Waiting for a planned maintenance window is so boring… So I am basically holding my breath and keep my fingers crossed for a DBA to not wreak havoc, cause general breakage across our environment and hoping for never getting the cue to start cleaning things up.

I had the strangest problems with a LAMP-server running a WordPress blog the other week. The only error messages from /var/log/httpd/error_log I found that seemed relevant was:

[error]  End of file found: apr_() file_read_full() failed

After some googling I found that file_read_full() came from mod_sflow and after that I found the issue quoted below in their bug-tracker.

Finally got to the bottom of this. I hadn’t realized that there was a difference between a SIGHUP restart and a graceful(SIGUSR1) restart. Unlike other daemons the log-rotation for apache typically does a graceful (SIGUSR1) restart, and this is the one that was being handled badly by mod-sflow 1.0.3 — although the error was only exposed if the server was relatively busy at the time.

https://code.google.com/p/mod-sflow/issues/detail?id=4

In our prepackaged logrotatate-script from the CentOS rpm Apache got reloaded, not restarted when logrotated.

 /var/log/httpd/*log {
    missingok
    notifempty
    sharedscripts
    delaycompress
    postrotate
        /sbin/service httpd reload > /dev/null 2>/dev/null || true
    endscript
}

I added copytruncate and changed reload to restart in the logrotate script and we’ll see if the problem still occurs.

References:

• https://code.google.com/p/mod-sflow/issues/detail?id=4
• https://code.google.com/p/mod-sflow/source/browse/trunk/mod_sflow.c

Two videos from Stockholm Ruby Usergroup’s meetup in November.

• http://danielnystrom.se/post/35141650112
• http://danielnystrom.se/post/35141603070

One of the best books I have read this year is Thomas A. Limoncelli’s Time Management for System Administrators.

It is somewhat dated in places (with references to PDA’s which mostly been replaced by mobile phones and other devices) but the general lessons are good and sound. I would recommend it to every one who struggles with juggling incidents, projects and customer obiigations as a System Administrator.

Yesterday I went to a meetup hosted at Spotify.

The talks where:

• Security Loves DevOps by James Turnbull from Puppetlabs
• Puppet Labs: The State of the Nation by James Turnbull from Puppetlabs
• Test Automation at Spotify by Kristian Karl from Spotify

The talk on bringing Security together with Development and Operations resonates well with the whole DevOps culture and I can’t get enough talks on the subject.

It was interesting to here from Puppetlabs and James about the somewhat strange design choices they done earlier on and nice to hear about the fixes planned. I also like their focus on transparency and investments in a more engaging community. I look forward to see what Puppetlabs planned for the next couple of years.

The last talk about testing at Spotify was educational, and as I am a operations guy and do not worry much about end users experience in my day to day work, it put things into perspective. We all have to mind our customers! It was nice of Spotify to show us some of their techniques for testing and educational to see a live demo off the login test.

All in all it was a great evening and it’s always nice to get some nice beers and some food for free. Big thanks to the Devops meetup group!