While experimenting with LXC, the linux virtual container, which by the way is shaping up to be a viable replacement for openvz, I ran into an annoying issue of syslog-ng taking up 100% of CPU time inside the container. Stumped, I tried to add the -d flag to the syslog command line, but it did not yield any clues.

Armed with strace, and attaching to the rouge process, the following spat out of the console again and again.

gettimeofday({1287484365, 501293}, NULL) = 0
lseek(8, 0, SEEK_END)                   = -1 ESPIPE (Illegal seek)
write(8, "Oct 19 19:39:57 login[439"..., 105) = -1 EAGAIN (Resource temporarily unavailable)

The key lines were lseek and write, both trying to write to file descriptor 8. To find out what fd 8 was, all I had to do was ls -al /proc/7411/fd/8 – The culprit was /dev/tty12. Now having looked into syslog-ng.conf, I was reminded of the fact that By default messages are logged to tty12.... So it seems, tty12 is somehow denying access to syslog. Being in LXC, I decided to check out tty12 by doing lxc-console -n container -t 12. To my surprise, syslog-ng was instantly unclogged as log messages were released into console. It looked as if the tty12 buffer was clogged up.

Regardless of the reason, the easy fix is to stop syslog-ng logging to tty12 as I’m never going look at that far away console. Commenting the console_all lines, all was fixed. This would probably never have happened if I had used metalog :/