Running docker inside LXC containers In my have proven to be no small task container, at least in my setup of a Gentoo host and Ubuntu LXC guest. One of the remaining issues is the cpuset.cpus error with Docker 1.4 and 1.5
/sys/fs/cgroup/cpu/lxc/cpuset/cpuset.cpus: no such file or directory
I have found a way to get around this issue.
I have a Linux box that runs both LXC and VirtualBox side by side. All the containers and VMs are set to bridge mode. Unfortunately for some reason, the VirtualBox VMs can’t talk to LXC containers. Pinging yields a response, but any TCP connection fails. Both LXC and VBox can talk to the host fine. This is how you can resolve the issue.
A few days ago I attempted to build a Ubuntu LXC container to run Docker on a Gentoo host. It ended up being a failure. I was getting “devices.deny: permission denied” errors. However, not willing to give up, I kept looking at the issue and finally solved it!
Docker is the newest craze in the devops world. It’s a tool that assists with application containerization using Linux Container technology. I decided to give it a try, but do it with a twist: I want to run docker inside a LXC container, essentially, run docker containers inside LXC containers. This inception style setup has a few benefits – It allows docker and its dependencies to be contained, isolated from the host machine. It also allows testing of different docker versions on different containers. It my case, I want to run docker under Ubuntu 14.04, without reformatting my entire Gentoo host.
Over the weekend I installed Avahi (the open source bonjour equivalent) and bumped into a strange error while trying to restart the service. /var/log/message says chroot.c: fork() failed: Resource temporarily unavailable. Searching the interwebs revealed it is an issue with LXC and setrlimit.
The setrlimit call can limit set cetain limitations on processes. One such limitation is NPROC, the number of processes that can have the same UID. Using setrlimit NPROC can enhance security by preventing unexpected forking, like when an attacker is trying spawn a new process. However, the server I am running on uses LXC, and avahi is installed on the host. In LXC, the containers themselves have isolation between one another, but the host sees all processes. The PIDs of container processes are remapped but their UIDs stay the same. Thus, you will get UID collisions where user 102 of container can refer to say ntp, while 102 of host can refer to avahi. Because the host sees and accounts for all processes, setrlimit on avahi (102) of say 3 processes will also count existing processes in containers with UID 102 (such as ntp) and thus breach the limit and unable to spawn.
The only way to solve this is to edit avahi.conf and set rlimit-nproc or just disable rlimits altogether using the --no-rlimits switch.
I guess as LXC and control groups becomes more common, developers will need to adjust their assumptions about users and processes.
While experimenting with LXC, the linux virtual container, which by the way is shaping up to be a viable replacement for openvz, I ran into an annoying issue of syslog-ng taking up 100% of CPU time inside the container. Stumped, I tried to add the -d flag to the syslog command line, but it did not yield any clues.
Armed with strace, and attaching to the rouge process, the following spat out of the console again and again.
gettimeofday({1287484365, 501293}, NULL) = 0
lseek(8, 0, SEEK_END) = -1 ESPIPE (Illegal seek)
write(8, "Oct 19 19:39:57 login[439"..., 105) = -1 EAGAIN (Resource temporarily unavailable)
The key lines were lseek and write, both trying to write to file descriptor 8. To find out what fd 8 was, all I had to do was ls -al /proc/7411/fd/8 – The culprit was /dev/tty12. Now having looked into syslog-ng.conf, I was reminded of the fact that By default messages are logged to tty12.... So it seems, tty12 is somehow denying access to syslog. Being in LXC, I decided to check out tty12 by doing lxc-console -n container -t 12. To my surprise, syslog-ng was instantly unclogged as log messages were released into console. It looked as if the tty12 buffer was clogged up.
Regardless of the reason, the easy fix is to stop syslog-ng logging to tty12 as I’m never going look at that far away console. Commenting the console_all lines, all was fixed. This would probably never have happened if I had used metalog :/