Running docker inside LXC containers In my have proven to be no small task container, at least in my setup of a Gentoo host and Ubuntu LXC guest. One of the remaining issues is the cpuset.cpus error with Docker 1.4 and 1.5
/sys/fs/cgroup/cpu/lxc/cpuset/cpuset.cpus: no such file or directory
I have found a way to get around this issue.
Docker in its younger days was dependent on LXC to create its containers. That changed some time ago when it migrated to using its own code in the form of libcontainer. You could however still opt to use LXC. A change to the execution driver from native (libcontainer) to lxc and the cpuset.cpus error was gone. You can check your current execution driver by running docker info. native-0.2 is libcontainer:
# docker info Containers: 22 Images: 3 Storage Driver: overlay Backing Filesystem: extfs Execution Driver: native-0.2 Kernel Version: 3.18.7-gentoo Operating System: Gentoo/Linux (containerized) CPUs: 4 Total Memory: 7.735 GiB
To change the execution driver, you can run the docker daemon with the “-e lxc” option, or add it to the config of your startup script. In Gentoo this was /etc/conf.d/docker. Change DOCKER_OPTS:
# any other random options you want to pass to docker DOCKER_OPTS="-e lxc"
After the change, docker info should show LXC:
# docker info Containers: 22 Images: 3 Storage Driver: overlay Backing Filesystem: extfs Execution Driver: lxc-1.0.7 Kernel Version: 3.18.7-gentoo Operating System: Gentoo/Linux (containerized) CPUs: 4 Total Memory: 7.735 GiB
However, after switching to LXC, I still had a few hurdles to jump through. First, there was a bug that made LXC look for sysRq when it doesn’t exist. This is a LXC bug. For now, I recompiled my kernel with CONFIG_MAGIC_SYSRQ=y. You can find the magic key option under “Kernel development” menu.
Next issue was to do with LXC failing to set devices.allow:
lxc-start: cgfs.c: do_setup_cgroup_limits: 1908 Error setting devices.allow to c 4:0 rwm for ****
To fix this, I added lxc.cgroup.devices.allow = c 4:* rwm to the LXC config of the parent container.
This was enough for my Gentoo LXC guest to run docker. For my Ubuntu LXC guest, I got this error:
lxc-cgmanager.c: lxc_cgmanager_escape: 329 call to cgmanager_move_pid_abs_sync(name=openrc) failed: invalid request
so I disabled CGManager:
initctl stop cgmanager echo manual | sudo tee /etc/init/cgmanager.override
Now I can use all the latest docker features like overlayfs!