Running docker inside LXC containers In my have proven to be no small task container, at least in my setup of a Gentoo host and Ubuntu LXC guest. One of the remaining issues is the cpuset.cpus error with Docker 1.4 and 1.5

/sys/fs/cgroup/cpu/lxc/cpuset/cpuset.cpus: no such file or directory

I have found a way to get around this issue.

Docker in its younger days was dependent on LXC to create its containers. That changed some time ago when it migrated to using its own code in the form of libcontainer. You could however still opt to use LXC. A change to the execution driver from native (libcontainer) to lxc and the cpuset.cpus error was gone. You can check your current execution driver by running docker info. native-0.2 is libcontainer:

# docker info
Containers: 22
Images: 3
Storage Driver: overlay
Backing Filesystem: extfs
Execution Driver: native-0.2
Kernel Version: 3.18.7-gentoo
Operating System: Gentoo/Linux (containerized)
CPUs: 4
Total Memory: 7.735 GiB

To change the execution driver, you can run the docker daemon with the “-e lxc” option, or add it to the config of your startup script. In Gentoo this was /etc/conf.d/docker. Change DOCKER_OPTS:

# any other random options you want to pass to docker
DOCKER_OPTS="-e lxc"

After the change, docker info should show LXC:

# docker info
Containers: 22
Images: 3
Storage Driver: overlay
Backing Filesystem: extfs
Execution Driver: lxc-1.0.7
Kernel Version: 3.18.7-gentoo
Operating System: Gentoo/Linux (containerized)
CPUs: 4
Total Memory: 7.735 GiB

However, after switching to LXC, I still had a few hurdles to jump through. First, there was a bug that made LXC look for sysRq when it doesn’t exist. This is a LXC bug. For now, I recompiled my kernel with CONFIG_MAGIC_SYSRQ=y. You can find the magic key option under “Kernel development” menu.

Next issue was to do with LXC failing to set devices.allow:

lxc-start: cgfs.c: do_setup_cgroup_limits: 1908 Error setting devices.allow to c 4:0 rwm for ****

To fix this, I added lxc.cgroup.devices.allow = c 4:* rwm to the LXC config of the parent container.

This was enough for my Gentoo LXC guest to run docker. For my Ubuntu LXC guest, I got this error:

lxc-cgmanager.c: lxc_cgmanager_escape: 329 call to cgmanager_move_pid_abs_sync(name=openrc) failed: invalid request

so I disabled CGManager:

initctl stop cgmanager
echo manual | sudo tee /etc/init/cgmanager.override

Now I can use all the latest docker features like overlayfs!