Docker in an LXC container on Gentoo

Docker is the newest craze in the devops world. It’s a tool that assists with application containerization using Linux Container technology. I decided to give it a try, but do it with a twist: I want to run docker inside a LXC container, essentially, run docker containers inside LXC containers. This inception style setup has a few benefits – It allows docker and its dependencies to be contained, isolated from the host machine. It also allows testing of different docker versions on different containers. It my case, I want to run docker under Ubuntu 14.04, without reformatting my entire Gentoo host.

Preparing the host

The host machine is running Gentoo. I installed lxc-1.0.6-r1. Docker allows several storage backends. The most popular is AUFS, which allows copy-on-write semantics, making it super-fast in provisioning new docker containers. AUFS isn’t natively supported in the kernel, so patches and kernel recompilation are required. I used aufs-sources which is basically gentoo-sources plus the AUFS patches. If you are copying the .config from your last kernel compilation, don’t forget to run make oldconfig to select the AUFS filesystem option. I picked version 3.16.7 as it was the closest to my current running kernel.

Ubuntu LXC container

I picked Ubuntu (14.04) initially because it seems to be the best supported distro for docker and related tools (like Dokku). I initialised an Ubuntu LXC container on the Gentoo host with the following command which download and installed a virtualised Ubuntu creatively called ‘ubun’:

# lxc-create -t download -n ubun -- -d ubuntu -r trusty -a amd64

Recent LXC versions allow you to expose cgroup inside a container as long as you are running a privileged container (i.e. root). This is necessary for launching nested containers.  Add this line to your LXC config:

lxc.mount.auto = cgroup

If you have apparmor (i.e. Ubuntu host) then you also need to relax app armor’s restrictions with this line:

lxc.aa_profile = lxc-container-default-with-nesting

The container config also needs networking support to be added.

# Network configuration
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = lan
lxc.network.ipv4 = 192.168.0.100/24
lxc.network.hwaddr = aa:bb:cc:dd:ee:01
lxc.network.name = eth0

Then I started the container. At some point during the startup, the container froze for a minute or so. The network also came up as DHCP instead of the statically configured IP. It turns out I have to remove the auto eth0 and iface eth0 lines from /etc/network/interfaces to allow the LXC configuration to take effect.

After that, I installed docker (1.4.1) according to their instructions and ran:

 # sudo docker run -i -t ubuntu /bin/bash

First, docker spent ten minutes downloading the 200 MB Ubuntu docker image, and after that, when it tried to start the instance, it resulted in this error:

FATA[0000] Error response from daemon: Cannot start container 4e871cc02faee6efe7fafd4e4ee172f76bdfe817093fa9ad594f79eb6f8ab320: open /sys/fs/cgroup/cpu/lxc/devices/lxc/ubun/docker/4e871cc02faee6efe7fafd4e4ee172f76bdfe817093fa9ad594f79eb6f8ab320/devices.deny: permission denied

Unfortunately this where I got stuck. Google turned up nothing useful, but I would put it down to the fact that this is running under a Gentoo kernel instead of the Ubuntu 14.04 kernel.

Update: I solved this issue later by commenting out all cgroup mounting operations inside the container. See updated post Docker running on Ubuntu LXC on Gentoo host.

The next best thing would be to run a Gentoo container with docker inside the Gentoo host.

Gentoo LXC container – Success! (somewhat)

So I created a Gentoo LXC container with:

 # lxc-create -t download -n g2 -- -d gentoo -r current -a amd64

and emerge’d docker expecting a somewhat smoother experience. Unfortunately I was in no such luck. Running the same command to start a Ubuntu docker instance, a different error occurred.

FATA[0000] Error response from daemon: Cannot start container 297a6d29a18fcd802d1d20035518f4cd6b076e98e2fed803cd3a54a63d1524df: open /sys/fs/cgroup/cpu/lxc/cpuset/cpuset.cpus: no such file or directory

Update: Workaround to the cpuset.cpus issue – Use the LXC execution driver

It seems like other users have experienced the same issue. A tip from the issue thread suggests it’s a bug in Docker 1.4. After downgrading to 1.3.3, the docker container ran:

 # docker run -i -t ubuntu /bin/bash
root@01ffd4694656:/#

Finally, I can get started.

Leave a Reply

Your email address will not be published. Required fields are marked *