Docker under Ubuntu 16.04, set up for direct-lvm storage
Recently i wanted to install docker on my ubuntu server the right way and use direct-lvm as storage layer for the devicemapper. For production environments this is the preferred method. Unfortunately it was not working out of the box, which is why i am writing this and maybe prevent some users from getting as much headaches as i was getting.
Preparing LVM
First step is to make some space in our volume group or even make a dedicated one for docker. If you want to do so, you have to create a new partition. You may have many partitions inside one volume group but you cannot have multiple volume groups on one partition.
I am not a fan of partitions, so i choose to use my existing volume group and reduce the the size of my logical volume in order to have some space for the docker's storage.
The command vgdisplay shows my existing volume groups and its properties. Interesting for us is the VG Size, which shows the overall volume group size, and the Alloc PE / Size, which shows the allocated size by existing logical volumes.
# vgdisplay --- Volume group --- VG Name ubuntu-vg System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 3 VG Access read/write VG Status resizable MAX LV 0 Cur LV 2 Open LV 2 Max PV 0 Cur PV 1 Act PV 1 VG Size 36,80 GiB PE Size 4,00 MiB Total PE 9422 Alloc PE / Size 9416 / 36,78 GiB Free PE / Size 6 / 24,00 MiB VG UUID e5LZI7-Uo19-9LBg-3Mcj-YgXm-traf-bGdgS8
In this example of my virtual host nearly all allocatable size is allocated already, so next step is to reduce the logical volume's size so there is enough space to build the docker's storage withing our existing volume group.
We cannot resize mounted logical volumes and since our root filesystem is on the only logical volume we have to boot into a rescue environment and perform our changes from there.
After booting into the rescue system i can reduce my local volume as follows:
#$ lvreduce --resizefs -L 20G /dev/ubuntu-vg/root fsck von util-linux 2.27.1 /dev/mapper/ubuntu--vg-root: 226401/2281104 Dateien (0.1% nicht zusammenhängend), 1417926/9118720 Blöcke resize2fs 1.42.13 (17-May-2015) Die Größe des Dateisystems auf /dev/mapper/ubuntu--vg-root wird auf 5242880 (4k) Blöcke geändert. Das Dateisystem auf /dev/mapper/ubuntu--vg-root is nun 5242880 (4k) Blöcke lang. Size of logical volume ubuntu-vg/root changed from 34,79 GiB (8905 extents) to 20,00 GiB (5120 extents). Logical volume root successfully resized.
I decided to reduce my root filesystem to 20G, which leaves me about 15G of free space for my docker storage. The --resizefs automatically resizes the underlying filesystem to its new size, so we don't have to do this manually. To figure out, which logical volume you have to resize, use the lvdisplay command. It shows all available logical volumes within a volume group:
#$ lvdisplay ubuntu-vg --- Logical volume --- LV Path /dev/ubuntu-vg/root LV Name root VG Name ubuntu-vg LV UUID 4KUUhN-PHJ2-5Q26-82Du-os3m-HusW-uKE2eI LV Write Access read/write LV Creation host, time ubuntu, 2016-06-30 08:18:44 +0000 LV Status available # open 0 LV Size 20,00 GiB Current LE 5120 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 252:0 --- Logical volume --- LV Path /dev/ubuntu-vg/swap_1 LV Name swap_1 VG Name ubuntu-vg LV UUID s4o2Jm-F0Sz-SysU-x8f7-eJaX-RRoJ-7piRMB LV Write Access read/write LV Creation host, time ubuntu, 2016-06-30 08:18:44 +0000 LV Status available # open 0 LV Size 2,00 GiB Current LE 511 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 252:1
With this we have successfully prepared our volume group ubuntu-vg to hold our new docker storage.
Install docker
Next we have to install docker. Since docker is supported very well by most distributions we have a recent version in the ubuntu repository. So all we have to do is
#$ sudo apt-get install docker.io ...
After successful installation we can see the docker daemon is up and running:
sudo service docker status
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Di 2016-07-12 15:25:26 CEST; 21s ago
Docs: https://docs.docker.com
Main PID: 4993 (docker)
CGroup: /system.slice/docker.service
└─4993 /usr/bin/docker daemon -H fd://
Jul 12 15:25:26 docker-lvm docker[4993]: time="2016-07-12T15:25:26.400038358+02:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Jul 12 15:25:26 docker-lvm docker[4993]: time="2016-07-12T15:25:26.415527884+02:00" level=info msg="Firewalld running: false"
Jul 12 15:25:26 docker-lvm docker[4993]: time="2016-07-12T15:25:26.457709818+02:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a
Jul 12 15:25:26 docker-lvm docker[4993]: time="2016-07-12T15:25:26.519826610+02:00" level=warning msg="Your kernel does not support swap memory limit."
Jul 12 15:25:26 docker-lvm docker[4993]: time="2016-07-12T15:25:26.560283918+02:00" level=info msg="Loading containers: start."
Jul 12 15:25:26 docker-lvm docker[4993]: time="2016-07-12T15:25:26.560353891+02:00" level=info msg="Loading containers: done."
Jul 12 15:25:26 docker-lvm docker[4993]: time="2016-07-12T15:25:26.560365431+02:00" level=info msg="Daemon has completed initialization"
Jul 12 15:25:26 docker-lvm docker[4993]: time="2016-07-12T15:25:26.560382383+02:00" level=info msg="Docker daemon" commit=20f81dd execdriver=native-0.2 graphdriver=aufs version=1.10.3
Jul 12 15:25:26 docker-lvm systemd[1]: Started Docker Application Container Engine.
Jul 12 15:25:26 docker-lvm docker[4993]: time="2016-07-12T15:25:26.587885630+02:00" level=info msg="API listen on /var/run/docker.sock"
For convenience i put my user into the docker group so i don't have to sudo all the docker commands:
#$ sudo usermod -aG docker <username>
after log out - log in my user is able to perform docker commands since it has access to docker's communication socket file:
#$ docker info Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 0 Server Version: 1.10.3 Storage Driver: aufs Root Dir: /var/lib/docker/aufs Backing Filesystem: extfs Dirs: 0 Dirperm1 Supported: true Execution Driver: native-0.2 Logging Driver: json-file Plugins: Volume: local Network: null host bridge Kernel Version: 4.4.0-28-generic Operating System: Ubuntu 16.04 LTS OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 3.859 GiB Name: docker-lvm ID: HQAE:HBS4:6RIR:7T4F:PTRU:3JBZ:ZVDM:T4LH:H24F:XUVW:VL22:SHKT WARNING: No swap limit support
What we can see now is that docker is using aufs as storage driver, which seems to be fine in development environments but also lacks performance, which is why we want to use direct-lvm. To do so we follow the instructions from the docker's website.
configuring logical volume for direct-lvm
Since we already have a volume group we don't need to create one. therefore we skip steps 1,2, 3 and 4 and resume creating a logical volume named thinpool. Since we already have a logical volume in our volume group we obviously cannot use 95% of the volume group's size for our thinpool. Instead for the thinpool itself we take 95% of whats left and for the thinpool metadata 25% of whats left of whats left (:
#$ sudo lvcreate --wipesignatures y -n thinpool ubuntu-vg -l 95%FREE Logical volume "thinpool" created. #$ sudo lvcreate --wipesignatures y -n thinpoolmeta ubuntu-vg -l 25%FREE Logical volume "thinpoolmeta" created.
Next we convert our pool into a thinpool:
#$ sudo lvconvert -y --zero n -c 512K --thinpool ubuntu-vg/thinpool --poolmetadata ubuntu-vg/thinpoolmeta WARNING: Converting logical volume ubuntu-vg/thinpool and ubuntu-vg/thinpoolmeta to pool's data and metadata volumes. THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.) Converted ubuntu-vg/thinpool to thin pool.
Next we configure autoextension for the thinpool:
#$ sudo mkdir -p /etc/lvm/profile
#$ sudo tee /etc/lvm/profile/docker-thinpool.profile <<EOF
activation {
thin_pool_autoextend_threshold=80
thin_pool_autoextend_percent=20
}
EOF
afterwards we enable the profile:
#$ sudo lvchange --metadataprofile docker-thinpool ubuntu-vg/thinpool
Logical volume "thinpool" changed.
To verify that the thinpool is monitored, do:
$# sudo lvs -o+seg_monitor
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Monitor
root ubuntu-vg -wi-ao---- 20,00g
swap_1 ubuntu-vg -wi-ao---- 2,00g
thinpool ubuntu-vg twi-a-t--- 14,07g 0,00 0,02 monitored
All we have to do now is to tell the docker-daemon to use this storage for its images and containers. Since Ubuntu 16.04 is now using systemd the /etc/default/docker file is not applying anymore. The easiest way to configure the new storage is to create a file named /etc/docker/daemon.json:
#$ sudo tee /etc/docker/daemon.json <<EOF
{
"storage-driver": "devicemapper",
"storage-opts": [
"dm.thinpooldev=/dev/mapper/ubuntu--vg-thinpool",
"dm.use_deferred_removal=true"
]
}
EOF
Now we stop the docker-daemon, remove the folder /var/lib/docker and start the daemon again:
$# sudo systemctl stop docker
$# sudo rm -rf /var/lib/docker
$# sudo systemctl start docker
Congratulations! Now we have an up and running docker-daemon with a direct-lvm storage layer.
The headache
After all this i was very upset, that after reboot the docker-daemon was not coming up.
#$ sudo systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Di 2016-07-12 16:22:33 CEST; 1min 30s ago
Docs: https://docs.docker.com
Process: 1880 ExecStart=/usr/bin/docker daemon -H fd:// $DOCKER_OPTS (code=exited, status=1/FAILURE)
Main PID: 1880 (code=exited, status=1/FAILURE)
Jul 12 16:22:33 docker-lvm systemd[1]: Starting Docker Application Container Engine...
Jul 12 16:22:33 docker-lvm docker[1880]: time="2016-07-12T16:22:33.304838820+02:00" level=fatal msg="Error starting daemon: error initializing graphdriver: devicemapper: Non existing device ubuntu--vg-thinpool"
Jul 12 16:22:33 docker-lvm systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Jul 12 16:22:33 docker-lvm systemd[1]: Failed to start Docker Application Container Engine.
Jul 12 16:22:33 docker-lvm systemd[1]: docker.service: Unit entered failed state.
Jul 12 16:22:33 docker-lvm systemd[1]: docker.service: Failed with result 'exit-code'.
Jul 12 16:22:33 docker-lvm systemd[1]: docker.service: Start request repeated too quickly.
Jul 12 16:22:33 docker-lvm systemd[1]: Failed to start Docker Application Container Engine.
The status messages indicated that my thinpool was not available. lvdisplay said the same:
#$ sudo lvdisplay ubuntu-vg/thinpool
--- Logical volume ---
LV Name thinpool
VG Name ubuntu-vg
LV UUID Or2cLx-ENFq-sZqh-kxUw-0FEQ-rEzi-vXCi3C
LV Write Access read/write
LV Creation host, time docker-lvm, 2016-07-12 15:57:06 +0200
LV Pool metadata thinpool_tmeta
LV Pool data thinpool_tdata
LV Status NOT available
LV Size 14,07 GiB
Current LE 3601
Segments 1
Allocation inherit
Read ahead sectors auto
LV Status NOT available
So i tried to activate it:
#$ sudo lvchange -ay ubuntu-vg/thinpool
/usr/sbin/thin_check: execvp failed: Datei oder Verzeichnis nicht gefunden
Check of pool ubuntu-vg/thinpool failed (status:2). Manual repair required!
After some research i found that the tools for check and repair are configured in /etc/lvm/lvm.conf and there i found the following paragraph:
# Configuration option global/thin_check_executable.
# The full path to the thin_check command.
# LVM uses this command to check that a thin metadata device is in a
# usable state. When a thin pool is activated and after it is
# deactivated, this command is run. Activation will only proceed if
# the command has an exit status of 0. Set to "" to skip this check.
# (Not recommended.) Also see thin_check_options.
# (See package device-mapper-persistent-data or thin-provisioning-tools)
# This configuration option has an automatic default value.
# thin_check_executable = "/usr/sbin/thin_check"
So i looked for the packages device-mapper-persistent-data and thin-provisioning-tools and i found that there is a package thin-provisioning-tools, which is not installed:
#$ sudo apt-cache policy thin-provisioning-tools
thin-provisioning-tools:
Installiert: (keine)
Installationskandidat: 0.5.6-1ubuntu1
Versionstabelle:
0.5.6-1ubuntu1 500
500 http://de.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
After installing it i could finally activate my docker storage again:
#$ sudo lvchange -ay ubuntu-vg/thinpool
And also after reboot the docker's logical volume was activated and the docker-daemon was started successfully.