LXD, ZFS and bridged networking on Ubuntu 16.04 LTS+

LXD works perfectly fine with a directory-based storage backend, but both speed and reliability are greatly improved when ZFS is used instead. 16.04 LTS saw the first officially supported release of ZFS for Ubuntu and having just set up a fresh LXD host on Elastichosts utilising both ZFS and bridged networking, I figured it’d be a good time to document it.

In this article I’ll walk through the installation of LXD, ZFS and Bridge-Utils on Ubuntu 16.04 and configure LXD to use either a physical ZFS partition or loopback device combined with a bridged networking setup allowing for containers to pick up IP addresses via DHCP on the (v)LAN rather than a private subnet.

To get started, let’s install our packages. They can all be installed with one command as follows:

sudo apt-get install lxd zfsutils-linux bridge-utils

However for this I will output the commands and the result for each package individually:

sudo apt-get install lxd

Reading package lists... Done
Building dependency tree       
Reading state information... Done
lxd is already the newest version (2.0.0-0ubuntu4).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

sudo apt-get install zfsutils-linux

Reading package lists... Done
Building dependency tree       
Reading state information... Done
[...]
The following NEW packages will be installed:
  libnvpair1linux libuutil1linux libzfs2linux libzpool2linux zfs-doc zfs-zed
  zfsutils-linux
0 upgraded, 7 newly installed, 0 to remove and 16 not upgraded.
Need to get 884 kB of archives.
[...]
Setting up zfs-doc (0.6.5.6-0ubuntu8) ...
Setting up libuutil1linux (0.6.5.6-0ubuntu8) ...
Setting up libnvpair1linux (0.6.5.6-0ubuntu8) ...
Setting up libzpool2linux (0.6.5.6-0ubuntu8) ...
Setting up libzfs2linux (0.6.5.6-0ubuntu8) ...
Setting up zfsutils-linux (0.6.5.6-0ubuntu8) ...
[...]
Setting up zfs-zed (0.6.5.6-0ubuntu8) ...
zed.service is a disabled or a static unit, not starting it.
Processing triggers for libc-bin (2.23-0ubuntu3) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for systemd (229-4ubuntu4) ...

sudo apt-get install bridge-utils

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  bridge-utils
0 upgraded, 1 newly installed, 0 to remove and 16 not upgraded.
Need to get 28.6 kB of archives.
[...]
Preparing to unpack .../bridge-utils_1.5-9ubuntu1_amd64.deb ...
Unpacking bridge-utils (1.5-9ubuntu1) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up bridge-utils (1.5-9ubuntu1) ...

You’ll notice I’ve installed LXD, ZFS and bridge utils. LXD should be installed by default on any 16.04 host as is shown by the output above, however should there be any updates this will bring them down before we begin.

ZFS and bridge utils are not installed by default; ZFS needs to be installed to run our storage backend and bridge utils is required in order for our bridged interface to work.

With the relevant packages installed, we can now move on to configuration. We’ll start by configuring the bridge as before this is complete we won’t be able to obtain DHCP addresses for containers within LXD.

We’ll begin by opening /etc/network/interfaces in a text editor. I like vim:

sudo vim /etc/network/interfaces

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet dhcp

This is the default interfaces file. What we’ll do here is add a new bridge named br0. The simplest edit to make to this file is as follows:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto br0
iface br0 inet dhcp
	bridge_ports eth0

iface eth0 inet manual

This will set the eth0 interface to manual and create a new bridge that piggybacks directly off it.
If you wish to create a static interface while you’re editing this file, the following may help you:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto br0 
iface br0 inet static
        address 192.168.0.44
        netmask 255.255.255.0
        network 192.168.0.0
        broadcast 192.168.0.255
        gateway 192.168.0.1
        # dns-* options are implemented by the resolvconf package, if installed
        dns-nameservers 8.8.8.8 8.8.4.4 #google-dns
        dns-search localdomain.local #optional-line
        # bridge options
        bridge_ports eth0
iface eth0 inet manual

Following any edits, it’s a good idea to restart the interfaces to force the changes to take place. Obviously if you’re connected via SSH this will disconnect your session. You’ll need to have physical access to the machine/VM.

sudo ifdown eth0 && sudo ifup eth0 && sudo ifup br0

We’ll begin by opening /etc/netplan/01-netcfg.yaml in a text editor. I like vim:

# This file is generated from information provided by
# the datasource. Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
 version: 2
 renderer: networkd
 ethernets:
   eth0:
     dhcp4: false
 bridges:
   br0:
     interfaces: [eth0]
     dhcp4: false
     addresses: [192.168.1.99/24]
     gateway4: 192.168.1.1
     nameservers:
       addresses: [1.1.1.1,8.8.8.8]
     parameters:
       forward-delay: 0

All of the above bolded lines have been added/modified for a static IP bridge. Edit to suit your environment and then run the following to apply changes:

sudo netplan apply

Running ifconfig on the CLI will now confirm the changes have been applied:

jason@ubuntu-lxdzfs:~$ ifconfig
<b>br0</b>       Link encap:Ethernet  HWaddr 00:0c:29:2f:cd:30  
          inet addr:192.168.0.44  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe2f:cd30/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2235187 errors:0 dropped:37359 overruns:0 frame:0
          TX packets:111487 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:2108302272 (2.1 GB)  TX bytes:8296995 (8.2 MB)

eth0      Link encap:Ethernet  HWaddr 00:0c:29:2f:cd:30  
          inet6 addr: fe80::20c:29ff:fe2f:cd30/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:161568870 errors:0 dropped:0 overruns:0 frame:0
          TX packets:132702 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:185204051222 (185.2 GB)  TX bytes:23974566 (23.9 MB)

[...]

With the bridge up and running we can now begin to configure LXD. Before we start setting up containers, LXD requests we run sudo lxd init to configure the package. As part of this, we’ll be selecting our newly created bridge for network connectivity and configuring ZFS as LXD will take care of both during setup.

For this guide I’ll be using a dedicated hard drive for the ZFS storage backend, though the same procedure can be used for a dedicated partition if you don’t have a spare drive handy. For those wishing to use a loopback file for testing, the procedure is slightly different and will be addressed below.

First we’ll run sudo fdisk -l to list the available disks & partitions on the server, here’s a relevant snippet of the output I get:

Disk /dev/sda: 20 GiB, 21474836480 bytes, 41943040 sectors
[...]

Device     Boot    Start      End  Sectors  Size Id Type
/dev/sda1  *        2048 36132863 36130816 17.2G 83 Linux
/dev/sda2       36134910 41940991  5806082  2.8G  5 Extended
/dev/sda5       36134912 41940991  5806080  2.8G 82 Linux swap / Solaris


Disk /dev/sdb: 20 GiB, 21474836480 bytes, 41943040 sectors
[...]

Device     Boot    Start      End  Sectors  Size Id Type
/dev/sdb1           2048 41940991 41938944  20G  83 Linux

Make a note of the partition or drive to be used. In this example we’ll use partition sdb1 on disk /dev/sdb

Start the configuration of LXD by running sudo lxd init

jason@ubuntu-lxd-tut:~$ sudo lxd init
Name of the storage backend to use (dir or zfs): zfs
Create a new ZFS pool (yes/no)? yes
Name of the new ZFS pool: lxd
Would you like to use an existing block device (yes/no)? yes
Path to the existing block device: /dev/sdb1
Would you like LXD to be available over the network (yes/no)? no
Do you want to configure the LXD bridge (yes/no)? yes
Warning: Stopping lxd.service, but it can still be activated by:
  lxd.socket
LXD has been successfully configured.

Let’s break the above options down:

Name of the storage backend to use (dir or zfs): zfs

Here we’re defining ZFS as our storage backend of choice. The other option, DIR, is a flat-file storage option that places all containers on the host filesystem under /var/lib/lxd/containers/ (though the ZFS partition is transparently mounted under the same path and so accessed equally as easily). It doesn’t benefit from features such as compression and copy-on-write however, so the performance of the containers using the DIR backend simply won’t be as good.

Create a new ZFS pool (yes/no)? yes
Name of the new ZFS pool: lxd

Here we’re creating a brand new ZFS pool for LXD and giving it the name of “lxd”. We could also choose to use an existing pool if one were to exist, though as we left ZFS unconfigured it does not apply here.

Would you like to use an existing block device (yes/no)? yes
Path to the existing block device: /dev/sdb1

Here we’re opting to use a physical partition rather than a loopback device, then providing the physical location of said partition.

Would you like LXD to be available over the network (yes/no)? no

It’s possible to connect to LXD from other LXD servers or via the API from a browser (see https://linuxcontainers.org/lxd/try-it/ for an example of this).

As this is a simple installation we won’t be utilising this functionality and it is as such left unconfigured. Should we wish to enable it at a later date, we can run:

lxc config set core.https_address [::]
lxc config set core.trust_password some-secret-string

Where some-secret-string is a secure password that’ll be required by other LXD servers wishing to connect in order to admin the LXD host or retried non-public published images.

Do you want to configure the LXD bridge (yes/no)? yes

Here we tell LXD to use our already-preconfigured bridge. This opens a new workflow as follows:

We don’t want LXD to create a new bridge for us, so we’ll select no here.

LXD now knows we may have our own bridge already set up, so we’ll select yes in order to declare it.

Finally we’ll input the bridge name and select OK. LXD will now use this bridge.

And with that, LXD will finish configuration and ready itself for use.

In version 2.5, the above purple bridge workflow has been retired in favour of the new lxc network command.

With lxd init complete above, add the br0 interface to the default profile with:

lxc network attach-profile br0 default eth0

If by accident the lxdbr0 interface was configured, it must be first detached from the default profile with:

lxc network detach-profile lxdbr0 default eth0

It’ll be obvious if this needs to be done as running lxc network attach-profile br0 default eth0 will result in the error error: device already exists.

With that complete, LXD will now successfully use the pre-configured bridge.

Run sudo lxd init as above, but use the following options instead.

Name of the storage backend to use (dir or zfs): zfs
Create a new ZFS pool (yes/no)? yes
Name of the new ZFS pool: lxd-loop
Would you like to use an existing block device (yes/no)? no
Size in GB of the new loop device (1GB minimum): 20

The size in GB of the ZFS partition is important, we don’t want to run out of space any time soon. Although ZFS partitions may be resized, it’s better to be a little generous now and not have to worry about reconfiguring it later.

Since it’s entirely possible we may in the future wish to run multiple LXD containers, it’s a good idea to already increase the number of open files and inode limits, this will prevent the dreaded “too many open files” errors which commonly occur with container solutions.

For the inode limits, open the sysctl.conf file as follows:

sudo vim /etc/sysctl.conf

Now add the following lines, as recommended by the LXD project

fs.inotify.max_queued_events = 1048576
fs.inotify.max_user_instances = 1048576
fs.inotify.max_user_watches = 1048576

It should look as follows:

After saving the file we’ll need to reboot, but not yet as we’ll also configure the open file limits.

Open the limits.conf file as follows:

sudo vim /etc/security/limits.conf

Now add the following lines. 100K should be enough:

* soft nofile 100000
* hard nofile 100000

It should look as follows:

Once the server is rebooted (this is important!) the new limits will apply and we’ll have future-proofed the server for now.

sudo reboot

With our bridge set up, our ZFS storage backend created and LXD fully configured, it’s time to test everything is working as it should be.

We’ll first get a quick overview of our ZFS storage pool using sudo zpool list lxd

jason@ubuntu-lxd-tut:~$ sudo zpool list lxd
NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
lxd   19.9G   646M  19.2G         -     2%     3%  1.00x  ONLINE  -

With ZFS looking fine, we’ll run a simple lxc info to generate our client certificate and verify the configuration we’ve chosen for LXD:

jason@ubuntu-lxd-tut:~$ lxc info
Generating a client certificate. This may take a minute...

apicompat: 0
auth: trusted
environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    [...]
    -----END CERTIFICATE-----
  driver: lxc
  driverversion: 2.0.0
  kernel: Linux
  kernelarchitecture: x86_64
  kernelversion: 4.4.0-21-generic
  server: lxd
  serverpid: 5135
  serverversion: 2.0.0
  storage: zfs
  storageversion: "5"
config:
  storage.zfs_pool_name: lxd
public: false

It would appear the storage backend is correctly using our ZFS pool: “lxd”. If we now take a look at the default profile using:

lxc profile show default

We should see LXD using br0 as the default container eth0 interface:

jason@ubuntu-lxd-tut:~$ lxc profile show default
name: default
config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic

Success! The only thing left to do now is launch a container.

We can use the official Ubuntu image repo and spin up a Xenial container with the alias xen1 using the command:

lxc launch ubuntu:xenial xen1

Which should return an output like this:

jason@ubuntu-lxd-tut:~$ lxc launch ubuntu:xenial xen1
Creating xen1
Retrieving image: 100%
Starting xen1

Now, we can use lxc list to get an overview of all containers including their IP addresses:

jason@ubuntu-lxd-tut:~$ lxc list
+------+---------+----------------------+------+------------+-----------+
| NAME |  STATE  |        IPV4          | IPV6 |    TYPE    | SNAPSHOTS |
+------+---------+----------------------+------+------------+-----------+
| xen1 | RUNNING | 192.168.0.197 (eth0) |      | PERSISTENT | 0         |
+------+---------+----------------------+------+------------+-----------+

We can see the xen1 container has picked up an IP from our DHCP server on the LAN, which is exactly what we want.

Finally, we can use lxc exec xen1 bash to gain CLI access to the container we’ve just launched:

jason@ubuntu-lxd-tut:~$ lxc exec xen1 bash
root@xen1:~# 

While a little long-winded, setting up LXD with a ZFS storage backend and utilising a bridged interface for connecting containers directly to the LAN isn’t overly difficult, and it’s only gotten easier as LXD has matured to version 2.0.

Are you brand new to LXD? I thoroughly recommend you take a look at LXD developer Stéphane Graber’s incredible LXD blog series to get up to speed.