Lennart Poettering: Walkthrough for Portable Services

Portable Services with systemd v239


contains a great number of new features. One of them is first class
support for Portable
. In
this blog story I’d like to shed some light on what they are and why
they might be interesting for your application.

What are “Portable Services”?

The “Portable Service” concept takes inspiration from classic
chroot() environments as well as container management and brings a
number of their features to more regular system service management.

While the definition of what a “container” really is is hotly debated,
I figure people can generally agree that the “container” concept
primarily provides two major features:

  1. Resource bundling: a container generally brings its own file system
    tree along, bundling any shared libraries and other resources it
    might need along with the main service executables.

  2. Isolation and sand-boxing: a container operates in a name-spaced
    environment that is relatively detached from the host. Besides
    living in its own file system namespace it usually also has its own
    user database, process tree and so on. Access from the container to
    the host is limited with various security technologies.

Of these two concepts the first one is also what traditional UNIX
chroot() environments are about.

Both resource bundling and isolation/sand-boxing are concepts systemd
has implemented to varying degrees for a longer time. Specifically,
have been around for a long time, and so have been the various

systemd provides. The Portable Services concept builds on that,
putting these features together in a new, integrated way to make them
more accessible and usable.

OK, so what precisely is a “Portable Service”?

Much like a container image, a portable service on disk can be just a
directory tree that contains service executables and all their
dependencies, in a hierarchy resembling the normal Linux directory
hierarchy. A portable service can also be a raw disk image, containing
a file system containing such a tree (which can be mounted via a
loop-back block device), or multiple file systems (in which case they
need to follow the Discoverable Partitions

and be located within a GPT partition table). Regardless whether the
portable service on disk is a simple directory tree or a raw disk
image, let’s call this concept the portable service image.

Such images can be generated with any tool typically used for the
purpose of installing OSes inside some directory, for example dnf
or debootstrap. There are very few requirements made
on these trees, except the following two:

  1. The tree should carry systemd unit

    for relevant services in them.

  2. The tree should carry
    (or /etc/os-release) OS release information.

Of course, as you might notice, OS trees generated from any of today’s
big distributions generally qualify for these two requirements without
any further modification, as pretty much all of them adopted
/usr/lib/os-release and tend to ship their major services with
systemd unit files.

A portable service image generated like this can be “attached” or
“detached” from a host:

  1. “Attaching” an image to a host is done through the new

    command. This command dissects the image, reading the os-release
    information, and searching for unit files in them. It then copies
    relevant unit files out of the images and into
    /etc/systemd/system/. After that it augments any copied service
    unit files in two ways: a drop-in adding a RootDirectory= or
    RootImage= line is added in so that even though the unit files
    are now available on the host when started they run the referenced
    binaries from the image. It also symlinks in a second drop-in which
    is called a “profile”, which is supposed to carry additional
    security settings to enforce on the attached services, to ensure
    the right amount of sand-boxing.

  2. “Detaching” an image from the host is done through portable
    . It reverses the steps above: the unit files copied out are
    removed again, and so are the two drop-in files generated for them.

While a portable service is attached its relevant unit files are made
available on the host like any others: they will appear in systemctl
, you can enable and disable them, you can start them
and stop them. You can extend them with systemctl edit. You can
introspect them. You can apply resource management to them like to any
other service, and you can process their logs like any other service
and so on. That’s because they really are native systemd services,
except that they have ‘twist’ if you so will: they have tougher
security by default and store their resources in a root directory or

And that’s already the essence of what Portable Services are.

A couple of interesting points:

  1. Even though the focus is on shipping service unit files in
    portable service images, you can actually ship timer units, socket
    units, target units, path units in portable services too. This
    means you can very naturally do time, socket and path based
    activation. It’s also entirely fine to ship multiple service units
    in the same image, in case you have more complex applications.

  2. This concept introduces zero new metadata. Unit files are an
    existing concept, as are os-release files, and — in case you opt
    for raw disk images — GPT partition tables are already established
    too. This also means existing tools to generate images can be
    reused for building portable service images to a large degree as no
    completely new artifact types need to be generated.

  3. Because the Portable Service concepts introduces zero new metadata
    and just builds on existing security and resource bundling
    features of systemd it’s implemented in a set of distinct tools,
    relatively disconnected from the rest of systemd. Specifically, the
    main user-facing command is
    and the actual operations are implemented in
    systemd-portabled.service. If
    you so will, portable services are a true add-on to systemd, just
    making a specific work-flow nicer to use than with the basic
    operations systemd otherwise provides. Also note that
    systemd-portabled provides bus APIs accessible to any program
    that wants to interface with it, portablectl is just one tool
    that happens to be shipped along with systemd.

  4. Since Portable Services are a feature we only added very recently
    we wanted to keep some freedom to make changes still. Due to that
    we decided to install the portablectl command into
    /usr/lib/systemd/ for now, so that it does not appear in $PATH
    by default. This means, for now you have to invoke it with a full
    path: /usr/lib/systemd/portablectl. We expect to move it into
    /usr/bin/ very soon though, and make it a fully supported
    interface of systemd.

  5. You may wonder which unit files contained in a portable service
    image are the ones considered “relevant” and are actually copied
    out by the portablectl attach operation. Currently, this is
    derived from the image name. Let’s say you have an image stored in
    a directory /var/lib/portables/foobar_4711/ (or alternatively in
    a raw image /var/lib/portables/foobar_4711.raw). In that case the
    unit files copied out match the pattern foobar*.service,
    foobar*.socket, foobar*.target, foobar*.path,

  6. The Portable Services concept does not define any specific method
    how images get on the deployment machines, that’s entirely up to
    administrators. You can just scp them there, or wget them. You
    could even package them as RPMs and then deploy them with dnf if
    you feel adventurous.

  7. Portable service images can reside in any directory you
    like. However, if you place them in /var/lib/portables/ then
    portablectl will find them easily and can show you a list of
    images you can attach and suchlike.

  8. Attaching a portable service image can be done persistently, so
    that it remains attached on subsequent boots (which is the default),
    or it can be attached only until the next reboot, by passing
    --runtime to portablectl.

  9. Because portable service images are ultimately just regular OS
    images, it’s natural and easy to build a single image that can be
    used in three different ways:

    1. It can be attached to any host as a portable service image.

    2. It can be booted as OS container, for example in a container
      manager like systemd-nspawn.

    3. It can be booted as host system, for example on bare metal or
      in a VM manager.

    Of course, to qualify for the latter two the image needs to
    contain more than just the service binaries, the os-release file
    and the unit files. To be bootable an OS container manager such as
    systemd-nspawn the image needs to contain an init system of some
    form, for example
    systemd. To
    be bootable on bare metal or as VM it also needs a boot loader of
    some form, for example


In the previous section the “profile” concept was briefly
mentioned. Since they are a major feature of the Portable Services
concept, they deserve some focus. A “profile” is ultimately just a
pre-defined drop-in file for unit files that are attached to a
host. They are supposed to mostly contain sand-boxing and security
settings, but may actually contain any other settings, too. When a
portable service is attached a suitable profile has to be selected. If
none is selected explicitly, the default profile called default is
used. systemd ships with four different profiles out of the box:

  1. The
    profile provides a medium level of security. It contains settings to
    drop capabilities, enforce system call filters, restrict many kernel
    interfaces and mount various file systems read-only.

  2. The
    profile is similar to the default profile, but generally uses the
    most restrictive sand-boxing settings. For example networking is turned
    off and access to AF_NETLINK sockets is prohibited.

  3. The
    profile is the least strict of them all. In fact it makes almost no
    restrictions at all. A service run with this profile has basically
    full access to the host system.

  4. The
    profile is mostly identical to default, but also turns off network access.

Note that the profile is selected at the time the portable service
image is attached, and it applies to all service files attached, in
case multiple are shipped in the same image. Thus, the sand-boxing
restriction to enforce are selected by the administrator attaching the
image and not the image vendor.

Additional profiles can be defined easily by the administrator, if
needed. We might also add additional profiles sooner or later to be
shipped with systemd out of the box.

What’s the use-case for this? If I have containers, why should I bother?

Portable Services are primarily intended to cover use-cases where code
should more feel like “extensions” to the host system rather than live
in disconnected, separate worlds. The profile concept is
supposed to be tunable to the exact right amount of integration or
isolation needed for an application.

In the container world the concept of “super-privileged containers”
has been touted a lot, i.e. containers that run with full
privileges. It’s precisely that use-case that portable services are
intended for: extensions to the host OS, that default to isolation,
but can optionally get as much access to the host as needed, and can
naturally take benefit of the full functionality of the host. The
concept should hence be useful for all kinds of low-level system
software that isn’t shipped with the OS itself but needs varying
degrees of integration with it. Besides servers and appliances this
should be particularly interesting for IoT and embedded devices.

Because portable services are just a relatively small extension to the
way system services are otherwise managed, they can be treated like
regular service for almost all use-cases: they will appear along
regular services in all tools that can introspect systemd unit data,
and can be managed the same way when it comes to logging, resource
management, runtime life-cycles and so on.

Portable services are a very generic concept. While the original
use-case is OS extensions, it’s of course entirely up to you and other
users to use them in a suitable way of your choice.


Let’s have a look how this all can be used. We’ll start with building
a portable service image from scratch, before we attach, enable and
start it on a host.

Building a Portable Service image

As mentioned, you can use any tool you like that can create OS trees
or raw images for building Portable Service images, for example
debootstrap or dnf --installroot=. For this example walkthrough
run we’ll use mkosi, which is
ultimately just a fancy wrapper around dnf and debootstrap but
makes a number of things particularly easy when repetitively building
images from source trees.

I have pushed everything necessary to reproduce this walkthrough
locally to a GitHub
. Let’s check it out:

$ git clone https://github.com/systemd/portable-walkthrough.git

Let’s have a look in the repository:

  1. First of all,
    is the main source file of our little service. To keep things
    simple it’s written in C, but it could be in any language of your
    choice. The daemon as implemented won’t do much: it just starts up
    and waits for SIGTERM, at which point it will shut down. It’s
    ultimately useless, but hopefully illustrates how this all fits
    together. The C code has no dependencies besides libc.

  2. walkthroughd.service
    is a systemd unit file that starts our little daemon. It’s a simple
    service, hence the unit file is trivial.

  3. Makefile
    is a short make build script to build the daemon binary. It’s
    pretty trivial, too: it just takes the C file and builds a binary
    from it. It can also install the daemon. It places the binary in
    /usr/local/lib/walkthroughd/walkthroughd (why not in
    /usr/local/bin? because it’s not a user-facing binary but a system
    service binary), and its unit file in
    /usr/local/lib/systemd/walkthroughd.service. If you want to test
    the daemon on the host we can now simply run make and then
    ./walkthroughd in order to check everything works.

  4. mkosi.default
    is file that tells mkosi how to build the image. We opt for a
    Fedora-based image here (but we might as well have used Debian
    here, or any other supported distribution). We need no particular
    packages during runtime (after all we only depend on libc), but
    during the build phase we need gcc and make, hence these are the
    only packages we list in BuildPackages=.

  5. mkosi.build
    is a shell script that is invoked during mkosi’s build logic. All
    it does is invoke make and make install to build and install
    our little daemon, and afterwards it extends the
    distribution-supplied /etc/os-release file with an additional
    field that describes our portable service a bit.

Let’s now use this to build the portable service image. For that we
use the mkosi tool. It’s
sufficient to invoke it without parameter to build the first image: it
will automatically discover mkosi.default and mkosi.build which
tells it what to do. (Note that if you work on a project like this for
a longer time, mkosi -if is probably the better command to use, as
it that speeds up building substantially by using an incremental build
mode). mkosi will download the necessary RPMs, and put them all
together. It will build our little daemon inside the image and after
all that’s done it will output the resulting image:

Because we opted to build a GPT raw disk image in mkosi.default this
file is actually a raw disk image containing a GPT partition
table. You can use fdisk -l walkthroughd_1.raw to enumerate the
partition table. You can also use systemd-nspawn -i
to explore the image quickly if you need.

Using the Portable Service Image

Now that we have a portable service image, let’s see how we can
attach, enable and start the service included within it.

First, let’s attach the image:

# /usr/lib/systemd/portablectl attach ./walkthroughd_1.raw
(Matching unit files with prefix 'walkthroughd'.)
Created directory /etc/systemd/system/walkthroughd.service.d.
Written /etc/systemd/system/walkthroughd.service.d/20-portable.conf.
Created symlink /etc/systemd/system/walkthroughd.service.d/10-profile.conf → /usr/lib/systemd/portable/profile/default/service.conf.
Copied /etc/systemd/system/walkthroughd.service.
Created symlink /etc/portables/walkthroughd_1.raw → /home/lennart/projects/portable-walkthrough/walkthroughd_1.raw.

The command will show you exactly what is has been doing: it just
copied the main service file out, and added the two drop-ins, as

Let’s see if the unit is now available on the host, just like a regular unit, as promised:

# systemctl status walkthroughd.service
● walkthroughd.service - A simple example service
   Loaded: loaded (/etc/systemd/system/walkthroughd.service; disabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/walkthroughd.service.d
           └─10-profile.conf, 20-portable.conf
   Active: inactive (dead)

Nice, it worked. We see that the unit file is available and that
systemd correctly discovered the two drop-ins. The unit is neither
enabled nor started however. Yes, attaching a portable service image
doesn’t imply enabling nor starting. It just means the unit files
contained in the image are made available to the host. It’s up to the
administrator to then enable them (so that they are automatically
started when needed, for example at boot), and/or start them (in case
they shall run right-away).

Let’s now enable and start the service in one step:

# systemctl enable --now walkthroughd.service
Created symlink /etc/systemd/system/multi-user.target.wants/walkthroughd.service → /etc/systemd/system/walkthroughd.service.

Let’s check if it’s running:

# systemctl status walkthroughd.service
● walkthroughd.service - A simple example service
   Loaded: loaded (/etc/systemd/system/walkthroughd.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/walkthroughd.service.d
           └─10-profile.conf, 20-portable.conf
   Active: active (running) since Wed 2018-06-27 17:55:30 CEST; 4s ago
 Main PID: 45003 (walkthroughd)
    Tasks: 1 (limit: 4915)
   Memory: 4.3M
   CGroup: /system.slice/walkthroughd.service
           └─45003 /usr/local/lib/walkthroughd/walkthroughd

Jun 27 17:55:30 sigma walkthroughd[45003]: Initializing.

Perfect! We can see that the service is now enabled and running. The daemon is running as PID 45003.

Now that we verified that all is good, let’s stop, disable and detach the service again:

# systemctl disable --now walkthroughd.service
Removed /etc/systemd/system/multi-user.target.wants/walkthroughd.service.
# /usr/lib/systemd/portablectl detach ./walkthroughd_1.raw
Removed /etc/systemd/system/walkthroughd.service.
Removed /etc/systemd/system/walkthroughd.service.d/10-profile.conf.
Removed /etc/systemd/system/walkthroughd.service.d/20-portable.conf.
Removed /etc/systemd/system/walkthroughd.service.d.
Removed /etc/portables/walkthroughd_1.raw.

And finally, let’s see that it’s really gone:

# systemctl status walkthroughd
Unit walkthroughd.service could not be found.

Perfect! It worked!

I hope the above gets you started with Portable Services. If you have
further questions, please contact our mailing

Further Reading

A more low-level document explaining details is shipped
along with systemd

There are also relevant manual pages:

For further information about mkosi see its homepage.

Source From: fedoraplanet.org.
Original article title: Lennart Poettering: Walkthrough for Portable Services.
This full article can be read at: Lennart Poettering: Walkthrough for Portable Services.


Random Article You May Like

Leave a Reply

Your email address will not be published. Required fields are marked *