User guide for osbuild-composer

osbuild-composer is a service for building customized operating system images (currently only Fedora and RHEL). These images can be used with various virtualization software such as QEMU, VirtualBox, VMWare and also with cloud computing providers like AWS, Azure or GCP.

This guide contains instructions on installing osbuild-composer service and its basic usage.

Installation

To get started with osbuild-composer on your local machine, you can install the CLI interface or the Web UI, which is part of Cockpit project.

CLI interface

For CLI only, run the following command to install necessary packages:

$ sudo dnf install osbuild-composer composer-cli

To enable the service, run this command:

$ sudo systemctl enable --now osbuild-composer.socket

Verify that the installation works by running composer-cli:

$ sudo composer-cli status show

If you prefer to run this command without sudo privileges, add your user to the weldr group:

$ sudo usermod -a -G weldr <user>
$ newgrp weldr

Web UI

If you prefer the Web UI interface, known as an Image Builder, install the following package:

$ sudo dnf install cockpit-composer

and enable cockpit and osbuild-composer services:

$ sudo systemctl enable --now osbuild-composer.socket
$ sudo systemctl enable --now cockpit

Basic concepts

osbuild-composer works with a concept of blueprints. A blueprint is a description of the final image and its customizations. A customization can be:

• an additional RPM package
• enabled service
• custom kernel command line parameter, and many others. See Blueprint reference for more details.

An image is defined by its blueprint and image type, which is for example qcow2 (QEMU Copy On Write disk image) or AMI (Amazon Machine Image).

Finally, osbuild-composer also supports upload targets, which are cloud providers where an image can be stored after it is built. Currently supported cloud providers are AWS and Azure.

Example blueprint

name = "base-image-with-tmux"
description = "A base system with tmux"
version = "0.0.1"

[[packages]]
name = "tmux"
version = "*"

The blueprint is in TOML format.

Blueprints management using composer-cli

osbuild-composer provides a storage for blueprints. To store a blueprint.toml blueprint file, run this command:

$ composer-cli blueprints push blueprint.toml

To verify that the blueprint is available, list all currently stored blueprints:

$ composer-cli blueprints list
base-image-with-tmux

To display the blueprint you have just added, run the command:

$ sudo composer-cli blueprints show base-image-with-tmux
name = "base-image-with-tmux"
description = "A base system with tmux"
version = "0.0.1"
modules = []
groups = []

[[packages]]
name = "tmux"
version = "*"

Image types

osbuild-composer supports various types of output images. To see all supported types, run this command:

$ composer-cli compose types

Building an image

An image is specified by a blueprint and an image type. It will use the same distribution version (e.g. Fedora 33) and architecture (e.g. aarch64) as the host system.

To build a customized image, start by choosing the blueprint and image type you would like to build. To do so, run the following commands:

$ sudo composer-cli blueprints list
$ sudo composer-cli compose types

and trigger a compose (example using the blueprint from the previous section):

$ composer-cli compose start base-image-with-tmux qcow2
Compose ab71b61a-b3c4-434f-b214-1e16527766ff added to the queue

Note that the compose is assigned with a Universally Unique Identifier (UUID), that you can use to monitor the image build progress:

$ composer-cli compose info ab71b61a-b3c4-434f-b214-1e16527766ff
ab71b61a-b3c4-434f-b214-1e16527766ff RUNNING  base-image-with-tmux 0.0.1 qcow2            2147483648
Packages:
    tmux-*
Modules:
Dependencies:

At this time, the compose is in a "RUNNING" state. Once the compose reaches the "FINISHED" state, you can download the resulting image by running the following command:

$ sudo composer-cli compose results ab71b61a-b3c4-434f-b214-1e16527766ff
ab71b61a-b3c4-434f-b214-1e16527766ff.tar: 455.18 MB
$ fd
ab71b61a-b3c4-434f-b214-1e16527766ff.tar
$ tar xf ab71b61a-b3c4-434f-b214-1e16527766ff.tar
$ fd 
ab71b61a-b3c4-434f-b214-1e16527766ff-disk.qcow2
ab71b61a-b3c4-434f-b214-1e16527766ff.json
ab71b61a-b3c4-434f-b214-1e16527766ff.tar
logs
logs/osbuild.log

From the example output above, the resulting tarball contains not only the qcow2 image, but also a JSON file, which is the osbuild manifest (see the Developer guide for more details), and a directory with logs.

For more options, see the help text for composer-cli:

$ sudo composer-cli compose help

Tip: Booting the image with qemu

If you want to quickly run the resulting image, you can use qemu:

$ qemu-system-x86_64 \
                -enable-kvm \
                -m 3000 \
                -snapshot \
                -cpu host \
                -net nic,model=virtio \
                -net user,hostfwd=tcp::2223-:22 \
                ab71b61a-b3c4-434f-b214-1e16527766ff-disk.qcow2 

Be aware that you must specify a way to access the machine in the blueprint. For example, you can create a user with known password, set an SSH key, or enable cloud-init to use a cloud-init ISO file.

Uploading an image to AWS

osbuild-composer provides the users with a convenient way to upload images directly to AWS right after the image is built. To use this feature, you need one additional configuration file for the cloud provider. In this example, the cloud provider is AWS:

provider = "aws"

[settings]
accessKeyID = "AWS_ACCESS_KEY_ID"
secretAccessKey = "AWS_SECRET_ACCESS_KEY"
bucket = "AWS_BUCKET"
region = "AWS_REGION"
key = "IMAGE_KEY"

But be careful here, if you are using IAM Roles, which is the recommendation, according to the AWS documentation, you need a specific policy that will allow VM import from the S3 bucket to EC2. See the AWS documentation for more details.

Once everything is configured, you can trigger a compose as usual with additional image name and cloud provider profile:

$ sudo composer-cli compose start base-image-with-tmux ami IMAGE_KEY aws-config.toml

where IMAGE_KEY will be the name of your VM Image, once it is uploaded to EC2.

Managing repositories

There are two kinds of repositories used in osbuild-composer:

1. Custom 3rd party repositories - use these to include packages that are not available in the official Fedora or RHEL repositories.
2. Official repository overrides - use these if you want to download base system RPMs from elsewhere than the official repositories. For example if you have a custom mirror in your network. Keep in mind that this will disable the default repositories, so the mirror must contain all necessary packages!

Custom 3rd party repositories

These are managed using composer-cli (see the manpage for complete reference). To add a new repository, create a TOML file like this:

id = "k8s"
name = "Kubernetes"
type = "yum-baseurl"
url = "https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64"
check_gpg = false
check_ssl = false
system = false

and add it using composer-cli sources add <file-name.toml>. Verify its presence using composer-cli sources list and its content using composer-cli sources info <id>.

Official repository overrides

osbuild-composer does not inherit the system repositories located in /etc/yum.repos.d/. Instead, it has its own set of official repositories defined in /usr/share/osbuild-composer/repositories. To override the official repositories, define overrides in /etc/osbuild-composer/repositories. This directory is meant for user defined overrides and the files located here take precedence over those in /usr.

The configuration files are not in the usual "repo" format. Instead, they are simple JSON files.

Important note: osbuild-composer can only create images for the distribution and architecture it is running on. For example, if you are running Fedora 33 on x86_64, osbuild-composer will create all images as Fedora 33 for x86_64. Building other distributions and architectures except for the host is not supported.

Defining official repository overrides

To set your own repositories, create this directory if it does not exist already:

$ sudo mkdir -p /etc/osbuild-composer/repositories

Based on the system you are running (see /etc/os-release if you are not sure), determine the name of a new JSON file:

• Fedora 32 - fedora-32.json
• Fedora 33 - fedora-33.json
• Already released RHEL 8 - rhel-8.json
• Pre-release RHEL 8.4 - rhel-84.json

Then, create the JSON file with the following structure (or copy the file for your distribution from /usr/share/osbuild-composer/ and modify its content):

{
    "<ARCH>": [
        {
            "name": "<REPO NAME>",
            "metalink": "",
            "baseurl": "",
            "mirrorlist": "",
            "gpgkey": "",
            "check_gpg": "",
            "metadata_expire": "",
        }
    ]
}

Specify only one value for the following attributes: metalink, mirrorlist, or baseurl. The remaining fields are optional.

For example, assuming that the host OS is Fedora 33 running on x86_64, create /etc/osbuild-composer/repositories/fedora-33.json with this content:

{
    "x86_64": [
        {
            "name": "fedora",
            "metalink": "https://mirrors.fedoraproject.org/metalink?repo=fedora-33&arch=x86_64",
            "gpgkey": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n\nmQINBF4wBvsBEADQmcGbVUbDRUoXADReRmOOEMeydHghtKC9uRs9YNpGYZIB+bie\nbGYZmflQayfh/wEpO2W/IZfGpHPL42V7SbyvqMjwNls/fnXsCtf4LRofNK8Qd9fN\nkYargc9R7BEz/mwXKMiRQVx+DzkmqGWy2gq4iD0/mCyf5FdJCE40fOWoIGJXaOI1\nTz1vWqKwLS5T0dfmi9U4Tp/XsKOZGvN8oi5h0KmqFk7LEZr1MXarhi2Va86sgxsF\nQcZEKfu5tgD0r00vXzikoSjn3qA5JW5FW07F1pGP4bF5f9J3CZbQyOjTSWMmmfTm\n2d2BURWzaDiJN9twY2yjzkoOMuPdXXvovg7KxLcQerKT+FbKbq8DySJX2rnOA77k\nUG4c9BGf/L1uBkAT8dpHLk6Uf5BfmypxUkydSWT1xfTDnw1MqxO0MsLlAHOR3J7c\noW9kLcOLuCQn1hBEwfZv7VSWBkGXSmKfp0LLIxAFgRtv+Dh+rcMMRdJgKr1V3FU+\nrZ1+ZAfYiBpQJFPjv70vx+rGEgS801D3PJxBZUEy4Ic4ZYaKNhK9x9PRQuWcIBuW\n6eTe/6lKWZeyxCumLLdiS75mF2oTcBaWeoc3QxrPRV15eDKeYJMbhnUai/7lSrhs\nEWCkKR1RivgF4slYmtNE5ZPGZ/d61zjwn2xi4xNJVs8q9WRPMpHp0vCyMwARAQAB\ntDFGZWRvcmEgKDMzKSA8ZmVkb3JhLTMzLXByaW1hcnlAZmVkb3JhcHJvamVjdC5v\ncmc+iQI4BBMBAgAiBQJeMAb7AhsPBgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAK\nCRBJ/XdJlXD/MZm2D/9kriL43vd3+0DNMeA82n2v9mSR2PQqKny39xNlYPyy/1yZ\nP/KXoa4NYSCA971LSd7lv4n/h5bEKgGHxZfttfOzOnWMVSSTfjRyM/df/NNzTUEV\n7ORA5GW18g8PEtS7uRxVBf3cLvWu5q+8jmqES5HqTAdGVcuIFQeBXFN8Gy1Jinuz\nAH8rJSdkUeZ0cehWbERq80BWM9dhad5dW+/+Gv0foFBvP15viwhWqajr8V0B8es+\n2/tHI0k86FAujV5i0rrXl5UOoLilO57QQNDZH/qW9GsHwVI+2yecLstpUNLq+EZC\nGqTZCYoxYRpl0gAMbDLztSL/8Bc0tJrCRG3tavJotFYlgUK60XnXlQzRkh9rgsfT\nEXbQifWdQMMogzjCJr0hzJ+V1d0iozdUxB2ZEgTjukOvatkB77DY1FPZRkSFIQs+\nfdcjazDIBLIxwJu5QwvTNW8lOLnJ46g4sf1WJoUdNTbR0BaC7HHj1inVWi0p7IuN\n66EPGzJOSjLK+vW+J0ncPDEgLCV74RF/0nR5fVTdrmiopPrzFuguHf9S9gYI3Zun\nYl8FJUu4kRO6JPPTicUXWX+8XZmE94aK14RCJL23nOSi8T1eW8JLW43dCBRO8QUE\nAso1t2pypm/1zZexJdOV8yGME3g5l2W6PLgpz58DBECgqc/kda+VWgEAp7rO2A==\n=EPL3\n-----END PGP PUBLIC KEY BLOCK-----\n",
            "check_gpg": true
        }
    ]
}

Building OSTree image

This section contains a guide for building OSTree commits. As opposed to the "traditional" image types, these commits are not directly bootable so although they basically contain a full operating system, in order to boot them, they need to be deployed. This can, for example, be done via the Fedora installer (Anaconda).

OSTree is a technology for creating immutable operating system images and it is a base for Fedora CoreOS, Fedora IoT, Fedora Silverblue, and RHEL for Edge. For more information on OSTree, see their website.

Overview of the intended result

As mentioned above, osbuild-composer produces OSTree commits which are not directly bootable. The commits are inside a tarball to make their usage more convenient. In order to deploy them, you will need:

• Fedora installation ISO - such as netinst (https://getfedora.org/en/server/download/)

• HTTP server to serve the content of the tarball to the Fedora virtual machine booted from the ISO

• Kickstart file that instructs Anaconda (Fedora installer) to use the OSTree commit from the HTTP server

In this guide, a container running Apache httpd will be used as the HTTP server.

The result will look like this:

 _________________          ____________________________
|                 |        |                            |
|                 |------> | Fedora VM with mounted ISO |
|                 |        |  - Anaconda                |
|  Fedora Host OS |        |____________________________|
|                 |                |
|                 |         _______|________________________
|                 |        |                                |
|                 |------->| Fedora container running httpd |
|_________________|        |  serving content of the tarball|
                           |  and the kickstart file        |
                           |________________________________|

Note: If you would like to understand what is inside the tarball, read the upstream OSTree documentation.

Building an OSTree commit

Start by creating a blueprint for your commit. Using your favorite text editor, vi, create a file named fishy.toml with this content:

name = "fishy-commit"
description = "Fishy OSTree commit"
version = "0.0.1"

[[packages]]
name = "fish"
version = "*"

Now push the blueprint to osbuild-composer using composer-cli:

$ composer-cli blueprints push fishy.toml

And start a build:

$ composer-cli compose start fishy-commit fedora-iot-commit
Compose 8e8014f8-4d15-441a-a26d-9ed7fc89e23a added to the queue

Monitor the build status using:

$ composer-cli compose status

And finally when the compose is complete, download the result:

$ composer-cli compose image 8e8014f8-4d15-441a-a26d-9ed7fc89e23a
8e8014f8-4d15-441a-a26d-9ed7fc89e23a-commit.tar: 670.45 MB

Writing a Kickstart file

As mentioned above, the Kickstart file is meant for the Anaconda installer. It contains instructions on how to install the system.

Create a file named ostree.ks with this content:

lang en_US.UTF-8
keyboard us
timezone UTC
zerombr
clearpart --all --initlabel
autopart
reboot
user --name=core --groups=wheel --password=foobar
ostreesetup --nogpg --url=http://10.0.2.2:8000/repo/ --osname=iot --remote=iot --ref=fedora/33/x86_64/iot

For those interested in all the options, you can read Anaconda’s documentation.

The crucial part is on the last line. Here, ostreesetup command is used to fetch the OSTree commit. Now for those wondering about the IP address, this tutorial uses qemu to boot the virtual machine and 10.0.2.2 is an address which you can use to reach the host system from the guest: User Networking.

Setting up an HTTP server

Now that the kickstart file and OSTree commit are ready, create a container running HTTP server and serving those file. Start by creating a Dockerfile:

FROM fedora:latest
RUN dnf -y install httpd && dnf clean all
ADD *.tar *.ks /var/www/html
EXPOSE 80
CMD ["/usr/sbin/httpd", "-D", "FOREGROUND"]

Make sure you have everything in the build directory (keep in mind that the UUID is random, so it will be different in your case):

$ ls
8e8014f8-4d15-441a-a26d-9ed7fc89e23a-commit.tar
Dockerfile
ostree.ks

Build the container image:

$ podman build -t ostree .

And run it:

$ podman run --rm -p 8000:80 ostree

Note: You might be wondering why to bother with a container when you can just use "python -m http.server". The problem is that OSTree produces way too many requests and the Python HTTP server simply fails to keep up with OSTree.

Running a VM and applying the OSTree commit

Start with downloading the Netinstall image from here: https://getfedora.org/en/server/download/

Create an empty qcow2 image. That is an image of a hard drive for the virtual machine (VM).

$ qemu-img create -f qcow2 disk-image.img 5G

Run a VM using the hard drive and mount the installation ISO:

$ qemu-system-x86_64 \
          -enable-kvm \
          -m 3000 \
          -snapshot \
          -cpu host \
          -net nic,model=virtio \
          -net user,hostfwd=tcp::2223-:22 \
          -cdrom $HOME/Downloads/Fedora-Server-netinst-x86_64-33-1.2.iso \
          disk-image.img

Note: To prevent any issue, use the latest stable Fedora host OS for this tutorial.

This command instructs qemu (the hypervisor) to:

• Use KVM virtualization (makes the VM faster).
• Increase memory to 3000MB (some processes can get memory hungry, for example dnf).
• Snapshot the hard drive image, don't override its content.
• Use the same CPU type as the host uses.
• Connect the guest to a virtual network bridge on the host and forward TCP port 2223 from the host to the SSH port (22) on the guest (makes it easier to connect to the guest system).
• Mount the installation ISO.
• Use the hard drive image created above.

At the initial screen, use arrow keys to select the "Install Fedora 33" line and press TAB key. You’ll see a line of kernel command line options appear below. Something like:

vmlinuz initrd=initrd.img inst.stage2=hd:LABEL=Fedora quiet

Add a space and this string:

inst.ks=http://10.0.2.2:8000/ostree.ks

Resulting in this kernel command line:

vmlinuz initrd=initrd.img inst.stage2=hd:LABEL=Fedora quiet inst.ks=http://10.0.2.2:8000/ostree.ks

The IP address 10.0.2.2 is again used here, because the VM is running inside Qemu.

Press "Enter", the Anaconda GUI will show up and automatically install the OSTree commit created above.

Once the system is installed and rebooted, use username "core" and password "foobar" to login. You can change the credentials in the kickstart file.

Building a RHEL for Edge Installer

The following describes how to build a boot ISO which installs an OSTree-based system using the "RHEL for Edge Container" in combination with the "RHEL for Edge Installer" image types. The workflow has the same result as the Building OSTree Image guide with the new image types automating some of the steps.

Process overview

1. Create and load a blueprint with customizations.
2. Build a rhel-edge-container image.
3. Load image in podman and start the container.
4. Create and load an empty blueprint.
5. Build a rhel-edge-installer image, pointing the ostree-url to http://10.0.2.2:8080/repo/ and setting the ostree-ref to rhel/edge/demo.

The rhel-edge-container image type creates an OSTree commit and embeds it into an OCI container with a web server. When the container is started, the web server serves the commit as an OSTree repository.

The rhel-edge-intaller image type pulls the commit from the running container and creates an installable boot ISO with a kickstart file configured to use the embedded OSTree commit.

Detailed workflow

Build the container and serve the commit

Start by creating a blueprint for the commit. The content below is an example and can be modified to fit your needs. For this guide, we will name the file example.toml.

name = "example"
description = "RHEL for Edge Installer example"
version = "0.0.3"

[[packages]]
name = "vim-enhanced"
version = "*"

[[packages]]
name = "tmux"
version = "*"

[customizations]

[[customizations.user]]
name = "user"
description = "Example User"
password = "$6$uvdfeuHQYM6kUaea$fvvzyu.Z.u89TVCB2tq8UEc52XDFGnAqCo75BX3zu8OzIbS.EKMo/Saammb151sLrdzmlESnpNEPrJ7h5b0c6/"
groups = ["wheel"]

Now push the blueprint to osbuild-composer using composer-cli:

$ composer-cli blueprints push example.toml

And start the container build:

$ composer-cli compose start-ostree --ref "rhel/edge/example" example rhel-edge-container
Compose 8e8014f8-4d15-441a-a26d-9ed7fc89e23a added to the queue

The value for --ref can be changed but must begin with an alphanumeric character and contain only alphanumeric characters, /, _, -, and ..

Monitor the build status using:

$ composer-cli compose status

When the compose is FINISHED, download the result:

$ composer-cli compose image 8e8014f8-4d15-441a-a26d-9ed7fc89e23a
8e8014f8-4d15-441a-a26d-9ed7fc89e23a-rhel84-container.tar: 670.45 MB

Load the container into registry:

$ cat 8e8014f8-4d15-441a-a26d-9ed7fc89e23a-rhel84-container.tar | podman load
Getting image source signatures
Copying blob 82934cd3e69d done
Copying config d11911c3dc done
Writing manifest to image destination
Storing signatures
Loaded image(s): @d11911c3dc4bee46cabd52b91c87f48b8a7d450fadc8cfbeb69e2de98b413521

Tag the image for convenience and start the container:

$ podman tag d11911c3dc4bee46cabd52b91c87f48b8a7d450fadc8cfbeb69e2de98b413521 localhost/edge-example
$ podman run --rm -d -p 8080:80 --name ostree-repo localhost/edge-example

Note: The -d option detaches the container and leaves it running in the background. You can also remove the option to keep the container attached to the terminal.

Build the installer

Start by creating a simple blueprint for the installer. The blueprint must not have any customizations or packages; only a name, and optionally a version and a description. Add the content below to a file and name it empty.toml:

name = "empty"
description = "Empty blueprint"
version = "0.0.1"

The rhel-edge-installer image type does not support customizations or package selection, so the build will fail if any are specified.

Push the blueprint:

$ composer-cli blueprints push empty.toml

Start the build:

$ composer-cli compose start-ostree --ref "rhel/edge/example" --url http://10.0.2.2:8000/repo/ empty rhel-edge-installer
Compose 09d98a67-a401-4613-9a5b-b93f8a6e695f added to the queue

The --ref argument must match the one from the rhel-edge-container compose. The --url in this case is IP address of the container. This tutorial uses qemu to boot the virtual machine and 10.0.2.2 is an address which you can use to reach the host system from the guest: User Networking.

Monitor the build status using:

$ composer-cli compose status

When the compose is FINISHED, download the result:

$ composer-cli compose image 09d98a67-a401-4613-9a5b-b93f8a6e695f
 09d98a67-a401-4613-9a5b-b93f8a6e695f-rhel84-boot.iso: 1422.61 MB

The downloaded image can then booted to begin the installation. If you used the blueprint in this guide, use the username "user" and password "password42" to login.

Blueprint reference

Blueprints are simple text files in TOML format that describe which packages to install into the image, allowing to specify the packages version. They can also define a limited set of customizations to make to the final image.

A basic blueprint looks like this:

name = "base"
description = "A base system with bash"
version = "0.0.1"

[[packages]]
name = "bash"
version = "4.4.*"

Where:

• name field is the name of the blueprint. It can contain spaces, but they will be converted to - when it is written to disk. It should be short and descriptive.
• description can be a longer description of the blueprint, it is only used for display purposes.
• version is a semver compatible version number. If a new blueprint is uploaded with the same version the server will automatically bump the PATCH level of the version. If the version doesn't match it will be used as is, for example, uploading a blueprint with version set to 0.1.0 when the existing blueprint version is 0.0.1 will result in the new blueprint being stored as version 0.1.0.

Packages and modules

[[packages]] and [[modules]] entries describe the package names and matching version glob to be installed into the image.

The package names must match the names exactly, and the versions can be an exact match or a filesystem-like glob of the version using * wildcards and ? character matching.

Currently there are no differences between packages and modules in osbuild-composer. Both are treated like an rpm package dependency.

For example, to install tmux-2.9a and openssh-server-8.* packages, add this to your blueprint:

[[packages]]
name = "tmux"
version = "2.9a"

[[packages]]
name = "openssh-server"
version = "8.*"

Groups

The [[groups]] entries describe a group of packages to be installed into the image. Package groups are defined in the repository metadata. Each group has a descriptive name used primarily for display in user interfaces and an ID more commonly used in kickstart files. Here, the ID is the expected way of listing a group.

Groups have three different ways of categorizing their packages: mandatory, default, and optional. For purposes of blueprints, just mandatory and default packages will be installed. There is no mechanism for selecting optional packages.

For example, if you want to install the anaconda-tools group, add the following to your blueprint:

[[groups]]
name="anaconda-tools"

groups is a TOML list, so each group needs to be listed separately, like packages but with no version number.

Customizations

The [customizations] section can be used to configure the hostname of the final image. for example:

[customizations]
hostname = "baseimage"

This is optional and can be left out to use the defaults.

Kernel command-line arguments

This allows you to append arguments to the bootloader's kernel command line.

For example:

[customizations.kernel]
append = "nosmt=force"

SSH Keys

Set an existing user's ssh key in the final image:

[[customizations.sshkey]]
user = "root"
key = "PUBLIC SSH KEY"

The key will be added to the user's authorized_keys file.

Warning: key expects the entire content of ~/.ssh/id_rsa.pub

Additional user

Add a user to the image, and/or set their ssh key. All fields for this section are optional except for the name. The following is a complete example:

[[customizations.user]]
name = "admin"
description = "Administrator account"
password = "$6$CHO2$3rN8eviE2t50lmVyBYihTgVRHcaecmeCk31L..."
key = "PUBLIC SSH KEY"
home = "/srv/widget/"
shell = "/usr/bin/bash"
groups = ["widget", "users", "wheel"]
uid = 1200
gid = 1200

If the password starts with $6$, $5$, or $2b$ it will be stored as an encrypted password. Otherwise it will be treated as a plain text password.

Warning: key expects the entire content of ~/.ssh/id_rsa.pub

Additional group

Add a group to the image. Name is required and GID is optional:

[[customizations.group]]
name = "widget"
gid = 1130

Timezone

Customizing the timezone and the NTP servers to use for the system:

[customizations.timezone]
timezone = "US/Eastern"
ntpservers = ["0.north-america.pool.ntp.org", "1.north-america.pool.ntp.org"]

The values supported by timezone can be listed by running the command:

$ timedatectl list-timezones

If no timezone is setup, the system will default to using UTC. The NTP servers are also optional and will default to using the distribution defaults, which are suitable for most uses.

Some image types have already NTP servers setup, for example, Google cloud image, and they cannot be overridden, because they are required to boot in the selected environment. But the timezone will be updated to the one selected in the blueprint.

Locale

Customize the locale settings for the system:

[customizations.locale]
languages = ["en_US.UTF-8"]
keyboard = "us"

The values supported by languages can be listed by running can be listed by running the command:

$ localectl list-locales 

The values supported by keyboard can be listed by running the command:

 $ localectl list-keymaps`

Multiple languages can be added. The first one becomes the primary, and the others are added as secondary. You must include one or more languages or keyboards in the section.

Firewall

By default the firewall blocks all access, except for services that enable their ports explicitly, like sshd. The following command can be used to open other ports or services. Ports are configured using the port:protocol format:

[customizations.firewall]
ports = ["22:tcp", "80:tcp", "imap:tcp", "53:tcp", "53:udp"]

Numeric ports, or their names from /etc/services can be used in the ports enabled/disabled lists.

The blueprint settings extend any existing settings in the image templates. Thus, if sshd is already enabled, it will extend the list of ports with those already listed by the blueprint.

If the distribution uses firewalld, you can specify services listed by firewall-cmd --get-services in a customizations.firewall.services section:

[customizations.firewall.services]
enabled = ["ftp", "ntp", "dhcp"]
disabled = ["telnet"]

Remember that the firewall.services are different from the names in /etc/services.

Both are optional, if they are not used leave them out or set them to an empty list []. If you only want the default firewall setup this section can be omitted from the blueprint.

NOTE: The Google and OpenStack templates explicitly disable the firewall for their environment. This cannot be overridden by the blueprint.

Systemd services

This section can be used to control which services are enabled at boot time. Some image types already have services enabled or disabled in order for the image to work correctly, and cannot be overridden. For example, ami image type requires sshd, chronyd, and cloud-init services. Without them, the image will not boot. Blueprint services do not replace this services, but add them to the list of services already present in the templates, if any.

The service names are systemd service units. You may specify any systemd unit file accepted by systemctl enable, for example, cockpit.socket:

[customizations.services]
enabled = ["sshd", "cockpit.socket", "httpd"]
disabled = ["postfix", "telnetd"]

Example Blueprint

The following blueprint example will:

• install the tmux, git, and vim-enhanced packages
• set the root ssh key
• add the groups: widget, admin users and students

name = "example-custom-base"
description = "A base system with customizations"
version = "0.0.1"

[[packages]]
name = "tmux"
version = "*"

[[packages]]
name = "git"
version = "*"

[[packages]]
name = "vim-enhanced"
version = "*"

[customizations]
hostname = "custombase"

[[customizations.sshkey]]
user = "root"
key = "A SSH KEY FOR ROOT"

[[customizations.user]]
name = "widget"
description = "Widget process user account"
home = "/srv/widget/"
shell = "/usr/bin/false"
groups = ["dialout", "users"]

[[customizations.user]]
name = "admin"
description = "Widget admin account"
password = "$6$CHO2$3rN8eviE2t50lmVyBYihTgVRHcaecmeCk31LeOUleVK/R/aeWVHVZDi26zAH.o0ywBKH9Tc0/wm7sW/q39uyd1"
home = "/srv/widget/"
shell = "/usr/bin/bash"
groups = ["widget", "users", "students"]
uid = 1200

[[customizations.user]]
name = "plain"
password = "simple plain password"

[[customizations.user]]
name = "bart"
key = "SSH KEY FOR BART"
groups = ["students"]

[[customizations.group]]
name = "widget"

[[customizations.group]]
name = "students"

Developer guide

In this section, you will find a description of the source code in osbuild organization.

The following scheme describes how separate components communicate with each other:

In the very basic use case where osbuild-composer is running locally, the "pool of workers" also lives on the user's host machine. The osbuild-composer and osbuild-worker processes are spawned by systemd. We don't support any other means of spawning these processes, as they rely on systemd to open sockets, create state directories etc. Additionally, osbuild-worker spawns osbuild as a subprocess to create the image itself. The whole image building machinery is spawned from a user process, for example, composer-cli.

osbuild

A CLI tool for building OS images. It takes manifest as an input and produces an image as an output. The manifest consists of:

• sources section
• pipeline

In our usual use-case, that is tied to Fedora and RHEL, not applicable to other non-RPM distros, the sources section contains an org.osbuild.files section, which is a list of RPMs described by their name, hash, and URL for downloading. We do not support metalink at the moment.

This section is, very often, a source of build failures. This happens because we can only include a single link and RPM repos are often instable. Furthermore, we need to set a timeout for the curl download, because we want the build to timeout eventually in case the RPMs are unavailable, but it sometimes fails on slow Internet connection as well.

The pipeline consists of a series of stages and ends with an assembler. A stage is our unit of filesystem tree modification and it is implemented as a standalone executable. For example, we have a stage for installing RPM packages, adding a user, enabling systemd service, or setting a timezone.

The difference between a stage and an assembler is that the former takes a read-write filesystem-tree and performs a certain modification to it, whereas the latter takes a read-only filesystem tree and produces an output artifact.

The pipeline contains one more "nested" pipeline, which does not have an assembler. It is called a "build" pipeline.

High level goals

1. reproducibility
2. extensibility

The ideal case for building images would be that, given the same input manifest, the output image would always be the same no matter what machine was used for building it. Where "the same" is defined as a binary equivalent. The world of IT is, of course, not ideal therefore we define reproducibility as a functional equivalence (that is the image behaves the same when built on different machines) and we limit the set of build machines only to those running the same distribution, in the same version, and on the same architecture. That means if you want to build a Fedora 33 aarch64 image, you need a Fedora 33 aarch64 machine.

It is possible to run a RHEL pipeline on Fedora, for example, but we do not test it and therefore we can't promise it will produce the correct result.

The advantage of the stage/assembler model is that any user can extend the tool with their own stage or assembler.

How osbuild works in practise

The following subsections describe how OSBuild tries to achieve the outlined high level goals.

Manifest versions

OSBuild accepts two versions of manifests. Both manifests are plain JSON files. The following sections contain examples of both (note that comments are not allowed in JSON, so the examples below are not actually valid JSON).

Version 1

The version 1 manifest is built around the idea that an artifact is produced by downloading files from the Internet (e.g. RPMs), using them to build and modify a filesystem tree (using stages), and finally using a read-only version of the final filesystem tree as an input to a assembler which produces the desired artifact.

{
   # This version contains 2 top-level keys.
   # First sources, these get downloaded from a network and are available
   # in the stages.
   "sources": {},
   # Second is a pipeline, which can optionally contain a nested "build"
   # pipeline.
   "pipeline": {
      # The build pipeline is used to create a build container that is
      # later used for building the actual OS artifact. This is mostly
      # to increase reproducibility and host-guest separation.
      # Also note that this is optional.
      "build": {
         "pipeline": {
            "stages": [
               {
                  "name": "",
                  "options": {}
               },
               {
                  "name": "",
                  "options": {}
               }
            ],
            "runner": ""
         }
      },
      # The pipeline itself is a list of osbuild stages.
      "stages": [
         {
            "name": "",
            "options": {}
         },
         {
            "name": "",
            "options": {}
         }
      ],
      # And finally exactly one osbuild assembler.
      "assembler": {
         "name": "",
         "options": {}
      }
   },
}

Version 2

Version 2 is more complicated because OSBuild needed to cover additional use cases like OSTree commit inside of a OCI container. In general that is an artifact inside of another artifact. This is why it comes with multiple pipelines.

{
   # This version has 3 top-level keys.
   # The first one is simply a version.
   "version": "2",
   # The second one are sources as in version 1, but keep in mind that in this
   # version, stages take inputs instead of sources because inputs can be both
   # downloaded from a network and produced by a pipeline in this manifest.
   "sources": {},
   # This time the 3rd entry is a list of pipelines.
   "pipelines": [
      {
         # A custom name for each pipeline. "build" is used only as an example.
         "name": "build",
         # The runner is again optional.
         "runner": "",
         "stages": [
            {
               # The "type" is same as "name" in v1.
               "type": "",
               # The "inputs" field is new in v2. You can specify what goes to
               # the stage. Example inputs are RPMs and OSTree commits from the
               # "sources" section, but also filesystem trees built by othe
               # pipelines.
               "inputs": {},
               "options": {}
            }
         ]
      },
      {
         # Again only example name.
         "name": "build-fs-tree",
         # But this time the pipeline can use the previous one as a build pipeline.
         # The name:<something> is a reference format in OSBuild manifest v2.
         "build": "name:build",
         "stages": []
      },
      {
         "name": "do-sth-with-the-tree",
         "build": "name:build",
         "stages": [
            {
               "type": "",
               "inputs": {
                  # This is an example of how to use the filesystem tree built by
                  # another pipeline as an input to this stage.
                  "tree": {
                     "type": "org.osbuild.tree",
                     "origin": "org.osbuild.pipeline",
                     "references": [
                        # This is a reference to the name of the pipeline above.
                        "name:build-fs-tree"
                     ]
                  }
               },
               "options": {}
            }
         ]
      },
      {
         # In v2 the assembler is a pipeline as well.
         "name": "assembler",
         "build": "name:build",
         "stages": []
      }
   ]
}

Components of osbuild

OSBuild is designed as a set of loosely coupled or independent components. This subsection describes each of them separately so that the following section can describe how they work together.

Object Store

Object store is a directory (also a class representing it) that contains multiple filesystem trees. Each filesystem tree lives in a directory whose name represents hash of the pipeline resulting in this tree. In OSBuild, a user can specify a "checkpoint" which stores particular filesystem tree inside of the Object Store.

Build Root

It is a directory where OSBuild modules (stages and assemblers) are executed. The directory contains full operating system which is composed of multiple things:

• Executables and libraries needed for building the OS artifact (these are either from the host or created in a build pipeline).
• Directory where the resulting filesystem tree resides.
• Few directories bind-mounted directly from the host system (like /dev)
• API sockets for communication between the stage running inside a container and the osbuild process running outside of it (directly on the host).

Sources

Sources are artifacts that are downloaded from the Internet. For example, generic files downloaded with curl, or OSTree commits downloaded using libostree.

Inputs

Inputs are a generalization of the concept of sources, but this time an "input" can be both downloaded, as sources are, or generated using osbuild pipeline. That means one pipeline can be used as an input for another pipeline so you can have an artifact inside of an artifact (for example OSTree commit inside of a container).

APIs

OSBuild allows for bidirectional communication from the build container to the osbuild process running on the host system. It uses Unix-domain sockets and JSON-based communication (jsoncomm) for this purpose. Examples of available APIs:

• osbuild - provides basic osbuild features like passing arguments to the stage inside the build container or reporting exceptions from the stage back to the host
• remoteloop - helps with setting up loop devices on the host and forwarding them to the container
• sources - runs a source module and returns the result

What happens during simplified osbuild run

This section puts the above concepts into context. It does not aim to describe all the possible code paths. To understand osbuild properly, you need to read the source code, but it should help you get started.

During a single osbuild run, this is what usually happens:

1. Preparation
   1. Validate the manifest schema to make sure it is either v1 or v2 manifest
   2. Object Store is instantiated either from an empty directory or from already existing one which might contain already cached filesystem trees.
2. Processing the manifest
   1. Download sources
   2. Run all pipelines sequentially
3. Processing a pipeline (one of N)
   1. Check the Object Store for cached filesystem trees and start from there if it already contains parially built artifact
4. Processing a module (stage or assembler)
   1. Create a BuildRoot, which means initializing a bwrap container, mounting all necessary directories, and forwarding API sockets.
   2. From the build container, use the osbuild API to get arguments and run the module
5. If an assembler is present in the manifest, run it and store the resulting artifact in the output directory

Issues that do not fit into the high level goals

Bootstrapping the build environment

The "build" pipeline was introduced to improve reproducibility. Ideally, given a build pipeline, one would always get the same filesystem tree. But, to create the first filesystem tree, you need some tools. So, where go you get them from? Of course from the host operating system (OS). The problem with getting tools from the host OS this is that the host can affect the final result.

We've already had this issue many times, because most of the usual CLI tools were not created with reproducibility in mind.

The struggle with GRUB

The standard tooling for creating GRUB does not fit to our stage/assembler concept because it wants to modify the filesystem tree and create the resulting artifact at the same time. As a result we have our own reimplementation of these tools.

Running OSBuild from sources

It is not strictly required to run OSBuild installed from an RPM package. If you attempt to run osbuild from the command line in combination with an SELinux stage in the manifest it will most likely fail. For example:

$ python3 -m osbuild

The cause of error is a lack of proper labelling of the python3 executable, all stages and assemblers. Creating two additional files resolves the problem:

1. New entrypoint which will soon have the right SELinux label, let's call it osbuild-cli:

#!/usr/bin/python3

import sys

from .osbuild.main_cli import osbuild_cli as main


if __name__ == "__main__":
   r = main()
   sys.exit(r)

2. A script to relabel all the files that need it:

#!/bin/bash

LABEL=$(matchpathcon -n /usr/bin/osbuild)

echo "osbuild label: ${LABEL}"

chcon ${LABEL} osbuild-cli

find . -maxdepth 2 -type f -executable -name 'org.osbuild.*' -print0 |
   while IFS= read -r -d '' module; do
   chcon ${LABEL} ${module}
   done

Now run the script and use the entrypoint to execute OSBuild from git checkout.

osbuild-composer

It is a web service for building OS images. The core of osbuild-composer, which is common to all APIs, is osbuild manifests generation a job queuing. If an operating system is to be supported by osbuild-composer, it needs the manifest generation code in internal/distro directory. So far, we only focus on RPM based distributions, such as Fedora and RHEL. The queuing mechanism is under heavy development at the moment.

Interfacing with dnf package manager

We use our custom wrapper for dnf, which we call simply dnf-json, because its interface goes like this:

• Stdin - takes a JSON object
• Stdout - returns a JSON object
• Return code is used only for dnf-json internal errors, not for errors in the operation specified on the input. Those errors are reported in the returned JSON object.

Local API - Weldr

This API comes from the Lorax-composer project. osbuild-composer was created as a drop-in replacement for Lorax which influenced many design decisions. It uses Unix-Domain socket, so it is meant for local usage only. There are two clients:

• composer-cli
• cockpit-composer (branded as Image Builder in the Cockpit console)

Activate this API by invoking systemctl start osbuild-composer.socket. Systemd will create a socket at /run/weldr/api.socket.

Remote APIs - Cloud and Koji

Both are under heavy development.

Latest RPM builds

While developing osbuild and osbuild composer it is convenient to download the latest RPM builds directly from upstream. The repositories in the osbuild organization don't use any automation from Copr or Packit. Instead, the RPMs are built directly in the Jenkins CI and stored in AWS under the commit hash which allows anyone to download precisely the version built from a desired commit.

The URL is specified in the mockbuild.sh scripts in the osbuild and osbuild-composer repositories:

• mockbuild.sh in osbuild-composer
• mockbuild.sh in osbuild

And the final resulting URL is displayed in the Jenkins output (available only from Red Hat VPN).

Common trap: If you click on a link to a repo, such as:

http://osbuild-composer-repos.s3-website.us-east-2.amazonaws.com/osbuild-composer/rhel-8.4/x86_64/6b67ca34caf0ff9d31fabb398f50533c1e41c847/

you will get HTTP 403 because that's a directory and we don't allow directory listing. If you append a known file path, such as repodata/repomd.xml you will see that the repo is there:

http://osbuild-composer-repos.s3-website.us-east-2.amazonaws.com/osbuild-composer/rhel-8.4/x86_64/6b67ca34caf0ff9d31fabb398f50533c1e41c847/repodata/repomd.xml

Testing strategy

Let me start with a quote:

As the team obsessed with immutable test dependencies, how could we use ..

One osbuild developer in one PR fixing one more piece of infrastructure which could still change.

TODO: what do we test in each repo

TODO: rpmci, rpmrepo

osbuild-composer

This section provides a basic summary of the various types of testing done for osbuild-composer. Detailed information about testing can be found in the upstream repository.

Unit tests

There is pretty heavy mocking in the osbuild-composer codebase.

HTTP API is unit-tested without any network communication (there is no socket), only the HTTP request/responses are tested.

Integration tests

These test cases live under test/cases and each of them is a standalone script. Some of them invoke additional binaries which live under cmd if not specified otherwise.

1. api.sh [aws|azure|gcp] - test the Cloud API (running at localhost:443)

   • Provisions osbuild-composer and locally running remote worker.

   • Creates a request for compose and uploads the image to specified cloud provider. Currently AWS, Azure and GCP are supported.

   • The uploaded image is used for a VM instance in the respective cloud environment, booted and connected to via SSH. This is currently tested only for AWS and GCP.

   • Requires credentials for the respective cloud provider to work properly.

2. aws.sh

   Use osbuild-composer "the way we expect our customers to use it". That means provision osbuild-composer and use Weldr API to build an AMI image and upload it to EC2. Then use the aws CLI tool to spawn a VM from the image and make sure it boots and can be accessed.

   • Requires AWS credentials

3. base_tests.sh

   This script runs binaries implemented as part of osbuild-composer codebase in golang. It provisions osbuild-composer and then runs the tests in a loop.

   1. osbuild-composer-cli-tests - Weldr API tests using composer-cli

      • Executing composer-cli utility
      • Invoke multiple image builds

   2. osbuild-weldr-tests - Weldr API tests using golang library from internal/client

      • These live directly in the internal directory, which is a bit odd given that all other tests live under cmd/, but there might be a reason for this.
      • They invoke a build of a qcow2 image

   3. osbuild-dnf-json-tests - These make sure the interface to dnf still works

      • This binary will execute dnf-json multiple times and it will also run multiple dnf depsolving tasks in parallel. It is possible that it will require a high amount of RAM.

      • My guess would be at least 2GB memory for a VM running this test.

   4. osbuild-auth-tests - Make sure the TLS certificate authentication works as expected for the koji api and worker api sockets.

      • A certificate authority is created for these tests and the files are stored in /etc/osbuild-composer-test/ca
      • The certificates live in the standard configuration directory: /etc/osbuild-composer
      • Multiple certificates are created:
        • For osbuild-composer itself (let's say a "server" certificate)
        • For osbuild-worker
        • For a client application, in this case the test binary
        • For kojihub

4. image_tests.sh

   Possibly the most resource-hungry test case. It builds an image for all supported image types for all supported distributions on all supported architectures (note that every distro has a different set of aches and arches have different set of supported types, e.g. there is no s390x image for AWS because there is no such machine). The "test cases" are defined in test/cases/manifests and they contain a boot type (where to spawn the VM), compose request (what to ask Weldr API for), and finally the expected manifest. Osbuild-composer should generate the same manifest, build the image successfully, optionally upload it to a cloud provider, boot the image, and finally verify it is running.

   • Require AWS, Openstack, and Azure credentials

5. koji.sh

   Runs a koji instance in a container. It sets up certificates and Kerberos KDC because osbuild-composer uses Kerberos to authenticate with Koji.

6. ostree.sh

   This test case creates an OSTree commit, boots it, then it creates a commit with an upgrade on top of the previous commit and makes sure the VM can upgrade to the new one.

   • Uses libvirt to run the VM

7. qemu.sh

   Create a qcow2 image and boot it using libvirt.

Leaking resources

The cloud-cleaner binary was created to clean up all artifacts (like images, but also registered AMIs, security groups, etc.) that could be left behind. Not all executables in our CI have proper error handling and clean up code and what is even worse, if Jenkins fails and takes down all running jobs, it is possible that the clean-up code will not run even if it is implemented.

Possibly leaking resources:

1. api.sh test case:

   • Image uploaded to AWS, Azure or GCP

2. aws.sh test case:

   • Image uploaded to EC2

   • VM running in EC2

Glossary