Table of Contents:

This guide will explore a number of workflows for retrieving, patching, building and packaging the Apertis kernel. This guide will suggest workflows both utilising the Debian infrastructure on which Apertis is built and a workflow that eschews the Debian infrastructure and thus is usable by third party integrators who may need access to a patched kernel tree for integration, but who’s workflow would make using the Apertis infrastructure difficult.

This document is targeted towards development of Apertis and where possible, we would strongly advise using the Apertis SDK image and the Debian tooling provided for it, as this workflow will be more efficient (“option 1” where present). We provide a guide to get to a built kernel (though not packaged) for those not able to use the provided infrastructure (“option 2”).

Required knowledge

Prior experience with the following topics will help with understanding the workflows presented below:

• Familiarity with using git
• Building and developing the Linux kernel
• Debian packaging
• Familiarity with basic Linux commands

Building locally patched kernel

Generating patched kernel source

Apertis provides a version of the Linux kernel, which is based on the upstream kernel with a number of patches applied to it and packaged using Debian packaging. We want to use this version of the kernel as a starting point for our modifications. As with most Debian packages, the upstream source is stored separately from any distribution specific modifications, scripting and metadata. In the case of the Apertis kernel, all the required components are stored in a single git repository, however on different branches. To retrieve the git repository run the following git command:

$ git clone https://gitlab.apertis.org/pkg/linux
$ cd linux

We will create a fresh branch, based on the branch of interested (in this case, we’re interested in the v2023: Apertis release) into which we will create any changes - that way we have the original branch available which will help with updating later. We will assume we have a project called “foo” for the sake of this guide.

$ git checkout -b foo/v2023 origin/apertis/v2023

Option 1: Apertis tools

We are going to use Debian’s git-buildpackage (gbp) to generate a patched source tree in a git branch, with the existing Apertis patches applied as commits.

First ensure that you have gbp installed:

$ sudo apt update
$ sudo apt install git-buildpackage

We can then use the following command to generate the source tree:

$ gbp pq import

Option 2: Manually constructing kernel tree

If git-buildpackage is not available, it is possible to use git-quiltimport to generate a patched kernel source tree (abet loosing out on a lot of the automation that the git-buildpackage approach provides. To get a patched kernel setup in a branch, run the following commands:

$ git quiltimport --patches debian/patches --author "Unknown <unknown@example.com>"

Adding modifications

The extracted and patched kernel source can be used as a baseline for porting or developing additional features. Existing patches can be imported using git am, which will import each patch as a separate commit. If developing additional drivers/board support it is advisable to break down this work into well explained atomic commits.

As a developer you will need to build and test your modifications during development prior to packaging. There is no single recommended way to achieve this as the most efficient way will depend on the development setup available to you. The suggested approach is to simply build the kernel image in the correct format and copy it to a location where the firmware can be made to boot the kernel.

In the kernel source, which was cloned to a directory called linux, add the file drivers/misc/trivial.c containing the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

// SPDX-License-Identifier: GPL-2.0

#include <linux/init.h>
#include <linux/module.h>

static int trivial_init(void)
{
    pr_info("Trivial module init\n");

    return 0;
}
module_init(trivial_init);

static void trivial_exit(void)
{
    pr_info("Trivial module exit\n");
}
module_exit(trivial_exit);

MODULE_AUTHOR("Martyn Welch <martyn.welch@collabora.co.uk");
MODULE_DESCRIPTION("Trivial Driver");
MODULE_LICENSE("GPL");

The following section needs to be patched into the configuration system in drivers/misc/Kconfig (This needs to be added somewhere between the menu ... and endmenu lines):

1
2
3
4

config TRIVIAL
        tristate "Trivial driver"
        help
          Trivial example driver to show how to integrate code into the kernel.

Finally we need to add the following to the makefile drivers/misc/Makefile:

1

obj-$(CONFIG_TRIVIAL)       += trivial.o

Add and commit these changes to git:

$ git add -f drivers/misc/trivial.c drivers/misc/Kconfig drivers/misc/Makefile
$ git commit -m "Add trivial driver"

Building

Now that we have made changes to the kernel, it would be advisable to build the kernel to ensure that the changes which have been made don’t introduce build failures.

We can build the configs used by Apertis with the following command:

$ make -f debian/rules setup

This command generates the configuration files that can be used for various platforms under `debian/config`. We will us the standard 64-bit x86 configuration as an example here, tweaking it to build our additional driver and building in a separate directory to keep the source tree clean:

$ mkdir ../test-build
$ cp debian/config/amd64/config ../test-build/.config
$ echo "CONFIG_TRIVIAL=m" >> ../test-build/.config
$ yes "" | make O=../test-build oldconfig
$ make -j $(nproc) O=../test-build

If cross-compilation is needed for your tests, ensure the required cross-compiler is installed and pass both the desired architecture and cross-compiler prefix to make as required. For example, for 32-bit ARM:

$ yes "" | make ARCH=arm O=../test-build oldconfig
$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- -j $(nproc) O=../test-build

Using with NFS root file systems

Now that we have a binary kernel, we can boot the board. Assuming the config is setup correctly (you’ll need the relevant network and other drivers built in to the kernel, along with NFS root support) you can boot with a pre-built NFS root. The Apertis provides rootfs for various architectures, for example the x86_64 (amd64) Apertis v2021.0 nfs root can be found here:

https://images.apertis.org/release/v2023/v2023.1/amd64/nfs/

Packaging

Once local changes have been made, they can be added to the Apertis packaging so that the modifications can be provided in a packaged kernel.

Updating patch series

So far we have a git repository holding a number of branches, including:

1. A packaging branch with the contents of the debian/ directory.
2. A pristine source branch, holding the broadly unaltered upstream source.
3. A local branch, which has the patches held in the packaging branch applied to the pristine source as git commits.

We’ve made one or more changes that have been applied to the patched source. We now want these changes to be added back into the packaging branch so that we can push these changes into the packaged kernel. We can do this with the following command:

$ gbp pq export --no-patch-numbers

This will switch us back to the packaging branch, with the changes that we’ve committed previously applied as patches under debian/patches/.

Depending on how “clean” the existing patch series is, at this point gbp may make a surprising number of changes to the existing patch series. Most of these changes will be to tweak metadata and formatting to match that preferred by the current version of git. However reordering of the files patches has been seen and this makes it hard to determine whether unintended changes have occurred. In addition, comments in the series file (debian/patches/series) will have been removed. One or more additional patches will also be present in debian/patches/ (the local changes) and will have been added to the series file.

Before committing the new changes we will revert the extraneous changes. Whilst these may make the series look cleaner, applying these changes makes it hard to follow what modifications were actually made. These changes also increase the delta between the modified Apertis kernel and the upstream Apertis kernel (and for that matter the upstream from which the Apertis kernel is derived) making it harder to update at a later date. These changes can be reverted with the following command:

$ git checkout HEAD debian/patches/

This will leave us just with the additional patch that was added in debian/patches/:

$ ls debian/patches/*.patch
debian/patches/Add-trivial-driver.patch

The debian/patches/ directory contains patches that get applied to the original source. Typically Debian packages provide a single set of patches and a single series file that informs the patch management tool (typically quilt) the order in which to apply the patches. The kernel is a little different in this regard. Where as most packages are built for a handful of fairly generic architectures, the kernel is built multiple times with differing configurations and even patches applied. In order to accommodate this and to make it easier to track which patches are applicable to which platforms/features the kernel uses configuration fragments, split into generic and more specific fragments that are combined depending on the architecture and even specific board for which we are building (this is the reason we needed the make -f debian/rules setup command in the building section).

For our custom kernel we are going to build a “standard”, non-realtime, kernel. Create a directory debian/patches/<project_name> (where <project_name> is the name of the project). The patch names and paths (relative to the debian/patches/ directory) should be appended to the debian/patches/series file in the order in which they need to be applied.

For the above example, assuming the project name is foo, make the directory debian/patches/foo and move the generated patch:

$ mkdir -p debian/patches/foo
$ mv debian/patches/Add-trivial-driver.patch debian/patches/foo/

Add the following to the end of debian/patches/series:

# Trivial driver example
foo/Add-trivial-driver.patch

These changes need to be committed into the git repository:

$ git add -f debian/patches/foo/ debian/patches/series
$ git commit -m "Add trivial driver kernel patch"
[foo/v2023 d246c910d5] Add trivial driver kernel patch
 2 files changed, 68 insertions(+), 1 deletion(-)
 create mode 100644 debian/patches/foo/Add-trivial-driver.patch

Updating kernel package configuration

Adding the driver into the kernel is not enough to make it build, the config option that we have added (CONFIG_TRIVIAL) needs to be enabled. In this instance this can be done to build the driver into the kernel, or a a kernel module that can be loaded into the kernel at runtime.

The kernel is built for a large number of platform variants, whilst some of these have significant similarities to each other, and thus can use the same binary kernel, many different builds need to be performed to support boards which have different architectures. Still, there are certain kernel configuration options, typically those that aren’t linked to specific hardware, that we want to build for all platforms and those that are only required for specific hardware. In order to accommodate this without needless repetition of the kernel configuration, the configuration is built from a number of configuration fragments found in the packaging repository under debian/config/.

The main fragment for all platforms is debian/config/config, under the debian/config/ directory there are a number of directories for the individual architectures for which the kernel can be built. Under each of these directories there is another config file that is for the architecture specific options. There may also be one or more config.<sub_architecture> config files for configuration options specific to a sub-architecture build.

We will treat the trivial driver as architecture independent and build it as a module by running the following to add CONFIG_TRIVIAL=m to the end of debian/config/config:

echo "CONFIG_TRIVIAL=m" >> debian/config/config

This change will need committing:

$ git add -f debian/config/config
$ git commit -m "Enable trivial driver for all builds"
[foo/v2023 1e913d77ee] Enable trivial driver for all builds
 1 file changed, 1 insertion(+)

Update changelog

As we have now modified the Debian metadata, we should update the changelog. This is important as the version specified in the changelog determines that which will be used when creating the binary packages and thus whether the newly built kernel is considered as an upgrade from that already installed should you attempt an upgrade with apt.

We can use git-buildpackage to automate much of this process. First though we want to ensure that git is configured with the correct user name and email:

$ git config --local user.name "User Name"
$ git config --local user.email "user@example.com"

Once these are set we can run the following:

$ gbp dch --git-author -l foo -R

We pass -l foo so that the local suffix foo is added to the version number rather than incrementing the Apertis version number it’s self. Incrementing the Apertis version number should only be done by the Apertis developers as incrementing it would collide with the version numbering used by the next Apertis change.

We also pass -R which marks the changes as released. If we don’t do this, the first line of the changelog entry would list as “UNRELEASED” rather than “apertis”. This might be done if mulitple changes are expected to be added to the next release, with the changelog entry being updated each time and only marked as released once all changes are applied.

The changelog (debian/changelog) will be updated with an entries based on the subject of the added commits:

$ git diff
diff --git a/debian/changelog b/debian/changelog
index 08ea067978..630a2fc3be 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,11 @@
+linux (6.1.20-1+apertis4~v2023foo1) apertis; urgency=medium
+
+  [ User ]
+  * Add trivial driver kernel patch
+  * Enable trivial driver for all builds
+
+ -- User Name <user@example.com>  Mon, 26 Jun 2023 13:28:19 +0100
+
 linux (6.1.20-1+apertis4~v2023) apertis; urgency=medium

   * Merge apertis v2024dev1

Now that we have a new version, we also need to update a number of kernel configuration files to reflect this change in version number. This can be done with the following command:

$ debian/apertis/update-metadata

The scope of the changes that this will introduce will depend a little on what changes have been made to the debian metadata. In this instance it’s relatively minor, changing the following files:

• debian/build/version-info
• debian/config.defines.dump
• debian/control.md5sum
• debian/rules.gen

These need to be committed to git, along with the changelog change:

$ git add -f debian/changelog debian/build/version-info debian/config.defines.dump debian/control.md5sum debian/rules.gen
$ DEBIAN_PACKAGE=$(dpkg-parsechangelog -S Source)
$ DEBIAN_VERSION=$(dpkg-parsechangelog -S Version)
$ git commit -s -m "Release ${DEBIAN_PACKAGE} version ${DEBIAN_VERSION}"

Building package

First we need to ensure that the packages required for the build installed:

$ sudo apt update
$ sudo apt build-dep linux

The following script can be used to perform the build (save as build-debian-kernel one directory above the kernel source):

#!/bin/sh
set -x

PARALLEL=$(/usr/bin/nproc)

export DEB_BUILD_PROFILES="pkg.linux.notools nodoc noudeb cross nopython"
export DEB_BUILD_OPTIONS=parallel=$PARALLEL

if [ -n "$1" ] ; then
  ARCH="-a$1"
else
  ARCH=""
fi

gbp buildpackage \
  --git-debian-branch=foo/v2023 \
  --git-ignore-new \
  --git-prebuild='debian/rules debian/control || true' \
  --git-builder='debuild -eDEBIAN_KERNEL_USE_CCACHE=1 -i -I' \
  --git-export-dir='~/build' \
  ${ARCH} \
  -B -d -uc -us

Make the script executable with:

$ chmod +x ../build-debian-kernel

It can be run with the required architecture as a command line option. This will limit the build to the supplied architecture, rather than building for all available architectures, so using this option is recommended.

The script uses git build package (gbp) to generate the binary kernel packages. As mentioned before gbp needs to be able to find a original source archive. We use --git-debian-branch=foo/v2023 to tell gbp to use the branch foo/v2023 to generate an upstream tarball.

We also specify a build area via the --git-export-dir option. As we have specified this as ~/build, which causes the build and the build processes artifacts to be extracted/stored in a directory called build in the users home directory.

Let’s build the example for amd64 (substitute amd64 for the desired architecture if your requirements differ):

$ mkdir ~/build
$ ../build-debian-kernel amd64

The built kernel packages will be available in ~/build.

Installing package and testing

To test download the v2023 SDK and boot it in virtualbox.

We can see that it’s running a 6.1 kernel:

user@apertis:~$ uname -a
Linux apertis 6.1.0-7-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.20-1+apertis4~v2023 (2023-04-2 x86_64 GNU/Linux

Copy the new kernel deb file to the SDK (this can be done via ssh) and install it:

user@apertis:~$ sudo dpkg -i linux-image-6.1.0-7-amd64_6.1.20-1+apertis4~v2023foo1_amd64.deb
(Reading database ... 113774 files and directories currently installed.)
Preparing to unpack linux-image-6.1.0-7-amd64_6.1.20-1+apertis4~v2023foo1_amd64.deb ...
Unpacking linux-image-6.1.0-7-amd64 (6.1.20-1+apertis4~v2023foo1) over (6.1.20-1+apertis4~v2023bv2023.0db1) ...
/etc/kernel/postrm.d/zz_removekernel:
/boot/efi is a mountpoint
Setting up linux-image-6.1.0-7-amd64 (6.1.20-1+apertis4~v2023foo1) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-6.1.0-7-amd64
...

Reboot and we boot with the new kernel:

user@apertis:~$ uname -a
Linux apertis 6.1.0-7-amd64 #1 SMP PREEMPT_DYNAMIC Apertis 6.1.20-1+apertis4~v2023foo1 (2023 x86_64 GNU/Linux

We now have the trivial driver available:

user@apertis:~$ sudo modinfo trivial
filename:       /lib/modules/6.1.0-7-amd64/kernel/drivers/misc/trivial.ko
license:        GPL
description:    Trivial Driver
author:         Martyn Welch <martyn.welch@collabora.co.uk
depends:
retpoline:      Y
intree:         Y
name:           trivial
vermagic:       6.1.0-7-amd64 SMP preempt mod_unload modversions

Which we can load and unload, receiving the log messages when we do (we need to increase the log level seen on the console to get the messages):

user@apertis:~$ sudo dmesg -WH &
[1] 987
user@apertis:~$ sudo modprobe trivial
[Jun28 15:50] Trivial module init
user@apertis:~$ sudo rmmod trivial
[  +7.967435] Trivial module exit
user@apertis:~$ fg
sudo dmesg -WH
^C

Maintenance

It is important to consider the effort required to effectively maintain a modified kernel within the Apertis ecosystem. There are a number of approaches that can be taken and depending on the modifications made to the Apertis kernel, one or more are likely to be suitable.

Upstreaming changes

It may be viable for some or all of the changes made to be upstreamed to the Apertis kernel. For any change to be considered for this approach it’s likely that (at least) the following criteria would need to be met:

• Changes must be signed off
• Changes follow the kernel coding style
• Changes need to be provided as a set of well formatted patches
• The changes are either:
  • A generic bugfix
  • Controlled via kernel configuration options so as to be easily disabled. That is to say, they must not impact the usability of the Apertis kernel for other Apertis users. A good way to achieve this is to ensure the changes are self contained.

The advantage to upstreaming the changes is that such changes will no longer require to be actively maintained as this will be carried out by the Apertis team.

Local maintenance of modified Apertis kernel

It is likely that some Apertis users will make changes to the kernel that are either inappropriate to be pushed into the upstream Apertis kernel or for which they are unable or willing to make available in the upstream kernel. Such changes will need to be managed by the team responsible for them. The Apertis kernel, as with other Apertis packages, will be updated over time both to fix bugs and to stay up-to-date with changes to the Linux kernel. Those who are using a locally modified kernel will need to port their changes over to the updated kernel. It is expected that these modifications will be stored in a fork of the Apertis kernel packaging repository.

The following guidelines will help minimise the effort required to successfully achieve this:

• As with upstreaming, ensure patches do one thing, are as self contained as possible and have a good description of the intent of the patch in the commit message. Well described patches are easier to follow and understand at a later date. If patches are broken down to achieve a single change it is easier to test each patch as they are ported.

• Ensure that commits don’t break the kernel build. Ensuring patches build provides another data point to ensure correct porting after each patch is applied to the updated tree. It also retains the ability to bisected the kernel which can be invaluable when tracking down bugs.

We will consider 2 cases, a minor Apertis update (for example 4.19.37-5apertis4 to 4.19.37-5apertis5) and a major update (for example 4.19.37-5apertis4 to 4.19.52-1apertis1).

Updating after minor upstream changes

It is expected that minor updates represent no change in the upstream sources. Unless the local patches happen to collide with the additional upstream patches, then usually the patches will just apply cleanly. Generally for minor changes, adding the local patches into debian/patches, updating debian/patches/series and the changelog is all that is required.

Updating after major upstream changes

Major updates represent a change in the upstream source. There is a greater chance that such changes will require changes to the patches in the series. Some changes may also be applied upstream (should they have been back-ports or previously submitted upstream for inclusion). These need to be left out and the remainder ported to the new kernel.

An example of how to achieve this:

• Generate patched kernel source of locally modified version using gbp pq import.
• Fetch and checkout latest packaging branch from Apertis.
• Generate patched kernel source of new version using gbp pq import.
• Cherry pick additional patches from original locally modified tree, performing any modifications necessary to have each patch apply and build.
• Use gbp pq export to add local changes into packaging patch series.
• Use gbp dch to update changelog, tweak as necessary.

Updating the kernel from an upstream stable git branch

The Apertis release flow stipulates that each Apertis release should include the latest mainline kernel LTS release. Due to Debian’s release cadence being approximately half that of Apertis’, there are 2 Apertis releases for each Debian stable release and the latest LTS is not always packaged by Debian. As a result the Apertis development team expected that they will need to pull from the mainline kernel LTS tree to satisfy it’s requirements. For example, Apertis v2024 is tracking a newer version of the Linux kernel (6.6.x) than Debian Bookworm (6.1.x) on which it is based.

This process may be necessary if you need to maintain a kernel from a revision other than one provided by Apertis.

In such cases, special care needs to be taken to update packages from their respective upstreams:

• Using the GitLab web UI, check to ensure that the relevant update branch exists, in the case of the Apertis v2020 release, kernel updates should be made on the apertis/v2020-security branch. Create the required branch if it doesn’t exist.

• Clone the existing package from Apertis GitLab and checkout the relevant branch:

  $ APERTIS_RELEASE=v2020
  $ git clone -b apertis/${APERTIS_RELEASE}-security git@gitlab.apertis.org:pkg/linux.git linux-apertis
  

• Separately clone the Linux kernel LTS repository and checkout the relevant stable branch:

  $ UPSTREAM_RELEASE=5.4
  $ git clone -b linux-${UPSTREAM_RELEASE}.y git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git linux-stable
  

• Create a gbp pq patch branch and switch back to normal branch. The patch branch will be needed when rebasing the Debian patches:

  $ cd linux-apertis
  $ gbp pq import
  $ gbp pq switch
  

• Determine the latest stable release and derive the Apertis version to use. To ensure collisions don’t occur with any future Debian releases, we will mark the release as the zeroth “Debian” release (-0) and first Apertis pre-release (+apertis1):

  $ KERNEL_RELEASE=`git -C ../linux-stable describe`
  $ RELEASE="${KERNEL_RELEASE#v}-0+apertis1"
  

• Add a changelog entry for newest linux-5.4.y kernel release (this is needed to get genorig.py to use the right release):

  $ DEBEMAIL="User Name <user@example.com>" dch -v "$RELEASE" ""
  $ git add -f debian/changelog
  $ git commit -s -m "Add changelog entry for update to $RELEASE"
  

• Run debian/bin/genorig.py to generate debianised kernel source tarball:

  $ debian/bin/genorig.py ../linux-stable
  

• Import the updated source tarball:

  $ git branch upstream/linux-${UPSTREAM_RELEASE}.y origin/upstream/${UPSTREAM_RELEASE}.y
  $ gbp import-orig --upstream-branch=upstream/linux-${UPSTREAM_RELEASE}.y --debian-branch=apertis/${APERTIS_RELEASE}-security <path to orig tarball>
  

• Rebase the Debian patches on the new kernel version:

  $ gbp pq rebase
  

  This may require some manual intervention following the normal git rebasing process.

• Export updated patch series:

  $ gbp pq export
  

• Tweak debian/patches/series file so that it retains the comments that gbp pq wants to drop. If no patches have been dropped during rebase then:

  $ git checkout debian/patches/series
  $ git add -f debian/patches
  $ git commit -s -m "Update the debian patches for $RELEASE update"
  

• Update changelog to released state (set distribution to apertis), document any patches that have been dropped as a result of the update, update the debian metadata and scripts to match the updated kernel version and create new commit:

  $ dch -D apertis
  $ git add -f debian/changelog
  $ debian/apertis/update-metadata
  $ git commit -s -m "Release linux version $RELEASE"
  

• Run dpkg-buildpackage to generate the debian packaging, this is needed to use the tools to update the pristine-lfs branch:

  $ mkdir ../build
  $ gbp buildpackage --git-ignore-new \
      --git-debian-branch=apertis/${APERTIS_RELEASE}-security \
      --git-prebuild='debian/rules debian/control || true' \
      --git-export-dir='../build' \
      --no-sign -us -S
  

• Import the generated kernel tarball into pristine-lfs branch with import-tarballs:

  $ git clone git@gitlab.apertis.org:infrastructure/packaging-tools.git ../packaging-tools
  $ git branch pristine-lfs origin/pristine-lfs
  $ ../packaging-tools/import-tarballs ../build/<kernel .dsc file>
  $ git push origin pristine-lfs
  

• Push kernel update to a branch to be reviewed:

  $ git push origin apertis/${APERTIS_RELEASE}-security:wip/${GITLAB_USER}/${KERNEL_RELEASE}-kernel-update
  

Version numbering

The updated nature of the package is reflected in the version number given to the package and this will have an impact on the version number that should be assigned to the newly rebased tree (new upstream, plus or or more local changes). As an example will assume 3 local revisions have been made since the last rebase on 4.19.37-5+apertis4, following our previous example the version would expected to be 4.19.37-5+apertis4foo3, should a new Apertis release be made (4.19.37-5+apertis5) and assuming that the local changes are still needed, then the local version with these inclusions with constitute the first local release on top of the new upstream, thus becoming 4.19.37-5+apertis5foo1.