Table of Contents:

This document aims as a single resource for all information related to Apertis Release Process. It covers the Major Release process as well as the Point Release process.

Apertis Infrastructure

The Apertis project is hosted on a couple of infrastructure services, which are tightly coupled to each other, providing end-to-end automation. This includes:

• Landing new packages into Apertis
• Landing new security and general updates into Apertis
• Landing downstream changes into Apertis

This document stands as a comprehensive guide to all package related processes as outlined above.

At a higher level, Apertis infrastructure includes:

• GitLab instance for source code hosting, CI, Automation, Image building etc
• OBS (Open Build Service) for package building
• aptly for APT repository management
• images.apertis.org for Apertis image hosting
• LAVA for Quality Assurance Testing

Development/Preview Releases

As described in Release flow and product lines Apertis ships three types of releases: development, preview and stable. This section describes the release process for development and preview releases.

Development/Preview Release Steps

As a high level overview, these are the steps to be executed during the release cycle, to perform a release:

• Keep release schedule wiki page updated
• Announce soft feature freeze
• Check available disk space on OBS Server
• Full archive rebuild to ensure all the packages are buildable from source
• Announce hard feature freeze
• Update the freshness status of all the design documents
• Write the release notes
• Confirm that no release blocker issue is present
• Publish Release Candidate (RC1) images and announce hard code freeze
• RC images validation (QA)
• Check available disk space on OBS Server
• Perform Release Branching
• Publish Release
• Update channels for Apertis packages Dashboard

Release Branching

Once the Release Preparation is complete, the next step in the Release Process is to run the Release Branching steps.

At a higher level, Release Branching includes the following steps which needs to be run in close co-ordination.

• Branching all package repositories with the new release
• Branching all image recipes
• Branching all test recipes
• Branching docker image recipes
• Branching all OBS repositories

Release Branching through GitLab CI

The release process has been semi-automated with the GitLab CI framework. The following screenshots/notes will give a run through of the steps, to make a release.

Some terms in regard to release branching:

• RELEASE: The current release codename
• NEXT_RELEASE: The new release codename that should be branched to
• PREV_STABLE_RELEASE: The previous stable release codename. Only used when branching a new stable release (i.e. not a dev or pre release)
• PURPOSE: Use string branching to instruct CI that it is a release procedure job
• NOACT: Set to 0 to trigger the release

By default, the release procedure jobs will run in dry-run mode, i. e. with NOACT set to 1. After running the release process in dry-run mode, go through the job logs to uncover potential issues. When ready, re-run the CI pipeline jobs with NOACT set to 0 to actually apply the changes.

This is the initial step to trigger release branching. Here:

• Specify the current RELEASE to be branched, i.e. v2025dev2
• NEXT_RELEASE is the name of the new release that is to be branched, i.e. v2025dev3
• If the new release to be branched is a stable release (i.e. not a dev or pre release), then PREV_STABLE_RELEASE needs to be defined, i.e. when branching v2025, this variable will be defined on v2024
• PURPOSE is set to branching to tell the CI Pipeline that this is a release branching job.

Note: NOACT has been deliberately left out in this example screenshot. So, in this case, the entire procedure will run in dry-run mode. To run in effective mode, pass NOACT with value 0

The above example complete the semi-automated aspect of the Release Branching steps. For more information, please check Release Branching examples.

project pkg/foo: settings: save changes -> error: 400: {'base': ["Could not change HEAD: branch 'apertis/v2024dev3' does not exist"]}

Post CI Manual Steps

After the completion of the Release Branching CI Jobs, a certain set of manual steps need to be performed, which are outlined below.

• Add a directory for the release to the LAVA NFS path at: /srv/images/public/lava/nfs/ on host images.apertis.org. The LAVA test jobs pick the kernel and initrd image from the release’s directory path. Refer to the README for details.
• Instantiate a new timer to rotate images for the new release (eg. On host aurora, run command systemctl enable --now rotate-images@v2024.timer).
• Update the ci-package-builder to target the next release.
• Update the gitlab-rulez recipe to point to the new development release.
  • Set the default development branch for all git projects.
  • Set the protection settings onto all apertis/* branches, which were unprotected in the initial release steps.
• Update the dashboard’s channels to track the new release.
• Trigger a build pipeline on the newly created branch of base-files package, after the new docker images have been built.
  • This step should automatically update the release specific attributes in the new package and auto propose the changes through a new merge request.
  • Validate the newly created pipeline ensuring that it generates the right set of changes with respect to the new release.
  • Validate the new package is built proper and pushed to OBS and the APT repositories.

Stable Point Release

Point Releases for the Stable Release happen on a timely cadence as defined in the Release Schedule. A Stable Point Release includes:

• Security Updates to software packages (funneled through $RELEASE-security). These include security vulnerability fixes.
• Normal updates to software packages (funneled through $RELEASE-updates). These include bug fixes to existing version of software packages.

Stable Point Release Steps

• Keep release schedule wiki page updated
• Announce soft feature freeze
• Announce hard feature freeze
• Check available disk space on OBS Server
• Perform Release folding
• Full archive rebuild to ensure all the packages are buildable from source
• Write the release notes
• Confirm that no release blocker issue is present
• Publish Release Candidate (RC1) images and announce hard code freeze
• RC images validation (QA)
• Publish Release
• Update channels for Apertis packages Dashboard

Several of the steps above were already described in the previous section, the following section will describe those steps specific for stable point releases.

Folding Stable Point Release

Once the Release Preparation is complete, the next step in the Release Process is to run the Release Folding steps.

At a higher level, Release Folding is the process of merging the updates from security and updates branches into the main ones, and includes the following steps which needs to be run in close co-ordination.

• Fold changes in security and updates branches into the main ones
• Confirm that newer versions of the packages are available in the main repositories
• Drop security and updates branches for the release in GitLab
• Drop security and updates repositories for the release in OBS

These changes are performed by the fold-security-update-branches script from the apertis-infrastructure/release-scripts/gitlab/, which automates many of the steps in a Stable Point Release Preparation:

• List the packages in the :security, :updates and other similar repositories
• Fast Forward Merge --ff-only the respective security and updates branches on Git. For example, apertis/v2024-security and apertis/v2024-updates should be merged back to apertis/v2024. Note: Not able to perform a fast forward merge means something got broken and needs to be investigated and fixed.
• Poke the pipeline and ensure that all the new packages (from the merged back changes to apertis/v2024) get built on OBS and land into the main repository.
• Drop the updates, security branches (Eg. apertis/v2024-updates, apertis/v2024-security)
• Delete the packages from the updates, security etc repositories on OBS (Eg. apertis:v2024:updates:target, apertis:v2024:security:development)

The script can be run in a semi-automatic mode through a GitLab CI pipeline. Similarly as with release branching, the pipeline runs in the dry-run mode by default. To actually perform the changes, set NOACT variable to 0 when triggering the pipeline.

The pipeline is organised in the following stages:

1. Fold. In this stage, the script submits merge requests to all packages with updates in updates and security. Packages where these merge requests can be automatically merged, are set to auto-merge once the pipelines succeed.
2. Manual review. Since the script cannot properly handle all updates yet, manual merges are often necessary. See below for more details.
3. Cleanup. After performing the reviews and verifying that all packages have been built and published in the main repositories, run this step to clean-up the GitLab branches and public apt repositories.

To make the review process easier, each component is processed in a separate job in the “fold” child pipeline:

At the end of each job’s logs, there’s a summary with status of each package and links to the merge requests:

The meaning of each status:

• CLEAN: this package needs no further processing other than possibly a clean-up
• NEEDS-MERGE: this package is out of sync and needs a merge, but the dry-run mode is in force
• MERGE: a merge request has been submitted for this package
• SKIP: a package was found in one of the OBS repositories, but no corresponding branch was found in GitLab
• ERROR: an error occurred during processing this package, check the detailed logs above

A known situation when the pipeline submits a merge request that GitLab cannot process is when a new package has been introduced in updates or security. In such cases, the merge request is created against a non-existent branch, and it cannot be edited or merged using the GitLab UI, and the branch needs to be manually pushed.

Common tasks during release

OBS disk space

OBS is the build server used by Apertis to build the packages before they are published. During the process of the release it is heavily used by different stages, like archive rebuild, branching and folding.

Archive Rebuilds

Archive Rebuilds are done before a release to ensure that the entire repository of packages are in good shape, can build proper, and that all their inter-dependencies are satisfied. The recommended timeline for an archive-wide rebuild is right after the Feature Freeze. Once a freeze is in place, we don’t anticipate new changes to be introduced, and thus this is the optimal time to re-build all the packages slated for the release.

Note: An archive-wide rebuild is a time consuming task. Depending on the build infrastructure, it can take large amount of time. Thus, it is important to account for this time and plan a rebuild accordingly to ensure we have proper build images for the first Release Candidate, which is typically 2 weeks from the Feature Freeze date.

Some notes about the Archive Rebuild task

• Rebuilds are usually not required to be triggered manually.
• Archive wide rebuild of packages should never be triggered on the main (i.e. default) repositories. Use separate rebuild repositories and delete them once the rebuilds complete. This is done by editing the project’s Meta Configuration:
  • Disable publishing of the rebuild repositories by adding the following snippet:

     <publish>
       <enable repository="default"/>
       <disable repository="rebuild"/>
     </publish>
  

  • Create the rebuild repositories with this snippet:

     <repository name="rebuild" rebuild="local" block="never">
       <path project="apertis:v2024:target" repository="default"/>
       <arch>x86_64</arch>
       <arch>armv7hl</arch>
       <arch>aarch64</arch>
     </repository>
  

  • Each rebuild repository depends on its default variant enabling it to have binary packages to bootstrap the rebuild. Without this dependency, packages will be in an unresolvable state on OBS.
  • Ensure that all rebuild repositories have rebuild="local" block="never" in the project meta configuration. This is an important step needed to ensure that packages don’t get into cyclic rebuilds everytime a relevant library is built.
  • The rebuild will start as soon as OBS has computed the repositories’ state. Otherwise, rebuilds can be triggered with: osc rebuild apertis:v2024:target --all -r rebuild
  • Once all packages have been successfully built, delete rebuild repositories.
• Never trigger an archive wide rebuild on the primary default repositories.
• In case of last minute changes in a package which could trigger dependencies issues, it is possible to trigger the rebuild of packages in the direct dependency chain by setting rebuild="direct". This will instruct OBS to compute the list of packages that directly depend on new version of packages and schedule a rebuild of them. It is also possible to trigger the rebuild of the whole dependency chain by setting rebuild="transitive", however this will trigger a big amount of rebuilds so it should be used with caution.

Publish Release Candidate

• Run the copy-release script on the images server (aurora) to prepare the RC.
• copy-release -v 20201203.0116 v2021pre.0rc1
• copy-release -v -i 20201203.0116 v2021pre.0rc1
• First for the public images and then second for Internal images (-i option).
• The source folder is picked from the daily image builds.
• Send release announcement for the release.

Build Suffix

For every release (Developer, Preview, Product), we add a build suffix string to the packages, which relates to the release name of Apertis. The build suffix gets added to every built .deb package. Having a build suffix helps determine which Apertis release the .deb package was built for.

The suffix string is constructed of: b + Release Name + b<B_CNT>

• Where, the initial b is for backward compatibility ensuring the new suffix string can work well together with older release packages
• String Release Name will refer to the Apertis release’s name
• String, b<B_CNT> refers to being a binary build along with the build count

For example, for an Apertis Developer Release of v2025dev0, we add string bv2025dev0b<B_CNT> to the project configuration on OBS. Similarly, for an Apertis Product Release of v2024.0, we add string bv2024.0b<B_CNT> to the project configuration on OBS

Stable Build Suffix

The Build Suffix is a crucial metadata for an Apertis Stable release. Setting it correct is important for smooth package upgrades across different repository components of a Stable Apertis Release.

In a typical Apertis Release, we have multiple repository components

• Main repository components like: apertis:v2024:development, apertis:v2024:target
• Security repository components like: apertis:v2024:security:development, apertis:v2024:security:target
• Updates repository components like: apertis:v2024:updates:development, apertis:v2024:updates:target
• Backports repository components like: apertis:v2024:backports:development, apertis:v2024:backports:target

For packages landing into an Apertis Stable Release, it would land through either of security or updates repository components. Packages accumulate into either of the repositories and eventually get folded into the main respective repository, just like the stable release procedure followed in the Debian project.

The above structure allows Apertis users to have immediate access to updates through either of the repository channels, provided those repositories are enabled in their apt configuration. For users that only follow the main repository, the same changes land into main on every Stable Point Release.

Given the structure above, of the lifetime of a package, across different repositories, it is important to set the correct revision for the package builds; to ensure that users’ have a smooth package upgrade experience. In short, we need to ensure that stable package updates, when finally landing into the main repository have a higher build revision than the rest of repository components.

To achieve the above goal, consider the below project config metadata:

bv2024.0db<B_CNT> in v2024
bv2024.0cb<B_CNT> in v2024-security
bv2024.0bb<B_CNT> in v2024-updates
bv2024.0ab<B_CNT> in v2024-backports

In the above example, the revisioning is carefully ordered to ensure that when Stable Update packages finally land into the main repository, they have the proper revision.

Consider the above with an example of package foo and version 1.0-1. Under Apertis, its typical reflection would be something like: foo-1.0-1+apertis0bv2024.0db1_amd64.deb which would be the initial version in the main repository. Over the course, we may push a new security update reflecting as foo-1.0-3+apertis0bv2024.0cb1_amd64.deb which would live for a defined time until the next Stable Point Release in the security repository, like apertis:v2024:security:development. Eventually, as part of the Stable Point Release, the package will be folded into the main repository, at which point, its reflection in the main repository will become: foo-1.0-3+apertis0bv2024.0db1_amd64.deb.

Over its lifecycle in the Apertis v2024 release so far, package foo has had 3 versions: 1.0-1+apertis0bv2024.0db1, 1.0-3+apertis0bv2024.0cb1 and 1.0-3+apertis0bv2024.0db1

From apt/dpkg point of view:

$ dpkg --compare-versions 1.0-1+apertis0bv2024.0db1 lt 1.0-3+apertis0bv2024.0cb1 && echo "Latter version is greater"
Latter version is greater

Similarly, now for the example security update folded into main:

$ dpkg --compare-versions 1.0-3+apertis0bv2024.0cb1 lt 1.0-3+apertis0bv2024.0db1 && echo "Latter version is greater"
Latter version is greater

$ dpkg --compare-versions 1.0-3+apertis0bv2024.0cb2 lt 1.0-3+apertis0bv2024.0db1 && echo "Latter version is greater"
Latter version is greater

The above repository versioning scheme is reliable across the entire lifespan of an Apertis Stable Release. No further tweaks to project config are required.

Appendix

Recommendations for downstream distributions

Downstreams should use the same workflow and tools described in this section to automate and ensure a smooth release process. To do so they are encouraged to extend Apertis release scripts and CI to include their own repositories and other required delta. As in any other piece of software in the Apertis ecosystem it is recommended that downstreams propose new changes to these tools to make the release process smoother and more reliable.

Helper scripts to ease Apertis development

release-scripts/: contains helper scripts to ease release process.

• update-* scripts create branches in key Git repositories for the current release and commit changes necessary to build images for the next release.
• create-branches is a script to create branches for the current release in all Git repositories. This script requires curl and jq.
• add-new-repo is a script which updates the reprepro configuration at the OBS backend host, and it has to run there as root. It depends on OSC clone plugin.
• do-branching.sh is the main script used to branch the next release off in preparation for the current release.
  • In addition to RELEASE and NEXT_RELEASE it also reacts on NOACT skipping the execution of most commands when set to any non-empty value. NOVERIFY (currently hardcoded to ‘y’) skips the verification of the package copy process.
  • NOTE: The branching script expects root access on certain hosts to be configured in ~/.ssh/config

Release Branching by CI examples

The previous sections showed the steps to launch the branching through GitLab CI. This section provides additional information to developers on the use of this particular CI.

This is an overview of all the release branching jobs queued in the pipeline

This is an example of the same jobs, with their dependencies chalked out. There are multiple jobs that depend on one another. This is to ensure that jobs, that depend on certain outputs or tasks to be completed, are run only after.

This example highlights the dependency in action. A job is only run after its parent dependency job is run successfully. Jobs that do not have a connecting line are independent jobs that are run in parallel. In this example, most jobs in the branch stage are independent and run in parallel

This example highlights Manual Jobs and their dependencies. Manual Jobs are differentiated with the Play Button. A Manual Job is to be run manually by a user. In the context of this document, a manual job constitutes of certain commands that the user is expected to run by hand on respective servers. Most manual jobs, in this case, are to be run on the OBS Backend server. For each manual job, the exact commands will be displayed in the job’s console view.

This example highlights the exact set of commands that should be manually run. Most commands to be manually run, are to be run on specific servers, which will be mentioned in the console output for the particular job.

Most manual jobs are made dependent on other job. This is to ensure that steps are performed in a particular order. In the above example, the dependency is defined as: obs_prjconf => obs_reprepro => obs_clone_binary, where obs_prjconf runs a certain set of tasks, which are needed by obs_reprepro. Similarly obs_reprepro performs a certain set of tasks which are a pre-requisite for obs_clone_binary. The job chain is strictly defined and users should ensure to successfully run the manual jobs on respective servers, before progressing to the next job.

Skipping jobs

There can be scenarios during the Release Branching wherein the user would be required to re-run the entire pipeline. Under such situation, it can be desired to have certain specific jobs to be skipped from the consecutive pipeline run. For example, in first run, branch-create-mass-branches was run and it created all the branches in respect to the NEXT_RELEASE. For such cases, on consecutive run, it is desired to skip the branch-create-mass-branches job. Below is an example on how to invoke the pipeline while defining a set of jobs to be skipped.

In the above example screenshot, we specify that the branch-create-mass-branches and branch-update-tests-projects jobs be skipped. The important bit is to pass the FORCE_JOB_SKIP variable with the required values.

As was specified in the previous job invocation, this example pipeline job run has the branch-create-mass-branches and branch-update-tests-projects skipped.

Release Branching manual steps

Given the number of steps involved in Release Branching, the following Branching Checklists should be executed in close co-ordination.

Branching Checklists

There are a couple of checks to be performed on individual host machines. The machine names mentioned are specific and should be adopted to users’ environment

• Host niobium: Hosts the OBS service
• Host images.apertis.org: Hosts the Apertis images repository
• Host gitlab.apertis.org: Hosts the Apertis GitLab instance

Branching Preparation

• set up ~/.ssh/config such that the niobium alias is available with appropriate root privilege.

Host niobium
    User andrewshR

Host images.apertis.org
    User andrewshR

• Check available disk space on OBS Server
  • Release Branching is a task that will consume a significant amount of disk space. Thus, it is very important to ensure that enough free space is available before commencing a release branching. Depending upon the size of a project, the amount of disk space required can vary.
  • Running out of disk space, halfway during the release branching process, may have severe side-effects. Hence, it is advised to consider over provision of free disk space before initiating a release branching.
• Install the jq package. It is already packaged in Apertis
• Install the osc-plugin-clone plugin for the OBS client
• Install the osc-plugins-dput. It is already packaged in Apertis
• Create a GitLab personal access token and configure it in $HOME/.gitlab-apertis-org-token, e.g. GITLAB_TOKEN=XXXXxxxxxXXXXXX > $HOME/.gitlab-apertis-org-token
  • Make sure your API token is active. When you create your API token, you have an option to set an expiry date. So make sure to verify the validity of your granted token
• Ensure that the osc tool is configured with the Admin user credentials for OBS. This is essential for the creation of additional repositories
• Check out the apertis-infrastructure repository
• Install gitlab-rulez tool and unprotect the current release branch for all repositories.
  • Verify that all relevant package groups, like pkg/*, have push_access_level: DEVELOPER_ACCESS in apertis-infrastructure/gitlab-scripts/rulez.yaml
  • Apply the rule with gitlab-rulez apply apertis-infrastructure/gitlab-scripts/rulez.yaml
• Change the working directory to release-scripts/
• Launch export RELEASE=v2019; export NEXT_RELEASE=v2020dev0; ./do-branching.sh which will:
  • Set new release to string v2020dev0 and set current release to string v2019, for example.
  • Create a new debootstrap script entry on the image builder host
  • Update the debootstrap package on OBS
  • Update the apertis-customizations, apertis-image-customization, apertis-tests repositories, both public and internal
  • Create branches in every repository on gitlab.apertis.org
  • Add the next-release aptly entry to the OBS configuration files (/etc/obs/BSConfig.*.pm files).
  • Restart OBS publisher and srcserver services to pick up the new aptly entry.
  • Stop the OBS schedulers
  • Fork the OBS projects, copying packages and metadata
  • Clone and republish the OBS binary repositories
  • Restart the OBS schedulers
• Update build suffix on OBS Project configuration (eg. bv2021.0b<B_CNT>, bv2020dev0b<B_CNT>). Please refer to section Build Suffix and Stable Build Suffix for more details about it

Removing an old distribution - Reprepro

Perform the following steps to remove an old distributions. This would usually be the obsolete Beta and Preview distribution releases, like: v2019dev0, v2019pre

• Remove the entry for the distribution from the reprepro configuration file. Example: /srv/obs/repos/shared/apertis/internal/apertis/conf/distributions

• Run the following reprepro command to apply the changes

  # sudo -u obsrun reprepro --gnupghome /srv/obs/gnupg/ -Vb /srv/obs/repos/shared/apertis/internal/apertis --delete clearvanished
  Deleting vanished identifier 'u|v2019dev0|proprietary|amd64'.
  Deleting vanished identifier 'u|v2019dev0|proprietary|arm64'.
  Deleting vanished identifier 'u|v2019dev0|proprietary|armhf'.
  Deleting vanished identifier 'u|v2019pre|demo|amd64'.
  Deleting vanished identifier 'u|v2019pre|demo|armhf'.
  Deleting vanished identifier 'u|v2019pre|nothumb|amd64'.
  Deleting vanished identifier 'u|v2019pre|nothumb|armhf'.
  Deleting vanished identifier 'u|v2019pre|proprietary|amd64'.
  Deleting vanished identifier 'u|v2019pre|proprietary|armhf'.
  

Dashboard update

At the end of the release process, it’s necessary to update the channels tracked by the Apertis packages Dashboard. This ensures that we focus on current releases. As soon as we publish a new release, it can be marked as closed (with status: closed) in the channels.yaml, this will hide some reports which are no longer of interest. Two weeks after a release, it can be completely removed from this channels file. This period allows us to keep an eye on a release after its publication in case of potential issues while downstream projects are synchronizing.