Testing on a immutable rootfs

Tests should be self-contained and not require changes to the rootfs: any change to the rootfs causes the tested system to diverge from the one used in production, reducing the value of testing.

Changing the rootfs is also impossible or very cumbersome to do with some deployment systems such as dm-verity or OSTree. Other systems may simply not ship package management systems like apt/dpkg due to size constraints, making package dependencies not viable.

Tests meant to be run on targets should then be self-sufficient and should not require dependencies to be installed on the rootfs.

Writing self-contained tests

The approach used by Apertis to create self contained tests is to ship all the required test code via dedicated git repositories under the tests GitLab group which get deployed by LAVA on the device before executing the tests.

Those repositories can either contain interpreted code that does not need to be built, for instance by using the dash POSIX shell, or they include binaries built on OBS and synchronized with a GitLab CI/CD pipeline.

For instance, the apparmor-dbus testcase involves:

  1. The apparmor-dbus.yaml testcase definition
  2. The tests/apparmor-dbus git repository
  3. The apparmor-dbus-tester deb package

The testcase definition

Both manual and automated testcases are defined via YAML files in the apertis-test-cases.

The YAML files are used to generate the qa.apertis.org website and the automated ones are executed by LAVA.

Usually testcase need supporting files and scripts and these are shipped via git repositories with the below directive:

1
2
3
4

install:
  git-repos:
    - url: https://gitlab.apertis.org/tests/apparmor-dbus.git
      branch: 'apertis/v2021'

The git repository

The easiest way to deploy supporting files, scripts and binaries on the device under test is by storing them in standalone git repositories that then get unpacked on the device before the test is executed.

In the simplest case the support git repository only ships a shell script and maybe some static test samples. Keep in mind that the Apertis reference images do not ship bash on fixedfunction/HMI images but use dash, so stick to POSIX features and avoid bashisms.

If instead compiled binaries are needed, they need to be built on OBS and regularly synchronized in the git repository. Apertis provides a reusable GitLab pipeline to retrieve those external binaries and commit them in the git repository. To set it up:

  1. Create a new GitLab project under the test group.

  2. Add a subtree of the common folder. For convenience, use the common-subtree.sh script to add a git-subtree into your repository:

    1

    ../common/common-subtree.sh add

    See the README for further details

  3. Add an external-binaries.cfg file which describes the binaries that need to be retrieved and committed. All the files that would otherwise need to be installed via apt should be listed here: check what the dependencies of the test are and inspect their contents, filtering out the files which are not needed to execute the test. The format is a simple list of $PACKAGE $FILEPATH on each line. The retrieved binaries will be committed under the appropriate $ARCH/bin subdirectory in the repository. An optional prefix= attribute is supported to control where the retrieved file should be installed, for instance from the ofono test repository:

    apertis-tests-apparmor-ofono /usr/lib/apertis-tests/apparmor/ofono/libofonod-malicious-override.so.0 prefix=lib
apertis-tests-apparmor-ofono /usr/lib/apertis-tests/apparmor/ofono/libofonod-malicious-override.so prefix=lib

  4. Add a .gitlab-ci.yml which defines the GitLab CI/CD pipeline that is responsible of keeping the binaries updated by pulling them from the built packages and committing them to the repository:

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

    variables:
  osname: apertis
  release: v2021
include:
  - project: tests/helper-tools
    ref: apertis/v2021
    file: /gitlab-ci/update-test-binaries.yaml
update-test-binaries:
  stage: build
  extends:
    - .update-test-binaries

  5. Configure a CI/CD schedule to run the pipeline daily.

The deb package

In some cases the needed binaries can be extracted from existing packages.

For instance the glib-gio-fs testcases sources them from the upstream libglib2.0-tests package.

In other cases a new, dedicated package needs to be created to host the test utilities. No specific requirements are needed on this package, but conventionally they are named ${TESTCASE}-tester, like apparmor-dbus-tester.

References