Getting from source code to an image suitable for loading into a target device is typically a two step process. The first step is creating binary packages and the second combining them to create an image. A more detailied explanation of this process is documented in the Apertis workflow.
The Apertis project automates the creation of the official images, which are built for the reference hardware and supported by the project. Whilst these images may prove enough for early exploration of the Apertis platform, they are unlikely to be suitable for direct use in a specific bespoke use case. The automation used to build these images is not currently available for direct use by unofficial projects on the Apertis infrastructure.
If you are in the fortunate position that your target hardware is supported by Apertis and are just looking to include extra packages already available in Apertis, then this can be achieved on the Apertis infrastructure and is documented in How to Build Your First Image on Apertis.
It is very likely that as a developer looking at Apertis you have your own bespoke hardware, a piece of hardware that isn’t one of the reference boards or additional hardware attached that isn’t currently supported in the reference images. In this guide we look at building an Apertis image locally with support to boot on such a board.
Whilst this process currently can’t be carried out on the Apertis infrastructure from a basic account. The Apertis project is open to the idea of creating dedicated project areas for projects on the Apertis infrastructure which might be able to take advantage of the Apertis’ automation.
The Target Hardware
We are going to use a low cost development board that is not currently supported by Apertis as an example. As the focus of this guide is on the development workflow, we will pick a board that has reasonable upstream support, but which will need the configuration of a few packages to be modified and these packages rebuilt to build an image that will boot. The board chosen for this is the Orange Pi Zero2.
If you are looking to target a less well supported or custom hardware platform where you need to add support, there is a guide for building, patching and maintaining the Apertis kernel which provides some guidance on how to approach this.
Picking a version of Apertis
It is important to consider which version of Apertis is going to be used. At any given time, Apertis has an old and new stable release and a version being actively developed. As this guide is acting as a demonstration and as there’ll be no product release the latest Apertis stable release will be used, v2024 at time of writing.
To make best use of the release support of the Apertis project, developers are advised to take the Apertis release flow and schedule into account when creating a product release roadmap and to pick the branch on which they perform their development carefully.
Summary of the Required Changes
We will base our initial image on the Fixed Function image type, as a nice small image with which to test booting our desired hardware. Our initial target will to be to have an image built that can be written to an SD card, which boots to a login prompt on the serial console. As the Orange Pi Zero2 is an ARM64 board, Apertis already provides basic support for the architecture, however there are a few key packages that we’ll need to check/modify to successfully boot:
- Arm Trusted Firmware
- U-Boot
- Linux kernel
As Arm boards don’t generally provide a standardised way of booting, with each vendor of Arm based SoCs having their own requirements to boot, we will need to create a HWpack in the form of a hardware specific Debos recipe that can extend and massage the Apertis ARM64 OSpack into a form that can boot.
We have picked an 64-bit Arm based board due to it being a bit more complex to boot compared with an 64-bit Intel based board. With that said, it is likely that any custom board will require some steps to be taken to customise the boot firmware, kernel and/or image contents. It should also be noted that we have picked an architecture that is already supported in the Apertis project, targeting an as yet unsupported architecture would require significantly more effort and is out of scope for this guide. Please contact the Apertis developers if you have requirements that you need help with.
Download and Setup a Devroot
We are going to be downloading the git source trees for packages we need to modify and buildng them locally. To do this we need a suitable toolchain and environment to perform these builds. Whilst a lot of packages can be successfully built with a cross compiler tool chain, there are others that can’t be easily setup to do this. Apertis provides Devroots to act as a build environment, enabling packages to be built as though they are being built on a machine of the target architecture.
The process for setting up a devroot is documented in
Development containers using devroot-enter.
Download the latest v2024 ARM 64-bit devroot, install it and spawn a devroot
container using the devroot_enter. This command will automatically bind the
user’s home directory, but we will need to find it manually:
$ devroot-enter ${DEVROOT_PATH}
...
user@devroot:/tmp$ cd ${HOMEDIR}
user@devroot:/home/user$
We will install
git-buildpackage
which is a useful tool to help with building debian packages and setup the name
and email used by Git and git-buildpackage:
user@devroot:/home/user$ sudo apt update
...
user@devroot:/home/user$ sudo apt install git-buildpackage
...
user@devroot:/home/user$ git config --global user.name "External Developer"
user@devroot:/home/user$ git config --global user.email "external@example.org"
user@devroot:/home/user$ export DEBEMAIL="External Developer <external@example.org>"
Building Arm Trusted Firmware
Looking at the documentation for building U-Boot for the Allwinner Arm SoCs, we can see that Arm Trusted Firmware (ATF) is a dependency for building U-Boot, so we’ll start with that.
The Apertis project already has a repository for the Arm trusted Firmware, we can clone that repsoitory locally:
user@devroot:/home/user$ git clone https://gitlab.apertis.org/pkg/arm-trusted-firmware.git
user@devroot:/home/user$ cd arm-trusted-firmware
In order to make it a little easier to track changes made here and updates made
by the Apertis project to ATF over time (allowing easier updating in the
future) we will create a branch under a distinct project name. For example if
we use “orange” as our project name, we will create an orange/v2024 branch
from the apertis/v2024 branch:
user@devroot:/home/user/arm-trusted-firmware$ git checkout -b orange/v2024 origin/apertis/v2024
We will also be using this project name as a suffix to the version number when creating our customised “downstream” packages.
Looking at the
hardware documentation
we can see that the device has the Allwinner H616 CPU. The
U-Boot documentation
helpfully tells us for this SoC we need to build ATF for the sun50i_h616
platform.
Looking in debian/rules
we find a list of platforms that the package is built for, so we add sun50i_h616 to this:
|
|
Commit this change:
user@devroot:/home/user/arm-trusted-firmware$ git commit -s -m "Add sun50i_h616 to build platforms" debian/rules
We should also create a changelog entry for a new release and commit it:
user@devroot:/home/user/arm-trusted-firmware$ gbp dch --git-author --ignore-branch -l orange -R
user@devroot:/home/user/arm-trusted-firmware$ DEBIAN_PACKAGE=$(dpkg-parsechangelog -S Source)
user@devroot:/home/user/arm-trusted-firmware$ DEBIAN_VERSION=$(dpkg-parsechangelog -S Version)
user@devroot:/home/user/arm-trusted-firmware$ git commit -s -m "Release ${DEBIAN_PACKAGE} version ${DEBIAN_VERSION}" debian/changelog
Note that we pass -l orange when calling gbp dch. This uses orange as a
suffix for a local “downstream” revision of the Apertis package. See our
documentation on versioning for more details.
We will now build it locally using the devroot. In order to build ATF successfully, we need to install it’s build dependencies:
user@devroot:/home/user/arm-trusted-firmware$ sudo apt build-dep arm-trusted-firmware
...
user@devroot:/home/user/arm-trusted-firmware$
We can then use git-buildpackage to build ATF:
user@devroot:/home/user/arm-trusted-firmware$ gbp buildpackage --git-ignore-branch -B -d -uc -us
In order for the U-Boot build to use the binaries in the package of ATF that was just built, we must install the new package in the devroot:
user@devroot:/home/user/arm-trusted-firmware$ cd ..
user@devroot:/home/user$ sudo dpkg -i arm-trusted-firmware_*_arm64.deb
Selecting previously unselected package arm-trusted-firmware.
(Reading database ... 30207 files and directories currently installed.)
Preparing to unpack arm-trusted-firmware_2.10.0+dfsg-1+apertis2orange1_arm64.deb ...
Unpacking arm-trusted-firmware (2.10.0+dfsg-1+apertis2orange1) ...
Setting up arm-trusted-firmware (2.10.0+dfsg-1+apertis2orange1) ...
user@devroot:/home/user$
We will now have the required binary available:
user@devroot:/home/user$ ls /usr/lib/arm-trusted-firmware/sun50i_h616/
bl31.bin
user@devroot:/home/user$
Building U-Boot
Now that we have the required ATF binaries we can modify U-Boot to build for our board. Start by cloning the Apertis U-Boot repository (as we did with ATF) and creating a downstream project branch:
user@devroot:/home/user$ git clone https://gitlab.apertis.org/pkg/u-boot.git
user@devroot:/home/user$ cd u-boot
user@devroot:/home/user/u-boot$ git checkout -b orange/v2024 origin/apertis/v2024
Like with ATF, we need to update the package configuration to build for the Orange Pi Zero2. The U-Boot package builds for a lot of different boards already, with the configuration for these split out from debian/rules into debian/targets/mk. To add the Pi Zero, we add the following entries to this file:
|
|
Commit this change, create a changelog entry for a new release and commit that too:
user@devroot:/home/user/u-boot$ git commit -s -m "Add Orange Pi Zero2 to built platforms" debian/targets.mk
user@devroot:/home/user/u-boot$ gbp dch --git-author --ignore-branch -l orange -R
user@devroot:/home/user/u-boot$ DEBIAN_PACKAGE=$(dpkg-parsechangelog -S Source)
user@devroot:/home/user/u-boot$ DEBIAN_VERSION=$(dpkg-parsechangelog -S Version)
user@devroot:/home/user/u-boot$ git commit -s -m "Release ${DEBIAN_PACKAGE} version ${DEBIAN_VERSION}" debian/changelog
As with building ATF we need to ensure that all required dependencies are install in the devroot:
user@devroot:/home/user/u-boot$ sudo apt build-dep u-boot
The U-Boot package will build U-Boot for numerous target devices, which takes a
long time. We can limit the number of devices it’s going to build for by
setting DEB_BUILD_PROFILES in the environment prior to running gbp buildpackage. Here we limit building to just the target we are interested in:
user@devroot:/home/user/u-boot$ export DEB_BUILD_PROFILES=pkg.uboot.platform.orangepi_zero2
user@devroot:/home/user/u-boot$ gbp buildpackage --git-ignore-branch -B -d -uc -us
This generates a bunch of U-Boot binary packages in the parent directory,
including one for sunxi based devices such as the Orange Pi Zero2:
user@devroot:/home/user/u-boot$ cd ..
user@devroot:/home/user$ ls u-boot-sunxi*.deb
u-boot-sunxi_2024.01+dfsg-1+apertis2orange1_arm64.deb
Checking kernel
Downloading the current Apertis kernel and extract it (it is easy to do this in the devroot), we find that it already ships the device tree for the OrangePi Zero2:
user@devroot:/home/user$ apt download "linux-image-*-arm64"
Get:1 https://repositories.apertis.org/apertis v2024/target arm64 linux-image-6.6.13-arm64 arm64 6.6.13-1+apertis2bv2024.0db1 [82.5 MB]
Get:2 https://repositories.apertis.org/apertis v2024-updates/target arm64 linux-image-6.6.0-2-arm64 arm64 6.6.1-0+apertis2bv2024.0bb1 [61.0 MB]
Fetched 144 MB in 49s (2934 kB/s)
user@devroot:/home/user$ dpkg -X linux-image-6.6.13-arm64_6.6.13-1+apertis2bv2024.0db1_arm64.deb linux-deb
./
./boot/
...
./usr/share/lintian/overrides/
./usr/share/lintian/overrides/linux-image-6.6.13-arm64
user@devroot:/home/user$ ls linux-deb/boot/dtbs/*-arm64/allwinner/*zero2*
linux-deb/boot/dtbs/6.6.13-arm64/allwinner/sun50i-h616-orangepi-zero2.dtb
A quick look at the config used to build it also suggests it’s been built with support for the H616, so we’ll assume for now that the Apertis kernel is ready to boot on the OrangePi Zero2:
user@devroot:/home/user$ grep H616 linux-deb/boot/config-*-arm64
CONFIG_PINCTRL_SUN50I_H616=y
CONFIG_PINCTRL_SUN50I_H616_R=y
CONFIG_SUN50I_H616_CCU=y
Image scripts
The Apertis images are assembled using Debos in a multi-stage process. Debos uses YAML based configuration files, each stage is configured by it’s own configuration file and these files are usually stored in the apertis-image-recipes repository.
To build an image, the first stage is to build an OSpack (the rootfs without the basic hardware-specific components like bootloader, kernel and hardware-specific support libraries) which tends to be a common artifact used between boards with the same architecture, this is then enhanced for a specific device or use case by a HWpack.
It is best to build these images from the appropriate image-builder docker
container, as documented in the apertis-image-recipes
README.md.
This needs to be run in the SDK or another Linux environment where docker is
available. Here we will assume use of the SDK.
Clone the apertis-image-recipes repository into a runing SDK instance and
create a downstream project branch:
user@sdk:~$ git clone https://gitlab.apertis.org/infrastructure/apertis-image-recipes.git
...
user@sdk:~$ cd apertis-image-recipes
user@sdk:~/apertis-image-recipes$ git checkout -b orange/v2024 origin/apertis/v2024
Building an OSpack
As previously mentioned, the aim is to build a fixedfunction image, to do
this we start by building the appropriate OSpack (ospack-fixedfunction.yaml).
user@sdk:~/apertis-image-recipes$ RELEASE=v2024
user@sdk:~/apertis-image-recipes$ docker run \
-i -t --rm \
-w /recipes \
-v $(pwd):/recipes \
--security-opt label=disable \
-u $(id -u):$(id -g) \
--group-add=$(getent group kvm | cut -d : -f 3) \
--device /dev/kvm \
--cap-add=SYS_PTRACE \
--tmpfs /scratch:exec \
registry.gitlab.apertis.org/infrastructure/apertis-docker-images/$RELEASE-image-builder \
debos ospack-fixedfunction.yaml -t suite:$RELEASE -t architecture:arm64
Building an image with a HWpack
HWpacks are typically hardware specific and as Apertis doesn’t currently support the Orange Pi Zero2, a new HWpack will need to be created. The main differentiator which requires different HWpacks is the firmware that needs to be loaded to boot the device and where.
We create a recipe for debos to perform the steps required to make a bootable
image. We can base this on an existing recipe. In this instance we have based
it on image-rk3399.yaml and made the following changes:
|
|
Going through these changes one at a time:
|
|
A fairly obvious change, we switch the name of the sbc from rock-pi-4-rk3399
to orangepi_zero2. Note however that the name chosen is the name used in the
U-Boot packaging (it’s used later in the script as a directory name).
|
|
Looking at the documentation
we need to copy the u-boot-sunxi-with-spl.bin binary we built to 8MB into the
boot device. With 512B sectors, this is at sector 16.
Since sector 16 is within the gpt partition space, we will instead utilise a msdos partition table.
|
|
The rk3399 is an officially supported board and the offical Apertis U-Boot build is used. The Orange Pi Zero2 build has required us build a U-Boot package locally, so instead of installing a U-Boot package from the Apertis repositories, we need to :
- Add an overlay holding the binary U-Boot package,
- Install these binary packages using
dpkgwhen building the image, - Delete the binary packages from the image.
In addition to the changes in the recipe, the U-Boot package
u-boot-sunxi_2023.01+dfsg-1+apertis6orange1_arm64.deb has been copied to a
new overlay in the local
repository:
$ mkdir -p overlays/local-debs/home/user
$ cp ../u-boot-sunxi*.deb overlays/local-debs/home/user/
Additionally from the above patch segment (and the one below) we can see that rockchip installs 2 pieces of firmware at different offsets. With the Orange Pi Zero2 we only install one and thus one of these actions gets removed.
|
|
Lastly we have the step where the U-Boot binary is written to the required
offset in the image. We are simply modifying this section to use the desired
binary (/usr/lib/u-boot/orangepi_zero2/u-boot-sunxi-with-spl.bin, provided by
the u-boot-sunxi package) and write it to the required offset (sector 16,
which is an 8MB offset).
We can now build the final image, again using the appropriate image-builder
docker container:
user@sdk:~/apertis-image-recipes$ docker run \
-i -t --rm \
-w /recipes \
-v $(pwd):/recipes \
--security-opt label=disable \
-u $(id -u):$(id -g) \
--group-add=$(getent group kvm | cut -d : -f 3) \
--device /dev/kvm \
--cap-add=SYS_PTRACE \
--tmpfs /scratch:exec \
registry.gitlab.apertis.org/infrastructure/apertis-docker-images/$RELEASE-image-builder \
debos image-h616.yaml -t suite:$RELEASE -t architecture:arm64
Installing the image on an SD card
The standard Debos recipes, on which we have based the recipe for H616 image
creation, create ancillary files when generating the image to allow images to
be quickly written on to SD cards and alike using
bmaptool.
The image writing process is quite simple as the Debos scripts have already generated the image with the required partitions and firmware installed as required so that the image can just be written to the beginning of the SD card:
$ sudo bmaptool copy apertis-v2024-fixedfunction-arm64-orangepi_zero2.img.gz $SDCARD
Make sure to set $SDCARD to the device file of the SD card.
Insert the SD card into the board, attach the serial console cable and power supply and boot:
U-Boot SPL 2024.01+dfsg-1+apertis2orange1-00002-g5a39a00d9-dirty (Feb 23 2024 - 20:05:46 +0000)
DRAM: 1024 MiB
Trying to boot from MMC1
NOTICE: BL31: v2.10.0 (debug):apertis/2.10.0+dfsg-1+apertis2-2-gcb99379-dirty
NOTICE: BL31: Built : 18:18:20, Feb 23 2024
NOTICE: BL31: Detected Allwinner H616 SoC (1823)
NOTICE: BL31: Found U-Boot DTB at 0x4a0b3458, model: OrangePi Zero2
INFO: ARM GICv2 driver initialized
INFO: Configuring SPC Controller
INFO: PMIC: Probing AXP305 on RSB
INFO: PMIC: aldo1 voltage: 3.300V
INFO: PMIC: aldo2 voltage: 3.300V
INFO: PMIC: aldo3 voltage: 3.300V
INFO: PMIC: bldo1 voltage: 1.800V
INFO: PMIC: dcdcd voltage: 1.500V
INFO: PMIC: dcdce voltage: 3.300V
INFO: BL31: Platform setup done
INFO: BL31: Initializing runtime services
INFO: BL31: cortex_a53: CPU workaround for erratum 855873 was applied
INFO: BL31: cortex_a53: CPU workaround for erratum 1530924 was applied
INFO: PSCI: Suspend is unavailable
INFO: BL31: Preparing for EL3 exit to normal world
INFO: Entry point address = 0x4a000000
INFO: SPSR = 0x3c9
INFO: Changed devicetree.
U-Boot 2024.01+dfsg-1+apertis2orange1-00002-g5a39a00d9-dirty (Feb 23 2024 - 20:05:46 +0000) Allwinner Technology
CPU: Allwinner H616 (SUN50I)
Model: OrangePi Zero2
DRAM: 1 GiB
Core: 54 devices, 22 uclasses, devicetree: separate
WDT: Not starting watchdog@30090a0
MMC: mmc@4020000: 0
Loading Environment from FAT... Unable to use mmc 0:1...
In: serial@5000000
Out: serial@5000000
Err: serial@5000000
Allwinner mUSB OTG (Peripheral)
Net: eth0: ethernet@5020000using musb-hdrc, OUT ep1out IN ep1in STATUS ep2in
MAC de:ad:be:ef:00:01
HOST MAC de:ad:be:ef:00:00
RNDIS ready
, eth1: usb_ether
starting USB...
Bus usb@5200000: USB EHCI 1.00
Bus usb@5200400: USB OHCI 1.0
scanning bus usb@5200000 for devices... 1 USB Device(s) found
scanning bus usb@5200400 for devices... 1 USB Device(s) found
scanning usb for storage devices... 0 Storage Device(s) found
Hit any key to stop autoboot: 0
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found /extlinux/extlinux.conf
Retrieving file: /extlinux/extlinux.conf
1: Apertis v2024 6.6.13-arm64
Retrieving file: /vmlinuz-6.6.13-arm64
Retrieving file: /initrd.img-6.6.13-arm64
append: root=UUID=17f9cad5-d684-4509-931c-d260268cbe06 rootwait rw quiet splash plymouth.ignore-serial-consoles fsck.mode=auto fsck.repair=yes
Retrieving file: /dtbs/6.6.13-arm64/allwinner/sun50i-h616-orangepi-zero2.dtb
Moving Image from 0x40080000 to 0x40200000, end=424d0000
## Flattened Device Tree blob at 4fa00000
Booting using the fdt blob at 0x4fa00000
Working FDT set to 4fa00000
Loading Ramdisk to 483f4000, end 49ffff51 ... OK
Loading Device Tree to 00000000483ed000, end 00000000483f3bfb ... OK
Working FDT set to 483ed000
Starting kernel ...
...
Apertis v2024 apertis ttyS0
apertis login: