Often target platforms are not supported by Apertis. This may be due to the way the Apertis components are built/configured or due to these components lacking support for the required hardware. Whilst they may lack support in Apertis, these boards frequently have reference images based on another Linux variant with the associated source code being available. This support is often provided in a Yocto-based deliverable. The support generally provides good coverage of the vendors hardware, however the quality of implementation can vary quite dramatically.

When evaluating a platform for use in a product, it is important to evaluate both the technical and business related factors to determine whether it represents a cost effective solution. This document aims to:

• highlight deficiencies commonly found in vendor hardware support,
• provide some ways in which the quality of vendor provided support can be quantified,
• suggestions to help with evaluating the changes to be integrated into Apertis, and

Hardware packages and OSpacks

Hardware packages (HWpacks) provide tailored Linux kernels, bootloaders and any specific drivers required to enable Apertis OSpacks to be run on target boards. Apertis provides hardware packages that enable it to be used on several reference boards across a number of architectures.

This document focuses on the Linux kernel as it is expected that a version of the Linux kernel will be required by all Apertis users. Whilst a Bootloader will be required for a device, these can vary far more and thus do not provide such a good example.

Common vendor kernel issues

Whilst the software provided by the vendor typically does a good job of showcasing their hardware it is typically not provided in a form that would be considered to follow best practice. Common issues are outlined below.

Limited enablement of hardware and core functionality in provided configuration

The configuration provided when building the kernel and other low-level components is generally tailored to just provide the support required to demo the vendors hardware. As a result this configuration is generally inappropriate for direct use in products as they will typically require drivers for hardware not provided on the vendors reference hardware and require additional functionality to be enabled to support application specific workloads.

Disregard for common code

The vendor is typically only interested in providing working support for their hardware and this is frequently done in the most expedient manner possible. As a result both the parameters for and logic of common code, which exist to support multiple existing pieces of hardware, are modified without regard for the continued operation of the other hardware. This is problematic when attempting to utilise a common source tree for multiple projects, utilizing different hardware platforms, where the changes made for one platform break support implemented for others.

This can sometimes result in build failures when attempting to enable hardware not found in the vendor provided configuration (see “Limited Enablement of Hardware and Core Functionality In Provided Configuration”).

No “clean” patch series implementing the vendors changes

Frequently the vendor will provide a git tree containing the kernel source and any target specific changes. The commits from which these changes are comprised are not usually provided as a clean set of changes on top of the mainline version (such as would be achieved by performing git rebase when shifting focus to a newer version of the underlying component), but are scattered through the commit history as subsequent mainline versions have been added through the use of git merge. This can make it difficult to generate a clean set of patches to apply the vendor BSP to the mainline kernel source of the same version and to port support to a newer kernel version to take advantage of security and bug fixes.

Unrelated changes

The changes to the source code might not only be changes for the required processor/target board, but also other changes like:

• Realtime patches
• Some improvements from perspective of vendor not directly related to hardware support.
• Addition of hardware support for other hardware provided by the vendor but which is not of consequence for the projects target platform

These changes may be scattered throughout the commit history (see No “clean” patch series implementing the vendors changes and typically lack useful commit messages to enable the intention behind any given change from being understood. This makes resolving conflicting changes between the newer mainline and BSP specific changes difficult to rectify.

This leads to problems when upgrading with newer versions of the mainline kernel.

Compatibility with ospacks

The Apertis operating system packages (ospacks) rely on the kernel with which they are paired to provide certain functionality. This functionality is expected to be provided using the common APIs found in the mainline kernel. Vendors sometimes either modify/extend or utilize custom APIs to expose functionality supported by their product. This combined with limitations in functionality enabled in the vendors kernel config can lead to the kernel not providing the functionality expected by the ospack, either in a form it can use or at all.

Lack of Mainline Support for Vendors Hardware

The Apertis project recommends hardware support is submitted for inclusion in the mainline kernel. This results in:

• Greater testing of common code.
• Ability to easily generate a single binary, supporting multiple target boards with a common architecture.
• Best support for security updates due to being able to efficiently track the active upstream code changes.
• Best compatibility with Apertis.

Many vendors do not actively work with the mainline kernel developers to get support for their hardware into the mainline kernel. The use of a vendor specific kernel reduces the upfront time and effort required to build a solution, but results in a significant increase in long term maintenance effort. Whilst more effort is required to achieve mainline kernel support, this effort frequently results in a higher quality of support being achieved than what would be expected from a vendor kernel.

There are often a number of changes that need to be made before the mainline kernel supports a target board. Sometimes the mainline kernel can be used directly though typically the following additions are required (in increasing effort):

• The kernel configuration has to be adopted
• A new device tree (.DTB) has to be added
• Board specific drivers have to be added
• Addition of support for unsupported system on chip.

The board specific drivers are available in the vendor specific kernel. With some kernel knowledge, the drivers can usually be ported to the mainline kernel (though this can be difficult for low quality vendor kernels). If the platform is important to you and the vendor has no interest in upstreaming support themselves, taking on the challenge of upstreaming support yourself may be beneficial in the long term.

Evaluating the quality of the vendor BSP

With the increasing complexity and requirement for devices to be internet enabled, the old “embedded” approach of freezing and hardening a BSP no longer works. Any issues that are not detected during the hardening process may result in serious security vulnerabilities being left present in shipped devices. As a result there is an increasing requirement to perform “in the field” updates to shipped devices so that upgraded kernels and other system components to resolve security issues.

Some vendors do not yet provide on going support for newer kernels, nor work to include support in mainline kernels. Some will use development practices that hinder the ability of a product team to perform these updates themselves when necessary. It is therefore imperative that the quality of the vendors software support be evaluated when selecting hardware for a project as this can drastically effect the economic viability of a vendors offering over the lifetime of a product.

Understanding Kernel Versioning

Each mainline kernel is defined by its version number. This is typically in the form of 2 or 3 dot separated numbers, where these numbers are:

<major version>.<minor version>.<stable release>

This may have other parts appended to it after a dash with these extra parts being used to denote “release candidate” kernels or vendor specific kernels.

The mainline kernel is released as a 2-number version, such as 5.3. The minor version is incremented with each official release. Every so often the major version number is incremented and the minor is returned to zero. There is no defined point at which this occurs and it is at Linus Torvalds' discretion as to when this happens.

Once a release is made, this is considered stable. Bugfixes will be applied to this until the next kernel release. Each bugfix release is released with a 3 number version, with the “stable release” part incrementing for each release. Some kernels will be supported for an extended period and are know as longterm stable (LTS) releases. Theses are the kernels that are used by Apertis.

Determine what kernel upgrades can be expected by the vendor

The vendor support for upgrades to newer versions varies a lot. Some vendors will do a single code drop with no support provided beyond that. Others provide upgrades, but without offering an upgrade plan. Most of the time it will be months between updates, during which time the vendor kernel will lack security and bug fixes that may be found in the mainline and longterm stable kernels.

Age and maintenance state of the related mainline kernel

Each vendor kernel is based on a mainline kernel. It is important to find out which kernel the vendors support is based on and determine it’s age and it’s support status. Wikipedia maintains a full list of release kernel versions, noting both their age and anticipated support “end of life” (EOL) dates, whilst kernel.org lists the projected EOL dates for still supported LTS releases along with detailing the latest release for each supported version.

When shipping a product, we do not want to be following the mainline development as a source of bug and security fixes as this will result in significant changes being pulled in when updating the kernel, which is likely to result in both unwanted changes to the features and performance on the target product and is likely to result in difficulties porting any vendor or product specific kernel modifications.

As stable kernel releases aim to only include security and bug fixes, this results in significantly less changes to the code base, reducing the risk of feature or performance change and reducing or eliminating issues with porting modifications. From a long term maintenance perspective it is therefore beneficial for a product to use a vendor that bases their vendor kernel on the recent LTS kernel (one with sufficient time before it reaches it’s end of life, as determined by the project time frames), even more so if they actively track the stable point releases.

Is a rebase to a longterm stable kernel needed?

Since development of the mainline kernel is always going on, the quality of these parts varies dependant on maturity. Also security issues like vulnerabilties are handled during this development process with security fixes backported to the kernels supported stable releases.

Apertis tracks this development effort to ensure the bug fixes, security fixes and feature improvements made during the mainline development process are available in Apertis, with each Apertis release including the latest Longterm Stable kernel with point releases tracking changes on that branch. Any vendor BSP integrated into Apertis should also be able to update regularly to include these fixes and improvements, however many vendors do not provide kernel updates leaving it to their users to either perform these updates themselves or remain on older kernels versions with known security vulnerabilities.

Is the vendor working to integrate support into the mainline kernel?

Even vendors that provide good on going support for their hardware via vendor trees lag behind the fixes provided in the mainstream kernel, as they require these releases to be made before porting any of their changes. Additionally, projects with long expected lifetimes may exceed the period for which a vendor is willing to provide support.

If a vendor “upstreams” support for their device, integrating that support right into the mainline kernel, the vendor kernel is typically no longer required. A correctly configured mainline kernel can be used, removing any lag and typically enabling the device to be used long after the vendor stops actively supporting it without significant addition effort and cost.

Are there qualified test results for vendor BSP ?

Some vendors provide tests and results for their specific images to check hardware and kernel features. These tests usually cover specific hardware functions and less common features of the kernel. So additional tests have to be planned and organized, probably got from mainline kernel. Since the image is a new binary (compared to images of mainline kernel), these tests have to be redone best at automatic test systems like LAVA (see also Apertis integration testing with LAVA).

How is the responsibility to maintain the vendor BSP organized ?

Developing and purchasing a product includes product liability for the company over product lifetime. This includes aspects for security (loss of personal data) and safety (risk to customers physically). The Linux kernel potentially plays a very important role in both, thus ensuring bugs and security issues get fixed promptly and reliably is critical. These updates either have to be done by the product manufacturer or the supplier of the vendor providing the kernel support. As vendors will typically stop providing support after a defined period, the timing of this needs to be understood in relation to the products life time. If the vendors support period would end before that of the product, then it needs to be understood (and factored in) that at such a time it becomes the responsibility of the product team to organize such maintenance support.

Evaluate the Quality of Vendor Added Drivers

Any drivers added by a vendor can be found by analyzing a diff between the vendor’s kernel sources against the related mainline kernel source. Drivers should be fairly self contained under drivers/, with minor changes in:

• Startup files
• Makefile(s)
• Kconfig(s)

A deep review of implementation and testing of the relevant kernel driver is recommended. This is a time consuming task.

Practical checks

The following practical checks can be made to help determine the quality of the vendor supplied kernel source.

Check the Ability to Build Mainline Supported Devices

Well integrated vendor support will result in a kernel that can still be built for other machines which share the same architecture and which are defined in the mainline kernel. Attempting to build using the configs provided in the configs/ folder for your target architecture (such as arch/arm/configs for 32-bit ARM devices). Build problems may indicate incomplete or unintended changes to common parts of the code.

Check the Ability to Build for Machines With Different Architectures

This is similar to testing configs for other machines from your target architecture, but can point to additional issues where changes have been made to very common code in architecture specific ways. This will require a toolchain supporting the relevant architecture. Build problems may indicate incomplete or unintended changes to common parts of the code.

Attempting to build a multi-target kernel binary

As long as sufficient similarities exist, it is possible to build a single kernel binary that can be used with multiple different machines with different peripherals and even processors.

The kernel provides configs that build a kernel binary which can run on multiple different devices. An example is arch/arm/configs/multi_v7_defconfig, which will build a kernel for many ARMv7 compatible devices. If such a configuration exists that may be compatible with the device your interested in, it may be instructive to attempt to add the target device to such a config and build it.

The need to build separate kernel binaries may indicate, that some driver interfaces are not implemented optimally. This will lead to reduced test depth and increased build efforts.

Determining quantity and quality of vendor kernel mondifications

It is typically not possible to utilise the git log to compare a vendor kernel to the mainline kernel from which it was built due to the previously mentioned development practices. As a result it is typical to need to utilise standard “diff” tools to make the comparison.

The differences highlighted by these tools need to be examined to identify the changes relating to target hardware. It would be typical to see:

• Hardware drivers and related header files
• Files to integrate hardware drivers like Kconfig(s) or Makefile(s)
• Addition of hooks to common files
• Expanded support in shared architecture files.

Thus the majority of the changes required to support a target device should be expected to be limited to those files providing drivers and support for the applicable architecture.

It is important to identify changes in other areas, including those not related to the target hardware, such as:

• Changes of significant complexity to common headers and source files
• Addition of unrelated support, such as those provided by the real-time patch set
• Unintended changes, e.g. changes of header files with contents of older kernel versions
• Unrelated improvements (at least from viewpoint of the supplier of the vendor kernel)
• Reformatting indenting and code style of entire files, masking subtle changes

The bigger the changes unrelated to the target hardware and the more spread the changes are, the greater the effort that will be required to upgrade to newer kernel versions.

Attempt to build a kernel using the Debian config with Debian patches applied

When adding kernel support for a device to Apertis, it is advised to base this on the Apertis kernel to enable you to easier take advantage of security updates. If the vendor kernel and the Apertis kernel are both based on a similar kernel, in order to test compatibility between the changes introduced by the vendor and those applied to the Apertis kernel it can be instructive to apply the Apertis patches to the vendor kernel and attempt to build with the Apertis kernel configuration (which is likely to have support for a broader number of features enabled).

As is customary with Debian packaging, changes to an upstream source and any other files (such as the component configuration file) are stored in the .debian.tar.xz file. The relevant files for Apertis can be found on the Apertis package repository.

Extracting this archive will create a debian/ folder. The patches applied by Apertis can be found in debian/patches (the series file provides the order and enables the series to be quickly applied with tools such as quilt). The Debian configuration is split in to a generic portion (in debian/config/config), architecture configuration (in debian/config/kernelarch-<arch>) and more fine grained architecture config (in debian/config/<arch>), with a “full” configuration comprised of the options found in one of each of these sets.

Applying these patches and building the kernel with this kernel configuration can highlight problems in common parts, which may make upgrades difficult without loss of functionality.