Table of Contents:

Apertis has a fairly standard boot process which utilises systemd to perform user space initialisation. This document describes steps that can be taken to profile and optimise the boot process.

Startup sequence modification rules

A number of simple steps need to be followed when adding services to the boot process to ensure that an optimal boot process can be achieved:

Do

  • Configure systemd services to only depend upon other services they absolutely require to function: This helps simplify and shorten the boot process.
  • Ensure less important services depend upon later stages of the systemd startup phase: This allows the system to bring up critical functionality sooner, which can make the startup feel shorter from the perspective of the user.

Don’t

  • Add arbitrary delays to services: Timing delays are fragile and slow the boot process, systemd supports a wide variety of ways to describe dependencies between services, enabling then to be started when the services they depend on are available.
  • Remove actual dependencies from .service files in an attempt to shorten the boot process: systemd is very good at minimizing the impact of the sequencing of service activation on the total boot time, so it is more important to ensure that all of a service’s requirements are well described to enable it to optimise the process well.

Profilng the boot process

There are two tools provided by systemd that are useful to profile and optimise the system’s startup time, systemd-analyze and systemd-bootchart.

systemd-analyze

Is a tool to analyze a system boot performance, retrieve statistics in order to improve the system startup time. There are different arguments to retrieve information from the system and service manager.

Following are some useful options with the explanation taken from the systemd-analyze manpage.

systemd-analyze time prints the time spent in the kernel before userspace has been reached, the time spent in the initial RAM disk (initrd) before normal system userspace has been reached, and the time normal system userspace took to initialize. Note that these measurements simply measure the time passed up to the point where all system services have been spawned, but not necessarily until they fully finished initialization or the disk is idle.

# systemd-analyze time
Startup finished in 1.412s (kernel) + 1.869s (initrd) + 14.158s (userspace) = 17.440s

systemd-analyze blame prints a list of all running units, ordered by the time they took to initialize. This information may be used to optimize boot-up times. Note that the output might be misleading as the initialization of one service might be slow simply because it waits for the initialization of another service to complete.

# systemd-analyze blame
          4.601s user@1000.service
          3.529s systemd-journal-flush.service
          2.760s plymouth-start.service
          2.025s avahi-daemon.service
          1.657s connman.service
          1.639s systemd-udev-trigger.service
          1.579s systemd-sysctl.service
          1.408s bootlogs.service
          1.341s dev-mqueue.mount
          1.119s systemd-tmpfiles-setup.service
          1.090s auditd.service
          1.050s motd.service
          1.032s sys-kernel-debug.mount
           979ms apparmor.service
           899ms systemd-modules-load.service
           884ms systemd-random-seed-load.service
           879ms systemd-tmpfiles-setup-dev.service
           809ms iptables.service
           784ms connman-vpn.service
           723ms systemd-fsck-root.service
           688ms ofono.service
           653ms plymouth-read-write.service
           632ms systemd-logind.service
           442ms run-lock.mount
           361ms run-user.mount
           307ms plymouth-quit.service
           292ms plymouth-quit-wait.service
           262ms systemd-remount-fs.service
           120ms rollbackd.service
            89ms systemd-user-sessions.service
            80ms systemd-udevd.service
            75ms systemd-update-utmp-runlevel.service

These numbers provide the relative duration of each service during the boot process (though their absolute time is only accurate if this is run on the target hardware).

systemd-analyze options can be used in conjunction with the --user argument which shows the performance data for user session instead of the system manager. For example the user@1000.service unit starts a set of services so a finer grained information can be gathered with:

# systemd-analyze --user blame
           3.321s pulseaudio.service
           1.687s xorg.service
           1.408s tracker-miner-fs.service

systemd-analyze critical-chain \[UNIT...\] prints a tree of the time-critical chain of units (for each of the specified UNITs or for the default target otherwise). The time after the unit is active or started is printed after the @ character. The time the unit takes to start is printed after the + character. Note that the output might be misleading as the initialization of one service might depend on socket activation and because of the parallel execution of units.

# systemd-analyze critical-chain
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

graphical.target @13.871s
└─multi-user.target @13.871s
  └─user@1000.service @9.268s +4.601s
    └─systemd-user-sessions.service @9.159s +89ms
      └─basic.target @6.344s
        └─systemd-ask-password-plymouth.path @6.343s
          └─-.mount @523ms

systemd-analyze plot prints an SVG graphic detailing which system services have been started at what time, highlighting the time they spent on initialization.

# systemd-analyze plot > plot.svg
# eog plot.svg

1024px

systemd-bootchart

systemd-bootchart is a tool that generates a SVG graph with information about processes resources utilization (cpu, memory and I/O).

It can be executed at any time but is usually ran at boot time to collect information during the boot process in order to analyse and optimise it.

systemd-bootchart can be executed at boot time by passing init=/lib/systemd/systemd-bootchart to the kernel command line.

There are different options that can be passed to bootchart to tune its execution. These can also be set on /etc/systemd/bootchart.conf. Refer to the systemd-bootchart manpage for information about the different options.

systemd-bootchart is provided in the development Apertis repository. Ensure this is installed and change the kernel command line by editing /boot/extlinux/extlinux.conf and adding init=/lib/systemd/systemd-bootchart to the end of the append argument for the required boot option. systemd-bootchart will be invoked as the init process and it will in turn fork and execute the real init so the system can be booted as normal while bootchart collect the system information during the boot process.

After a few seconds (20 by default but configurable in bootchart.conf) the data collection stops and the graph is generated. A message similar to the following will be shown:

systemd-bootchart wrote /run/log/bootchart-19700101-0000.svg

Displaying the svg generated will yield a graph similar to this:

1024px