See Application Layout for more details of how store and built-in application bundles are arranged.

Assumptions

  • Store application bundles are arranged according to the Application Layout.
  • Built-in application bundles are arranged according to the Application Layout.
  • Platform upgrades are somewhat frequent, although not as frequent as store application bundle installation, upgrade or removal.
  • Rollbacks are supported, but are relatively infrequent.

Requirements

Application bundles

Suppose com.example.BuiltInApp is a built-in application bundle. A platform update from version 42 to version 43 results in BuiltInApp being updated. Later, the user requests a system rollback. It is possible that the version of BuiltInApp in platform v43 has migrated user files to a format not understood by the version in platform v42, so we must roll back the user data.

  • During upgrade, the platform must take a snapshot of the data stored on behalf of BuiltInApp in platform v42. The upgraded system v43 must have a separate read/write data area for BuiltInApp, mounted in the same place and starting from the same state as that snapshot. (System Updates & Rollback design, §6.3; Applications design, §4.2.2)
  • During rollback, the platform must discard the data stored by the version of BuiltInApp that was present in platform v43, replacing it with the data stored by the version of BuiltInApp that was present in platform v42 (System Updates & Rollback, §6.3; Applications, §4.2.4), with the exception that cached files may be deleted instead of being rolled back.
  • Because built-in applications are upgraded and rolled back as part of the platform, it is not necessary to support rolling back built in applications’ data individually; in other words, it would be acceptable to have a single snapshot that contains the data for all built-in applications together.
  • Because we have assumed that upgrades and rollbacks will not be very frequent, it is acceptable for performing an upgrade or rollback involving n built-in application bundles to take O(n) operations; in other words, it would also be acceptable to have a snapshot per built-in application even though it is not necessary.

Suppose net.example.ThirdPartyApp is a store application bundle that was installed before the upgrade described above.

  • During upgrade, under certain circumstances (if the platform API level increased between platform v42 and v43), the platform must take a snapshot of ThirdPartyApp and its data. (System Updates & Rollback, §6.4)
  • During rollback, under certain circumstances, the platform must restore that snapshot. (System Updates & Rollback, §6.4)

Platform services

Some platform services store configuration, state and/or cached files outside the scope of a particular application bundle. There are various use-cases for how these interact with upgrades and rollbacks.

System-level components may store configuration and state that is valid for the system as a whole, independent of users. For example, if ConnMan is configured to provide a wireless hotspot in the car, the SSID and password should be preserved during upgrades, but rolled back during rollbacks.

  • There must be a storage location for user-independent system configuration and state, which has upgrade/rollback behaviour similar to a built-in application: upgrades preserve it, but rollbacks roll it back.
  • The Applications design calls for /var/lib to have this behaviour. (Applications design, §3.2)

User-level platform components may store configuration and state on a per-user basis, independent of any particular application bundle. For example, the Evolution data server stores address book and calendar data for a particular user.

  • There should be a storage location for user-specific platform-level configuration and state, which again has upgrade/rollback behaviour similar to a built-in application: upgrades preserve it, but rollbacks roll it back.

System-level components may store configuration and state that is only valid for a particular operating system image. For example, the state of the dpkg and apt package databases is invalidated by platform upgrades.

  • There should be a storage location for user-independent system configuration and state that is tied to a particular operating system image.
  • We anticipate that this will only be needed in special cases (such as apt), so this may be a special case that is only available to specific components.
  • We do not anticipate that a similar feature will be needed for per-user configuration and state.

Certain system-level components may store configuration and state that is valid for the system as a whole, and must be preserved across upgrades and rollbacks. For example, the list of blacklisted platform upgrades (System Updates & Rollback, §3.1) should be preserved and extended after a rollback. It would also be sensible to avoid /var/cache and /var/log participating in snapshots. (Applications design, §3.2)

  • There must be a storage location for system configuration and state that survive rollbacks. Platform components must be careful to use extensible formats with robust parsers for anything in this location, since it may have been written by a newer version of the same software.
  • The Applications design calls for /var to have this behaviour. (Applications design, §3.2)
  • /var/cache must be cleaned during system rollback. (Applications design, §3.2)
  • /var/log must not be cleaned during system rollback, since it is likely to be key to analyzing why a rollback was required. (Applications design, §3.2)

We anticipate that large media files should be stored in shared space (as of September 2015, /home/shared). It would not be appropriate to carry out snapshots or rollback for these files.

  • There must be a storage location that is not subject to snapshot and rollback. Large media files should be stored there.
  • User-level components may write other data to this location, with the same caveats as for system-level non-rollback data. (Applications design, §4.1.5)

System- and user-level components may have cached files that are unimportant and may be deleted. For example, Tracker stores a database of media files per user. This is non-critical and can be rebuilt if necessary, at some performance penalty.

Application bundles that extend the platform

Certain application bundles provide extensions for the operating system. ‘’(System Updates & Rollback, §6.2; Applications, §4.3).

  • See the Applications design document for details of the requirements in this area.

Developer considerations

Mounting btrfs subvolumes can lead to difficulties with debugging and documenting the system architecture: it is non-obvious which physical locations (subvolumes) correspond to which logical locations (mounts).

To make the system more debuggable, we recommend that the root of each underlying btrfs volume has a standard mount path, for example /System and /General. These could either be mounted at boot time, or left unmounted but configured as noauto in /etc/fstab so that they can be mounted for debugging.

Production code should not normally use these locations, although the mount management service may use them if desired.

This would facilitate references to physical locations as, for example, /System/root_1/usr or /General/apertis_extensions in developer documentation.

Security considerations

If ordinary user processes can access an outdated setuid binary that has an exploitable security vulnerability, then they may be able to escalate privileges using the vulnerability, even if the latest version of the corresponding binary has been fixed.

As a result, old snapshots should be mounted below a directory that is not accessible by ordinary user processes. We recommend that when the root of a btrfs volume is mounted, for example at /System and /General, it should be owned by root, with rwx------ permissions.

Similarly, AppArmor rules should be configured so that those mount points can only be accessed by the snapshot/subvolume manager (Ribchester) and by unconfined processes.

Implementation as of September 2015

Logical layout

Directories that are a mount point for a separate subvolume are emphasized in this diagram. See Physical layout, below, for how they are actually arranged on-disk.

  • /
    • Applications
    • bin (early-boot programs)
    • boot (kernel and initramfs files)
    • dev (virtual filesystem for device nodes)
    • etc (system configuration)
    • home
      • shared (shared files for all users)
      • user (personal files for user named “user”)
    • lib (early-boot libraries)
    • media (transient mount points for removable devices)
      • user (removable devices for user named “user”)
        • USBSTICK (a USB drive with a label)
        • 1234_5678 (a USB drive with a serial number)
    • mnt (empty, not used on Apertis)
    • opt (empty, not used on Apertis)
    • proc (virtual filesystem for process information)
    • root -> var/root (sysadmin’s home directory, should not be used on Apertis)
    • run (transient runtime data for the system)
      • dbus (contains the system-level D-Bus socket)
      • user (parent directory for users’ runtime data areas)
        • 1000 (transient runtime data for uid 1000)
    • sbin (early-boot programs)
    • srv (not used by Apertis)
    • sys (virtual filesystems for system information)
    • tmp (temporary directory for all users, avoid using it)
    • usr (static data and programs)
      • Applications
      • bin (platform programs)
      • lib (platform libraries and data)
      • share (platform data)
        • arm-linux-gnueabihf etc. (platform libraries with “multiarch” support)
    • var (variable platform data)
      • cache (system-wide caches)
      • lib (assorted variable platform data)
        • MILDENHALL_extensions (platform plugins installed by applications) (should be called apertis_extensions)
        • Services (FIXME: document this)
        • …

Where not otherwise stated, directories have the same purpose as their equivalent in Debian.

Root filesystem

The root directory is mounted read-only. Its subdirectories /bin, /boot, /lib, /usr and so on contain the static files of the Apertis platform.

Either the system_1 or system_2 subvolume from the system partition is mounted here, whichever one is currently in use.

/Applications

The applications subvolume on the general partition is mounted at /Applications.

/Applications/(store app bundle)

Store applications are installed in a subdirectory of /Applications named after their bundle ID.

The current applications/System/Application-Subvolumes/(bundle ID) subvolume from the general partition is mounted here.

/Applications/(built-in app bundle)

Built-in application bundles’ program files are stored in subdirectories of /usr/Applications, but that location is read-only, so their read/write data is stored in a corresponding subdirectory of /Applications.

The current applications/System/Application-Subvolumes/(bundle ID) subvolume from the general partition is mounted here.

/etc

/etc is physically stored on the root directory, but is remounted to be read/write. It holds basic system configuration.

/home

The home subvolume from the general partition is mounted here. One subdirectory per user contains user-specific but non-app-specific data.

/home/shared

The shared subdirectory of /home contains files available to all users.

/usr/Applications

/usr/Applications is an Apertis-specific subdirectory of /usr. It contains built-in application bundles, each in its own directory.

/var

/var is physically stored on the root directory, but is remounted to be read/write. It holds the mutable state of the system.

/var/lib/MILDENHALL_extensions

The applications/System/System-Subvolumes/var/lib/MILDENHALL_extensions subvolume from the general partition is mounted here.

/var/lib/Services

The applications/System/System-Subvolumes/var/lib/Services subvolume from the general partition is mounted here.

Physical layout

There are two major partitions.

System partition

The system partition contains the Apertis platform.

Mount points (locations where other filesystems will be mounted) are physically empty directories.

Subvolumes are shown in bold here.

  • (system partition)
    • system_1
      • Applications
        • bin
        • boot
        • …
    • system_2
      • Applications
        • bin
        • boot
        • …

General partition

The general partition contains store application bundles and user data.

Again, subvolumes are shown in bold here.

  • (general partition)
    • applications
      • System
      • Application-Subvolumes
        • com.example.BuiltInApp
        • com.example.StoreApp
      • Snapshots
      • System-Subvolumes
        • var
          • lib
            • MILDENHALL_extensions
            • Services
      • com.example.BuiltInApp (empty directory to use as a mount point)
      • com.example.StoreApp (empty directory to use as a mount point)
    • home
      • shared
      • user

Differences between this setup and the design documents

The Applications design document calls for /var and /var/lib to be separate, with /var read/write and not participating in rollbacks, while /var/lib does participate in rollbacks. (§7 “Application Storage”

The Applications design document calls for the directory shared by all users to be /home/_Shared.

The Applications design document calls for the application subvolumes to have version-dependent names.

The Applications design document documents a more complex directory hierarchy with Application/Bundle/{bin,lib,...}, User Storage/$user and Everyone Storage instead of {bin,lib,...}, users/$uid and {cache,config,data} inside an application bundle’s data directory.

The Applications design document calls for the general partition to be mounted at /General.

The System Updates & Rollback design document calls for separate home, var and Applications subvolumes within the general partition, with var mounted at /var.

Many system-wide files such as /Applications are currently owned by the user named “user”, which prevents correct multi-user behaviour. We recommend having a strict division between system files (user-independent and only written by root or other system uids), shared files (user-independent but written by all users), and user-dependent files.

The /var/lib/Services directory is not present in the design documents. Its interaction with rollback is not clear. We recommend clarifying the desired interaction with rollback and multi-user, and revisiting this part of the design.

Recommendations

This section is incomplete.

Design principles

When choosing where files are to be situated, we recommend answering these questions as a guide:

  • Is the file intended to be read by system uids, or by a single user, or by multiple users?
  • Is the file intended to be written by nobody (read-only data), by system uids, or by a single user, or by multiple users?
  • Is the file intended to be read by platform components, by a single application bundle, and/or by all application bundles?
  • Is the file intended to be written by platform components, by a single application bundle, and/or by multiple application bundles?
  • If the file is not read-only, which category does it most closely fit: configuration, cache, or valuable data? To choose between these, consider what the user-visible impact would be for deleting the file. If the user would be angry to have lost something that they created, then the file was valuable data; if the user would be at annoyed that some settings/preferences had returned to their default values, then the file was configuration; and if the user would not notice the difference (except for perhaps a performance impact), it was cache.
  • If the application or platform is rolled back from version 2 to version 1, is the file expected to be kept intact at its version-2 state, rolled back to its version-1 state, or deleted?

Logical layout

We recommend that mount points for the system and general partitions are reserved at /System and /General respectively. The top-level directory of each partition should be owned by root and have permissions rwx------ (see Security considerations).

Ribchester or another platform component may mount those locations, if they are required to be mounted in order to make platform components work as desired. If not, we recommend that they are listed in /etc/fstab with the noauto flag, so that they are not mounted by default but developers can easily mount them for debugging.

Physical layout

This section is written with the assumption that the root directories of the system and general partitions are available at /System and /General respectively.

The current layout of the system partition, with two subvolumes /System/root_1 and /System/root_2, is suitable and should be kept.

/General/applications should not be mounted on /Applications. Instead, we recommend that /Applications should be a tmpfs (in-memory file system), which should be populated with empty directories to act as mount points for application bundles. All directories in that tmpfs should be owned by root, with the standard permissions (rwxr-xr-x).

All subdirectories and subvolumes that are not specific to a particular application bundle, such as /General/applications/System, should be moved from /General/applications into /General.

References