Freedesktop SDK cross compilers

June 25, 2019 6-minute read

In upcoming Freedesktop SDK 19.08 release, we will introduce cross compilers. x86-64 will be able to cross-compile to all other available architectures. aarch64 will be able to cross-compiler to armv7.

Motivation

Legacy 32-bit support

Freedesktop SDK does not provide support for multilib. Multilib is having /usr/lib and /usr/lib64 as library directories and support for -m32 in GCC.

In order to support applications providing both 64-bit and 32-bit binaries, the 32-bit library directory was mounted in a different directory. The original path of 32-bit libraries would conflict with the path of the 64-bit ones. So the 32-bit library were in a directory that they were not intended to be. This directory was added in LD_LIBRARY_PATH originally, and later in ld.so.cache.

However, due to recurrent issues with some 32-bit games in Steam, multiarch library paths were introduced in the 18.08 release. The library directories are like like /usr/lib/x86_64-linux-gnu. Multiarch comes from Debian. It is a solution that is more generic than multilib, more future-proof, and cross-compilation and emulation friendly.

32-bit runtime is extracted from the 32-bit build, and then added to at the right library path as an extension to the 64-bit runtime.

Unfortunately, applications needing to build extra 32-bit libraries that were not in the Freedesktop SDK would have needed some complicated hacks when using Flatpak Builder. This is because -m32 is not supported. The 32-bit libraries need to be built with the 32-bit SDK and then extracted and reimported.

Providing a cross-compiler as well as a multiarch SDK would solve this issue.

Steam was the only application needing 32-bit support. At first, no extra 32-bit runtime that was not in Freedesktop SDK was required. This has changer. Also now Lutris, which is available as beta Flatpak application on Flathub requires 32-bit Wine and is waiting for 32-bit cross-compilers.

Building applications for smaller devices

Phones, tablets and netbooks often use ARM. But developers typically develop on x86-64. Most developers do not own an ARM board to build their applications.

For now, those who wanted to test building Flatpak applications for ARM needed to use QEMU. The issue is that running the compiler on emulation is potentially slow. And it is not unlikely that applications need to build some big libraries like WebKit. This would take an awful long time.

Cross-compile with FlatpakBuilder

Let’s say we want to cross-compile GNU Hello, from an x86_64 machine to aarch64.

First we need to install the 19.08 SDK. This will come soon as beta on flathub. But for the moment, we can add remote https://cache.sdk.freedesktop.org/freedesktop-sdk.flatpakrepo

We will need to install:

org.freedesktop.Sdk//19.08, the build SDK.
org.freedesktop.Sdk.Extension.toolchain-aarch64//19.08, the cross compiler extension. It contains mostly the linker and the compiler.
org.freedesktop.Sdk.Compat.aarch64//19.08, the SDK for the target runtime. This contains the libraries to link to, the pkg-config files, and the architecture dependent headers.
org.freedesktop.Platform/aarch64/19.08, the target runtime. This is not used to build, but it is required to be installed by Flatpak Builder.

We will then create file org.gnu.Hello.yml:

app-id: org.gnu.Hello
# We need to force here the target architecture for the runtime
runtime: org.freedesktop.Platform/aarch64
runtime-version: '19.08'
# To avoid confusion we can set the architecture for the SDK. But this is
# not important.
sdk: org.freedesktop.Sdk/x86_64
command: hello
sdk-extensions:
# We need to ensure the toolchain is available
- org.freedesktop.Sdk.Extension.toolchain-aarch64
# As well as the target SDK
- org.freedesktop.Sdk.Compat.aarch64

modules:
  - name: hello
    buildsystem: autotools
    build-options:
      # Add the cross compiler to the PATH
      append-path: /usr/lib/sdk/toolchain-aarch64/bin
      # binutils are available there in a different path. GCC knows
      # where to get them. But if we need access to those tools,
      # they can be added to the path that way:
      # prepend-path: /usr/lib/sdk/toolchain-aarch64/aarch64-unknown-linux-gnu/bin

      # In this case we do not need pkg-config. But if needed,
      # we can add this path:
      # prepend-pkg-config-path: /usr/lib/aarch64-linux-gnu/pkgconfig
      env:
        # Because default CFLAGS are architecture-specific, we need
	# to reset them to something more portable.
        CFLAGS: -g -O2
        CXXFLAGS: -g -O2
      config-opts:
      # This is a standard option for autoconf to cross-comile
      - --host=aarch64-unknown-linux-gnu
    sources:
      - type: archive
        url: https://ftp.gnu.org/gnu/hello/hello-2.10.tar.gz
        sha256: 31e066137a962676e89f69d1b65382de95a7ef7d914b8cb956f41ea72e0f516b

With this we can now build our application.

# We build and commit the app in repo/
flatpak-builder build org.gnu.Hello.yml --repo=repo

# We create a new repository containing the commits for aarch64
ostree init --mode archive-z2 --repo=aarch64-repo

for i in app/org.gnu.Hello \
         runtime/org.gnu.Hello.Debug \
	 runtime/org.gnu.Hello.Locale
do
     # We rename the commits to for them to be for aarch64
     flatpak build-commit-from \
        --src-ref=${i}/x86_64/master \
	--src-repo=repo \
	aarch64-repo \
	${i}/aarch64/master
done

# Now we can create a bundle for our application
flatpak build-bundle --arch=aarch64 \
                     aarch64-repo \
		     org.gnu.Hello-aarch64.bundle \
		     org.gnu.Hello

We will then get a bundle called org.gnu.Hello-aarch64.bundle. We can transfer it to a device to be installed and tested.

Cross-compile with BuildStream

Let’s do the same exercise but using BuildStream instead.

To cross-compile in BuildStream, cross-compilers/gcc-<arch>.bst needs to be used. The runtime also needs to be imported. For the moment only the standard runtime is provided (glibc, GCC’s libstdc++ and GCC’s runtime). This is because BuildStream does not allow yet to import artifacts built from different architectures. For our example, GNU Hello, we do not need anything else than glibc and GCC’s runtime. This runtime is available from cross-compilers/standard-libs-<arch>.bst

Here is the project.conf.

name: hello
ref-storage: project.refs

variables:
  # Application files should be installed in /app for flatpak applications
  prefix: /app

  # Debugdir is still unchange, we will extract files at the right place
  debugdir: /usr/lib/debug

# We enable flatpak related plugins
plugins:
- origin: pip
  package-name: buildstream-external
  elements:
    flatpak_image: 0
    flatpak_repo: 0

environment:
  # We add the aarch64 toolchain in the path
  # If some tools use binutils without querying `gcc -print-prog-name`,
  # we can add `/usr/lib/sdk/toolchain-aarch64/aarch64-unknown-linux-gnu/bin`.
  PATH: '/usr/lib/sdk/toolchain-aarch64/bin:/usr/bin'

  # In theory we should add pkg-config path. In practice,
  # we do not yet have the runtime available on BuildStream.
  PKG_CONFIG_PATH: '/usr/lib/aarch64-linux-gnu/pkgconfig:'

elements:
  # Autotools supports cross-compilation out of the box. We enable it
  # by default for all our elements.
  autotools:
    conf-global: >-
      --build=x86_64-unknown-linux-gnu
      --host=aarch64-unknown-linux-gnu

The Freedesktop SDK junction freedesktop-sdk.bst:

kind: junction

config:
  options:
    bootstrap_build_arch: "x86_64"
    target_arch: "x86_64"

sources:
- kind: git
  url: https://gitlab.com/freedesktop-sdk/freedesktop-sdk.git
  track: master

And here is the element hello.bst.

kind: autotools

depends:
- filename: public-stacks/buildsystem-autotools.bst
  junction: freedesktop-sdk.bst
  type: build

  # This is the cross-compiler
- filename: cross-compilers/gcc-aarch64.bst
  junction: freedesktop-sdk.bst
  type: build

  # This is the aarch64 runtime to which we need to link
- filename: cross-compilers/standard-libs-aarch64.bst
  junction: freedesktop-sdk.bst
  type: build

sources:
- kind: tar
  url: https://ftp.gnu.org/gnu/hello/hello-2.10.tar.gz

The main flatpak element org.gnu.Hello.bst:

kind: flatpak_image

depends:
- filename: hello.bst
  type: build

config:
  directory: '%{prefix}'
  exclude:
  - debug

  metadata:
    Application:
      name: org.gnu.Hello
      runtime: org.freedesktop.Platform/aarch64/19.08
      sdk: org.freedesktop.Sdk/aarch64/19.08
      command: hello

    'Extension org.gnu.Hello.Debug':
      directory: lib/debug
      autodelete: true
      no-autodownload: true

The debug extension element org.gnu.Hello.Debug.bst:

kind: flatpak_image

depends:
- filename: hello.bst
  type: build

config:
  directory: '%{debugdir}%{prefix}'

  metadata:
    Runtime:
      name: org.gnu.Hello.Debug

    ExtensionOf:
      ref: 'app/org.gnu.Hello/x86_64/master'

And finally the flatpak repository flatpak-release-deps.bst:

kind: flatpak_repo

depends:
- filename: org.gnu.Hello.bst
  type: build
- filename: org.gnu.Hello.Debug.bst
  type: build
- filename: deploy-tools/flatpak.bst
  junction: freedesktop-sdk.bst
  type: build

config:
  arch: aarch64
  branch: master

  environment:
  - deploy-tools/flatpak.bst

In order to build and bundle our application we now need to run the following commands.

# First we need to track separately junctions.
bst track freedesktop-sdk.bst
# Then we track our elements.
bst build --track hello.bst
# We build the flatpak repository and extract it
bst build flatpak-repo.bst
bst checkout flatpak-repo.bst repo

# We then convert the repository to a bundle
flatpak build-bundle --arch=aarch64 \
                     repo \
		     org.gnu.Hello-aarch64.bundle \
		     org.gnu.Hello

We will then get a bundle called org.gnu.Hello-aarch64.bundle similar to the one built with the FlatpakBuilder method.

Aknowledgements

This work has been sponsored by Codethink Ltd.