RFC-0139: Bazel SDK

RFC-0139: Bazel SDK
StatusAccepted
Areas
  • General
Description

Plan and requirements to design and distribute an official Bazel SDK.
Gerrit change
Authors
Reviewers
Date submitted (year-month-day)2021-11-16
Date reviewed (year-month-day)2021-11-10

Edit this RFC

Edit RFC metadata

Summary

In order to support Workstation Out of Tree, the workstation.git repository will include a Bazel SDK that will initially support building and testing a product from pre-built artifacts. There are also plans to support driver development out of tree. To support both use-cases, this RFC proposes that the Workstation Bazel SDK is productionized for general distribution. Prototype rules are currently developed in workstation.git. A detailed design for general distribution will be included in a future RFC.

"SDK" does not refer to a replacement or change of Fuchsia's current supported API.

The Fuchsia project makes available an IDK and a GN C++ SDK. The IDK documentation details that the IDK, also distributed with the name "core SDK" on CIPD, is intended to be a base to build a "frontend SDK" on top of. There is an existing generated Bazel SDK frontend in fuchsia.git, however it is only intended as a test for the IDK, and is not supported for general usage. To avoid confusion, it should be renamed to "Selftest SDK"; the rest of this document will refer exclusively to the proposed out-of-tree Bazel SDK as the "SDK" or "Bazel SDK", while the IDK, or "core SDK" will be referred to exclusively as the "IDK".

Motivation

According to RFC-0095, Workstation will be built out-of-tree using Bazel. This proposal is for a roadmap to make these rules generally available for out-of-tree development beyond Workstation. Several important workflows are not currently supported by the current GN C++ SDK, most notably including product and driver development. These use cases should be supported separately from building the Fuchsia platform.

Supporting a generic, language-agnostic build-system such as Bazel is useful for a variety of reasons:

Generally, products built on Fuchsia – workstation included – can be expected to rely on a variety of dependencies from different sources. These include libraries, executables, components, packages, configuration files, and more. These artifacts may exist as source code or binary files in different languages and architectures, and may come from a combination of first party, third party, public, and private sources.

The IDK includes Fuchsia's complete API surface and a suite of development tools. While these vary in scope and purpose, the IDK by definition does not include generic build tools, and direct usage is discouraged - some tools are most conveniently used from scripts as opposed to invoked manually at a terminal.

"The contents of the IDK represent the most basic contract that the Fuchsia platform developers offer to prospective developers. The Fuchsia IDK is not suitable for immediate consumption. It does not contain any reference to toolchains or build systems, and in fact does not require any specific instance of these."

The existing GN C++ SDK is productively used by C++ projects already using GN, such as Flutter and Chromium, however there is no canonical Fuchsia SDK that supports official Fuchsia languages besides C++. GN cannot easily be extended to support all of Fuchsia's official languages, and rules cannot be easily composed unless they are designed to be combined with each other.

Bazel is a natural fit for building Fuchsia products and Fuchsia software. Bazel is supported by Google and an active community of third-party developers. It supports a wide range of runtime-targets and language toolchains by means of build-rules written in the Starlark language. These build rules can be published, shared across projects, and are used in production by both independent developers and large corporations. Starlark is fully testable, and offers testing utilities and a testing guide. Bazel is designed to be extended with new languages, new build-actions, and new means of integrating external dependencies. As a project, Bazel is focused on solving generic issues of distributed software builds and version management and has a clear scope of responsibility.

Stakeholders

Facilitator: hjfreyer@google.com

Reviewers: hjfreyer, mangini, chaselatta, amathes, sethladd, maruel, shayba, crjohns, ejia, chandarren, lijiaming, dworsham

Consulted: Driver team, SDK team, Infra team, Workstation Team.

Socialization: Sent to eng-council-discuss@fuchsia.dev

Design

A detailed technical design will be a follow-up to this RFC. This RFC outlines broad requirements for the RFC but does not specify how to implement them.

Versioning

The rules should at all times be pinned to the latest LTS version of Bazel. Bazelisk and a .bazelversion file should be the recommended way of installing and invoking Bazel for use with the Bazel SDK.

Each release of the Bazel SDK will also be pinned to a release version of the IDK and any required toolchains, such as Clang. The Bazel SDK will fetch the pinned IDK and toolchain archives depending on the host OS. The IDK and toolchain archives should also be overridable to local files for development.

Code Location

Workstation.git includes prototype Bazel rules to support assembling products, compiling C++ components, and compiling Flutter components. The rules are pinned against a particular version of the Fuchsia IDK and make use of Distributed Product Integration.

Development will move to a new repository called fuchsia-bazel-rules.git. This is because releases of the Fuchsia IDK are not created from a single checkout of fuchsia.git, and in order to consume the latest version of the IDK, the Bazel rules must exist downstream from fuchsia.git.

Distribution

A later RFC will clarify with specific design details for distribution.

IDK

The Bazel SDK will include a repository rule for instantiating a Bazel external repository with the contents of an IDK archive as well as generated BUILD targets for the contents of the IDK, including tools, language-specific libraries and FIDL definitions. The archive should be downloaded by ffx or directly from CIPD. The rule may download ffx first in order to obtain the IDK.

Building

The following are the primary items that the Bazel SDK must provide:

C++, Dart, and Flutter toolchains for building libraries and executables. Rust is not needed initially for building Workstation but can be added later. The Bazel toolchain definitions should be hosted in the same repository as the Bazel SDK, but could be moved externally in the future.

Providers and rules for assembling a Fuchsia product from a combination of local and remote Fuchsia packages.

Actions for deploying a particular product image to a Fuchsia device, or emulator.

Rules for compiling and packaging drivers that can be included in product definitions.

A Bazel toolchain that wraps tools from the IDK. The IDK is fetched by Bazel (potentially via ffx, which may also be fetched by Bazel) to build products according to the version specified in BUILD/WORKSPACE rules. This is for making IDK binary tools available to Bazel rules.

A provider and rule for Fuchsia packages that contains a package manifest and associated files, similar to the in-tree fuchsia_package rule.

A provider and rule for creating Fuchsia components, with a component manifest and associated files, similar to the in-tree fuchsia_component rule.

Testing

This section refers to testing-related Bazel rules, not testing the Bazel SDK itself. The primary requirement is to support OOT System Testing.

Dependencies and External Repositories

These will be provided as analysis-phase repository rules.

  • Downloading build artifacts that are built by external systems and hosted on TUF. See Phase 3 of Workstation OOT RFC.
  • Repository rules for building against a local checkout of the Fuchsia tree.
  • Repository rules for building against a particular version of the Fuchsia IDK, downloaded from CIPD.

Implementation

  • Prototype rules are being developed in workstation.git
  • These rules will move to a new git repository (fuchsia-bazel-rules.git).
  • A follow-up RFC will be published detailing the specifics of both distribution and the public API surface of the Bazel rules.

Testing

Initially, testing will run continuously on Fuchsia infrastructure as part of workstation.git's continuous testing and release process. Tests should be able to run in a Mac or Linux environment. Windows support is not planned at this time, and will not be considered until there is a Windows distribution of the IDK. To keep the rules compatible with that possible eventuality, rules will avoid use of run_shell and actions that rely on built-in toolchains. Go's toolchain is a good candidate, as it is well supported on Linux, macOS, and Windows, with good cross-compiling capabilities from all three platforms. See rules_go.

The Bazel SDK should include Starlark unit testing. Each rule in the public API of the Bazel SDK will include tests and examples that will be build as part of CI.

Bazel can download artifacts (dependencies, toolchains, etc) as part of the analysis phase of its build process. To support this in Fuchsia CI, Bazel offers several mechanisms for downloading any necessary artifacts in advance: Bazel offline builds.

When the rules move from workstation.git to fuchsia-bazel-rules.git, the CI recipe will follow the same design as the workstation.git recipe.

Documentation

Stardoc should run along with the continuous testing to generate formatted documentation within the repository. This documentation should be integrated into fuchsia.dev where additional documentation will explain the recommended way for users to install Bazel (using bazelisk).

Drawbacks, alternatives, and unknowns

Drawbacks

  • Driver development on Windows. The Fuchsia IDK does not support Windows so rules that depend on it will not work on Windows systems.

Alternatives

  • Expand the GN C++ SDK to support the above workflows.
    • Pros
      • Builds on existing work, incremental.
      • Some rules may be migrated directly from fuchsia.git with minimal changes.
      • Most Fuchsia developers have some experience with GN.
    • Cons
      • There is not a simple version scheme for releases of GN - projects are tied to specific GN releases, and there's no guarantee of forwards or backwards compatibility. This is important because these rules are expected to be used in conjunction with rules distributed by other parties.
      • GN is not designed for build rules to work across multiple projects.
        • Build rules are generally written to support being built in a specific project - usually Chromium.
        • It is common for GN rules to define/reference a build_with_chromium variable to selectively support being built as part of Chromium, or use dependencies whose rules were written for use in the Chromium project.
      • The above reasons fall below Fuchsia's intended level of API stability.
  • Use another build system such as CMake, Buck, Meson, etc.
    • Pros
      • Another system may be more popular with users, and therefore more familiar to work with.
      • Bazel is a pretty large dependency, and runs a Java daemon. Some alternatives are much leaner.
    • Cons
      • None are supported directly by Google
      • Most do not have an as large multi-language rule ecosystem as Bazel.
      • Most do not support content-addressed storage for distributed builds.

Prior art and references