Tests and test suites

Overview

This page provides style and best practices for defining and structuring tests and test_suite() targets in BUILD.bazel files and connecting them to the GN build graph. The guidance on this page is in addition to the common Bazel style guide and best practices and the BUILD.bazel files style guide. For instructions on how to define, build, and run host and device test targets, see Bazel tests.

This page is part of the Bazel style guide and best practices.

Test suite structure

In Fuchsia's Bazel build graph, tests are organized and connected to the build in three layers:

  1. Individual packages: Each BUILD.bazel file defines private *_test targets and exposes them via package test_suite() targets named "tests" and/or "host_tests" (see Defining tests and test suites in individual packages).

  2. Aggregation packages: Dedicated packages for an area or subsystem group the package test_suite() targets from individual packages into aggregation test_suite() targets (for example, "bluetooth_tests").

  3. GN build graph integration: While GN drives the overall build, bazel_test_suite() targets in BUILD.gn reference the aggregation test_suite() targets in the same directory so the tests are included in tests.json (see Connecting Bazel test suites to the GN build graph).

Individual tests MUST NOT be referenced outside the package in which they are defined, and packages do not recursively include tests or test_suite() targets from their subpackages. See Why only expose test_suite() targets? for the rationale behind this structure.

Example

//src/connectivity/bluetooth/lib/packet-encoding/BUILD.bazel (individual package):

test_suite(
    name = "tests",
    tests = [],  # Include all tests in the package.
    visibility = ["//src/connectivity/bluetooth:__pkg__"],
)

# Private test target(s) automatically included in ":tests" above.
...

//src/connectivity/bluetooth/BUILD.bazel (aggregation package):

test_suite(
    name = "bluetooth_tests",
    tests = [
        "//src/connectivity/bluetooth/lib/packet-encoding:tests",
        ...
    ],
)

//src/connectivity/bluetooth/BUILD.gn (GN integration):

import("//build/bazel/bazel_test_suite.gni")

bazel_test_suite("bazel_tests") {
  target_tests = [ "//src/connectivity/bluetooth:bluetooth_tests" ]
}

group("tests") {
  testonly = true
  deps = [
    ":bazel_tests",
    ...
  ]
}

Defining and structuring Bazel tests and test suites

Target names and visibility

  • Target names: The names of test targets always end in "_test". The names of Bazel test_suite() targets should be or end in "tests" or "host_tests" as appropriate.

  • Test target visibility: Individual tests MUST NOT be referenced outside the package in which they are defined. Test targets must always have package-private visibility, which is generally the default (see Target visibility).

  • test_suite() target visibility: test_suite() targets should only be visible to package(s) containing the aggregation test_suite() targets that depend on them (in their tests attribute). Always use :__pkg__ rather than :__subpackages__.

  • Referencing tests and test_suite() targets: Tests and test_suite() targets must only be referenced by test_suite() targets (in their tests attribute). Do NOT put them in a filegroup() or similar.

Platform and CPU architecture compatibility

Target platform compatibility (target_compatible_with)

Test targets can and should specify target_compatible_with as appropriate (for example, target_compatible_with = HOST_OS_CONSTRAINTS on host tests; see target_compatible_with).

However, do not rely on Bazel test compatibility to filter test targets and only run compatible ones. Specifically, do not group tests for different platforms in a single test_suite() (or test_suite() targets containing tests for different platforms in a single test_suite()).

It is TBD whether test_suite() targets should specify target_compatible_with.

CPU architecture

Avoid defining separate tests or test_suite() targets for, for example, each CPU architecture where possible. Ideally, a single test target can be built for all CPU architectures.

In rare cases, there may be CPU architecture-specific tests, such as unit tests for a SIMD library written in assembly. In these cases, seek guidance from the Build team. (Once we have some experience with these scenarios, we can update this text.)

Defining tests and test suites in individual packages

Exposing package test_suite() targets

Each Bazel package (BUILD.bazel file) should expose one or more package test_suite() targets that depend on the tests defined in that package (in the single BUILD.bazel file, not in any of its subpackages).

These targets should be named "tests" or "host_tests" depending on the target platform. All such test_suite() targets should be the first targets in a BUILD.bazel file, ideally immediately after load() and package() statements.

Including all tests in the package

All tests defined in a Bazel package must be a dependency of one of the test_suite() targets in the same package. (In the future, this may change to "at least one".) This includes tests defined by macros used to define non-test targets.

If all tests in a package should be in a given test_suite(), specify an empty tests attribute with a comment noting that all tests in the package are included:

test_suite(
    name = "tests",
    tests = [],  # Include all tests in the package.
    visibility = [...],
)

When possible, this is easier to maintain, requires one less step when adding tests, and ensures that a test is not missed.

Defining a default test_suite() in all packages

Where possible, all BUILD.bazel files should have a test_suite() named "tests" or "host_tests" with an empty tests attribute (tests = []) regardless of whether they explicitly define tests. Each such test_suite() should be referenced from an appropriate aggregation test_suite() target.

This ensures that any tests added to the package in the future - either explicitly or by a macro - will be included in the build graph. If we can enforce this, it also prevents forgetting to add a test to the build graph.

The primary exceptions to this are packages that contain tests for both Fuchsia and host because it is not currently possible to use the same test target in both cases and thus the empty tests attribute mechanism cannot be used.

Modifying a test_suite() with an empty tests attribute

When modifying an existing test_suite() whose tests attribute is either empty or not specified, you MUST ensure that all tests defined by the package are explicitly added to this or another test_suite() in the package. The only reason this should occur is if a package's tests are being divided among multiple test_suite() targets.

Do not depend on subpackages

Do not depend on tests or test_suite() targets in the package's subpackages. The exception is aggregation test_suite() targets, but a single package should generally not have both types of test_suite() targets.

This is a departure from the common pattern in GN where group("tests") depended on all such targets in its subdirectories, which recursively did the same. We do NOT want to repeat that pattern in Bazel.

Aggregation test_suite() targets

These test_suite() targets group package test_suite() targets from individual packages (for example, for an area or subsystem) and should have names that describe their scope and/or purpose. They should still end in "tests" or "host_tests" as appropriate. There may be more than one of each such target. As a hypothetical example, "bluetooth_tests" might apply to all products while "bluetooth_le_tests" might include additional tests to run on products that support Bluetooth LE.

In general, packages (BUILD.bazel files) containing such aggregation test_suite() targets should not themselves define other targets, especially test targets. For example, //src/connectivity/bluetooth/. (Note also that the BUILD.gn file only defines group() targets.)

Depending on the aggregation targets

In the future, this section will describe optional further aggregation and inclusion in the test bundles for each product.

For now, the existing dependencies to the GN tests group() targets that depend on the bazel_test_suite() targets are used.

Rust unit tests

In GN, the rustc_*() templates accept a with_unit_tests parameter that when true automatically generates unit test targets from the tests in the source files.

Connecting Bazel test suites to the GN build graph

Targets defined with the bazel_test_suite() GN template should only reference aggregation test_suite() targets defined in the same directory (in a BUILD.bazel file next to the BUILD.gn file). They must not reference individual tests.

Each aggregation test_suite() target must be referenced in the appropriate field of a target defined by the bazel_test_suite() GN template in the same directory (in a BUILD.gn file next to the BUILD.bazel file) so that its tests are included in tests.json. Use full Bazel labels starting with "//" rather than relative ":" labels (for example, "//src/connectivity/bluetooth:bluetooth_tests"), and reference the targets in the GN template's parameters as follows:

  • Targets whose names end in "host_tests": host_tests

  • All others (ending in "tests"): target_tests

Appendix: Why only expose test_suite() targets?

Advantages of only exposing test_suite() targets rather than individual tests include:

  • visibility only needs to be specified for the test_suite() target(s) and not for every individual test target.

  • This is especially relevant for macros such as Fuchsia's rustc_*() and ffx_plugin() that support with_host_unit_tests and/or with_unit_tests (see Rust unit tests) because the visibility of the test targets cannot be specified independent of the production target. If the individual tests were referenced outside the Bazel package, the visibility of the non-test target would need to include the Bazel package that depends on the test.

  • Which tests exist and what they are named is abstracted away from the aggregation test_suite() targets.

  • There is no need for OWNERS approval for the package containing the aggregation test_suite() target(s) when adding, renaming, or removing tests.

  • It is more likely that someone will remember to update test_suite() target(s) in the same BUILD.bazel file than one (or multiple) somewhere else in the tree.

  • Test inclusion can be handled automatically in some cases by specifying an empty tests attribute (see Defining tests and test suites in individual packages above).