Fuchsia tests.json file format

Fuchsia's build system exports structured information about all tests included in a build in a tests.json file in the build output directory (e.g. $FUCHSIA_DIR/out/default/tests.json).

Fuchsia's continuous integration infrastructure uses tests.json to determine which tests to run, and fx test uses it to determine which tests may be run locally. tests.json and its schema are implementation details of the platform build and test and are not part of the SDK.

This document describes the file's contents and how the file is generated.

Generating tests.json

The tests.json file is generated immediately after gn-gen in regenerator.py.

The fields for tests.json entries are collected using the GN metadata mechanism. tests.json entries are registered from GN files by using the test_spec GN template.

tests.json is automatically generated by fx set. The tests.json format is limited by what can readily be generated from GN using GN's relatively limited metadata mechanism. To get around these limitations, the Fuchsia build also produces file called test-list.json, which is generated by post-processing tests.json as described below.

tests.json consumers

Various tools and scripts ingest the tests.json file since it is the canonical list of available tests for a given Fuchsia build. This section surveys the most notable consumers of the format.

test_list_tool

The test_list_tool processes tests.json. Using the canonical list of tests, it uses the contained package manifest information to load the build packages and annotate each test with information about how it should be executed and tags for additional categorization. For example, it can determine if a test component runs hermetically, and tags the test with "hermetic": true.

The test_list_tool outputs this information to a test-list.json file adjacent to tests.json, and it is run as one of the last build steps after all packages are built. It uses the same unique name key as tests.json so that entries can be matched.

fx search-tests

The fx search-tests script provides fuzzy matching on test names to help developers find which tests are available to run. It uses the values in tests.json as keys for fuzzy matching, in addition to searching for non-included tests through the source tree.

fx test

The fx test command is the developer-facing entrypoint to test execution in the Fuchsia source tree. It uses tests.json to determine the canonical set of tests, and then finds associated data in test-list.json and test_components.json to determine how a test should be executed.

botanist / testsharder

In Infra, testsharder depends on tests.json to shard test entries based on environment dimensions. The output of that process is another file containing fields derived from tests.json which is then passed to botanist on the bots where tests run.

tests.json structure

tests.json is a JSON file consisting of a single array. Each element of the array corresponds to a single Test Target entry in the build.

Test Target entries have two top-level fields, environments and test corresponding to Test Environments and Test Info respectively.

Fields

environments field

The environments field is an array of Test Environment entries. Each Test Environment entry specifies a single environment the test may run in, and this field is used by testsharder to shard tests between device configurations.

Each Test Environment entry has a single field dimensions with any number of arbitrary dimensions. Typically, however, there are two sets of fields that are most common:

device_type - Specifies the required device type for the test. For example:
```
"environments": [
    {
        "dimensions": {
            "device_type": "QEMU"
        }
    }
]
```
This denotes a test that can run on the QEMU device configuration. Note: This does not mean that the test runs on any QEMU device, rather it means that the test runs on a QEMU instance based on the configuration used by botanist.

cpu and os - Specifies the required CPU architecture and operating system for a host test. For example:

"environments": [
    {
        "dimensions": {
            "cpu": "x64",
            "os": "Linux"
        }
    }
]

This specifies a test that can run on an x64 Linux system.

tags - Specifies other properties for sharding. For example:
```
"environments": [
  {
    "tags": [ "e2e-isolated" ]
  }
]
```
This specifies a test with tag "e2e-isolated".

Tests with an environment tag will not be run in continuous integration builds unless the build configuration specifically opts in to running tests with that tag, in which case only tests with that tag will run.

test field

The test field provides information about how to categorize, run, and configure the test. Test executors like fx test and botanist use this data to determine the correct binary and command line to execute the test. It also contains information that is useful for other tooling integrations, for instance, to rebuild test targets between test invocations.

There are generally two types of test: Host and Device. Host tests refer to a binary path on the host system that is run to execute the test. Device tests refer to a component URL that is run on the target system using Test Manager.

Some host tests require a Fuchsia device and some do not. An end-to-end test (E2E) has an os that is not "fuchsia," but still has a device_type specified in its environments.

The test object contains the following fields, split by type of test:

All tests

name - Name for the test. This value is unique within tests.json. Typically contains the test component URL for device tests and the path to the test binary relative to the output directory for host tests.
cpu - The CPU architecture this test runs on. Redundant with the environments entry for host tests, but the only source of this information for device tests.
os - The OS this test runs on. Redundant with the environments entry for host tests, but the only source of this information for device tests. This field is used by botanist to determine if a test should be run as a subprocess on the host machine or run over SSH to a target device.
label - The GN label that produced this test entry. Used to know which targets to rebuild when executing tests from fx test, but only for host tests for which this label actually outputs the real test binary. Other types of tests use different label fields such as package_label.

Host tests only

path - The path to the test binary, relative to the output directory. Present only for host tests.
runtime_deps - The path (relative to the output directory) to a JSON file containing a list of file paths (relative to the output directory) to files that should be put in a place known to a host test for use at runtime. The format of the file is a JSON array of file path strings.

Device tests only

has_generated_manifest - True if this is a test component whose manifest was generated by a GN rule, omitted if the test is a host test, false otherwise. Used for test categorization and tracking of tests using custom manifests.
build_rule - Generated by the build for further test categorization. For example, it is meaningful to know if a test was was created by the fuchsia_unittest_package or the fuchsia_test_package rule.
log_settings - A dict that may contain these fields:
- max_severity - This field overrides the default maximum log severity for tests. By default tests that emit an ERROR log will fail, but this field instructs test executors to run in a mode where a different level of logs causes failure.
- min_severity - Used to instruct the test components to emit logs at this level rather than the default. By default tests choose their own minimum log severity.
package_label - The GN label that produced the package this test is contained in. Only present for device tests. Used by fx test to rebuild the packages containing tests before they are run.
component_label - The GN label that produces the test component for this test. Used for fuzzy matching test names when no matching tests are found in fx search-tests (in addition to all other label fields). Only present for device tests.
package_manifests - List of paths (relative to output directory) for each package manifest involved in constructing the test package. Used to determine which packages to republish after building tests in fx test. Only present for device tests.
package_url - The test component URL corresponding to this test. This is the URL that Test Manager is requested to execute for device tests.
parallel - Override for default test runner parallelism (as an integer). Typically used to instruct test executors to tell ffx test not to run test cases in parallel for tests that have crosstalk. Setting this value to 1 disables parallel test case execution, while a number > 1 forces parallel test case execution if supported by the individual test runner.

Examples

Host Test

[
  {
    "environments": [
      {
        // This test runs on the infra shard for Linux x64.
        "dimensions": {
          "cpu": "x64",
          "os": "Linux"
        }
      }
    ],
    "test": {
      "cpu": "x64",
      "label": "//src/performance/trace2json:trace2json_tests(//build/toolchain:host_x64)",
      "name": "host_x64/trace2json_tests",
      "os": "linux",
      // This is the path to execute the test both locally and in infra.
      "path": "host_x64/trace2json_tests",
      // The deps listed in this file must be placed in a known
      // location for the test.
      "runtime_deps": "host_x64/gen/src/performance/trace2json/trace2json_tests.deps.json"
    }
  }
]

Device Test

[
  {
    "environments": [
      // This test runs on the AEMU shard in infra.
      {
        "dimensions": {
          "device_type": "AEMU"
        }
      }
    ],
    "test": {
      // This test was created using fuchsia_unittest_package
      "build_rule": "fuchsia_unittest_package",
      "component_label": "//src/diagnostics/lib/sampler-config:sampler-config-tests_component(//build/toolchain/fuchsia:x64)",
      "cpu": "x64",
      // This test specified its own manifest.
      "has_generated_manifest": false,
      "label": "//src/diagnostics/lib/sampler-config:sampler-config-tests_test_sampler-config-tests_component(//build/toolchain/fuchsia:x64)",
      "log_settings": {
        // Any ERROR or FATAL logs will force this test to fail.
        "max_severity": "WARN"
      },
      "name": "fuchsia-pkg://fuchsia.com/sampler-config-tests#meta/sampler-config-tests.cm",
      "os": "fuchsia",
      "package_label": "//src/diagnostics/lib/sampler-config:sampler-config-tests(//build/toolchain/fuchsia:x64)",
      // These manifests can be used for deep inspection of the
      // contents of this test.
      "package_manifests": [
        "obj/src/diagnostics/lib/sampler-config/sampler-config-tests/package_manifest.json"
      ],
      "package_url": "fuchsia-pkg://fuchsia.com/sampler-config-tests#meta/sampler-config-tests.cm"
    }
  }
]