Google is committed to advancing racial equity for Black communities. See how.

The Fuchsia Test Runner Framework

Integrating testing frameworks with the Component Framework

The Fuchsia Component Framework allows developers to create components in a variety of languages and runtimes. Fuchsia's own code uses a diverse mix of programming languages for components, including C/C++, Rust, Dart, and Go.

The Test Runner Framework uses Component Framework runners as an integration layer between various testing runtimes and a common Fuchsia protocol for launching tests and receiving their results. This makes for an inclusive design that on one hand allows developers to bring their language and testing framework of choice, and on the other hand allows building and testing Fuchsia on a variety of systems and targeting different hardware.

The Test Manager

The test_manager component is responsible for running tests on a Fuchsia device. Test manager exposes the fuchsia.test.manager.RunBuilder protocol, which allows launching test suites.

Each test suite is launched as a child of test manager. Test suites are offered capabilities by test manager that enable them to do their work while maintaining isolation between the test and the rest of the system. For instance hermetic tests are given the capability to log messages, but are not given the capability to interact with real system resources outside of their sandbox. Test manager uses only one capability from the test realm, a controller protocol that test suites expose. This is done to ensure hermeticity (test results aren't affected by anything outside of their intended sandbox) and isolation (tests don't affect each other or the rest of the system).

The test manager controller itself is offered to other components in the system in order to integrate test execution with various developer tools. Tests can then be launched with such tools as fx test and ffx.

The test suite protocol

The test suite protocol, fuchsia.test.Suite, is used by the test manager to control tests, such as to invoke test cases and to collect their results.

Test authors typically don't need to implement this protocol. Instead, they rely on a test runner to do this for them. For instance, you might write a test in C++ using the GoogleTest framework, and then use gtest_runner in your component manifest to integrate with the Test Runner Framework.

Test runners

A language and runtime-inclusive framework

Test runners are reusable adapters between the Test Runner Framework and common languages & frameworks used by developers to write tests. They implement the fuchsia.test.Suite protocol on behalf of the test author, allowing developers to write idiomatic tests for their language and framework of choice.

Component manifests for simple unit tests can be generated by the build rules. Generated component manifests for v2 tests will include the appropriate test runner based on their build definition. For instance a test executable that depends on the GoogleTest library will include the GoogleTest runner in its generated manifest.

Inventory of test runners

The following test runners are currently available for general use:

GoogleTest runner

A runner for tests written in C/C++ using the GoogleTest framework. Use this for all tests written using GoogleTest.

Common GoogleTest features are supported, such as disabling tests, running only specified tests, running the same test multiple times, etc'. Standard output, standard error, and logs are captured from the test.

In order to use this runner, add the following to your component manifest:

{
    include: [ "//src/sys/test_runners/gtest/default.shard.cml" ]
}

By default GoogleTest test cases run serially (one test case at a time).

Rust runner

A runner for tests written in the Rust programming language and following Rust testing idioms. Use this for all idiomatic Rust tests (i.e. tests with modules that set the attribute [cfg(test)]).

Common Rust testing features are supported, such as disabling tests, running only specified tests, running the same test multiple times, etc'. Standard output, standard error, and logs are captured from the test.

In order to use this runner, add the following to your component manifest:

{
    include: [ "//src/sys/test_runners/rust/default.shard.cml" ]
}

By default Rust test cases run in parallel, at most 10 cases at a time.

Go test runner

A runner for tests written in the Go programming language and following Go testing idioms. Use this for all tests written in Go using import "testing".

Common Go testing features are supported, such as disabling tests, running only specified tests, running the same test multiple times, etc'. Standard output, standard error, and logs are captured from the test.

In order to use this runner, add the following to your component manifest:

{
    include: [ "//src/sys/test_runners/gotests/default.shard.cml" ]
}

By default Go test cases run in parallel, at most 10 cases at a time.

ELF test runner

The simplest test runner - it waits for your program to terminate, then reports that the test passed if the program returned zero or that it failed for any non-zero return value.

Use this test runner if your test is implemented as an ELF program (for instance an executable written in C/C++) but it does not use a common testing framework that's supported by existing runners and you'd rather not implement a bespoke test runner.

In order to use this runner, add the following to your component manifest:

{
    include: [ "//src/sys/test_runners/elf/default.shard.cml" ]
}

If you are using in-tree unit test GN templates, and you are not already using a test framework with a dedicated test runner, add the following to your build deps:

fuchsia_unittest_package("my-test-packkage") {
    // ...
    deps = [
        // ...
        "//src/sys/testing/elftest",
    ]
}

Legacy test runner

Legacy tests are tests that were written before the Test Runner Framework was introduced. The legacy test runner offers a simple adapter between the modern test framework and legacy tests that were not converted to modern ones. For help with migrations see this guide. It is not recommended to use the legacy test runner in new tests.

The legacy test runner detects if a test passed or failed by observing its return code, with zero indicating success and non-zero indicating failure.

All legacy tests are automatically wrapped in a modern test and executed using the legacy test runner. The launch URL of the wrapper will be derived from the wrapped test's launch URL. For instance:

    fuchsia-pkg://fuchsia.com/package#meta/test_component.cmx

will become:

    fuchsia-pkg://fuchsia.com/package#meta/test_component.cm

The legacy test runner does not understand concepts such as test cases (or filtering on them), running multiple test cases in parallel, etc. It does however forward arguments to the test, so you can pass arguments that are specific to the underlying test framework. For instance, to run just a specific test case from a gtest:

fx test <test> -- --gtest_filter=MyTestCase

To run Rust tests, at most 5 test cases at a time:

fx test <test> -- --test-threads=5

To suppress this behavior set wrap_cmx_test_with_cml_test to false on fuchsia_test_package or fuchsia_unittest_package. Don't forget to file a bug and track the reason for the exclusion.

Change your BUILD.gn to exclude your legacy test:

import("//build/components.gni")

# This is your legacy test
fuchsia_test_component("simple_test_legacy") {
  component_name = "simple_test"
  manifest = "meta/simple_test.cmx"
  deps = [ ":simple_test_bin" ]
}

# Exclude your test from auto-wrapping.
fuchsia_test_package("simple_test") {
  test_components = [ ":simple_test_legacy" ]

  # TODO(fxbug.dev/XXXXX) : Excluding the test due to ...
  # Remove below line once the issue is fixed.
  wrap_cmx_test_with_cml_test = false
}

Controlling parallel execution of test cases

When using fx test to launch tests, they may run each test case in sequence or run multiple test cases in parallel up to a given limit. The default parallelism behavior is determined by the test runner. To manually control the number of test cases to run in parallel use test spec:

fuchsia_test_package("my-test-pkg") {
  test_components = [ ":my_test_component" ]
  test_specs = {
    # control the parallelism
    parallel = 10
  }
}

To override the value specified in the test spec, pass the parallel option when invoking fx test:

fx test --parallel=5 <test_url>

Running test multiple times

To run a test multiple times use:

 fx test --count=<n> <test_url>

If an iteration times out, no further iteration will be executed.

Passing arguments

Custom arguments to the tests can be passed using fx test:

fx test <test_url> -- <custom_args>

Individual test runners have restrictions on these custom flags:

GoogleTest runner

Note the following known behavior change:

--gtest_break_on_failure: As each test case is executed in a different process, this flag will not work.

The following flags are restricted and the test fails if any are passed as fuchsia.test.Suite provides equivalent functionality that replaces them.

  • --gtest_filter - Instead use:
 fx test --test-filter=<glob_pattern> <test_url>

--test-filter may be specified multiple times. Tests that match any of the given glob patterns will be executed.

  • --gtest_also_run_disabled_tests - Instead use:
 fx test --also-run-disabled-tests <test_url>
  • --gtest_repeat - See Running test multiple times.
  • --gtest_output - Emitting gtest json output is not supported.
  • --gtest_list_tests - Listing test cases is not supported.

Rust runner

The following flags are restricted and the test fails if any are passed as fuchsia.test.Suite provides equivalent functionality that replaces them.

  • - Instead use:
 fx test --test-filter=<glob_pattern> <test_url>

--test-filter may be specified multiple times. Tests that match any of the given glob patterns will be executed.

  • --nocapture - Output is printed by default.
  • --list - Listing test cases is not supported.

Go test runner

Note the following known behavior change:

-test.failfast: As each test case is executed in a different process, this flag will only influence sub-tests.

The following flags are restricted and the test fails if any are passed as fuchsia.test.Suite provides equivalent functionality that replaces them

  • -test.run - Instead use:
 fx test --test-filter=<glob_pattern> <test_url>

--test-filter may be specified multiple times. Tests that match any of the given glob patterns will be executed.

A runtime-agnostic, runtime-inclusive testing framework

Fuchsia aims to be inclusive, for instance in the sense that developers can create components (and their tests) in their language and runtime of choice. The Test Runner Framework itself is language-agnostic by design, with individual test runners specializing in particular programming languages or test runtimes and therefore being language-inclusive. Anyone can create and use new test runners.

Creating new test runners is relatively easy, with the possibility of sharing code between different runners. For instance, the GoogleTest runner and the Rust runner share code related to launching an ELF binary, but differ in code for passing command line arguments to the test and parsing the test's results.

Temporary storage

To use temporary storage in your test, add the following to your component manifest:

{
    include: [ "//src/sys/test_runners/tmp_storage.shard.cml" ]
}

At runtime, your test will have read/write access to /tmp. The contents of this directory will be empty when the test starts, and will be deleted after the test finishes.

Tests that don't specify a custom manifest and instead rely on the build system to generate their component manifest can add the following dependency:

fuchsia_unittest_package("foo-tests") {
  deps = [
    ":foo_test",
    "//src/sys/test_runners:tmp_storage",
  ]
}

Hermeticity

A test is hermetic if it uses or offers no capabilities from the test root's parent. The tests are by default hermetic unless explicitly stated otherwise.

Hermetic capabilities for tests

There are some capabilities which all tests can use which do not violate test hermeticity:

Protocol Description
fuchsia.boot.WriteOnlyLog Write to kernel log
fuchsia.logger.LogSink Write to syslog
fuchsia.process.Launcher Launch a child process from the test package
fuchsia.sys2.EventSource Access to event protocol

The hermeticity is retained because these capabilities are carefully curated to not allow tests to affect the behavior of system components outside the test realm or of other tests.

To use these capabilities, there should be a use declaration added to test's manifest file:

// my_test.cml
{
    use: [
        ...
        {
            protocol: [
              "fuchsia.logger.LogSink"
            ],
        },
    ],
}

Tests are also provided with some default storage capabilities which are destroyed after the test finishes execution.

Storage Capability Description Path
data Isolated data storage directory /data
cache Isolated cache storage directory /cache
temp Isolated in-memory temporary storage directory /tmp

Add a use declaration in test's manifest file to use these capabilities.

// my_test.cml
{
    use: [
        ...
        {
            storage: "data",
            path: "/data",
        },
    ],
}

The framework also provides some capabilities to all the components and can be used by test components if required.

Legacy non-hermetic tests

These tests that were introduced before hermetic testing was enforced. They could access some pre-defined capabilities outside of the test realm. A capability accessed by non-hermetic test from outside its test realm is called a system capability.

To use a system capability, a test must explicitly mark itself to run in non-hermetic "system" realm as shown below.

// my_component_test.cml

{
    include: [
        // Select the appropriate test runner shard here:
        // rust, gtest, go, etc.
        "//src/sys/test_runners/rust/default.shard.cml",
    ],
    program: {
        binary: "bin/my_component_test",
    },
    facets: {
        "fuchsia.test": {
            type: "system"
        },
    },
    use: [
        {
            protocol: [ "fuchsia.sysmem.Allocator" ],
        },
    ],
}

Possible values of fuchsia.test.type:

Value Description
hermetic Hermetic realm
system Legacy non hermetic realm

Below is the list of system capabilities provided to legacy non-hermetic tests:

Protocols:

fuchsia.boot.ReadOnlyLog
fuchsia.hwinfo.Board
fuchsia.hwinfo.Device
fuchsia.hwinfo.Product
fuchsia.kernel.RootJobForInspect
fuchsia.kernel.Stats
fuchsia.kernel.VmexResource
fuchsia.net.http.Loader
fuchsia.scheduler.ProfileProvider
fuchsia.sys2.ComponentResolver
fuchsia.sysmem.Allocator
fuchsia.tracing.provider.Registry
fuchsia.vulkan.loader.Loader
fuchsia.sysinfo.SysInfo

Directories:

root-ssl-certificates
config-data
dev-input-report
dev-display-controller
dev-goldfish-address-space
dev-goldfish-control
dev-goldfish-pipe
dev-goldfish-sync
dev-gpu
dev-mediacodec

Performance

When writing a test runner that launches processes, the runner needs to provide a library loader implementation.

Test runners typically launch individual test cases in separate processes to achieve a greater degree of isolation between test cases. However this can come at a significant performance cost. To mitigate this, the test runners listed above use a caching loader service which reduces the extra overhead per process launched.

Test roles

Components in the test realm may play various roles in the test, as follows:

  • Test driver: The component that actually runs the test, and implements (either directly or through a test runner) the fuchsia.test.Suite protocol. This role may be, but is not necessarily, owned by the test root.
  • Capability provider: A component that provides a capability that the test will exercise somehow. The component may either provide a "fake" implementation of the capability for test, or a "real" implementation that is equivalent to what production uses.
  • Component under test: A component that exercises some behavior to be tested. This may be identical to a component from production, or a component written specifically for the test intended to model production behavior.

Further reading