- Project leads: shayba@google.com, crjohns@google.com
- Area: Testing

## Problem statement

### Missing platform surface for testing

Software developers targeting Fuchsia can write [components](https://fuchsia.dev/docs/concepts/components/v2/introduction),
[build them](https://fuchsia.dev/docs/development/components/build), and [test them](https://fuchsia.dev/docs/development/testing/components/README). However these
critical developer journeys are fully and continuously tested only for


[](https://fuchsia.dev/docs/glossary#in-tree)


developers, and not at all for


[](https://fuchsia.dev/docs/glossary#out-of-tree)


developers.

Today there are several teams that develop and test components out-of-tree. We
sometimes refer to these teams as "partners", because the Fuchsia team maintains
a close engagement with them. This arrangement is costly to maintain and
impossible to scale for the following reasons:

- Out-of-tree testing relies on deprecated platform features, protocols, and
  tools. Most notably: using the SSH protocol to issue commands on the target
  (such as with [`fx shell`](https://fuchsia.dev/reference/tools/fx/cmd/shell) or [`fssh`](https://fuchsia.dev/reference/tools/sdk/fssh.md)), `dash` as a
  system interface, [`fx log`](https://fuchsia.dev/reference/tools/fx/cmd/log) to collect system-wide logs during the
  test, and using the SCP protocol (such as with [`fx scp`](https://fuchsia.dev/reference/tools/fx/cmd/scp)) to collect
  other test artifacts as side effects on a global mutable filesystem. These
  produce unreliable behavior that leads to flakes and is difficult to
  troubleshoot, which can at least be partially attributed to the brittle nature
  of text-based protocols. Furthermore the transport affords for a single
  character stream, a great fit for a system-wide log (e.g. serial log or
  syslog) but not for multiple log streams or large binary artifacts such as
  state dumps and screenshots from tests.

- Various text-based protocols that lack methods and practices for ensuring ABI
  stability or orchestrating ABI evolution.

- Tests written as legacy components aka CFv1 which enjoy a
  lesser degree of isolation, suffer from a greater degree of flakiness, and
  don't benefit from [new testing tools](https://fuchsia.dev/docs/development/testing/components/realm_builder).

- Bespoke, inconsistent rules and scripts for defining tests and capturing their
  results and associated diagnostics.

- Inconsistent support for various test frameworks. For instance, while all
  out-of-tree partners support C++ and GoogleTest, only some partners support
  Dart, and none support Rust despite its popularity for in-tree component
  development.

- Inconsistent support for additional instrumentation that adds more value to
  tests, such as [sanitizers](https://fuchsia.dev/docs/contribute/testing/sanitizers) and [coverage](https://fuchsia.dev/docs/contribute/testing/coverage).

- Some tools and source code is distributed to partners not via the
  [Fuchsia IDK](https://fuchsia.dev/docs/development/idk/README), the set of [SDK tools](https://fuchsia.dev/reference/tools/sdk/ffx). For instance the
  `TestWithEnvironment` helper class is
  [manually copied to the Flutter repository on GitHub](https://bugs.fuchsia.dev/p/fuchsia/issues/detail?id=73984) to unblock
  integration testing needs.

To overcome these issues, the Fuchsia team has offered dedicated support to key
partners. This arrangement often produces tailored solutions that aren't
portable between different customers. Furthermore, support for customer issues
often happens inside the customer's source repository, which may be convenient
for the customer but doesn't scale to supporting a general public audience of
developers.

We now have a breadth of experiences and observations from the wild to inform us
on how to create more generalized testing solutions. The time is right to take
these insights and build out platform support for these use cases, thus creating
a more capable SDK as well as reducing and removing bespoke solutions that carry
a fixed maintenance cost over a diminishing value proposition.

### Platform/Infra surface for testing

Fuchsia's in-house testing infrastructure (aka "infra") exhibits most of the
same problems listed above and is affected in similar ways to Fuchsia partners.
Since Fuchsia's infra doesn't continuously exercise platform solutions and SDK
tools for testing, there is a missed opportunity for continuous quality
assurance for said solutions and tools.

### A growing need for out-of-tree testing

There are several current and upcoming projects that are expected to increase
the scope of out-of-tree development \& testing targeting Fuchsia. These include:

- *[Compatibility Test Suite](https://fuchsia.dev/docs/contribute/governance/rfcs/0015_cts)* (CTS) tests will be able to run outside
  of Fuchsia's in-tree build \& test system, though their source code will be
  hosted on [fuchsia.git](https://fuchsia.googlesource.com/fuchsia).

- *[Flutter-on-Fuchsia Velocity](https://fuchsia.dev/docs/contribute/roadmap/2021/flutter_on_fuchsia_velocity)* expects to build and test a
  Flutter embedder on Fuchsia and a Flutter runner for components and their
  tests out-of-tree, with at least some integration tests being upstreamed to
  the Flutter project.

- *[Drivers as Components](https://fuchsia.dev/docs/contribute/roadmap/2020/overview#implementing_drivers_as_components)* will include a demonstration of a driver built
  and tested out-of-tree, to realize the promise of
  robust hardware support on Fuchsia\] via
  [driver ABI stability](https://fuchsia.dev/docs/contribute/roadmap/2021/stable_driver_runtime).

- *Support for running existing tests on Fuchsia in [LLVM](https://llvm.org/) and
  [Rust](https://www.rust-lang.org/) projects* will require out-of-tree C++ and Rust testing
  support.

Currently these projects cannot be successfully completed as they depend on
missing support for out-of-tree testing.

## Solution statement

We will create a platform solution for testing that works exclusively based on
tools and protocols that are publicly available in the Fuchsia SDK.

### Host-side

We will use [FFX](https://fuchsia.dev/docs/development/tools/ffx/getting-started) as the entry point for out-of-tree testing. We will finish
developing [`ffx test`](https://fuchsia.dev/reference/tools/sdk/ffx.md#test) to handle all host-side aspects of testing. We
will rely on the established FFX technologies and practices, such as
configuration management, target device discovery, and the [Overnet](https://cs.opensource.google/fuchsia/fuchsia/+/main:/src/connectivity/overnet/README.md)
communication suite.

We will replace existing host tools with `ffx` tools. Tools such as
[`testrunner`](https://cs.opensource.google/fuchsia/fuchsia/+/main:/tools/testing/testrunner/README.md) and [Botanist](https://cs.opensource.google/fuchsia/fuchsia/+/main:/tools/botanist/) currently perform tasks in
Fuchsia CI/CQ - such as device discovery, device setup, test orchestration, test
artifact collection - that can be incrementally handed off to `ffx`. Some of
these handoffs will require building equivalent `ffx` plugins up to parity, for
instance bringing up `ffx test` support for running Bringup tests over a serial
connection. The payoff is that we'd get to continuously verify our work on modern
tools that are portable between in-tree and out-of-tree use cases using our
existing rich and robust corpus of in-tree tests and in-tree automation.

We will port aspects of working with [sanitizers](https://fuchsia.dev/docs/contribute/testing/sanitizers) and [test coverage](https://fuchsia.dev/docs/contribute/testing/coverage)
from tools that are only available in-tree such as [tefmocheck](https://cs.opensource.google/fuchsia/fuchsia/+/main:/tools/testing/tefmocheck/README.md) and [covargs](https://cs.opensource.google/fuchsia/fuchsia/+/main:/tools/debug/covargs/) to
[ffx plugins](https://fuchsia.dev/docs/development/tools/ffx/development/README).

### Target-side

We will grow the [Test Runner Framework](https://fuchsia.dev/docs/development/testing/components/test_runner_framework) (TRF) to accommodate for the needs
of out-of-tree testing.

TRF includes an on-device Overnet daemon, a component to manage/schedule tests,
an isolated realm for hermetic testing, a selection of test runners that support
a variety of languages and frameworks for writing tests, and [FIDL](https://fuchsia.dev/docs//concepts/fidl/overview) protocols to
connect all of the above. TRF supports both in-tree and out-of-tree testing
workflows. It replaces a test runtime that only worked in-tree and only
supported CFv1 components.

The priority customer for TRF so far has been in-tree testing, with success
measured in terms of the portion of tests that run on TRF. At the time of
writing more than 70% of Fuchsia in-tree tests have been migrated to TRF, with
modern (CFv2) tests running exclusively on TRF. By the end of 2021 we expect all
remaining tests except [ZBI tests](https://fuchsia.dev/reference/tools/fx/cmd/run-boot-test) to run under TRF, thanks to
an upcoming compatibility layer.

Once all component tests are migrated to TRF, we will turn down the legacy
target-side v1-only in-tree test runtime. This will allow us to focus on
improving the new testing runtime, improvements that will benefit both in-tree
and out-of-tree developers.

To improve the developer experience, out-of-tree developers will be able to
[include test runners from the SDK](https://fuchsia.dev/docs/contribute/governance/rfcs/0106_manifest_includes_in_sdk). Through this mechanism,
out-of-tree developers will have access to the existing inventory of test
runners - [gtest](https://fuchsia.dev/docs/development/testing/components/test_runner_framework#gtest-runner), [rust](https://fuchsia.dev/docs/development/testing/components/test_runner_framework#rust-runner), [Go](https://fuchsia.dev/docs/development/testing/components/test_runner_framework#gotest-runner), and
arbitrary [ELF](https://fuchsia.dev/docs/development/testing/components/test_runner_framework#elf-test-runner) binaries - as well as the upcoming Dart and Flutter
test runners. TRF will also provide the foundation for developing more advanced
testing strategies, such as [stress tests](https://fuchsia.dev/docs/development/testing/stress_tests) and
[CTS tests](https://fuchsia.dev/docs/contribute/governance/rfcs/0015_cts). These testing strategies will be expressed as runners
that may also be provided in the SDK.

In addition, out-of-tree developers will be able to create and use test runners
of their own. This is thought to be feasible today, but it has not been
demonstrated yet. We should create a first out-of-tree test runner so we can
speak about this workflow with more confidence.

### Test execution control

We will finish the development and rollout of new protocols to control test
execution.

The new protocols are defined in [FIDL](https://fuchsia.dev/docs//concepts/fidl/overview) form (allowing for ABI stability \&
evolution, crucial for out-of-tree) and are natively carried by [Overnet](https://cs.opensource.google/fuchsia/fuchsia/+/main:/src/connectivity/overnet/README.md). The
new protocols don't have specific knowledge of Fuchsia infra, therefore
sharpening the so-called platform/infra contract.

The new protocols allow for better layering and separation of concerns. For
instance the host side is responsible for test selection and requesting
execution on the target. The target-side test manager takes responsibility for
actually running the tests, as opposed to the legacy system in which a host tool
manually executes each test individually on a target device. The responsibility
of parallelizing test execution to maximize resource utilization is transferred
to the target, which is better equipped to handle this responsibility.

Lastly, the new protocols are not based on a character stream (SSH). This allows
for information to flow both ways with multiple streams of instructions,
results, and diagnostics, as well as large test inputs and outputs that may be
in binary format.

### Test results

Test results will no longer be constrained to a suite-level pass/fail outcome,
but will be enumerated in fine detail and with structured format. Diagnostics
collected during the test, such as logs captured from the test realm throughout
the duration of the test, will be organized in a manner associated with the
tests that produced them. There will be standard support for other artifacts
from tests, such as profiles collected during test runtime or large test outputs
such as screenshots taken during the test. The schema for the result format will
be published in the SDK to support processing by out-of-tree tools.

### Documentation

Developer guides such as [this](https://fuchsia.dev/docs/development/testing/components/test_runner_framework#troubleshooting) will be
reviewed, edited, and simplified to be useful to out-of-tree developers. In-tree
and out-of-tree testing workflows will be unified such that these guides won't
have information specific to Fuchsia in-tree/out-of-tree developers, or separate
sections for such different audiences.

A new onboarding guide detailing the "Journey of Testing" will be developed.
This guide will provide an entrypoint to developers who need to ensure their
code is properly tested with both unit and integration tests. The goals of this
guide are to 1) aid developers in making the correct choice about which type of
test to write, and 2) quickly bring developers to a point where their tests are
prepared to take advantages of more advanced testing strategies (such as
[CTS](https://fuchsia.dev/docs/contribute/governance/rfcs/0015_cts) and [Stress Testing](https://fuchsia.dev/docs/development/testing/stress_tests)).

### Virtualization support

Virtual targets such as emulators are very useful and popular for testing.
Fuchsia currently offers to [download a distribution of qemu](https://fuchsia.dev/docs/development/idk/download#qemu)
that has been tested to work with Fuchsia. However there are additional tools
for working with virtualized targets, such as [`fx qemu`](https://fuchsia.dev/reference/tools/fx/cmd/qemu) and
[`fx gce`](https://fuchsia.dev/reference/tools/fx/cmd/gce), that are only available in-tree.

We will map the gap between in-tree and out-of-tree support for running Fuchsia
on virtualized targets, and address it to close testing workflow gaps as needed.

## Dependencies

- The FFX tool and associated stack.
- The Fuchsia IDK, and any SDK frontends used by out-of-tree developers.
- Making [RealmBuilder](https://fuchsia.dev/docs/development/testing/components/realm_builder) available out-of-tree.
- Exposing [RealmBuilder](https://fuchsia.dev/docs/development/testing/components/realm_builder) via the SDK. This includes the underlying protocol, and at least one client library.
- Extending RealmBuilder to [support additional
  languages](https://fuchsia.dev/docs/development/testing/components/realm_builder#language-feature-matrix).

## Risks and mitigations

N/A

## Not in scope

### End-to-end testing (aka system testing)

This proposal focuses on component testing, which takes the form of unit testing
a single component or integration testing that spans multiple components. System
tests, also known as end-to-end (or e2e) tests, don't exercise specific
component instances but rather the entire system under test. As such they differ
in many ways from component tests such as in developer needs and use cases, and
in the platform's ability to offer isolation between e2e tests.

The current popular solution for e2e tests is
[SL4F (Scripting Layer for Fuchsia)](https://fuchsia.dev/docs/development/testing/sl4f).
The implementation includes a target-side daemon that is included in
some Fuchsia images and can perform a documented set of system automation tasks,
and a [client library written in Dart](https://cs.opensource.google/fuchsia/fuchsia/+/main:/sdk/testing/sl4f/client/) that is available in the
Fuchsia SDK.

Additionally, there exist CFv1 component tests that are arguably system tests.
This is possible because the legacy CFv1 test runtime allows
accessing real system services, as a legacy compromise that is intentionally not
supported in CFv2 testing. For the purpose of this discussion we consider such
strictly non-hermetic CFv1 component tests to be system tests as well.

Current challenges in scaling e2e test development include:

- Adding facades to SL4F requires changing platform code and redistributing
  Fuchsia system images. Out-of-tree developers cannot extend SL4F's
  capabilities to automate a system.

- SL4F does not rely on ffx, but rather uses its own transport layer, protocols,
  target discovery, and configuration. These differences add to the ongoing
  maintenance cost and introduce inconsistencies and friction in the developer
  experience.

- Only a Dart client library is provided.

Since system tests are so uniquely different than component tests, this topic is
covered in a [separate roadmap document](https://fuchsia.dev/docs/contribute/roadmap/2021/oot_system_testing).
[](https://issues.fuchsia.dev/issues/new?component=1439230&template=1881640&description=Issue+on:+https://fuchsia.dev/docs/contribute/roadmap/2021/oot_component_testing.md "Report a bug or suggest improvements on this page") [](https://cs.opensource.google/fuchsia/fuchsia/+/main:docs/contribute/roadmap/2021/oot_component_testing.md "View the source code of this page") [](https://ci.android.com/edit?repo=fuchsia/fuchsia/main&file=docs/contribute/roadmap/2021/oot_component_testing.md&cloneRepo=fuchsia/fuchsia:refs/heads/main&fullPath=fuchsia/docs/contribute/roadmap/2021/oot_component_testing.md "Edit the source code of this page")