| RFC-0015: Compatibility Tests for Fuchsia (CTF) | |
|---|---|
| Status | Accepted |
| Areas |
|
| Description | Requirements, goals, and context for a Compatibility Tests for Fuchsia (CTF). |
| Issues | |
| Gerrit change | |
| Authors | |
| Reviewers | |
| Date submitted (year-month-day) | 2020-11-13 |
| Date reviewed (year-month-day) | 2020-12-09 |
Summary
This document presents requirements, design, and implementation strategy for a set of Compatibility Tests for Fuchsia (CTF). The CTF will offer a way of testing platform implementations to make sure that they behave according to Fuchsia's specifications. Fuchsia developers will write tests that guarantee compatibility across changes to both the source and behavior of the platform. When these tests pass, they will guarantee that a particular release, when run on a particular device, is compatible with a target API level and target ABI revision, as defined in RFC-0002.
For the purposes of this document, we refer to API and ABI jointly as the platform surface area. A platform surface area element is a named and versioned identifier associated with the platform surface area (e.g., a method name). Future RFCs may formalize these definitions.
Motivation
All of the open-source tests for Fuchsia platform behavior are currently (December 2020) built and run as part of the platform build. As the platform evolves, we keep the tests passing at head. As a result, we have no tests that guarantee backwards compatibility with older versions of the platform.
Currently, we use a number of product tests to ensure compatibility and stability. These product tests are difficult to triage: because they rely on the stability of the product, and target many different parts of the platform, it is difficult for platform engineers to determine where the bug may be.
At the same time, developers are writing code that targets older versions of the Fuchsia platform surface area. In this document, we refer to such developers as end developers.
As we roll out API breaking changes, we have no safeguards in place that raise a flag when we break compatibility with end developers' code. Over the course of the project, undocumented API changes have frequently been released that cause external code to stop building.
Furthermore, we are currently building out strong guarantees about backwards ABI compatibility. As of 9 November 2020, we require a six-week ABI compatibility window, but have no enforcement mechanism for it.
We need a set of tests that we can execute independently from the platform build that identify clearly when we break our contracts with end developers. This will help ensure that we maintain compatibility with externally developed code, and provide more easily triaged, targeted test coverage for the parts of the platform that are currently only exercised by product tests.
In the long term, we will also need a set of tests that system integrators can execute to know if they are producing a compliant Fuchsia implementation.
Fuchsia's CTF will offer a way of testing platform implementations to make sure that they are compatible with particular platform releases (as defined in RFC-0002). We aspire to have a test for each documented behavior in the platform surface area.
When we create a release, we can use the CTF to tell us about the compatibility of its surface area with that of other release versions.
When someone is developing a device running Fuchsia, and wants to see if it is compatible with a given SDK, they can take the CTF and the SDK with which it wants to demonstrate compatibility, pass the tests, and have confidence that their product correctly implements Fuchsia's semantics - it will be "Fuchsia compatible".
When a developer wants to understand how to write code against a particular API, or using a particular ABI, they will be able to use these tests as reference.
RFC-0002 allows a platform implementation to provide partial support for backwards compatibility. CTF will provide a way to test partial compatibility.
Note that the CTF is not intended as a complete solution for platform evolution and backwards compatibility. It's not likely that CTF tests will cover every last use case. API and ABI will still have to be designed with future use cases in mind. See the section on drawbacks and alternatives for additional discussion.
Design
The CTF design involves balancing ease of development with the need to build and run the CTF itself outside of the Fuchsia repository. The requirements are as follows:
There should be a CTF test for every documented behavior of every platform surface area element. Although we expect this to become a hard requirement eventually, this RFC does not specify such a requirement.
CTF tests may not rely on any internal details of a particular system image. To the extent that they rely on other platform code that is not itself subject to test, that code must be bundled as part of the CTF and also not rely on any internal details of a particular system image.
CTF tests must be updated by developers (that is, tests must be added or modified) when adding or changing elements of the platform surface area.
It must be possible to determine the API level and ABI revision of Fuchsia that a given CTF artifact targets.
CTF tests that are not included in the test suite as prebuilt artifacts must be written in languages supported by the SDK being used to test (see the supported languages document and the Language section below for more details).
Authoring the Tests
We develop CTF tests alongside their corresponding SDK elements. Today, that means we develop the tests in fuchsia.git. While it would be nice if CTF developers had the same experience as the out of tree developers who use the SDK, there are too many advantages to in-tree development to ignore:
Because feature development is done alongside test development, in-tree development of the tests will allow test authors to use a workflow with which they are familiar, as well as submit the test in the same CL as the feature.
Because the feature will be submitted at the same time as the test, there is no need for any machinery to align the CTF and the version that it qualifies.
CTF tests will use build-time enforcement to ensure that CTF tests can only depend on SDK elements or other pre-approved CTF code. One of the dangers of developing in-tree is that we may accidentally take on dependencies on platform implementation details that are not exposed via the SDK. CTF tests must only access publicly facing elements of the platform to prevent accidentally relying on implementation details. CTF tests may use platform code that is appropriate for writing tests (e.g., zxtest); such code will ship as part of the CTF artifact.
CTF tests must not take dependencies on third party libraries that rely on the SDK for their Fuchsia support. Third party libraries that require SDK elements to support Fuchsia are going to be built against a particular SDK. We must make sure that our tests are as decoupled as possible from anyone else's SDK dependencies, as third party code may rely on platform features that we need to exclude from the tests. For example, if we rely on a test suite that heavily uses locking, it may be inappropriate for testing features of Zircon used to implement locking. Because of this restriction, we will use zxtest rather than gtest.
An artifact containing the CTF tests relevant to a given SDK will be published alongside that SDK. This artifact will also contain build system support sufficient to build and run the CTF tests outside of the tree. It will not contain a toolchain.
The tests must exercise language support thoroughly. See the section on Language Support for more details.
Implementation
Coverage Requirements
All updates to Fuchsia platform surface area elements should include tests that exercise the documented surface. This includes, but is not limited to, C/C++ headers, FIDL API, the FIDL wire format, and any surface described by the Fuchsia System Interface document. If the surface area element can be accessed by developers via an SDK, it must be tested.
We recognize that it may not be practical to require tests at this point. As the CTF and platform grows, we expect this requirement will become more strict.
Almost all changes that require API review should have CTF tests, and API reviewers should review with that in mind. The final review will be made by testability reviewers, who should only approve platform surface area changes if they are appropriately covered by CTF tests.
All tests are subject to the same review requirements as any other code submitted to the tree. Note that this does not mean that tests must be run as part of the commit queue, although we expect most will be. Examples of tests that might not be run as part of the commit queue include manual tests and tests that take longer than the commit queue allows.
Directory structure
The structure of the //sdk/ctf/tests directory mirrors the structure of
released SDKs. Tests go in the directory that mirrors the one where the
interface under test is found in an SDK. For example:
- Tests for host tools should go in
//sdk/ctf/tests/tools - Tests for FIDL interfaces should go in the appropriate
subdirectory of
//sdk/ctf/tests/fidl. For example, tests forfuchsia.sysmemshould go in//sdk/ctf/tests/fidl/fuchsia.sysmem. - Tests for libraries should go in the appropriate subdirectory of
//sdk/ctf/tests/pkg. For example, tests forasync-loopshould go in//sdk/ctf/tests/pkg/async-loop.
If Fuchsia developers are not clear on where to put their tests, they should consult the OWNERS of the relevant directory.
Build support
CTF tests target API and ABI that are available through externally-available
SDKs. Build support ensures that tests only depend on API elements that are
either available via an SDK, or allowlisted for use within the CTF. All build
targets that are not allowlisted must use the cts_ rule variants found in
//sdk/ctf/build instead of the standard fuchsia.git rules (i.e., use
ctf_fuchsia_component, ctf_executable, and so on).
The allowlist for non-SDK code can be found in
//sdk/ctf/build/allowed_ctf_deps.gni. Test authors who believe they need an
additional inclusion should reach out to the OWNERS of this directory.
Language
Target-side tests
All API tests must be written in languages supported by the SDK they test. In most cases, this implies C++. ABI tests may be written in any language; in order to avoid having to build external support for languages we do not support via the SDK, if an ABI test needs to be in another language, we will include it as a prebuilt binary or package (whichever is more appropriate).
Tests for particular headers must be written in a language that supports that header. As of this writing, C headers target C11 and C++11 and above, and C++ headers target C++14 and above.
CTF tests may restrict themselves to a particular language version. For example, we may decide that particular tests are restricted to C++14 in order to guarantee that headers maintain C++14 compatibility.
Host-side tests
Language restrictions for target-side tests are not applicable to host-side
tests. The language for host-side tests is test-specific. If it will require
the CTF to depend on a new toolchain, the decision should be made in
consultation with the CTF team. For end-to-end tests and scripts that run on
the host, as of this writing, we support the use of Dart (and, specifically
sl4f). As supported languages change, documentation will be made available
about which languages are supported for host-side testing.
Test Requirements
Tests should contain a check for every documented assertion about a particular
API or ABI. For example, if we have a class fit::basic_string_view, and it
has a method size that is documented to return the size of the string_view, we
would have a test that creates a string_view, calls the size method, and
asserts that the return value is correct.
We recognize that it may be difficult to do this in some cases, and that some tests may require specific device setup that may be hard to replicate. We recommend that developers start working on testing early in the development cycle. The long-term goal is to make CTF testing a requirement for all changes to the platform surface area.
Tests should reflect best practices about the usage of a given API. Informally, if an end developer copies a test's usage of the API, the test author would believe that developer is using the API correctly. Tests should, to the extent possible, not depend on undocumented, application-specific invariants. In the future, in the case of widespread use of undocumented behaviors outside of the Fuchsia tree, we may need to add tests for use cases that do not follow recommended usages.
Wherever possible, tests should avoid creating test doubles (e.g., mocks and fakes) for the internal state of the target device. The intent of the CTF is to make sure the entire device behaves correctly, not to make sure that a particular component behaves correctly in isolation.
However, this does not mean that CTF tests cannot benefit from fakes in some environments. For example, for the purposes of using CTF tests to ensure platform stability, we may find it useful to exercise tests that require real hardware or manual input, such audio or connectivity tests, in an automated environment that does not have those features available. While a CTF test itself should avoid the use of test doubles, the device under test can use fake drivers that feed the test fake data. CTF tests can rely on such drivers in cases where using real hardware is not practical.
In addition, CTF tests should maintain isolation from each other. If one test fails, or if one aspect of the system under test misbehaves, then ideally this failure should be localized rather than affect other tests. Lack of isolation between tests is sometimes referred to as "test cross-talk". For instance consider a test that changes global state in a device settings component. If the test fails to restore the original state before it terminates, or if another component on the system changes the global state during test execution, then cross-talk may occur. For such a test to be isolated, the test author might consider creating an affordance for locally-scoped settings rather than mutating global state.
If necessary, tests may require manual intervention to pass. We recommend that developers thoroughly investigate the possibility of automation.
Deployment
CTF artifacts will be generated alongside the SDK artifacts that contain the relevant platform surface elements. Because of the soft transition requirements of RFC-0002, we expect that every SDK build will successfully execute the CTF associated with the previous build of the same SDK. As a proof of concept, we will implement infrastructure to guarantee this.
CTF artifacts will contain a test harness and build rules for gn. They will not contain a build system or toolchain; this must be supplied in the test execution environment. We will document which toolchains are known to be compatible with a given CTF.
Examples
Test examples can be found in fuchsia.git at //sdk/ctf/.
Performance
This change will have the following performance impact:
- An increase in time to run all platform tests, stemming from an increased number of tests.
- No impact on production performance, because the changes are test-only.
Security considerations
Because changes associated with this RFC are test-only, they are a low security risk. Tests are not expected to interact with untrusted data from external sources.
Privacy considerations
Because changes associated with this RFC are test-only, they are a low privacy risk. Tests are not expected to interact with user data.
Testing
This proposal will increase the testing matrix for the platform. For example, given the six-week ABI stability guarantee, all ABI tests from the CTF generated six weeks earlier than a given build should be run and complete successfully against that build.
The new requirements in this proposal will also increase the overall number of platform tests.
As many required properties of the test framework as is practical will be enforced automatically; for example, the framework will automatically check that only allowed dependencies are included.
Documentation
Documentation on how to write CTF tests will be included in //docs. There
will be updates to testability and API process documents to reflect new CTF test
authorship requirements. The steps needed to run CTF out of tree will be
documented, so that end developers and system integrators can do them
independently.
Drawbacks, alternatives, and unknowns
The chief drawback of this proposal is that it creates a significant new testing requirement for all changes to the platform surface area.
It is not a goal of the CTF effort to provide a complete solution to evolution and backwards compatibility issues. APIs and ABIs will have to be designed carefully to ensure that developers can migrate their code at a reasonable cost. For example, the FIDL team evolves language bindings with extreme care: they have a clear specification for how bindings ought to work, and actively tracks how conformant the various bindings are.
The CTF approach is a standard industry approach to maintaining backwards compatibility. Other approaches include:
- Simply being careful. We know empirically that this does not work by itself.
- Not evolving the platform. Obviously, simply never making changes is not practical. Most scaled down versions of this (for example, shipping most of an application's dependencies with it, or providing a virtual environment for every application) are at odds with Fuchsia's design principles and product goals.
- Formal verification. We do not consider formal verification to be a scalable alternative to testing.
Prior art and references
Android solves this problem by releasing a CTF with their product. Developers of new Android devices must ensure that their devices pass the CTF.
As part of its Windows Hardware Compatibility Program, Microsoft produces a Windows Hardware Lab Kit that they distribute to developers of new Windows hardware.