This document standardizes the way we define and organize FIDL tests in the GN build system, respecting the following goals:
- Name consistently. If Rust uses
fx test fidl_rust_conformance_tests
, then Go should usefx test fidl_go_conformance_tests
. There should be no need to continually look these up because they're all different. - Run what you need. If there are multiple test executables in a given
category, it should be easy to only run the one you care about using
fx test
. - Run on host. Wherever possible, we should provide ways to run tests on host, and in a predictable way. The quick edit-build-run cycle on host can greatly improve productivity.
- Follow best practices. Our approach should be aligned with Fuchsia best
practices, e.g. following the fuchsia-pkg URL specification, working well
with
fx test
, etc.
Terminology
This document uses the following terminology:
- target: a GN target defined in a BUILD.gn file
- toolchain: see
fx gn help toolchain
- host: a developer's platform, specifically linux or mac
- device: a Fuchsia platform, either physical or emulated (i.e. qemu)
- package: a Fuchsia package; the unit of distribution in Fuchsia
- component: a Fuchsia component; the unit of executable software in Fuchsia
Naming
General guidelines:
- Use underscores, not hyphens.
- End names with the plural
_tests
rather than the singular_test
. - Use full, descriptive, unique names for packages, components, and binaries.
The last point means preferring full names like fidl_rust_conformance_tests
over contextual names like conformance_tests
. It might seem verbose and
redundant to repeat "fidl" and "rust" at the directory, package, component, and
binary level. But the fact is that these names must be unique, and it is better
to make them unique in a consistent way rather than remembering odd rules like
fidl-bindings-test
is for Dart and fidl-test
is for C.
Names should use the following scheme, joining parts with underscores:
tool [ bindings ] [ category [ subcategory ] ] tests
Where tool is one of:
- fidl: FIDL runtime support
- fidlc: FIDL compiler frontend
- fidlgen: FIDL compiler backend
- gidl, measure_tape, etc.: other tools
And the other parts are:
- bindings
- One of c, llcpp, hlcpp, rust, go, dart (note: hlcpp, not cpp)
- category, subcategory
- Example categories: conformance, types, parser, lib
- Do not use: frontend, backend, bindings (tool distinguishes these)
Hierarchy
Every BUILD.gn file that defines tests should include a "tests"
group:
group("tests") {
testonly = true
deps = [ ... ] # not public_deps
}
These are aggregated in "tests" groups of BUILD.gn files in parent directories.
The root "tests" group (for some portion of the codebase, e.g.
src/lib/fidl/BUILD.gn) should be included in bundles/fidl/BUILD.gn. This enables
fx set ... --with //bundles/fidl:tests
to include all FIDL tests in the build.
(The tests are also run in CQ because //bundles/buildbot:core
includes
//bundles/fidl:tests
.)
Binary names
Normally test binary names are based on the target name. For example, a
test("some_tests") { ... }
target would produce a some_tests
binary.
However, for a single test you often need multiple targets (source sets,
components, packages, etc.) with unique names. Therefore, the examples in this
document use target names like some_tests_bin
and override the binary name
using the output_name
parameter:
test("some_tests_bin") {
output_name = "some_tests"
...
}
This also works for rustc_test
, go_test
, etc.
Device tests
Assume we have a :fidl_foo_tests_bin
target that produces a fidl_foo_tests
binary. To wrap this in a package, start with a fuchsia_unittest_package
:
import("//src/sys/build/components.gni")
fuchsia_unittest_package("fidl_foo_tests") {
deps = [ ":fidl_foo_tests_bin" ]
}
We can now run the test by package name or component name (they are the same in
this case) with fx test fidl_foo_tests
.
For multiple device tests, collect them all in a single package with
fuchsia_test_package
. For example, suppose we split fidl_foo_tests
into
fidl_foo_unit_tests
and fidl_foo_integration_tests
:
import("//src/sys/build/components.gni")
fuchsia_unittest_component("fidl_foo_unit_tests") {
deps = [ "fidl_foo_unit_tests_bin" ]
}
fuchsia_unittest_component("fidl_foo_integration_tests") {
deps = [ "fidl_foo_integration_tests_bin" ]
}
fuchsia_test_package("fidl_foo_tests") {
test_components = [
":fidl_foo_unit_tests",
":fidl_foo_integration_tests",
]
}
If your test requires any component features, services, etc. beyond the
fuchsia_unittest_component
defaults, you must write a component manifest file:
# BUILD.gn
import("//src/sys/build/components.gni")
fuchsia_unittest_package("fidl_foo_tests") {
manifest = "meta/fidl_foo_tests.cmx"
deps = [ ":fidl_foo_tests_bin" ]
}
# meta/fidl_foo_tests.cmx
{
"program": {
"binary": "bin/fidl_foo_tests"
},
"sandbox": {
"services": [
"fuchsia.logger.LogSink", # some example services
"fuchsia.process.Launcher"
]
}
}
If you're grouping multiple components into a single fuchsia_test_package
,
each fuchsia_unittest_component
needs its own manifest.
For more information on package and component templates, see Building components.
Host tests
Assume we have a :fidl_bar_tests_bin
target that produces a fidl_bar_tests
binary. We must ensure that GN is in $host_toolchain
when it reaches that
target, otherwise it will try to build it for Fuchsia:
groups("tests") {
testonly = true
deps = [ ":fidl_bar_tests_bin($host_toolchain)" ]
}
(Always put ($host_toolchain)
in the BUILD.gn file's tests
group, not in
//bundles/fidl:tests.)
This will create a test_spec entry named host_x64/fidl_bar_tests
, which will
end up in out/default/tests.json:
{
"command": [ "host_x64/fidl_bar_tests", "--test.timeout", "5m" ],
"cpu": "x64",
"label": "//PATH/TO/BAR:fidl_bar_tests_bin(//build/toolchain:host_x64)",
"name": "host_x64/fidl_bar_tests",
"os": "linux",
"path": "host_x64/fidl_bar_tests",
"runtime_deps": "host_x64/gen/PATH/TO/BAR/fidl_bar_tests_bin.deps.json"
}
Running fx test fidl_bar_tests
works because of the "name" field in
tests.json.
Host/Device tests
Tests that run both on host and device fall in two categories. In the first category, the test target simply builds under either toolchain. For example:
import("//src/sys/build/components.gni")
rustc_test("fidl_rust_conformance_tests_bin") {
output_name = "fidl_rust_conformance_tests" # host test name
...
}
fuchsia_unittest_package("fidl_rust_conformance_tests") { # device test name
deps = [ ":fidl_rust_conformance_tests_bin" ]
}
group("tests") {
testonly = true
deps = [
":fidl_rust_conformance_tests_bin($host_toolchain)",
":fidl_rust_conformance_tests",
]
}
Or, if the component is packaged with other tests:
import("//src/sys/build/components.gni")
rustc_test("fidl_rust_conformance_tests_bin") {
output_name = "fidl_rust_conformance_tests" # host test name
...
}
fuchsia_unittest_component("fidl_rust_conformance_tests") { # device test name
deps = [ ":fidl_rust_conformance_tests_bin" ]
}
fuchsia_test_package("fidl_rust_tests") {
test_components = [
":fidl_rust_conformance_tests",
...
]
}
group("tests") {
testonly = true
deps = [
":fidl_rust_conformance_tests_bin($host_toolchain)",
":fidl_rust_tests",
]
}
We can now run the test both ways:
- on device:
fx test fidl_rust_conformance_tests --device
- on host:
fx test fidl_rust_conformance_tests --host
In the second category, the device and host tests share source code, but they
are sufficiently different that they must be defined by separate targets. This
requires wrapping the host test definition in if (is_host) { ... }
to prevent
GN complaining about multiple targets producing the same output. For example:
import("//src/sys/build/components.gni")
source_set("conformance_test_sources") {
...
}
test("fidl_hlcpp_conformance_tests_bin") {
output_name = "fidl_hlcpp_conformance_tests"
...
deps = [
":conformance_test_sources",
...
]
}
if (is_host) {
test("fidl_hlcpp_conformance_tests_bin_host") {
output_name = "fidl_hlcpp_conformance_tests" # host test name
...
deps = [
":conformance_test_sources",
...
]
}
}
fuchsia_unittest_package("fidl_hlcpp_conformance_tests") { # device test name
deps = [ ":fidl_hlcpp_conformance_tests_bin" ]
}
group("tests") {
testonly = true
deps = [
":fidl_hlcpp_conformance_tests_bin_host($host_toolchain)",
":fidl_hlcpp_conformance_tests",
]
}
Now, we can run the test both ways:
- on device:
fx test fidl_hlcpp_conformance_tests --device
- on host:
fx test fidl_hlcpp_conformance_tests --host
C++ Zircon tests
C++ tests in Zircon are usually defined using zx_test
. This template behaves
differently from most others like test
, rustc_test
, go_test
, etc.: it
appends -test
to the binary name by default. Do not rely on this, for two
reasons:
- The naming guidelines require a
_tests
suffix, not-test
. - This behaviour might change during build unification.
Instead, specify the target and binary name explicitly:
zx_test("fidlc_unit_tests") {
output_name = "fidlc_unit_tests" # explicit binary name
...
}
Rust unit tests
Rust libraries are often defined like this:
rustc_library("baz") {
with_unit_tests = true
...
}
This automatically creates a baz_test
target that builds a baz_lib_test
binary. Do not use this, for two reasons:
- The naming guidelines require a
_tests
suffix, not_test
. - It will be deprecated soon.
Instead of with_unit_tests
, write a separate rustc_test
target with an
appropriate name:
rustc_library("baz") {
...
}
rustc_test("fidl_baz_tests") {
...
}
Grouping
Suppose we have the following test structure:
- FIDL Rust
- Device
- Conformance
- Integration
- Host
- Conformance
- Device
We should have test targets for the leaves:
fx test fidl_rust_conformance_tests
fx test fidl_rust_integration_tests
We should not make additional targets for running various subsets of the
tests. Using fx test
, we can already
- run all tests:
fx test //path/to/fidl/rust
- run all device tests:
fx test //path/to/fidl/rust --device
- run all host tests:
fx test //path/to/fidl/rust --host