This document describes the directory format used to collect and store results
and outputs for tests run on Fuchsia. It is currently output by
ffx test run when the
--output-directory option is specified.
A single execution of
ffx test produces a test run. A test run is
comprised of any number of suite runs, and each suite run is comprised of
any number of test cases.
A suite run is an execution of a test component. The test component is most
commonly identified by its URL, for example,
A test case is an execution of a single test case contained in a suite.
In addition, test runs, suites, or test cases may produce artifacts. These include outputs from the test such as stdout, stderr, and syslog. Artifacts may be either a file or a directory.
Test output is stored as a directory. The directory contains the results for a
single test run and any suite runs and test cases it contains. The root of the
directory contains a single JSON file called
run_summary.json and any number
of subdirectories. Each subdirectory contains artifacts for either the test
run, or a single suite run or test case.
The JSON file is always called
run_summary.json and is always located in the
root level of the directory.
run_summary.json contains the complete set of
results for the overall test run, suite runs within the test run, and test
cases within each suite run. It also contains the name of each artifact
subdirectory and a list of artifacts within it. Artifacts are always located in
the top level of an artifact subdirectory.
The names of subdirectories and artifacts within the subdirectories are not
specified as part of the schema. Instead, the actual names of subdirectories
and artifacts are listed in
Artifacts may consist of either a single file or a directory. The following artifacts exist today and are known to the scheme.
|SYSLOG||Syslog collected from all components running in a test suite.|
|RESTRICTED_LOG||Any high severity logs that caused a test to be failed.|
|STDOUT||stdout collected from a single component in a test.|
|STDERR||stderr collected from a single component in a test.|
|REPORT||A copy of the stdout output produced by ffx test.|
|CUSTOM||A set of arbitrary files produced by a component in a test.|
Tools that need to process test results should begin by parsing
run_summary.json, as it is the only part of the output directory with a
location defined in the schema. The remaining contents of the directory are
run_summary.json. Tools should not assume any particular naming
convention for artifacts or subdirectories in the output.
This example output was generated by running
ffx test against an
$ ffx test run --output-directory /tmp/ffx-out fuchsia-pkg://fuchsia.com/stdout-test#meta/stdout-test.cm
The contents of the
/tmp/ffx-out directory generated by
ffx test is
available in examples/testing.
RFC-0163: RFC proposing this output.