Fuchsia test output format

This document describes the directory format used to collect and store results and outputs for tests run on Fuchsia. It is currently output by ffx test run when the --output-directory option is specified.

Background

A single execution of ffx test produces a test run. A test run is comprised of any number of suite runs, and each suite run is comprised of any number of test cases.

A suite run is an execution of a test component. The test component is most commonly identified by its URL, for example, fuchsia-pkg://fuchsia.com/run_test_suite_integration_tests#meta/passing-test-example.cm.

A test case is an execution of a single test case contained in a suite.

In addition, test runs, suites, or test cases may produce artifacts. These include outputs from the test such as stdout, stderr, and syslog. Artifacts may be either a file or a directory.

Overview

Test output is stored as a directory. The directory contains the results for a single test run and any suite runs and test cases it contains. The root of the directory contains a single JSON file called run_summary.json and any number of subdirectories. Each subdirectory contains artifacts for either the test run, or a single suite run or test case.

Directory layout

The JSON file is always called run_summary.json and is always located in the root level of the directory. run_summary.json contains the complete set of results for the overall test run, suite runs within the test run, and test cases within each suite run. It also contains the name of each artifact subdirectory and a list of artifacts within it. Artifacts are always located in the top level of an artifact subdirectory.

The names of subdirectories and artifacts within the subdirectories are not specified as part of the schema. Instead, the actual names of subdirectories and artifacts are listed in run_summary.json.

`run_summary.json` contents

The exact schema for run_summary.json is published on fuchsia.dev. In Fuchsia source, it is available under the //sdk/schema directory.

Artifacts

Artifacts may consist of either a single file or a directory. The following artifacts exist today and are known to the scheme.

Type	Description
SYSLOG	Syslog collected from all components running in a test suite.
RESTRICTED_LOG	Any high severity logs that caused a test to be failed.
STDOUT	stdout collected from a single component in a test.
STDERR	stderr collected from a single component in a test.
REPORT	A copy of the stdout output produced by ffx test.
CUSTOM	A set of arbitrary files produced by a component in a test.

Parsing

Tools that need to process test results should begin by parsing run_summary.json, as it is the only part of the output directory with a location defined in the schema. The remaining contents of the directory are described by run_summary.json. Tools should not assume any particular naming convention for artifacts or subdirectories in the output.

Example

This example output was generated by running ffx test against an example test.

$ ffx test run --output-directory /tmp/ffx-out fuchsia-pkg://fuchsia.com/stdout-test#meta/stdout-test.cm

The contents of the /tmp/ffx-out directory generated by ffx test is available in examples/testing.