Test metadata

Fuchsia uses TESTING.json5 files to provide additional metadata for tests on a per-directory basis. This metadata is used to organize and visualize test coverage in dashboards and reporting tools.

Purpose

The primary purpose of TESTING.json5 is to associate a directory (and the tests within or under it) with a specific category and optionally other metadata.

The primary use cases include:

  • Visualizing test coverage across different areas of the operating system.
  • Identifying areas that lack tests or have uncategorized tests.

Location

TESTING.json5 files should be placed at the root of a component, subsystem, or major directory in the source tree. For example:

  • src/diagnostics/TESTING.json5

Files placed in a directory apply to that directory and all its subdirectories, unless overridden by another TESTING.json5 file deeper in the tree.

Format

The file is a JSON5 document containing a single object with a coverage field. The coverage field contains zero or more of the following fields:

  • category: An optional string representing the category.
  • subcategory: An optional string providing a more specific subcategory.
  • tags: An optional list of strings representing tags.

Here is the schema:

{
  coverage: {
    category: "Category Name",
    subcategory: "Subcategory Name",
    tags: ["tag1", "tag2"],
  },
}

Overriding behavior

A TESTING.json5 file applies to all subdirectories, but individual fields can be overridden as follows:

  • The coverage.category and coverage.subcategory fields are exclusive. If a subdirectory has one of those fields set, it overrides the matching field from the nearest TESTING.json5 parent.
  • The coverage.tags field is additive. If a subdirectory has a coverage.tags field, the tags are merged with the tags from each ancestor TESTING.json5 file.

Viewing categories

The fx test-category tool can be used to view the computed categories for any directory in the tree. Use this tool to verify that the computed categories look reasonable and to find uncategorized directories.

Pass a directory to the tool to see the final computed categories:

fx test-category src/diagnostics/archivist

This shows output:

src/diagnostics/archivist:
  category: "Diagnostics"
  subcategory: "Archivist"
  tags: []

Pass the --stats flag to show instead the aggregated statistics for the given subdirectory or the whole tree:

fx test-category src/diagnostics --stats

This shows output:

Categories and Subcategories:
  Diagnostics: 214
    Archivist: 45
    Test Only: 38
    None: 32
    Libraries: 26
    Detect: 18
    Utilities: 16
    Sampler: 16
    Persistence: 9
    Triage: 9
    Tools: 5

Tags:
  lib: 26

Pass the --web flag to launch a simple web page that allows you to interactively navigate and view categories for the tree:

fx test-category --web

This shows output:

View categories at http://localhost:6240?files=metadata.json
Not seeing the categories you expect? Run `touch BUILD.gn` and then `fx build`

Press enter to stop the server.

Category style guide

The format of TESTING.json5 is intentionally loose to support incremental refinement of categories over time, and getting categories wrong temporarily is very low cost.

There are only a few hard rules to follow, which are enforced automatically:

  • Every directory has a category.

    This is satisfied by the top-level //TESTING.json5 file.

  • A directory may not have a subcategory if its category is "Uncategorized".

  • Every TESTING.json5 file must be formatted by fx format-code.

Suggested category layout

Consider the following recommendations:

  • Provide categories for the top-level directory of a subsystem or component, but not for every subdirectory.

    For example, src/diagnostics can set category: "Diagnostics", and subdirectories src/diagnostics/lib and src/diagnostics/archivist simply inherit that category.

  • Categories may generally map to a team name, but do not have to. They should generally match the directory naming.

    For example, category "Diagnostics" matches the team who owns the directory.

  • Set subcategories in subdirectories of a categorized directory. These should relate to the purpose of the subdirectory.

    For example, src/diagnostics/archivist sets subcategory: "Archivist" to group all Archivist-related tests together.

  • Set tags for cross-team or cross-component concerns.

    For example, src/diagnostics/lib and src/lib/diagnostics both set the "lib" tag. We can then group coverage for all "libs" across the tree.

Examples

Given the following source layout:

src/my_team/
src/my_team/component1/
src/my_team/component2/
src/my_team/lib/
src/my_team/lib/component1_client
src/lib/my_team/server_library

Consider creating the following files:

// src/my_team/TESTING.json5
{
  coverage: {
    category: "My Team",
  },
}

// src/my_team/component1/TESTING.json5
{
  coverage: {
    subcategory: "Component1",
  },
}

// src/my_team/component2/TESTING.json5
{
  coverage: {
    subcategory: "Component2",
  },
}

// src/my_team/lib/TESTING.json5
{
  coverage: {
    tags: ["lib"],
  },
}

// src/lib/my_team/TESTING.json5
{
  coverage: {
    category: "My Team",
    tags: ["lib"],
  },
}