Diagnostics Persistence: Saving Inspect across reboot

Diagnostics Persistence is a service that stores specific Inspect data on device across one or more reboots. If you need to track failure states, historical metrics, or other telemetry that must survive a restart, configure this service to save your data automatically.

Behavior and use cases

Persistence is particularly useful for recovering diagnostic data after:

• Unexpected reboots: Persistence saves data periodically based on your configured frequency and survives device restarts.
• Component crashes: Persistence never drops a value once it has been saved. For example, if your component publishes data in the first sample, but drops that data or crashes prior to the next sample, Persistence retains the original data in the persistence file.

Quickstart guide

To configure Persistence to save your component's Inspect data, follow these steps:

• Identify your Inspect data
• Create a configuration file
• Estimate max_bytes
• Update the build

1. Identify your Inspect data

Select the specific data you need to save. You will need the exact INSPECT: selector for your component's data (e.g., INSPECT:core/pkg-resolver:root/resolver_service/active_package_resolves:*).

2. Create a configuration file

Create a new .persist file in either //src/diagnostics/config/persistence or //vendor/*/diagnostics/config/persistence.

The file uses JSON5. Define the parameters for the data you wish to persist:

[
  {
    tag: "cache-fallbacks", // Unique name
    service_name: "pkg-resolver", // Grouping for tags
    max_bytes: 500, // Max size of the persisted data
    min_seconds_between_fetch: 3600, // How frequently to sample the data
    selectors: [
      "INSPECT:core/pkg-resolver:root/resolver_service:cache_fallbacks_due_to_not_found",
      "INSPECT:core/pkg-resolver:root/resolver_service/active_package_resolves:*",
    ],
  },
]

3. Estimate max_bytes

The size of the persisted data is enforced at runtime. If your selectors fetch more data than max_bytes, all of the saved data for this tag will be permanently dropped and replaced with a single error string instead.

To estimate the correct max_bytes limit:

1. Run your component on a device and populate the Inspect data you wish to persist. For more information on populating Inspect data, see Codelab: Using Inspect.
2. Run ffx inspect show locally with your exact selectors, and pipe the output through jq to strip away un-persisted data. Finally, count the bytes using wc -c. For example:

ffx --machine json inspect show \
    'core/pkg-resolver:root/resolver_service:cache_fallbacks_due_to_not_found' \
    'core/pkg-resolver:root/resolver_service/active_package_resolves:*' \
    | jq -c '.[] | pick(.moniker, .payload.root)' \
    | wc -c

1. Add a generous buffer (e.g., 20-50%) to this total to account for string length variations, future field additions, and JSON formatting overhead.

4. Update the build

Add your new configuration file to the build by adding it to the diagnostics-persistence package configuration in //bundles/assembly/BUILD.gn.

Find the package_name = "diagnostics-persistence" block and add your .persist file to the files list:

      package_name = "diagnostics-persistence"
      files = [
        {
          source = "//src/diagnostics/config/persistence/netstack.persist"
          destination = "netstack.persist"
        },
+       {
+         source = "//src/sys/pkg/bin/pkg-resolver/pkg-resolver.persist"
+         destination = "pkg-resolver.persist"
+       },
      ]

Reading persisted data

On the next boot (after the software update check completes), the saved data is re-published into Inspect.

The data is hosted by the diagnostics-persistence component. The original path is prefixed with your configured service_name and tag.

$ ffx inspect show core/diagnostics/persistence
core/diagnostics/persistence:
  root:
    persist:
      pkg-resolver:
        cache-fallbacks:
          core/pkg-resolver:
            resolver_service:
              cache_fallbacks_due_to_not_found: 2

Configuration reference

The .persist JSON5 file format expects an array of objects, where each object defines a Persistence Tag. Each tag accepts the following fields:

Privacy considerations

Persistence is a powerful tool for debugging, but it can also be a privacy risk if not used carefully.

• Cross-boot linkage: Enabling persist_across_boot preserves saved data across boots, accumulating historical data. This creates a long-term record of device usage, which could be used to track users across multiple boots. This can also violate other privacy safeguards, such as fingerprinting a device or user across a time-limited pseudonymous ID.

• Data retention: Persistence data is stored on the device and can be accessed by anyone with physical access to the device. It is important to consider the sensitivity of the data you are persisting and whether it should be protected with additional security measures.

• Data minimization: Only persist the data that you need to debug your component. Avoid persisting unnecessary data, as this can increase the privacy risk.

FAQ

Does Persistence work with Lazy Nodes?

Yes. Persistence provides built-in support for Inspect Lazy Nodes. Persistence registers its required selectors and fetch frequencies with the Archivist. At each interval, the Archivist actively queries the component's selectors, which triggers the evaluation of any Lazy Nodes, allowing the system to save their dynamically generated data.