Updating the VMO file format

This document describes how to update or extend the Component Inspection File Format.

While extending the format, it is imperative that you don't break any existing functionality, in particular, the Inspect readers and validation tests. However, packing all changes into a single change can be overwhelming to review. In general, there should be 5-9 changes or steps in a chain to alter the VMO format:

  1. (Possibly not applicable) Choose a type number by updating the VMO Format Docs.
  2. Update the Rust reader.
  3. Update the C++ reader.
  4. Update the Rust writer.
  5. Update the C++ writer.
  6. (Potentially) Update the validator.
  7. Update the documentation.
  8. (Not a change)(Optional) Send a feature announcement.

Choosing a type number

View the type table in the Inspect file format and choose an available type number. There are a total of 256 possible types in the current specification.

In order to reserve the new type, update the Inspect file format.

Implementation

It is cumbersome to test a reader or writer without the other and difficult to get the proper block-level API without both a reader and writer.

To test a reader and writer:

  1. Pick a language and design and implement the feature entirely in that language, using unit tests to model actual usage of the API.

  2. Split the changes into separate reader and writer changes, stacking the writer on top of the reader.

    At this point, tests in the reader that rely on the writer API are probably broken in the reader change.

  3. Rebase into that change and rewrite the tests (keeping around the original version of the tests) using lower level functionality.

    Typically, you can put all changes to block-code in the reader change, making it possible but messy to write tests for the reader API.

  4. The test will be ugly. Rebase into the writer change and remove the modified tests, replacing them with the original tests that are written in terms of the high level API.

  5. Use the two changes for reference and duplicate them in the second language.

  6. (Optional) Depending on the contents of your change, you may have to update the validator tests as you go, since changes to the existing format of existing blocks will likely break them.

The following sections outline how to put together each change, but in practice, take these as hints for designing the whole system cohesively before splitting the design as described above.

Update the Rust implementation

The examples in this section create a new type called MyFoo.

Set up

  1. Include tests:
fx set core.x64 --with //src/lib/diagnostics:tests

  1. Run tests:

fx test inspect-format-tests fuchsia-inspect-tests

Reader change

Bitfield updates

  1. If defining a new Block Type, update the BlockType enum.

  2. Update the methods and functions defined for BlockType.

  3. If changing the fields in an existing block or creating a new BlockType, update the bitfield layout.

  4. Run inspect-format-tests to verify changes compile:

fx test inspect-format-tests

  1. Update the block definition to include methods for reading and writing the new fields. It is appropriate to include write-functionality at this time in the block library; it is next to impossible to write a reader test without it.

  2. Write block tests that exercise the new functionality.

  3. Write tests that make assertions on the first 8-16 bytes of the block.

    Typically this means writing the expected contents as a &[u8] that contains hex values and asserting its equivalence to the buffer the block is using as a container.

Update the reader

You can find the reader code in mod.rs. The tests in this change will probably be tricky, because the high-level API writer doesn't exist yet.

Writer change

State

The primary changes here will be in the State functionality. This is where blocks can be allocated and converted into the new type. If all you're doing is modifying an existing block, this is likely the only place you need to make changes.

Creating a new value type

There is a types directory where you can add a new file for your type.

  1. Create the new type in the file created. Use an existing type as an example. Types always have access to internal State. Use this to create the necessary methods on your new type, calling into the methods created in State.

  2. Add a method to Node for creating the new type.

  3. Ensure that your type has RAII semantics in the VMO. If your type is a value, this is probably done automatically by the boilerplate copied from an existing type in step 1.

Finally, go back and update the tests from the Reader change to use the new API!

Update the C++ implementation

The examples in this section create a new type called MyFoo.

As noted above, this section should be two changes in Gerrit.

Set up

  1. Include tests:
fx set core.x64 --with //zircon/system/ulib/inspect:tests
  1. Run tests.
fx test inspect-cpp-unittest

Reader change

Bitfield updates

This section describes how to define the bitfields for your new type.

Update the block definition.

  1. Change BlockType to include your new type. For example: kMyFoo = #;

  2. If your type needs a new header (typically if it is not a VALUE), define the header bitfields for your type with a struct. For example: struct MyFooBlockFields final : public BlockFields.

  3. If your type needs a new payload (it requires using the second 8 bytes of the block), define the payload bitfields for your type with a struct. For example: struct MyFooBlockPayload final.

  4. If your type contains enums (such as format), define a new enum at the top of block.h. For example: enum class MyFooBlockFormat : uint8_t.

Implement the type reader

This section describes how to make your new type readable.

Update the Inspect hierarchy based on your type:

A value (child of Node)

  1. Update PropertyFormat enum with a new number for your type. This must be sequential in this specific enum and does not need to match the format type ordinal you chose.

  2. Create a new value type. For example, using MyFooValue = internal::Value<T, static_cast<size_t>(PropertyFormat::kMyFoo)>;

  3. Update PropertyValue variant with the new value. Note: The index in fit::internal::variant must match the value of PropertyFormat.

Not a value

  1. You need to make your own in-memory representation objects in the hierarchy file.

  1. Update the actual reader.

  2. Update InnerScanBlocks to dispatch your type. If you are creating a new Property, you may only have to add your BlockType.

  3. If you need a custom parser, implement InnerParseMyFoo, which takes a parent (if needed) and the pointer to the scanned block.

Writer change

Type wrapper declaration

This section describes how to declare a C++ RAII-style wrapper for your new type.

Type wrappers contain indices of blocks that are owned by the type. You are responsible for implementing operations on those blocks, including creation and deletion, in State action updates.

Update the writer types definition.

Determine if you can reuse an existing wrapper or if you need a bespoke type:

Reuse

  1. If you need to support Add, Subtract, and Set: using MyFoo = internal::NumericProperty<T>, where T is the argument type to those operations.

  2. If you need to support Set: using MyFoo = internal::Property<T>, where T is the argument type to Set.

  3. If you need to support numeric operations on an array: using MyFood = internal::ArrayProperty<T>, where T is the argument type for slots in the array.

  4. If you need to support inserting to a histogram: using MyFoo = internal::{Linear,Exponential}Histogram<T>, where T is the argument to Insert.

Bespoke

  1. Create a new type wrapper. For example class MyFoo final.

  2. Ensure your class has internal::State as a friend class. Note: See class Link for a copyable starting point.

State action updates

The State class is the actual implementation for all operations on all types. This section describes how to implement the operations you will need to complete your wrapper implementation.

  1. Update State header:

    1. Add Create and Free methods. For example: MyFoo CreateMyFoo(<args>); void FreeMyFoo(MyFoo* property); where args typically includes name, parent, and some initial value.

    2. Add methods for each operation you need to support on your type. For example, if your type can be Set, void SetMyFoo(MyFoo* property, T), where T is the same type from your update to types.h.

  2. Update State:

    1. Implement your new type's methods. The implementation varies between the different types. This section provides a high-level overview of what each method must do:

      • MyFoo CreateMyFoo(Args...) is responsible for allocating a number of blocks, setting their values, and returning them wrapped in a MyFoo. You may use a private constructor to create MyFoo from the BlockIndex objects it wraps. Various internal helpers exist to simplify this operation. See CreateIntProperty for an example.

      • void FreeMyFoo(MyFoo* property) is responsible for freeing all blocks wrapped by the MyFoo. There are sometimes particular ordering requirements or updates necessary for freeing blocks. See InnerFreeValue for an example of how values are freed.

      • Operations, such as void SetMyFoo(MyFoo* property, T value) change the value of blocks allocated to MyFoo to implement the operation. See SetIntProperty for an example.

Implement the type wrapper

This section describes how to implement the wrapper methods declared previously.

  1. Update the writer type definitions:

    • If you used an existing templated type, you need to override each method for your new base type T. For example, if you typed using MyFoo = internal::Property<T>, you will write: template<> void internal::Property<T>::OPERATION(...) { ... }

    • If you created your own type, simply create definitions for the methods you declared. You need to do the following:

      • Make your constructor call state_->CreateMyFoo(...);

      • Make your destructor call state_->FreeMyFoo(...);

      • Make your other methods call the corresponding implementation on State.

      • Have all your constructors and methods check that state_ is not null before calling.

Implement tests

  1. Update state unit tests with tests for your low-level operations.

  2. Update reader unit tests with tests for your high-level reader implementation.

Example change chain

  1. C++ Reader
  2. Rust Reader
  3. Rust Writer
  4. Rust Validator Changes
  5. C++ Validator Changes
  6. Documentation Updates