RFC-0024: Mandatory source compatibility

Summary

Establish a source-compatibility standard for FIDL language bindings, as well as a process for evolving that standard.

Motivation

Today, there are few written rules for the code generated by language bindings. It's expected that they conform to a specific wire ABI, but aside from that, binding authors are given lots of leeway over how they shape their APIs. Any change to a FIDL definition could cause arbitrary changes to the generated bindings.

In practice, users expect a sort of "common sense" list of things that should be source-compatible, such as defining a new top-level type. However, there's no explicit rule saying that this is the case. While this case seems somewhat absurd, it illustrates how a lack of specification can ruin users' expectations. Real examples of this that have occurred in practice include adding fields to tables, adding a new xunion variant, or adding a new defaulted field to a struct. Users could reasonably expect that these changes wouldn't be source-breaking, but there's no standard specifying this, and all of these changes cause source-level breakage in one or more language bindings today (e.g. due to positional initializers in C++ or Go, or struct patterns in Rust).

Furthermore, there are a number of extremely useful extensions to FIDL language bindings that have been rejected in the past due to their interaction with source compatibility. Examples of this include adding a copy or clone function to types that don't contain handles. Types that contain arbitrary handles cannot be cloned, so adding a handle to a type would prevent it from offering a clone function (or prevent it from offering a clone function that worked, at any rate). A change to introduce conditional inclusion of a clone function to generated Rust bindings based on the absence of handles has been rejected multiple times due to its effects on source compatibility. As a result, Fuchsia developers have had to manually roll their own clone functions and add wrapper types for the FIDL generated types that clone via these hand-rolled methods. This document proposes a consistent standard against which we can evaluate functionality like this, hopefully providing a more ergonomic, user-friendly, and boilerplate-free experience to developers.

Design

The Process

This FTP establishes an initial set of source compatibility constraints. This list will be tracked in a document in the Fuchsia source tree. Additional source compatibility constraints must be added using the FTP process. To facilitate easy addition of source compatibility rules related to new features, the "Backwards Compatibility" section of the FTP template will be amended to include a suggestion to introduce new source compatibility constraints (where applicable).

Definitions: Source Compatibility and Transitionability

Changes below are required to be either source-compatible (i.e., non-source-breaking) or transitionable.

Source-compatible changes must not cause source-breakage to any valid (compiling) usage of the public API of the generated FIDL bindings. There's some reasonable argument over the definition and feasibility of restricting which features are part of the "public API", and which aren't; so for the purposes of this document we consider the "public API" to be any use of the generated bindings that does not require either extraordinary language gymnastics (e.g. reflection) or explicit developer intent to violate privacy (e.g. calling __private_dont_use_me_function_2()). All other APIs exposed (e.g., positional initialization, pattern-matching, etc.) must be constrained so that user code cannot be broken by source-compatible changes to FIDL libraries.

Transitionable changes are changes for which it is possible to write code that compiles both before and after the change. Each transitionable source-compatibility rule must specify exactly what "use" of the API must be possible during the transition.

Initial Source Compatibility Constraints

The following are a list of changes that must be source-compatible:

• Adding a new top-level item (protocol, type, or constant).
  • Motivation: users expect that declaring new protocols, types, and constants can be done without breaking existing users of the FIDL library.
  • Exemption: usages with "*" or blanket imports from a namespace may experience breakage as a result of ambiguities between multiple items from different libraries with the same name.
• Adding a field to a non-strict table.
  • Motivation: tables are designed for easy extensibility and should support additional fields without breakage. To opt into breakage, the strict modifier can be used.
• Adding a variant to a non-strict extensible union.
  • Motivation: extensible unions are designed for easy extensibility and should support additional variants without breakage. To opt-in-to breakage, the strict modifier can be used.
• Adding a member to a non-strict enum
  • Motivation: non-strict enums are implicitly opting into expansibility and should be expandable without source breakage.
• Adding a member to a non-strict "bits"
  • Motivation: non-strict bits are implicitly opting into expansibility and should be expandable without source breakage
• Adding [Layout = "Simple"] to an existing protocol
  • Motivation: [Layout = "Simple"] exists in order to enable usage in simple C bindings. Existing protocols that conform should not require a breaking source change in order to specify that they can be used in the simple C bindings.
• Adding [MaxHandles] to an existing type
  • Motivation: [MaxHandles] exists to provide extra information about a type so that it can be used more permissively. It should not require a breaking source change in order to specify that a type already contains a fixed maximum number of handles and may be assumed to continue containing at most that number of handles.

The following are a list of changes that must be transitionable:

• Adding [Transitional] to a method
  • Use: it must be possible to implement a protocol and supply an implementation of a method using the same source both before and after the addition of the [Transitional] attribute to that method.
  • Motivation: it must be possible to gradually add or remove methods to protocols so long as all existing implementations can be gradually adapted.
• Adding a new [Transitional] method
  • Use: it must be possible to implement a protocol using the same source both before and after the addition of a new [Transitional] method (though the API need not allow implementation of the method during the transition).
  • Motivation: it must be possible to gradually add or remove methods to protocols so long as all existing implementations can be gradually adapted.
• Removing a [Transitional] method
  • Use: it must be possible to implement a protocol using the same source both before and after the removal of a [Transitional] method (though the API need not allow implementation of the method during the transition).
  • Motivation: it must be possible to gradually add or remove methods to protocols so long as all existing implementations can be gradually adapted.
• Removing a field of a non-strict table
  • Use: it must be possible to create a table and access its fields (except the one being removed) using the same source both before and after the removal of a table field.
  • Motivation: tables are designed to be evolved easily and should support removal without breakage. To opt into breakage, the strict modifier can be used on the table.
• Removing a variant of a non-strict extensible union
  • Use: it must be possible to create an xunion and access its variants (except the one being removed) using the same source both before and after the removal of an xunion variant.
  • Motivation: xunions are designed to be evolved easily and should support removal without breakage. To opt into breakage, the strict modifier can be used on the table.
• Marking a type as strict
  • Use: it must be possible to access all fields of a table or "bits" and all variants of an enum or xunion using the same source both before and after strict is added.
  • Motivation: strict is intended to be added to a type declaration once that type has stabilized, allowing increased reasoning and developer tooling. However, this is only required as a transitionable change and not a non-breaking change because extensible types may wish to allow access to unrecognized fields or variants. These capabilities don't make sense for a strict type, as unrecognized fields or variants would be rejected.
• Adding [Transitional] to a member of an enum or bits, field of a table, or variant of an extensible union.
  • Use: it must be possible to access all non-transitional members/bits/fields/variants and to construct values of the enum/bits/table/extensible union that do not include the [Transitional] value using the same source both before and after the introduction of [Transitional].
  • Motivation: it must be possible to gradually remove members, fields, or variants.
• Adding a new member of an enum or bits, field of a table, or variant of an extensible union marked as [Transitional].
  • Use: it must be possible to access all non-transitional members/bits/fields/variants and to construct values of the enum/bits/table/extensible union that do not include the [Transitional] value using the same source both before and after the introduction of the new [Transitional] field.
  • Motivation: it must be possible to gradually add members, fields, or variants.
• Removing a member of an enum or bits, field of a table, or variant of an extensible union marked as [Transitional].
  • Use: it must be possible to access all non-transitional members/bits/fields/variants and to construct values of the enum/bits/table/extensible union that do not include the [Transitional] value using the same source both before and after the removal of the [Transitional] field.
  • Motivation: it must be possible to gradually remove members, fields, or variants.

The following are potential constraints that have been omitted from this list, including justification as to why they have been omitted:

• Adding or removing fields (defaulted or not) from structs
  • This is an ABI-breaking change and would require other significant efforts to ensure a compatible transition. Making this a non-breaking change requires eliminating anything that does "for all fields"-style reasoning about a type, including automatic method derivation (e.g. "does this type contain any floats"), positional initializers, and exhaustive field matching and construction.
• Adding or removing fields/variants (defaulted or not) from strict tables and xunions
  • strict is intended to enable additional developer tooling that relies on "for all fields"-style reasoning about a type, including automatic method derivation (e.g. "does this type contain any floats"), positional initializers, and exhaustive field matching and construction. Forcing this to be a non-breaking change would inhibit this purpose.
• Adding handle-containing fields or variants to a type not marked with [MaxHandles]
  • Adding fields to a strict type or a struct is already a source-breaking change for other reasons, so adding a field with a handle is similarly a breaking change and may affect the APIs generated as a result.

Implementation strategy

This FTP establishes the initial proposed language compatibility standard. Bugs will be filed and assigned to one author of each language binding to ensure that their languages bindings are compliant.

Ergonomics

This change makes FIDL easier to use by setting clear standards for source compatibility, allowing for automatic checking as well as easier manual checking of FIDL changes' source-compatibility, as well as offering bindings authors clearer guidance on source compatibility, allowing them the freedom to make bindings that are language-idiomatic while still respecting standard requirements of the project.

Documentation and examples

Following the acceptance of this FTP, the process established by the FTP as well as the source compatibility rules themselves will be published along with other FIDL reference documentation.

Backwards compatibility

Application of the guidance proposed may require changes to bindings and uses of those bindings, it is up to the respective binding authors to navigate such changes.

This section (the "backwards compatibility" section) will be amended to include the following text:

"If you are introducing a new data type or language feature, consider what changes you would expect users to make to FIDL definitions without breaking users of the generated code. If your feature places any new source compatibility restrictions on the generated language bindings, list those here."

Note that you should include the source compatibility text as an actual link to this FTP, that is:

[source compatibility](/docs/contribute/governance/rfcs/0024_mandatory_source_compatibility.md)

Performance

This FTP does not restrict runtime behavior, although the restrictions on source APIs may cause language binding authors to design more or less performant APIs. The feasibility of creating performant bindings in supported languages should be considered when new source compatibility restrictions are introduced.

This feature might affect compile-time performance as a result of pushing towards patterns that require heavier inlining and compiler optimizations in order to be performant (e.g. optimizing away a complex builder API into a simple struct initialization). Bindings authors should strive to make design choices that don't significantly hamper compile times, but the compile-time-consequence of a particular language API should not necessarily prevent the introduction of a new source compatibility restriction.

Security

This feature does not affect security.

Testing

Many source compatibility rules are of the form "there cannot exist any user code that compiled before this change but not after this change." Unfortunately, these restrictions are difficult or impossible to test because they would require enumerating every possible usage of the API before the change.

However, we can (and should) add items to the FIDL change test suite to show that there does exist some usage of the API before the change that remains valid after the change. This is a necessary but not sufficient condition for meeting the source compatibility requirements.

Drawbacks, alternatives, and unknowns

• Don't introduce a specification like this. Allow bindings authors to choose how breaking or non-breaking they want their changes to be. This is roughly similar to the current status de jure, but would give bindings authors more flexibility than they are granted de facto under the current system, in which some source-compatibility-hostile changes have received pushback.
• Create a specification for which changes are allowed to be source-breaking, rather than which ones are not allowed to be source-breaking. This is tougher to enforce and would require bindings authors to anticipate changes under which their bindings must remain source-compatible.
• A slight modification would be to specify both changes that are and are not, with unspecified changes defaulting one way or another -- this is essentially the same as either this FTP or the alternative above depending on the default, although it sets up a more official expectation around documenting the effects of different FIDL changes.

Prior art and references

Previous attempts have been made to introduce evolvability restrictions via the [MaxHandles] attribute. This design and the intended modifications to it have been discussed in earlier parts of this proposal.