RFC-0041: Support for unifying services and devices

Summary

Introduce the notion of a service — a collection of protocols, where there may be one or more instances of the collection.

Motivation

Today, within the component framework, a service is defined as a single protocol, and only one instance of that protocol may exist in the namespace of a process under /svc. This prevents us from describing more complex relationships:

• A service that is expressed in two different forms, depending on the consumer — e.g., when there are two different versions of the protocol, like FontProvider and FontProviderV2
• A service that is split in two, in order to grant features based on levels of access — e.g., regular access versus administrative access, like Directory and DirectoryAdmin, where the latter provides privileged access
• A service that is comprised of many different protocols for use by different consumers — e.g., like Power for power management, and Ethernet for network stacks
• A service that has multiple instances — e.g., multiple audio devices offering AudioRenderer, or multiple printers exposing Printer

Providing this flexibility allows a service to be more clearly expressed, without resorting to the use of workarounds like service hubs. With that flexibility, we can define devices as services. Concretely, we plan to evolve /svc/$Protocol which implies "only one protocol per process namespace" to:

/svc/$Service/$Instance/$Member

Which instead introduces two additional indirections: a service (e.g., printer, ethernet), and an instance (e.g., default, deskjet_by_desk, e80::d189:3247:5fb6:5808). A path to a protocol will then consist of the following parts:

• $Service — the fully-qualified type of the service, as declared in FIDL
• $Instance — the name of an instance of the service, where "default" is used by convention to indicate the preferred (or only) instance made available
• $Member — a service member name, as declared in FIDL, where the declared type of that member indicates the intended protocol

Design

Flavours of Services

Let's first consider various flavours of service we aim to support:

• A single, unique protocol: ONE instance, ONE protocol:

  /svc/fuchsia.Scheduler/default/profile_provider
  

• A composite of multiple protocols: ONE instance, MANY protocols:

  /svc/fuchsia.Time/default/network
                        .../rough
  

• Multiple instances of a service, with a single protocol: MANY instances, ONE protocol:

  /svc/fuchsia.hardware.Block/0/device
                          .../1/device
  

• Multiple instances, with different sets of protocols: MANY instances, MANY protocols:

  /svc/fuchsia.Wlan/ff:ee:dd:cc:bb:aa/device
                                  .../power
                .../00:11:22:33:44:55/access_point
                                  .../power
  

Language

To introduce the notion of a service to FIDL and support the various flavours, we will make the following changes to the FIDL language:

1. Add a service keyword.
2. Remove the Discoverable attribute.

The service keyword will allow us to write a service declaration, which we can use to define a set of protocols as members of a service. For example, we can declare the different flavours of service as follows:

• A single, unique protocol: ONE instance, ONE protocol:

  service Scheduler {
    fuchsia.scheduler.ProfileProvider profile_provider;
  };
  

• A composite of multiple protocols: ONE instance, MANY protocols:

  service Time {
    fuchsia.time.Provider network;
    fuchsia.time.Provider rough;
  };
  

• Multiple instances of a service, with a single protocol: MANY instances, ONE protocol:

  service Block {
    fuchsia.hardware.block.Device device;
  };
  

• Multiple instances, with different sets of protocols: MANY instances, MANY protocols

  service Wlan {
    fuchsia.hardware.ethernet.Device device;
    fuchsia.wlan.AccessPoint access_point;
    fuchsia.hardware.Power power;
  };
  

A service declaration may have multiple members that use the same protocol, but each member declaration must use a different identifier. See "a composite of multiple protocols" above.

When an instance of a service may contain a different set of protocols from another instance, the service declaration declares all possible protocols that may be present in any instance. See "multiple instances, with different sets of protocols" above.

A service declaration makes no mention of the names of specific instances of a service or the URI of the components that offer the service, this is left to the purview of the component framework based on component manifest declarations and use of its APIs at runtime.

Language bindings

Language bindings will be modified to make connecting to a service more convenient. Specifically, they will become more service-oriented, for example:

• Connect to the "default" instance of a service, with a single protocol: ONE instance, ONE protocol:

  • C++:

  Scheduler scheduler = Scheduler::Open();
  ProfileProviderPtr profile_provider;
  scheduler.profile_provider().Connect(profile_provider.NewRequest());
  

  • Rust:

  let scheduler = open_service::<Scheduler>();
  let profile_provider: ProfileProviderProxy = scheduler.profile_provider();
  

• Connect to the "default" instance of a service, with multiple protocols: ONE instance, MANY protocols:

  • C++:

  Time time = Time::Open();
  ProviderPtr network;
  time.network().Connect(&network);
  ProviderPtr rough;
  time.rough().Connect(&rough);
  

  • Rust:

  let time = open_service::<Time>();
  let network = time.network();
  let rough = time.rough();
  

• Connect to multiple instances of a service, with a single protocol: MANY instances, ONE protocol:

  • C++:

  Block block_0 = Block::OpenInstance("0");
  DevicePtr device_0;
  block_0.device().Connect(&device_0);
  
  Block block_1 = Block::OpenInstance("1");
  DevicePtr device_1;
  block_1.device().Connect(&device_1);
  

  • Rust:

  let block_0 = open_service_instance::<Block>("0");
  let device_0 = block_0.device();
  let block_1 = open_service_instance::<Block>("1");
  let device_1 = block_1.device();
  

• Connect to multiple instances of a service, with multiple protocols: MANY instances, MANY protocols:

  • C++:

  Wlan wlan_a = Wlan::OpenInstance("ff:ee:dd:cc:bb:aa");
  DevicePtr device;
  wlan_a.device().Connect(&device);
  Power power_a;
  wlan_a.power().Connect(&power_a);
  
  Wlan wlan_b = Wlan::OpenInstance("00:11:22:33:44:55");
  AccessPoint access_point;
  wlan_b.access_point().Connect(&access_point);
  Power power_b;
  wlan_b.power().Connect(&power_b);
  

  • Rust:

  let wlan_a = open_service_instance::<Wlan>("ff:ee:dd:cc:bb:aa");
  let device = wlan_a.device();
  let power_a = wlan_a.power();
  
  let wlan_b = open_service_instance::<Wlan>("00:11:22:33:44:55");
  let access_point = wlan_b.access_point();
  let power_b = wlan_b.power();
  

The following illustrates the proposed function signatures.

Note that the Open() and OpenInstance() methods also accept an optional parameter to specify the namespace. By default, the process's global namespace will be used (can be retrieved using fdio_ns_get_installed).

// Generated code.
namespace my_library {
class MyService final {
public:
  // Opens the "default" instance of the service.
  //
  // |ns| the namespace within which to open the service or nullptr to use
  // the process's "global" namespace as defined by |fdio_ns_get_installed()|.
  static MyService Open(fdio_ns_t* ns = nullptr) {
    return OpenInstance(fidl::kDefaultInstanceName, ns);
  }

  // Opens the specified instance of the service.
  //
  // |name| the name of the instance, must not be nullptr
  // |ns| the namespace within which to open the service or nullptr to use
  // the process's "global" namespace as defined by |fdio_ns_get_installed()|.
  static MyService OpenInstance(const std::string& instance_name,
                                fdio_ns_t* ns = nullptr);

  // Opens the instance of the service located within the specified directory.
  static MyService OpenAt(zxio_t* directory);
  static MyService OpenAt(fuchsia::io::DirectoryPtr directory);

  // Opens a directory of available service instances.
  //
  // |ns| the namespace within which to open the service or nullptr to use
  // the process's "global" namespace as defined by |fdio_ns_get_installed()|.
  static fidl::ServiceDirectory<MyService> OpenDirectory(fdio_ns_t* ns = nullptr) {
    return fidl::ServiceDirectory<MyService>::Open(ns);
  }

  // Gets a connector for service member "foo".
  fidl::ServiceConnector<MyService, MyProtocol> foo() const;

  // Gets a connector for service member "bar".
  fidl::ServiceConnector<MyService, MyProtocol> bar() const;

  /* more stuff like constructors, destructors, etc... */
}

And the bindings code:

/// FIDL bindings code.
namespace fidl {
constexpr char[] kDefaultInstanceName = "default";

// Connects to a particular protocol offered by a service.
template <typename Service, typename Protocol>
class ServiceConnector final {
public:
   zx_status_t Connect(InterfaceRequest<Protocol> request);
};

// A directory of available service instances.
template <typename Service>
class ServiceDirectory final {
public:
  // Opens a directory of available service instances.
  //
  // |ns| the namespace within which to open the service or nullptr to use
  // the process's "global" namespace as defined by |fdio_ns_get_installed()|.
  static ServiceDirectory Open(fdio_ns_t* ns = nullptr);

  // Gets the underlying directory.
  zxio_t* directory() const;

  // Gets a list of all available instances of the service.
  std::vector<std::string> ListInstances();

  // Opens an instance of the service.
  Service OpenInstance(const std::string& name);

  // Begins watching for services to be added or removed.
  //
  // Invokes the provided |callback| to report all currently available services
  // then reports incremental changes.  The callback must outlive the returned
  // |Watcher| object.
  //
  // The watch ends when the returned |Watcher| object is destroyed.
  [[nodiscard]] Watcher Watch(WatchCallback* callback,
                              async_dispatcher_t* dispatcher = nullptr);

  // Keeps watch.
  //
  // This object has RAII semantics.  The watch ends once the watcher has
  // been destroyed.
  class Watcher final {
  public:
    // Ends the watch.
    ~Watcher();
  };

  // Callback invoked when service instances are added or removed.
  class WatchCallback {
  public:
    virtual void OnInstanceAdded(std::string name) = 0;
    virtual void OnInstanceRemoved(std::string name) = 0;
    virtual void OnError(zx_status_t error) = 0;
  };
}

Language bindings will further expand upon these by offering convenient methods of iterating through instances of a service, and watching for new instances to become available.

Service Evolution

To evolve a service, we can add new protocols to it over time. In order to maintain source compatibility, existing protocols should not be removed, otherwise source compatibility may be broken as users may depend on the code generated from the service by language bindings.

As all protocols within a service are effectively optional, they may or may not be provided at runtime and components should be built for that eventuality, it simplifies the set of problems we face when evolving a service:

• Adding a protocol member to a service can be done at any time
• Removing a protocol member should be avoided (for source compatibility)
• Renaming a protocol member involves adding a new protocol member, and leaving the existing protocol member

To evolve a service itself, we have a similar set of restrictions. A service is not guaranteed to exist within a component's namespace, and a service can be visible at multiple different locations within a namespace, therefore:

• Adding a service can be done at any time
• Removing a service should be avoided (for source compatibility)
• Renaming a service involves duplicating a service and using a new name, whilst keeping the original copy of the service (for source compatibility)

Possible Extensions

We expect service instances to eventually become 'first class' and be allowed to be part of messages, just like protocol P handles can be passed around as P or request<P>. This might take the form of something like service_instance<S> for a service S. We will make sure that this extension is possible, without putting working behind it today.

We leave the door open to (and plan on) expanding the kinds of members possible beyond solely allowing protocols. For instance, we may want to have a VMO (handle<vmo>) exposed by a service:

service DesignedService {
    ...
    handle<vmo>:readonly logo; // gif87a
};

Implementation strategy

This proposal should be implemented in phases, so as not to break existing code.

Phase 1

1. Modify component_manager, so that components v2 supports the new directory schema for services.
2. Modify appmgr and sysmgr, so that components v1 supports the new directory schema for services.

Phase 2

1. Add support for service declarations.
2. Modify the language bindings to generate services.

Phase 3

1. For all protocols that have a Discoverable attribute, create appropriate service declarations. > Note: at this stage, we should verify that there are no name > collisions possible between the old and new directory schemas for services.
2. Migrate all source code to use services.

Phase 4

1. Remove all Discoverable attributes from FIDL files.
2. Remove support for Discoverable from FIDL and the language bindings.
3. Remove support for the old directory schema from component_manager, appmgr, and sysmgr.

Documentation and examples

We would need to expand the FIDL tutorial to explain the use of service declarations, and how they interact with protocols. We would then explain the different structures of a service: singleton vs multi-instance, and how the language bindings can be used.

Glossary

A protocol declaration describes a set of messages that may be sent or received over a channel and their binary representation.

A service declaration describes a capability that is offered as a unit by a service provider. It consists of a service name and zero-or-more named member protocols that clients use to interact with the capability.

The same protocol may appear more than once as a member of a service declaration, with the member's name indicating the intended interpretation of a protocol:

service Foo {
    fuchsia.io.File logs;
    fuchsia.io.File journal;
};

A component declaration describes a unit of executable software, including the location of the component's binaries and the capabilities (such as services) that it intends to use, expose, or offer to other components.

This information is typically encoded as a component manifest file within a package:

// frobinator.cml
{
    "uses": [{ "service": "fuchsia.log.LogSink" }],
    "exposes": [{ "service": "fuchsia.frobinator.Frobber" }],
    "offers": [{
        "service": "fuchsia.log.LogSink",
        "from": "realm",
        "to": [ "#child" ]
    }],
    "program": { "binary": ... }
    "children": { "child": ... }
}

A service instance is a capability that conforms to a given service declaration. On Fuchsia, it is represented as a directory. Other systems may use different service discovery mechanisms.

A component instance is a particular instance of a component with its own private sandbox. At runtime, it uses service instances offered by other components through opening directories in its incoming namespace. Conversely, it exposes its own service instances to other components by presenting them in its outgoing directory. The component manager acts as a broker for service discovery.

• A component instance is often (but not always) one-to-one with a process.
• Component runners can often run multiple component instances within the same process each with its own incoming namespace.

Idiomatic Use of Services

Backwards compatibility

This proposal will deprecate, and eventually remove the Discoverable attribute from FIDL.

There are no changes to the wire format.

If you are introducing a new data type or language feature, consider what changes you would expect users to make to FIDL definitions without breaking users of the generated code. If your feature places any new source compatibility restrictions on the generated language bindings, list those here.

Performance

This should have no impact on IPC performance when connecting to the default instance of a service, or an instance known a priori.

To connect to a different instance, where the instance ID is not known a priori, will require the user to list the service's directory and locate the instance before connecting.

There will be a minimal impact on build and binary size, as service definitions must be generated by backends for particular language bindings.

Security

This proposal will allow us to enforce more fine-grained access control, as we can split a service into separate protocols with different access rights.

This proposal has no other effect on security.

Testing

Unit tests in the compiler, and changes to the compatibility test suite to check that protocols contained within services can be connected to.

Drawbacks, alternatives, and unknowns

The following questions are explored:

• Why do service declarations belong in FIDL?
• What is the difference between a protocol, a service, and a component?
• Is the proposed flat topology for service instances sufficiently expressive?
• How should we extend services over time?
• If a component instance wishes to expose multiple services that relate to a single underlying logical resource, how is that expressed?

Q1: Why do service declarations belong in FIDL?

Response

• We use FIDL to describe Fuchsia's system API including the protocols that components exchange.
• The same protocols may be used in many ways depending on the situation. Representing the various uses of these protocols as services makes it easier for developers to access the right set of protocols for each situation.
• FIDL already provides language bindings that can readily be extended to provide developers a consistent and convenient way to access these services.

Discussion

• [ianloic] But what about component manifests? Why not use FIDL to describe those too?
• [jeffbrown] component manifests describe concepts that go well beyond IPC concerns
• [abdulla] describing services in component manifests would lead to duplication of the description of those services
• [ianloic] could we generate the skeleton of a component from its manifest?
• [drees] putting service declarations in FIDL is imposing a specific structure, does this make sense on other platforms?
• [jeffbrown] we want declarations of services to be external to components because they need to be shared between components, it is the point of agreement for service exchange
• [ianloic] service declarations for overnet likely to be similar
• [pascallouis] Is it is good to start simple based on what we know we need now. We can adapt later as needed.
• [pascallouis] FIDL is Fuchsia first so it makes sense to introduce features that only make sense in that context given the information we have today but that over time could be generalized for other contexts
• [dustingreen] what about a separate file?
• [pascallouis] those files would be very small and lonely, opportunities for static type checking if we keep them in FIDL, seems low risk to move it later if needed

Q2: What is the difference between a protocol, a service, and a component?

Response

• A protocol declaration describes a set of messages that may be sent or received over a channel and their binary representation.
• A service declaration describes a capability that is offered as a unit by a service provider. It consists of a service name and zero-or-more named member protocols that clients use to interact with the capability.
  • The same protocol may appear more than once as a member of a service declaration; the member's name indicates the intended interpretation of a protocol.
    • e.g., service Foo { fuchsia.io.File logs; fuchsia.io.File journal; };

• A component declaration describes a unit of executable software, including the location of the component's binaries and the capabilities (such as services) that it intends to use, expose, or offer to other components.

  • This information is typically encoded as a component manifest file within a package. Example:

    // frobinator.cml
    {
        "uses": [{ "service": "fuchsia.log.LogSink" }],
        "exposes": [{ "service": "fuchsia.frobinator.Frobber" }],
        "offers": [{ "service": "fuchsia.log.LogSink",
                     "from": "realm", "to": [ "#child" ]}],
        "program": { "binary": ... }
        "children": { "child": ... }
    }
    

• A service instance is a capability that conforms to a given service declaration. On Fuchsia, it is represented as a directory. Other systems may use different service discovery mechanisms.

• A component instance is a particular instance of a component with its own private sandbox. At runtime, it uses service instances offered by other components through opening directories in its incoming namespace. Conversely, it exposes its own service instances to other components by presenting them in its outgoing directory. The component manager acts as a broker for service discovery.

  • A component instance is often (but not always) one-to-one with a process.
  • Component runners can often run multiple component instances within the same process each with its own incoming namespace.

Discussion

• [ianloic] what guidance should we offer for choosing protocol composition vs. service declarations?
• [abdulla] protocol composition indicates that the protocol themselves are highly related vs. service is indicating that a set of capabilities (possibly unrelated) are being jointly offered
• [pascallouis] compose multiplexes protocols over a single channel so has implications for message ordering vs. individual protocols of a service have different channels
• [jeffbrown] can delegate in different places, not related, composition doesn't get you this functionality, services allow "discovery" at runtime, e.g. listing which protocols are available

Q3: Is the proposed flat topology for service instances sufficiently expressive?

Response

• A flat topology is easy to use because there is no need to recursively traverse paths to locate all instances. This impacts both ease of use and performance.
• A flat topology can be just as expressive as a hierarchical topology when relevant information is encoded in the instance names, e.g., /svc/fuchsia.Ethernet/rack.5,port.9/packet_receiver.
• Services can be accessed from different locations using Open(), Open(namespace), and OpenAt(directory). In other words, not all services need to come from `/svc" in the process's global namespace. This allows for the creation of arbitrary service topologies, if necessary.

Q4: How should we extend services over time?

Response

• We can add new members to existing service declarations. Adding a new member doesn't break source or binary-compatibility because each member is effectively optional (attempting to connect to the protocol is an operation that can fail).
• We can remove existing members from service declarations. Removing (or renaming) an existing member may break source and binary compatibility and may require a careful migration plan to mitigate adverse impact.
• The service's documentation should provide clear expectations for how the service is intended to be used or implemented, particularly when such usage is not obvious, e.g., explain what features of the service are deprecated and slated for removal.

• Anticipated pattern for versioning: add new members to a service as protocols evolve. Protocol enumeration (listing directories) allows clients to discover what is supported. Example:

  • In version 1...

    service Fonts {
        FontProvider provider;
    };
    
    protocol FontProvider {
        GimmeDaFont(string font_name) -> (fuchsia.mem.Buffer ttf);
    };
    

  • In version 2, an incremental update...

    service Fonts {
        FontProvider provider;
        FontProvider2 provider2;
    };
    
    protocol FontProvider2 {
        compose FontProvider;
        GetDefaultFontByFamily(string family) -> (string family);
    };
    

  • In version 3, a complete redesign...

    service Fonts {
        [Deprecated]
        FontProvider provider;
        [Deprecated]
        FontProvider provider2;
        TypefaceChooser typeface_chooser;
    }
    
    protocol TypefaceChooser {
        GetTypeface(TypefaceCriteria criteria);
    };
    
    table TypefaceCriteria {
        1: Family family;
        2: Style style;
        3: int weight;
    };
    

Q5: If a component instance wishes to expose multiple services that relate to a single underlying logical resource, how is that expressed?

Response

• A component would define multiple services that are exposed through its component manifest. Example:

  // frobinator.cml
  {
      ...
      "exposes": [
          { "service": "fuchsia.frobinator.Fooer" },
          { "service": "fuchsia.frobinator.Barer" },
      ],
      ...
  }
  

• The component would then implement these services on top of the single underlying resource, but users of these services need not be aware of that fact.