Audio Signal Processing

Overview

The signal processing interface is available to be potentially used by audio composite drivers. This interface SignalProcessing is a FIDL protocol used by the Composite protocol to provide audio signal processing capabilities.

The SignalProcessing protocol is defined to control signal processing hardware and their topologies. We define processing elements (PEs) as a logical unit of audio data processing provided by an audio driver, and we define topologies as the arrangement of PEs in pipelines and controls associated with them.

The SignalProcessing protocol allows hardware vendors to implement drivers with stable application binary interfaces (ABIs), and allow system integrators to configure drivers to perform differently based on system or product requirements using these interfaces for run-time configurations.

The SignalProcessing protocol composes the Reader signal processing protocol. Signal processing methods that only retrieve information are part of the Reader protocol, the rest are part of the SignalProcessing protocol itself. This separation allows clients of this interface to compose the Reader signal processing protocol into their own protocol if they require providing a read only subset of functionality to their own clients.

The SignalProcessing protocol and associated definitions are part of the fuchsia.hardware.audio.signalprocessing FIDL library.

Topologies

Each driver can have its own topology. Drivers can abstract from applications the topologies exposed by other drivers as needed for a particular configuration or product. Note that it is possible although not required to expose topologies to applications, in particular to audio_core.

Notes:

  • Topologies are not meant to fully describe the audio pipeline state/format/configuration in and out of every PE. The intent is to describe what can be changed/rearranged by the client based on its knowledge, configuration (for instance from metadata) and specific business logic.
  • Topologies include one or more processing elements of type RingBuffer, if (and only if) ring buffers are supported by the driver.
  • Topologies include one or more processing elements of type DaiInterconnect, if (and only if) DAI interconnects are supported by the driver. *Topologies include one or more processing elements of type PackeetStream, if (and only if) packet streams are supported by the driver.
  • Topologies must include at least one of these element types.

Processing Elements

A PE (defined in the fuchsia.hardware.audio.signalprocessing FIDL library as Element) is expected to be hardware-provided functionality managed by a particular driver (but it could be emulated in software, as any other driver functionality). A pipeline is composed of one or more PEs and a topology is composed of one or more pipelines.

We refer to the server as the driver that is providing the signal processing protocol. We refer to the client as the user of the functionality, e.g. an application such as audio_core.

Basic operation

The client is responsible for requesting and then configuring any signal processing capabilities. Once the server provides its PEs by replying to a client's GetElements, the client may issue WatchElementState calls (see hanging get pattern) to retrieve PE state and SetElementState to dynamically control the PEs parameters as needed. For instance, to retrieve the gain of a PE of type GAIN, the client issues WatchElementState calls, one to retrieve the initial state (the driver will reply to the first WatchElementState sent by the client), and subsequent ones to get notified of updates to the ElementState that includes the gain. Similarly, to retrieve the state of a PE of type EQUALIZER, which is composed of multiple bands in its bands_state, a client would issue a WatchElementState that would retrieve the initial state (the driver will reply to the first WatchElementState sent by the client) including for instance frequency fields for each band.

Also after the server provides its PEs by replying to a client's GetElements, the client may request available topologies with the GetTopologies method. If more than one topology is returned by GetTopologies, then SetTopology can be used to pick the topology to use, and WatchTopology can be used to observe updates to the active topology.

GetElements

GetElements allows to optionally get a list of all PEs. For instance this method may be called by a client on a driver abstracting a hardware codec. Once the list of PEs is known to the client, the client may configure the PEs based on the parameters exposed by the PE types.

SetElementState

SetElementState allows a client to control the state of a PE using an id returned by GetElements. PEs of different types may have different state exposed to clients, the SetElementState parameter state has a different type depending on the type of PE.

WatchElementState

WatchElementState allows a client to monitor the state of a PE using an id returned by GetElements. PEs of different types may have different state exposed to clients, the WatchElementState parameter state has a different type depending on the type of PE.

The state of a PE is composed of values that may be changed directly by the client via a call to SetElementState, or indirectly for instance by calling SetElementState on a different PE, or independent of the client for instance due to a plug detect change.

GetTopologies

GetTopologies allows to optionally get a list of topologies. For instance this method may be called by a client on a driver abstracting a hardware codec. Once the list of topologies is known to the client, the client may configure the server to use a particular topology.

SetTopology

SetTopology allows a client to control which topology is used by the server. Only one topology can be selected at any time.

WatchTopology

WatchTopology allows a client to observe the currently active topology id via a hanging get, and to be notified asynchronously when the topology changes.

Processing elements types

The PEs returned by GetElements support a number of different types of signal processing defined by the PE types and parameters. PE types define standard signal processing (e.g. GAIN, DELAY, EQUALIZER, etc), vendor specific signal processing (VENDOR_SPECIFIC e.g. a type not defined in the SignalProcessing protocol), and CONNECTION_POINTs used to construct multi-pipeline topologies (allow for routing and mixing definitions, see Connection points below).

Each individual PE may have one or more inputs and one or more output channels. For routing and mixing, PEs may make the number of output channels different from the number of input channels.

Data in each channel (a.k.a. the signal that is processed) may be altered by the PE. For instance if there is a single PE of type AGL in a pipeline that includes a DAI_INTERCONNECT element with DaiFormat number_of_channels set to 2, then AGL (Automatic Gain Limiting) can be started or stopped for these 2 channels by a client calling SetElementState with state started set to true (or false if the AGL Element's can_stop was true).

If optional fields in the different PE types are not included, then the state of the processing element is not changed with respect to the particular field. For instance, if an EqualizerBandState in a SetElementState call does not include an optional frequency then the equalizer's band frequency state is not changed.

Vendor specific data

ElementState vendor_specific_data is an optional parameter that can be specified for any processing element. This allows processing elements to specify an opaque object to be either sent to the driver as part of a SetElementState or received from a driver as part of a WatchElementState.

In addition to opaque data for any type, a processing element of type VENDOR_SPECIFIC allows drivers to specify a type that is not defined in the SignalProcessing protocol, for instance something that is not standard yet or is not meant to be standardized and provided only by a specific vendor. A processing element of type VENDOR_SPECIFIC specifies a vendor_specific variant in TypeSpecificElement and TypeSpecificElementState. In addition, it may specify opaque data to be sent to or received from the driver using the ElementState vendor_specific_data parameter, just like any other processing element type.

Topologies

The topologies returned by GetTopologies support different arrangements for the PEs returned by GetElements. GetTopologies may advertise one or multiple topologies.

One topology

If one topology is advertised, i.e. GetTopologies returns a vector with one element, then all PEs are part of this explicit single pipeline. Ordering in this case is explicit. For instance, if GetElements returns 2 PEs:

  1. Element: id = 1, type = AUTOMATIC_GAIN_LIMITER (AGL)
  2. Element: id = 2, type = EQUALIZER (EQ)

The one Topology element returned by GetTopologies will list an id and a processing_elements_edge_pairs vector explicitly advertising the order in which signal processing is performed, in this example:

  1. Topology: id = 1, processing_elements_edge_pairs = vector with one element with processing_element_id_from = 1 and processing_element_id_to = 2.

This advertises this one topology with one pipeline:

                +-------+    +-------+
Input signal -> |  AGL  | -> |  EQ   | -> Output signal
                +-------+    +-------+

In this topology the beginning (where the input signal is input into the pipeline) and the end of the pipeline (where the output signal is output from the pipeline) are implicit.

If only one topology is advertised, then the contents are informational only since the client can't change the use of one and only topology.

Multiple topologies

If multiple topologies are advertised, i.e. GetTopologies returns a vector with multiple elements, then PEs may be used in multiple configurations, i.e. topologies. Each topology explicitly lists a number of PEs and their ordering, i.e. ordering in this case is explicit. The arrangement and ordering of PEs define a pipeline.

By listing only the specific arrangements and ordering of PEs supported, servers restrict what combinations of pipelines are valid.

For instance, if GetElements returns 6 PEs:

  1. Element: id = 1, type = AUTOMATIC_GAIN_LIMITER (AGL)
  2. Element: id = 2, type = EQUALIZER (EQ)
  3. Element: id = 3, type = SAMPLE_RATE_CONVERSION (SRC)
  4. Element: id = 4, type = GAIN
  5. Element: id = 5, type = DYNAMICS (DRC1)
  6. Element: id = 6, type = DYNAMICS (DRC2) parameters different from DRC1 parameters.

The Topology elements returned by GetTopologies will list an id and a processing_elements_edge_pairs for each topology, in this example:

  1. Topology: id = 1, processing_elements_edge_pairs =
    • processing_element_id_from = 3 and processing_element_id_to = 2.
    • processing_element_id_from = 2 and processing_element_id_to = 4.
    • processing_element_id_from = 4 and processing_element_id_to = 5.
    • processing_element_id_from = 5 and processing_element_id_to = 1.
  2. Topology: id = 2, processing_elements_edge_pairs =
    • processing_element_id_from = 2 and processing_element_id_to = 4.
    • processing_element_id_from = 4 and processing_element_id_to = 6.

This advertises two topologies with one pipeline each:

                +-------+    +-------+    +-------+    +-------+    +-------+
Input signal -> |  SRC  | -> |  EQ   | -> | GAIN  | -> |  DRC1 | -> |  AGL  | -> Output signal
                +-------+    +-------+    +-------+    +-------+    +-------+

                +-------+    +-------+    +-------+
Input signal -> |  EQ   | -> | GAIN  | -> |  DRC2 | -> Output signal
                +-------+    +-------+    +-------+

Connection points

The PEs of type CONNECTION_POINT allow for:

  1. Mixing multiple channels within a single pipeline.
  2. Mixing multiple channels from different pipelines.
  3. Repeating channels.
  4. Expanding a single pipeline into multiple pipelines ones (scatter).