fuchsia.media

A protocol for monitoring the usage activity of the AudioRenderers and AudioCapturers.

WatchCaptureActivity

Notifies the client whenever there is a change in the set of active AudioCaptureUsages. It returns immediately the first time that it is called.

Removed: 31 Deprecated: 26

Request

<EMPTY>

Response

Name	Type
`active_usages`	`vector<AudioCaptureUsage>:4`

WatchCaptureActivity2

Notifies the client whenever there is a change in the set of active AudioCaptureUsages. It returns immediately the first time that it is called.

Request

<EMPTY>

Response

Name	Type
`payload`	`ActivityReporter_WatchCaptureActivity2_Result`

WatchRenderActivity

Notifies the client whenever there is a change in the set of active AudioRenderUsages. It returns immediately the first time that it is called.

Removed: 31 Deprecated: 26

Request

<EMPTY>

Response

Name	Type
`active_usages`	`vector<AudioRenderUsage>:5`

WatchRenderActivity2

Notifies the client whenever there is a change in the set of active AudioRenderUsages. It returns immediately the first time that it is called.

Request

<EMPTY>

Response

Name	Type
`payload`	`ActivityReporter_WatchRenderActivity2_Result`

Audio

Defined in fuchsia.media/audio.fidl

CreateAudioCapturer

Creates an AudioCapturer which either captures from the current default audio input device, or loops-back from the current default audio output device based on value passed for the loopback flag.

Request

Name	Type
`audio_capturer_request`	`server_end:AudioCapturer`
`loopback`	`bool`

CreateAudioRenderer

Request

Name	Type
`audio_renderer_request`	`server_end:AudioRenderer`

AudioCapturer

AudioCapturer

An AudioCapturer is an interface returned from an fuchsia.media.Audio's CreateAudioCapturer method, which may be used by clients to capture audio from either the current default audio input device, or the current default audio output device depending on the flags passed during creation.

**Format support**

See (Get|Set)StreamType below. By default, the captured stream type will be initially determined by the currently configured stream type of the source that the AudioCapturer was bound to at creation time. Users may either fetch this type using GetStreamType, or they may choose to have the media resampled or converted to a type of their choosing by calling SetStreamType. Note: the stream type may only be set while the system is not running, meaning that there are no pending capture regions (specified using CaptureAt) and that the system is not currently running in 'async' capture mode.

**Buffers and memory management**

Audio data is captured into a shared memory buffer (a VMO) supplied by the user to the AudioCapturer during the AddPayloadBuffer call. Please note the following requirements related to the management of the payload buffer.

The payload buffer must be supplied before any capture operation may start. Any attempt to start capture (via either CaptureAt or StartAsyncCapture) before a payload buffer has been established is an error.
The payload buffer may not be changed while there are any capture operations pending.
The stream type may not be changed after the payload buffer has been set.
The payload buffer must be an integral number of audio frame sizes (in bytes)
When running in 'async' mode (see below), the payload buffer must be at least as large as twice the frames_per_packet size specified during StartAsyncCapture.
The handle to the payload buffer supplied by the user must be readable, writable, mappable and transferable.
Users should always treat the payload buffer as read-only.

**Synchronous vs. Asynchronous capture mode**

The AudioCapturer interface can be used in one of two mutually exclusive modes: Synchronous and Asynchronous. A description of each mode and their tradeoffs is given below.

**Synchronous mode**

By default, AudioCapturer instances are running in 'sync' mode. They will only capture data when a user supplies at least one region to capture into using the CaptureAt method. Regions supplied in this way will be filled in the order that they are received and returned to the client as StreamPackets via the return value of the CaptureAt method. If an AudioCapturer instance has data to capture, but no place to put it (because there are no more pending regions to fill), the next payload generated will indicate that their has been an overflow by setting the Discontinuity flag on the next produced StreamPacket. Synchronous mode may not be used in conjunction with Asynchronous mode. It is an error to attempt to call StartAsyncCapture while the system still regions supplied by CaptureAt waiting to be filled.

If a user has supplied regions to be filled by the AudioCapturer instance in the past, but wishes to reclaim those regions, they may do so using the DiscardAllPackets method. Calling the DiscardAllPackets method will cause all pending regions to be returned, but with NO_TIMESTAMP as their StreamPacket's PTS. See "Timing and Overflows", below, for a discussion of timestamps and discontinuity flags. After a DiscardAllPackets operation, an OnEndOfStream event will be produced. While an AudioCapturer will never overwrite any region of the payload buffer after a completed region is returned, it may overwrite the unfilled portions of a partially filled buffer which has been returned as a result of a DiscardAllPackets operation.

Asynchronous mode

While running in 'async' mode, clients do not need to explicitly supply shared buffer regions to be filled by the AudioCapturer instance. Instead, a client enters into 'async' mode by calling StartAsyncCapture and supplying a callback interface and the number of frames to capture per-callback. Once running in async mode, the AudioCapturer instance will identify which payload buffer regions to capture into, capture the specified number of frames, then deliver those frames as StreamPackets using the OnPacketCapture FIDL event. Users may stop capturing and return the AudioCapturer instance to 'sync' mode using the StopAsyncCapture method.

It is considered an error to attempt any of the following operations.

To attempt to enter 'async' capture mode when no payload buffer has been established.
To specify a number of frames to capture per payload which does not permit at least two contiguous capture payloads to exist in the established shared payload buffer simultaneously.
To send a region to capture into using the CaptureAt method while the AudioCapturer instance is running in 'async' mode.
To attempt to call DiscardAllPackets while the AudioCapturer instance is running in 'async' mode.
To attempt to re-start 'async' mode capturing without having first stopped.
To attempt any operation except for SetGain while in the process of stopping.

Synchronizing with a StopAsyncCapture operation

Stopping asynchronous capture mode and returning to synchronous capture mode is an operation which takes time. Aside from SetGain, users may not call any other methods on the AudioCapturer interface after calling StopAsyncCapture (including calling StopAsyncCapture again) until after the stop operation has completed. Because of this, it is important for users to be able to synchronize with the stop operation. Two mechanisms are provided for doing so.

The first is to use StopAsyncCapture (not the NoReply variant). When the user's callback has been called, they can be certain that stop operation is complete and that the AudioCapturer instance has returned to synchronous operation mode.

The second way to determine that a stop operation has completed is to use the flags on the packets which get delivered via the user-supplied AudioCapturerCallback interface after calling StopAsyncCapture. When asked to stop, any partially filled packet will be returned to the user, and the final packet returned will always have the end-of-stream flag (kFlagsEos) set on it to indicate that this is the final frame in the sequence. If there is no partially filled packet to return, the AudioCapturer will synthesize an empty packet with no timestamp, and offset/length set to zero, in order to deliver a packet with the end-of-stream flag set on it. Once users have seen the end-of-stream flag after calling stop, the AudioCapturer has finished the stop operation and returned to synchronous operating mode.

**Timing and Overflows**

All media packets produced by an AudioCapturer instance will have their PTS field filled out with the capture time of the audio expressed as a timestamp given by the reference clock timeline. Note: this timestamp is actually a capture timestamp, not a presentation timestamp (it is more of a CTS than a PTS) and is meant to represent the underlying system's best estimate of the capture time of the first frame of audio, including all outboard and hardware introduced buffering delay. As a result, all timestamps produced by an AudioCapturer should be expected to be in the past relative to 'now' on the stream's reference clock timeline.

The one exception to the "everything has an explicit timestamp" rule is when discarding submitted regions while operating in synchronous mode. Discarded packets have no data in them, but FIDL demands that all pending method-return-value callbacks be executed. Because of this, the regions will be returned to the user, but their timestamps will be set to NO_TIMESTAMP, and their payload sizes will be set to zero. Any partially filled payload will have a valid timestamp, but a payload size smaller than originally requested. The final discarded payload (if there were any to discard) will be followed by an OnEndOfStream event.

Two StreamPackets delivered by an AudioCapturer instance are 'continuous' if the first frame of audio contained in the second packet was captured exactly one nominal frame time after the final frame of audio in the first packet. If this relationship does not hold, the second StreamPacket will have the STREAM_PACKET_FLAG_DISCONTINUITY bit set in its flags field.

Even though explicit timestamps are provided on every StreamPacket produced, users who have very precise timing requirements are encouraged to always reason about time by counting frames delivered since the last discontinuity, rather than simply using the raw capture timestamps. This is because the explicit timestamps written on continuous packets may have a small amount of rounding error based on whether or not the units of the capture timeline reference clock are divisible by the chosen audio frame rate.

Users should always expect the first StreamPacket produced by an AudioCapturer to have the discontinuous flag set on it (as there is no previous packet to be continuous with). Similarly, the first StreamPacket after a DiscardAllPackets or a Stop/Start cycle will always be discontinuous. After that, there are only two reasons that a StreamPacket will ever be discontinuous:

The user is operating in synchronous mode and does not supply regions to be filled quickly enough. If the next continuous frame of data has not been captured by the time it needs to be purged from the source buffers, an overflow has occurred and the AudioCapturer will flag the next captured region as discontinuous.
The user is operating in asynchronous mode and some internal error prevents the AudioCapturer instance from capturing the next frame of audio in a continuous fashion. This might be high system load or a hardware error, but in general it is something which should never normally happen. In practice, however, if it does, the next produced packet will be flagged as being discontinuous.

**Synchronous vs. Asynchronous Trade-offs**

The choice of operating in synchronous vs. asynchronous mode is up to the user, and depending on the user's requirements, there are some advantages and disadvantages to each choice.

Synchronous mode requires only a single Zircon channel under the hood and can achieve some small savings because of this. In addition, the user has complete control over the buffer management. Users specify exactly where audio will be captured to and in what order. Because of this, if users do not need to always be capturing, it is simple to stop and restart the capture later (just by ceasing to supply packets, then resuming later on). Payloads do not need to be uniform in size either, clients may specify payloads of whatever granularity is appropriate.

The primary downside of operating in synchronous mode is that two messages will need to be sent for every packet to be captured. One to inform the AudioCapturer of the instance to capture into, and one to inform the user that the packet has been captured. This may end up increasing overhead and potentially complicating client designs.

Asynchronous mode has the advantage requiring only 1/2 of the messages, however, when operating in 'async' mode, AudioCapturer instances have no way of knowing if a user is processing the StreamPackets being sent in a timely fashion, and no way of automatically detecting an overflow condition. Users of 'async' mode should be careful to use a buffer large enough to ensure that they will be able to process their data before an AudioCapturer will be forced to overwrite it.

AddPayloadBuffer

Adds a payload buffer to the current buffer set associated with the connection. A StreamPacket struct reference a payload buffer in the current set by ID using the StreamPacket.payload_buffer_id field.

A buffer with ID id must not be in the current set when this method is invoked, otherwise the service will close the connection.

Request

Name	Type
`id`	`uint32`
`payload_buffer`	`handle<vmo>`

BindGainControl

Binds to the gain control for this AudioCapturer.

Request

Name	Type
`gain_control_request`	`server_end:fuchsia.media.audio/GainControl`

CaptureAt

Explicitly specifies a region of the shared payload buffer for the audio input to capture into.

Request

Name	Type
`payload_buffer_id`	`uint32`
`payload_offset`	`uint32`
`frames`	`uint32`

Response

Name	Type
`captured_packet`	`StreamPacket`

DiscardAllPackets

Request

<EMPTY>

Response

<EMPTY>

DiscardAllPacketsNoReply

Request

<EMPTY>

GetReferenceClock

Retrieves the stream's reference clock. The returned handle will have READ, DUPLICATE and TRANSFER rights, and will refer to a zx::clock that is MONOTONIC and CONTINUOUS.

Request

<EMPTY>

Response

Name	Type
`reference_clock`	`handle<clock>`

GetStreamType

Gets the currently configured stream type. Note: for an AudioCapturer which was just created and has not yet had its stream type explicitly set, this will retrieve the stream type -- at the time the AudioCapturer was created -- of the source (input or looped-back output) to which the AudioCapturer is bound. Even if this matches the client's desired format, SetPcmStreamType must still be called.

Request

<EMPTY>

Response

Name	Type
`stream_type`	`StreamType`

OnEndOfStream

Indicates that the stream has ended.

Response

<EMPTY>

OnPacketProduced

Delivers a packet produced by the service. When the client is done with the payload memory, the client must call ReleasePacket to release the payload memory.

Response

Name	Type
`packet`	`StreamPacket`

ReleasePacket

Releases payload memory associated with a packet previously delivered via OnPacketProduced.

Request

Name	Type
`packet`	`StreamPacket`

RemovePayloadBuffer

Removes a payload buffer from the current buffer set associated with the connection.

A buffer with ID id must exist in the current set when this method is invoked, otherwise the service will will close the connection.

Request

Name	Type
`id`	`uint32`

SetPcmStreamType

Sets the stream type of the stream to be delivered. Causes the source material to be reformatted/resampled if needed in order to produce the requested stream type. Must be called before the payload buffer is established.

Request

Name	Type
`stream_type`	`AudioStreamType`

SetReferenceClock

Sets the reference clock that controls this capturer's playback rate. If the input parameter is a valid zx::clock, it must have READ, DUPLICATE, TRANSFER rights and refer to a clock that is both MONOTONIC and CONTINUOUS. If instead an invalid clock is passed (such as the uninitialized zx::clock()), this indicates that the stream will use a 'flexible' clock generated by AudioCore that tracks the audio device.

SetReferenceClock cannot be called after the capturer payload buffer has been added. It also cannot be called a second time (even before capture). If the client wants a reference clock that is initially CLOCK_MONOTONIC but may diverge at some later time, they should create a clone of the monotonic clock, set this as the stream's reference clock, then rate-adjust it subsequently as needed.

Request

Name	Type
`reference_clock`	`handle<clock>?`

SetUsage

Sets the usage of the capture stream. This may be changed on the fly, but packets in flight may be affected. By default, Capturers are created with the FOREGROUND usage.

DEPRECATED - use SetUsage2 and AudioCaptureUsage2

Request

Name	Type
`usage`	`AudioCaptureUsage`

SetUsage2

Sets the usage of the capture stream. This may be changed on the fly, but this may affect packets in flight. By default, Capturers are created with the FOREGROUND usage.

Request

Name	Type
`usage`	`AudioCaptureUsage2`

StartAsyncCapture

Places the AudioCapturer into 'async' capture mode and begin to produce packets of exactly 'frames_per_packet' number of frames each. The OnPacketProduced event (of StreamSink) will be used to inform the client of produced packets.

Request

Name	Type
`frames_per_packet`	`uint32`

StopAsyncCapture

Stops capturing in 'async' capture mode and (optionally) deliver a callback that may be used by the client if explicit synchronization is needed.

Request

<EMPTY>

Response

<EMPTY>

StopAsyncCaptureNoReply

Request

<EMPTY>

AudioConsumer

Interface for playing and controlling audio.

BindVolumeControl

Binds to this AudioConsumer volume control for control and notifications.

Request

Name	Type
`volume_control_request`	`server_end:fuchsia.media.audio/VolumeControl`

CreateStreamSink

Creates a StreamSink for the consumer with the indicated properties.

Multiple stream sinks may be acquired using this method, but they are intended to be used sequentially rather than concurrently. The first stream sink that's created using this method is used as the sole source of packets incoming to the logical consumer until that stream sink is closed or the EndOfStream method is called on that sink. At that point, the second stream sink is used, and so on.

If an unsupported compression type is supplied, the stream_sink_request request will be closed with an epitaph value of ZX_ERR_INVALID_ARGS.

Request

Name	Type
`buffers`	`vector<handle<vmo>>:16`
`stream_type`	`AudioStreamType`
`compression`	`Compression?`
`stream_sink_request`	`server_end:StreamSink`

OnEndOfStream

Indicates that the last packet prior to the end of the stream has been rendered.

Response

<EMPTY>

SetRate

Requests to change the playback rate of the renderer. 1.0 means normal playback. Negative rates are not supported. The new rate will be reflected in the updated status. The default rate of any newly created StreamSink is 1.0.

Request

Name	Type
`rate`	`float32`

Start

Starts rendering as indicated by flags.

media_time indicates the packet timestamp that corresponds to reference_time. Typically, this is the timestamp of the first packet that will be rendered. If packets will be supplied with no timestamps, this value should be NO_TIMESTAMP. Passing a media_time value of NO_TIMESTAMP chooses the default media time, established as follows: 1. When starting for the first time, the default media time is the timestamp on the first packet sent to the stream sink. 2. When resuming after stop, the default media time is the media time at which the stream stopped.

reference_time is the monotonic system time at which rendering should be started. For supply-driven sources, this must be the time at which the first packet was (or will be) sent plus a lead time, which must be in the range indicated in the AudioConsumerStatus. For demand-driven sources, the client must ensure that the lead time requirement is met at the start time. Passing the default value of 0 for reference_time causes the consumer to choose a start time based on the availability of packets, the lead time requirements, and whether LOW_LATENCY has been specified.

The actual start time will be reflected in the updated status.

Request

Name	Type
`flags`	`AudioConsumerStartFlags`
`reference_time`	`zx/Time`
`media_time`	`int64`

Stop

Stops rendering as soon as possible after this method is called. The actual stop time will be reflected in the updated status.

Request

<EMPTY>

WatchStatus

Gets the current status of the consumer using the long get pattern. The consumer responds to this method when the status changes - initially with respect to the initial status value and thereafter with respect to the previously-reported status value.

Request

<EMPTY>

Response

Name	Type
`status`	`AudioConsumerStatus`

AudioCore

Defined in fuchsia.media/audio_core.fidl

BindUsageVolumeControl

Binds to a volume control protocol for the given usage.

DEPRECATED - Use BindUsageVolumeControl2 and AudioRenderUsage2

Request

Name	Type
`usage`	`Usage`
`volume_control`	`server_end:fuchsia.media.audio/VolumeControl`

BindUsageVolumeControl2

Binds to a volume control protocol for the given usage.

Request

Name	Type
`usage`	`Usage2`
`volume_control`	`server_end:fuchsia.media.audio/VolumeControl`

CreateAudioCapturer

Creates an AudioCapturer which either captures from the current default audio input device, or loops-back from the current default audio output device based on value passed for the loopback flag.

Request

Name	Type
`loopback`	`bool`
`audio_in_request`	`server_end:AudioCapturer`

CreateAudioCapturerWithConfiguration

Creates an AudioCapturer according to the given requirements.

pcm_stream_type sets the stream type of the stream to be delivered. It causes the source material to be reformatted/resampled if needed in order to produce the requested stream type.

usage is used by Fuchsia to make decisions about user experience. See AudioCaptureUsage for more details.

configuration must be initialized to a variant, or no capturer can be created.

Request

Name	Type
`stream_type`	`AudioStreamType`
`configuration`	`AudioCapturerConfiguration`
`audio_capturer_request`	`server_end:AudioCapturer`

CreateAudioRenderer

Creates an AudioRenderer which outputs audio to the default device.

Request

Name	Type
`audio_out_request`	`server_end:AudioRenderer`

GetDbFromVolume

Queries the decibel value that maps to a volume percentage [0, 1] for a particular usage. This is the same mapping as used by the VolumeControl from BindUsageVolumeControl.

DEPRECATED - Use GetDbFromVolume2 and AudioRenderUsage2

Request

Name	Type
`usage`	`Usage`
`volume`	`float32`

Response

Name	Type
`gain_db`	`float32`

GetDbFromVolume2

Queries the decibel value that maps to a volume percentage [0, 1] for a particular usage. This is the same mapping as used by the VolumeControl from BindUsageVolumeControl.

Request

Name	Type
`usage`	`Usage2`
`volume`	`float32`

Response

Name	Type
`payload`	`AudioCore_GetDbFromVolume2_Result`

GetVolumeFromDb

Queries the volume percentage [0, 1] that maps to a gain_db value for a particular usage. This is the same mapping as used by the VolumeControl from BindUsageVolumeControl.

DEPRECATED - Use GetVolumeFromDb2 and AudioRenderUsage2

Request

Name	Type
`usage`	`Usage`
`gain_db`	`float32`

Response

Name	Type
`volume`	`float32`

GetVolumeFromDb2

Queries the volume percentage [0, 1] that maps to a gain_db value for a particular usage. This is the same mapping as used by the VolumeControl from BindUsageVolumeControl.

Request

Name	Type
`usage`	`Usage2`
`gain_db`	`float32`

Response

Name	Type
`payload`	`AudioCore_GetVolumeFromDb2_Result`

LoadDefaults

Re-loads the platform policy configuration. Falls back to a default config if the platform does not provide a config.

Request

<EMPTY>

ResetInteractions

Re-initializes the set of rules that are currently governing the interaction of streams in audio_core. The default behavior is 'NONE'.

Request

<EMPTY>

SetCaptureUsageGain

Sets gain for this capture usage. By default, all capture usages are set to Unity (0 db).

DEPRECATED - Use SetCaptureUsageGain2 and AudioCaptureUsage2

Request

Name	Type
`usage`	`AudioCaptureUsage`
`gain_db`	`float32`

SetCaptureUsageGain2

Sets gain for this capture usage. By default, all capture usages are set to Unity (0 db).

Request

Name	Type
`usage`	`AudioCaptureUsage2`
`gain_db`	`float32`

SetInteraction

Sets how audio_core handles interactions of multiple active streams simultaneously. If streams of Usage active are processing audio, and streams of Usage affected are as well, the Behavior specified will be applied to the streams of Usage affected.

DEPRECATED - Use SetInteraction2 and AudioRenderUsage2

Request

Name	Type
`active`	`Usage`
`affected`	`Usage`
`behavior`	`Behavior`

SetInteraction2

Request

Name	Type
`active`	`Usage2`
`affected`	`Usage2`
`behavior`	`Behavior`

SetRenderUsageGain

Sets the gain for this render usage. By default, all render usages are set to Unity (0 db).

DEPRECATED - Use SetRenderUsageGain2 and AudioRenderUsage2

Request

Name	Type
`usage`	`AudioRenderUsage`
`gain_db`	`float32`

SetRenderUsageGain2

Sets the gain for this render usage. By default, all render usages are set to Unity (0 db).

Request

Name	Type
`usage`	`AudioRenderUsage2`
`gain_db`	`float32`

AudioDeviceEnumerator

Defined in fuchsia.media/audio_device_enumerator.fidl

AddDeviceByChannel

Deprecation

StreamConfig is not supported anymore, instead use an Audio Composite , see Audio Drivers Architecture

DEPRECATED

Request

Name	Type
`device_name`	`string:256`
`is_input`	`bool`
`channel`	`client_end:fuchsia.hardware.audio/StreamConfig`

GetDeviceGain

Gain/Mute/AGC control

Note that each of these operations requires a device_token in order to target the proper input/output.

The Get command returns the device_token of the device whose gain is being reported, or ZX_KOID_INVALID in the case that the requested device_token was invalid or the device had been removed from the system before the Get command could be processed.

Set commands which are given an invalid device token are ignored and have no effect on the system. In addition, users do not need to control all of the gain settings for an audio device with each call. Only the settings with a corresponding flag set in the set_flags parameter will be affected. For example, passing SetAudioGainFlag_MuteValid will cause a SetDeviceGain call to care only about the mute setting in the gain_info structure, while passing (SetAudioGainFlag_GainValid | SetAudioGainFlag_MuteValid) will cause both the mute and the gain status to be changed simultaneously.

Request

Name	Type
`device_token`	`uint64`

Response

Name	Type
`device_token`	`uint64`
`gain_info`	`AudioGainInfo`

GetDevices

Obtain the list of currently active audio devices.

Request

<EMPTY>

Response

Name	Type
`devices`	`vector<AudioDeviceInfo>`

OnDefaultDeviceChanged

DEPRECATED

Response

Name	Type
`old_default_token`	`uint64`
`new_default_token`	`uint64`

OnDeviceAdded

Events sent when devices are added or removed, or when properties of a device change.

Response

Name	Type
`device`	`AudioDeviceInfo`

OnDeviceGainChanged

Response

Name	Type
`device_token`	`uint64`
`gain_info`	`AudioGainInfo`

OnDeviceRemoved

Response

Name	Type
`device_token`	`uint64`

SetDeviceGain

Request

Name	Type
`device_token`	`uint64`
`gain_info`	`AudioGainInfo`
`valid_flags`	`AudioGainValidFlags`

AudioRenderer

Defined in fuchsia.media/audio_renderer.fidl

AudioRenderers can be in one of two states at any time: configurable or operational. A renderer is considered operational whenever it has packets queued to be rendered; otherwise it is configurable. Once an AudioRenderer enters the operational state, calls to "configuring" methods are disallowed and will cause the audio service to disconnect the client's connection. The following are considered configuring methods: AddPayloadBuffer, SetPcmStreamType, SetStreamType, SetPtsUnits, SetPtsContinuityThreshold.

If an AudioRenderer must be reconfigured, the client must ensure that no packets are still enqueued when these "configuring" methods are called. Thus it is best practice to call DiscardAllPackets on the AudioRenderer (and ideally Stop before DiscardAllPackets), prior to reconfiguring the renderer.

AddPayloadBuffer

A buffer with ID id must not be in the current set when this method is invoked, otherwise the service will close the connection.

Request

Name	Type
`id`	`uint32`
`payload_buffer`	`handle<vmo>`

BindGainControl

Binds to the gain control for this AudioRenderer.

Request

Name	Type
`gain_control_request`	`server_end:fuchsia.media.audio/GainControl`

DiscardAllPackets

Discards packets previously sent via SendPacket or SendPacketNoReply and not yet released. The response is sent after all packets have been released.

Request

<EMPTY>

Response

<EMPTY>

DiscardAllPacketsNoReply

Discards packets previously sent via SendPacket or SendPacketNoReply and not yet released.

Request

<EMPTY>

EnableMinLeadTimeEvents

Enables or disables notifications about changes to the minimum clock lead time (in nanoseconds) for this AudioRenderer. Calling this method with 'enabled' set to true will trigger an immediate OnMinLeadTimeChanged event with the current minimum lead time for the AudioRenderer. If the value changes, an OnMinLeadTimeChanged event will be raised with the new value. This behavior will continue until the user calls EnableMinLeadTimeEvents(false).

The minimum clock lead time is the amount of time ahead of the reference clock's understanding of "now" that packets needs to arrive (relative to the playback clock transformation) in order for the mixer to be able to mix packet. For example...

Let the PTS of packet X be P(X)
Let the function which transforms PTS -> RefClock be R(p) (this function is determined by the call to Play(...)
Let the minimum lead time be MLT

If R(P(X)) < RefClock.Now() + MLT Then the packet is late, and some (or all) of the packet's payload will need to be skipped in order to present the packet at the scheduled time.

The value min_lead_time_nsec = 0 is a special value which indicates that the AudioRenderer is not yet routed to an output device. If Play is called before the AudioRenderer is routed, any played packets will be dropped. Clients should wait until min_lead_time_nsec > 0 before calling Play.

Request

Name	Type
`enabled`	`bool`

EndOfStream

Indicates the stream has ended. The precise semantics of this method are determined by the inheriting interface.

Request

<EMPTY>

GetMinLeadTime

While it is possible to call GetMinLeadTime before SetPcmStreamType, there's little reason to do so. This is because lead time is a function of format/rate, so lead time will be recalculated after SetPcmStreamType. If min lead time events are enabled before SetPcmStreamType (with EnableMinLeadTimeEvents(true)), then an event will be generated in response to SetPcmStreamType.

Request

<EMPTY>

Response

Name	Type
`min_lead_time_nsec`	`int64`

GetReferenceClock

Retrieves the stream's reference clock. The returned handle will have READ, DUPLICATE and TRANSFER rights, and will refer to a zx::clock that is MONOTONIC and CONTINUOUS.

Request

<EMPTY>

Response

Name	Type
`reference_clock`	`handle<clock>`

OnMinLeadTimeChanged

Response

Name	Type
`min_lead_time_nsec`	`int64`

Pause

Immediately puts the AudioRenderer into the paused state and then report the relationship between the media and reference timelines which was established (if requested).

If the AudioRenderer is already in the paused state when this called, the previously-established timeline values are returned (if requested).

Request

<EMPTY>

Response

Name	Type
`reference_time`	`int64`
`media_time`	`int64`

PauseNoReply

Request

<EMPTY>

Play

Immediately puts the AudioRenderer into a playing state. Starts the advance of the media timeline, using specific values provided by the caller (or default values if not specified). In an optional callback, returns the timestamp values ultimately used -- these set the ongoing relationship between the media and reference timelines (i.e., how to translate between the domain of presentation timestamps, and the realm of local system time).

Local system time is specified in units of nanoseconds; media_time is specified in the units defined by the user in the SetPtsUnits function, or nanoseconds if SetPtsUnits is not called.

The act of placing an AudioRenderer into the playback state establishes a relationship between 1) the user-defined media (or presentation) timeline for this particular AudioRenderer, and 2) the real-world system reference timeline. To communicate how to translate between timelines, the Play() callback provides an equivalent timestamp in each time domain. The first value ('reference_time') is given in terms of this renderer's reference clock; the second value ('media_time') is what media instant exactly corresponds to that local time. Restated, the frame at 'media_time' in the audio stream should be presented at 'reference_time' according to the reference clock.

Note: on calling this API, media_time immediately starts advancing. It is possible (if uncommon) for a caller to specify a system time that is far in the past, or far into the future. This, along with the specified media time, is simply used to determine what media time corresponds to 'now', and THAT media time is then intersected with presentation timestamps of packets already submitted, to determine which media frames should be presented next.

With the corresponding reference_time and media_time values, a user can translate arbitrary time values from one timeline into the other. After calling SetPtsUnits(pts_per_sec_numerator, pts_per_sec_denominator) and given the 'ref_start' and 'media_start' values from Play, then for any 'ref_time':

media_time = ( (ref_time - ref_start) / 1e9 * (pts_per_sec_numerator / pts_per_sec_denominator) ) + media_start

Conversely, for any presentation timestamp 'media_time':

ref_time = ( (media_time - media_start) * (pts_per_sec_denominator / pts_per_sec_numerator) * 1e9 ) + ref_start

Users, depending on their use case, may optionally choose not to specify one or both of these timestamps. A timestamp may be omitted by supplying the special value 'NO_TIMESTAMP'. The AudioRenderer automatically deduces any omitted timestamp value using the following rules:

Reference Time If 'reference_time' is omitted, the AudioRenderer will select a "safe" reference time to begin presentation, based on the minimum lead times for the output devices that are currently bound to this AudioRenderer. For example, if an AudioRenderer is bound to an internal audio output requiring at least 3 mSec of lead time, and an HDMI output requiring at least 75 mSec of lead time, the AudioRenderer might (if 'reference_time' is omitted) select a reference time 80 mSec from now.

Media Time If media_time is omitted, the AudioRenderer will select one of two values.

If the AudioRenderer is resuming from the paused state, and packets have not been discarded since being paused, then the AudioRenderer will use a media_time corresponding to the instant at which the presentation became paused.
If the AudioRenderer is being placed into a playing state for the first time following startup or a 'discard packets' operation, the initial media_time will be set to the PTS of the first payload in the pending packet queue. If the pending queue is empty, initial media_time will be set to zero.

Return Value When requested, the AudioRenderer will return the 'reference_time' and 'media_time' which were selected and used (whether they were explicitly specified or not) in the return value of the play call.

Examples

A user has queued some audio using SendPacket and simply wishes them to start playing as soon as possible. The user may call Play without providing explicit timestamps -- Play(NO_TIMESTAMP, NO_TIMESTAMP).
A user has queued some audio using SendPacket, and wishes to start playback at a specified 'reference_time', in sync with some other media stream, either initially or after discarding packets. The user would call Play(reference_time, NO_TIMESTAMP).
A user has queued some audio using SendPacket. The first of these packets has a PTS of zero, and the user wishes playback to begin as soon as possible, but wishes to skip all of the audio content between PTS 0 and PTS 'media_time'. The user would call Play(NO_TIMESTAMP, media_time).
A user has queued some audio using SendPacket and want to present this media in synch with another player in a different device. The coordinator of the group of distributed players sends an explicit message to each player telling them to begin presentation of audio at PTS 'media_time', at the time (based on the group's shared reference clock) 'reference_time'. Here the user would call Play(reference_time, media_time).

Request

Name	Type
`reference_time`	`int64`
`media_time`	`int64`

Response

Name	Type
`reference_time`	`int64`
`media_time`	`int64`

PlayNoReply

Request

Name	Type
`reference_time`	`int64`
`media_time`	`int64`

RemovePayloadBuffer

Removes a payload buffer from the current buffer set associated with the connection.

A buffer with ID id must exist in the current set when this method is invoked, otherwise the service will will close the connection.

Request

Name	Type
`id`	`uint32`

SendPacket

Sends a packet to the service. The response is sent when the service is done with the associated payload memory.

packet must be valid for the current buffer set, otherwise the service will close the connection.

Request

Name	Type
`packet`	`StreamPacket`

Response

<EMPTY>

SendPacketNoReply

Sends a packet to the service. This interface doesn't define how the client knows when the sink is done with the associated payload memory. The inheriting interface must define that.

packet must be valid for the current buffer set, otherwise the service will close the connection.

Request

Name	Type
`packet`	`StreamPacket`

SetPcmStreamType

Sets the type of the stream to be delivered by the client. Using this method implies that the stream encoding is AUDIO_ENCODING_LPCM.

This must be called before Play or PlayNoReply. After a call to SetPcmStreamType, the client must then send an AddPayloadBuffer request, then the various StreamSink methods such as SendPacket/SendPacketNoReply.

Request

Name	Type
`type`	`AudioStreamType`

SetPtsContinuityThreshold

Sets the maximum threshold (in seconds) between explicit user-provided PTS and expected PTS (determined using interpolation). Beyond this threshold, a stream is no longer considered 'continuous' by the renderer.

Defaults to an interval of half a PTS 'tick', using the currently-defined PTS units. Most users should not need to change this value from its default.

Example: A user is playing back 48KHz audio from a container, which also contains video and needs to be synchronized with the audio. The timestamps are provided explicitly per packet by the container, and expressed in mSec units. This means that a single tick of the media timeline (1 mSec) represents exactly 48 frames of audio. The application in this scenario delivers packets of audio to the AudioRenderer, each with exactly 470 frames of audio, and each with an explicit timestamp set to the best possible representation of the presentation time (given this media clock's resolution). So, starting from zero, the timestamps would be..

[ 0, 10, 20, 29, 39, 49, 59, 69, 78, 88, ... ]

In this example, attempting to use the presentation time to compute the starting frame number of the audio in the packet would be wrong the majority of the time. The first timestamp is correct (by definition), but it will be 24 packets before the timestamps and frame numbers come back into alignment (the 24th packet would start with the 11280th audio frame and have a PTS of exactly 235).

One way to fix this situation is to set the PTS continuity threshold (henceforth, CT) for the stream to be equal to 1/2 of the time taken by the number of frames contained within a single tick of the media clock, rounded up. In this scenario, that would be 24.0 frames of audio, or 500 uSec. Any packets whose expected PTS was within +/-CT frames of the explicitly provided PTS would be considered to be a continuation of the previous frame of audio. For this example, calling 'SetPtsContinuityThreshold(0.0005)' would work well.

Other possible uses: Users who are scheduling audio explicitly, relative to a clock which has not been configured as the reference clock, can use this value to control the maximum acceptable synchronization error before a discontinuity is introduced. E.g., if a user is scheduling audio based on a recovered common media clock, and has not published that clock as the reference clock, and they set the CT to 20mSec, then up to 20mSec of drift error can accumulate before the AudioRenderer deliberately inserts a presentation discontinuity to account for the error.

Users whose need to deal with a container where their timestamps may be even less correct than +/- 1/2 of a PTS tick may set this value to something larger. This should be the maximum level of inaccuracy present in the container timestamps, if known. Failing that, it could be set to the maximum tolerable level of drift error before absolute timestamps are explicitly obeyed. Finally, a user could set this number to a very large value (86400.0 seconds, for example) to effectively cause all timestamps to be ignored after the first, thus treating all audio as continuous with previously delivered packets. Conversely, users who wish to always explicitly schedule their audio packets exactly may specify a CT of 0.

Note: explicitly specifying high-frequency PTS units reduces the default continuity threshold accordingly. Internally, this threshold is stored as an integer of 1/8192 subframes. The default threshold is computed as follows: RoundUp((AudioFPS/PTSTicksPerSec) * 4096) / (AudioFPS * 8192) For this reason, specifying PTS units with a frequency greater than 8192x the frame rate (or NOT calling SetPtsUnits, which accepts the default PTS unit of 1 nanosec) will result in a default continuity threshold of zero.

Request

Name	Type
`threshold_seconds`	`float32`

SetPtsUnits

Sets the units used by the presentation (media) timeline. By default, PTS units are nanoseconds (as if this were called with numerator of 1e9 and denominator of 1). This ratio must lie between 1/60 (1 tick per minute) and 1e9/1 (1ns per tick).

Request

Name	Type
`tick_per_second_numerator`	`uint32`
`tick_per_second_denominator`	`uint32`

SetReferenceClock

Sets the reference clock that controls this renderer's playback rate. If the input parameter is a valid zx::clock, it must have READ, DUPLICATE, TRANSFER rights and refer to a clock that is both MONOTONIC and CONTINUOUS. If instead an invalid clock is passed (such as the uninitialized zx::clock()), this indicates that the stream will use a 'flexible' clock generated by AudioCore that tracks the audio device.

SetReferenceClock cannot be called once SetPcmStreamType is called. It also cannot be called a second time (even if the renderer format has not yet been set). If a client wants a reference clock that is initially CLOCK_MONOTONIC but may diverge at some later time, they should create a clone of the monotonic clock, set this as the stream's reference clock, then rate-adjust it subsequently as needed.

Request

Name	Type
`reference_clock`	`handle<clock>?`

SetUsage

Sets the usage of the render stream. This method may not be called after SetPcmStreamType is called. The default usage is MEDIA.

DEPRECATED - Use SetUsage2 and AudioRenderUsage2

Request

Name	Type
`usage`	`AudioRenderUsage`

SetUsage2

Sets the usage of the render stream. This method may not be called after SetPcmStreamType is called. The default usage is MEDIA.

Request

Name	Type
`usage2`	`AudioRenderUsage2`

ProfileProvider

Defined in fuchsia.media/profile_provider.fidl

RegisterHandlerWithCapacity

Register a thread as a media thread. This notifies the media subsystem that this thread should have an elevated scheduling profile applied to it in order to meet audio or video deadlines.

name is the name of a system scheduling role to apply to the thread given by thread_handle -- different products may customize the underlying scheduling strategy based on the requested role. period is the suggested interval to be scheduled at. period may be zero if the thread has no preferred scheduling interval. capacity is the proportion of the scheduling interval the thread needs to be running to achieve good performance or to meet the scheduling deadline defined by period. capacity may be zero if the workload has no firm runtime requirements. Note that capacity should be a good faith estimate based on the worst case runtime the thread requires each period. Excessive capacity requests may be rejected or result in scaling back the performance of other threads to fit resource limits.

Capacity, max runtime, and period have the following relationship:

capacity = max runtime / period

Where:

0 <= max runtime <= period and 0 <= capacity <= 1

For heterogeneous systems, the capacity should be planned / measured against the highest performance processor(s) in the system. The system will automatically adjust the effective capacity to account for slower processors and operating points and will avoid processors and operating points that are too slow to meet the requested scheduling parameters (provided they are reasonable).

Returns the period and capacity (actually maximum runtime) that was applied, either of which may be zero to indicate not applicable.

Request

Name	Type
`thread_handle`	`handle<thread>`
`name`	`string:64`
`period`	`zx/Duration`
`capacity`	`float32`

Response

Name	Type
`period`	`zx/Duration`
`capacity`	`zx/Duration`

RegisterMemoryRange

Register a memory range as being used for media processing. This notifies the media subsystem that this memory should have an elevated memory profile applied to it in order to meet audio or video deadlines.

name is the name of a system memory role to apply to the memory given by vmar_handle -- different products may customize the underlying memory strategy based on the requested role.

Request

Name	Type
`vmar_handle`	`handle<vmar>`
`name`	`string:64`

Response

<EMPTY>

UnregisterHandler

Reset a thread's scheduling profile to the default.

Request

Name	Type
`thread_handle`	`handle<thread>`
`name`	`string:64`

Response

<EMPTY>

UnregisterMemoryRange

Reset a memory range's memory profile.

Request

Name	Type
`vmar_handle`	`handle<vmar>`

Response

<EMPTY>

SessionAudioConsumerFactory

Interface for creating audio consumers bound to a session.

CreateAudioConsumer

Creates an AudioConsumer, which is an interface for playing audio, bound to a particular session. session_id is the identifier of the media session for which audio is to be rendered.

Request

Name	Type
`session_id`	`uint64`
`audio_consumer_request`	`server_end:AudioConsumer`

SimpleStreamSink

Defined in fuchsia.media/stream.fidl

A StreamSink that uses StreamBufferSet for buffer management.

AddPayloadBuffer

A buffer with ID id must not be in the current set when this method is invoked, otherwise the service will close the connection.

Request

Name	Type
`id`	`uint32`
`payload_buffer`	`handle<vmo>`

DiscardAllPackets

Discards packets previously sent via SendPacket or SendPacketNoReply and not yet released. The response is sent after all packets have been released.

Request

<EMPTY>

Response

<EMPTY>

DiscardAllPacketsNoReply

Discards packets previously sent via SendPacket or SendPacketNoReply and not yet released.

Request

<EMPTY>

EndOfStream

Indicates the stream has ended. The precise semantics of this method are determined by the inheriting interface.

Request

<EMPTY>

RemovePayloadBuffer

Removes a payload buffer from the current buffer set associated with the connection.

A buffer with ID id must exist in the current set when this method is invoked, otherwise the service will will close the connection.

Request

Name	Type
`id`	`uint32`

SendPacket

Sends a packet to the service. The response is sent when the service is done with the associated payload memory.

packet must be valid for the current buffer set, otherwise the service will close the connection.

Request

Name	Type
`packet`	`StreamPacket`

Response

<EMPTY>

SendPacketNoReply

Sends a packet to the service. This interface doesn't define how the client knows when the sink is done with the associated payload memory. The inheriting interface must define that.

packet must be valid for the current buffer set, otherwise the service will close the connection.

Request

Name	Type
`packet`	`StreamPacket`

StreamBufferSet

Defined in fuchsia.media/stream.fidl

Manages a set of payload buffers for a stream. This interface is typically inherited along with StreamSink or StreamSource to enable the transport of elementary streams between clients and services.

AddPayloadBuffer

A buffer with ID id must not be in the current set when this method is invoked, otherwise the service will close the connection.

Request

Name	Type
`id`	`uint32`
`payload_buffer`	`handle<vmo>`

RemovePayloadBuffer

Removes a payload buffer from the current buffer set associated with the connection.

A buffer with ID id must exist in the current set when this method is invoked, otherwise the service will will close the connection.

Request

Name	Type
`id`	`uint32`

StreamProcessor

Overview of operation:

Create

create via CodecFactory - see CodecFactory
create via LicenseSession - see LicenseSession

Get input constraints

OnInputConstraints() - sent unsolicited by stream processor shortly after stream processor creation.

Provide input buffers ()

SetInputBufferPartialSettings()
or AddBuffer, if StreamProcessor reported support for dynamic buffers via CodecFactory - in this case, QueueInputFormatDetails and QueueInputEndOfStream do not require a prior AddBuffer, but QueueInputPacket still does.

Deliver input data

QueueInputPacket() + OnFreeInputPacket(), for as long as it takes, possibly working through all input packets repeatedly before...

Get output constraints and format

OnOutputConstraints()
This is not sent until after at least one QueueInput* message is sent by the client, even if the underlying processor behind the StreamProcessor doesn't fundamentally need any input data to determine its output constraints. This server behavior prevents clients taking an incorrect dependency on the output constraints showing up before input is delivered. * A client must tolerate this arriving as late as after substantial input data has been delivered, including lots of input packet recycling via OnFreeInputPacket(). * This message can arrive more than once before the first output data.

Provide output buffers * SetOutputBufferPartialSettings() / CompleteOutputBufferPartialSettings()
Data flows, with optional EndOfStream * OnOutputPacket() / RecycleOutputPacket() / QueueInputPacket() / OnFreeInputPacket() / QueueInputEndOfStream() / OnOutputEndOfStream()

Semi-trusted StreamProcessor server - SW decoders run in an isolate (with very few capabilities) just in case the decoding SW has a vulnerability which could be used to take over the StreamProcessor server. Clients of the stream processor interface using decoders and processing streams of separate security contexts, to a greater extent than some other interfaces, need to protect themselves against invalid server behavior, such as double-free of a packet_index and any other invalid server behavior. Having fed in compressed data of one security context, don't place too much trust in a single StreamProcessor instance to not mix data among any buffers that StreamProcessor server has ever been told about. Instead, create separate StreamProcessor instances for use by security-separate client-side contexts. While the picture for HW-based decoders looks somewhat different and is out of scope of this paragraph, the client should always use separate StreamProcessor instances for security-separate client-side contexts.

Descriptions of actions taken by methods of this protocol and the states of things are given as if the methods are synchronously executed by the stream processor server, but in reality, as is typical of FIDL interfaces, the message processing is async. The states described are to be read as the state from the client's point of view unless otherwise stated. Events coming back from the server are of course delivered async, and a client that processes more than one stream per StreamProcessor instance needs to care whether a given event is from the current stream vs. some older soon-to-be-gone stream.

The Sync() method's main purpose is to enable the client to robustly prevent having both old and new buffers allocated in the system at the same time, since media buffers can be significantly large, depending. The Sync() method achieves this by only delivering it's response when all previous calls to the StreamProcessor protocol have actually taken effect in the StreamControl ordering domain. Sync() can also be used to wait for the stream processor server to catch up if there's a possibility that a client might otherwise get too far ahead of the StreamProcessor server, by for example requesting creation of a large number of streams in a row. It can also be used during debugging to ensure that a stream processor server hasn't gotten stuck. Calling Sync() is entirely optional and never required for correctness - only potentially required to de-overlap resource usage.

It's possible to re-use a StreamProcessor instance for another stream, and doing so can sometimes skip over re-allocation of buffers. This can be a useful thing to do for cases like seeking to a new location - at the StreamProcessor interface that can look like switching to a new stream.

AddBuffer

Add buffers previously created with the help of ParticipateInBufferAllocation.

For input buffers, the client can send QueueInputFormatDetails or QueueInputEndOfStream before any AddBuffer messages. At least one input buffer must be added before a valid QueueInputPacket can be sent. Most clients will want to continue quickly adding buffers up to at least buffer_count_for_server_current to avoid the codec potentially stalling, and typically a low number of buffers beyond that to keep the pipeline running smoothly. Input buffers beyond the first input buffer can be added after the first QueueInputPacket.

Even if buffer_constraints_version_ordinal is current, the server must not close the channel if the buffer isn't consistent with the current buffer_constraints_version_ordinal (per sysmem GetVmoInfo given consistent StreamProcessor constraints). Instead, the server must send a new OnOutputConstraints. This simplifies some edge cases for some clients, particularly when a client can't reliably detect whether a newly-obtained buffer was actually allocated after buffer_constraints_version_ordinal changed, or may have been cached from before.

The client can add additional buffers to the same port and buffer_lifetime_ordinal at any time using this message. If the buffer_lifetime_ordinal is no longer the most recent, the message will be ignored, the handle to the buffer dropped, and any later RemoveBuffer message re. the same buffer will complete immediately.

If dynamic_buffers_input_max or dynamic_buffers_output_max is exceeded by the sum of buffers added by all AddBuffer calls with the same buffer_lifetime_ordinal, the server may close the channel. Servers are required to close the channel in this case if performance degradation or un-tested behavior would result from adding too many buffers.

Switching to a new buffer_lifetime_ordinal starts the process of removing buffers associated with an old buffer_lifetime_ordinal. However, until the remove is complete, those buffers can still be used by the codec as normal. See also RemoveBuffer, which can be used to detect when removal is complete, regardless of whether the RemoveBuffer started the removal (including when not using dynamic buffers).

The buffer stays added until removal later completes. The buffer remains added across potentially multiple buffer re-uses. Removal can be initiated (and/or confirmed/fenced) by the client using RemoveBuffer. The codec server can unilaterally initiate buffer removal; if the server does this, it must send a new buffer_constraints_version_ordinal with action_required true. Some other client-initiated messages can also begin buffer removal, such as CloseCurrentStream with release_input_buffers and/or release_output_buffers. The buffers are also automatically removed and released if the client closes the StreamProcessor client_end or the server closes the StreamProcessor server_end.

Mixing of AddBuffer and SetInputBufferPartialSettings / SetOutputBufferPartialSettings is not allowed while there are any buffers still active on the port. To successfully switch modes for a port, most clients will want to just start over with a new StreamProcessor instance. This can also be done reliably by first ensuring that every added buffer under the port up to "now" (including those with older buffer_lifetime_ordinal) has completed a RemoveBuffer request. Only then is it known ok to switch modes for that port under the same StreamProcessor. Most clients will just use one way or the other to add buffers and never need to switch to the other way, especially within the same StreamProcessor.

If a client might plausibly "spam" creation of many new buffer_lifetime_ordinal values without the buffers seeing any actual usage in between, the client should consider starting a Sync every few buffer_lifetime_ordinal(s) to fence cleanup of old buffer_lifetime_ordinal values, and avoid getting ahead of Sync completions by more than 16 buffer_lifetime_ordinal values. Else the channel may close from a backlog of new buffer_lifetime_ordinal(s) getting too far ahead of closing out old ones. The threshold of 16 is well below the enforcement threshold. Clients don't need to Sync if they won't be spamming new buffer_lifetime_ordinal values, or if added buffers will see at least some actual usage visible to the client before being replaced again.

All buffers of the same port and buffer_lifetime_ordinal must share the same [fuchsia.sysmem2/SingleBufferSetttings]. The client can ensure this in various ways. One way is to use ParticipateInBufferAllocation then AddBuffer for at least the first buffer, before ParticipateInBufferAllocation for any subsequent buffers. Another way is to observe a mismatch in SingleBufferSettings before sending AddBuffer and bump to the next odd buffer_lifetime_ordinal value for the AddBuffer.

Added: NEXT

Request

Name	Type
`payload`	`StreamProcessorAddBufferRequest`

CloseCurrentStream

This "closes" the current stream, leaving no current stream. In addition, this message can optionally release (and unregister) input buffers or output buffers.

If there has never been any active stream, the stream_lifetime_ordinal must be zero or the server will close the channel. If there has been an active stream, the stream_lifetime_ordinal must be the most recent active stream whether that stream is still active or not. Else the server will close the channel.

Multiple of this message without any new active stream in between is not to be considered an error, which allows a client to use this message to close the current stream to stop wasting processing power on a stream the user no longer cares about, then later decide that buffers should be released and send this message again with release_input_buffers and/or release_output_buffers true to get the buffers released, if the client is interested in trying to avoid overlap in resource usage between old buffers and new buffers (not all clients are).

Request

Name	Type
`stream_lifetime_ordinal`	`uint64`
`release_input_buffers`	`bool`
`release_output_buffers`	`bool`

CompleteOutputBufferPartialSettings

After SetOutputBufferPartialSettings(), the server won't send OnOutputConstraints(), OnOutputFormat(), OnOutputPacket(), or OnOutputEndOfStream() until after the client sends CompleteOutputBufferPartialSettings().

This message isn't permitted after AddBuffer. When using dynamic buffers the server can send OnOutputConstraints, OnOutputFormat, OnOutputPacket, or OnOutputEndOfStream at any time after the first QueueInputPacket. In the case of OnOutputPacket there must also be at least one output buffer that's been added but not yet fully removed (RemoveBuffer not yet complete).

Some clients may be able to send CompleteOutputBufferPartialSettings() immediately after SetOutputBufferPartialSettings() - in that case the client needs to be prepared to receive output without knowing the buffer count or packet count yet - such clients may internally delay processing the received output until the client has heard from sysmem (which is when the client will learn the buffer count and packet count).

Other clients may first wait for sysmem to allocate, prepare to receive output, and then send CompleteOutputBufferPartialSettings().

Request

Name	Type
`buffer_lifetime_ordinal`	`uint64`

EnableForceOutputBuffersFixedImageSize

For video decoders, this forces the output buffers to be reallocated if the image size needs to change. This is wasteful as it forces extra buffer reallocations given typical video bitrate control strategies (applicable to both streaming and RTC) involving shifting the image dimensions up and down repeatedly as a logical video/stream plays, sometimes even if network conditions remain fairly stable. This extra buffer reallocation cost is incurred vs. baseline whether the dimension switching is achieved within a single StreamProcessor stream or by using a new StreamProcessor stream for new dimensions.

A client should not send this message unless the client really must force the output buffers to be reallocated every time the output image size changes.

Sending this message more than once closes the channel. If sent, this message must be sent prior to any SetInputBufferPartialSettings, SetOutputBufferPartialSettings, ParticipateInBufferAllocation, or AddBuffer.

This message is only permitted if this StreamProcessor is a video decoder.

This message is only permitted when [fuchsia.mediacodec/CodecFactory.DetailedCodecDescription.supports_dynamic_buffers] is true.

Added: NEXT

Request

<EMPTY>

EnableOldOutputBuffers

This informs the StreamProcessor that the client is prepared to handle output packets that specify a buffer with buffer_lifetime_ordinal older than the most recent buffer_lifetime_ordinal.

If the client doesn't send this message, the StreamProcessor will omit any such output, even if DetailedCodecDescription.supports_dynamic_buffers is true. For relevant decoders such as VP9 decoders, not sending this message can result in output that isn't bistream spec compliant, and the output can be visually different than intended by the bitstream.

Such streams are only possible with some bitstream formats (such as VP9), and are rare, but can happen and can be valid per the bitstream spec. For example, this can be specified by a VP9 bitstream using show_existing_frame to output an old-dimensions buffer after having already output a new-dimensions buffer.

Most clients that send this message will also want to use RemoveBuffer to know when it becomes safe to stop tracking an old buffer.

Most of the time this makes no difference as most bitstreams don't actually emit old buffers, even if the bitstream spec would allow it. Old output buffers are especially rare for RTC streams which typically don't have any frame reordering in the first place.

In most video streaming scenarios that use dimension switching as part of their bitrate control strategy (among those that I've observed), at the StreamProcessor layer the new dimensions are part of a new stream instead of being spliced together as a continuation of the old stream. That said, using a continuation of the old stream is also a completely valid way to implement dimension switching. When a stream switch occurs as part of dimension switching, the decoder state is not retained and there won't be any old buffer(s) emitted after new buffer(s), since the new stream doesn't know anything about old buffers filled by the old stream.

Clients which haven't tested their ability to handle old output buffers should not send this message. Clients decoding bitstreams like VP9 for decoder compliance testing purposes should send this message (and use a VP9 decoder with DetailedCodecDescrption.supports_dynamic_buffers true). Clients which are required to support old output frames and/or fully comply with a relevant bitstream spec should/must send this message, and should test using a test stream that outputs packets referencing an old output buffer.

Sending this message more than once closes the channel. If sent, this message must be sent prior to the client establishing the first output buffer_lifetime_ordinal. This requirement avoids ambiguity re. free/busy status of packets of old buffer_lifetime_ordinal(s), as the server can auto-recycle packets with old buffer_lifetime_ordinal on behalf of the client when this message was not sent by the client.

This message is only permitted when [fuchsia.mediacodec/CodecFactory.DetailedCodecDescription.supports_dynamic_buffers] is true.

Added: NEXT

Request

<EMPTY>

EnableOnStreamFailed

Permit the server to use OnStreamFailed() instead of the server just closing the whole StreamProcessor channel on stream failure.

If the server hasn't seen this message by the time a stream fails, the server will close the StreamProcessor channel instead of sending OnStreamFailed().

Request

<EMPTY>

EnableSameOutputBufferConcurrentlyInFlight

This informs the StreamProcessor that the client is prepared to handle output packets that specify the same buffer as another packet that's also concurrently in flight to the client (not yet recycled).

Most bitstream formats don't do this. In formats that can do this such as VP9, most actual bitstreams don't do this.

As an example, in VP9, a stream can cause the same output buffer to be used by another emitted output packet/frame by using show_existing_frame on the same VP9 reference/held frame slot more than once without decoding a new frame into that slot in between.

Clients that need to achieve full bitstream spec compliance for such a bitstream format must send this message.

If this message is not in effect, the server will omit any such output, which can result in output that is not compliant to the relevant bitstream spec, and which can be visually different than the stream intended.

Sending this message more than once closes the channel. If sent, this message must be sent prior to the client establishing the first output buffer_lifetime_ordinal.

This message is only permitted when [fuchsia.mediacodec/CodecFactory.DetailedCodecDescription.supports_dynamic_buffers] is true.

Added: NEXT

Request

<EMPTY>

FlushEndOfStreamAndCloseStream

This message is optional; a client isn't required to send this ever.

Terminology note: In the name of this message, "flush" means flush through, sometimes called "drain". This messge does not discard. To discard, just QueueInputPacket with a new stream_lifetime_ordinal without first waiting for OnOutputEndOfStream of the old stream and without sending FlushEndOfStreamAndCloseStream.

There is currently no way to cancel the flush effect of this message short of the client closing the StreamProcessor channel.

This message is only valid after QueueInputEndOfStream() for this stream. The stream_lifetime_ordinal input parameter must match the stream_lifetime_ordinal of the QueueInputEndOfStream(), else the server will close the channel.

A client can use this message to flush through (drain, not discard) the last input data of a stream so that the stream processor server generates corresponding output data for all the input data before the server moves on to the next stream, without forcing the client to wait for OnOutputEndOfStream() before queueing data of another stream.

The difference between QueueInputEndOfStream() and FlushEndOfStreamAndCloseStream(): QueueInputEndOfStream() is a promise from the client that there will not be any more input data for the stream (and this info is needed by some stream processors for the stream processor to ever emit the very last output data). The QueueInputEndOfStream() having been sent doesn't prevent the client from later completely discarding the rest of the current stream by closing the current stream (with or without a stream switch). In contrast, FlushEndOfStreamAndCloseStream() is a request from the client that all the previously-queued input data be processed including the logical "EndOfStream" showing up as OnOutputEndOfStream() (in success case) before moving on to any newer stream - this essentially changes the close-stream handling from discard to flush-through for this stream only.

A client using this message can start providing input data for a new stream without that causing discard of old stream data. That's the purpose of this message - to allow a client to flush through (not discard) the old stream's last data (instead of the default when closing or switching streams which is discard).

Because the old stream is not done processing yet and the old stream's data is not being discarded, the client must be prepared to continue to process OnOutputConstraints() messages until the stream_lifetime_ordinal is done. The client will know the stream_lifetime_ordinal is done when OnOutputEndOfStream(), OnStreamFailed(), or the StreamProcessor channel closes.

Request

Name	Type
`stream_lifetime_ordinal`	`uint64`

OnFreeInputPacket

The server sends this message when the stream processor is done reading the data in this packet (but not necessarily done processing its own copy of the data). This message is sent regardless of any changes to stream_lifetime_ordinal, stream closure, buffer_lifetime_ordinal changes (as/when supported), etc.

When a new input buffer_lifetime_ordinal is created, all input packets start with the client, and there is no initial OnFreeInputPacket.

On receiving this event, the client can determine which input buffer is becoming free for re-use based on a packet_index to buffer_index correspondence maintained by the client. This correspondence can be fixed or dynamic, depending on client preference.

For video decoders, an OnFreeInputPacket can be relied on to indicate that all prior packets are done reading from their input buffers as well. However, the client isn't required to track this, as all OnFreeInputPacket(s) will be sent by the server (just not necessarily in the same order as QueueInputPacket, even for video decoders).

Re-use of input buffer buffer_index values is independent from re-use of input packet packet_index values in general, but some clients may choose to always set packet_index to the same value as buffer_index.

OnFreeInputPacket() does not imply that the data in the input packet has been processed successfully or that processing of the input bytes has been attempted yet, only that the input data in the associated input buffer is no longer needed by the StreamProcessor. If a client needs to know which input data has generated corresponding output, using timestamp_ish values for that is recommended.

The order of OnFreeInputPacket is not guaranteed to be the same as the order of QueueInputPacket, and OnFreeInputPacket can occur after an OnStreamFailed.

Response

Name	Type
`free_input_packet`	`PacketHeader`

OnInputConstraints

The server sends this shortly after StreamProcessor creation to indicate input buffer constraints. The "min" and "max" input constraints don't change for the life of the StreamProcessor.

The "max" values for buffer size and count are large enough to support the most demanding format the server supports on input. The "recommended" values should be workable for use with the input FormatDetails conveyed during StreamProcessor creation. The "recommended" values are not necessarily suitable if the client uses QueueInputFormatDetails() to change the input format. In that case it's up to the client to determine suitable values, either by creating a new StreamProcessor instance instead, or knowing suitable values outside the scope of this protocol.

See comments on StreamBufferConstraints.

This message is guaranteed to be sent unsolicited to the StreamProcessor client during or shortly after StreamProcessor creation. Clients should not depend on this being the very first message to arrive at the client.

The "min" and "max" input constraints are guaranteed not to change for a given StreamProcessor instance. The "recommended" values may effectively change when the server processes QueueInputFormatDetails(). There is not any way in the protocol short of creating a new StreamProcessor instance for the client to get those new "recommended" values.

The first buffer_constraints_version_ordinal from the server (for input) will always be 1.

Response

Name	Type
`input_constraints`	`StreamBufferConstraints`

OnOutputConstraints

This event informs the client of new output constraints.

This message is ordered with respect to other output (such as output packets, output format, output end-of-stream).

Before the first OnOutputPacket() of a stream, the server guarantees that at least one OnOutputConstraints() and exactly one OnOutputFormat() will be sent. The client must tolerate multiple OnOutputConstraints() (and 1 OnOutputFormat() message) before the first output packet. As long as the client hasn't moved to a new stream, the server won't send another OnOutputConstraints() until after the client has configured output buffers.

This message can be sent mid-stream by a server, when the server needs new output buffers. The server won't generate more output until the client has added new output buffers via SetOutputBufferPartialSettings or AddBuffer.

The server will not send any OnOutputPacket() for this stream until after the client has configured/re-configured output buffers.

All clients, even those which don't want to support any mid-stream output buffer re-config or mid-stream OnOutputFormat() are required to deal with 1..multiple OnOutputConstraints() messages before the first output packet, and one OnOutputFormat() messages before the first output packet.

Response

Name	Type
`output_config`	`StreamOutputConstraints`

OnOutputEndOfStream

After QueueInputEndOfStream() is sent by the StreamProcessor client, within a reasonable duration the corresponding OnOutputEndOfStream() will be sent by the StreamProcessor server. Similar to QueueInputEndOfStream(), OnOutputEndOfStream() is sent a maximum of once per stream.

No more stream data for this stream will be sent after this message. All input data for this stream was processed.

While a StreamProcessor client is not required to QueueInputEndOfStream() (unless the client wants to use FlushEndOfStreamAndCloseStream()), if a StreamProcessor server receives QueueInputEndOfStream(), and the client hasn't closed the stream, the StreamProcessor server must generate a corresponding OnOutputEndOfStream() if nothing went wrong, or must send OnStreamFailed(), or must close the server end of the StreamProcessor channel. An ideal StreamProcessor server would handle and report stream errors via the error_ flags and complete stream processing without sending OnStreamFailed(), but in any case, the above-listed options are the only ways that an OnOutputEndOfStream() won't happen after QueueInputEndOfStream().

There will be no more OnOutputPacket() or OnOutputConstraints() messages for this stream_lifetime_ordinal after this message - if a server doesn't follow this rule, a client should close the StreamProcessor channel.

The error_detected_before bool has the same semantics as the error_detected_before bool in OnOutputPacket().

Response

Name	Type
`stream_lifetime_ordinal`	`uint64`
`error_detected_before`	`bool`

OnOutputFormat

This message is sent by the server before the first output packet of any stream, and potentially mid-stream between output packets of the stream, ordered with respect to output packets, and ordered with respect to OnOutputConstraints().

The server guarantees that the first packet of every stream will be preceeded by an OnOutputFormat().

The server guarantees that there will be an OnOutputFormat() between an OnOutputConstraints() and an OnOutputPacket(). In other words, the client is essentially allowed to forget what the output format is on any OnOutputConstraints(), because the server promises a subsequent OnOutputFormat() before any OnOutputPacket().

The server won't send OnOutputFormat() (and therefore also won't send OnOutputPacket()) until the client has re-configured output buffers.

The server is allowed to send an OnOutputFormat() mid-stream between two output packets.

A server won't send two adjacent OnOutputFormat messages without any output packet in between. However an OnOutputFormat message doesn't guarantee a subsequent packet, because for example the server could send OnOutputEndOfStream or OnStreamFailed instead, or the client can move on to a new stream and the server may not send a packet from the old stream despite having sent OnOutputFormat.

A client that does not wish to seamlessly handle mid-stream output format changes should either ensure that no stream processed by the client ever has any mid-stream format change, or the client should ensure that any retry of processing starts the new attempt at a point logically at or after the point where the old format has ended and the new format starts, else the client could just hit the same mid-stream format change again.

An example of this message being sent mid-stream is mid-stream change of dimensions of video frames output from a video decoder.

Not all servers will support seamless handling of format change. Those that do support seamless handling of format change may require that the format change not also require output buffer re-config, in order for the handling to be seamless. See the comment block for OnOutputConstraints() for more discussion of how servers and clients should behave - in particular when they don't seamlessly handle output constraint change and/or output format change.

If this message isn't being sent by the server when expected at the start of a stream, the most common reason is that a OnOutputConstraints() hasn't been processed by the client (by configuring output buffers using SetOutputBufferPartialSettings or AddBuffer).

Response

Name	Type
`output_format`	`StreamOutputFormat`

OnOutputPacket

This is how the stream processor emits an output packet to the stream processor client.

Order is significant.

The client should eventually call RecycleOutputPacket() (possibly after switching streams multiple times), unless the buffer_lifetime_ordinal has moved on. A stream change doesn't change which packets are busy with the client vs. free with the server.

The relevant buffer is always the one specified in the packet's buffer_index field.

For low-level buffer types that support it, a StreamProcessor is free to emit an output packet before the low-level buffer actually has any usable data in the buffer, with the mechanism for signalling the presence of data separate from the OnOutputPacket() message. For such low-level buffer types, downstream consumers of data from the emitted packet must participate in the low-level buffer signalling mechanism to know when it's safe to consume the data. This is most likely to be relevant when using a video decoder and gralloc-style buffers.

The error_ bool(s) allow (but do not require) a StreamProcessor server to report errors that happen during an AU or between AUs.

The scope of error_detected_before starts at the end of the last delivered output packet on this stream, or the start of stream if there were no previous output packets on this stream. The scope ends at the start of the output_packet.

The error_detected_before bool is separate so that discontinuities can be indicated separately from whether the current packet is damaged.

The scope of error_detected_during is from the start to the end of this output_packet.

Response

Name	Type
`output_packet`	`Packet`
`error_detected_before`	`bool`
`error_detected_during`	`bool`

OnOutputTimestampHasNoOutput

Most clients can ignore the OnOutputTimestampHasNoOutput message.

OnOutputTimestampHasNoOutput is only sent by StreamProcessor servers with supports_dynamic_buffers true.

When the server determines that a previous input packet with a given timestamp_ish value will never generate any output, the server will send this message with the timestamp_ish value.

This message can be thought of as a stand-in for OnOutputPacket when the server eventually determines that the timestamp will never generate any output; this is why the name starts with "OnOutput". Similar to messages indicating output, the server will not send this message when there is an in-progress stream-driven output re-config in progress (as occurs starting with OnOutputConstraints).

While servers should send this message as soon as allowed, feasible, and practical, there is no specific timeliness guarantee regarding when this message will be sent by a server. For some input packets that don't generate any output, the server may send this message fairly soon after consuming the corresponding input packet. For other input packets that don't generate any output, the server may not send this message until the server can prove that the corresponding input packet will never generate any output via delayed means such as, for example, logic involving max_num_reorder_frames, or any relevant bitstream-format-specific logic. The server is allowed to not send this message until substantially more input is received and processed by the server. The client must not attempt to wait on reception of this message before sending more input, as that would create a potential deadlock.

When sent, this message will be sent at some time after the OnFreeInputPacket message for the corresponding input packet that had the same timestamp_ish value. The client must not delay re-use of the input buffer until after reception of OnOutputTimestampHasNoOutput because that would create a potential deadlock.

When sent, this message will be sent before the OnOutputEndOfStream message for the stream_lifetime_ordinal stream. A server is not required to send OnOutputTimestampHasNoOutput for all unresolved (in terms of output yes/no) input timestamp_ish values prior to sending OnOutputEndOfStream, and the OnOutputEndOfStream unambiguously resolves the (output yes/no) status of any remaining input timestamp_ish values tracked by the client.

The server must not output the indicated timestamp_ish value in a later OnOutputPacket, unless the client is not ensuring that timestamp_ish values are unique and the client is re-using the timestamp_ish value on another input packet. The server-side mechanism to achieve this robustly can be bitstream format specific and codec implementation specific.

This message is not required to be sent by the server when the reason for an input packet not generating output is the client switching to a new stream without FlushEndOfStreamAndCloseStream, stream failure, or codec failure.

If the client is ensuring that all input packets with a given stream_lifetime_ordinal have unique timestamp_ish values, then the client can use this message to determine which input packet will never generate any output.

If the client is not ensuring that all input packets have a unique timestamp_ish value, then the client should always ignore this message, as the protocol makes no particular gaurantees regarding this message in this case (it may not be possible for any such client to disambiguate which input packet is meant, the server may not send this message as many times as there were input packets with the same timestamp_ish value, the server may not send this message at all once it notices any non-unique timestamp_ish values, etc).

Effective and robust use of this message may require the client to assign timestamp_ish values as sequential unsigned 64 bit numbers, and keep any corresponding real timestamps locally in the client.

If a client doesn't have any specific reason to pay attention to this event, the client should ignore this event. Most clients can completely ignore this event with no adverse impact.

Added: NEXT

Response

Name	Type
`payload`	`StreamProcessorOnOutputTimestampHasNoOutputRequest`

OnStreamFailed

The stream has failed, but the StreamProcessor instance is still usable for a new stream.

This message is only ever sent by the server if the client previously sent EnableOnStreamFailed(). If the client didn't send EnableOnStreamFailed() then the server closes the StreamProcessor channel instead.

StreamProcessor server implementations are encouraged to handle stream errors (and ideally to also report them via error_ bools of OnOutputPacket() and OnOutputEndOfStream()) without failing the whole stream, but if a stream processor server is unable to do that, but still can cleanly contain the failure to the stream, the stream processor server can (assuming EnableOnStreamFailed() was called) use OnStreamFailed() to indicate the stream failure to the client without closing the StreamProcessor channel.

An ideal StreamProcessor server handles problems with input data without sending this message, but sending this message is preferred vs. closing the server end of the StreamProcessor channel if the StreamProcessor server can 100% reliably contain the stream failure to the stream, without any adverse impact to any later stream.

No further messages will arrive from the server regarding the failed stream. This includes any OnOutputEndOfStream() that the client would have otherwise expected.

Response

Name	Type
`stream_lifetime_ordinal`	`uint64`
`error`	`StreamError`

ParticipateInBufferAllocation

This message results in channel closure unless supports_dynamic_buffers is set to true.

This participates in allocation of buffers to be used with AddBuffer later. The client can get VMO handles for these buffers by also participating in the sysmem allocation, using the client's own related sysmem token (associated with the same logical buffer collection). It's up to the client to separately set any constraints needed by the client using the client's own related sysmem token, if any.

Some clients may prefer to use SetInputBufferPartialSettings and/or SetOutputBufferPartialSettings. Servers must support those messages.

In handling this message, if allow_single_buffer is set to true, the server must not constrain the number of buffers allocated. The server must set min_buffer_count to 1, and must leave max_buffer_count un-set or set it to 0xFFFFFFFF, and must leave all min_buffer_count_* fields un-set. The sender can set min_buffer_count and max_buffer_count to the same value if the intent is to allocate exactly that many buffers. If allow_single_buffer is un-set or set to false, the server will indicate needed buffer counts to sysmem.

The server's BufferCollection channel (created from the passed-in sysmem2_token) may see ZX_CHANNEL_PEER_CLOSED at any time, but in particular, the server shouldn't expect the BufferCollection channel to remain connected to sysmem beyond the server sending SetConstraints. For this reason, the server may not be able to call WaitForAllBuffersAllocated or similar, so the server should just send SetConstraints, Close, then close the server's BufferCollection client_end. This means the server in general shouldn't attempt to get VMO handles for these buffers while processing this message.

The server should not assume that these buffers will necessarily ever be added with AddBuffer to this StreamProcessor instance or any other StreamProcessor instance (owned by the server or not). These buffers may instead be dropped, or as a less-common example, possibly added to a different codec served by a different server implementation which also participated in the same sysmem buffer collection allocation.

For input buffers, AddBuffer of the allocated buffer(s) to a different StreamProcessor instance of the same codec (same per CodecFactory) is likely to work, but using the same StreamProcessor instance is recommended when feasible.

In contrast, for output buffers, AddBuffer of the allocated buffer(s) to a different StreamProcessor instance of the same codec (same per CodecFactory) can't (within reason) be made work in general, especially for video decoders. Therefore, for output buffers, the same StreamProcessor instance must be used for this message and AddBuffer. While a client may currently be able to get away with using different StreamProcessor instances for this message and AddBuffer for output buffers for some codecs, this may break at any time without it being considered a server-side bug.

The allocated buffers can later be added using AddBuffer (piecemeal), and can be removed (piecemeal) using RemoveBuffer.

Multiple different ParticipateInBufferAllocation messages can have their buffers later added to the same StreamProcessor instance using the same buffer_lifetime_ordinal. This can be useful if the client wants to allocate buffers incrementally, or dynamically adjust the number of buffers, potentially while actively processing. See also the buffer_lifetime_ordinal field of this message.

Server implementations may use sysmem to help verify buffer compatibility later when buffers are added with AddBuffer.

Added: NEXT

Request

Name	Type
`payload`	`StreamProcessorParticipateInBufferAllocationRequest`

QueueInputEndOfStream

Inform the server that all QueueInputPacket() messages for this stream have been sent.

If the stream isn't closed first (by the client, or by OnStreamFailed(), or StreamProcessor channel closing), there will later be a corresponding OnOutputEndOfStream().

The corresponding OnOutputEndOfStream() message will be generated only if the server finishes processing the stream before the server sees the client close the stream (such as by starting a new stream). A way to force the server to finish the stream before closing is to use FlushEndOfStreamAndCloseStream() after QueueInputEndOfStream() before any new stream. Another way to force the server to finish the stream before closing is to wait for the OnOutputEndOfStream() before taking any action that closes the stream.

In addition to serving as an "EndOfStream" marker to make it obvious client-side when all input data has been processed, if a client never sends QueueInputEndOfStream, no amount of waiting will necessarily result in all input data getting processed through to the output. Some stream processors have some internally-delayed data which only gets drained (pushed through) by additional input data or by this EndOfStream marker. In that sense, this message can be viewed as a drain at InputData domain level, but the drain only takes effect if the stream processor even gets that far before the stream is just closed at StreamControl domain level. This message is not alone sufficient to act as an overall drain at StreamControl level. For that, send this message first and then send FlushEndOfStreamAndCloseStream (at which point it becomes possible to queue input data for a new stream without causing discard of this older stream's data). Alternately, the client can wait for the OnOutputEndOfStream before closing the current stream.

After a client sends QueueInputEndOfStream for a stream, if the client then sends for the same stream any of QueueInputPacket, QueueInputFormatDetails, QueueInputEndOfStream, the server will close the StreamProcessor channel.

Request

Name	Type
`stream_lifetime_ordinal`	`uint64`

QueueInputFormatDetails

If the input format details are still the same as specified during StreamProcessor creation, this message is unnecessary and does not need to be sent.

If the stream doesn't exist yet, this message creates the stream.

The server won't send OnOutputConstraints() until after the client has sent at least one QueueInput* message.

All servers must permit QueueInputFormatDetails() at the start of a stream without failing, as long as the new format is supported by the StreamProcessor instance. Technically this allows for a server to only support the exact input format set during StreamProcessor creation, and that is by design. A client that tries to switch formats and gets a StreamProcessor channel failure should try again one more time with a fresh StreamProcessor instance created with CodecFactory using the new input format during creation, before giving up.

These format details override the format details specified during stream processor creation for this stream only. The next stream will default back to the format details set during stream processor creation.

This message is permitted at the start of the first stream (just like at the start of any stream). The format specified need not match what was specified during stream processor creation, but if it doesn't match, the StreamProcessor channel might close as described above.

Request

Name	Type
`stream_lifetime_ordinal`	`uint64`
`format_details`	`FormatDetails`

QueueInputPacket

This message queues input data to the stream processor for processing.

If the stream doesn't exist yet, this message creates the new stream.

The server won't send OnOutputConstraints() until after the client has sent at least one QueueInput* message.

When using dynamic buffers the server can send OnOutputConstraints, OnOutputFormat, OnOutputPacket, or OnOutputEndOfStream at any time after the first QueueInputPacket. In the case of OnOutputPacket there must also be at least one output buffer that's been added but not yet fully removed (RemoveBuffer not yet complete).

The client must continue to deliver input data via this message even if the stream processor has not yet generated the first OnOutputConstraints, and even if the StreamProcessor is generating OnFreeInputPacket for previously-queued input packets. The input data must continue as long as there are free packets to be assured that the server will ever generate the first OnOutputConstraints.

The server will close the channel if this packet refers to an old buffer_lifetime_ordinal. Clients that need to deliver input images of different dimensions to a video encoder can either (a) allocate buffers large enough to contain the range of needed image sizes and use those to deliver all the input images, (b) keep their own sets of buffers used for different image dimensions and re-add old buffers with AddBuffer, moving to a new buffer_lifetime_ordinal each time image dimensions change, or (c) allocate new buffers each time image dimensions change. The (a) option is preferred, for clients that have the flexibiilty to store images of varying dimensions in a single set of buffers. Clients that require each buffer to have a single image size can use option (b) or (c).

Request

Name	Type
`packet`	`Packet`

RecycleOutputPacket

After the client is done with an output packet, the client needs to tell the stream processor that the output packet can be re-used for more output, using this message.

It's not permitted to recycle an output packet that's already free with the stream processor server.

If a client is using EnableOldOutputBuffers, the client must recycle all packets when done with them, even those of old buffer_lifetime_ordinal. This is also permitted behavior for all clients.

If a client is not using EnableOldOutputBuffers, the client may optionally omit this message for packets with an old buffer_lifetime_ordinal. In other words, packets from before an explicit or implicit output buffer de-configuration don't need to be recycled if the client isn't using and won't be using EnableOldOutputBuffers.

Request

Name	Type
`available_output_packet`	`PacketHeader`

RemoveBuffer

When using dynamic buffers, this call removes a buffer as soon as the buffer can be removed without adversely impacting any ongoing processing or an in-flight output packet referring to the buffer.

If a client wants to remove a buffer with an in-flight output packet referring to the buffer, the client must RecycleOutputPacket for that in-flight output packet before the RemoveBuffer will complete.

When using SetInputBufferPartialSettings / SetOutputBufferPartialSettings, this call doesn't initiate removal of the buffer. This call will complete when the buffer is done removing due to other reasons, such as a new buffer_lifetime_ordinal starting.

Until RemoveBuffer completes, the codec is still allowed to send OnOutputPacket messages referencing the buffer, and the codec may still have a VMO handle open to the buffer. After RemoveBuffer completes, the codec guarantees that no subsequent output packet will reference the buffer, and that the server holds no VMO handles to the buffer.

The client may need to recycle an output packet before the RemoveBuffer call can complete. Clients should take care to avoid blocking packet recycling while the RemoveBuffer request is in progress, since this would create a potential deadlock.

The server closing any VMO handles to the buffer prior to completing this call is important for the client's ability to prevent memory usage spikes.

If the client has "paused" processing by not providing any more input, the client will potentially need to send CloseCurrentStream before RemoveBuffer will complete. When using dynamic buffers, setting release_input_buffers or release_output_buffers to true is not necessary; the RemoveBuffer is explicitly telling the codec to release a specific buffer. When not using dynamic buffers, the client will need to set release_input_buffers or release_output_buffers to true, since RemoveBuffer alone doesn't initiate removal when not using dynamic buffers. A current "paused" stream needs to be stopped because frames can be held as reference frames, and codecs are never reqiured to copy their output data.

For any video decoder output buffers being removed while there's an active stream (being fed input or not), it's typically best for the client to assume that removal of a video decoder output buffer may take a very long duration. This is because bitstreams, especially non-standard-compliant bitstreams, but in some cases potentially even standard-compliant bitstreams, can keep a video decoder output buffer in the set of active reference frames (aka DPB) indefinitely. The server is not required to notice that a stream is not conforming to a bitstream standard in this regard.

In the case of h.264 decode, a standard-complient bitstream will limit the reorder delay to no longer than the max DPB occupancy. However, a server is not required to detect or reject non-compliant streams that potentially keep a frame in the DPB for longer.

For HEVC, the situation is similar to h.264 (IIUC).

In the case of VP9, there is nothing in the bitstream spec that limits the reorder delay (IIUC), meaning a frame can potentially stay in VP9's set of 8 reference frames (aka DPB) indefinitely. The server is not required to detect or mitigate this.

Until this call completes, the server may still be using the buffer. For output buffers, the server can still send OnOutputPacket message(s) that references this buffer, but only up until the RemoveBuffer completion message is sent by the server. The client must continue to RecycleOutputPacket for packets that reference the buffer, until RemoveBuffer completes.

When using dynamic buffers, upon receiving this message, the server will stop selecting the buffer for any new usage (as in, for any usage that moves the buffer from "free" to "not free" within the server). This applies even if the server has no other buffers available for use (aka no other "free" buffers). Any existing usage of the buffer is not ended early by this call alone.

When not using dynamic buffers, upon receiving this message, the server will just remember to complete this call shortly after the buffer has completed removal triggered by some other cause (removal is not triggered/caused by this call).

For video decoders, when using dynamic buffers, typically the client should take care to avoid removing too many output buffers for continued decode to be possible. If this occurs, the decoder will wait until the client adds another output buffer with AddBuffer. This can be a deadlock if the client never sends that AddBuffer. Due to DPB mechanism(s) and frame reordering, sending a single AddBuffer doesn't necessarily guarantee another OnOutputPacket, since additional output buffers can be needed before the server can send OnOutputPacket.

The client must not call RemoveBuffer on the same buffer more than once, whether overlapping in time or not. The server should enforce this when not enforcing would require tracking additional concurrent requests. The server is not required to enforce this when enforcing would use more server memory.

The server is allowed to complete this request quickly with success for buffer_lifetime_ordinal and buffer_index combinations that were never real buffers, but must close the channel if the buffer_lifetime_ordinal hasn't been started by the client yet (no removing potential future buffers).

When using dynamic buffers, a buffer_lifetime_ordinal and buffer_index combination may be re-used after completion of RemoveBuffer, but only if the buffer_lifetime_ordinal is the current buffer_lifetime_ordinal. In other words, no adding buffers under an old buffer_lifetime_ordinal.

Assuming a valid historical buffer is specified, successful completion of this call means the buffer has been fully released by the server and won't be referenced in any subsequent OnOutputPacket.

Added: NEXT

Request

Name	Type
`payload`	`StreamProcessorRemoveBufferRequest`

Response

Name	Type
`payload`	`StreamProcessor_RemoveBuffer_Result`

SetInputBufferPartialSettings

When the client is not using dynamic buffers, a single SetInputBufferPartialSettings() provides the StreamProcessor with the client-specified input settings and a BufferCollectionToken which the StreamProcessor will use to convey constraints to sysmem. Both the client and the StreamProcessor will be informed of the allocated buffers directly by sysmem via their BufferCollection channel (not via the StreamProcessor channel).

The client must not QueueInput...() until after sysmem informs the client that buffer allocation has completed and was successful.

The server should be prepared to see QueueInput...() before the server has necessarily heard from sysmem that the buffers are allocated - the server must tolerate either ordering, as the QueueInput...() and notification of sysmem allocation completion arrive on different channels, so the client having heard that allocation is complete doesn't mean the server knows that allocation is complete yet. However, the server can expect that allocation is in fact complete and can expect to get the allocation information from sysmem immediately upon requesting the information from sysmem.

Mixing of AddBuffer and SetInputBufferPartialSettings is not allowed while there are any buffers still active on the port. To successfully switch modes for a port, most clients will want to just start over with a new StreamProcessor instance. This can also be done reliably by first ensuring that every added buffer under the port up to "now" (including those with older buffer_lifetime_ordinal) has completed a RemoveBuffer request. Only then is it known ok to switch modes for that port under the same StreamProcessor. Most clients will just use one way or the other to add buffers and never need to switch to the other way, especially within the same StreamProcessor.

Request

Name	Type
`input_settings`	`StreamBufferPartialSettings`

SetOutputBufferPartialSettings

This is the replacement for SetOutputBufferSettings().

When the client is using sysmem to allocate buffers, this message is used instead of SetOutputBufferSettings()+AddOutputBuffer(). Instead, a single SetOutputBufferPartialSettings() provides the StreamProcessor with the client-specified output settings and a BufferCollectionToken which the StreamProcessor will use to convey constraints to sysmem. Both the client and the StreamProcessor will be informed of the allocated buffers directly by sysmem via their BufferCollection channel (not via the StreamProcessor channel).

Configuring output buffers is required after OnOutputConstraints() is received by the client with buffer_constraints_action_required true and stream_lifetime_ordinal equal to the client's current stream_lifetime_ordinal (even if there is an active stream), and is permitted any time there is no current stream.

Closing the current stream occurs on the StreamControl ordering domain, so after a CloseCurrentStream() or FlushEndOfStreamAndCloseStream(), a subsequent Sync() completion must be received by the client before the client knows that there's no longer a current stream.

Mixing of AddBuffer and SetOutputBufferPartialSettings is not allowed while there are any buffers still active on the port. To successfully switch modes for a port, most clients will want to just start over with a new StreamProcessor instance. This can also be done reliably by first ensuring that every added buffer under the port up to "now" (including those with older buffer_lifetime_ordinal) has completed a RemoveBuffer request. Only then is it known ok to switch modes for that port under the same StreamProcessor. Most clients will just use one way or the other to add buffers and never need to switch to the other way, especially not within the same StreamProcessor instance.

See also CompleteOutputBufferPartialSettings().

Request

Name	Type
`output_settings`	`StreamBufferPartialSettings`

Sync

On completion, all previous StreamProcessor calls have done what they're going to do server-side, except for processing of data queued using QueueInputPacket().

The main purpose of this call is to enable the client to wait until CloseCurrentStream() with release_input_buffers and/or release_output_buffers set to true to take effect, before the client allocates new buffers and re-sets-up input and/or output buffers. This de-overlapping of resource usage can be worthwhile for media buffers which can consume resource types whose overall pools aren't necessarily vast in comparison to resources consumed. Especially if a client is reconfiguring buffers multiple times.

Note that Sync() prior to allocating new media buffers is not alone sufficient to achieve non-overlap of media buffer resource usage system wide, but it can be a useful part of achieving that.

The Sync() transits the Output ordering domain and the StreamControl ordering domain, but not the InputData ordering domain.

This request can be used to avoid hitting kMaxInFlightStreams which is presently 10. A client that stays <= 8 in-flight streams will comfortably stay under the limit of 10. While the protocol permits repeated SetInputBufferSettings() and the like, a client that spams the channel can expect that the channel will just close if the server or the channel itself gets too far behind.

Request

<EMPTY>

Response

<EMPTY>

StreamSink

Defined in fuchsia.media/stream.fidl

Consumes a stream of packets. This interface is typically inherited along with StreamBufferSet to enable the transport of elementary streams from clients to services.

DiscardAllPackets

Discards packets previously sent via SendPacket or SendPacketNoReply and not yet released. The response is sent after all packets have been released.

Request

<EMPTY>

Response

<EMPTY>

DiscardAllPacketsNoReply

Discards packets previously sent via SendPacket or SendPacketNoReply and not yet released.

Request

<EMPTY>

EndOfStream

Indicates the stream has ended. The precise semantics of this method are determined by the inheriting interface.

Request

<EMPTY>

SendPacket

Sends a packet to the service. The response is sent when the service is done with the associated payload memory.

packet must be valid for the current buffer set, otherwise the service will close the connection.

Request

Name	Type
`packet`	`StreamPacket`

Response

<EMPTY>

SendPacketNoReply

Sends a packet to the service. This interface doesn't define how the client knows when the sink is done with the associated payload memory. The inheriting interface must define that.

packet must be valid for the current buffer set, otherwise the service will close the connection.

Request

Name	Type
`packet`	`StreamPacket`

StreamSource

Defined in fuchsia.media/stream.fidl

Produces a stream of packets. This interface is typically inherited along with StreamBufferSet to enable the transport of elementary streams from services to clients.

DiscardAllPackets

Request

<EMPTY>

Response

<EMPTY>

DiscardAllPacketsNoReply

Request

<EMPTY>

OnEndOfStream

Indicates that the stream has ended.

Response

<EMPTY>

OnPacketProduced

Delivers a packet produced by the service. When the client is done with the payload memory, the client must call ReleasePacket to release the payload memory.

Response

Name	Type
`packet`	`StreamPacket`

ReleasePacket

Releases payload memory associated with a packet previously delivered via OnPacketProduced.

Request

Name	Type
`packet`	`StreamPacket`

Usage2AudioConsumerFactory

Interface for creating audio consumers for local rendering.

CreateAudioConsumer

Creates an AudioConsumer, which is an interface for playing audio, given a usage value. Audio submitted to such a consumer is always rendered locally.

Request

Name	Type
`usage`	`AudioRenderUsage2`
`audio_consumer_request`	`server_end:AudioConsumer`

UsageAudioConsumerFactory

Interface for creating audio consumers for local rendering.

Removed: 31 Deprecated: 26

CreateAudioConsumer

Request

Name	Type
`usage`	`AudioRenderUsage`
`audio_consumer_request`	`server_end:AudioConsumer`

UsageGainListener

A protocol for watching changes to usage gain settings.

The channel will close when the device is not present.

OnGainMuteChanged

Called immediately on connection and afterward any time the usage gain setting changes.

Clients must respond to acknowledge the event. Clients that do not acknowledge their events will eventually be disconnected.

Note: This API does not have mute reporting implemented; muted is always false.

Request

Name	Type
`muted`	`bool`
`gain_dbfs`	`float32`

Response

<EMPTY>

UsageGainReporter

A protocol for setting up watchers of usage gain.

RegisterListener

Connects a listener to a stream of usage gain setting changes for usage on the device identified by device_token. Usage Gain is not set directly by any client; it is a translation of the usage volume setting for each device, summed with active muting/ducking gain adjustments.

Devices may map the same volume level to different dbfs, so a device_unique_id is needed to identify the device.

AudioDeviceEnumerator provides programmatic access to devices and their unique ids if it is necessary for a client to select an id at runtime.

DEPRECATED - Use RegisterListener2 and Usage2

Request

Name	Type
`device_unique_id`	`string:36`
`usage`	`Usage`
`usage_gain_listener`	`client_end:UsageGainListener`

RegisterListener2

Devices may map the same volume level to different dbfs, so a device_unique_id is needed to identify the device.

AudioDeviceEnumerator provides programmatic access to devices and their unique ids if it is necessary for a client to select an id at runtime.

Request

Name	Type
`device_unique_id`	`string:36`
`usage`	`Usage2`
`usage_gain_listener`	`client_end:UsageGainListener`

UsageReporter

A protocol for setting up watchers of audio usages.

Watch

DEPRECATED - Use Watch2, Usage2 and UsageWatcher2

Request

Name	Type
`usage`	`Usage`
`usage_watcher`	`client_end:UsageWatcher`

Watch2

Request

Name	Type
`usage`	`Usage2`
`usage_watcher`	`client_end:UsageWatcher2`

UsageWatcher

A protocol for listening to changes to the policy state of an audio usage.

User actions, such as lowering the volume or muting a stream, are not reflected in this API.

DEPRECATED - Use Watch2, Usage2 and UsageWatcher2

OnStateChanged

Called on first connection and whenever the watched usage changes. The provided usage will always be the bound usage; it is provided so that an implementation of this protocol may be bound to more than one usage.

Clients must respond to acknowledge the event. Clients that do not acknowledge their events will eventually be disconnected.

Request

Name	Type
`usage`	`Usage`
`state`	`UsageState`

Response

<EMPTY>

UsageWatcher2

OnStateChanged

Clients must respond to acknowledge the event. Clients that do not acknowledge their events will eventually be disconnected.

Request

Name	Type
`usage`	`Usage2`
`state`	`UsageState`

Response

<EMPTY>

STRUCTS

AacConstantBitRate

Field	Type	Description	Default
`bit_rate`	`uint32`	Bits per second	No default

AacEncoderSettings

Field	Type	Default
`transport`	`AacTransport`	No default
`channel_mode`	`AacChannelMode`	No default
`bit_rate`	`AacBitRate`	No default
`aot`	`AacAudioObjectType`	No default

AacTransportAdts

AAC inside ADTS

<EMPTY>

AacTransportLatm

AAC inside LATM

Field	Type	Description	Default
`mux_config_present`	`bool`	Whether MuxConfiguration stream element is present	No default

AacTransportRaw

Raw AAC access units.

<EMPTY>

ActivityReporter_WatchCaptureActivity2_Response

Field	Type	Description	Default
`active_usages`	`vector<AudioCaptureUsage2>:8`		No default

ActivityReporter_WatchRenderActivity2_Response

Field	Type	Description	Default
`active_usages`	`vector<AudioRenderUsage2>:8`		No default

AudioCompressedFormatAac

<EMPTY>

AudioCompressedFormatSbc

<EMPTY>

AudioCore_GetDbFromVolume2_Response

Defined in fuchsia.media/audio_core.fidl

Field	Type	Description	Default
`gain_db`	`float32`		No default

AudioCore_GetVolumeFromDb2_Response

Defined in fuchsia.media/audio_core.fidl

Field	Type	Description	Default
`volume`	`float32`		No default

AudioDeviceInfo

Defined in fuchsia.media/audio_device_enumerator.fidl

Field	Type	Default
`name`	`string`	No default
`unique_id`	`string`	No default
`token_id`	`uint64`	No default
`is_input`	`bool`	No default
`gain_info`	`AudioGainInfo`	No default
`is_default`	`bool`	No default

AudioGainInfo

Defined in fuchsia.media/audio_device_enumerator.fidl

Field	Type	Description	Default
`gain_db`	`float32`		No default
`flags`	`AudioGainInfoFlags`		No default

AudioStreamType

Defined in fuchsia.media/stream_type.fidl

Describes the type of an audio elementary stream.

Field	Type	Default
`sample_format`	`AudioSampleFormat`	No default
`channels`	`uint32`	No default
`frames_per_second`	`uint32`	No default

Compression

Defined in fuchsia.media/stream_type.fidl

Describes the compression applied to a stream. This type can be used in conjunction with AudioStreamType or VideoStreamType to represent a medium-specific compressed type.

Field	Type	Description	Default
`type`	`CompressionType`	The type of compression applied to the stream. This is generally one of the ENCODING values, though `AUDIO_ENCODING_LPCM` and `VIDEO_ENCODING_UNCOMPRESSED` must not be used, because those encodings are regarded as uncompressed.	No default
`parameters`	`vector<uint8>:8192?`	Type-specific, opaque 'out-of-band' parameters describing the compression of the stream.	No default

EncryptionPattern

EncryptionPattern

Pattern encryption utilizes a pattern of encrypted and clear 16 byte blocks over the protected range of a subsample (the encrypted_bytes of a SubsampleEntry). This structure specifies the number of encrypted data blocks followed by the number of clear data blocks.

Field	Type	Description	Default
`clear_blocks`	`uint32`		No default
`encrypted_blocks`	`uint32`		No default

Metadata

Defined in fuchsia.media/metadata.fidl

Field	Type	Description	Default
`properties`	`vector<Property>`		No default

Parameter

Parameter

Generic parameter.

We want to minimize use of this generic "Parameter" structure by natively defining as many stream-specific parameter semantics as we can.

Field	Type	Default
`scope`	`string`	No default
`name`	`string`	No default
`value`	`Value`	No default

PcmFormat

PcmFormat

PCM audio format details.

Field	Type	Default
`pcm_mode`	`AudioPcmMode`	No default
`bits_per_sample`	`uint32`	No default
`frames_per_second`	`uint32`	No default
`channel_map`	`vector<AudioChannelId>:16`	No default

Property

Defined in fuchsia.media/metadata.fidl

Field	Type	Description	Default
`label`	`string`		No default
`value`	`string`		No default

SbcEncoderSettings

Settings for an SBC Encoder.

SBC Encoders take signed little endian 16 bit linear PCM samples and return encoded SBC frames. SBC encoder PCM data in batches of sub_bands * block_count PCM frames. This encoder will accept PCM data on arbitrary frame boundaries, but the output flushed when EOS is queued may be zero-padded to make a full batch for encoding.

Field	Type	Description	Default
`sub_bands`	`SbcSubBands`		SbcSubBands.SUB_BANDS_8
`allocation`	`SbcAllocation`		SbcAllocation.ALLOC_LOUDNESS
`block_count`	`SbcBlockCount`		SbcBlockCount.BLOCK_COUNT_4
`channel_mode`	`SbcChannelMode`		No default
`bit_pool`	`uint64`	SBC bit pool value.	No default

StreamPacket

Defined in fuchsia.media/stream.fidl

Describes a packet consumed by StreamSink or produced by StreamSource.

Field	Type	Description	Default
`pts`	`int64`	Time at which the packet is to be presented, according to the presentation clock.	NO_TIMESTAMP
`payload_buffer_id`	`uint32`	ID of the payload buffer used for this packet. When this struct is used with `StreamBufferSet`, this field is the ID of a payload buffer provided via `StreamBufferSet.AddPayloadBuffer`. In that case, this value must identify a payload buffer in the current set. Other interfaces may define different semantics for this field.	No default
`payload_offset`	`uint64`	Offset of the packet payload in the payload buffer. This value plus the `payload_size` value must be less than or equal to the size of the referenced payload buffer.	No default
`payload_size`	`uint64`	Size in bytes of the payload. This value plus the `payload_offest` value must be less than or equal to the size of the referenced payload buffer.	No default
`flags`	`uint32`	An bitwise-or'ed set of flags (see constants below) describing properties of this packet.	0
`buffer_config`	`uint64`	The buffer configuration associated with this packet. The semantics of this field depend on the interface with which this struct is used. In many contexts, this field is not used. This field is intended for situations in which buffer configurations (i.e. sets of payload buffers) are explicitly identified. In such cases, the `payload_buffer_id` refers to a payload buffer in the buffer configuration identified by this field.	0
`stream_segment_id`	`uint64`	The stream segment associated with this packet. The semantics of this field depend on the interface with which this struct is used. In many contexts, this field is not used. This field is intended to distinguish contiguous segments of the stream where stream properties (e.g. encoding) may differ from segment to segment.	0

StreamProcessor_RemoveBuffer_Response

<EMPTY>

StreamType

Defined in fuchsia.media/stream_type.fidl

Describes the type of an elementary stream.

Field	Type	Description	Default
`medium_specific`	`MediumSpecificStreamType`	Medium-specific type information.	No default
`encoding`	`string:255`	Encoding (see constants below). This value is represented as a string so that new encodings can be introduced without modifying this file.	No default
`encoding_parameters`	`vector<uint8>?`	Encoding-specific parameters, sometimes referred to as 'out-of-band data'. Typically, this data is associated with a compressed stream and provides parameters required to decompress the stream. This data is generally opaque to all parties except the producer and consumer of the stream.	No default

SubpictureStreamType

Defined in fuchsia.media/stream_type.fidl

Describes the type of a subpicture elementary stream.

<EMPTY>

SubsampleEntry

SubsampleEntry

A subsample is a byte range within a sample consisting of a clear byte range followed by an encrypted byte range. This structure specifies the size of each range in the subsample.

Field	Type	Description	Default
`clear_bytes`	`uint32`		No default
`encrypted_bytes`	`uint32`		No default

TextStreamType

Defined in fuchsia.media/stream_type.fidl

Describes the type of a text elementary stream.

<EMPTY>

TimelineFunction

Defined in fuchsia.media/timeline_function.fidl

A TimelineFunction represents a relationship between a subject timeline and a reference timeline with a linear relation.

For example, consider a common use case in which reference time is the monotonic clock of a system and subject time is intended presentation time for some media such as a video.

reference_time is the value of the monotonic clock at the beginning of playback. subject_time is 0 assuming playback starts at the beginning of the media. We then choose a reference_delta and subject_delta so that subject_delta / reference_delta represents the desired playback rate, e.g. 0/1 for paused and 1/1 for normal playback.

Formulas

With a function we can determine the subject timeline value s in terms of reference timeline value r with this formula (where reference_delta > 0):

s = (r - reference_time) * (subject_delta / reference_delta) + subject_time

And similarly we can find the reference timeline value r in terms of subject timeline value s with this formula (where subject_delta > 0):

r = (s - subject_time) * (reference_delta / subject_delta) + referenc_time

Choosing time values

Time values can be arbitrary and our linear relation will of course be the same, but we can use them to represent the bounds of pieces in a piecewise linear relation.

For example, if a user performs skip-chapter, we might want to describe this with a TimelineFunction whose subject_time is the time to skip to, reference_time is now plus some epsilon, and delta ratio is 1/1 for normal playback rate.

Field	Type	Description	Default
`subject_time`	`int64`	Added: 31	No default
`reference_time`	`int64`	Added: 31	No default
`subject_delta`	`uint32`	Added: 31	No default
`reference_delta`	`uint32`	Added: 31	No default

VideoStreamType

Defined in fuchsia.media/stream_type.fidl

Describes the type of a video elementary stream.

Field	Type	Description	Default
`pixel_format`	`fuchsia.images/PixelFormat`	Deprecated: 13	No default
`color_space`	`ColorSpace`		No default
`width`	`uint32`	Dimensions of the video frames as displayed in pixels.	No default
`height`	`uint32`		No default
`coded_width`	`uint32`	Dimensions of the video frames as encoded in pixels. These values must be equal to or greater than the respective width/height values.	No default
`coded_height`	`uint32`		No default
`pixel_aspect_ratio_width`	`uint32`	The aspect ratio of a single pixel as frames are intended to be displayed.	No default
`pixel_aspect_ratio_height`	`uint32`		No default
`stride`	`uint32`	The number of bytes per 'coded' row in the primary video plane.	No default

VideoUncompressedFormat

VideoUncompressedFormat

Uncompressed video format details.

Field	Type	Default
`image_format`	`fuchsia.sysmem/ImageFormat_2`	No default
`fourcc`	`uint32`	No default
`primary_width_pixels`	`uint32`	No default
`primary_height_pixels`	`uint32`	No default
`secondary_width_pixels`	`uint32`	No default
`secondary_height_pixels`	`uint32`	No default
`planar`	`bool`	No default
`swizzled`	`bool`	No default
`primary_line_stride_bytes`	`uint32`	No default
`secondary_line_stride_bytes`	`uint32`	No default
`primary_start_offset`	`uint32`	No default
`secondary_start_offset`	`uint32`	No default
`tertiary_start_offset`	`uint32`	No default
`primary_pixel_stride`	`uint32`	No default
`secondary_pixel_stride`	`uint32`	No default
`primary_display_width_pixels`	`uint32`	No default
`primary_display_height_pixels`	`uint32`	No default
`has_pixel_aspect_ratio`	`bool`	false
`pixel_aspect_ratio_width`	`uint32`	1
`pixel_aspect_ratio_height`	`uint32`	1

Void

<EMPTY>

ENUMS

AacAudioObjectType strict

Type: uint32

Name	Value	Description
MPEG2_AAC_LC	`0`	MPEG-2 Low Complexity
MPEG4_AAC_LC	`1`	MPEG-4 Low Complexity

AacChannelMode strict

Type: uint32

Name	Value	Description
MONO	`0`
STEREO	`2`

AacVariableBitRate strict

Type: uint32

Variable bit rate modes. The actual resulting bitrate varies based on input signal and other encoding settings.

See https://wiki.hydrogenaud.io/index.php?title=Fraunhofer_FDK_AAC#Bitrate_Modes

Name	Value	Description
V1	`1`
V2	`2`
V3	`3`
V4	`4`
V5	`5`

AudioBitrateMode strict

Type: uint32

Name	Value	Description
UNSPECIFIED	`0`
CBR	`1`
VBR	`2`

AudioCaptureUsage strict

Type: uint32

Defined in fuchsia.media/audio_core.fidl

Usages annotating the purpose of the stream being used to capture audio. The AudioCaptureUsage is used by audio policy to dictate how audio streams interact with each other.

Name	Value	Description
BACKGROUND	`0`	Stream is used to capture audio while in the background. These streams may be active at any the time and are considered privileged. Example: Listening for Hotwords
FOREGROUND	`1`	Stream is intended to be used for normal capture functionality. Streams that are used for audio capture while the stream creator is in the foreground should use this. Example: Voice Recorder
SYSTEM_AGENT	`2`	Stream is for interaction with a system agent. This should only be used once a user has signalled their intent to have the interaction with an interested party. Examples: Assistant, Siri, Alexa
COMMUNICATION	`3`	Stream is intended to be used for some form of real time user to user communication. Voice/Video chat should use this.

AudioCaptureUsage2 flexible

Type: uint32

Defined in fuchsia.media/audio_core.fidl

Usages annotating the purpose of the stream being used to capture audio. The AudioCaptureUsage is used by audio policy to dictate how audio streams interact with each other.

Name	Value	Description
BACKGROUND	`0`	Stream is used to capture audio while in the background. These streams may be active at any the time and are considered privileged. Example: Listening for Hotwords
FOREGROUND	`1`	Stream is intended to be used for normal capture functionality. Streams that are used for audio capture while the stream creator is in the foreground should use this. Example: Voice Recorder
SYSTEM_AGENT	`2`	Stream is for interaction with a system agent. This should only be used once a user has signalled their intent to have the interaction with an interested party. Examples: Assistant, Siri, Alexa
COMMUNICATION	`3`	Stream is intended to be used for some form of real time user to user communication. Voice/Video chat should use this.

AudioChannelId strict

Type: uint32

AudioChannelId

Used in specifying which audio channel is for which speaker location / type.

TODO(b/529511595): Do we need/want more channel IDs than this?

Name	Value	Description
SKIP	`0`
LF	`1`
RF	`2`
CF	`3`
LS	`4`
RS	`5`
LFE	`6`
CS	`7`
LR	`8`
RR	`9`
END_DEFINED	`10`
EXTENDED_CHANNEL_ID_BASE	`1862270976`
MAX	`2147483647`

AudioOutputRoutingPolicy strict

Type: uint32

Defined in fuchsia.media/audio_core.fidl

DEPRECATED

Name	Value	Description
ALL_PLUGGED_OUTPUTS	`0`
LAST_PLUGGED_OUTPUT	`1`

AudioPcmMode strict

Type: uint32

Name	Value	Description
LINEAR	`0`
ALAW	`1`
MULAW	`2`

AudioRenderUsage strict

Type: uint32

Defined in fuchsia.media/audio_core.fidl

Usage annotating the purpose of the stream being used to render audio. An AudioRenderer's usage cannot be changed after creation. The AudioRenderUsage is used by audio policy to dictate how audio streams interact with each other.

Name	Value	Description
BACKGROUND	`0`	Stream is intended to be used for ambient or background sound. Streams that can be interrupted without consequence should use this.
MEDIA	`1`	Stream is intended to be used for normal functionality. Streams that are part of normal functionality should use this.
INTERRUPTION	`2`	Stream is intended to interrupt any ongoing function of the device. Streams that are used for interruptions like notifications should use this.
SYSTEM_AGENT	`3`	Stream is for interaction with a system agent. This should be used in response to a user initiated trigger.
COMMUNICATION	`4`	Stream is intended to be used for some form of real time user to user communication. Voice/Video chat should use this.

AudioRenderUsage2 flexible

Type: uint32

Defined in fuchsia.media/audio_core.fidl

Usage annotating the purpose of the stream being used to render audio. An AudioRenderer's usage cannot be changed after creation. The AudioRenderUsage2 value is used by audio policy to dictate how audio streams interact with each other. This enum expands the original (strict) enum |AudioRenderUsage| with additional stream types.

Name	Value	Description
BACKGROUND	`0`	Stream is intended to be used for ambient or background sound. Streams that can be interrupted without consequence should use this.
MEDIA	`1`	Stream is intended to be used for normal functionality. Streams that are part of normal functionality should use this.
INTERRUPTION	`2`	Stream is intended to interrupt any ongoing function of the device. Streams that are used for interruptions like notifications should use this.
SYSTEM_AGENT	`3`	Stream is for interaction with a system agent. This should be used in response to a user initiated trigger.
COMMUNICATION	`4`	Stream is intended to be used for some form of real time user to user communication. Voice/Video chat should use this.
ACCESSIBILITY	`5`	Stream is intended to be used for accessibility or other non-system- agent speech output, such as screen reader TTS.

AudioSampleFormat strict

Type: uint32

Defined in fuchsia.media/stream_type.fidl

Enumerates the supported audio sample formats.

Name	Value	Description
UNSIGNED_8	`1`	8-bit unsigned samples, sample size 1 byte.
SIGNED_16	`2`	16-bit signed samples, host-endian, sample size 2 bytes.
SIGNED_24_IN_32	`3`	24-bit signed samples in 32 bits, host-endian, sample size 4 bytes.
FLOAT	`4`	32-bit floating-point samples, sample size 4 bytes.

Behavior flexible

Type: uint32

Defined in fuchsia.media/audio_core.fidl

The behaviors applied to streams when multiple are active.

Name	Value	Description
NONE	`0`	Mix the streams.
DUCK	`1`	Apply a gain to duck the volume of one of the streams. (-14.0db)
MUTE	`2`	Apply a gain to mute one of the streams. (-160.0db)

CodecProfile flexible

Type: uint32

A list of permitted codec profiles. This list should be flexible since codecs can and will be added in the future. This can contain both video and audio profiles if needed.

Name	Value	Description
H264PROFILE_BASELINE	`0`
H264PROFILE_MAIN	`1`
H264PROFILE_EXTENDED	`2`
H264PROFILE_HIGH	`3`
H264PROFILE_HIGH10PROFILE	`4`
H264PROFILE_HIGH422PROFILE	`5`
H264PROFILE_HIGH444PREDICTIVEPROFILE	`6`
H264PROFILE_SCALABLEBASELINE	`7`
H264PROFILE_SCALABLEHIGH	`8`
H264PROFILE_STEREOHIGH	`9`
H264PROFILE_MULTIVIEWHIGH	`10`
VP8PROFILE_ANY	`11`
VP9PROFILE_PROFILE0	`12`
VP9PROFILE_PROFILE1	`13`
VP9PROFILE_PROFILE2	`14`
VP9PROFILE_PROFILE3	`15`
HEVCPROFILE_MAIN	`16`
HEVCPROFILE_MAIN10	`17`
HEVCPROFILE_MAIN_STILL_PICTURE	`18`
MJPEG_BASELINE	`19`

ColorSpace strict

Type: uint32

Defined in fuchsia.media/stream_type.fidl

Name	Value	Description
UNKNOWN	`0`
NOT_APPLICABLE	`1`
JPEG	`2`
HD_REC709	`3`
SD_REC601	`4`

Lc3FrameDuration flexible

Type: uint32

Added: HEAD

Name	Value	Description
D10_MS	`1`
D7P5_MS	`2`

Port flexible

Type: uint32

This enum indicates the StreamProcessor input or output. This is used in messages which can apply to either input or output. For example, the StreamProcessor.ParticipateInBufferAllocation message includes a port field.

Added: NEXT

Name	Value	Description
INVALID	`0`
INPUT	`1`
OUTPUT	`2`

SbcAllocation strict

Type: uint32

Name	Value	Description
ALLOC_LOUDNESS	`0`
ALLOC_SNR	`1`

SbcBlockCount strict

Type: uint32

Name	Value	Description
BLOCK_COUNT_4	`4`
BLOCK_COUNT_8	`8`
BLOCK_COUNT_12	`12`
BLOCK_COUNT_16	`16`

SbcChannelMode strict

Type: uint32

Name	Value	Description
MONO	`0`
DUAL	`1`
STEREO	`2`
JOINT_STEREO	`3`

SbcSubBands strict

Type: uint32

Name	Value	Description
SUB_BANDS_4	`4`
SUB_BANDS_8	`8`

StreamError strict

Type: uint32

StreamError

This error code encapsulates various errors that might emanate from a StreamProcessor server. It can be sent either as an OnStreamFailed event or as an epitaph for the channel.

Name	Value	Description
UNKNOWN	`1`	An internal error with an unspecified reason.
INVALID_INPUT_FORMAT_DETAILS	`2`	The client provided invalid input format details.
INCOMPATIBLE_BUFFERS_PROVIDED	`3`	The server received buffers that are not suitable for the operation to be performed. An example of this would be if a Decoder received output buffers that are too small to decode a frame into.
EOS_PROCESSING	`4`	Processing of input EOS (end of stream) failed, so the stream failed. Currently this can occur if a core codec watchdog fires while processing EOS.
DECODER_UNKNOWN	`16777217`	An internal decoder error with an unspecified reason.
DECODER_DATA_PARSING	`16777218`	Input data that can't be parsed. Only some parsing problems/errors are reported this way. Corrupt input data may be reported as other StreamError, or may not cause a StreamError.
ENCODER_UNKNOWN	`33554433`	An internal encoder error with an unspecified reason.
DECRYPTOR_UNKNOWN	`50331649`	An internal decryptor error with an unspecified reason.
DECRYPTOR_NO_KEY	`50331650`	The requested KeyId is not available for use by the Decryptor. The client may try again later if that key becomes available.

VideoColorSpace strict

Type: uint32

Deprecated: NEXT Added: 7

Name	Value	Description
INVALID	`0`

TABLES

AudioCompressedFormatCvsd

AudioCompressedFormatCvsd contains no fields for now since we will be using the parameter values recommended by Bluetooth Core Spec v5.3 section 9.2.

Added: HEAD

Ordinal	Field	Type	Description

AudioCompressedFormatLc3

AudioCompressedFormatLc3 contains no fields. The required parameters for setting up the decoder would be embedded as Codec_Specific_Configuration (Assigned Numbers section 6.12.5) in FormatDetails.oob_bytes.

Added: HEAD

Ordinal	Field	Type	Description

AudioConsumerStatus

Represents the status of the consumer. In the initial status, error and presentation_timeline are absent. The lead time fields are always present.

Ordinal	Field	Type	Description
1	`error`	`AudioConsumerError`	If present, indicates an error condition currently in effect. Absent if no error.
2	`presentation_timeline`	`TimelineFunction`	If present, indicates the current relationship between the presentation timeline and local monotonic clock, both in nanosecond units. If not present, indicates there is no relationship. Absent initially. 'Presentation timeline' refers to the `pts` (presentation timestamp) values on the packets. This timeline function can be used to determine the local monotonic clock time that a packet will be presented based on that packet's `pts` value.
3	`min_lead_time`	`uint64`	Indicates the minimum lead time in nanoseconds supported by this `AudioConsumer`. Or in other words, how small of a gap between the `media_time` provided to `AudioConsumer.Start` and the pts on the first packet can be. Values outside this range will be clipped.
4	`max_lead_time`	`uint64`	Indicates the maximum lead time in nanoseconds supported by this `AudioConsumer`. Or in other words, how large of a gap between the `media_time` provided to `AudioConsumer.Start` and the pts on the first packet can be. Values outside this range will be clipped.

CvsdEncoderSettings

Settings for CVSD Encoders. It contains no fields for now since we will be using the parameter values recommended by Bluetooth Core Spec v5.3 section 9.2.

Added: HEAD

Ordinal	Field	Type	Description

DecryptedFormat

DecryptedFormat

This describes the format of the decrypted content. It is required to be sent by the StreamProcessor server prior to the delivery of output packets. Currently, there is no additional format details for decrypted output.

Ordinal	Field	Type	Description
1	`ignore_this_field`	`bool`

EncryptedFormat

EncryptedFormat

The stream format details payload of a decrypting stream processor. This is a sparsely populated table to specify parameters necessary for decryption other than the data stream. It is only necessary to update fields if they changed, but not an error if the same value is repeated.

Ordinal	Field	Type	Description
3	`init_vector`	`InitVector`	`init_vector` is used in combination with a key and a block of content to create the first cipher block in a chain and derive subsequent cipher blocks in a cipher block chain. Usage: It is required to be set prior to the delivery of input packets to a decryptor. This may be changed multiple times during a data stream.
4	`subsamples`	`vector<SubsampleEntry>`	`subsamples` is used to identify the clear and encrypted portions of a subsample. Usage: For whole sample encryption, this parameter should not be sent. This may be changed multiple times during a data stream.
5	`pattern`	`EncryptionPattern`	`pattern` is used to identify the clear and encrypted blocks for pattern based encryption. Usage: This is not allowed for CENC and CBC1 and required for CENS and CBCS. If required, it must be set prior to the delivery of input packets to a decryptor. This may be changed multiple times during a data stream.
6	`scheme`	`string`	`scheme` specifies which encryption scheme to use, such as `fuchsia.media.ENCRYPTION_SCHEME_CENC`. Usage: It is required to be set prior to delivery of input packets. Changing the scheme mid-stream is only permitted in some scenarios. Once an encrypted scheme is selected for a stream, the scheme may only be set to `fuchsia.media.ENCRYPTION_SCHEME_UNENCRYPTED` or that same initial encrypted scheme. The scheme may be set to `fuchsia.media.ENCRYPTION_SCHEME_UNENCRYPTED` at any point.
8	`key_id`	`KeyId`	`key_id` identifies the key that should be used for decrypting subsequent data. Usage: It is required to be set prior to delivery of input packets to a decryptor. This may be changed multiple times during a data stream.

FormatDetails

FormatDetails

This describes/details the format on input or output of a StreamProcessor (separate instances for input vs. output).

Ordinal	Field	Type	Description
1	`format_details_version_ordinal`	`uint64`
2	`mime_type`	`string`
3	`oob_bytes`	`vector<uint8>`
4	`domain`	`DomainFormat`
5	`pass_through_parameters`	`vector<Parameter>`
6	`encoder_settings`	`EncoderSettings`	Instructs an encoder on how to encode raw data. Decoders may ignore this field but are entitled to rejected requests with this field set because it doesn't make sense.
7	`timebase`	`uint64`	The number of ticks of the timebase of input packet timestamp_ish values per second. The timebase is only used used for optional extrapolation of timestamp_ish values when an input timestamp which applies to byte 0 of the valid portion of the input packet does not correspond directly to byte 0 of the valid portion of any output packet. Leave unset if timestamp extrapolation is not needed, either due to lack of timestamps on input, or due to input being provided in increments of the encoder's input chunk size (based on the encoder settings and calculated independently by the client). Set if timestamp extrapolation is known to be needed or known to be acceptable to the client.
8	`profile`	`CodecProfile`	The codec profile for the given encoder or decoder. For encoders: This value is completely optional. A client may want to specify the codec profile used for protocol compatibility reasons (i.e. WebRTC). However if the value is not set then the the encoder is free to choose any supported codec profile. For decoders: This value is optional but providing it is best practice (at least of unencrypted streams). Container formats include the encoded profile and this should be provided to the decoder. Certain formats like VP9 and AV1 include the encoded profile in their elementary bitstream so it is possible for those decoders to figure out the profile the stream is encoded in. Regardless, clients should provide the encoded profile if possible.

H264EncoderSettings

Settings for H264 Encoders.

Ordinal	Field	Type	Description
1	`bit_rate`	`uint32`	Target bits per second for encoded stream. If omitted, interpreted as 200,000.
2	`frame_rate`	`uint32`	Target frames per second for encoded stream. If omitted, interpreted as 30.
3	`gop_size`	`uint32`	Number of pictures per keyframe. Setting to 0 will disable key frame encoding, except for if force_key_frame is set to true. If omitted, interpreted as 8.
4	`variable_frame_rate`	`bool`	Whether to enable frame rate adjustments in order to meet target bitrate. If omitted, interpreted as false.
5	`min_frame_rate`	`uint32`	Lowest frame rate allowed if `variable_frame_rate` is enabled. If omitted, interpreted as 10.
6	`force_key_frame`	`bool`	If true, next frame encoded will be a key frame. If omitted, interpreted as false.
7	`quantization_params`	`H264QuantizationParameters`	Allow customization of quantization parameters for encoding. Each frame submitted after setting this will use the new values. If omitted, no change from encoder defaults is made.

H264QuantizationParameters

Customization of h264 encoder parameters for macroblock quantization. The values can range from 0 to 51, with lower numbers indicating higher quality/bitrate. While encoders should support these fields if feasible, some encoders may ignore these fields. It's ok to not set this table, or not set some of the fields in this table, as encoders can determine their own defaults. If the targeted bitrate can't be achieved with the specified values, then the user should expect the resulting encoded stream bitrate to differ from the requested bitrate.

Ordinal	Field	Type	Description
1	`i_base`	`uint32`	Starting value for quantization of key frames.
2	`i_min`	`uint32`	Smallest allowed value for quantization of key frames.
3	`i_max`	`uint32`	Largest allowed value for quantization of key frames.
4	`p_base`	`uint32`	Starting value for quantization of predicted frames.
5	`p_min`	`uint32`	Smallest allowed value for quantization of predicted frames.
6	`p_max`	`uint32`	Largest allowed value for quantization of predicted frames.

HevcEncoderSettings

Settings for HEVC/H265 Encoders.

Ordinal	Field	Type	Description
1	`bit_rate`	`uint32`	Target bits per second for encoded stream. Defaults to 200,000 if omitted.
2	`frame_rate`	`uint32`	Target frames per second for encoded stream. Defaults to 30 if omitted.
3	`gop_size`	`uint32`	Number of pictures per keyframe. Defaults to 8 if omitted.

InputAudioCapturerConfiguration

Configuration for a capturer which will receive a stream from an input device.

Ordinal	Field	Type	Description
1	`usage`	`AudioCaptureUsage`	DEPRECATED - Use usage2 and AudioCaptureUsage2
2	`usage2`	`AudioCaptureUsage2`

Lc3EncoderSettings

Settings for LC3 Encoders. Contains parameters stated in LC3 Specification v1.0. Contains fields that are not represented by PcmFormat.

Added: HEAD

Ordinal	Field	Type	Description
1	`nbytes`	`uint16`	External byte count values to be used for the frame encoding of audio. According to the LC3 Specification v1.0, each audio channel could have different output byte size value, but for Fuchsia use case, we use the same `nbytes` value for all channels.
2	`frame_duration`	`Lc3FrameDuration`	Frame duration is used together with sampling frequency to determine the frame size.

LoopbackAudioCapturerConfiguration

Configuration for a capturer which will receive a loopback stream a system output.

Ordinal	Field	Type	Description

MSbcEncoderSettings

Settings for the mSBC encoder. There are no settings as the mSBC encoder uses the values required by HFP Spec v1.8.

Added: HEAD

Ordinal	Field	Type	Description

Packet

A Packet represents a chunk of input or output data to or from a stream processor.

stream processor output:

While the Packet is outstanding with the client via OnOutputPacket(), the stream processor will avoid modifying the referenced output data. After the client calls RecycleOutputPacket(packet_index), the stream processor is notified that the client is again ok with the referenced data changing.

stream processor input:

The client initially has all packet_index(es) available to fill, and later gets packet_index(s) that are again ready to fill via OnFreeInputPacket(). The client must not modify the referenced data in between QueueInputPacket() and OnFreeInputPacket().

Ordinal	Field	Type	Description
1	`header`	`PacketHeader`
2	`buffer_index`	`uint32`	Which buffer this packet refers to. A given in-flight interval of a packet refers to a specfic buffer using buffer_lifetime_ordinal (in PacketHeader) and buffer_index. A client should be prepared to handle output packets that refer to a buffer_lifetime_ordinal that's old (with buffer_index scoped to the old buffer_lifetime_ordinal). Streams that do this are not common (in this author's experience so far). Some clients may prefer to just ignore output packets with an old buffer_lifetime_ordinal, or notice the old buffer_lifetime_ordinal and fail the stream client-side. Clients that do want to fully handle such streams (for bitstream formats that can do this in the first place) can use RemoveBuffer completion to detect when each buffer of an old buffer_lifetime_ordinal will never again be referenced in any subsequent output packet. If the dimensions of the frame in the old buffer are different than the previous output frame, OnOutputFormat will be sent in between the output packets (as usual). A client should be prepared to handle (as in "not crash when") two or more output packets reference the same buffer, without the first output packet having been recycled back to the StreamProcessor yet. There is intentionally no guarantee that packets in-flight with the client will all refer to unique buffers, at least due to VP9 show_existing_frame, and possibly similar coding tools in other bitstream formats. Some layers above the direct client may not support this, in which case failing the stream client-side may be acceptable if the client has no need to support decode of streams that do this. More sophisticated translation strategies in this situation are outside the scope of this documentation. Clients intending to handle DRM-protected streams should keep in mind that copying frame contents with the CPU is likely not possible. To be clear, this author has never seen a DRM-protected stream that does this, and the only non-DRM-protected streams that this author has seen do this are test streams. Still, client authors should be aware that streams "in the wild" can potentially do this. RemoveBuffer is supported both when using dynamic buffers, and when not using dynamic buffers. When using RemoveBuffer with non-dynamic buffers, there is no AddBuffer involved or required, as SetInputBufferPartialSettings or SetOutputBufferPartialSettings was used to add all the buffers of the buffer_lifetime_ordinal. In the case of non-dynamic buffers however, the RemoveBuffer won't actaully start the removal of a buffer. But it will complete when the buffer is fully done removing. The packet has an associated buffer_lifetime_ordinal and buffer_index only while the packet is in-flight, not while the packet is free. When not using dynamic buffers, this value is the same as the sysmem buffer_index. When using dynamic buffers, this value is not the same as the sysmem buffer_index (other than by chance), nor is it guaranteed to be a small number. By default, for output buffers, this buffer_index value is guaranteed to be unique among buffers with the same buffer_lifetime_ordinal currently in-flight with the client. For decoding bitstream formats that can output the same buffer backing different output packets where the multiple packets are in the output queue at the same time (possibly with different timestamp_ish values), a client can send EnableSameOutputBufferConcurrentlyInFlight to indicate that the client is prepared to handle more than one output packet concurrently in flight referencing the same output buffer. Streams where this makes any difference are rare outside of bitstream format tests.
3	`stream_lifetime_ordinal`	`uint64`	The value 1 is the lowest permitted value after stream processor creation. Values sent by the client must be odd. Values must only increase. A stream_lifetime_ordinal represents the lifetime of a stream. All messages that are specific to a stream have the stream_lifetime_ordinal value and the value is the same for all messages relating to a given stream.
4	`start_offset`	`uint32`	Which part of the relevant buffer is this packet using. These are valid for input data that's in-flight to the stream processor, and are valid for output data from the stream processor. For compressed formats and uncompressed audio, the data in [start_offset, start_offset + valid_length_bytes) is the contiguously valid data referred to by this packet. For uncompressed video frames, FormatDetails is the primary means of determining which bytes are relevant. The offsets in FormatDetails are relative to the start_offset here. The valid_length_bytes must be large enough to include the full last line of pixel data, including the full line stride of the last line (not just the width in pixels of the last line). Despite these being filled out, some uncompressed video buffers are of types that are not readable by the CPU. These fields being here don't imply there's any way for the CPU to read an uncompressed frame.
5	`valid_length_bytes`	`uint32`	This must be > 0. The semantics for valid data per packet vary depending on data type as follows. uncompressed video - A video frame can't be split across packets. Each packet is one video frame. uncompressed audio - Regardless of float or int, linear or uLaw, or number of channels, a packet must contain an non-negative number of complete audio frames, where a single audio frame consists of data for all the channels for the same single point in time. Any stream-processor-specific internal details re. lower rate sampling for LFE channel or the like should be hidden by the StreamProcessor server implementation. compressed data input - A packet must contain at least one byte of data. See also stream_input_bytes_min. Splitting AUs at arbitrary byte boundaries is permitted, including at boundaries that are in AU headers. compressed data output - The stream processor is not required to fully fill each output packet's buffer.
6	`timestamp_ish`	`uint64`	This value is not strictly speaking a timestamp. It is an arbitrary unsigned 64-bit number that, under some circumstances, will be passed by a stream processor unmodified from an input packet to the exactly-corresponding output packet. For timestamp_ish values to be propagated from input to output the following conditions must be true: promise_separate_access_units_on_input must be true has_timestamp_ish must be true for a given input packet, to have that timestamp_ish value (potentially) propagate through to an output packet the StreamProcessor instance itself decides (async) that the input packet generates an output packet - if a given input never generates an output packet then the timestamp_ish value on the input will never show up on any output packet - depending on the characteristics of the input and output formats, and whether a decoder is willing to join mid-stream, etc this can be more or less likely to occur, but clients should be written to accommodate timestamp_ish values that are fed on input but never show up on output, at least to a reasonable degree (not crashing, not treating as an error).
7	`start_access_unit`	`bool`	If promise_separate_access_units_on_input (TODO(dustingreen): or any similar mode for output) is true, this bool must be set appropriately depending on whether byte 0 is or is not the start of an access unit. The client is required to know, and required to set this boolean properly. The server is allowed to infer that when this boolean is false, byte 0 is the first byte of a continuation of a previously-started AU. (The byte at start_offset is "byte 0".) If promise_separate_access_units_on_input is false, this boolean is ignored.
8	`known_end_access_unit`	`bool`	A client is never required to set this boolean to true. If promise_separate_access_units_on_input is true, for input data, this boolean must be false if this packet does not contain the last byte of the up to one AU contained or partially contained in this packet, and this boolean may be true if the last byte of the up-to-one AU is in this packet. A client delivering one AU at a time that's interested in the lowest possible latency via the decoder should set this boolean to true when it can be set to true. If promise_separate_access_units_on_input is false, this boolean is ignored. Some bitstream formats have the concept of an access unit delimiter or similar. A client wishing to achieve low latency may choose to inject correctly-formed access unit delimiters (or similar) after each access unit as an alternate way to signal to a decoder that the access unit has ended. However, such a client is encouraged to also set this field to true when possible, to ensure that all layers of a decoder notice. Injecting an access unit delimiter is allowed but unnecessary (for latency reduction purposes) if this is set to true (for a properly-functioning decoder). A client never needs to strip away any access unit delimiters that are potentially already present in the stream - there is no requirement that the setting of this field match the existence/non-existence of any access unit delimiter(s).
9	`key_frame`	`bool`	Used for compressed video packets. If not present should be assumed to be unknown. If false, indicates the packet is not part of a key frame. If true, indicates the packet is part of a key frame.

PacketHeader

PacketHeader

When referring to a free packet, we use PacketHeader alone instead of Packet, since while a packet is free it doesn't really have meaningful offset or length etc.

A populated Packet also has a PacketHeader.

Ordinal Field Type Description

1

buffer_lifetime_ordinal

uint64

This is which buffer configuration lifetime this header is referring to.

See [fuchsia.mediacodec/StreamBufferPartialSettings.buffer_lifetime_ordinal].

For QueueInputPacket(), a server receiving a buffer_lifetime_ordinal that isn't the current input buffer_lifetime_ordinal will close the channel.

For output packets, this can be an old buffer_lifetime_ordinal only if EnableOldOutputBuffers was sent by the client. See also RemoveBuffer to determine when it's safe to stop tracking old output buffers (only relevant to some video bitstream formats such as VP9, and only relevant to decoders). Streams that want to emit old output buffers are rare outside of bitstream format test streams.

2

packet_index

uint32

When using SetInputBufferPartialSettings or SetOutputBufferPartialSettings to allocate buffers, the valid range of packet_index is from 0..buffer_count-1, where buffer_count is the length of [fuchsia.sysmem2/BufferCollectionInfo.buffers], considering input and output separately.

When using ParticipateInBufferAllocation and AddBuffer to allocate and add buffers, the valid range of packet_index is unconstrained (any uint32 value). The client and server must still ensure that packet_index values aren't re-used until the other end has indicated the packet_index value is again available and can be re-used.

The number of concurrently in-flight packet_index values is limited per port (input vs. output) and buffer_lifetime_ordinal. If a client never puts the same input buffer in flight using more than one input packet concurrently, this inherently ensures the client will conform to this limit. Most clients can safely ignore this limit for output from the server. The limit is as follows: * When sending a packet, the sender must ensure that the number of concurrently in-flight packet_index values under the port (input or output) and buffer_lifetime_ordinal does not exceed the high-water-mark number of buffers that have concurrently existed from the server's point of view under the buffer_lifetime_ordinal so far. When using SetInputBufferPartialSettings, this is buffer_count as defined above.

When using AddBuffer, the high-water-mark value is determined slightly differently for input vs. output.
- For input, this high-water-mark value is the maximum value so far under the buffer_lifetime_ordinal of the total number of AddBuffer messages regarding the buffer_lifetime_ordinal minus the total number of RemoveBuffer messages regarding the buffer_lifetime_ordinal.
  - If a client never puts an input buffer in flight more than once concurrently using multiple different input packet_index values, then the client will inherently stay under this limit for input.
- For output, this high-water-mark value is the maximum so far under the buffer_lifetime_ordinal of the total number of AddBuffer messages regarding the buffer_lifetime_ordinal minus the total number of completed buffer removals under the buffer_lifetime_ordinal. This is without regard to which buffer removals had a corresponding RemoveBuffer completion.
  - If a client wishes to (optionally) validate this server behavior, the client can rely on the server to not complete a RemoveBuffer until after completion of buffer removal server-side (else the server is failing to conform to RemoveBuffer semantics or failing to conform to this limit).

For input, the client chooses an available packet_index value arbitrarily when sending an input packet with QueueInputPacket, and the client doesn't re-use the value until the server has sent OnFreeInputPacket with that packet_index value.

For output, the server chooses an available packet_index value arbitrarily (among all uint32 values) when sending an output packet with OnOutputPacket, and doesn't re-use the value until the client has sent RecycleOutputPacket with that packet_index value.

Available packet_index values can be queued in arbitrary order.

Both the client and server should validate the packet_index against the known bound (if any; see above) and disconnect if it's out of bounds.

The packet_index values don't imply anything about order of use of packets. The client should not expect the ordering to remain the same over time - the stream processor is free to hold on to an input or output packet for a while during which some other packet_index values may be re-used multiple times.

For a given properly-functioning StreamProcessor instance, output packet_index values will be unique among concurrently-outstanding packets.

Servers should validate that a client isn't double-using a packet and clients may similarly validate packet_index values from the server.

StreamBufferConstraints

This struct conveys the buffer_constraints_version_ordinal.

Historically this table conveyed more fields than it currently does, but those fields are all deprecated in favor of using sysmem instead.

There are separate instances of this struct for stream input and stream output.

Notes about fields:

For uncompressed video, separate and complete frames in their separate buffers (buffer-per-packet mode) are always a requirement.

Ordinal	Field	Type	Description
1	`buffer_constraints_version_ordinal`	`uint64`	This is a version number the server sets on the constraints to allow the server to determine when the client has caught up with the latest constraints sent by the server. The server won't emit output data until the client has configured output settings and buffers with a buffer_constraints_version_ordinal >= the latest buffer_constraints_version_ordinal that had buffer_constraints_action_required true. See buffer_constraints_action_required comments for more. A buffer_constraints_version_ordinal of 0 is not permitted, to simplify initial state handling. Other than 0, both odd and even version ordinals are allowed (in contrast to the stream_lifetime_ordinal, neither the client nor server ever has a reason to consider the latest version to be stale, so there would be no benefit to disallowing even values).
2	`default_settings`	`StreamBufferSettings`	DEPRECATED - Ignore. Use fuchsia.sysmem.BufferCollection.SetConstraints()
3	`per_packet_buffer_bytes_min`	`uint32`	DEPRECATED - Ignore. Use fuchsia.sysmem.BufferCollection.SetConstraints()
4	`per_packet_buffer_bytes_recommended`	`uint32`	DEPRECATED - Ignore. Use fuchsia.sysmem.BufferCollection.SetConstraints()
5	`per_packet_buffer_bytes_max`	`uint32`	DEPRECATED - Ignore. Use fuchsia.sysmem.BufferCollection.SetConstraints()
6	`packet_count_for_server_min`	`uint32`	DEPRECATED - Ignore. Use fuchsia.sysmem.BufferCollection.SetConstraints()
7	`packet_count_for_server_recommended`	`uint32`	DEPRECATED - Ignore. Use fuchsia.sysmem.BufferCollection.SetConstraints()
8	`packet_count_for_server_recommended_max`	`uint32`	DEPRECATED - Ignore. Use fuchsia.sysmem.BufferCollection.SetConstraints()
9	`packet_count_for_server_max`	`uint32`	DEPRECATED - Ignore. Use fuchsia.sysmem.BufferCollection.SetConstraints()
10	`packet_count_for_client_min`	`uint32`	DEPRECATED - Ignore. Use fuchsia.sysmem.BufferCollection.SetConstraints()
11	`packet_count_for_client_max`	`uint32`	DEPRECATED - Ignore. Use fuchsia.sysmem.BufferCollection.SetConstraints()
12	`single_buffer_mode_allowed`	`bool`	DEPRECATED - Ignore. Obsolete.
13	`is_physically_contiguous_required`	`bool`	DEPRECATED - Ignore. Use fuchsia.sysmem.BufferCollection.SetConstraints()
14	`buffer_count_for_server_current`	`uint32`	If a codec has `[fuchsia.mediacodec/DetailedCodecDescription.supports_dynamic_buffers]` set to true, this field must be set by the server and will be the minimum number of input or output buffers that the server needs to be able to concurrently use (or continue using) to guarantee forward progress. For video decoders, this number assumes a bitstream-standard-compliant stream. Any time the server doesn't have this many buffers available for the server's concurrent use (DPB size + 1), processing may pause, but will not fail - in this case the client can add more buffers with AddBuffers to get the processing to start again. When the DPB needs to reference more images, recycling an output buffer won't necessarily cause processing to start again, as the image may still be in the DPB, so the buffer can't be re-used yet despite being recycled by the client. For input, the codec may require this many buffers before the codec will process any input. Currently, for input, this value is static per StreamProcessor instance. The client is allowed to send QueueInput...() messages before this many input buffers have been added, but should not expect any output to necessarily be emitted until at least this many input buffers have been added. This also applies for QueueInputFormatDetails and QueueInputEndOfStream before any QueueInputPacket, for a newly starting stream, so the first two can be sent while there are still zero added input buffers so far. Codecs that only need one input buffer to make forward progress (non-continuous non-smooth forward progress still counts) should set this to one. Most clients will want to add a low number of additional buffers to keep the pipeline running smoothly, to avoid stalling processing while an OnFreeInputPacket is on its way from the codec to the client. Decoders that copy input data into a separate stream buffer (sometimes treated as circular) typically will only need one input buffer for forward progress to be possible, but most clients will still want to add a low number of additional input buffers for smoother thread scheduling, even if such a decoder core might be able to run at 100% utilization with a single input buffer. For video decoder output, if a client were to only `AddBuffers` this many buffers overall, every time `OnOutputPacket` happened, the decoder would be potentially stalled until `RecycleOutputPacket` is received at the server. For this reason, most clients will prefer to allocate a few more buffers, accounting for other parts of the pipeline (locally in the client or not) that may also need to "camp" on buffers for reasons unrelated to video decode, and possibly also a low number of additional buffers to keep the pipeline running smoothly, to avoid stalling decode while a free buffer on its way to being recycled or similar. For video decoder output, when set, this value will always be less than or equal to `[fuchsia.mediacodec/DetailedCodecDescription.dynamic_buffers_video_decoder_output_safe]`. This value can be updated by a new OnOutputConstraints from the server with action_required false. However, despite "action_required" being false, the client must still ensure that the number of buffers currently available for concurrent use by the codec is at least this many. If a client using dynamic buffers with a video decoder doesn't want to pay attention to this field, the client should still pay attention to `[fuchsia.mediacodec/DetailedCodecDescription.dynamic_buffers_video_decoder_output_safe]`. Or the client can instead use SetInputBufferPartialSettings/SetOutputBufferPartialSettings which has approximately the same (potentially adverse) effect in terms of overall buffer count (assuming a correctly operating pipeline). While these less complicated client implementation options might initially seem like they're wasting some memory, paying attention to buffer_count_for_server_current (in contrast to only dynamic_buffers_video_decoder_output_safe) only saves memory if the stream has the needed optional fields with values less than the max DPB size, which is unfortunately not guaranteed. In particular, h.264 streams without max_num_reorder_frames or max_dec_frame_buffering set in vui_parameters (or anywhere else) seem more common than not (to this author, among streams looked at so far), though this could depend strongly on which particular set of streams a client wants to decode. Client implementers may want to check some of the relevant streams for relevant bitstream header fields before caring about this field and the dynamic changes to the value conveyed by this field, for video decoders. Also, client implementers are free to use other strategies to save memory, such as being stingy with buffer counts but adding buffers dynamically and quickly if it seems like the codec might be starving for buffers (such as if downstream queued output buffers start to run low). This sort of strategy is up to the client, and it's on the client to get the timing right if it goes with that sort of strategy (not super easy to get perfectly correct / tuned). The platform makes no specific guarantee re. how quickly an additional buffer can be allocated, so a client attempting to run with fewer than dynamic_buffers_video_decoder_output_safe may encounter stream rendering jank or similar, including mid-stream. The platform makes no promises regarding the ability to avoid jank in this situation, such as if there's a mid-stream header that increases this value for output (not common). Layers above are encouraged to permit storing smaller images in larger buffers, so that mid-stream dimensions changes don't each need to re-allocate buffers. Instead, it's almost always better to just keep using existing larger buffers in this case, to avoid possibility of rendering jank due to allocating buffers not always being as fast as desired combined with output buffer count not necessarily having sufficient extra buffers to absorb the buffer reallocation in a timing sense. However, if a layer above really must re-allocate buffers when the image size changes and the needed buffer size to hold the images goes down, such a client must set force_new_buffers_for_new_dimensions passed to CreateDecoder. Otherwise a video decoder is permitted to start putting smaller images in the existing larger buffers, with OnOutputFormat message(s) indicating this. Added: NEXT
15	`size`	`fuchsia.math/SizeU`	Clients are free to ignore this field. This field is set iff `[fuchsia.mediacodec/DetailedCodecDescription.supports_dynamic_buffers]` is true and the indicating port is outputting uncompressed video. This field is provided for informational purposes, for clients that may want to know this information. However, the StreamProcessor and sysmem protocols work fine without the client ever reading this field. This is the required coded size, without any cropping down to display size. If the client gets directly involved as a sysmem participant, the client should not set constraints which are incompatible with this size, or allocation will fail. This field should not be taken to imply that a given buffer only has a single allowed image size. That is incorrect in general. Rather, this field indicates a size which will be covered by the required_min_size and required_max_size set by the StreamProcessor sysmem participant using token passed into SetInputBufferPartialSettings, SetOutputBufferPartialSettings, or ParticipateInBufferAllocation. Despite this size indicating a specific width and height, the allocated buffers may permit a range of image sizes (for video decoder output or video encoder input). Added: NEXT
16	`pixel_format`	`fuchsia.images2/PixelFormat`	Clients are free to ignore this field. This field is set iff `[fuchsia.mediacodec/DetailedCodecDescription.supports_dynamic_buffers]` is true and the indicating port is conveying uncompressed video. This field is provided for informational purposes, for clients that may want to know this information. However, the StreamProcessor and sysmem protocols work fine without the client ever reading this field. Among the pixel formats supported by the StreamProcessor server which can carry the full bit depth of the images, and which are broadly supported for carrying images of the needed bit depth, and assuming linear pixel_format_modifier, this is the pixel format that is most performant for the StreamProcessor considered in isolation. If multiple qualified formats are essentially the same performance, this should be the most common format (or chosen arbitrarily among roughly equally-common formats). Most StreamProcessor servers will hard code a particular format to put in this field for each supported bit depth. Currently, there is no anticipated need for a list here, but please don't hesitate to reach out if a non-test scenario is encountered which can only be addressed by having a list here (or by attempting several test allocations via ParticipateInBufferAllocation, or by adding a sysmem feature). In general, sysmem is not guaranteed to select this pixel format, since that depends on the full set of pixel formats supported by the StreamProcessor instance and other sysmem participants. If only the client and StreamProcessor are sysmem participants (such as in some tests), if the client requires this pixel format, buffer allocation can still succeed. Typical non-test clients will not need or want to constrain the sysmem allocation to only allow this pixel format, as that would be more likely to fail allocation in comparison to letting sysmem pick a format mutually supported by all participants. Added: NEXT

StreamBufferPartialSettings resource

Ordinal	Field	Type	Description
1	`buffer_lifetime_ordinal`	`uint64`	Each buffer is associated with a `buffer_lifetime_ordinal` value. There is a separate buffer_lifetime_ordinal for input vs. output. Values must be odd. Values must only increase (increasing by more than 2 is permitted). When using `SetInputBufferPartialSettings` or `SetOutputBufferPartialSettings`, re-use of the same value (considering input and output separately) for the new message is not allowed. Using these messages, all the buffers of a given buffer_lifetime_ordinal value are allocated as one sysmem buffer collection. When a new buffer_lifetime_ordinal starts, that also automatically starts un-registering and releasing any buffers associated with prior buffer_lifetime_ordinal values, but does not force them to complete un-registration (or release) if they're still used by the codec for reference purposes (VP9 can do this across a mid-stream dimensions switch). A buffer can still be output by the server (as indicated by an output packet) until un-registration completes. If the client wants the server to drop all buffers quickly, the client can send `CloseCurrentStream` with `release_input_buffers` and/or `release_output_buffers` set. This of course also stops decode of the current stream. Closing the `StreamProcessor` client end is another option.
2	`buffer_constraints_version_ordinal`	`uint64`	This value indicates which version of constraints the client is/was aware of so far. For input, this must always be 0 because constraints don't change for input (settings can change, but there's no settings vs current constraints synchronization issue on input). For output, this allows the server to know when the client is sufficiently caught up before the server will generate any more output. When there is no active stream, a client is permitted to re-configure buffers again using the same buffer_constraints_version_ordinal.
3	`single_buffer_mode`	`bool`	Deprecated: 23 DEPRECATED - Ignore. Obsolete.
4	`packet_count_for_server`	`uint32`	Deprecated: 23 DEPRECATED - Ignore. Use fuchsia.sysmem.BufferCollection.SetConstraints()
5	`packet_count_for_client`	`uint32`	Deprecated: 23 DEPRECATED - Ignore. Use fuchsia.sysmem.BufferCollection.SetConstraints()
6	`sysmem_token`	`client_end:fuchsia.sysmem/BufferCollectionToken`	Deprecated: 23
7	`sysmem2_token`	`client_end:fuchsia.sysmem2/BufferCollectionToken`	The client end of a BufferCollectionToken channel, which the StreamProcessor will use to deliver constraints to sysmem and learn of buffers allocated by sysmem. The client guarantees that the token is already known to sysmem (via BufferCollectionToken.Sync(), BufferCollection.Sync(), or BufferCollectionEvents.OnDuplicatedTokensKnownByServer()). Added: 23

StreamBufferSettings

Deprecated. Use SetStreamBufferPartialSettings() and StreamBufferPartialSettings instead.

DEPRECATED - Ignore. Use SetStreamBufferPartialSettings instead.

Ordinal	Field	Type	Description
1	`buffer_lifetime_ordinal`	`uint64`	DEPRECATED - Ignore. Use SetStreamBufferPartialSettings instead.
2	`buffer_constraints_version_ordinal`	`uint64`	DEPRECATED - Ignore. Use SetStreamBufferPartialSettings instead.
3	`packet_count_for_server`	`uint32`	DEPRECATED - Ignore. Use SetStreamBufferPartialSettings instead.
4	`packet_count_for_client`	`uint32`	DEPRECATED - Ignore. Use SetStreamBufferPartialSettings instead.
5	`per_packet_buffer_bytes`	`uint32`	DEPRECATED - Ignore. Use SetStreamBufferPartialSettings instead.
6	`single_buffer_mode`	`bool`	DEPRECATED - Ignore. Use SetStreamBufferPartialSettings instead.

StreamOutputConstraints

The stream-processor-controlled output configuration, including both StreamBufferConstraints for the output and FormatDetails for the output.

Ordinal Field Type Description

1

stream_lifetime_ordinal

uint64

A client which always immediately re-configures output buffers on receipt of OnOutputConstraints() with buffer_constraints_action_required true can safely ignore this field.

A client is permitted to ignore an OnOutputConstraints() message even with buffer_constraints_action_required true if the client knows the server has already been told to discard the remainder of the stream with the same stream_lifetime_ordinal or if this stream_lifetime_ordinal field is set to 0. The server is required to re-send needed output config via OnOutputConstraints() with new stream_lifetime_ordinal and buffer_constraints_action_required true, if the most recent completed server-side output config isn't what the server wants/needs yet for the new stream.

2

buffer_constraints_action_required

bool

When the buffer constraints are delivered, they indicate whether action is required. A false value here permits delivery of constraints which are fresher without forcing a buffer reconfiguration. If this is false, a client cannot assume that it's safe to immediately re-configure output buffers. If this is true, the client can assume it's safe to immediately configure output buffers once.

When not using dynamic buffers, a false value is deprecated and not sent by any known servers.

When using dynamic buffers, a false value can be sent by the server to update buffer_count_for_server_current, but that value will always be less than or equal to dynamic_buffers_video_decoder_output_safe. A client that prefers to just use dynamic_buffers_video_decoder_output_safe can ignore new StreamOutputConstraints if buffer_constraints_action_required false (assuming the client has no other reason to pay attention to the message). If a client is using buffer_count_for_server_current to potentially save a few buffers for streams that explicitly allow for that (unfortunately streams which specify a lower required DPB size are fairly rare AFAICT), the client must pay attention to new values of buffer_count_for_server_current even when buffer_constraints_action_required false, and ensure that the codec will have a buffer count available for the codec's use that's consistent with the new value. The codec will not set true or false depending on how many buffers the codec currently has available / free / seemingly for the codec's use, because there's no way for the codec to reliably detect that, as buffers can be opportunistically handed to the codec, yet actually reserved for potential use (and potential "camping") by the downstream pipeline (or similar).

A client is permitted to ignore buffer constraint versions which have buffer_constraints_action_required false. The server is not permitted to change buffer_constraints_action_required from false to true for the same buffer_constraints_version_ordinal.

For each configuration, a client must use new buffers, never buffers that were previously used for anything else, and never buffers previously used for any other StreamProcessor purposes. This rule exists for multiple good reasons, relevant to both mid-stream changes, and changes on stream boundaries. A client should just use new buffers each time.

When this is true, the server has already de-refed as many low-level output buffers as the server can while still performing efficient transition to the new buffers and will de-ref the rest asap. A Sync() is not necessary to achieve non-overlap of resource usage to the extent efficiently permitted by the formats involved.

If buffer_constraints_action_required is true, the server must not deliver more output data until after output buffers have been configured (or re-configured) by the client.

3

buffer_constraints

StreamBufferConstraints

StreamOutputFormat

Ordinal Field Type Description

1

stream_lifetime_ordinal

uint64

A client is permitted to ignore an OnOutputFormat() message even with buffer_constraints_action_required true if the client knows the server has already been told to discard the remainder of the stream with the same stream_lifetime_ordinal or if this stream_lifetime_ordinal field is set to 0. The server is required to re-send needed output config via OnOutputConstraints() with new stream_lifetime_ordinal and buffer_constraints_action_required true, if the most recent completed server-side output config isn't what the server wants/needs yet for the new stream.

The server is required to send an OnOutputFormat() before the first output packet of a stream.

2

format_details

FormatDetails

If format_details.format_details_version_ordinal changes, the client should inspect the new format details and determine if it must adjust to the new format. The server guarantees that if the format has changed, then format_details.format_details_version_ordinal will change, but a change to format_details.format_details_version_ordinal does not guarantee that the format details actually changed. Servers are strongly encouraged to not change format_details.format_details_version_ordinal other than before the first output data of a stream unless there is a real mid-stream format change in the stream. Unnecessary mid-stream format changes can cause simpler clients that have no need to handle mid-stream format changes to just close the channel. Format changes before the first output data of a stream are not "mid-stream" in this context - those can be useful for stream format detection / setup reasons.

Note that in case output buffers don't really need to be re-configured despite a format change, a server is encouraged, but not required, to set buffer_constraints_action_required false on the message that conveys the new format details. Simpler servers may just treat the whole output situation as one big thing and demand output buffer reconfiguration on any change in the output situation.

A client may or may not actually handle a new buffer_constraints with buffer_constraints_action_required false, but the client should always track the latest format_details.

An updated format_details is ordered with respect to emitted output packets, and applies to all subsequent packets until the next format_details with larger version_ordinal. A simple client that does not intend to handle mid-stream format changes should still keep track of the most recently received format_details until the first output packet arrives, then lock down the format details, handle those format details, and verify that any format_details.format_details_version_ordinal received from the server is the same as the locked-down format_details, until the client is done with the stream. Even such a simple client must tolerate format_details.format_details_version_ordinal changing multiple times before the start of data output from a stream (any stream - the first stream or a subsequent stream). This allows a stream processor to request that output buffers and output format be configured speculatively, and for the output config to be optionally adjusted by the server before the first data output from a stream after the server knows everything it needs to know to fully establish the initial output format details. This simplifies stream processor server implementation, and allows a clever stream processor server to guess it's output config for lower latency before any input data, while still being able to fix the output config (including format details) if the guess turns out to be wrong.

Whether the format_details.format_details_version_ordinal will actually change mid-stream is a per-stream-processor and per-stream detail that is not specified in comments here, and in most cases also depends on whether the format changes on the input to the stream processor. Probably it'll be fairly common for a client to use a format which technically supports mid-stream format change, but the client happens to know that none of the streams the client intends to process will ever have a mid-stream format change.

StreamProcessorAddBufferRequest resource

Ordinal	Field	Type	Description
1	`port`	`Port`	Must be set. Which port (input or output) the buffer is for.
2	`buffer_constraints_version_ordinal`	`uint64`	Typically this is the latest buffer_constraints_version_ordinal known to the client based on OnInputConstraints/OnOutputConstraints. Older values are not harmful to correctness. The server will ignore this message if the server has already created a later buffer_constraints_version_ordinal value that has action_required true. If the server ignores this message, a later RemoveBuffer for this buffer will complete quickly and without creating an error. A client is not required to ensure that every call to AddBuffer with the same buffer_lifetime_ordinal uses the same buffer_constraints_version_ordinal, but the client must ensure that all buffer_constraints_version_ordinal values which share the same buffer_lifetime_ordinal are within the same action_required true interval. Each such interval starts with a value that has action_required true, and lasts until but not including the next value with action_required true, or until the current value if there is not yet a next value with action_required true. If this would create a problem for a server, that server should set action_required true instead of false.
3	`buffer_lifetime_ordinal`	`uint64`	These values are scoped to input/output separately. The first message from the client with a new (higher) buffer_lifetime_ordinal value begins removing all buffers on the same port with lower buffer_lifetime_ordinal. However, until removal is complete, the older buffers can still be used by the codec. The AddBuffer message adds the buffer to this buffer_lifetime_ordinal.
4	`buffer_index`	`uint32`	The buffer_index values under a buffer_lifetime_ordinal must be unique among all added but not yet fully removed buffers. This value is initially allocated/specified by the client using this field, and then both the client and the server will then refer to a this added buffer uniquely using input vs output, buffer_lifetime_ordinal, and the buffer_index (all three parts are needed to unambiguously specify this buffer). See RemoveBuffer doc comments for how to safely remove a buffer and potentially re-use a buffer_index value. In contrast to using SetInputBufferPartialSettings/SetOutputBufferPartialSettings, when using AddBuffer and RemoveBuffer, the StreamProcessor buffer_index value has no relation to the sysmem buffer_index (under the sysmem buffer_collection_id). Instead, the StreamProcessor buffer_index is specified by this field, not by using sysmem's buffer_index. This way, two buffers from two sysmem collections having the same sysmem buffer_index (under separate buffer_collection_id(s)) can be specified unambiguously in StreamProcessor messages.
5	`buffer`	`handle<vmo>`	This is a VMO handle that the server can use to access the sysmem buffer. This VMO is required to have been provided by sysmem, and required to be compatible with the server's sysmem constraints. The buffer_constraints_version_ordinal check occurs first, and failure of that check just silently ignores the remainder of this message without checking the buffer field (buffer is just dropped). If the buffer_constraints_version_ordinal check passes, but `buffer` isn't a sysmem-provided VMO or isn't consistent with the server's sysmem constraints, the server will close the StreamProcessor server_end. The client can avoid triggering this failure case by allocating the buffers per the following. For video decoder input buffers, the client can allocate the buffer using sysmem, with participation of this or another instance of the same codec implementation, using ParticipateInBufferAllocation with Port.INPUT. The same StreamProcessor instance isn't required in this case, only the same codec implementation. If using a different StreamProcessor instance (but still from the same codec implementation), the client should take care to create the two StreamProcessor instances with the same requirements specified to CodecFactory (for example with secure_input_mode and secure_output_mode set the same way). Video decoder server implementations with supports_dynamic_buffers true are required to support allocating input buffers using one StreamProcessor instance then using them with a different StreamProcessor instance (in contrast, for output buffers, this is not required). For video decoder output buffers, this can be achieved by allocating the buffer using sysmem, with participation of the same StreamProcessor instance, using ParticipateInBufferAllocation with Port.OUTPUT. Servers should verify that this VMO is (a) provided by sysmem, and (b) meets the server's current sysmem constraints. The server can verify both (a) and (b) by checking the result of sysmem's CheckVmoConstraints before using this VMO. If that check fails, the server can close the StreamProcessor server_end.

StreamProcessorOnOutputTimestampHasNoOutputRequest

Ordinal	Field	Type	Description
1	`stream_lifetime_ordinal`	`uint64`	All servers will set this field. This is the `stream_lifetime_ordinal` value from the input `Packet` that will never generate any output packet.
2	`timestamp_ish`	`uint64`	All servers will set this field. This is the `timestamp_ish` value from the input `Packet` that will never generate any output packet under the `stream_lifetime_ordinal` having this `timestamp_ish` value.

StreamProcessorParticipateInBufferAllocationRequest resource

Ordinal	Field	Type	Description
1	`port`	`Port`	Which port (input or output) the buffer(s) are for. Must be set.
2	`buffer_constraints_version_ordinal`	`uint64`	For input, this field must be set to 1, at least for now. For output, this field must set to the latest known (to the client) buffer_constraints_version_ordinal value from the server. If the value is less than the latest buffer_constraints_version_ordinal created by the server which had action_reqiured true, then the server will set generic sysmem constraints (just min_buffer_count 1) and otherwise ignore this message. This is intended to allow allocation to succeed (to allow simpler client error handling for some clients); in this case a new buffer_constraints_version_ordinal will reach the client and the client can catch up to the latest. If this value is a future value, the server will drop the sysmem2_token which will cause allocation failure. Must be set.
3	`sysmem2_token`	`client_end:fuchsia.sysmem2/BufferCollectionToken`	This sysmem token is used by the server to inform sysmem directly of any applicable buffer constraints. These constraints are allowed to (and in some cases must) depend on how this StreamProcessor instance was created. For example, secure_input_mode/secure_output_mode during StreamProcessor creation causes the server to set secure_required for input/output respectively (as specified by Port.INPUT or Port.OUTPUT respectively). Must be set.
4	`buffer_lifetime_ordinal`	`uint64`	If set, and there's been at least one AddBuffer previously sent with this buffer_lifetime_ordinal on the same StreamProcessor instance which hasn't yet been removed server-side, the new buffer(s) will be allocated with identical SingleBufferSettings, or fail allocation. If any of the conditions in this paragraph aren't met, the effect is the same as if this field is not set. If this field is not set, the sysmem constraints set by the server will be suitable for a buffer of a new buffer_lifetime_ordinal, and the new buffer's SingleBufferSettings might not match any previous buffer's SingleBufferSettings. If the server has a current stream and `port` indicates the output port, the constraints set will allow the stream to be correctly processed across the switch to new buffer_lifetime_ordinal (assuming the client is using a new buffer_lifetime_ordinal and adds the new buffer(s) subsequently etc). If port indicates the input port, the constraints set will be generic for the input port regardless of any current stream. If this mechanism isn't used, the client must still ensure that all AddBuffer with a given StreamProcessor instance, port, and buffer_lifetime_ordinal have identical SingleBufferSettings. One way to do this is to allocate all the buffers of a buffer_lifetime_ordinal under a single sysmem buffer collection. Another way is for the client to tell sysmem directly that a new buffer collection must match the SingleBufferSettings of a prior collection using `[fuchsia.sysmem2/BufferCollectionConstraints.must_match_vmo]` client-side. It's not recommended for a client to attempt to force a new collecton's SingleBufferSettings to match an old collection's SingleBufferSettings without sysmem's help and without StreamProcessor's help via this field, because sysmem intentionally allows clients participating in the same collection to be using different versions of the sysmem interfaces, so the client may not have enough information to successfully force the SingleBufferSettings to match without using `must_match_vmo`, for example if the client is unaware of a new SingleBufferSettings field (or similar). The most common StreamProcessor server implementation (CodecImpl) will handle this field using `must_match_vmo` on behalf of the client as long as there's still at least one buffer of the buffer_lifetime_ordinal active in the server. The ability to dynamically add more buffers to an existing sysmem collection may be added to sysmem at some point; that would give the client another way to ensure identical SingleBufferSettings within a buffer_lifetime_ordinal, but only if/when all sysmem collection participants indicate support for a dynamic sysmem buffer collection. For output, setting this field is strongly recommended if a client is adding additional output buffers to an existing buffer_lifetime_ordinal using a new/different sysmem buffer collection, and setting this field is recommended for all output buffer allocations. For input, if a client is using the same StreamProcessor instance for this message and AddBuffer (which is itself recommended when feasible), use of this field is recommended for those input buffers. If set, this value must match the buffer_lifetime_ordinal later sent in AddBuffer. Clients should set this when feasible; see above.
5	`allow_single_buffer`	`bool`	If set to true, the server will leave all `buffer_count_*` fields un-set when sending SetConstraints to sysmem, and will set `min_buffer_count` to 1. If un-set or set to false, the server will fill out buffer count fields when sending SetConstraints to sysmem based on the current value of `buffer_count_for_server_current`, without any extra slack. The client can add slack as desired via its own retained token. Regardless of whether this field is set to true, it's up to the client to `AddBuffer` at least `buffer_count_for_server_current` buffers so that processing can make progress.

StreamProcessorRemoveBufferRequest resource

Ordinal Field Type Description

1

port

Port

Must be set. Which port (input or output) the buffer is for.

2

buffer_lifetime_ordinal

uint64

Scoped to input or output, this is the buffer_lifetime_ordinal under which the buffer_index is meaningful.

3

buffer_index

uint32

The buffer_index of the buffer previously established (and most recently established) by AddBuffer, or by SetInputBufferPartialSettings / SetOutputBufferPartialSettings. The scope of this value is under input vs. output and under the buffer_lifetime_ordinal value.

When using dynamic buffers, the buffer_index values that are between AddBuffer starting and RemoveBuffer completing (the active buffer_index values) must be unique for a given Port value and buffer_lifetime_ordinal value.

Clients using dynamic buffers may wish to not re-use a buffer_index value quickly upon completion of RemoveBuffer just to make debugging/diagnosing easier, but technically re-use of buffer_index in a subsequent AddBuffer is permitted as soon as RemoveBuffer completes (from the StreamProcessor point of view - the client may have its own reasons to not re-use that quickly). AddBuffer under an old (not current) buffer_lifetime_ordinal is never allowed.

UsageStateDucked

A state of audio usages in which a policy decision has been made to temporarily lower the volume of all streams with this usage.

Ordinal	Field	Type	Description

UsageStateMuted

A state of audio usages in which a policy decision has been made to temporarily mute the volume of all streams with this usage.

Ordinal	Field	Type	Description

UsageStateUnadjusted

A state of audio usages in which no policy actions are taken on any streams with the usage.

Ordinal	Field	Type	Description

UNIONS

AacBitRate strict

Ordinal	Variant	Type	Description
1	`constant`	`AacConstantBitRate`
2	`variable`	`AacVariableBitRate`

AacTransport flexible

Ordinal	Variant	Type
1	`raw`	`AacTransportRaw`
2	`latm`	`AacTransportLatm`
3	`adts`	`AacTransportAdts`

ActivityReporter_WatchCaptureActivity2_Result strict

Ordinal	Variant	Type	Description
1	`response`	`ActivityReporter_WatchCaptureActivity2_Response`
3	`framework_err`	`internal`

ActivityReporter_WatchRenderActivity2_Result strict

Ordinal	Variant	Type	Description
1	`response`	`ActivityReporter_WatchRenderActivity2_Response`
3	`framework_err`	`internal`

AudioCapturerConfiguration strict

Configuration for an audio Capturer.

Ordinal	Variant	Type	Description
1	`loopback`	`LoopbackAudioCapturerConfiguration`
2	`input`	`InputAudioCapturerConfiguration`

AudioCompressedFormat flexible

Ordinal	Variant	Type	Description
1	`aac`	`AudioCompressedFormatAac`
2	`sbc`	`AudioCompressedFormatSbc`
3	`cvsd`	`AudioCompressedFormatCvsd`	Added: HEAD
4	`lc3`	`AudioCompressedFormatLc3`	Added: HEAD

AudioConsumerError strict

Represents a AudioConsumer error condition.

Ordinal	Variant	Type	Description
1	`place_holder`	`Void`

AudioCore_GetDbFromVolume2_Result strict

Defined in fuchsia.media/audio_core.fidl

Ordinal	Variant	Type	Description
1	`response`	`AudioCore_GetDbFromVolume2_Response`
3	`framework_err`	`internal`

AudioCore_GetVolumeFromDb2_Result strict

Defined in fuchsia.media/audio_core.fidl

Ordinal	Variant	Type	Description
1	`response`	`AudioCore_GetVolumeFromDb2_Response`
3	`framework_err`	`internal`

AudioFormat strict

AudioFormat

Ordinal	Variant	Type	Description
1	`compressed`	`AudioCompressedFormat`
2	`uncompressed`	`AudioUncompressedFormat`

AudioUncompressedFormat strict

AudioUncompressedFormat

Ordinal	Variant	Type	Description
1	`pcm`	`PcmFormat`

CryptoFormat flexible

CryptoFormat

Crypto (encrypted or decrypted) format details.

Ordinal	Variant	Type	Description
1	`encrypted`	`EncryptedFormat`
2	`decrypted`	`DecryptedFormat`

DomainFormat strict

DomainFormat

Ordinal	Variant	Type
1	`audio`	`AudioFormat`
2	`video`	`VideoFormat`
3	`crypto`	`CryptoFormat`

EncoderSettings flexible

Settings for encoders that tell them how to encode raw formats.

Ordinal	Variant	Type	Description
1	`sbc`	`SbcEncoderSettings`
2	`aac`	`AacEncoderSettings`
3	`h264`	`H264EncoderSettings`
4	`hevc`	`HevcEncoderSettings`
5	`cvsd`	`CvsdEncoderSettings`	Added: HEAD
6	`lc3`	`Lc3EncoderSettings`	Added: HEAD
7	`msbc`	`MSbcEncoderSettings`	Added: HEAD

MediumSpecificStreamType strict

Defined in fuchsia.media/stream_type.fidl

A union of all medium-specific stream type structs.

Ordinal	Variant	Type
1	`audio`	`AudioStreamType`
2	`video`	`VideoStreamType`
3	`text`	`TextStreamType`
4	`subpicture`	`SubpictureStreamType`

StreamProcessor_RemoveBuffer_Result strict

Ordinal	Variant	Type	Description
1	`response`	`StreamProcessor_RemoveBuffer_Response`
3	`framework_err`	`internal`

Usage strict

Defined in fuchsia.media/audio_core.fidl

Ordinal	Variant	Type	Description
1	`render_usage`	`AudioRenderUsage`
2	`capture_usage`	`AudioCaptureUsage`

Usage2 flexible

Defined in fuchsia.media/audio_core.fidl

Ordinal	Variant	Type	Description
1	`render_usage`	`AudioRenderUsage2`
2	`capture_usage`	`AudioCaptureUsage2`

UsageState flexible

The state of audio policy enforcement on a stream or set of streams.

Ordinal	Variant	Type
1	`unadjusted`	`UsageStateUnadjusted`
2	`ducked`	`UsageStateDucked`
3	`muted`	`UsageStateMuted`

Value strict

Value

Generic "value" for use within generic "Parameter" struct.

Ordinal	Variant	Type
1	`bool_value`	`bool`
2	`uint64_value`	`uint64`
3	`int64_value`	`int64`
4	`string_value`	`string`
5	`bytes_value`	`vector<uint8>`

VideoCompressedFormat strict

VideoCompressedFormat

Compressed video format details.

Ordinal	Variant	Type	Description
1	`temp_field_todo_remove`	`uint32`	Temp field to make the compiler happy until we have at least one real field. Do not reference this field in code.

VideoFormat strict

VideoFormat

Video (compress or uncompressed) format details. In this context, "uncompressed" can include block-based image compression formats that still permit fairly fast random access to image data.

Ordinal	Variant	Type	Description
1	`compressed`	`VideoCompressedFormat`
2	`uncompressed`	`VideoUncompressedFormat`

BITS

AudioConsumerStartFlags strict

Type: uint32