fuchsia.mediacodec

Added: 7

PROTOCOLS

CodecFactory

Defined in fuchsia.mediacodec/codec_factory.fidl

The purpose of the media::CodecFactory interface is to create media::StreamProcessor instances for decoders and encoders.

The interface methods don't attempt to homogenize all codec types, preferring to have a separate dedicated message for decoders.

AttachLifetimeTracking

AttachLifetimeTracking:

Attach an eventpair endpoint to the next Create{X}(), so that the codec_end will be closed when the number of buffers allocated reaches 'buffers_remaining'. Multiple eventpair endpoints can be attached per create, with an enforced limit of CODEC_FACTORY_LIFETIME_TRACKING_EVENTPAIR_PER_CREATE_MAX.

The lifetime signalled by this event is intended to track all resources used by the codec, including sysmem-allocated buffers created internally by the codec. The sysmem buffer collections visible to the client, for input and output buffers, are not included in the resources tracked here, because those can be tracked separately via fuchsia.sysmem.BufferCollection.AttachLifetimeTracking(). It is permitted to send a duplicate of codec_end to both this AttachLifetimeTracking() and also to fuchsia.sysmem.BufferCollection.AttachLifetimeTracking(). The ZX_EVENTPAIR_PEER_CLOSED will happen when both/all lifetimes are fully over. This conveniently avoids needing multiple separate async waits by the client.

In the case of server process crashes, or failure of a codec to plumb codec_end to sysmem, ZX_EVENTPAIR_PEER_CLOSED signalled on the peer of codec_end may occur shortly before all resources are freed.

A maximum of CODEC_FACTORY_LIFETIME_TRACKING_EVENTPAIR_PER_CREATE_MAX calls to AttachLifetimeTracking() are allowed before any Create{X}(). There is no way to cancel an attach short of closing the CodecFactory channel. Closing the client end of the eventpair doesn't subtract from the number of pending attach(es). For this reason, it can be good to only send attach message(s) immediately before the relevant Create{X}(), when it's known by the client that both the attach message(s) and the Create{X}() messages will be sent.

Closing the client's end doesn't result in any action by the server. If the server listens to events from the client end at all, it is for debug logging only.

The server intentionally doesn't "trust" any bits signalled by the client. This mechanism intentionally uses only ZX_EVENTPAIR_PEER_CLOSED which can't be triggered early, and is only triggered when all handles to codec_end are closed. No meaning is associated with any of the other signal bits, and clients should functionally ignore any other signal bits on either end of the eventpair or its peer.

The codec_end may lack ZX_RIGHT_SIGNAL or ZX_RIGHT_SIGNAL_PEER, but must have ZX_RIGHT_DUPLICATE (and must have ZX_RIGHT_TRANSFER to transfer without causing CodecFactory channel failure).

Request

NameType
codec_end handle<eventpair>

CreateDecoder

CreateDecoder:

decoder_params - See CreateDecoder_Params comments for required and optional parameters for creating a decoder.

decoder - a Codec.NewRequest() which will hopefully be connected to a Codec server, or the Codec channel will get closed if no suitable codec can be found. We don't return any additional Codec-specific status here because finding the Codec is allowed to be fully async, so we don't necessarily yet know on return from this method which Codec will be selected, if any.

Rough sequence to create a decoder:

factory = ConnectToEnvironmentService(CodecFactory); CreateDecoder_Params params; [fill out params] CreateDecoder(params, decoder_request);

See use_media_decoder code for more detail.

Request

NameType
decoder_params CreateDecoder_Params
decoder server_end<fuchsia.media/StreamProcessor>

CreateEncoder

CreateEncoder:

encoder_params - See CreateEncoder_Params comments for required and optional parameters for creating a decoder.

encoder - a Codec.NewRequest() which will hopefully be connected to a Codec server, or the Codec channel will get closed if no suitable codec can be found. We don't return any additional Codec-specific status here because finding the Codec is allowed to be fully async, so we don't necessarily yet know on return from this method which Codec will be selected, if any.

Request

NameType
encoder_params CreateEncoder_Params
encoder server_end<fuchsia.media/StreamProcessor>

GetDetailedCodecDescriptions

A client should call |GetDetailedCodecDescriptions()| to get a list of codecs supported either by software implementations or by underlying hardware.

Added: 11

Request

<EMPTY>

Response

NameType
payload CodecFactoryGetDetailedCodecDescriptionsResponse

OnCodecList

Driver-based local CodecFactory(s) will send this once shortly after the main CodecFactory connects to the driver-local CodecFactory.

For now, the main CodecFactory will not send this.

A SW-based local CodecFactory(s) will not send this event.

Each codec in the list must be separately-described, for clean aggregation.

Deprecated: 11 Added: 7

Response

NameType
codecs vector<CodecDescription>[256]

STRUCTS

CodecDescription

Defined in fuchsia.mediacodec/codec_factory.fidl

Deprecated.

Rather than listening for OnCodecList, clients needing codec information prior to just requesting a codec with CodecFactory.CreateDecoder or CodecFactory.CreateEncoder should instead use GetDetailedCodecDescriptions to get the DetailedCodecDescription table instead, which has per-profile-entry equivalents of all these fields. Clients with no real need for codec information prior to requesting a codec can call CodecFactory.CreateDecoder or CodecFactory.CreateEncoder with relevant requirements set in that request, and then StreamProcessor.Sync to see if a codec was actually created successfully.

In contrast to OnCodecList which uses FIDL structs (due to ordering of historical events), GetDetailedCodecDescriptions uses FIDL tables so is not expected to need to be deprecated, since we can add new table fields as needed, and gradually deprecate old table fields if appropriate, without deprecating the whole thing.

Per-codec servers do not need to fill out this struct or send OnCodecList, as the main CodecFactory will construct the OnCodecList info from the GetDetailedCodecDescriptions info, so that each codec can (optionally) stop sending OnCodecList immediately, rather than having to wait for all clients to stop listening to OnCodecList, which may take a while. All codecs do need to implement GetDetailedCodecDescriptions however.

Deprecated: 11 Added: 7

FieldTypeDescriptionDefault
codec_type CodecType

Decoder or encoder.

No default
mime_type string[256]

The mime type of the compressed format. For decoders this is the mime type of the input. For encoders, this is the mime type of the output.

No default
can_stream_bytes_input bool

For each of these fields, the default is the most-capable setting, but if a codec doesn't support the most-capable behavior, then the codec must override the default.

true
can_find_start bool true
can_re_sync bool true
will_report_all_detected_errors bool true
is_hw bool true
split_header_handling bool true

ENUMS

CodecType strict

Type: uint32

Defined in fuchsia.mediacodec/codec_factory.fidl

NameValueDescription
0
1

SecureMemoryMode strict

Type: uint32

Defined in fuchsia.mediacodec/codec_factory.fidl

Whether buffers need to be secure. If not specified, the default is OFF.

This enum may have additional values added later; code handling this type should be written with this in mind. For example, in C++, having a "default" case in any switch statement on this type will avoid compilation warnings/errors when a new value is added.

NameValueDescription
0
1

TABLES

CodecFactoryGetDetailedCodecDescriptionsResponse

Defined in fuchsia.mediacodec/codec_factory.fidl

OrdinalFieldTypeDescription
codecs vector<DetailedCodecDescription>[256]

CreateDecoder_Params

Defined in fuchsia.mediacodec/codec_factory.fidl

OrdinalFieldTypeDescription
input_details fuchsia.media/FormatDetails

Input mime type for a decoder.

The recognized mime types for now: video/h264 video/vp9 audio/aac input_details.oob_bytes must be an AudioSpecificConfig() as defined by AAC spec. audio/sbc input_details.oob_bytes must be Codec Specific Information Elements for SBC as defined by the A2DP spec.

promise_separate_access_units_on_input bool

This must be true in order for the client to be permitted to put a timestamp on an input packet, which is in turn required to get any timestamps on any output packets.

It is always legal to provide separate Access Units (henceforth AUs) to a decoder, but this boolean must be true for a decoder to accept and propagate timestamp values.

This must be true when creating a video encoder, or the CodecFactory channel will close.

If not set, interpreted as false.

require_can_stream_bytes_input bool

Require that the selected codec be capable of accepting input where AUs are not separated into separate packets.

This does not imply that the decoder can find the start of the first AU; for that see require_can_find_start. This does not imply that the decoder can re-sync on its own if the stream data is damaged; for that see require_can_re_sync.

If both promise_separate_access_units_on_input and require_can_stream_bytes_input are true, the CodecFactory channel will close.

If this is false, the client must feed separate AUs on the fuchsia.ui.input. This must be false for a video encoder, and if true the CodecFactory channel will close.

Unless a client demands a decoder capable of taking concatenated AUs (require_can_stream_bytes_input true), the client must feed a decoder separate AUs. This means the client cannot have parts of two separate AUs in the same packet, unless require_can_stream_bytes_input is true.

If not set, interpreted as false.

require_can_find_start bool

A decoder is allowed to be capable of streaming bytes but not capable of searching for the start of the first usable AU. To require both, set both require_can_stream_bytes_input and require_can_find_start. Setting require_can_find_start without require_can_stream_bytes_input is invalid.

With require_can_stream_bytes_input true but require_can_find_start false, the client must start the first packet with the start of an AU, but can send a stream of bytes after that.

If not set, interpreted as false.

require_can_re_sync bool

On problematic input data, all decoders are expected to at least be able to close the channel rather than getting stuck in a failed and/or broken state.

A decoder returned from a request with require_can_re_sync is potentially able to handle damaged input without closing the Codec channel. Such a Codec is encouraged, but not required, to also satisfy requirements of require_report_all_detected_errors.

If not set, interpreted as false.

require_report_all_detected_errors bool

Sometimes a client would rather fail an overall use of a decoder than fail to notice data corruption. For such scenarios, the client can specify require_report_all_detected_errors. For any codec returned from a request with require_report_all_detected_errors set, on detection of any input data corruption the codec will report in one or more of these ways:

  • closing the Codec channel
  • OnStreamFailed()
  • error_detected_before
  • error_detected_during

If false, a codec may silently skip past corrupted input data.

No decoder can detect all corruption, because some corruption can look like valid stream data. This requirement is only to request a codec that is written to attempt to detect and report input stream corruption.

This flag is not intended to be 100% bulletproof. If a client needs robust assurance that all detectable stream corruption is always detected, this flag is not enough of a guarantee to achieve that. Since some stream corruption is inherently non-detectable in any case, such a client should consider using stronger techniques upstream to ensure that corruption can be detected with the needed probability very close to 1.

This flag being true doesn't imply anything about whether the codec will discard damaged data vs. producing corresponding damaged output. Only that the codec will set error_detected_* bools to true when appropriate.

Regardless of this setting, not all timestamp_ish values provided on input are guaranteed to show up on output.

If not set, interpreted as false.

require_hw bool

If true, require that the returned codec is HW-accelerated. See also require_sw.

If not set, interpreted as false.

permit_lack_of_split_header_handling bool

permit_lack_of_split_header_handling

This field is a temporary field that will be going away.

TODO(dustingreen): Remove this field once we're down to zero codecs with problems handling split headers.

By default, a Codec instance is required to handle "split headers", meaning that a client is allowed to deliver parts of an AU one byte at a time, including parts near the beginning of the AU, and the codec is required to tolerate and handle that properly. However, unfortunately not all codecs properly support split headers. If a client is willing to permit such a codec to be used, the client can set this to true. Clients are not encouraged to set this, but setting it may be necessary to find a codec for some formats for now. If a client sets this to true, the client should deliver data of each AU with many contiguous non-split bytes from the start of each AU. The client is not strictly required to deliver one AU at a time, only to ensure that either all the AU bytes are in a single packet or that many bytes at the start of each AU are in a single packet.

The specification for how a client should use this and how a client should behave if setting this to true is intentionally vague, because lack of support for header splitting is not ideal, and is expected to be temporary, and all codecs should handle split headers in the long run. The main intent of this field is to avoid giving an innocent client using default value of false here a codec that can't properly handle split headers. This is not an attempt at a mechanism to fully work around a codec that doesn't handle split headers.

If not set, interpreted as false.

secure_output_mode SecureMemoryMode

If set to ON, the decoder must support secure buffers on output, and must reject non-secure buffers on output.

If set to OFF or not set, the created decoder will reject secure buffers on output by closing the StreamProcessor channel.

If secure_input_mode ON, secure_output_mode must also be ON.

secure_input_mode SecureMemoryMode

If set to ON, the decoder must support secure buffers on input and must reject non-secure buffers on input.

If set to OFF or not set, the created decoder will reject secure buffers on input by closing the StreamProcessor channel.

If secure_input_mode ON, secure_output_mode must also be ON.

require_sw bool

If true, require that the returned codec is entirely SW-based, not HW-accelerated (other than possibly using vector CPU instructions). This can be useful for testing purposes or other special scenarios, but is not recommended for performance-sensitive scenarios. Also, some builds may lack a SW-based decoder for some formats. See also require_hw.

If not set, interpreted as false.

CreateEncoder_Params

Defined in fuchsia.mediacodec/codec_factory.fidl

Parameters used to request an encoder.

OrdinalFieldTypeDescription
input_details fuchsia.media/FormatDetails

The format of the uncompressed input data.

This field should be a raw mime_type (e.g. 'video/raw') and uncompressed format details for the encoder to use when reading buffers.

To be elibigible an encoder must support the input format.

require_hw bool

If true, require that the returned codec is HW-accelerated.

If not set, interpreted as false.

DecoderProfileDescription

Defined in fuchsia.mediacodec/codec_factory.fidl

Specification of the supported parameters of a given video decoder.

Fields in this table with the same name as fields in CodecDescription have the same meaning as the fields in the CodecDescription struct when the corresponding field in this table is set.

When a corresponding field is un-set, each of these fields is interpreted according to the corresponding doc comment on the field in this table (doc comments on fields in the CodecDescription struct re. struct field defaults are not relevant to the interpretation of un-set fields of this table).

For video decoders, the following is always required:

  • Handling split input payload (distinct from split header), when an input frame is too large for a single input buffer. The next portion (possibly the remainder) of the timstamped input chunk is delivered in the next input packet.

For audio decoders, the following is always required:

  • Concatenation of multiple compressed audio chunks via the input is always permitted.
Added: 11

OrdinalFieldTypeDescription
profile fuchsia.media/CodecProfile

The codec profile that this decoder supports. If the client wants to use this profile, the requirements specified in this table must be adhered to.

min_image_size fuchsia.math/SizeU

The minimum image size this decoder supports for the given |profile|. Decoders must set this field, and this field must specify a size that is >= the min size defined for the profile in the codec spec.

This size refers to the pixel layout in memory, not the display_rect which can be smaller than the fuchsia.Images2.ImageFormat.size / fuchsia.sysmem.ImageFormat2.coded_width/height.

max_image_size fuchsia.math/SizeU

The maximum image size this decoder supports for the given |profile|.

Decoders must set this field, and this field must specify a size that is <= the max size defined for the profile in the codec spec.

This size refers to the pixel layout in memory, not the display_rect which can be smaller than the fuchsia.Images2.ImageFormat.size / fuchsia.sysmem.ImageFormat2.coded_width/height.

By setting this field, a decoder is not required to fail to decode a stream that specifies a size that is larger than the profile of the stream would normally allow. The decoder may or may not fail to decode a stream which is not a listed profile or not within the size bounds of a listed profile.

allow_encryption bool

This |profile| entry can apply to encrypted input data. If require_encryption is false or unset, this |profile| entry can also apply to unencrypted input data.

This will be un-set (even when allow_input_protection or require_input_protection are true) until Fuchsia supports decryption as part of decode (in contrast to it being a separate step involving protected memory in between). When allow_encryption is false/un-set but allow_input_protection is true, a client setting up DRM decode should set up decryption as a separate step prior to decode with protected memory in between the decrypt and decode.

require_encryption bool

This |profile| entry applies only when input data is encrypted. This |profile| entry does not apply for unencrypted input data. Un-set is interepted as false.

If this is set to true and there is no profile with require_encryption false in the same DetailedCodecDescription, then this decoder only supports encrypted input.

This will be un-set until Fuchsia supports decryption as part of decode (even when allow_input_protection or require_input_protection are true). See also allow_encryption, allow_input_protection, require_input_protection.

allow_input_protection bool

This |profile| entry can apply when input data delivered via protected memory. Whether to protect input and which protected "heap" to use (if protecting input) is determined during sysmem constraints aggregation and via DRM mechanisms. See also require_input_protection, allow_encryption, require_encryption.

require_input_protection bool

This |profile| entry applies only when input data is delivered via protected memory. Whether to protect input and which protected "heap" to use (if protecting input) is determined during sysmem constraints aggregation, and by separate DRM mechanisms.

If this is set to true and there is no profile with require_input_protection false in the same DetailedCodecDescription, then this decoder only supports protected input.

Output protection is negotiated separately during output buffer constraints aggregation in sysmem, and via DRM mechanisms.

See also allow_input_protection, allow_encryption, require_encryption.

can_stream_bytes_input bool

If set to true, the decoder can handle an input chunk payload (containing compressed data) that's split across a packet boundary (between header and payload, or between payload bytes), and the decoder can also handle more than one input chunk containing compressed input data in a single input packet. See also split_header_handling to determine if the decoder can also handle split headers.

Un-set means false.

While this field always indicates whether it's ok to split an input chunk across a packet boundary or not, and this field being set always implies it's ok to have a single input packet with bytes from more than one input chunk including when the input chunks contain compressed input data, there is some additional somewhat-subtle meaning that differs between video and audio decoders.

Audio decoders are always required to permit more than one input chunk in a single input packet, including mulitple input chunks which each contain compressed input data (not just prepended "context" headers), regardless of this field being set or unset. For audio decoders, this field only indicates whether splitting an input chunk across packets is allowed.

Audio decoders shouldn't set split_header_handling to true unless they also set can_stream_bytes_input to true, since allowing splitting header bytes is meaningless unless it's allowed to split an input chunk.

For video decoders, if this field is un-set or false, the decoder may not be able to handle bytes of more than one input chunk in a single input packet. However, for all decoders (both video and audio) it's always ok for preceding "context" headers such as the h264 SPS and PPS headers to be in the same input packet as the following input chunk containing compressed input data such as an h264 slice (for example). A video decoder is always required to permit continuation of an input chunk in the next packet, with the split between header byte and compressed input data byte or between two compressed input data bytes (but not necessarily between two header bytes; see also split_header_handling).

It's expected to be fairly common for a decoder to set can_stream_bytes_input to true, but leave split_header_handling un-set or set split_header_handling to false, due to parsing limitations in the HW, FW, or driver. Decoders which set can_stream_bytes_input false or leave can_stream_bytes_input un-set may be forcing input data to be re-packed in some scenarios. Profiles which set allow_encryption are encouraged to also set can_stream_bytes_input if feasible, as re-packing input data can be more difficult in some scenarios involving encryption.

can_find_start bool

If set to true, the decoder can scan forward at the start of a stream to find the start of the first fully-present input chunk, even if the input data starts at a byte that's in the middle of a chunk.

For both video and audio decoders, setting this field to true also indicates the ability to handle (skip or only partly use) any input chunks that are not possible to decode (or not possible to fully decode) due to lack of referenced prior data.

If un-set or false, the decoder may not be able to scan forward to sync up with the input stream unless the input stream starts with the beginning of a suitable chunk.

can_re_sync bool

If set to true, the decoder can resynchronize with the input stream (eventually) despite a missing input chunk, and can handle partial input chunks without failing the stream or the StreamProcessor control connection.

will_report_all_detected_errors bool

If set to true, the decoder makes efforts to indicate partial or detected-as-missing or detected-as-corrupt input chunks by setting error_detected_before and/or error_detected_during as appropriate. Even when this field is set to true, it is worth noting that in general it is not possible for a decoder to detect all possible data corruptions, as codecs generally aren't inclined to include error detection bits within the stream. This is not a replacment for a real (as in robust with high probability against random bit flips and bit insertion/deletion) data integrity check.

split_header_handling bool

If set to true, the decoder can handle the bytes of any header being split across packet boundaries. If false or un-set, the decoder requires all header bytes of a header to be within the same packet.

Starting a new packet at the spec-defined boundary between two headers that are officially in separate input chunks is always permitted.

For purposes of this field, headers like the h264 SPS and PPS headers are considered separate input chunks, and so such headers that have their own input chunk can be delivered in a separate packet regardless of the setting of this field, and a subsequent h264 slice header can be in yet another packet.

It is always permissible regardless of the setting of this field for "context" headers (such as h264 SPS and PPS headers) to be delivered in the same packet as the subsequent chunk which conveys compressed image data (such as an h264 slice).

Splitting a single header between codec_oob_bytes and the stream data is never supported (as in, never to be relied on by clients), not even when this field is set to true.

See also can_stream_bytes_input.

Video decoders with can_stream_bytes_input true but split_header_handling false can handle continuing an chunk in the next packet when split between header and payload or between payload bytes, and can handle bytes of more than one input chunk in a single input packet, but cannot handle splitting a header across packets. Video decoders with can_stream_bytes_input false and split_header_handling true cannot tolerate bytes of more than one input chunk in a single packet, but can tolerate continuation of an input chunk in a following packet regardless of where the split occurs (such as in the middle of an h264 SPS, PPS, or slice header).

Audio decoders shouldn't set split_header_handling to true unless they also set can_stream_bytes_input to true, since for an audio decoder, allowing splitting header bytes is meaningless (for audio decoders) unless also allowing splitting an input chunk.

DetailedCodecDescription

Defined in fuchsia.mediacodec/codec_factory.fidl

Clients needing codec information prior to just requesting a codec with CodecFactory.CreateDecoder or CodecFactory.CreateEncoder should use GetDetailedCodecDescriptions to get this table, which has details re. the codec and the profile entries supported by the codec.

Clients with no real need for codec information prior to requesting a codec can simply use CodecFactory.CreateDecoder or CodecFactory.CreateEncoder with relevant requirements set in that request, and then call StreamProcessor.Sync (round trip) to see if a codec was created successfully.

Added: 11

OrdinalFieldTypeDescription
codec_type CodecType

Decoder or encoder.

mime_type string[256]

The mime type of the compressed format. For decoders this is the mime type of the input. For encoders, this is the mime type of the output.

is_hw bool

If this decoder/encoder uses underlying hardware to perform its operations.

profile_descriptions ProfileDescriptions

A list of profile descriptions that describe what codec profiles this encoder/decoder supports along with requirements for using each profile.

EncoderProfileDescription

Defined in fuchsia.mediacodec/codec_factory.fidl

Added: 11

OrdinalFieldTypeDescription
profile fuchsia.media/CodecProfile

UNIONS

ProfileDescriptions strict

Defined in fuchsia.mediacodec/codec_factory.fidl

Added: 11
OrdinalVariantTypeDescription
decoder_profile_descriptions vector<DecoderProfileDescription>[256]

A list of |DecoderProfileDescription| that describe what codec profiles this decoder supports along with requirements that must be adhered to if the client is to use the decoder. The CodecFactory guarantees to the client that each |DecoderProfileDescription| within |decoder_profile_descriptions| will have an unique |profile|.

encoder_profile_descriptions vector<EncoderProfileDescription>[256]

CONSTANTS

NameValueTypeDescription
CODEC_FACTORY_CODEC_LIST_SIZE_MAX 256 uint32

Constrain the length of the vector indicated via OnCodecList, and constrain the max items in any vector within the response from GetDetailedCodecDescriptions. The overall enforced limit for codec caps information is the max channel message size however, since nested vectors each of max size could otherwise exceed the max channel message size.

CODEC_FACTORY_LIFETIME_TRACKING_EVENTPAIR_PER_CREATE_MAX 16 uint32

More than this many calls to AttachLifetimeTracking() without a Create{X}() call will result in CodecFactory channel closing from the server end.

CODEC_FACTORY_MAX_MIME_TYPE_LENGTH 256 uint32

Constrain the mime_type to a size that won't cause problems.