[FFmpeg-devel] [PATCH] avcodec/mlpdec: Add decoding of object audio data

[Massimo Eynard]

On 24/03/2025 00:00, James Almer wrote:
> On 3/23/2025 6:47 PM, Hendrik Leppkes wrote:
>> On Sun, Mar 23, 2025 at 9:35 PM James Almer <jamrial at gmail.com> wrote:
>>>
>>> On 3/23/2025 4:33 PM, Massimo Eynard wrote:
>>>> On 23/03/2025 20:01, James Almer wrote:
>>>>> On 3/22/2025 2:49 PM, Massimo Eynard wrote:
>>>>>> This patch adds support for decoding the fourth MLP substream
>>>>>> which contains the 16-channel presentation used for Atmos
>>>>>> audio objects.
>>>>>>
>>>>>> By default only the first three substreams are decoded
>>>>>> unless the new extract_objects flag is enabled as the resulting
>>>>>> presentation contains audio object feeds instead of classic
>>>>>> loudspeaker feeds.
>>>>>>
>>>>>> As this introduces interpolation of primitive matrices, precision
>>>>>> has been increased to 2.18 fixed point. Therefore this requires
>>>>>> DSP code upgrade which has been done for C and x86 implementations
>>>>>> but not the ARM implementation.
>>>>>>
>>>>>> Adds two FATE tests using existing atmos.thd sample to reflect
>>>>>> changes.
>>>>>>
>>>>>> Signed-off-by: Massimo Eynard <eynard.massimo at gmail.com>
>>>>>> ---
>>>>>>     libavcodec/arm/mlpdsp_armv5te.S  |   2 +-
>>>>>>     libavcodec/arm/mlpdsp_init_arm.c |   3 +-
>>>>>>     libavcodec/mlp.h                 |  10 +-
>>>>>>     libavcodec/mlp_parse.c           |  31 ++-
>>>>>>     libavcodec/mlp_parse.h           |   1 +
>>>>>>     libavcodec/mlp_parser.c          |  11 +-
>>>>>>     libavcodec/mlpdec.c              | 389 +++++++++++++++++++++++++++----
>>>>>>     libavcodec/mlpdsp.c              |  50 +++-
>>>>>>     libavcodec/mlpdsp.h              |  25 ++
>>>>>>     libavcodec/x86/mlpdsp.asm        |  19 +-
>>>>>>     tests/fate/truehd.mak            |  10 +
>>>>>>     11 files changed, 476 insertions(+), 75 deletions(-)
>>>>>
>>>>> With atmos.thd i get:
>>>>>
>>>>>> [aist#0:0/truehd @ 00000209caf3ee00] Guessed Channel Layout: 7.1.4
>>>>>> Input #0, truehd, from '../samples/truehd/atmos.thd':
>>>>>>     Duration: N/A, start: 0.000000, bitrate: N/A
>>>>>>     Stream #0:0: Audio: truehd (Dolby TrueHD + Dolby Atmos), 48000 Hz, 7.1.4, s32 (24 bit)
>>>>>
>>>>> Which is unlikely to be correct. The file has 11 (or 12) objects, which is exported as 12 channels in an unspecified layout, and automatically assumed to be a 7.1.4 fixed layout.
>>>>>
>>>>
>>>> This is caused by `guess_input_channel_layout` (in `ffmpeg_demux.c`) which tries to assume a layout.
>>>> Would using `AV_CHANNEL_ORDER_CUSTOM` with all channels set to `AV_CHAN_UNKNOWN` (for unknown position, except LFE if present) be a better solution?
>>>
>>> Possibly, but it may make the stream undecodable unless you remap the
>>> channels (probably with a filter in the filterchain).
>>>
>>> Is there no better representation for the output? What are these 12
>>> channels the sample exports? 16 channels (as you say the MLP substream
>>> contains) would match Ambisonics 3rd order, but i assume that doesn't
>>> apply here, unless you should also be outputting something else.
>>>
>>
>> Its object-based audio. Every extra "channel" represents an audio
>> object at any arbitrary position in space, as defined by separate
>> metadata, which you are then supposed to mix together for your final
>> speaker configuration.
>> Typically, the "bed" channels (eg. the base 7.1) will contain audio
>> that doesn't require much localization information, music, background
>> noises, and the objects will contain audio which is more relevant to
>> have full spatial localization. A mixer is then tasked based on the
>> spatial metadata and knowledge of the physical speaker configuration
>> to mix the objects for ideal spatial representation.
>>
>> We don't have a channel layout that would identify this sort of setup
>> as of yet, nevermind a mixer that could actually deal with it, or even
>> exporting the metadata from the TrueHD stream, but baby steps I
>> suppose.
> 
> So we'd need a new layout (or pseudo-channel) where you set arbitrary coordinates? Sort of like what Apple defined in https://developer.apple.com/documentation/coreaudiotypes/audio-channel-coordinates
> 

That would be the best approach I guess. Atmos in TrueHD is the same as in E-AC-3 (except for the audio coding part of course) which is described in section 4 of ETSI TS 103 420.
In the specification, the audio "channels" for objects are called "audio object essences" which are supplied to a mixer/renderer alongside the metadata.
Section 4.4 describes the metadata interface.

However the purpose of this patch is only to decode the essences. What should I do for now?

>>
>> FWIW, taking all this into account, I fully agree that it should by
>> default output the 7.1 representation that everyone can actually
>> process, because the bed+objects representation is rather unexpected
>> and unhandleable at this time.
> Agree.
> 
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".