[FFmpeg-devel] Integrating the mod engine into FFmpeg - what is the best design approach?

[Sebastian Vater]

Michael Niedermayer a ?crit :
> On Sat, Jul 17, 2010 at 07:41:35PM +1000, Peter Ross wrote:
>   
>> On Sat, Jul 17, 2010 at 10:07:43AM +0200, Michael Niedermayer wrote:
>>     
>>> On Sat, Jul 17, 2010 at 03:04:17PM +1000, Peter Ross wrote:
>>>       
>>>> On Fri, Jul 16, 2010 at 04:15:26PM +0200, Vitor Sessak wrote:
>>>>         
>>>>> On 07/16/2010 03:46 PM, Sebastian Vater wrote:
>>>>>           
>>>>>> Hello dears!
>>>>>>
>>>>>> I had a discussion with Vitor and Stefano about the best way to
>>>>>> integrate the mod engine into FFmpeg.
>>>>>>             
>>>>>> I see just one disadvantage here, simply extraction of metadata and
>>>>>> removing it, passing the rest to AVPacket requires parsing of all module
>>>>>> files twice and also manipulating them (correct the offsets, etc.). This
>>>>>> would require duplicate code in the demuxer and decoder, which I would
>>>>>> like to avoid, if possible.
>>>>>>             
>>>> The same problem exists for image metadata. Simple solution would be
>>>> to permit codecs to get and set metadata.
>>>>         
>>> id like to point out that i wanted AVMetadata in lavc when it was written
>>> already but people where against it.
>>>
>>> so maybe its time to move it over ...
>>>       
>> I feel that this should not hold up Sebastian's project.
>>     
>
> i did not suggest that it should
>   

Yes, we can do this later. Anyway, in the meanwhile I made some thoughts
on a more precise generic integration plan for lavseq.
As you currently see, I have a directory lavseq containing:
avsequencer.h (the connection to rest of FFmpeg).

To give an overview of the AVSequencer:
At root, we have avsequencer.h, i.e. the AVSequencerContext which is the
only structure linking to remaining of FFmpeg.

root:
AVSequencerContext contains a list of modules (module.h and module
handling is implemented in module.c), the playback handler and a list of
available mixing engines.

depth 1:
AVSequencerModule contains songs, instruments, keyboard definitions,
arpeggio definitions and envelope structures.

depth 2:
AVSequencerSong contains tracks, an order list which references the
track numbers being played for each channel (since the sequencer is
internally track instead of pattern based to allow different speeds.

AVSequencerInstrument contains samples, which can be assigned using the
keyboard definition. For example, I tell the instrument to use sample
number 1 for C-5 but number 2 instead for C-6.
Instruments also determine how envelopes are used (you can assign them
to vibrato, tremolo, volume handling, etc.).

AVSequencerEnvelope contains the actual envelope data and also it
properties like loop points.

AVSequencerKeyboard contains the octave/note -> sample mapping for all
notes from C-0 to B-9 which are 120 entries (10 octaves * 12 notes per
octave).

AVSequencerArpeggio is mostly like AVSequencerEnvelope with the
difference you can specify a custom arpeggio layout and the structure is
designed for that.

depth 3:
AVSequencerSample contains the sample loop points, auto vibrato
envelopes and also the PCM data (the PCM data should later be obtained
by the lavc, so you can also directly use ogg/mp3/flac/wav/etc.). It
also contains a reference to AVSequencerSynth if it is a programmable
synth sound.

depth 4:
AVSequencerSynth contains a list of "machine code" instructions for
programming the synth sound "DSP", a symbol table for human-readability
and properties like initial variables (16 general purpose registers).

Hierarchy overview:
SequencerContext (avsequencer.[hc])
    Module (module.[hc])
        Song (song.[hc])
            Track (track.[hc])
                TrackData
                    TrackDataEffect
            OrderList (order.[hc])
                OrderData
        Instrument (instr.[hc])
            Sample (sample.[hc])
                SynthSound (synth.[hc])
        Envelope (instr.[hc])
        Keyboard (instr.[hc])
        Arpeggio (instr.[hc])
    Mixer (allmixers.c, mixer.[hc])
        Null mixer (null_mix.[hc])
        Low quality PCM mixer (lq_mix.[hc])
        High quality PCM mixer (hq_mix.[hc])
        FUTURE: OPL2/3 (AdLib/etc.) FM synthesizer
                        SID chip FM (as found in C64) synthesizer
                        Floating point mixers
    Player (player.[hc])

Open discussion points are:
1. Best way of integration into rest of FFmpeg
2. How to do the mixer so it finally can playback channels in a way like:
    Channel 0: raw PCM
    Channel 1: ogg file
    Channel 2: mp3 file
    [..]
    Channel 62: ImpulseTracker instrument file
    Channel 63: GUS patch sound
    Channel 64: flac file
    (while all these can be played with different loop points, volumes,
    panning positions, etc.)
3. What's the best approach for writing demuxers / decoders using the
    AVSequencer

-- 

Best regards,
                   :-) Basty/CDGS (-: