[FFmpeg-devel] Allow interrupt callback for AVCodecContext
Don Moir
donmoir at comcast.net
Mon Jan 6 22:08:48 CET 2014
----- Original Message -----
From: "Ronald S. Bultje" <rsbultje at gmail.com>
To: "FFmpeg development discussions and patches" <ffmpeg-devel at ffmpeg.org>
Sent: Monday, January 06, 2014 3:59 PM
Subject: Re: [FFmpeg-devel] Allow interrupt callback for AVCodecContext
> Hi,
>
> On Mon, Jan 6, 2014 at 2:48 PM, Don Moir <donmoir at comcast.net> wrote:
>
>>
>> ----- Original Message ----- From: "Ronald S. Bultje" <rsbultje at gmail.com>
>> To: "FFmpeg development discussions and patches" <ffmpeg-devel at ffmpeg.org>
>> Sent: Monday, January 06, 2014 9:52 AM
>>
>> Subject: Re: [FFmpeg-devel] Allow interrupt callback for AVCodecContext
>>
>>
>> Hi,
>>>
>>> On Mon, Jan 6, 2014 at 3:56 AM, Don Moir <donmoir at comcast.net> wrote:
>>>
>>>
>>>> ----- Original Message ----- From: "Ronald S. Bultje" <
>>>> rsbultje at gmail.com>
>>>> To: "FFmpeg development discussions and patches" <
>>>> ffmpeg-devel at ffmpeg.org>
>>>> Sent: Monday, January 06, 2014 8:44 AM
>>>>
>>>> Subject: Re: [FFmpeg-devel] Allow interrupt callback for AVCodecContext
>>>>
>>>>
>>>> Hi,
>>>>
>>>>>
>>>>> On Mon, Dec 16, 2013 at 1:21 AM, Don Moir <donmoir at comcast.net> wrote:
>>>>>
>>>>> ----- Original Message ----- From: "Don Moir" <donmoir at comcast.net>
>>>>>
>>>>>>
>>>>>> To: "FFmpeg development discussions and patches" <
>>>>>>> ffmpeg-devel at ffmpeg.org
>>>>>>> >
>>>>>>> Sent: Monday, December 16, 2013 2:03 AM
>>>>>>> Subject: Re: [FFmpeg-devel] Allow interrupt callback for
>>>>>>> AVCodecContext
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ----- Original Message ----- From: "Ronald S. Bultje" <
>>>>>>>
>>>>>>>> rsbultje at gmail.com>
>>>>>>>> To: "FFmpeg development discussions and patches" <
>>>>>>>> ffmpeg-devel at ffmpeg.org>
>>>>>>>> Sent: Wednesday, January 01, 2014 11:14 AM
>>>>>>>> Subject: Re: [FFmpeg-devel] Allow interrupt callback for
>>>>>>>> AVCodecContext
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Mon, Dec 16, 2013 at 1:30 AM, Don Moir <donmoir at comcast.net>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ----- Original Message ----- From: "Ronald S. Bultje" <
>>>>>>>>>
>>>>>>>>>> rsbultje at gmail.com>
>>>>>>>>>> To: "FFmpeg development discussions and patches" <
>>>>>>>>>> ffmpeg-devel at ffmpeg.org>
>>>>>>>>>> Sent: Wednesday, January 01, 2014 10:52 AM
>>>>>>>>>> Subject: Re: [FFmpeg-devel] Allow interrupt callback for
>>>>>>>>>> AVCodecContext
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Dec 16, 2013 at 2:07 AM, Don Moir <donmoir at comcast.net>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> For now just seeing what you think about this. This is about
>>>>>>>>>>> thread
>>>>>>>>>>> based
>>>>>>>>>>>
>>>>>>>>>>> apps where this makes sense.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> When attempting to do a new seek or waiting to close a video, I
>>>>>>>>>>>> find
>>>>>>>>>>>> that
>>>>>>>>>>>> I am waiting on avcodec_decode_video2 to return before I can
>>>>>>>>>>>> continue.
>>>>>>>>>>>> Depending on machine and video, this wait time can be up to about
>>>>>>>>>>>> 50ms
>>>>>>>>>>>> but
>>>>>>>>>>>> normally wait time about 5 to 20 ms or so.
>>>>>>>>>>>>
>>>>>>>>>>>> You might say 'so what' and I would agree for a simple player app
>>>>>>>>>>>> it
>>>>>>>>>>>> does
>>>>>>>>>>>> not make that much difference.
>>>>>>>>>>>>
>>>>>>>>>>>> If you are trying to stay on a timeline, or in case of scrubbing,
>>>>>>>>>>>> or
>>>>>>>>>>>> for
>>>>>>>>>>>> editing apps, this wait time does make a difference. That is, you
>>>>>>>>>>>> can not
>>>>>>>>>>>> move on until avcodec_decode_video2 has returned.
>>>>>>>>>>>>
>>>>>>>>>>>> I can pretty much seek instantly to zero for any file except
>>>>>>>>>>>> when I
>>>>>>>>>>>> have
>>>>>>>>>>>> to wait on avcodec_decode_video2 if that be the case.
>>>>>>>>>>>>
>>>>>>>>>>>> For me, it's normal for any intense process like decoding to be
>>>>>>>>>>>> interruptible but this is not the case for AVCodecContext in
>>>>>>>>>>>> ffmpeg.
>>>>>>>>>>>> This
>>>>>>>>>>>> is strange, don't you think?
>>>>>>>>>>>>
>>>>>>>>>>>> For AVFormatContext you have:
>>>>>>>>>>>>
>>>>>>>>>>>> typedef struct AVIOInterruptCB {
>>>>>>>>>>>> int (*callback)(void*);
>>>>>>>>>>>> void *opaque;
>>>>>>>>>>>> } AVIOInterruptCB;
>>>>>>>>>>>>
>>>>>>>>>>>> I would use this model for AVCodecContext but change naming to:
>>>>>>>>>>>>
>>>>>>>>>>>> typedef struct AVInterruptCB {
>>>>>>>>>>>> int (*callback)(void*);
>>>>>>>>>>>> void *opaque;
>>>>>>>>>>>> } AVInterruptCB;
>>>>>>>>>>>>
>>>>>>>>>>>> Then make name changes to whereever and add to AVCodecContext.
>>>>>>>>>>>>
>>>>>>>>>>>> This callback could be implemented piecemeal whereever needed
>>>>>>>>>>>> over
>>>>>>>>>>>> time,
>>>>>>>>>>>> hitting the more intense processes first.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Just open a (potentially pre-cached) new AVCodecContext, it'll
>>>>>>>>>>>> be
>>>>>>>>>>>>
>>>>>>>>>>>> even
>>>>>>>>>>> faster than your solution.
>>>>>>>>>>>
>>>>>>>>>>> Ronald
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> hmmm. That's a thought for seeking I suppose but does not apply
>>>>>>>>>>> to
>>>>>>>>>>>
>>>>>>>>>> waiting
>>>>>>>>>> to close. Why do I care about close time ? Because another video
>>>>>>>>>> has
>>>>>>>>>> come
>>>>>>>>>> in to replace it or variations of it. This can happen rapidly.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> This is where more evolved languages have the concept of garbage
>>>>>>>>> collection. For your purpose, you simply have a queue where you push
>>>>>>>>> "AVCodecContexts I don't need anymore" into, and while the
>>>>>>>>> application
>>>>>>>>> is
>>>>>>>>> in the idle loop, you pop it and destroy items left in it.
>>>>>>>>>
>>>>>>>>> Really, I understand your use case, but you don't want to add all
>>>>>>>>> kind
>>>>>>>>> of
>>>>>>>>> clever hacks in AVCodecContext to get this kind of stuff done.
>>>>>>>>> You're
>>>>>>>>> not
>>>>>>>>> using a shared I/O resource that may be protected by a cookie or
>>>>>>>>> worse
>>>>>>>>> for
>>>>>>>>> pay-per-view video, and you're not in any sort of kernel wait, so
>>>>>>>>> there's
>>>>>>>>> no reason to add these kind of hacks. It's a logical thought, but
>>>>>>>>> this
>>>>>>>>> problem has been solved already and there's better, easier and
>>>>>>>>> faster
>>>>>>>>> solutions out there that do not involve adding hacks in every single
>>>>>>>>> FFmpeg
>>>>>>>>> decoder to actually support this.
>>>>>>>>>
>>>>>>>>> Ronald
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Yeah really did not like the notion of changing decoders and I
>>>>>>>> don't
>>>>>>>> like adding anything that might not be needed, but I will see what I
>>>>>>>> can do
>>>>>>>> with your suggestions. I never know what ffmpeg can tolerate. I had
>>>>>>>> asked
>>>>>>>> in libav-user but get the same old BS there when asking about things
>>>>>>>> like
>>>>>>>> this.
>>>>>>>>
>>>>>>>>
>>>>>>>> Ok tried some test code allocating new context and that worked
>>>>>>> pretty
>>>>>>> well.
>>>>>>>
>>>>>>> I had to do this to get consistent results:
>>>>>>>
>>>>>>> AVCodec *codec ... already have it
>>>>>>>
>>>>>>> AVCodecContext *new_context = avcodec_alloc_context3 (NULL);
>>>>>>> avcodec_copy_context (new_context,old_context);
>>>>>>> avcodec_open2 (new_context,codec,NULL);
>>>>>>>
>>>>>>> The following worked for at least one file but for failed for others
>>>>>>> like
>>>>>>> Theora etc.
>>>>>>>
>>>>>>> AVCodecContext *new_context = avcodec_alloc_context3 (codec);
>>>>>>> avcodec_open2 (new_context,codec,NULL);
>>>>>>>
>>>>>>> For Theora it failed in avcodec_open2 saying 'missing side data' or
>>>>>>> similiar.
>>>>>>>
>>>>>>> Using a cached context the wait time is zero. Executing the 3
>>>>>>> statements
>>>>>>> above on slower machine is about 1 to 4 ms. It's also not like it
>>>>>>> always
>>>>>>> stuck in avcodec_decode_video2 either. In this case I don't need a new
>>>>>>> context and wait time is zero.
>>>>>>>
>>>>>>> Thanks Ronald.
>>>>>>>
>>>>>>>
>>>>>>> Ronald says:
>>>>>>
>>>>>> Just open a (potentially pre-cached) new AVCodecContext, it'll be even
>>>>>>
>>>>>> faster than your solution.
>>>>>>>
>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> and
>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>> Really, I understand your use case, but you don't want to add all kind
>>>>>> of
>>>>>>
>>>>>> clever hacks in AVCodecContext to get this kind of stuff done. You're
>>>>>>>
>>>>>>>> not
>>>>>>>>> using a shared I/O resource that may be protected by a cookie or
>>>>>>>>> worse
>>>>>>>>> for
>>>>>>>>> pay-per-view video, and you're not in any sort of kernel wait, so
>>>>>>>>> there's
>>>>>>>>> no reason to add these kind of hacks. It's a logical thought, but
>>>>>>>>> this
>>>>>>>>> problem has been solved already and there's better, easier and
>>>>>>>>> faster
>>>>>>>>> solutions out there that do not involve adding hacks in every single
>>>>>>>>> FFmpeg
>>>>>>>>> decoder to actually support this.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Using a cached and open AVCodecContext does work but it's like
>>>>>>>> trying
>>>>>>>>
>>>>>>> to
>>>>>> kill an ant with a sledgehammer. Using a cached and not opened context
>>>>>> helps some but you still loose time when opening it. So best results
>>>>>> are
>>>>>> when using a pre-cached open context.
>>>>>>
>>>>>> This means you will have allocated resources and most likely opened
>>>>>> threads in your cached context that are not doing anything.
>>>>>>
>>>>>> So you have to ask what the real hack is. Keeping an opened cached
>>>>>> context
>>>>>> or having an interrupt callback. An interrupt callback does not use any
>>>>>> additional resources but then it has to be implemented for every
>>>>>> decoder.
>>>>>> An opened context works now, but uses addtional resources and mostly
>>>>>> opened
>>>>>> threads that are more or less dormant. Having an interruptible intense
>>>>>> process is normal and not a hack and you should not have to beat it to
>>>>>> death to get it to work.
>>>>>>
>>>>>> I restrict the context to have at most 2 threads. Yes I could limit it
>>>>>> to
>>>>>> no new threads, but I get better results with 2. So you have to be
>>>>>> careful.
>>>>>> If the thread_count is zero, which is the default, then it will choose
>>>>>> the
>>>>>> number of threads based on the cpu count. You will have this number of
>>>>>> opened threads in a cached opened context. On one of my machines this
>>>>>> would
>>>>>> be 8 opened and unused threads for cached open context, but I set
>>>>>> thread_count to 2, getting diminishing returns on greater number of
>>>>>> threads.
>>>>>>
>>>>>>
>>>>>
>>>>> I'll bite. Please do define expensive, as Reimar also said. Do you mean
>>>>> "cpu usage"? Or "memory usage"? Or something else?
>>>>>
>>>>>
>>>> I was definning expensive as allocating threads and memory that do
>>>> nothing
>>>> for what should be a simple process.
>>>>
>>>
>>>
>>> But isn't that a completely academic concern?
>>>
>>
>> Don't think so as something is not good somewhere. Did some more checking
>> and tried to figure the cost of this more accurately. It varies on codec
>> and some other things so trying to get a handle on it.
>>
>> With H264 file, 2 threads, 2 opened context, one in use and one cached.
>>
>> Initially all is good. Then when I seek, I swap the context. The 'in use'
>> context which might still be in avcode_decode_video2 is flushed later but
>> for test I waited on video and flushed both cached and original context.
>>
>> When the seek is complete, I see about double the amount of memory being
>> used for the file. This a large video file and goes from about 30mb to 60mb
>> after seek give or take.
>
>
> Right, because you didn't flush the old context yet. You need to flush and
> reset it for the reference frames to be released. Until then, it's as alive
> as it could be, consuming any bit as much memory as the running one.
>
I guess you missed reading this:
>>The 'in use' context which might still be in avcode_decode_video2 is flushed later but
>>for test I waited on video and flushed both cached and original context.
For test I waiting on video and flushed both the old context and new context. Normally the old context is flushed later (very soon
later), but for testing I flushed both to be sure. In either case it did not make any difference.
More information about the ffmpeg-devel
mailing list