[FFmpeg-devel] Allow interrupt callback for AVCodecContext

Michael Niedermayer michaelni at gmx.at
Wed Jan 8 13:13:25 CET 2014


On Mon, Jan 06, 2014 at 11:50:22PM -0500, Don Moir wrote:
> 
> ----- Original Message ----- From: "Don Moir" <donmoir at comcast.net>
> To: "FFmpeg development discussions and patches" <ffmpeg-devel at ffmpeg.org>
> Sent: Monday, January 06, 2014 5:12 PM
> Subject: Re: [FFmpeg-devel] Allow interrupt callback for AVCodecContext
> 
> 
> >
> >----- Original Message ----- From: "Ronald S. Bultje"
> ><rsbultje at gmail.com>
> >To: "FFmpeg development discussions and patches" <ffmpeg-devel at ffmpeg.org>
> >Sent: Monday, January 06, 2014 5:04 PM
> >Subject: Re: [FFmpeg-devel] Allow interrupt callback for AVCodecContext
> >
> >
> >>Hi,
> >>
> >>On Mon, Jan 6, 2014 at 4:08 PM, Don Moir <donmoir at comcast.net> wrote:
> >>
> >>>
> >>>----- Original Message ----- From: "Ronald S. Bultje" <rsbultje at gmail.com>
> >>>To: "FFmpeg development discussions and patches" <ffmpeg-devel at ffmpeg.org>
> >>>Sent: Monday, January 06, 2014 3:59 PM
> >>>
> >>>Subject: Re: [FFmpeg-devel] Allow interrupt callback for AVCodecContext
> >>>
> >>>
> >>> Hi,
> >>>>
> >>>>On Mon, Jan 6, 2014 at 2:48 PM, Don Moir <donmoir at comcast.net> wrote:
> >>>>
> >>>>
> >>>>>----- Original Message ----- From: "Ronald S. Bultje" <
> >>>>>rsbultje at gmail.com>
> >>>>>To: "FFmpeg development discussions and patches" <
> >>>>>ffmpeg-devel at ffmpeg.org>
> >>>>>Sent: Monday, January 06, 2014 9:52 AM
> >>>>>
> >>>>>Subject: Re: [FFmpeg-devel] Allow interrupt callback for AVCodecContext
> >>>>>
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>>>
> >>>>>>On Mon, Jan 6, 2014 at 3:56 AM, Don Moir <donmoir at comcast.net> wrote:
> >>>>>>
> >>>>>>
> >>>>>> ----- Original Message ----- From: "Ronald S. Bultje" <
> >>>>>>>rsbultje at gmail.com>
> >>>>>>>To: "FFmpeg development discussions and patches" <
> >>>>>>>ffmpeg-devel at ffmpeg.org>
> >>>>>>>Sent: Monday, January 06, 2014 8:44 AM
> >>>>>>>
> >>>>>>>Subject: Re: [FFmpeg-devel] Allow interrupt callback for AVCodecContext
> >>>>>>>
> >>>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>>
> >>>>>>>>On Mon, Dec 16, 2013 at 1:21 AM, Don Moir <donmoir at comcast.net>
> >>>>>>>>wrote:
> >>>>>>>>
> >>>>>>>> ----- Original Message ----- From: "Don Moir" <donmoir at comcast.net>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> To: "FFmpeg development discussions and patches" <
> >>>>>>>>>
> >>>>>>>>>>ffmpeg-devel at ffmpeg.org
> >>>>>>>>>>>
> >>>>>>>>>>Sent: Monday, December 16, 2013 2:03 AM
> >>>>>>>>>>Subject: Re: [FFmpeg-devel] Allow interrupt callback for
> >>>>>>>>>>AVCodecContext
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> ----- Original Message ----- From: "Ronald S. Bultje" <
> >>>>>>>>>>
> >>>>>>>>>> rsbultje at gmail.com>
> >>>>>>>>>>>To: "FFmpeg development discussions and patches" <
> >>>>>>>>>>>ffmpeg-devel at ffmpeg.org>
> >>>>>>>>>>>Sent: Wednesday, January 01, 2014 11:14 AM
> >>>>>>>>>>>Subject: Re: [FFmpeg-devel] Allow interrupt callback for
> >>>>>>>>>>>AVCodecContext
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Hi,
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, Dec 16, 2013 at 1:30 AM, Don Moir <donmoir at comcast.net>
> >>>>>>>>>>>>wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> ----- Original Message ----- From: "Ronald S. Bultje" <
> >>>>>>>>>>>>
> >>>>>>>>>>>> rsbultje at gmail.com>
> >>>>>>>>>>>>>To: "FFmpeg development discussions and patches" <
> >>>>>>>>>>>>>ffmpeg-devel at ffmpeg.org>
> >>>>>>>>>>>>>Sent: Wednesday, January 01, 2014 10:52 AM
> >>>>>>>>>>>>>Subject: Re: [FFmpeg-devel] Allow interrupt callback for
> >>>>>>>>>>>>>AVCodecContext
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Mon, Dec 16, 2013 at 2:07 AM, Don Moir <donmoir at comcast.net>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> For now just seeing what you think about this. This is about
> >>>>>>>>>>>>>>thread
> >>>>>>>>>>>>>>based
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> apps where this makes sense.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>When attempting to do a new seek or waiting to close a video, I
> >>>>>>>>>>>>>>>find
> >>>>>>>>>>>>>>>that
> >>>>>>>>>>>>>>>I am waiting on avcodec_decode_video2 to return before I can
> >>>>>>>>>>>>>>>continue.
> >>>>>>>>>>>>>>>Depending on machine and video, this wait time can be up to
> >>>>>>>>>>>>>>>about
> >>>>>>>>>>>>>>>50ms
> >>>>>>>>>>>>>>>but
> >>>>>>>>>>>>>>>normally wait time about 5 to 20 ms or so.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>You might say 'so what' and I would agree for a simple player
> >>>>>>>>>>>>>>>app
> >>>>>>>>>>>>>>>it
> >>>>>>>>>>>>>>>does
> >>>>>>>>>>>>>>>not make that much difference.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>If you are trying to stay on a timeline, or in case of
> >>>>>>>>>>>>>>>scrubbing,
> >>>>>>>>>>>>>>>or
> >>>>>>>>>>>>>>>for
> >>>>>>>>>>>>>>>editing apps, this wait time does make a difference. That is,
> >>>>>>>>>>>>>>>you
> >>>>>>>>>>>>>>>can not
> >>>>>>>>>>>>>>>move on until avcodec_decode_video2 has returned.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>I can pretty much seek instantly to zero for any file except
> >>>>>>>>>>>>>>>when I
> >>>>>>>>>>>>>>>have
> >>>>>>>>>>>>>>>to wait on avcodec_decode_video2 if that be the case.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>For me, it's normal for any intense process like decoding to be
> >>>>>>>>>>>>>>>interruptible but this is not the case for AVCodecContext in
> >>>>>>>>>>>>>>>ffmpeg.
> >>>>>>>>>>>>>>>This
> >>>>>>>>>>>>>>>is strange, don't you think?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>For AVFormatContext you have:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>typedef struct AVIOInterruptCB {
> >>>>>>>>>>>>>>>    int (*callback)(void*);
> >>>>>>>>>>>>>>>    void *opaque;
> >>>>>>>>>>>>>>>} AVIOInterruptCB;
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>I would use this model for AVCodecContext but change naming to:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>typedef struct AVInterruptCB {
> >>>>>>>>>>>>>>>    int (*callback)(void*);
> >>>>>>>>>>>>>>>    void *opaque;
> >>>>>>>>>>>>>>>} AVInterruptCB;
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>Then make name changes to whereever and add to AVCodecContext.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>This callback could be implemented piecemeal whereever needed
> >>>>>>>>>>>>>>>over
> >>>>>>>>>>>>>>>time,
> >>>>>>>>>>>>>>>hitting the more intense processes first.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Just open a (potentially pre-cached) new AVCodecContext, it'll
> >>>>>>>>>>>>>>>be
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> even
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>faster than your solution.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>Ronald
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> hmmm. That's a thought for seeking I suppose but does not apply
> >>>>>>>>>>>>>>to
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> waiting
> >>>>>>>>>>>>>to close. Why do I care about close time ? Because another video
> >>>>>>>>>>>>>has
> >>>>>>>>>>>>>come
> >>>>>>>>>>>>>in to replace it or variations of it. This can happen rapidly.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> This is where more evolved languages have the concept of garbage
> >>>>>>>>>>>>collection. For your purpose, you simply have a queue where you
> >>>>>>>>>>>>push
> >>>>>>>>>>>>"AVCodecContexts I don't need anymore" into, and while the
> >>>>>>>>>>>>application
> >>>>>>>>>>>>is
> >>>>>>>>>>>>in the idle loop, you pop it and destroy items left in it.
> >>>>>>>>>>>>
> >>>>>>>>>>>>Really, I understand your use case, but you don't want to add all
> >>>>>>>>>>>>kind
> >>>>>>>>>>>>of
> >>>>>>>>>>>>clever hacks in AVCodecContext to get this kind of stuff done.
> >>>>>>>>>>>>You're
> >>>>>>>>>>>>not
> >>>>>>>>>>>>using a shared I/O resource that may be protected by a cookie or
> >>>>>>>>>>>>worse
> >>>>>>>>>>>>for
> >>>>>>>>>>>>pay-per-view video, and you're not in any sort of kernel wait, so
> >>>>>>>>>>>>there's
> >>>>>>>>>>>>no reason to add these kind of hacks. It's a logical thought, but
> >>>>>>>>>>>>this
> >>>>>>>>>>>>problem has been solved already and there's better, easier and
> >>>>>>>>>>>>faster
> >>>>>>>>>>>>solutions out there that do not involve adding hacks in every
> >>>>>>>>>>>>single
> >>>>>>>>>>>>FFmpeg
> >>>>>>>>>>>>decoder to actually support this.
> >>>>>>>>>>>>
> >>>>>>>>>>>>Ronald
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Yeah really did not like the notion of changing decoders and I
> >>>>>>>>>>>>
> >>>>>>>>>>>don't
> >>>>>>>>>>>like adding anything that might not be needed, but I will see what
> >>>>>>>>>>>I
> >>>>>>>>>>>can do
> >>>>>>>>>>>with your suggestions. I never know what ffmpeg can tolerate. I had
> >>>>>>>>>>>asked
> >>>>>>>>>>>in libav-user but get the same old BS there when asking about
> >>>>>>>>>>>things
> >>>>>>>>>>>like
> >>>>>>>>>>>this.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Ok tried some test code allocating new context and that worked
> >>>>>>>>>>>
> >>>>>>>>>>pretty
> >>>>>>>>>>well.
> >>>>>>>>>>
> >>>>>>>>>>I had to do this to get consistent results:
> >>>>>>>>>>
> >>>>>>>>>>AVCodec *codec ... already have it
> >>>>>>>>>>
> >>>>>>>>>>AVCodecContext *new_context = avcodec_alloc_context3 (NULL);
> >>>>>>>>>>avcodec_copy_context (new_context,old_context);
> >>>>>>>>>>avcodec_open2 (new_context,codec,NULL);
> >>>>>>>>>>
> >>>>>>>>>>The following worked for at least one file but for failed for others
> >>>>>>>>>>like
> >>>>>>>>>>Theora etc.
> >>>>>>>>>>
> >>>>>>>>>>AVCodecContext *new_context = avcodec_alloc_context3 (codec);
> >>>>>>>>>>avcodec_open2 (new_context,codec,NULL);
> >>>>>>>>>>
> >>>>>>>>>>For Theora it failed in avcodec_open2 saying 'missing side data' or
> >>>>>>>>>>similiar.
> >>>>>>>>>>
> >>>>>>>>>>Using a cached context the wait time is zero. Executing the 3
> >>>>>>>>>>statements
> >>>>>>>>>>above on slower machine is about 1 to 4 ms. It's also not like it
> >>>>>>>>>>always
> >>>>>>>>>>stuck in avcodec_decode_video2 either. In this case I don't need a
> >>>>>>>>>>new
> >>>>>>>>>>context and wait time is zero.
> >>>>>>>>>>
> >>>>>>>>>>Thanks Ronald.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Ronald says:
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Just open a (potentially pre-cached) new AVCodecContext, it'll be
> >>>>>>>>>even
> >>>>>>>>>
> >>>>>>>>> faster than your solution.
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>>>  and
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>> Really, I understand your use case, but you don't want to add all
> >>>>>>>>>kind
> >>>>>>>>>of
> >>>>>>>>>
> >>>>>>>>> clever hacks in AVCodecContext to get this kind of stuff done.
> >>>>>>>>>You're
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> not
> >>>>>>>>>>>
> >>>>>>>>>>>>using a shared I/O resource that may be protected by a cookie or
> >>>>>>>>>>>>worse
> >>>>>>>>>>>>for
> >>>>>>>>>>>>pay-per-view video, and you're not in any sort of kernel wait, so
> >>>>>>>>>>>>there's
> >>>>>>>>>>>>no reason to add these kind of hacks. It's a logical thought, but
> >>>>>>>>>>>>this
> >>>>>>>>>>>>problem has been solved already and there's better, easier and
> >>>>>>>>>>>>faster
> >>>>>>>>>>>>solutions out there that do not involve adding hacks in every
> >>>>>>>>>>>>single
> >>>>>>>>>>>>FFmpeg
> >>>>>>>>>>>>decoder to actually support this.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>  Using a cached and open AVCodecContext does work but it's like
> >>>>>>>>>>>>
> >>>>>>>>>>>trying
> >>>>>>>>>>>
> >>>>>>>>>>> to
> >>>>>>>>>>
> >>>>>>>>>kill an ant with a sledgehammer. Using a cached and not opened
> >>>>>>>>>context
> >>>>>>>>>helps some but you still loose time when opening it. So best results
> >>>>>>>>>are
> >>>>>>>>>when using a pre-cached open context.
> >>>>>>>>>
> >>>>>>>>>This means you will have allocated resources and most likely opened
> >>>>>>>>>threads in your cached context that are not doing anything.
> >>>>>>>>>
> >>>>>>>>>So you have to ask what the real hack is. Keeping an opened cached
> >>>>>>>>>context
> >>>>>>>>>or having an interrupt callback. An interrupt callback does not use
> >>>>>>>>>any
> >>>>>>>>>additional resources but then it has to be implemented for every
> >>>>>>>>>decoder.
> >>>>>>>>>An opened context works now, but uses addtional resources and mostly
> >>>>>>>>>opened
> >>>>>>>>>threads that are more or less dormant. Having an interruptible
> >>>>>>>>>intense
> >>>>>>>>>process is normal and not a hack and you should not have to beat it
> >>>>>>>>>to
> >>>>>>>>>death to get it to work.
> >>>>>>>>>
> >>>>>>>>>I restrict the context to have at most 2 threads. Yes I could limit
> >>>>>>>>>it
> >>>>>>>>>to
> >>>>>>>>>no new threads, but I get better results with 2. So you have to be
> >>>>>>>>>careful.
> >>>>>>>>>If the thread_count is zero, which is the default, then it will
> >>>>>>>>>choose
> >>>>>>>>>the
> >>>>>>>>>number of threads based on the cpu count. You will have this number
> >>>>>>>>>of
> >>>>>>>>>opened threads in a cached opened context. On one of my machines this
> >>>>>>>>>would
> >>>>>>>>>be 8 opened and unused threads for cached open context, but I set
> >>>>>>>>>thread_count to 2, getting diminishing returns on greater number of
> >>>>>>>>>threads.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>I'll bite. Please do define expensive, as Reimar also said. Do you
> >>>>>>>>mean
> >>>>>>>>"cpu usage"? Or "memory usage"? Or something else?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I was definning expensive as allocating threads and memory that do
> >>>>>>>nothing
> >>>>>>>for what should be a simple process.
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>But isn't that a completely academic concern?
> >>>>>>
> >>>>>>
> >>>>>Don't think so as something is not good somewhere. Did some more checking
> >>>>>and tried to figure the cost of this more accurately. It varies on codec
> >>>>>and some other things so trying to get a handle on it.
> >>>>>
> >>>>>With H264 file, 2 threads, 2 opened context, one in use and one cached.
> >>>>>
> >>>>>Initially all is good. Then when I seek, I swap the context. The 'in use'
> >>>>>context which might still be in avcode_decode_video2 is flushed later but
> >>>>>for test I waited on video and flushed both cached and original context.
> >>>>>
> >>>>>When the seek is complete, I see about double the amount of memory being
> >>>>>used for the file. This a large video file and goes from about 30mb to
> >>>>>60mb
> >>>>>after seek give or take.
> >>>>>
> >>>>
> >>>>
> >>>>Right, because you didn't flush the old context yet. You need to flush and
> >>>>reset it for the reference frames to be released. Until then, it's as
> >>>>alive
> >>>>as it could be, consuming any bit as much memory as the running one.
> >>>>
> >>>>
> >>>I guess you missed reading this:
> >>>
> >>>
> >>> The 'in use' context which might still be in avcode_decode_video2 is
> >>>>>flushed later but
> >>>>>for test I waited on video and flushed both cached and original context.
> >>>>>
> >>>>
> >>>For test I waiting on video and flushed both the old context and new
> >>>context. Normally the old context is flushed later (very soon later), but
> >>>for testing I flushed both to be sure. In either case it did not make any
> >>>difference.
> >>
> >>
> >>Did you avcodec_close() it?
> >
> >No. That is almost pointless since the whole thing here is doing
> >something fast. The re-open takes too much time. Caching in itself
> >doesn't make too much sense if you have to close and reopen each
> >time. The allocation takes very little time. It can help a little
> >but not too much.
> 
> The way it is right now with cached context is extreme overkill to
> perform a fairly simple task. You have memory allocations per thread
> that can be quite large and that depends on codec. For h264 it can
> be large and for others not as much. The whole process of caching a
> context has some merit but the overhead and time required to close
> and reopen reduces it's effectiveness.
> 
> Essentially, everytime you want to use a cached context, you will be
> closing threads, freeing memory, etc and then reopening the threads
> and reallocating memory (silly). The reason for this is it's the
> only way to free the context accumulated memory. flush does not do
> this and I don't know whats up with that.
> 
> Backing off from this but of course would still like
> avcodec_decode_video2 to be interruptible or a reasonable way to
> avoid waiting on it. Cached context the way it stands now is
> anything but reasonable.  Possibly if flush actually flushed the
> memory it would be more attractive.
> 
> The AVCodecContext.execute function which can be set by user would
> almost work but that is only used for slices and it does not look
> like its complete to me.

are you using frame threads ?
if you want lowest latency possibly , frame threads probably are
not a good idea

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

I have often repented speaking, but never of holding my tongue.
-- Xenocrates
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20140108/0b6090f7/attachment.asc>


More information about the ffmpeg-devel mailing list