[Libav-user] Encoding with variable frame rate

[Brad O'Hearne]

On May 20, 2013, at 6:01 AM, Robert Krüger <krueger at lesspain.de> wrote:

> On Mon, May 20, 2013 at 2:06 PM, Carl Eugen Hoyos <cehoyos at ag.or.at> wrote:
>> Robert Krüger <krueger at ...> writes:
>> 
>>> On Mon, May 20, 2013 at 12:11 PM, Carl Eugen Hoyos wrote:
>>>> FFmpeg's mov muxer does not support vfr.

>>> What happens when one feeds the muxer with packets
>>> that have timestamps with differing offsets?

Robert, 

For what its worth, this is essentially the problem I recently had to tackle. You may have seen some recent discussions about pts/dts and the nature of time-base. While there are some things that can be done if you control a capture mechanism, if you don't control that mechanism (i.e. receiving capture data from a third-party API), in general, there is no such thing as a fixed frame rate. 

In my use-case, I am using QTKit as the capture mechanism, which while you can set the desired frame-rate, doesn't deliver that specified FPS per se. It delivers pts timestamp, dts timestamp, a timescale, and a duration. That is sufficient data and logically all that is needed to properly encode the captured video. But if you specify that you want 30 FPS, and set FFmpeg's time_base.den to 30, but then QTKit decides to deliver you 15FPS with each frame having double the 30FPS duration (1/15 of a sec instead of 1/30), this is a problem because FFmpeg must have its time_base.den frames per second. If you don't give it that, your video will play back at an unexpected speed -- if you give it 15FPS for example, it will take 2 seconds in the source data's time to reach 30 FPS, but FFmpeg processes that in one second, and hence your video will end up playing back at double the speed. 

This was my dilemma -- capture is by definition a variable frame rate (it may end up coming at a fixed frame rate, but that isn't guaranteed, so you have to plan for variable), but FFmpeg must have fixed frame rate. So what I had to do was make variable frame rate look like a fixed frame rate. In short, you have to skip encoding frames entirely if they come too quickly (i.e. a higher frame rate than time_base.den), and you have to add frames if they are received too slowly (i.e. lower frame rate than time_base.den) by encoding the same frame more than once. You also have to convert your pts / dts received to the frame count relative again to the time_base. 

It isn't pretty. But it worked. I hope that helps. 

Brad