[FFmpeg-trac] #7281(avformat:new): fMP4 Single File Generation is excruciatingly slow

[FFmpeg]

#7281: fMP4 Single File Generation is excruciatingly slow
------------------------------------+------------------------------------
             Reporter:  ronak2121   |                    Owner:
                 Type:  defect      |                   Status:  new
             Priority:  normal      |                Component:  avformat
              Version:  git-master  |               Resolution:
             Keywords:  hls         |               Blocked By:
             Blocking:              |  Reproduced by developer:  0
Analyzed by developer:  0           |
------------------------------------+------------------------------------

Comment (by ronak2121):

 I analyzed the issue, but I'm filing this ticket so I can get help from
 the community to fix this issue. The fix for this issue looks involved
 with many use cases to test for once we fix the problem.

 Just to summarize, I'm including the research I've done on this issue so
 far:

 Hey Carl,

 So I dug into this more today and I have root caused what's exactly
 happening here.

 The problematic code is this:
 https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/hlsenc.c#L1368
 This is where the filename is set and the next line actually opens the
 file.

 The logic for this hls_window method is the following:

 1. Make a new temporary file.
 2. Write out a new HLS manifest header.
 3. Loop through all available segments and write out all of the entries
 for them.
 4. Close the temporary file when finished.
 5. Rename the temporary file to the target file name.
 6. Rinse and repeat for every single fragment.

 Therefore, if you can imagine a 153 hour audio file, we write out a
 totally new HLS manifest 550800 times (153 * 60 * 60 assuming a 1s
 fragment duration) that gets progressively larger as each fragment is
 generated.

 This is a classic O(N^2) algorithm implementation, instead of:

 1. Creating the destination file up front & write the manifest header.
 2. Append the new segment to the file.
 3. If this is the last segment, write out EXT-X-ENDLIST.

 There's no looping involved, nor the need to make temporary files.

 FYI that I've noticed the same sort of pattern being applied to MPEG DASH:
 https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/dashenc.c#L786

 To implement something like this, looks like we'd have to significantly
 re-engineer the code. Do you have any pointers on how to go about doing
 this? Or, would you be able to help do this?

 Thanks for all your help,

 Ronak

--
Ticket URL: <https://trac.ffmpeg.org/ticket/7281#comment:2>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker