[FFmpeg-trac] #10111(avcodec:new): Memory leak with vaapi encoder

[FFmpeg]

#10111: Memory leak with vaapi encoder
--------------------------------------+----------------------------------
             Reporter:  Steve Browne  |                     Type:  defect
               Status:  new           |                 Priority:  normal
            Component:  avcodec       |                  Version:  4.4.3
             Keywords:                |               Blocked By:
             Blocking:                |  Reproduced by developer:  0
Analyzed by developer:  0             |
--------------------------------------+----------------------------------
 Summary of the bug:
 Calling avcodec_send_frame multiple times in a row without an
 avcodec_receive_packet in between with an AVFrame that does not store data
 in data[0] will result in a memory leak. The example where I ran into this
 was with VAAPI. This could potentially apply to other hardware encoders
 that don't store anything in data[0] as well though.

 The problem lies in encode_send_frame_internal because it uses
 dst->data[0] to determine if avci->buffer_frame is already pointing to
 something. Since vaapi contexts only store the surface ID in data[3] it
 will gladly think buffer_frame is unused and overwrite it without actually
 encoding it or freeing it. I don't know if all contexts should just store
 something in data[0] or if that check something else like buf[0], but it's
 very easy for this to cause a memory leak.

 The simple workaround for this is to just set data[0] = data[4] for vaapi.

 How to reproduce:
 Call avcodec_send_frame multiple times in a row without
 avcodec_receive_packet.

 It looks like this was actually fixed with
 67aceaf4ad641a4d34c3ec70b532efdc60483e3d, but I'm using an older version.
 It would probably be nice if this were backpatched to avoid others running
 into this.

 Feel free to close this, but I just wanted to write something for it
 because I couldn't find anything indicating a memory leak with vaapi
 encoding when searching and there's nothing in the changelog that would
 suggest that resolves a memory leak either.
-- 
Ticket URL: <https://trac.ffmpeg.org/ticket/10111>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker