[FFmpeg-trac] #9666(avcodec:closed): munmap_chunk(): invalid pointer encoding SEI with libx265

[FFmpeg]

#9666: munmap_chunk(): invalid pointer encoding SEI with libx265
-------------------------------------+-----------------------------------
             Reporter:  Brad Hards   |                    Owner:  (none)
                 Type:  defect       |                   Status:  closed
             Priority:  normal       |                Component:  avcodec
              Version:  unspecified  |               Resolution:  invalid
             Keywords:  libx265      |               Blocked By:
             Blocking:               |  Reproduced by developer:  0
Analyzed by developer:  0            |
-------------------------------------+-----------------------------------
Changes (by mkver):

 * status:  new => closed
 * resolution:   => invalid

Comment:

 Your code adds ever more AV_FRAME_DATA_SEI_UNREGISTERED side data to the
 frame (the old ones are not discarded). And libx265's behaviour upon
 encountering this is buggy. Here is the relevant code that copies the
 input NAL units to the internal frames used by libx265:

 {{{
     int numPayloads = pic_in->userSEI.numPayloads + toneMapPayload +
 userPayload;
     frame->m_userSEI.numPayloads = numPayloads;

     if (frame->m_userSEI.numPayloads)
     {
         if (!frame->m_userSEI.payloads)
         {
             frame->m_userSEI.payloads = new x265_sei_payload[numPayloads];
             for (int i = 0; i < numPayloads; i++)
                 frame->m_userSEI.payloads[i].payload = NULL;
         }
         for (int i = 0; i < numPayloads; i++)
         {
             x265_sei_payload input;
             if ((i == (numPayloads - 1)) && toneMapPayload)
                 input = toneMap;
             else if (m_enableNal)
                 input = seiMsg;
             else
                 input = pic_in->userSEI.payloads[i];

             if (!frame->m_userSEI.payloads[i].payload)
                 frame->m_userSEI.payloads[i].payload = new
 uint8_t[input.payloadSize];
             memcpy(frame->m_userSEI.payloads[i].payload, input.payload,
 input.payloadSize);
             frame->m_userSEI.payloads[i].payloadSize = input.payloadSize;
             frame->m_userSEI.payloads[i].payloadType = input.payloadType;
         }
 }}}

 The internal frames are reused, so frame might not be clean; in particular
 frame->m_userSEI might already be set. And in this case the above code
 simply presumes that this array is already big enough. Which is just not
 true (in particular not in this case).

 A few lines below the same error happens again with the payload arrays.
-- 
Ticket URL: <https://trac.ffmpeg.org/ticket/9666#comment:1>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker