[FFmpeg-trac] #10286(avutil:new): segfault calling av_hwframe_transfer_data on qsv decoded frame if MFXVideoVPP_Init fails

[FFmpeg]

#10286: segfault calling av_hwframe_transfer_data on qsv decoded frame if
MFXVideoVPP_Init fails
--------------------------------------+----------------------------------
             Reporter:  Steve Browne  |                     Type:  defect
               Status:  new           |                 Priority:  normal
            Component:  avutil        |                  Version:  5.1.2
             Keywords:                |               Blocked By:
             Blocking:                |  Reproduced by developer:  0
Analyzed by developer:  0             |
--------------------------------------+----------------------------------
 Summary of the bug:
 I know this has worked before even in the same version so I think there's
 an Intel driver bug causing my `MFXVideoVPP_Init` to not be successful. I
 bet if I reboot it would be fine, but currently it's returning
 MFX_WRN_PARTIAL_ACCELERATION. This happens in hwcontext_qsv.c.

 Once a machine gets into that state or if your GPU for whatever reason
 only has partial support it tries to fallback and do the hwframe transfer
 by other means. What happens in this case is it eventually gets to
 `qsv_transfer_data_child` in hwcontext_qsv.c where it creates a dummy
 frame, but only sets data[3] and not data[0]. After that this dummy frame
 works its way down to `d3d11va_transfer_data` in hwcontext_d3d11va.c where
 it expects the texture to be in data[0] and does no null check so it ends
 up segfaulting. Additionally it looks at data[0] for the index which won't
 be set either so it would always use index 0 which may be fine.

 How to reproduce:
 I don't really know how you would reproduce this without just simply
 forcing `MFXVideoVPP_Init` to fail when creating the session so it falls
 back. So just comment that line out and set the error code to simulate it.
 Then it's just a matter of decoding a frame with h264_qsv and calling
 `av_hwframe_transfer_data` with the `AV_HWFRAME_TRANSFER_DIRECTION_FROM`
 flag.
-- 
Ticket URL: <https://trac.ffmpeg.org/ticket/10286>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker