[FFmpeg-trac] #7108(avutil:new): d3d11va_device_create() crashes when the graphics device is disabled.

[FFmpeg]

#7108: d3d11va_device_create() crashes when the graphics device is disabled.
-------------------------------------+-------------------------------------
             Reporter:  haviet       |                     Type:  defect
               Status:  new          |                 Priority:  important
            Component:  avutil       |                  Version:  3.4
             Keywords:  d3d11va      |               Blocked By:
  crash                              |  Reproduced by developer:  1
             Blocking:               |
Analyzed by developer:  0            |
-------------------------------------+-------------------------------------
 Summary of the bug:
 1) Environment:
 Purpose: Decoding video via HW Acceleration
         Before the crash: It decodes video well via both CUDA, DXVA2 and
 D3D11VA.
         Crash condition: Disables the graphics device.
         Graphics device: NVIDIA
         OS: Windows 10 x64 Pro
         Version:
         ffmpeg version N-90264-g80798e3857 Copyright (c) 2000-2018 the
 FFmpeg developers
   built with Microsoft (R) C/C++ Optimizing Compiler Version 19.13.26128
 for x64
   configuration: --toolchain=msvc --arch=x86_64 --target-os=win64
         libavutil      56.  8.100 / 56.  8.100
         libavcodec     58. 14.100 / 58. 14.100
         libavformat    58. 10.100 / 58. 10.100
         libavdevice    58.  2.100 / 58.  2.100

 2) How to reproduce:
         In Windows 10 OS, go to the menu as below and disable the graphics
 device
         Computer Manager -> Device Manager -> Display adapters -> Graphics
 device.

 Then make a call to the function
         av_hwdevice_ctx_create(&pHwDeviceCtx, AV_HWDEVICE_TYPE_D3D11VA,
 "hw", nullptr, 0);

         This shall trigger the crash in d3d11va_device_create() function.
         The order of calls leading to the crash is:
         -> av_hwdevice_ctx_create
         -> d3d11va_device_create
         -> ID3D11Device_QueryInterface

         Note that I had tried other device types as below, though the
 function return an error but it did not cause crash.

     AV_HWDEVICE_TYPE_CUDA,      <-- OK
     AV_HWDEVICE_TYPE_DXVA2,     <-- OK
     AV_HWDEVICE_TYPE_QSV,       <-- OK
     AV_HWDEVICE_TYPE_OPENCL,    <-- OK
     AV_HWDEVICE_TYPE_D3D11VA,   <-- crashed


 {{{

 //      function is crashed
 static int d3d11va_device_create(AVHWDeviceContext *ctx, const char
 *device,
                                  AVDictionary *opts, int flags)
 {
     AVD3D11VADeviceContext *device_hwctx = ctx->hwctx;

     //  other lines ...

     hr = mD3D11CreateDevice(pAdapter, pAdapter ? D3D_DRIVER_TYPE_UNKNOWN :
 D3D_DRIVER_TYPE_HARDWARE, NULL, creationFlags, NULL, 0,
                    D3D11_SDK_VERSION, &device_hwctx->device, NULL, NULL);

 //      Right here the hr status is:
         hr = 0x887a0004 : The specified device interface or feature level
 is not supported on this system.

         // The block "if (FAILED(hr))" should be placed here.

     if (pAdapter) {
         DXGI_ADAPTER_DESC2 desc;
         hr = IDXGIAdapter2_GetDesc(pAdapter, &desc);
         if (!FAILED(hr)) {
             av_log(ctx, AV_LOG_INFO, "Using device %04x:%04x (%ls).\n",
                    desc.VendorId, desc.DeviceId, desc.Description);
         }
         IDXGIAdapter_Release(pAdapter);
     }
     if (FAILED(hr)) {
         av_log(ctx, AV_LOG_ERROR, "Failed to create Direct3D device
 (%lx)\n", (long)hr);
         return AVERROR_UNKNOWN;
     }

         //      Crashed here due to device_hwctx->device is NULL
         //
     hr = ID3D11Device_QueryInterface(device_hwctx->device,
 &IID_ID3D10Multithread, (void **)&pMultithread);
     if (SUCCEEDED(hr)) {
         ID3D10Multithread_SetMultithreadProtected(pMultithread, TRUE);
         ID3D10Multithread_Release(pMultithread);
     }
 }

 }}}

 I realized that the call to mD3D11CreateDevice() returns
 hr = 0x887a0004 : "The specified device interface or feature level is not
 supported on this system."
 But hr value is not checked before moving to the next statement.
 That is the cause.

--
Ticket URL: <https://trac.ffmpeg.org/ticket/7108>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker