[FFmpeg-devel] [PATCH] Prevents crash from CreateDeviceEx and proper fallback to classic d3d9.

[wm4]

On Thu, 4 Jan 2018 00:26:01 +0100
Hendrik Leppkes <h.leppkes at gmail.com> wrote:

> On Wed, Jan 3, 2018 at 11:18 PM, wm4 <nfxjfg at googlemail.com> wrote:
> > On Wed,  3 Jan 2018 13:59:34 -0800
> > Humberto Ribeiro <mont3z.claro5 at gmail.com> wrote:
> >  
> >> ---
> >>  libavutil/hwcontext_dxva2.c | 6 +++++-
> >>  1 file changed, 5 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/libavutil/hwcontext_dxva2.c b/libavutil/hwcontext_dxva2.c
> >> index 2ddd4be..44ebdbc 100644
> >> --- a/libavutil/hwcontext_dxva2.c
> >> +++ b/libavutil/hwcontext_dxva2.c
> >> @@ -485,7 +485,11 @@ static int dxva2_device_create9ex(AVHWDeviceContext *ctx, UINT adapter)
> >>      if (FAILED(hr))
> >>          return AVERROR_UNKNOWN;
> >>
> >> -    IDirect3D9Ex_GetAdapterDisplayModeEx(d3d9ex, adapter, &modeex, NULL);
> >> +    hr = IDirect3D9Ex_GetAdapterDisplayModeEx(d3d9ex, adapter, &modeex, NULL);
> >> +    if (FAILED(hr)) {
> >> +        IDirect3D9Ex_Release(d3d9ex);
> >> +        return AVERROR_UNKNOWN;
> >> +    }
> >>
> >>      d3dpp.BackBufferFormat = modeex.Format;
> >>  
> >
> > LGTM, but when does it happen? Why would creation of IDirect3D9Ex
> > succeed, but this method fail? The strangest thing being that
> > CreateDeviceEx apparently fails, even though modeex is
> > zero-initialized and doesn't contain any pointers.
> >  
> 
> I noticed something about this some time ago - D3DDISPLAYMODEEX should
> not be entirely zero-initialized before this call, instead its "Size"
> member should be set to sizeof(D3DDISPLAYMODEEX) - see
> https://msdn.microsoft.com/en-us/library/windows/desktop/bb172549(v=vs.85).aspx
> Currently it always causes IDirect3D9Ex_GetAdapterDisplayModeEx to
> fail due to that (at least on my hardware), but it didn't seem to have
> any impact on the later create device call in my case.

Oh, so possibly entirely our/my fault.