[FFmpeg-devel] [PATCH] gdigrab: Fix hwnd parameter issues

[Martin Storsjö]

Converting from an integer to HWND (which is a pointer) requires
an explicit cast, otherwise Clang errors out like this:

    src/libavdevice/gdigrab.c:280:14: error: incompatible integer to pointer conversion assigning to 'HWND' (aka 'struct HWND__ *') from 'long' [-Wint-conversion]
      280 |         hwnd = strtol(name, &p, 0);
          |              ^ ~~~~~~~~~~~~~~~~~~~

(With GCC and MSVC, this was a mere warning, but with recent Clang,
this is an error.)

After adding a cast, all compilers also warn something like this:

    src/libavdevice/gdigrab.c:280:16: warning: cast to 'HWND' (aka 'struct HWND__ *') from smaller integer type 'long' [-Wint-to-pointer-cast]
      280 |         hwnd = (HWND) strtol(name, &p, 0);
          |                ^~~~~~~~~~~~~~~~~~~~~~~~~~

On Windows, long types are 32 bit, so to get a usable pointer, we
need to use long long. And interpret it as unsigned long long
while at it - i.e. using strtoull.

Finally, right above it, the code triggered the following warning:

    src/libavdevice/gdigrab.c:278:15: warning: mixing declarations and code is incompatible with standards before C99 [-Wdeclaration-after-statement]
      278 |         char *p;
          |               ^
---
 libavdevice/gdigrab.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/libavdevice/gdigrab.c b/libavdevice/gdigrab.c
index 41ef370f2b..b2858ecd89 100644
--- a/libavdevice/gdigrab.c
+++ b/libavdevice/gdigrab.c
@@ -274,10 +274,10 @@ gdigrab_read_header(AVFormatContext *s1)
     } else if (!strcmp(filename, "desktop")) {
         hwnd = NULL;
     } else if (!strncmp(filename, "hwnd=", 5)) {
-        name = filename + 5;
         char *p;
+        name = filename + 5;
 
-        hwnd = strtol(name, &p, 0);
+        hwnd = (HWND) strtoull(name, &p, 0);
 
         if (p == NULL || p == name || p[0] == '\0')
         {
-- 
2.34.1