[FFmpeg-user] Stack size issues with Vulkan Video and musl

[Riteo Siuga]

I'm pretty sure I sent this mail wrong the first time. Sorry if this
is a duplicate.

Hi, I hope this isn't the wrong place to ask this.

Calling the command to test Vulkan Video from the HWAccelIntro page
in the wiki on my musl setup causes a segfault.

The command: `ffmpeg -init_hw_device "vulkan=vk:0"-hwaccel vulkan
-hwaccel_output_format vulkan -i INPUT -f null - -benchmark`

(BTW, should I break commands too here?)

After some debugging it looks like the stack gets corrupted after
putting an array of `StdVideoH264ScalingLists` into it.

That makes sense because musl has a very small default stack size
(128k), which apparently is not always enough for proper Vulkan Video
decoding.

This is further confirmed by some quick calculations from
`StdVideoH264ScalingLists`' typedef and the aforementioned
stack-breaking declaration.

>From Vulkan's "vk_video/vulkan_video_codec_h264std.h" (with macros
evaluated):

```
typedef struct StdVideoH264ScalingLists {
uint16_t    scaling_list_present_mask;
uint16_t    use_default_scaling_matrix_mask;
uint8_t     ScalingList4x4[6][16];
uint8_t     ScalingList8x8[6][64];
} StdVideoH264ScalingLists
```

2B + 2B + 8B * 6 * 16 + 8B * 6 * 64 = 3844B

>From ffmpeg's "libavcodec/vulkan_h264.c" (with macros evaluated):

```
StdVideoH264ScalingLists vksps_scaling[32];
```

3844B * 32 = 123008B = ~123KB

That's almost the whole default thread stack size on musl!

A simple workaround (that I confirmed fixes the segfault) would be to
change the default thread size through the PT_GNU_STACK program header
(set in ffmpeg's case by configuring with
`--extra-ldflags='-Wl,-z,stack-size=SIZE'`) but I wonder if a simple
patch to set the decoding thread's stack size to something bigger
through `pthread_attr_setstacksize` would be appreciated.

Thanks in advance.