[FFmpeg-devel] Is this the expected behavior in libswscale/input.c

[Ronald S. Bultje]

Hi,

On Thu, Jun 9, 2016 at 4:02 PM, Dan Parrot <dan.parrot at mail.com> wrote:

> Line 72 of libswscale/input.c is:
> dstU[i] = (ru*r + gu*g + bu*b + (0x10001<<(RGB2YUV_SHIFT-1))) >>
> RGB2YUV_SHIFT;
>
> The definition of macro RGB2YUV_SHIFT in libswscale/swscale_internal.h
> is on line 417:
> #define RGB2YUV_SHIFT 15
>
> By examining the result of executing line 72 in input.c it appears that
> the radix used for macro RGB2YUV_SHIFT is hexadecimal. So that 0x10001
> is shifted left by 20 in subexpression 0x10001<<(RGB2YUV_SHIFT-1) with
> the result being 0x100000.
>
> I was expecting a left-shift of 14, given the macro definition.
>
> Should the macro be interpreted as decimal 15 or hexadecimal 15?


It's decimal 15, it should shift by 14. Check disassembly, though... It's
almost impossible that your compiler would read this as hex and still be
able to produce runnable applications. Maybe submit a bug report to your
compiler's bug tracker?

Ronald