[FFmpeg-devel] [PATCH] swscale: fix gbrap16 alpha channel issues

[James Cowgill]

Hi,

On 02/08/17 23:21, Michael Niedermayer wrote:
> On Wed, Aug 02, 2017 at 03:32:04PM +0100, James Cowgill wrote:
>> Hi,
>>
>> On 02/08/17 14:18, Michael Niedermayer wrote:
>>> On Tue, Aug 01, 2017 at 02:46:22PM +0100, James Cowgill wrote:
>>>> Fixes filter-pixfmts-scale test failing on big-endian systems due to
>>>> alpSrc not being cast to (const int32_t**).
>>>>
>>>> Also fixes distortions in the output alpha channel values by copying the
>>>> alpha channel code from the rgba64 case found elsewhere in output.c.
>>>>
>>>> Fixes ticket 6555.
>>>>
>>>> Signed-off-by: James Cowgill <James.Cowgill at imgtec.com>
>>>> ---
>>>>  libswscale/output.c                 | 15 ++++++++-------
>>>>  tests/ref/fate/filter-pixfmts-scale |  4 ++--
>>>>  2 files changed, 10 insertions(+), 9 deletions(-)
>>>>
>>>> diff --git a/libswscale/output.c b/libswscale/output.c
>>>> index 9774e9f327..8e5ec0a256 100644
>>>> --- a/libswscale/output.c
>>>> +++ b/libswscale/output.c
>>>> @@ -2026,17 +2026,18 @@ yuv2gbrp16_full_X_c(SwsContext *c, const int16_t *lumFilter,
>>>>                      const int16_t **lumSrcx, int lumFilterSize,
>>>>                      const int16_t *chrFilter, const int16_t **chrUSrcx,
>>>>                      const int16_t **chrVSrcx, int chrFilterSize,
>>>> -                    const int16_t **alpSrc, uint8_t **dest,
>>>> +                    const int16_t **alpSrcx, uint8_t **dest,
>>>>                      int dstW, int y)
>>>>  {
>>>>      const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(c->dstFormat);
>>>>      int i;
>>>> -    int hasAlpha = (desc->flags & AV_PIX_FMT_FLAG_ALPHA) && alpSrc;
>>>> +    int hasAlpha = (desc->flags & AV_PIX_FMT_FLAG_ALPHA) && alpSrcx;
>>>>      uint16_t **dest16 = (uint16_t**)dest;
>>>>      const int32_t **lumSrc  = (const int32_t**)lumSrcx;
>>>>      const int32_t **chrUSrc = (const int32_t**)chrUSrcx;
>>>>      const int32_t **chrVSrc = (const int32_t**)chrVSrcx;
>>>> -    int A = 0; // init to silence warning
>>>> +    const int32_t **alpSrc  = (const int32_t**)alpSrcx;
>>>
>>>> +    int A = 0xFFFF << 14;
>>>
>>> unused value
>>
>> The initial value of A is unused in the old code, but not in the new code.
> 
> IIRC all uses are under hasAlpha and it is writen to in that case first

Sorry, you're right. I think I was looking at the code from yuv2rgba64.
I'll send a v2.

Thanks,
James