[FFmpeg-trac] #10716(swscale:open): swscale produces bright green when going from grayf32 to gbrpf32

[FFmpeg]

#10716: swscale produces bright green when going from grayf32 to gbrpf32
-------------------------------------+-------------------------------------
             Reporter:  Leo Izen     |                    Owner:  Niklas
                                     |  Haas
                 Type:  defect       |                   Status:  open
             Priority:  normal       |                Component:  swscale
              Version:  git-master   |               Resolution:
             Keywords:  swscale      |               Blocked By:
             Blocking:               |  Reproduced by developer:  1
Analyzed by developer:  1            |
-------------------------------------+-------------------------------------
Changes (by Niklas Haas):

 * analyzed:  0 => 1
 * owner:  (none) => Niklas Haas
 * reproduced:  0 => 1

Comment:

 I had a look at this, and I think the error is down to a confusion about
 what internal formats swscale is using.

 In input.c and output.c and many other places, swscale follows the rule of
 using 15-bit intermediate if output bpc is <= 8, and 19-bit (inside
 int32_t) intermediate otherwise. See e.g. the comments on hyScale() on
 swscale_internal.h. These are also the coefficients that
 yuv2gbrpf32_full_X_c() is using.

 In contrast to this, the plane init code in slice.c (function fill_ones)
 is assuming that we use 35-bit intermediates (inside 64-bit integers) for
 this case, seemingly added by commit
 b4967fc71c63eae8cd96f9c46cd3e1fbd705bbf9 with no further justification.

 As such, I propose simply dropping this case from fill_ones(), as 64-bits
 precision is way more than conceivably necessary for any realistic use
 case, and even 32 bits of precision is arguably overkill for anything
 other than maintaining bit exactness with some abstract mathematical
 reference. (Something that would require also extending the precision of
 all conversion tables, yuv coefficients, scaling filters and so on)
-- 
Ticket URL: <https://trac.ffmpeg.org/ticket/10716#comment:2>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker