[FFmpeg-devel] [RFC] swscale dithering

[Niklas Haas]

On Mon, 24 Mar 2025 21:04:46 +0100 Michael Niedermayer <michael at niedermayer.cc> wrote:
> Hi Niklas
> 
> On Mon, Mar 24, 2025 at 01:43:19PM +0100, Niklas Haas wrote:
> > Hi all,
> > 
> > As part of my ongoing swscale rewrite, we have both the opportunity and the
> > need to make a central decision about how to apply rounding and/or dithering.
> > 
> > Some particular cases I want to point out and gather feedback on include:
> > 
> 
> all IMHO:
> 
> 
> > 1. Should we dither and/or accurately round when scaling up full range
> >    content? For example, say you are converting from full-range rgb24 to
> 
> by default, yes
> 
> 
> [...]
> 
> > 2. At what bit depth does dithering become negligible? For context, the
> 
> I think we should consistently always apply it by default
> 
> also especially if someone does work with lets say 32bit, that person has
> some strange requirements already and direct human vison may not be it.
> 
> 
> [...]
> 
> > 3. Should we dither per-channel after conversion from grayscale to RGB? For
> 
> in general, yes
> 
> 
> [...]
> > 
> > 3. What should we make of the SWS_ACCURATE_RND and SWS_BITEXACT flags? I am
> >    personally thinking that SWS_BITEXACT should become a no-op flag, with
> >    bit exact output being the default behavior of all new implementations.
> >    But What about SWS_ACCURATE_RND?
> > 
> >    I am thinking that SWS_ACCURATE_RND should essentially be the switch that
> >    toggles our preferred resolution of question 1. So in other words, with
> >    SWS_ACCURATE_RND specified, full range upconversions should go through an
> >    accurate dither pass, while being relaxed to the simple (x << 2) | (x >> 6)
> >    upconversion in the absence of this flag.
> > 
> >    How should this flag relate to question 2? With the flag specified, I am
> >    thinking that we should also force dithering even at 16 bit depth, and
> >    skip dithering in this case only in the flag's absence. If so, what
> >    bit depth should the cutoff threshold be, for when to skip accurate
> >    dithering? I am thinking to simply use the 12/14 bit SDR/HDR threshold as
> >    appropriate for the content type.
> > 
> > This would lead to the following conversions, as an illustration:
> > 
> > SWS_ACCURATE_RND specified:
> > 
> > - rgb24 -> yuv420p10:       full dithering
> > - rgb24 -> yuv420p12:       full dithering
> > - rgb24 -> rgb30:           full dithering
> > - rgb24 -> rgba64:          full dithering
> > - yuva444p -> yuva444p10:   scale YUV, dither alpha
> > - yuva444p14 -> yuva444p16: scale YUV, dither alpha
> > - yuv444p10 -> yuv444p14:   left shift, no dithering needed
> > 
> > SWS_ACCURATE_RND absent:
> > 
> > - rgb24 -> yuv420p10:       full dithering
> > - rgb24 -> yuv420p12:       truncate if SDR, full dithering if HDR
> > - rgb24 -> rgb30:           truncate
> > - rgb24 -> rgba64:          truncate
> > - yuva444p -> yuva444p10:   left shift YUV, truncate alpha
> > - yuva444p14 -> yuva444p16: left shift YUV, truncate alpha
> > 
> > Does this seem reasonable?
> 
> IMHO in the accurate mode, dither should always be on
> its also easier to understand

That seems reasonable. It does mean some conversions are necessarily going
to get slower than the status quo.

> 
> but there could be a flag for vissual percetion
> 
> SWS_VISSUAL_PERCEPTION (or some better name)
> some flag that uses less accurate and faster operations when their
> effect is expected to be vissually impercivable

This could be the behavior of SWS_DITHER_AUTO, which is not clearly defined
either way.

In a much earlier thread we discussed the idea of adding quality "presets",
which seems like a good thing to consider as well.

How about this proposal?

1. SWS_ACCURATE_RND is added to the default flags.
2. When SWS_ACCURATE_RND is absent, the implementation may truncate instead
   of rounding; to enable e.g. fast alpha upconversions.
3. SWS_DITHER_AUTO implies dithering only when the result is visually needed,
   and skips dithering otherwise
4. When a specific dither mode is requested, dithering is always performed,
   at whatever bit depth.

Tying this into the quality presets, I suggest the "default" quality prefix
imply (accurate_rnd + dither=auto). One of the "slower" presets could
always enable dithering, while one of the "faster" presets could skip
accurate rounding.

> 
> thx
> 
> [...]
> 
> -- 
> Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
> 
> I have often repented speaking, but never of holding my tongue.
> -- Xenocrates
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".