[FFmpeg-devel] [RFC]] swscale modernization proposal

[Niklas Haas]

On Sat, 29 Jun 2024 15:41:58 +0800 Zhao Zhili <quinkblack at foxmail.com> wrote:
> 
> 
> > On Jun 22, 2024, at 21:13, Niklas Haas <ffmpeg at haasn.xyz> wrote:
> > 
> > Hey,
> > 
> > As some of you know, I got contracted (by STF 2024) to work on improving
> > swscale, over the course of the next couple of months. I want to share my
> > current plans and gather feedback + measure sentiment.
> > 
> > ## Problem statement
> > 
> > The two issues I'd like to focus on for now are:
> > 
> > 1. Lack of support for a lot of modern formats and conversions (HDR, ICtCp,
> >   IPTc2, BT.2020-CL, XYZ, YCgCo, Dolby Vision, ...)
> > 2. Complicated context management, with cascaded contexts, threading, stateful
> >   configuration, multi-step init procedures, etc; and related bugs
> > 
> > In order to make these feasible, some amount of internal re-organization of
> > duties inside swscale is prudent.
> > 
> > ## Proposed approach
> > 
> > The first step is to create a new API, which will (tentatively) live in
> > <libswscale/avscale.h>. This API will initially start off as a near-copy of the
> > current swscale public API, but with the major difference that I want it to be
> > state-free and only access metadata in terms of AVFrame properties. So there
> > will be no independent configuration of the input chroma location etc. like
> > there is currently, and no need to re-configure or re-init the context when
> > feeding it frames with different properties. The goal is for users to be able
> > to just feed it AVFrame pairs and have it internally cache expensive
> > pre-processing steps as needed. Finally, avscale_* should ultimately also
> > support hardware frames directly, in which case it will dispatch to some
> > equivalent of scale_vulkan/vaapi/cuda or possibly even libplacebo. (But I will
> > defer this to a future milestone)
> > 
> > After this API is established, I want to start expanding the functionality in
> > the following manner:
> > 
> > ### Phase 1
> > 
> > For basic operation, avscale_* will just dispatch to a sequence of swscale_*
> > invocations. In the basic case, it will just directly invoke swscale with
> > minimal overhead. In more advanced cases, it might resolve to a *sequence* of
> > swscale operations, with other operations (e.g. colorspace conversions a la
> > vf_colorspace) mixed in.
> > 
> > This will allow us to gain new functionality in a minimally invasive way, and
> > will let API users start porting to the new API. This will also serve as a good
> > "selling point" for the new API, allowing us to hopefully break up the legacy
> > swscale API afterwards.
> > 
> > ### Phase 2
> > 
> > After this is working, I want to cleanly separate swscale into two distinct
> > components:
> > 
> > 1. vertical/horizontal scaling
> > 2. input/output conversions
> > 
> > Right now, these operations both live inside the main SwsContext, even though
> > they are conceptually orthogonal. Input handling is done entirely by the
> > abstract callbacks lumToYV12 etc., while output conversion is currently
> > "merged" with vertical scaling (yuv2planeX etc.).
> > 
> > I want to cleanly separate these components so they can live inside independent
> > contexts, and be considered as semantically distinct steps. (In particular,
> > there should ideally be no more "unscaled special converters", instead this can
> > be seen as a special case where there simply is no vertical/horizontal scaling
> > step)
> > 
> > The idea is for the colorspace conversion layer to sit in between the
> > input/output converters and the horizontal/vertical scalers. This all would be
> > orchestrated by the avscale_* abstraction.
> > 
> > ## Implementation details
> > 
> > To avoid performance loss from separating "merged" functions into their
> > constituents, care needs to be taken such that all intermediate data, in
> > addition to all involved look-up tables, will fit comfortably inside the L1
> > cache. The approach I propose, which is also (afaict) used by zscale, is to
> > loop over line segments, applying each operation in sequence, on a small
> > temporary buffer.
> > 
> > e.g.
> > 
> > hscale_row(pixel *dst, const pixel *src, int img_width)
> > {
> >    const int SIZE = 256; // or some other small-ish figure, possibly a design
> >                          // constant of the API so that SIMD implementations
> >                          // can be appropriately unrolled
> > 
> >    pixel tmp[SIZE];
> >    for (i = 0; i < img_width; i += SIZE) {
> >        int pixels = min(SIZE, img_width - i);
> > 
> >        { /* inside read input callback */
> >            unpack_input(tmp, src, pixels);
> >            // the amount of separation here will depend on the performance
> >            apply_matrix3x3(tmp, yuv2rgb, pixels);
> >            apply_lut3x1d(tmp, gamma_lut, pixels);
> >            ...
> >        }
> > 
> >        hscale(dst, tmp, filter, pixels);
> > 
> >        src += pixels;
> >        dst += scale_factor(pixels);
> >    }
> > }
> > 
> > This function can then output rows into a ring buffer for use inside the
> > vertical scaler, after which the same procedure happens (in reverse) for the
> > final output pass.
> > 
> > Possibly, we also want to additionally limit the size of a row for the
> > horizontal scaler, to allow arbitrary large input images.
> 
> I did a simple benchmark to compare the performance between libswscale and
> libyuv. With Apple M1 arm64, libyuv is about 10 times faster than libswscale for
> unscaled rgba to yuv420p. After recently aarch64 neon optimizations, libyuv is
> still 5 times faster than libswscale. The situation isn’t much better with scaled
> conversion.
> 
> Sure libswscale has more features and can be more precise than libyuv. Hope
> we can catch up the performance after your refactor.

AFAICT, libyuv does not do any dithering nor advanced filtering, which
swscale is capable of. They also do processing at a pretty low bit
depth, e.g. converting from 8-bit yuv420p straight to 8-bit rgba before
scaling at, you guessed it, 8-bit resolution. (libswscale would use
15-bit here)

That said, there's probably still something we can/should learn from
this implementation.

> 
> > 
> > ## Comments / feedback?
> > 
> > Does the above approach seem reasonable? How do people feel about introducing
> > a new API vs. trying to hammer the existing API into the shape I want it to be?
> > 
> > I've attached an example of what <avscale.h> could end up looking like. If
> > there is broad agreement on this design, I will move on to an implementation.
> > <avscale.h>_______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel at ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > 
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".