[FFmpeg-devel] [PATCH 0/4] avdevice/dshow: implement capabilities API

[Diederick C. Niehorster]

Reorganized a bit for easier replying.

Also, while i think this is an important discussion, i do not see why
it should stop de-deprecation of a good API. Deprecating the device
capabilities API cleaned up avformat a bit, but various other function
pointers are left. A redesign would clean them all up at once, if
thats the direction taken. As such, having the device capabilities api
does not create a hindrance to a redesign. As i argued, having a use
example of important functionality helps, not hinders the redesign
effort. And lets not forget it offers users of the current API (me at
least) functionality they need now, not at some indeterminate
timepoint in the future. So, lets not stop work on the current
avdevice component (my patch set) while figuring out the way forward.
Lets keep the below discussion separate, they are two separate points
to decide on I think. What is your view on this Anton / how should
this issue be decided?

On Fri, Jun 11, 2021 at 5:24 PM Anton Khirnov <anton at khirnov.net> wrote:
>
> It is also about new functionality, since one of the main stated
> advantages of libavdevice is that it can be used transparently by all
> programs that use libavformat. New libavdevice-specific APIs
> go against this.
>
> I see a contradiction here. On one hand you're saying that the
> usefulness of lavd comes from it having the same API as lavf. But then
> you want to add a whole bunch of libavdevice-specific APIs. So any
> program that wants to use them has to be specifically programmed for
> libavdevice anyway. And libavformat API is saddled with extra frameworks
> that are of no use to "normal" (de)muxing.
>
> At that point, why insist on accessing lavd through the lavf API? You
> can have a lavd-specific API that can cleanly export everything that is
> specific to capture devices, without ugly hacks that are there
> currently.

I do not think there is a contradiction. Note also that i did not say
i "want to add a whole bunch of libavdevice-specific APIs." For (not
unimporant) convenience, as an API user, i want to use some of the
already existing API to enhance my use of avdevices.

Quoting a bit from Nicolas' reply allows me to make the point that
there is no contradiction:

On Fri, Jun 11, 2021 at 3:17 PM Nicolas George <george at nsup.org> wrote:
> Input devices are demuxers with a few extra methods; output devices are
> muxers with a few extra methods. We already have the beginning of a
> class/interface hierarchy:
>
>         formats
>           |
>           +---- muxers
>           |       |
>           |       +---- output devices
>           |
>           +---- demuxers
>                   |
>                   +---- input devices

Code in my (C++) program looks just like that. I have a
significantly-size class with code for accessing ffmpeg demuxers,
originally written to be able to read files. Then i realized access to
webcams would be important too. It was very simply added by deriving
my webcam class from the general ffmpeg input class and adding a
little bit of convenience functions to deal with device discovery,
proper URL formatting (so users don't have to prepend "video=" for
dshow). I.e., almost all of the code is shared between my class for
general avformats and the avdevice class.

But, for more advanced use than just streaming in packets/frames, I
need to gain a bit of control over the avdevice. These are convenience
things, even if not unimportant, and appropriate APIs are already
available. Thats what i implemented for dshow. Right now i needed
another DirectShow library to do device discovery and given a device,
format discovery. The ffmpeg API is (was) already there, just needed
to be implemented. I also needed a way to pause/resume (just stopping
reading packets like you can do when the input is a file does not work
for a realtime source, where it leads to a full packet list buffer,
and a kinda nasty restart as first you get a bunch of irrelevant old
frames, and may lose a few of the first ones you do want as the buffer
was still full). These bits of additional API thus helps make using
avdevices better, but even without them, i (and i assume many other
happy users out there) already have a very workable solution.

> > (I do see Mark's list of
> > opportunities that a new API offer, copied below). I see Nicolas argue
> > this entanglement of internals is not a problem in practice, and i
> > suppose there is a certain amount of taste involved here.
>
> Do note that Nicolas' position in library separation is quite unorthodox
> --- I am not aware of anyone else supporting it, several people strongly
> disagree with it. It also disagrees with our current practice.
> [...]
>
> The function pointers, various private APIs, contents of
> AVFormatInternal, etc. This is a pretty big deal, since it restricts
> what we can do to libavformat internals without breaking ABI.

This may be above my pay grade, but: ABI here would be ABI between the
various dlls, not user-facing right? (They are internals after all,
and hidden behind opaque pointers). I assume the intention is that a
user uses a set of ffmpeg dlls from the same build, not mix and match?
Are there such issues in that case?

I understand the beauty of clean design and generic interfaces.
Less/no entanglement allows moving faster and more flexibly with
individual components. But i also think that there is a limit to how
separated different components can become. As i wrote in my previous
message:
1. lavd being accessible through the lavf interface is super
convenient and important.
2. if we would not want to lose the possibility to use avdevices
drop-in in the place of AVFormats, if making avdevices separate
completely from avfromats, some kind of adapter/wrapper component that
has access to internals of both is unavoidable.

So a goal, to my personal taste, could/should certainly be to minimize
use of internal APIs across library boundaries. But not eliminate it
completely at the cost of functionality.

I read somewhere on the list that avpriv_ cross-library solutions are
considered a hack. Why? Mixing them up with other actually internal to
a library functions is perhaps what makes things appear messy. Moving
declarations for functions that are not meant to be used by users, but
are part of a cross-library internal API into their own header may
clean things up and make it even clearer where links between the
libraries exist.

When it comes to avdevice, having some data on the situation may be
good. I had a look at how some avdevices (those i have available in my
build environment here on windows/msvc without external libs) depend
on internals of other libraries.
1. #include "libavformat/internal.h": dshow, vfwcap, gdigrab and lavfi
all include this only for the avpriv_set_pts_info() function
2. lavfi furthermore includes libavformat/avio_internal.h and
libavutil/internal.h. I am not sure why, commenting them out does not
lead to any build warnings or errors.

There may be other libraries with closer entanglement, but it seems
minimal for these four devices: one avpriv_ function and the function
pointers in the AVFormat structs. grepping for avpriv_ in the whole
avdevice library, it is indeed mostly avpriv_set_pts_info() and
avpriv_open() that are used.

I note that the internal field of AVFormat does not appear to be
accessed in any of the avdevices at all. Just grep the avdevice
directory for "internal": not a single use.

>
> One thing mentioned by Mark that you cite above is that lavf is designed
> around working with encoded data in the form of AVPackets, whereas for
> some devices handled by lavd it would be better to use decoded frames
> (or even hw surfaces) wrapped in an AVFrame.

Understood.

Cheers,
Dee