[FFmpeg-devel] [PATCH] libavdevice: Add KMS/DRM output device

[Mark Thompson]

On 20/01/2021 11:53, Nicolas George wrote:
> Mark Thompson (12021-01-19):
>> Can you explain in more detail your use of libavdevice here?
>>
>> My general sense is that while they might sometimes be convenient to
>> use if you are already working in a libav* context, they are so
>> limited (by both the API and the implementation) that most users who
>> want video output to a screen will turn more full-featured players to
>> implement this.
>>
>> (To be clear here, I am talking specifically about outputs to a screen
>> like fbdev/xv, not about output to non-screen-like devices such as
>> v4l2 which certainly do have other use-cases.)
> 
> For starter, since libavcodec and libavformat are supposed to be the
> preferred way of dealing with audio and video, we should assume that
> people who develop projects about it will likely have their data already
> in AVFrame and AVPacket format, so the fact that these devices work with
> the same kind of API is a huge plus.
> 
> Second, even though they are limited, they do the work, and for a lot of
> projects that is plenty enough. Their existence in libavdevices avoids
> reinventing the wheel, and their common API makes it easy to switch,
> getting us portability for cheap.
> 
> Also, I will turn the board on you: why do YOU object to these
> components?

Because the libavformat API is not suitable for (especially) these use-cases, so including them in the library feels like a hack that happened to be convenient for some subset of people even though it doesn't really fit and wants something else.

Those components do feel the worst to me, but this is really a general complaint about libavdevice as a whole.  I would prefer that is either mostly moved into the ffmpeg utility (where it has clear use and could be better integrated), or updated to have a device-oriented API (if there are external users, hence the question).

For an API, the things which immediately come to mind are:
* Handle frames as well as packets.
   * Including hardware frames - DRM objects from KMS/V4L2, D3D surfaces from Windows desktop duplication (which doesn't currently exist but should).
* Clear core option set - currently almost everything is set by inconsistent private options; things like pixel/sample format, frame/sample rate, geometry and hardware device should be common options to all.
* Asynchronicity - a big annoyance in current recording scenarios with the ffmpeg utility is that both audio and video capture block, and do so on the same thread which results in skipped frames.
* Capability probing - the existing method of options which log the capabilities are not very useful for API users.

(Not meant to be an exhaustive list.)

Thanks,

- Mark