[FFmpeg-devel] [PATCH 03/16] avutil: add ioctl definitions for tegra devices

[averne]

Le 31/05/2024 à 23:16, Timo Rothenpieler a écrit :
> On 31.05.2024 23:06, averne wrote:
>> Le 30/05/2024 à 22:42, Rémi Denis-Courmont a écrit :
>>> Le torstaina 30. toukokuuta 2024, 22.43.05 EEST averne a écrit :
>>>> These files are taken with minimal modifications from nvidia's Linux4Tegra
>>>> (L4T) tree. nvmap enables management of memory-mapped buffers for hardware
>>>> devices. nvhost enables interaction with different hardware modules
>>>> (multimedia engines, display engine, ...), through a common block, host1x.
>>>>
>>>> Signed-off-by: averne <averne381 at gmail.com>
>>>> ---
>>>>   libavutil/Makefile       |   2 +
>>>>   libavutil/nvhost_ioctl.h | 511 +++++++++++++++++++++++++++++++++++++++
>>>>   libavutil/nvmap_ioctl.h  | 451 ++++++++++++++++++++++++++++++++++
>>>>   3 files changed, 964 insertions(+)
>>>>   create mode 100644 libavutil/nvhost_ioctl.h
>>>>   create mode 100644 libavutil/nvmap_ioctl.h
>>>>
>>>> diff --git a/libavutil/Makefile b/libavutil/Makefile
>>>> index 6e6fa8d800..9c112bc58a 100644
>>>> --- a/libavutil/Makefile
>>>> +++ b/libavutil/Makefile
>>>> @@ -52,6 +52,8 @@ HEADERS = adler32.h
>>>>               \ hwcontext_videotoolbox.h
>>>>   \ hwcontext_vdpau.h                                             \
>>>> hwcontext_vulkan.h                                            \ +
>>>> nvhost_ioctl.h                                                \ +
>>>> nvmap_ioctl.h                                                 \ iamf.h
>>>>                                                    \ imgutils.h
>>>>                                        \ intfloat.h
>>>>                            \ diff --git a/libavutil/nvhost_ioctl.h
>>>> b/libavutil/nvhost_ioctl.h
>>>> new file mode 100644
>>>> index 0000000000..b0bf3e3ae6
>>>> --- /dev/null
>>>> +++ b/libavutil/nvhost_ioctl.h
>>>> @@ -0,0 +1,511 @@
>>>> +/*
>>>> + * include/uapi/linux/nvhost_ioctl.h
>>>
>>> Well, then that should be provided by linux-libc-dev or equivalent. I don't
>>> think that this should be vendored into FFmpeg.
>>>
>>
>> Agreed. On L4T this is provided by nvidia-l4t-kernel-headers, but
>> on the HOS side there is no such equivalent yet. If this patch
>> series moves forward, I will integrate the relevant bits in libnx
>> and get rid of those headers.
>> As for the hardware definitions (in the following patch), I think
>> they should be put in nv-codec-headers.
> 
> I disagree there, the nv-codec-headers track the upstream codec SDK.
> Making it track two independent things would be a mess.
> 
> This patchset is implementing parts of an entire video-driver into FFmpeg. Which really looks out of scope to me.
> Can't all that be moved into a library, which then also comes with the necessary headers for applications to use it?
> 
> On that note, how stable are those headers? Given they're part of the nvidia driver, couldn't they randomly change at any time?

Hi, firstly those uapi headers are stable as can be. They haven't 
changed throughout all the L4T releases, from the TX1 to the latest 
SoC. You can check the packages here: 
https://repo.download.nvidia.com/jetson. 
Ideally I would use a new uapi nvidia has been working on upstream: 
https://github.com/torvalds/linux/blob/master/include/uapi/drm/tegra_drm.h. 
Unfortunately, the kernel that comes with the Jetson Nano is too old 
for it. The Switch also doesn't have this API.
Secondly, the hardware headers in the next patch cannot change, 
since they represent hardware structures and constants. Also one can 
note how nvidia preserves backwards compatibility between engine 
releases: new features are programmed through extensions to the main 
structures (see for instance the v1/v2/v3 substructures in 
nvdec_hevc_pic_s).

I'll also respond to Rémi's question here. The nvdec hwaccel relies 
on the cuda framework, which is present on L4T (not cuvid though) 
but not the Switch. So I would have to implement a good part of the 
cuda runtime on the Switch, while making FFmpeg and cuda on L4T happy 
about it. That feels like even more work than a vaapi implementation, 
and frankly, as an unpaid hobbyist, I really don't feel like doing 
what should be nvidia's job.
That leaves the option of moving the code to a library of my own and 
abstract away the hardware details. I don't think that comes with much 
benefit, except moving parts of the code outside FFmpeg, and you'll 
notice there isn't so much of it (the decoder implementations are 
usually shorter than their vaapi counterpart, for instance). It also 
comes at some performance cost, like I said in my previous answer to 
Rémi. 
Finally, I'm not sure coming up with yet another hwaccel API for a 
niche platform would be any more acceptable for upstreaming into 
FFmpeg.