[FFmpeg-devel] [PATCH] hwcontext_videotoolbox: add vt_device_derive

[Zhao Zhili]

> 在 2024年2月23日，上午4:19，Gnattu OC via ffmpeg-devel <ffmpeg-devel at ffmpeg.org> 写道：
> 
> Actually, if you examine the `hwcontext_videotoolbox.c` file, you will find that the current function for `vt_device_create` is essentially a no-op: it only returns an error if a user attempts to select a device, otherwise it just returns. This is generally how VideoToolbox works. If the `create_device` function is already a no-op, then the derive operation will certainly also be a no-op.

Device derive means inter operation between different hwcontext, which doesn’t supported by videotoolbox yet, so a no-op implementation for now is misleading.

I’m trying to implement opencl and videotoolbox inter operation, the no-op implementation can make sense afterwards.

> All other hardware filters can utilize the derive API, and there are use cases that depend on this API. Thus, it seems inappropriate to me that VideoToolbox users cannot use the derive API just because VideoToolbox does not require any device creation.
> 
>> A libavfilter API user will set the correct hw_device_ctx on each filter and your use-case will work properly
> 
> Can't this just be the modification of the `hwupload` filter? Instead of derive, the user just set the which initialized hardware to use, as `-init_hw_device` can already be called multiple times and creating alias for each hardware.
> 
>> On Feb 23, 2024, at 03:50, Mark Thompson <sw at jkqxz.net> wrote:
>> 
>> On 22/02/2024 19:39, ChenLiucheng via ffmpeg-devel wrote:
>>>> On Feb 23, 2024, at 03:28, Mark Thompson <sw at jkqxz.net> wrote:
>>>> 
>>>>> On 22/02/2024 18:46, gnattu via ffmpeg-devel wrote:
>>>>>> There is no device context to be setup, nor devices to be
>>>>>> selected with VideoToolbox. Just a simple return would allow
>>>>>> us to use derived device in filters like
>>>>>> `hwupload=derive_device=videotoolbox`
>>>>>> Signed-off-by: Gnattu OC <gnattuoc at me.com>
>>>>>> ---
>>>>>> libavutil/hwcontext_videotoolbox.c | 9 +++++++++
>>>>>> 1 file changed, 9 insertions(+)
>>>>>> diff --git a/libavutil/hwcontext_videotoolbox.c b/libavutil/hwcontext_videotoolbox.c
>>>>>> index fe469dc161..d13199eca7 100644
>>>>>> --- a/libavutil/hwcontext_videotoolbox.c
>>>>>> +++ b/libavutil/hwcontext_videotoolbox.c
>>>>>> @@ -759,6 +759,14 @@ static int vt_device_create(AVHWDeviceContext *ctx, const char *device,
>>>>>>     return 0;
>>>>>> }
>>>>>> +static int vt_device_derive(AVHWDeviceContext *device_ctx,
>>>>>> +                            AVHWDeviceContext *src_ctx, AVDictionary *opts,
>>>>>> +                            int flags)
>>>>>> +{
>>>>>> +    // There is no context to be setup with VT, just return.
>>>>>> +    return 0;
>>>>>> +}
>>>>>> +
>>>>>> const HWContextType ff_hwcontext_type_videotoolbox = {
>>>>>>     .type                 = AV_HWDEVICE_TYPE_VIDEOTOOLBOX,
>>>>>>     .name                 = "videotoolbox",
>>>>>> @@ -766,6 +774,7 @@ const HWContextType ff_hwcontext_type_videotoolbox = {
>>>>>>     .frames_priv_size     = sizeof(VTFramesContext),
>>>>>>       .device_create        = vt_device_create,
>>>>>> +    .device_derive        = vt_device_derive,
>>>>>>     .frames_hwctx_size    = sizeof(AVVTFramesContext),
>>>>>>     .frames_init          = vt_frames_init,
>>>>>>     .frames_get_buffer    = vt_get_buffer,
>>>>> 
>>>>> This derivation behaviour doesn't make any sense inside libavutil.  Features which are only for the ffmpeg utility should be implemented inside the ffmpeg utility.
>>>>> 
>>>>> (Also, try -init_hw_device.)
>>>>> 
>>> Well this does make sense.
>>> 
>>> If only one type of hardware device is in use, then `-init_hw_device` will suffice. However, if we use hardware filters with different device types, such as OpenCL, and we want to switch to a VideoToolbox filter later in the chain, we will have a issue. Since there is no `hwmap` between OpenCL and VideoToolbox, we need to perform a `hwdownload` followed by a `hwupload`. If the `derive_device` option cannot be used with `hwupload` due to `device_derive` not being implemented, the filter will struggle to find the correct device for uploading. It may attempt to upload to the OpenCL device without `device_derive` set if it comes after an OpenCL filter.
>>> 
>> 
>> Yes, it is unfortunate that the ffmpeg utility is missing this feature.
>> 
>> A libavfilter API user will set the correct hw_device_ctx on each filter and your use-case will work properly, but the ffmpeg utility does not fully expose the functionality of libavfilter to be able to do this.
>> 
>> This is mostly for syntax reasons; implementation would be straightforward.  If you can think of a good way to represent this in options to the ffmpeg utility (which doesn't become too horrible with escaping) then I would very much welcome suggestions.
>> 
>> Thanks,
>> 
>> - Mark
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel at ffmpeg.org
>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>> 
>> To unsubscribe, visit link above, or email
>> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email