[FFmpeg-devel] [PATCH] avformat/ffrtmpcrypt: Fix int-conversion warning

[Martin Storsjö]

Hi Frank,

On Fri, 22 Dec 2023, Frank Plowman wrote:

> The gcrypt definition of `bn_new` used to use `AVERROR`, however it is
> called in `dh_generate_key` and `ff_dh_init` which return pointers. As a
> result, compiling with gcrypt and the ffrtmpcrypt protocol resulted in an
> int-conversion warning. GCC 14 may upgrade these to errors [1].

(FWIW, the issue that bn_new was used in functions that don't return an 
error-signaling integer was present when this macro was added originally 
in d50b5d547f4070678c88aa095b5292c872e2c1dc to.)

The change LGTM, but the wording here is slightly confusing IMO. The 
problem isn't with using per se AVERROR, that's just a macro for 
generating suitable integers, the issue is more about the fact that we're 
returning from a macro, without knowing the actual context where the macro 
is invoked.

WDYT about this wording?

> The gcrypt definition of `bn_new` used to use the return statement on 
> errors, with an AVERROR return value, regardless of the signature of the 
> function where the macro is used - it is called in `dh_generate_key` and 
> `ff_dh_init` which return pointers. As a result, compiling with gcrypt 
> and the ffrtmpcrypt protocol resulted in an int-conversion warning. GCC 
> 14 may upgrade these to errors [1].

// Martin