[FFmpeg-devel] [V2 1/2] lavc/vaapi_encode: fix the caculation overflow

[Li, Zhong]

> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces at ffmpeg.org] On Behalf
> Of Pengfei Qu
> Sent: Monday, March 26, 2018 1:51 PM
> To: ffmpeg-devel at ffmpeg.org
> Cc: Qu, Pengfei <pengfei.qu at intel.com>
> Subject: [FFmpeg-devel] [V2 1/2] lavc/vaapi_encode: fix the caculation
> overflow
> 
> this fix the overflow during the caculation before value assignment.
> 
> Signed-off-by: Pengfei Qu <Pengfei.Qu at intel.com>
> ---
>  libavcodec/vaapi_encode.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/libavcodec/vaapi_encode.c b/libavcodec/vaapi_encode.c index
> 36c85a3..78347d4 100644
> --- a/libavcodec/vaapi_encode.c
> +++ b/libavcodec/vaapi_encode.c
> @@ -1168,9 +1168,9 @@ static av_cold int
> vaapi_encode_init_rate_control(AVCodecContext *avctx)
>              rc_target_percentage = 100;
>          } else {
>              rc_bits_per_second   = avctx->rc_max_rate;
> -            rc_target_percentage = (avctx->bit_rate * 100) /
> rc_bits_per_second;
> +            rc_target_percentage = (unsigned long)(avctx->bit_rate *
> + 100) / rc_bits_per_second;
>          }
> -        rc_window_size = (hrd_buffer_size * 1000) / avctx->bit_rate;
> +        rc_window_size = (unsigned long)(hrd_buffer_size * 1000) /
> + avctx->bit_rate;

Should better to use " uint64_t".

>      }
> 
>      ctx->rc_params.misc.type = VAEncMiscParameterTypeRateControl;
> --
> 2.9.3