[FFmpeg-cvslog] avcodec/hqxdsp: Fix multiple runtime error: signed integer overflow: 248220 * 21407 cannot be represented in type 'int' in idct_col()

Michael Niedermayer git at videolan.org
Mon May 8 13:35:20 EEST 2017


ffmpeg | branch: master | Michael Niedermayer <michael at niedermayer.cc> | Mon May  8 12:04:09 2017 +0200| [5d5118f81bd51b9c33500616b3c637123e8e4691] | committer: Michael Niedermayer

avcodec/hqxdsp: Fix multiple runtime error: signed integer overflow: 248220 * 21407 cannot be represented in type 'int' in idct_col()

Fixes: 1405/clusterfuzz-testcase-minimized-5011491835084800

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpeg
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=5d5118f81bd51b9c33500616b3c637123e8e4691
---

 libavcodec/hqxdsp.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/libavcodec/hqxdsp.c b/libavcodec/hqxdsp.c
index feff9c0b68..04a65e7767 100644
--- a/libavcodec/hqxdsp.c
+++ b/libavcodec/hqxdsp.c
@@ -39,18 +39,18 @@ static inline void idct_col(int16_t *blk, const uint8_t *quant)
     s6 = (int) blk[6 * 8] * quant[6 * 8];
     s7 = (int) blk[7 * 8] * quant[7 * 8];
 
-    t0  =  (s3 * 19266 + s5 * 12873) >> 15;
-    t1  =  (s5 * 19266 - s3 * 12873) >> 15;
-    t2  = ((s7 * 4520  + s1 * 22725) >> 15) - t0;
-    t3  = ((s1 * 4520  - s7 * 22725) >> 15) - t1;
+    t0  =  (int)(s3 * 19266U + s5 * 12873U) >> 15;
+    t1  =  (int)(s5 * 19266U - s3 * 12873U) >> 15;
+    t2  = ((int)(s7 * 4520U  + s1 * 22725U) >> 15) - t0;
+    t3  = ((int)(s1 * 4520U  - s7 * 22725U) >> 15) - t1;
     t4  = t0 * 2 + t2;
     t5  = t1 * 2 + t3;
     t6  = t2 - t3;
     t7  = t3 * 2 + t6;
     t8  = (t6 * 11585) >> 14;
     t9  = (t7 * 11585) >> 14;
-    tA  = (s2 * 8867 - s6 * 21407) >> 14;
-    tB  = (s6 * 8867 + s2 * 21407) >> 14;
+    tA  = (int)(s2 * 8867U - s6 * 21407U) >> 14;
+    tB  = (int)(s6 * 8867U + s2 * 21407U) >> 14;
     tC  = (s0 >> 1) - (s4 >> 1);
     tD  = (s4 >> 1) * 2 + tC;
     tE  = tC - (tA >> 1);



More information about the ffmpeg-cvslog mailing list