[FFmpeg-trac] #9945(avcodec:new): undefined behavior at libavcodec/jpeg2000dwt.c:84:18

[FFmpeg]

#9945: undefined behavior at libavcodec/jpeg2000dwt.c:84:18
-------------------------------------+-------------------------------------
             Reporter:  Johannes     |                     Type:  defect
  Kauffmann                          |
               Status:  new          |                 Priority:  normal
            Component:  avcodec      |                  Version:  git-
                                     |  master
             Keywords:  jpeg2000dwt  |               Blocked By:
             Blocking:               |  Reproduced by developer:  0
Analyzed by developer:  0            |
-------------------------------------+-------------------------------------
 UBSan complains about undefined behaviour when running {{{make check}}}

 How to reproduce:
 {{{
 mkdir build && cd build
 ../configure --toolchain=gcc-usan --enable-debug=2 --disable-stripping
 --disable-doc
 UBSAN_OPTIONS=print_stacktrace=1:halt_on_error=1 make -j8 check
 }}}

 ... which prints:
 {{{
 Test j2k-dwt failed. Look at tests/data/fate/j2k-dwt.err for details.
 make: *** [/home/user/ffmpeg/tests/Makefile:307: fate-j2k-dwt] Error 1
 make: *** Waiting for unfinished jobs....
 }}}


 {{{j2k-dwt.err}}} contains:
 {{{
 src/libavcodec/jpeg2000dwt.c:84:18: runtime error: left shift of negative
 value -66
     #0 0x55863fe454bd in ff_dwt_encode.cold
 (/home/user/ffmpeg/build/libavcodec/tests/jpeg2000dwt+0x144bd)
     #1 0x55863fe5a046 in test_dwt src/libavcodec/tests/jpeg2000dwt.c:44
     #2 0x55863fe465f0 in main src/libavcodec/tests/jpeg2000dwt.c:134
     #3 0x7fd8ddf2fd8f in __libc_start_call_main
 ../sysdeps/nptl/libc_start_call_main.h:58
     #4 0x7fd8ddf2fe3f in __libc_start_main_impl ../csu/libc-start.c:392
     #5 0x55863fe48204 in _start
 (/home/user/ffmpeg/build/libavcodec/tests/jpeg2000dwt+0x17204)
 }}}

 Tested with:
 ffmpeg git dd2ea014ef273157fe9a0e928e77841fbbee9b2f
 GCC 11.2.0
 Ubuntu 22.04.
-- 
Ticket URL: <https://trac.ffmpeg.org/ticket/9945>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker