[FFmpeg-trac] #1566(avcodec:new): incorrect assembly code in libavcodec/x86/dsputil_mmx.c

[FFmpeg]

#1566: incorrect assembly code in libavcodec/x86/dsputil_mmx.c
-------------------------------------+-------------------------------------
             Reporter:  yang         |                     Type:  defect
               Status:  new          |                 Priority:  important
            Component:  avcodec      |                  Version:  git-
             Keywords:  dsputil_mmx  |  master
             Blocking:               |               Blocked By:
Analyzed by developer:  0            |  Reproduced by developer:  0
-------------------------------------+-------------------------------------
 Summary of the bug:
 In file libavcodec/x86/dsputil_mmx.c, function
 ff_put_pixels_clamped_mmx(), there are two assembly code blocks. In the
 first block (in the unrolled loop), the instructions "movq   8%3, %%mm1
 \n\t" etc have problem.
 For above instruction, it is clear what the programmer wants: a load from
 p + 8.  But this assembly code doesn’t guarantee that.  It only works if
 the compiler puts p in a register to produce an instruction like this:
 “movq 8(%edi), %mm1”.  During compiler optimization,  it is possible that
 the compiler will be able to constant propagate into p.  Suppose p =
 &x[10000].  Then operand 3 can become 10000(%edi), where %edi holds &x.
 And the instruction becomes “movq 810000(%edx)”.  That is, it will stride
 by 810000 instead of 8.
 This will cause the segmentation fault.
 This error was fixed in the second block of the assembly code, but not in
 the unrolled loop.

 How to reproduce:
 {{{
 This error is exposed when we build the ffmpeg using Intel C++ Compiler,
 IPO+PGO optimization. The ffmpeg was crashed when decoding a mjpeg video.
 }}}

-- 
Ticket URL: <https://ffmpeg.org/trac/ffmpeg/ticket/1566>
FFmpeg <http://ffmpeg.org>
FFmpeg issue tracker