[FFmpeg-trac] #6763(swscale:new): swscale: Out-of-bounds memory accesses

[FFmpeg]

#6763: swscale: Out-of-bounds memory accesses
---------------------------------+---------------------------------------
             Reporter:  Gramner  |                     Type:  defect
               Status:  new      |                 Priority:  normal
            Component:  swscale  |                  Version:  unspecified
             Keywords:           |               Blocked By:
             Blocking:           |  Reproduced by developer:  0
Analyzed by developer:  0        |
---------------------------------+---------------------------------------
 Many assembly functions in swscale will read past the end of their input
 buffers which causes segfaults and/or bus errors if the buffer happens to
 be located near the end of a memory page and the next page is invalid.

 Aligning input buffers isn't even enough for formats like RGB24 (and
 requiring alignment would be a bad idea anyway since it wouldn't work with
 memory-mapped input files for example).

 Using swscale with x264 CLI seems to be a fairly consistent way to trigger
 such out-of-bounds crashes. This command line for example will cause
 segfaults in ff_rgb24ToY_avx():

 ./x264 -o /dev/null --input-csp rgb --input-res 512x512 <any_input_file>

 If asm is disabled in swscale the problem goes away.

--
Ticket URL: <https://trac.ffmpeg.org/ticket/6763>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker