[FFmpeg-trac] #10144(swscale:new): Segfault only on macOS in sws_scale() for DCI 2K resolution

[FFmpeg]

#10144: Segfault only on macOS in sws_scale() for DCI 2K resolution
-------------------------------------+----------------------------------
             Reporter:  LoadingByte  |                     Type:  defect
               Status:  new          |                 Priority:  normal
            Component:  swscale      |                  Version:  5.1.2
             Keywords:               |               Blocked By:
             Blocking:               |  Reproduced by developer:  0
Analyzed by developer:  0            |
-------------------------------------+----------------------------------
 '''Summary of the bug:'''

 On macOS only, libswscale segfaults when converting certain pixel formats
 on frames with a width of 2048 pixels. This is unfortunate as the DCI 2K
 standard specifies exactly that width. The issue stems from libswscale
 reading 1 byte beyond the input frame's buffer. As such, the current best
 workaround is to enlarge that buffer by 1 byte. Notice that frames
 converted with the workaround on macOS and without the workaround on
 Windows or Linux exactly match.

 '''How to reproduce:'''

 Compile and run the following program on macOS. The h variable can be set
 to any value and the bug still persists. Other outPixFmts like
 YUV422P/YUV422P10LE/YUV444P10LE and other inPixFmts like BGR24 also
 provoke the issue. The workaround appends "+ 1" to the buffer length in
 the calloc() call.

 {{{
 #include <libavutil/imgutils.h>
 #include <libswscale/swscale.h>

 AVFrame *alloc_frame(int w, int h, int pix_fmt) {
     AVFrame *frame = av_frame_alloc();
     frame->format = pix_fmt;
     frame->width = w;
     frame->height = h;
     return frame;
 }

 int main(void) {
     int w = 2048;
     int h = 858;
     int inPixFmt = AV_PIX_FMT_RGB24;
     int outPixFmt = AV_PIX_FMT_BGR24;

     AVFrame *inFrame = alloc_frame(w, h, inPixFmt);
     AVFrame *outFrame = alloc_frame(w, h, outPixFmt);
     struct SwsContext *swsCtx = sws_getContext(w, h, inPixFmt, w, h,
 outPixFmt, 0, NULL, NULL, NULL);

     av_frame_get_buffer(outFrame, 0);
     av_image_fill_arrays(inFrame->data, inFrame->linesize, calloc(w * h *
 3, 1), inPixFmt, w, h, 1);

     sws_scale(swsCtx, inFrame->data, inFrame->linesize, 0, h,
 outFrame->data, outFrame->linesize);
 }
 }}}
-- 
Ticket URL: <https://trac.ffmpeg.org/ticket/10144>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker