[FFmpeg-trac] #3451(swscale:new): sws_scale crashes in high resolutions when using mmx optimization

[FFmpeg]

#3451: sws_scale crashes in high resolutions when using mmx optimization
-------------------------------------+-------------------------------------
             Reporter:  miro82       |                     Type:  defect
               Status:  new          |                 Priority:  normal
            Component:  swscale      |                  Version:
             Keywords:  4k vertical  |  unspecified
  flip mmx                           |               Blocked By:
             Blocking:               |  Reproduced by developer:  0
Analyzed by developer:  0            |
-------------------------------------+-------------------------------------
 Hi, sws_scale crashes in high resolutions (4k & 8k) when using mmx
 optimization and a negative stride for the input frame. I'm developing a
 render engine that uses openGL to render hi-res video for planetariums.
 OpenGL is storing the pixel information vertically flipped compared to
 FFmpeg so I'm flipping the frame in sws_scale. The code i'm using is:


 {{{
 // Fill picture with image
 // pixels is a unsigned char buffer
 avpicture_fill((AVPicture*)tmpFrame, pixels, PIX_FMT_BGR24, mWidth,
 mHeight);

 // Flipping frame
 tmpFrame->data[0] += tmpFrame->linesize[0]*(mHeight-1);
 // Flipping frame
 tmpFrame->linesize[0] = -tmpFrame->linesize[0];

 //convert BGR24 to YUV420
 int ret = sws_scale(sContext, tmpFrame->data, tmpFrame->linesize, 0,
 mHeight, outFrame->data, outFrame->linesize);
 }}}

 The code above works fine for resolutions up to 2k but crashes if I go
 above that. I'm using XCode and the trace looks like this:

 Error line: 1629 rgb2rgb_template.c
 Thread 1, Queue : com.apple.main-thread
 #0      0x0000000101a75ec8 in rgb24toyv12_mmxext at
 /Users/miran46/Downloads/ffmpeg/libswscale/x86/rgb2rgb_template.c:1629
 #1      0x0000000101a47561 in bgr24ToYv12Wrapper at
 /Users/miran46/Downloads/ffmpeg/libswscale/swscale_unscaled.c:1314
 #2      0x0000000101a416c8 in sws_scale at
 /Users/miran46/Downloads/ffmpeg/libswscale/swscale.c:1075
 #3      0x0000000100006b64 in Encoder::addFrame(unsigned char*) at
 /Users/miran46/code/projects/FFMpegCapture/hmap/Encoder.cpp:160

 This problem is resolved by building FFmpeg with: --disable-mmx
 When disabling mmx I have no problems to encode 4096 x 4096 HEVC video.
 The SwsContext is inited using the following code:


 {{{
 sContext = sws_getContext(mWidth, mHeight, AV_PIX_FMT_BGR24,
                 mWidth, mHeight, AV_PIX_FMT_YUV420P,
                 SWS_FAST_BILINEAR, NULL, NULL, NULL);
 }}}


 Best regards,

 Miroslav

--
Ticket URL: <https://trac.ffmpeg.org/ticket/3451>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker