[FFmpeg-trac] #8747(swscale:new): libswscale 4.3 crash if output buffer is not 16 bytes aligned for yuv2rgb conversion

[FFmpeg]

#8747: libswscale 4.3 crash if output buffer is not 16 bytes aligned for yuv2rgb
conversion
------------------------------------+-----------------------------------
             Reporter:  melanconj   |                    Owner:
                 Type:  defect      |                   Status:  new
             Priority:  important   |                Component:  swscale
              Version:  git-master  |               Resolution:
             Keywords:  regression  |               Blocked By:
             Blocking:              |  Reproduced by developer:  0
Analyzed by developer:  0           |
------------------------------------+-----------------------------------

Comment (by melanconj):

 Replying to [comment:6 cehoyos]:
 > Replying to [comment:3 melanconj]:
 > > Apologies, I should have mentioned our real world use case in the
 report. We've noticed this regression when trying to integrate 4.3 in our
 application. Our test infrastructure has been crashing consistently with
 the new version. In our case, the application is a C# program doing
 interop into swscale.
 >
 > I suspect the problem is that FFmpeg offers C libraries and that there
 may be no system with SSSE3 support where your crash can be reproduced
 using libc's allocation functions. Since you are not using C, you may have
 to compile FFmpeg with --disable-ssse3.

 I'm not sure I understand. The second sample I provided shows a small
 enough buffer will not be aligned. The buffer is also large enough for a
 small but common-enough video resolution in our usecases. So a C program
 can hit the issue on its own.

 The issue #8532 I linked initially also shows the same crash for the input
 buffer. The fix seems to have changed the ssse3 assembly to use an
 unaligned mov for the input buffer. I'm believe the output buffer should
 be treated the same way.

 > Can't you use FFmpeg's allocation functions?

 We integrate with a few other libraries directly and so far have not
 needed to rely on the libraries allocations for frame buffers. We manage
 all that from our own code. This would be a non-trivial change to our
 architecture.

 >
 > There is something else that I would like to understand: Why didn't your
 application crash before using SSE2-optimized functions that also need
 16-byte alignment?

 I'm not very familiar with the ffmpeg codebase, nor with assembly and SSE.
 From looking at libswscale/x86/yuv_2_rgb.asm I see two implementations,
 the MMX one and the new SSSE3. The MMX, which I assume is the older one
 and is the SSE2 your are mentioning, seems to move from the input/output
 buffers using movq, which does not have an alignment requirement.

 The new SSSE3 uses mova, which I understand to be an aligned-move. The fix
 for the input-buffer in issue #8532 in commit
 828f7db5d9fdf9052bb4b6d1b528009fece5bb10 seems to have simply replaced
 them with movu, which does not require 16-byte alignment. I believe that
 since the output buffer may also be user-controlled, then alignment should
 not be assumed and it should use movu too.

--
Ticket URL: <https://trac.ffmpeg.org/ticket/8747#comment:10>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker