[FFmpeg-devel] lurking bugs in the mmx-related assembler code (?)

[wm4]

On Sat, 1 Oct 2016 17:37:50 +0200
u-h8zb at aetey.se wrote:

> Hello,
> 
> Would someone familiar with the mmx assembler parts of ffmpeg
> look over and check that the fp-state is restored as needed?
> 
> This is crucial with musl libc which uses floating point in its
> malloc() implementation.
> 
> My troubleshooting strongly suggests that it is the mmx code which is
> the culprit of ffmpeg misbehavior (the same issue has been recently
> fixed in libtheora).
> 
> Ffmpeg built against musl behaves erratically on certain combinations of
> inputs and outputs, which may also depend on other settings.
> Most often ffmpeg hangs forever quite early while beginning the conversion
> (at "frame=    0 fps...." or even before showing this) but sometimes it
> segfaults at that point or while being manually interrupted.
> 
> At least the internal theora decoder is affected but there may be other
> misbehaving decoders or encoders. I think I observed similar breakage with
> other file types than ogg/theora but did not document all the tested
> combinations.
> 
> Affected ffmpeg versions: tested 2.8.8, 3.0.2, 3.1.3, same behaviour,
> also when building with various gcc versions (4.x, 5.x, 6.x).
> 
> Disabling assembler in ffmpeg gets rid of the problem.
> 
> Given patches/fixes I will be glad to test them and report here.
> 
> Regards,
> Rune
> P.S. an attempt to report via trac did not succeed, that's why posting
> to the list

AFAIK most MMX code in FFmpeg does not run emms (i.e. keeps the FPU
state trashed) until returning to the API user.