[FFmpeg-devel] MMX function crash with icc -fp-stack-check

[Yves Lefebvre]

Hi,

I spent some time chasing a strange floating point bug in a custom 
encoder under gstreamer on a 64bit xeon. By compiling under icc and 
using the -fp-stack-check option, I was able to narrow it down to 
ffdeinterlace filter MMX code. This option of icc cause a segfault when 
it detect that the floating point stack is not in a proper state. 
Turning off MMX code prevent the crash in ffdeinterlace, and it also 
prevent my strange floating point bug to occur.

I download the latest ffmpeg and compiled it using icc and 
-fp-stack-check option. I was able to reproduce the exact same problem :

under gdb :
Starting program: /home/yves/ffmpeg/ffmpeg_g -i test.avi -vcodec 
rawvideo -an -deinterlace out2.avi
FFmpeg version UNKNOWN, Copyright (c) 2000-2009 Fabrice Bellard, et al.
  configuration: --prefix=/home/yves/ffmpeg/install --cc=icc 
-fp-stack-check
  libavutil     50. 2. 0 / 50. 2. 0
  libavcodec    52.22. 3 / 52.22. 3
  libavformat   52.32. 0 / 52.32. 0
  libavdevice   52. 1. 0 / 52. 1. 0
  libswscale     0. 7. 1 /  0. 7. 1
  built on Mar 23 2009 17:12:05, gcc: Intel(R) C++ gcc 4.2 mode
Input #0, avi, from 'test.avi':
  Duration: 00:00:00.66, start: 0.000000, bitrate: 9133 kb/s
    Stream #0.0: Video: rawvideo, yuv420p, 176x144, 30 tbr, 30 tbn, 30 tbc
Output #0, avi, to 'out2.avi':
    Stream #0.0: Video: rawvideo, yuv420p, 176x144, q=2-31, 200 kb/s, 30 
tbn, 30 tbc
Stream mapping:
  Stream #0.0 -> #0.0
Press [q] to stop encoding

Program received signal SIGSEGV, Segmentation fault.
0x0000000000555a7e in avpicture_deinterlace (dst=0x15d83b0, 
src=0x15cef20, pix_fmt=22867920, width=22867744, height=176)
    at libavcodec/imgconvert.c:1578
1578                deinterlace_bottom_field(dst->data[i],dst->linesize[i],
(gdb)


The input file was made with gstreamer : gst-launch videotestsrc 
num-buffers=20 ! clockoverlay halign=right valign=bottom  
shaded-background=true ! ffmpegcolorspace ! video/x-raw-yuv, width=176, 
height=144 !  avimux ! filesink location=test.avi

The problem is the asm code of imgconvert.c in deinterlace_line 
function. If I force the function deinterlace_line not to use MMX code, 
I got no error. However, using an encoder (mpeg4 for example) give other 
error :

Starting program: /home/yves/ffmpeg/ffmpeg_g -i test.avi -vcodec mpeg4 
-an -deinterlace out2.avi
FFmpeg version UNKNOWN, Copyright (c) 2000-2009 Fabrice Bellard, et al.
  configuration: --prefix=/home/yves/ffmpeg/install --cc=icc 
-fp-stack-check
  libavutil     50. 2. 0 / 50. 2. 0
  libavcodec    52.22. 3 / 52.22. 3
  libavformat   52.32. 0 / 52.32. 0
  libavdevice   52. 1. 0 / 52. 1. 0
  libswscale     0. 7. 1 /  0. 7. 1
  built on Mar 23 2009 17:12:05, gcc: Intel(R) C++ gcc 4.2 mode
Input #0, avi, from 'test.avi':
  Duration: 00:00:00.66, start: 0.000000, bitrate: 9133 kb/s
    Stream #0.0: Video: rawvideo, yuv420p, 176x144, 30 tbr, 30 tbn, 30 tbc
File 'out2.avi' already exists. Overwrite ? [y/N] y
Output #0, avi, to 'out2.avi':
    Stream #0.0: Video: mpeg4, yuv420p, 176x144, q=2-31, 200 kb/s, 30 
tbn, 30 tbc
Stream mapping:
  Stream #0.0 -> #0.0
Press [q] to stop encoding

Program received signal SIGSEGV, Segmentation fault.
0x00000000005e70a8 in mb_var_thread (c=0x10728890, arg=0xd0) at 
libavcodec/mpegvideo_enc.c:1955
1955                int sum = s->dsp.pix_sum(pix, s->linesize);
(gdb)

Anyway, reconfiguring with --disable-mmx didn't crash anymore.

I would like to know if someone has an idea about the cause of this 
problem. Maybe it is some clobbered register in the asm code ?

Finally, the above occur on a Xeon 64 bits. Doing the same on a 32 bits 
machines show another problem that didn't disappear with --disable-mmx :

gdb --args ./ffmpeg_g -i test.avi -vcodec rawvideo -an -deinterlace out2.avi
GNU gdb Red Hat Linux (6.3.0.0-1.143.el4rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host 
libthread_db library "/lib/tls/libthread_db.so.1".

(gdb) r
Starting program: /home/yves/ffmpeg/ffmpeg_g -i test.avi -vcodec 
rawvideo -an -deinterlace out2.avi

Program received signal SIGSEGV, Segmentation fault.
0x0817b86e in av_opt_set_defaults2. () at libavcodec/opt.c:439
439                    av_set_q(s, opt->name, val);


I am using icc version :
icc (ICC) 10.1 20080312

Regards,
Yves