[Ffmpeg-devel] segfault: ff_fdct_sse2

[RayLing Hwang]

Hi, 
I have stuck on this for two weeks. Someone please help out. Thansk in advance.
 
1. I wrote a small program to extract partial frames out mpeg1/2 for video preview.
    I basically followed apiexample.c and output_example.c.
2.   I have compliled ffmpeg 0.4.9-pre1 with Msys/mingw with configure --enable_shared
3. It runs ok on windows 2000, but crashed on XP home-edition. But we need it to be run
on XP home.
4. The crash seems to be related to program address, by changing unimportant input options(e.g. change  log file name from abc.log to aaaaaaaaaaaaaaaaaa.log ) it may run
ok sometime. 
5. change log of ffmpeg for this release says "fdct sse2" --- is it a fix for problems like mine?
6. I run ffmpeg to convert our mpg files, it crashes depending on input option.
    eg,  A. ffmpeg -target ntsc_vcd A.mpg runs ok 
          B. ffmpeg -target ntsc_vcd B.mpg crashes 
               but ffmpeg -target ntsc_vcd -s 352x288 B.mpg runs ok
7. decoing is fine, ffplay plays all our mpg files ok.
8. I have run about 10 mpeg files, all die at the same place.
8. Attached information from gdb

 
0x65643b83 in ff_fdct_sse2 (block=0x525018) at i386/fdct_mmx.c:353
353         asm volatile(

#0  0x00653bb3 in ff_fdct_sse2 (block=0xbb4de8) at i386/fdct_mmx.c:353
#1  0x04400440 in ?? ()
#2  0x00649142 in dct_quantize_SSE2 (s=0xb71b18, block=0xbb4de8, n=0, 
    qscale=5, overflow=0x22606c) at i386/mpegvideo_mmx_template.c:46
#3  0x0046dc4c in encode_mb (s=0xb71b18, motion_x=0, motion_y=0)
    at mpegvideo.c:3704
#4  0x0046af6b in encode_thread (c=0xb85af8, arg=0xb71b18) at mpegvideo.c:4581
#5  0x00446fe0 in avcodec_default_execute (c=0xb85af8, 
    func=0x468d40 <encode_thread>, arg=0xb71bdc, ret=0x0, count=1)
    at utils.c:342
#6  0x004605c4 in encode_picture (s=0xb71b18, picture_number=0)
    at mpegvideo.c:4885
#7  0x0045e227 in MPV_encode_picture (avctx=0xb85af8, buf=0xbbdb40 "", 
    buf_size=200000, data=0xb78850) at mpegvideo.c:2149
#8  0x004471d8 in avcodec_encode_video (avctx=0xb85af8, buf=0xbbdb40 "", 
    buf_size=200000, pict=0xb78850) at utils.c:478

eax            0x66ba60 6732384
ecx            0x66ba70 6732400
edx            0xbb4de8 12275176
ebx            0x225e80 2252416
esp            0x225e80 0x225e80
ebp            0x225f08 0x225f08
esi            0x0      0
edi            0xb71b18 12000024
eip            0x653bb3 0x653bb3
eflags         0x10202  66050
cs             0x1b     27
ss             0x23     35
ds             0x23     35
es             0x23     35
fs             0x3b     59
gs             0x0      0
st0            -nan(0x1000100010001)    (raw 0xffff0001000100010001)
st1            0        (raw 0xffff0000000000000000)
st2            -nan(0x1000100010001)    (raw 0xffff0001000100010001)
st3            0        (raw 0xffff0000000000000000)
st4            0        (raw 0xffff0000000000000000)
st5            -nan(0xffffffffffffffff) (raw 0xffffffffffffffffffff)
st6            0        (raw 0xffff0000000000000000)
st7            -nan(0xffffffffffffffff) (raw 0xffffffffffffffffffff)
fctrl          0xffff037f       -64641
fstat          0xffff0120       -65248
ftag           0xffffaaaa       -21846
fiseg          0x1b     27
fioff          0x50dbc8 5299144
foseg          0xffff0023       -65501
fooff          0x22e29c 2286236
fop            0x5d8    1496

 
(gdb) disass $pc-32 $pc+32
Dump of assembler code from 0x653b93 to 0x653bd3:
0x653b93 <ff_fdct_sse2+739>:    cmc    
0x653b94 <ff_fdct_sse2+740>:    int3   
0x653b95 <ff_fdct_sse2+741>:    paddd  %xmm7,%xmm3
0x653b99 <ff_fdct_sse2+745>:    paddd  %xmm2,%xmm1
0x653b9d <ff_fdct_sse2+749>:    paddd  %xmm6,%xmm3
0x653ba1 <ff_fdct_sse2+753>:    paddd  %xmm6,%xmm1
0x653ba5 <ff_fdct_sse2+757>:    psrad  $0x11,%xmm3
0x653baa <ff_fdct_sse2+762>:    psrad  $0x11,%xmm1
0x653baf <ff_fdct_sse2+767>:    packssdw %xmm3,%xmm1
0x653bb3 <ff_fdct_sse2+771>:    movdqa %xmm1,0x0(%edx)
0x653bb8 <ff_fdct_sse2+776>:    movq   0x40(%ebx),%xmm2
0x653bbd <ff_fdct_sse2+781>:    movq   0x48(%ebx),%xmm0
0x653bc2 <ff_fdct_sse2+786>:    movdqa 0x20(%ecx),%xmm3
0x653bc7 <ff_fdct_sse2+791>:    movdqa 0x30(%ecx),%xmm7
0x653bcc <ff_fdct_sse2+796>:    movq   %xmm2,%xmm1
0x653bd0 <ff_fdct_sse2+800>:    pshuflw $0x1b,%xmm0,%xmm0

 

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com