[FFmpeg-trac] #5080(undetermined:new): Excessive HTTP GETs reading MP4 from web server

FFmpeg trac at avcodec.org
Thu Dec 10 20:27:24 CET 2015


#5080: Excessive HTTP GETs reading MP4 from web server
--------------------------------------+----------------------------------
             Reporter:  nealzebub     |                     Type:  defect
               Status:  new           |                 Priority:  normal
            Component:  undetermined  |                  Version:  2.8.3
             Keywords:  http          |               Blocked By:
             Blocking:                |  Reproduced by developer:  0
Analyzed by developer:  0             |
--------------------------------------+----------------------------------
 OS: RHEL 3.14.33-26.47.amzn1.x86_64
 Build: ffmpeg-2.8.1-64bit-static
 Obtained from: ​http://johnvansickle.com/ffmpeg/

 == Summary of the bug:==
 When FFmpeg/FFprobe reads an MP4 file from a web server that allows
 partial download, the application may use an excessive number of GETs.
 This occurs with files in which audio/video frames are interleaved where
 audio-video frames exceed a difference in file offset, presumably
 exceeding some I/O buffer size. I have encountered many files that bring
 out this behavior, so it's not a corner case.

 Doubling probesize to 10000000 does not decrease the number of GETs. All
 these GETs can cause a dramatic slowdown of encoding speed(4fps vs.
 90fps), depending upon the response time of the service/server.

 To develop my presumption of frame "position", I used FFprobe's to print
 that information.
 {{{
 ./ffprobe -hide_banner -v trace -show_entries
 packet=pos,pts_time,codec_type
 "http://transcodes.s3.amazonaws.com/content/iPhone4_upload_via_3G.MP4" -of
 compact 2>&1 | grep 'Content-Range\|codec_type'
 }}}

 And an excerpt from the attached output of that command:
 {{{
 packet|codec_type=audio|pts_time=3.436553|pos=42024
 packet|codec_type=video|pts_time=2.466667|pos=61374
 [http @ 0x3f7a4a0] header='Content-Range: bytes 31559-684616/684617'
 packet|codec_type=audio|pts_time=3.482993|pos=42182
 packet|codec_type=audio|pts_time=3.529433|pos=42317
 packet|codec_type=video|pts_time=2.533333|pos=66016
 packet|codec_type=audio|pts_time=3.575873|pos=42473
 packet|codec_type=video|pts_time=2.600000|pos=66407
 packet|codec_type=audio|pts_time=3.622313|pos=42618
 packet|codec_type=video|pts_time=2.666667|pos=67537
 packet|codec_type=audio|pts_time=3.668753|pos=48642
 packet|codec_type=audio|pts_time=3.715193|pos=48785
 packet|codec_type=video|pts_time=2.733333|pos=68161
 [http @ 0x3f7a4a0] header='Content-Range: bytes 48926-684616/684617'
 packet|codec_type=audio|pts_time=3.761633|pos=48926
 packet|codec_type=video|pts_time=2.800000|pos=69005
 packet|codec_type=audio|pts_time=3.808073|pos=49058
 }}}



 == How to reproduce: ==
 ''Please note that this is extremely dependent on the source file and must
 be over HTTP, which is why I made it available.''
 {{{
 ./ffmpeg -hide_banner -v trace -report -i
 "https://transcodes.s3.amazonaws.com/content/iPhone4_upload_via_3G.MP4" -y
 output.mp4 2>&1 | grep 'Content-Range\|frame='
 }}}


 == Log Output: ==
 I had to use a fairly verbose logging level to show the http requests and
 frames processed. Please see the attached report to this bug.

 Here's an excerpt, though:

 {{{
 [https @ 0x48f2640] request: GET /content/iPhone4_upload_via_3G.MP4
 HTTP/1.1
 User-Agent: Lavf/56.40.101
 Accept: */*
 Range: bytes=65677-
 Connection: close
 Host: transcodes.s3.amazonaws.com
 Icy-MetaData: 1

 [libx264 @ 0x48fca20] frame=   0 QP=23.32 NAL=3 Slice:I Poc:0   I:112  P:0
 SKIP:0    size=6416 bytes
 [libx264 @ 0x48fca20] frame=   1 QP=23.35 NAL=2 Slice:P Poc:4   I:0
 P:105  SKIP:7    size=1053 bytes
 [libx264 @ 0x48fca20] frame=   2 QP=31.00 NAL=0 Slice:B Poc:2   I:0
 P:65   SKIP:47   size=87 bytes
 [libx264 @ 0x48fca20] frame=   3 QP=23.65 NAL=2 Slice:P Poc:8   I:0
 P:99   SKIP:13   size=957 bytes
 [libx264 @ 0x48fca20] frame=   4 QP=31.00 NAL=0 Slice:B Poc:6   I:0
 P:56   SKIP:55   size=79 bytes
 [libx264 @ 0x48fca20] frame=   5 QP=23.66 NAL=2 Slice:P Poc:12  I:0
 P:104  SKIP:8    size=1133 bytes
 [libx264 @ 0x48fca20] frame=   6 QP=29.67 NAL=0 Slice:B Poc:10  I:0
 P:60   SKIP:47   size=134 bytes
 [libx264 @ 0x48fca20] frame=   7 QP=23.84 NAL=2 Slice:P Poc:16  I:0
 P:98   SKIP:14   size=884 bytes
 [libx264 @ 0x48fca20] frame=   8 QP=31.00 NAL=0 Slice:B Poc:14  I:0
 P:62   SKIP:49   size=84 bytes
 [libx264 @ 0x48fca20] frame=   9 QP=23.73 NAL=2 Slice:P Poc:20  I:0
 P:108  SKIP:4    size=1056 bytes

 [https @ 0x48f2640] request: GET /content/iPhone4_upload_via_3G.MP4
 HTTP/1.1
 User-Agent: Lavf/56.40.101
 Accept: */*
 Range: bytes=82048-
 Connection: close
 Host: transcodes.s3.amazonaws.com
 Icy-MetaData: 1

 [libx264 @ 0x48fca20] frame=  10 QP=31.00 NAL=0 Slice:B Poc:18  I:0
 P:48   SKIP:62   size=68 bytes
 [libx264 @ 0x48fca20] frame=  11 QP=23.79 NAL=2 Slice:P Poc:24  I:0
 P:107  SKIP:5    size=1079 bytes
 [libx264 @ 0x48fca20] frame=  12 QP=31.50 NAL=0 Slice:B Poc:22  I:0
 P:48   SKIP:64   size=56 bytes
 [libx264 @ 0x48fca20] frame=  13 QP=23.96 NAL=2 Slice:P Poc:28  I:0
 P:105  SKIP:7    size=971 bytes
 [libx264 @ 0x48fca20] frame=  14 QP=30.33 NAL=0 Slice:B Poc:26  I:0
 P:69   SKIP:39   size=137 bytes
 [libx264 @ 0x48fca20] frame=  15 QP=23.62 NAL=2 Slice:P Poc:32  I:0
 P:112  SKIP:0    size=1858 bytes
 [libx264 @ 0x48fca20] frame=  16 QP=29.60 NAL=0 Slice:B Poc:30  I:0
 P:64   SKIP:46   size=196 bytes
 [libx264 @ 0x48fca20] frame=  17 QP=23.32 NAL=2 Slice:P Poc:36  I:0
 P:105  SKIP:7    size=1228 bytes
 [libx264 @ 0x48fca20] frame=  18 QP=29.05 NAL=0 Slice:B Poc:34  I:0
 P:83   SKIP:27   size=170 bytes
 [libx264 @ 0x48fca20] frame=  19 QP=23.24 NAL=2 Slice:P Poc:40  I:0
 P:107  SKIP:5    size=1039 bytes
 [libx264 @ 0x48fca20] frame=  20 QP=29.67 NAL=0 Slice:B Poc:38  I:0
 P:74   SKIP:33   size=181 bytes
 [libx264 @ 0x48fca20] frame=  21 QP=23.45 NAL=2 Slice:P Poc:44  I:0
 P:110  SKIP:2    size=1133 bytes

 [https @ 0x48f2640] request: GET /content/iPhone4_upload_via_3G.MP4
 HTTP/1.1
 User-Agent: Lavf/56.40.101
 Accept: */*
 Range: bytes=90655-
 Connection: close
 Host: transcodes.s3.amazonaws.com
 Icy-MetaData: 1
 }}}

--
Ticket URL: <https://trac.ffmpeg.org/ticket/5080>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker


More information about the FFmpeg-trac mailing list