[FFmpeg-user] Reducing image2pipe png decoder latency

[Maxim Khitrov]

Hi all,

I'm trying to encode two image streams into a single h264 video, with
the second stream scaled and overlaid over the first one. The video is
then streamed in real time as mpeg-ts, so I'm trying to reduce latency
as much as possible.

I have an image generator program that prints the current timecode to
stdout and burns it into each pair of images, so by taking a
screenshot of the terminal and ffplay windows, I can see exactly how
much latency (in frames) there is between the images that are sent to
ffmpeg and the live video stream.

When both images are JPGs, the latency is 3 frames at 30 fps. However,
I need the second image to be transparent, so the production version
will overlay PNG over JPG, and this combination results in 11 frames
of latency, which isn't ideal. At first, I thought this was the result
of RGB to YUV conversion or maybe some other filter graph computation,
but this delay seems independent of the image size, leading me to
believe that something else is going on. At the same time, if I only
encode one image stream (no filter graph), both JPG-only and PNG-only
versions result in ~3 frames of latency.

I'm running out of ideas for what else could be responsible for the
additional buffering and/or computational delays, so I thought I'd ask
here. The command and ffmpeg output are below. If you have any other
recommendations for improving real-time encode performance, please
share them.

Thanks,
Max

ffmpeg -f image2pipe -avioflags direct -fflags nobuffer -fpsprobesize
0 -framerate 30 -probesize 22023 -an -c:v mjpeg -pixel_format yuvj420p
-video_size 1280x720 -i \\.\pipe\ffmpeg-OTdHYsp8CvAJMsnK.0 -f
image2pipe -avioflags direct -fflags nobuffer -fpsprobesize 0
-framerate 30 -probesize 5811 -an -c:v png -pixel_format rgba
-video_size 640x360 -i \\.\pipe\ffmpeg-OTdHYsp8CvAJMsnK.1
-filter_complex
'sws_flags=fast_bilinear;[1:v]format=yuva420p[f1];[f1][0:v]scale2ref=w=iw:h=ih[s1][s0];[s0][s1]overlay=eval=init[v0]'
-filter_threads 1 -map '[v0]' -c:v libx264 -profile:v high -preset
superfast -tune zerolatency -x264-params
keyint=90:min-keyint=30:bframes=0 -intra-refresh 1 -r 30 -thread_type
slice -f mpegts -

[image2pipe @ 000001f9e6b9eec0] Stream #0: not enough frames to
estimate rate; consider increasing probesize
Input #0, image2pipe, from '\\.\pipe\ffmpeg-Mb7lgdkRbbyjzpol.0':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: mjpeg, yuvj420p(bt470bg/unknown/unknown),
1280x720, 30 tbr, 30 tbn, 30 tbc
[image2pipe @ 000001f9e6bcc5c0] Stream #0: not enough frames to
estimate rate; consider increasing probesize
Input #1, image2pipe, from '\\.\pipe\ffmpeg-Mb7lgdkRbbyjzpol.1':
  Duration: N/A, bitrate: N/A
    Stream #1:0: Video: png, rgba(pc), 640x360, 30 tbr, 30 tbn, 30 tbc
Stream mapping:
  Stream #0:0 (mjpeg) -> scale2ref:ref
  Stream #1:0 (png) -> format
  overlay -> Stream #0:0 (libx264)
[libx264 @ 000001f9e6ba1dc0] using cpu capabilities: MMX2 SSE2Fast
SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 000001f9e6ba1dc0] profile High, level 3.1, 4:2:0, 8-bit
Output #0, mpegts, to 'pipe:':
  Metadata:
    encoder         : Lavf58.35.100
    Stream #0:0: Video: h264 (libx264), yuvj420p(pc), 1280x720,
q=-1--1, 30 fps, 90k tbn, 30 tbc (default)
    Metadata:
      encoder         : Lavc58.62.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
[mjpeg @ 000001f9e6bcaf00] error count: 64
[mjpeg @ 000001f9e6bcaf00] error y=2 x=1
[mjpeg @ 000001f9e6bcaf00] mjpeg_decode_dc: bad vlc: 0:0
(000001f9e6bcb5c8)bits/s speed=0.968x
[mjpeg @ 000001f9e6bcaf00] error dc
[mjpeg @ 000001f9e6bcaf00] error y=44 x=57
[mjpeg @ 000001f9e6bcaf00] mjpeg_decode_dc: bad vlc: 0:0
(000001f9e6bcb5c8)bits/s speed=0.975x
[mjpeg @ 000001f9e6bcaf00] error dc
[mjpeg @ 000001f9e6bcaf00] error y=44 x=68
frame=  310 fps= 30 q=23.0 Lsize=    1663kB time=00:00:10.33
bitrate=1318.2kbits/s speed=0.998x
video:1552kB audio:0kB subtitle:0kB other streams:0kB global
headers:0kB muxing overhead: 7.149707%
[libx264 @ 000001f9e6ba1dc0] frame I:1     Avg QP: 5.79  size: 10606
[libx264 @ 000001f9e6ba1dc0] frame P:309   Avg QP: 9.00  size:  5108
[libx264 @ 000001f9e6ba1dc0] mb I  I16..4: 92.3%  0.2%  7.5%
[libx264 @ 000001f9e6ba1dc0] mb P  I16..4: 82.6%  1.4%  1.4%  P16..4:
6.9%  0.0%  0.0%  0.0%  0.0%    skip: 7.8%
[libx264 @ 000001f9e6ba1dc0] 8x8 transform intra:1.6% inter:94.1%
[libx264 @ 000001f9e6ba1dc0] coded y,uvDC,uvAC intra: 1.5% 4.3% 3.0%
inter: 26.6% 35.7% 5.2%
[libx264 @ 000001f9e6ba1dc0] i16 v,h,dc,p: 78% 11% 10%  0%
[libx264 @ 000001f9e6ba1dc0] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu:  4%  5%
45% 40%  0%  0%  0%  0%  6%
[libx264 @ 000001f9e6ba1dc0] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 61% 19%
13%  1%  1%  1%  1%  1%  1%
[libx264 @ 000001f9e6ba1dc0] i8c dc,h,v,p: 72% 12% 16%  0%
[libx264 @ 000001f9e6ba1dc0] Weighted P-Frames: Y:1.0% UV:0.0%
[libx264 @ 000001f9e6ba1dc0] kb/s:1230.27