Hello,
I have an interesting problem I am working through and I am hoping to get some insight as to a root cause. I am taking antenna feeds that are converted to multicast into a transcoding server that is running FFMPEG. I am transcoding the streams using h264_nvenc encoder and outputting them onto the network. The issue is that the receiving servers are getting continuity errors on the transcoded streams periodically. You can also see these in the video playback in the form of very minor pixilation and slight audio skips. This only seems to effect HD stream, and only when the receiving server is virtualized.
What I have tried:
Testing in 3 separate physical locations/networks
Verified that only virtual machines are effected. Physical baremetal linux installed machines do not see continuity errors.
Verified that the problem is only present on HD transcoded streams. SD transcoded streams and the original source streams do not display continuity issues.
I noticed variation in packet size so I forced consistent packet size with the ffmpeg command.
2 separate physical transcoder servers.
2 separate models of Nvidia cards. P2000 and RTX4000.
Changing the virtual machines NIC driver from VMXNET3 to E1000.
Increasing buffers of the Linux OS's that are receiving the channels.
Verified that it was not a resource issue on the graphics card by reducing to a single stream with one transcoded output.
Verified the GPU is not under load. It is currently using less than 10% of the NVENC chip to process one of these streams.
Verified that the continuities are an issue on both audio streams and the video stream.
Verified the issue is present across multiple versions of FFMPEG including a newly compiled from source version.
Command that I am using to transcode:
ffmpeg -hwaccel nvdec -i 'udp://@239.200.200.100:6001?fifo_size=1000000&overrun_nonfatal=1' \
-vcodec h264_nvenc -bf:v 2 -g 120 -rc cbr -b:v 5500K -minrate 5500k -maxrate 5500k -bufsize 11000k -profile:v high -level 4.0 -acodec libfdk_aac -ac 2 -b:a 128k -flush_packets 0 -f mpegts 'udp://@239.200.200.103:59103'
==Command on the transcoding server to kick off the process:
james@transcoder:~$ ffmpeg -hwaccel nvdec -i 'udp://@239.1.1.93:59093?overrun_nonfatal=1' -vcodec h264_nvenc -bf:v 2 -g 120 -rc cbr -b:v 5500K -minrate 5500k -maxrate 5500k -bufsize 11000k -profile:v high -level 4.0 -acodec libfdk_aac -ac 2 -b:a 128k -flush_packets 0 -f mpegts 'udp://@239.100.100.3:59003'
ffmpeg version N-93005-gd92f06e Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 7 (Ubuntu 7.3.0-27ubuntu1~18.04)
configuration: --prefix=/home/circle/ffmpeg_build --pkg-config-flags=--static --extra-cflags=-I/home/circle/ffmpeg_build/include --extra-ldflags=-L/home/circle/ffmpeg_build/lib --extra-libs='-lpthread -lm' --bindir=/home/circle/bin --enable-gpl --enable-libaom --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-nonfree --enable-nvenc
libavutil 56. 26.100 / 56. 26.100
libavcodec 58. 44.100 / 58. 44.100
libavformat 58. 26.100 / 58. 26.100
libavdevice 58. 6.101 / 58. 6.101
libavfilter 7. 48.100 / 7. 48.100
libswscale 5. 4.100 / 5. 4.100
libswresample 3. 4.100 / 3. 4.100
libpostproc 55. 4.100 / 55. 4.100
[mpeg2video @ 0x5630735fc900] Invalid frame dimensions 0x0.
Last message repeated 37 times
Input #0, mpegts, from 'udp://@239.1.1.93:59093?overrun_nonfatal=1':
Duration: N/A, start: 52458.920733, bitrate: N/A
Program 3
Stream #0:0[0x31]: Video: mpeg2video (Main) ([2][0][0][0] / 0x0002), yuv420p(tv, progressive), 1280x720 [SAR 1:1 DAR 16:9], Closed Captions, 59.94 fps, 59.94 tbr, 90k tbn, 119.88 tbc
Stream #0:1[0x34](eng): Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, 5.1(side), fltp, 384 kb/s
Stream #0:2[0x35](spa): Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, stereo, fltp, 192 kb/s
Stream mapping:
Stream #0:0 -> #0:0 (mpeg2video (native) -> h264 (h264_nvenc))
Stream #0:1 -> #0:1 (ac3 (native) -> aac (libfdk_aac))
Press [q] to stop, [?] for help
Output #0, mpegts, to 'udp://@239.100.100.3:59003':
Metadata:
encoder : Lavf58.26.100
Stream #0:0: Video: h264 (h264_nvenc) (High), nv12, 1280x720 [SAR 1:1 DAR 16:9], q=-1--1, 5500 kb/s, 59.94 fps, 90k tbn, 59.94 tbc
Metadata:
encoder : Lavc58.44.100 h264_nvenc
Side data:
cpb: bitrate max/min/avg: 5500000/0/5500000 buffer size: 11000000 vbv_delay: -1
Stream #0:1(eng): Audio: aac (libfdk_aac), 48000 Hz, stereo, s16, 128 kb/s
Metadata:
encoder : Lavc58.44.100 libfdk_aac
frame=18186 fps= 60 q=19.0 size= 226856kB time=00:05:04.10 bitrate=6111.1kbits/s speed=1.01x
==FFProbe from the receiving server:
ffprobe udp://@239.100.100.3:59003
ffprobe version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2007-2018 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
libavutil 54. 31.100 / 54. 31.100
libavcodec 56. 60.100 / 56. 60.100
libavformat 56. 40.101 / 56. 40.101
libavdevice 56. 4.100 / 56. 4.100
libavfilter 5. 40.101 / 5. 40.101
libavresample 2. 1. 0 / 2. 1. 0
libswscale 3. 1.101 / 3. 1.101
libswresample 1. 2.101 / 1. 2.101
libpostproc 53. 3.100 / 53. 3.100
[h264 @ 0x208f4e0] decode_slice_header error
[h264 @ 0x208f4e0] no frame!
[h264 @ 0x208f4e0] non-existing PPS 0 referenced
Last message repeated 1 times
Input #0, mpegts, from 'udp://@239.100.100.3:59003':
Duration: N/A, start: 37.005333, bitrate: N/A
Program 1
Metadata:
service_name : Service01
service_provider: FFmpeg
Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], Closed Captions, 59.94 fps, 59.94 tbr, 90k tbn, 119.88 tbc
Stream #0:1[0x101](eng): Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 127 kb/s
==Debugging from the receiving VM:
james@transcoder:~$ ffmpeg -v debug -i udp://@239.100.100.3:59003 -f null /dev/null
ffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
libavutil 54. 31.100 / 54. 31.100
libavcodec 56. 60.100 / 56. 60.100
libavformat 56. 40.101 / 56. 40.101
libavdevice 56. 4.100 / 56. 4.100
libavfilter 5. 40.101 / 5. 40.101
libavresample 2. 1. 0 / 2. 1. 0
libswscale 3. 1.101 / 3. 1.101
libswresample 1. 2.101 / 1. 2.101
libpostproc 53. 3.100 / 53. 3.100
Splitting the commandline.
Reading option '-v' ... matched as option 'v' (set logging level) with argument 'debug'.
Reading option '-i' ... matched as input url with argument 'udp://@239.100.100.3:59003'.
Reading option '-f' ... matched as option 'f' (force format) with argument 'null'.
Reading option '/dev/null' ... matched as output url.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option v (set logging level) with argument debug.
Successfully parsed a group of options.
Parsing a group of options: input url udp://@239.100.100.3:59003.
Successfully parsed a group of options.
Opening an input file: udp://@239.100.100.3:59003.
[udp @ 0x1e4bc20] end receive buffer size reported is 131072
[mpegts @ 0x1e4b400] Format mpegts probed with size=2048 and score=100
[mpegts @ 0x1e4b400] stream=0 stream_type=1b pid=100 prog_reg_desc=
[mpegts @ 0x1e4b400] stream=1 stream_type=f pid=101 prog_reg_desc=
[mpegts @ 0x1e4b400] Before avformat_find_stream_info() pos: 0 bytes read:8832 seeks:0
[h264 @ 0x1e6fd20] non-existing PPS 0 referenced
Last message repeated 1 times
[h264 @ 0x1e6fd20] decode_slice_header error
[h264 @ 0x1e6fd20] no frame!
[h264 @ 0x1e6fd20] non-existing PPS 0 referenced
[mpegts @ 0x1e4b400] max_analyze_duration 5000000 reached at 5013333 microseconds st:1
[mpegts @ 0x1e4b400] rfps: 60.000000 0.007957
[mpegts @ 0x1e4b400] rfps: 59.940060 0.000000
Last message repeated 1 times
[mpegts @ 0x1e4b400] After avformat_find_stream_info() pos: 3997328 bytes read:3997952 seeks:0 frames:553
Input #0, mpegts, from 'udp://@239.100.100.3:59003':
Duration: N/A, start: 389.901333, bitrate: N/A
Program 1
Metadata:
service_name : Service01
service_provider: FFmpeg
Stream #0:0[0x100], 310, 1/90000: Video: h264 (High), 2 reference frames ([27][0][0][0] / 0x001B), yuv420p(left), 1280x720 [SAR 1:1 DAR 16:9], 1001/120000, Closed Captions, 59.94 fps, 59.94 tbr, 90k tbn, 119.88 tbc
Stream #0:1[0x101](eng), 243, 1/90000: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 144 kb/s
Successfully opened the file.
Parsing a group of options: output url /dev/null.
Applying option f (force format) with argument null.
Successfully parsed a group of options.
Opening an output file: /dev/null.
Successfully opened the file.
detected 8 logical cores
[graph 0 input from stream 0:0 @ 0x23b51a0] Setting 'video_size' to value '1280x720'
[graph 0 input from stream 0:0 @ 0x23b51a0] Setting 'pix_fmt' to value '0'
[graph 0 input from stream 0:0 @ 0x23b51a0] Setting 'time_base' to value '1/90000'
[graph 0 input from stream 0:0 @ 0x23b51a0] Setting 'pixel_aspect' to value '1/1'
[graph 0 input from stream 0:0 @ 0x23b51a0] Setting 'sws_param' to value 'flags=2'
[graph 0 input from stream 0:0 @ 0x23b51a0] Setting 'frame_rate' to value '60000/1001'
[graph 0 input from stream 0:0 @ 0x23b51a0] w:1280 h:720 pixfmt:yuv420p tb:1/90000 fr:60000/1001 sar:1/1 sws_param:flags=2
[AVFilterGraph @ 0x1e7c0e0] query_formats: 3 queried, 2 merged, 0 already done, 0 delayed
[graph 1 input from stream 0:1 @ 0x1e4b300] Setting 'time_base' to value '1/48000'
[graph 1 input from stream 0:1 @ 0x1e4b300] Setting 'sample_rate' to value '48000'
[graph 1 input from stream 0:1 @ 0x1e4b300] Setting 'sample_fmt' to value 'fltp'
[graph 1 input from stream 0:1 @ 0x1e4b300] Setting 'channel_layout' to value '0x3'
[graph 1 input from stream 0:1 @ 0x1e4b300] tb:1/48000 samplefmt:fltp samplerate:48000 chlayout:0x3
[audio format for output stream 0:1 @ 0x1e6ba00] Setting 'sample_fmts' to value 's16'
[audio format for output stream 0:1 @ 0x1e6ba00] auto-inserting filter 'auto-inserted resampler 0' between the filter 'Parsed_anull_0' and the filter 'audio format for output stream 0:1'
[AVFilterGraph @ 0x1e6be20] query_formats: 4 queried, 6 merged, 3 already done, 0 delayed
[auto-inserted resampler 0 @ 0x1e74400] [SWR @ 0x1e8e2a0] Using fltp internally between filters
[auto-inserted resampler 0 @ 0x1e74400] ch:2 chl:stereo fmt:fltp r:48000Hz -> ch:2 chl:stereo fmt:s16 r:48000Hz
Output #0, null, to '/dev/null':
Metadata:
encoder : Lavf56.40.101
Stream #0:0, 0, 1001/60000: Video: rawvideo, 1 reference frame (I420 / 0x30323449), yuv420p(left), 1280x720 [SAR 1:1 DAR 16:9], 1001/60000, q=2-31, 200 kb/s, 59.94 fps, 59.94 tbn, 59.94 tbc
Metadata:
encoder : Lavc56.60.100 rawvideo
Stream #0:1(eng), 0, 1/48000: Audio: pcm_s16le, 48000 Hz, stereo, s16, 1536 kb/s
Metadata:
encoder : Lavc56.60.100 pcm_s16le
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> rawvideo (native))
Stream #0:1 -> #0:1 (aac (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
[h264 @ 0x1eb06c0] Frame num gap 27 24
[h264 @ 0x1eb06c0] Frame num gap 27 25
[h264 @ 0x1eb06c0] no picture
[mpegts @ 0x1e4b400] Continuity check failed for pid 257 expected 15 got 7
[mpegts @ 0x1e4b400] PES packet size mismatch
[aac @ 0x23b4920] Number of bands (67) exceeds limit (40).
Error while decoding stream #0:1: Invalid data found when processing input
[aac @ 0x23b4920] channel element 2.6 is not allocated
Error while decoding stream #0:1: Invalid data found when processing input
[mpegts @ 0x1e4b400] Continuity check failed for pid 4096 expected 10 got 12
[mpegts @ 0x1e4b400] Continuity check failed for pid 256 expected 9 got 3
[mpegts @ 0x1e4b400] Continuity check failed for pid 0 expected 10 got 13
[h264 @ 0x1e7b5c0] Frame num gap 2 0
[h264 @ 0x1f43160] concealing 260 DC, 260 AC, 260 MV errors in I frame
[mpegts @ 0x1e4b400] Continuity check failed for pid 256 expected 4 got 6
[mpegts @ 0x1e4b400] Continuity check failed for pid 0 expected 10 got 12
[mpegts @ 0x1e4b400] Continuity check failed for pid 4096 expected 10 got 12
[mpegts @ 0x1e4b400] Continuity check failed for pid 256 expected 2 got 6
[mpegts @ 0x1e4b400] Continuity check failed for pid 0 expected 13 got 15
[mpegts @ 0x1e4b400] Continuity check failed for pid 4096 expected 13 got 15
[mpegts @ 0x1e4b400] Continuity check failed for pid 17 expected 10 got 11
[h264 @ 0x20d6480] left block unavailable for requested intra4x4 mode -1 at 0 25
[h264 @ 0x20d6480] error while decoding MB 0 25, bytestream 5224
[h264 @ 0x20d6480] concealing 1649 DC, 1649 AC, 1649 MV errors in P frame
frame= 6908 fps= 62 q=-0.0 Lsize=N/A time=00:01:56.09 bitrate=N/A
video:648kB audio:21744kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Input file #0 (udp://@239.100.100.3:59003):
Input stream #0:0 (video): 6960 packets read (79709163 bytes); 6909 frames decoded;
Input stream #0:1 (audio): 5438 packets read (1856092 bytes); 5436 frames decoded (5566464 samples);
Total: 12398 packets (81565255 bytes) demuxed
Output file #0 (/dev/null):
Output stream #0:0 (video): 0 frames encoded; 6908 packets muxed (663168 bytes);
Output stream #0:1 (audio): 5436 frames encoded (5566464 samples); 5436 packets muxed (22265856 bytes);
Total: 12344 packets (22929024 bytes) muxed
12345 frames successfully decoded, 2 decoding errors
[AVIOContext @ 0x1e6c100] Statistics: 88821952 bytes read, 0 seeks
Exiting normally, received signal 2.
Thanks,
-James
CONFIDENTIALITY NOTICE: This e-mail including attachments is covered by the Electronic Communications Privacy Act, 18 U.S.C. ??2510-2521, is confidential and may contain information that is privileged, confidential or otherwise protected from use and disclosure. If you are not the intended recipient, you are hereby notified that any review, disclosure, copying, or dissemination of this transmission, or taking of any action in reliance on its contents, or other use is strictly prohibited. If you have received this transmission in error, please reply to the sender listed above immediately and permanently delete this message from your inbox.
