February 2022 - ffmpeg-user - lists.ffmpeg.org

Video Notation -- new home, with specimen pictures
by Mark Filipak 10 Feb '22

10 Feb '22

Hello All, Thanks to Ralph Spitzner generous contribution of space on his server, the notation has a new home http://videonotation.spitzner.org including specimen pictures to illustrate the elements of the notation. Thank you, Ralph.

1 0

Video Notation, A Video Lingua Franca -- Home
by Mark Filipak 09 Feb '22

09 Feb '22

Hello, All, Steven Kan has very graciously provided a home for my work. https://www.kan.org/VideoNotation/Video%20Notation.01.Preface.html Thank you, Steven. To Y'all: Kindly ask questions, submit suggestions/corrections, etc. through ffmpeg-user. To quote Max, "Let the rumpus begin!" Regards, Mark Filipak.

3 3

Video Notation, A Video Lingua Franca
by Mark Filipak 09 Feb '22

09 Feb '22

This message is apparently the best I can do. The ffmpeg-user system apparently won't allow HTML file attachments. I'm sorry. ...I'm amazed by the number of people who can't handle ZIP. Simply open the ZIP and extract the 11 enclosed HTML files to a directory and then click any one of them. If you don't know how to open a ZIP, perhaps your email client does -- just double-click on the attachment and follow your email client's cues. Regards, Mark Filipak.

6 6

off: How to merge multiple H.264 streams into a single H.264 stream in HW/FPGA?
by Tamás Forgács 09 Feb '22

09 Feb '22

I encode camera images with several parallel H.264 baseline I-frame only encoder cores in FPGA (more parallel cores are required to get the encoding done fast enough without external memory). I use an opensource [encoder][1]. On simulator I use 352x288 yuv420 input image and I slice it to 2 horizontal halves. The decoder is on another embedded device and I can decode the separated slices correctly. But, instead of several decoded video sequences, I would need to have just one vido sequence (with the whole image) from one H.264 byte stream. It is preferred to merge the H.264 streams on the FPGA (instead of doing this by post processing the decoded slice video sequences on the embedded host). So, the input image data looks like this: .--.-----.--.--.-----.--. |MB| |MB|MB| |MB| | 0| ... |10|11| ... |21| |--. '--|--' '--| |MB| | | |22| | | |--' | | | Slice | Slice | | 0 | 1 | '-----------'----------' And the encoder's block diagram is this (just one encoder core shown, without merging): YUV420 MacroBlocks (16*16 pixel) | | chroma | | luma | | .----------------.-------. | | | | | | | v v v v reconstruct intra8x8cc intra4x4 ---> header ^ | | | | v v | inv.transform coretransform | ^ | | | | | v | dequantise | dctransform | ^ ^ | | | | | v v | inv.dc.transform <-'---------- quantise | | | v | buffer | | | v | cavlc | | | v | tobytes <------------' | v H.264 byte stream The encoded h264 data from a singel core (parsed with [H264Naked,][2] see [parse.txt][3], [ref.yuv420][4] and [ref.264][5]) contains 3 NAL-s: in the first one there is a Sequence Parameter Set, in the second a Picture Parameter Set and in the third a Coded slice of an IDR picture. The SPS and PPS headers are static and these are generated on the host. The IDR header is generated by the encoder on the FPGA. Currently I use a constant Quantization Parameter (i.e. VBR, but it is planned to implement CBR, with variable QP later). To get the encoded streams merged, I have removed the IDR header from the slices (except from slice 0) in the header block. Then I buffer the output of CAVLC blocks from each encoder for a whole MB row (this is the encoded MB data, MB headers can be identified from the output of header block). Then from this buffer I drive all the CAVLC data to a new instance of tobytes block in this sequence: - CAVLC data of one MB row from slice 0 - CAVLC data of one MB row from slice 1 - the above is repeated till the end of the frame. When I decode this merged H.264 stream, then I get the following error with ffmpeg (see [merged_tobytes.264][6] and [ffplay_error.txt][7]): ``` [h264 @ 0x7f5f0c00ac00] dquant out of range (-112) at 13 1B f=0/0 [h264 @ 0x7f5f0c00ac00] error while decoding MB 13 1 [h264 @ 0x7f5f0c00ac00] concealing 396 DC, 396 AC, 396 MV errors in I frame ``` With JM reference decoder I get this error ([jm_dec_error.txt][8]): ``` mb_qp_delta is out of range (-112) illegal chroma intra pred mode! ``` I think the problem is that the decoder does not know about the slices and expects neighboring MB data from other slice which are not available in the encoded data. I have checked the Macroblock prediction syntax in the [standard][9] (chapter 7.3.5.1), but I do not see how should I correct this (I have very little knowledge about H.264). On the encoder side, I see that intra4x4 and intra8x8cc are using neighboring MB prediction and pixel data from the top and left MB neighbors when those are available (we have no top MB neighbors in the first MB row and we have no left MB neighbor at first MB of each MB row, in each slice). I have also tried to use FMO and declared slice groups (slice_group_map_type=2) as described [here][10], but FMO and slice groups are not supported by typical H.264 decoders (see ffmpeg [issue][11], I think it is used only in broadcast equipments). In fact, I had also got error from JM reference decoder even when I have declared the slice groups (see [merged_tobytes_with_slice_grops.264][12] and [jm_dec_slice_groups_error.txt][13]): ``` warning: Intra_8x8_Horizontal prediction mode not allowed at mb 0 illegal chroma intra pred mode! ``` Any help is appreciated. Reference photo is taken from [here][14]. PS: if someone can advise a deblocker which could be used with this FPGA core, that would be also nice. [1]: https://github.com/bcattle/hardh264/ [2]: https://github.com/shi-yan/H264Naked [3]: https://www.dropbox.com/s/kktelcy06jmpa0t/h264naked_ref.txt?dl=0 [4]: https://www.dropbox.com/s/mqkohwten6qax6z/ref.yuv420?dl=0 [5]: https://www.dropbox.com/s/vbon3ul5jimkq60/ref.264?dl=0 [6]: https://www.dropbox.com/s/ic3cms9p40g80od/merged_tobytes.264?dl=0 [7]: https://www.dropbox.com/s/hkombdr087z7knf/ffplay_error.txt?dl=0 [8]: https://www.dropbox.com/s/s42hxmjea5ul3qd/jm_dec_error.txt?dl=0 [9]: https://www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC-H.264-202108-I!!PDF… [10]: https://en.wikipedia.org/wiki/Flexible_Macroblock_Ordering [11]: https://www.mail-archive.com/ffmpeg-issues@lscube.org/msg03024.html [12]: https://www.dropbox.com/s/8mnavr5y4dgr469/merged_tobytes_with_slice_grops.2… [13]: https://www.dropbox.com/s/wgxnup7m19atm5j/jm_dec_slice_groups_error.txt?dl=0 [14]: http://%20https://unsplash.com/photos/NuIbSaztf_g

1 0

Re: [FFmpeg-user] Video Notation, A Video Lingua Franca
by Steven Kan 08 Feb '22

08 Feb '22

Of course! If you want, I can host them here: https://www.kan.org/VideoNotation/Video%20Notation.01.Preface.html I don’t have an unlimited hosting account, but I can’t imagine these will get that much traffic. Of course if you don’t want them there, let me know and I’ll delete them. > On Feb 8, 2022, at 12:49 PM, Mark Filipak <markfilipak.noreply(a)gmail.com> wrote: > > On 2022-02-08 12:12, Steven Kan wrote: >> Hi, >> Can you fax them to me? > > You're making a joke, right? > > -- Mark. >

2 2

Video Notation, A Video Lingua Franca
by Mark Filipak 08 Feb '22

08 Feb '22

On suggestion, I tried web hosting suppliers and failed. I've attached the 11 HTML pages -- no ZIPs. I don't know how I could make it easier. Suppose I used a file sharing service? You'd have to download the HTLM pages, right? Well, the HTML pages are attached, so you don't have to download them. They're already here. Just save them to your hard drive (160 KBytes) and click any one of them. Regards, Mark Filipak.

4 5

setts difficulty
by Mark Filipak 08 Feb '22

08 Feb '22

Help! Command: ffmpeg -report -i 24'fps[24pps].vob -filter_complex "[0:a]asetpts=expr=N/TB*1001/24000[a]" -bsf setts=pts=N/TB*1001/24000:dts=N/TB*1001/24000 -target ntsc-dvd -map 0:v -map "[a]" -codec:v copy -codec:a ac3 -dn -r 24000/1001 foo.vob 24'fps[24pps].vob, the source video, is soft telecined. foo.vob, the target, freezes within the 1st second. The audio plays to the end. I'm trying to fix the PTSs & DTSs via 'setts'. I have evidence, below. Issue: Since 24'fps[24pps].vob is soft telecined, the metadata says 29.970fps but we all know that the pictures in those frames are actually 24pps (i.e. it's not hard telecined). Any clue how to proceed? The FFmpeg LOG is below, and an FFprobe is at the end. Clue: What are these 3 FFmpeg LOG lines about? Any ideas? ----- Multiple -c, -codec, -acodec, -vcodec, -scodec or -dcodec options specified for stream 0, only the last option '-c:v copy' will be used. Multiple -r options specified for stream 0, only the last option '-r 24000/1001' will be used. Multiple -c, -codec, -acodec, -vcodec, -scodec or -dcodec options specified for stream 1, only the last option '-c:a ac3' will be used. ----- I DID NOT have multiple '-c' & '-r' directives. Did FFmpeg add some internal settings without notifying me? Clue: The following FFprobe output showing the target's 1st 2 key frames. Note that the PTSs are screwy. ----- frames.frame.0.key_frame=1 frames.frame.0.pts=45000 <= should be zero frames.frame.0.pkt_dts=45000 : frames.frame.12.key_frame=1 frames.frame.12.pts=45012 <= should be 3753 frames.frame.12.pkt_dts=45012 ----- That deltaPTS implies 90000 FPS !! Comment: It's stuff like this that impels me to use 'settb=expr=1001/24000,setpts=expr=N' (which always works) but 'settb' & 'setpts' mandate transcoding. I'm trying to avoid transcoding. Thanks for your patience and help, Mark Filipak. ===== Begin FFmpeg LOG: ffmpeg -report -i 24'fps[24pps].vob -filter_complex "[0:a]asetpts=expr=N/TB*1001/24000[a]" -bsf setts=pts=N/TB*1001/24000:dts=N/TB*1001/24000 -target ntsc-dvd -map 0:v -codec:v copy -map "[a]" -codec:a ac3 -dn -r 24000/1001 foo.vob ffmpeg started on 2022-02-07 at 17:57:06 Report written to "ffmpeg-20220207-175706.log" Log level: 48 ffmpeg version 2022-02-03-git-e1a14479a8-full_build-www.gyan.dev Copyright (c) 2000-2022 the FFmpeg developers built with gcc 11.2.0 (Rev7, Built by MSYS2 project) configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-bzlib --enable-lzma --enable-libsnappy --enable-zlib --enable-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libdav1d --enable-libdavs2 --enable-libuavs3d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libaom --enable-libopenjpeg --enable-libvpx --enable-mediafoundation --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-libshaderc --enable-vulkan --enable-libplacebo --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint libavutil 57. 19.100 / 57. 19.100 libavcodec 59. 20.100 / 59. 20.100 libavformat 59. 17.101 / 59. 17.101 libavdevice 59. 5.100 / 59. 5.100 libavfilter 8. 26.101 / 8. 26.101 libswscale 6. 5.100 / 6. 5.100 libswresample 4. 4.100 / 4. 4.100 libpostproc 56. 4.100 / 56. 4.100 Input #0, mpeg, from '24'fps[24pps].vob': Duration: 00:00:09.98, start: 0.500000, bitrate: 43021 kb/s Stream #0:0[0x1e0]: Video: mpeg2video (Main), yuv420p(tv, progressive), 720x480 [SAR 32:27 DAR 16:9], Closed Captions, 29.97 fps, 59.94 tbr, 90k tbn Side data: cpb: bitrate max/min/avg: 8401600/0/0 buffer size: 1835008 vbv_delay: N/A Stream #0:1[0x80]: Audio: ac3, 48000 Hz, 5.1(side), fltp, 384 kb/s Multiple -c, -codec, -acodec, -vcodec, -scodec or -dcodec options specified for stream 0, only the last option '-c:v copy' will be used. Multiple -r options specified for stream 0, only the last option '-r 24000/1001' will be used. Multiple -c, -codec, -acodec, -vcodec, -scodec or -dcodec options specified for stream 1, only the last option '-c:a ac3' will be used. Stream mapping: Stream #0:1 (ac3) -> asetpts:default Stream #0:0 -> #0:0 (copy) asetpts:default -> Stream #0:1 (ac3) Press [q] to stop, [?] for help Output #0, dvd, to 'foo.vob': Metadata: encoder : Lavf59.17.101 Stream #0:0: Video: mpeg2video (Main), yuv420p(tv, progressive), 720x480 [SAR 32:27 DAR 16:9], q=2-31, 6000 kb/s, 29.97 fps, 59.94 tbr, 90k tbn Side data: cpb: bitrate max/min/avg: 8401600/0/0 buffer size: 1835008 vbv_delay: N/A Stream #0:1: Audio: ac3, 48000 Hz, 5.1(side), fltp, 448 kb/s Metadata: encoder : Lavc59.20.100 ac3 [dvd @ 000001c8b5094900] buffer underflow st=0 bufi=13621 size=21760 0.0kbits/s speed=1.37x [dvd @ 000001c8b5094900] buffer underflow st=0 bufi=15645 size=21760 [dvd @ 000001c8b5094900] buffer underflow st=0 bufi=17669 size=21760 et cetera ===== Begin FFprobe LOG: Showing the 1st 2 key frames. Note that the PTSs are wrong. ffprobe version 2022-02-03-git-e1a14479a8-full_build-www.gyan.dev Copyright (c) 2007-2022 the FFmpeg developers [mpeg @ 0000023e3f669900] start time for stream 0 is not set in estimate_timings_from_pts [mpeg @ 0000023e3f669900] stream 1 : no PTS found at end of file, duration not set Input #0, mpeg, from 'D:\D\Setup\Video\ffmpeg\HowTo ffmpeg, vapoursynth\24'fps[24pps]24fps & 48KHz[x0.999]48KHz\foo.vob': Duration: 00:00:55.53, start: 0.500000, bitrate: 7823 kb/s Stream #0:0[0x1bf]: Data: dvd_nav_packet Stream #0:1[0x1e0]: Video: mpeg2video (Main), yuv420p(tv, progressive), 720x480 [SAR 32:27 DAR 16:9], Closed Captions, 29.97 fps, 29.97 tbr, 90k tbn Side data: cpb: bitrate max/min/avg: 8401600/0/0 buffer size: 1835008 vbv_delay: N/A Stream #0:2[0x80]: Audio: ac3, 48000 Hz, 5.1(side), fltp, 448 kb/s Unsupported codec with id 98312 for input stream 0 frames.frame.0.key_frame=1 frames.frame.0.pts=45000 <= should be zero frames.frame.0.pkt_dts=45000 : frames.frame.12.key_frame=1 frames.frame.12.pts=45012 <= should be 3753 frames.frame.12.pkt_dts=45012

2 1

Video Notation, A Video Lingua Franca
by Mark Filipak 08 Feb '22

08 Feb '22

Update: I just checked the archive and there were no attachments. So ZIP worked but HTML apparently doesn't work. I'm trying again. There should be 11 HTML attachments, 160 KBytes, total. On suggestion, I tried web hosting suppliers and failed. I've attached the 11 HTML pages -- no ZIPs. I don't know how I could make it easier. Suppose I used a file sharing service? You'd have to download the HTLM pages, right? Well, the HTML pages are attached, so you don't have to download them. They're already here. Just save them to your hard drive (160 KBytes) and click any one of them. Regards, Mark Filipak.

1 0

Video Notation, A Video Lingua Franca 1+2of11.zip (8.8 KB)
by Mark Filipak 07 Feb '22

07 Feb '22

Greetings! For those interested in an object-oriented video notation for concise descriptions of video structures... ___ This attachment: / => Video Notation.01.Preface.html => Video Notation.02.Teasers.html Video Notation.03.EnhancedTerminology.html Video Notation.04.Reference.html Video Notation.05.EncodingOfDVD&BlurayContent.html Video Notation.06.AboutAudio.html Video Notation.07.RecoveringTheCameraShots.html Video Notation.08.BasicPrimitives.html Video Notation.09.PulldownPrimitives.html Video Notation.10.AdvancedInterpolations.html Video Notation.11.SeenInTheWild.html Warm Regards, Mark Filipak.

3 5

Video Notation, A Video Lingua Franca 9+10+11of11.zip (8.3 KB) -- links fixed
by Mark Filipak 07 Feb '22

07 Feb '22

Greetings! For those interested in an object-oriented video notation for concise descriptions of video structures... Video Notation.01.Preface.html Video Notation.02.Teasers.html Video Notation.03.EnhancedTerminology.html Video Notation.04.Reference.html Video Notation.05.EncodingOfDVD&BlurayContent.html Video Notation.06.AboutAudio.html Video Notation.07.RecoveringTheCameraShots.html Video Notation.08.BasicPrimitives.html ___ This attachment: / => Video Notation.09.PulldownPrimitives.html => Video Notation.10.AdvancedInterpolations.html => Video Notation.11.SeenInTheWild.html Warm Regards, Mark Filipak.

1 0