should I shoot the dog?
I've spent 2 days studying frame.h, pixfmt.h, dpx.c, packet.h, and escape124.c. I haven't learned a damn thing. Despite their vagueness and ambiguity, reading and understanding H.222 & H.262 are dead easy by comparison [1]. I just want to understand the frame structures that ffmpeg creates, and that ffmpeg uses in processing and filtering. Are Y, Cb, Cr separate buffers? That would be logical. Or are the Y, Cb, Cr values combined and organized similarly to macroblocks? I've found some code that supports that. Or are the Y, Cb, Cr values thrown together, pixel-by-pixel. That would be logical, too. I really can't understand how anyone can architect these things without making some pictures. Can anyone here help me, or should I shoot the dog? Notes: [1] Reading and understanding H.222 & H.262 is slightly easier than self-administered appendectomy. -- The U.S. political problem? Amateurs are doing the street fighting. The Princeps Senatus and the Tribunus Plebis need their own armies.
Am 29.09.2020 um 04:28 schrieb Mark Filipak (ffmpeg):
I just want to understand the frame structures that ffmpeg creates, and that ffmpeg uses in processing and filtering. Are Y, Cb, Cr separate buffers? That would be logical. Or are the Y, Cb, Cr values combined and organized similarly to macroblocks? I've found some code that supports that. Or are the Y, Cb, Cr values thrown together, pixel-by-pixel. That would be logical, too.
As far as I understood it, that depends on the pixel format. For example there are "packed" pixel formats rgb24, bgr24, argb, rgba, abgr, bgra,rgb48be, rgb48le, bgr48be, bgr48le. And there are "planar" pixel formats gbrp, bgrp16be, bgrp16le. Michael
On 09/29/2020 04:06 AM, Michael Koch wrote:
Am 29.09.2020 um 04:28 schrieb Mark Filipak (ffmpeg):
I just want to understand the frame structures that ffmpeg creates, and that ffmpeg uses in processing and filtering. Are Y, Cb, Cr separate buffers? That would be logical. Or are the Y, Cb, Cr values combined and organized similarly to macroblocks? I've found some code that supports that. Or are the Y, Cb, Cr values thrown together, pixel-by-pixel. That would be logical, too.
As far as I understood it, that depends on the pixel format. For example there are "packed" pixel formats rgb24, bgr24, argb, rgba, abgr, bgra,rgb48be, rgb48le, bgr48be, bgr48le. And there are "planar" pixel formats gbrp, bgrp16be, bgrp16le.
Hi Michael, "Packed" and "planar", eh? What evidence do you have? ...Share the candy! Now, I'm not talking about streams. I'm talking about after decoding. I'm talking about the buffers. I would think that a single, consistent format would be used. ? ? ? ? ? So, why am I interested in ffmpeg's internal video buffer format? ...I've been here for about 1/2 year now, watching the ffmpeg, slow motion train wreck. It seems to me that the ffmpeg patricians assume that everyone knows the formats just because the patricians do, and have documented based on that assumption. Because we plebians don't know the format, and we don't know that we don't know, the patricians get frustrated with us and become short tempered and then the word "troll" flies. I'm just a simple engineer. To understand an architecture, all I need is the structures, preferably as pictures, and maybe a bit of the processing flow, preferably via flow diagrams (i.e. step '1', then step '2', then step '3', etc.) -- I'm a visual kinda guy -- but I usually don't need to know the processing. Examining the source code doesn't work for me. 'C' code is just too cryptic and I'm too ignorant. -- The U.S. political problem? Amateurs are doing the street fighting. The Princeps Senatus and the Tribunus Plebis need their own armies.
Hi Mark, So when you talk about the decoded frames, there is no concept of macroblocks. There are simple video frames with Y, Cb, Cr samples. How those samples are organized and their sizes are determined by the AVFrame format.
"Packed" and "planar", eh? What evidence do you have? ...Share the candy!
Now, I'm not talking about streams. I'm talking about after decoding. I'm talking about the buffers. I would think that a single, consistent format would be used.
When dealing with typical consumer MPEG-2 or H.264 video, the decoded frames will typically have what's referred to as "4:2:0 planar" format. This means that the individual Y/Cb/Cr samples are not contiguous. If you look at the underlying data that makes up the frame as an array, you will typically have W*H Y values, followed by W*H/4 Cb values, and then there will be W*H/4 Cr values. Note that I say "values" and not "bytes", as the size of each value may vary depending on the pixel format. Unfortunately there is no "single, consistent format" because of the variety of different ways in which the video can be encoded, as well as performance concerns. Normalizing it to a single format can have a huge performance cost, in particular if the original video is in a different colorspace (e.g. the video is YUV and you want RGB). Generally speaking you can set up the pipeline to always deliver you a single format, and ffmpeg will automatically perform any transformations necessary to achieve that (e.g. convert from packed to planer, RGB to YUV, 8-bit to 10-bit, 4:2:2 to 4:2:0, etc). However this can have a severe performance cost and can result in quality loss depending on the transforms required. The codec will typically specify its output format, largely dependent on the nature of the encoding, and then announce AVFrames that conform to that format. Since you're largely dealing with MPEG-2 and H.264 video, it's almost always going to be YUV 4:2:0 planar. The filter pipeline can then do conversion if needed, either because you told it to convert it or because you specified some filter pipeline where the individual filter didn't support what format it was being given. See libavutil/pixfmt.h for a list of all the possible formats in which AVFrames can be announced by a codec. Devin -- Devin Heitmueller, Senior Software Engineer LTN Global Communications o: +1 (301) 363-1001 w: https://ltnglobal.com e: devin.heitmueller@ltnglobal.com
On 09/29/2020 09:20 AM, Devin Heitmueller wrote:
Hi Mark,
Hi Devin. Thanks much! Your response came in while I was composing my previous message. I see (below) that performance is a major issue. That absolutely makes sense because, after accuracy, speed is the next most important objective (and for some use cases, may actually be more important). I imagine that format-to-format conversion is probably the most optimized code in ffmpeg. Is there a function library dedicated solely to format conversion? I ask so that, in what I write, I can assure users that the issues are known and addressed. For my modest purposes, a sketch of planar v. packed is probably all that's needed. I think you've made "planar" clear. Thank you for that. I can imagine that the structure of packed is multitudinous. Why is it called "packed"? How is it packed? Are the luma & chroma mixed in one buffer (analogous to blocks in macroblocks) or split into discrete buffers? How are they spacially structured? Is there any special sub structures (analogous to macroblocks in slices)? Are the sub structures, if any, format dependent?
So when you talk about the decoded frames, there is no concept of macroblocks. There are simple video frames with Y, Cb, Cr samples. How those samples are organized and their sizes are determined by the AVFrame format.
"Packed" and "planar", eh? What evidence do you have? ...Share the candy!
Now, I'm not talking about streams. I'm talking about after decoding. I'm talking about the buffers. I would think that a single, consistent format would be used.
When dealing with typical consumer MPEG-2 or H.264 video, the decoded frames will typically have what's referred to as "4:2:0 planar" format. This means that the individual Y/Cb/Cr samples are not contiguous. If you look at the underlying data that makes up the frame as an array, you will typically have W*H Y values, followed by W*H/4 Cb values, and then there will be W*H/4 Cr values. Note that I say "values" and not "bytes", as the size of each value may vary depending on the pixel format.
Unfortunately there is no "single, consistent format" because of the variety of different ways in which the video can be encoded, as well as performance concerns. Normalizing it to a single format can have a huge performance cost, in particular if the original video is in a different colorspace (e.g. the video is YUV and you want RGB). Generally speaking you can set up the pipeline to always deliver you a single format, and ffmpeg will automatically perform any transformations necessary to achieve that (e.g. convert from packed to planer, RGB to YUV, 8-bit to 10-bit, 4:2:2 to 4:2:0, etc). However this can have a severe performance cost and can result in quality loss depending on the transforms required.
The codec will typically specify its output format, largely dependent on the nature of the encoding, and then announce AVFrames that conform to that format. Since you're largely dealing with MPEG-2 and H.264 video, it's almost always going to be YUV 4:2:0 planar. The filter pipeline can then do conversion if needed, either because you told it to convert it or because you specified some filter pipeline where the individual filter didn't support what format it was being given.
See libavutil/pixfmt.h for a list of all the possible formats in which AVFrames can be announced by a codec.
Devin
-- The U.S. political problem? Amateurs are doing the street fighting. The Princeps Senatus and the Tribunus Plebis need their own armies.
Hi Mark, On Tue, Sep 29, 2020 at 10:51 AM Mark Filipak (ffmpeg) <markfilipak@bog.us> wrote:
I imagine that format-to-format conversion is probably the most optimized code in ffmpeg. Is there a function library dedicated solely to format conversion? I ask so that, in what I write, I can assure users that the issues are known and addressed.
Most of these sorts of conversions for video are performed by the swscale library, which is part of ffmpeg. When ffmpeg constructs a filter pipeline it looks at the input and output formats for each filter in the chain, and if they don't support the same format an instance of the vf_scale filter is automatically inserted into the pipeline between the two filters to perform the conversion. It's worth noting that in some cases this can cause significant performance problems and more than once I've found a performance bottleneck to be that a filter performing conversion was automatically inserted without my knowledge. A good deal of effort has been put into optimizing swscale, but it's a huge matrix of conversions that are possible and hence some work better than others. Hence performance can vary greatly depending on the transform being done.
For my modest purposes, a sketch of planar v. packed is probably all that's needed. I think you've made "planar" clear. Thank you for that. I can imagine that the structure of packed is multitudinous. Why is it called "packed"? How is it packed? Are the luma & chroma mixed in one buffer (analogous to blocks in macroblocks) or split into discrete buffers? How are they spacially structured? Is there any special sub structures (analogous to macroblocks in slices)? Are the sub structures, if any, format dependent?
The following describes the differences between packed an planar: https://wiki.videolan.org/YUV With packed the luma and chroma are interleaved (YCbCrYCbCr). With planar they are not (YYY...CbCbCb...CrCrCr). There is also something known as "semi-planar" where the Y is separate but CbCr is interleaved (YYY...CbCrCbCrCbCr). Whether the buffers themselves are contiguous is implementation dependent. In ffmpeg you get pointers to the individual buffers for each plane, but there is nothing to say that under the hood the second buffer isn't immediately after the first buffer in the underlying memory. Of course you should never write code that makes this assumption, and perform operations against each plane without assuming the data is adjacent to the buffer for another plane. Devin -- Devin Heitmueller, Senior Software Engineer LTN Global Communications o: +1 (301) 363-1001 w: https://ltnglobal.com e: devin.heitmueller@ltnglobal.com
On Tue, 29 Sep 2020 10:48:42 -0400 "Mark Filipak (ffmpeg)" <markfilipak@bog.us> wrote:
Hi Devin. Thanks much!
Your response came in while I was composing my previous message. I see (below) that performance is a
Because it reverses the normal order of reading! Why not top post?
On 09/29/2020 11:09 AM, Dave Stevens wrote:
On Tue, 29 Sep 2020 10:48:42 -0400 "Mark Filipak (ffmpeg)" <markfilipak@bog.us> wrote:
Hi Devin. Thanks much!
Your response came in while I was composing my previous message. I see (below) that performance is a
Because it reverses the normal order of reading!
Why not top post?
Hi Dave, Top posting is discouraged in the ffmpeg-user list. I personally loathe top posting and prefer an interleaved, call-and-response model. However, in the cited case, I felt that call-and-response would not have worked and would simply have been boring and "me too". In other words, I just wanted to acknowledge Devin's contribution and thank him one time, in one place. -- The U.S. political problem? Amateurs are doing the street fighting. The Princeps Senatus and the Tribunus Plebis need their own armies.
Am 29.09.2020 um 14:58 schrieb Mark Filipak (ffmpeg):
On 09/29/2020 04:06 AM, Michael Koch wrote:
Am 29.09.2020 um 04:28 schrieb Mark Filipak (ffmpeg):
I just want to understand the frame structures that ffmpeg creates, and that ffmpeg uses in processing and filtering. Are Y, Cb, Cr separate buffers? That would be logical. Or are the Y, Cb, Cr values combined and organized similarly to macroblocks? I've found some code that supports that. Or are the Y, Cb, Cr values thrown together, pixel-by-pixel. That would be logical, too.
As far as I understood it, that depends on the pixel format. For example there are "packed" pixel formats rgb24, bgr24, argb, rgba, abgr, bgra,rgb48be, rgb48le, bgr48be, bgr48le. And there are "planar" pixel formats gbrp, bgrp16be, bgrp16le.
Hi Michael,
"Packed" and "planar", eh? What evidence do you have? ...Share the candy!
As far as I know, this is not described in the official documentation. You can find it for example here: https://video.stackexchange.com/questions/16374/ffmpeg-pixel-format-definiti...
Now, I'm not talking about streams. I'm talking about after decoding. I'm talking about the buffers. I would think that a single, consistent format would be used.
There is no single consistent format used internally. See Gyan's answer here: http://ffmpeg.org/pipermail/ffmpeg-user/2020-September/050031.html Michael
On 09/29/2020 09:37 AM, Michael Koch wrote:
Am 29.09.2020 um 14:58 schrieb Mark Filipak (ffmpeg):
On 09/29/2020 04:06 AM, Michael Koch wrote:
Am 29.09.2020 um 04:28 schrieb Mark Filipak (ffmpeg):
I just want to understand the frame structures that ffmpeg creates, and that ffmpeg uses in processing and filtering. Are Y, Cb, Cr separate buffers? That would be logical. Or are the Y, Cb, Cr values combined and organized similarly to macroblocks? I've found some code that supports that. Or are the Y, Cb, Cr values thrown together, pixel-by-pixel. That would be logical, too.
As far as I understood it, that depends on the pixel format. For example there are "packed" pixel formats rgb24, bgr24, argb, rgba, abgr, bgra,rgb48be, rgb48le, bgr48be, bgr48le. And there are "planar" pixel formats gbrp, bgrp16be, bgrp16le.
Hi Michael,
"Packed" and "planar", eh? What evidence do you have? ...Share the candy!
As far as I know, this is not described in the official documentation. You can find it for example here: https://video.stackexchange.com/questions/16374/ffmpeg-pixel-format-definiti...
Thanks for that. It saved me some time. ...So, what does "planar" mean? What does "packed" mean?
Now, I'm not talking about streams. I'm talking about after decoding. I'm talking about the buffers. I would think that a single, consistent format would be used. There is no single consistent format used internally. See Gyan's answer here: http://ffmpeg.org/pipermail/ffmpeg-user/2020-September/050031.html
And thanks for that. So, ffmpeg really is a Tower of Babel, eh? That does not seem wise to me. That seems like a great way to generate bugs in addition to confusion. Now, I imagine that converting to a lingua franca would blow up processing time, so it isn't done. However, if there are format-to-format regular expressions for conversions, may I suggest that those regular expressions be published? Also, if Y Cb & Cr are separate buffers, may I suggest that ffmpeg publish that? In school, I learned that inputs and outputs should be fully characterized, not suggested, not implied, but fully characterized as structures. That takes time, and it takes review and perfection by informed people, but that time is worth the investment. -- The U.S. political problem? Amateurs are doing the street fighting. The Princeps Senatus and the Tribunus Plebis need their own armies.
Am 29.09.2020 um 16:26 schrieb Mark Filipak (ffmpeg):
On 09/29/2020 09:37 AM, Michael Koch wrote:
Am 29.09.2020 um 14:58 schrieb Mark Filipak (ffmpeg):
On 09/29/2020 04:06 AM, Michael Koch wrote:
Am 29.09.2020 um 04:28 schrieb Mark Filipak (ffmpeg):
I just want to understand the frame structures that ffmpeg creates, and that ffmpeg uses in processing and filtering. Are Y, Cb, Cr separate buffers? That would be logical. Or are the Y, Cb, Cr values combined and organized similarly to macroblocks? I've found some code that supports that. Or are the Y, Cb, Cr values thrown together, pixel-by-pixel. That would be logical, too.
As far as I understood it, that depends on the pixel format. For example there are "packed" pixel formats rgb24, bgr24, argb, rgba, abgr, bgra,rgb48be, rgb48le, bgr48be, bgr48le. And there are "planar" pixel formats gbrp, bgrp16be, bgrp16le.
Hi Michael,
"Packed" and "planar", eh? What evidence do you have? ...Share the candy!
As far as I know, this is not described in the official documentation. You can find it for example here: https://video.stackexchange.com/questions/16374/ffmpeg-pixel-format-definiti...
Thanks for that. It saved me some time. ...So, what does "planar" mean? What does "packed" mean?
Here is an example for a very small image with 3 x 2 pixels. In (packed) RGB24 format: RGBRGBRGBRGBRGBRGB In (planar) GBRP format: GGGGGGBBBBBBRRRRRR Michael
On 09/29/2020 10:46 AM, Michael Koch wrote:
Am 29.09.2020 um 16:26 schrieb Mark Filipak (ffmpeg):
On 09/29/2020 09:37 AM, Michael Koch wrote:
Am 29.09.2020 um 14:58 schrieb Mark Filipak (ffmpeg):
On 09/29/2020 04:06 AM, Michael Koch wrote:
Am 29.09.2020 um 04:28 schrieb Mark Filipak (ffmpeg):
I just want to understand the frame structures that ffmpeg creates, and that ffmpeg uses in processing and filtering. Are Y, Cb, Cr separate buffers? That would be logical. Or are the Y, Cb, Cr values combined and organized similarly to macroblocks? I've found some code that supports that. Or are the Y, Cb, Cr values thrown together, pixel-by-pixel. That would be logical, too.
As far as I understood it, that depends on the pixel format. For example there are "packed" pixel formats rgb24, bgr24, argb, rgba, abgr, bgra,rgb48be, rgb48le, bgr48be, bgr48le. And there are "planar" pixel formats gbrp, bgrp16be, bgrp16le.
Hi Michael,
"Packed" and "planar", eh? What evidence do you have? ...Share the candy!
As far as I know, this is not described in the official documentation. You can find it for example here: https://video.stackexchange.com/questions/16374/ffmpeg-pixel-format-definiti...
Thanks for that. It saved me some time. ...So, what does "planar" mean? What does "packed" mean?
Here is an example for a very small image with 3 x 2 pixels. In (packed) RGB24 format: RGBRGBRGBRGBRGBRGB
Oh, dear, that's what "packed" means? ...very misleading name, eh? How are fields handled? Are the pixels assumed to be unfielded (meaning so-called "progressive")?
In (planar) GBRP format: GGGGGGBBBBBBRRRRRR
What about fields? In macroblock format, samples are 1st spacially divided vertically into by-16 slices, then spacially divided within slices into by-16 macroblocks, then, within macroblocks, divided by into (combined) colorant-field blocks: Ytop Ytop Ybottom Ybottom Cb Cr, and, within Cb Cr colorants, into field half-blocks, and finally, interleaved by 2 levels of interleaving. ...An overly complicated and (to me) ill-conceived set of datasets that illustrates (to me) that the video "engineers" of the Motion Pictures Expert Group are lightweight engineers and have hacked a "system". It is structure to the field-level that I'm most interested in, but a deep dive would be fun. -- The U.S. political problem? Amateurs are doing the street fighting. The Princeps Senatus and the Tribunus Plebis need their own armies.
On Tue, Sep 29, 2020 at 11:29 AM Mark Filipak (ffmpeg) <markfilipak@bog.us> wrote:
Oh, dear, that's what "packed" means? ...very misleading name, eh? How are fields handled? Are the pixels assumed to be unfielded (meaning so-called "progressive")?
So the topic of how interlaced video is handled in subsampled video is something we could spend an hour on by itself. In the Luma space, the Y samples are organized in interleaved form (i.e. lines of top/bottom/top/bottom). Because of chroma subsampling and the fact that multiple lines can share chroma samples, this gets tricky. In the simple progressive case for 4:2:0, you'll have the first Chroma sample corresponding to the first two luma samples on line 1 and the first two luma samples on line 2. If the video frame is interlaced however, the first chroma sample corresponds to the first two luma samples on line 1 and the first two luma samples on line 3. The first chroma sample on the second line of chroma corresponds with the first two luma samples on line 2 and the first two luma samples on line 4. This is known as "interlaced chroma" and a Google search will reveal lots of cases where it's done wrong and what the effects are. This is the article I usually refer people to: https://hometheaterhifi.com/technical/technical-reviews/the-chroma-upsamplin... The above article does a really good job explaining the behavior (far better than I could do in the one paragraph above). Devin -- Devin Heitmueller, Senior Software Engineer LTN Global Communications o: +1 (301) 363-1001 w: https://ltnglobal.com e: devin.heitmueller@ltnglobal.com
On 09/29/2020 11:44 AM, Devin Heitmueller wrote:
On Tue, Sep 29, 2020 at 11:29 AM Mark Filipak (ffmpeg) <markfilipak@bog.us> wrote:
Oh, dear, that's what "packed" means? ...very misleading name, eh? How are fields handled? Are the pixels assumed to be unfielded (meaning so-called "progressive")?
So the topic of how interlaced video is handled in subsampled video is something we could spend an hour on by itself. In the Luma space, the Y samples are organized in interleaved form (i.e. lines of top/bottom/top/bottom). ...
Top/bottom/top/bottom, especially if full lines, seems like straightforward interlaced to me. Or do I misunderstand?
... Because of chroma subsampling and the fact that multiple lines can share chroma samples, this gets tricky. ...
Our messages crossed in transit, but I'm going to assume that you've seen my "In macroblock format..." post (in this subject thread).
... In the simple progressive case for 4:2:0, you'll have the first Chroma sample corresponding to the first two luma samples on line 1 and the first two luma samples on line 2. ...
I assume you meant to write "and the *next* two luma samples on line 2". That 'sounds' like what I'm calling sample-quads. The following is from the glossary I'm working on (reformatted to fit email). YCbCr420 sampleset: A sampleset with sample-quads: .---.---. ¦ S ¦ S ¦ :---:---: ¦ S ¦ S ¦ '---'---', reduced to 1/4 chrominance resolution: .---.---. .-------. .-------. ¦ Y ¦ Y ¦ ¦ ¦ ¦ ¦ :---:---: ¦Cb ¦ ¦Cr ¦ ¦ Y ¦ Y ¦ ¦ ¦ ¦ ¦ '---'---' '-------' '-------', distinguished by binary metadata: 'chroma_format' = 01. (See "Cb420 & Cr420 macroblocks", "Y macroblock".) YCbCr422 sampleset: A sampleset with sample-quads: .---.---. ¦ S ¦ S ¦ :---:---: ¦ S ¦ S ¦ '---'---', reduced to 1/2 chrominance resolution: .---.---. .-------. .-------. ¦ Y ¦ Y ¦ ¦Cb ¦ ¦Cr ¦ :---:---: :-------: :-------: ¦ Y ¦ Y ¦ ¦Cb ¦ ¦Cr ¦ '---'---' '-------' '-------', distinguished by binary metadata: 'chroma_format' = 10. (See "Cb422 & Cr422 macroblocks", "Y macroblock".) YCbCr444 sampleset: A sampleset with sample-quads: .---.---. ¦ S ¦ S ¦ :---:---: ¦ S ¦ S ¦ '---'---', having full chrominance resolution: .---.---. .---.---. .---.---. ¦ Y ¦ Y ¦ ¦Cb ¦Cb ¦ ¦Cr ¦Cr ¦ :---:---: :---:---: :---:---: ¦ Y ¦ Y ¦ ¦Cb ¦Cb ¦ ¦Cr ¦Cr ¦ '---'---' '---'---' '---'---', distinguished by binary metadata: 'chroma_format' = 11. (See "Cb444 & Cr444 macroblocks", "Y macroblock".)
... If the video frame is interlaced however, the first chroma sample corresponds to the first two luma samples on line 1 and the first two luma samples on line 3. The first chroma sample on the second line of chroma corresponds with the first two luma samples on line 2 and the first two luma samples on line 4.
I have pictures of those, too. What do you think of the above pictures? Do you a, like them, or b, loathe them, or c, find them unnecessary?
This is known as "interlaced chroma" and a Google search will reveal lots of cases where it's done wrong and what the effects are. This is the article I usually refer people to:
https://hometheaterhifi.com/technical/technical-reviews/the-chroma-upsamplin...
The above article does a really good job explaining the behavior (far better than I could do in the one paragraph above).
I've seen that produce mild combing. I'll read your reference. -- -- The U.S. political problem? Amateurs are doing the street fighting. The Princeps Senatus and the Tribunus Plebis need their own armies.
On Tue, Sep 29, 2020 at 12:28 PM Mark Filipak (ffmpeg) <markfilipak@bog.us> wrote:
Top/bottom/top/bottom, especially if full lines, seems like straightforward interlaced to me. Or do I misunderstand?
You do not misunderstand. I was offering the context of how the Y-data is organized, which is straightforward compared to the explanation that followed on how the chroma data relates to the luma data (and how that relationship differs depending on whether the frame represents interlaced vs. progressive video).
... Because of chroma subsampling and the fact that multiple lines can share chroma samples, this gets tricky. ...
Our messages crossed in transit, but I'm going to assume that you've seen my "In macroblock format..." post (in this subject thread).
... In the simple progressive case for 4:2:0, you'll have the first Chroma sample corresponding to the first two luma samples on line 1 and the first two luma samples on line 2. ...
I assume you meant to write "and the *next* two luma samples on line 2".
No, what I wrote is what I intended. The 4:2:0 pixel format has four luma samples, two from the first line, and the two that are directly below those on the second line.
That 'sounds' like what I'm calling sample-quads. The following is from the glossary I'm working on (reformatted to fit email).
I would encourage you stop trying to invent new terminology, in particular as you are still learning the basics like "What is 4:2:0 planar format?" You're better off explaining what 4:2:0 is than showing a diagram of some pixels and giving it a name which is the equivalent to 4:2:0.
YCbCr420 sampleset: A sampleset with sample-quads: .---.---. ¦ S ¦ S ¦ :---:---: ¦ S ¦ S ¦ '---'---', reduced to 1/4 chrominance resolution: .---.---. .-------. .-------. ¦ Y ¦ Y ¦ ¦ ¦ ¦ ¦ :---:---: ¦Cb ¦ ¦Cr ¦ ¦ Y ¦ Y ¦ ¦ ¦ ¦ ¦ '---'---' '-------' '-------', distinguished by binary metadata: 'chroma_format' = 01. (See "Cb420 & Cr420 macroblocks", "Y macroblock".)
YCbCr422 sampleset: A sampleset with sample-quads: .---.---. ¦ S ¦ S ¦ :---:---: ¦ S ¦ S ¦ '---'---', reduced to 1/2 chrominance resolution: .---.---. .-------. .-------. ¦ Y ¦ Y ¦ ¦Cb ¦ ¦Cr ¦ :---:---: :-------: :-------: ¦ Y ¦ Y ¦ ¦Cb ¦ ¦Cr ¦ '---'---' '-------' '-------', distinguished by binary metadata: 'chroma_format' = 10. (See "Cb422 & Cr422 macroblocks", "Y macroblock".)
YCbCr444 sampleset: A sampleset with sample-quads: .---.---. ¦ S ¦ S ¦ :---:---: ¦ S ¦ S ¦ '---'---', having full chrominance resolution: .---.---. .---.---. .---.---. ¦ Y ¦ Y ¦ ¦Cb ¦Cb ¦ ¦Cr ¦Cr ¦ :---:---: :---:---: :---:---: ¦ Y ¦ Y ¦ ¦Cb ¦Cb ¦ ¦Cr ¦Cr ¦ '---'---' '---'---' '---'---', distinguished by binary metadata: 'chroma_format' = 11. (See "Cb444 & Cr444 macroblocks", "Y macroblock".)
The diagrams are probably fine, but probably not how I would draw them given they blur the relationship between packed and planar. Either it's packed, in which case you should probably show 4:2:2 as YCbYCr, or it's planer in which case the Cb/Cr samples should not be adjacent per line (i.e. have all the Y lines followed by all the Cr/Cb lines). You may wish to take into your account your newfound understanding of packed vs. planar to redo these diagrams representing the data as either one or the other. I would probably also refrain from using the term "macroblock" to describe the raw decoded video, as macroblocks are all about how the pixels are organized in the compressed domain. Once they are decoded there is no notion of macroblocks in the resulting video frames.
... If the video frame is interlaced however, the first chroma sample corresponds to the first two luma samples on line 1 and the first two luma samples on line 3. The first chroma sample on the second line of chroma corresponds with the first two luma samples on line 2 and the first two luma samples on line 4.
I have pictures of those, too. What do you think of the above pictures? Do you a, like them, or b, loathe them, or c, find them unnecessary?
I would probably see if you can find drawings already out there. For example the Wikipedia article on YUV has some pretty good representations for pixel arrangement in various pixel formats. So does the LinuxTV documentation.
This is known as "interlaced chroma" and a Google search will reveal lots of cases where it's done wrong and what the effects are. This is the article I usually refer people to:
https://hometheaterhifi.com/technical/technical-reviews/the-chroma-upsamplin...
The above article does a really good job explaining the behavior (far better than I could do in the one paragraph above).
I've seen that produce mild combing. I'll read your reference.
Yes, it often manifests visually as a combing effect, but it's not as pronounced as a typical deinterlacing artifact since it's only the chroma lines that are rendered in the wrong order. Hence it is most obvious with certain content types like animation (where you are more likely to have sharp transitions between certain colors). Also, deinterlacing artifacts are most visible with high motion, whereas interlaced chroma artifacts can be visible even with static non-moving video. Devin -- Devin Heitmueller, Senior Software Engineer LTN Global Communications o: +1 (301) 363-1001 w: https://ltnglobal.com e: devin.heitmueller@ltnglobal.com
On 09/29/2020 12:57 PM, Devin Heitmueller wrote:
On Tue, Sep 29, 2020 at 12:28 PM Mark Filipak (ffmpeg) <markfilipak@bog.us> wrote: -snip- I would encourage you stop trying to invent new terminology ... -snip-
With due respect to you, I'm not trying to invent new terminology. I'm trying to create extended terminology that builds on the existing terminology. But we shall see, eh? If what I do is crap, then I'll be the first to throw it away. I've thrown away weeks of work in the past.
YCbCr420 sampleset: A sampleset with sample-quads: .---.---. ¦ S ¦ S ¦ :---:---: ¦ S ¦ S ¦ '---'---', reduced to 1/4 chrominance resolution: .---.---. .-------. .-------. ¦ Y ¦ Y ¦ ¦ ¦ ¦ ¦ :---:---: ¦Cb ¦ ¦Cr ¦ ¦ Y ¦ Y ¦ ¦ ¦ ¦ ¦ '---'---' '-------' '-------', distinguished by binary metadata: 'chroma_format' = 01. (See "Cb420 & Cr420 macroblocks", "Y macroblock".)
YCbCr422 sampleset: A sampleset with sample-quads: .---.---. ¦ S ¦ S ¦ :---:---: ¦ S ¦ S ¦ '---'---', reduced to 1/2 chrominance resolution: .---.---. .-------. .-------. ¦ Y ¦ Y ¦ ¦Cb ¦ ¦Cr ¦ :---:---: :-------: :-------: ¦ Y ¦ Y ¦ ¦Cb ¦ ¦Cr ¦ '---'---' '-------' '-------', distinguished by binary metadata: 'chroma_format' = 10. (See "Cb422 & Cr422 macroblocks", "Y macroblock".)
YCbCr444 sampleset: A sampleset with sample-quads: .---.---. ¦ S ¦ S ¦ :---:---: ¦ S ¦ S ¦ '---'---', having full chrominance resolution: .---.---. .---.---. .---.---. ¦ Y ¦ Y ¦ ¦Cb ¦Cb ¦ ¦Cr ¦Cr ¦ :---:---: :---:---: :---:---: ¦ Y ¦ Y ¦ ¦Cb ¦Cb ¦ ¦Cr ¦Cr ¦ '---'---' '---'---' '---'---', distinguished by binary metadata: 'chroma_format' = 11. (See "Cb444 & Cr444 macroblocks", "Y macroblock".)
The diagrams are probably fine, but probably not how I would draw them given they blur the relationship between packed and planar. Either it's packed, in which case you should probably show 4:2:2 as YCbYCr, or it's planer in which case the Cb/Cr samples should not be adjacent per line (i.e. have all the Y lines followed by all the Cr/Cb lines). You may wish to take into your account your newfound understanding of packed vs. planar to redo these diagrams representing the data as either one or the other.
Thank you, Devin. Yes, the diagrams are incomplete. And, yes, I will do diagrams that take planar v. packed into account. I will post them when completed. May I also say that I appreciate your attitude: That seekers are not stupid or trolls. Regarding "adjacent per line", the references to "Cb444 & Cr444 macroblocks", "Y macroblock" make that clear, but I will revise the note to better indicate that the chroma subsamples are not adjacent. Regarding "4:2:2 as YCbYCr" packed, I can't fully visualize it because, I think, there should be 4 Y samples, not 2. But don't explain it, though. Not yet. Wait until I post a diagram of it and then let me know what you think and how that diagram is wrong. :-) I don't want to exploit your generosity. I'll do the grunt work.
I would probably also refrain from using the term "macroblock" to describe the raw decoded video, as macroblocks are all about how the pixels are organized in the compressed domain. Once they are decoded there is no notion of macroblocks in the resulting video frames.
Got it. Regarding "compressed domain" (in which macroblocks are sparse), that's what I initially thought, but H.262 pretty strongly implies that macroblocks also apply to raw video. That seems logical to me (as datasets prior to compression). Unrelated: In the glossary, I seek to always have "distinguished by" clauses so that readers can be sure about when and where a particular definition applies.
... If the video frame is interlaced however, the first chroma sample corresponds to the first two luma samples on line 1 and the first two luma samples on line 3. The first chroma sample on the second line of chroma corresponds with the first two luma samples on line 2 and the first two luma samples on line 4.
I have pictures of those, too. What do you think of the above pictures? Do you a, like them, or b, loathe them, or c, find them unnecessary?
I would probably see if you can find drawings already out there. For example the Wikipedia article on YUV has some pretty good representations for pixel arrangement in various pixel formats. So does the LinuxTV documentation.
Thanks for the tips.
This is known as "interlaced chroma" and a Google search will reveal lots of cases where it's done wrong and what the effects are. This is the article I usually refer people to:
https://hometheaterhifi.com/technical/technical-reviews/the-chroma-upsamplin...
The above article does a really good job explaining the behavior (far better than I could do in the one paragraph above).
I've seen that produce mild combing. I'll read your reference.
Yes, it often manifests visually as a combing effect, but it's not as pronounced as a typical deinterlacing artifact since it's only the chroma lines that are rendered in the wrong order. Hence it is most obvious with certain content types like animation (where you are more likely to have sharp transitions between certain colors). Also, deinterlacing artifacts are most visible with high motion, whereas interlaced chroma artifacts can be visible even with static non-moving video.
Indeed. I have cited exactly what you cite in the various definitions, re: motion object edges v. stationary object edges, and also the effects on verticals v. diagonals. -- The U.S. political problem? Amateurs are doing the street fighting. The Princeps Senatus and the Tribunus Plebis need their own armies.
On Tue, Sep 29, 2020 at 10:26:20AM -0400, Mark Filipak (ffmpeg) wrote:
On 09/29/2020 09:37 AM, Michael Koch wrote:
Am 29.09.2020 um 14:58 schrieb Mark Filipak (ffmpeg):
On 09/29/2020 04:06 AM, Michael Koch wrote:
Am 29.09.2020 um 04:28 schrieb Mark Filipak (ffmpeg):
I just want to understand the frame structures that ffmpeg creates, and that ffmpeg uses in processing and filtering. Are Y, Cb, Cr separate buffers? That would be logical. Or are the Y, Cb, Cr values combined and organized similarly to macroblocks? I've found some code that supports that. Or are the Y, Cb, Cr values thrown together, pixel-by-pixel. That would be logical, too.
As far as I understood it, that depends on the pixel format. For example there are "packed" pixel formats rgb24, bgr24, argb, rgba, abgr, bgra,rgb48be, rgb48le, bgr48be, bgr48le. And there are "planar" pixel formats gbrp, bgrp16be, bgrp16le.
Hi Michael,
"Packed" and "planar", eh? What evidence do you have? ...Share the candy!
As far as I know, this is not described in the official documentation. You can find it for example here: https://video.stackexchange.com/questions/16374/ffmpeg-pixel-format-definiti...
Thanks for that. It saved me some time. ...So, what does "planar" mean? What does "packed" mean?
Now, I'm not talking about streams. I'm talking about after decoding. I'm talking about the buffers. I would think that a single, consistent format would be used. There is no single consistent format used internally. See Gyan's answer here: http://ffmpeg.org/pipermail/ffmpeg-user/2020-September/050031.html
And thanks for that. So, ffmpeg really is a Tower of Babel, eh? That does not seem wise to me. That seems like a great way to generate bugs in addition to confusion.
Now if that is not trolling out of pure ignorance I dunno what is. I'm really asking mysefl why are there still people feeding this poor troll.
Now, I imagine that converting to a lingua franca would blow up processing time, so it isn't done. However, if there are format-to-format regular expressions for conversions, may I suggest that those regular expressions be published? Also, if Y Cb & Cr are separate buffers, may I suggest that ffmpeg publish that?
In school, I learned that inputs and outputs should be fully characterized, not suggested, not implied, but fully characterized as structures. That takes time, and it takes review and perfection by informed people, but that time is worth the investment.
-- The U.S. political problem? Amateurs are doing the street fighting. The Princeps Senatus and the Tribunus Plebis need their own armies. _______________________________________________ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-user
To unsubscribe, visit link above, or email ffmpeg-user-request@ffmpeg.org with subject "unsubscribe".
On 9/29/2020 12:43 PM, Paul B Mahol wrote:
On Tue, Sep 29, 2020 at 10:26:20AM -0400, Mark Filipak (ffmpeg) wrote:
And thanks for that. So, ffmpeg really is a Tower of Babel, eh? That does not seem wise to me. That seems like a great way to generate bugs in addition to confusion.
Now if that is not trolling out of pure ignorance I dunno what is.
Actually, it sounds like it might be an accurate representation of the entire codebase. Cryptic code without explanations _is_ a good way to cause bugs, at least that's my experience (which goes back a fair ways). That said, as with many systems which evolve over time, the interfaces and understandings morph although the explanations often don't. And when there isn't a single entity (is there?) to direct the process and a single cohesive view of how things should go, chaos happens. One example was a while ago when someone suggested that a specific option was not consistent with other related ones- the great hew and cry was "WE CAN"T CHANGE THAT!!!" instead of "Hmm, it might make more sense to change this one for consistency's sake, yes, it'll break some older usages, but it'll be more logical in general."
I'm really asking mysefl why are there still people feeding this poor troll.
Because he's contributing to the overall knowledge? A lot of things are being described that I've found to be quite useful and _should_ be part of the overall documentation, be it in code comments or separate files. A note to Mark F.- If you refuse to try understanding even the basics of 'c', which shouldn't be that difficult to anyone who has ever programmed, you're holding yourself back. z!
On Tue, Sep 29, 2020 at 01:07:45PM -0700, Carl Zwanzig wrote:
On 9/29/2020 12:43 PM, Paul B Mahol wrote:
On Tue, Sep 29, 2020 at 10:26:20AM -0400, Mark Filipak (ffmpeg) wrote:
And thanks for that. So, ffmpeg really is a Tower of Babel, eh? That does not seem wise to me. That seems like a great way to generate bugs in addition to confusion.
Now if that is not trolling out of pure ignorance I dunno what is.
Actually, it sounds like it might be an accurate representation of the entire codebase. Cryptic code without explanations _is_ a good way to cause bugs, at least that's my experience (which goes back a fair ways).
It is cryptic only for ignorant folks. But ignornat folks should not be here at all, because they are, guess what, ignorant.
That said, as with many systems which evolve over time, the interfaces and understandings morph although the explanations often don't. And when there isn't a single entity (is there?) to direct the process and a single cohesive view of how things should go, chaos happens. One example was a while ago when someone suggested that a specific option was not consistent with other related ones- the great hew and cry was "WE CAN"T CHANGE THAT!!!" instead of "Hmm, it might make more sense to change this one for consistency's sake, yes, it'll break some older usages, but it'll be more logical in general."
I'm really asking mysefl why are there still people feeding this poor troll.
Because he's contributing to the overall knowledge? A lot of things are being described that I've found to be quite useful and _should_ be part of the overall documentation, be it in code comments or separate files.
A note to Mark F.- If you refuse to try understanding even the basics of 'c', which shouldn't be that difficult to anyone who has ever programmed, you're holding yourself back.
z!
_______________________________________________ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-user
To unsubscribe, visit link above, or email ffmpeg-user-request@ffmpeg.org with subject "unsubscribe".
ig·no·rant /ˈiɡnərənt/ adjective lacking knowledge, information, or awareness about a particular thing. On 9/29/2020 1:40 PM, Paul B Mahol wrote:
It is cryptic only for ignorant folks.
Gee, maybe that's why they're asking questions- to gain a "knowledge, information, or awareness about a particular thing". Some parts of ffmpeg are rather complex and it's irrational to expect everyone else to understand them. (And RTFS is not usually a good answer.)
But ignornat folks should not be here at all, because they are, guess what, ignorant.
Sounds like you're equating "user" with "ignorant" which is a fine way to insult quite a lot of people. Since this is the ffmpeg -users- list, not the -developers- list, it's not surprising that some people don't or can't read the code (which isn't a requirement to using it, last I checked) or don't grok the subtleties. Is that really such a problem? When someone does try to understand things, insulting them off does not increase anyone's understanding. Remember that not everyone has your knowledge of the code, and it's ridiculous to expect it. z!
participants (6)
-
Carl Zwanzig -
Dave Stevens -
Devin Heitmueller -
Mark Filipak (ffmpeg) -
Michael Koch -
Paul B Mahol