[FFmpeg-trac] #9132(ffmpeg:open): Wrong pixel format/output when converting video to yuv444p*

[FFmpeg]

#9132: Wrong pixel format/output when converting video to yuv444p*
------------------------------------+----------------------------------
             Reporter:  viley       |                    Owner:  (none)
                 Type:  defect      |                   Status:  open
             Priority:  important   |                Component:  ffmpeg
              Version:  git-master  |               Resolution:
             Keywords:  regression  |               Blocked By:
             Blocking:              |  Reproduced by developer:  1
Analyzed by developer:  0           |
------------------------------------+----------------------------------
Comment (by jeeb):

 1) Nobody stops you testing these patches before they land since it's not
 like they're hidden somewhere (and also with the hindsight of going back
 to this point of the message after writing the rest of it, you could have
 seen this issue that you are now complaining about was there even before).

 2) Can you please be less rambly and all over the place? It's really hard
 to understand what and where since the steps and expected and received
 results are all over the place. For the record, vf_scale '''never''' set
 the matrix coefficients, so unless out->format as a pix_fmt didn't for
 some reason had a mis-matching RGB flag I do not know how the flying
 f-word I could have made this any worse than it was. I just stopped a
 completely incorrect configuration from going forwards.

 Thus, step by step trying to decrypt your message:

 {{{
 ffmpeg -v verbose -i ffmpeg_samples\ticket_9374\rec.mkv -vf
 scale=out_color_matrix=bt709:flags=accurate_rnd,format=yuv420p -c:v
 libx264 -crf 10 -color_primaries 1 -color_trc 1 TESTCASE-yuvnew.mp4
 }}}
 leads to a result that I'd expect, which seems completely fine on
 playback.
 {{{
 Stream #0:0[0x1](und): Video: h264 (High), 1 reference frame (avc1 /
 0x31637661), yuv420p(tv, unknown/bt709/bt709, left), 1920x1200, 952 kb/s,
 25 fps, 25 tbr, 12800 tbn (default)
 }}}

 The matrix coefficients are not set to BT.709 or anything else specific
 because the filter '''never did that'''. One needs to go look into the
 swscale code to actually '''verify and add code''' to set the value
 according to swscale's behavior. And I've touched swscale enough during
 the past few months to not want to touch that thing for a while, so excuse
 me. The zscale filter has options for all these three values which is why
 if they're configured you then get them passed through and received from
 zimg's configuration. So no, I've not made this specific conversion any
 worse. At all.

 Alright, so let's go to the next step where you forcibly set the matrix
 coefficients through an AVOption for the encoder.
 {{{
 ffmpeg -v verbose -i ffmpeg_samples\ticket_9374\rec.mkv -vf
 scale=out_color_matrix=bt709:flags=accurate_rnd,format=yuv420p -c:v
 libx264 -crf 10 -color_primaries 1 -color_trc 1 -colorspace 1 TESTCASE-
 yuvnewer.mp4
 }}}
 That results in:
 {{{
 Stream #0:0[0x1](und): Video: h264 (High), 1 reference frame (avc1 /
 0x31637661), yuv420p(tv, bt709, left), 1920x1200, 952 kb/s, 25 fps, 25
 tbr, 12800 tbn (default)
 }}}
 Apparently when all values match the separate logging is skipped. So that
 says that all the metadata you've set is correctly passed on. Still
 everything good.

 Finally, the example where you convert YCbCr to RGB, and then forcibly set
 the output encoder's primaries and transfer to BT.709.
 {{{
 ffmpeg -v verbose -i TESTCASE-yuvnewer.mp4 -vf scale=flags=accurate_rnd
 -c:v libx264rgb -crf 10 -color_primaries 1 -color_trc 1 rgbsupper.mp4
 }}}
 Encoder configuration looks good to me (pix_fmt is RGB):
 {{{
  Stream #0:0(und): Video: h264 (avc1 / 0x31637661), rgb24(pc, bt709,
 progressive), 1920x1200, q=2-31, 25 fps, 12800 tbn (default)
 }}}
 vf_scale's internal conversion also seems Just Fine
 {{{
 [Parsed_scale_0 @ 000002009cc31700] w:1920 h:1200 fmt:yuv420p sar:0/1 ->
 w:1920 h:1200 fmt:rgb24 sar:0/1 flags:0x40000
 }}}

 But yes, it seems like for some reason when the encoded output is marked
 as full range4:4:4 YCbCr.
 {{{
 Stream #0:0[0x1](und): Video: h264 (High 4:4:4 Predictive), 1 reference
 frame (avc1 / 0x31637661), yuvj444p(pc, bt709, left), 1920x1200, 2230
 kb/s, 25 fps, 25 tbr, 12800 tbn (default)
 }}}

 I am not sure, though, why you think this is the fault of my change and
 why you have started pointing the finger at me.

 As I noted, vf_scale never set the matrix coefficients to anything. It
 just copied whatever metadata the input AVFrame might have had, and passed
 it on. I have now stopped that in a single case which is clearly wrong to
 fix a single problematic case.

 And if you don't trust me, just revert that commit I added and see what
 you get out of that last YCbCr->RGB conversion of yours. I just tested it
 here locally and my changes have nothing to do with it. You have
 successfully hit a separate bug somewhere along the range, most likely
 another case of where vf_scale should not be passing the input AVFrame's
 metadata out as-is. These bugs have been there since at least
 7e350379f87e7f74420b4813170fe808e2313911 , and the only reason why things
 hit my commit from late 2020 is because I for the first time in the
 history of ffmpeg.c actually plugged the input pipe into the output pipe,
 so people see all these bugs that always were there but were not visible
 in ffmpeg.c (But would have confused the heck out of other libavfilter
 users).
-- 
Ticket URL: <https://trac.ffmpeg.org/ticket/9132#comment:23>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker