[FFmpeg-user] Lossless encoding to ffv1 changes frames md5

Paul B Mahol onemda at gmail.com
Tue Dec 17 15:53:46 EET 2019


On 12/17/19, Alberto Mattea <alberto at mattea.info> wrote:
> In data martedì 17 dicembre 2019 14:33:47 CET, Carl Eugen Hoyos ha scritto:
>> Am Di., 17. Dez. 2019 um 14:26 Uhr schrieb Carl Eugen Hoyos
>> <ceffmpeg at gmail.com>:
>> >
>> > Am Di., 17. Dez. 2019 um 14:18 Uhr schrieb Alberto Mattea
>> > <alberto at mattea.info>:
>> > >
>> > > Hi all, I'm trying to losslessly convert some videos from grassvalley
>> > > hqx to ffv1.
>> > > Unfortunately it seems the conversion is not really lossless, as the
>> > > frames md5 changes after the conversion.
>> > > Here's what I'm doing:
>> > >
>> > > - Calculate md5 of the original video frames
>> > >
>> > > $ ffmpeg -i 01.avi -f framemd5 orig.framemd5
>> > > ffmpeg version N-51066-ge52070e89c-static
>> > > https://johnvansickle.com/ffmpeg/  Copyright (c) 2000-2019 the FFmpeg
>> > > developers
>> > >   built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
>> > >   configuration: --enable-gpl --enable-version3 --enable-static
>> > > --disable-debug --disable-ffplay --disable-indev=sndio
>> > > --disable-outdev=sndio --cc=gcc-6 --enable-fontconfig --enable-frei0r
>> > > --enable-gnutls --enable-gmp --enable-libgme --enable-gray
>> > > --enable-libaom --enable-libfribidi --enable-libass --enable-libvmaf
>> > > --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb
>> > > --enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband
>> > > --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libvorbis
>> > > --enable-libopus --enable-libtheora --enable-libvidstab
>> > > --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp
>> > > --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libdav1d
>> > > --enable-libxvid --enable-libzvbi --enable-libzimg
>> > >   libavutil      56. 36.101 / 56. 36.101
>> > >   libavcodec     58. 65.100 / 58. 65.100
>> > >   libavformat    58. 35.101 / 58. 35.101
>> > >   libavdevice    58.  9.101 / 58.  9.101
>> > >   libavfilter     7. 68.100 /  7. 68.100
>> > >   libswscale      5.  6.100 /  5.  6.100
>> > >   libswresample   3.  6.100 /  3.  6.100
>> > >   libpostproc    55.  6.100 / 55.  6.100
>> > > Input #0, avi, from '01.avi':
>> > >   Duration: 00:15:04.40, start: 0.000000, bitrate: 202751 kb/s
>> > >     Stream #0:0: Video: hqx (CHQX / 0x58514843), yuv422p16le(10 bpc,
>> > > progressive), 1632x1200 [SAR 1:1 DAR 34:25], 202756 kb/s, 25 fps, 25
>> > > tbr, 25 tbn, 25 tbc
>> > > Stream mapping:
>> > >   Stream #0:0 -> #0:0 (hqx (native) -> rawvideo (native))
>> > > Press [q] to stop, [?] for help
>> > > Output #0, framemd5, to 'orig.framemd5':
>> > >   Metadata:
>> > >     encoder         : Lavf58.35.101
>> > >     Stream #0:0: Video: rawvideo (Y3[10][16] / 0x100A3359),
>> > > yuv422p16le(10 bpc), 1632x1200 [SAR 1:1 DAR 34:25], q=2-31, 1566720
>> > > kb/s, 25 fps, 25 tbn, 25 tbc
>> > >     Metadata:
>> > >       encoder         : Lavc58.65.100 rawvideo
>> > > frame=22610 fps= 50 q=-0.0 Lsize=    1767kB time=00:15:04.40 bitrate=
>> > > 16.0kbits/s speed=1.98x
>> > > video:172966500kB audio:0kB subtitle:0kB other streams:0kB global
>> > > headers:0kB muxing overhead: unknown
>> > >
>> > > - Convert
>> > >
>> > > $ ffmpeg -i 01.avi -c:v ffv1 01ffv1.mkv
>> > > ffmpeg version N-51066-ge52070e89c-static
>> > > https://johnvansickle.com/ffmpeg/  Copyright (c) 2000-2019 the FFmpeg
>> > > developers
>> > >   built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
>> > >   configuration: --enable-gpl --enable-version3 --enable-static
>> > > --disable-debug --disable-ffplay --disable-indev=sndio
>> > > --disable-outdev=sndio --cc=gcc-6 --enable-fontconfig --enable-frei0r
>> > > --enable-gnutls --enable-gmp --enable-libgme --enable-gray
>> > > --enable-libaom --enable-libfribidi --enable-libass --enable-libvmaf
>> > > --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb
>> > > --enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband
>> > > --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libvorbis
>> > > --enable-libopus --enable-libtheora --enable-libvidstab
>> > > --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp
>> > > --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libdav1d
>> > > --enable-libxvid --enable-libzvbi --enable-libzimg
>> > >   libavutil      56. 36.101 / 56. 36.101
>> > >   libavcodec     58. 65.100 / 58. 65.100
>> > >   libavformat    58. 35.101 / 58. 35.101
>> > >   libavdevice    58.  9.101 / 58.  9.101
>> > >   libavfilter     7. 68.100 /  7. 68.100
>> > >   libswscale      5.  6.100 /  5.  6.100
>> > >   libswresample   3.  6.100 /  3.  6.100
>> > >   libpostproc    55.  6.100 / 55.  6.100
>> > > Input #0, avi, from '01.avi':
>> > >   Duration: 00:15:04.40, start: 0.000000, bitrate: 202751 kb/s
>> > >     Stream #0:0: Video: hqx (CHQX / 0x58514843), yuv422p16le(10 bpc,
>> > > progressive), 1632x1200 [SAR 1:1 DAR 34:25], 202756 kb/s, 25 fps, 25
>> > > tbr, 25 tbn, 25 tbc
>> > > Stream mapping:
>> > >   Stream #0:0 -> #0:0 (hqx (native) -> ffv1 (native))
>> > > Press [q] to stop, [?] for help
>> > > [ffv1 @ 0x6e1ce80] bits_per_raw_sample > 8, forcing range coder
>> > > Output #0, matroska, to '01ffv1.mkv':
>> > >   Metadata:
>> > >     encoder         : Lavf58.35.101
>> > >     Stream #0:0: Video: ffv1 (FFV1 / 0x31564646), yuv422p16le(10 bpc),
>> > > 1632x1200 [SAR 1:1 DAR 34:25], q=2-31, 200 kb/s, 25 fps, 1k tbn, 25
>> > > tbc
>> > >     Metadata:
>> > >       encoder         : Lavc58.65.100 ffv1
>> > > frame=22610 fps= 17 q=-0.0 Lsize=35631195kB time=00:15:04.36
>> > > bitrate=322759.1kbits/s speed=0.691x
>> > > video:35630872kB audio:0kB subtitle:0kB other streams:0kB global
>> > > headers:0kB muxing overhead: 0.000908%
>> > >
>> > > - Calculate md5 of the converted frames (I've stopped it after a few
>> > > hundred frames, just to check the beginning)
>> > >
>> > > $ ffmpeg -i 01ffv1.mkv -f framemd5 converted.framemd5
>> > > ffmpeg version N-51066-ge52070e89c-static
>> > > https://johnvansickle.com/ffmpeg/  Copyright (c) 2000-2019 the FFmpeg
>> > > developers
>> > >   built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
>> > >   configuration: --enable-gpl --enable-version3 --enable-static
>> > > --disable-debug --disable-ffplay --disable-indev=sndio
>> > > --disable-outdev=sndio --cc=gcc-6 --enable-fontconfig --enable-frei0r
>> > > --enable-gnutls --enable-gmp --enable-libgme --enable-gray
>> > > --enable-libaom --enable-libfribidi --enable-libass --enable-libvmaf
>> > > --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb
>> > > --enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband
>> > > --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libvorbis
>> > > --enable-libopus --enable-libtheora --enable-libvidstab
>> > > --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp
>> > > --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libdav1d
>> > > --enable-libxvid --enable-libzvbi --enable-libzimg
>> > >   libavutil      56. 36.101 / 56. 36.101
>> > >   libavcodec     58. 65.100 / 58. 65.100
>> > >   libavformat    58. 35.101 / 58. 35.101
>> > >   libavdevice    58.  9.101 / 58.  9.101
>> > >   libavfilter     7. 68.100 /  7. 68.100
>> > >   libswscale      5.  6.100 /  5.  6.100
>> > >   libswresample   3.  6.100 /  3.  6.100
>> > >   libpostproc    55.  6.100 / 55.  6.100
>> > > Input #0, matroska,webm, from '01ffv1.mkv':
>> > >   Metadata:
>> > >     ENCODER         : Lavf58.35.101
>> > >   Duration: 00:15:04.40, start: 0.000000, bitrate: 322745 kb/s
>> > >     Stream #0:0: Video: ffv1 (FFV1 / 0x31564646), yuv422p10le,
>> > > 1632x1200, SAR 1:1 DAR 34:25, 25 fps, 25 tbr, 1k tbn, 1k tbc (default)
>> > >     Metadata:
>> > >       ENCODER         : Lavc58.65.100 ffv1
>> > >       DURATION        : 00:15:04.400000000
>> > > Stream mapping:
>> > >   Stream #0:0 -> #0:0 (ffv1 (native) -> rawvideo (native))
>> > > Press [q] to stop, [?] for help
>> > > Output #0, framemd5, to 'converted.framemd5':
>> > >   Metadata:
>> > >     encoder         : Lavf58.35.101
>> > >     Stream #0:0: Video: rawvideo (Y3[10][10] / 0xA0A3359),
>> > > yuv422p10le, 1632x1200 [SAR 1:1 DAR 34:25], q=2-31, 979200 kb/s, 25
>> > > fps, 25 tbn, 25 tbc (default)
>> > >     Metadata:
>> > >       DURATION        : 00:15:04.400000000
>> > >       encoder         : Lavc58.65.100 rawvideo
>> > > frame=  406 fps= 31 q=-0.0 Lsize=      32kB time=00:00:16.24 bitrate=
>> > > 16.1kbits/s speed=1.23x
>> > > video:3105900kB audio:0kB subtitle:0kB other streams:0kB global
>> > > headers:0kB muxing overhead: unknown
>> > >
>> > > - Compare hashes
>> > >
>> > > orig.framemd5:
>> > > #format: frame checksums
>> > > #version: 2
>> > > #hash: MD5
>> > > #software: Lavf58.35.101
>> > > #tb 0: 1/25
>> > > #media_type 0: video
>> > > #codec_id 0: rawvideo
>> > > #dimensions 0: 1632x1200
>> > > #sar 0: 1/1
>> > > #stream#, dts,        pts, duration,     size, hash
>> > > 0,          0,          0,        1,  7833600,
>> > > 39d64300c8b749b0776423a80746d8dc
>> > > 0,          1,          1,        1,  7833600,
>> > > 12e7317d49727740187d2e23439d87d8
>> > > 0,          2,          2,        1,  7833600,
>> > > 12e7317d49727740187d2e23439d87d8
>> > > 0,          3,          3,        1,  7833600,
>> > > 0b3b0136b008ec9d67e900595be0c923
>> > > 0,          4,          4,        1,  7833600,
>> > > 97cd9b7689d1f49d1e1988a3613ffa72
>> > > 0,          5,          5,        1,  7833600,
>> > > 97cd9b7689d1f49d1e1988a3613ffa72
>> > > 0,          6,          6,        1,  7833600,
>> > > 1632e4b3ba7dc28e9dde698f3b7ec9e7
>> > > 0,          7,          7,        1,  7833600,
>> > > 796bda137d768f4ad31eb7166ea16447
>> > > 0,          8,          8,        1,  7833600,
>> > > 796bda137d768f4ad31eb7166ea16447
>> > > 0,          9,          9,        1,  7833600,
>> > > 612c13b4824bd427faa51efa7a09660d
>> > > ...
>> > >
>> > > converted.framemd5:
>> > > #format: frame checksums
>> > > #version: 2
>> > > #hash: MD5
>> > > #software: Lavf58.35.101
>> > > #tb 0: 1/25
>> > > #media_type 0: video
>> > > #codec_id 0: rawvideo
>> > > #dimensions 0: 1632x1200
>> > > #sar 0: 1/1
>> > > #stream#, dts,        pts, duration,     size, hash
>> > > 0,          0,          0,        1,  7833600,
>> > > 3ed693fc02ba36cca92a2153afa960ac
>> > > 0,          1,          1,        1,  7833600,
>> > > 23d7482899eb8c854b26444dcdf38db4
>> > > 0,          2,          2,        1,  7833600,
>> > > 23d7482899eb8c854b26444dcdf38db4
>> > > 0,          3,          3,        1,  7833600,
>> > > 1fc392823ebc14001ee0e44c68d710e6
>> > > 0,          4,          4,        1,  7833600,
>> > > d63335f1c34132680e571549da51eac7
>> > > 0,          5,          5,        1,  7833600,
>> > > d63335f1c34132680e571549da51eac7
>> > > 0,          6,          6,        1,  7833600,
>> > > 09304bca4578458c353ecc119de0bd87
>> > > 0,          7,          7,        1,  7833600,
>> > > 0ec1e49008d1f70385dc4a1636559a30
>> > > 0,          8,          8,        1,  7833600,
>> > > 0ec1e49008d1f70385dc4a1636559a30
>> > > 0,          9,          9,        1,  7833600,
>> > > 78b22236d03517e5ceea3bd390ac2d0b
>> > > ...
>> > >
>> > > Any idea of why this is happening?
>> >
>> > The input has 10 bit precision, therefore only 10 bit are encoded.
>>
>> The more difficult question is how big hqx precision really is:
>> The original commit specified 10 bit, but a comment in the code (same
>> commit)
>> indicates 12 bit.
>>
>> Carl Eugen
>
> Hey, thanks for the instant answer.
> So if I understand correctly no information is being lost, it's just that
> the 10 bits are represented over <10 bits plus 6 bits of padding> instead of
> over <16-bits where the least significant 6 bits always happen to be zero>.
> Is it correct?
> I'm assuming there is padding since if I extract a raw yuv frame from the
> hqx and the ffv1 they are both the size you would expect if they were
> 16-bit.

Thing is decoder's output should be changed so it will be 12/10 bit
pixel format.
Instead of using 16bit format. Assuming 4 upper bits are not having useful info.

>
> Alberto
>
>
>
>
> _______________________________________________
> ffmpeg-user mailing list
> ffmpeg-user at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> ffmpeg-user-request at ffmpeg.org with subject "unsubscribe".


More information about the ffmpeg-user mailing list