Verifying lossless rewrapping/transcoding in one step?
Hi everyone :) I was wondering if maybe there's a quicker way to verify lossless transcoding or container rewrapping than these 3 steps: 1. generate content hash during transcoding. 2. generate content hash of output file. 3. diff source-hash with output-hash (using whatever diff-tool). I'm doing this in bash scripts, but I'm wondering if it would be possible to tell ffmpeg something like this: "Please generate the output and compare source and target's content hash and only return exit code 0 if they match" Is that already implemented, or would it be possible to do that? Thank you very much in advance, Peter B.
On Fri, 26 Jul 2019 at 21:02, Peter B. <pb@das-werkstatt.com> wrote:
Hi everyone :)
I was wondering if maybe there's a quicker way to verify lossless transcoding or container rewrapping than these 3 steps:
1. generate content hash during transcoding. 2. generate content hash of output file. 3. diff source-hash with output-hash (using whatever diff-tool).
I'm doing this in bash scripts, but I'm wondering if it would be possible to tell ffmpeg something like this:
"Please generate the output and compare source and target's content hash and only return exit code 0 if they match"
Is that already implemented, or would it be possible to do that?
Thank you very much in advance, Peter B.
Hey there, You could use a muxer such as framemd5: https://ffmpeg.org/ffmpeg-formats.html#framemd5-1 See the link above for usage. -- まさひろ はやし
On 26/07/2019 22:54, Dennis Mungai wrote:
You could use a muxer such as framemd5: https://ffmpeg.org/ffmpeg-formats.html#framemd5-1
Thanks for your reply, but I've been using framemd5 for years. I meant, if there's a way that I don't have to code a script around the ffmpeg call to do the actual validation? Kind regards, Peter B.
Hi everyone :) I assume no answer means there's no option to do that. Would anyone be willing to implement it? And how much could that approximately cost? Thank you very much in advance, Peter B.
On Fri, Aug 02, 2019 at 14:51:36 +0200, Peter B. wrote:
I assume no answer means there's no option to do that.
No, there isn't. I believe in using the right tools for the right tasks. If it can be glued together with a script, why worry about modifying an existing program to do the same? That said, regarding your request, I believe the showstopper is in the concept: To re-calculate the checksum of the output, it needs to be decoded again. That's not something ffmpeg can do within its binary, it can't use the output and chain it as its own input. You *can* do that by scripting (obviously), and you could construct some program with the libav* libraries. But at the end of the day, you want the features of ffmpeg (the command line tool), which you can't just reeinvent. So basically, you need to prop something into the command line tool which can chain encodes and decodes. And that's no small task, I believe, unlike "yet another special muxer/demuxer". That's my take on it: Way too much effort for the gain. Cheers, Moritz
Dear Moritz, Thank you for your explanation! On 03/08/2019 14:07, Moritz Barsnick wrote:
I believe in using the right tools for the right tasks. If it can be glued together with a script, why worry about modifying an existing program to do the same?
I completely agree, but... ;) One reason (for me) is that I haven't found a generic way to script this check for inhomogenous source material. I don't know how many A/V streams, in which configuration, etc. I know I can ffprobe, then parse, etc - but: The 2nd reason (also for me) is, that I've now written the same functionality in different scripts for slightly different use cases. And the 3rd reason (for me...) is: It's actually not-so-trivial to explain this validation-check approach when I do FFmpeg workshops (for commandline newbies, and people who don't know what a "diff" even is) :D
That said, regarding your request, I believe the showstopper is in the concept: To re-calculate the checksum of the output, it needs to be decoded again. That's not something ffmpeg can do within its binary, it can't use the output and chain it as its own input.
That's a very interesting insight! Thank you very much. I was sure there's a reason, but now I finally know. Thank you!
That's my take on it: Way too much effort for the gain.
Hm... I'm thinking about how (under the given circumstances you've mentioned), this could still be made a bit "easier"... Would it be possible to, for example, modify the hash muxer to output a hash for each stream - in a way that one could easily parse that textfile and know which hash belongs to which stream? For example, instead of "CRC32=...." let's say: "v:0:CRC32=... v:1:CRC32=... a:0:CRC32... a:1:CRC32... a:2... " Something like that. Just an idea... :D Thank you very much again! Peter
On Mon, Aug 5, 2019 at 4:16 PM Peter B. <pb@das-werkstatt.com> wrote:
Dear Moritz,
Thank you for your explanation!
On 03/08/2019 14:07, Moritz Barsnick wrote:
I believe in using the right tools for the right tasks. If it can be glued together with a script, why worry about modifying an existing program to do the same?
I completely agree, but... ;)
One reason (for me) is that I haven't found a generic way to script this check for inhomogenous source material. I don't know how many A/V streams, in which configuration, etc. I know I can ffprobe, then parse, etc - but:
The 2nd reason (also for me) is, that I've now written the same functionality in different scripts for slightly different use cases.
And the 3rd reason (for me...) is: It's actually not-so-trivial to explain this validation-check approach when I do FFmpeg workshops (for commandline newbies, and people who don't know what a "diff" even is)
:D
That said, regarding your request, I believe the showstopper is in the concept: To re-calculate the checksum of the output, it needs to be decoded again. That's not something ffmpeg can do within its binary, it can't use the output and chain it as its own input.
That's a very interesting insight! Thank you very much. I was sure there's a reason, but now I finally know. Thank you!
That's my take on it: Way too much effort for the gain.
Hm... I'm thinking about how (under the given circumstances you've mentioned), this could still be made a bit "easier"...
Would it be possible to, for example, modify the hash muxer to output a hash for each stream - in a way that one could easily parse that textfile and know which hash belongs to which stream?
For example, instead of "CRC32=...." let's say: "v:0:CRC32=... v:1:CRC32=... a:0:CRC32... a:1:CRC32... a:2... "
Hash muxer does not work like that. Why would you need hash muxer for multiple streams? You can use several maps to produce hash of more stream at once, each output directed to separate file. ffmpeg.exe -i fate-suite/mxf/C0023S01.mxf -vn -map 0:1 -f framecrc first_audio_stream.framecrc -map 0:2 -f framecrc second_audio_stream.framecrc
Hi Paul! On 05/08/2019 16:25, Paul B Mahol wrote:
On Mon, Aug 5, 2019 at 4:16 PM Peter B. <pb@das-werkstatt.com> wrote:
[...] For example, instead of "CRC32=...." let's say: "v:0:CRC32=... v:1:CRC32=... a:0:CRC32... a:1:CRC32... a:2... "
Hash muxer does not work like that.
Oh :(
Why would you need hash muxer for multiple streams?
Because I want to verify that every stream was transferred losslessly from source to target. For both: audio and video. (and actually also non-AV tracks, any time-based metadata, etc)
You can use several maps to produce hash of more stream at once, each output directed to separate file.
ffmpeg.exe -i fate-suite/mxf/C0023S01.mxf -vn -map 0:1 -f framecrc first_audio_stream.framecrc -map 0:2 -f framecrc second_audio_stream.framecrc
Thanks! However, that's what I'm already doing (-f hash), but I'm writing for fully-automated transcoding scenarios, so I need to parse the source file to know how many streams (of which type), and then have another part that puts together the right commandline recipe, and then write code that generates names for each stream-hash output file, and then, and then I need to keep track of all these files, and then ... And I believe I'm neither the first nor the last person to write code for automating this. So I thought: wouldn't it be great to find a nicer way of validating "output-fidelity"? :) Any ideas? What I don't understand: If the hash muxer output can generate a single hash for the sums of all streams, I presume it has access to all streams. Since it can also output individual hashes to individual files, why isn't it possible to do the same, but store the hashcodes in variables (associated with their source stream), and then write that information in one single file? I like puzzles :) Thank you very much in advance! Peter B.
On Mon, Aug 5, 2019 at 11:13 PM Peter B. <pb@das-werkstatt.com> wrote:
Hi Paul!
On 05/08/2019 16:25, Paul B Mahol wrote:
On Mon, Aug 5, 2019 at 4:16 PM Peter B. <pb@das-werkstatt.com> wrote:
[...] For example, instead of "CRC32=...." let's say: "v:0:CRC32=... v:1:CRC32=... a:0:CRC32... a:1:CRC32... a:2... "
Hash muxer does not work like that.
Oh :(
Why would you need hash muxer for multiple streams?
Because I want to verify that every stream was transferred losslessly from source to target. For both: audio and video. (and actually also non-AV tracks, any time-based metadata, etc)
You can use several maps to produce hash of more stream at once, each output directed to separate file.
ffmpeg.exe -i fate-suite/mxf/C0023S01.mxf -vn -map 0:1 -f framecrc first_audio_stream.framecrc -map 0:2 -f framecrc second_audio_stream.framecrc
Thanks!
However, that's what I'm already doing (-f hash), but I'm writing for fully-automated transcoding scenarios, so I need to parse the source file to know how many streams (of which type), and then have another part that puts together the right commandline recipe, and then write code that generates names for each stream-hash output file, and then, and then I need to keep track of all these files, and then ...
And I believe I'm neither the first nor the last person to write code for automating this. So I thought: wouldn't it be great to find a nicer way of validating "output-fidelity"? :)
Any ideas?
What I don't understand: If the hash muxer output can generate a single hash for the sums of all streams, I presume it has access to all streams. Since it can also output individual hashes to individual files, why isn't it possible to do the same, but store the hashcodes in variables (associated with their source stream), and then write that information in one single file?
I like puzzles :) Thank you very much in advance!
If I understand correctly, basically need hash of each stream separately (for framehash this already works). That is quite possible to be implemented. Feel free to open feature request on FFmpeg bug tracker.
On Tue, Aug 06, 2019 at 10:07:40 +0200, Paul B Mahol wrote:
If I understand correctly, basically need hash of each stream separately (for framehash this already works).
That is quite possible to be implemented. Feel free to open feature request on FFmpeg bug tracker.
Just to show that it works, I have hacked up a quick proof of concept. (Hacked up means: Possibly not clean, not enough reuse of code, ...) From this for a video stream: ./ffmpeg_g -f lavfi -i testsrc -map 0 -t 1 -f hash - [...] SHA256=c747429bf5d240958db9545b184e2b226f826e67495ad5d46245794bc1b84078 and this for an audio stream: ./ffmpeg_g -f lavfi -i anoisesrc=seed=0 -map 0 -t 1 -f hash - [...] SHA256=4a3aa64bd7ef86ff2afb43c9346076792497cdcfcd741f10a1bea2e016294417 I can now produce this (note the new muxer): ./ffmpeg_g -f lavfi -i testsrc -f lavfi -i anoisesrc=seed=0 -map 0 -map 1 -t 1 -f streamhash - SHA256=c747429bf5d240958db9545b184e2b226f826e67495ad5d46245794bc1b84078 SHA256=4a3aa64bd7ef86ff2afb43c9346076792497cdcfcd741f10a1bea2e016294417 I just couldn't quickly figure out how to prefix the stream specifiers (e.g. 0:0, 0:1). I'm not hapy with submitting a hack as a patch to ffmpeg-devel though. ;-) But I can provide the patch here or somewhere else. Cheers, Moritz
On Tue, Aug 6, 2019 at 12:14 PM Moritz Barsnick <barsnick@gmx.net> wrote:
On Tue, Aug 06, 2019 at 10:07:40 +0200, Paul B Mahol wrote:
If I understand correctly, basically need hash of each stream separately (for framehash this already works).
That is quite possible to be implemented. Feel free to open feature request on FFmpeg bug tracker.
Just to show that it works, I have hacked up a quick proof of concept. (Hacked up means: Possibly not clean, not enough reuse of code, ...)
From this for a video stream: ./ffmpeg_g -f lavfi -i testsrc -map 0 -t 1 -f hash - [...] SHA256=c747429bf5d240958db9545b184e2b226f826e67495ad5d46245794bc1b84078
and this for an audio stream: ./ffmpeg_g -f lavfi -i anoisesrc=seed=0 -map 0 -t 1 -f hash - [...] SHA256=4a3aa64bd7ef86ff2afb43c9346076792497cdcfcd741f10a1bea2e016294417
I can now produce this (note the new muxer): ./ffmpeg_g -f lavfi -i testsrc -f lavfi -i anoisesrc=seed=0 -map 0 -map 1 -t 1 -f streamhash - SHA256=c747429bf5d240958db9545b184e2b226f826e67495ad5d46245794bc1b84078 SHA256=4a3aa64bd7ef86ff2afb43c9346076792497cdcfcd741f10a1bea2e016294417
I just couldn't quickly figure out how to prefix the stream specifiers (e.g. 0:0, 0:1).
Look how framehash does it?
I'm not hapy with submitting a hack as a patch to ffmpeg-devel though. ;-) But I can provide the patch here or somewhere else.
On Tue, Aug 06, 2019 at 12:21:58 +0200, Paul B Mahol wrote:
I just couldn't quickly figure out how to prefix the stream specifiers (e.g. 0:0, 0:1). Look how framehash does it?
Yeah, too obvious. I was looking for "0:1", but the first digit is meaningless, and the second one is just the stream index (which I use to iterate anyway). I'll try to rework it to give the "hash" muxer a "-per_stream" boolean option, then I can (and must) reuse all the code. Cheers, Moritz
On 06/08/2019 13:31, Moritz Barsnick wrote:
Yeah, too obvious. I was looking for "0:1", but the first digit is meaningless, and the second one is just the stream index (which I use to iterate anyway). I'll try to rework it to give the "hash" muxer a "-per_stream" boolean option, then I can (and must) reuse all the code.
Wooow! That'd be awesome! Thank you soooo much in advance, Peter
On Wed, Aug 07, 2019 at 15:10:06 +0200, Peter B. wrote:
I'll try to rework it to give the "hash" muxer a "-per_stream" boolean option, then I can (and must) reuse all the code.
Wooow! That'd be awesome!
I sent a non-hackish implementation to Peter privately. I'll clean it up and post it to ffmpeg-devel eventually, at the risk of its usefulness or quality being doubted. Or improved. Cheers, Moritz
On Wed, Aug 7, 2019 at 4:26 PM Moritz Barsnick <barsnick@gmx.net> wrote:
On Wed, Aug 07, 2019 at 15:10:06 +0200, Peter B. wrote:
I'll try to rework it to give the "hash" muxer a "-per_stream" boolean option, then I can (and must) reuse all the code.
Wooow! That'd be awesome!
I sent a non-hackish implementation to Peter privately. I'll clean it up and post it to ffmpeg-devel eventually, at the risk of its usefulness or quality being doubted. Or improved.
If you do not do that soon, I will.
Cheers, Moritz _______________________________________________ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-user
To unsubscribe, visit link above, or email ffmpeg-user-request@ffmpeg.org with subject "unsubscribe".
participants (4)
-
Dennis Mungai -
Moritz Barsnick -
Paul B Mahol -
Peter B.