FFV1 slicecrc: "hashxx" instead of "CRC" for speed?
Hi everyone :) This is merely a question of interest. Not a request, complaint or trolling of any kind :) I'm happy about any answer. I'm curious. Would it possibly have a significant impact on coding speed of FFV1's slicecrc option (Where a CRC is calculated for each frame slice), to use a faster algorithm instead (if one exists)? I'm wondering if, for example something like "xxHash" may warrant a try? (Haven't found CRC vs xxHash benchmarks yet, but I'm still looking) Anyhow, Thanks for any of your time :D
Peter B.:
Hi everyone :)
This is merely a question of interest. Not a request, complaint or trolling of any kind :) I'm happy about any answer. I'm curious.
Would it possibly have a significant impact on coding speed of FFV1's slicecrc option (Where a CRC is calculated for each frame slice), to use a faster algorithm instead (if one exists)? I'm wondering if, for example something like "xxHash" may warrant a try? (Haven't found CRC vs xxHash benchmarks yet, but I'm still looking)
The format of the checksum is determined by the FFV1 specification and not by the decoder/encoder implementation. - Andreas
On Tue, 25 Oct 2022, 21:32 Peter B., <pb@das-werkstatt.com> wrote:
Hi everyone :)
Would it possibly have a significant impact on coding speed of FFV1's slicecrc option (Where a CRC is calculated for each frame slice), to use a faster algorithm instead (if one exists)? I'm wondering if, for example something like "xxHash" may warrant a try? (Haven't found CRC vs xxHash benchmarks yet, but I'm still looking)
Anyhow, Thanks for any of your time :D
I haven't benchmarked the overhead of the CRC in FFv1 vs the encode or decode process. But FFmpeg's CRC could be optimised using multiple unrolled tables or SIMD or other approaches. Regards, Kieran Kunhya
On 10/26/22, Kieran Kunhya <kierank@obe.tv> wrote:
On Tue, 25 Oct 2022, 21:32 Peter B., <pb@das-werkstatt.com> wrote:
Hi everyone :)
Would it possibly have a significant impact on coding speed of FFV1's slicecrc option (Where a CRC is calculated for each frame slice), to use a faster algorithm instead (if one exists)? I'm wondering if, for example something like "xxHash" may warrant a try? (Haven't found CRC vs xxHash benchmarks yet, but I'm still looking)
Anyhow, Thanks for any of your time :D
I haven't benchmarked the overhead of the CRC in FFv1 vs the encode or decode process. But FFmpeg's CRC could be optimised using multiple unrolled tables or SIMD or other approaches.
It is already using multiple tables.
Regards, Kieran Kunhya _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
On Tue, Oct 25, 2022 at 10:31:48PM +0200, Peter B. wrote:
Hi everyone :)
This is merely a question of interest. Not a request, complaint or trolling of any kind :) I'm happy about any answer. I'm curious.
Would it possibly have a significant impact on coding speed of FFV1's slicecrc option (Where a CRC is calculated for each frame slice), to use a faster algorithm instead (if one exists)? I'm wondering if, for example something like "xxHash" may warrant a try? (Haven't found CRC vs xxHash benchmarks yet, but I'm still looking)
CRC is not a random pick for error detection, CRC has specfic properties like for example that whole classes of errors are guranteed to be detected a random hash will not gurantee that. for example a properly designed 8bit CRC will detect every single 1 byte change always. a 8 bit hash like taking 8 bits from SHA256 will detect only 255 out of 256 1 byte changes also CRC can be done in parallel if you have cores to spare, you can split a chunk in as many chunks as you want do the CRC for each and combine them with a bit of math and CRC is in the specification so using somehing else would require changing the spec thx [...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB He who knows, does not speak. He who speaks, does not know. -- Lao Tsu
On 10/26/22 15:13, Michael Niedermayer wrote:
CRC is not a random pick for error detection, CRC has specfic properties like for example that whole classes of errors are guranteed to be detected a random hash will not gurantee that. for example a properly designed 8bit CRC will detect every single 1 byte change always. a 8 bit hash like taking 8 bits from SHA256 will detect only 255 out of 256 1 byte changes
I'm a bit confused here, aren't there only 255 one-byte changes in the first place? The 256th is when the byte remains unchanged. That aside I don't believe xxhash is designed to be an error-correcting hash, but rather one that's used for fast database lookup. - Leo Izen (thebombzen)
Leo Izen (12022-11-03):
I'm a bit confused here, aren't there only 255 one-byte changes in the first place? The 256th is when the byte remains unchanged.
Most data is made of more than one byte. -- Nicolas George
On 11/3/22 09:19, Nicolas George wrote:
Leo Izen (12022-11-03):
I'm a bit confused here, aren't there only 255 one-byte changes in the first place? The 256th is when the byte remains unchanged.
Most data is made of more than one byte.
Ah, so it's on average 255 out of every 256 one-byte changes across the entire bitstream, got it. Thanks. - Leo Izen (thebombzen)
Thanks for all your replies! Now I can refer to this, when being asked "Why CRC and not ...?" :) Kind regards, Peter
participants (7)
-
Andreas Rheinhardt -
Kieran Kunhya -
Leo Izen -
Michael Niedermayer -
Nicolas George -
Paul B Mahol -
Peter B.