Mesa Turns To BLAKE3 For Faster Vulkan Shader Hashing

[ryao]

I was actually wondering why they didn't pick xx so thanks for that!
In that case do you think it would be bad to use it for an FS checksum?

ZFS added Blake3 as a checksum option. xxhash is not usable for on-disk checksums in ZFS since ZFS uses 256-bit checksums while xxhash is at most 128-bit. The Fletcher4 checksum algorithm in ZFS generates 256-bit hashes and has comparable (memory speed) performance.

[markg85]

@pabs
Thank you for that reply! That's much appreciated

I am running on Arch linux so i'm going with your assumption that i'm on the fast SHA path in both sha256sum and openssl. That seems to match my expectations too.

[AndyChow]

Stop saying AES-NI. It has nothing to do with SHA.
AES-NI is for optimized AES encryption.
https://en.m.wikipedia.org/wiki/Intel_SHA_extensions is for optimized SHA hashing.

Therefore what you claim is like comparing an apple (the fruit) with a pinetree. It just doesn't make sense, you're comparing 2 completely different things.

As for benchmarks. I don't have fancy graphs or numbers as this is from memory and from about a year ago. On the raspberry pi (I tried V4) blake3 was stupidly fast for calculating a checksum for a file. Like ~3x faster. You can try that yourself with the "b3sum" tool.

On desktop hardware the difference wasn't that extreme but still like 1.5x.

Which, in my book, means it's just a vastly superior hashing algorithm compared to sha (I used sha256sum). If you want benchmarks, run them yourself. You likely already have sha256sum on your PC, just install b3sum and go compare till your hearts content 😉

I couldn't benchmark at the time for reasons beyond my control. Someone else provided a bench showing on an X1 laptop sha256 was about twice as fast as blake2b. Since blake3 is 6 times faster than blake2b. I'd expect it to be 3 times faster than sha256. It's not a trivial question really. I would use xxhash over crc for non-suspect stuff, since it's faster, but the acceleration makes the reverse true in practice. That's why I'm often skeptical at "faster algo" claims.

Newer ARM chips tend to have those crypto hashing accelerations. RK3588 for example supposedly has such accelerators.

As for which branding Intel uses for it's instructions, sorry if I don't remember which group provides which instructions. AES-NI made sense. That's the header under which I would group the hashing stuff too.

[oconnor663]

Since blake3 is 6 times faster than blake2b

The performance comparison between BLAKE3 and BLAKE2b (really between any two hash functions) depends completely on which architecture you measure them on. There's no simple rule of thumb. In fact, if you're on a CPU with say 16 cores, there's going to be an order of magnitude difference just between single-threaded BLAKE3 and multithreaded BLAKE3!

When we publish benchmarks, we try to be very careful to specify exactly what machine they ran on, and whether BLAKE3 was using one thread or multiple threads. But it's understandable that as these things get filtered through articles and forum posts, some of the finer details get lost.

[geearf]

ZFS added Blake3 as a checksum option. xxhash is not usable for on-disk checksums in ZFS since ZFS uses 256-bit checksums while xxhash is at most 128-bit. The Fletcher4 checksum algorithm in ZFS generates 256-bit hashes and has comparable (memory speed) performance.

Hello,

comparable to which? Blake3?

Thank you!

[ryao]

Hello,

comparable to which? Blake3?

Thank you!

Comparable to xxhash. Blake3 does not run at memory speed.

[geearf]

Comparable to xxhash. Blake3 does not run at memory speed.

Got it, thank you!

[evil_core]

Comparable to xxhash. Blake3 does not run at memory speed.

But how does Blake3 compares to Fletcher4 in context of ZFS?

I'm creating new pool and wonder if should I choose Fletcher4 or Blake3?

As I understand Blake3 should allow me for background dedup, like BTRFS does.
All hardware supports SIMD2, so I understand that's requirement for h/w accel in ZFS.

Unfortunately I was unable to find any articles/benchmarks or even commments about Blake3 vs Fletcher4 performance in context of ZFS.

[ryao]

But how does Blake3 compares to Fletcher4 in context of ZFS?

I'm creating new pool and wonder if should I choose Fletcher4 or Blake3?

As I understand Blake3 should allow me for background dedup, like BTRFS does.
All hardware supports SIMD2, so I understand that's requirement for h/w accel in ZFS.

Unfortunately I was unable to find any articles/benchmarks or even commments about Blake3 vs Fletcher4 performance in context of ZFS.

It has about 10x less throughput than fletcher4. Use /proc/spl/kstat/zfs/fletcher_4_bench and /proc/spl/kstat/zfs/chksum_bench to compare. The former uses bytes per second. The latter uses binary megabytes per second.