BLAKE3 Cryptographic Hash Implementation Preparing For v1.0 Release

[Palu Macil]

What systems? On a Zen2 SHA2-256 is over 2x faster on long messages and this increases to like 4x for very short messages,

It's slower per iteration but uses twice the block size. When hashing files, per byte matters. When calculating a merkle tree, iteration matters. I haven't seen anything like your claim for throughput, though I wish I had the time to benchmark at this moment. I will tip my hat to you and admit I cannot speak as someone whom has benchmarked this comparison myself, but I have read it multiple times. At some point I'll do some additional due diligence.

HMACs are not provably secure unlike other, proper constructions that were designed for the task instead of working around limitations of hash functions that are vulnerable to length extension.
This has nothing to do with collisions per se, but when you use another hack, namely truncation, to gain resistance to length extension you also weaken collision resistance.

HMAC is actually provably as secure as the underlying hash function and can use any hash function (SHA256, BLAKE3, etc). That property is one of the strengths of an HMAC and is exactly what it was designed for. You SHOULD use HMAC for an authenticated hash. There are valid alternatives, but if you don't know a lot about why to pick one, it is a solid choice. I'm a fan of ChaCha20-Poly1305 for some cases, but most of the time I reach for HMAC.

You don't have a length extension vulnerability with an HMAC even if the underlying hash has that vulnerability. You can read up on that if you like. From the wiki page on the vuln: "When a Merkle–Damgård based hash is misused as a message authentication code with construction H(secret ‖ message), and message and the length of secret is known, a length extension attack allows anyone to include extra information at the end of the message and produce a valid hash without knowing the secret. Since HMAC does not use this construction, HMAC hashes are not prone to length extension attacks."

In other words, it has NOTHING to do with collisions. It also does not affect any properly constructed authentication algo. If you don't try rolling your own there, you don't have to worry about it.

HMAC requires you to call the hash function twice, sequentially. A keyed hash function that is resistant to length extension does not require such a construction. BLAKE3 further optimized this in not even requiring you to hash the key because it uses they key as part of its IV so to speak.

From this, it's clear you do know what you're talking about, so I want to make sure I recognize that I see that and could be missing something. I merely have a masters in cybersecurity and work as a developer in the same field, but I admit that the math in cryptography is challenging to me, and there are concepts I need to go over several times because I start to understand the nuances. Furthermore, I am not specialized in cryptoanalysis, and my job is not to make any professional advisement on this area.

As for selling products I have to tell you that I'm not a salesman.

If it wasn't for this one point, I would be likely to be experimenting with BLAKE3 already. When you write security software for people, they require you to use certain algos. Sometimes they require that you call out to libraries that ship with the OS and/or a set of audited DLLs. You can't really sell to Fortune 500 companies without caring what is accepted by their requirements.

All that said, if I was working for a chat app like Signal, I would be making other choices. Would my choices be based upon throughput performance? No, not for a chat app. But there are plenty of other tools where BLAKE3 might be the best choice.

[Palu Macil]

I think this discussion is missing something important. Maybe Blake3 is also really useful in the non-security space?

If you're going to use it for storage or something, you wind up competing with far faster things like xxhash, farmer hash, murmurHash, etc.

EDIT: There is something to be said about the fact that sometimes you think something has no security implications and then later you realize it does.

[arQon]

For one thing, Intel (so is AMD for that matter) is perfectly inept enough to break its own CPU instructions without help from anyone else.

If only there was some way to test the outputs of those instructions for correctness. Maybe something like a function that turns an arbitrary-length stream of data into a consistent checksum of some kind, with enough result bits that the chances of two different streams producing the same result is infinitesimally small. I'm sure I've seen something like that mentioned a few times in the past.

Not sure what we'd call it though. "Secure checksum" is a bit of a mouthful. It's kind of like a hash though, so we could just call it that for a while until we come up with a better name...

[xnor]

Evidence of particular cases yes. Cryptography is not an all or nothing field.

So when something as simple as a random number generator has a backdoor everything based on it is just somewhat less secure? No, it is an all or nothing field, because that single backdoor allows you to break any encryption, any authentication, any signature algorithm that makes use of it.
And the NSA is by far not the only entity of which we have evidence for deliberately undermining information security. Not only to spy on foreign countries but also their own people.

Throwing the baby out with the bathwater is insane. Unless you have direct evidence of the NSA undermining a particular implementation it's a red herring argument.

Nah, it's the only reasonable position to take. You don't assume generally good intentions, especially from an organization that specializes in espionage, communication interception, decryption ... That would be quite gullible and naive.
Instead you should require evidence and mathematical proofs about the security, and for everything they had their hand in you should require even stronger evidence and more proofs and should be more skeptical, not less.

The NSA isn't a monolithic entity. Parts are dedicated to offensive operations, and parts are dedicated to defensive operations.

NSA doesn't publish the algorithms used for high security defensive operations.
The public algorithms are used for short-term storage (which can be securely erased) and interoperability.

[stormcrow]

NIST, I assume, isn't subverting security but the NSA is. Anyway, the point was that you don't need a tinfoil hat to have concerns when the NSA is mentioned with secure standards in the same sentence since we have evidence.


But BLAKE using AVX instructions is at the same speed or faster.
Also, what SHA in hardware? The broken SHA1 that shouldn't be used anymore or SHA256 which is susceptible to length extension attacks, which limits its use or requires more complex constructions (with multiple calls that reduce speed)?

Evidence of particular cases yes. Cryptography is not an all or nothing field. Throwing the baby out with the bathwater is insane. Unless you have direct evidence of the NSA undermining a particular implementation it's a red herring argument. For one thing, Intel (so is AMD for that matter) is perfectly inept enough to break its own CPU instructions without help from anyone else. For another, the most wide spread Linux kernel security system, seLinux, was invented by the NSA and has regularly withstood analysis. The NSA isn't a monolithic entity. Parts are dedicated to offensive operations, and parts are dedicated to defensive operations.

[xnor]

You use SHA512 because it's faster than SHA256 anyway (at least in 64-bit systems).

What systems? On a Zen2 SHA2-256 is over 2x faster on long messages and this increases to like 4x for very short messages,

A length extension attack is related to using a hash in a MAC. I don't believe it can cause collisions in the hashes. The problem is with authenticated MACs. The answer is to use a well tested, standard algorithm provided by your standard library like an HMAC. The HMAC is available in virtually all standard libraries and very difficult to get wrong. Ironically, if you use BLAKE instead for an authenticated MAC, you should still be using it within HMAC, which means your "extra work" point isn't true.

Any kind of authentication where these NIST-approved secure hash functions are used can be vulnerable to this problem. This includes APIs that use such hashes as authentication tags, web services, authenticated encryption ...
HMACs are not provably secure unlike other, proper constructions that were designed for the task instead of working around limitations of hash functions that are vulnerable to length extension.
This has nothing to do with collisions per se, but when you use another hack, namely truncation, to gain resistance to length extension you also weaken collision resistance.

BLAKE2 and 3 don't have this problem because they add a finalization step from which you cannot recover the hash function's state.

HMAC requires you to call the hash function twice, sequentially. A keyed hash function that is resistant to length extension does not require such a construction. BLAKE3 further optimized this in not even requiring you to hash the key because it uses they key as part of its IV so to speak.

BLAKE2 is similar in terms of security to SHA3. So if you can use both SHA512 and BLAKE in an HMAC, why do you pick SHA512? Well, you do so to be able to sell your product to anyone. Nobody in the US is buying software outside the mainstream best-vetted cryptography. In many industries they can't for regulatory or certification reasons.

Similar in security but much, much faster. As for selling products I have to tell you that I'm not a salesman.

[ddriver]

There is something important missing from the picture - actual cpu load. If this algorithms is just better at saturating the cpu pipeline, this will inevitably come at the expense of other running processes. It will be faster to hash, but it will drag other workloads lower, so that massive throughput is unlikely to translate to huge real world performance improvement, much like the vast difference between synthetic and real world performance for SATA vs PCIE4 ssd.

[microcode]

Sure it's a little faster

Dude, it's almost ten times faster.

[ChamPro]

I think this discussion is missing something important. Maybe Blake3 is also really useful in the non-security space? I can think of all kinds of use in the storage space. skeevy420 mentioned it earlier for ZFS. I can see this being incredibly useful for metadata analytics; applications that scan an entire file space, hashing every file, and keeping track of when they change in a NoSQL database. The speed improvement over MD5 or SHA1 is significant when you're looking at over a billion files.

[Davonious]

OK - talking about the NSA/NIST subverting the security industry may work on an internet forum, but I assume you wouldn't be bringing that up in meetings with your customers. You'll be saying "our product follows FIPS recommendations"

True, true. We recently dropped a vendor due to their use of non-FIPS certified algorithms. The product in and of itself was fine, but we'd formally raised the issue of non-FIPS algorithms before. After a year (or more) of "we're working on it" and no timeline for implementation, at license renewal we went with a competitor.

Sad thing is that the crypto features seemed secondary/tertiary (at best) to the core functionality of the product.