Unity Continues Crunching More Out Of Crunch Texture Compression

[schlemil]

A dual GPL+commercial license setup would have allowed him to prove his point (the reason he chose an opensource license in the first place), it would still be free for FOSS users (that are a very poor source of income anyway, and are more likely to contribute back code instead), and it would still be monetizable for any commercial project that wanted to use it in proprietary code (because GPL is the worst license to try to integrate in proprietary code EVER).

Not really. Nobody would be using his stuff and nobody would know about it. He tried to market it and didn't have any success. His code wasn't ready for prime time and needed a lot of work. The only reason it went somewhere is that it was free under a commercially usable license.

[richgel999]

So Rich Geldreich is going to sit in his own sad proprietary bubble trying to convince people to buy his proprietary solutions for offering some mildly better compression, while the community takes his previous open-source project and improves it to make it even better *and* open-source *and* free ... *and* ... actually used by people. I see that as a win. It is a good thing for everyone, except maybe Rich Geldreich (from his POV at least).

I used to have a pretty good opinion of him back when he was working at Valve and I read about his work. With this whole recent fiasco, I have lost a lot of respect for him. Not that whether he has a random internet dude's respect matters to him or anything, he doesn't even know who I am ... but anyway ... I wish him all the best.

I don't know why I spend the time replying to posters hiding behind handles here, but Phoronix has been a good supporter of my work so I check in once in a while.

First, the original crunch lib consists of two separate pieces of tech: CRN mode and RDO mode. RDO mode was intended for fast desktop encoding, CRN for cloud. Most current AAA users of the original crunch library use RDO compression, not .CRN (what Unity decided to use), so they can combine it with a state of the art LZ solution like Zstd. The ones that need the higher compression mode (CRN) use massive clouds to do the encoding, because their datasets are enormous. I heavily optimized RDO mode, and it's easily many times faster than the original .CRN mode. Unity's work has impacted CRN encoding perf, but not RDO. All Unity had to do was replace the .CRN frontend with the already optimized RDO frontend (which solves the same problem) and they would instantly have had much faster .CRN compression, and would have saved themselves a bunch of time and effort. But they didn't write the library and didn't even bother to ask the author what would be best to do.

Anyhow, the more companies that use crunch, the better. Our copyright will remain on the code. The more open source projects that take our legacy tech and branch it somehow, the more influence we've had and continue to have. When we walk into a game studio, or Netflix, or Microsoft, or Sony, or Google, etc. they are already using our tech before they even talk to us at Binomial, and that's invaluable.

crunch, LZHAM, and miniz (my other open source compression libs) were released to advance the state of the art in data compression, which I have successfully done. The teams at RAD and Google working on their later lossless codecs closely studied and profiled LZHAM. I did all this 100% for free, unlike many open source devs who are paid large sums of money by some corporation to do the work. Now I have code in endless repos and my open and closed source data compression technology has shipped in major engines/products.

As for Unity branching crunch: It's not some nebulous "community" doing this work for free for the benefit of all. It's one paid Unity Technologies employee who has been reverse engineering my old software and algorithms for years. Unity's in a desperate feature war (to grab more AAA customers) and wanted faster compression, and we think GPU format specific solutions like crunch along with CPU encoding/transcoding are more or less a dead end. GPU format independence is where things are going, because you can encode once vs. X times and get all the GPU formats that matter. Unity deploying crunch has helped our business because all the other engine teams/companies now need to compete with them along this axis.

Some of the things being posted here are just comical. If you think about it, the crunch lib was so good that Unity execs sat down an engineer for *years* to try and figure out how it worked and improve it. The skillsets involved in creating a new technology and working codebase completely from scratch aren't the same as the ones involved in optimizing an already completed, working solution.

The largest non-game users of crunch (the ones who run it on their massive clouds, and the ones that need a super reliable encoder) will not deploy Unity's variant due to reliability/quality concerns.

-Rich Geldreich

[richgel999]

Not really. Nobody would be using his stuff and nobody would know about it. He tried to market it and didn't have any success. His code wasn't ready for prime time and needed a lot of work. The only reason it went somewhere is that it was free under a commercially usable license.

Have you ever used Google Maps by any chance? They were the first to deploy crunch in 2012, into a cloud consisting of over 100k machines, years before Unity looked at crunch. To get it there, I had to port it to Linux specifically for Google. I was told that if it wasn't for crunch the Javascript version of Google Maps wouldn't have been shippable.

crunch was brought into Google's private repo years before Unity ever looked at it. Honestly, Unity and video games are in no way "prime time".

[tajjada]

I don't know why I spend the time replying to posters hiding behind handles here, but Phoronix has been a good supporter of my work so I check in once in a while.

First, the original crunch lib consists of two separate pieces of tech: CRN mode and RDO mode. RDO mode was intended for fast desktop encoding, CRN for cloud. Most current AAA users of the original crunch library use RDO compression, not .CRN (what Unity decided to use), so they can combine it with a state of the art LZ solution like Zstd. The ones that need the higher compression mode (CRN) use massive clouds to do the encoding, because their datasets are enormous. I heavily optimized RDO mode, and it's easily many times faster than the original .CRN mode. Unity's work has impacted CRN encoding perf, but not RDO. All Unity had to do was replace the .CRN frontend with the already optimized RDO frontend (which solves the same problem) and they would instantly have had much faster .CRN compression, and would have saved themselves a bunch of time and effort. But they didn't write the library and didn't even bother to ask the author what would be best to do.

Anyhow, the more companies that use crunch, the better. Our copyright will remain on the code. The more open source projects that take our legacy tech and branch it somehow, the more influence we've had and continue to have. When we walk into a game studio, or Netflix, or Microsoft, or Sony, or Google, etc. they are already using our tech before they even talk to us at Binomial, and that's invaluable.

crunch, LZHAM, and miniz (my other open source compression libs) were released to advance the state of the art in data compression, which I have successfully done. The teams at RAD and Google working on their later lossless codecs closely studied and profiled LZHAM. I did all this 100% for free, unlike many open source devs who are paid large sums of money by some corporation to do the work. Now I have code in endless repos and my open and closed source data compression technology has shipped in major engines/products.

As for Unity branching crunch: It's not some nebulous "community" doing this work for free for the benefit of all. It's one paid Unity Technologies employee who has been reverse engineering my old software and algorithms for years. Unity's in a desperate feature war (to grab more AAA customers) and wanted faster compression, and we think GPU format specific solutions like crunch along with CPU encoding/transcoding are more or less a dead end. GPU format independence is where things are going, because you can encode once vs. X times and get all the GPU formats that matter. Unity deploying crunch has helped our business because all the other engine teams/companies now need to compete with them along this axis.

Some of the things being posted here are just comical. If you think about it, the crunch lib was so good that Unity execs sat down an engineer for *years* to try and figure out how it worked and improve it. The skillsets involved in creating a new technology and working codebase completely from scratch aren't the same as the ones involved in optimizing an already completed, working solution.

The largest non-game users of crunch (the ones who run it on their massive clouds, and the ones that need a super reliable encoder) will not deploy Unity's variant due to reliability/quality concerns.

-Rich Geldreich

Thank you for caring enough to write this highly-informative reply. I have learned new things from it.

FWIW, I do not hold the views I expressed in my comment from 4 months ago anymore. In retrospect, sorry for the harsh comment.

I wish you best of luck in your work. You have done a lot to advance the state of the art in data compression, as you said. Thank you for it.