LZHAM + Crunch Now Placed Under The Public Domain

[illwieckz]

I have three pull requests opened since 2017 on your BinomialLLC's crunch repository […] The reason I cannot compile that tool on a 64 bit Linux system in 2020 is the error I reported you in 2013, the error I delivered you a fix for in 2013.

richgel999 I made a typo: I reported the issue and delivered you the fix in 2017. The fix itself was implemented by someone else in February of 2014 and never been merged. So, the fix for Linux compilation on 64bit systems has been ready for at least 6 years and 6 months. The PR from 2017 is the submission I made myself on Github. The PR for the rtopmip features is from July of 2014. All those PR I talked about are part of an initiative explicitly named “merging to upstream”: https://github.com/DaemonEngine/crunch/issues/2

Anyway, no one pull request from anyone got merged in your tree after January of 2017. There is no code change since this date. All commits since January of 2017 in your tree are readme and license tweaks. The Unity tree is based on the latest version of your code (the only missing commits are some about readme rewriting and license wording).

Because my concern is to make and keep Crunch usable and to prevent fragmentation, the Dæmon repository hosts five special branches:

• https://github.com/DaemonEngine/crunch/tree/googlecode
  The original tree that was hosted on Google Code,
• https://github.com/DaemonEngine/crunch/tree/binomialllc
  Your actual tree on Github,
• https://github.com/DaemonEngine/crun...e/unvanquished
  The original tree previously used by Unvanquished/Daemon with patches sitting on top of a now old version of your tree,
• https://github.com/DaemonEngine/crunch/tree/unity
  The Unity tree,
• https://github.com/DaemonEngine/crunch/tree/master
  The current tree maintained by Unvanquished/Daemon with patches rebased on Unity tree.

About maintenance of Crunch by Unvanquished/Daemon, a special effort is done to systematically create a pull-request on what is considered upstream at the time when a change is merged, and a special effort is done to minimize intrusive changes to reduce the risk of something behaving differently once upstreamed. The whole purpose of such maintenance work is to make sure one day the Unvanquished/Daemon tree can be ditched, when upstream would be maintained again and would have fixed the blocking issues and implemented the required features.

The Crunch tree maintained by Unvanquished/Daemon provides:

• Your own tree up to the latest code-related commits you made,
• Unity patches above your own commits (making the code faster and producing smaller files),
• Linux 64bit compilation fixes (make crunch and software embedding crnlib to compile on this system and architecture),
• Fix for malloc in crn_decomp on macOS and FreeBSD (make software embedding crnlib to compile on those systems),
• Support for user-supplied definition of CRND_ASSERT to be used in (make crnlib easier to embed in an software),
• CMake support (make software embedding crnlib easier to integrate crnlib in the toolchain, also install crunch the File Hierarchy Standard way),
• Makefile fixes (to make compilation working again for those who may still rely in it),
• Option for top mip-level renormalization (a special option for normal maps to make Z reconstruction easier when using DXT5/BC3),
• A first class embedding in other software when used as a git submodule.

Note that many people just reimplement the Linux fixes or the CMake toolchain every now and then after your branch has stopped development in 2017.

Also, if what you call RDO is not a format but some decision optimization techniques (maybe the ones used by the -bitrate option I've never seen used?), 90% of my previous comment is still fully relevant because as I said, mistakes I would say about what is RDO is not a concern: my concern is to keep Crunch being alive, being usable and fits the need of users.

Today the Unvanquished/Damon tree of Crunch is probably the only one making it possible to compile and run the Crunch tool outside of Windows and to embed crnlib in software with neat submodule integration and toolchain integration with compilation supported on four operating systems.

Also, there are strategic and legal reasons why this code is explicitly and overtly being placed into the Public Domain, without a license, in the US

Just for the sake of curiosity, is it for a proprietary software or another free software (like some code that would be reused in Basis)? Because the ZLIB license was already very permissive. I'm doing a lot of stuff under CC0 1.0 license, public domain dedication but the wording about strategic and legal reasons pique my curiosity.

[illwieckz]

crunch has two completely different modes: RDO BC1-5, and intermediate (.CRN) BC1-5.

Hi Rich, it's nice to see you there! I was talking about .CRN, the thing the Crunch name is about, the thing the crn library is about. When I'm talking about file incompatibilities, this can't be between .CRN and something else but between .CRN and .CRN. If RDO and CRN are two different formats, then an engine can implement two loaders, right?

The question would be: is there a way to tell a file produced by Crunch is using CRN or RDO?

if yes, RDO would be out of topic when talking about CRN incompatibilities because this could be handled separately.
If no, Unity wouldn't really be in fault to have produced a variant that is not distinguishable since upstream would have done it.

Note that I don't know about RDO at all, so I don't have the answer to that question. Please answer Yes or No to the previous question so we know if you don't have to mention RDO in a conversation about .CRN incompatibilities, or we know you made two formats we can't detect at runtime and made two loaders for them.

I don't know about RDO:

• There is no mention of RDO in the README in the repository,
• There is no mention of RDO in the user interface of the provided binary¹, I found no string about RDO in it,
• There is no mention of RDO in source code, there is no string about it,
• The code does not compile today on an actual Ubuntu Linux LTS, so I was not able today to try an actual binary,
• I looked at print_usage() function and I have seen no mention about RDO,
• I looked at process_files() function and I have only seen mentions about TGA, BMP, DDS, KTX, CRN, and PNG, I found nothing about RDO here and nothing to detect a file is CRN or RDO.

¹ Since this windows binary is 5 years old, it's better to delete it or people may get unexpected results.

So, is there a way to tell if a file is using CRN or RDO?

Like many developers who are paid fortunes to work on open source projects, I don't work for free. I work on open source projects that bring in actual income from large corporations, like Basis Universal (the successor to the crunch lib)

Within the boundaries of that topic I know who you are. I have not talked about money. I don't know why you talk about money. You have nothing to prove. I know you're not involved in the Unity fork. I know about Basis and your involvement in it and I know it's a successor to CRN and I'm waiting for it. I know about your opinion on ASTC license and you can count on me to support you on that opinion.

I know what you said when you said “Think twice before open sourcing your work”. But I also know why the solution is never to not open source a work because the work is not the problem. What I know is that opensource is not about software, it's about people. I know the need for not opensourcing a work is a canary for a person to know his situation is not secured. I know that the need for not opensourcing a work is a wrong need to “live with it”, the wrong need to live with the fact were are not secured, instead of securing ourselves. I know that free software is about making sure we as people don't subject ourselves to a situation that does not secure us, because we have a canary telling us we are not secured and reminding us we need to be secured. I'm happy to see you can now take the risk to place your work not only under a free license but under public domain while you know about the risk, so maybe you're secured enough to afford that? That would be good news!

Note that at this point of my message, I may already have made numerous obvious mistakes about CRN and RDO, but that's not my concern. My concern is to provide people formats they can use. My concern is to provide people working methods and software integrating the formats they can use. My concern is to provide people software that can detect and load correctly the format they can use. Not understanding RDO and CRN is not my concern. My concern is to be able to provide one loader per format, this requires ways to know about and detect the format, this requires up to date documentation not to describe the variants but to name the variants, this requires bits in the format dedicated to detect the variants.

If you understood what the library does

I don't have to understand what the library does. I have to understand what people can do with it. I have to understand who is who, what fixes is for which problem, what feature for which need. Also I'm sure there are many things I understand on some other topics you don't understand or know anything about and if one day we work together I would not ask you to understand them because if one day we have to do things together it would be because each other would have something the other doesn't have. I don't know why you talk about understanding. I have not talked about understanding.

Also the described problem you answer to is not about understanding this or that more or less than any level. There is no competition about understanding, neither in knowledge. The topic is all about making people have their needs fulfilled thanks to various people understanding a bit of this or knowing a bit of that, not everything.

This thread is about knowledge. The same way you are able to acknowledge I can understand things that are not in your own scope I can acknowledge you can understand things that are not in my scope. I can acknowledge the name of the format variants, the way they can be detected, the way different loaders can be implemented for each variant, etc. Where is the knowledge about RDO? Where is the knowledge about detecting format variants and implementing different loaders for each variant?

The RDO BC1-5 modes are used by numerous games and some engines (like Unreal), and Unity's part-time developer didn't touch (or even test) the RDO encoder.

I talked about CRN and you introduce RDO and are making a difference between CRN and RDO. I talked about the fact your CRN format and Unity one were not compatible using your CRN encoder/decoder or Unity CRN encoder/decoder. If Unity didn't touch the RDO decoder, can you tell people why we would be concerned about it? Is there something that prevents me and others from implementing both an RDO loader using your code and an CRN loader using Unity code? You said Unity didn't touch or even test the RDO encoder, do you know if that Unity developer didn't know about RDO to begin with? If you do know he didn't know, can you explain he didn't know?

Why do you mention that the developer was part-time? I don't need to know if you are part-time. You don't need to know if I am part-time. What is this information about?

Unity optimized my first .CRN encoder, but not my second RDO encoder (which I optimized a great deal, years ago)

How can people use it? I'm not talking about understanding about RDO, I'm talking about knowledge. How can people use it, implement it? How can people recommend it?

If you want RDO, you can try to use Unity's repo - but good luck.

I don't know about RDO, you said Unity's repo did not touch or even test the RDO encoder, why would you recommend me to use Unity's code for something that is now knowledgeable to not be the tool for? At this point I don't even know if I need RDO, why would you recommend me what you think is wrong for it?

The statistics you're talking about are only for the .CRN mode, which few games/engines outside of Unity actually use because the transcoder is too slow.

Of course the statistics I'm talking about are only for the CRN mode, I did not talk about RDO at all. I don't know about RDO. I know the CRN transcoder is too slow, Unity knows the CRN encoder is too slow. That's because Unity knew the CRN encoder was too slow Unity worked to reduce that slowness. That's because I know the CRN encoder was too slow I made those benchmarks to know if Unity really managed to reduce that slowness, and they did.

Maybe Unity's effort to reduce CRN slowness is still far behind what you did on RDO. But those are different formats, right? We can make two loaders, right? That must not be a problem is Unity's CRN is incompatible with RDO, right? You said it's completely different, right?

The root code they forked from is now in the Public Domain, so they or the public can do literally whatever they want with it, including changing the license in the forked version

Yes of course, they can put their additional commits in public domain too. But wait… that's not the topic.

So, is there a way to tell if a file is using CRN or RDO?

Also: how to select CRN mode or RDO mode with Crunch?

crunch has two completely different modes: RDO BC1-5, and intermediate (.CRN) BC1-5

You talked about things being “completely different”, so is there a way to tell if a file is using CRN or RDO?

Note that I said that within the boundaries of that topic I already know about you, but in fact you already know about me.

I have one issue opened since 2017 on your BinomialLLC's crunch repository:

• https://github.com/BinomialLLC/crunch/issues/11
  This is a cross-platform compilation issue (32/64bit)

I have three pull requests opened since 2017 on your BinomialLLC's crunch repository:

• https://github.com/BinomialLLC/crunch/pull/13
  It's the fix for Linux compilation on 64bit platform (the issue I just talked about),
• https://github.com/BinomialLLC/crunch/pull/14
  It's a fix for a type issue,
• https://github.com/BinomialLLC/crunch/pull/12
  It's a feature implementation.

Note that those three pull requests were not about code I wrote:

• I was the one reporting you the issue,
• I was the one shipping the fixes up to you and making sure they are rebased on your latest tree,
• I was the one giving you knowledge about the errors and the fixes,
• I was the one giving you knowledge about the needs and the features,
• I was the one who put you in touch with the specialists who implemented the fixes and the features, those with whom you can talk about understanding.

I don't know why this code I delivered to you was never merged in. You talked about money and said you would not work for free. I did not ask you to fix the issues. I did not ask you to implement the features. I delivered the fixes, others implemented the fixes and features and I did the work to ensure you to get the fixes and features, you did not have to implement the fixes, you did not have to implement the features. You did not answer.

Well, I was curious and I looked at the reason why I'm not able today to compile and run your tool, the last version of your branch, the one you just set in the public domain. The reason I cannot compile that tool on a 64 bit Linux system in 2020 is the error I reported you in 2013, the error I delivered you a fix for in 2013.

I know the other Crunch tool based on Unity enhancements and maintained by Dæmon/Unvanquished compiles on Windows, Linux and macOS, and I know the CRN library from that other tree is integrated in game engines running on those OS, and I know the CRN library form that other tree is integrated in game edition tools compilable and running on Windows, Linux, macOS and FreeBSD. Knowing those things is part of my concern, alongside the efficiency of the tools. Until today I have found no knowledge about RDO from your tree, and the Linux 64 bit fix for your tree is yet to be merged after three years.

I said earlier that if one day we have to do things together it would be because each other would have something the other doesn't have. You see, I know the true strength of patience and after three years I know it's still possible to do things together. I'm still there.

[richgel999]

So, if you still need Crunch, please, really please not use the old code from Binomial repository. To be honest I don't know why he never merged Unity's work on Crunch. The code from Binomial repository is dead and must not be used.

If you understood what the library does, you would know that there are two completely different encoders in the crunch library: .CRN and RDO. Unity optimized my first .CRN encoder, but not my second RDO encoder (which I optimized a great deal, years ago). The statistics you're talking about are only for the .CRN mode, which few games/engines outside of Unity actually use because the transcoder is too slow.

The RDO BC1-5 modes are used by numerous games and some engines (like Unreal), and Unity's part-time developer didn't touch (or even test) the RDO encoder. I've thoroughly tested both the RDO and .CRN code in the Binomial repo, but not the Unity repo. If you want RDO, you can try to use Unity's repo - but good luck.

[richgel999]

Well, my comment was flagged… this code is abandonware, and did not get the last updates, please don't use it: it produces files incompatible with current software: the Unity game engine, the Dæmon game engine, the NetRadiant game level editor, etc. This other tree has performances/efficiency patches from Unity and maintenance and compilation fixes an the three main OS from Unvanquished/Dæmon: https://github.com/DaemonEngine/Crunch

crunch is used by numerous companies, and I'll gladly update it if someone pays me to do it. Like many developers who are paid fortunes to work on open source projects, I don't work for free. I work on open source projects that bring in actual income from large corporations, like Basis Universal (the successor to the crunch lib). 99% of game companies won't pay a dime to support projects like this. They instead work their employees until they drop for peanuts, and expect open source authors to give away their code for free.

crunch has two completely different modes: RDO BC1-5, and intermediate (.CRN) BC1-5. A part-time Unity engineer optimized only the .CRN mode. In the process, Unity made the .CRN encoder basically unreadable/unmaintainable. Unity didn't touch the RDO encoder at all, because they don't use it, and they could have very possibly broken it. The root code they forked from is now in the Public Domain, so they or the public can do literally whatever they want with it, including changing the license in the forked version.

Also, there are strategic and legal reasons why this code is explicitly and overtly being placed into the Public Domain, without a license, in the US (with an unlicense-style fallback clauses for jurisdictions that don't recognize Public Domain).

-Rich

[jntesteves]

Very interesting graphs, in particular "transfer+processing". Too bad Zstd is considered to have just one level, when it has been like more than 12 for quite some time.

I think you mean more than 20. The original first release of zstd had 19 levels IIRC.

[uxmkt]

https://quixdb.github.io/squash-benc...-decompression

Very interesting graphs, in particular "transfer+processing". Too bad Zstd is considered to have just one level, when it has been like more than 12 for quite some time.

[illwieckz]

Well, my comment was flagged… this code is abandonware, and did not get the last updates, please don't use it: it produces files incompatible with current software: the Unity game engine, the Dæmon game engine, the NetRadiant game level editor, etc. This other tree has performances/efficiency patches from Unity and maintenance and compilation fixes an the three main OS from Unvanquished/Dæmon: https://github.com/DaemonEngine/Crunch

[illwieckz]

The thing is that Unity improved it and broke compatibility in the meantime (for the good anyway):

Unity guys said that their modified crunch tool “can compress up to 2.5 times faster, while providing about 10% better compression ratio”. So We did a test on our own asset repository, re-crunching all the ressources and textures packages. The given corpus at the time we did the the test produced 1797 .crn files. We benchmarked both our old crunch tool based on BinomialLLC code and the new Unity one with our patches applied. We rebuilt the binaries so there is no bias on that part. And here come the results :

BinomialLLC’s crunch:

• compression time: 115 minutes 8 seconds
• produced file size: 269 megabytes

Unity’s crunch:

• compression time: 26 minutes 43 seconds
• produced file size: 239 megabytes

source: https://unvanquished.net/unvanquished-area-51/

Also, Unity is not merging fixes, so maintained version of Crunch is there (we makes sure it compiles under Linux, macOS and Windows):

https://github.com/DaemonEngine/Crunch

Michael Seeing public domain is good, but what is public domain is now obsolete, less performant, less efficient, and dead, incompatible code.

As an example, see abandonware source code releases.

Yes, this old Binomial's crunch is abandonware, look at Dæmon/Unvanquished Crunch for maintained one. Please, people, please don't use that old code from Binomial, it's unmaintained, less efficient than it can be, and a much much much slower than it can be. Also, there may be no way for a Crunch loaded to dectect that old format so your precious files would just produce pure garbage (or fail miserably) on modern editing tools and engines.

Also, Rich Geldreich is now working on another project, Basis: https://www.phoronix.com/scan.php?pa...WebGL-New-Exts
So, if you still need Crunch, please, really please not use the old code from Binomial repository. To be honest I don't know why he never merged Unity's work on Crunch. The code from Binomial repository is dead and must not be used.

[tildearrow]

I don't know why, but I feel this about source code releases:

- People who don't understand what open-source is release it under public domain.
- People who do know open-source exists release it under an open-source license.

As an example, see abandonware source code releases.

[linuxgeex]

I agree!

Are there any comparisons out there?

https://quixdb.github.io/squash-benchmark/#ratio-vs-decompression