The GCC Git Conversion Heats Up With Hopes Of Converting Over The Holidays

[Terrablit]

I'm kind of sad I'm late to the party celebrating ESR's "achievements".

Realistically speaking, in the two years of this project's delays, an intern could have manually walked through every commit in the history and committed it anew to git. Then someone could have reviewed the entire thing and done any commit revision necessary to adjust dates, commit messages and attribution. Various other options, like a two stage conversion (simple, then revising pass) could have worked as well. There were plenty of options, but ESR claimed he was experienced, and said he'd deliver results.

Instead after a year we got a "tool" that somehow didn't work with less than 64GB of ram. We got no intermediate progress, but a lot of far more competent people dedicated their time to cleaning up after ESR, and a lot of people donated to get him better computers when he's clearly not competent enough to need what he already had. Then he started to learn Go, but wasn't competent enough to convert the project. So he called out to get other people to do the work again.

There's very little indication that he's working hard on this. And there's little indication of results. And every time it comes up, he blames his tools or the scope of the work, but dissuades anyone else from working on it unless it's under his direction. But there's no indication he knows what he's doing other than his self-confidence and his distrust in anything else. SRSLY, reposurgeon code is worse than RenPy code. If that's what you're writing, you don't need Threadripper to write it. And Moore's law's never going to keep up with being able to run it as is, as the target repo gets bigger every day.

Letting him get involved in this only slowed it down. Maybe his last-minute scramble to prove he hasn't wasted everyone's time will bear fruit, but I'd rather see Maxim succeed than him. Maxim shows a lot more progress in a lot less time with a lot less bluster. He didn't call for handouts regularly, and he describes his process clearly and succinctly instead of pretending like it's magic or a great feat of skill.

If ESR wins, I'm afraid he'll slow down again without competition. Instead of getting the job done, he'll probably just find some random excuse to write a shitty clone of the JVM for "performance reasons". In a way, it's amazing. I've never seen anyone else with more than two years of experience NIH everything while being so bad at reinventing the wheel.

[smitty3268]

If ESR wins, I'm afraid he'll slow down again without competition. Instead of getting the job done, he'll probably just find some random excuse to write a shitty clone of the JVM for "performance reasons". In a way, it's amazing. I've never seen anyone else with more than two years of experience NIH everything while being so bad at reinventing the wheel.

The good news is it looks like they are committed to picking a solution next Monday, and then doing the conversion at the end of the year. If the reposurgeon conversion isn't fully working by next Monday, it will have presumably eliminated itself by then and Maxim's solution will win by default.

Doesn't sound like they're willing to wait any longer.

[stormcrow]

And the gcc developers who prefer git are probably using it, with the git-svn integration, to let the upstream project continue to look like svn while having the ability to use tools that are a bit more modern and familiar and integrated. I have gone the git-svn route for a few past projects until the rest of the community agreed to do a (good enough) conversion. That said, those projects were not the size, complexity, or broken history of gcc, so I think we can understand that large projects without corporate commitment, resources, and decision making (such as what happened with Microsoft and their conversion to git for the Windows team) are not going to happen quickly.

I didn't think about that when I wrote my reply, but your suggestion also makes sense. Probably plenty of contributors using existing compatibility tooling that more or less makes the conversion moot (to them).

[GI_Jack]

ESR is a giant fucking tool and always has been. Some suckers just bought him a new rig, and that is fucking hillarious.

If RAM was an issue, he could have gotten a decently fast SSD, perhaps even two in RAID-0, perhaps on any modern board with NVMe and just used them as 1TB+ or swap.

[starshipeleven]

If RAM was an issue, he could have gotten a decently fast SSD, perhaps even two in RAID-0, perhaps on any modern board with NVMe and just used them as 1TB+ or swap.

That's... not what you do when you need a lot of RAM.

[anarki2]

I don't know why some of you are so salty or angry about this.

I just see a guy who wants to carefully and cleanly convert the data and has been working on doing this in the spare time he has available.

Any cowboy with some basic coding knowledge could do a fast conversion using existing tools or even some homebrew software. However, the result would almost certainly be an imperfect conversion that would leave a trail of confusion and problems for years to come. But that cowboy would have "done his job" and received praise for doing it, so he probably wouldn't care. I've met those kind of coders; they are numerous. There are far fewer coders who are methodical, far-thinking and considerate.

If I was a GCC dev I'd prefer that this guy takes another 2 years doing this task properly instead of it being done quickly with potential messy fallout.

There's a difference between being pedantic and being an incompetent fool just stabbing in the dark without a clue.

[alcalde]

I don't know why some of you are so salty or angry about this.

I'm salty because he decided to post an entire rant/essay disowning Python, which he blamed for his problems. Python programmers who looked at his code found it to be a nightmare, and translating it to Go just meant porting crappy code from one language to another. I swear ESR told me that the Python version (and the subsequent port) didn't employ any parallelization and his knowledge of Python debugging tools seemed scant. In the end, he achieves an order of magnitude less performance gain than he predicted, eating up months of time, showing it wasn't the language at all that was the problem, but the programmer. Similarly, his memory problems were caused by his insistence of trying to keep everything in memory at once, which others who had worked on similar projects tried and failed to convince him wasn't necessary. Meanwhile, those who didn't follow this closely or read his blog simply saw headlines about Python being a terrible language and some famous programmer abandoning it, tarnishing the language's reputation. That's not fair.

[alcalde]

Anyone claiming "he should just do this" or "just do that" probably never had a "pleasure" of converting huge SVN repo to Git.

His original blog post announcing the rewrite to Go had several people in the comments section who had just that experiments and couldn't for the life of them understand what he was doing, including the need to load everything into memory at once. Not one could convince him that it was the approach he was using, not the language, that was the problem.

[alcalde]

ESR: I didn't port to Go to fix bugs, I ported for better performance.

The program only needs to be run once. It would not have taken the months to run that it took to convert.

Python is a wonderful language for prototyping a tool like this, but it's too slow and memory-hungry for use at the GCC conversion's scale. Also doesn't parallelize worth a damn.

In exchanges with him in the comment section of his blog post about the conversion, his displayed almost no (accurate) knowledge of writing parallelized Python code at all. I swear he said at the time that the code didn't employ parallelization either then.

[dreamer_]

His original blog post announcing the rewrite to Go had several people in the comments section who had just that experiments and couldn't for the life of them understand what he was doing, including the need to load everything into memory at once. Not one could convince him that it was the approach he was using, not the language, that was the problem.

I can't comment on this particular decision, as I don't know the details - but faced with a similar situation in the past I made a similar decision and it turned out to be correct.

Except instead of writing my own tool from scratch I implemented additional functionality to BFG. BFG (written in Scala) loads everything into memory to speed up rewrite operation of whole DAG at once. It works several orders of magnitude faster than using git-filter-branch - when doing large migrations in the past I used it as one of several steps of converting repo to Git (it was used to rewrite binary artefacts from SVN to git-lfs, fix historical commit messages, and remove some broken commits causing problems from SVN history) - the speed of operation allowed me to run migration step on each commit to SVN (so I had new repo snapshot less than a minute after new commit landed in SVN) and allowed me to track progress in the original repo and find problematic areas of migration early (it would've been impossible if I worked with full SVN dumps offline).

I imagine ESR might face similar problems I had and writing dedicated migration tool to deal with issues specific to GCC might be the correct approach.