GCC's Conversion To Git Is Being Held Up By RAM, a.k.a. Crazy DDR4 Prices

[anarki2]

That's the stupidest article I've read in a good while. Stupid on way too many levels So I won't even bother to elaborate

[grok]

Get a Threadripper 1900X and 96GB RAM (reuse your 64GB, add another 32GB) : this would be the cheapest way or reasonably cheap way to do it, if coming from a Core i7/Xeon E3 platform. Update firmware (what AMD calls AGESA). It'd be a bit better with Threadripper 2900X as it's the same with lower memory latency but it'll be available some months from now. Either can be overclocked. It would be a rare example of overclocked system with ECC.

Am I the only one here confused about what exactly it is that's taking up all this RAM? If he's just transferring files to git, why does that need so much RAM, and, why couldn't just segment it? If he's compiling code, couldn't he just use fewer threads? There's probably something big or obvious that I'm not noticing here.

If you RTFA and follow links you can reach there :

(...)
> The Great Beast is a semi-custom PC optimized for doing graph theory
> on working sets gigabytes wide - its design emphasis is on the best
> possible memory caching. If I dropped back to a conventional machine
> the test times would go up by 50% (benchmarked, that's not a guess),
> and they're already bad enough to make test cycles very painful.
> I just saw elapsed time 8h30m36.292s for the current test - I had it
> down to 6h at one point but the runtimes scale badly with increasing
> repo size, there is intrinsically O(n**2) stuff going on.
>
> My first evasive maneuver is therefore to run tests with my browser
> shut down. That's working. I used to do that before I switched from
> C-Python to PyPy, which runs faster and has a lower per-object
> footprint. Now it's mandatory again. Tells me I need to get the
> conversion finished before the number of commits gets much higher.
>
> More memory would avoid OOM but not run the tests faster. More cores
> wouldn't help due to Python's GIL problem - many of reposurgeon's
> central algorithms are intrinsically serial, anyway. Higher
> single-processor speed could help a lot, but there plain isn't
> anything in COTS hardware that beats a Xeon 3 cranking 3.5Ghz by
> much. (The hardware wizard who built the Beast thinks he might be able
> to crank me up to 3.7GHz later this year but that hardware hasn't
> shipped yet.)
(...)

follows talk about the software which is a custom tool. That's the gist of the problem, a custom and complex migration tool that I assume puts everything in memory because it's easier and faster and writing this shit is costly in Unix bearded admin time.

So he's using a Xeon E3 I suppose, which is similar to a branded version of i7 6700 and such and has good memory performance in term of latency. Unless he meant a Xeon E5-1620 v3. Argh!! If so just upgrade to 96GB asap!
I had to run the fucking puzzle. E5-1620v3 is a Haswell on 2011-3 with 10MB L3, 3.5GHz, 3.6GHz turbo. If updating to E5-1620v4 that's 10MB L3, 3.5GHz, 3.8GHz turbo and can run the RAM at DDR 2400 speed instead of 2133. (and some +5% from Broadwell vs Haswell)
Buy some fucking 4x8GB DDR 2400 ECC or 4x16GB DDR 2400 ECC. Depending on whether your system has 4x16GB or 8x8GB ECC. I assume it is unbuffered ECC, as it has better performance than buffered (registered). If you have buffered buy buffered.

Yet final edit if you do have a Xeon E3 platform (socket 1151, dual channel memory, max 64GB) you do have to migrate to Threadripper or Xeon E5 or Xeon-W.
Talos Lite with POWER9 should be good also, as it just matches a high end PC but is considered good on memory performance..

There was also an announcement about new high density Samsung DDR4. It was notable because it said it would allow 128GB in a laptop (four sticks of 32GB unbuffered). It's near future stuff like the Threadripper 2900X. This means a latest mobile Xeon in the laptop would do the job. Running a serial task would mean running at turbo speed so possibly 4.0GHz in a laptop, with low memory latency (btw core desktop i9 is slower! seen on gaming benchmarks! Xeon-W is Xeon-branded desktop Core i9)
Holy crap the Coffee Lake Xeon mobile has even stupider high single core speed : 4.4GHz or 4.8GHz, and 12MB L3 https://hothardware.com/news/intel-c...rocessors-leak . Sure you don't need a damn laptop. I wanted to sum up everything though.
Older fast CPU that can take "old" DDR3 Registered ECC from discarded servers meaning cheap RAM on ebay : https://ark.intel.com/products/75779...Cache-3_70-GHz https://ark.intel.com/products/64621...-GTs-Intel-QPI

[Weasel]

This problem exists because of Python in the first place. You know, sometimes "throwing more hardware at the problem" doesn't work.

For all the people who like to moan about everything, that's about an 8-hour day's gross wages for a freelance developer who charges $162.50/hr, and there are plenty of people out there charging more than that.

What country is that wage from? I'd like to relocate.

[Venemo]

Why not just rent an Amazon EC2 instance with enough RAM for the duration of this operation?
Or just ask Red Hat (who sponsors GCC anyway) for a machine with enough RAM, until this is done?

[discordian]

follows talk about the software which is a custom tool. That's the gist of the problem, a custom and complex migration tool that I assume puts everything in memory because it's easier and faster and writing this shit is costly in Unix bearded admin time.

probably some sparse accesses where you would have to load/page-in several KB from HDD for getting a couple dozen bytes.

Still, he refused to use some build-servers with 128GB of Ram because his "beast" is supposedly faster. Thats all nice and fine, but I dont get why this is problem with a one-time-import, I bet it could be done before he made substantial improvements or got RAM shipped.

[dwagner]

The complete linux git repo I am working with has 766962 commits (so three times that of gcc?) - and is only a few GB in size.
At work, I have converted SVN repositories with about as many revisions and about the same size to git repositories - yes, the once-off "git svn" invocation was a process taking several days, but otherwise no particular issues in terms of RAM usage.

[Niarbeht]

This is Eric S. Raymond computer build https://www.youtube.com/watch?v=875NbdL39A0

Awww, now I'm bein' all nostalgic for when Tek Syndicate was super awesome and stuff.

[grok]

probably some sparse accesses where you would have to load/page-in several KB from HDD for getting a couple dozen bytes.

Still, he refused to use some build-servers with 128GB of Ram because his "beast" is supposedly faster. Thats all nice and fine, but I dont get why this is problem with a one-time-import, I bet it could be done before he made substantial improvements or got RAM shipped.

It could be to make a point. In particular it means there are tasks where he doesn't give a shit about a 1024-core CPU or a 1024-node cluster but would be better served by a dual core 5GHz CPU with 256GB RAM. It's not too far off, this is the kind of workload when visiting a mainstream media news site on Windows 10 or Gnome 3.

[grok]

probably some sparse accesses where you would have to load/page-in several KB from HDD for getting a couple dozen bytes.

I wonder about this too, I fall for the alleged algorithmic complexity and argument from authority.

[Veto]

FTA:

My first evasive maneuver is therefore to run tests with my browser shut down. That's working.

No shit, Shirlock