X.Org/FreeDesktop.org Is Looking For Sponsors Or May Have To Cut Continuous Integration Hosting

[J.G.]

infrastructure was previously housed on servers at the likes of Portland State University and MIT, but their GitLab hosting setup is found in Google's cloud

What a waste of money. They even had an alternative and chose to funnel sponsorship funding and donations right through to Google.

Google is willing to hire an admin

It would be better for Google to hire a financial responsibility advisor for them.

[starshipeleven]

It would be better for Google to hire a financial responsibility advisor for them.

It would be better if it was not hired by Google, but I agree on the general idea.

[allquixotic]

it's not bandwith but CPU power and system resources to run the Continuous Integration build bots and automated testing infrastructure

On the first page of this thread someone quoted Matt Turner as saying:

We're paying 75K USD for the bandwidth to transfer data from the GitLab cloud instance.

Even if that $75k includes significant compute resources and they simply weren't mentioned, having full unconstrained access to 320 Sandy Bridge cores as in my pricing example is going to be WAY more than they need. My example of a $30k/year budget was a significant over-estimate of what they need. I believe such a purchase would cover all of freedesktop.org and X.org's hosting needs, including web traffic, wiki, bug reporting, source control (GitLab, etc.), and CI. If their infrastructure budget is around $30k/year and that amount is sustainable given current donations, then this would work for them.

A few additional thoughts for cost-savings:

1. If they build their software systems so additional nodes can easily be added to expand support, they could start with a more modest cluster, and only add nodes once the demand for resources is demonstrated.
2. The above pricing bakes in cost for the HDD storage. If they decided they wanted all large binary (built artifact) storage to be on Wasabi instead, and just use the local SSDs for temporary scratch space during build processing, they could save $20 per server per month, or $4800/year off of the bill. This all depends on whether 2 x 240 GB SSD is enough for the local storage space -- if it's not, the HDDs will be necessary. BTW, as a simple experiment, for $4800/year they could store 66 TB of data on Wasabi for a whole year. If one build of LibreOffice consumes 1 GB, that would be enough for 66,000 builds of LibreOffice.

[starshipeleven]

On the first page of this thread someone quoted Matt Turner as saying:

oh wow, that post wasn't there when I first read the thread.

Huge bandwith usage is what you get when you set up a CI without any form of download cache.

Every commit the system pulls down sources of all stuff and rebuilds. Complex projects need to pull down a lot of stuff that isn't just in their own source tree.

Sane admins usually set up a local cache for the stuff. For example the OpenWrt infrastructure is ALWAYS rebuilding development snapshots (so they are refreshed daily or so), but there is a local cache server (either in the same group or in the same server) that keeps all the tarballs and whatnot of all sources of components needed when rebuilding.

[Jabberwocky]

GCloud/AWS/Azure are all scams, unfortunately. They promise you the world, give you estimated bills, then you realise they lied with a straight face, and hide behind enough technicalities and small print that it's still cheaper to just be extorted.

I have seen this happening over, and over, and over. My current company pays gcloud only about 30x what the estimated costs were. That's still 20x what we paid before at Hetzner.

So, please, don't ever trust these large cloud providers' promises.

Rule #1: Don't trust any sales person ever! Due diligence dictates that you plan and calculate yourself.

I've been using AWS and GCP for over a decade. I can say no platform is perfect, but it saved my customers a lot of opportunity cost as well as time and money in the long run.

In my personal capacity I'm pro homelab and DIY. In my professional space I know there are no platforms on this planet that allows quick, large scale, stable deployments and automation in so many regions like AWS or GCP.

It's important to not have all your eggs in one basket, cloud providers have a list of services to prevent you from moving to another provider, terraform and k8s helps to make cloud more agnostic in a sane way cost effective way. You can abstract other services but the development and maintenance overhead is huge. It's still not child's play to setup and maintain standalone k8s from scratch. One should not get tricked by out of sight out of mind.

I'm a fan of https://www.packet.com/ but most CTOs don't seem to trust them. I really hope they will do more marketing and expand to more locations!

PS: There are legacy code bases (unmaintained c usually) that does not run on the cloud, but I deliberately choose to not work with companies that operate that way. Cloud is usually not their style either.

[EarthMind]

GCloud/AWS/Azure are all scams, unfortunately. They promise you the world, give you estimated bills, then you realise they lied with a straight face, and hide behind enough technicalities and small print that it's still cheaper to just be extorted.

I have seen this happening over, and over, and over. My current company pays gcloud only about 30x what the estimated costs were. That's still 20x what we paid before at Hetzner.

So, please, don't ever trust these large cloud providers' promises.

I'm quite familiar with Azure and can verify that it's VERY unclear what your costs are at the end of the month. Although they are investing a lot of effort into budget limiting and prediction (way more than AWS afaik), they should be clear about exact pricing when you're setting up new environments. They are all but transparent on that and we still have to go to the general pricing documentation to make a calculation, but that's a lot of effort each process and even then it doesn't guarantee exact cost calculation.

And I haven't even spoken about their ridiculous pricing for mediocre performance. Storage performance on Azure is really bad, unless you pay high premium prices.

Smaller providers are more transparent about the costs though, e.g. Digitalocean, Upcloud etc.

[grigi]

I think the big issue is that the CI often needs to run on physical devices with specific hardware. So they will have this BW issue whereever they are? I'd imagine the specific hardware would often run off-site, so that a developer can inspect it manually in case something goes wonky?

[Ironmask]

exactly. Let Pottering integrate it into systemd

... Sometimes I feel like half the users here are markov chains

[skeevy420]

Contrary to popular opinion, capitalism does not imply scamming customers.
Scamming people out of their money is a completely parallel thing to the economic structure and is prevalent in communism and socialism too.

There is more than one form of capitalism. Take early 1800s capitalism where you lived in a company town, got paid in company money, shopped at the company store, lived in company provided housing and basically resulted in a bastardized mix of communism and capitalism. Legacy/Monopolistic Capitalism.

Then you have 1980s capitalism where it's based on nothing but short-term goals and milking everything dry before moving on to another asset and repeating that to increase profits to shareholders at the expense of everything else -- I call it as Predystopian Capitalism....but it'll also go full circle back to Monopolistic Capitalism if left unchecked because we'll be back just like we were in the 1800s...only replace "a handful of really rich guys" with "a handful of really rich mega corps"...and remember, "really rich mega corps" are people too.

Then you have 1950s capitalism. Socialistic Capitalism. It's pretty much the same as 1980s capitalism, only there are still high taxes and regulations. High taxes are used to fund society and regulations make sure Flint, MI doesn't happen. Lower taxes and you can't afford to fund regulators which results in funk-ass, poisoned drinking water.

But unchecked, pure capitalism....that's literally nothing but getting whatever you can from whoever you can get it from with the only regards being your own, personal gain and outcome. Pure capitalism requires fascist, socialist, and communist laws to prevent people from screwing each other over...and whoever allowed the fine print laws to be passed needs to be taken out back and dealt with. I shouldn't need a goddamn magnifying glass to ensure I'm not about to be screwed over.

[flux242]

Contrary to popular opinion, capitalism does not imply scamming customers.
Scamming people out of their money is a completely parallel thing to the economic structure and is prevalent in communism and socialism too.

geez, what an idiot! Contrary to popular opinion cola,mc'donalds and comic books won't make you free or smart, it'll fokk you up to be stupid like that one above