X.Org Is Getting Their Cloud / Continuous Integration Costs Under Control

You may recall from earlier this year that the X.Org/FreeDesktop.org cloud costs were growing out of control primarily due to their continuous integration setup. They were seeking sponsorships to help out with these costs but ultimately they've attracted new sponsors while also better configuring/optimizing their CI configuration in order to get those costs back at more manageable levels.

The X.Org/FreeDesktop.org server/cloud expenses grew from a few hundred dollars per month before jumping at the end of 2018 quickly into thousands per month. Following subsidies/sponsorships from Google and Gitlab, their expenses really began ballooning at the start of 2020 when hitting more than six thousand dollars per month in cloud expenses, an unmanageable level for a open-source project relying upon sponsorships.

Of their $6,300+ month cloud bill in January, around $3,700 of that were networking costs, another $2,258 in compute costs, and around $360 in Kubernetes costs.

In working to get the costs under control, the volunteer FreeDesktop.org admins began analyzing logs and enabling other features for figuring out how to better optimize/cache their CI setup to reduce the exploding costs. One of their big helpers in this feat was the Grafana open-source analytics/visualization application.

Among the optimizations as a result were configuring their RAID array, garbage collecting registry images, and remediations on their artifacts dropped bandwidth usage from 3.5TB per week down to around 150GB. Google Compute costs were reduced from an updated Nginx, Git tweaking, and other tuning.

The FreeDesktop.org costs have now dropped from around $6k per month down to around $3k per month for the past several months. However, FreeDesktop.org has a budget of around $2500 USD per month for infrastructure.

Further reducing costs is now being pursued by moving out of Google Cloud, Packet.com sponsoring now a lot of their continuous integration setup, and Packet.com hosting their Gitlab infrastructure too. The GStreamer project also stepped up in paying for their own CI runners.

More details via the XDC2020 presentation above (the presentation starts at around 75 minutes into the stream) and this slide deck by Red Hat's Benjamin Tissoires.