Google Has A Problem With Linux Server Reboots Too Slow Due To Too Many NVMe Drives

[CommunityMember]

Guess I would be interested in knowing just what OS you are using

In the hyperscaler cloud space one will find the hyperscaler reboots servers (including bare metal servers) all the time, and every minute lost can be (substantial) lost revenue at hyperscale (along with annoyed customers who want their systems available now, now now).

[CommunityMember]

AFAIK it can't be anything else than a write back cache that causes the NVMe driver to wait so the question you should ask yourself is what crazy sized write back caches that Google uses that takes 4.5 seconds to flush on a NVMe...

Hyperscalers have hyperscalers issues (of around a terabyte of memory). While my desktop is not going to have a terabyte of memory any time soon, the problems of the HPC and Hyperscalers today will be seen on workstations in a decade or less, so improvements today are beneficial.

[onlyLinuxLuvUBack]

echostream packs 108 nvme drives in a 2u server...

https://echostreams.com/products/flachesan2n108m-un

[cl333r]

I just realized that with this patch if I buy a second NVMe my PC will shutdown 4.5 seconds faster. So it's not just about cloud servers.

[F.Ultra]

Umm. But WHY is it taking 4.5 seconds?
There is nothing that should take 4.5 seconds on an NVMe drive.
Even if it has a power hold-up cache, it would be flushed rather immediately.
Buggy drive or bugged specification?

AFAIK it can't be anything else than a write back cache that causes the NVMe driver to wait so the question you should ask yourself is what crazy sized write back caches that Google uses that takes 4.5 seconds to flush on a NVMe...

[Developer12]

Google almost certainly have more NVMe drives per server than they're letting on. 16 is probably just enough to cross the 60 second threshold. They're a hyperscalar, they pack things as densely as possible.

At the same time, this may well be a holdover from their previous generation of servers. It's just only now finally working is way from their prod-kernel into the mainline.

[logical]

Ah remember the old days IBM p43 servers booting just within 7 minutes. Those fond memories.

How about an IBM RS/6000 J40 (microchannel, PowerPC) A full boot with 8 processors took about 45 minutes. A quick boot was 15 minutes. :-)

[Vistaus]

Wow, so many comments and nobody is wondering why few seconds matter. After all you'll reboot a desktop only once per day, servers have uptime of months. So shouldn't matter even if it takes few hours to boot.

I reboot my desktop only after a kernel update or if otherwise deemed necessary.

[dekernel]

Any rolling release will push updates requiring a reboot on an almost daily basis.

Good point. I guess I just don't equate "servers" and "rolling releases" in the same sentence for just such reasons.

[bug77]

Guess I would be interested in knowing just what OS you are using that requires daily updates because even Windows 2003 could go a full week before required reboots from the Windows Updates.

Any rolling release will push updates requiring a reboot on an almost daily basis.