Ceph Sees "Lots Of Exciting Things" For Linux 5.3 Kernel

[Nite_Hawk]

Thanks for the pointers. SuperMicro's been understandably putting EPYCs in their NVMe storage servers (for the incredible 128 PCI-E lanes), so I think we'll be good to go.

The new rome chips look like they are going to have a lot of potential!

The concept is to have many small, easy-to-replace, easy-to-scale-in-increments parts, hence the 1U form factor. These usually have 10 NVMe drives. You need more IOPS or space? Just add one more node and be done with it

After a quick glance something between EPYC 7251 and 7351 should do, Intel P4610 SSDs, plus 128-ish GB RAM, but picking the CPU is like stabbing in the dark, for RAM I still need to read through the Ceph planning guides. Any suggestions? Our current choice is AS-1113S-WN10RT, Thinkmate has a configurator for the parts and disks.

Once a couple of my PRs land, it looks like we can drive about 40-45K 4K randwrite IOPS using about 10 2.3GHz broadwell era xeon cores from a single P3700 NVM drive. Optanes potentially will do better, but it's the CPU usage that we need to get down to make use of 10 NVMe drives in a single 1U node. We're getting closer though and there's a lot of really good work going on now so I'm excited to see how all of these things come together.

For networking we'll prolly settle with some Mellanox Connect-X dual 100G card and a 32x100G switch from FiberStore.

I've got an older Mellanox 40GbE switch that has been working well in the lab. No complaints with it.

[anarki2]

The new rome chips look like they are going to have a lot of potential!

Once a couple of my PRs land, it looks like we can drive about 40-45K 4K randwrite IOPS using about 10 2.3GHz broadwell era xeon cores from a single P3700 NVM drive. Optanes potentially will do better, but it's the CPU usage that we need to get down to make use of 10 NVMe drives in a single 1U node. We're getting closer though and there's a lot of really good work going on now so I'm excited to see how all of these things come together.

I've got an older Mellanox 40GbE switch that has been working well in the lab. No complaints with it.

Any suggestion for distro? RHEL/CentOS seems an obvious choice, but their ancient kernel versions make me wonder.

[Nite_Hawk]

Any suggestion for distro? RHEL/CentOS seems an obvious choice, but their ancient kernel versions make me wonder.

If you can hold out for another month or two CentOS 8 will be coming out soonish:

https://blog.centos.org/2019/07/centos-community-newsletter-july-2019-1907/#centos8

EPEL will probably take longer though.

[anarki2]

If you can hold out for another month or two CentOS 8 will be coming out soonish:

https://blog.centos.org/2019/07/centos-community-newsletter-july-2019-1907/#centos8

EPEL will probably take longer though.

Yes, I know, but that kernel's already 1 year old, and it will stay that way for a good 2-3 years Do RH backport Ceph improvements into EL?

[Nite_Hawk]

Yes, I know, but that kernel's already 1 year old, and it will stay that way for a good 2-3 years Do RH backport Ceph improvements into EL?

I think so, but frankly after working for RH for 5 years I still don't understand how kernel backport decisions are made. Not my area really though. I did hear a rumor that io_uring is going to make it into RHEL8/CentOS8 at some point, but I don't know if that's going to be in the form of a backport or a kernel upgrade.

[anarki2]

I think so, but frankly after working for RH for 5 years I still don't understand how kernel backport decisions are made. Not my area really though. I did hear a rumor that io_uring is going to make it into RHEL8/CentOS8 at some point, but I don't know if that's going to be in the form of a backport or a kernel upgrade.

Thanks a lot for all the info, appreciated