Linux 5.11 Adding VirtIO-MEM "Big Block Mode"

[phoronix]

Phoronix: Linux 5.11 Adding VirtIO-MEM "Big Block Mode"

The previously covered VirtIO memory (VirtIO-MEM) work on its Big Block Mode "BBM" will be landing with Linux 5.11...

http://www.phoronix.com/scan.php?page=news_item&px=Linux-5.11-VirtIO-Big-Block-Mode

[Jabberwocky]

" For example, gigantic pages in the hypervisor result on x86-64 in a device block size of 1 GiB"

Could someone expand on a user story for this?
Real world scenario...I've read the patch posting and previous article...I'm a little lost to be honest..

Honestly I have no idea. I did a few calculations and could not find a valid reason for this. You probably need more than 1TiB RAM and hosting mirco-services/lambda-functions for this to be an limitation, actually even with 1TiB RAM it should still be difficult to hit the max mappings.

Here follows my estimations:
Host/Linux mem block: 128 MiB
Host mem size: 1 TiB
Max memory mappings: 32k (for conservative estimation lets say each entity has another mapping for disk)

Total mem / (max mem mappings * host mem block size)
1099511627776 / ( 32000 * 134217728 ) = 0.256

That number is way too low to impact any services that I have worked with. Perhaps my understanding is wrong or there are much more mappings per entity (like network/usb/gpu).

All I know is that this won't impact my use cases any time soon. I welcome over-engineered software when I don't have to pay for it. Just wondering why it's important to someone else.

[davidhildenbrand]

Hi,

in general, virtio-mem prefers small granularity (which was the primary first use case), however, there are two main use cases in the future to be covered:

1. Using gigantic pages via hugetlbfs in the hypervisor. Gigantic pages (e.g., 1 GB) can increase VM performance by reducing the number of TLB entries, similar to huge pages (e.g., 2 MB). Some information can be had from https://pvk.ca/Blog/2014/02/18/how-b...-1gb-pages-be/. For now, using them with virtio-mem resulted in Linux guests not being able to make us of such memory. IIRC, with VFIO, gigantic pages can also help mapping bigger physical chunks into the IOMMU (e.g., the huge page size in IOMMU tables might differ to the huge page size used by the CPU).

2. Limiting the number of mappings in the hypervisor. You are right that with 32..64k mappings (e.g., VFIO), it won't be an issue with reasonably small VMs. However, in the future

a) We cannot assume that virtio-mem can consume that many mappings in the future. We might have other consumers of a lot of mappings (e.g., virtio-fs in DAX mode might be one candidate where this issue just popped up recently when resizing files).

b) With VFIO, we don't want to have that many DMA mappings. Creating each separate mapping is expensive, managing each mapping is expensive. Each mapping comes with memory overhead in the Linux kernel. The Linux kernel might actually be configured to provide less that 64k max DMA mappings.

c) We might have very large VMs. More stuff for the far instead of near future.

Long story short, it won't affect setups that don't care about VFIO and that don't care about gigantic pages - that is, no existing setups. (Note: BBM is actually much simpler than SBM implementation wise)

Please let me know if you have further questions, cheers.