Intel Optimization Around Batched TLB Flushing For Folios Looks Great

[sinepgib]

Yes i know that x86-64 cpus can do 4k and 2m pages, now if i assume every pages occupies a bit in the tlb 2048/4 you save 512bit 64byte or is this a brain fart ? for 64 bit if you go 64 bit it´s 8x8 so every address would need 8 byte so one pointer for a 4 k page is 8 byte and then that * 512 is 4096 byte i realy dont understand that bitmap stuff, and that does not explain how you fold lets say 20 4k pages into a folio and save ram there that would mean you need 160 bytes but you actually point it to one 8 byte header ? but where is the info to split those 20 pages after that so it cant be a 19 times compression it needs some info somewhere thats why i asked how ?

edit 20*8 is actually 160 byte that would mean 20 pointers vs 1 thats a significant reduction but how do you split the 20 pages after that ????

edit2 if i read my own post i c that from a 4k page to 2m page its probably better to use 8 byte for 2m then to use 8 byte for 4k you save 4088 bytes allmost a hole page

There's some overhead beyond the pointers themselves. The mappings of virtual to physical addresses are stored in a page table that has several levels as a trie. Leaves are called page table entries, and one of the things they store is whether or not the page is "huge". Because it's tied to the MMU you only need that single bit to know whether the mapping talks about the next 4k or the next 2M. This also means you have more overhead than just a pointer per page. I don't know from memory how much space it takes, but stuff like access permissions are also stored here, etc. The trie structure compresses this overhead a bit tho, if the virtual addresses share a prefix (say, the whole of your stack will have the same prefix, and the heap if you use sbrk I think will do as well) you don't need to repeat it for every page.
IIRC the TLBs just mirror the page tables, so it should be exactly the same amount of saving that you get in main memory.

In principle you don't split them. To split them if you need to you'll have to basically recreate the mapping with the smaller pages. Otherwise, it's just a bigger block.

[Times Two]

Phoronix: Intel Optimization Around Batched TLB Flushing For Folios Looks Great

A patch worked on by an Intel engineer for batched TLB flushing for page migration with folios is showing some promising results and currently working its way to the mainline kernel...

https://www.phoronix.com/news/Migrat...ch-TLB-Flush-F

WTF is this? Batched TLB flushing for folios? How many people in the world know what "page folios" are? Yeah right, 20, 100, 400? Something like that. This is an Intel private concept that is not defined in a clear, easily found or understanding manner anywhere on the net. Granted, it's no worse than many of the profligate NAMES and 3-letter acronyms that apparently everyone beside myself was conscious of at birth. My bad I guess. But really, I'm a technically knowledgeable person that reads and at times comments on sites such as phoronix and realworldtech, but I think I'm understandably annoyed by the bandying about of extremely uncommon 3 letter acronyms and things with longer names that are essentially corporate obfuscation. All without the slightest courtesy of pointing one, at initial reference, to what the thing, the thing, being talked about is.

Yes, yes, I now have a fair idea about what the referenced "folios" are, but still, this is just ridiculous. How many other phoronix readers have wasted x-minutes or hours into figuring out what this might possibly be, in general (which would be nice) but also possibly in the specific (seeing as that seems to play a not insignificant part).

[milkylainen]

WTF is this?

Dude, relax. Be happy that you learnt something new.
Apparently it was interesting enough to make you peek into the subject.

[milkylainen]

I guess I'll have to re-read both folios and the batched flush in detail to be sure. But I would assume folios are just a representation of behavior, rather than a change in behavior itself, i.e. they represent continuous pages in places that already asked for continuous pages, not cause places to expect those.

Mmm. Probably so. Anyway, I was just curious.
So many system/architectural questions regarding batch handling of compound pages.
Thanks for answering.

[sinepgib]

WTF is this? Batched TLB flushing for folios? How many people in the world know what "page folios" are? Yeah right, 20, 100, 400? Something like that. This is an Intel private concept that is not defined in a clear, easily found or understanding manner anywhere on the net. Granted, it's no worse than many of the profligate NAMES and 3-letter acronyms that apparently everyone beside myself was conscious of at birth. My bad I guess. But really, I'm a technically knowledgeable person that reads and at times comments on sites such as phoronix and realworldtech, but I think I'm understandably annoyed by the bandying about of extremely uncommon 3 letter acronyms and things with longer names that are essentially corporate obfuscation. All without the slightest courtesy of pointing one, at initial reference, to what the thing, the thing, being talked about is.

Yes, yes, I now have a fair idea about what the referenced "folios" are, but still, this is just ridiculous. How many other phoronix readers have wasted x-minutes or hours into figuring out what this might possibly be, in general (which would be nice) but also possibly in the specific (seeing as that seems to play a not insignificant part).

1. This is not at all Intel specific, nor is this particular 3-letter acronym specific to any architecture. TLBs are a quite basic concept: they are just a cache for page tables that all modern chips have! Folios are simply a type in the Linux kernel that is also used regardless of architecture. They are also named the same for all of them.
2. If you consider learning these concepts a waste of time, you are neither "technically knowledgeable" nor even interested. And it's OK, as long as you don't claim to be and complain that people with actual technical interests get news about it.
3. If you don't like to look for it, just don't. Nobody forces you. You can skip articles, you know?

[sinepgib]

Mmm. Probably so. Anyway, I was just curious.
So many system/architectural questions regarding batch handling of compound pages.
Thanks for answering.

If you liked it, you could remind me in ~2 weeks (I'm really overworked until then so I don't have the time nor the energy) to look up the details in the Intel manual for you
I would suggest you look for it, but it's quite opaque if you didn't already study some of the concepts and I don't want to make assumptions. On the batching I'll seriously need to re-read the article. I did read the folios part and indeed it's just letting the type system know it will deal with compound pages beforehand to skip some special casing and what not, but it shouldn't change behavior other than by reducing branching.

[sinepgib]

OK, I'm not 100% sure I got it right, but I think this should actually help latency because migration would not be considered complete until all pages in the folio got flushed, and flushing them one by one means the latency cost is paid once per page. The batch makes it so you do one flush for the whole folio IIUC, paying that cost only once. This is about migrating a process between NUMA nodes BTW, where each processor has its own TLB and gets invalidated during the migration due to (I think, not an expert in NUMA) the contents of the memory being moved to a different node causing changes on the page table. But that's at least how this looks to me.

EDIT: correction, flush is per whole folio, and it's whole folios that get batched. The proper explanation is here: https://lore.kernel.org/all/20230116...dd3048f37d329d
But still, essentially this will reduce latency, at least in average, since once you migrate the process it will mostly run on another socket and the quicker you are to move the pages the quicker you are to stop paying the latency toll of accessing a node not owned by that CPU.

[Times Two]

Dude, relax. Be happy that you learnt something new.
Apparently it was interesting enough to make you peek into the subject.

I'm actually not all that agitated and I take all of Michael's fantastic work with great seriousness and gratitude. But Intel "folios" or "page folios" shouldn't be bandied about without links for at least the first decade of their existence. As far as I could see there's no clear and concise definition of WTF this is. After spending some time on the matter it seems to me that such a definition is eminently achievable, but still not forthcoming from this company (Intel) of 120K employees. Fail!

[Times Two]

1. This is not at all Intel specific, nor is this particular 3-letter acronym specific to any architecture. TLBs are a quite basic concept: they are just a cache for page tables that all modern chips have! Folios are simply a type in the Linux kernel that is also used regardless of architecture. They are also named the same for all of them.
2. If you consider learning these concepts a waste of time, you are neither "technically knowledgeable" nor even interested. And it's OK, as long as you don't claim to be and complain that people with actual technical interests get news about it.
3. If you don't like to look for it, just don't. Nobody forces you. You can skip articles, you know?

The specific "folios" here are all very, very Intel specific. Seems to me that Intel's concept is fairly general and could be brought over to general use, but really, nobody's heard of if it before, not in any real sense of the word heard.

PS. I'm LMAO, from your kind guidance regarding TLBs and the like, but I also truly, truly appreciate your advice in items 2 & 3. Hahahaha!

[sinepgib]

The specific "folios" here are all very, very Intel specific. Seems to me that Intel's concept is fairly general and could be brought over to general use, but really, nobody's heard of if it before, not in any real sense of the word heard.

https://lwn.net/Articles/849538/

Not Intel specific. It took me about 10" to Google it, so you can too. Even the affected region is not Intel specific, but shared memory management. They are new, that much is true, but even the author doesn't work for Intel according to kernel newbies, but Oracle. Research before ranting, or follow advices 2 and 3.