EXT4 Case-Insensitive Directories/File-Name Lookups Coming With Linux 5.2

[FastCode]

Can we opt out of windows crap?

[Azrael5]

Because you've posted this in multiple threads now:

Care to list your system specs? It's extremely doubtful that stuff like duplicated kernel timers and case insensitivity are the cause of your slowdowns.

I only ask because I haven't really had any day to day, simple activity overloads since my Athlon 64 x2 with 2gb DDR2 or Q6600 with 3GB DDR2 days. Once I put 8GB in that Q6600 12 years ago, that has been a thing of the past outside of the HDD slowdowns and "the web browser has yet another memory eating bug".

Personally, I consider a Core2+ CPU with 4 threads and 8GB to be the minimum one should use for a Gnome or KDE desktop. Yeah, my KDE desktop uses less than 600mb on boot...but not once I start doing stuff. I should add that one has to change the default distribution values to make that work, like maybe needing to enable zswap, limit the size of /tmp, ZFS ARC limiting, lower the swap value; things like that. Sometimes I think the Linux devs forget about the average day-to-day computers when I see things like systemd ganking half the ram for /tmp.

The reason I say all that is that I'm at 4.9GB usage right now with Firefox with 5 tabs open (one is STiscovery on pause), PulseEffects, SC-Controller, Steam, & Lutris and, if this were an 8GB system and I a noob, I'd be using swap space since Manjaro/systemd has /tmp setup to use half of the system's available ram by default. ZFS on my system can use up to another half of the remaining ram. Someone with an 8GB system using systemd only has 4GB available to the OS. Add in ZFS and your down to 2GB available until ZFS frees up the ARC which will then cause drive slowdowns...can't win for losing there.

All that said, I do have 48GB of ram, 16GB allocated to the OS, so there is a really good chance that Firefox, other programs, the kernel in general, all aren't being as conservative on my system as they might on a lower ram system. Every tweak I mentioned above is in use on my system outside of limiting /tmp.

Yes. My system has 8GB of RAM and a dual core APU.... I use KDE Neon. IF 7GB of RAM are occupied the mess begins. All the OS crashes because of a loop in the hard drive which continuously works for no reason.

[starshipeleven]

Yes. My system has 8GB of RAM and a dual core APU.... I use KDE Neon. IF 7GB of RAM are occupied the mess begins. All the OS crashes because of a loop in the hard drive which continuously works for no reason.

The system is moving RAM to disk to "make more space" in RAM, which kinda sucks in the modern day with modern RAM sizes.

Remove swap file or remove the swap partition entry from the /etc/fstab.

Next time you get into low memory situation the system will kill applications to free ram instead of freezing and crashing.

[oiaohm]

I get storing the encoding for some filesystems, although with ext4 (and other fs exclusive to *nix) this could have just been fixed to UTF8 with a small revision.

This is kind of on the right track but the problem is unicode is screwed.

test=𝚝𝚎𝚜𝚝 is the same right?? sorry the answer would be no. Yes I could have picked closer https://lingojam.com/WeirdTextGenerator does show you a lot of the different unicode ways a simple bit of text could be.

So you may want to limit what chars users can use in file names to avoid insanity. The worst fact is that different styles of same letter have different unicode numbers.

What I wont ever get is what purpose languages and case-insensivity have lost in the kernel.

The term that is used is casefold. Casefold can be case-insensitivity it can also be you use like two versions of x lower case with different unicode numbers and due to unicode being limited by the casefold set this becomes 1 predictable x char.

Have problems with different UTF8 encodings denoting the same human interpretation -> do normalization in userspace
Have problems with case -> do normalization in userspace (to either lower or upper case)

Yes I would kind of like this as userspace but not the way you are thinking.

Want to still save some CamelCase or special kind encoding for display to the user -> save that in an extent (or whatever metadata extension is available)

Part of this process is creating the metadata slot to be in the VFS for this information.

Apply that to all filenames (like interposing the libc open function) and you don't end up with any kind of race, quick byte-compare lookups and alot less baggage in the kernel.

Problem here is like interposing the libc open function really does not work when the applications can just syscall kernel open function instead. To do case-insensivity/casefold(case fold covers extra issues) you need it kernel level intercept this is not avoidable if you really want to enforce it. Should the casefold processing be in kernel space or should it be a usermode mode helper to the kernel.

https://www.kernel.org/doc/htmldocs/kernel-api/API-call-usermodehelper-setup.html

With the mess of unicode I would like to see the case fold char processing as a user mode helper so it can be updated without updating the kernel. So still a kernel patch but done differently to current form.

Some would think this is coping OS X and Windows the reality is the Unicode system is stuffed and filter is required to have sanity.

[ssokolow]

The system is moving RAM to disk to "make more space" in RAM, which kinda sucks in the modern day with modern RAM sizes.

Remove swap file or remove the swap partition entry from the /etc/fstab.

Next time you get into low memory situation the system will kill applications to free ram instead of freezing and crashing.

That won't necessarily fix it. I have swap disabled but my SSD-less system still thrashes to a stand-still before triggering the OOM killer when my 16GB of RAM gets into the range above 90% utilization because there isn't enough "unused" RAM to act as a disk cache.

The solution I settled on was a daemon called earlyoom which starts killing processes (at a configurable point and with configurable preferences for which kinds of processes get killed first or must not be killed) before the RAM gets too full.

(I set it to preferentially kill Firefox and Chrome content processes since they're so easy to restart and amended its list of things to never kill which came with things like the X server already defined as --avoid in my /etc/default/earlyoom.)

[oiaohm]

That won't necessarily fix it. I have swap disabled but my SSD-less system still thrashes to a stand-still before triggering the OOM killer when my 16GB of RAM gets into the range above 90% utilization because there isn't enough "unused" RAM to act as a disk cache.

At this stage having swap disabled with Linux kernel is not recommend. You might be using a zram or the like for a ram based swap instead of a disc one. You can in fact lose a few GB of memory into slub allocated and other data structure fragmentation that the Linux kernel needs swap to defrag and compact at this stage. So no swap might be making your problems little worse than they should be. This was covered in a Linux plumbers conference video and a LCA video. There is work to one day properly fix these problems but until they are the kick problem structure to swap while we replace it or move it to another memory location is a Linux kernel used hack work around so the result of disabling swap is disable the work around.

I do agree that the earlyoom is a good idea. There is work at moment look at changing how the oom killer in kernel works so it triggers sooner instead of waiting until you are completely in screwed.

[reavertm]

Why would anyone want this?

To set it for SteamApps dir for instance, to have game mods work out of the box for games like new XComs or Total War series.

To make it practical, it should narrow down to ASCII probably, otherwise those Unicode tables...
Only reason I play those games on Windows, too lazy too fix mods for case sensitivity. And setting up NTFS just for SteamApps will hurt performance.

[skeevy420]

Yes. My system has 8GB of RAM and a dual core APU.... I use KDE Neon. IF 7GB of RAM are occupied the mess begins. All the OS crashes because of a loop in the hard drive which continuously works for no reason.

You would mention an APU... or yet another thing that takes ram away from the system...you can configure it to use less ram in the BIOS/UEFI, but that really won't fix your underlying problems.

For starters -- use zswap if you aren't already because it's the closest thing we have to downloading more ram; add the following to your kernel command line -- "zswap.enabled=1 zswap.compressor=lz4 zswap.max_pool_percent=25 zswap.zpool=z3fold". The only one you may need to tune is the max_pool_percent; 10 to 25 percent are sane values...leave the rest the same.

Tweaking your swappiness may improve things for you. Create "/etc/sysctl.d/10-vm_swappiness.conf" and add the line "vm.swappiness = 10" The lower the number, the longer your system will hold off on using the swap. IMHO, don't go below 10 unless you're in the "assloads of ram" category.

Since it's KDE, check that Baloo isn't running (balooctl status), and then disable it if it is (balooctl stop && balooctl disable). Baloo is handy, but it is also a resource killer until it does its thing...the problem is Baloo Scanning + HDD = Suckfest for days.

In regards to /tmp, the Arch Wiki has a nice write up on tweaking it to your needs (edit /etc/fstab, use a size= option, limit it to 512mb to 1gb).

Read ssokolow's post 35. It has some good information to help you.

If I were you I'd remove the swap space from the HDD and put a new swap space on a USB drive so the OS HDD doesn't become overloaded with swap and OS crap when the system is under stress since it seems like OOM then HDD Swap is the root cause of your problems (especially if Baloo is doing its thing).

One other thing to consider is you disk io scheduler. With spinning rust...noop, none, or deadline...in my experiences, don't bother with any of the rest since they mainly help with flash and SDDs. On 5.0.9 with BTRFS and ZFS, I use none on my spinning rust. Any of the other available disk schedulers reminds me of the Pentium 2 days (seriously, it's that bad). For me, setting it to "none" is the difference between double clicking a song in Clementine and having it play instantly to having a 3-5 second lag before it plays.

And, honestly, if none of that helps then you really have three choices left:

1) Get more ram
2) Get an SSD for the OS
3) Both 1 & 2

Oh, this is a bit of an old-school tweak I don't see much anymore but it really does help -- when you partition your drive, put your swap space on the inner part of the platter so it's on the fastest-read part of the HDD. That really isn't an option for you now, but it is something to keep in mind with future OS installs on HDDs (especially 5400rpms).

[starshipeleven]

That won't necessarily fix it. I

On OpenSUSE Tumbleweed it works fine as long as I remove the swap.
I reached points where I was using so much RAM with VMs that it killed Thunderbird for example (OOM killer: sacrificing children... how cool is that)

Maybe you need to configure the memory settings to get the OOM killer to start acting sooner?
I remember I had to do this for Android. I don't remember offhand what I had to change as it's long since I upgraded to a better smartphone.
see this for the docs on the type of settings I did change https://www.mjmwired.net/kernel/Docu.../sysctl/vm.txt

Googling seems like setting vm.admin_reserve_kbytes=262144 would help.

[starshipeleven]

future OS installs on HDDs (especially 5400rpms).

For the love of Zod, DO NOT install on 5400rpm hard drives (or NAS drives like WD red, which are 5400 drives too).