EXT4 Case-Insensitive Directories/File-Name Lookups Coming With Linux 5.2

[bison]

Can we opt out of windows crap?

I hope so, because I routinely name my makefiles "Makefile", and at times make a copy named "makefile" for temporary use, which "make" will read before "Makefile". When I'm done with the copy I just delete it and I'm back to where I was without having edited the project makefile.

As a general rule it seems that a lot of "improvements" manage to damage my workflow.

[starshipeleven]

Can we opt out of windows crap?

Even better, it's opt-in and can be enabled for files in specific directories.

[skeevy420]

For the love of Zod, DO NOT install on 5400rpm hard drives (or NAS drives like WD red, which are 5400 drives too).

They shouldn't even be created anymore, but they are what come with laptops, game consoles, and eBay specials.

That swap platter location trick does help with low memory craptops and is something to keep in mind when using any spinning rust.

[discordian]

This is kind of on the right track but the problem is unicode is screwed.

test=𝚝𝚎𝚜𝚝 is the same right?? sorry the answer would be no. Yes I could have picked closer https://lingojam.com/WeirdTextGenerator does show you a lot of the different unicode ways a simple bit of text could be.

So you may want to limit what chars users can use in file names to avoid insanity. The worst fact is that different styles of same letter have different unicode numbers.

Yeah, and AFAIK the ligatures stuff is even worse, the point is should the kernel support various (potentially changing in the future) usecases (like windows, like Standard XY),
or should this insanity be limited to applications?

Unicode covers more then "logical descriptive text" (ie. without styles). The insanity is to stuff a MS Word text search into the filesystem's name lookup.

The term that is used is casefold. Casefold can be case-insensitivity it can also be you use like two versions of x lower case with different unicode numbers and due to unicode being limited by the casefold set this becomes 1 predictable x char.


Yes I would kind of like this as userspace but not the way you are thinking.


Part of this process is creating the metadata slot to be in the VFS for this information.

Problem here is like interposing the libc open function really does not work when the applications can just syscall kernel open function instead. To do case-insensivity/casefold(case fold covers extra issues) you need it kernel level intercept this is not avoidable if you really want to enforce it. Should the casefold processing be in kernel space or should it be a usermode mode helper to the kernel.

I would not want to be this transparently handled in the kernel. Want a samba share?
Either get samba doing the normalization and metadata-storage for all operations (and just tell the user to not write directly to the hosting directory),
or mount a FUSE filesystem transparently doing that dirty work.

All usecases will have some specific requirements this doesnt scale if stuck into a kernel filesystem.
What if I want to host read-only from a squashfs?

https://www.kernel.org/doc/htmldocs/kernel-api/API-call-usermodehelper-setup.html

With the mess of unicode I would like to see the case fold char processing as a user mode helper so it can be updated without updating the kernel. So still a kernel patch but done differently to current form.

Some would think this is coping OS X and Windows the reality is the Unicode system is stuffed and filter is required to have sanity.

Would this usermode helper compare each filename for a match using the complex rules?
Cause if that's the case, using a FUSE on top that normalizes the filename and stores normalized filenames on any underlying filesystem (supporting extends or similar metadata),
would likely be an order of magnitude faster for lookups.

[Azrael5]

You would mention an APU... or yet another thing that takes ram away from the system...you can configure it to use less ram in the BIOS/UEFI, but that really won't fix your underlying problems.

For starters -- use zswap if you aren't already because it's the closest thing we have to downloading more ram; add the following to your kernel command line -- "zswap.enabled=1 zswap.compressor=lz4 zswap.max_pool_percent=25 zswap.zpool=z3fold". The only one you may need to tune is the max_pool_percent; 10 to 25 percent are sane values...leave the rest the same.

Tweaking your swappiness may improve things for you. Create "/etc/sysctl.d/10-vm_swappiness.conf" and add the line "vm.swappiness = 10" The lower the number, the longer your system will hold off on using the swap. IMHO, don't go below 10 unless you're in the "assloads of ram" category.

Since it's KDE, check that Baloo isn't running (balooctl status), and then disable it if it is (balooctl stop && balooctl disable). Baloo is handy, but it is also a resource killer until it does its thing...the problem is Baloo Scanning + HDD = Suckfest for days.

In regards to /tmp, the Arch Wiki has a nice write up on tweaking it to your needs (edit /etc/fstab, use a size= option, limit it to 512mb to 1gb).

Read ssokolow's post 35. It has some good information to help you.

If I were you I'd remove the swap space from the HDD and put a new swap space on a USB drive so the OS HDD doesn't become overloaded with swap and OS crap when the system is under stress since it seems like OOM then HDD Swap is the root cause of your problems (especially if Baloo is doing its thing).

One other thing to consider is you disk io scheduler. With spinning rust...noop, none, or deadline...in my experiences, don't bother with any of the rest since they mainly help with flash and SDDs. On 5.0.9 with BTRFS and ZFS, I use none on my spinning rust. Any of the other available disk schedulers reminds me of the Pentium 2 days (seriously, it's that bad). For me, setting it to "none" is the difference between double clicking a song in Clementine and having it play instantly to having a 3-5 second lag before it plays.

And, honestly, if none of that helps then you really have three choices left:

1) Get more ram
2) Get an SSD for the OS
3) Both 1 & 2

Oh, this is a bit of an old-school tweak I don't see much anymore but it really does help -- when you partition your drive, put your swap space on the inner part of the platter so it's on the fastest-read part of the HDD. That really isn't an option for you now, but it is something to keep in mind with future OS installs on HDDs (especially 5400rpms).

thanks very much for your suggestions, I appreciate it. What seems absurd is that none developer pay attention to this problem, so the user has to apply workarounds on his own. It's really annoying and absurd. User can think that if no developer is able to fix it how can the user replace a developer!?

[Azrael5]

The system is moving RAM to disk to "make more space" in RAM, which kinda sucks in the modern day with modern RAM sizes.

Remove swap file or remove the swap partition entry from the /etc/fstab.

Next time you get into low memory situation the system will kill applications to free ram instead of freezing and crashing.

Thanks, so the swap is the responsible of the mess.

[oiaohm]

Yeah, and AFAIK the ligatures stuff is even worse, the point is should the kernel support various (potentially changing in the future) usecases (like windows, like Standard XY),
or should this insanity be limited to applications?

The problem here is if you limit to applications how are all applications on the same settings.

I would not want to be this transparently handled in the kernel. Want a samba share?
Either get samba doing the normalization and metadata-storage for all operations (and just tell the user to not write directly to the hosting directory),
or mount a FUSE filesystem transparently doing that dirty work.

Samba doing the normalization runs into issues when native programs access the folders and put files in that should not be there.

Fuse ends up with very high overhead and lots of structure duplication and memory consume. You still have the problem if the file system gets modified behind fuses back with incompatible files..

All usecases will have some specific requirements this doesnt scale if stuck into a kernel filesystem.
What if I want to host read-only from a squashfs?

The same casefold system that ext4 is using xfs will be using. So its not suck into a single kernel file system any more. The meta data saying this is normalised could be added to squashfs in time.

Would this usermode helper compare each filename for a match using the complex rules?

This is absolutely what you don't want to.

Cause if that's the case, using a FUSE on top that normalizes the filename and stores normalized filenames on any underlying filesystem (supporting extends or similar metadata), Would likely be an order of magnitude faster for lookups.

This does not work. But it close to what you are after. Since fuse in this is on top of the file system not the file system an application could write non normalised into a directory expecting normalised.

What you are after todo.
1) Normalise each file name string once.
2) Have the normalised in the VFS lookup hash table.

What I would have in the usermode helper is the normalise due to changes in unicode this is likely to change over time.

If you cannot make sure the stored file names in a particular location are normalise you are force to perform a normalisation check every time you search the directory this is why samba and fuse solutions are horrible cpu heavy because they are doing "compare each filename for a match using the complex rules" and they don't have a choice in the mater.

So yes putting this into the file system as proper file system meta data and storage limitations that applications cannot bypass and screw up is more than an order of magnitude faster than samba or fuse solution. The fact we have xfs and ext4 so far we can expect more file systems to pick-up the casefold support. Of course the file systems that don't you can still use the crappy low performance samba/fuse solutions.

[MoonMoon]

Some Linux users: Linux is all about choice. Of course not the choice to have parts of the filesystem case-insensitive, only the choices I deem to be OK.

[starshipeleven]

Some Linux users: Linux is all about choice. Of course not the choice to have parts of the filesystem case-insensitive, only the choices I deem to be OK.

Translating it to meme for the sake of younger generations.

someone: (making the wrong choices)

some linux users: *TRIGGERED*

[skeevy420]

Some Linux users: Linux is all about choice. Of course not the choice to have parts of the filesystem case-insensitive, only the choices I deem to be OK.

Pretty much that.