Announcement

Collapse
No announcement yet.

EXT4 Getting Faster Case-Insensitive Performance

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • phoronix
    started a topic EXT4 Getting Faster Case-Insensitive Performance

    EXT4 Getting Faster Case-Insensitive Performance

    Phoronix: EXT4 Getting Faster Case-Insensitive Performance

    The Linux 5.2 kernel brings optional per-directory case-insensitive filenames/folders while with the Linux 5.3 kernel that new EXT4 feature will see better performance...

    http://www.phoronix.com/scan.php?pag...nsitive-Faster

  • oiaohm
    replied
    Originally posted by xfcemint View Post
    Case-sensitive comparison is equally fast as case insensitive (more precisely, the difference is negligible when used in file-system operations.
    https://lwn.net/ml/linux-fsdevel/201...collabora.com/
    No case-sensitive comparison at file system driver level is in fact slower than case-sensitive you have a longer code path. If what you are doing is like a Linux kernel build having the file system folder where it is as case insensitive does hurt by taking longer.

    Originally posted by xfcemint View Post
    It is not fine (to emulate case-insensitive file-open by retrieving a list of all files in a directory) because:
    - it is slower
    - it requires extra code and testing
    This is true at kernel level as well. Just not as much as processing full list in userspace.

    Originally posted by xfcemint View Post
    - requires code duplication
    Kernel level avoid code duplication this is the advantage of doing in kernel space is you don't have to duplicate up the hash table and this table can be cached between runs.

    Originally posted by xfcemint View Post
    - requires extra error checking, memory allocation and other things which experts know very well about, but amateur's don't
    - requires extra case handling (when two synonymous files exist in the same dir).
    These is true at kernel level as well. Casefold at kernel level requires extra memory allocation and extra checking.
    Bugs in file system means two synonymous files may exist so true at kernel level.

    Originally posted by xfcemint View Post
    - etc, etc...
    You did not mention these. I would suspect most of them effect kernel level as well. There is very little difference in issues between implementing in kernel space and implementing in userspace. Duplicate processing issue/Duplicated code is the fairly much the total performance hit difference between the user space and kernel implementation. All the other performance/memory hits from being case insensitive other than the Duplicate processing issue/Duplicated code stay the same when you move the processing of case insensitive from user space to kernel space.

    Originally posted by xfcemint View Post
    In short, it is not fine because it is not an elegant solution. An elegant solution is to provide this functionality on the kernel level.
    Yes it might be more elegant for the kernel to provide case insensitive over userspace. But case insensitive is not a free no matter where you implement it. Cheapest in performance cost is always case sensitive.

    The advantage of implementing case insensitive in kernel space is avoiding some duplication and race condition problems. The avoid some duplication increases performance but it does not change the fact that case insensitive processing is still slower than case sensitive processing even when in kernel space.

    Case insensitive search is always a longer code path than a case sensitive one. The reason why Linux kernel developers resisted adding case insensitive support for so long is the fact it adds extra code into file system layer to be debugged and harms performance. Of course those emulating case insensitive in user-space being wine and samba were being hit worse.

    Leave a comment:


  • xfcemint
    replied
    Originally posted by starshipeleven View Post
    Still true, even if it's "more efficient" for your own usecase it's still not as efficient as case-sensitive and applications that don't need that (the vast majority) are getting hit for no reason.
    Case-sensitive comparison is equally fast as case insensitive (more precisely, the difference is negligible when used in file-system operations.
    But you should have known that. Only amateurs don't know this. So, perhaps, you just don't know what you are talking about here.
    Originally posted by starshipeleven View Post
    Only applications actually having issues with that are Samba and Wine stuff. But there is no reason to do that with a new application.
    If some applications are having issues with it today, than it is reasonable to expect that there might be future applications having the same problem.
    Wine is not old. It is a modern app. The problem is that there is no other way to do it, so Wine will suffer as long as this functionality is missing from the kernel.

    Maybe you are suggesting to just delete Wine and Samba? Or an even better solution: don't use Linux at all, and your problems are solved!

    Originally posted by starshipeleven View Post
    You aren't making much sense then, is your application requiring the user to set this filesystem/folder option on all folders he keeps his documents and stuff just because your application needs to find his stuff?
    Oh, you don't even understand how it's supposed to work? Do you always take strong positions about things you don't understand?

    Originally posted by starshipeleven View Post
    I implied that if all current applications can do it you can do it too for your specific usecase and it will be fine.
    It is not fine (to emulate case-insensitive file-open by retrieving a list of all files in a directory) because:
    - it is slower
    - it requires extra code and testing
    - requires code duplication
    - requires extra error checking, memory allocation and other things which experts know very well about, but amateur's don't
    - requires extra case handling (when two synonymous files exist in the same dir).
    - etc, etc...

    In short, it is not fine because it is not an elegant solution. An elegant solution is to provide this functionality on the kernel level.

    Leave a comment:


  • starshipeleven
    replied
    Originally posted by carewolf View Post
    It isn't needed. It could be extra API.
    Afaik there is no real requirement for an extra API (beyond internet trolls posing as software developers). This is just a quick workaround for some problematic applications, that's it.

    Leave a comment:


  • carewolf
    replied
    Originally posted by starshipeleven View Post

    Another guy seeing thigs that are only in his own mind.

    This is a filesystem feature that is needed by one of the possible usecases. Like Samba shares, or Wine application folders.

    It's not on by default and it's specifically designed to be enabled on a folder basis.
    It isn't needed. It could be extra API.

    Ironically Windows went the other way. They have have Linux APIs that makes it possible to store files that otherwise would name clash with normal Windows APIs.. NTFS have always been able to handle both, it is just different APIs.

    Leave a comment:


  • oiaohm
    replied
    Originally posted by xfcemint View Post
    1. FS Driver has additional information and privileges that user-space apps and libraries don't have, therefore enabling for a more efficient implementation.
    Not the cause of the advantage. The cause of the performance advantage to doing it kernel driver side is filename hashing. File names are first searched for by hash.

    Filling this hash table with casefolded make a case-insensitive search more effective at the file system level and the hash table is shared between processes so does not have to be started every time and some file systems allow you to save the hash table in the file system metadata between runs.

    Remember casefolding the look-up request is not free so case-insensitive always has more overhead than case-sensitive.

    Originally posted by xfcemint View Post
    2. It is easier to turn a case-insensitive file search into a case-sensitive than vice-versa.
    Also not true. Both kill you if you attempt todo it from userspace.

    https://lwn.net/ml/linux-fsdevel/201...collabora.com/

    About time you look at the patches. To turn case-insensitive file search into a case-sensitive you basically have to reverse the above patch.

    The problem is the filename lookup hash what is is filled with.

    A pure case-sensitive search is always faster than a case folded case-insensitive search. Turning a case-insensitive search into a case-sensitive one adds extra processing.

    Basically you are better to start with a case-sensitive search and extend it to include case-insensitive than attempt to convert a case-insensitive search into a case-sensitive one.

    Please also note a directory really cannot be both case-sensitive and case-insensitive at the same time as this leads to conflict issues on what is and is not allowed.



    Leave a comment:


  • starshipeleven
    replied
    Originally posted by xfcemint View Post
    Not true.
    Still true, even if it's "more efficient" for your own usecase it's still not as efficient as case-sensitive and applications that don't need that (the vast majority) are getting hit for no reason.

    While it can be done in a couple of lines, the actual code for this is at least 10 times bigger than the normal "file open", and prone to errors.
    If you are rewriting the same code over and over from scratch it's your own issue. It's a long-solved problem.
    The real problem is that such a code is extremely slow.
    I didn't notice that on any GUI or CLI application doing it.

    Only applications actually having issues with that are Samba and Wine stuff. But there is no reason to do that with a new application.

    I didn't say that. Nothing even close. I am arguing that "case-insensitive" handling is better. I didn't say anything about forcing it on anybody. The solution with per-directory preference is quite acceptable to me.
    You aren't making much sense then, is your application requiring the user to set this filesystem/folder option on all folders he keeps his documents and stuff just because your application needs to find his stuff? That's very convenient for end-user experience.

    Reminds me of some shit Windows applications that require the user to set Windows system folder to world writable to work at all.

    1. Not true for Wine and similar apps.
    That's legacy unmaintainable crap. (the applications you run with Wine, not Wine itself) You can just work around it and it's fine.
    2. Irrelevant argument. We are arguing what's better, not whether there will be an immediate performance impact.
    Performance impact is part of the assessment about what is "better".

    No, you are not.
    Yes I am, but you are just choosing to ignore what does not fit your worldview.

    Of course you never said it. You just implied it.
    More of the above, you are seeing only what you want to see.

    I implied that if all current applications can do it you can do it too for your specific usecase and it will be fine.

    Leave a comment:


  • xfcemint
    replied
    Originally posted by starshipeleven View Post
    case-insensitive is inefficient too, you are just moving the efficiency issue to someone else (everyone) for what is just your specific usecase.
    Not true. Any of the following reasons is sufficient to refute what you say:
    1. FS Driver has additional information and privileges that user-space apps and libraries don't have, therefore enabling for a more efficient implementation.
    2. It is easier to turn a case-insensitive file search into a case-sensitive than vice-versa.

    Originally posted by starshipeleven View Post
    [open a file given a user-specified name, case-insensitive]
    I can do that with a couple lines of shell script, if you have any issue in doing the same with a compiled program I assume it's your own problem.
    While it can be done in a couple of lines, the actual code for this is at least 10 times bigger than the normal "file open", and prone to errors.
    But that's not the real problem. The real problem is that such a code is extremely slow.
    So, your response does not count, because the solution you want to provide is not sufficiently fast. Your "solution" is not good enough.

    Originally posted by starshipeleven View Post
    and your usecase is always better. Who cares about everyone else that don't need that. Let's hit everyone.
    I didn't say that. Nothing even close. I am arguing that "case-insensitive" handling is better. I didn't say anything about forcing it on anybody. The solution with per-directory preference is quite acceptable to me.

    Originally posted by starshipeleven View Post
    and humans aren't going to generate enough files that you will see a performance impact for your usecase.
    1. Not true for Wine and similar apps.
    2. Irrelevant argument. We are arguing what's better, not whether there will be an immediate performance impact.

    Originally posted by starshipeleven View Post
    [playing "but muh humans" card]
    I'm playing the same card.
    No, you are not. Or you are playing it so badly that it doesn't make any difference.

    Originally posted by starshipeleven View Post
    [starshipeleven: I already said that all noteworthy user-facing applications can do case-insensitive searching already.]
    [xfcemint: Of course, and we won't ever need any new applications.]
    Look ma! A strawman argument! I never said that.
    Of course you never said it. You just implied it. You implied that there is nothing of much relevance besides current "noteworthy user-facing applications".

    If you want it phrased diffferently: your original argument is irrelevant because there are many important things besides current "noteworthy user-facing applications".

    Leave a comment:


  • oiaohm
    replied
    Originally posted by Weasel View Post
    Invalid point. Nobody forces you to use it or enable it on the directory.

    You want a simple argument for it? Wine performance.
    It not just for wine. NFS, Samba and Wine have usage cases for case insensitive. Problem is all three also have usage cases for case sensitive.

    So really it a case we need both. One or the other does not cut it.

    There has been a lot of history stubbornness over case insensitive and case sensitive with the stupid logic that you have to choose 1 or the other absolutely when correct answer is create a system that provide both solutions.

    Both windows 10 and Linux are going the per folder route for choosing if you are case insensitive or case sensitive it seams like the right option.

    Leave a comment:


  • starshipeleven
    replied
    Originally posted by Weasel View Post
    Invalid point. Nobody forces you to use it or enable it on the directory.
    I'm arguing against a guy that wants it enabled across the board because he can't deal with case-insensitive file search or something in his own applications.

    Leave a comment:

Working...
X