KDE Starts April With Big Performance Jump For Local I/O + 50~95% Faster Samba Transfers

[bug77]

In my experience, file operations (copy, delete) are generally slow in KDE. At least way slower then doing the same from Konsole, so I'm not sure what the problem is (Konsole is still KDE). Or maybe it's just Krusader, though that's supposed to be updated to Qt5, so it should be in good shape.

[NotMine999]

Just FYI, if you unmount a device in KDE it will immediately tells you is OK to remove it, even if the device light activity is still blinking.

Sounds like a critical bug because the misleading behavior of the system message ("OK to remove it") compared to the blinking device activity light could lead to data corruption.

Has a bug report been filed by anybody on this issue?

Goofy UI stuff like this makes me wonder if KDE is being written by script-kiddies.

[thxcv]

Sounds like a critical bug because the misleading behavior of the system message ("OK to remove it") compared to the blinking device activity light could lead to data corruption.

Has a bug report been filed by anybody on this issue?

Goofy UI stuff like this makes me wonder if KDE is being written by script-kiddies.

You shouldn't believe everything you read on the internet..

[caligula]

How does one manage to still lose 10 to 20% of performance when we're talking about just 100 MB/s of throughput while a modern CPU core can easily handle 500 MB/s of TCP traffic and man GB/s of local memory transfers, you ask?

By dragging on an unnecessarily convoluted, arcane and superseded concept like KIO.

The other file managers also have performance issues. Ever tried GIO/GVfs or XFCE/Thunar? XFCE even had race conditions that made the whole file manager window crash when e.g. deleting files. What a joke.

[caligula]

In my experience, file operations (copy, delete) are generally slow in KDE. At least way slower then doing the same from Konsole, so I'm not sure what the problem is (Konsole is still KDE). Or maybe it's just Krusader, though that's supposed to be updated to Qt5, so it should be in good shape.

Graphical file managers are slow. Many other file managers also accumulate "file lists" before large operations. Generating these lists can sometimes be slower than traversing the file tree while actually performing the operation (assuming one does not allocate huge heaps of memory for this list/log).

[polarathene]

One thing that really annoy me is the way a file transfer progress is shown. The progress bar visualize the reading rate of the file, not the writing. So reading can long finish before the writing.

Actually, afaik it's not the read I/O you're seeing, but write I/O to the kernel buffer(RAM). I ran into this same issue a couple years back when writing to some old cheap USB stick over USB 2.0, which while saying that people can point out it should be slow, the same amount of data that took only 10 minutes to copy via Windows took around 7 hours iirc with KDE/Dolphin at the time.

I know there has been times(including on Gnome too) where either progress was very fast, the rate was much faster than could be written to the device at, and progress bar filled up to close to 100%, then became sluggish and slow with an ever decreasing rate, that could take a very long time to complete. On Gnome I recall this when copying over an ISO I think to a USB stick, instead of just flashing/burning the ISO to the storage media via linux with dd which would probably have been a better choice.

The other scenario was as M@GOid has described, where the progress bar would complete instead of become a snail, and claim the transfer was successful. I believe this was how KDE was behaving vs Gnome? I remember transferring a video and trying to play it after the transfer worked perfectly fine, I could go through the entire video playback jumping through it thinking I had verified it was ok. Then when I unmounted and mounted again, the video file was there but it was corrupt. Turned out I was previously viewing the video file not stored on the USB device, but actually from the RAM buffer it was still writing from to that location... I cannot confirm the safe to unmount / successful unmount UI, this was back in 2016/2017, I may have been refused and that verification step might have been me sick of waiting over an hour for something I knew transferred much faster on Windows, thinking the inability to unmount was some bug, and force removed it physically(likely).

In either case, I did discover the issue was as mentioned, the data was written to a buffer in memory as an optimization(write sequentially in bulk to target vs direct source to destination which can perform worse). IIRC, the default buffer size is 10% of your systems total RAM, I had a 32GB machine, so anything under 3.2GB qualifies for filling that before any flush/write starts. These are called vm_dirty_bytes/vm_dirty_ratio or something like that, kernel tunables, but they oddly are not device specific for these operations, so while you can lower them(I think 15MB was a good size, it would write 15MB, then write 15MB before transferring the next chunk), it can potentially reduce performance for internal transfers.

With the much lower buffer size, you'd then get an accurate progress bar(much more accurate at least). I recall testing with a USB device that had an activity LED which would only last a few seconds at most after the progress bar from Plasma finished and said the transfer was complete. Unmounting would be successful too. Progress bar was steady as it was proven to be tied to that buffer(it'd fill up and flush to target).

If a user did not experience that, it's possible that their distro had chosen a more sane default for modern systems, or that they did not have a large amount of RAM paired with a large transfer to slow media. Filesystems may have also played a role in it, I think I noticed FAT32/exFAT(over FUSE) had more issues than UDF? It might be a slightly separate issue related to filesystem support/info about being able to get some sort of metadata/info about the transfer, I dunno.

All I know is it was/is(?) a real issue, hopefully this is helpful to someone

This entire post looks completely untrue from my experience. Including the nonexistent progress bar.

Non-existent progress bar? I've definitely had such for both Gnome and KDE transfers, don't know what you're on about. See above for past experiences and cause/resolution.

[bug77]

Graphical file managers are slow. Many other file managers also accumulate "file lists" before large operations. Generating these lists can sometimes be slower than traversing the file tree while actually performing the operation (assuming one does not allocate huge heaps of memory for this list/log).

I don't know, file managers don't seem to be slow on Windows...

[Volta]

I don't know, file managers don't seem to be slow on Windows...

They're very slow. Especially when extracting large archive, but then entire Windows crawls. It also happens when installing games on Steam. USB transfer:

Fedora 32 Gnome 3: 65MB/s to 62MB/s. (ext4 HDD to ext4 USB).
Windows 10: 55MB/s and then ~40MB/s most of the time to finally drop to 15MB/s. (NTFS HDD to NTFS USB).

What needs to be checked is the real end of transfer. On Gnome it took few seconds to unmount. (still damn faster than Windows) and on Windows it took me few seconds to find the unmount icon in the trash. I mean tray..

[M@GOid]

Actually, afaik it's not the read I/O you're seeing, but write I/O to the kernel buffer(RAM). I ran into this same issue a couple years back when writing to some old cheap USB stick over USB 2.0, which while saying that people can point out it should be slow, the same amount of data that took only 10 minutes to copy via Windows took around 7 hours iirc with KDE/Dolphin at the time.

I know there has been times(including on Gnome too) where either progress was very fast, the rate was much faster than could be written to the device at, and progress bar filled up to close to 100%, then became sluggish and slow with an ever decreasing rate, that could take a very long time to complete. On Gnome I recall this when copying over an ISO I think to a USB stick, instead of just flashing/burning the ISO to the storage media via linux with dd which would probably have been a better choice.

The other scenario was as M@GOid has described, where the progress bar would complete instead of become a snail, and claim the transfer was successful. I believe this was how KDE was behaving vs Gnome? I remember transferring a video and trying to play it after the transfer worked perfectly fine, I could go through the entire video playback jumping through it thinking I had verified it was ok. Then when I unmounted and mounted again, the video file was there but it was corrupt. Turned out I was previously viewing the video file not stored on the USB device, but actually from the RAM buffer it was still writing from to that location... I cannot confirm the safe to unmount / successful unmount UI, this was back in 2016/2017, I may have been refused and that verification step might have been me sick of waiting over an hour for something I knew transferred much faster on Windows, thinking the inability to unmount was some bug, and force removed it physically(likely).

In either case, I did discover the issue was as mentioned, the data was written to a buffer in memory as an optimization(write sequentially in bulk to target vs direct source to destination which can perform worse). IIRC, the default buffer size is 10% of your systems total RAM, I had a 32GB machine, so anything under 3.2GB qualifies for filling that before any flush/write starts. These are called vm_dirty_bytes/vm_dirty_ratio or something like that, kernel tunables, but they oddly are not device specific for these operations, so while you can lower them(I think 15MB was a good size, it would write 15MB, then write 15MB before transferring the next chunk), it can potentially reduce performance for internal transfers.

With the much lower buffer size, you'd then get an accurate progress bar(much more accurate at least). I recall testing with a USB device that had an activity LED which would only last a few seconds at most after the progress bar from Plasma finished and said the transfer was complete. Unmounting would be successful too. Progress bar was steady as it was proven to be tied to that buffer(it'd fill up and flush to target).

If a user did not experience that, it's possible that their distro had chosen a more sane default for modern systems, or that they did not have a large amount of RAM paired with a large transfer to slow media. Filesystems may have also played a role in it, I think I noticed FAT32/exFAT(over FUSE) had more issues than UDF? It might be a slightly separate issue related to filesystem support/info about being able to get some sort of metadata/info about the transfer, I dunno.

All I know is it was/is(?) a real issue, hopefully this is helpful to someone




Non-existent progress bar? I've definitely had such for both Gnome and KDE transfers, don't know what you're on about. See above for past experiences and cause/resolution.

Finally a decent answer. Something that is getting rarer around here.

I believe what you are describing is correct. At last in the latest KDE Neon with the latest stable version this still occur, unfortunately.

The first time I noticed something wasn't right was when I saw a friend doing a file transfer on the Mate DE, so after some tests I discovered this strange behavior on KDE.

[polarathene]

I believe what you are describing is correct. At last in the latest KDE Neon with the latest stable version this still occur, unfortunately.

Check the vm dirty kernel tunables(sysctl or something? should be able to google it). If you've got the defaults and a fair amount of RAM, then adjust the tunables like I said to much smaller amount than 10% of your RAM. Should resolve the issue, you can make the change and it'll reset to whatever it was before next boot unless you save a config to override it. That way you can have a script to toggle if you want to selectively reduce the buffer for transfers to USB sticks.

You can try to get your distro to adjust the defaults, but other users might not like it impacting other storage transfers(I haven't tested to see impact, might be negligible). Nothing to do with KDE though afaik, the Mate DE probably had different vm dirty settings. Dolphin/KIO is a different thing which has been known to add overhead, but has had many updates over recent years to reduce that, it used to be very bad, one of the worst ways to transfer data via GUI on linux iirc?