~5 Minutes Of Coding Yields A 6%+ Boost To Linux I/O Performance

[caligula]

Kodi does that for me, but more or less only if there is a background scan going on that looks for metadata of new files. Otherwise it still takes a couple of seconds, but not to bad.
I guess it also needs to close all those database connections/files. Because it has lots of databases. One for video, one for pictures, one for music, one for textures, one for.... yeah, it goes on.
Normally an application has one database with loads of tables, which means fewer files/connections to sync/close at the end.

But this behavior is also true for Kodi on windows last time I used it, so I guess that is not Linux-specific.

My Kodi installation only contains a database of movies and TV series. No pictures or music. There are tons of videos though. Also it doesn't use mysql or any real db, I think the db files are built with sqlite.

[caligula]

I can't find an article that address this, but years ago i took a class on computer architecture and I remember we were taught that on all systems, regardless of OS, a portion of system ram equal to the amount of vram available is set aside by the OS.

That was an issue when the systems had limited memory addressing. E.g. old 16-bit DOS systems only had one megabyte worth of memory addresses. In 32-bit systems you have 4 GB of address space and 64-bits provides 2^32 x 4 GB of addresses. There's no real advantage in shadowing the main memory addresses with vram.

[archkde]

Ah, makes sense. I was gonna quote the paper I've read but couldn't quite remember the title, and a quick google search on GC overhead made it obvious it was gonna take a lot of time to find it again without a more of the exact title.

It's actually not hard to derive this formula directly: under the assumption that live memory stays constant and garbage is produced at a uniform rate, GC overhead is directly proportional to GC frequency, which is inversely proportional to the amount of garbage collected each time. (And yes, the formula should have been 1/(x-1) and I just made a sign error.)

[Xake]

My Kodi installation only contains a database of movies and TV series. No pictures or music. There are tons of videos though. Also it doesn't use mysql or any real db, I think the db files are built with sqlite.

Those might be the databases you know of. However there are more.

https://kodi.wiki/view/Databases

Kodi uses SQLite, an open source light-weight SQL database-engine, to store all the library, add-on, EPG, textures and PVR related data using a number of specific databases.

For SQLite that means one file per database.
And you will have:
* an add-on database telling kodi that is empty because you have no add-ons, but it will still be opened together with Kodi, and closed with Kodi, ready if you want to install an add-on
* The same goes for picture and music, however these three should not pose a larger problem, only adding at most seconds to close time
* "textures" database which contains links to all posters, thumbnails and other graphics shown in you videos/music lists
If you have a large set with many episodes and movies this will be huge
The same goes for you "MyVideo"-database that contains all info about the video files themselves, the scraped info about what the video-file contains, how it is related to other videofiles (movie collections, series), actors/actresses, cross-references for these, watch-lists and more.

Also, SQLite is good at pretty small databases. Large ones? Not so much. That also means that every time you close Kodi it needs to:
* Finish all its current queries
* close the SQLite "connections" correctly, so that all locks and alike things in the SQLite-files is handled
* Sync the files to filesystem

For large databases with large datasets this can take a while, and I am unsure if Kodi does this async or sync (that is closes databases in parallel or one at a time).

Adding to this and related size: Kodi is _NOT_ good at cleaning out the database. This by design to not remove stuff just because you for example lost your NAS-connection.
My database contains references to loads of files removed ages ago, even if the movies does not show up in the lists, because they where cleared away by a "clean library". These references are only removed if I before removing/moving the file from the context menu remove the reference in the library.

For example of this:

https://forum.kodi.tv/showthread.php?tid=356874

[coder]

For SQLite that means one file per database.
And you will have:
* an add-on database telling kodi that is empty because you have no add-ons, but it will still be opened together with Kodi, and closed with Kodi, ready if you want to install an add-on
* The same goes for picture and music, however these three should not pose a larger problem, only adding at most seconds to close time
* "textures" database which contains links to all posters, thumbnails and other graphics shown in you videos/music lists

Many distros still have a default limit of 1024 open files per process. Closing a file takes microseconds. I understand sqlite is more heavy-weight, but since most of these databases will not have pending writes, there shouldn't be a lot of cleanup to do.

As someone who regularly deals with very resource-intensive processes with lots of open connections and files (up to 16k), I'm telling you: there's no good reason it should take more than a couple seconds to shut down, if that.

I'm sure the bulk of the problem is some garbage code around their threading and synchronization. This stuff is hard to get right, if you don't know what you're doing or have a pretty clear plan from the outset.

[Xake]

As someone who regularly deals with very resource-intensive processes with lots of open connections and files (up to 16k), I'm telling you: there's no good reason it should take more than a couple seconds to shut down, if that.

Is that against a remote mysql-server or local SQLite-file?
As someone working with resource-intensive processes against MySQL-clusters I can say that the finishing and disconnect of the MySQL-server is fast.
Closing mysqld directly after before it has "settled" and synced everything? Not so much.

I actually do not work that much with SQLite, but my limited experience is that it makes the application slower at shutdown then working against a mysql-server, as the application in itself becomes the SQL-server having to handle closing, cleanup of locks and the other stuff that for example MySQL defers til its own shutdown.

I guess you have to do a fast perf of Kodi to actually find what takes time. My instance (running MyVideo/MyMusic on a remote mysql server) takes <5 seconds as long as it does not have a scan in progress it needs to settle.

[Nth_man]

[...] My entire KDE desktop will freeze up occasionally when I'm copying RAWs from my camera's SD card to my PC for editing. That happened with my previous system, too. KDE and a high IO load don't always mix.

GNOME has a hard time even reaching 60hz on low end hardware. [...]

Occasionally, something like that happened to me, then I started using a method that RealNC wrote about. Maybe that can be useful for you, too.

[cl333r]

But it's not. You listed the DE, browser + 6 tabs, IM, IDE, terminal, audio player. On my system
- XFCE DE uses around 350 MB (includes also all headless background services, fwiw)
- Firefox - depends on pages, but more than 300 MB per page seems quite a lot
- don't know about Telegram, but Pidgin uses less than 100 MB
- terminals (konsole, gnome-terminal, xfce-terminal etc.) only use few megabytes of RAM
- audio player like Audacious uses less than 100 MB.

I don't use VS Code, but I suspect it's the worst offender here, maybe followed by the browser.

I don't know how much FF used back then, now it's at 1.2GiB (4 tabs), Visual Studio Code - 473MiB, Telegram - 383MiB.
The only thing raising eyebrows to me is FF, but then again.. meh, got plenty of memory anyway.

[coder]

Is that against a remote mysql-server or local SQLite-file?
Closing mysqld directly after before it has "settled" and synced everything? Not so much.

Yeah, probably because it has lots of pending writes and other bookkeeping to update. Unike this Kodi example we're discussing, which should be mostly just reading its databases.

as the application in itself becomes the SQL-server having to handle closing, cleanup of locks and the other stuff that for example MySQL defers til its own shutdown.

Closing files and deleting locks are both trivial operations.

I guess you have to do a fast perf of Kodi to actually find what takes time.

I'd try getting a stack dump immediately after telling it to exit. Then, look at what all the threads are doing. I'll bet you'll either find deadlocks or some background threads/processes with long polling intervals.

[Xake]

Yeah, probably because it has lots of pending writes and other bookkeeping to update. Unike this Kodi example we're discussing, which should be mostly just reading its databases.

Mostly reads, but still some writes, and the needed locks.

Closing files and deleting locks are both trivial operations.

True. However a CoW-filesystem and alike can make this take more time.

I'd try getting a stack dump immediately after telling it to exit. Then, look at what all the threads are doing. I'll bet you'll either find deadlocks or some background threads/processes with long polling intervals.

Why stacktrace? Better set perf to record the pid of kodi and initiate a shutdown. That will get you information on what it wastes time on.

Me myself has a load of videos os well, and my old laptop with a i7-5500U, 8gb memory and an old SATA-SSD with webbrowser and many tabs open in the background having MyVideos and MyMusic in a remote MySQL reached over n-wifi take about 3 seconds to exit. It takes shorter time on the HTPC which has even more oddities but newer CPU. So if yours takes longer something seems odd with your installation or setup.