Lennart Poettering Talks Up His New Linux Vision That Involves Btrfs

[justmy2cents]

If the difference in packaging is the main problem, and Lennart proposes to just give this job to upstream, what is the point in actually having different distributions? IIRC the core business of distributions is packaging ...
Also, how does Lennart want to deal with situations like different programs requiring different versions of a service like DBUS running? Those two will never be able to communicate with each other!
Finally, if you are going to give upstream total control over the dependency versions, you close the door to post-release bugfixes of those dependencies, OTOH if you let the dependency be updated, the "testability for devs"-point is moot. You can't have both (for all I know. If you know better, please enlighten me)

dbus and rest ipc frameworks are troubling to me too. what would be the benefit if one would have to know which software can work with some other or not. i mean idea of simply updating snapshot differences is nice, download diffs and let btrfs handle them. what couldn't be nicer... which leads to second troubling thing that is shitstorm you end up with. this goes to commercial developers more or less. they won't update apps to target newer frameworks, they'll just enforce old one to be installed with their app. when money matters more than quality, this is guaranteed effect. it won't be snapshots by choice, it will be snapshot by must.

how will it for example handle trivial things like clipboard if implementation breaks ABI and you have one app using old one and one app using new one?

partially this could be probably solved by introducing enforced ABI versioning instead of software versions. for example Qt5.3 must be compatible with Qt5.2. as soon as next one breaks compatibility, it must be snapshot must be called Qt6. that way there would only be one image per specific requirements and system could update it freely from new Qt5.x

as long as you allow whole distros to be packaged like this, what's to stop some adobe like lame vendor going ballistic and require xyzlinux-1.4.3.2-after_breakfast_version_with_some_patches_to_break _xyzlinux-1.4.3.2_completely_since_they_modified_dbus_to_inc ompatible. at that point everyone is stuck with that release. and worse, incompatibilities like that could bring linux from "hard to use" to "doesn't work at all"

[emblemparade]

A problem that Poettering and others here in the discussion are ignoring is that of curation.

I don't just want to be able to run software, I want it carefully curated for my operating system. The problem is that upstream devs might have very different ideas about how to deploy their software than that of my operating system, and there's no trivial way to align those cultures. Take two examples from two different words:

Firefox: You know you can just download Mozilla's Linux distribution from their site, but you also know that you'd much rather have the Ubuntu/Fedora version, carefully patched and tuned to work with the ecosystem of your operating system, with things like user management, desktop integration, plugins (Flash), etc.

Apache Web Server: Debian and Fedora have different philosophies for configuration, for installing plugins, integrating with init systems, etc. Just downloading the binaries is a very small part of the job of deploying the product.

The reason we enjoy using Debian/Fedora/Arch/whatever is that we get a carefully curated experience.

Rolling-release distributions are impossible to carefully curate due to so many interdependencies. "DLL hell", as some people here mentioned, is a very real problem. It's true that you can have GTK+2 and GTK+3 on the same operating system, because that's how the GTK+ project planned it. But you cannot have different versions of GTK+3 together, and different releases of GTK+3 are indeed breaking upgrades. And of course every upstream project has their own way of handling version releases, many with breaking changes to APIs and ABIs that do not easily allow for multiple versions to exist together. And of course applications that use such libraries might feature their own peculiar bugs, sometimes having to do with specific point releases of dependencies. For popular libraries and frameworks these problems multiply very, very quickly.

I think the 6-month plus 2-year LTS release cycles have proven themselves in nicely balancing freshness and stability. However, I would like to see a more coherent upgrade path for important software. It's nice that Firefox gets upgraded within a week, but of course we want other important packages upgraded, too.

For example, we currently have "PPAs" for Ubuntu, on Launchpad as well as repositories in other places, but those are an incoherent mess of "official PPAs" and private endeavors, with their own installation instructions. What I'd like to see is Ubuntu taking leadership on this mess, and curate a space for upgrades and other extras, and have it cleanly integrated into the Software Center (separately for each Ubuntu released version). So you would have a tab with auto-upgrades for existing Ubuntu software, which much always be official in some way, either managed directly by upstream (they sometimes do it already) or by a dedicated packager that somehow has proven themselves to whomever is managing this in the Ubuntu project. There should be an equally easy way to remove these add-ons (ppapurge behind the scenes).

We see this in other software distribution systems. Eclipse has the Eclipse Marketplace. From within Eclipse, you can browse the marketplace, and if you like something, you click the link and it adds that repository. Easy to remove it if you don't like something. (Unfortunately Eclipse doesn't curate it very well.) Firefox has "Add Ons" of various kinds, easy to add and remove (but doesn't exactly have the "repository" aspect, so not such as a good implementation example.)

There are few individual attempts to curate such systems for Linux (Ubuntu Tweak), but of course they are too small and scattered. It needs support directly in Ubuntu to work well, and of course has to work out of the box.

This can be integrated into Launchpad. PPA after all means "Personal Package Archive", it was never intended for official distribution, just a way for devs to test packages before having them integrated into the operating system. (That's also what the name "Launchpad" intends.) But there can be a new kind of archive there, which will immediately be visible in the Software Center for all users. This can of archive must be approved by the curating team.

[psychoticmeow]

And if you took a poll of the Linux user base you'd find quite easily that these "problems" of his are of no problem to probably 99% of users.

Nobody cares about this, but I'll bet we're all going to be forced to care because it'll be coming one way or another.

Do the words "self selecting" mean anything to you?

[Isedonde]

That's a comment from his g+ stream.

Thanks, that answered my question. So runtimes will be updated to fix security / grave functionality issues, without breaking binary apps that depend on that runtime.

However, where's the difference to the technique we use today for handling library dependencies and ABI, sonames? I don't see how bundling multiple libraries into a single runtime is going to improve the abi stability issue. The same can be achieved using sonames, AFAIUI.

Where's the difference?

Using new Lennart idea:
* Rhythmbox depends on the GNOME_3_12 runtime, and links (among others) against libgio from that runtime
* Gnome fixes a CVE in libgio. Gnome releases updated GNOME_3_{8,10,12,whatever} runtimes containing the new libgio, without breaking ABI, so Rhythmbox uses the new libgio without recompiling and the CVE is fixed.
* Gnome does ABI breaking changes to some library in the gnome runtime, releases GNOME_3_14 (but somehow fails to update Rhythmbox for it). Now Rhythmbox "3.12" still works, because the old GNOME_3_12 is still available, containing the ABI compatible libgio.
* Same as before, but this time GNOME_3_12 runtime is no longer available because 10 years have passed: I can no longer use rhythmbox (required GNOME_3_12 runtime not found).


Using sonames:
* Rhythmbox links against libgio-2.0.so, provided by some distro package libgio2.
* Gnome fixes a CVE in libgio. Gnome releases updated libgio-2.0.so source code without breaking ABI. The distro provides their users with an updated libgio2 package and Rhythmbox uses the contained libgio-2.0.so library without recompiling, so the CVE is fixed.
* Gnome does ABI breaking changes in libgio. Thus, they must choose a new soname and call it libgio-3.0.so, which is shipped in my distro in the libgio3 package. The old Rhythmbox still works, because the libgio2 package is still available, containing the libgio-2.0.so library.
* Same as before, but this time libgio2 was dropped from the distro because 10 years have passed: I can no longer use rhythmbox (shared library libgio-2.0.so not found)

I could go on. The only difference I found: With RUNTIMEs, stuff breaks as soon as one of the libs contained in a runtime breaks ABI and thus a new RUNTIME is released, because sooner or later everything must migrate to the new RUNTIME. With good old sonames, stuff only breaks when one of the actually used libraries breaks ABI, because sooner or later everything that uses this specific library must migrate to the new library ABI.

There seems to be a perceived difference with RUNTIMEs having a longer support period. I don't think that's valid, because why would someone maintain an old RUNTIME for 5 years, but drop an old soname library after 2 months already? Right now, most libraries are removed from distros as soon as every depending package has been migrated to the new library version. It's pretty rare to see different library sonames of the same library in one distro release. It's simply too much work to maintain multiple library versions without a good reason to do so. And that won't change with RUNTIMEs.

Sure, you can always use an old runtime ? Just like you can always use an old .so file. However, since nobody does security / bug fix work for old libraries or old RUNTIMEs, this is dangerous.


TL;DR: You can only have third-party binaries, and binaries in general, work for longer periods of time without re-compiling if all used libraries maintain ABI compatibility. If a lib provides long-time ABI stability, this is already possible today. If a library does not provide ABI stability, then you can't fix that by adding layers of BTRFS features around the libs. New library versions fail to work because of new ABI, and old library versions have security issues / bugs, no matter if they're installed through apt or BTRFS-send-and-receive.

Anyway, the Lennart method probably has other advantages over today's standards unrealted to shared libraries. So I'm looking forward to new developments in that area.

[justmy2cents]

Where's the difference?

Using new Lennart idea:
...

Using sonames:
...

now take your example in this scenario. rhythmbox never was updated because it works, dbus changes ABI and communication and all remote like apps for rhythmbox either don't work or they are forced to use same old runtime. you get stuck with same old runtime for ages just because one relevant application didn't update/fix. now if those remote apps were part of desktop, will you also use old desktop on that same runtime just because new one would break rhythmbox?

as good as it sounds (mind you, i love systemd and i'm mostly fond of lennarts ideas), issue here is that as much as it makes it "just works" it also provides excuse in a way of "no need to update/fix, we provide you with OUR runtime". another questionable thing here is where the developers "runtime" ends. if they used some custom tweaked kernel/glibc, that one app might just force you to boot in whole new other os. and here is the nightmare, let's say some adobe comes and provides this plagued version of whole os as runtime leading to other vendors following them...

i'd be much happier if this idea wouldn't go so low and if frameworks would be sandboxed and created the same way steam-runtime is. here it is, build it for your self for your distro. at the moment you create this runtime package you're ok to run anything requiring "xyz-3.x". but, same way would needed to be done from apps where first thing that app should do is saying "i require qt-3, dbus-1... ABI runtimes" and then started with correct environment

[liam]

Lennart strikes me as a visionary with poor vision. It's nothing personal at all. I've never met the guy. He just wants to take Linux down a certain path that I don't want to go. And he has the influence to make it happen. I do not, of course, lay all the blame on him. There is a lot of bad thinking in the Linux world these days.

People are always so excited about revolutionary change. But history proves that revolutions don't always end well.

History indicates, thus far, that his vision is pretty damn good. All of his major projects have seen high uptake. PA would be almost perfect if ALSA included the necessary timing apparatus. Avahi does its (small) job well. SD is providing the base linux has always failed to provide.
Inarguably, he's making things better for the normal user.
If you have specfic issues with his proposal lets hear them, and I'll try to answer.

[liam]

A problem that Poettering and others here in the discussion are ignoring is that of curation.

I don't just want to be able to run software, I want it carefully curated for my operating system. The problem is that upstream devs might have very different ideas about how to deploy their software than that of my operating system, and there's no trivial way to align those cultures. Take two examples from two different words:

Firefox: You know you can just download Mozilla's Linux distribution from their site, but you also know that you'd much rather have the Ubuntu/Fedora version, carefully patched and tuned to work with the ecosystem of your operating system, with things like user management, desktop integration, plugins (Flash), etc.

Apache Web Server: Debian and Fedora have different philosophies for configuration, for installing plugins, integrating with init systems, etc. Just downloading the binaries is a very small part of the job of deploying the product.

The reason we enjoy using Debian/Fedora/Arch/whatever is that we get a carefully curated experience.

Rolling-release distributions are impossible to carefully curate due to so many interdependencies. "DLL hell", as some people here mentioned, is a very real problem. It's true that you can have GTK+2 and GTK+3 on the same operating system, because that's how the GTK+ project planned it. But you cannot have different versions of GTK+3 together, and different releases of GTK+3 are indeed breaking upgrades. And of course every upstream project has their own way of handling version releases, many with breaking changes to APIs and ABIs that do not easily allow for multiple versions to exist together. And of course applications that use such libraries might feature their own peculiar bugs, sometimes having to do with specific point releases of dependencies. For popular libraries and frameworks these problems multiply very, very quickly.

I think the 6-month plus 2-year LTS release cycles have proven themselves in nicely balancing freshness and stability. However, I would like to see a more coherent upgrade path for important software. It's nice that Firefox gets upgraded within a week, but of course we want other important packages upgraded, too.

For example, we currently have "PPAs" for Ubuntu, on Launchpad as well as repositories in other places, but those are an incoherent mess of "official PPAs" and private endeavors, with their own installation instructions. What I'd like to see is Ubuntu taking leadership on this mess, and curate a space for upgrades and other extras, and have it cleanly integrated into the Software Center (separately for each Ubuntu released version). So you would have a tab with auto-upgrades for existing Ubuntu software, which much always be official in some way, either managed directly by upstream (they sometimes do it already) or by a dedicated packager that somehow has proven themselves to whomever is managing this in the Ubuntu project. There should be an equally easy way to remove these add-ons (ppapurge behind the scenes).

We see this in other software distribution systems. Eclipse has the Eclipse Marketplace. From within Eclipse, you can browse the marketplace, and if you like something, you click the link and it adds that repository. Easy to remove it if you don't like something. (Unfortunately Eclipse doesn't curate it very well.) Firefox has "Add Ons" of various kinds, easy to add and remove (but doesn't exactly have the "repository" aspect, so not such as a good implementation example.)

There are few individual attempts to curate such systems for Linux (Ubuntu Tweak), but of course they are too small and scattered. It needs support directly in Ubuntu to work well, and of course has to work out of the box.

This can be integrated into Launchpad. PPA after all means "Personal Package Archive", it was never intended for official distribution, just a way for devs to test packages before having them integrated into the operating system. (That's also what the name "Launchpad" intends.) But there can be a new kind of archive there, which will immediately be visible in the Software Center for all users. This can of archive must be approved by the curating team.

He addressed this but few, appear, to have read his post and his subsequent comments.
I'm continually amazed how many people think Lennart is a not very bright guy.

[anda_skoa]

how will it for example handle trivial things like clipboard if implementation breaks ABI and you have one app using old one and one app using new one?

Not a very likely problem for clipboard. Clipboard is almost certainly something that is based on a communication between at least to programs, mabye three if there is a clipboard manager involved.
Such communication happens via system level interaces such as sockets. The system level API/ABI of such things almost never changes.

partially this could be probably solved by introducing enforced ABI versioning instead of software versions. for example Qt5.3 must be compatible with Qt5.2. as soon as next one breaks compatibility, it must be snapshot must be called Qt6.

That is exactly how it is handled already.

Cheers,
_

[justmy2cents]

Not a very likely problem for clipboard. Clipboard is almost certainly something that is based on a communication between at least to programs, mabye three if there is a clipboard manager involved.
Such communication happens via system level interaces such as sockets. The system level API/ABI of such things almost never changes.

"almost" in your comment should tell you everything. was much more specific in my later comment. we're at time when linux is moving towards completely new display server, sandboxing of applications, etc

That is exactly how it is handled already.

Cheers,
_

actually, i can't see this in his proposal, since he also specifies whole required /usr is carried in runtime snapshot. that would implicate that we could have one dbus in kde snapshot and one in gnome where they could be incompatible if dbus broke ABI and communication protocol between the 2 versions and that would take Linux desktop few years back, not forward

his approach of sticking everything into snapshot does guarantee you have ABI stability for specific runtime, but it doesn't guarantee interop between 2 runtime snapshots. and no interop between snapshots might lead to really bleak future

proposal is severely lacking definition what exactly snapshot is and how fragmented snapshots are. it also only takes perfect future by forgetting imperfect. just look at Autodesk Maya on linux and compare it to Blender. taken this in question Autodesk Maya will require 3 years old snapshot of whole OS that was modified and they will require same snapshot for the next century, while Blender will always move with progress. and Maya is more or less typical commercial application where vendor is not better or worse than others

[liam]

Having some problems parsing this, but I'll attempt an answer since you are using good netiquette (thanks for that!).

However, where's the difference to the technique we use today for handling library dependencies and ABI, sonames? I don't see how bundling multiple libraries into a single runtime is going to improve the abi stability issue. The same can be achieved using sonames, AFAIUI.

Where's the difference?

Using new Lennart idea:
* Rhythmbox depends on the GNOME_3_12 runtime, and links (among others) against libgio from that runtime
* Gnome fixes a CVE in libgio. Gnome releases updated GNOME_3_{8,10,12,whatever} runtimes containing the new libgio, without breaking ABI, so Rhythmbox uses the new libgio without recompiling and the CVE is fixed.
* Gnome does ABI breaking changes to some library in the gnome runtime, releases GNOME_3_14 (but somehow fails to update Rhythmbox for it). Now Rhythmbox "3.12" still works, because the old GNOME_3_12 is still available, containing the ABI compatible libgio.
* Same as before, but this time GNOME_3_12 runtime is no longer available because 10 years have passed: I can no longer use rhythmbox (required GNOME_3_12 runtime not found).

This solution doesn't solve the long-term support issues linux desktops have, but it does fix short-term breaks, which represent the majority of issues users have.

Using sonames:
* Rhythmbox links against libgio-2.0.so, provided by some distro package libgio2.
* Gnome fixes a CVE in libgio. Gnome releases updated libgio-2.0.so source code without breaking ABI. The distro provides their users with an updated libgio2 package and Rhythmbox uses the contained libgio-2.0.so library without recompiling, so the CVE is fixed.
* Gnome does ABI breaking changes in libgio. Thus, they must choose a new soname and call it libgio-3.0.so, which is shipped in my distro in the libgio3 package. The old Rhythmbox still works, because the libgio2 package is still available, containing the libgio-2.0.so library.
* Same as before, but this time libgio2 was dropped from the distro because 10 years have passed: I can no longer use rhythmbox (shared library libgio-2.0.so not found)

This isn't quite the problem it solves. Any decent distro will support MAJOR api breaks, for major libraries, by providing both libraries, but every other break can fall through the crack.

I could go on. The only difference I found: With RUNTIMEs,

Just want to mention this point: runtimes are the level at which distros would deal, but apps will be provided the ability to bundle AND build against the runtime. This would let vendors, or just lazy devs, the ability to more easily distribute software on linux without having to use complicated scripting, or build multiple packages against all supported distros.

stuff breaks as soon as one of the libs contained in a runtime breaks ABI and thus a new RUNTIME is released, because sooner or later everything must migrate to the new RUNTIME. With good old sonames, stuff only breaks when one of the actually used libraries breaks ABI, because sooner or later everything that uses this specific library must migrate to the new library ABI.

AIUI, this shouldn't cause any additional breakage. The runtime provides the common shared libraries(this would be, I would imagine, pretty much everything in your distro's repos). You simply won't see a difference here (unless something breaks, in which case the difference will be that you won't notice it broke, and when they fix the issue it will be pushed to your system, repeat and rinse).

There seems to be a perceived difference with RUNTIMEs having a longer support period. I don't think that's valid, because why would someone maintain an old RUNTIME for 5 years, but drop an old soname library after 2 months already? Right now, most libraries are removed from distros as soon as every depending package has been migrated to the new library version. It's pretty rare to see different library sonames of the same library in one distro release. It's simply too much work to maintain multiple library versions without a good reason to do so. And that won't change with RUNTIMEs.

I'm not sure that's the case. Runtimes just provide a more convenient unit for distributions. Again, this doesn't really address the long-term support issue of linux (excepting the bundled apps which should work for very long periods, depending on what they assume).

I hope this helps, at least somewhat.