APT 1.3 Released For Debian Linux Distributions

[starshipeleven]

unapproved post for sdack above

[sdack]

Sigh. ...

Here is an illustration for you.

[B]Deinterleaved download, extraction and installation sequence:[/B]
D1 D2 D3 D4 D5
               X1 X2 X3 X4 X5
                              I1 I2 I3 I4 I5
|---------------- Timeline ---------------->

[B]Interleaved download and installation sequence:[/B]
D1 D2 D3 D4 D5
   X1 X2 X3 X4 X5
      I1 I2 I3 I4 I5
|---- Timeline ---->

D1..D5 show the downloads, X1..X5 show the extractions, I1..I5 show the installations. The order in which the packages are being installed (I1...I5) is preserved with interleaving.

Conflict resolution doesn't happen at this stage. It already took place (and determines which packages can and cannot be installed, which have to be removed, which have to be downloaded and installed and in which order this needs to happen).

If it helps you then think about what happens when you update your distro each week... Each week do you perform a download, an extraction and an installation, and you don't need to download 52 packages in one year only so you can install them in the next year. This is what conflict resolution does for you.

[starshipeleven]

Here is an illustration for you.

And here is one for you:

[B]Unsafe interleaved download and installation sequence:[/B]
D1 D2 D3 D4 D5
  X1 X2 X3 X4 X5
    I1 I2 I3 I4 I5
|---- Timeline ---->

[B]Safe interleaved download and installation sequence:[/B]
D1 D2 D3 D4 D5
  X1 X2 X3 X4 X5
                 I1 I2 I3 I4 I5
|---- Timeline ---->

The installation can start ONLY when all stuff is on disk, not before, or you risk breakage if you can't finish downloading.

Sure this is a case where the package 1 is a dependency of package 2 which is a dependency of package 3 and so on, so it may not show fully how it's gonna be in a real situation where you download more than one dependency chain at the time.

Contrary to your own wrong opinion, this way you still see most if not all potential benefit from interleaving (downloading/extracting is the most time-consuming part usually), but you are making it in a safe way for updates so you don't screw up the system if the download cannot finish for some reason.

A more realistic illustration, where we have 2 different dependency chains (dep chain 1 and dep chain 2), would be like this:

[B]Safe interleaved download and installation sequence:[/B]
D1.1 D1.2 D1.3
  X1.1 X1.2 X1.3
                I1.1 I1.2 I1.3
D2.1 D2.2 D2.3 D2.4 D2.5 D2.6
  X2.1 X2.2 X2.3 X2.4 X2.5 X2.6
                               I2.1 I2.2 I2.3 I2.4 I2.5 I2.6
|---- Timeline ---->

The order in which the packages are being installed (I1...I5) is preserved with interleaving.

Yes, the issue I'm talking about here is if you are UPDATING packages, not if you are installing new ones.

When you update packages you have all old packages installed already and if you start installing new stuff before you have finished downloading, you risk to end up with a system where you have newer packages as dependencies of older packages, causing breakage.

Therefore, installation MUST be done ONLY after everything has been downloaded and extracted and is ready. Or you must handle differently updating from simple installation from scratch of new packages (as if it is installing from scratch what you say is fine, it's an issue only if updating packages), anyway you must keep this point in mind or you risk breakage.

[sdack]

The installation can start ONLY when all stuff is on disk, not before, or you risk breakage if you can't finish downloading.

No. Only the package currently being installed needs to be present on disk. When you have two packages A and B, and A depends on B, then B needs to be installed before to A. That's all. B does not depend on A. If A and B depended on each other then you'd have a circular dependency, which isn't allowed by definition.

[starshipeleven]

No. Only the package currently being installed needs to be present on disk.

How about you read my post as I'm talking ot the situation where you are UPGRADING packages that ARE INSTALLED ALREADY?
Where you have already package 1 2 3 4 and you need to upgrade (i.e. overwrite) them with newer?

If you allow installation BEFORE download/extraction is complete and internet connection fails or there are other issues, you may end with new package 1 and 2 but with old 3 and 4. Many programs don't like mixing old and new packages, and you cannot assume it won't cause breakage.

[franglais125]

If you allow installation BEFORE download/extraction is complete and internet connection fails or there are other issues, you may end with new package 1 and 2 but with old 3 and 4. Many programs don't like mixing old and new packages, and you cannot assume it won't cause breakage.

sdack doesn't seem to get your point. Perhaps he would understand with an example?

For sdack:
Example (these packages are fictitious and simplified for obvious reasons):
1) I want to UPGRADE xserver, mesa and networkmanager.
2) My package manager decides to start with networkmanager.
3) Here is the current situation:
- networkmanager was downloaded, extracted and about to finish installation.
- xserver was downloaded, extracted and is BEING installed
- mesa is currently downloading
4) OOPS: connection interrupted, due to networkmanager. Restart required...
Now here is the current situation:
- netwokrmanager: awaiting reboot
- xserver: was being installed... has to finish
- mesa: couldn't download!
5) Now xserver finishes installation, you have to reboot... (Remember, no internet!)
6) Kaboom: xserver is incompatible with old mesa (which was never downloaded), you have no graphics


Now, you can interject and say that you open whatever terminal, or that package managers should be intelligent enough to know the smart order. In any case, either:
- You make life hard for non-savvy users
- You acknowledge that this is non-trivial, and some extra logic must be added to sorting updates.

In the end, what happens is that there are more and more corner cases like this to consider.

[sdack]

- xserver: was being installed... has to finish
- mesa: couldn't download!
5) Now xserver finishes installation, you have to reboot... (Remember, no internet!)
6) Kaboom: xserver is incompatible with old mesa (which was never downloaded), you have no graphics

This situation by definition would require xserver to depend on mesa. Mesa would have to be installed before xserver and not afterwards, or else would your package manager have resolved the dependencies wrong.

Just realize that dependencies form a tree, not a graph. In a tree can you always find a starting point for your sequence. If this is an update or a new installation is irrelevant for the problem. If it was a graph then you could end up having circular dependencies, which would make it impossible to install anything safely - you could not determine which package would need to be installed before another. This is why circular dependencies aren't allowed by definition or else would your packages be unmanageable.

By the way, you can install an xserver with and without mesa. It's not a problem for a package manager. And you can install an xserver with the wrong driver or have no xserver installed at all. Some of the things a package manager allows may seem wrong to a user, but aren't from a package manager's point of view and are allowed.

[franglais125]

This situation by definition would require xserver to depend on mesa. Mesa would have to be installed before xserver and not afterwards, or else would your package manager have resolved the dependencies wrong.

Just realize that dependencies form a tree, not a graph. In a tree can you always find a starting point for your sequence. If this is an update or a new installation is irrelevant for the problem. If it was a graph then you could end up having circular dependencies, which would make it impossible to install anything safely - you could not determine which package would need to be installed before another. This is why circular dependencies aren't allowed by definition or else would your packages be unmanageable.

It is not as simple as you try to put it there. I understand what you mean, but it's not as simple as "A depends on B, therefore install B first". As I said, my example was a simplification.

Particularly:
mutter in debian (say jessie for instance: https://packages.debian.org/jessie/mutter) requires mutter-common to be EQUAL to some version, and note that mutter-common is the dependency here.
Now same example as above: you upgrade mutter-common, but mutter never finished the download. Voila: you end up with a broken system, even if things were installed following the dependency order.

By the way, you can install an xserver with and without mesa. It's not a problem for a package manager. And you can install an xserver with the wrong driver or have no xserver installed at all. Some of the things a package manager allows may seem wrong to a user, but aren't from a package manager's point of view and are allowed.

I'm aware of this, as I said in the example, it was simplified and fictitious. This is beside the point anyway... Replace xserver and mesa with whatever combination of packages which results in a broken system upon booting.


============

In the end you can come up with sets of packages to be downloaded before proceeding to installation (like mutter and mutter-common should be downloaded together, etc.). But as you see, the logic to follow keeps only getting worse: dependencies, block of dependencies, does it affect network?
You can say "hey, you can install package A without any problems since it shouldn't affect the network". Well, shit happens (also known as bugs). Your network goes down (because bugs, misconfiguration, whatever), then your system is screwed.

If I were a maintainer for a package, I wouldn't want to be held responsible for screwing someone else's installation process. I would be extremely wary of saying something like "100% you can install this in parellel".

[sdack]

It is not as simple as you try to put it there. ...

It is and this is what you don't get. You only imagine it would be a lot more complicated, because you don't really know how it works and that is already blocking you from understanding it. You could not possibly manage 100,000 packages without setting up some rules and having ways to work around problems.

Take gtk2 and gtk3 for example... Obviously is gtk3 the newer version, but it comes with a different API and one which breaks with the older. So how was this solved? It was solved by allowing both versions to exist as two distinct packages even though one is technically the upgrade to the other. This solution allows an application to depend on the old version while another application depends on the newer version. It also allows upgrading gtk2 and gtk3 as long as their API doesn't change. It also allows to upgrade an application to the newer API without affecting other applications that still need gtk2.

This isn't a new problem and you'll probably already have seen it. So I'm wondering what it is that is holding you back here.

Now do yourself a favour and don't just look at mutter, but do look at mutter-common. Can you see that mutter depends on mutter-common, but that mutter-common does not depend on mutter? It means you have to install mutter-common before mutter and that you cannot install mutter before you've installed mutter-common. So you do download mutter-common, then you install it, and while it's installing can you start downloading mutter. Once mutter-common has finished installing and mutter finished downloading do you install mutter. What then happens when mutter doesn't download is that you simply have mutter-common installed (to no effect) and it will show up as auto-removeable until you've finally installed mutter. It's no different from a user cancelling an installation or a power outage. A newer mutter-common will also not break anything, because it needs to be compatible by definition, or else come as a new and distinct package just as it's always been done.

[franglais125]

Now do yourself a favour and don't just look at mutter, but do look at mutter-common. Can you see that mutter depends on mutter-common, but that mutter-common does not depend on mutter? It means you have to install mutter-common before mutter and that you cannot install mutter before you've installed mutter-common. So you do download mutter-common, then you install it, and while it's installing can you start downloading mutter. Once mutter-common has finished installing and mutter finished downloading do you install mutter. What then happens when mutter doesn't download is that you simply have mutter-common installed (to no effect) and it will show up as auto-removeable until you've finally installed mutter. It's no different from a user cancelling an installation or a power outage. A newer mutter-common will also not break anything, because it needs to be compatible by definition, or else come as a new and distinct package just as it's always been done.

Sure, mutter-common is installed before, just as you say, we agree there. I didn't say the opposite.
If you are trying to install mutter and mutter-common, this might not be so much of a problem.

So, maybe I wasn't clear: I'm assuming an upgrade, not an installation (as starshipeleven points out).
Yes, it does make a difference. Not from the package manager, but from the end user POV.
If someone is installing it, then it didn't have it before, probably didn't needed the package in question to even install it.
If someone is upgrading (a critical package like say gnome-shell, linux-image, etc), they need the package the package in question to have a functional pc in the first place.

So, for an end-user upgrading his system, mutter-common won't be simply "auto-removable" end of the story.
For a non-savvy end-user it could mean a broken installation and no way of repairing it.

Let me give you an example of what could happen:

1) My old grandpa decides to update his system with his new package manager that works as you explain.
2) The package manager upgrades mutter-common, but another concurrent update breaks the connection, and mutter never gets downloaded.
3) So a reboot is necessary to bring the pc online again.
4) He reboots and gets a broken system, can't login.
5) Calls me: "my computer broke".
6) Me: "..."

See? That's what could happen with a packagame manager that tries to be too smart.
Sure, myself I won't care as I can work my way around such problems. But you have to think of what would happen if someone can't.

At this point I don't care that mutter-common is autoremovable. From my grandpa's perspective his pc is broken.