Announcement

Collapse
No announcement yet.

Arch Linux's Pacman 6.0 Enters Alpha With Parallel Downloads Support

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #41
    Originally posted by dirlewanger88

    No, it really doesn't. The slowness is due to Python being dog slow and the developers of dnf being incompetent.

    C O P E. dnf is slow as molasses and basically unusable.
    Python & Rust & C++ Oh My.

    Comment


    • #42
      Originally posted by dirlewanger88

      No, it really doesn't. The slowness is due to Python being dog slow and the developers of dnf being incompetent.

      C O P E. dnf is slow as molasses and basically unusable.
      effectively instant dependency resolution, normal download times for new installations and faster than normal download times for updates, sub-second package installations. Oh no it's so terribly slow it's utterly unusable .

      dnf, zypper, and other RPM based package managers don't really have a noticeable time difference vs pacman, meanwhile they're faster than debian based because dpkg takes forever and if you're unlucky gentoo can takes minutes to resolve dependencies.

      Comment


      • #43
        Originally posted by schmidtbag View Post
        But if you compare the size of the file to the download rate, you're not really losing any performance. If you download a 500KB file in less than a second, you've technically only transferred 500KB/s, so it's going to make the download rate seem slower than it actually is. So, if you're downloading 10x 500KB files in parallel and another batch where you download them in serial, as long as your network connection isn't bottlenecked, they should all complete at roughly the same duration.
        EDIT:
        Whoops made a mistake in my last sentence.
        If my connection can handle speeds of up to 10MB/s and I want to download 10 small files of 1MB each, then if I download them serially my bandwidth goes wasted because all of it will go into downloading each single file, so in practice the download speed will never reach its full 10 MB/s potential and I'll still need 10 seconds to download these 10 files. But if I have the option to download them in parallel, it means I can have them downloaded in a single second, with my full bandwidth being utilized during that time period, i.e. 1MB/s for each file (obviously without taking into consideration stuff like overhead, speed fluctuations, server capacity etc).

        Originally posted by skeevy420 View Post
        While the AUR might be unofficial, some distributions that use pacman include AUR support in their tools and pacman wrappers so AUR upgrade speed does factor in to an Arch upgrade to some extent. My standard process is pacman for system updates, aurutils to build and repo up AUR updates, pacman again to install my AUR updates.

        FWIW, Arch doesn't give the ability to automagically compile and install them via the system's package manager or even have an official tool to do it. The actual Arch Way is to enter a clean chroot, build it there with makepkg, and then install it on the actual system with pacman. I suppose there's makepkg -sif outside of the chroot, but, IMHO, that should be treated more like a quick hack than the proper way.
        Sure, I didn't mean to say that the AUR should not be treated as part of Arch, in fact it's one of the main reasons to use Arch in the first place (at least until flakpaks really take off and make it more or less useless - a sad day that will be). All I'm saying is that compiling stuff from sources is a time bottleneck that has nothing to do with how fast pacman can resolve dependencies or install a package. I could be running Arch on an ancient Core 2 Duo or on a bleeding edge Ryzen 5900x, and if I tried to install a custom kernel from the AUR, then pacman would be more or less just as fast resolving the dependencies and then installing the end result on both of them - but the total compilation time of that kernel... hoo boy!

        Regarding the automagic: makepkg is very much an official tool, as it's the tool that builds the packages in the official Arch repos, which pacman then downloads and installs. And it's a tool that allows you to create a PKGBUILD for whatever may tickle your fancy (variations of packages in the repos, obscure open source projects that no one cares about enough to create a package for, your own pet projects, etc) and make it into a normal system package complete with dependency tracking and (if you upload it to the AUR) update handling by pacman. I'd say this counts as Arch giving us the ability to automagically compile and install stuff via the system's package manager. Also, as an aside, I don't consider makepkg -si a quick hack, it's what I've always used whenever I install PKGBUILDs manually

        Comment


        • #44
          Originally posted by Nocifer View Post
          If my connection can handle speeds of up to 10MB/s and I want to download 10 small files of 1MB each, then if I download them serially my bandwidth goes wasted because all of it will go into downloading each single file, so in practice the download speed will never reach its full 10 MB/s potential and I'll still need 10 seconds to download these 10 files. But if I have the option to download them in parallel, it means I can have them downloaded in a single second, with my full bandwidth being utilized during that time period, i.e. 1MB/s for each file (obviously without taking into consideration stuff like overhead, speed fluctuations, server capacity etc).
          How would the bandwidth go wasted? It seems like what you're implying is those 10x 1MB files would take 1 second to download, which just simply isn't true. In IPv4, each packet has 20 bytes of data and I believe IPv6 supports a 3KB payload. The speed of your internet determines how quickly each packet is sent, but in both cases they're a small chunk of data. If your connection speed is 10MB/s, that doesn't mean you instantaneously download 10MB per file, it means that's your maximum bandwidth. So if you're downloading 10x 1MB files, they will download in approximately 1 second.

          Comment


          • #45
            Originally posted by Nocifer View Post
            Sure, I didn't mean to say that the AUR should not be treated as part of Arch, in fact it's one of the main reasons to use Arch in the first place (at least until flakpaks really take off and make it more or less useless - a sad day that will be). All I'm saying is that compiling stuff from sources is a time bottleneck that has nothing to do with how fast pacman can resolve dependencies or install a package. I could be running Arch on an ancient Core 2 Duo or on a bleeding edge Ryzen 5900x, and if I tried to install a custom kernel from the AUR, then pacman would be more or less just as fast resolving the dependencies and then installing the end result on both of them - but the total compilation time of that kernel... hoo boy!

            Regarding the automagic: makepkg is very much an official tool, as it's the tool that builds the packages in the official Arch repos, which pacman then downloads and installs. And it's a tool that allows you to create a PKGBUILD for whatever may tickle your fancy (variations of packages in the repos, obscure open source projects that no one cares about enough to create a package for, your own pet projects, etc) and make it into a normal system package complete with dependency tracking and (if you upload it to the AUR) update handling by pacman. I'd say this counts as Arch giving us the ability to automagically compile and install stuff via the system's package manager. Also, as an aside, I don't consider makepkg -si a quick hack, it's what I've always used whenever I install PKGBUILDs manually
            If Flats get all their printing and peripheral device issues solved then maybe they'll take off. Until then I'll keep on treating them like glorified demo programs -- If you want printing support, please "purchase" this program from the Apt Store. Even if flats do get better, the AUR will still be there. I doubt that Mesa, the kernel, etc will start Flatting up RC and git masters of their software. Shit. Flat a Kernel. Lol. That's evidence enough that the AUR will be there.

            I'm not saying that makepkg isn't official. But building from the AUR with just makepkg -si is unofficial and a quick hack when compared to the technically correct way...even though most all of us do it that way and that's basically how most all the AUR Helpers work... The technically correct way is to create a clean chroot, enter that, build our package with makepkg in there, and then install that built package to the main system. It's a lot of extra steps, but it's to ensure you don't pollute your system with lots of unnecessary build dependencies, pollute the build with data from the active OS, or mess up the system due to a build error or a nefariously written PKGBUILD. If you don't plan on sharing your packages, makepkg -si is usually fine. Things like Mesa, however, have sticked warnings in the comment section from the maintainer about how you shouldn't build it on the active system and only in a clean chroot.

            Comment


            • #46
              Stopped using Arch, but Arch is the best distro out there, as worldwide polls confirm.[1][2][3][4]

              [1] = <Error! You need to be registered donator to view this link.>
              [2] = <Error! you need to be registered donator to view this link.>
              [3] = <Error! you need to be registered donator to view this link.>
              [4] = <Error! you need to be registered donator to view this link.>

              Comment


              • #47
                Originally posted by skeevy420 View Post
                The technically correct way is to create a clean chroot, enter that, build our package with makepkg in there, and then install that built package to the main system. It's a lot of extra steps, but it's to ensure you don't pollute your system with lots of unnecessary build dependencies, pollute the build with data from the active OS, or mess up the system due to a build error or a nefariously wri.....
                Yeah. No. Nonsense. Use yay and:
                Code:
                $ pacman -Qtdq
                Last edited by ArchLinux; 07 December 2020, 11:15 AM.

                Comment


                • #48
                  Originally posted by schmidtbag View Post
                  It seems like what you're implying is those 10x 1MB files would take 1 second each to download, which just simply isn't true. [...] So even if you're downloading 10x 1MB files serially, they will download in approximately 1 second.
                  If you meant to say it like that (with the bold words I've added) then your answer makes sense, otherwise it's a bit of a contradiction. Anyway, I'm not saying that downloading a 1MB file over a 10MB/s (or 100Mbps if you prefer) connection would take a full 1 second (that number was just a figure of speech), but it certainly wouldn't take the 0.1 seconds that you seem to be implying yourself, because the connection doesn't begin the download at a speed of 10MB/s but at 0MB/s, and it takes some time to increase.

                  Because of that, and also because as I've already said the connection speed constantly resets after each file, when downloading 10x 1MB files serially it never has the chance to reach its full 10MB/s speed for each file, so downloading them all ends up taking much more than the theoretical 1 second. On the other hand, when downloading those same files in parallel, the connection ends up utilizing the full bandwidth of your connection, so downloading 10x 1MB files takes much closer to the theoretical 1 second (theoretical because in this case, too, it takes some time for the connection to increase its speed from 0MB/s).

                  Originally posted by skeevy420 View Post

                  If Flats get all their printing and peripheral device issues solved then maybe they'll take off. Until then I'll keep on treating them like glorified demo programs -- If you want printing support, please "purchase" this program from the Apt Store. Even if flats do get better, the AUR will still be there. I doubt that Mesa, the kernel, etc will start Flatting up RC and git masters of their software. Shit. Flat a Kernel. Lol. That's evidence enough that the AUR will be there.

                  I'm not saying that makepkg isn't official. But building from the AUR with just makepkg -si is unofficial and a quick hack when compared to the technically correct way...even though most all of us do it that way and that's basically how most all the AUR Helpers work... The technically correct way is to create a clean chroot, enter that, build our package with makepkg in there, and then install that built package to the main system. It's a lot of extra steps, but it's to ensure you don't pollute your system with lots of unnecessary build dependencies, pollute the build with data from the active OS, or mess up the system due to a build error or a nefariously written PKGBUILD. If you don't plan on sharing your packages, makepkg -si is usually fine. Things like Mesa, however, have sticked warnings in the comment section from the maintainer about how you shouldn't build it on the active system and only in a clean chroot.
                  Regarding Flatpaks, yes, they're not much to speak of for the time being, and there's some stuff that will always need to be installed in the normal way (basically all the GUI-less backend-type stuff, from kernels to databases to compilers to basic shell utilities), but the AUR iswhat it is because of the vast amount of user facing apps it contains, not because of the XanMod kernel or the multiple variations of vim that someone somewhere decided to upload. Basically the AUR is a perfect alternative to stuff like PPAs. And because I seriously expect that within the next ~5 years or so we'll be using Flatpaks like there's no tomorrow, and because Flatpaks were created precisely to take the place of stuff like PPAs, I also expect that the AUR will still be there, sure, but it will be a repo for the more technically oriented users like us and not the envy-inducing, user-friendly, vastly huge "PPA" that it is today.

                  Regarding makepkg, I'm aware of all that stuff, and I again understand what you're saying; I was simply discussing it for the sake of discussing it, it has nothing to do with the discussion about pacman's speed. As for that, I still believe that package compilation time is completely separate from dependency resolution and package installation, the two things that usually count as "speed" when talking about packagers managers; and because it's also a) reliant on hardware power and b) completely optional and a side effect of the user's choice to use the AUR, I don't really think it's fair for it to be measured as part of the overall updating process.

                  And to drive my point home better, I'd probably say the same even if we were talking about Gentoo's package manager, because even there, package compilation speed is different from the speed with which you resolve dependencies, decide what to compile, and finally install the end result. Although in this case I'd of course accept that compilation time can and should be counted as a (negative) part of the overall updating process.

                  Originally posted by ArchLinux View Post
                  Stopped using Arch, but Arch is the best distro out there, as worldwide polls confirm.
                  Might I ask then why you stopped using it..?
                  Last edited by Nocifer; 07 December 2020, 06:51 PM. Reason: There were more posts in need of a reply.

                  Comment


                  • #49
                    Originally posted by Nocifer View Post
                    If you meant to say it like that (with the bold words I've added) then your answer makes sense, otherwise it's a bit of a contradiction. Anyway, I'm not saying that downloading a 1MB file over a 10MB/s (or 100Mbps if you prefer) connection would take a full 1 second (that number was just a figure of speech), but it certainly wouldn't take the 0.1 seconds that you seem to be implying yourself, because the connection doesn't begin the download at a speed of 10MB/s but at 0MB/s, and it takes some time to increase.

                    Because of that, and also because as I've already said the connection speed constantly resets after each file, when downloading 10x 1MB files serially it never has the chance to reach its full 10MB/s speed for each file, so downloading them all ends up taking much more than the theoretical 1 second. On the other hand, when downloading those same files in parallel, the connection ends up utilizing the full bandwidth of your connection, so downloading 10x 1MB files takes much closer to the theoretical 1 second (theoretical because in this case, too, it takes some time for the connection to increase its speed from 0MB/s).
                    Yes, I see how you're confused by what I said, I wasn't being very clear. The TLDR is: whether you have a single 10MB file, 10x 1MB files downloaded serially, or 10x 1MB files downloaded in parallel, they will in total complete in about 1 second if your download rate is 10MB/s.
                    The speed doesn't take time to increase and the speed doesn't "reset" after each file either; unless you have some crappy ISP, you get your full bandwidth immediately. The reason you probably think the transfer rate "accelerates" is because applications tend to tell you an average speed. If you haven't yet downloaded 10MB for a single transfer, well, how can it truthfully tell you that your maximum download speed is 10MB/s? As mentioned before, data is transferred via packets, which don't contain a lot of data. Of programs that do immediately show you the right download speed, they probably estimate the download rate based on how quickly those packets are sent and received. That being said, sometimes they estimate incorrectly. My internet's maximum speed is about 14MB/s but I've seen small files download at 17MB/s. It didn't literally download that quickly (it didn't even download that much data), but it was the closest estimation that could be made in such a short amount of time. Anyway, the small size of those packets is convenient, because it allows you to download from multiple different sources at once, which is why you're able to stream videos or music while downloading a file and not forcing one to be paused. It seems like you're downloading in parallel, but you're really not. Network connections are serialized (or full-duplex serialized). So, when you're downloading 10 files simultaneously, what you're really doing is just downloading one file for a short burst, pausing it, switching to another one to download in a short burst, pause that, and so on. So, if network speeds were to ramp up speed in the way you suggest, they would always be at their slowest speed when downloading multiple files.

                    Although it isn't significant, it's actually more likely that downloading files in parallel would slow things down, because drives on both ends have to spend more time seeking, and the CPUs have to synchronize with the tasks downloading the information. But, since networks tend to be far slower than the rest of the system, this performance difference is negligible. And of course, if you have some crappy ISP like skeevy420 has where they put a speed cap per file, then yeah, downloading multiple files is definitely faster.

                    Comment


                    • #50
                      Originally posted by dirlewanger88
                      Isn't it funny how rank amateurs are always the most vocal...?
                      It's more funny when people insist they know better yet never elaborate. Prove me wrong.

                      Comment

                      Working...
                      X