Originally posted by dirlewanger88
Announcement
Collapse
No announcement yet.
Arch Linux's Pacman 6.0 Enters Alpha With Parallel Downloads Support
Collapse
X
-
Originally posted by dirlewanger88
No, it really doesn't. The slowness is due to Python being dog slow and the developers of dnf being incompetent.
C O P E. dnf is slow as molasses and basically unusable.
dnf, zypper, and other RPM based package managers don't really have a noticeable time difference vs pacman, meanwhile they're faster than debian based because dpkg takes forever and if you're unlucky gentoo can takes minutes to resolve dependencies.
Comment
-
Originally posted by schmidtbag View PostBut if you compare the size of the file to the download rate, you're not really losing any performance. If you download a 500KB file in less than a second, you've technically only transferred 500KB/s, so it's going to make the download rate seem slower than it actually is. So, if you're downloading 10x 500KB files in parallel and another batch where you download them in serial, as long as your network connection isn't bottlenecked, they should all complete at roughly the same duration.
EDIT:
Whoops made a mistake in my last sentence.
Originally posted by skeevy420 View PostWhile the AUR might be unofficial, some distributions that use pacman include AUR support in their tools and pacman wrappers so AUR upgrade speed does factor in to an Arch upgrade to some extent. My standard process is pacman for system updates, aurutils to build and repo up AUR updates, pacman again to install my AUR updates.
FWIW, Arch doesn't give the ability to automagically compile and install them via the system's package manager or even have an official tool to do it. The actual Arch Way is to enter a clean chroot, build it there with makepkg, and then install it on the actual system with pacman. I suppose there's makepkg -sif outside of the chroot, but, IMHO, that should be treated more like a quick hack than the proper way.
Regarding the automagic: makepkg is very much an official tool, as it's the tool that builds the packages in the official Arch repos, which pacman then downloads and installs. And it's a tool that allows you to create a PKGBUILD for whatever may tickle your fancy (variations of packages in the repos, obscure open source projects that no one cares about enough to create a package for, your own pet projects, etc) and make it into a normal system package complete with dependency tracking and (if you upload it to the AUR) update handling by pacman. I'd say this counts as Arch giving us the ability to automagically compile and install stuff via the system's package manager. Also, as an aside, I don't consider makepkg -si a quick hack, it's what I've always used whenever I install PKGBUILDs manually
Comment
-
Originally posted by Nocifer View PostIf my connection can handle speeds of up to 10MB/s and I want to download 10 small files of 1MB each, then if I download them serially my bandwidth goes wasted because all of it will go into downloading each single file, so in practice the download speed will never reach its full 10 MB/s potential and I'll still need 10 seconds to download these 10 files. But if I have the option to download them in parallel, it means I can have them downloaded in a single second, with my full bandwidth being utilized during that time period, i.e. 1MB/s for each file (obviously without taking into consideration stuff like overhead, speed fluctuations, server capacity etc).
Comment
-
Originally posted by Nocifer View PostSure, I didn't mean to say that the AUR should not be treated as part of Arch, in fact it's one of the main reasons to use Arch in the first place (at least until flakpaks really take off and make it more or less useless - a sad day that will be). All I'm saying is that compiling stuff from sources is a time bottleneck that has nothing to do with how fast pacman can resolve dependencies or install a package. I could be running Arch on an ancient Core 2 Duo or on a bleeding edge Ryzen 5900x, and if I tried to install a custom kernel from the AUR, then pacman would be more or less just as fast resolving the dependencies and then installing the end result on both of them - but the total compilation time of that kernel... hoo boy!
Regarding the automagic: makepkg is very much an official tool, as it's the tool that builds the packages in the official Arch repos, which pacman then downloads and installs. And it's a tool that allows you to create a PKGBUILD for whatever may tickle your fancy (variations of packages in the repos, obscure open source projects that no one cares about enough to create a package for, your own pet projects, etc) and make it into a normal system package complete with dependency tracking and (if you upload it to the AUR) update handling by pacman. I'd say this counts as Arch giving us the ability to automagically compile and install stuff via the system's package manager. Also, as an aside, I don't consider makepkg -si a quick hack, it's what I've always used whenever I install PKGBUILDs manually
I'm not saying that makepkg isn't official. But building from the AUR with just makepkg -si is unofficial and a quick hack when compared to the technically correct way...even though most all of us do it that way and that's basically how most all the AUR Helpers work... The technically correct way is to create a clean chroot, enter that, build our package with makepkg in there, and then install that built package to the main system. It's a lot of extra steps, but it's to ensure you don't pollute your system with lots of unnecessary build dependencies, pollute the build with data from the active OS, or mess up the system due to a build error or a nefariously written PKGBUILD. If you don't plan on sharing your packages, makepkg -si is usually fine. Things like Mesa, however, have sticked warnings in the comment section from the maintainer about how you shouldn't build it on the active system and only in a clean chroot.
Comment
-
Stopped using Arch, but Arch is the best distro out there, as worldwide polls confirm.[1][2][3][4]
[1] = <Error! You need to be registered donator to view this link.>
[2] = <Error! you need to be registered donator to view this link.>
[3] = <Error! you need to be registered donator to view this link.>
[4] = <Error! you need to be registered donator to view this link.>
Comment
-
Originally posted by skeevy420 View PostThe technically correct way is to create a clean chroot, enter that, build our package with makepkg in there, and then install that built package to the main system. It's a lot of extra steps, but it's to ensure you don't pollute your system with lots of unnecessary build dependencies, pollute the build with data from the active OS, or mess up the system due to a build error or a nefariously wri.....
Code:$ pacman -Qtdq
Last edited by ArchLinux; 07 December 2020, 11:15 AM.
Comment
-
Originally posted by schmidtbag View PostIt seems like what you're implying is those 10x 1MB files would take 1 second each to download, which just simply isn't true. [...] So even if you're downloading 10x 1MB files serially, they will download in approximately 1 second.
Because of that, and also because as I've already said the connection speed constantly resets after each file, when downloading 10x 1MB files serially it never has the chance to reach its full 10MB/s speed for each file, so downloading them all ends up taking much more than the theoretical 1 second. On the other hand, when downloading those same files in parallel, the connection ends up utilizing the full bandwidth of your connection, so downloading 10x 1MB files takes much closer to the theoretical 1 second (theoretical because in this case, too, it takes some time for the connection to increase its speed from 0MB/s).
Originally posted by skeevy420 View Post
If Flats get all their printing and peripheral device issues solved then maybe they'll take off. Until then I'll keep on treating them like glorified demo programs -- If you want printing support, please "purchase" this program from the Apt Store. Even if flats do get better, the AUR will still be there. I doubt that Mesa, the kernel, etc will start Flatting up RC and git masters of their software. Shit. Flat a Kernel. Lol. That's evidence enough that the AUR will be there.
I'm not saying that makepkg isn't official. But building from the AUR with just makepkg -si is unofficial and a quick hack when compared to the technically correct way...even though most all of us do it that way and that's basically how most all the AUR Helpers work... The technically correct way is to create a clean chroot, enter that, build our package with makepkg in there, and then install that built package to the main system. It's a lot of extra steps, but it's to ensure you don't pollute your system with lots of unnecessary build dependencies, pollute the build with data from the active OS, or mess up the system due to a build error or a nefariously written PKGBUILD. If you don't plan on sharing your packages, makepkg -si is usually fine. Things like Mesa, however, have sticked warnings in the comment section from the maintainer about how you shouldn't build it on the active system and only in a clean chroot.
Regarding makepkg, I'm aware of all that stuff, and I again understand what you're saying; I was simply discussing it for the sake of discussing it, it has nothing to do with the discussion about pacman's speed. As for that, I still believe that package compilation time is completely separate from dependency resolution and package installation, the two things that usually count as "speed" when talking about packagers managers; and because it's also a) reliant on hardware power and b) completely optional and a side effect of the user's choice to use the AUR, I don't really think it's fair for it to be measured as part of the overall updating process.
And to drive my point home better, I'd probably say the same even if we were talking about Gentoo's package manager, because even there, package compilation speed is different from the speed with which you resolve dependencies, decide what to compile, and finally install the end result. Although in this case I'd of course accept that compilation time can and should be counted as a (negative) part of the overall updating process.
Originally posted by ArchLinux View PostStopped using Arch, but Arch is the best distro out there, as worldwide polls confirm.Last edited by Nocifer; 07 December 2020, 06:51 PM. Reason: There were more posts in need of a reply.
Comment
-
Originally posted by Nocifer View PostIf you meant to say it like that (with the bold words I've added) then your answer makes sense, otherwise it's a bit of a contradiction. Anyway, I'm not saying that downloading a 1MB file over a 10MB/s (or 100Mbps if you prefer) connection would take a full 1 second (that number was just a figure of speech), but it certainly wouldn't take the 0.1 seconds that you seem to be implying yourself, because the connection doesn't begin the download at a speed of 10MB/s but at 0MB/s, and it takes some time to increase.
Because of that, and also because as I've already said the connection speed constantly resets after each file, when downloading 10x 1MB files serially it never has the chance to reach its full 10MB/s speed for each file, so downloading them all ends up taking much more than the theoretical 1 second. On the other hand, when downloading those same files in parallel, the connection ends up utilizing the full bandwidth of your connection, so downloading 10x 1MB files takes much closer to the theoretical 1 second (theoretical because in this case, too, it takes some time for the connection to increase its speed from 0MB/s).
The speed doesn't take time to increase and the speed doesn't "reset" after each file either; unless you have some crappy ISP, you get your full bandwidth immediately. The reason you probably think the transfer rate "accelerates" is because applications tend to tell you an average speed. If you haven't yet downloaded 10MB for a single transfer, well, how can it truthfully tell you that your maximum download speed is 10MB/s? As mentioned before, data is transferred via packets, which don't contain a lot of data. Of programs that do immediately show you the right download speed, they probably estimate the download rate based on how quickly those packets are sent and received. That being said, sometimes they estimate incorrectly. My internet's maximum speed is about 14MB/s but I've seen small files download at 17MB/s. It didn't literally download that quickly (it didn't even download that much data), but it was the closest estimation that could be made in such a short amount of time. Anyway, the small size of those packets is convenient, because it allows you to download from multiple different sources at once, which is why you're able to stream videos or music while downloading a file and not forcing one to be paused. It seems like you're downloading in parallel, but you're really not. Network connections are serialized (or full-duplex serialized). So, when you're downloading 10 files simultaneously, what you're really doing is just downloading one file for a short burst, pausing it, switching to another one to download in a short burst, pause that, and so on. So, if network speeds were to ramp up speed in the way you suggest, they would always be at their slowest speed when downloading multiple files.
Although it isn't significant, it's actually more likely that downloading files in parallel would slow things down, because drives on both ends have to spend more time seeking, and the CPUs have to synchronize with the tasks downloading the information. But, since networks tend to be far slower than the rest of the system, this performance difference is negligible. And of course, if you have some crappy ISP like skeevy420 has where they put a speed cap per file, then yeah, downloading multiple files is definitely faster.
Comment
Comment