GNU Wget2 Reaches Beta With Faster Download Speeds, New Features

[S.Pam]

(unless wget2 has a different flagset from wget)

It's not a drop in replacement.

[uxmkt]

One defining feature was the beautiful progress bar. No such thing in wget2; even curl's is not as appealing. That and there being practically no command-line option compatibility makes wget2 a flop. How many projects were rewritten, only to find themselves in limbo? It's better modifying an existing code base (even if the hardship feels longer - it usually isn't).

[blackiwid]

j

How many projects were rewritten, only to find themselves in limbo? It's better modifying an existing code base (even if the hardship feels longer - it usually isn't).

You seem to understand how opensource project often work financially. I doubt that the author of wget got any or much money from creating the "product". Others made indirectly maybe some money like Redhat and other Distris, or people that used that tools for work the end customer.

But how much do you think specifically donated to the author(s) of wget? If they have even a way to donate money directly.

So the only value software developers get out of such project is recognition and if they them self created a Tool for them self that fits their requirements.

So for many it comes down to the latter. I well tried to rewrite a tool I wrote that had some (small) recognition but because nobody gave me money you do what makes most sense for you, and if that means radical code changes and rewrites of big parts with loosing many features in the hope to get a better more robust code base you do it.

Because it's hard enough to get your tools updated and keep it in a state that is compilable / runnable.

So you measure success in a way commercial proprietary software would be measured, like user counts and stuff like that. While for opensource it often is a success if the codebase is better you need less work to keep it stable and it has all the features that you personally want.

Look at exim or other email servers. How did they become a thing? a person on a university literally copy pasted emails from one server to other computers, and then the same person wrote a script to automate that FOR HIM, that was the goal, and then because it's done why not share it, but writing what you likes is often the primary driver.

[Ardje]

That sounds really strange, the bsd sockets interface is completely agnostic regarding IPv4 vs IPv6 for everything other than setup so I don't really understand how "IPv4 was deeply in the code"?

How do you mean completely agnostic? wget and most network utilities are older than 20 years. They are based on everything IPv4. Even something simple as a ':' test in a host address can be deeply embedded, requiring a rewrite of the address processing of the utility, which is usually deeply embedded. These days we frown upon a utility doing a quick ':' strchr. Yet you can still find it all over the place.
Maybe you are to young to know there is nothing agnostic about the bsd sockets. Maybe these days they are correctly documented, and the deprecated functions are marked correctly as that: *DO NOT USE*.
Anyway: take 30 year old code, copy paste, and voila, someone made another utility. The fact that the sockets interface was not agnostic 30 years ago, still hurts code development these days. One by one libraries become agnostic. But we are still not there. And still people are advising the wrong interface.
Source: I'm doing ports/fixes for IPv6 for over 15 years when needed. My network designs for big POS setups are based on IPv6, specifically IPv6 link local.

[Niarbeht]

Everything is being parallelized as chip designs get wider and slower.

Single-core perf has been improving with time, what the actual fuck are you talking about?

[SystemCrasher]

I read "faster wget" as axel. Small funny program that happens to do one thing and do it well: faster, parallelized downloads, easily configurable. Also rather good at resuming (yes, wget, your approach sucks at it hell a lot! With wget it easy to overwrite half-downloaded huge file, being forced to start from scratch again! And at least wget "1" had no notion of resume by default, one have to explicitly request that!). Axel got no these high-profile buzzwords though, it just does its job rather than boasts.

As concrete example: I had really picky, misconfigured host that is willing to serve ~50-100kb and then suddenly drop connection for no apparent reason - and I needed large file from that server like a hell. Axel winged it. Within reasonable time frame. Something that wget failed.

[F.Ultra]

How do you mean completely agnostic? wget and most network utilities are older than 20 years. They are based on everything IPv4. Even something simple as a ':' test in a host address can be deeply embedded, requiring a rewrite of the address processing of the utility, which is usually deeply embedded. These days we frown upon a utility doing a quick ':' strchr. Yet you can still find it all over the place.
Maybe you are to young to know there is nothing agnostic about the bsd sockets. Maybe these days they are correctly documented, and the deprecated functions are marked correctly as that: *DO NOT USE*.
Anyway: take 30 year old code, copy paste, and voila, someone made another utility. The fact that the sockets interface was not agnostic 30 years ago, still hurts code development these days. One by one libraries become agnostic. But we are still not there. And still people are advising the wrong interface.
Source: I'm doing ports/fixes for IPv6 for over 15 years when needed. My network designs for big POS setups are based on IPv6, specifically IPv6 link local.

If so then you are talking about code outside of the bsd sockets interface, i.e parsing host/port strings and what not. And FYI I wrote network code long before bsd sockets where a widely used thing.

The BSD sockets interface are protocol agnostic in that once you have set the address family and host/port on the socket then all operations on that socket are completely agnostic of what you use as the underlying transport, long before IPv6 this was used for e.g both AF_UNIX and AF_INET. Hell I even had to work with AF_IPX back in the day.

edit: would be fun to know which functions that you think where non-documented back in the day and that have anything whatsoever to do with IPv4 vs IPv6 and which ones you think are deprecated?

[Ardje]

edit: would be fun to know which functions that you think where non-documented back in the day and that have anything whatsoever to do with IPv4 vs IPv6 and which ones you think are deprecated?

Then you probably know the difference between gethostbyname and getaddrinfo.
The difference is huge.
You then also probably know that handling of <hostname or address>%<zone> portnumber/name is handled totally by getaddrinfo, which gives you an address structure which contains all, including port numbers and zones.
Using gethostbyname you only get an IP, with getaddrinfo you get a struct sockaddr .
So the old way was to transfer all options seperately and parsing everything by hand, which has a huge impact in how C(!) programs are designed and what data they put in the structures.
Now to resolve or not to resolve, because most programs did not even use that. They just look for numbers and dots, if that's the case it is an ipv4 address and you use other functions.
Anyway: the way it was is that from top to bottom nobody used sockaddr.
Fixing an application like wget that was for 100% not using getaddrinfo involved a lot of hacking.
And they might have fixed a lot, but http_proxy=http://[fe80::123%eth0]:3128 is still a problem.
Now back to others: since everybody persistently used gethostbyname and family and not getaddrinfo as has been adviced since 2000 or so (none of the manuals stated the deprecated status of gethostbyname until a few years after), the bad examples became the way to program.
A good example is this:
https://www.php.net/manual/en/functi...hostbyaddr.php and look at all those functions at the right bar.
There are a lot of these examples for python, java, other languages that just did not understand.

Anyway: gethostbyname is part of the BSD socket library/API/family. So no, it was never agnostic until getaddrinfo was introduced in 1995. And that was introduced before PHP ever saw the light.

Also note that glibc had a bad implementation of a pluggable getaddrinfo, because zone info could not be forwarded between a plugin and getaddrinfo. On linux this resulted that you could not ssh to a .local address that had it's ipv6 link-local advertised. I think that should now work for a year(!) now.

Also, I never understood that people had to use strchr(..,":") or other multiline constructions to test for IPv4 while a single sscanf would have sufficed. Scraping microseconds of something that's done once.

[F.Ultra]

Then you probably know the difference between gethostbyname and getaddrinfo.
The difference is huge.
You then also probably know that handling of <hostname or address>%<zone> portnumber/name is handled totally by getaddrinfo, which gives you an address structure which contains all, including port numbers and zones.
Using gethostbyname you only get an IP, with getaddrinfo you get a struct sockaddr .
So the old way was to transfer all options seperately and parsing everything by hand, which has a huge impact in how C(!) programs are designed and what data they put in the structures.
Now to resolve or not to resolve, because most programs did not even use that. They just look for numbers and dots, if that's the case it is an ipv4 address and you use other functions.
Anyway: the way it was is that from top to bottom nobody used sockaddr.
Fixing an application like wget that was for 100% not using getaddrinfo involved a lot of hacking.
And they might have fixed a lot, but http_proxy=http://[fe80::123%eth0]:3128 is still a problem.
Now back to others: since everybody persistently used gethostbyname and family and not getaddrinfo as has been adviced since 2000 or so (none of the manuals stated the deprecated status of gethostbyname until a few years after), the bad examples became the way to program.
A good example is this:
https://www.php.net/manual/en/functi...hostbyaddr.php and look at all those functions at the right bar.
There are a lot of these examples for python, java, other languages that just did not understand.

Anyway: gethostbyname is part of the BSD socket library/API/family. So no, it was never agnostic until getaddrinfo was introduced in 1995. And that was introduced before PHP ever saw the light.

Also note that glibc had a bad implementation of a pluggable getaddrinfo, because zone info could not be forwarded between a plugin and getaddrinfo. On linux this resulted that you could not ssh to a .local address that had it's ipv6 link-local advertised. I think that should now work for a year(!) now.

Also, I never understood that people had to use strchr(..,":") or other multiline constructions to test for IPv4 while a single sscanf would have sufficed. Scraping microseconds of something that's done once.

None of which have to do with weather sockets are or are not transport agnostic. You are still talking about converting "arbitrary string representations of host/ip and port" to the values to put into the sockaddr structure, even the very fact that no such function existed makes it transport agnostic.

This is not to say that I don't agree with you that there exists many broken parsers out there, just that the bsd sockets itself is completely agnostic, the bugs that you talk about is always in the code that sets up the sockets, not in the sockets themselves.

Don't also forget that your original claim was that IPv4 was deeply entrenched in the wget network code which is what I was objecting to since IPv4 vs IPv6 is a very minor detail in the wget codebase and is only used when setting up the socket (so only done in a single place) and thus not deeply entrenched in the network code.

http_proxy is parsed using the same function as the fetch URI which handles IPv6 addresses perfectly fine.

Small disclaimer here is that I'm a code contributor to wget (although it was a long time since last).

[S.Pam]

Missing is FTP support