Rust-Written Coreutils 0.0.25 With Improved GNU Compatibility

[pabloski]

coreutils is one thing, findutils is another etc. Busybox utilities composes a lot of "utils/tools" suites, from coreutils, to net-tools, procps, and many more. even if you were to combine all of uutils projects into a single binary it wouldn't be a complete busybox replacement, but it would be closer.

I see your point, but splitting the userland is a precise choice. Regrouping the parts would pollute coreutils. What is needed is another project, like the old Rustybox. Sadly this one didn't go forward and was abandoned.

The lucky part is that Rust projects are always made of a library part and an executable implementing the interface and other small bits. So you can take coreutils, findutils, procps, diffutils, etc... and build a Busybox-like solution.

But honestly few people feel the need for it. I was a Busybox enthusiast in the old days, but today I just want a different userland from the GNU one. And I am more into a BSD userland and less into other solutions. I like Chimera Linux for this reason, but they aren't exactly having success in their endeavour.

So for now I accept to use the bloated GNU userland on my main workstation, with two other machines running FreeBSD!

[anarki2]

reposting for unnapproved bot

With the "full uutils" suite, I want to see them all compiled into a single binary, kind of like a psuedo busybox replacement, it's not everything, but it's getting there, they currently have coreutils, findutils, procps, acl, util-linux, bsdutils and ofc diff-utils.

Compiling that into a single binary could be quite nice.

Why? So that you have to patch 50 megs instead of 500 kB whenever there's a fix for a single util?

[Quackdoc]

I see your point, but splitting the userland is a precise choice. Regrouping the parts would pollute coreutils. What is needed is another project, like the old Rustybox. Sadly this one didn't go forward and was abandoned.

The lucky part is that Rust projects are always made of a library part and an executable implementing the interface and other small bits. So you can take coreutils, findutils, procps, diffutils, etc... and build a Busybox-like solution.

But honestly few people feel the need for it. I was a Busybox enthusiast in the old days, but today I just want a different userland from the GNU one. And I am more into a BSD userland and less into other solutions. I like Chimera Linux for this reason, but they aren't exactly having success in their endeavour.

So for now I accept to use the bloated GNU userland on my main workstation, with two other machines running FreeBSD!

busybox/toybox are still fundemental tools for embedded and otherwise lower storage devices. A single binary could stand to save a lot of space then multiple binaries.

Why? So that you have to patch 50 megs instead of 500 kB whenever there's a fix for a single util?

having a single statically built binary with LTO can save a lot of space

[archkde]

Why? So that you have to patch 50 megs instead of 500 kB whenever there's a fix for a single util?

More like you have 1 binary with 2 MB instead of dozens with 500 kB each. (I guess the absolute numbers might be quite a bit higher here though because of Rust.)

[q66_]

But honestly few people feel the need for it. I was a Busybox enthusiast in the old days, but today I just want a different userland from the GNU one. And I am more into a BSD userland and less into other solutions. I like Chimera Linux for this reason, but they aren't exactly having success in their endeavour.

what would count as "having success"? ~10000 commits per year by ~40 authors and packaging close to 2000 different software on 5 CPU architectures including all major web browsers and other tricky stuff in alpha phase not enough?

[Quackdoc]

More like you have 1 binary with 2 MB instead of dozens with 500 kB each. (I guess the absolute numbers might be quite a bit higher here though because of Rust.)

I gotchu, putting it in quotes so it's easier to read, also note that I am not doing any additonal things to save file size, no upx, no full stripping, no rebuilding std, no codegen units etc. so there is plenty of headroom left to play with and optimize away since cargo really doesn't do a good job at that by default (which is fair imo, it's not like too many people actually care now).

uutils actually uses fat lto by default so setting thin will cause it to baloon up in size a bit

RUSTFLAGS="-C strip=debuginfo" cargo build --release --features unix: (Single binary) 10.76Mb

RUSTFLAGS="-C strip=debuginfo" cargo build --release -p uu_arch -p uu_base32 -p uu_base64 -p uu_basename -p uu_basenc -p uu_cat -p uu_chgrp -p uu_chmod -p uu_chown -p uu_chroot -p uu_cksum -p uu_comm -p uu_cp -p uu_csplit -p uu_cut -p uu_date -p uu_dd -p uu_df -p uu_dir -p uu_dircolors -p uu_dirname -p uu_du -p uu_echo -p uu_env -p uu_expand -p uu_expr -p uu_factor -p uu_false -p uu_fmt -p uu_fold -p uu_groups -p uu_hashsum -p uu_head -p uu_hostid -p uu_hostname -p uu_id -p uu_install -p uu_join -p uu_kill -p uu_link -p uu_ln -p uu_logname -p uu_ls -p uu_mkdir -p uu_mkfifo -p uu_mknod -p uu_mktemp -p uu_more -p uu_mv -p uu_nice -p uu_nl -p uu_nohup -p uu_nproc -p uu_numfmt -p uu_od -p uu_paste -p uu_pathchk -p uu_pinky -p uu_pr -p uu_printenv -p uu_printf -p uu_ptx -p uu_pwd -p uu_readlink -p uu_realpath -p uu_rm -p uu_rmdir -p uu_seq -p uu_shred -p uu_shuf -p uu_sleep -p uu_sort -p uu_split -p uu_stat -p uu_stdbuf -p uu_sum -p uu_sync -p uu_tac -p uu_tail -p uu_tee -p uu_test -p uu_timeout -p uu_touch -p uu_tr -p uu_true -p uu_truncate -p uu_tsort -p uu_tty -p uu_uname -p uu_unexpand -p uu_uniq -p uu_unlink -p uu_uptime -p uu_users -p uu_vdir -p uu_wc -p uu_who -p uu_whoami -p uu_yes: (Many binaries) 111.21MiB

Single binary with lto=thin: 14.84 Mib

Multiple binaries with lto=thin:135.04

uutils does have a profile small and profile fast release configs, with the profiles flags being listed below, to give into context, compiling the multi call binary with default lto and release-small is 7.04 MiB, which is not bad, but it could be made better by setting codegen units and building std and a couple other things

[profile.release-fast]
inherits = "release"
panic = "abort"

# A release-like profile that is as small as possible.
[profile.release-small]
inherits = "release"
opt-level = "z"
panic = "abort"
strip = true

[bug77]

Does this yield any measurable performance improvements over GNU Coreutils?

If it's anything like ripgrep, hell yeah. But I suspect ripgrep is a best-case scenario.
However, you are asking the wrong question here. Since the UNIX (and, by extension Linux) philosophy is about small, specialized utilities that work together to get a job done, having safe, small utilities working together makes is that much harder to exploit command chaining.

[imaami]

So is this also not dynamically linked to system libraries?

[Blademasterz]

The downside of rust and rust programs for me is the compile time even for a simple programs it takes twice or even 3 times longer than C++ written porgrams

[imaami]

I had a look at the Debian Sid packages for coreutils and rust-coreutils. Normal coreutils is separate executables, while rust-coreutils contains a single binary and symlinks.

Normal GNU coreutils is 6.287 MiB, all executables _combined_. The single binary in rust-coreutils is 20.998 MiB. And the single-binary build is supposed to be the size-optimized choice. If the rust version was separate binaries, it would be 20x larger in total.

Rust binaries are ridiculous bloat as long as they can't figure out a stable ABI and dynamic linking.

Someone here was complaining about GNU coreutils being bloat. Not sure how anyone can say that with a straight face when looking at the compiled result.