Making The Case For Using Rust At Low Levels On Linux Systems

[mmstick]

C++ can also export a C interface. I made a small library for IRC showing how this can be done here, which uses Boost ASIO and cppformat while providing a C interface: https://github.com/computerquip/cqirc/blob/master/

We all know that C++ can export a C interface. That isn't what the argument is referring to.

[computerquip]

We all know that C++ can export a C interface. That isn't what the argument is referring to.

Uh, you said, "C++ really isn't good for software libraries either due to C++ not having a stable exportable interface." Unless I'm taking advantage of the verbatim quote? Your argument is that C++ doesn't have, or isn't capable of, a stable exportable interface which is bologna.

[mmstick]

Some few objections & observations:
1) Low level code is not supposed to be large and complex in first place. If it is not a case, it is not a language to fix but project management I guess.

There are a lot of projects that have to be large and complex while also being written in a low level language. The Linux kernel, for example, would largely benefit from being rewritten in Rust. We could easily drastically reduce the amount of SLOC and design APIs that are more generic, and yet just as fast, much easier.

2) Strong types are double edged sword and could be annoying as well.

Rust's types are not annoying in the least. If you actually try developing software with it, it's easy to hate languages that do not provide the same level of guarantees and finesse that Rust's type system is able to pull off.

3) Furthermore, all low-level runtime environments are written in C. They are using C types anyway. Be it syscalls, low/mid level libs or something, you have to call it unless you're in mood to write whole OS and usermode from scratch (good luck with it, come back in ~10 years and show your superb achievements).

Rust's libcore interfaces directly with LLVM primitives, which is much lower level than C, and very much like a modern version of Assembly. It is what allows Rust to achieve C-like performance today, and better-than-C performance in some areas, and will allow better-than-C performance across the board in the future. SIMD support and other CPU-specific instructions can be exposed to Rust through the LLVM primitives.

4) Dealing with conversion of types is absolutely last thing one may want in low level program. Whatever, Pascal/Delphi are pain to use for low level things due to the fact virtually all OSes are using C as native implementation language. Therefore all low level calls are using C types all over place and doing conversions is no fun and even gives birth to whole new class of mistakes and errors.

Rust does not perform any sort of type conversions at runtime. Everything, including C types, are desugared to LLVM primitives and then converted to machine code. If you've ever actually tried to work with C libraries inside Rust, it is actually quite easy and convenient. It does not open up a 'whole class of mistakes and errors' in the least. The main downside is just knowing the standard quirks of C libraries such as what different error codes returned by C functions actually mean, as C libraries do not have a concept of sum types like Option and Result.

5) Sometimes it could be handy to be able to enter kernel side. A bit of kernel superpower does not hurts on low level. But wait, I could only use C to do that. Needless to say learning 2 different languages is a bit ridiculous and redundant, right?

The beauty of Rust is that you can do both programming lower level than C with libcore, and programming higher level than C++ with libstd. There's no need to have two different languages if you keep everything in Rust, which supports both low level and high level programming without hindering either's capabilities.

6) Writing perfect programs is cool. However there're thousands of C progs and libs. If you're McLeod, you would not mind rewriting everything in Rust. Mortals are really better off changing mere 20 lines of code in lib or prog which "almost does the thing". C got huge advantage here, you could literally write few lines of code yourself, while reusing the rest. Rust seems to be bad at it. It lacks bindings to most libs. Could it be used to create "universally reusable" lib like C, callable from almost any language around? Ability to create new building bricks which are reusable by others matters.

It's actually not that hard to convert existing C libraries into Rust. There was a recent talk at the last Rust convention about this very subject. The Zopfli library from Google was translated to Rust piece by piece, one function at a time. She wrote Rust unit tests along the way to not only ensure that the Rust replacements were tested, but also that the C functions could be tested too.

Although it can be argued that rewriting from scratch using idiomatic Rust is more ideal and can lead to faster implementations, large projects can easily start by just adding rust to the build system and converting one function at a time.

As for bindings to C libraries, there are tools to automatically generate bindings from C headers so this really isn't a problem. Providing a binding on Crates.io is best if you also go a step farther and wrap the C bindings up so that the library is more idiomatic to Rust, such as adding error handling.

7) Mozilla have shown us what they're up to, by enforcing signatures on extensions, with no way around. Relying on toolchain from someone like THIS looks shortsighted and hazardous to say the least. Especially when they pedal "package manager" theme. Wouldn't they introduce signatures on their package manager as well, with Mozilla being the only central authority deciding what is "good" and what is "bad", with no means of override?

How does this relate to Rust? Do you even know how Cargo works? It would be impossible for them to require signatures to Cargo crates when Cargo merely pulls libraries directly from git URLs. The Crates website is just an index of crates that conveniently map versions to precise git commits.

8) I'm really fine with installing somelib and somelib-dev using my system-wide package manager. I really do not want some third-party crap to bring some files and execute code in my system. It screws up OS management and brings winduz-like practices on my head.

You clearly do not understand how or what Cargo is. Cargo is not a system package manager like APT, DNF, or pacman. Cargo does not manage packages on your operating system. Cargo is a build tool that manages crate dependencies automatically. Rust software still has to be packaged into debs and rpms and PKGBUILDs when distributing Rust software to users on their platforms.

[mmstick]

Uh, you said, "C++ really isn't good for software libraries either due to C++ not having a stable exportable interface." Unless I'm taking advantage of the verbatim quote? Your argument is that C++ doesn't have, or isn't capable of, a stable exportable interface which is bologna.

The argument isn't about the existence of a C ABI, which is very hard to perform in comparison to Rust. The issue that C++ poses on the system is bad enough that there is a section in the Gentoo wiki dedicated to it. Ever tried updating a Gentoo installation after a GCC update and having all the C++ software on the system break because libstdc++ was updated? Anyone who maintains packages for a Linux distribution or uses Gentoo will understand that any time GCC is updated, you have to also re-release all your C++ packages, and the same goes for when a C++ library is updated, all packages relying on that library have to be rebuilt and re-released.

With Rust software, you do not even need to have the Rust standard library installed. Rust software is linked statically at compile-time, and thus will never break because a package was updated on the system, unless that package is a dependency to Rust that is dynamically linked. Cargo then allows you to perform application-wide LTO on your build to optimize it and reduce binary size greatly.

[computerquip]

The argument isn't about the existence of a C ABI, which is very hard to perform in comparison to Rust. The issue that C++ poses on the system is bad enough that there is a section in the Gentoo wiki dedicated to it. Ever tried updating a Gentoo installation after a GCC update and having all the C++ software on the system break because libstdc++ was updated? Anyone who maintains packages for a Linux distribution or uses Gentoo will understand that any time GCC is updated, you have to also re-release all your C++ packages, and the same goes for when a C++ library is updated, all packages relying on that library have to be rebuilt and re-released.

With Rust software, you do not even need to have the Rust standard library installed. Rust software is linked statically at compile-time, and thus will never break because a package was updated on the system, unless that package is a dependency to Rust that is dynamically linked. Cargo then allows you to perform application-wide LTO on your build to optimize it and reduce binary size greatly.

That is a completely different message than the first.
That said, you don't *actually* have to recompile those packages.
The very wiki page you're talking about explains it better than I can: https://wiki.gentoo.org/wiki/Upgrading_GCC#ABI_changes

[SystemCrasher]

There are a lot of projects that have to be large and complex while also being written in a low level language. The Linux kernel, for example, would largely benefit from being rewritten in Rust.

This bold statement needs some rock-solid proof and I'm yet to see one. Say, these nitty-gritty web devs pissed off C all the time for manual memory management and so on. The result? Web servers are usually hacked via these high-level web apps, not C. Isn't it funny? Then, do you understand e.g. kernel starts up in very constrained environment, where you do NOT have memory allocation at all, unless you bring it up yourself, most HW is offline and basically you can't rely on any cool standard libs crap and plenty of usual facilities (e.g. notion of "thread" could be really missing at some point, and in Linux it is fairly large user-mode glibc which creates full-featured POSIX things, Linux kernel uses own notion of LWPs wihch are both "processes" and "threads" and a bit beyond of both concepts).

Has Rust been crafted with all these use cases in mind? Where you could really get rid of smartass helpers and services and just do things in simplest, robust, extremely predictable ways? Say, C could be put into mode it works like a "code generator" similar to "high-level assembler". One could take really precise control over environment. Like very exact memory layout, memory accesses and so on. This is very important in uninitialized system, because one wrong move and everything is FUBAR.

Even -nostdlib is non-standard GCC (and maybe clang?) extension, not something C mandates. Yet that's what makes it possible to use gcc to create truly low-level code like kernels, boot loaders, all kinds of self-sufficient firmwares and so no. And, honestly, it is very convenient when all this crap using same language, using same understanding of e.g. data structures (a bit tricky in C, but C99+ makes it much better). Even usermode things MAY need e.g. custom memory allocator to get best performance. When language pedals huge runtime lib and memory management so much, I'm getting worried about these issues, to say the least. I doubt Mozilla ever cared about something like this.

Even usermode low-level programs sometimes may want to do "wrong" things. Say, one of my programs had to read exact PHYSICAL memory address. This is different from VM address, and takes rather unusual Linux-specific trickery & calculations. This is also Fundamentarlly Wrong Way of accessing mmaped devices. However it has been 100x easier compared to any other options and since it has been read-only access on perfectly RO HW status register, it is safe. And saved me shitload of time compared to e.g. writing kernel driver and so on. I needed just one fucking value from HW status register at certain physical address, to understand how to proceed next. Fortunately most of you would never stumble on this code because it has benen really task-specific stuff for embedded Linux device. I wonder if Rust could handle this at all. Let's assume I want to read unsigned 32-bit value from physical mem address 0xDEADC0DE. I know HW is little-endian and ok with unaligned mem accesses (may or may not be a case, btw). How it supposed to look in Rust?

We could easily drastically reduce the amount of SLOC and design APIs that are more generic, and yet just as fast, much easier.

The only way to reduce SLOC is to make calls/primitives more comples and higher-level, add smartass runtime libs and so on. This kind of crap tend to get on the way when it comes down to low-level code. Calling black-boxes doing complicate processing is not very funny and could lead to really odd failure modes.

Rust's types are not annoying in the least. If you actually try developing software with it, it's easy to hate languages that do not provide the same level of guarantees and finesse that Rust's type system is able to pull off.

I'm okay with C99 types for low level kind of stuff. It is convenient to have fixed-width types to implement e.g. comm protocols & portable data files exchange and so on, so the very lowest levels could be crafted in a very predictable ways. Then, if I use my program to e.g. fetch some data from e.g. microcontroller running my firmware, it is really handy both are using same C language, so I could eventually reuse the same include for communication protocol definitions, etc. Bringing new types system is not okay unless you could also handle most of microcontrollers as well. Because dealing with different type systems in different places is PITA and extremely error prone kind of thing. C does it simple and light. It is a bit dumbass, but that's what low-level kinds of things truly need. That's why C stays with us since 70s despite numerous attempts to replace it.

Rust's libcore interfaces directly with LLVM primitives, which is much lower level than C,

First of all, calling some lib is quite high level thing on its own and dooms low level programmers to face a lot of woes and issues. Not to mention I like GCC toolchain, so being inclined on LLVM is rather counts as disadvantage for me. I really need to translate text into machine code, everything else should be very optional when it comes down to really low level kind of things.

and very much like a modern version of Assembly.

I'm using C in this mode sometimes. And it's not just about some fucking lib. It is about whole toolchain. Even after code has been generated, I may really want to take control on how e.g. linking happens and how final code placement happens in memory. E.g. in micocontrollers I have some minimal early init loader, which does minimal pre-firmware init and prepares exec environment for running main() of firmware. Making sure it does some bare minimum similar to what libc normally does, but subject to extreme constraints of environment. So I have to be able to put certain crap at certain physical locations and be able to load it using my simple loader (few commands in assembly aka "startup code"). This have to be done to bring up environment and program code to state C program & compilers expect.

It is what allows Rust to achieve C-like performance today, and better-than-C performance in some areas, and will allow better-than-C performance across the board in the future. SIMD support and other CPU-specific instructions can be exposed to Rust through the LLVM primitives.

On other hand it means one can't really use it to create "early" environments, firmwares and somesuch. These things would still use C. Interfacing them, their protocols, data structures and somesuch is often needed in low level programs. So it seems Rust isn't really low level kind of thing, right? Calling some lib or SIMD is cool. But it makes assumption it is available. Getting superb performance is good. Getting way too smartass runtime misbehaving in way one can't override on the way is not.

C is good in that you virtually never hit the wall with decent C toolchain like GCC. Basically, toolchain like GCC allows one virtually everythnig assembly language permits to do but in more portable and convenient ways (i.e. you only have to break portability when you really have to do something truly HW-specific). Once some lib behaves poorly or gets in the way, one could get rid of it, redefining runtime environment from lowest levels as needed.

That's what allows to use C to create boot loader, firmware, kernels and so on. What is LLVM if we've just been powered on? There is only code and some unused memory. Everything else is have to be created by launched code. There is no notion of "LLVM" or "memory allocation" at this point. Actually, whole memory allocation kind of thing is optional in C. One could use fully static memory allocation instead. This way one could create system which can't run out of memory. At least not by usual means. At most one could provoke "stack overflow" based things, but if there're no malloc, it can't fail either. Very handy in applications like microcontroller firmwares demanding reliability (erm, default Linux attitude about overcommit isn't reliability best friend btw, but only low-level system ppl suspect this :P).

Rust does not perform any sort of type conversions at runtime. Everything, including C types, are desugared to LLVM primitives and then converted to machine code. If you've ever actually tried to work with C libraries inside Rust, it is actually quite easy and convenient.

C is good in one property: I could get rid of fucking libs. Even standard ones, which is really valuable option for low level things. If lib gets on the way, just chop it and reimplement what you really need the way you need. That's what makes C unbeatable on low level: assembly-level control over execution environment without actually resorting to assembly, which is utterly non-portable and its SLOC really bad and at the end of day, it is easy to beat compiler in some local part of code, but once code reaches certain size, human proves to be worse in tracking global resisters usage and ways to optimize it. So these days assembly is only practical for writting few hottest points of code where performance is paramount. Like e.g. encryption algo or audio/video codec, where replacing crappy compiler-generated code could give a big performance win and code is small enough.

It does not open up a 'whole class of mistakes and errors' in the least.

OTOH C is fairly easy to understand if you have low level mind. Then it is fairly easy to avoid these, all it takes is understanding on how computers are working and relying on self rather than on "superb" blackbox-style runtime. OTOH it is really nasty when one can't get rid of misbehaving runtime/lib, etc. Even C isn't perfect in this regard and it is actually about "toolchain" where one may want to take control over e.g. linking as well, etc.

The main downside is just knowing the standard quirks of C libraries such as what different error codes returned by C functions actually mean, as C libraries do not have a concept of sum types like Option and Result.

C is good in sense you could get rid of any lib. Even standard libs, if desired. That's what makes it suitable for low level use in first place. Uhm, at least toolchain I use could do that, so I could write e.g. microcontroller firmware and user-space program talking to this device, heavily reusing code in the process. OTOH talking to C-typed HW from different typed program is doomed to be pain and would bring plenty of weird bugs. Sure, C got some odds, but they're relatively easy and fairly predictable. Though most annoying shortcoming is probably lack of math overflow handling. Virtually all CPUs I know actually set math overflow flag in HW, and it quite cheap in terms of speed to catch math overflow. But still not free, at least like 1 extra assembly command have to be emitted, and it could kill performance a lot if happens in hot spot. So sometimes getting things as dumb and simple as possible really matters.

The beauty of Rust is that you can do both programming lower level than C with libcore, and programming higher level than C++ with libstd. There's no need to have two different languages if you keep everything in Rust, which supports both low level and high level programming without hindering either's capabilities.

C could already program lowest levels. With no fucking libs at all, no runtime, no memory management, only bare CPU and physical memory (multi-task OSes expose VM, it is different kind of thing). Even assembly can't go lower than that. That's what allows to use C for writing boot loaders, firmwares and kernels. Say, Cortex M microcontrollers are intentionally designed the way one could bring CPU online wtith virtually no assembly code at all (i.e. assembly "startup" being zero or very close to this point).

It's actually not that hard to convert existing C libraries into Rust. There was a recent talk at the last Rust convention about this very subject. The Zopfli library from Google was translated to Rust piece by piece, one function at a time. She wrote Rust unit tests along the way to not only ensure that the Rust replacements were tested, but also that the C functions could be tested too.

IMHO, imcompatibility with shitload of pre-existing libs is a disaster. C is funny in the fact it could be drastically low level, so I could access physical mem location where e.g. HW status register resides. But it could also be fairly high-level either. Basically, one could attach few problem-specific libs and, erm, C code SLOC could compete with super-high level things like Python/Ruby/JS/etc while still beating 'em to dust in terms of speed, reliability and predictable behavior.

Although it can be argued that rewriting from scratch using idiomatic Rust is more ideal and can lead to faster implementations, large projects can easily start by just adding rust to the build system and converting one function at a time.

This makes false assumption devs got shitload of free time to deal with tasks not anyhow related to achieving their initial goals. This is wrong assumption to say the least.

As for bindings to C libraries, there are tools to automatically generate bindings from C headers so this really isn't a problem. Providing a binding on Crates.io is best if you also go a step farther and wrap the C bindings up so that the library is more idiomatic to Rust, such as adding error handling.

I do not really get why it is not a standard feature of language. Ignoring pre-existing C software and libs? Cool, good luck to ignore your kernel and glibc for next 20 years. Not to mention their calls are using C types (not best ones actually, it sometimes plays poor joke on us, sure).

How does this relate to Rust? Do you even know how Cargo works? It would be impossible for them to require signatures to Cargo crates when Cargo merely pulls libraries directly from git URLs. The Crates website is just an index of crates that conveniently map versions to precise git commits.

First of all, I really hate when someone is trying to reinvent package management. I already have system-wide package manager. It tracks all files I use. I'm happy with it. It allows me to keep my systems patched in easy, efficient and consistent ways. So for me it rather counts as disadvantage. Doing apt-get install libsomething-dev is both simple, comes from really trusted source and there're good established practices on who could commit what, when and why not to mentoin real package managers do a real integrity and authenticy checks while allowing me to add my own key or entirely overriding it, if desired. This way neither any kind of network attacker could pwn my systems in obvious ways nor e.g. system maintainers could have exclusive control on what's going on. I could add my own URLs as repos, there're no some central site I can't override, etc. It does not gives maintainers exclusive control over what's going on in my system, I can always have override. I could even entirely disable their keys and repos, though at this point I'll have a lot of fun rebuilding all packages myself. That's how I like it. Ppl arond mozilla proven they are not ok about it and rather trying to lock devs & users on self, as seen on extansions sigs. Not something to be taken lightly. Sure, I'm very suspicious about any products related to Mozilla, not even Google gone so evil. So last thing I want is a package manager designed and operated by someone from mozilla and some crappy central site locations without means of override.

You clearly do not understand how or what Cargo is. Cargo is not a system package manager like APT, DNF, or pacman. Cargo does not manage packages on your operating system. Cargo is a build tool that manages crate dependencies automatically.

And who told I want it to download some crap from network without my consent? Then, what about centralized locations and means of override? Could I have my own repos/lists, getting all Mozilla spiggots out of the way? If no, I wouldn't touch it with 10' pole for the time being. Getting locked down on centarlized repo/list managed by Mozilla thing is absolutely last thing I may want.

Rust software still has to be packaged into debs and rpms and PKGBUILDs when distributing Rust software to users on their platforms.

And I'm also really okay with apt and somesuch when buildnig C programs. It is really easy to write apt-get build-dep <program>. Not like if I need another program to do that. I really want apt to be the only thing which fiddles with libs and somesuch. I do not need or want fucking mess like Windows where each program brings own installer/packagemanager and so on. C is good in this regard. It comes with no strings attached. Package manager does the rest most of the time, even for building programs. While being vendor-lock free so I could enjoy same level of control over whole infrastructure maintainers have and even override maintainers entirely, if I'm ready to get what it takes. That's what makes Linux and C so beautiful. I never hit dead end I can't overcome, its a wonderful feeling XD.

[ssokolow]

Has Rust been crafted with all these use cases in mind?

Yes. Rust was explicitly designed to be an alternative "successor to C" by people fed up with having to use C++ when it clearly had lost the plot with regards to C and was too constrained by backwards compatibility with design mistakes in existing C++.

In fact, it's been specifically designed to allow you to reason about your program's low-level behaviour even more effectively than with C, by specifying much less undefined behaviour for the compiler to play around with than the C spec.

Even -nostdlib is non-standard GCC (and maybe clang?) extension

Rust has equivalent pragmas. What runtime it has (standard library, stack trace on panic, jemalloc as the default allocator, etc.) can be opted out of, and there are various efforts, both past and in progress, to write OS kernels (rustboot, redox, etc.), kernel modules (rust.ko, winapi-kmd-rs, etc.) and microcontroller firmware (Zinc, AVR-Rust, etc.) in it.

IMHO, imcompatibility with shitload of pre-existing libs is a disaster.

Rust isn't incompatible with the existing C libraries. The efforts to translate C libraries to Rust are purely to take advantage of Rust's much stricter compile-time checks (primarily letting the "borrow checker" statically verify memory safety at compile time rather than by adding something at runtime like a garbage collector.)

Bindings in rust aren't like bindings in other languages. Rust's compatibility with C libraries is very similar to C++'s and Rust bindings exist mostly to enforce invariants and bridge the gap between how idiomatic C and idiomatic Rust do things. (eg. A wrapper around a database library would hook it up to Rust's RAII-style model for closing handles when they go out of scope.) Given that one of Rust's design goals is "zero-cost abstraction", the compiler's also very good at optimizing the wrappers to code at least as good as what you'd hand-write. (In the case of iterators, for example, you get safety without bounds checks because the compiler can prove, at compile time, that they're unnecessary. In addition, the closure-based API gets optimized to the same code you'd get by writing a for loop without closures.)

And who told I want it to download some crap from network without my consent? Then, what about centralized locations and means of override? Could I have my own repos/lists, getting all Mozilla spiggots out of the way? If no, I wouldn't touch it with 10' pole for the time being. Getting locked down on centarlized repo/list managed by Mozilla thing is absolutely last thing I may want.

It's possibly to specify a crate's location explicitly (similar to how pip for Python will accept URLs in addition to package names), local overrides are supported, and a flag that means "NEVER do network requests. Die with an error if you have to." is in development. (In fact, lack of that is the main remaining blocker for Rust getting integrated into Firefox's build system.)

[ssokolow]

From my perspective, the most severe problem of C++ is the amount of time it takes to compile. In theory there do exist methods for making typical C++ code compile 10x faster but unfortunately C++ compiler implementations aren't there yet.

That is one area they're still working on in Rust. They prioritized stuff that couldn't be improved later without breaking existing code, such as deciding on a balance point between type inference and explicitness which makes things comfortable without causing "action at a distance" and massive error messages as C++ is prone to.

[mmstick]

rant

tl;dr: It's apparent that you have no idea what you are talking about. Talk about FUD, you're full of it..

[SystemCrasher]

tl;dr: It's apparent that you have no idea what you are talking about. Talk about FUD, you're full of it..

Yes, I'm full of FUD when it comes down to Mozilla and especially anything relying on their nasty online services, they've both surpassed Google in terms of aggressive data mining (like "telemetry" backdoor crap they are planting here and there) and weird restrictions (like mozilla being the only authority who could sign extensions). Thanks to Mozilla's own actions I'm definitely not going to touch their products with 10' pole. Seems they want to be MS/Apple style ecosystem and I wish them good luck heading this direction. I'm not going to be fond of ppl trying to push dependence on such entity either.