The Latest Progress On Rust For The Linux Kernel

[billyswong]

Wow the discussion thread flooded with so many "borrow checker" "borrow semantic" argument~ Hey guys, don't waste time on terminology. It doesn't matter the spirit / concept of borrow checking applied to runtime / std libs shall be called "borrow checker" authentically. The doc already said it is "Enforcing Borrowing Rules at Runtime with RefCell<T>" in chapter 15.5. When one said "there are borrow checker in Rust runtime" casually, it is about the spirit, not the exact borrow checker module in compiler.

[billyswong]

On various platforms (Linux included), you don't really have any choice, because the kernel defaults to overcommit semantics to compensate for applications that waste a ton of memory on Windows by mapping much more memory than they actually write to.

That means that, even if you're handling malloc failure perfectly well, malloc will return success but, when you try to write to that memory, the kernel may suddenly realize it's overpromised what memory is available and have to kill something after all the branches for handling allocation failure in the software have already been told it was successful.

Interesting knowledge! This might make Rust memory allocation failure being panic a rational choice! Here memory allocation error may trigger not when it is allocated but when it is actually used, where it is not possible to do a try_read() or try_write(). So it may be best practice to isolate the program into processes / threads / whatever grouping such that when a group of code may allocate and access a new big chunk of memory, the panic is isolated and the unwinding won't fall through the bottom.

Anyone know if my idea make sense or not???

[mdedetrich]

Interesting knowledge! This might make Rust memory allocation failure being panic a rational choice! Here memory allocation error may trigger not when it is allocated but when it is actually used, where it is not possible to do a try_read() or try_write(). So it may be best practice to isolate the program into processes / threads / whatever grouping such that when a group of code may allocate and access a new big chunk of memory, the panic is isolated and the unwinding won't fall through the bottom.

Anyone know if my idea make sense or not???

Yes and no, threads are expensive and they have overhead especially so you then get into granularity issues (i.e. you don't want to create a new thread every time you allocate memory just for isolation reasons).

[ssokolow]

Interesting knowledge! This might make Rust memory allocation failure being panic a rational choice! Here memory allocation error may trigger not when it is allocated but when it is actually used, where it is not possible to do a try_read() or try_write(). So it may be best practice to isolate the program into processes / threads / whatever grouping such that when a group of code may allocate and access a new big chunk of memory, the panic is isolated and the unwinding won't fall through the bottom.

Anyone know if my idea make sense or not???

In Rust 1.0, threads were the only way to catch panics. The primary motivation for adding std:anic::catch_unwind was so C APIs could translate panics into C-style return values without having to spin up hidden threads inside the library.

As for isolation, you can certainly wrap catch_unwind around your unit of work (eg. an individual image in a batch thumbnailing operation, or the handling of an individual network request) and rely on Rust's ownership semantics and things like Mutex poisoning to ensure that stuff that's been left in an inconsistent state is not observable from outside the panicking thread before it gets cleaned up.

[jacob]

Yes, we don't agree. Just because I wasn't able to allocate a larger block, I shouldn't be allowed to even try allocating some smaller one because it *might* not succeed? This is crazy and bogus. Then why should I be allowed to allocate anything at all? There is always a chance a larger allocation might have failed. In fact, any allocation, even if it is the first one, might fail. And what about not being able to allocate 100MB for an image? Then there is a very high likelyhood I could still allocate 100 bytes for displaying an error message. This has nothing to do with undefined behavior, as long as the allocation function always returns a well-defined result (even in the case of failure), all is well. I could argue a forced termination is less defined than an error-result, because the result of a program- or thread-termination could be undefined at the application level (undefined hardware-state, loss of unsaved work, corruption of calculation result etc...), while the app developer would have been able to properly handle the failed allocation (just not continue).

On the other topic: Yes, I know about unwindable panics, and that the global allocator is used by default in most if not all crates. I wrote about this in an earlier post here in this thread, and I can only say the same: The problem is this facility is unstable/experimental/not yet released/call it whatever you want, and progress on it has been very slow over the years. And even when it gets officially stable, another part of the same Rust proposal is the introduction of fallible APIs, and those are the ones which won't get automatically used by crates, and will have to trickle down to them over a long period of time.

Actually catch_unwind has been stable since 1.9.0. It is true that fallible allocation such as .try_reserve() is still nightly-only, but in my opinion that's almost a sidetrack from what I personally consider one of Rust's current main weaknesses, which is the lack of a (stable) placement allocation.

On the other hand your point about allocating 100MB for images is kind of bogus, because if you need that, then 1) you would set your hardware requirements accordingly and 2) if it fails, the program wouldn't be able to perform its primary function of processing images, and the rest is essentially irrelevant. If you are so concerned about being able to display an error message, you can also allocate and build the pop-up first, just not show it, and ONLY THEN allocate your 100MB for images (and display the popup from catch_unwind if it fails). Another way to look at it, Rust currently HAS fallible allocation in exactly the same way as C++ or Java: through runtime exceptions (because of course panic/catch_unwind is nothing but exceptions propagation). What it doesn't have in stable versions is idiomatic fallible allocation that would return Option<T> or Result<T,E>.

There is also a fundamental difference between a panicking allocation and runtime assertions such as that in RefCell. The latter is a development tool to detect a condition that is supposed to never happen in a well designed program, and when it occurs it signals a bug that must be corrected. It's the same as assert() in C, which is and should be used, but there is no way to recover from it and that's not its purpose. On the other hand, the former (allocation and other similar problems) result from external conditions that cannot be predicted or fixed during development and in one way or another, they prevent the program from doing what it's supposed to do. If that happens, the program *MAY* be able to do something to warn the user etc., but it's absolutely not guaranteed, because the idea that if you fail to allocate 100MB, you could still allocate at least 2Kb is nothing more or less than keeping fingers crossed with zero guarantees as to what the outcome will be. In the best case, it would work, in the worst case, you would get a cascade of exceptions and/or null pointers that will result in a crash that will be extremely difficult to diagnose and even more so to reproduce reliably.

[oiaohm]

Wow the discussion thread flooded with so many "borrow checker" "borrow semantic" argument~ Hey guys, don't waste time on terminology. It doesn't matter the spirit / concept of borrow checking applied to runtime / std libs shall be called "borrow checker" authentically. The doc already said it is "Enforcing Borrowing Rules at Runtime with RefCell<T>" in chapter 15.5. When one said "there are borrow checker in Rust runtime" casually, it is about the spirit, not the exact borrow checker module in compiler.

The problem I have here is that the "borrow checker" module in the complier could be doing like what gcc does with printf scanf ... with C where it checking that the runtime RefCell usage looks sane at the build time. From my point of view the borrow checker in the compiler of rust is defective because it has not be implemented out to cover everything it should. Yes some cases with RefCell the borrwer checker module in the complier should not cause the build to fail but should throw a warning that the code looks dicey.

Yes the problem here that we have people making the rust compiler putting a false divide between what is compiler functionality and what is standard library functionality. This false divide makes them not see that the compiler should be checking bits of the standard libraries functionality to pick up human errors in coding before they cause end user problems.

You could argue that gcc should not check for use after free in C because that is the C standard library functionality yes people use to argue this with C but if you attempt that today when writing a C compiler everyone would think you are a idiot. Yet that exactly the same argue rust people make to me over RefCell. Yes the rust people need to wake up on this point.

[jacob]

The problem I have here is that the "borrow checker" module in the complier could be doing like what gcc does with printf scanf ... with C where it checking that the runtime RefCell usage looks sane at the build time. From my point of view the borrow checker in the compiler of rust is defective because it has not be implemented out to cover everything it should. Yes some cases with RefCell the borrwer checker module in the complier should not cause the build to fail but should throw a warning that the code looks dicey.

Yes the problem here that we have people making the rust compiler putting a false divide between what is compiler functionality and what is standard library functionality. This false divide makes them not see that the compiler should be checking bits of the standard libraries functionality to pick up human errors in coding before they cause end user problems.

You could argue that gcc should not check for use after free in C because that is the C standard library functionality yes people use to argue this with C but if you attempt that today when writing a C compiler everyone would think you are a idiot. Yet that exactly the same argue rust people make to me over RefCell. Yes the rust people need to wake up on this point.

The problem is that RefCell can only be checked at runtime. Could the compiler have some logic to deal with certain simple cases where it can prove statically that multiple mutating borrows are impossible, and optimise the runtime assertion out? Absolutely, but from a practical point of view implementing it would be lots of work for a fairly minimal benefit, because

1. In practice it would likely not cover much more than what you can use normal mutable references for, which the static borrow checker already handles;
2. It would be poor use of developer's time and resources to expand considerably the already very advanced static analysis capabilities of rustc just to handle a comparatively obscure standard library feature that is rarely used (but the added complexity, slowness and potential bugs of the compiler would affect everyone, even developers of low level code that doesn't actually use the standard library at all)
3. The only tangible result would be to remove a test and conditional jump from the generated code, which is hardly noticeable - even with Spectre etc mitigations, the effect is not even measurable unless you have that in some tight running inner loop, in which case you have some major design problems anyway.

[oiaohm]

The problem is that RefCell can only be checked at runtime.

Except this is not 100 true.

Could the compiler have some logic to deal with certain simple cases where it can prove statically that multiple mutating borrows are impossible, and optimise the runtime assertion out?

No not optimise runtime assertion out that not the problem I am 100 percent refering to.

https://doc.rust-lang.org/std/cell/struct.RefCell.html

There is simple panic code here.

use std::cell::RefCell;
let c = RefCell::new(5);
let m = c.borrow_mut();
let b = c.borrow(); // this causes a panic

Could you statically prove that code like this is always going to panic when run. Yes you can. Why does code like this build without at least a warning.

Absolutely, but from a practical point of view implementing it would be lots of work for a fairly minimal benefit, because

1. In practice it would likely not cover much more than what you can use normal mutable references for, which the static borrow checker already handles;
2. It would be poor use of developer's time and resources to expand considerably the already very advanced static analysis capabilities of rustc just to handle a comparatively obscure standard library feature that is rarely used (but the added complexity, slowness and potential bugs of the compiler would affect everyone, even developers of low level code that doesn't actually use the standard library at all)
3. The only tangible result would be to remove a test and conditional jump from the generated code, which is hardly noticeable - even with Spectre etc mitigations, the effect is not even measurable unless you have that in some tight running inner loop, in which case you have some major design problems anyway.

A lot of what you write here is correct on one hand but incorrect on another that you most likely would not detect more issues than you would if it was a normal not RefCell mutable reference as in Cell but remember Cell checking logic is not being run against the RefCell code because if it was the "let b = c.borrow(); // this causes a panic" would not be build-able code.

Yes the result would not be a performance fix. The result would developer being informed that have coded something that could panic on the end user at runtime that currently builds without warning or error. This is not about making the program faster this is making the program less crash prone.

I like how you call it obscure standard library feature. This is a feature of the rust standard library that as part of standard operation says it perfectly fine to crash the application and cause user to loss all their work.

Performance issues are one thing. RefCell was not coded to make sure that you have to include a proper handler to deal with the case it goes wrong either.

Please note I am not saying deal with the RefCell problem is simple. Those making advanced static analysis for C and C++ don't have a easy time either.

[jacob]

use std::cell::RefCell;
let c = RefCell::new(5);
let m = c.borrow_mut();
let b = c.borrow(); // this causes a panic

Could you statically prove that code like this is always going to panic when run. Yes you can. Why does code like this build without at least a warning.

This is exactly the kind of trivial case that can be handled statically. Obviously for the hell of me I can't see why would someone use RefCell here instead of the exactly equivalent code:

let mut c = 5;
let m = &mut c;
let b = &c;

which wouldn't pass the borrow checker. But for the sake of the argument, let's assume that they do. On the first attempt to run this, RefCell's assertion will fail and they will have to fix it. Now of course, it is possible that this kind of code would be in some rarely used execution path and wouldn't fail in normal usage scenarios, that brings us back to the tests coverage problem. So obviously no, the current model is not perfect and does not do EVERYTHING that could possibly done statically. Yes, the borrow checker could be improved to model RefCell in simple cases like that. Is it worth all the effort though? That's the real question. If someone comes and volunteers to contribute to the compiler to implement specifically this, it's likely to be a decision between the (arguably meagre) benefit vs the cost of increased complexity and hardcoding library-specific rules within the compiler itself. But I don't think it's likely the core team itself would consider working on this anytime soon given the arguably much higher priority issues that are in greater demand too (generic associated types, improvements of const generics, further developments of the async support and ecosystem, etc.)

I like how you call it obscure standard library feature. This is a feature of the rust standard library that as part of standard operation says it perfectly fine to crash the application and cause user to loss all their work.

Many standard library features in any language, even ones much more widely used than RefCell, can crash a program. Even in Ada, and don't even get me started on C++... The point of RefCell though is precisely that it is NOT fine at all to crash the application: the condition in which it would crash can't be predicted statically except in trivial cases, but it is not supposed to ever happen and if it does, it's an indicator that the program must be fixed. Unlike the other category of errors, like the allocation failures, which are due to external conditions and not to whether the program is well written or not.

Please note I am not saying deal with the RefCell problem is simple. Those making advanced static analysis for C and C++ don't have a easy time either.

Years ago I did actually work on a static analyser for C++. It taught me, to paraphrase the G-Man, that a C++ program can be totally f*ed up in every way possible and even in some that are essentially impossible Ever since I've appreciated Rust for what it has to offer.

[ssokolow]

Yes the problem here that we have people making the rust compiler putting a false divide between what is compiler functionality and what is standard library functionality. This false divide makes them not see that the compiler should be checking bits of the standard libraries functionality to pick up human errors in coding before they cause end user problems.

That was a conscious decision. Rust was designed with the intent to work toward minimizing the coupling between the compiler and the standard library so that efforts to provide alternative standard libraries (eg. a klibc for the Linux kernel) won't be inherently crippled compared to the main one.

Heck, that's why one of the ongoing efforts is to get to a point where the functionality of Xargo (a Cargo wrapper which automates building for platforms without precompiled standard library distributions on offer) is merged into Cargo.

Checking bad uses of RefCell is allowed... it just belongs as a lint in Clippy, rather than as part of rustc.

It's basically the same mindset that has them artificially limit type inference. Yes, rustc's type inference is capable of inferring function argument types like Haskell's, as is demonstrated by that working for Rust's closure syntax... but normal functions are intentionally required to explicitly specify their argument types because that's part of their interface and Rust's designers made a conscious decision that changes to the innards of a function should not implicitly alter the interface it exposes.

Clippy has plenty of lints for stuff like that which can be turned into errors using annotations right within your source code, while rustc does not. It also has various lints that default to being errors, like out_of_bounds_indexing. (And remember that Clippy is part of the core Rust distribution, just like rustc and rustdoc.)