The Latest Progress On Rust For The Linux Kernel

[mdedetrich]

I find it hilarious that oiaohm is coming up with really contrived scenarios about people misusing RefCell (even though its evident that in practice this is a non issue i.e. if you look at actual projects written in Rust and ask Rust programmers/community). Its clear here that he hasn't programmed in Rust non trivially and is just quoting random sources like documentation without having actual experience in the language.

The mere fact that so much effort on the part of weasel wording and misconstrued arguing is required to try and "prove" how much of an issue RefCell is, it's a testament to how well Rust is designed.

Especially compared to C, which in reality has a million ways to do things incorrectly and here 10 pages are spent arguing on something as ridiculous as RefCell when in reality Rust has other actual and real problems.

[jacob]

...

Basically it all comes down to the fact that RefCell, like unsafe{} or performUnsafeIO in Haskell, lets you shoot yourself in the foot, and shouldn't be used carelessly. If what you are ultimately arguing is that there should be some kind of opt-in to that, maybe a compiler switch or pragma to enable RefCell at a crate or module level, that could actually make sense. If you want a warning message each time RefCell is used, personally I don't see the point since if you revert to RefCell, you are already dealing with some difficulties with your code.

As for the linting capabilities of modern C/C++ compilers, it's important to realise that that's not something that Rust should take inspiration from. Rather, it's a piecemeal substitute for features that Rust already has - in other words, gcc or clang with their linter are not innovating above rustc, rather they are trying to at least somewhat close the gap between what they do and what rustc can do.

[oiaohm]

This is a seriously contrived scenario. Basically you are supposing that the user is:

• a beginner who struggles to get past the borrow checker even in this simplest of cases; but
• at the same time, knows about RefCell and how to use it; and
• understands that this is just equivalent to a mut& that he couldn't get to work in the first place, and can instruct the compiler about it.

Really?

https://www.justanotherdot.com/posts...w-checker.html

Points 1 and 2 are explained really simply. A person does not have to have skill to find a blog that will give them the information to hang themselves. This is true for rust as any other program language.
Point 3 the equivalent would not be done novice programmer. This would be stuff the standard library developers have put in place so the compiler know what the functions are so can in fact generate correct warnings and at times run correct processing.

Of course person who is not very skilled having to type more lines is more likely screw themselves up as well.

Even leaving aside the total implausibility of the situation, let's say that you implement support for something like #[mut_equivalent] to be used with RefCell. What exactly is the point of that? In every case you could use it, you could just as well use the mut&, which is a better and more correct solution. Ultimately your linter would do nothing but print a warning "use mut instead" every time this annotation would be used. In fact mdedetrich's analogy above is exactly right: RefCell is like performUnsafeIO in Haskell, something to be used as a desperate last resort and probably only temporarily until the code is refactored. There is absolutely no point in having any linting for performUnsafeIO because in every case where it would apply, you should rather use the normal IO monad.

Haskell example there is a case where people have asked for a warning option.
https://www.generacodice.com/en/arti...8221-exception
Yes the response from Haskell developers have been very much like you. The developer should get it right answer. There are lot of cases with haskell performUnsafeIO that code turns up that a person has a comment I will refactor this later then never does.

Do think what mdedetrich wrote about the processing time that rustc is getting up the practical limit on what can be statically solved. That there are cases to allow particular items to be runtime borrow checked because this will save build time.

This is the hard point rust a some point will have to work out how balance between static borrow checking and runtime borrow checking. Static borrow checking can cost insane amount of processing time due to having to process every possible outcome. Runtime borrow checking little slower at runtime but only has to process the used paths.

You think about your arguement that "usage of RefCell is a desperate last resort and probably only temporary code until the code is refactored". This is a case for a warning so that when you build program you can see that part in there you should remove at some point. So the bit you planned to refactor does not end up lost in the code base. Yes gcc diagnostic options allow you to add a warning or general message where ever you like for this stuff as well.

Please note warning option does not mean that compiler has to show it every time. Just it need to be possible for the compiler to show. Currently the rustc complier cannot be set to show if code is using any runtime borrow checking this is weakness. The compiler cannot answer question program may want answered. Lint solution here also has to be custom coded to answer these questions. Now if the standard library provides the extra data this allows the compiler and the lint solution to be generically coded for these problems.

Think about it jacob you are using refcell or other runtime borrow checker parts to reduce down build time you still may want to check that they are right using the static borrow checker.

Rust developers have not had the serous talk about what to-do when you hit the wall that you cannot always static borrow check any more because is going to take too much build time. The rust compiler need to be able to work with the runtime borrow checker. Also need to talk about making sure stuff is not a silent path to failure unless the developer writing the program decided for that to happen.

Of course something as complex as the Linux kernel cannot 100 percent statically solve all the time either this is why some things are statically checked and other are runtime checked and there is overlap between the two coming the Linux implementation for C. Yes that overlap means in cases where you cannot statically solve due to build time limit the item will still be checked at runtime so preventing that particular problem completely going undetected.

[jacob]

The answer is horrible simple. Person does not know better and using Refcell got rid of their build error and this was not replaced with a warning so they now believe they have done the right thing.



Please note I said at least a warning. Lets say rust had __attribute__() like gcc has. Items like RefCell in the standard could have a __attribute__() added to raise a warning. So you swap failure to build by using RefCell for warning that you need to validate this code at runtime. And by declaring what it is with a __attribute__ you could have the option to check it like a normal mut. That could be useful if you do have a runtime crash.

This is a seriously contrived scenario. Basically you are supposing that the user is:

• a beginner who struggles to get past the borrow checker even in this simplest of cases; but
• at the same time, knows about RefCell and how to use it; and
• understands that this is just equivalent to a mut& that he couldn't get to work in the first place, and can instruct the compiler about it.

Really?

Even leaving aside the total implausibility of the situation, let's say that you implement support for something like #[mut_equivalent] to be used with RefCell. What exactly is the point of that? In every case you could use it, you could just as well use the mut&, which is a better and more correct solution. Ultimately your linter would do nothing but print a warning "use mut instead" every time this annotation would be used. In fact mdedetrich's analogy above is exactly right: RefCell is like performUnsafeIO in Haskell, something to be used as a desperate last resort and probably only temporarily until the code is refactored. There is absolutely no point in having any linting for performUnsafeIO because in every case where it would apply, you should rather use the normal IO monad.

There is a problem with this logic. More and more lint functionality of C and C++ is being integrated into the compiler. Why developers are lazy. This is the reality if the compiler builds a lot of developers will not run test-suites and will not run lint tools. Yes why Linus of the Linux kernel wants -Werror on by default to reduce the amount of defective code landing on his desk.

But that comparison is entirely flawed. Linting functionality added to C/C++ invovles things like strcpy() which should never be used, no ifs or buts (the only linting functionality here is "is it used at all? Then warning"), and things like printf() format strings, which are 1) used all the time and 2) trivial to check. Conversely, RefCell is 1) rarely used and 2) impossible to model statically except in irrelevant cases.

[oiaohm]

How is that not just "ask every legitimate user of RefCell to add a 'Yes, I'm not an idiot and I know what I'm doing. Disable the warning.' annotation to their projects"?

I'm reminded of one of Raymond Chen's posts on his blog, The Old New Thing, where he's talking about the difficulty in designing UAC to be useful without causing "alert fatigue", where people just tune out and start clicking "allow" automatically.

To be correct having every legitimate user of RefCell add 'Yes, I'm not an idiot and I know what I'm doing. Disable the warning.' is a valid move.

https://gcc.gnu.org/onlinedocs/gcc/Diagnostic-Pragmas.html

Gcc and LLVM have that functionality for C. Think about it when you use RefCell if you have to write large code because its high risk code it makes the safe code the more compact and simpler to read. This is about encouraging developers to use the safer route when possible,

Yes you are right there is a balance between alert fatigue and alerts. Remember alert fatigue is not a reason not to have a fire alarm. Alert fatigue is not a reason to be missing means to generate a warning/error.

It would be a option that you cannot use any runtime borrow checking unless you put in extra code to enable runtime borrow checking. Again this is about having the standard library and the compiler work with each other.

From what you have said with the [...] you have the frame work at least partly their to implement proper warnings in case users attempt to use RefCell to fix a build error and don't know what they are really doing.

Remember currently RefCell in lines of code is the same size as mut or equal that is safely processed by the compiler.

[ssokolow]

Please note I said at least a warning. Lets say rust had __attribute__() like gcc has.

Rust has attributes and makes them even more of a first-class thing than that. The Rust equivalent to __attribute__((...)) is #[...] before the AST node in question or or #![...] as the first child of it and users are introduced to them much earlier and more readily than in C.

For example, #[must_use] is an annotation you can apply to a type or function to get a warning if the return value is neither saved nor explicitly discarded with let _ = and stuff like #[derive(Debug, Clone)] is how you can ask the compiler to synthesize common functionality for structs and enums so long as their fields all implement said traits (interfaces).

#[rustfmt::skip] is how you ask rustfmt not to touch the formatting of something when reformatting a file and #![forbid(unsafe_code)] at the top of the root file of a project is how you ask the Rust compiler to disallow unsafe use across the entire project without exceptions. (deny allows exceptions and allow/warn/deny/forbid are how you set the behaviour of "warnings", even though not all of them warn by default)

There's even a standard way for procedural macros to hook in custom derives and, within what's fed to a procedural macro through any means, the macro can accept and process whatever attributes it likes. (That's how Serde and StructOpt provide very nice declarative APIs for parsing and serializing stuff.)

It's also how the standard library achieves "collapse Option<NonNull<T>> into the same bit pattern as a nullable pointer to T" by using attributes to annotate when types have holes in their possible range that an Option's None value can be folded into.

Items like RefCell in the standard could have a __attribute__() added to raise a warning. So you swap failure to build by using RefCell for warning that you need to validate this code at runtime.

How is that not just "ask every legitimate user of RefCell to add a 'Yes, I'm not an idiot and I know what I'm doing. Disable the warning.' annotation to their projects"?

I'm reminded of one of Raymond Chen's posts on his blog, The Old New Thing, where he's talking about the difficulty in designing UAC to be useful without causing "alert fatigue", where people just tune out and start clicking "allow" automatically.

And by declaring what it is with a __attribute__ you could have the option to check it like a normal mut. That could be useful if you do have a runtime crash.

Are you still misunderstanding that the point of RefCell is to be used in situations where the compiler can't "check it like a normal mut"?

[oiaohm]

Actually this is the hallmarks of a well designed language, the more you couple the standard library with the compiler the less portable the compiler becomes and the harder it becomes for the compiler to reason about programs because now you have assumptions in the stdlib that aren't entirely correct for some given context being placed into the compiler (as a person who works with Java/Scala + JDK I have dealt with this many times).

Java and Scala don't have anything like __attribute__() this does come from another language this idea.

Also taking into account on what I said earlier, I don't know what bundling the standard library and compiler will achieve. Its not like you can break the laws of mathematics to magically make the borrow checker better (if your complaining that the compile time borrow checker is not good enough), and moving more of the borrow semantics into runtime is opposite of what Rust stands for from a design principle.

This is a point people bring up to raise a warning does not mean you have to have always proved their is a problem. So having a __attribute__() that you could put on functions like RefCell so if they are used the compiler will raise a warning that you have runtime borrow checking in this program that need to validated because if its not this program could crash without warning. This stop some people swapping normal mut or Cell that will be checked for a RefCell. Because they got rid of a build failure and replaced it with a warning instead of nothing.

Please note I did not say bundling standard library with compiler. There need to be way for compiler and standard library to work with each other. A way that the standard library functions can be marked that the compiler can provide sane warnings. And at times provide extra checking as in this function is like X function it usages should be the same Y static borrow checked function even that it runtime borrow checking.

Java and Scala did attempt hard code this in the compiler. Where is java and Scala equal to the __attribute__() feature.

mdedetrich basically there is a need for the standard library function decelerations to be able to provide the compiler with enough information to display valid warnings when user is possible doing something dangerous and at times to allow the compiler to perform deeper lint/static checks.

mdedetrich think about it why can you use runtime borrow checking with rust and it does not raise a warning.

The best you can do is to limit the use of RefCell by making sure that the borrow checker can incrementally validate more and more correct programs but at one point you essentially hit diminishing returns and/or create other problems (such as increasing compile times). There is a strong argument that we have pretty much already hit that limit.

Think about it how are you limiting the usage of RefCell if usage does not raise a warning. Making sure complier inform users when they are using runtime borrow checking would not result in running into the mathematical problem. If likes __attribute__() are applied correct code that uses runtime borrow checking it would be possible with a compiler flag to force checking when you have a problem or are able to take the extended compile times.

Having the static and runtime borrow checking being able to be used as a combined solution.

mdedetrich think about this at the moment you can only run the compiler you cannot look at the source code how would you be able to tell at the moment what programs used static borrow checking and what programs are using runtime borrow checking. This is a weakness of the rust compiler design and language of rust not allowing the standard language library and other libraries to provide compiler with more details to provide better developer information on what they just built.

[oiaohm]

This is exactly the kind of trivial case that can be handled statically. Obviously for the hell of me I can't see why would someone use RefCell here instead of the exactly equivalent code:

let mut c = 5;
let m = &mut c;
let b = &c;

The answer is horrible simple. Person does not know better and using Refcell got rid of their build error and this was not replaced with a warning so they now believe they have done the right thing.

which wouldn't pass the borrow checker. But for the sake of the argument, let's assume that they do. On the first attempt to run this, RefCell's assertion will fail and they will have to fix it. Now of course, it is possible that this kind of code would be in some rarely used execution path and wouldn't fail in normal usage scenarios, that brings us back to the tests coverage problem. So obviously no, the current model is not perfect and does not do EVERYTHING that could possibly done statically. Yes, the borrow checker could be improved to model RefCell in simple cases like that. Is it worth all the effort though? That's the real question. If someone comes and volunteers to contribute to the compiler to implement specifically this, it's likely to be a decision between the (arguably meagre) benefit vs the cost of increased complexity and hardcoding library-specific rules within the compiler itself. But I don't think it's likely the core team itself would consider working on this anytime soon given the arguably much higher priority issues that are in greater demand too (generic associated types, improvements of const generics, further developments of the async support and ecosystem, etc.)

Please note I said at least a warning. Lets say rust had __attribute__() like gcc has. Items like RefCell in the standard could have a __attribute__() added to raise a warning. So you swap failure to build by using RefCell for warning that you need to validate this code at runtime. And by declaring what it is with a __attribute__ you could have the option to check it like a normal mut. That could be useful if you do have a runtime crash.

That was a conscious decision. Rust was designed with the intent to work toward minimizing the coupling between the compiler and the standard library so that efforts to provide alternative standard libraries (eg. a klibc for the Linux kernel) won't be inherently crippled compared to the main one.

Good in theory there is reason why this does not work out its human interaction with compliers.

Checking bad uses of RefCell is allowed... it just belongs as a lint in Clippy, rather than as part of rustc.

There is a problem with this logic. More and more lint functionality of C and C++ is being integrated into the compiler. Why developers are lazy. This is the reality if the compiler builds a lot of developers will not run test-suites and will not run lint tools. Yes why Linus of the Linux kernel wants -Werror on by default to reduce the amount of defective code landing on his desk.

[mdedetrich]

which wouldn't pass the borrow checker. But for the sake of the argument, let's assume that they do. On the first attempt to run this, RefCell's assertion will fail and they will have to fix it. Now of course, it is possible that this kind of code would be in some rarely used execution path and wouldn't fail in normal usage scenarios, that brings us back to the tests coverage problem. So obviously no, the current model is not perfect and does not do EVERYTHING that could possibly done statically. Yes, the borrow checker could be improved to model RefCell in simple cases like that. Is it worth all the effort though? That's the real question. If someone comes and volunteers to contribute to the compiler to implement specifically this, it's likely to be a decision between the (arguably meagre) benefit vs the cost of increased complexity and hardcoding library-specific rules within the compiler itself. But I don't think it's likely the core team itself would consider working on this anytime soon given the arguably much higher priority issues that are in greater demand too (generic associated types, improvements of const generics, further developments of the async support and ecosystem, etc.)

One thing to add ontop of this is that its impossible for a turing complete type checker to ensure that every theoritical program is properly checked because that is essentially the halting problem (or more specially Gödel's incompleteness theorem)

In other words, you will always find an example where the type checker at compile time is unable to prove that a correct program is correct, so a solution like RefCell will always have to exist.

The best you can do is to limit the use of RefCell by making sure that the borrow checker can incrementally validate more and more correct programs but at one point you essentially hit diminishing returns and/or create other problems (such as increasing compile times). There is a strong argument that we have pretty much already hit that limit.

And yeah, pointing out the flaws of RefCell by causing it to panic when the borrow checker at compile time can catch such errors is an asinine and misleading example, people use RefCell as a last resort. its like asInstanceOf/null in Scala or unsafePerformIO in Haskell, you just don't use it unless you have a really good reason.

Yes the problem here that we have people making the rust compiler putting a false divide between what is compiler functionality and what is standard library functionality. This false divide makes them not see that the compiler should be checking bits of the standard libraries functionality to pick up human errors in coding before they cause end user problems.

Actually this is the hallmarks of a well designed language, the more you couple the standard library with the compiler the less portable the compiler becomes and the harder it becomes for the compiler to reason about programs because now you have assumptions in the stdlib that aren't entirely correct for some given context being placed into the compiler (as a person who works with Java/Scala + JDK I have dealt with this many times).

Also taking into account on what I said earlier, I don't know what bundling the standard library and compiler will achieve. Its not like you can break the laws of mathematics to magically make the borrow checker better (if your complaining that the compile time borrow checker is not good enough), and moving more of the borrow semantics into runtime is opposite of what Rust stands for from a design principle.

[ssokolow]

Yes the problem here that we have people making the rust compiler putting a false divide between what is compiler functionality and what is standard library functionality. This false divide makes them not see that the compiler should be checking bits of the standard libraries functionality to pick up human errors in coding before they cause end user problems.

That was a conscious decision. Rust was designed with the intent to work toward minimizing the coupling between the compiler and the standard library so that efforts to provide alternative standard libraries (eg. a klibc for the Linux kernel) won't be inherently crippled compared to the main one.

Heck, that's why one of the ongoing efforts is to get to a point where the functionality of Xargo (a Cargo wrapper which automates building for platforms without precompiled standard library distributions on offer) is merged into Cargo.

Checking bad uses of RefCell is allowed... it just belongs as a lint in Clippy, rather than as part of rustc.

It's basically the same mindset that has them artificially limit type inference. Yes, rustc's type inference is capable of inferring function argument types like Haskell's, as is demonstrated by that working for Rust's closure syntax... but normal functions are intentionally required to explicitly specify their argument types because that's part of their interface and Rust's designers made a conscious decision that changes to the innards of a function should not implicitly alter the interface it exposes.

Clippy has plenty of lints for stuff like that which can be turned into errors using annotations right within your source code, while rustc does not. It also has various lints that default to being errors, like out_of_bounds_indexing. (And remember that Clippy is part of the core Rust distribution, just like rustc and rustdoc.)