Updated Rust Code For Linux Kernel Patches Posted

[dremon_nl]

2. my example was not about C++ and its STL, so I fail to see how it's relevant here. You mean incompetent developers will magically stop polluting their crapware with garbage dependencies, etc?

I mean you are comparing apples with oranges, it's not even about the language but about the standard library. Rust stdlib is analogous to libstdc++: it is richer, easier to work with and hence bigger than barefoot libc. But it's not crapware and it's not bloatware, and it's not mandatory at all. You can opt for `no_std` feature and even use `libc` crate if you want. And it's not relevant for kernel development anyway, so stop gaslighting about the nonsense like "crapware is trying to sneak into linux kernel". You can't use libc in the kernel either.

3. C links to libc, right. You forgot to mention that rust also links to libc. and nobody asked for its stdlib (see section 1). So stop gaslighting.

I didn't forget to mention that, may be just read my comment again.

[ssokolow]

no it's not. 6MB is approximately a size of, for example, the entire Alpine Linux docker container.

Well, first, my ~/.cargo/bin/rg is 4MB after stripping, not 6MB. I'll give the benefit of the doubt and assume that it was 6MB before because of how the precompiled userland stdlib is "release build, with un-stripped debuging info".

Second, looking at those comparison charts, it appears that Silver Searcher doesn't have unicode support or at least has incomplete unicode support. As someone who's worked with unicode, I can tell you that the unicode character classification tables that would have been embedded into ripgrep to achieve that are massive all on their own and would bulk out a tool in any language if you embed them in the binary.

Let's be honest about this sort of thing.

There's nothing that can justify such difference in (presumably) tiny single-purpose console utilities. https://tonsky.me/blog/disenchantment/ for reference.

I haven't read your link in full yet, so I'll have to get back to you on that, but skimming through it, the overall thrust looks like what I've already been saying, my reason for making heavy use of uMatrix, and writing my JavaScript or TypeScript from scratch to progressively enhance HTML that's already functional, and my reason for avoiding browser-based apps whenever possible.

Using "programmer time is more valuable" as an excuse to justify how far the waste has gone is inexcusable, but finding the actual excusable trade-off point is more difficult... especially when things like "A Windows install is 27 gigs because dynamic linking forced the OS to carry along WinSxS dependencies for applications you'll probably never install" exist.

And by the way, ripgrep doesn't support lookaheads and back references, which is the hardest part of regular expressions.

True, but not because Andrew Gallant was incompetent or lazy. Rather, it's an explicit design decision for the underlying regex engine to ensure that "all searches execute in linear time with respect to the size of the regular expression and search text" as part of hardening the engine against DoS attacks when used with untrusted regexes. There's a whole section in the manual on how to configure the engine for that use-case.

[darkonix]

That is presuming that module developer will stick to the code that rust module wrapper expose. Also there are particular cases with modules where allocation failure where this is expected outcome and should not lead to module failure with the Linux kernel.

https://linux-kernel.vger.kernel.narkive.com/jyUQhL5H/patch-use-attribute-malloc-for-gcc-3-2

The attribute tags have been added over the Linux kernel for a long time.

https://elixir.bootlin.com/linux/v4.3/source/arch/x86/mm/pgtable.c#L354

Do notice something here all the allocs that have the malloc flag applied in the Linux kernel do have if fail return NULL.

I don't see any attribute in the pgtable.c linked. Are you sure that's the right link?

I'm also a bit confused by the attribute(malloc). By default it only seems to be helpful to optimizing and does not add any extra validation:
"Using the attribute with no arguments is designed to improve optimization by relying on the aliasing property it implies."

https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html

And the validation is only to check that "Associating a function with a deallocator helps detect calls to mismatched allocation and deallocation functions and diagnose them under the control of options such as -Wmismatched-dealloc.". That doesn't seem to be very useful in this use case.

I'm curious about what is the advantage or disadvantage of not using the attribute.

[ssokolow]

OK, I've read it and I fully agree.

Hell, one of the big reasons I'm working to replace my use of Python with Rust for userland stuff is that, once I got over the initial learning hump, it was a way to get more compile-time correctness than Python, abstractions that aren't cripplingly slow, and get it more CPU-efficient and more memory-efficient than what I was used to in Python. Win win win. Heck, if you're using something like py2exe for an equivalent to static linking, it's even more compact than using Python.

...and resource-efficiency when memory-safety is non-negotiable (and I don't like the UI/UX direction GTK is going) is the big reason I use PyQt with QWidget APIs for my GUI apps. Put as much as possible in C++, use Python for the glue, and, if there's something that can be split out as a well-encapsulated backend, use rust-cpython or PyO3 to avoid the Python waste there.

There's a reason my Firefox, Ungoogled Chromium, and Thunderbird are the heaviest non-game things on my system by an order of magnitude, and I loathe them.

I also have something of my own to say about this tweet they quoted:

@tveastman: I have a Python program I run every day, it takes 1.5 seconds. I spent six hours re-writing it in rust, now it takes 0.06 seconds. That efficiency improvement means I’ll make my time back in 41 years, 24 days :-)

Idiot. You rewrite it in Rust because MyPy+PyLint+Flake8, despite trying their best, can't make up for a dynamically typed language with exception-based error handling and nullable variables being crap for maintainability.

You rewrite it in Rust because, if you don't need to modify it, it's trivial to make a statically linked binary which will continue to operate until your OS kernel decides that it's time for 64-bit x86 to go the way of m68k or PPC on macOS or Win16 on Windows. That is, long after version bumps to your dependencies would have broken things in Python. (Ask me how I know.)

You rewrite it in Rust so it's easy to reuse the code in libraries that can expose a C ABI and be loaded from whatever language you find yourself needing to use.

You write it in Rust the first time if you need an excuse to justify saving CPU time.

[oiaohm]

I don't see any attribute in the pgtable.c linked. Are you sure that's the right link?

https://linux-kernel.vger.kernel.narkive.com/jyUQhL5H/patch-use-attribute-malloc-for-gcc-3-2

+extern pgd_t *pgd_alloc(struct mm_struct *) malloc_function;
Its in the first link that malloc_function here is in fact __attribute__((malloc)) when running with a compiler that support it.

Yes it declared in linux/include/asm-i386/pgalloc.h that is a C header that is basically the C version of a wrapper. Covert into a rust wrapper you are losing details.

I'm also a bit confused by the attribute(malloc). By default it only seems to be helpful to optimizing and does not add any extra validation:
"Using the attribute with no arguments is designed to improve optimization by relying on the aliasing property it implies."

https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html

And the validation is only to check that "Associating a function with a deallocator helps detect calls to mismatched allocation and deallocation functions and diagnose them under the control of options such as -Wmismatched-dealloc.". That doesn't seem to be very useful in this use case.

I'm curious about what is the advantage or disadvantage of not using the attribute.

You are just about to learn why there is such thing as Linux kernel C. That __attribute__((malloc)) is not only picked up by gcc its also picked up by sparse the static analyzation tool of Linux kernel. So it does in fact check more than the mismatched-dealloc. Including if you check if the result was null before using.

There are extra attributes the complier picks up and there are extra attributes sparse picks up. Sparse has some very specialist attribute configuration like making sure you are not attempting to use a userspace pointer in kernel space and vice verse.

https://www.kernel.org/doc/htmldocs/kernel-api/API---copy-to-user.html

Notice the __user here this is not a C standard attribute this is sparse attribute.. That a attribute that market that the item there has to be user space.

Then there are more attributes that are added for sprase to detect locking issues. Yes issues that can result in a deadlock/frozen system. As locks are altered inside the kernel the added attributes on functions that sprase processes change.

Notice what the rust developers said we write the wrapper once we are done. Sorry no. The include files of the Linux kernel are not static. The include files of the Linux kernel have a lot of extra details so that sparse can detect problems. Guess what your general rust wrapper throws away all those details.

With something like the Linux kernel have something the rust wrapper that is we need to check "X' C language include that it has exactly 'y' function declaration because if it does not there could be some alteration resulting in the current wrapper being incompatible. Note this will not catch everything because sparse rules themselves get extended.

This is the catch Linux kernel has used features of C to hide extra details about functions for the static analyzation tool that has rules designed particular for the Linux kernel.

Yes the specialist linux kernel rules in sparse need to be applied to rust modules. Yes for rust to be a first rate linux kernel language there need to be a Linux Kernel rust. Yes this is a rust where rules that are used in sparse kernel wide in the C sections are also being checked inside rust. Same goes for the __attribute__ flags.

This is not that I hate rust. The simple reality the decades of Linux kernel developer has lead to some very specialist static analyzation that is required to make sure that kernel space runs properly. Yes the reality is the Linux kernel is not general C because of how much stuff has been added for static analyzation over the years be it compiler performed or sparse(static anlyzation) tool performed.. The big problem I have is the rust developers idea that you can just throw all this stuff out. This is throwing the baby out with the bath water the custom static analyzation of the LInux kernel has decades of development based off the decades of different kernel space problems found.

Is the static analyzation of the Linux kernel a finished work the answer is no its a work in progress and this is also why the idea is write wrapper once we are done is just totally wrong.

[dragonn]

Is the static analyzation of the Linux kernel a finished work the answer is no its a work in progress and this is also why the idea is write wrapper once we are done is just totally wrong.

I am fully for Rust in Linux kernel, but I must say thank you, that the first post on this whole thread with raises a valid concern and you definitely know what you are talking about.
I expect that some of that static analyzation will be just need to redone in Rust, but I think most of it can be done via macros, no need to "kernel Rust". Macros in Rust are much more powerful then in C, for example I am often using a library for working with sql db called sqlx, by using macros it can check sql queries against a existing database an abort a compilation with an error if any of them in your program fails.

[ssokolow]

for example I am often using a library for working with sql db called sqlx, by using macros it can check sql queries against a existing database an abort a compilation with an error if any of them in your program fails.

Yeah. Rust's procedural macros are pretty powerful and attribute macros are often enough to extend the compiler itself with things like __user and __attribute__((malloc)). No external linting tool needed.

[oiaohm]

I am fully for Rust in Linux kernel, but I must say thank you, that the first post on this whole thread with raises a valid concern and you definitely know what you are talking about.
I expect that some of that static analyzation will be just need to redone in Rust, but I think most of it can be done via macros, no need to "kernel Rust". Macros in Rust are much more powerful then in C, for example I am often using a library for working with sql db called sqlx, by using macros it can check sql queries against a existing database an abort a compilation with an error if any of them in your program fails.

Yeah. Rust's procedural macros are pretty powerful and attribute macros are often enough to extend the compiler itself with things like __user and __attribute__((malloc)). No external linting tool needed.

Please note I did not say the procedural macros of rust could not do the same tests as sparse. The problem is the way C is linked in rust there is no detection if the flags has changed.

Lets say a __attribute__((malloc)) on a function has been changed to add __attribute__((assume_aligned (16))) you might perform a reallocation on that pointer now and not put it back on the wrong alignment. Of course modern C compiler can warn you of this problem. The way rust imports C is not going to pick up that the wrapper that was code in rust is now missing this C attribute or is now missing a new assigned sparse attribute equal.

The current way rust does it C imports does not make sure that the rust code is in alignment with the C code for the rules being applied..

ssokolow this is not a question can rust do X. This is question can the combined rust + C code base both be doing the same X check and balances at the same time. With the second question will you have errors when the check and balances are X in C and Y in rust (as in miss aligned checks and balances).

Without having the checks and balances aligned between Linux kernel C and Rust there is going to be errors happen that should not happen.

Yes it should be expected that at times someone will change a sparse flag or add a extra C compiler attribute that the Rust wrapper is going to throw a build error if it has not been updated to match. The fact at the moment the wrapper code will not error in these cases allows the checks and balances in the rust and Linux kernel C code to go out of alignment.

I don't see this as a unfix-able problem. More than rust developers need to admit that they cannot take in C throw away all the __attribute__/static analyzation notes in C and expect everything to fine. Do this is going to result at failure in the wrapper code.

Basically made the core code in the module higher quality only to have it let down my poor quality wrapper code caused by the simple error of throwing key information out and not making sure that key information is still in alignment.

Yes another area would be having rust procedural macros aligned to sparse rules so that if sparse checking rule has been updated that the procedural macro checking the same thing fails with error that its out of alignment now and need fixed.

Rust development has a problem with we are better we don't have to make sure that we integrate into C code bases correctly because C is inferior. Yes the C standard is inferior to rust in lot of ways. But compilers and static analyzation tools have extended C along way that C is not as inferior as it first appears. There are a lot of rules that C compiler and C static anlyzation are enforcing that the rust compiler need to enforce as well or rust C wrappers are going to be defective.

Yes in Linux kernel being defective in way of causing a kernel panic at some point because the C wrapper is wrong if this is not fixed. Yes this would be just a matter of time.

[ssokolow]

I certainly agree that these are concerns and that, if they want Rust support in the kernel this badly, they'll need to do something like contributing patches to bindgen.

[oiaohm]

I certainly agree that these are concerns and that, if they want Rust support in the kernel this badly, they'll need to do something like contributing patches to bindgen.

Linux kernel is worse due to sparse. But rust/c hybrid projects not picking up the __attribute__ and other compiler features defined in the include files when importing C functions is also still a problem to long term stability.

Lot of ways the way rust makes C interfaces needs a look over. Being able to define the C include file and the C function declare the rust import is meant to line up in the rust code base with would go a long way and would allow a checking function.

Please do note -fsanitize=alignment that you can turn on with gcc and clang(llvm) will cause warning with __attribute__((assume_aligned (16))) tagged stuff in C.

Modern day C compiler extend well past the C standard specifications and there is a lot of information in the C include files to guide the compiler these days. Of course sparse just add a extra level of extra information and processing.

Rust developers have at this stage taken the import of C functions way too lightly and have ignored the extra attributes that are in the modern C include files. Yes all the extra information in __attribute__ is not in the C standard.

C compiler developers have been attempting to reduce how problematic the C language is and they have done this by basically extending the C language without adding those extensions to the standard.

C language is a mess ways than one. Yes ignoring how compilers have extended C to reduce the problems of C as rust C interface currently do just plays into running into all those problems the C compiler developers added those extensions to prevent all over again. Ok at least with rust the defects will be restricted to the wrapper code but even so we should not want these preventable defects at all.