Google Wants To See Rust Code In The Linux Kernel, Contracts The Main Developer

[NobodyXu]

It's not perfect... and it's actually part of the standard library. (The std::any module)

Building a runtime reflection system for Rust 🦀️ (Part 1) is a good starting point.

Thanks for the link again.

I would take my time and read it, very interested in how people would approach this problem.

[silmeth]

Regarding the use of `Self` of Traits, I think the use of it is not common. And if one is decided to use polynomial, they can simply replace it with `&dyn Trait` for `&Self` or `Box<dyn Trait>` for `Self`.

It is. Take the Clone trait – if you want to be able to copy the objects you take in, you need it (or something like it), with method returning an instance of Self. If you carry around a Vec<Box<dyn SomeTrait>> (vector of trait objects) and you want to clone it, you have a problem. Or the Add trait – say you want to make a polymorphic function summing all elements of a vector – it will take Vec<T> where T is something you can add to other Ts. But the add() method of the Add trait in this case returns Self, so it’s not object safe…

Swift handles this by ‘reabstraction’ – the compiler boxes stuff under the hood and all objects behave the same way, so you can easily have vector of cloneable or addable protocol objects.

So you’d need to make a trait that returns trait objects on the clone function, but if you create a ObjectCloneable trait that has a function returning Box<dyn ObjectCloneable> instead of Self, then you remove all the other info about the type, now you cannot use the other interesting traits of the object. You have to return something like Box<dyn ObjectCloneable + SomeTrait> instead. But now you have to have a special cloning trait for each trait combination that you use in your program…

Maybe libraries who want generics to be polynomial as in swift can make the generics a wrapper of the actual implementation that takes trait objects.
They can even implemented a macro for it that automatically wraps the actual implementation with a generic, or they can just export the polynomial version directly.

Yes, I think you’re right here. If you really want to expose a polymorphic function in a dynamic lib, this is probably the way to go. I don’t think it’d ever become part of the official Rust ABI.

Regarding the lifetime, I don’t think they ever need to be encoded in the ABI.
They only need to present in the module (not dynamic library) and let the compiler know about the lifetime at compile time.

On the pointers, I am pretty sure they are not part of the ABI. (…)

Yeah, they are not part of the ABI proper – as I wrote, they’re compiled away. Perhaps I touched a bit different problem. My thinking was that if you compiled your program against version A of the library, and you upgrade it – if the API of the functions you use changed but the ABI did not (eg. a function that used to take references now takes Boxed objects; or a function that used to return static references now returns references to parts of the arguments passed) – you’d rather want linking errors that random segfaults from something that is supposed to be purely safe Rust. So the full API information with lifetimes, pointer types, etc. is still needed.

Also, it would need completely new heap-allocation types (like Box, Vec, etc.) – because the library could be compiled against a different allocator than the program using it, so the library wouldn’t ever be able to deallocate any object passed to it from outside directly (while current Box implementation just deallocates directly on drop – so maybe you would just need to always use a different ExternalBox type in public APIs…?), etc., etc.

These all are problems that could be solved somehow – but I don’t think there are people in the Rust community willing to do so just to have Rust-native dynamic linking – which doesn’t seem to be needed by most, and those who really need dynamic linking can use C-ABI public API through Rust FFI today.

[NobodyXu]

It is. Take the Clone trait – if you want to be able to copy the objects you take in, you need it (or something like it), with method returning an instance of Self. If you carry around a Vec<Box<dyn SomeTrait>> (vector of trait objects) and you want to clone it, you have a problem. Or the Add trait – say you want to make a polymorphic function summing all elements of a vector – it will take Vec<T> where T is something you can add to other Ts. But the add() method of the Add trait in this case returns Self, so it’s not object safe…

Swift handles this by ‘reabstraction’ – the compiler boxes stuff under the hood and all objects behave the same way, so you can easily have vector of cloneable or addable protocol objects.

So you’d need to make a trait that returns trait objects on the clone function, but if you create a ObjectCloneable trait that has a function returning Box<dyn ObjectCloneable> instead of Self, then you remove all the other info about the type, now you cannot use the other interesting traits of the object. You have to return something like Box<dyn ObjectCloneable + SomeTrait> instead. But now you have to have a special cloning trait for each trait combination that you use in your program…

Maybe the `clone` method can be defined for the combination of the traits instead, like this:

trait Trait: trait1 + trait2 + … {
    pub fn clone(&self) -> Box<dyn Trait>;
}

https://play.rust-lang.org/?version=...7b0ad779dbb51a

Yeah, they are not part of the ABI proper – as I wrote, they’re compiled away. Perhaps I touched a bit different problem. My thinking was that if you compiled your program against version A of the library, and you upgrade it – if the API of the functions you use changed but the ABI did not (eg. a function that used to take references now takes Boxed objects; or a function that used to return static references now returns references to parts of the arguments passed) – you’d rather want linking errors that random segfaults from something that is supposed to be purely safe Rust. So the full API information with lifetimes, pointer types, etc. is still needed.

I think this kind of problem is already solved using library versioning and dependency management via apt, yum, dnf and etc.

Tricks is to put the version into the dynamic library and symlink the library + major version to the current used minor version of dynamic library, e.g. libpython3.9.so is a symlink to libpython3.9.1.so

I saw this on debian, gentoo and possibly all distros.

This is not a problem common to rust, C/C++ also suffers it.

Even if the argument to the API didn’t change, the semantics might change slightly, so they still need the library versioning,

Also, it would need completely new heap-allocation types (like Box, Vec, etc.) – because the library could be compiled against a different allocator than the program using it, so the library wouldn’t ever be able to deallocate any object passed to it from outside directly (while current Box implementation just deallocates directly on drop – so maybe you would just need to always use a different ExternalBox type in public APIs…?), etc., etc.

I think it is already there. Looking at `std::boxed::Box` https://doc.rust-lang.org/std/boxed/struct.Box.html and I can see the generic contains parameter A for allocator, which defaults to Global and the impl in std is only for the Gloabl allocator.

[silmeth]

This is not a problem common to rust, C/C++ also suffers it.

Even if the argument to the API didn’t change, the semantics might change slightly, so they still need the library versioning,

Except that C/C++ don’t give the strong guarantee that if you don’t use unsafe code yourself directly, you can’t have any memory unsafety (barring unsafe-related bugs in libraries, but implementation details changes without changes to the API should never matter).

Rust guarantees that if you are allowed to call this function, it will not segfault or do other Bad Stuff™ (the worst can happen is a panic). API-unaware dynamic libraries would break this promise – because upgrading the library would possibly suddenly introduce double-frees, use-after-frees, etc. in safe Rust.

I think it is already there. Looking at `std::boxed::Box` https://doc.rust-lang.org/std/boxed/struct.Box.html and I can see the generic contains parameter A for allocator, which defaults to Global and the impl in std is only for the Gloabl allocator.

This is part of the effort to allow different per-container allocators (Rust std used to require you to use one global allocator for every heap-based container, but there’s push to allow finer control and use of multiple different allocators in one binary, relevant RFC). But note that it’s still a regular generic (ie. statically monomorphised) type parameter. I guess it’d be possible to create containers keeping their allocator dynamically – but that wouldn’t be the same type as the current Box<T, Allocator>.

[ssokolow]

These all are problems that could be solved somehow – but I don’t think there are people in the Rust community willing to do so just to have Rust-native dynamic linking – which doesn’t seem to be needed by most, and those who really need dynamic linking can use C-ABI public API through Rust FFI today.

Partly because of third-party solutions like the abi_stable crate.

[NobodyXu]

Except that C/C++ don’t give the strong guarantee that if you don’t use unsafe code yourself directly, you can’t have any memory unsafety (barring unsafe-related bugs in libraries, but implementation details changes without changes to the API should never matter).

Rust guarantees that if you are allowed to call this function, it will not segfault or do other Bad Stuff™ (the worst can happen is a panic). API-unaware dynamic libraries would break this promise – because upgrading the library would possibly suddenly introduce double-frees, use-after-frees, etc. in safe Rust.

Can you elaborate on why adding major version to the library name doesn’t solve this problem please?

My understanding is that, if you add major version to the library name, then it would ensure the same library with two different major version would have different name.

Program depends on the one major version either isn’t effected at all and still uses the old one or failed at dynamic linkings stage and requires rebuild/upgrade.

I also think that change in minor version number should not modify the behavior of existing APIs and should only containsnew APIs or bug fix to internal of the dynamic library.

This is part of the effort to allow different per-container allocators (Rust std used to require you to use one global allocator for every heap-based container, but there’s push to allow finer control and use of multiple different allocators in one binary, relevant RFC). But note that it’s still a regular generic (ie. statically monomorphised) type parameter. I guess it’d be possible to create containers keeping their allocator dynamically – but that wouldn’t be the same type as the current Box<T, Allocator>.

Maybe Box<T, &’static dyn Alloc> lol

[silmeth]

Can you elaborate on why adding major version to the library name doesn’t solve this problem please?

My understanding is that, if you add major version to the library name, then it would ensure the same library with two different major version would have different name.

Program depends on the one major version either isn’t effected at all and still uses the old one or failed at dynamic linkings stage and requires rebuild/upgrade.

I also think that change in minor version number should not modify the behavior of existing APIs and should only containsnew APIs or bug fix to internal of the dynamic library.

Because you can still copy-paste the lib file, change its name, and force it to be linked against. Because somebody can introduce a breaking API-change and not bump the version in a sem-ver compatible way. Because someone might not even bump the version number for some internally used artifacts deployed by hand. Etc.

In the case of static linking, if one does such shenanigans (either somebody just edits library sources by hand, changing the API, or cargo downloads incompatible version, believing blindly in the sem-ver number, or someone copy-pastes the wrong library version pretending it’s a different version…), the compiler just rejects the code – it sees that the usage is not compatible with the API exposed – the statically linked program is rejected during type-checking.

But in case of API-unaware dynamic linking all might seem to work fine until segfault or, worse, some wrong logic on incorrect memory being executed.

It all boils down to the guarantees you want to keep in the dynamic linking case. I’d like all the guarantees Rust gives when statically linking to be still upheld without just depending on the reported version number not lying about API compatibility.

Maybe Box<T, &’static dyn Alloc> lol

Yeah, I think that could work, actually.

[NobodyXu]

Because you can still copy-paste the lib file, change its name, and force it to be linked against. Because somebody can introduce a breaking API-change and not bump the version in a sem-ver compatible way. Because someone might not even bump the version number for some internally used artifacts deployed by hand. Etc.

In the case of static linking, if one does such shenanigans (either somebody just edits library sources by hand, changing the API, or cargo downloads incompatible version, believing blindly in the sem-ver number, or someone copy-pastes the wrong library version pretending it’s a different version…), the compiler just rejects the code – it sees that the usage is not compatible with the API exposed – the statically linked program is rejected during type-checking.

But in case of API-unaware dynamic linking all might seem to work fine until segfault or, worse, some wrong logic on incorrect memory being executed.

It all boils down to the guarantees you want to keep in the dynamic linking case. I’d like all the guarantees Rust gives when statically linking to be still upheld without just depending on the reported version number not lying about API compatibility.

While that would work, I would argue its cost is too much.

Currently, some mangled symbols in std library can go up to 300 bytes due to the namespaces.

I would say it can take an alternative approach of hashing existing API in sha512 automatically and store it in .rodata section.

Say you release version 0.1, then all APIs in 0.1 are hashed and stored in .rodata
When 0.2 is released, all APIs in 0.1 are hashed again and stored in .rodata and all new APIs introduced in 0.2 is hashed and stored in .rodata

These hashes can be checked at runtime to see if the API/ABIs are changed.

If creating a hash for APIs introduced in each release is too complex, just calculate a hash for all APIs and deny minor version changes as well.

Yeah, I think that could work, actually.

After a second thought, it is actually reasonable for the custom allocators to be stored globally.

If one want to pass around the allocated data just like any normal heap allocated data, the the allocator has to be stored the allocator globally, otherwise you would have to limit the scope the allocated data and introduces lifetime, which might still be useful in some cases, but if one is writing custom allocator for data structure like list, tree, etc, which requires custom allocator for better performance, then it only make sense to use a global custom allocator.

[birdie]

116 messages from people absolute most of whom have never submitted a single line of code to the kernel and most of whom have never written a single line of code. LMAO.

I won't even attempt to read any comments in this "discussion". Rust or not in the Linux kernel - that's of very little essence. What matters is stable API/ABI and Rust won't solve it.

[lsatenstein]

If Rust is to be used, I would say, why not, but not for time-critical processes.
The gcc or clang compilers, over the years, have become enhanced highly optimizing. The compilers review the code and is able to perform optimizations better than many humans.

The Rust compiler does not appear to have the attention to optimization that has been built into gcc or clang over the past 10+ years.