XCP-ng Initating Effort To Rewrite Xen Components In Rust

[archkde]

They are fine languages, but neither can match the performance of Rust. Both are also garbage collected, which makes them unsuitable for certain application areas. Go also doesn't provide the same safety guarantees.

You probably don't want to write the hypervisor in Go or OCaml because of the garbage collection, that's right. But for userspace tools, it should be fine (both have a pretty low GC latency). Go is not fully memory safe, but I haven't actually seen that being an issue in practice (unlike the shitshow that are C and C++). I don't know how mature the paralaelism is in OCaml now, since I have started writing my own code in Rust before multicore was merged.

[jacob]

You probably don't want to write the hypervisor in Go or OCaml because of the garbage collection, that's right. But for userspace tools, it should be fine (both have a pretty low GC latency). Go is not fully memory safe, but I haven't actually seen that being an issue in practice (unlike the shitshow that are C and C++). I don't know how mature the paralaelism is in OCaml now, since I have started writing my own code in Rust before multicore was merged.

I agree that for userland tools they are great languages, Go in particular has become quite popular in the Linux container management ecosystem. Canonical's Snap package manager is also written in it IIRC.

I haven't looked at OCaml literally in more than 20 years so I'm not sure where it's at now in terms of parallelism. I think back then the GC was not thread safe and thus OCaml only supported concurrency/"green threading" but not actual parallel execution. But I could be wrong about this.

[OneTimeShot]

Wow - this project has lasted much longer than I thought it would have. This stuff does the remote configuration of a XenServer computer cluster (create VMs, set up networks, file storage, etc). It does what libvirt does basically.

I'd be interested if there is any functionality that libvirt is missing that this has (I doubt it).

[patrick1946]

Go is not fully memory safe, but I haven't actually seen that being an issue in practice (unlike the shitshow that are C and C++).

Actually it would be nice if people don't think that C and C++ are the same language. C++ is quite compatible with C but that's it. You can handle resources in C++ quite well but you are not forced to. Many still write a quite outdated C++. That is the drawback of compatibility.

And the compiler is not ensuring that your resource management is right to the extent that Rust doing it. You have to use tests and reviews. For many applications it is okay because they are not security sensitive. You have to write tests anyway because many errors are not detectable are compile time.

But people don't do write tests or the quality of the tests are bad. That is not a C++ only problem but especially C++ because many programmers are older and don't want to change.

I really would like if there is pressure to a more modern, safer C++ subset. With modules they could introduce epochs. I think it is not economical to rewrite all the C++ code in different language.

[Quackdoc]

I would sum up Xen this way: it's a different but very interesting design (vs KVM). However, like KVM, there's little chances you might use Xen "alone" if you want to scale a bit. What matters in this case is the toolstack around. In a way, XCP-ng is closer to VMware (regarding the design). That's not a coincidence that almost all our new comers are from VMware

I am not the most familiar with vmware, but have used it a bit, though since most of my use cases require emulated 3d (IE. virgl/venus) in linux, qemu via libvirt is what im most familiar with, but lately i've been doing stuff where that isn't so necessary.

for xcp-ng/xen is there any discussion around potentially using crates from rust-vmm should testing the rust migration waters and the conclusion is that it seems viable? I can see both pros and cons, it seems like it could reduce longterm maintenance a bit, on the trade off of needing to spin off upstream should the two paths not align. at the very least the general direction of rust-vmm, being safe and reliable. at the very least, IIRC both crosvm and firecracker use bits of it.

can't wait to check out xcp-ng though, will certainly be looking into it soon

[archkde]

I haven't looked at OCaml literally in more than 20 years so I'm not sure where it's at now in terms of parallelism. I think back then the GC was not thread safe and thus OCaml only supported concurrency/"green threading" but not actual parallel execution. But I could be wrong about this.

I think "multicore" (parallelism) support was merged into 5.0, but as said, I haven't used that version myself.

[archkde]

Actually it would be nice if people don't think that C and C++ are the same language. C++ is quite compatible with C but that's it. You can handle resources in C++ quite well but you are not forced to. Many still write a quite outdated C++. That is the drawback of compatibility.

C and C++ are certainly not the same language (that's why I used the plural), but both are a safety shitshow because of their lacking memory safety. Even the C++ way of handling resources "quite well" is not particularly great in this aspect.

[patrick1946]

C and C++ are certainly not the same language (that's why I used the plural), but both are a safety shitshow because of their lacking memory safety. Even the C++ way of handling resources "quite well" is not particularly great in this aspect.

Yes but much of the code gets never in contact with the network. Only in webbrowser where you develop in a networked ui platform get this big surface. I am not sure that exposing so much is smart but you can make money with data collection, so it is desirable for everybody who want to be rich. ;-)

Actually most of this breaches I have seen were using c functions which uses \0 or use pointers into variable length data. That kind of code is hacky.

[ssokolow]

But... why "Riir"? I mean, I don't know Rust and I assume it must be a good thing (TM) since so many C/C++ project move to it, but why rewriting something that already works? Wouldn't be better to just start something new instead? Porting existing code without adding anything new, being forced to keep compatibility with what exists, is among the most boring things a programmer could be asked to do. And it gains you nothing in the short term, so you only delay innovation of your existing product.

I'd rewrite from Go to Rust if I were using it for three reasons:

1. Go is hostile to FFI by design, while Rust embraces [2] interoperability. (This was a big draw for me. It being easy to write stuff that can be reused from other languages without giving up compile-time safety checks.)
2. Go feels like its developers stopped following developments in programming language design in the 1990s and that's made worse by their aversion to adding quality-of-life features like generics until years after ecosystem API trends have solidified. Rust, by contrast, has had, from the beginning, sum types, generics, no exceptions, no null/nil, and a solid implementation of monadic error handling.
   
   

   Amen. Yes to everything else you said to, but this nailed it for me. Go is braindead AF. And I'm not even talking about complicated shit, just basic "make my life less mindless and plodding for 1 fucking second, it will be more readable, cleaner, and everyone would prefer it, but no".
   
   -- Uptown-Dog

3. Rust gives you much more ability to encode invariants into the compiler so you can relax when making changes or reviewing a PR by something like Github Dependabot because the compiler will catch much more.

I don't really write low-level stuff on modern systems, but I imagine there's one other reason that's on their minds:

Rust works well in both the high-level/application domain and the low-level/system domain so, if they like the results, moving stuff to rust could achieve the same breadth of functionality in fewer languages, which broadens the range of potential contributors (how many people wandering by are likely to know Go and Ocaml if they want to implement/fix a feature that crosses both of those components?) and reduces the complexity of the build environment they need to set up.

(For my DOS retro-hobby project, I've made it a rule that mandatory dependencies for the build process must either be vendored or coming from the Open Watcom C/C++ installer and the Watcom Make file must Just Work™ without extra setup steps.)

[ssokolow]

Yes, that I mean by difficult. I really wanted to like Rust but I find Rust quite metaphysical approach not seducing. It is not very pragmatic. I got the impression that it was written by people who dislike C++ and want to develop a better C. I understand that there are bad ideas in C++ but in my view templates are not. They are a very powerful generic implementation and with concepts they got manageable. In Rust in many cases you have to use macros. Even if they are good macros they are to my understanding not typed. This is a big drawback to me. The generics are quite limited too. And then Rust has this hidden exception support with panic. This goes on. I have the impression that it wants to fix the same problems like Java but in a faster way. For that the resource model has to be limited which makes it difficult. I am not sure if that is enough to be called a C++ successor. I am much more inclined to think that it will find it's niche like golang.

I think the biggest problem of Rust will be that the other languages will not go away. You have to interface too them easily. But if your resource model is different it gets much harder.

1. You're exactly right about Rust's relationship to C and C++. C++ was "Here at Subversion, we're making a better CVS" while Rust is "We still need something to serve CVS's role, but let's go back to first principles and try again". (Bearing in mind that, in many of its design decisions, Rust is an ML-family language like Ocaml with a coat of carefully selected C++ syntax elements to make it friendlier to mainstream programmers.)
   
   That's why the semantics are so different. It's more of the lineage that began with ML in 1973 than of the ALGOL 68 lineage like C is. The Rust compiler was even originally written in Ocaml before it became self-hosting.
   
   That's also where it got things like let for variable declaration, postfix type annotations separated by colons, 'a for lifetimes (Ocaml's equivalent to <T> is 'a and lifetimes in Rust are a special kind of generic parameter), Option/Some/None (option in Ocaml), match, fn (fun or function in Ocaml with fn being more from fn(x) in math papers), -> and => (just -> in Ocaml) for function returns and match arms, and () as the void return type.
2. C++ templates are dynamically/duck-typed (eg. SFINAE) and Rust developers wanted statically typed generics. Among other reasons, to make error message complexity more manageable than "On second thought, I'll just stare at the code some more". Rust's macros are token-based and more like C++'s templates and they're what you're supposed to use when you want duck-typing. (Rust's declarative macro syntax is just a variant of their match syntax with some extra syntax like $argument to keep things clean when you're writing code that generates code.)
3. Rust's macros are not completely untyped. Instead, they operate on token types (eg. expression, literal, identifier, token tree, etc.) and pattern matches (You can build a DSL in Rust's declarative if you want to thanks to the pattern matching.)
4. Rust's panic! is equivalent to C's abort() and the top-level Cargo.toml (Makefile) has final say on whether panics unwrap or abort. They weren't catchable in Rust 1.0 and catch_unwind was added specifically for translating unwinding panics into error codes when writing a Rust library which exposes a C interface. Panics are specifically meant for handling programmer errors in the same way abort() is (eg. they're the building block for Rust's asserts) and you will not be well-received if you treat them as exceptions. (I still don't trust the competence of author of an executable parsing crate which I managed to induce to panic! rather than Err(...) on untrusted input and that was about not being careful enough about "Untrusted input should never be able to induce a panic. That's a DoS vulnerability.", not about "panics are for programmer errors/bugs, not for user errors".)
5. It's been explicitly said by the developers about their goals that there should be no room above assembly language and below Rust to need another language.
6. The resource model isn't "limited"... it's just safe-by-default. Do you want a shared_ptr? Use Arc<T> or even Rc<T> (Rust doesn't do "was libpthread linked?" shenanigans to switch between atomic and non-atomic refcounting). Do you want to use a raw C-style pointer? Use one. Just remember that you can only dereference them inside unsafe blocks. Do you really need RTTI-style type tricks like downcasting? Look into implementing the Any trait for your struct to tell the compiler to not throw away the type information.
7. Rust was explicitly co-designed with the Servo browser engine and evolved around the use-case of replacing C++ in Firefox. Google has been using it to replace C and C++ in Android (not Java) and they're quite pleased with the results. The Chromium team are gearing up to use it in Chrome. Patches are landing in the Linux kernel to support drivers written in Rust and Asahi Lina is very happy with the experience of using it to write the GPU driver for M1 macs. I'd say that's pretty strong evidence in support for it being suitable for replacing C++ in many use-cases.