Rust GCC Code Generator "rustc_codegen_gcc" Can Now Bootstrap Rustc

[arQon]

If you speak of CORBA and (D)COM, I think the D-Bus in Linux ecosystem is also in the same league.

No, not even close. dbus is primitive and clumsy, and was still unstable last time I looked (which is somewhat worrying, since this is a well-defined problem). A quick search for examples today suggests that whether or not it's now stable, the "primitive and clumsy" part is still very much there.

> But none of them seems to be something low level stuff such as glibc eager to interface in.

Sure, but that's a sensible choice made by the glibc devs, and doesn't in any way preclude a dev or a third party having or creating an IDL version of it. Regardless of its importance, glibc is still only one of thousands of libraries, and the root issue here is that ANYTHING that is Not C (including some "dialects" of C) is always going to need SOME sort of glue to talk to anything else: and the only competent solution that's appeared so far is an IDL.

Your question was "will there ever be, a non-C ABI for everyone to talk to safely ...", and CORBA / DCOM is it. The deficiencies of dbus, or musings about what other non-existent options might appear in the future just because someone reinvents the wheel by choice or through ignorance, is unfortunately an unrelated conversation that I don't have time for ATM.

[ssokolow]

Your question was "will there ever be, a non-C ABI for everyone to talk to safely ...", and CORBA / DCOM is it. The deficiencies of dbus, or musings about what other non-existent options might appear in the future just because someone reinvents the wheel by choice or through ignorance, is unfortunately an unrelated conversation that I don't have time for ATM.

Does CORBA support intra-process (non-IPC) operation like non-D COM does? ...because that's what the question is about. If you want to... say... interface a Rust rewrite of ffmpeg with a Swift API for some kind of iDevice computer vision coprocessor, you don't want to be tossing 30-60 frames per-second of HD (or bigger) video over a typical IPC channel.

[arQon]

Does CORBA support intra-process (non-IPC) operation like non-D COM does? ...because that's what the question is about. If you want to... say... interface a Rust rewrite of ffmpeg with a Swift API for some kind of iDevice computer vision coprocessor, you don't want to be tossing 30-60 frames per-second of HD (or bigger) video over a typical IPC channel.

Don't remember that far back, sorry. My gut feel is "no", but that's "at the time" and the answer may be different now even if it was the case then.

I'm not sure I buy your ffmpeg argument though. ANY competent IPC mechanism will use shm for localhost, so even the worst case is "just" an extra memory copy. Unpleasant for 4K, but not a problem for 1080p even on ancient x86 systems. A Pi etc certainly wouldn't appreciate it though, certainly.

I'd *hope* that there'd be the equivalent of a splice in there by now, even if the transport was still nominally IPC, but IDK what the current state of play is, sorry.

[NobodyXu]

ANY competent IPC mechanism will use shm for localhost, so even the worst case is "just" an extra memory copy.

This is not true, as many simply use socket or pipe, since it is far easier and provide good enough performance.

[ssokolow]

Don't remember that far back, sorry. My gut feel is "no", but that's "at the time" and the answer may be different now even if it was the case then.

I'm not sure I buy your ffmpeg argument though. ANY competent IPC mechanism will use shm for localhost, so even the worst case is "just" an extra memory copy. Unpleasant for 4K, but not a problem for 1080p even on ancient x86 systems. A Pi etc certainly wouldn't appreciate it though, certainly.

I'd *hope* that there'd be the equivalent of a splice in there by now, even if the transport was still nominally IPC, but IDK what the current state of play is, sorry.

Still, it seems kind of ridiculous to imagine that every application that wants to use more than one language is going to be following in Firefox and Chrome's footsteps in spraying itself across two or more processes, purely because nobody can agree on an in-process IDL.

At least Python has the excuse of following in Apache's mpm_prefork's footsteps because it hasn't managed to find a performant way to get rid of the Global Interpreter Lock... and actual "one process per..." parallelism has a long history.

[arQon]

This is not true, as many simply use socket or pipe, since it is far easier and provide good enough performance.

erm... what do you think the transport for a local socket is? Do you imagine that the packets actually go out on the wire, get bounced back, and are then read in again, or something like that?

Regardless, I think that when the concern is "performance", which was ssokolow's question, the goal is unlikely to be "list other options that you admit are generally slower". Once you start moving enough data, ANY additional rounds of copying are NOT "good enough performance", regardless of how true that is for more trivial cases.

It does come down to the implementation, which at least one of us isn't current on for Linux - but high-volume / low-latency / performant IPC is both far from being a new problem and something that an awful lot of systems rely on. There'll be "good" implementations somewhere, because using a crappy one means spending *literally millions* of dollars more on hardware. Using "but this is easier!" as an excuse doesn't cut it in those situations, un/fortunately.

[NobodyXu]

erm... what do you think the transport for a local socket is? Do you imagine that the packets actually go out on the wire, get bounced back, and are then read in again, or something like that?

I know that they don't go out on the wire, as any connection to localhost would be written to the buffer and directly pushed to the other process.

However, compared to shmem solution you mentioned, using socket/pipe still incurs a large overhead, namely the context switching, two copies between user space and kernel space.

Even with io-uring, it will still have a large overhead since the two copies cannot be eliminated, at least, not now.

It does come down to the implementation, which at least one of us isn't current on for Linux - but high-volume / low-latency / performant IPC is both far from being a new problem and something that an awful lot of systems rely on. There'll be "good" implementations somewhere, because using a crappy one means spending *literally millions* of dollars more on hardware. Using "but this is easier!" as an excuse doesn't cut it in those situations, un/fortunately.

Not everything need the most performant IPC though.

When you look at the web server with WSGI/FastCGI, the dynamic content generator (mostly written in python and PHP) communicates with the web server (nginx or apache) using socket.

They could use something better, like shmem, but most of the time it doesn't matter, as the bottleneck is often in python and PHP, or the bottleneck is solved using caching in the web server.

[arQon]

Not everything need the most performant IPC though.

* Bangs head on desk...

Yes, I know! Hence "we ended up adopting CORBA and DCOM even for purely in-house components that were running on the same machine and could have just used native calling instead".

> Even with io-uring, it will still have a large overhead

Then it's not "even with" - you can't choose to use the wrong tool for the job and then blame the tool for not being the right one...

I think we all understand that there are fast ways, slow ways, easy ways, and hard ways of doing IPC, and that what matters is picking the appropriate one for the task at hand. Let's get back on topic.

[arQon]

Still, it seems kind of ridiculous to imagine that every application that wants to use more than one language is going to be following in Firefox and Chrome's footsteps in spraying itself across two or more processes, purely because nobody can agree on an in-process IDL.

That's certainly undesirable, yes - but even in the worst case scenario, once you HAVE that marshalling layer there's nothing keeping your two languages from running in the same process, as long as you have ANY way of making a single call with just a couple of well-defined types. And we seem to have managed that pretty well so far, because those types ARE well-defined.
The Chrome/etc mess isn't about IPC, it's about complex (and highly clusterfucked) code constantly tripping over its own feet. There IS an in-process IDL, and it's not one that "nobody can agree on", it's one that everyone already *has*: it's "whatever the underlying native types of the CPU are". In every non-worst case, the native types of a language will be the same as C's, and there's no need for translation, because that's what the CPU is using. IDLs are great, but they're overkill for your case, as you know - and, I suspect, overkill for almost every other case you can imagine, because that's just not how the world works. (Even what actually *was* the worst possible case, during the nightmares of the 16- to 32-bit transition, was resolved trivially (albeit not prettily :P) by thunking).

I think it's fair to say that "over-engineering" for such scenarios always turns out to be masturbatory, driven by an unrealistic appraisal of either the value of the code or the effort needed to deal with the problem IF (not "when") it actually happens. That's not to say no project ever does it, but it's out of ego, not need.
If some language chooses to e.g. represent ints as strings internally, and is incapable of turning them into the machine's register type, that's a problem with that language, not every other language in history, and that language will sink into oblivion as it deserves. I don't think you have to worry about it.

[ssokolow]

That's certainly undesirable, yes - but even in the worst case scenario, once you HAVE that marshalling layer there's nothing keeping your two languages from running in the same process, as long as you have ANY way of making a single call with just a couple of well-defined types. And we seem to have managed that pretty well so far, because those types ARE well-defined.
The Chrome/etc mess isn't about IPC, it's about complex (and highly clusterfucked) code constantly tripping over its own feet. There IS an in-process IDL, and it's not one that "nobody can agree on", it's one that everyone already *has*: it's "whatever the underlying native types of the CPU are". In every non-worst case, the native types of a language will be the same as C's, and there's no need for translation, because that's what the CPU is using. IDLs are great, but they're overkill for your case, as you know - and, I suspect, overkill for almost every other case you can imagine, because that's just not how the world works. (Even what actually *was* the worst possible case, during the nightmares of the 16- to 32-bit transition, was resolved trivially (albeit not prettily :P) by thunking).

I think it's fair to say that "over-engineering" for such scenarios always turns out to be masturbatory, driven by an unrealistic appraisal of either the value of the code or the effort needed to deal with the problem IF (not "when") it actually happens. That's not to say no project ever does it, but it's out of ego, not need.
If some language chooses to e.g. represent ints as strings internally, and is incapable of turning them into the machine's register type, that's a problem with that language, not every other language in history, and that language will sink into oblivion as it deserves. I don't think you have to worry about it.

The problem that https://gankra.github.io/blah/c-isnt-a-language/ pointed out is that there's a lot that isn't defined by the machine types... to the point where, despite 128-bit integers on x86_64 being defined by the AMD64 SysV ABI document, GCC and LLVM implement them differently.

Let alone all the other things about "C ABI" that people just assume are standardized in some fashion, rather than being "delegate to libclang and hope you haven't tripped over one of the areas where it hasn't yet managed to match the relevant GCC/MSVC implementation details".

For example, what's the calling convention? Well, it depends on the target triple. What about a "word-sized integer"? Well, if you're talking to a language that delegates to C and you're on most 64-bit platforms, you're talking to a language that piggy-backed on the decision to keep int 32-bit for ease of porting existing code.

What's being called for is a concerted effort to move away from "do what C does on the given platform" to a C-independent standard for "When we say argument 0 is a machine integer, we mean passing such and such a 64-bit integer in register X and it's the callee's responsibility to preserve the contents of registers Y and Z if they're used by the callee. As for not-quite-primitives everyone uses like strings, they're laid out like this..."