Rust GCC Code Generator "rustc_codegen_gcc" Can Now Bootstrap Rustc

[oiaohm]

There are two fixes for this using wasm:
- We can simply compile all the languages to wasm
- We can embed the wasm runtime (interpreter/jvm) into the application

The first option is supported by C/C++, C#/. NET, Rust, Java, Python, Elixir, Go
The second option is supported by all languages which supports C/C++ API/ABI and Rust.


Is there any tool to compile C/C++/Rust/Java/Go to CLR?

Edit:

It looks like C++ can be compiled to CLR by using microsft's compiler:

The reality here you don't get how much what you just wrote is exactly the same as .net. You need to remember .net push was done before rust and go existed. Yes Microsoft did C/C++ and java compilers for .net.

https://docs.microsoft.com/en-us/arc...-native-c-code Yes back in 2005 here is embedding .net interpreter in any application that supports C/C++ interface.

But that is still no where near the support of compiling C++ to wasm.

And notice that it can only compile "A version of C++" to CLR, while clang can literally compile any valid C++ code to wasm.

What protection does the languages lost when compiled to wasm?

You are wrong that all C++ code can built to WASM. Microsoft C++ could only do a restricted set to CLR due to enforcing foreign function interface standard.

C and C++ does not have a valid code safety. Also some code that builds with gcc does not build with clang the 128bit integer differences and others. Emscripten allows you to use gcc-llvm and clang for C and C++ to get to wasm yes the clang and gcc issue of using different format 128bit integer differences and others raise their head.

The reality is ssokolow idea that wasm fixes foreign function interface is wrong. The same problems with jvm, .net, parrot and others. Yes there you end up restricting the language due to mandating foreign function interface leading to miss match problems(.net/java) or you don't restrict it so having incompatibilities yes WASM is have incompadiblities(wasm/parrot and others).

Programming Languages are not in agreement on lots of things and implements like clang vs gcc are not in agreement on lots of things. Rust is having to have a gcc backend so that rust is compatible with gcc built C. I do expect wasm to fracture same ways that when mixing languages/implementations compilers you will have issues.

They could definitely do better on that one, maybe providing a new mode similar to pointer tagging in ARM (sorry I forgot the name), which ises 128 bit to store address, where 64 bit is used to encode a tag?

https://developer.arm.com/-/media/Arm%20Developer%20Community/PDF/Arm_Memory_Tagging_Extension_Whitepaper.pdf

The ARM MTE you are kind of referencing to is not a bullet proof as one would hope. ARM MTE it still possible to buffer overflow as to allocations of memory next to each other could end up with the same tag.

Yes ARM MTE has cost 4 bits of extra memory for every 16 bytes of memory. Yes 16 possible values for the key collisions are absolutely likely.

We are not yet seeing a proper fix to these problems in silicon MTE is improvement over nothing but its not the fix.

No arm did not end up using 128bit address. Different mainframes that uses 128 bit address for proteciton. One is first 64 bits does not change when the memory is allocated and contains the end of the allocation. Yes this is simple when cpu progress the 128 bit address it first checks that the second half is not less than the first half by at least 8 byte then using the 8 byte end value stored at the first half location to validate that the address in the second half is not pass that. Yes this has a performance cost for mainframe users where errors are really bad this is not a problem. This also provides that metadata about a value can be aligned before the start. The mainframe one prevents buffer overflows and buffer overreads .

Next there is pointer and offset again pointer points to allocation size. Then there allocation table-pointer. Yes these are all have a 128 bit address split it in two. First halve pointing to something to validate the pointer operation.

There are ways to make CPU instructions not allow buffer overflow problems. All the ones that absolutely work have a performance cost. MTE and other tagging like it is the trade off of safety for performance as in less protection from buffer overflows but still better than nothing yet has very low performance overhead due to no add/subtract and no having to access other memory.

I have seen this problem enough times its a lot harder to sell me a magical fix. Lot of ways I see the only fix is that major CPU decide to include instructions for pointers and storage that allow extra meta data that is enforceable by the CPU then programming languages writers will have to alter to make their languages more suitable.

[NobodyXu]

You are wrong that all C++ code can built to WASM. Microsoft C++ could only do a restricted set to CLR due to enforcing foreign function interface standard.

C and C++ does not have a valid code safety. Also some code that builds with gcc does not build with clang the 128bit integer differences and others. Emscripten allows you to use gcc-llvm and clang for C and C++ to get to wasm yes the clang and gcc issue of using different format 128bit integer differences and others raise their head.

The reality is ssokolow idea that wasm fixes foreign function interface is wrong. The same problems with jvm, .net, parrot and others. Yes there you end up restricting the language due to mandating foreign function interface leading to miss match problems(.net/java) or you don't restrict it so having incompatibilities yes WASM is have incompadiblities(wasm/parrot and others).

Programming Languages are not in agreement on lots of things and implements like clang vs gcc are not in agreement on lots of things. Rust is having to have a gcc backend so that rust is compatible with gcc built C. I do expect wasm to fracture same ways that when mixing languages/implementations compilers you will have issues.

If wasm provides its own standardized u128 implementation, provide a way to convert it to clang/gcc's u128, and then requires all interface defined in wasm to use it, then the problem is solved for wasm.

https://developer.arm.com/-/media/Arm%20Developer%20Community/PDF/Arm_Memory_Tagging_Extension_Whitepaper.pdf

The ARM MTE you are kind of referencing to is not a bullet proof as one would hope. ARM MTE it still possible to buffer overflow as to allocations of memory next to each other could end up with the same tag.

Yes ARM MTE has cost 4 bits of extra memory for every 16 bytes of memory. Yes 16 possible values for the key collisions are absolutely likely.

We are not yet seeing a proper fix to these problems in silicon MTE is improvement over nothing but its not the fix.

No arm did not end up using 128bit address. Different mainframes that uses 128 bit address for proteciton. One is first 64 bits does not change when the memory is allocated and contains the end of the allocation. Yes this is simple when cpu progress the 128 bit address it first checks that the second half is not less than the first half by at least 8 byte then using the 8 byte end value stored at the first half location to validate that the address in the second half is not pass that. Yes this has a performance cost for mainframe users where errors are really bad this is not a problem. This also provides that metadata about a value can be aligned before the start. The mainframe one prevents buffer overflows and buffer overreads .

Next there is pointer and offset again pointer points to allocation size. Then there allocation table-pointer. Yes these are all have a 128 bit address split it in two. First halve pointing to something to validate the pointer operation.

There are ways to make CPU instructions not allow buffer overflow problems. All the ones that absolutely work have a performance cost. MTE and other tagging like it is the trade off of safety for performance as in less protection from buffer overflows but still better than nothing yet has very low performance overhead due to no add/subtract and no having to access other memory.

I have seen this problem enough times its a lot harder to sell me a magical fix. Lot of ways I see the only fix is that major CPU decide to include instructions for pointers and storage that allow extra meta data that is enforceable by the CPU then programming languages writers will have to alter to make their languages more suitable.

I am actually referrening to something like this https://www.cl.cam.ac.uk/research/se...cheri-faq.html which provides pointer provenance.

I know that does not and cannot rule out all memory related bugs out there, but it will certainly help identify bugs.

[oiaohm]

If wasm provides its own standardized u128 implementation, provide a way to convert it to clang/gcc's u128, and then requires all interface defined in wasm to use it, then the problem is solved for wasm.

I said cursed for a reason. The difference is a binary storage difference between clang and gcc that can and does make it way all the way out to disc. So you have file incompatibility caused by the compiler. Yes wasm add own standard effectively risks just creating another implementation of u128 with different incompatibilities.

I am actually referrening to something like this https://www.cl.cam.ac.uk/research/se...cheri-faq.html which provides pointer provenance.

I know that does not and cannot rule out all memory related bugs out there, but it will certainly help identify bugs.

That one is to meet USA Mil requirements for mainframe implementations in particular use cases. That is the independent register solution that is very nice. Where all registers have a second register that points to metadata somewhere in memory.
Some of the existing mainframes exceed USA Mil the metadata containing 3 values instead of 2:
1) start of the storage space.
2) end of the storage space.
3) type of the storage.

This was a 32 bit system not a 64 bit one so those were all 32 bit values. 32 bit worth of unique values is a lot for types. Yes type casting is possible inside this system due to the fact metadata can overlap start and end of storage values with each other so you can have two or more metadata for the same memory.

Yes the mainframe chips with this could throw a type error. This does not rule out all memory related bugs but does prevent a lot and that is seen in real world implementations. The mainframe version meta data prevents buffer overflows and buffer over reads and lot of type issues there are still memory issues yes but way less common because the CPU simply does not allow as many memory issues. The risc-v and mips implementation funded by darpa is in fact a cut down implementation of what has been in different mainframes.

I do note that this Cheri implementation uses tagged memory as fall back the mainframe implementations for USA mil in the use cases requiring this no tagged memory fall back it and it used the metadata on memory or nothing works. I do mean nothing works once the system gets to a particular point in the boot if metadata is not setup the mainframe stops.

What ARM is offering at this stage is very weak and what x86 offers is insanely weak against memory issues. Yes something like cheri for mips and risc-v is serous CPU solution to the memory issues. Of course cheri still behind compared to the best mainframe solutions that contain type as well.

This is why I view the problem with C and programming languages being deeper in. C on mainframe US MIl for particular usages buffer overflows basically don't happen the metadata prevents 99.999%+ of the time. Most common issue is buffer underrun/underwrite on those mainframe systems this normally a locking issue.

For software security and stability the most common used CPU ISA and implementations are horrible bad. The scary point the is the first time metadata was done on memory in mainframes for military usage is 1995 so we are over 25 years late to the party. Yes intel and amd is doing a lot to protect virtual machines by encrypted memory but they have not done a lot fix the buffer overflow problems or the data type problem that does effect basically all users with software issues.

Fixed at the ISA is where we really need this fixed so that it can be hardware optimised so it has the least performance overhead. There have been enough failed attempts to fix it by software.

[NobodyXu]

I said cursed for a reason. The difference is a binary storage difference between clang and gcc that can and does make it way all the way out to disc. So you have file incompatibility caused by the compiler. Yes wasm add own standard effectively risks just creating another implementation of u128 with different incompatibilities.

Yes, the situation is indeed hard to fix unless one party of GCC/LLVM is willing to change their implementation to match other, or come up with a unified implementation.

This makes me wonder if MSVC also have different u128 implementation compared to GCC/LLVM.

IMHO in this situation, if wasm wants to support u128, it will have to create its own implementation as this is the most simple method to archive portability and stability in the interface, as otherwise they depends on specific compiler implementation.

Or, if x86_64 and arm comes up with hardware support for 128-bit integer, then this will be resolved for the newly released chips.

That one is to meet USA Mil requirements for mainframe implementations in particular use cases. That is the independent register solution that is very nice. Where all registers have a second register that points to metadata somewhere in memory.
Some of the existing mainframes exceed USA Mil the metadata containing 3 values instead of 2:
1) start of the storage space.
2) end of the storage space.
3) type of the storage.

This was a 32 bit system not a 64 bit one so those were all 32 bit values. 32 bit worth of unique values is a lot for types. Yes type casting is possible inside this system due to the fact metadata can overlap start and end of storage values with each other so you can have two or more metadata for the same memory.

Yes the mainframe chips with this could throw a type error. This does not rule out all memory related bugs but does prevent a lot and that is seen in real world implementations. The mainframe version meta data prevents buffer overflows and buffer over reads and lot of type issues there are still memory issues yes but way less common because the CPU simply does not allow as many memory issues. The risc-v and mips implementation funded by darpa is in fact a cut down implementation of what has been in different mainframes.

I do note that this Cheri implementation uses tagged memory as fall back the mainframe implementations for USA mil in the use cases requiring this no tagged memory fall back it and it used the metadata on memory or nothing works. I do mean nothing works once the system gets to a particular point in the boot if metadata is not setup the mainframe stops.

What ARM is offering at this stage is very weak and what x86 offers is insanely weak against memory issues. Yes something like cheri for mips and risc-v is serous CPU solution to the memory issues. Of course cheri still behind compared to the best mainframe solutions that contain type as well.

This is why I view the problem with C and programming languages being deeper in. C on mainframe US MIl for particular usages buffer overflows basically don't happen the metadata prevents 99.999%+ of the time. Most common issue is buffer underrun/underwrite on those mainframe systems this normally a locking issue.

For software security and stability the most common used CPU ISA and implementations are horrible bad. The scary point the is the first time metadata was done on memory in mainframes for military usage is 1995 so we are over 25 years late to the party. Yes intel and amd is doing a lot to protect virtual machines by encrypted memory but they have not done a lot fix the buffer overflow problems or the data type problem that does effect basically all users with software issues.

Fixed at the ISA is where we really need this fixed so that it can be hardware optimised so it has the least performance overhead. There have been enough failed attempts to fix it by software.

This sounds even more amazing than what I heard of pointer proverance on arm.

The problem with this is the added overhead, the pointer requires more than 2x space.

Maybe we can start with simple fix, like using fat pointer to store C pointers, just like how Rust does it, though Rust does not use fat pointer to store pointers to struct and arrays which size are known at compile time, only for slice and unsized struct.
This would already eliminates some overflow bugs.

IMHO it will be best to do these memory related checking at compile time, like Rust does, and only add runtime checking for parts cannot be checked at compile time, like the FFI boundary, syscall, or calling into assembly indirectly.

[oiaohm]

Yes, the situation is indeed hard to fix unless one party of GCC/LLVM is willing to change their implementation to match other, or come up with a unified implementation.

This makes me wonder if MSVC also have different u128 implementation compared to GCC/LLVM.

IMHO in this situation, if wasm wants to support u128, it will have to create its own implementation as this is the most simple method to archive portability and stability in the interface, as otherwise they depends on specific compiler implementation.

Or, if x86_64 and arm comes up with hardware support for 128-bit integer, then this will be resolved for the newly released chips.

Microsoft solution with MSVC so far is simply don't support using hardware 128bit integer at all. If programmer wants to use this they have to write assembly third party bit of software. Yes the include assembler with MSVC does not contain the instructions to use x86 hardware 128bit integer in any form.

https://docs.microsoft.com/en-us/cpp...?view=msvc-170 Microsoft solution to 128 bit integer is used 2 64 bit integers.

This 128bit issue is why number of programs have been ported to gcc or clang from MSVC because you get to remove a stack of big ridden assembly code and use a lot less complex built process. Its just a horrible that gcc and clang are not using the same implementation or close enough it does not matter.

This sounds even more amazing than what I heard of pointer proverance on arm.

The problem with this is the added overhead, the pointer requires more than 2x space.

Maybe we can start with simple fix, like using fat pointer to store C pointers, just like how Rust does it, though Rust does not use fat pointer to store pointers to struct and arrays which size are known at compile time, only for slice and unsized struct.
This would already eliminates some overflow bugs.

IMHO it will be best to do these memory related checking at compile time, like Rust does, and only add runtime checking for parts cannot be checked at compile time, like the FFI boundary, syscall, or calling into assembly indirectly.

Check at the compile time means you have to have all the compilers on the same page or use only 1 compiler and possibly only one language. Remember you are wanting FFI boundary at times with third party provided binaries that could be built with different compilers. This is not a easy problem.

Mainframe usage for likes weapon systems being correct is more import than performance. Really for reducing of buffer overflows and over read nothing beats the extra meta data solution just a pitty it has such a high performance cost and memory cost. We do need to be benchmarking the proposed solutions against the best of the mainframe world has done.

Risc-v we could see some of the mainframe class memory protections in time in more affordable hardware.

The most insane mainframes stablity have what is called multi execution this is where a process is run 4 times on different cores and the result has to match that has a performance overhead that makes this metadata stuff look like nothing. Yes those systems you can pull a complete cpu module out and the system does not skip a beat because the CPU are basically old school raid.

[NobodyXu]

Microsoft solution with MSVC so far is simply don't support using hardware 128bit integer at all. If programmer wants to use this they have to write assembly third party bit of software. Yes the include assembler with MSVC does not contain the instructions to use x86 hardware 128bit integer in any form.

https://docs.microsoft.com/en-us/cpp...?view=msvc-170 Microsoft solution to 128 bit integer is used 2 64 bit integers.

This 128bit issue is why number of programs have been ported to gcc or clang from MSVC because you get to remove a stack of big ridden assembly code and use a lot less complex built process. Its just a horrible that gcc and clang are not using the same implementation or close enough it does not matter.

Not implementing `u128` at all seems like a feasible way to solve the FFI problems in wasm, as it is an extension to C, not something that is guaranteed.

Check at the compile time means you have to have all the compilers on the same page or use only 1 compiler and possibly only one language. Remember you are wanting FFI boundary at times with third party provided binaries that could be built with different compilers. This is not a easy problem.

I know this is difficult when FFI is involved, that's why I think having hardware support for checking use of pointer accross FFI will be great, combined use of memory-safe languages like Rust.

Mainframe usage for likes weapon systems being correct is more import than performance. Really for reducing of buffer overflows and over read nothing beats the extra meta data solution just a pitty it has such a high performance cost and memory cost. We do need to be benchmarking the proposed solutions against the best of the mainframe world has done.

Risc-v we could see some of the mainframe class memory protections in time in more affordable hardware.

The most insane mainframes stablity have what is called multi execution this is where a process is run 4 times on different cores and the result has to match that has a performance overhead that makes this metadata stuff look like nothing. Yes those systems you can pull a complete cpu module out and the system does not skip a beat because the CPU are basically old school raid.

I heard that back in the days, the rockets launched by NASA does the same calculation by 4 times, or something like that.

I suspected that they also use this kind of CPU?

[oiaohm]

I heard that back in the days, the rockets launched by NASA does the same calculation by 4 times, or something like that.

I suspected that they also use this kind of CPU?

When you don't have enough shielding against radiation run the process multi times comes required.

https://www.researchgate.net/figure/Triple-modular-redundant-TMR-architecture_fig3_2891855

You also find multi times in cars and heavy machinery. Triple is more common than quad. The quad I wrote as insane because generally it makes very little difference NASA normally uses Triple for core critical and that goes back to day one. Yes they all use a majority rules model. The quad is so that you can remove a core and still be running in triple yes mainframe N+1 as in the N you need +1 spare gets you to quad.

Yes this is a very different form of computing because it insane to a point.

Not implementing `u128` at all seems like a feasible way to solve the FFI problems in wasm, as it is an extension to C, not something that is guaranteed.

Remember this has just cursed you self by this choice the same way MSVC is that is resulting in developers migrating away from MSVC. This is really just cursed there is no correct answer that does not get you one way or the other. u128 its not the end of the story there is 128 bit signed. Yes depending on how you implement 128bit in arm or x86 simd instructions the signed bit for 128bit can be in different places. To be exact here you are implementing on arm you can do 128bit signed I think 8 different ways and x86 about 4 in the simd and that using the native machine code. This problem is not that its extension to C its that at the cpu instruction set level there is not agreement.

The correct answer might be stuff it we will have multi types for 128bit signed and unsigned because of lack of agreement at the machine level. Yes of course those making compilers and the like don't want to have to do 128bit integer code paths 4 to 8 times. Remember it does not end at 128bit there is 256 and 512 integer support coming in x86 and arm hardware welcome to the next level of failure to agree.

This is why I say the problem deeper in than C the problems are truly coming up from the cpu instruction sets and cpu implementations resulting in everyone having hell.

[NobodyXu]

When you don't have enough shielding against radiation run the process multi times comes required.

https://www.researchgate.net/figure/Triple-modular-redundant-TMR-architecture_fig3_2891855

You also find multi times in cars and heavy machinery. Triple is more common than quad. The quad I wrote as insane because generally it makes very little difference NASA normally uses Triple for core critical and that goes back to day one. Yes they all use a majority rules model. The quad is so that you can remove a core and still be running in triple yes mainframe N+1 as in the N you need +1 spare gets you to quad.

Yes this is a very different form of computing because it insane to a point.

Yeah, the space is a harsh place for human beings and electronics alike.

I heard that SpaceX uses multiple commodity computer to archive the same thing though.

Remember this has just cursed you self by this choice the same way MSVC is that is resulting in developers migrating away from MSVC. This is really just cursed there is no correct answer that does not get you one way or the other. u128 its not the end of the story there is 128 bit signed. Yes depending on how you implement 128bit in arm or x86 simd instructions the signed bit for 128bit can be in different places. To be exact here you are implementing on arm you can do 128bit signed I think 8 different ways and x86 about 4 in the simd and that using the native machine code. This problem is not that its extension to C its that at the cpu instruction set level there is not agreement.

The correct answer might be stuff it we will have multi types for 128bit signed and unsigned because of lack of agreement at the machine level. Yes of course those making compilers and the like don't want to have to do 128bit integer code paths 4 to 8 times. Remember it does not end at 128bit there is 256 and 512 integer support coming in x86 and arm hardware welcome to the next level of failure to agree.

This is why I say the problem deeper in than C the problems are truly coming up from the cpu instruction sets and cpu implementations resulting in everyone having hell.

Well, that's true that developers could move away because of that, but I don't think use of 128-bit integer is that common.

The problem of 128-bit integer is indeed partially because of the underlying CPU instruction, but also because GCC and LLVM refuses to collaborate.

[oiaohm]

Yeah, the space is a harsh place for human beings and electronics alike.

I heard that SpaceX uses multiple commodity computer to archive the same thing though.

Yes that still meets runs the operations 3 times compare the results and take the 2 that match as correct.

Well, that's true that developers could move away because of that, but I don't think use of 128-bit integer is that common.

The problem of 128-bit integer is indeed partially because of the underlying CPU instruction, but also because GCC and LLVM refuses to collaborate.

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60846 The 128-bit integer problems is horrible common the most common 128-bit unsigned integer required is called ipv6.

Its not just a failure of Gcc and LLVM to agree it a failure of C and C++ standards to include a define for 128bit integer as well. This ipv6 problem was known to be coming from 1995. Yes network packet stuff for games, voip... can be very time sensitive. I cannot remember the common signed 128bit interger but there is something else horrible common that need 128bit signed that is time sensitive.

So yes this is a problem that had over 2 decades heads up that it was coming with 128bit unsigned integers. This is kind of way MSVC avoid the problem does not really work and instead just forces programmers to write in assembly for internet packet handling and other things that need 128 bit integers with all the extra security risks that brings.

There are particular use cases for 256 bit as well but these are not as common and at this stage not as time sensitive. 128 bit integer problems are truly cursed with ipv6 and others you in fact do need 128-bit integer support so avoid the problem is not really option. LLVM and gcc developers having a disagreement over what way results in the best performance to implement 128-bit integer happens because its a time sensitive problem.

Yes there are many programming languages that have not defined 128-bit integer support as well. Welcome to next level of cursed.

[NobodyXu]

The 128-bit integer problems is horrible common the most common 128-bit unsigned integer required is called ipv6.

This is not true.

Ipv6 address is not expressed as a 128-bit integer, thus it is not a problem here.

Looking athttps://man7.org/linux/man-pages/man7/ipv6.7.html, the ipv6 address is expressed using `struct in6_addr`, further investigation into the kernel API https://docs.huihoo.com/doxygen/linu...in6__addr.html, it shows that the Linux kernel treats it as a union, it can be interpreted as 16 u8, or 8 u16, or 4 u32.

When looking at the source code of glibc implementation of `inet_pton6` https://code.woboq.org/userspace/gli...tml#inet_pton6, a function to convert IPv4 and IPv6 addresses from text to binary form, it does not use any 128-bit integer at all.

In fact, the RFC for ipv6 does not involve 128-bit integer at all.

An ipv6 address is defined as:

An IPv6 address is represented as eight groups of four hexadecimal digits,
each group representing 16 bits.

The groups are separated by colons.

The ipv6 address is essentially 8 `u16` integers that are encoded separately, so there is absolutely no 128-bit integer involved for ipv6 socket.