Python 3.11 Performance Benchmarks Show Huge Improvement

[sinepgib]

you are right, I forgot that, but I cannot find the RwLock in std lib https://docs.python.org/3/library/threading.html

My memory failed me. While mutex is part of the stdlib, the read write lock I used was part of a PyPI package.

[NobodyXu]

Python already has mutex and rwlock in its standard library IIRC. That's because the GIL only prevents races at the individual opcode level, not complex objects.

you are right, I forgot that, but I cannot find the RwLock in std lib https://docs.python.org/3/library/threading.html

[sinepgib]

I was mainly thinking about locking.
IMHO if GIL is removed, then having some language construct for locking might be desired, probably because I come from compiled PL like Rust and is used to explicitly add locking.
I have not looked too much into IronPython or Jython, though I do hear that they do not have a GIL due to different API and JIT.

Python already has mutex and rwlock in its standard library IIRC. That's because the GIL only prevents races at the individual opcode level, not complex objects.

[NobodyXu]

[*]Because neither compile() nor eval() are in the C/C++ standards, it is possible that a high-performance 8-bit emulator could run faster in plain CPython than in plain C/C++. The term "plain C/C++" means that the main part of the emulator isn't using non-standard C/C++ features (such as: libgccjit.so).

Does it matter?
You can write a C/C++/Rust interpreter or a JIT compiler that compiles C/C++/Rust to some internal bitcode or to native code and run them anyway.

The "compile()" function in python also calls into the python interpreter written in C, it's not written in Python.

[NobodyXu]

It isn't necessary because it is possible to search the space for near-optimal solutions without human intervention during the search.

And in order for the python program to search the space, you would have to write the algorithm first before the program is started, which can also be done in AOT PL.

A trivial example would be the CPython interpreter itself.
Since it accepts almost all valid python code and is written in C, an AOT pl, it can be considered as a C program that takes file "*.py" as data.
Since python is interpreted by the C program, anything can be done by the python must be able to be done by C.

[marios]

[*]Because neither compile() nor eval() are in the C/C++ standards, it is possible that a high-performance 8-bit emulator could run faster in plain CPython than in plain C/C++. The term "plain C/C++" means that the main part of the emulator isn't using non-standard C/C++ features (such as: libgccjit.so).

What are you talking about? libgccjit.so is neither a C nor C++ feature. It is a shared object accompanied by some .h files just like every C dynamic library. And the ability to use libraries is definitely part of both C and C++ standard. And code that includes those .h files and uses the functionality is perfectly compliant to the standard. The fact that it is not built in the syntax of the language is good practice.

[NobodyXu]

None of those are visible to pure Python code. In fact, aren't some of the alternative interpreters GIL-less? I may be confused, but I think IronPython or Jython don't have a GIL, and pure Python is supposed to run fine. Maybe we mean different things with "language construct"? I interpret a syntax/semantic change at the Python language level (i.e., I'm not counting changes to native extensions, those will surely break and are not always compatible between implementations).

I was mainly thinking about locking.
IMHO if GIL is removed, then having some language construct for locking might be desired, probably because I come from compiled PL like Rust and is used to explicitly add locking.
I have not looked too much into IronPython or Jython, though I do hear that they do not have a GIL due to different API and JIT.

[sinepgib]

Yeah you are right.
Removing the GIL would certainly break the C API, reference counting/change to GC and requires to add lock somehow, thus probably requires new language construct.

None of those are visible to pure Python code. In fact, aren't some of the alternative interpreters GIL-less? I may be confused, but I think IronPython or Jython don't have a GIL, and pure Python is supposed to run fine. Maybe we mean different things with "language construct"? I interpret a syntax/semantic change at the Python language level (i.e., I'm not counting changes to native extensions, those will surely break and are not always compatible between implementations).

[NobodyXu]

If you implement an emulator of an 8-bit machine (such as: Atari 130XE) and compile the emulator using an AOT compiler, then it is mathematically impossible for the AOT compiler to account for the fact that you loaded a a particular game (such as: Bruce Lee) into the 8-bit emulator, especially if it is a game that was developed after the file "Atari130XE.exe" was created by the AOT compiler. ---- If the emulator was implemented in Python (or some other language capable of compiling and loading arbitrary code at run-time) then the emulator could translate selected parts of the currently loaded 8-bit machine-code directly into Python bytecode.

In order for that python program to account for that "Atari130XE.exe" and specialize for it, you must implement the python code requires for identifying that program and add that specialized treatment when you are writing that python program.
Which means that you already put the information into the program before it even starts running.

Then, what's the problem with doing the same thing with AOT program?
You already put the information into the python program prior to running it, you can absolutely do the same thing in AOT program.

You see, AOT is not inferior to dynamic language, the later does not give you more advantages when it comes to such optimization or the general ease-of-use since every case would still needs to be coded AOT before the program is up and running.

[NobodyXu]

Yes, it's not there by accident after all. Regarding requirements, I don't think it requires JIT nor changing the high level part of the language. It will probably require breaking the C API to some degree, and it will either require more sophisticated techniques for reference counting, a hit for simply taking locks or promoting the GC to be the only memory management mechanism.

Yeah you are right.
Removing the GIL would certainly break the C API, reference counting/change to GC and requires to add lock somehow, thus probably requires new language construct.