Pyston Now 95% Faster Than CPython, But Dropbox Just Stopped Supporting It

[nevion]

Solved? You mean like memory leaks are just fixed now? Or errno vs, overloading a return value to indicate an error? And intermediate level developers never return stack values in function results, the compilers fixed that? No more buffer overflows? C's problems are the very nature of C, you can't "solve" them. Dropbox would probably have never been if they used C.

C in isolation from it's problems, which is fairly possible given an approach similar to either Cython or Pyinline (or something with strict boundaries like OpenCL, which places some restrictions on what you can do) minimizes those issues. There are other ways of wrangling the C-beast in - the alternative is time after time failure.

[ldo17]

You mean like memory leaks are just fixed now?

There are ways to minimize the chance of both memory leaks and double-frees in C code, without sacrificing its sheer speed. It all comes down to structured programming and avoiding gotos.

The example I like to offer is this one. You will see the same kind of structuring technique used throughout. But the point is really made in the most complex sections, with dynamic allocation happening inside a loop, like that beginning at line 477: note how you can determine just by inspecting the code, that all control paths will correctly manage memory, no matter what errors might occur.

[microcode]

Solved? You mean like memory leaks are just fixed now? Or errno vs, overloading a return value to indicate an error? And intermediate level developers never return stack values in function results, the compilers fixed that? No more buffer overflows? C's problems are the very nature of C, you can't "solve" them. Dropbox would probably have never been if they used C.

This is kind of interesting event, they built their own compiler to speed things up, it makes benchmarks run twice as quick and it makes their workload only 10% faster. I doubt they needed to build a compiler to know it, but they have an architecture problem. They're going to rebuild a lot of stuff no matter what they do if they choose to address it. Switching to Go seems like a very sane and rational thing to do. It's much safer than C, it performs in that Java to C level, it's popular and people are writing it with a quickness. There is probably a culture within the company that makes discussing such architectural flaws difficult. Plus, customers aren't complaining about performance, it's a margin issue which just suggests what everyone already knows, dropbox is getting close to a liquidity event and they think they can make it spectacular if they pick up a few points on ops expenses..

I mean that tooling can get you out of C's problems. Memory leaks can be solved by using valgrind or a malloc debugger, or by using a garbage collector, or carefully deallocating heap at the end of your scope. Buffer overflows can be eliminated by using bounded buffer descriptors, or known-length buffers. Regarding errno, it is an important design choice, since errno requires no allocation to function. It can function correctly if your stack is corrupted, or if you're out of memory (important for reporting out-of-memory errors). I understand that go's solution (tuples in return) works fine, but it also makes prototyping harder since you have to separate out that value even if you're just trying something and know it won't fail.
Returning stack values in functions is fine as long as they fit in the return type. I'm guessing you mean pointers to stack values?

Sure, maybe not 100% of things are exactly as easy, but at least the solutions are established. In Go, if you can't make your program agree with the garbage collector (no matter how excellent it is) then there is no escape.

[Sonadow]

Seems from the FAQ that Pyston was only for Python 2 anyway.

Anything that still tries to build on Python 2 is a waste of time.

Firefox can only be built on Python 2.

So Firefox is a waste of time.

[nevion]

Firefox can only be built on Python 2.

So Firefox is a waste of time.

Firefox isn't a python runtime or library? Apples and bananas?

[ssokolow]

It seems I'm not the only one who thinks this way mate. I wrote about grumpy before you said anything and your worry validates my own concern. It was the first thing that came in my mind when I read about their decision.

I guess we can see things the way it should.

Well done mate

For me, PyQt5 and Django (especially the schema migration support and the ecosystem of reusable apps) are Python's killer apps, both of which support Python 3. (In fact, *buntu 14.04 only offers PyQt5 packages for Python 3)

For everything else, I've been aiming to move to Rust as a happy medium between Haskell's powerful type system and the easier-to-reason-about performance of imperative languages.

[Nelson]

I mean that tooling can get you out of C's problems. Memory leaks can be solved by using valgrind or a malloc debugger, or by using a garbage collector, or carefully deallocating heap at the end of your scope. Buffer overflows can be eliminated by using bounded buffer descriptors, or known-length buffers. Regarding errno, it is an important design choice, since errno requires no allocation to function. It can function correctly if your stack is corrupted, or if you're out of memory (important for reporting out-of-memory errors). I understand that go's solution (tuples in return) works fine, but it also makes prototyping harder since you have to separate out that value even if you're just trying something and know it won't fail.
Returning stack values in functions is fine as long as they fit in the return type. I'm guessing you mean pointers to stack values?

Sure, maybe not 100% of things are exactly as easy, but at least the solutions are established. In Go, if you can't make your program agree with the garbage collector (no matter how excellent it is) then there is no escape.

Valgrind and malloc debugging doesn't fix memory leaks, they help you identify them, so long as you build for it, build unit tests and such. It's nearly impossible to valgrind a really large system. The problem here is dramatically more fundamental and their post already told the story. They aren't careful coders, they built a unicorn company running hard and fast. So much so that it made logical sense to attempt to build a new python compiler to try and squeeze some performance out. How often is the solution to attempt to build a new compiler? Effort and energy have been spent on it too, it's not the first new python compiler... Sure, if you write perfect code and build great architecture every time, you can't go wrong with C, but they didn't and they don't. I'll say it again, they wouldn't exist anymore had they initially chosen C, they aren't the right kind of people.


Can you demonstrate some cases where you can't get a long with the Go GC? Ideally something similar to what dropbox is doing. They aren't poking registers, they aren't bit banging real-time protocols, there stuff works as it is in Python, it's just not fast enough for their finance guys.

[ldo17]

There seems to be this implicit assumption that using a GC-based language somehow does away with memory leaks. It doesn’t.

[caligula]

There seems to be this implicit assumption that using a GC-based language somehow does away with memory leaks. It doesn’t.

The problem is more common in mediocre imperative languages like Java where you use static variables. Purely functional languages don't use static variables that much.

[caligula]

I mean that tooling can get you out of C's problems. Memory leaks can be solved by using valgrind or a malloc debugger, or by using a garbage collector, or carefully deallocating heap at the end of your scope. Buffer overflows can be eliminated by using bounded buffer descriptors, or known-length buffers. Regarding errno, it is an important design choice, since errno requires no allocation to function. It can function correctly if your stack is corrupted, or if you're out of memory (important for reporting out-of-memory errors). I understand that go's solution (tuples in return) works fine, but it also makes prototyping harder since you have to separate out that value even if you're just trying something and know it won't fail.
Returning stack values in functions is fine as long as they fit in the return type. I'm guessing you mean pointers to stack values?

Sure, maybe not 100% of things are exactly as easy, but at least the solutions are established. In Go, if you can't make your program agree with the garbage collector (no matter how excellent it is) then there is no escape.

I don't get why people insist on using C. The ML category of languages (SML, OCaml, Rust, Swift, ...) have a safe programming model, yet compile to efficient native code. They've been here long and people know how they work. The performance is very similar to C assuming an optimized compiler and unboxed types. There's very little overhead for basic abstractions and stuff like functional loops can be easily optimized. If you had any idea how compilers work, you'd know that the intermediate format of any C frontend is a functional static single assignment form, which is closer to ML than C. Of course the garbage collection introduces some overhead, but then you should probably compare ML to D instead. Even then, the safe memory model enables using more efficient garbage collection. C/D style languages can't guarantee that more efficient GC works so they resort to conservative GC.

Another big plus with ML style languages is a more expressive set of basic types. For example tagged unions / variants or tuples are natively supported. C programmers can only emulate them poorly. C might be ok for kernels and drivers in userspace, but for anything else, pick a ML based language. The abstractions are more in line with higher level coding conventions.

Third thing is, these better languages have a sane syntax. C's syntax sucks ass. One example of such suckage are the type signatures for functions and/or pointers. Also C's lack of modules and reliance on pre-compiler is a huge pain in the ass and ruins the compilation speed.