D Language Still Showing Promise, Advancements

[bnolsen]

D (and Go) may never match C++ for resource efficiency. But both languages supposedly can compile hundred-thousand LOC projects in less than ten seconds. That puts them at a developer feedback loop almost as good as a scripting language and performance probably within 10x of C++ (depending upon the application). I imagine there are tens of thousands of software projects that would find that combination of features better than what they have now. Obviously in some places - where absolute control of resources or real-time constraints are present - you have to stay with C++ or C.

I would say this was one of the major features I was looking for. I was hoping that 'D' would be a smaller, simpler, more orthogonal language than c++ with a couple of decades of experience to effectively streamline it. I got a bit nervous when I saw the huge number of reserved keywords for a relatively new language. Kind of reminded me of the ALGOL family which crushed itself by over-featurization.

I wouldn't mind something like a 'd-lite' which has as part of its goal the ability to trivially implement a full compiler without a bunch of brain damage as well. Just a bit of rambling.

[stqn]

I would say this was one of the major features I was looking for. I was hoping that 'D' would be a smaller, simpler, more orthogonal language than c++ with a couple of decades of experience to effectively streamline it. I got a bit nervous when I saw the huge number of reserved keywords for a relatively new language. Kind of reminded me of the ALGOL family which crushed itself by over-featurization.

I wouldn't mind something like a 'd-lite' which has as part of its goal the ability to trivially implement a full compiler without a bunch of brain damage as well. Just a bit of rambling.

They couldn?t stop adding stuff to D v2 and it?s unfortunate. At first I liked D (v1) very much, but after buying and reading the D book I realized that v2 had gotten even more complicated than C++ and was not good at all. Which is why I switched to Go. I haven?t done a lot of things with it yet, but so far so good!

[elanthis]

Have you watched the presentation I linked to? Because there it was suggested that it should be possible to define how the GC works, either by environment variables or compiler switches.

That isn't relevant. I don't want to control how the GC works globally. I want to control independent memory domains, some of which function in ways antithetical to GC. C++ allows this. C allows this. Just about every other language does not.

And so we keep using C++ for systems work and C# for higher-level tools work while the academics keep making new text->machine code toys like D/Go/Rust/etc. that don't solve the actual problems we need solving without also creating a bunch more. Likewise, C/C++ (and C#/Java) solve the problems of tool ecosystems, in that said ecosystems _actually exist_. I'm sure there's some people here who still think Vi or Emacs are the be-all-end-all of code editing, but those of us who like to get more stuff done during the day (and get paid more on account of said higher productivity) have moved on to highly tools-assisted workflows. The replacements to C++ are generally _worse_ here which is kind of hard to grok if you think about how insanely difficult it is to build half-assed tools for C++ (which is a reason that Clang is so freaking awesome). If your new fancy language doesn't have a top-notch IDE and debugger to go along with it, it's just a toy, best left to the dusty corners of CS history where it'll inevitably end up anyway. Objective-C for instance has lived on, despite its warts, because Apple gives it first-class support in XCode. Likewise, C# has Visual Studio and Java has Eclipse. C++ has varying levels of support, Visual Studio + Visual Assist often being considered the best. These are non-trivial pieces of software that amount to far, far more than just the syntax highlighting plugged into Notepad++ or Sublime or Emacs or whatever.

He seemed to not advertize to use extensively the smart pointers, but just for cases of shared instances. So maybe there is a common wisdom in which C++ performance of reference counting should be avoided as much as possible by avoiding using smart pointers at least extensively.

I prefer Herb Sutters approach. Pervasively use smart pointers, and with the C++14 make_unique, *never* create an object with new, ever. unique_ptr is as efficient as a raw pointer for everything it does and has a few bytes memory overhead. And make everything unique (while preferring to pass ownership with std::move) until you need shared.

Indeed, _always_ use smarter pointers except in the cases where you are explicitly passing in a temporary-lived reference to an object, in which case use a reference. Raw pointers have relatively few use cases left in C++11-and-onward world.

Right. To ciplogic: shared semantics are not what a smart pointer is necessarily for. unique_ptr and similar types literally don't even support shared semantics and have _zero_ overhead compared to a raw pointer (except maybe in debug mode with inlining turned off and debug checks turned on). The primary difference is that, unlike a raw pointer, they enforce ownership and lifetime.

You can use smart pointers to handle things that aren't individually allocated with operator new, too. We use them rather extensively to safely and efficiently handle ownership and safely weak references to pool-allocated objects. A smart pointer is just any user-defined type that acts like a pointer and "has smarts." It might not even contain a pointer internally but rather an index to a contiguous chunk of memory in a pool or to a value computed on access (by requiring that you "lock" the pointer while using it so that you pay that cost only once and then are safe; see std::weak_ptr for one way that is done). A smart pointer type that just wraps an integer index is not at all that uncommon.

Have an syntax that has inclusive syntax but store it with half closed interval convention.

One-off special syntax is almost always a bad idea. It took me some years to realize why, so I get where you're coming from. The problem you're trying to solve can be easily solved with simple stdlib updates. Your solution is also relatively non-extensible, as in it doesn't really provide anything of value outside of your use case. It isn't even immediately obvious how to apply it to user-defined types. It isn't applicable to any of the many other places that might want an interval, like the new ranged-based for-statement.

You could achieve very similar by overloading operator[] to take a std::interval and then return a std::array_view (the latter being proposed for some future version of C++ already) and then you can do something like:

auto data = array[std::interval(1, 17)];

The same type could be used for the for-statement:

for (auto i : std::interval(1, 17)) foo(i);

It can be used with algorithms:

auto in = interval(1, 17)
std::copy(std::begin(in), std::end(in), std::back_inserter(vec));

And so on. Relatively simple, requires no parser additions, implementable _today_ on the existing standard (even in C++98, minus the use of auto in that example), and the idiom is applicable to many other uses of operator[] besides just arrays and many other interfaces that might want to take a numeric interval. Clear and obvious win compared to the one-off special-case syntax addition you propose. The only real downside is that it doesn't work for C-style arrays... but stop using those.

Bike-shedding over how to name it to be clear it's half-closed I leave up to people who give a crap about such things.

[plonoma]

@elanthis
Thanks for reading and responding to this thread.

Relatively non-extensible is a good thing, no feature creep! Simple and well contained.

Do like the fact that it can be done with simple std lib updates.
The amount of text you need is a lot compared to the short and expressive syntax of the D slices and stuff.

One-off special syntax is almost always a bad idea. It took me some years to realize why, so I get where you're coming from.

Could you elaborate on this please? What you mean with the One-off special syntax. Could you tell me why and how you realized why it's a bad idea?

[bnolsen]

I'm sure there's some people here who still think Vi or Emacs are the be-all-end-all of code editing, but those of us who like to get more stuff done during the day (and get paid more on account of said higher productivity) have moved on to highly tools-assisted workflows.

That's pretty high and mighty. I can do far more with hooking a bunch of these "out of date" tools together than you probably can with your highly integrated tools that force a specific workfow onto you. It's about learning how to better use the language and use it better, not about learning how to push buttons on what IDE is pushed on you.

My experience has been that these IDE programmers live in a little box and have very little understanding of the actual systems they work on. And they write sloppier code that the IDE lets them get away with rather than try to deal with overall design improvements to make everything better.

[ciplogic]

That isn't relevant. I don't want to control how the GC works globally. I want to control independent memory domains, some of which function in ways antithetical to GC. C++ allows this. C allows this. Just about every other language does not.(...)

This is factually false.
Look here: http://stackoverflow.com/questions/1...t-size-c-sharp or Miguel's blog entry about Mono/.Net performance: http://tirania.org/blog/archive/2012/Apr-04.html

C++ allow this? Does it allow to control how the GC works globally? How? Can you make a C++ generational GC, can you say how to enable it? Can you manage a moving GC? Can you make the marking or sweeping multi-core? It looks to me that you cannot do any of these. Also, the GC implementation in Mono/.Net allows you to get some memory pressure status, can you get this from C++?

Right. To ciplogic: shared semantics are not what a smart pointer is necessarily for. unique_ptr and similar types literally don't even support shared semantics and have _zero_ overhead compared to a raw pointer (except maybe in debug mode with inlining turned off and debug checks turned on). The primary difference is that, unlike a raw pointer, they enforce ownership and lifetime.

You can use smart pointers to handle things that aren't individually allocated with operator new, too. We use them rather extensively to safely and efficiently handle ownership and safely weak references to pool-allocated objects. A smart pointer is just any user-defined type that acts like a pointer and "has smarts." It might not even contain a pointer internally but rather an index to a contiguous chunk of memory in a pool or to a value computed on access (by requiring that you "lock" the pointer while using it so that you pay that cost only once and then are safe; see std::weak_ptr for one way that is done). A smart pointer type that just wraps an integer index is not at all that uncommon.

I agree that SmartPointers are not only for allocated objects, they were historically used to manage COM interfaces, do I recall it right?

I fully understand that smart-pointers are just about managing resources, a smarter RAII with shared semantics. Please anyway read about two items that I was arguing for: SmartPointers do bring overhead (and sometimes is significant), when a reference in a GCed heap doesn't (so the GC is slow can't be replaced with: "I have no GC, so use smart-pointers instead"), there are not many people in C++ world which encourage you to use Smart-Pointers everywhere (given their drawbacks), which is in fact not (so) told in the GC world. Most GC implementations work with GC and stack, no unique, shared, reference, raw pointers, and such. The smart-pointers are optimized with the ARC model for performance (so the LLVM/Clang people noticed that ref-counting can be a bit expensive).

[elanthis]

That isn't relevant. I don't want to control how the GC works globally. I want to control independent memory domains, some of which function in ways antithetical to GC. C++ allows this. C allows this. Just about every other language does not.(...)

This is factually false.
Look here: http://stackoverflow.com/questions/1...t-size-c-sharp or Miguel's blog entry about Mono/.Net performance: http://tirania.org/blog/archive/2012/Apr-04.html

C++ allow this? Does it allow to control how the GC works globally? How? Can you make a C++ generational GC, can you say how to enable it? Can you manage a moving GC? Can you make the marking or sweeping multi-core? It looks to me that you cannot do any of these. Also, the GC implementation in Mono/.Net allows you to get some memory pressure status, can you get this from C++?

I don't think you fully understand what you're quoting, or how good memory use in a well-architected C/C++ program works. That second link practically makes my point for me. No, we don't control a GC globally in C++. We don't use a GC at all in C++. We don't _need_ a generational GC because we don't fragment the memory space excessively or generate craploads of garbage objects in the first place. C# needs a high-end GC becuase the language requires it; some surprising basic constructs in C# generate garbage like it's going out of style.

The most state of the art garbage collector is still lightyears behind a decently-skilled programmer who actually understands what his data is, how it is used, and how the machine works. Think of it like the crews that pick up garbage on the side of the highway: the safest, cheapest, fastest, most efficient cleanup crew is still totally unnecessary if you don't throw garbage on the side of the road in the first place.

Even with the best Mono can do, the best V8 can do, etc., game developers still go _to length_ to work around the GC in those runtimes. Honest. I've shipped a major game using C#/Mono for most of its game logic. It is painful. You spend a lot of time looking at those fancy charts Mono can spit out just to figure out how to make the GC never run outside of level loading. In C++, we don't even think about, even when targeting the SPUs on the PS3. So long as we're not totally brain-dead it's just not a real issue. A GC, even the best of the best, _is_ an issue that has to be actively worked around to maintain low latency in updates.

I won't argue that a good GC can beat bad manual memory management code, nor that a GC can maintain very good "overall bandwidth" of memory management, or handle defragmentation for code generating tons of small objects in a global memory domain. I don't _care_ about bad code, bandwidth over latency, or defragmenting sloppy allocations. I want to avoid those in the first place. The last one in particular requires an approach to memory management that a GC actively thwarts due to the nature of how a GC works; it needs to be responsible for everything in a domain for it to work efficiently (else you end up with the now dated Boehm-Demers-Weiser collector) and at that point there's no reason for multiple domains and hence no way to get true benefits from them. No thanks.

Relatively non-extensible is a good thing, no feature creep! Simple and well contained.

The problem is that, 5 years from now, who's to say we won't have an "even better" way to structure things that some smart-ass dropout from MIT has yet to think up but then for back-compat reasons that one-off syntax is already used up and in the way? With a library-only approach, we can just stop using the "old" way, like we did with std::auto_ptr. If that behavior had been baked into the language somehow, we'd be screwed.

I get super frustrated by the C++ committee and some of their responses to even simple and improvements to the language (having proposed some myself, only one of which is being considered for standardization), but for my complaints at their sluggishness, there are very few post-standardization C++ changes I can disagree with. All the bits of C++ I despise came from C or pre-standardization C++. The process is excruciatingly slow, but it works. Compare to the constant "oops we done goofed" deprecations in PHP, the divisive releases of Python3 and Perl6, or the "hey, why did we add std.kitchen.sink and why is it semantically broken" nature of D/D2/Phobos/whatever-next-week-brings.

The library approach is also handy because, sometimes, the language just gets it wrong, and a library allows replacement in ways that language builtins do not. We don't use std::unordered_map, for instance. It's a fine data structure if you need its particular set of constraints, but you can implement a significantly faster hash table if you don't mind accepting a somewhat different set of constraints. unordered_map was designed for a very generalized set of use cases and has efficiency limitations you maybe don't want and guarantees you maybe don't need. In a language with built-in dictionary types, you're stuck with whatever they provide. Hence the problem with C pointers and arrays; we can make replacements that are equally efficient but semantically superior, but now the "default" way to do things is stuck being used for the worse semantics. Having the foresight to predict those problems is unrealistic; being conservative and avoiding hard-coded decisions over semantics is your best option. (To a limit; sometimes you have to bite and bullet and just hope you made the right choice for decades to come, of course.)

That's pretty high and mighty. I can do far more with hooking a bunch of these "out of date" tools together than you probably can with your highly integrated tools that force a specific workfow onto you.

No, you can't. I worked in those kinds of tools exclusively for over a decade with the same "real men use [Vim/Emacs]" mentality. Been there, done that, found out I was just young and stubborn. There are some interesting tricks you can do faster, sure, but those are "every once in a while" things not the "20 times a day" things. Optimize for the common case.

You have a dated idea of what a modern IDE can do (I did, having first used relics like TurboC++ and VS6 until years later trying VS2008+VAX) if you think Vim or Emacs is more powerful in ways that matter for software engineering. They can manipulate _text_ way better, sure, but a program is not really text. It is merely represented by text for our human consumption as we haven't figured out a more efficient medium for it yet. The compiler quickly converts that text into more abstract, semantically-meaningful representations as soon as it can. A good IDE does this as well. The IDE also understand that once you start debugging a program, it is _really_ not just text anymore. To do a complex refactoring with a good IDE, you hit a key-combo, maybe type in a new name or signature, and you're done; drink a coffee, watch the numbers tick on your bank account, or do whatever else you want while the Emacs user spends so time developing his one-off LISP macro to do the same (and then fix up all the false-positive matches it'll end up with). The IDE gives you a deeply integrated debugger with super painless visualization of values, drag-n-drop instruction pointers, etc. You can have visual debuggers for the modern GPU-heavy world, quite powerful multithreaded debuggers and visualizers, interactive charts and graphics, etc. The build system is integrated deeply in ways you literally can't even do with many UNIX-originated build systems. And then, most of the high-power plugins of Vim/Emacs can _also_ be written as plugins to Eclipse or Visual Studio or whatever you're using, often in a much more productive language like with full access to those semantic-code data structures and build-system integration. Even the things you can do with those text-oriented tools are just interfaced less efficiently require far more time-wasting to use. This is vaguely similar to why dynamically-typed languages have started dropping out of favor again and things like Dart and TypeScript and Rust and Go are popping up; people are realizing that a language which supports tooling (even if just compile-time semantics checking and not edit-time semantic manipulation) is far more important in most larger cases than a language which is excessively concise and easier for lazy typists to slop out.

... That was a big block of words. I think you get the idea. I think this comic sums up my whole point, and my problem with Linux as a desktop OS despite the "Year of Linux Desktop" first rolling around 14 years ago:

My experience has been that these IDE programmers live in a little box and have very little understanding of the actual systems they work on. And they write sloppier code that the IDE lets them get away with rather than try to deal with overall design improvements to make everything better.

_Bad programmers_ write sloppy code. IDEs make it easier for bad programmers to get by; you can't complain that a tool meant to make writing code easier is indiscriminate about whom it helps. It's the same horrible argument that Linus uses to attack C++: "Shitty programmers use C++ so I'm going to be elitist and stick to C." It's outright silly.

[zanny]

It is also worth mentioning C++ provides you with all the tools necessary to write your own GC. If you want a generational GC, what you have to do is have some local gcnew() function that returns a custom pointer construct with pointer semantics (ie, gc_ref or something). The pointers are separately allocated from the underlying objects, and the GC can move and update the pointers at whim (the exact same way Java / C# implement their reference systems). You just can't use &foo or pass around references to the object because the memory addresses are liable to be invalidated at a moments notice.

You don't even really need a custom pointer construct, you just need &* or ** pointers, and the gc can maintain an array or better optimized data structure outside its gc space to guarantee those objects don't move, but the underlying allocated objects can in its generational gc process or however you want to do things. It also depends either on using OS signals to preempt the program to run the gc or have it threaded off (the latter is better I think, especially with C++11 threads).

I mean, most of these higher level languages have their GCs implemented in C++ or C. They use their own custom data structures to contain language constructs but are still doing the same thing on their own data types, albeit it is easier to optimize for a small object pool than the entire range of possible C++ types.

"Shitty programmers use C++ so I'm going to be elitist and stick to C." It's outright silly.

I must be a shitty programmer, because the other day I was implementing pulseaudio support in a backend I was working on and went "holy crap, they literally have a struct of function pointers because they don't have inheritance" called mainloop_api. I mean, it works from the API standpoint, but I could never imagine writing that in C and thinking "yep, this is so much better than doing polymorphism and using virtual functions!"

[ciplogic]

I don't think you fully understand what you're quoting, or how good memory use in a well-architected C/C++ program works. That second link practically makes my point for me. No, we don't control a GC globally in C++. We don't use a GC at all in C++. We don't _need_ a generational GC because we don't fragment the memory space excessively or generate craploads of garbage objects in the first place. C# needs a high-end GC becuase the language requires it; some surprising basic constructs in C# generate garbage like it's going out of style.

The most state of the art garbage collector is still lightyears behind a decently-skilled programmer who actually understands what his data is, how it is used, and how the machine works. Think of it like the crews that pick up garbage on the side of the highway: the safest, cheapest, fastest, most efficient cleanup crew is still totally unnecessary if you don't throw garbage on the side of the road in the first place.

Even with the best Mono can do, the best V8 can do, etc., game developers still go _to length_ to work around the GC in those runtimes. Honest. I've shipped a major game using C#/Mono for most of its game logic. It is painful. You spend a lot of time looking at those fancy charts Mono can spit out just to figure out how to make the GC never run outside of level loading. In C++, we don't even think about, even when targeting the SPUs on the PS3. So long as we're not totally brain-dead it's just not a real issue. A GC, even the best of the best, _is_ an issue that has to be actively worked around to maintain low latency in updates.

I won't argue that a good GC can beat bad manual memory management code, nor that a GC can maintain very good "overall bandwidth" of memory management, or handle defragmentation for code generating tons of small objects in a global memory domain. I don't _care_ about bad code, bandwidth over latency, or defragmenting sloppy allocations. I want to avoid those in the first place. The last one in particular requires an approach to memory management that a GC actively thwarts due to the nature of how a GC works; it needs to be responsible for everything in a domain for it to work efficiently (else you end up with the now dated Boehm-Demers-Weiser collector) and at that point there's no reason for multiple domains and hence no way to get true benefits from them. No thanks.
(...)
_Bad programmers_ write sloppy code. IDEs make it easier for bad programmers to get by; you can't complain that a tool meant to make writing code easier is indiscriminate about whom it helps. It's the same horrible argument that Linus uses to attack C++: "Shitty programmers use C++ so I'm going to be elitist and stick to C." It's outright silly.

So what you're saying:
- an average programmer using GC can be faster than an average programmer using "not managed memory" (whatever this means)
- GCs have good throughput (maybe better than the malloc/new), but for games has to be avoided
- saying about crap-loads of objects, it seems you imply that in C++ you use everywhere const-references, move semantics and such, and you spill everywhere with new in GC, and every time has to be a full blown class, or Generics list. I think you are aware that there is the struct keyword in C#. You have to use it at times, really
- GCs and realtime games probably are not the best match, but what about not-so realtime but close to realtime games? Can't they use a GC? Imagine a game like Heroes of MM, or XNA games? Do they get all the freezes or things that you state that GC are painful for?
- at last, do you know that Mono as of 3.0 has a generational GC? You can use this in your advantage (probably you did). Of course, you have to take in account the memory management algorithm (I think you do this with games, as you say that you want to use memory pools)

This looks to me like a mediocre C# programmer (or a mediocre C++ programmer would do on the same years experience):

We don't _need_ a generational GC because we don't fragment the memory space excessively or generate craploads of garbage objects in the first place. C# needs a high-end GC becuase the language requires it; some surprising basic constructs in C# generate garbage like it's going out of style.

Have you heard about copying STL vectors, maps because people forget to use a reference or a const reference? Thanks goodness that C++ 11 comes with Move constructor. You can say: yeah, this is a beginner (the one that copies collections all the time), but it looks that you represent yourself as a beginner into C# world, and expect to not be call on it.
C#:
Q: How many objects are created when you use a lambda (for Linq)? Let's say to find all objects that match a condition?
A: 1 (for closure)? If the iteration happen (it is lazily loaded), you will create the 2nd object (the Enumerator)
C++:
Q: How many copies are involved if you don't put ++ before iterators in a loop?
A: As many objects are in the loop

Sure, you never make these pitfalls, but you just expect that all C# programmers do, so this is why you can write efficient C++ code, but not fast C# code, right?

[ciplogic]

It is also worth mentioning C++ provides you with all the tools necessary to write your own GC. If you want a generational GC, what you have to do is have some local gcnew() function that returns a custom pointer construct with pointer semantics (ie, gc_ref or something). The pointers are separately allocated from the underlying objects, and the GC can move and update the pointers at whim (the exact same way Java / C# implement their reference systems). You just can't use &foo or pass around references to the object because the memory addresses are liable to be invalidated at a moments notice.

You don't even really need a custom pointer construct, you just need &* or ** pointers, and the gc can maintain an array or better optimized data structure outside its gc space to guarantee those objects don't move, but the underlying allocated objects can in its generational gc process or however you want to do things. It also depends either on using OS signals to preempt the program to run the gc or have it threaded off (the latter is better I think, especially with C++11 threads).(...)

When you implement the GC, and you're using your custom handles, and your GC code will rewrite parts of stack, and you put blocking points for threads, you get eventually to need this VM (as the code overhead should be similar). Also the C++ coding will be so awkward, that would make most normal to write your code into these languages supporting the GC nicely. GC are tuned for years, .Net 4.0 (released with VS 2010) was tuning the pause times using a background GC. I can say that is out of reach of any medium company to write their GC because of this (as it takes years and years to tune them).

Based on this, I can say that is not possible to write a GC. Mono's generational GC was a 2-3 years work (at least), and around 1-2 years for tuning it. Mozilla's IonMonkey is actively working for a Generational GC, and the work for it is again longer than 1 year (of course, was not the only task). This bug: https://bugzilla.mozilla.org/show_bug.cgi?id=619558 was reported in December 2010, so I'm not sure if anyone would implement anything but a trivial GC.