Radeon HD 7000 Series Open-Source Still A Mess

[bridgman]

I can tell you from my experience that all you need to know is OpenGL and GLSL and you need to know them really well.

Yep. It's hard to find experienced graphics driver developers, so for new developers we end up looking for people who know OpenGL details (and/or DX for the Windows world) well enough that following the GL commands down to hardware doesn't seem like much of a stretch.

If you're trying to learn graphics *and* hardware/drivers at the same time, it's really difficult and IMO no documentation in the world is going to make that easy.

[bridgman]

Hmm, wouldn't it have been easier if the code was documented as it was written, by those who wrote the code in the first place? It doesn't sound like a very difficult thing to do. Now when it's incredibly complex, I can see it being a problem, but it makes me wonder if it's a problem just because of poor coding habits...

The problem is that most of the open source driver code wasn't written as a "one time top down project" where doing design documentation up front would be an obvious expectation. A lot of the big changes were made incrementally as time permitted over a few years, as bits of time became available. That makes it hard to keep detailed design documentation current.

In some cases (eg r600g shader compiler) the code was known to be temporary when it was written so it would be hard to justify requiring high quality documentation for the initial code (which is also the current code). I haven't asked him, but I imagine Jerome would have wanted to spend any time he had for documentation on the "final" design he had been working on, not the initial "write this so we can get the rest of the stack running" code that runs today.

Another challenge is that the underlying hardware is constantly evolving, so even the line between hardware and software keeps changing. That, in turn, drives change all through the stack and makes the old "a job done right never needs to be done again" line into a bit of a joke.

There probably are some sections of the code which the developers expect to keep around for a while and there's no reason why those sections couldn't / shouldn't be documented, but you might be surprised how many bits of the current driver stack the devs expect to have to rewrite in the next couple of years. That's why I think staying with higher level documentation (basically enough to get new developers to a general area of code and give them an idea about how that code relates to the rest of the stack) is the most likely to be successful for the near future.

[marek]

Maybe it's just me, but I'd say it's really the complexity which keeps people from contributing.
The threshold to enter that business seems rather high.

I can tell you from my experience that all you need to know is OpenGL and GLSL and you need to know them really well. Then there's only the will to contribute. There are also plenty of useful papers, articles, and conference slides about how to efficiently program for GPUs with APIs like OpenGL and Direct3D on both major vendors' websites. I think anyone who can write a decent game rendering engine (that's why you learned OpenGL in the first place, right?) has already enough knowledge to write GPU drivers. The rest can be learned on the job.

I doubt any kind of documentation for new GPU driver developers will be worth the invested resources its authors will put into it.

Looking back, I think the most difficult thing for me was building and installing Mesa for the first time. At least that should be documented somewhere.

[GreatEmerald]

Hmm, wouldn't it have been easier if the code was documented as it was written, by those who wrote the code in the first place? It doesn't sound like a very difficult thing to do. Now when it's incredibly complex, I can see it being a problem, but it makes me wonder if it's a problem just because of poor coding habits...

[bridgman]

There are conflicting views on the value of documentation though, even among the active developers. If you apply a classic triage model to the pool of potential developers and divide them into...

- those who will learn enough by asking questions & reading code to work effectively without documentation
- those who wouldn't be able to learn without documentation but would & will become effective developers if given a decent docco starting point
- those who would read the documentation but still not be able to deal with the complexity of the hardware & code in the time they have available

... the big debate is over the size of the middle category, the ones who would be helped by documentation. If the work contributed by that middle group is more than the work it takes existing developers to write and maintain documentation, then the community is ahead. If not, then the community loses.

The argument against a big push on documentation is that it would mostly help potential part-time developers, and the aggregate work available from the successful ones would be less than the time required to write & maintain the docco because of their part-time nature.

IMO the solution is to find the right level of detail. I think everyone agrees that documenting where to find the code and how to build/install it is worth doing, so the question is whether documenting one or two more levels of detail would help and where we reach the point of diminishing returns. I suspect we reach that point quite early, ie that the optimal point would be maybe 5-10 pages *total* of well-written design docco divided across the major component projects (eg a page or two for stack-level plus a page or two for each major component) and kept ruthlessly up to date.

That would be enough to let potential developers get a lot further into the code quickly, and hopefully enough that they can start tinkering and asking good questions (where "good" in this case means "other developers feel that answering the questions is a good use of their time").

One of the obvious challenges would be how to avoid the documentation growing past the point where it can be maintained, since documents (and code) tend to grow during periods of enthusiasm then rot when the enthusiasm wanes.

It would be nice if code and docco automatically got smaller when fewer people were available to work on them but nobody knows how to do that yet AFAIK

[entropy]

Graphics drivers get more complex every year as expectations go up for features and performance, and that in turn makes it harder for volunteer developers to find enough time to make real progress on the drivers, but the ability to "grab the wheel at any time" is there in principle.

Maybe it's just me, but I'd say it's really the complexity which keeps people from contributing.
The threshold to enter that business seems rather high.
There's this promising project to write documentation and guides for new GPU driver developers

http://cgit.freedesktop.org/~marcheu/lgd/

So far it's a very good read, IMHO, but it's incomplete and progress seems to be slow at best.
I'd say this is a *very* important project and a first version is hopefully completed soon (I doubt it).
It's not like it will automagically spit out better drivers immediately but it might help to lower
the threshold for new contributors, which are new to GPU hardware and the graphics stack.

AFAIK there's no comparable literature available yet. Otherwise please point me to it.
I'll buy it immediately. And I don't mean books on generic driver development on Linux.

[89c51]

So open source isn't to be held in as high regards as I thought it was, hm? I mean, it is certainly more desirable than closed source software, but considering how companies still hold a vast majority of the power as nothing seemingly gets done without them...

It has nothing to do with how high is regarded or code quality or whatever. The development would just be much much slower. Its different having someone code everyday as his job than someone doing it on his spare time.

[bridgman]

Not sure I agree. As long as the code is public and there are developers outside the HW vendor who know the code well enough to work on it, the community *has* the power. The question is if and when it makes sense for them to use it.

Graphics drivers get more complex every year as expectations go up for features and performance, and that in turn makes it harder for part-time developers to find enough time to make real progress on the drivers, but the ability to "grab the wheel at any time" is there in principle.

Most of the developers working for HW vendors or distros work as "part of the community" anyways, so you do get a largely vendor-independent group making most of the key architectural decisions. The transition to kernel modesetting was a good example, with developers from a half-dozen different companies working together on various bits of the solution, and in many cases the developers *were* independent when they started work on KMS but continued working on it after going to work at Red Hat, AMD, Intel etc...

I think you get a more accurate picture of how open source works if you imagine all the work being done by volunteers, then consider that some of the "volunteering" is done by individuals and some is done by (ie funded by) large companies. I can only be sure about AMD but it's probably safe to say that all the large companies expect/allow their developers to work as part of the community and to strike a balance between community priorities and company priorities.

[FutureSuture]

No.

In order to fix something you have to know can and have the will. Linux would have never been what it is now without companies paying developers. Most of the work on AMD FOSS drivers comes from people on the AMD payroll. And redhat i think.

So open source isn't to be held in as high regards as I thought it was, hm? I mean, it is certainly more desirable than closed source software, but considering how companies still hold a vast majority of the power as nothing seemingly gets done without them...

[89c51]

Excuse my ignorance, but when something is crap and open source, doesn't the community rush to fix it? I thought that was the very reason many Linux users want software to be open source. Where are all the zealots?

No.

In order to fix something you have to know can and have the will. Linux would have never been what it is now without companies paying developers. Most of the work on AMD FOSS drivers comes from people on the AMD payroll. And redhat i think.