Concerns Over Merging Drivers Back Into The X Server

[libv]

One thing you seem to completely miss (if I were you I'd say you were doing it deliberately, but I'm me so I suspect you just can't see it),

I release a -ati 6.14.4, that builds against libdrms back to 2.4.1, now there is a fix in libdrm 2.4.14 that really really really makes radeon work better, fixes crashes etc. How do I enforce this on the users? I don't *want* them running older libdrm's. In Intels case a lot of libdrm_intel changes went in that were fixes for this type of problem. At some point you have to realise that making a driver build against lots of different things, and making a useful driver are vastly different. In my experience users want the complete experience when they use your driver they have learned that from closed source drivers, we need to be able to provide something similiar but leveraging the advantages of being open source, which mainly means avoiding the amount of code duplication that occurs in the closed source drivers. Like when KMS came out lots of people upgrade kernel + -ati, but got a really sub-par experience with a 1.6 X server, this meant they got a bad opinion of the feature, now I have a choice, refuse to build/run against the 1.6 X server or just close bugs saying upgrade the X server, both of which require the users to upgrade their X server.

Also the other point raised on the list, that you haven't replied to, is you don't need to rebuild all the libs when a new proto gets pushed out, I push out new protos the whole time and we don't rebuild the complete client stack against them ever. Again if you want some of the features expose to clients of course you need to rebuild them. I'm not sure how you propose exposing new features otherwise.

It would be really nice if you could avoid talking like a paranoid lunatic, and stick to addressing points, you actually have some points based in reality that are useful to the discussion, but when it comes to the "Intel are out to screw you and I'm the only one that can see it" you do start to seem a bit ott, and it makes it easier for people to just ignore you.

Dave.

Heh. First of all, i listed the dri driver above. Secondly, a version bump of (the whole of) libdrm not only brings in bugfixes, it always brings in new features, which include new bugs and regressions.

In general, no effort is being made to backport bugfixes to older drm versions, and no effort is being made to keep any form of backwards compatibility. That being said, this is the general case, you are _now_ doing things slightly differently because the ghost of radeonhd is still there. As a great for instance, if radeonhd wasn't there, you would've dropped UMS months ago. But now, with radeonhd still around (despite your best efforts), you cannot force people to switch to KMS, as they still have an alternative to fall back to.

Now, you should not be forcing users to update. Ideally, you should give users working code to begin with. (Naturally) Failing that (as that's how graphics drivers are almost per definition), you should give them an easy route to get them a working graphics driver stack. You should not, under any circumstances, force your users to update most of their installation. Users will pick up new features as they upgrade their installation as part of natural maintenance. And then, as soon as debian stable and enterprise versions get upgraded, you should drop backwards compatibility. If you want to provide anything less, then that's up to you, but your users will rate you accordingly, and you will see drivers picked up, tested and debugged, less frequently.

This brings us to the core of the issue here: Most of the bugfixes go into the driver specific parts, not into the infrastructure parts. Libdrm is the key example there. Having one big libdrm with a single version, including all driver specific bits, is madness, and this has to be changed. Once this is done, you can then notice how stable the non-driver specific parts are, and that it is trivial to build the driver specific bits without having to update anything else. This gives you the freedom to tell users to update their driver specific libdrm part as needed. You _can_ force users to update that.

I handed the solution to that on a platter a few months ago: namely, create graphics driver stacks by cutting driver specific parts loose from the infrastructure. But, as fully expected, you just had to shoot that down.

As far as the libs go, this is just pure logic. The protocol headers are the API between the xserver and the client libraries. Dropping existing granularity is just stupid. And not rebuilding either xserver or relevant libraries/clients when changing the protocol headers is just asking for trouble. You might get away with it, but there be dragons.

I can only see two reasons as to why people are backing this. It is either stupidity, or it is bigger "goal", a certain direction that some people steer towards. For most people who tag along, it is some value of stupidity, but for those driving this, they should know better, which leaves just one other explanation.

So about "talking as a paranoid lunatic", i have been around people like you for far too long, and i have seen people like you at work far too often. As far as Keithp goes, I only recently came across David Wexelblat his email to [email protected], and it explained a lot of things, including several things which affected me directly. Most of all, it showed that this game has been going on for a looong time.

A nice case of people accusing me of "talking as a paranoid lunatic" is the following: Somewhere late oktober or early november 2007 Egbert asked me whether my "paranoia" surrounding a competing driver (to radeonhd) by the usual suspects wasn't completely off. I conceded, said that we might have just gotten away with it by the unholy amount of work we invested in radeonhd. Now, what did the usual suspects do a short while later?

[libv]

@libv remember when i asked you (on this forum) why you didn't fork all of xorg since you already have quite big patches for those drivers the way you think it should be done...

Now i'm wondering.. IF they (xorg) do decide to put the drivers back in xorg would you then fork it? I'm not trying to get you into forking that project.. i'm just curious how this scenario would go.. since i think xorg is right now playing a very dangerous game.

I am not in the habit of forking, and I am a driver developer with no ambitions beyond providing the best driver support that i can.

When people were busy tearing down Xfree86, i just took the (in one case, seemingly) active developers of the unichrome driver together and created a project to further this unichrome driver from outside the turmoil.

I could write a whole strategic analysis of why or why not a fork should happen and who it should by, but i'd rather go and play with hardware again (after a whole week of doing very little). Just know that i will not be driving a fork, i will however be happy to provide driver compatibility, and to dispense my insights.

That being said, only one company is in the habit of either actively forking or pushing forks forwards, and it is exactly this company that has nothing to gain from such a fork at this time. Those with actual enterprise desktops, who need working graphics drivers for recent hardware, they are the ones that are most likely to back a fork, even though they do not seem in the habit of doing so.

@everyone
This is about the versions and backporting.. What i just don't get with all this new KSM stuff and new oss radeon drivers is the backporting.. That driver and all of KMS is under freaking development so why backport anything at all? As for the "enterprise".. If they need to have kernel versions of a few years old along with xorg then also let them buy vga cards of a few years old or just use plain vesa.. DON'T BACKPORT stuff that's in development! For the people that can't get the latest dri because they run ubuntu or fedora.. MOVE AWAY to something rolling like archlinux. I really just don't get the backporting (illness) these days...

Both hardware vendors and enterprise desktop distribution vendors need to provide hardware enablement. This is a shared business goal and also a shared responsibility. But the usual suspects will claim that responsibility is purely at the enterprise desktop distribution vendors.

[airlied]

How would you know what is childsplay and what isn't. You've never backported either X or drm drivers.

Wow its like you have no clue about what our job at Red Hat is. Its just wierd how you actually think you know what we do inside Red Hat, despite you never having any dealings with us.

-ati driver

http://rpm.pbone.net/index.php3/stat/22/idpl/12384477/com/changelog.html

xserver

http://rpm.pbone.net/index.php3/stat/22/idpl/12953498/com/changelog.html

Intel driver

http://rpm.pbone.net/index.php3/stat/22/idpl/12958178/com/changelog.html

-nv driver

http://rpm.pbone.net/index.php3/stat/22/idpl/12832263/com/changelog.html

You might learn something about what our customers pay us for,

Dave.

[daniels]

also, no-one has ever said that merging (most) drivers back in means that it will be impossible to build anything out of tree. anyone who says/encourages this point of view is at best misinformed, and at worst lying. it just makes life much easier for the merged-in drivers during the development phase (apis can be smashed at will), and then when the server's released, the out-of-tree drivers can adapt to the new api (sound familiar?).

[daniels]

As far as the libs go, this is just pure logic. The protocol headers are the API between the xserver and the client libraries. Dropping existing granularity is just stupid. And not rebuilding either xserver or relevant libraries/clients when changing the protocol headers is just asking for trouble. You might get away with it, but there be dragons.

no, you're wrong. and since we're playing proof-by-assertion, i can't be bothered proving it. but trust me.

I can only see two reasons as to why people are backing this. It is either stupidity, or it is bigger "goal", a certain direction that some people steer towards. For most people who tag along, it is some value of stupidity, but for those driving this, they should know better, which leaves just one other explanation.

ok, so humour me. what is the end 'goal'? (i guess i fall into the 'stupidity' end of the spectrum by not even knowing what conspiracy i'm aiding, let alone be rewarded for it ...)

So about "talking as a paranoid lunatic", i have been around people like you for far too long, and i have seen people like you at work far too often. As far as Keithp goes, I only recently came across David Wexelblat his email to [email protected], and it explained a lot of things, including several things which affected me directly. Most of all, it showed that this game has been going on for a looong time.

A nice case of people accusing me of "talking as a paranoid lunatic" is the following: Somewhere late oktober or early november 2007 Egbert asked me whether my "paranoia" surrounding a competing driver (to radeonhd) by the usual suspects wasn't completely off. I conceded, said that we might have just gotten away with it by the unholy amount of work we invested in radeonhd. Now, what did the usual suspects do a short while later?

who are the 'usual suspects'? i guess you don't mean myself and j?r?me (being that we created avivo before radeonhd was public, thus meaning that radeonhd was completely ripping off avivo and its competitor ...), but are instead referring to dave (who also started before you) and maybe alex?

if you're going to accuse people of entertainingly insane conspiracies, at least have the balls to put it out there rather than occasionally muttering 'the usual suspects', and their 'goals', and that your paranoia wasn't actually paranoia because they were out to get you! this petty bitching at every turn without anything actually concrete is really not a good look.

so:
* who exactly is in the conspiracy/ies?
* what is the goal of these conspiracy/ies?
* how do they affect radeonhd/novell/you?

if you don't have the decency to enumerate these, please stop putting out weird blanket accusations of people conspiring against you without at least giving enough detail for them to prove you wrong (or right, who knows).

[MrCooper]

also, no-one has ever said that merging (most) drivers back in means that it will be impossible to build anything out of tree. anyone who says/encourages this point of view is at best misinformed, and at worst lying. it just makes life much easier for the merged-in drivers during the development phase (apis can be smashed at will), and then when the server's released, the out-of-tree drivers can adapt to the new api (sound familiar?).

In other words, out of tree drivers will be second class citizens at best (though when the people proposing this change mention what they might do with the in-tree drivers, I'm hardly hoping for the best in this context).

[MrCooper]

History showed us that updated drivers are usually not compatible with older Xorg releases anyway, [...]

Which driver(s) are you referring to? The only one I can think of offhand matching your description would be intel, the developers of which indeed haven't cared much about support for older servers. This is proposal is mostly a next step of theirs in the same vein.

[bridgman]

In other words, out of tree drivers will be second class citizens at best (though when the people proposing this change mention what they might do with the in-tree drivers, I'm hardly hoping for the best in this context).

We had a similar discussion at XD(C?) 2008 at Google in the "5 minute session" talks. The conclusion (to the extent there was one) seemed to be that the ideal situation would be highly integrated development and test (between X and drivers in this case) without hurting the ability to release independently when necessary.

To my little mind that implies a modular build system, and "continuous integration" development (monolithic-ish) between Xorg releases where the bulk of the testing (not all, but the dfault) is done with relatively recent versions of the other components (maybe a week old at most).

This doesn't actually require a monolithic source tree -- syncing and building multiple projects together would also work -- but one obvious downside of the current source tree (forest ?) is that API changes between core and driver require coordinated changes in 30-odd different git projects (one for each driver). As daniels said moving to a single tree would definitely reduce the effort required for internal API changes. On the other hand I don't see a lot of API changes between video drivers and core, and haven't heard any indication that there are plans to increase the rate of change there (but I could easily have missed something).

I still think the big win will come from encouraging driver & server devs to "sync and build the other parts as well" on a frequent basis during development rather than doing most of the development on older, static versions of the components they are not working on. That is what will drive release predictability and improve quality.

Combining the source trees will simplify and encourage a continuous integration dev/test model and simplify the implementation of API changes - the downside, I guess, is that developers and testers will need to download the entire server+all drivers tree to get the latest code for one piece. This is probably fine for most development, since we want all the new stuff to be tested together, but will be awkward for the individual release scenarios unless we encourage use of single-driver source tarballs and edgers-type distro-specific builds.

Out-of-cycle releases seem to be valuable in three areas - new hardware enablement between distro releases, critical bug fixes, and experimental "hey I heard the latest driver code might fix my XYZ problem" testing. The discussion on the xorg-devel list was a bit cryptic about whether good support for out-of-cycle releases was considered a clear requirement for the final solution, but I think daniels made it clear that this was the case.

[dfx.]

also, no-one has ever said that merging (most) drivers back in means that it will be impossible to build anything out of tree. anyone who says/encourages this point of view is at best misinformed, and at worst lying. it just makes life much easier for the merged-in drivers during the development phase (apis can be smashed at will), and then when the server's released, the out-of-tree drivers can adapt to the new api (sound familiar?).

also, no-one has ever said that it would be opposite. in fact, no one ever has said anything concrete about what will be changed, what may be changed and what will never be touched.
of course, maybe it just me who missed presentation of exact plans and proposals by all the parties.

so, you may consider me as uninformed.

personally, by "merged-in drivers" i understand three main ones (big and rapidly developing): radeon,intel and nouveau; and, maybe, bunch of small, stagnated ones.
here i can imagine Intel guys doing their stuff as they pleased, Red Hat & Google guys doing their stuff as they see fit. it's okay - they have a lot to be done and commercial support.

but what Nouveau guys think ? they showed A Lot of concern about them breaking api when Linus was pushing for inclusion. That much that they actually were _against_ of using their work in mainstream yet...

and in my understanding - because they are also ones with big plans BUT without commercial support or much of distribution support.

as Luc kinda said: "new features are new bugs" -> new features without chance of "downgrade" stuff selectively (broken or non-existent apis) are new regressions you can't ignore.
for people with full commercial support (testing and backporting) it's irrelevant but for people like me (with no one to decide what "stable" for my system but me) it is inevitable crap to endure (no chance to go-back with one simple command and need to wait for one particular fix in the whole thing).

you are awesome dudes and plentiful developers but _Are You THAT CONFIDENT in Yourselves_ ?

to be more precise - can you do that development "breakthrough" phase and NOT to piss off a lot of people with such unclear approach ?
because, again - in my understanding, it's how big-ass commercial "all-in-one" "drivers" are done. and they freaking pissing people off...
and for answer for "sound familiar ?" - it sounds like breaking a way for those monsters (one moster, actually - nvidia blob which still able to keep up for X somehow) and replacing it with other few monsters - "ours".

so, please, stop this personal rivalry of who did what and how much it worth and think for a second of how much your balls are actually weight.
but then again - your stuff is the only stuff which is most useful on all of my systems, so maybe they are weigh that much.

i have no idea - i'm uninformed and afraid that it will just kill all stagnated drivers (like geode), bring more regressions, made whole thing clumsy and "you" reluctant to release it often (until some critical mass of fixes is incorporated). besides - what is the point of such big changes if objective can be achieved by changing release policies and build-system ?

[bridgman]

dfx, have you read the discussion on the xorg-devel mailing list ? The discussion is still going on there and I think it's going OK...