Gentoo Developers Unhappy, Fork udev

[ryao]

Arguably, without RH backing, the entire F/OSS ecosystem would be appreciably damaged. RH builds on the work of lots of other people, and in return, contributes an awful lot of work back to the wider community.

I am tired and I appear to have missed this comment. I must thank curaga for highlighting it.

I think that there is an argument to be made the RedHat's involvement in the community causes damage. RedHat has no qualms about sending poorly tested patches upstream and letting the community fix the problems that they cause. I am told by developers on our virtualization team that this is exactly what RedHat has been doing with QEMU and that upstream QEMU was removed from our tree because of it. The increasing dependence of GNOME on systemd is another instance of damage that RedHat's influence has had. It has forced Gentoo, FreeBSD and numerous others to try to fix the damage. I started the eudev project because I noticed changes that RedHat employees have made to udev are harming our tree. There are likely other examples, but three should be sufficient.

The community would be better off if RedHat only contributed production ready changes. That would dramatically reduce the number of commits that Redhat employees make to various projects. However, many of us feel that RedHat is wasting our time on experimental patches and we would appreciate it if Redhat would stop doing things that make us feel that way. Forks such as eudev are what happen when Redhat's QA standards for upstream contributions annoy enough people.

[curaga]

Arguably, without RH backing, the entire F/OSS ecosystem would be appreciably damaged. RH builds on the work of lots of other people, and in return, contributes an awful lot of work back to the wider community. Oracle is pleased to try and build on the work of lots of other people, including all of RH's work, and contribute absolutely nothing back to the community. Surely you can see why RH may have a motive to try and discourage Oracle's approach.

Last I checked, Oracle still employed people working on X, btrfs, and gcc. Has this changed?

[ryao]

Yes, well, Mark claims lots of things. Ask yourself this: would the future maintenance prospects of the code shipped by Gentoo be affected more by Red Hat going bankrupt tomorrow, or by Canonical going bankrupt tomorrow?

Canonical has done wonders for Linux marketshare and our community grows in size as Ubuntu users have jumped ship. On the other hand, RedHat's actions have been fairly problematic. RedHat developers do not care about regressions that their patches cause in software that we distribute, which causes plenty of headaches for us. RedHat is also actively trying to chase other contributors to OSS projects away, such as Oracle and RisingTide Systems.

Which company do you think that we would prefer to go bankrupt tomorrow?

'Undercut' has a specific financial meaning. Gentoo does not appear to sell support contracts (or, really, anything else), nor do any Gentoo-derivative distributions, so far as I can tell. Gentoo is supported by volunteer work and donations, which is admirable.

A large chunk of RH's support comes from commercial support contracts. As RH believes in contributing to the F/OSS ecosystem, our overheads include all the money spent on this development. The Oracle UL business model is to take RHEL, repackage it under a different name, and sell support for it - without any of the overheads of contributing to upstream development. This is not a problem Gentoo has, as it does not engage in this kind of activity at all. RH is at a theoretical disadvantage in attempting to compete against Oracle to sell support for the same codebase while we are also paying for the development of a lot of that code, but Oracle isn't. Fortunately, RH's customers appear to understand that RH is in a better position to support the same code since we employ a lot of the people who write it, but still, surely you see the problem. Again, as long as no-one else in the F/OSS community really wants to just take the RHEL kernel - which is really just an old stable kernel with specific upstream backports in any case - and use it as the basis for independent development, which they don't, there really isn't a _problem_. There'd be a problem if RH was putting stuff in the RHEL kernel that it didn't contribute to the upstream kernel, but we don't do that.

Lets say that I put some software on the Apple App Store and I charge $0. Hypothetically speaking, someone else could undercut me by listing a fork of my software on the Apple App Store for $-0.01. I would clearly benefit from such behavior and I encourage it.

With that said, Gentoo does benefit from marketshare. A bigger community means that it is easier to recruit additional developers, we have more bug reports and the foundation receives additional donations that could be used to purchase better infrastructure that could be used to do more comprehensive QA testing on our tree. Please do not confuse our lack of pricing for a lack of competition. We are in heavy competition with other distributions.

A large chunk of RH's support comes from commercial support contracts. As RH believes in contributing to the F/OSS ecosystem, our overheads include all the money spent on this development. The Oracle UL business model is to take RHEL, repackage it under a different name, and sell support for it - without any of the overheads of contributing to upstream development. This is not a problem Gentoo has, as it does not engage in this kind of activity at all.

We do engage in such activity. Until recently, all of our projects have been lower profile than RedHat's projects. eudev is the first of our projects to receive widespread attention. It is also the first to be slashdotted before the official announcement.

If you look in various upstream commit logs, you will find plenty of @gentoo.org email addresses in commits. We do contribute to upstream development. We just do not believe in actively trying to obtain revenue from our contributions.

[AdamW]

Anyway, look, we're getting way off into the weeds here. I don't want to be mean or condescending and I'm sorry if I have, I just really am not sure if you're entirely aware of how big a hunk you're biting off here or how much of a (however transient) shitstorm you may (however inadvertently) have triggered. If I'm wrong and your fork turns out to be a raging success, great. That's how F/OSS works, and heaven knows I've been wrong enough times before. I guess I'm just saying, one, Linus likes to yell at people and you can't take his declarations that people are not fit to clean sewers entirely seriously, and two, when the guy who wrote the thing you're trying to fork says, hey, maybe you're not doing it right, it's probably a good idea to listen. I guess we'll see how things turn out.

[AdamW]

Mark Shuttleworth claims that Canonical is:

http://www.markshuttleworth.com/archives/1207

Yes, well, Mark claims lots of things. Ask yourself this: would the future maintenance prospects of the code shipped by Gentoo be affected more by Red Hat going bankrupt tomorrow, or by Canonical going bankrupt tomorrow?

They are non-profit organizations and we are in direct competition with them. With that said, I am a strong believer in permitting others to undercut OSS products. I openly encourage people to undercut us.

'Undercut' has a specific financial meaning. Gentoo does not appear to sell support contracts (or, really, anything else), nor do any Gentoo-derivative distributions, so far as I can tell. Gentoo is supported by volunteer work and donations, which is admirable.

A large chunk of RH's support comes from commercial support contracts. As RH believes in contributing to the F/OSS ecosystem, our overheads include all the money spent on this development. The Oracle UL business model is to take RHEL, repackage it under a different name, and sell support for it - without any of the overheads of contributing to upstream development. This is not a problem Gentoo has, as it does not engage in this kind of activity at all. RH is at a theoretical disadvantage in attempting to compete against Oracle to sell support for the same codebase while we are also paying for the development of a lot of that code, but Oracle isn't. Fortunately, RH's customers appear to understand that RH is in a better position to support the same code since we employ a lot of the people who write it, but still, surely you see the problem. Again, as long as no-one else in the F/OSS community really wants to just take the RHEL kernel - which is really just an old stable kernel with specific upstream backports in any case - and use it as the basis for independent development, which they don't, there really isn't a _problem_. There'd be a problem if RH was putting stuff in the RHEL kernel that it didn't contribute to the upstream kernel, but we don't do that.

[ryao]

And my point is I'm not sure that's a terribly smart policy, as along with Linus' tendency to explode at the slightest provocation, it leads to gigantic messes like this. Gentoo's image has been rather badly damaged among vaguely clued-up developers by the perception that it has made a fairly ad hoc and ill thought through decision to start using an immature udev fork. Even though that's not exactly what's actually happened, a policy where any sandbox project started on Gentoo resources by any Gentoo developer with no oversight is an 'official Gentoo project' is clearly likely to contribute to confusion here.

Feel free to email feedback on that policy to gentoo-project AT lists DOT gentoo DOT org.

Did you consider that perhaps Kay's response to Linus' hair-trigger flame was precisely the same as your response to my post?

I corrected my comments after I asked someone that I know for feedback on how well I expressed my thoughts. I consider the the two versions to be logically equivalent for the most part, but others seem to disagree. I apologize for my inability to distinguish between the two. It is by no means intentional.

With that said, Kay has been given multiple opportunities to clarify what he meant and he has refused to take them. That differs sharply from my own response.

That is an assumption you're making that I don't think is based in reality. Many Red Hat developers work on all sorts of projects that are developed in the open all the time. The only example you have cited to the contrary is a project which was not a Red Hat project at the time in the first place. Can you cite a single company with a better track record of public open source development than Red Hat?

Mark Shuttleworth claims that Canonical is:

http://www.markshuttleworth.com/archives/1207

None of those people, to my knowledge, is a major international enterprise software distributor with whom you are in direct competition. RH is a for-profit publicly traded company; whether you like that or not, it's a fact. It's also a rather successful for-profit publicly traded company, whose success allows it to pay the salary of rather a lot of F/OSS developers. Arguably, without RH backing, the entire F/OSS ecosystem would be appreciably damaged. RH builds on the work of lots of other people, and in return, contributes an awful lot of work back to the wider community. Oracle is pleased to try and build on the work of lots of other people, including all of RH's work, and contribute absolutely nothing back to the community. Surely you can see why RH may have a motive to try and discourage Oracle's approach. It's worth noting that none of the non-commercial RHEL clones have expressed any displeasure with RHEL's kernel release policies. It's also worth noting that Fedora's kernel is not released in the same way. It's *further* worth noting that all of the work of Red Hat's kernel engineers is contributed, promptly, to the upstream kernel. In practice, the RHEL kernel release policy negatively affects exactly one specific type of actor: the type of actor which wishes to take the RHEL kernel and modify it slightly for commercial purposes. It does not negatively affect any actor which simply wishes to release the RHEL kernel as is; they can do so easily. It does not negatively affect other actors who are participating properly and productively in the F/OSS ecosystem - like Gentoo - as they do not simply attempt to copy RHEL's kernel with slight modifications, they work from the collaborative upstream kernel base.

They are non-profit organizations and we are in direct competition with them. Their existence threatens our marketshare, but we do not try to undermine their efforts.

[AdamW]

AdamW, thanks for your response. However, I had made some corrections to clarify what I meant, although it looks like your response would have been the same either way.

1. My point is that there is no distinction between the two.

And my point is I'm not sure that's a terribly smart policy, as along with Linus' tendency to explode at the slightest provocation, it leads to gigantic messes like this. Gentoo's image has been rather badly damaged among vaguely clued-up developers by the perception that it has made a fairly ad hoc and ill thought through decision to start using an immature udev fork. Even though that's not exactly what's actually happened, a policy where any sandbox project started on Gentoo resources by any Gentoo developer with no oversight is an 'official Gentoo project' is clearly likely to contribute to confusion here.

2. Our QA rules are not yet in full effect. I would appreciate it if you would review our work after our first few tags and then let us know what you think of our commits.

I sympathize with you there, but welcome to the world of high-profile F/OSS development. Did you consider that perhaps Kay's response to Linus' hair-trigger flame was precisely the same as your response to my post? If you're planning on strapping in for the long haul, expect more of the same.

3. That does not change the fact that you are openly criticizing our work at a point in time that RedHat's own things would have been developed behind closed doors. It does not seem fair.

That is an assumption you're making that I don't think is based in reality. Many Red Hat developers work on all sorts of projects that are developed in the open all the time. The only example you have cited to the contrary is a project which was not a Red Hat project at the time in the first place. Can you cite a single company with a better track record of public open source development than Red Hat?

4. People resell Gentoo under different names and we are okay with that. We do not change how we do things because of them.

None of those people, to my knowledge, is a major international enterprise software distributor with whom you are in direct competition. RH is a for-profit publicly traded company; whether you like that or not, it's a fact. It's also a rather successful for-profit publicly traded company, whose success allows it to pay the salary of rather a lot of F/OSS developers. Arguably, without RH backing, the entire F/OSS ecosystem would be appreciably damaged. RH builds on the work of lots of other people, and in return, contributes an awful lot of work back to the wider community. Oracle is pleased to try and build on the work of lots of other people, including all of RH's work, and contribute absolutely nothing back to the community. Surely you can see why RH may have a motive to try and discourage Oracle's approach. It's worth noting that none of the non-commercial RHEL clones have expressed any displeasure with RHEL's kernel release policies. It's also worth noting that Fedora's kernel is not released in the same way. It's *further* worth noting that all of the work of Red Hat's kernel engineers is contributed, promptly, to the upstream kernel. In practice, the RHEL kernel release policy negatively affects exactly one specific type of actor: the type of actor which wishes to take the RHEL kernel, modify it slightly, and sell support for it. It does not negatively affect any actor which simply wishes to release the RHEL kernel as is; they can do so easily. It does not negatively affect other actors who are participating properly and productively in the F/OSS ecosystem - like Gentoo - as they do not simply attempt to copy RHEL's kernel with slight modifications, they work from the collaborative upstream kernel base.

[ryao]

1: You appear to have missed the part where I explicitly said:

"And probably still does to those who don't understand that it isn't really an official Gentoo-blessed project, just a small group of Gentoo devs." If a project doesn't require any review and can be started on a whim by any Gentoo dev with no oversight by anyone else, in what sense is it an 'official Gentoo-blessed project', exactly? Does Gentoo apply its name to any project started by anyone who has Gentoo commit privileges, with no review whatsoever of what that project entails? I'm not sure that's a terribly smart policy.

2: I am not a GNOME developer. I am a QA engineer. Never written any GNOME code, never written any code, never going to.

3: RH developers can certainly create projects in the open and develop them at any point, though they don't typically use RH resources to do so, as it only leads to confusion. As neatly illustrated in the current case. KVM was developed by a separate company called Qumranet, who were then bought by RH, who opened up the code. If you look into RH's history, you will note that this is a consistent pattern: we buy small, relatively closed-development-model companies, and open their code. To the benefit of all.

4: I don't have much to say on the kernel patch issue as it's nothing to do with me, but you may note - all of this is entirely public information - that it happened shortly after Oracle started releasing our product, commercially, under a different name and attempting to undercut us on service. For the many years where non-commercial RH clones existed but no commercial clones like Oracle's did, our kernel patches were available in separated form. You may draw your own conclusions.

AdamW, thanks for your response. However, I had made some corrections to clarify what I meant, although it looks like your response would have been the same either way.

1. My point is that there is no distinction between the two.

2. Our QA rules are not yet in full effect. I would appreciate it if you would review our work after our first few tags and then let us know what you think of our commits.

3. That does not change the fact that you are openly criticizing our work at a point in time that RedHat's own things would have been developed behind closed doors. It does not seem fair.

4. People resell Gentoo under different names and we are okay with that. We do not change how we do things because of them.

[AdamW]

eudev is "an official Gentoo-blessed project", but I don't think you understand what that means. Unlike developers in other organizations, any Gentoo developer can start a project for any reason. It does not need approval and it is as official as any other project. eudev would have been almost certainly developed behind closed doors for the first few months of its existence had it been a RedHat project. If I recall, your company did that with KVM. Your company also makes it difficult for independent review of changes its makes to GPL code in monolithic patches. If it weren't for Oracle's RedPatch (which is awesome), it would be impossible for most of us to audit your company's changes. When RedHat doesn't provide monolithic patches, we see single line commit messages in pull requests to various projects. The only exception to this that I have seen is the Linux kernel, where Linus rejects pull requests containing such commits.

All of the changes to eudev are developed in the open and none of our work is currently considered production ready. I wrote the commit that you cited because the kmod dependency broke things on Gentoo stable (think RHEL6) and keeping it in HEAD was problematic for development. We have observed many things to be committed to HEAD are non-functional (e.g. hwdb) when first committed, only to be fixed with additional patches later. You could claim that it would be natural for things to look like this because of how merges work, but when we snapshotted systemd, we obtained a new builtin called hwdb that was clearly broken and was later fixed in a slew of commits made before the 196 tag. We believe that new features should be introduced to HEAD in after multiple developers have verified it as working, not before. We are working toward a repository in which that is the case. However, we literally just started. I am currently reworking the kmod builtin in a branch. It will be merged after it has been verified to be working well on all target system configurations.

With that said, there is great irony to your comments given that you are a GNOME developer. Linus Torvalds has made terrific comments about the quality of your project's work. In specific, it is an "unholy mess" [1] and "GNOME Are In Total Denial" [2]. Your criticisms of our project would be valid had these commits been part of GNOME, but they are not.

1: http://digitizor.com/2011/08/04/linu...nome-for-xfce/
2: http://www.omgubuntu.co.uk/2012/09/l...n-total-denial

1: You appear to have missed the part where I explicitly said:

"And probably still does to those who don't understand that it isn't really an official Gentoo-blessed project, just a small group of Gentoo devs." If a project doesn't require any review and can be started on a whim by any Gentoo dev with no oversight by anyone else, in what sense is it an 'official Gentoo-blessed project', exactly? Does Gentoo apply its name to any project started by anyone who has Gentoo commit privileges, with no review whatsoever of what that project entails? I'm not sure that's a terribly smart policy.

2: I am not a GNOME developer. I am a QA engineer. Never written any GNOME code, never written any code, never going to.

3: RH developers can certainly create projects in the open and develop them at any point, though they don't typically use RH resources to do so, as it only leads to confusion. As neatly illustrated in the current case. KVM was developed by a separate company called Qumranet, who were then bought by RH, who opened up the code. If you look into RH's history, you will note that this is a consistent pattern: we buy small, relatively closed-development-model companies, and open their code. To the benefit of all.

4: I don't have much to say on the kernel patch issue as it's nothing to do with me, but you may note - all of this is entirely public information - that it happened shortly after Oracle started releasing our product, commercially, under a different name and attempting to undercut us on service. For the many years where non-commercial RH clones existed but no commercial clones like Oracle's did, our kernel patches were available in separated form. You may draw your own conclusions.

[ryao]

the fact of a fork doesn't make Gentoo as a whole look bad, but the fact that it's a fork by a bunch of guys who clearly don't have a clue what they're doing did make Gentoo as a whole look bad for a bit. And probably still does to those who don't understand that it isn't really an official Gentoo-blessed project, just a small group of Gentoo devs.

I mean, they clearly have no clue about copyright law. They say stuff like:

"I am afraid that I have to disappoint you. If we were using the waterfall model, I could outline some very nice long term goals for you, but we are doing AGILE development, so long term goals have not been well defined. Some short term goals have been defined, but I imagine that you are already familiar with them. I suggest asking again after our first tag."

which is a fundamental misunderstanding of what waterfall and agile actually *are*, and more to the point, boils down to 'we refuse to tell you what our plans for udev are and we don't actually really know ourselves, we're just poking stuff it feels like a good idea to poke'. This doesn't seem like a great maintenance plan for a core infrastructure component.

And then they commit stuff like https://github.com/gentoo/eudev/comm...4dc81c0589c8cb , which is just...well, Greg explains: http://thread.gmane.org/gmane.linux....26/focus=81281

A fork of udev in and of itself is not an invalid thing to do. This, however, appears to be a fork maintained by Laurel and Hardy. It is clearly doomed unless it gets taken over by people with clue. I'm not even a developer and I can see that they don't really have any idea what they're doing.

eudev is "an official Gentoo-blessed project", but I do not think that you understand what that means. Unlike developers in other organizations, any Gentoo developer can start a project for any reason. It does not need approval and it is as official as any other project. eudev would have been almost certainly developed behind closed doors for the first few months of its existence had it been a RedHat project. If I recall, your company did that with KVM. Your company also makes it difficult for independent review of changes its makes to GPL code in monolithic patches. If it weren't for Oracle's RedPatch (which is awesome), it would be impossible for most of us to audit your company's changes. When RedHat doesn't provide monolithic patches, we see single line commit messages in pull requests to various projects. The only exception to this that I have seen is Linus' tree, where Linus rejects pull requests containing such commit messages.

All of the changes to eudev are developed in the open and none of our work is currently considered production ready. I wrote the commit that you cited because the kmod dependency broke things on Gentoo stable (think RHEL6) and keeping it in HEAD was problematic for development. We have observed situations in systemd where things committed to HEAD in systemd are be non-functional when first committed, only to be fixed with additional patches later. You could claim that it would be natural for things to look like this because of how merges work, but when we snapshotted systemd, we obtained a new builtin called hwdb that was clearly broken. It was later fixed in a slew of commits made before the 196 tag. We believe that new features should be introduced to HEAD in after multiple developers have verified it as working, not before. We are working toward a repository in which that is the case. However, we literally just started. I am currently reworking the kmod builtin in a branch. It will be merged after it has been verified to be working well on all target system configurations.

By the way, I see that you are a GNOME developer. Please enlighten us about how the GNOME project would have approached this. I believe that many people would like to know.