Revising Our Motherboard Testing Process

[Michael]

Year to date we have only posted four motherboard reviews to date, which is quite dismal compared to when we were cranking out three new motherboards a month on top of our other reviews. However, a lot of factors attribute to the differences... This is due to lots of work on other types of articles, limited staff and budget, I have been very busy with some other ventures, but more importantly we are working on updating our motherboard evaluation process.

We have largely focused upon overclocking and Linux performance in past articles (along with compatibility notes), but we are now evaluating our evaluation process with focusing more upon the Linux compatibility and tweaking. We will still be benchmarking every motherboard that comes through our hands for testing, but in recent months we have seen more and more motherboards that run into compatibility issues or perform peculiarly with some distributions and other oddities. We are working on a new standard for our motherboard evaluation and review process. When time allows, I will elaborate a lot more on our plans (or David may also be able to shed some additional light).

However, before anything is finalized we would like to hear from our readers and forum goers on what areas you would like to see greater coverage or any other comments that you may have. So feel free to type away as we highly value reader feedback before enacting any major changes.

[Zooko]

I'm not interested in overclocking, but I'm very interested in compatibility and in Free Software drivers.

[ivanovic]

I'm not interested in overclocking, but I'm very interested in compatibility and in Free Software drivers.

Agreed! I would like to read more about what is supported by the open drivers directly included in the (vanilla-)kernel and what you need extra (propriatary?) drivers for. Some hints how to get some stuff working if there are general probs getting the hardware work are welcome, too.
Overlooking on the otherhand IMO is something that has not to be considerd this much, though I welcome at least some comments about the options provided in the BIOS, like fan management, turning off integrated devices and such.
Okay, if there are general speed differences between some hardware devices and general (speed)probs with a MB it should be mentioned, too.

[Michael]

Here's some further thoughts from the Tyan Toledo i965R review that will be published next week.

In recent months we have run into a number of new motherboards having operational problems with some of the latest Linux distributions -- from both AMD and Intel. However, we have been working on extensively overhauling our motherboard testing process.

We are still working on revising these testing guidelines and they are still a way out from being finalized. We will still be featuring benchmarks of the motherboard at hand compared to other similar motherboards, but we will be focusing more upon Linux compatibility with a number of different distributions, steps required to get the motherboard up and running, are any binary blobs required, and much more. With each new motherboard review we are looking at trying it out with the latest development build of Fedora and Ubuntu, the current version of Ubuntu/Fedora (or current - 1), and another distribution or two (*BSD and Solaris are also being evaluated).

Pending feedback from our readers and other stakeholders, we hope to issue a draft on our new testing guidelines within a matter of weeks. At that time we will be back with additional Linux performance / compatibility thoughts from the Tyan Toledo i965R.

[joshuapurcell]

I agree that the focus should be more on compatibility than overclocking ability (or performance), and if this means leaving performance statistics completely out of some articles/reviews in order to have enough time to thoroughly cover compatibility then I am for it.

But I think it's important to at least cover overclocking in most articles while also having the steps taken to overclock under Linux documented (this document/howto wouldn't need to be specific to a single piece of hardware though). My reason is because this is a hardware enthusiast site as well as being the only one around (as far as I know) that is also a Linux enthusiast site. While some motherboards will have special issues that make them less compatible with Linux, the vast majority of hardware released today will be able to run Linux without any show stoppers. Hardware enthusiasts, gamers, etc. want Linux to not only run on the hardware, but they want it to run fast.

I know you can't have performance before compatibility, but the average reader of any of the many Windows-centric hardware enthusiast sites doesn't even give a second thought to compatibility when reading the articles... they are just interested in how well things are performing. Since Linux is a second-class OS in the eyes of most hardware vendors, it makes sense to cover the compatibility topic where there are problems. But I don't think it should necessarily exclude the performance aspect.

[Michael]

I agree that the focus should be more on compatibility than overclocking ability (or performance), and if this means leaving performance statistics completely out of some articles/reviews in order to have enough time to thoroughly cover compatibility then I am for it.

But I think it's important to at least cover overclocking in most articles while also having the steps taken to overclock under Linux documented (this document/howto wouldn't need to be specific to a single piece of hardware though). My reason is because this is a hardware enthusiast site as well as being the only one around (as far as I know) that is also a Linux enthusiast site. While some motherboards will have special issues that make them less compatible with Linux, the vast majority of hardware released today will be able to run Linux without any show stoppers. Hardware enthusiasts, gamers, etc. want Linux to not only run on the hardware, but they want it to run fast.

I know you can't have performance before compatibility, but the average reader of any of the many Windows-centric hardware enthusiast sites doesn't even give a second thought to compatibility when reading the articles... they are just interested in how well things are performing. Since Linux is a second-class OS in the eyes of most hardware vendors, it makes sense to cover the compatibility topic where there are problems. But I don't think it should necessarily exclude the performance aspect.

We won't be eliminating our performance portion, that is for certain. We will, however, be focusing much more on compatibility. In early May we are looking to make a public statement about another addition to Phoronix's compatibility testing, which should prove to be a major statement.

[Svartalf]

In all honesty, Michael, I'm interested in all of it, including overclocking- but, I think your decision to worry about compatibility and stability reviewing first and foremost is a welcome change. You can always get some trusted people to do the overclock and peak performance evaluations (or the other way round...your call..) to increase staff.

I'd volunteer for either role!

Seriously, though, what's been done to date, coupled with what you're planning to do is greatly appreciated- Kyle and Anand haven't been putting as much effort into Linux stuff as they probably ought to until recently. Even then, it's not been the stuff I thought they ought to be doing.

[Michael]

Thanks for the kind words.

AnAndTech has done virtually nothing lately on Linux... In 2004 they did about a dozen Linux articles, and then one in 2005, and nothing since then.

[Michael]

Also, this new testing process is going to be integrated with the new phoronix relaunch -- http://www.phoronix.net/forums/showthread.php?t=1360

[Svartalf]

Thanks for the kind words.

AnAndTech has done virtually nothing lately on Linux... In 2004 they did about a dozen Linux articles, and then one in 2005, and nothing since then.

Heh... I've noticed that. It's why I'd worded things the way I did. It's why it was such a surprise that even [H]Consumer did their little piece in recent days. Perhaps they kind of got tired of having to field the "Where's the Linux Love?" that got asked at least once or twice at every [H]ardware Workshop at QuakeCon.

And you're welcome, Michael- nobody else in the enthusiasts space seem to actually care about what is a majorly growing segment, in spite of everything stacked against us.