Gosh, Our Test Farm Already Finds Big Regression

[phoronix]

Phoronix: Gosh, Our Test Farm Already Finds Big Regression

Back in October we made it possible to autonomously find performance regressions in a project's code-base by leveraging the Phoronix Test Suite atop Git's bisect command. A new Phoronix Test Suite module was created that would automatically run a binary search on a defined code-base and keep running the specified test(s) on each revision and then traverse to the next using Git bisect until the lone commit causing the defined regression was located...

http://www.phoronix.com/vr.php?view=Nzc5NA

[Nogotheg]

I'm curious if you could compare older versions with current kernels sometimes in the future. Just to see if 2.6.32 is really faster than 2.6.12/22/etc. (if possible - btrfs-performance in 2.6.12 wouldn't be very useful ;-) )

[FireBurn]

Seriously why cant you folk just raise a bug!!

[jbrown96]

And you are holding off disclosing this because???? Seriously, what do you have to gain by not submitting this. Every day you wait brings the release date for .33 closer, and makes it less likely that there will be time to create/test patches.

I don't even know what to call this move. It's not selfish because there's not really anything to gain/

[[Knuckles]]

And you are holding off disclosing this because???? Seriously, what do you have to gain by not submitting this. Every day you wait brings the release date for .33 closer, and makes it less likely that there will be time to create/test patches.

I don't even know what to call this move. It's not selfish because there's not really anything to gain/

.33 should still be 3 months away or something -- the merge window is still open and -rc1 hasn't even come out yet.
And if it's a regression, then a patch fixing it would still be accepted outside the merge window, and basically anytime until the final release.

[yoshi314]

And if it's a regression, then a patch fixing it would still be accepted outside the merge window, and basically anytime until the final release.

waiting and not disclosing security bugs is bad.

performance regressions are not as critical, but i think not disclosing them during merge window is a fatal mistake, because it might be very difficult to revert the problematic merge.

I don't even know what to call this move. It's not selfish because there's not really anything to gain/

publicity stunt.

[mirza]

Compier-related suggestions

Here are some suggestins:

1) It would be nice to have tests on both gcc and icc (and LLVM?). Gcc-generated code quality messure over time, on real world SW, is missing.

2) Tests for different versions of gcc, inluding ancient (3.x).

3) Tests for default compiler settings vs. CPU type specific options.

[mirv]

waiting and not disclosing security bugs is bad.

performance regressions are not as critical, but i think not disclosing them during merge window is a fatal mistake, because it might be very difficult to revert the problematic merge.

publicity stunt.

I'm normally quiet about this sort of thing, but I agree with the publicity stunt opinion. And this probably isn't the place to say it, but I do think Phoronix needs to have better wording in the articles to avoid that kind of impression - if I might exaggerate a little, less of "look aren't we great, here's sensationalist news" and more "here's something interesting".
Don't get me wrong, a solid automated testing framework to discover regressions would be very useful - just the article reads too much like an opinionated blog.
Of course, feel free to disagree with me, I won't be offended.

[yoshi314]

i have to applaud phoronix for kernel test farm. this will definitely be helpful. also, we need more people concerned with linux desktop performance, as contrary to most kernel developers (as proven by Con Kolivas' experience).

the only good reason for not disclosing a performance regression is that it has to verified in more test environments.

and this probably isn't the place to say it, but I do think Phoronix needs to have better wording in the articles to avoid that kind of impression - if I might exaggerate a little, less of "look aren't we great, here's sensationalist news" and more "here's something interesting".

the one in this article sounds to me this way - "here's some interesting performance regression in the linux kernel, but you have to wait one week when our next great product comes out to learn about it." . there's pretty much a marketing decision.


phoronix should generally disclose the regression as quickly as possible. if it's real - kernel test farm will already prove a valuable tool, regardless of its release date - today, tomorrow or maybe even never (Coverity is extremely useful, but it was never released to the public)

[Michael]

This delay is just not so the public interface can be finished up but to also ensure the issue can be reproduced on other platforms and that the commit that's isolated is indeed correct. This will hopefully be all completed by Monday so it's really not much of a delay.