Checksum and/or validation

[keithlm]

Michael --

Are there any plans to add a checksum and/or validation of some type to Phoronix result files?

To be more specific and add more detail to my question: I refer to a some type of checksum in the result file that is created using a "secret" method known only to a few people that will allow the checksum in the file to validate itself before the file/result can be uploaded to the Phoronix database?

(Example: Take the CRC of the file and divide it by the MD5Sum. Or add it and then divide. Or something similar... taking into account that putting the checksum in the file changes the CRC/MD5 etc. FINAL RESULT: Somebody attempting to manually change their results in the file before uploading to the Phoronix Database wouldn't know the "method" used and therefore couldn't upload the data.)

BTW: This idea came because one of the only criticisms I've seen of Phoronix-test-suite on other forums was that since it doesn't force validation that dishonest people can "doctor" their results before uploading them. (And as such the posters on those forums won't accept the results... Even if there are several people getting similar performance results.)

[Michael]

I have had some thoughts about both checksum and validation methods... Unfortunately, with PTS being open-source and written in PHP there isn't too much that can be done as far as a checksum goes, or at least what I have tried thinking of thus far... With the client being open and the communication between PTS and Phoronix Global being known, there is nothing that would stop the user from deleting the code that creates the checksum or even from them looking at the algorithm that generates the checksum. There could be some sort of validation methods but could be still easily circumvented.

The possibility for dishonest results is the reason why on Phoronix Global the test results aren't displayed at all until clicking on one of the listings, etc.

If you have any other ideas, I would enjoy hearing them since I really would like some sort of result checksumming, but I can't think of a way to implement it without it being easily circumvented.

Michael

[keithlm]

Locks only keep honest people honest. If people really want to cheat they will find ways.

=============
Anyway: just throwing out some ideas:

Add a "certified" field to the Phoronix Global. People can run tests and merge/upload all they want but if they want the "certified" then the results must be uploaded at run time or meet some other predetermined requirements that would need to be determined.

============

A more complicated method would be to keep an online keypair which is created when the test is run. The keypair would be a unique result set name and the CRC of the created file. The CRC would not be stored in the file but in the online database.

Later if the file is uploaded and/or merged it would validate against the keypair that was uploaded at the test run time.

Validating different sets of results in merged results might be more difficult. But you could probably validate each single file and then create a new keypair/checksum for the new merged file.

Of course in this case the "validation" would require being online when the test is run.

[Ex-Cyber]

What it comes down to is that you have to trust the client. There's no way to verify that the benchmark results are real, so it could just be hacked at that level no matter how much stuff is wrapped around it. If someone hacks up a shell script that reports 1337 fps and drops it in as Tremulous, what's to stop PTS from signing that with all the validation voodoo?

Locks only keep honest people honest.

A scheme like this would be less like a lock and more like a fiddly doorknob with a sign saying "please don't open this door".

[Michael]

Locks only keep honest people honest. If people really want to cheat they will find ways.

=============
Anyway: just throwing out some ideas:

Add a "certified" field to the Phoronix Global. People can run tests and merge/upload all they want but if they want the "certified" then the results must be uploaded at run time or meet some other predetermined requirements that would need to be determined.

============

A more complicated method would be to keep an online keypair which is created when the test is run. The keypair would be a unique result set name and the CRC of the created file. The CRC would not be stored in the file but in the online database.

Later if the file is uploaded and/or merged it would validate against the keypair that was uploaded at the test run time.

Validating different sets of results in merged results might be more difficult. But you could probably validate each single file and then create a new keypair/checksum for the new merged file.

Of course in this case the "validation" would require being online when the test is run.

Sadly, all of those ways still could be easily cheated with a completely open client...

I, however, have thought of the basis for a security implementation that could be found in a completely open client and would NOT be easily beaten. I would estimate a 90%+ success rate with detecting valid vs. invalid results. However, this method would be server intensive and I am not sure if the existing Phoronix Global server could handle the additional load.