FreeBSD 8.0 vs. Ubuntu 9.10 Benchmarks

[ivoras]

Statistical analysis

Some statistical analysis would be nice to have in the future. Here's a short description of a simple tool with which to do it.

[Kano]

I did not look at the code, but maybe it executes the external md5sum tool.

[risner]

Well, I'm about to give up.

I've just spend 8 hours trying to get Phoronix running in FreeBSD.

Running the install-sh program puts everything in /usr/bin and /usr/share/phoro* but doesn't make a pts / pts-core directory, so I need to cp those from the install directory to /usr/share/phor*/ before the program will think it is installed.

Once that is past, if I try to run an install, it says the checksum fails, but I can download the file and the checksum matches.

The only way I think I could manage to do these tests is to do them by hand and not use Phoronix-test-suite. Which I really don't have the time to do all these tests by hand. I've already spent close to 16 hours installing/prepping this.

Errors that I can't debug:
test# tar -xf phoronix-test-suite-2.0.0.tar
test# cd phoronix-test-suite
test# ./install-sh

Phoronix Test Suite Installation Completed

Executable File: /usr/bin/phoronix-test-suite
Documentation: /usr/share/doc/phoronix-test-suite/
Phoronix Test Suite Files: /usr/share/phoronix-test-suite/

test# phoronix-test-suite install compress-7zip

To run the Phoronix Test Suite locally you must first change directories to phoronix-test-suite/
or install the program using the install-sh script. For support visit: http://www.phoronix-test-suite.com/

test# cp -r pts pts-core /usr/share/phoronix-test-suite/
test# phoronix-test-suite install compress-7zip
dcop: not found

====================================
PHORONIX TEST SUITE - WELCOME
====================================

The Phoronix Test Suite is the most comprehensive testing and
...
Do you agree to these terms and wish to proceed (Y/n)? y


Checking For Needed External Dependencies.

====================================
Downloading Files For: compress-7zip
Estimated Download Size: 2.38 MB
====================================



Downloading File: p7zip_4.65_src_all.tar.bz2


The MD5 check-sum of the downloaded file is incorrect.
Failed URL: http://voxel.dl.sourceforge.net/sour...rc_all.tar.bz2
Attempting to re-download from another mirror.


Downloading File: p7zip_4.65_src_all.tar.bz2


The MD5 check-sum of the downloaded file is incorrect.
Failed URL: http://internode.dl.sourceforge.net/...rc_all.tar.bz2
Attempting to re-download from another mirror.
...
Download of Needed Test Dependencies Failed! Exiting.


Installation of compress-7zip test failed.

test# fetch http://ovh.dl.sourceforge.net/source...rc_all.tar.bz2
p7zip_4.65_src_all.tar.bz2 100% of 2442 kB 957 kBps
test# md5 p7zip_4.65_src_all.tar.bz2
MD5 (p7zip_4.65_src_all.tar.bz2) = f78ed232436dc8f8be25a2f95d4e5f9a
test# ls -l p7zip_4.65_src_all.tar.bz2
-rw-r--r-- 1 root 1000 2501418 Feb 14 2009 p7zip_4.65_src_all.tar.bz2
test# cat /usr/share/phoronix-test-suite/test-resources/compress-7zip/downloads.xml
<PhoronixTestSuite>
<Downloads>
<Package>
<URL>http://internode.dl.sourceforge.net/...rc_all.tar.bz2, http://ovh.dl.sourceforge.net/source...rc_all.tar.bz2, http://voxel.dl.sourceforge.net/sour...rc_all.tar.bz2, http://superb-east.dl.sourceforge.net/sourceforge/p7zip/p7zip_4.65_src_all.tar.bz2</URL>
<MD5>f78ed232436dc8f8be25a2f95d4e5f9a</MD5>
<FileSize>2501418</FileSize>
</Package>
</Downloads>
</PhoronixTestSuite>
test# cat /tmp/w.php
<?php
$md5file = md5_file("p7zip_4.65_src_all.tar.bz2");
file_put_contents("md5file.txt",$md5file);
?>
test# php /tmp/w.php

test# cat md5file.txt
f78ed232436dc8f8be25a2f95d4e5f9atest#
test#

[kraftman]

Why 250 Hz? To be more server oriented? If yes then he should enable and the "no forced preemption" option in the proccesor type and features and also to choose the correct type of processor.

Yep, it's noticeable in the test I gave a link. :>

http://unix.derkeiler.com/Mailing-Li.../msg00073.html - it's buried there somewhere

[Apopas]

I do not have enough knowledge to help you. For Ubuntu I would just turn off debugging, chose 250Hz option, disable some services, because sometimes they can 'eat' up to 100% cpu (afaik some package management related)...

Why 250 Hz? To be more server oriented? If yes then he should enable and the "no forced preemption" option in the proccesor type and features and also to choose the correct type of processor.

[kraftman]

No one here willing/able to help me? If not, I'll have to abandon running a new series of tests.

I've got the hardware and can run the test on the bare metal, I've got the time to do the OS installs and the optimizations I'm able to do (FreeBSD), but I don't have the time to learn how to run the phoronix-test-suite.

While I am responsible for a number of Linux systems (mostly RHEL and Gentoo), I wouldn't consider myself an expert on optimizing Linux.

If no one has the time or desire to help, that is fine. I understand. If you do have it, please help with these:
1) Install and Optimize Ubuntu on the hardware.
2) Run the Phoronix Test Suite with the same options as the original test.

I don't know how Michael Larabel ran it originally (what options.) I also can't get it to run on FreeBSD. It installs (with install-sh) but then errors out immediately with this:

To run the Phoronix Test Suite locally you must first change directories to phoronix-test-suite/
or install the program using the install-sh script. For support visit: http://www.phoronix-test-suite.com/

I'm running it from the directory and /usr/bin/ is where the suite binary is located.

If I can't get help soon, I'll need to put the new box into production middle of next week. If in production, it will have 8 production VM's on it which would make it worthless for any meaningful test.

I do not have enough knowledge to help you. For Ubuntu I would just turn off debugging, chose 250Hz option, disable some services, because sometimes they can 'eat' up to 100% cpu (afaik some package management related) and if you plan to optimize FreeBSD as much as possible I would also use writeback mode and Ext4 file system in Ubuntu. The best option would be probably to ask at lkml about how to optimize it

turning on writeback:

http://bbs.archlinux.org/viewtopic.php?pid=622697

I did not run PTS on FreeBSD, but on some Linux distros if PTS is already installed (if not I just cd to its directory and do ./phoronix-test-suite [...]):

phoronix-test-suite list-tests - to show available tests,
phoronix-test-suite install [test_name]
phoronix-test-suite benchmark [test_name] no special options are required afaik.

To make it running in Arch I had to change some option in php like described in this thread:

http://bbs.archlinux.org/viewtopic.php?id=46504


However, it seems there's nothing wrong with the results:

http://unix.derkeiler.com/Mailing-Li.../msg00073.html

[risner]

No one here willing/able to help me? If not, I'll have to abandon running a new series of tests.

I've got the hardware and can run the test on the bare metal, I've got the time to do the OS installs and the optimizations I'm able to do (FreeBSD), but I don't have the time to learn how to run the phoronix-test-suite.

While I am responsible for a number of Linux systems (mostly RHEL and Gentoo), I wouldn't consider myself an expert on optimizing Linux.

If no one has the time or desire to help, that is fine. I understand. If you do have it, please help with these:
1) Install and Optimize Ubuntu on the hardware.
2) Run the Phoronix Test Suite with the same options as the original test.

I don't know how Michael Larabel ran it originally (what options.) I also can't get it to run on FreeBSD. It installs (with install-sh) but then errors out immediately with this:

To run the Phoronix Test Suite locally you must first change directories to phoronix-test-suite/
or install the program using the install-sh script. For support visit: http://www.phoronix-test-suite.com/

I'm running it from the directory and /usr/bin/ is where the suite binary is located.

If I can't get help soon, I'll need to put the new box into production middle of next week. If in production, it will have 8 production VM's on it which would make it worthless for any meaningful test.

[clau]

Without context, you can't make that decision. There may be any number of scenarios that would dictate that sort of configuration. (For example, 3 server application licenses that require 3 OSes, but will be within the capacity of a single system). In that case you want to have the app in the right spot and/or VM. You then choose benchmarks that represent your application.

The intent of the benchmarking should be made clear, but without it you can't declare a benchmark invalid.

With the intent you can call into question two items
- the analysis and consequent construction of the testing.
- the analysis and conclusions of the testing.

Regards,

Matthew

I do agree with you, but I thought the context is rather clear, this being a thread for discussions about the results of the Phoronix's benchmarks.
Besides, if you want to compare the performance of 2 or more OSes, my guess is that you'd want to have the least possible interferences with the running OS.

[clau]

I was according to this:

http://people.freebsd.org/~kris/scaling/os-mysql.png

If 2.6.22 was tested here and there (also in benchmark I posted) I wonder what they base on saying some scaling problems were fixed in 2.6.22?

Ok, now I know:

http://jeffr-tech.livejournal.com/5705.html

There's gnu's malloc used and that's why 2.6.21* drops down.

And with Google's malloc:

http://ozlabs.org/~anton/linux/sysbench/

It could be like this. However, there's/was known problem with GNU's malloc library and I wonder what library was used (it seems it was used). There's also 2.6.22 test which I'm according to.

That's why I'd really like to see a new benchmark. Unfortunately I simply don't have the HW necessary for it at my disposal to do it myself.

[mtippett]

Comparing a benchmark ran in a VM with one ran in the host OS is completely pointless.
If you really want to do the benchmark yourself, you'd have to install both OSes directly on the hardware, no VM involved.

Without context, you can't make that decision. There may be any number of scenarios that would dictate that sort of configuration. (For example, 3 server application licenses that require 3 OSes, but will be within the capacity of a single system). In that case you want to have the app in the right spot and/or VM. You then choose benchmarks that represent your application.

The intent of the benchmarking should be made clear, but without it you can't declare a benchmark invalid.

With the intent you can call into question two items
- the analysis and consequent construction of the testing.
- the analysis and conclusions of the testing.

Regards,

Matthew