Specifying the result file and test identifier non-interactively

[tytso]

I want to be able to run "phoronix-test-suite batch-run local/fs-benchmarks" from a shell script, and I want to specify the result file and test identifier from either a command-line option, or an evironment variable, or by setting some magical config file.

I haven't been able to find a way to do that; is there such a way?

Thanks!

[Michael]

I want to be able to run "phoronix-test-suite batch-run local/fs-benchmarks" from a shell script, and I want to specify the result file and test identifier from either a command-line option, or an evironment variable, or by setting some magical config file.

I haven't been able to find a way to do that; is there such a way?

Thanks!

Yep, see environment variables of TEST_RESULTS_IDENTIFIER / TEST_RESULTS_NAME that should achieve the desired behavior. Documentation for details - http://phoronix-test-suite.com/docum...test-suite.pdf

Let me know if hitting any issues but there should be the env var coverage for all of the possible knobs.

[tytso]

Thanks, that's really helpful. One other question. I'm looking to add to disk details the ext4 file system features that are enabled, since the primary use case I have in mind is benchmarking file system differences (for example, when enabling the new ext4 fast_commit feature):

Dbench 4.0
12 Clients
MB/s > Higher Is Better
no-fast-commit . 607.75 |=============================
fast-commit .... 1050.18 |================================================= =

I think it would be better to add this a new disk property, as opposed to just wedging it into the "other disk details" property, which seems to be currently focused on RAID settings. Would you have an objection to doing this? Or would it better to jam it into other disk details if the file system type is ext4 (in which case we'd lose the RAID information when benchmarking ext4)?

Similarly, what is your opinion about how to record the VM configuration details when running on a Google Compute Engine VM? I could overload motherboard to be something like n2-standard-16, or we could add a new property which is specific to the GCE or AWS VM type.

I'm also interested in saving some additional system logs; for example, /proc/fs/ext4/<block_device>/options, and the contents of /proc/lock_stat after the test runs are completed (if the file is present, meaning that locking statistics are enabled). Are these changes you would be open to, or should I keep them as out-of-tree patches?

I'm currently launching PTS as part of "gce-xfstests pts", which is still fairly manual. My eventual goal is to pass in the test suite to run (I have a local fs-benchmarks test suite which is focused on file system benchmarking, and which skips certain tests that are primarily more about measuring disk speeds as opposed to benchmarking the file system), and to have the VM kexec into successive kernel versions and/or with the PTS disk configured with different file system features, so I don't have to run the tests manually. (gce-xfstests is a test automation framework which is optimized for file system developers. For more info, see https://thunk.org/gce-xfstests )

[Michael]

If you have an example command for the information you would like displayed when running on EXT4, I can work on getting it integrated into the PTS side. My initial feeling would be to just add it as a new listing under the Disk Details section but will give it more thought today.

Regarding the VM details, perhaps appending it to the 'System Layer' might make sense? Right now it should be reporting Xen, KVM, or other details. Thinking perhaps appending it there would work well. If there is a suitable means for reliably querying the cloud details, should be fairly straight forward for adding it there. Otherwise could add a new "Virtualization Details" in the footnotes.

[tytso]

The file system features can be extracted from "dumpe2fs -h <block device>" for all ext4 file systems. For that matter, saving the full output of "dumpe2fs -h" to system logs, which has quite a few other file system config parameter, would be handy. Also perhaps useful enough to extract out to file system info is the file system block size and/or cluster size. Those can be just as important as the file system feature in use. BTW, in addition to fetching the block size from dumpe2fs, you can get it in a file system independent, if it is mounted, via:

% stat -f -c %S /mnt
4096

The GCE machine type can be fetched from http://metadata.google.internal/comp...e/machine-type. If the hostname metadata.google.internal resolves, that's one way you can know that you are running on GCE. You can also figure this out by looking at the output of dmidecode (another useful thing to perhaps save in system-logs). There is some fancy retry logic when reading from metadata which I've implemented here[1]. For more information on the sort of things that can be fetched from the VM metadata, see [2].

[1] https://github.com/tytso/xfstests-bl.../lib/gce-funcs
[2] https://cloud.google.com/compute/doc...eving-metadata

Example:

root@pts:~# . /usr/local/lib/gce-funcs
root@pts:~# echo $(get_metadata_value machine-type)
projects/687721678153/machineTypes/n1-standard-16

You'll want to strip out everything before the last '/', e.g.:

root@pts:~# echo $(basename $(get_metadata_value machine-type))
n1-standard-16

Oh, one other thing that would be super-useful --- /proc/config.gz if it is present. That's because some kernel config options --- in particular lockdep and lock_stat --- have huge impacts on performance. So when validating test results to make sure all of the benchmarks were run with locking debugging/stats enabled --- or not enabled --- would be really helpful. I recently had to redo all of my ext4 benchmarks for v4.14, v4.19,, v5.4, v5.10-rcX because I wasn't sure the configs were comparable or not. And if I'm looking at results downloaded from openbenchmarking.org it would be helpful for me to confirm exactly how the kernel was compiled. And as I mentioned earlier, saving the contents of /proc/lock_stat (if present) *after* the benchmarks have been completed to system-logs would be super-useful for me.

Thanks!!

[tytso]

Oops, please use get_metadata_value_with_retries above:

root@pts:~# echo $(basename $(get_metadata_value_with_retries machine-type))
n1-standard-16

[Michael]

Thanks tytso the latest PTS Git should now log lock_stat, config.gz, and dmidecode as part of the system-logs collection. And, yes, the logs are captured after all tests are run.

There might be an issue when uploading results with config.gz to OpenBenchmarking.org since I believe I have checks there to ensure the logs are just plain text files. So looking at that now and may just make more robust kernel config file handling since the archived logs on OB are already compressed anyhow, so may have a better solution there this afternoon.

Taking a look as well at the improved cloud details, thanks.

[Michael]

Also with latest Git the /boot/config-[kernel] config will be archived if /proc/config.gz not present. Also tweaked it for storage constraints to drop comment lines and drop CONFIG_ prefixes to make it more manageable on the OB storage side.

[Michael]

tytso within latest PTS Git -

- The GCE instance type is now reported to the system table. I ended up doing this as appended to the "Google Compute Engine" motherboard string, since that is similar to how Amazon EC2 ends up exposing their instance type as part of the sysfs DMI information and thus how PTS ends up reporting it, so will be at least uniform for GCE/EC2 and was going to leverage the "Google Compute Engine" motherboard anyhow for determining when to query the Google metadata.

- The dumpe2fs output should now be saved to the system logs when PTS is running as root and the file-system where the tests are being installed to is EXT4.

- The block size should be reported as a note on the system table at the bottom if running a disk test as part of the benchmarks.

Continuing to play around with other tweaks and improvements, let me know if anything else comes to mind...

e.g. on clean GCE instance

system table layout at the moment - https://openbenchmarking.org/result/...FI-GCEQUERYN56

shortened kernel config display, dumpe2fs, etc - https://openbenchmarking.org/system/...ERYN56/Run%202

[tytso]

Great, thanks! I'll take a look in the next day or two, once I have a chance to build a new test appliance image which uses the PTS from a git tree.

In my case, I'll never have the /boot/config-$KVER present because I'm using kexec to boot the kernel under test. My plan is to have a list of kernel images stored in GCS, and I'll have the automation framework download each kernel in turn, kexec into that kernel, run the PTS test suite, and then kexec into the next kernel on the list. (I want to use the same GCE Persistent Disk to try to minimize variation, as opposed to simply launching multiple VM's with separate PD block devices and then merging the results afterwards.)