Announcement

Collapse
No announcement yet.

Quick, overall system performance suite?

Collapse
This is a sticky topic.
X
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • #61
    Hi timtaw , how did you get on with this? Meindata?
    Quick screening looks good, did you ever build a test profile?

    Comment


    • #62
      Well, I'm currently waiting for some feedback from the more experienced users here. I'm still unsure about how the exact options for each test should look like (e.g. take 'fio' - buffer or no buffer? What block size is most representative?) and if some of the tests are redundant (i.e. produce very similiar results and thus if some of them can be removed) or if vital tests that I haven't taken into consideration yet are missing.

      I will build and provide the final profiles, but first I'd like to bring up a discussion on how these profiles should look like and what makes sense and what doesn't make sense.

      Hopefully users with more experience than me can chime in!

      Comment


      • #63
        Okay, here's another stab at this topic.

        In the last couple of days I tried to gain some more insight into the various tests. Which tests are widely recognized and, ideally, scientifically recognized? Which tests allow long-term comparison and scale well? I've also sampled some of the tests performed by phoronix.com during the past five years in order to see which ones are widely used.

        Taking into account and weighting all these considerations, I came up with this proposal (I've created corresponding test suites but I'm not allowed to create an attachment).

        Again, feedback is welcome!

        Superfast screening
        Very fast and trivial test in order to get a first impression of a system's performance: Processor (single-threaded and multi-threaded), Disk and Memory.

        Processor (single-threaded):
        • SciMark (Computational Test: Fast Fourier Transform): 3 minutes



        Processor (multi-threaded):
        • Himeno Benchmark: 3 minutes



        Disk:
        • Flexible IO Tester (Type: Random Read; IO Engine: POSIX AIO; Buffered: No; Direct: Yes; Block Size: 4 KB): 6 minutes
        • Flexible IO Tester (Type: Random Write; IO Engine: POSIX AIO; Buffered: No; Direct: Yes; Block Size: 4 KB): 6 minutes



        Memory:
        • RAMspeed SMP (Type: Add; Benchmark: Integer): 6 minutes



        Estimated total runtime: 24 minutes
        Approx. Download size: 0,53 MB
        Approx. installed size: 5,87 MB


        Fast screening
        Fast and representative test of all essential subsystems: Processor (single-threaded and multi-threaded), Disk, Memory and Network.

        Processor (single-threaded):
        • SciMark (Computational Test: Test All Options): 16 minutes



        Processor (multi-threaded):
        • Himeno Benchmark: 3 minutes



        Disk:
        • Flexible IO Tester (Type: Test All Options; IO Engine: POSIX AIO; Buffered: No; Direct: Yes; Block Size: 4 KB): 15 minutes



        Memory:
        • RAMspeed SMP (Type: Test All Options; Benchmark: Integer): 26 minutes



        Network:
        • Loopback TCP Network Performance: 3 minutes



        Estimated total runtime: 1:03 hours
        Approx. Download size: 0,53 MB
        Approx. installed size: 5,87 MB


        Live screening
        Representative test of all essential subsystems: Processor (single-threaded and multi-threaded), Disk, Memory and Network. This suite is able to be run from a live medium on systems with an limited amount of RAM, which means the requirements are a small download and small on-disk test size and not-too-long runtime.

        [Contains all tests from 'Fast screening' plus:]

        Processor (single-threaded):
        • FLAC Audio Encoding: 2 minutes



        Processor (multi-threaded):
        • FFTE: 4 minutes
        • ebizzy: 2 minutes
        • BLAKE2: 1 minute
        • John The Ripper (Test: Test All Options): 9 minutes
        • C-Ray: 6 minutes
        • LAME MP3 Encoding: 2 minutes
        • Gzip Compression: 4 minutes
        • Smallpt: 4 minutes
        • Stockfish: 4 minutes



        Disk:
        • PostMark: 17 minutes



        System:
        • Hierarchical INTegration (Test: Test All Options): 51 minutes



        Memory:
        • CacheBench (Test: Test All Options): 20 minutes
        • Stream (Type: Test All Options): 41 minutes



        Estimated total runtime: 3:50 hours
        Approx. Download size: 84 MB
        Approx. installed size: 30 MB


        Standard screening
        Representative test of all essential subsystems: Processor (single-threaded and multi-threaded), Disk, Memory and Network. Because of the required on-disk-size this suite is intended to be installed on a target system where download size and on-disk test size do not matter.

        [Contains all tests from 'Live screening' plus:]

        Processor (multi-threaded):
        • Dbench (Client Count: 6): 37 minutes
        • SQLite (Test Target: Default Test Directory): 24 minutes



        Disk:
        • OpenSSL: 2 minutes
        • 7-Zip Compression: 4 minutes
        • x264: 3 minutes
        • GraphicsMagick (Operation: Test All Options): 16 minutes
        • Gcrypt Library: 4 minutes
        • GnuPG: 2 minutes
        • Primesieve: 15 minutes



        System:
        • Apache Benchmark: 5 minutes
        • NGINX Benchmark: 4 minutes
        • PostgreSQL pgbench: 1:35 hours



        Estimated total runtime: 7:21 hours
        Approx. Download size: 490 MB
        Approx. installed size: 2000 MB


        Long screening
        Representative test of all essential subsystems: Processor (single-threaded and multi-threaded), Disk, Memory and Network. Because of of the required on-disk-size this suite is intended to be installed on a target system where download size, on-disk test size and runtime do not matter.

        [Contains all tests from 'Standard screening' plus:]

        Processor (multi-threaded):
        • HPC Challenge (Test: G-HPL): 52 minutes
        • NAS Parallel Benchmarks (Test / Class: Test All Options): 39 minutes
        • FFTW (Build: Test All Options; Site: 2D FFT Size 32): 54 minutes
        • High Performance Conjugate Gradient: unknown



        Disk:
        • BlogBench (Test: Test All Options): 1:04 hours
        • Iozone (Record Size: Test All Options; File Size: 4GB; Disk Test: Test All Options): 2:13 hours
        • FS-Mark (Test: Test All Options): 1:03 hours



        Estimated total runtime: 14:06 hours
        Approx. Download size: 497 MB
        Approx. installed size: 2050 MB

        Comment


        • #64
          I am happy to have found this thread. Since 13.04 I have been running 1306245-SO-CALCULATE89 before and after Ubuntu release upgrades to make sure the performance has improved or at least not decreased. The problem is that some tests in this test suite can no longer be installed, it takes quite some time and I'm not quite sure of the effectiveness.

          So I see the benefit of having a short/quick standardized test suite to verify that changes lead to improvements or at least not drawbacks. If drawbacks are found i guess a longer or more detailed standardized test suite or subset may help to pinpoint the root cause.
          I guess a short/quick standardized test suite would attract more users and then one could use openbenchmarking.org to do several interesting comparisons:
          • how does similar configurations perform?
          • how would performance change with other hardware or with a different computer?

          I guess when the short/quick standardized test suite is run for the first time on a certain configuration, the user may be requested to run the more detailed standardized test.
          Maybe with much more data it would be possible to find out which tests correlate and then select the most compatible/quick test to replace in the standardized test suite.
          Maybe then even the distros would include the standardized short/quick test as part of distribution upgrades or just periodically to find and fix regressions.

          or is anything of this all ready easily available?

          Comment


          • #65
            Originally posted by fatal-man View Post
            or is anything of this all ready easily available?
            Not that I know of. There is a plethora of benchmarks out there, each with a very different degree of reliability, scalability and trustworthiness.

            According to my research over the past few weeks the following tests are the most (1) widely recognized, (2) scalable and (3) future-proof (e.g. a certain file format may become obsolete in the future or compressing a 2 GB test file may take 30 seconds today, but a few years ahead it may only take milliseconds, so the test should not be based on such simple calculations) tests available on pts:

            Processor (multi-threaded):
            • Himeno Benchmark
            • HPC Challenge (G-HPL - this is the LINPACK that is the base for the TOP500 supercomputer list)
            • High Performance Conjugate Gradient (aims to complement the LINPACK with a new, more practical, metric)
            • FFTE
            • NAS Parallel Benchmarks (seems to be quite old; is it still relevant?)



            Processor (single-threaded):
            • SciMark



            Memory:
            • Stream
            • RAMspeed SMP
            • CacheBench



            Filesystem:
            • Flexible IO Tester
            • Iozone
            • Dbench



            System:
            • Hierarchical INTegration (Practical test that ranks a computer system as a whole, including processors, memory and buses. While almost ancient it is scalable from small serial systems to supercomputers. Almost immune to artificial optimization.)
            • ebizzy



            I totally agree with your remarks. I'm excited about the large result base that Michael created with phoronix and openbenchmarking.org. And I believe that standardized test sets would perfectly complement this existing infrastructure and benefit all. When I initially came to phoronix, my first search was for standard test sets that are representative as I lacked a deeper understanding of each available test. Which tests are important? Which tests do have a large user base so my results are as comparable as possible?
            I assume that most people want a quick way to benchmark their systems so they can compare their machines with other systems. Benchmarking as such is an own science field that most users don't want and need to be bothered with.

            Standardized test sets would not only lend a helping hand to newcomers, it would also enlarge the amount of valid test results that everybody can compare to. There are no drawbacks to that - when somebody wants to tackle a certain aspect of his system that is not covered by standard test sets, he is free to do so anyway!

            I'm currently finishing performing test runs of the test sets I posted earlier. Results look promising although there are some modifications that seem to be practical. I'll report my findings in the next few days on this thread.
            Last edited by timtaw; 06-19-2017, 09:27 AM. Reason: Corrected spelling error

            Comment

            Working...
            X