Announcement

Collapse
No announcement yet.

Tests running longer on much more powerful machine

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Tests running longer on much more powerful machine

    I've recently upgraded my desktop after eight years. Moving from an i7-2600K/16GB RAM/SSD & spinning disk/GTX970 to a Ryzen 3950x/64GB RAM/PCIe4 NVME/2080 Super. (Hoping to get another long-lasting machine.) To get an idea of the performance differential, I ran the "timtaw screening-long" suite of tests (https://openbenchmarking.org/suite/t...screening-long) on the old machine before I sent it off to a loving home:

    OpenBenchmarking.org, Phoronix Test Suite, Linux benchmarking, automated benchmarking, benchmarking results, benchmarking repository, open source benchmarking, benchmarking test profiles


    I don't remember exactly how long it took to run (and I can't find the number on that page; is there a way to check?) but I'm sure it took less than 24 hours. However, I kicked off the same suite on the new box (running "phoronix-test-suite benchmark 2001299-HU-OLDBOXBAS88") at just after 11pm on Sunday Feb 23. It's now over 30 hours later and the tests are still running. (It's made it to pgbench v10.3 "Scaling: On-Disk - Test: Heavy Contention - Mode: Read Write". Still a ways to go...)

    Now, the raw numbers on the new machine are, of course, significantly better than the old ones. The test just before it, "Scaling: On-Disk - Test: Heavy Contention - Mode: Read Only" went from ~18K tps to ~100K tps. But the runtime is nothing like the estimate of "38 minutes" to run. Each individual test run (of three) is over an hour.

    In general, I'd expect benchmarks to either run faster on a more powerful machine (same amount of work, done faster) or at least the same time (more work done in X amount of time). Is there something about pgbench that makes it run longer when given more resources? At the current rate, I'm not likely to be able to use my new machine for another twelve hours or so...

  • #2
    Yes in the case of PostgreSQL on a newer/larger machine it may take longer. With PostgreSQL for the scaling it is based on the amount of RAM in the system for either using light I/O or heavy I/O based on "scaling: on-disk", so if you have a lot more RAM in the new system it could take longer based upon the CPU core / RAM capacity / storage.

    The "estimated time" is based on the per-test-profile based upon anonymous statistic reporting for all users on OpenBenchmarking.org, so the average run-time for that test generally based on thousands of samples.
    Michael Larabel
    https://www.michaellarabel.com/

    Comment


    • #3
      Originally posted by Michael View Post
      Yes in the case of PostgreSQL on a newer/larger machine it may take longer. With PostgreSQL for the scaling it is based on the amount of RAM in the system...
      And four times the RAM seems to mean well over four times the work. Awesome. The latest "Heavy Contention Read/Write" run took two hours and five minutes. If it decides to run any of the remaining ones another N=15 times, I won't see a command prompt until Wednesday or Thursday.

      Well, hopefully I won't need to run this suite again for another eight years. But when I do I guess I'll have to set aside a month for it to finish. Maybe there's a way to set some limits on how far pgbench will scale its workload?

      Comment


      • #4
        You could always just set SKIP_TESTS=pgbench environment variable prior to running that comparison But Pgbench is designed to scale to hundreds of CPU cores and hundreds of GB of RAM.
        Michael Larabel
        https://www.michaellarabel.com/

        Comment


        • #5
          Originally posted by Michael View Post
          You could always just set SKIP_TESTS=pgbench environment variable prior to running that comparison
          Guess I'll have to. In the meantime I'll chalk this one up to experience and use it as an opportunity to exercise patience.

          Is there documentation on other tests that scale their workload in a similar manner? Seems like it might be possible to calibrate/update the "Estimated Test Run-Time" based on local conditions, too.

          Comment


          • #6
            Originally posted by Ray Ingles View Post

            Guess I'll have to. In the meantime I'll chalk this one up to experience and use it as an opportunity to exercise patience.

            Is there documentation on other tests that scale their workload in a similar manner? Seems like it might be possible to calibrate/update the "Estimated Test Run-Time" based on local conditions, too.
            Pgbench is just the main one that comes to mind that uses dynamic scaling in such a noticeable manner based on CPU cores / RAM where as most of the other tests use static workloads.

            Michael Larabel
            https://www.michaellarabel.com/

            Comment


            • #7
              Well, it finished, after about 42 hours. Definitely better, but dang... single-threaded CPU performance hasn't gotten all that much better over the last eight years.

              OpenBenchmarking.org, Phoronix Test Suite, Linux benchmarking, automated benchmarking, benchmarking results, benchmarking repository, open source benchmarking, benchmarking test profiles


              Is it possible to include only a subset of the test runs? Anyway, it's fun to be able to compare all this in an automated way!

              Comment


              • #8
                Originally posted by Ray Ingles View Post
                Well, it finished, after about 42 hours. Definitely better, but dang... single-threaded CPU performance hasn't gotten all that much better over the last eight years.

                OpenBenchmarking.org, Phoronix Test Suite, Linux benchmarking, automated benchmarking, benchmarking results, benchmarking repository, open source benchmarking, benchmarking test profiles


                Is it possible to include only a subset of the test runs? Anyway, it's fun to be able to compare all this in an automated way!
                With PTS 9.4 there is a 'phoronix-test-suite run-subset' command for future reference.
                Michael Larabel
                https://www.michaellarabel.com/

                Comment


                • #9
                  Sorry, that was unclear. What I meant was, when showing the graphs. I have a couple aborted runs cluttering up the graphs a bit; it'd be nice to only display the two complete runs...

                  Comment


                  • #10
                    Originally posted by Ray Ingles View Post
                    Sorry, that was unclear. What I meant was, when showing the graphs. I have a couple aborted runs cluttering up the graphs a bit; it'd be nice to only display the two complete runs...
                    Ahhh okay, would be trivial to implement in the viewer, will add it to my TODO list.
                    Michael Larabel
                    https://www.michaellarabel.com/

                    Comment

                    Working...
                    X