Announcement

Collapse
No announcement yet.

The pts/apache test is a bit misleading

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • The pts/apache test is a bit misleading

    Hi

    Some comments to the pts/apache test in PTS.

    Before looking deeper into pts/apache, I thought the test gave some indication of how many apache web server requests the machine you test on is able to serve.
    But since the "ab" program from apache is used to do the testing, and this program is not multi threaded, I think the test is only showing how many http requests a CPU core can handle (send/receive).

    The test as described on http://openbenchmarking.org/test/pts/apache
    "
    This is a test of ab, which is the Apache benchmark program. This test profile measures how many requests per second a given system can sustain when carrying out 700,000 requests with 100 requests being carried out concurrently.
    "
    So the first sentence "This is a test of ab" tells what the test is, but I have a feeling that people who look at tests, and compare systems, and see a "Apache Benchmark" number will interpret this as the maximum number of requests Apache web service is able to serve. What do you think people interpret the "Apache Benchmark" as ?


    While running the test on my new pc, I see one core with 100% CPU, which is the one running "ab", while the overall system is at about 85% idle cpu. I get about 4000/requests per second using the ab.
    So it is clearly the "ab" which is the limiting factor in the test.

    I've also tested using Apache JMeter, which is a tool for web performance testing, among other stuff, and it is multi threaded. Running that tool on a separate machine, connecting to apache on my server, I got 12000 request/s, and the web server was still at about 85% idle CPU.
    When I ran JMeter test and "ab" test at the same time, I still got the 4000/s from ab and 12000/s from JMeter. I will run JMeter on another machine as well, and see what happens then.

    I know that the web server test might not be fully CPU bound, it might also be IO bound, but since the test is only requesting the same small html file, I do not think there will be any disk I/O for it.
    But of course there is a lot of IO due to writing the access and error log files.

    I would like PTS to have an improved test for web server performance.
    I think there should be several different "runs" for the test, with different values for the "apache mpm configuration", i.e. to control how many threads are used for processing requests.

    I also think that for example JMeter should be used to run the test, and then you could have several "runs", with different number of threads used in JMeter.
    It will of course be hard to measure the "max web server performance", since the "client program hitting the web server" will be running on the same server, and consuming CPU.
    But I think it should be possible to get a better view of "max web server performance" using such an approach, compared to using "ab" as is done now.

    I will try to look into defining a test with apache web server and apache jmeter as the test driver.

    Comments are appreciated

    Regards
    Alf Hogemark

  • #2
    I believe you should think of the current ab test as a "cpu benchmark", not as a "web server benchmark". The tool just happens to be named ab = Apache Bench.

    Comment

    Working...
    X