Announcement

Collapse
No announcement yet.

Frame latency analysis on Doom 3

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Frame latency analysis on Doom 3

    As some of you are aware, The Tech Report have been using a new type of analysis on FPS to analyse frame latencies (see http://techreport.com/review/21516/i...e-benchmarking).

    To do this yourself I have devised a methodology (which could be automated) using Doom 3.

    Step 1:
    Record frame timings (I have used the command from the OpenBenchmarking Doom3 Test Profile). The important bit is the com_speeds flag:

    Code:
    ./doom3 +exec doom3-pts.cfg +set sys_VideoRam 512MB +set r_mode -1 +timedemoquit demo1 +set com_speeds 1 > doom3.log
    Step 2:
    Parse log file using this Python script:
    Code:
    import re
    
    # Compile reg exp
    re1='.*?'	    # Non-greedy match on filler
    re2='(\\d+)'	# Integer Number 1
    r = re.compile(re1+re2+re1+re2+re1+re2+re1+re2+re1+re2,re.IGNORECASE|re.DOTALL)
    
    # Open log file
    frames = []
    f = open('doom3.log')
    for line in f.readlines():
        if line.startswith('frame:'):
            m = r.search(line)
            if m:
                frames.append( {'com_frameNumber': int(m.group(1)),
                                'com_frameMsec':   int(m.group(2)), 
                                'time_gameFrame':  int(m.group(3)), 
                                'time_frontend':   int(m.group(4)), 
                                'time_backend':    int(m.group(5))} )
    f.close()
    
    # Write to data file
    f = open('data.csv', 'w')
    f.write('Frame,FrameTime\n')
    for frame in frames:
        # Skip first frame
        if frame['com_frameNumber'] == 0:
            continue
        f.write('%i,%i\n' % (frame['com_frameNumber'], frame['com_frameMsec']))
    f.close()
    Step 3:
    Use this R script on resulting csv file:
    Code:
    dat <- read.csv("data.csv",header=T,colClasses=c("integer","numeric"))
    summary(dat)
    
    # Progression of frame times
    plot(dat, type='n', xlab="Frame number", 
        ylab="Frame time in ms (lower is better)")
    lines(dat)
    
    # Average fps
    max(dat$Frame)*1000/sum(dat$FrameTime)
    
    # Percentiles
    # It's the point below which 99% of all frames have been rendered. 
    # We're simply excluding the last 1% of frames, many of them potential outliers, 
    # to get a sense of overall smoothness. 
    quantile(dat$FrameTime, .99) 
    
    # Frame latencies by percentile
    p <- seq(0, 1, length.out=1000)^(1/3)
    quan <- data.frame(q = quantile(dat$FrameTime, probs = p), prob = p)
    plot(quan$prob, quan$q, type='n', xaxt='n', xlim=c(0.5,1),
      xlab="Proportion of frames rendered", ylab="Frame time in ms (lower is better)")
    axis(1, at=seq(0,1,by=.05), labels=paste(100*seq(0,1,by=.05), "%") )
    lines(quan$prob, quan$q)
    
    # Time spent beyond 50 ms
    sum(subset(dat,FrameTime>50)$FrameTime)
    
    # Time spent beyond 16.7 ms
    sum(subset(dat,FrameTime>16.7)$FrameTime)

    Example of resulting images:



  • #2
    I wasn't aware of com_speeds and thanks for the script as reference, I'll take a look into it for supporting test profiles with com_speeds and then having PTS-internal infrastructure for handling that parsing and shoving it into line graph.
    Michael Larabel
    http://www.michaellarabel.com/

    Comment


    • #3
      Not only the graphs are important but also the values which are calculated in the R script:

      - Average fps (not important for latency analysis)
      - 99% Percentile
      - Time spent beyond 50 ms
      - Time spent beyond 16.7 ms

      PS. Please note that the frametime for the first frame was so high, I ignored it (3.5 seconds)
      Last edited by thofke; 11-05-2012, 09:59 AM.

      Comment


      • #4
        Very nice. Looking forward to seeing latency articles.

        Comment


        • #5
          Note that this does not measure input-to-output latency, but the latency between consecutive frames.

          Maybe it could be called jitter analysis, as you analyse the undesired deviation from a certain periodic frame rate.

          This article describes better than the previous mentioned article what is possible using this method: http://techreport.com/review/23246/i...ith-today-cpus
          Last edited by thofke; 11-05-2012, 01:47 PM.

          Comment


          • #6
            Originally posted by thofke View Post
            Note that this does not measure input-to-output latency, but the latency between consecutive frames.

            Maybe it could be called jitter analysis, as you analyse the undesired deviation from a certain periodic frame rate.

            This article describes better than the previous mentioned article what is possible using this method: http://techreport.com/review/23246/i...ith-today-cpus
            Yes yes yes; THIS is how benchmarking is supposed to be done. That was one of Techreports best articles ever.

            But its a bit worrying that an older game like Doom3 has so many latency spikes; eyeballing the graph, the average frame latency looks to be about 20ms or so, indicating significant frame skipping...

            Comment


            • #7
              Now we can compare.

              Average fps:
              42.37472 (me)
              61.15589 (you)


              Green is your machine, red is mine.


              Striking here is that you have a higher average fps, but my machine has a lower frame time at the far right of the graph.

              That is also visible in the 99th percentile number:
              99% of your frames are drawn in 52.08ms or less, whereas my machine has drawn 99% of the frames in 42ms or less. I would say that my machine has less jitter.

              Comment


              • #8
                That means that every frame is taking 1/60 seconds (or 16.7ms). If you never experience a stuttering frame (i.e. higher than 16.7ms per frame), then maybe the demo is not a good benchmark or maybe is the demo very hard on resources?

                Comment


                • #9
                  Most likely the jitter is in your brain. Why don't you record one and compare?

                  Comment


                  • #10
                    Originally posted by Kano View Post
                    Most likely the jitter is in your brain. Why don't you record one and compare?
                    Why would you make such a non-constructive comment? If you do not have anything to add, please refrain from commenting.

                    Originally posted by Wikipedia
                    Jitter is the undesired deviation from true periodicity of an assumed periodic signal in electronics and telecommunications, often in relation to a reference clock source. Jitter may be observed in characteristics such as the frequency of successive pulses, the signal amplitude, or phase of periodic signals.
                    If you look at the frame rate of a game as a periodic signal or as successive pulses from the previous definition, any deviation from that periodicity could be called jitter.
                    Last edited by thofke; 11-06-2012, 05:11 AM.

                    Comment

                    Working...
                    X