Announcement

Collapse
No announcement yet.

Intergrating profiling of Mesa into PTS using OProfiler

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Intergrating profiling of Mesa into PTS using OProfiler

    Does the idea of intergrating profiling of Mesa into PTS using OProfiler sound plasable? It seems if it was possible to automate profiling with something like PTS that the job of finding and improving slow code paths in Mesa could be made a whole lot easier. For more on my idea see: http://www.itsqueeze.com/2014/05/thr...project-ideas/

  • #2
    Originally posted by tarceri View Post
    Does the idea of intergrating profiling of Mesa into PTS using OProfiler sound plasable? It seems if it was possible to automate profiling with something like PTS that the job of finding and improving slow code paths in Mesa could be made a whole lot easier. For more on my idea see: http://www.itsqueeze.com/2014/05/thr...project-ideas/
    I don't have much experience with oprofile but from what I know and my experience with utilizing other metrics during PTS testing, yes, it would be fairly easy to implement; almost within a few dozen lines of code could quite possibly be all that's needed.
    Michael Larabel
    http://www.michaellarabel.com/

    Comment


    • #3
      Originally posted by Michael View Post
      I don't have much experience with oprofile but from what I know and my experience with utilizing other metrics during PTS testing, yes, it would be fairly easy to implement; almost within a few dozen lines of code could quite possibly be all that's needed.
      Great this is good to know. Are you able to point out where in the code you would think changes would need to be added? I'm just looking for a starting point as it stands I'm not really sure how it all fits toghether so I'm kind of search for a needle in a hay stack. It would save time if you could point me in the right direction.

      Comment


      • #4
        Originally posted by tarceri View Post
        Great this is good to know. Are you able to point out where in the code you would think changes would need to be added? I'm just looking for a starting point as it stands I'm not really sure how it all fits toghether so I'm kind of search for a needle in a hay stack. It would save time if you could point me in the right direction.
        phoronix-test-suite/pts-core/modules/

        If from there you look at the timed_screenshot module that is very simple for simply showing processes to run concurrently during the actual testing process by taking a screenshot at a defined interval. There's a few other trivial modules in that directory that show for how to read the current value of the test being run, etc. Most likely you would want to create some oprofile PTS module in that directory.
        Michael Larabel
        http://www.michaellarabel.com/

        Comment


        • #5
          Originally posted by Michael View Post
          phoronix-test-suite/pts-core/modules/

          If from there you look at the timed_screenshot module that is very simple for simply showing processes to run concurrently during the actual testing process by taking a screenshot at a defined interval. There's a few other trivial modules in that directory that show for how to read the current value of the test being run, etc. Most likely you would want to create some oprofile PTS module in that directory.
          Nice. Thanks

          Comment


          • #6
            Originally posted by Michael View Post
            phoronix-test-suite/pts-core/modules/

            If from there you look at the timed_screenshot module that is very simple for simply showing processes to run concurrently during the actual testing process by taking a screenshot at a defined interval. There's a few other trivial modules in that directory that show for how to read the current value of the test being run, etc. Most likely you would want to create some oprofile PTS module in that directory.
            Ok, I've got something profiling in system-wide mode but it would be better if I could retrive the pid of the application being tested so that I can just profile that application. I will need to make a change to pts_test_execution.php to get the pid (havent worked out all the details of how to do this yet), and maybe add a new function to the module interface to be called right after the test is started (some kind of thread will be needed I'm guessing) in order for the module to kick off profiling using the retrived pid.

            Comment


            • #7
              Originally posted by tarceri View Post
              Ok, I've got something profiling in system-wide mode but it would be better if I could retrive the pid of the application being tested so that I can just profile that application. I will need to make a change to pts_test_execution.php to get the pid (havent worked out all the details of how to do this yet), and maybe add a new function to the module interface to be called right after the test is started (some kind of thread will be needed I'm guessing) in order for the module to kick off profiling using the retrived pid.
              hmmm okay, there's a few functions in PHP for being able to obtain the PID of a process. In pts_test_execution the shell_exec() might need to be reworked to use proc_open() instead and from there you can use proc_get_status() and from there should be able to read the PID as one of the elements in the returned array.

              Proper threading in PHP is a bit of a bitch and not always available; php5-pcntl is generally required and is shipped by default on most distributions but not all. It might be better to check for threading support within the module and then just have it wait from the existing API that calls right before test execution until it finds the PID placed within some file or something. If you want to fork into a separate thread, I have a template for that if you call pts_module:ts_fork_function('FUNCTION') where that function should be called asynchronously or pts_module:ts_timed_function(FUNCTION, TIME) where TIME is to keep calling that function from a separate thread every TIME seconds.

              Hope that might be of some help.
              Michael Larabel
              http://www.michaellarabel.com/

              Comment


              • #8
                Originally posted by Michael View Post
                hmmm okay, there's a few functions in PHP for being able to obtain the PID of a process. In pts_test_execution the shell_exec() might need to be reworked to use proc_open() instead and from there you can use proc_get_status() and from there should be able to read the PID as one of the elements in the returned array.
                Yeah I stumbled across proc_open() while googling. I have updated pts_client.php to use proc_open() rather than shell_exec() and have been able to retrieve the pid and call a module function from there.
                I've pushed my current work to github its still probably needs a bit more work (fyi I'm no php programmer) as I've only really tested with nexuiz so far and the way the pid is retrived seems a bit flaky. Currently the oprofile module simply triggers a single test run and profiles the entire run but I hope to (if I can find time) do some fancy stuff with the oprifle output and also try run multiple profile bursts on parts of the test were cpu peaks.
                If you are interested in what I've done so far the github branch is here: https://github.com/tarceri/phoronix-.../tree/oprofile

                Edit: When I say "its still probably needs a bit more work" I'm talking about the pts_client change. The module is a bit of a mess and needs quite a bit of work still

                Originally posted by Michael View Post
                Proper threading in PHP is a bit of a bitch and not always available; php5-pcntl is generally required and is shipped by default on most distributions but not all. It might be better to check for threading support within the module and then just have it wait from the existing API that calls right before test execution until it finds the PID placed within some file or something. If you want to fork into a separate thread, I have a template for that if you call pts_module:ts_fork_function('FUNCTION') where that function should be called asynchronously or pts_module:ts_timed_function(FUNCTION, TIME) where TIME is to keep calling that function from a separate thread every TIME seconds.

                Hope that might be of some help.
                Thanks for the tips I will probably create a modified version of pts_timed_function() to try profile cpu peaks.
                Last edited by tarceri; 05-18-2014, 12:43 AM.

                Comment


                • #9
                  Instead of trying to be hacky and timing the cpu peaks, the accurate solution would be to use oprofile's capability to record multiple events, and then present the functions ordered by "unhalted clocks in highest power state", for example.

                  Oprofile does not currently expose the power state as an event. You may be able to use some other event as a "high power state" indicator, or consider adding p-state event support to oprofile.

                  Comment


                  • #10
                    Originally posted by curaga View Post
                    Instead of trying to be hacky and timing the cpu peaks, the accurate solution would be to use oprofile's capability to record multiple events, and then present the functions ordered by "unhalted clocks in highest power state", for example.

                    Oprofile does not currently expose the power state as an event. You may be able to use some other event as a "high power state" indicator, or consider adding p-state event support to oprofile.
                    Thanks for pointing this out. I'll have to look into this further, to be honest I'm new to using oprofile so I'm not fully across all its features yet.

                    Comment


                    • #11
                      I think Josť Fonseca (vmware/apitrace) has been doing some Mesa profiling ( http://jrfonseca.googlecode.com/svn/...llium.oprofile ).

                      I've used his apitrace tool to do some very basic profiling to compare r600g with Catalyst (SwapBuffers taking 6-8 times longer with Mesa in my tests). Maybe it might be worth to identify the slow calls this way.

                      Comment


                      • #12
                        Originally posted by log0 View Post
                        I think Josť Fonseca (vmware/apitrace) has been doing some Mesa profiling ( http://jrfonseca.googlecode.com/svn/...llium.oprofile ).

                        I've used his apitrace tool to do some very basic profiling to compare r600g with Catalyst (SwapBuffers taking 6-8 times longer with Mesa in my tests). Maybe it might be worth to identify the slow calls this way.
                        Thanks I'll take a look at this too. Josť Fonseca's gprof2dot tool looks nice too it creates a good picture of whats going on and should give me a good example for processing the oprofile output.

                        Comment


                        • #13
                          tarceri:

                          Overall, the patch is looking good, one thing to point out though is that the pts_client shell_exec is called more often than just during the test execution, so the pts_module_manager::module_process('__post_test_ex ecution'); call would be inappropriate universally. That pts_client function is also called when installing tests and in a few other conditions too. Aside from that on the quick review the other code looks fine.
                          Michael Larabel
                          http://www.michaellarabel.com/

                          Comment


                          • #14
                            Originally posted by Michael View Post
                            tarceri:

                            Overall, the patch is looking good, one thing to point out though is that the pts_client shell_exec is called more often than just during the test execution, so the pts_module_manager::module_process('__post_test_ex ecution'); call would be inappropriate universally. That pts_client function is also called when installing tests and in a few other conditions too. Aside from that on the quick review the other code looks fine.
                            Yeah looks there are quite a few things I still need to work out. Anyway what ever happened to "Karsk" if you dont mind me asking?

                            Comment


                            • #15
                              Originally posted by tarceri View Post
                              Yeah looks there are quite a few things I still need to work out. Anyway what ever happened to "Karsk" if you dont mind me asking?
                              Karsk was for some enterprise research and ended up working out okay but my lack of time prevented me from polishing it up and pursuing a version for public usage.
                              Michael Larabel
                              http://www.michaellarabel.com/

                              Comment

                              Working...
                              X