Announcement

Collapse
No announcement yet.

Phoromatic Push vs. Pull Mechanisms

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Phoromatic Push vs. Pull Mechanisms

    I am performing scale testing against some massive Openstack environments where we create tens to hundreds of thousands of instances at a time. I want to be able to use Phoronix to generate load and perform benchmarking on these instances at the same time. I assumed Phoromatic would allow me to do this, but it looks like it wants to push to clients instead of setting up a job queue for instances to pull from. It is feasible to create a single "management" network for Phoromatic to directly communicate with the clients, but at this large scale we have issues.

    First, when we go above around 8,000 instances on a single subnet the dmsmasq process is no longer able to keep things alive and accessible due to the nature of the service keeping a flat file on disk for each network. This essentially creates a situation where the majority of instances are no longer reachable over the management network.

    Second, creating a management network for all tenants and all instances is very impractical, and does not reflect what tenants do in overlay space in any enterprise that is using multi-tenant capable data center management systems from this decade (I'm looking at you AT&T...)

    Lastly floating/public IP addresses are only available in limited quantities, and the cost of providing 1/4 million of them for a test is impractical.

    The number of processor threads required for a single instance to orchestrate test runs in this manner to run 250,000 hosts at the same time creates a situation where it takes a very long time to get everyone started, and many instances will actually complete their test runs before others have even started depending on the depth of the test package each instance is told to run, if the head node is even able to server that many clients in the first place.

    For this type of testing I would like to see Phoromatic designed in such a way that it can live somewhere on a public/floating IP address, and client instances call in to it for instrucitons from behind private networks. You would be able to create a test run for a future time, instances would call home to the head node every 5 minutes or so, download the required tests/suites for the next run, and then all execute at a pre-determined time. This could be accomplished by inserting a message queue on the phoromatic server behind the web socket. This method works great for Opscode Chef (using different mechanisms) and a single Chef instance on AWS can serve into the tens of thousands of hosts in this manner.

    I am working on a scale testing and load generating platform (Medusa) and would like to leverage Phoronix in this manner, but unless clients are able to pull the required test plans and schedules for themselves from the server instead of the server pushing them to each individual instance I probably cant use it to provide a representative scale test or generate the kinds of loads I want to see.

    You can find the Openstack Mythos project on Launchpad below if you are interested. It is in its infancy now, but I expect it to be very straightforward and comprehansive when it is completed, and my project team was selected as an alternate to present this and some other user oriented testing methods and theories at the Tokyo Openstack Summit later this month.

    Thanks.

  • #2
    Originally posted by spyderdyne View Post
    For this type of testing I would like to see Phoromatic designed in such a way that it can live somewhere on a public/floating IP address, and client instances call in to it for instrucitons from behind private networks. You would be able to create a test run for a future time, instances would call home to the head node every 5 minutes or so, download the required tests/suites for the next run, and then all execute at a pre-determined time. This could be accomplished by inserting a message queue on the phoromatic server behind the web socket. This method works great for Opscode Chef (using different mechanisms) and a single Chef instance on AWS can serve into the tens of thousands of hosts in this manner.

    I am working on a scale testing and load generating platform (Medusa) and would like to leverage Phoronix in this manner, but unless clients are able to pull the required test plans and schedules for themselves from the server instead of the server pushing them to each individual instance I probably cant use it to provide a representative scale test or generate the kinds of loads I want to see.
    Phoromatic already works like this. All the clients do is PULL and the server right now doesn't PUSH. There's been experiments with the WebSockets back-end, but right now by default, on the Phoromatic Server an HTTP server runs and all the clients connect (every minute or when necessary) and then pull everything relevant. So not sure if you just overlooked something or thought the WS back-end was used right now or what?
    Michael Larabel
    http://www.michaellarabel.com/

    Comment


    • #3
      That is awesome.

      I couldnt tell because it doesnt appear to be documented anywhere. Is there some more technical documentation than the PDF for this or do I have to read through the code?

      Thanks.

      Comment

      Working...
      X