It's time for the FreeBSD project to get a continuous benchmark system to test for performance regressions. At this point, we're weighing PTS vs a homegrown solution. But PTS is missing some features that we would like. We have four FreeBSD developers willing to work part time on adding them, if it can be done easily and if Phoronix would be willing to merge our changes upstream. However, we are all complete n00bs to PTS. Can some more experienced PTS developers please advise us on the ease of making these particular enhancements?
1) We'll need to add some extra benchmark programs. This looks easy.
2) We'll need to detect far more hardware and software configuration elements. Some of these (like sysctl settings) will be FreeBSD-specific and won't apply to other platforms. When viewing results, we'll want to be able to query by these elements. For numeric elements, like the kernel version of the amount of installed RAM, we'll want to be able to query a range.
3) We'll need more graph types. In particular, we need to be able to do a scatter or line graph of one or more results against another variable, like kernel version.
4) We need to incorporate networking. AFAICT, PTS benchmarks are restricted to a single host. But networking services like nfsd and pf are very imporant to FreeBSD users, so any benchmark suite must be able to exercise them. This means we'll need a small cluster of slave nodes to generate load for the machine under test. We'll probably have to record the slaves' hardware and software configurations, and we'll need PTS to command and control them. We'll also need an exclusive reservation system, so multiple systems running PTS don't try to use the same clients and the same time. Is this soft of functionality on the roadmap for PTS? Is it even possible with PTS's architecture?
1) We'll need to add some extra benchmark programs. This looks easy.
2) We'll need to detect far more hardware and software configuration elements. Some of these (like sysctl settings) will be FreeBSD-specific and won't apply to other platforms. When viewing results, we'll want to be able to query by these elements. For numeric elements, like the kernel version of the amount of installed RAM, we'll want to be able to query a range.
3) We'll need more graph types. In particular, we need to be able to do a scatter or line graph of one or more results against another variable, like kernel version.
4) We need to incorporate networking. AFAICT, PTS benchmarks are restricted to a single host. But networking services like nfsd and pf are very imporant to FreeBSD users, so any benchmark suite must be able to exercise them. This means we'll need a small cluster of slave nodes to generate load for the machine under test. We'll probably have to record the slaves' hardware and software configurations, and we'll need PTS to command and control them. We'll also need an exclusive reservation system, so multiple systems running PTS don't try to use the same clients and the same time. Is this soft of functionality on the roadmap for PTS? Is it even possible with PTS's architecture?
Comment