benchmarklib: a small benchmarking library

Background

Yet another C++ benchmark library I guess, but I couldn't find any that fulfilled my needs properly The requirements: output to text into a format easily parsable (for generating grahps afterwards). Low coupling, non intrusive, performant, provide basic statistics, ... .

So this one is easy to use, at first I implemented the timer using boost::posix_time::microsec_clock, only to discover that the resolution was not as good on windows. [Edit 15-02-2015: I now think the individual measures may be off a bit, but overall it does seem to average out to correct timings. I now added a boost::chrono based timer, which seems to offer good results on all platforms.] I don't understand why exactly, so I searched for a better timer (for windows) and found an example for an HRTimer [Edit 15-02-2015, forgot the link, but I found this]. So now you can choose Timer implementations ^_^

Available Timer implementations are:

• BoostTimerImpl - microseconds resolution on *nix systems, at most milliseconds on windows.
• BoostChronoTimerImpl - microseconds resolution on *nix systems and windows.
• WindowsHRTimerImpl - microseconds resolution on windows, windows specific.

Available Measure classes are:

• MeasureCounts - group individual measures per time interval.
• MeasureInterval - individual measures.

With both classes you can set the number of measures you require before the report file should be generated.

Download

Currently compiled as a static library for platform toolset Visual Studio 2012 - Windows XP (v110_xp). It is 32 bit, with flags /MD (Multi-threaded DLL).

Download link: benchmarklib-1.0.zip (md5 37ce3698a5af8b25555913f5151c85c9).

Using the Benchmark and Timer classes

Using the library

1. #include "benchmark.h"
2. Link against benchmarklib.lib

Using the MeasureCounts class

MeasureCounts counter(TimerFactory::Type::WindowsHRTimerImpl);

// Measure FPS
counter.setDescription("number of frames");
counter.setBatchSize(1000);

// Measure 30 times at 500 milliseconds interval (duration of 15 seconds)
counter.require(30);
counter.setBatchGroupSize(500);

// Results will be written to RESULT_EXAMPLE.TXT
counter.setOutput("EXAMPLE");

while ( ! counter.complete()) {

    /* do something */

    counter.measure();
}

Using the MeasureInterval class

MeasureInterval counter2(TimerFactory::Type::WindowsHRTimerImpl);

counter2.setDescription("milliseconds per frame");

// Measure 500 frames
counter2.require(500);

// Results will be written to RESULT_EXAMPLE2.TXT
counter2.setOutput("EXAMPLE2");

while ( ! counter2.complete()) {

    /* do something */

    counter2.measure();
}

Using the timer class

auto timer = TimerFactory::factory(TimerFactory::Type::WindowsHRTimerImpl);
timer->start();

/* do something */

std::cout << "Run time CPU : " << timer->end() << " milliseconds." << std::endl;

Example output (RESULT_EXAMPLE.TXT)

@@@ BENCHMARK @@@

profile "EXAMPLE"

measure "number of frames" / 1000 ms.

measure interval = 500 ms.


@@@ STATISTICS @@@

N          30
Mean       122.5427
S.E. Mean  0.2686
Std. Dev   1.4714
Variance   62.7829
Minimum    118.8119
Maximum    124.5059
Median     122.7723


@@@ HISTOGRAM @@@

                      #       
                      #       
                      #       
                      #       
                      #       
                      #       
                      #       
                #  #  #       
          #     #  #  #       
 #        #  #  #  #  #  #    
 #  #  #  #  #  #  #  #  #  # 
------------------------------
 0. 1. 2. 3. 4. 5. 6. 7. 8. 9.

0.  118.8119 <> 119.4446  =  2
1.  119.4446 <> 120.0772  =  1
2.  120.0772 <> 120.7099  =  0
3.  120.7099 <> 121.3426  =  3
4.  121.3426 <> 121.9752  =  2
5.  121.9752 <> 122.6079  =  4
6.  122.6079 <> 123.2406  =  4
7.  123.2406 <> 123.8733  =  11
8.  123.8733 <> 124.5059  =  2
9.  124.5059 <> 125.1386  =  1

Troubleshoot

Use this if your compiler does not support typed enums:

auto timer = TimerFactory::factory("WindowsHRTimer");

And for the benchmark classes as well.. (Edit: not yet supported for the Measure classes)

MeasureCounts counter("WindowsHRTimer");