criterion performance measurements

overview

State

	lower bound	estimate	upper bound
OLS regression	xxx	xxx	xxx
R² goodness-of-fit	xxx	xxx	xxx
Mean execution time	1.6791487133929636e-2	1.697416438479176e-2	1.716919097509231e-2
Standard deviation	3.627207595284326e-4	4.627260349117218e-4	5.720056786936559e-4

Outlying measurements have slight (8.004417610262879e-2%) effect on estimated standard deviation.

IORef

	lower bound	estimate	upper bound
OLS regression	xxx	xxx	xxx
R² goodness-of-fit	xxx	xxx	xxx
Mean execution time	8.58231040994432e-4	8.702263604572573e-4	8.827930883579915e-4
Standard deviation	3.576087539630157e-5	4.214309320254753e-5	5.0904587934349635e-5

Outlying measurements have moderate (0.39554525006899494%) effect on estimated standard deviation.

IORef(atomic)

	lower bound	estimate	upper bound
OLS regression	xxx	xxx	xxx
R² goodness-of-fit	xxx	xxx	xxx
Mean execution time	2.7434138781961873e-3	2.7827556676557072e-3	2.8343226013761652e-3
Standard deviation	1.2109129275312054e-4	1.491837052098112e-4	1.9596226414008268e-4

Outlying measurements have moderate (0.3577537094820758%) effect on estimated standard deviation.

mvar

	lower bound	estimate	upper bound
OLS regression	xxx	xxx	xxx
R² goodness-of-fit	xxx	xxx	xxx
Mean execution time	3.833062687144345e-3	3.8780427871184337e-3	3.9383938943527395e-3
Standard deviation	1.329784990345649e-4	1.6571312307181837e-4	2.1066327635436565e-4

Outlying measurements have moderate (0.2330818045447426%) effect on estimated standard deviation.

In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.

The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.

Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.

OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.

We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.