criterion performance measurements
overview
want to understand this report?
bytestring-read/short
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.7162801383115504e-7 | 4.769204655525962e-7 | 4.824817168885289e-7 |
| Standard deviation | 1.4440258334442982e-8 | 1.7961791748944985e-8 | 2.4622338983753537e-8 |
Outlying measurements have severe (0.5423715737334821%) effect on estimated standard deviation.
bytestring-read/long
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.2824232777865205e-6 | 1.309064283918206e-6 | 1.3313456049416185e-6 |
| Standard deviation | 6.835369784603116e-8 | 8.197904945920968e-8 | 1.1132575375994883e-7 |
Outlying measurements have severe (0.7509019367863308%) effect on estimated standard deviation.
bytestring-read(Lazy)/short
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.720538633606248e-7 | 4.766441322710024e-7 | 4.809489667663176e-7 |
| Standard deviation | 1.2472479919920693e-8 | 1.477374499552161e-8 | 1.8217749617819612e-8 |
Outlying measurements have moderate (0.44493756224885644%) effect on estimated standard deviation.
bytestring-read(Lazy)/long
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.3804265648109332e-6 | 1.39110634815362e-6 | 1.403287788638091e-6 |
| Standard deviation | 3.353693109640881e-8 | 3.926755408804221e-8 | 4.772243516668513e-8 |
Outlying measurements have moderate (0.37283917094986535%) effect on estimated standard deviation.
text/short
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.3878335323555526e-6 | 2.413706786810988e-6 | 2.429966635282845e-6 |
| Standard deviation | 4.386079407165844e-8 | 6.501160046379726e-8 | 1.1208279802972513e-7 |
Outlying measurements have moderate (0.33847830644093246%) effect on estimated standard deviation.
text/long
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.387946895109051e-5 | 2.4022854915991557e-5 | 2.4135753480789004e-5 |
| Standard deviation | 3.5000812903327943e-7 | 4.5126562380115054e-7 | 6.289332543022445e-7 |
Outlying measurements have moderate (0.1561881895590054%) effect on estimated standard deviation.
bytestring-lexing/short
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.63333521681301e-7 | 5.687934808419892e-7 | 5.746259757782843e-7 |
| Standard deviation | 1.6806849174816527e-8 | 1.967779346048279e-8 | 2.412419894619576e-8 |
Outlying measurements have moderate (0.4959114622123787%) effect on estimated standard deviation.
bytestring-lexing/long
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.576230001179116e-6 | 4.604762768607262e-6 | 4.63583807978982e-6 |
| Standard deviation | 7.747696358915707e-8 | 9.965712060155399e-8 | 1.2135331081571904e-7 |
Outlying measurements have moderate (0.23337041394013625%) effect on estimated standard deviation.
attoparsec/short
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.890083945927189e-6 | 5.949399177086641e-6 | 5.994201658095431e-6 |
| Standard deviation | 1.0195899821677136e-7 | 1.7625716794431162e-7 | 3.084325581610089e-7 |
Outlying measurements have moderate (0.36267359470690586%) effect on estimated standard deviation.
attoparsec/long
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.3393914645395654e-5 | 3.3590482716960386e-5 | 3.407035303137176e-5 |
| Standard deviation | 3.654433830820784e-7 | 9.554850257398814e-7 | 1.8555518230297776e-6 |
Outlying measurements have moderate (0.28947419475273783%) effect on estimated standard deviation.
read/short
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.987691536156147e-5 | 2.0034531430551893e-5 | 2.0254588774120766e-5 |
| Standard deviation | 4.41198752957701e-7 | 6.416330130285746e-7 | 8.697192233478371e-7 |
Outlying measurements have moderate (0.3591784178689366%) effect on estimated standard deviation.
read/long
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.01920330019204e-5 | 7.133864838016226e-5 | 7.311702396601722e-5 |
| Standard deviation | 2.9922454338168035e-6 | 4.997445319329486e-6 | 7.468833629637481e-6 |
Outlying measurements have severe (0.696745173080583%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.