criterion performance measurements
overview
want to understand this report?
bytestring-read/short
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.1396008220461417e-7 | 3.189484987655596e-7 | 3.267179134904764e-7 |
| Standard deviation | 1.5330876022516912e-8 | 2.1402561101793328e-8 | 2.9609288949899298e-8 |
Outlying measurements have severe (0.7988582835566432%) effect on estimated standard deviation.
bytestring-read/long
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.905791826528533e-7 | 8.007823575127015e-7 | 8.154883171934045e-7 |
| Standard deviation | 3.27909317658136e-8 | 4.1504569845528264e-8 | 5.785818832664791e-8 |
Outlying measurements have severe (0.6859142578154056%) effect on estimated standard deviation.
bytestring-read(Lazy)/short
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.487125348813807e-7 | 3.5366496156979684e-7 | 3.596248468466595e-7 |
| Standard deviation | 1.397742927595958e-8 | 1.864174781352251e-8 | 2.7480247707901868e-8 |
Outlying measurements have severe (0.7084818369097999%) effect on estimated standard deviation.
bytestring-read(Lazy)/long
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.538926249325165e-7 | 8.660879447608077e-7 | 8.834929520604016e-7 |
| Standard deviation | 3.302292322340885e-8 | 4.785423842418233e-8 | 7.380672483318267e-8 |
Outlying measurements have severe (0.7117524879820342%) effect on estimated standard deviation.
text/short
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5680126495200317e-6 | 1.5914702084801678e-6 | 1.6221243441662208e-6 |
| Standard deviation | 7.224679233269858e-8 | 9.105514852392167e-8 | 1.1300617840626193e-7 |
Outlying measurements have severe (0.7104281712516254%) effect on estimated standard deviation.
text/long
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.582545550681814e-5 | 1.5968123421088278e-5 | 1.6199754063956735e-5 |
| Standard deviation | 4.5015213769702427e-7 | 5.956381712636622e-7 | 9.217641029335763e-7 |
Outlying measurements have moderate (0.43947949968845335%) effect on estimated standard deviation.
bytestring-lexing/short
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1618298747761865e-6 | 1.1771914947868006e-6 | 1.1965925392858362e-6 |
| Standard deviation | 4.4693266297173854e-8 | 5.903754687232171e-8 | 7.617653135673274e-8 |
Outlying measurements have severe (0.659546115528433%) effect on estimated standard deviation.
bytestring-lexing/long
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.584600737196397e-6 | 7.69312404679779e-6 | 7.892418875252297e-6 |
| Standard deviation | 3.3980758954411386e-7 | 4.903427820138112e-7 | 7.671234581286082e-7 |
Outlying measurements have severe (0.7230620694155536%) effect on estimated standard deviation.
attoparsec/short
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.316632532494751e-6 | 4.356034585485529e-6 | 4.419744580744338e-6 |
| Standard deviation | 1.3062595007658229e-7 | 1.6669920705112044e-7 | 2.417222498388673e-7 |
Outlying measurements have moderate (0.49305814764564504%) effect on estimated standard deviation.
attoparsec/long
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.2872349527876183e-5 | 2.3148560927181788e-5 | 2.3544393111501485e-5 |
| Standard deviation | 7.712507422648525e-7 | 1.1313845090124473e-6 | 1.5477401317409832e-6 |
Outlying measurements have severe (0.5633536742245129%) effect on estimated standard deviation.
read/short
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.3647746813391396e-5 | 1.3893780204755124e-5 | 1.4214732248033707e-5 |
| Standard deviation | 6.977761544964352e-7 | 9.127629327342302e-7 | 1.4335829710420947e-6 |
Outlying measurements have severe (0.720112028107516%) effect on estimated standard deviation.
read/long
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.749322488648775e-5 | 4.841758832266267e-5 | 4.966193696363475e-5 |
| Standard deviation | 3.0173181643339984e-6 | 3.702486333766637e-6 | 4.584549725481118e-6 |
Outlying measurements have severe (0.745958125700868%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.