criterion performance measurements
overview
want to understand this report?
bytestring-read/short
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 3.871990513902483e-7 | 3.9031477517761356e-7 | 3.9286803745730374e-7 |
Standard deviation | 8.551939657211476e-9 | 9.97092154549203e-9 | 1.18101696922444e-8 |
Outlying measurements have moderate (0.35393137279368686%) effect on estimated standard deviation.
bytestring-read/long
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.0686758573931825e-6 | 1.0723947372418401e-6 | 1.0771486028883048e-6 |
Standard deviation | 1.144862828565305e-8 | 1.4065066700795679e-8 | 1.977759665797805e-8 |
Outlying measurements have moderate (0.11717920221630587%) effect on estimated standard deviation.
bytestring-read(Lazy)/short
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 3.7805978186745114e-7 | 3.8160236646923135e-7 | 3.8401049950293446e-7 |
Standard deviation | 7.204917149657172e-9 | 9.801250527953904e-9 | 1.5822210588421482e-8 |
Outlying measurements have moderate (0.3580828655056597%) effect on estimated standard deviation.
bytestring-read(Lazy)/long
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.1183901567294006e-6 | 1.133588220209855e-6 | 1.1449907244543843e-6 |
Standard deviation | 3.259245418355493e-8 | 4.5024080502700415e-8 | 5.73107422181539e-8 |
Outlying measurements have severe (0.5494867224326775%) effect on estimated standard deviation.
text/short
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.930886613294211e-6 | 1.9518169343387877e-6 | 1.968394028243429e-6 |
Standard deviation | 4.5462353576988114e-8 | 5.8823577206724994e-8 | 7.724078534664008e-8 |
Outlying measurements have moderate (0.39640478239024646%) effect on estimated standard deviation.
text/long
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 2.146947272177313e-5 | 2.1640820884982994e-5 | 2.1812504299087905e-5 |
Standard deviation | 4.6656808739144735e-7 | 5.992752110459349e-7 | 8.65696945863067e-7 |
Outlying measurements have moderate (0.2913262130533435%) effect on estimated standard deviation.
bytestring-lexing/short
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 4.5563088684618697e-7 | 4.5869478026091517e-7 | 4.6190834134970195e-7 |
Standard deviation | 8.504071760784233e-9 | 1.0173572244985155e-8 | 1.2165381489633621e-8 |
Outlying measurements have moderate (0.2910364617622874%) effect on estimated standard deviation.
bytestring-lexing/long
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 3.6668481925482002e-6 | 3.6865652316334563e-6 | 3.707939711787143e-6 |
Standard deviation | 5.228725779815216e-8 | 6.882608089738974e-8 | 9.285851291740723e-8 |
Outlying measurements have moderate (0.19030727137747097%) effect on estimated standard deviation.
attoparsec/short
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 4.6338644678388995e-6 | 4.697196672784298e-6 | 4.762070972359596e-6 |
Standard deviation | 1.8997350017632536e-7 | 2.1464929244854916e-7 | 2.464505160201899e-7 |
Outlying measurements have severe (0.5828289246255699%) effect on estimated standard deviation.
attoparsec/long
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 2.8893666021040232e-5 | 2.904531608702304e-5 | 2.9150802954544414e-5 |
Standard deviation | 2.82194140697655e-7 | 4.1149806826572184e-7 | 6.101645542854993e-7 |
Outlying measurements have slight (9.326277069336177e-2%) effect on estimated standard deviation.
read/short
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.607987059017719e-5 | 1.6246042175777217e-5 | 1.641152669475765e-5 |
Standard deviation | 5.067074461469622e-7 | 5.731979739249643e-7 | 6.835411916106077e-7 |
Outlying measurements have moderate (0.4136010909819513%) effect on estimated standard deviation.
read/long
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 6.153022782480029e-5 | 6.202506718700818e-5 | 6.244306727791669e-5 |
Standard deviation | 1.0392335589119437e-6 | 1.5272934899343277e-6 | 2.2187291810301526e-6 |
Outlying measurements have moderate (0.2177662453113236%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.