Herbie reports

The Herbie report

The Herbie report, which is output by the Herbie web commands, lists five items.

First, a brief summary of the results. For most uses, the “Average Error” number, which summarizes how accurate the input and output expressions are, is the most important number in this section. The other numbers are: the time Herbie took to improve the program; the precision Herbie assumed floating-point operations (which can be set at the command line); and the internal precision used to ensure accurate results.

A summary of results from a Herbie report.

Second, the input and output programs themselves. These are printed in standard mathematical syntax. Library functions not often used by mathematicians, including atan2, expm1, fma, hypot, lgamma, log1p, and logb are drawn with a sub- or super-script asterisk, while if statements are rendered as in a program.

Input and output program from a Herbie report.

Third, under Error, a graph of floating-point error versus input value. This is helpful for understanding the sorts of inputs Herbie is improving accuracy on. Sometimes, Herbie improved accuracy on some inputs at the cost of accuracy on other inputs that you care more about. In these cases, you can add a :precondition to restrict the inputs Herbie reasons about.

On these graphs, the red line is the error of the input program, while the blue line is the error of the output program (both can be toggled). For expressions with multiple variables, the variable on the horizontal axis can be selected. If Herbie decided to insert an if statement into the program, the locations of those if statements will be marked with vertical bars.

An error graph from a Herbie report. Note the variable selector (x is selected) and the toggles for the input and output program (both are toggled on).

Fourth, a derivation of the output from the input. For complex or unexpected programs, these can be helpful. Each substantive step in the derivation also lists the error, in bits, of that step's output.

The derivations may name rules built into Herbie, or may claim derivation steps are done by simplification, series expansion, or other Herbie strategies. The derivation will also call out splits of the input into regimes, and strategies Herbie is invoking. When one part of the term is colored blue, that is the only part of the term modified by the operation.

A short derivation from a Herbie report. Note the error at each step, in bits, in gray.

Fifth and finally, a breakdown of Herbie's runtime. This can usually be ignored. The colored bar is a timeline of Herbie's run, with each section of the bar sized proportionally to its runtime, and each color corresponding to a strategy; hover over that section of the bar to learn which strategy.

If you find a bug, include the code snippet in this section when filing the bug. Please also include the debug log linked from this block.

Runtime breakdown from a Herbie report.

We expect the report to grow more informative with future versions. Please get in touch if there is more information you'd like to see.