cockroachdb/apd

Exp needs a review

maddyblue opened this issue · 1 comments

Exp needs a review

Looks good with two nits:

  1. apd/context.go

    Line 756 in 78ac5e9

    return 0, errors.Wrap(err, "QuoInteger")
    should read Quo, not QuoInteger.

  2. apd/context.go

    Line 767 in 78ac5e9

    nf := math.Ceil((1.435*pf - 1.182) / math.Log10(pf/rf))
    Doing the Stage 3 calculation in floating-point without directed rounding makes me a little nervous, but I think that there's enough slack elsewhere in the analysis that, if log10 is correct to 1 ulp, then we're fine.

An easy way around the precision assertion is to just increase the working precision -- is this infeasible for some reason? The root cause seems to be that raising sum to the kth power creates a demanding accuracy requirement on sum, so I don't know that this limitation can be easily removed.