gadget-framework/mfdb

count data type running out of significant digits

Closed this issue · 1 comments

count was changed from INT to REAL in v3, to handle cases where the count is normalised to handle tow lengths. However, there's only scope for 6 significant digits here, so there's scope for running out of room for long integers. The rounding errors are compounded by SUM().

Switching to NUMERIC would solve this, with a < 1 sec performance hit on summing the entirety of example_iceland, and is probably a more sensible data type to choose.

Alternatively we could cast to DOUBLE before summing.

Maybe there should be a choice in data types?

In wip-7.x, will be done when that's released.