count data type running out of significant digits
Closed this issue · 1 comments
lentinj commented
count was changed from INT to REAL in v3, to handle cases where the count is normalised to handle tow lengths. However, there's only scope for 6 significant digits here, so there's scope for running out of room for long integers. The rounding errors are compounded by SUM()
.
Switching to NUMERIC would solve this, with a < 1 sec performance hit on summing the entirety of example_iceland, and is probably a more sensible data type to choose.
Alternatively we could cast to DOUBLE before summing.
Maybe there should be a choice in data types?
lentinj commented
In wip-7.x
, will be done when that's released.