openaq/openaq-db

Country code `--` is still in the country_stats table even after being removed from the sensor nodes

caparker opened this issue · 1 comments

Somehow it seems like the bad country code appeared again in the countries page
https://openaq.org/#/countries
Even though the code was removed from the sensor nodes 20 hours or more earlier.

This ended up being a more complicated problem that involved a few different issues. In the end it was not the -- that caused the problem but instead the 99 that we changed it to. The -- was causing problems with the website when it was used in urls and so it was changed to 99 which would still pass the fetcher check but not cause url issues. However, the automatic updates that are performed to update various data products where erroneously adding the country code 99 to any group that was called 99 (see here), which in our case was a node group.

The problem came from having sensors from more than one node in the same node group. This was allowed in the rollups table, which should not be the case, but was not allowed in a product of the rollups/nodes table, the sensor_stats table. This meant that the update failed before it could get to the country_stats table and therefor never removed the old -- data.

That method has been updated to ignore the country = 99 nodes and we have also updated the country updater to also update countries with a 99 code before applying the groups method. Then we removed all data in the rollup table that related to those nodes.