Verify updates of NL deaths
Closed this issue · 5 comments
Verify if "Deaths by epidemiological week over the past year" (081099d5-cb0f-445d-ba00-78f63cc49800) dataset should be used for NL death data. It seems NL HR-level death counts are no longer being updated in the underlying JSON dataset, as deaths have not incremented since 2023-06-21, despite being repeated in this deaths by epi week dataset. Check also the PHAC deaths dataset the next time it updated (Wednesday, September 6). If week-level data are preserved in the PHAC dataset, that can be used, otherwise we can use the "past year" dataset, treating it as a report with repeating entries, like SK reports (see the problem presented by ccodwg/Covid19CanadaArchive#294).
If NL is no longer reporting deaths at the HR-level, this will also need to be reported in the README (and potentially values.json).
The NL death dataset is now being updated with the PHAC dataset:
- NL deaths no longer reported at HR-level after 2023-06-21
- NL death data after this date is from PHAC dataset
- Start PHAC data on 2023-07-01 rather than 2023-06-24, because the cumulative data on 2023-06-24 would imply negative deaths since 2023-06-21
This could be done for the case dataset, for consistency, but the dashboard case dataset is daily whereas the PHAC dataset is weekly.
Verify if New_Deaths and Prev_deaths values may have received updates after (cumulative) Deaths did.
As a note, it is possible that when the NL dashboard reported weekly (and updated via an active_cumul dataset), that data were being reported up to the previous Saturday (i.e., using epi weeks), similar to the currently monthly update system. The active_cumul dataset used the edit date of the file (which was always a Wednesday) as the "as of date", rather than the previous Saturday as we do now.
It is impossible to verify if the previous dashboard had header text indicating this (as it does now, e.g., "(Current reporting period: epidemiological weeks 39, 40, 41 and 42 (September 24, 2023 to October 21, 2023"), since archived versions of the dashboard from the Wayback Machine do not load. Therefore, we will leave the historical data alone.
The new NL dashboard (#128) and missing reporting weeks means stitching together the death data requires another rethink.
Adding new NL data is a bit tricky. First of all, the data are now well and truly at the PT-level; there is no more hidden HR-level data. The old NL dashboard last reported up for the period from 2023-09-24 to 2023-10-21. The new dashboard has weekly death counts starting with the week ending 2023-09-02. It gives deaths in the current week (ending 2023-11-11) as 1 (current reporting period) and 3 (previous reporting period) for a total of 4. However, there are no "current week" reports of this style for the other weeks in between the current week and the final report week from the old dashboard (2023-10-21). It is not clear if the "death from previous reporting periods" in this first weekly report from the new dashboard account for the fact that no deaths were reported for those missing reporting weeks.
If we look at the final report from the old dashboard, covering the week ending 2023-09-30 to the week ending 2023-10-21, there were 2 deaths from the current reporting period and 1 death from the previous reporting period, for a total of 3 new deaths.
The second-to-last report, covering the week ending 2023-09-02 to the week ending 2023-09-23, gives 4 deaths from the current reporting and 0 deaths from the previous reporting period, for a total of 4 new deaths.
Here is what the death time series on the dashboard currently displays:
1 mortality NL 2023-09-02 0
2 mortality NL 2023-09-09 1
3 mortality NL 2023-09-16 2
4 mortality NL 2023-09-23 1
5 mortality NL 2023-09-30 1
6 mortality NL 2023-10-07 1
7 mortality NL 2023-10-14 0
8 mortality NL 2023-10-21 1
9 mortality NL 2023-10-28 1
10 mortality NL 2023-11-04 1
11 mortality NL 2023-11-11 1
So the second-to-last report period still includes 4 deaths.
The last report period now includes 3 deaths, as opposed to the 2 within-period deaths reported originally.
The week ending 2023-11-11 includes 1 death and the missing weeks of 2023-10-28 and 2023-11-04 each include 1 death.
Thus, the time series is consistent with the weekly report of 1 death from the current period (2023-11-11) and 3 deaths from previous reporting periods (one each from 2023-11-04, 2023-10-28, and the period from 2023-09-30 to 2023-10-21).
Therefore, a proposed solution would be to add a death of unknown HR to the final report from 2023-09-30 to 2023-10-23, then add one further death of unknown each to the weeks of 2023-10-28, 2023-11-04, and 2023-11-11.
Starting with the next update, we can simply use the weekly update numbers as usual.