NWFSCSlope and AFSCSlope Survey Data Scaling Problem
Ovec8hkin opened this issue · 7 comments
For Shortspine Thornyhead, the design-based indices of abundance that are calculated by Biomass.fn() for the NWFSC Slope and AFSC Slope surveys are very different when run in 2023, than the design-based indices that were reported in the most-recent assessment document (2013). In 2013, the survey biomass was reported as ~30k mt for each survey, while in 2023 the survey biomass is being reported as ~50k (AFSC Slope) and ~70k (NWFSC Slope) mt.
It was reported several weeks ago that there had been some problem within the package that had caused the biomass indices to be scaled incorrectly, but this appears to have been fixed for the combo and triennial surveys. Can anyone confirm that the NWFSC Slope and AFSC Slope survey data and index calculations are correct, or provide a potential reason as to why the indices now are so different from what they were 10 years ago?
@Ovec8hkin The unit issue is fixed for all surveys. There some major differences between 2013 and now in how surveys are processed. First, it is important to note the the indices in the 2013 model are not design-based (what the nwfscSurvey package calculates) but rather were estimated using a delta-glm approach. The methodology for estimating delta-glm indices has changed substantially since 2013. Additionally, this type of code package was not available for processing and expanding data from all West Coast surveys. I believe the raw data was provided to each assessment author who then did their own data processing on the data. If you have any of the data processing files (scripts) from 2013 I could look at them more thoroughly but without those it is difficult to know what would be causing any discrepencies.
The 2013 SSPT assessment reported both delta-GLM indices and design-based indices which were generally on the same scale and trend. The 2023 design-based indices for the two slope surveys are very different than what reported in 2013, though the other surveys (triennial and NWFSC Combo) are almost identical.
@iantaylor-NOAA Do you happen to have (or can access) the analysis scripts you used in 2013 for the SSPT indices?
I just shared this Google Drive folder with @chantelwetzel-noaa and @Ovec8hkin https://drive.google.com/drive/folders/0B5lSVg5Eq86uZGtycU1PX2pVaHc?resourcekey=0-czOoaIJ_a7OWgwfybVXRsw&usp=sharing (others may request access, but it's a cluttered mess--github has helped us get more organized over the past 10 years).
I found the attached .r file (renamed to .txt so it can be attached to a github issue) within the GLMM_results folder within the shared folder, but there may be others as well.
AlaskaSurveys.glmrun.r.txt
Also, if this issue is proving difficult to resolve, I would not worry too much about it. Both the AFSC Slope and NWFSC Slope surveys have short time series and are likely to have very little influence on the model results. Rescaled indices with the same trend should have zero influence on the model.
Given my current commitments, I will not be able to look into this in detail. @iantaylor-NOAA is correct that if this is only a rescaling issue, the impact on the model would be small and come out in the wash via q.
One more comment: although the initial mismatch a few weeks ago was caused by a recent change, I think there's no reason to believe that the remaining difference in slope survey scales is due to a problem with the current approach as opposed to something wrong with the 2013 analysis.
@iantaylor-NOAA @chantelwetzel-noaa Ok. I will take a look at the 2013 analyses and see if I can riddle out what the difference is. Feel free to close.