DOI-USGS/national-flow-observations

No `getters` file?

padilla410 opened this issue · 4 comments

Is it a problem that there is no getters file for accessing the shared S3 cache?

> nwis_dv_inventory <- readRDS(sc_retrieve('10_nwis_pull/inout/nwis_dv_inventory.rds.ind'))
Error: while reading 'getters.yml'
The file 'getters.yml' does not exist
In addition: Warning message:
In sc_retrieve("10_nwis_pull/inout/nwis_dv_inventory.rds.ind") :
  sc_retrieve now looks for recipes in 'getters.yml' (the 'scipiper.getters_file' option) by default, but that file does not exist

It is here, but the default value for the remake_file in that function was set to getters.yml in a past scipiper update. This pipeline probably predates that change.

To get around it, you need to use

nwis_dv_inventory <- readRDS(sc_retrieve('10_nwis_pull/inout/nwis_dv_inventory.rds.ind', remake_file = '10_nwis_pull.yml'))

I think you're using the latest version of scipiper, which defaults to remake_file = 'getters.yml'. This repo pre-dates this change, so there are a couple solutions.

  • We could make a getters.yml file and pull the appropriate targets into that file.
  • We could revert to an earlier version of scipiper
  • We could modify those sc_retrieve functions like Jordan indicated above. I think this is the easiest/best option for now.

As a connection to the other repo/project, the coop pipeline files use a different getter than the default in the lake-temperature-model-prep pipeline. See here

Ah great, thanks for the context. I am mostly interested in it because I am developing a little "proof of concept" snippet for my PR. I'll close this.