nextstrain/seasonal-flu

Add rules to the nextflu-private workflow to rename Auspice JSONs based on the build date and deploy the JSONs to the Nextstrain group

huddlej opened this issue · 0 comments

Before we can deploy newly built Auspice JSONs to the Nextstrain group, we need to rename them to a unique name that reflects the date they were built. The builds need to be renamed from names like h1n1pdm_2y_titers_ha.json to flu_seasonal_YYYY-MM-DD_h1n1pdm_2y_titers_ha.json where the YYYY-MM-DD is the date the builds were run. We currently rename these files manually to include the date.

The nextstrain-public builds provide an example of custom rules to rename the JSONs prior to deployment. The new names for the JSONs get defined as a template string in the build configuration parameter called auspice_name. The rename rules get invoked by the custom deploy rule which is the primary target of the workflow we run from GitHub Actions.

We should follow the same pattern for the nextflu-private builds, except we need to calculate the current date when the workflow runs. One approach could be to assign the current date to a global variable in the rename.smk Snakemake file, define an auspice_name for the builds with a non-wildcard placeholder for the date like "flu_seasonal_{{date}}_{lineage}_{{segment}}_{resolution}", and then pass the date value to the format call on the filename strings (e.g., this one and this one).

TODO:

  • Add custom rules to rename Auspice JSONs in nextflu-private profile
  • Define auspice_name string for builds in profiles/nextflu-private.yaml with reference to the date
  • Format the Auspice JSON names with the date calculated from datetime.date.today().strftime("%Y-%m-%d")
  • Add custom rule to deploy renamed Auspice JSONs in the nextflu-private profile using the Nextstrain Group secrets
  • Setup an AWS Cognito user with write access to the nextflu-private group (nextflu-private-bot)
  • Add nextflu-private username and password to seasonal-flu repo secrets for actions
  • Update GitHub Action for nextflu-private builds to add two lines below the current environment variables passed to the CLI which specify --env NEXTSTRAIN_USERNAME and --env NEXTSTRAIN_PASSWORD. These environment variables should already exist in the GitHub Action thanks to setup by the pathogen-repo-build action
  • Add nextstrain login command to the custom Snakemake deploy rule which will rely on credentials in environmental variables
  • Add a rule to the custom rename Snakefile to rename the forecast tip frequencies JSONs (e.g., from h3n2_2y_ha_cell_fra_cTiter_x-ne_star_forecast-tip-frequencies.json to flu_seasonal_YYYY-MM-DD_h3n2_2y_ha_cell_fra_cTiter_x-ne_star_forecast-tip-frequencies.json or just store as auspice/forecasts/{build_name}/{model}.json and copy to auspice/forecasts/{build_date}/{build_name}/{model}.json)