kaizen-ai/kaizenflow

Resume resampling Binance bid/ask data

Jd8111997 opened this issue · 0 comments

Jd8111997 commented

This is a reproduction of issue Resume resampling Binance bid/ask data from Cmamp.

Issue:

we have this data s3://cryptokaizen-data-test/v3/periodic_daily/airflow/resampled_1min/parquet/bid_ask/futures/v8/ccxt/binance/v2_0_0/

The process of resampling somehow got paused/lost but it will be easy to catch up.

Steps:

  1. Move the data from test bucket to preprod.
  2. Create a DAG to backfill the resampling history from April 15 up to present (using the archived_200ms data).
  3. Check if we have a daily preprod DAG - if yes, unpause it, if not, let's add it.
  4. Update Master Dataset Matrix
  5. Update Raw Data Gallery notebook https://github.com/cryptokaizen/cmamp/blob/eb5c16dd3bb670703a4210475656d02178eb5015/im_v2/common/notebooks/Master_raw_data_gallery.ipynb

CC : @sonaalKant @jsmerix