[Bug]: Duplicate product generations trigger InvalidChecksum from ASF DAAC
Closed this issue · 2 comments
Checked for duplicates
Yes - I've already checked
Describe the bug
Duplicate product generations:
Traceback (most recent call last):
File "/home/ops/verdi/ops/CNM_product_delivery/product_delivery/commons/exec_util.py", line 15, in wrapper
status = func(*args, **kwargs)
File "/home/ops/verdi/ops/CNM_product_delivery/product_delivery/scripts/../utils/update_es.py", line 463, in main
raise ValueError(error_message)
ValueError: CNM-R message reported an error:
dataset=OPERA_L2_CSLC-S1_T108-230160-IW3_20231119T022138Z_20231120T074019Z_S1A_VV_v1.0, error_code=PROCESSING_ERROR, error_message={"errorType":"InvalidChecksum","errorMessage":"Invalid checksum for S3 object s3://asf-cumulus-test-opera-staging/OPERA_L2_CSLC-S1_T108-230160-IW3_20231119T022138Z_20231120T074019Z_S1A_VV_v1.0/asf-cumulus-test/OPERA_L2_CSLC-S1_V1___1.0/OPERA_L2_CSLC-S1_T108-230160-IW3_20231119T022138Z_20231120T074019Z_S1A_VV_v1.0.iso.xml.md5 with type md5 and expected sum 9469a908e9731d499bfb5b04480db200","trace":["InvalidChecksum: Invalid checksum for S3 object s3://asf-cumulus-test-opera-staging/OPERA_L2_CSLC-S1_T108-230160-IW3_20231119T022138Z_20231120T074019Z_S1A_VV_v1.0/asf-cumulus-test/OPERA_L2_CSLC-S1_V1___1.0/OPERA_L2_CSLC-S1_T108-230160-IW3_20231119T022138Z_20231120T074019Z_S1A_VV_v1.0.iso.xml.md5 with type md5 and expected sum 9469a908e9731d499bfb5b04480db200"," at Object.validateS3ObjectChecksum (/var/task/webpack:/src/S3.ts:876:9)"," at runMicrotasks ()"," at processTicksAndRejections (node:internal/process/task_queues:96:5)"," at GranuleFetcher.verifyFile (/var/task/webpack:/GranuleFetcher.js:288:7)"," at handleDuplicateFile (/var/task/webpack:/src/granule.ts:284:27)"," at GranuleFetcher.ingestFile (/var/task/webpack:/GranuleFetcher.js:392:26)"," at async Promise.all (index 4)"," at GranuleFetcher.ingest (/var/task/webpack:/GranuleFetcher.js:135:29)"," at ingestGranule (/var/task/webpack:/index.js:57:58)"]}
What did you expect?
I expected [...]
Reproducible steps
affected products:
2 OPERA_L2_CSLC-S1_T108-230159-IW1_20231119T022133Z_20231120T074019Z_S1A_VV
2 OPERA_L2_CSLC-S1_T108-230159-IW2_20231119T022134Z_20231120T074019Z_S1A_VV
2 OPERA_L2_CSLC-S1_T108-230159-IW3_20231119T022135Z_20231120T074019Z_S1A_VV
2 OPERA_L2_CSLC-S1_T108-230160-IW1_20231119T022136Z_20231120T074019Z_S1A_VV
2 OPERA_L2_CSLC-S1_T108-230160-IW2_20231119T022137Z_20231120T074019Z_S1A_VV
2 OPERA_L2_CSLC-S1_T108-230161-IW1_20231119T022139Z_20231120T074019Z_S1A_VV
2 OPERA_L2_CSLC-S1_T108-230161-IW2_20231119T022140Z_20231120T074019Z_S1A_VV
2 OPERA_L2_CSLC-S1_T108-230161-IW3_20231119T022141Z_20231120T074019Z_S1A_VV
2 OPERA_L2_CSLC-S1_T108-230162-IW1_20231119T022142Z_20231120T074019Z_S1A_VV
2 OPERA_L2_CSLC-S1_T108-230162-IW2_20231119T022143Z_20231120T074019Z_S1A_VV
2 OPERA_L2_CSLC-S1_T108-230162-IW3_20231119T022143Z_20231120T074019Z_S1A_VV
2 OPERA_L2_CSLC-S1_T108-230163-IW1_20231119T022144Z_20231120T074019Z_S1A_VV
2 OPERA_L2_CSLC-S1_T108-230163-IW2_20231119T022145Z_20231120T074019Z_S1A_VV
2 OPERA_L2_CSLC-S1_T108-230163-IW3_20231119T022146Z_20231120T074019Z_S1A_VV
2 OPERA_L2_CSLC-S1_T108-230164-IW1_20231119T022147Z_20231120T074019Z_S1A_VV
2 OPERA_L2_CSLC-S1_T108-230164-IW2_20231119T022148Z_20231120T074019Z_S1A_VV
2 OPERA_L2_CSLC-S1_T108-230164-IW3_20231119T022149Z_20231120T074019Z_S1A_VV
2 OPERA_L2_CSLC-S1_T108-230165-IW1_20231119T022150Z_20231120T074019Z_S1A_VV
2 OPERA_L2_CSLC-S1_T108-230165-IW2_20231119T022151Z_20231120T074019Z_S1A_VV
2 OPERA_L2_CSLC-S1_T108-230165-IW3_20231119T022152Z_20231120T074019Z_S1A_VV
2 OPERA_L2_CSLC-S1_T108-230166-IW1_20231119T022153Z_20231120T074019Z_S1A_VV
2 OPERA_L2_CSLC-S1_T108-230166-IW2_20231119T022154Z_20231120T074019Z_S1A_VV
2 OPERA_L2_CSLC-S1_T108-230166-IW3_20231119T022155Z_20231120T074019Z_S1A_VV
2 OPERA_L2_CSLC-S1_T108-230167-IW1_20231119T022155Z_20231120T074019Z_S1A_VV
2 OPERA_L2_CSLC-S1_T108-230167-IW3_20231119T022157Z_20231120T074019Z_S1A_VV
2 OPERA_L2_RTC-S1-STATIC_T108-230722-IW3_20140403_S1A_30
2 OPERA_L2_RTC-S1-STATIC_T108-230723-IW1_20140403_S1A_30
2 OPERA_L2_RTC-S1-STATIC_T111-236951-IW1_20140403_S1A_30
2 OPERA_L2_RTC-S1-STATIC_T111-236971-IW2_20140403_S1A_30
2 OPERA_L2_RTC-S1-STATIC_T111-236971-IW3_20140403_S1A_30
2 OPERA_L2_RTC-S1-STATIC_T111-236974-IW3_20140403_S1A_30
2 OPERA_L2_RTC-S1-STATIC_T111-236975-IW1_20140403_S1A_30
2 OPERA_L2_RTC-S1-STATIC_T111-237087-IW3_20140403_S1A_30
2 OPERA_L2_RTC-S1-STATIC_T111-237116-IW2_20140403_S1A_30
2 OPERA_L2_RTC-S1-STATIC_T111-237118-IW2_20140403_S1A_30
2 OPERA_L2_RTC-S1-STATIC_T111-237127-IW2_20140403_S1A_30
2 OPERA_L2_RTC-S1-STATIC_T111-237127-IW3_20140403_S1A_30
2 OPERA_L2_RTC-S1-STATIC_T111-237130-IW3_20140403_S1A_30
2 OPERA_L2_RTC-S1-STATIC_T111-237136-IW2_20140403_S1A_30
2 OPERA_L2_RTC-S1-STATIC_T111-237143-IW3_20140403_S1A_30
2 OPERA_L2_RTC-S1-STATIC_T111-237145-IW2_20140403_S1A_30
2 OPERA_L2_RTC-S1-STATIC_T111-237149-IW2_20140403_S1A_30
2 OPERA_L2_RTC-S1-STATIC_T111-238252-IW3_20140403_S1A_30
4 OPERA_L2_CSLC-S1_T108-230160-IW3_20231119T022138Z_20231120T074019Z_S1A_VV
### Environment
```bash
- Version of this software [e.g. vX.Y.Z]
- Operating System: [e.g. MacOSX with Docker Desktop vX.Y]
...
Per @philipjyoon, Invalid checksum due to reprocessing of both static and non static products caused by overlapping Standard and STATIC trigger rules.
Closing as it is more related duplication of trigger rules. Not related to trigger rules.
Pending investigation from Scott, Phil.