[PM-1977] intersite transfers for large workflows broken
Closed this issue · 1 comments
mayani commented
for large workflows where the parent job of a compute job gets assigned a different relative directory (under the hierarchical naming scheme), the inter site transfer job created has an incorrect source and destination url. the source url ends up having the same relative dir prefix as the destination one
/scitech/shared/projects/XENONnT/workflows-to-keep
wimp_ac_optimization-discovery_power-202409191126.analyzer.txt
wimp_ac_optimization-discovery_power-202409191126.tar.gz
00/27/combine_ID0000613.sh: { "site_label": "staging-davs", "url": "gsidavs://xenon-gridftp.grid.uchicago.edu:2880/xenon/scratch/xudc/runs/00/23/wimp_ac_optimization-discovery_power-202409191126-6-combined_output.tar.gz" } Then later: 00/27/stage_inter_local_separate_ID0000614_0.out.000: 2024-09-19 14:37:09,089 INFO: gfal-copy -f -p -t 7200 -T 7200 'davs://xenon-gridftp.grid.uchicago.edu:2880/xenon/scratch/xudc/runs/00/24/wimp_ac_optimization-discovery_power-202409191126-6-combined_output.tar.gz' 'file:////scratch/xudc/workflows/wimp_ac_optimization-discovery_power-202409191126/scratch/runs/00/24/wimp_ac_optimization-discovery_power-202409191126-6-combined_output.tar.gz'