adjtomo/seisflows

Change Cluster-based run_script/submit_workflow script path location

Closed this issue · 1 comments

bch0w commented

System modules derived from the Cluster class use an external run script to execute tasks on compute nodes. This run script is essentially a python script that loads functions from Pickle files and executes them. This allows tasks to be partitioned from a master job to each compute node.

run_call = " ".join([
f"{self.run_call_header}",
f"{os.path.join(ROOT_DIR, 'system', 'runscripts', 'run')}",
f"--funcs {funcs_fid}",
f"--kwargs {kwargs_fid}",
f"--environment SEISFLOWS_TASKID={{task_id}},{self.environs}"
])

Currently the run calls (located in the run() function) points to a symlink to a Python script, which was used to make things clearer. However when SeisFlows is installed via Pip, these symlinks are not carried through.

I see two ways to address this:

  1. add a MANIFEST.in file to keep the symlinks
  2. change the paths from 'run' to 'run_funcs.py' (which is carried through on pip installation)

If 2 is chosen, it might be beneficial to define the path to this script as a class variable once (in Cluster) rather than having it floating around in lots of different places. The same should be done for the submit workflow function:

submit_call = " ".join([
f"{self.submit_call_header}",
f"{os.path.join(ROOT_DIR, 'system', 'runscripts', 'submit')}",
f"--workdir {workdir}",
f"--parameter_file {parameter_file}",
])

bch0w commented

This is addressed in 1163f46 with a Manifest.in file