Running on Chinook issue: No such file or directory: 'DATA/Par_file'

[ytian159]

I was trying to submit some forward simulation jobs on Chinoo04 via seisflows. I have specified my own DATA DADABASE and bin directory. But I have this error message:

2023-04-03 03:02:07 | INFO | forward.py -> run_forward_simulations():L394
evaluating objective function for source 20160518032548320
2023-04-03 03:02:07 | DEBUG | forward.py -> run_forward_simulations():L396
running forward simulation with 'Specfem3D'
Traceback (most recent call last):
File "/var/spool/slurm/job02423/slurm_script", line 110, in
func(**kwargs)
File "/import/c1/ERTHQUAK/ytian4/simutils/seisflows/seisflows/workflow/forward.py", line 409, in run_forward_simulations
self.solver.forward_simulation(
File "/import/c1/ERTHQUAK/ytian4/simutils/seisflows/seisflows/solver/specfem3d.py", line 148, in forward_simulation
setpar(key="ATTENUATION", val=".false`.", file="DATA/Par_file")
File "/import/c1/ERTHQUAK/ytian4/simutils/seisflows/seisflows/tools/specfem.py", line 137, in setpar
key_out, val_out, i = getpar(key, file, delim, match_partial)
File "/import/c1/ERTHQUAK/ytian4/simutils/seisflows/seisflows/tools/specfem.py", line 79, in getpar
lines = open(file, "r").readlines()
FileNotFoundError: [Errno 2] No such file or directory: 'DATA/Par_file'

I only changed those three paths. It could run before my changes. What was the problem? I'm running it at : '/import/c1/ERTHQUAK/ytian4/seisflows_workdir'. The parameter file is located here.

Thanks,
Yuan

[bch0w]

Hi @ytian159, looks like for some reason you have 4 solver directories but you have only specified ntask==2 in your parameter file. The two chosen events did not seem to initialize properly, and they are missing all files in their respective DATA/ directories:

That is, comparing: /import/c1/ERTHQUAK/ytian4/seisflows_workdir/scratch/solver/mainsolver/DATA/
to: /import/c1/ERTHQUAK/ytian4/seisflows_workdir/scratch/solver/20160518032548320/DATA/

SeisFlows only takes the first ntask events in path_specfem_data, in alphabetical order, which looks like it would be 20160114190410727 and 20160518032548320, but for some reason 20190411104245609 is linked as your mainsolver, which makes it seem like perhaps this was use for a previous run and now there are new events being added but not properly intialized.

I think the first thing to try is a seisflows restart to clear out the existing working directory and re-run the workflow (this will delete everything in the working directory so if you need these files, copy them somewhere safe before running this command). The expected behavior is that SeisFlows will run forward simulations for 20160114190410727 and 20160518032548320.

If you encounter the same issue then please let me know, there may be something else happening.

[ytian159]

Hi Bryant,
I still have the same problem after restarting seisflows. I changed the number of tasks and restarted several times. The error messages are the same.
Yuan

[bch0w]

Hi @ytian159, got it! There was an interesting thing going on where the initialization of the solver directory was ignoring all files which contained a string matching the source prefix (in your case, 'CMTSOLUTION'). Since your full directory contains the string 'CMTSOLUTION', all the files were getting ignored, and thus the empty DATA/ directory, and the missing DATA/Par_file.

I just pushed a fix in the latest devel branch (cb1065a) that should deal with this by only looking at the filenames (not full path), and only excluding files that start with the source prefix (not contain it). Hopefully that works for you and that you don't run into this issue again.

[ytian159]

Thank you so much. It has finished running without issue.
Yuan

[bch0w]

Closing as complete! Please re-open if this issue persists for you.