Bugfix: allow support for specfem acoustic domain/pressure synthetics
AbolfazlKhanMo opened this issue · 10 comments
Hi everyone,
I'm using Seisflows for a test FWI problem. I encountered AssertionError: cannot quantify misfit, missing observed or synthetic traces error. I am attaching parts of paramters.yaml which I think it might be useful. Please check what is wrong with this. Maybe I'm missing something which I need help understanding. There are no outputs when I run seisflows check.
FYI, I'm using CUBIT to create the geological model and mesh it.
MODULES
workflow: inversion
system: workstation
solver: specfem2d
preprocess: default
optimize: gradient
Inversion Workflow
data_case: synthetic
stop_after: evaluate_initial_misfit
export_traces: True
export_residuals: False
export_gradient: True
export_kernels: False
start: 1
end: 2
export_model: True
thrifty: False
iteration: 1
Workstation System
ntask: 1
nproc: 1
mpiexec: null
log_level: DEBUG
verbose: False
Solver SPECFEM
data_format: ascii
materials: acoustic
density: False
attenuation: False
smooth_h: 0.0
smooth_v: 0.0
components: PRE
source_prefix: SOURCE
multiples: False
Paths
path_workdir: /home/akhanmo/__FWI__/SPECFEM2D/specfem2d_workdir
path_scratch: /home/akhanmo/__FWI__/SPECFEM2D/specfem2d_workdir/scratch
path_eval_grad: /home/akhanmo/__FWI__/SPECFEM2D/specfem2d_workdir/scratch/eval_grad
path_output: /home/akhanmo/__FWI__/SPECFEM2D/specfem2d_workdir/output
path_model_init: /home/akhanmo/__FWI__/SPECFEM2D/specfem2d_workdir/OUTPUT_FILES_INIT
path_model_true: /home/akhanmo/__FWI__/SPECFEM2D/specfem2d_workdir/OUTPUT_FILES_TRUE
path_state_file: /home/akhanmo/__FWI__/SPECFEM2D/specfem2d_workdir/sfstate.txt
path_data: null
path_mask: null
path_eval_func: /home/akhanmo/__FWI__/SPECFEM2D/specfem2d_workdir/scratch/eval_func
path_par_file: /home/akhanmo/__FWI__/SPECFEM2D/specfem2d_workdir/parameters.yaml
path_log_files: /home/akhanmo/__FWI__/SPECFEM2D/specfem2d_workdir/logs
path_output_log: /home/akhanmo/__FWI__/SPECFEM2D/specfem2d_workdir/sflog.txt
path_specfem_bin: /home/akhanmo/__FWI__/SPECFEM2D/specfem2d_workdir/bin
path_specfem_data: /home/akhanmo/__FWI__/SPECFEM2D/specfem2d_workdir/DATA
path_solver: /home/akhanmo/__FWI__/SPECFEM2D/specfem2d_workdir/scratch/solver
path_preconditioner: null
Thank very much.
Hi @AbolfazlKhanMo, thanks for raising this issue. Can you paste the last 10 or so lines of your output log file (sflog.txt), that might help determine why this error is occurring.
My initial thought is that the solver failed to generate either the synthetic waveforms or the 'data', causing the preprocess module to throw this AssertionError when looking for the relevant files.
Thanks for the reply @bch0w. That can be possible because the example (Tape 2007) works perfectly. Here is the last part of the sflog.txt file for this specific problem.
////////////////////////////////////////////////////////////////////////////////
EVALUATING MISFIT FOR INITIAL MODEL
////////////////////////////////////////////////////////////////////////////////
2023-02-02 17:38:08 (I) | checking initial model parameters
2023-02-02 17:38:08 (I) | 3000.00 <= rho <= 3000.00
2023-02-02 17:38:08 (I) | 3000.00 <= vp <= 3000.00
2023-02-02 17:38:08 (I) | 0.00E+00 <= vs <= 0.00E+00
2023-02-02 17:38:08 (I) | checking true/target model parameters
2023-02-02 17:38:08 (I) | 3000.00 <= rho <= 3000.00
2023-02-02 17:38:08 (I) | 3000.00 <= vp <= 3500.00
2023-02-02 17:38:08 (I) | 0.00E+00 <= vs <= 0.00E+00
2023-02-02 17:38:08 (I) | preparing observation data for source 001
2023-02-02 17:38:08 (I) | running forward simulation w/ target model for 001
2023-02-02 17:38:08 (D) | running executable with cmd: 'bin/xmeshfem2D'
2023-02-02 17:38:08 (D) | running executable with cmd: 'bin/xspecfem2D'
2023-02-02 17:38:52 (I) | evaluating objective function for source 001
2023-02-02 17:38:52 (D) | running forward simulation with 'Specfem2D'
2023-02-02 17:38:52 (D) | running executable with cmd: 'bin/xmeshfem2D'
2023-02-02 17:38:52 (D) | running executable with cmd: 'bin/xspecfem2D'
2023-02-02 17:39:36 (D) | quantifying misfit with 'Default'
Thanks for that, could you check the following things and let me know if you see anything out of place?
Do the following directories have synthetic waveforms (.semd files)?
scratch/solver/001/traces/obs/*scratch/solver/001/traces/syn/*
Does the forward solver log file show any error messages?
scratch/solver/001/fwd_solver.log
If the forward solver ran without issue, the last few lines should be something like:
Saving model files to directory: ./DATA/
SAVE_MODEL type: binary
written to file(s): ./DATA/proc000000_rho.bin
written to file(s): ./DATA/proc000000_vp.bin
written to file(s): ./DATA/proc000000_vs.bin
written to file(s): ./DATA/proc000000_x.bin
written to file(s): ./DATA/proc000000_z.bin
written to file(s): ./DATA/proc000000_jacobian.bin
done
Saving forward arrays:
Saving Stacey absorbing boundary contributions...
Saving elastic last frame...
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Program SPECFEM2D:
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Tape-Liu-Tromp (GJI 2007)
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
D a t e : 12 - 09 - 2022 T i m e : 18:03:09
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
There are no files in the following folders:
traces/obs, traces/syn, and traces/adj
Also, the forward solver ran with no issues, and I got something like the same output you got.
By the way, in this case, which is an acoustic case, what would be the components in the Seisflows setup?
Okay so I think the problem lies in the process of moving the created waveform files from their output location (typically OUTPUT_FILES/) to their SeisFlows location (traces/*).
For this, SeisFlows is expecting a specific file format for the output seismograms, something like the following wildcard: OUTPUT_FILES/*.?X{comp}.sem?
Can you check if there are waveform files matching the following wildcard: scratch/solver/001/OUTPUT_FILES/*sem*? And if so, could you copy one of the filenames into a comment here?
I suspect acoustic simulations output waveforms with file names that do not match the expected wildcard, leading to the error we are seeing. I admit I have not done much work with acoustic domains and acoustic waveforms, so it's much appreciated that you're testing this out!
Right. So, I intentionally wanted to run acoustic simulations first to ensure I get what I expect. Here is one of the file names in the scratch/solver/001/OUTPUT_FILES/*sem* directory as you wanted:
AA.S0001.PRE.semp
Being said, I changed the case from acoustic to elastic to see what would happen and if will there be any changes. It turned out that it worked! After changing to the elastic case, I'm not encountering AssertionError: cannot quantify misfit, missing observed or synthetic traces anymore.
Here is the parameter file for the elastic case. I think the parameter components is the problem in the acoustic case, but I'm not sure. I'm just guessing.
# EDIT THE SEISFLOWS PARAMETER FILE
!seisflows par ntask 1 # set the number of sources/events to use
!seisflows par materials elastic # update Vp and Vs during inversion
!seisflows par end 2 # final iteration -- we will only run 1
!seisflows par data_case synthetic # synthetic-synthetic means we need both INIT and TRUE models
!seisflows par components X # X-component seismograms
!seisflows par step_count_max 5 # limit the number of steps in the line search
!seisflows par export_traces True # limit the number of steps in the line search
Great! Now that we've gotten to the root of the problem I can institute a fix.
So it seems the main issue here is that the output seismogram AA.S0001.PRE.semp will not match the current wildcard that SeisFlows is checking (*.?X{comp}.sem?), as the filename differs in format from the elastic case.
The tasks to fix this are then:
- If SeisFlows is running acoustic simulations, adjust to data filename wildcard to
*.*.PRE.semp - Ignore the
componentsparameter during acoustic simulations, since all waveforms will be in pressure
I'll try to get to this asap, and get back to you when I submit a PR. Thanks!
Just noting here for myself: this is a SPECFEM2D-specific issue. In SPECFEM2D, pressure (and potential) seismograms are saved differently only when using P-SV waves (not membrane waves). In SPECFEM3D, pressure seismograms are labelled the same as the others (e.g., NN.SSS.BXZ.semp).
The SPECFEM Par_file configuration that gets this different seismogram formatting is
P_SV = .true.
seismotype = 4
save_ASCII_seismograms = .true.See: https://github.com/SPECFEM/specfem2d/blob/master/src/specfem2D/write_seismograms.F90#L473-L488
Hi @AbolfazlKhanMo, I tried to address this issue in #164. Can you update your SeisFlows version and see if that allows you to run your original problem?