Problem running in mpi mode
Opened this issue · 4 comments
gnpang commented
Who may concern
I am running into a problem when I run the xspecfem2D using mpirun. I can run the code using 8 cores. However, if I would like to use more cores, say 16 cores, the xpecfem2D will run until the output says
"Preparing array allocations
arrays for elastic domains
done initialization"
Then it will hold it there, with no error output and no calculation results. I have talked to my department's IT group and we did not find any issues with the hardware side.
Does anyone of you meet this issue or can you help me figure it out?
danielpeter commented
could you attach your Par_file and output generated by the run?
gnpang commented
Please see the attachments for the Par_file I used and the output generated
by the run.
…On Thu, Nov 4, 2021 at 1:07 AM daniel peter ***@***.***> wrote:
could you attach your Par_file and output generated by the run?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#1134 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AC2FQ4PCQMJBLMXOHVGA55TUKIIIZANCNFSM5HJVQNNQ>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
--
Guanning Pang, Ph. D.
Postdoctoral Scholar
Earth and Atmospheric Sciences
Cornell University
**********************************************
**** Specfem 2-D Solver - MPI version ****
**********************************************
Running Git version of the code corresponding to commit 9e6c712
dating From Date: Mon Sep 20 15:23:03 2021 +0300
There are 16 MPI processes
Processes are numbered from 0 to 15
There is a total of 16 slices
NDIM = 2
NGLLX = 5
NGLLZ = 5
using single precision for the calculations
smallest and largest possible floating-point numbers are: 1.17549435E-38 3.40282347E+38
Reading the parameter file...
Title of the simulation: Test of SPECFEM2D with curved interfaces
Parameter file successfully read
Reading in SOURCE file...
Source 1
Position xs, zs = 50000.000000000000 800000.00000000000
Source type (1=force, 2=moment tensor): 1
Force source:
Angle of the source (deg) = 0.0000000000000000
Time function type (1=Ricker, 2=First derivative, 3=Gaussian, 4=Dirac, 5=Heaviside, 6,7=ocean type, 8=Read from file, 9=burst, 10=Sinusoidal, 11=Ormsby): 1
Ricker wavelet (second-derivative):
Frequency, delay = 1.0000000000000000 0.0000000000000000
Multiplying factor = 10000000000.000000
all sources are okay
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Program SPECFEM2D:
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Test of SPECFEM2D with curved interfaces
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
D a t e : 04 - 11 - 2021 T i m e : 08:37:18
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
C o n t r o l
=============
Number of spectral element control nodes. . .(npgeo) = 82369
Number of space dimensions. . . . . . . . . . (NDIM) = 2
C o n t r o l
=============
Display frequency . . . .(NSTEP_BETWEEN_OUTPUT_INFO) = 100
Color display . . . . . . . . . . . . . . . (colors) = 1
== 0 black and white display
== 1 color display
Numbered mesh . . . . . . . . . . . . . . .(numbers) = 0
== 0 do not number the mesh
== 1 number the mesh
C o n t r o l
=============
Seismograms recording type . . . . . . .(seismotype) = 1
Angle for first line of receivers. . . . .(anglerec) = 0.00
C o n t r o l
=============
Read external initial field. . . . . .(initialfield) = F
Add Bielak conditions (add_Bielak_conditions_bottom) = F
Add Bielak conditions .(add_Bielak_conditions_right) = F
Add Bielak conditions . .(add_Bielak_conditions_top) = F
Add Bielak conditions. .(add_Bielak_conditions_left) = F
Attenuation in solid . . .(ATTENUATION_VISCOELASTIC) = T
Attenuation in fluid . . (ATTENUATION_VISCOACOUSTIC) = F
Save grid in ASCII file or not . (output_grid_ASCII) = F
Save a file with total energy or not.(OUTPUT_ENERGY) = F
C o n t r o l
=============
Vector display type . . . . . . . . .(imagetype_postscript) = 1
Percentage of cut for vector plots. . . . . . . .(cutsnaps) = 1.00
Subsampling of velocity model display. (subsamp_postscript) = 1
I t e r a t i o n s
===================
Number of time iterations . . . . .(NSTEP) = 30000
Time step increment. . . . . . . . . .(DT) = 4.000000E-03
Total simulation duration . . . . . (ttot) = 1.200000E+02
reading mesh databases:
reading nodal coordinates...
-- Spectral Elements --
Number of spectral elements . . . . . . . . . (nspec) = 640998
Number of control nodes per element . . . . . (ngnod) = 4
Number of points in X-direction . . . . . . . (NGLLX) = 5
Number of points in Y-direction . . . . . . . (NGLLZ) = 5
Number of points per element. . . . . . (NGLLX*NGLLZ) = 25
Number of points for display . . . . . . (pointsdisp) = 6
Number of element material sets . . . . . . . (numat) = 3
Number of acoustic forcing elements (nelem_acforcing) = 0
Number of acoustic free surf (nelem_acoustic_surface) = 0
reading attenuation setup...
reading material properties..
M a t e r i a l s e t s : 2 D (p o r o) e l a s t i c i t y
======================================================
Number of material sets . . . . . . (numat) = 3
-------------------------------
-- Acoustic (fluid) material --
-------------------------------
Material set number. . . . . . . . (jmat) = 1
P-wave velocity. . . . . . . . . . . (cp) = 1.02830000E+04
Mass density. . . . . . . . . . (density) = 1.21240000E+04
Bulk modulus Kappa . . . . . . . .(kappa) = 1.28199284E+12
QKappa_attenuation. . . . . . . .(QKappa) = 9.99900000E+03
Qmu_attenuation. . . . . . . . . . .(Qmu) = 9.99900000E+03
----------------------------------------------------
-- Material described by external tomography file --
----------------------------------------------------
Material set number. . . . . . . . (jmat) = 2
----------------------------------------
-- Elastic (solid) isotropic material --
----------------------------------------
Material set number. . . . . . . . (jmat) = 3
P-wave velocity. . . . . . . . . . . (cp) = 1.10370000E+04
S-wave velocity. . . . . . . . . . . (cs) = 3.50500000E+03
Mass density. . . . . . . . . . (density) = 1.27040000E+04
Poisson's ratio. . . . . . . . .(poisson) = 4.43919536E-01
First Lame parameter Lambda. . . (lambda) = 1.23540453E+12
Second Lame parameter Mu. . . . . . .(mu) = 1.56068958E+11
Bulk modulus Kappa . . . . . . . .(kappa) = 1.39147349E+12
Young's modulus E. . . . . . . . .(young) = 4.51332901E+11
QKappa_attenuation . . . . . . .(QKappa) = 3.00000000E+02
Qmu_attenuation . . . . . . . . . . (Qmu) = 3.00000000E+02
reading interfaces informations...
number of interfaces = 3
reading absorbing boundary...
number of absorbing elements = 600
any absorbing boundary flag = T
PML boundary flag = F
Stacey boundary flag = T
Absorbing boundaries:
using Stacey absorbing boundary conditions
Total number of absorbing elements: 3208
nspec_left = 1604
nspec_right = 0
nspec_bottom = 1604
nspec_top = 0
reading acoustic forcing...
acoustic forcing = F
number of acoustic forcing boundary elements = 0
reading free surface information...
number of acoustic free surface boundary elements = 0
reading coupling surfaces...
number of fluid-solid edges = 268
number of fluid-poro edges = 0
number of solid-poro edges = 0
reading tangential curves...
number of tangential curve nodes = 0
reading axial elements...
number of axial elements = 0
done reading mesh databases
Setting up mesh
Generating global mesh numbering (fast version)...
Mesh numbering:
Total number of elements: 640998
Total number of acoustic elements = 160000
Total number of elastic/visco/poro elements = 480998
Approximate total number of grid points in the mesh
(with a few duplicates coming from MPI buffers): 10285388
Approximate number of acoustic degrees of freedom in the mesh: 2567344
Approximate number of elastic degrees of freedom in the mesh: 15436089
(there are 2 degrees of freedom per point for elastic elements)
Approximate total number of degrees of freedom in the mesh
(sum of the two values above): 18003433
(for simplicity viscoelastic or poroelastic elements, if any,
are counted as elastic in the above three estimates;
in reality they have more degrees of freedom)
Mesh dimensions:
Xmin,Xmax of the whole mesh = -200000.00000000003 600000.00000000000
Zmin,Zmax of the whole mesh = 0.0000000000000000 800000.00000000012
Assigning an external velocity and density model
model: tomo
tomographic model file: ./DATA/tomo_file.xyz
Number of grid points = NX*NZ: 1922801
done
setup material arrays
setting up anisotropic arrays
Domains:
total number of acoustic elements = 160000
total number of elastic elements = 480998
total number of poroelastic elements = 0
Mixed acoustic/elastic simulation
Beginning of fluid/solid edge detection
Checking fluid/solid edge topology...
End of fluid/solid edge detection
excluding common points between acoustic absorbing edges and acoustic/elastic matching interfaces, if any
Setting up MPI communication arrays
maximum number of MPI interfaces (for a single slice) = 7
total MPI interface points: 46048
checking acoustic interfaces:
total number of global acoustic interface points: 4316
total number of global points assembled by acoustic MPI interfaces: 4316
interfaces okay
main process:
number of MPI interfaces in acoustic domain = 1
number of MPI interfaces in elastic domain = 3
number of MPI interfaces in poroelastic domain = 0
number of outer elements = 657
number of inner elements = 81082
percentage of outer elements 0.803779602 %
percentage of inner elements 99.1962204 %
number of global nodes in outer elements = 13145
number of global nodes in inner elements = 1299821
determining communication phases:
elastic domains:
total number of outer/inner elements = 10371 470627
total percentage of outer elements 2.15614319 %
total percentage of inner elements 97.8438568 %
acoustic domains:
total number of outer/inner elements = 1078 158922
total percentage of outer elements 0.673751831 %
total percentage of inner elements 99.3262482 %
All mesh setup done successfully
sources:
Total number of sources: 1
Setting parameters for source 1
Source Type. . . . . . . . . . . . . . = Collocated Force
X-position (meters). . . . . . . . . . = 5.0000000000E+04
Y-position (meters). . . . . . . . . . = 8.0000000000E+05
Fundamental frequency (Hz) . . . . . . = 1.0000000000E+00
Time delay (s) . . . . . . . . . . . . = 0.0000000000E+00
Multiplying factor . . . . . . . . . . = 1.0000000000E+10
Angle from vertical direction (deg). . = 0.0000000000E+00
Source: 1
Onset time. . . . . . = -1.2000000000000000
Fundamental period. . = 1.0000000000000000
Fundamental frequency = 1.0000000000000000
The onset time is ok
Source: 1
*******************************
locating force source
*******************************
Force source:
original x: 50000.0000
original z: 800000.000
Closest estimate found: 0.00000000 m away
in rank 0
in element 10344
in acoustic domain
at xi,gamma coordinates = -1.0000000000000000 -1.0000000000000000
end of force source detection
receivers:
Total number of receivers = 21
********************
locating receivers
********************
reading receiver information from the ./DATA/STATIONS file
Station # 1 AA S0001
original x: 445000.000
original z: 750000.000
Closest estimate found: 0.00000000 m away
in rank 13
in element 185
in acoustic domain
at xi,gamma coordinates = -1.0000000000000000 1.0000000000000000
Distance from source: 398151.969 m
Station # 2 AA S0002
original x: 445500.000
original z: 750000.000
Closest estimate found: 0.00000000 m away
in rank 13
in element 185
in acoustic domain
at xi,gamma coordinates = -1.0000000000000000 0.0000000000000000
Distance from source: 398648.031 m
Station # 3 AA S0003
original x: 446000.000
original z: 750000.000
Closest estimate found: 0.00000000 m away
in rank 13
in element 185
in acoustic domain
at xi,gamma coordinates = -1.0000000000000000 -1.0000000000000000
Distance from source: 399144.094 m
Station # 4 AA S0004
original x: 446500.000
original z: 750000.000
Closest estimate found: 0.00000000 m away
in rank 13
in element 2464
in acoustic domain
at xi,gamma coordinates = 0.0000000000000000 -1.0000000000000000
Distance from source: 399640.156 m
Station # 5 AA S0005
original x: 447000.000
original z: 750000.000
Closest estimate found: 0.00000000 m away
in rank 13
in element 2464
in acoustic domain
at xi,gamma coordinates = 1.0000000000000000 -1.0000000000000000
Distance from source: 400136.219 m
Station # 6 AA S0006
original x: 447500.000
original z: 750000.000
Closest estimate found: 0.00000000 m away
in rank 13
in element 2465
in acoustic domain
at xi,gamma coordinates = -1.0000000000000000 0.0000000000000000
Distance from source: 400632.312 m
Station # 7 AA S0007
original x: 448000.000
original z: 750000.000
Closest estimate found: 0.00000000 m away
in rank 13
in element 2465
in acoustic domain
at xi,gamma coordinates = -1.0000000000000000 -1.0000000000000000
Distance from source: 401128.406 m
Station # 8 AA S0008
original x: 448500.000
original z: 750000.000
Closest estimate found: 0.00000000 m away
in rank 13
in element 28382
in acoustic domain
at xi,gamma coordinates = 0.0000000000000000 -1.0000000000000000
Distance from source: 401624.500 m
Station # 9 AA S0009
original x: 449000.000
original z: 750000.000
Closest estimate found: 0.00000000 m away
in rank 13
in element 28379
in acoustic domain
at xi,gamma coordinates = -1.0000000000000000 1.0000000000000000
Distance from source: 402120.625 m
Station # 10 AA S0010
original x: 449500.000
original z: 750000.000
Closest estimate found: 0.00000000 m away
in rank 13
in element 28379
in acoustic domain
at xi,gamma coordinates = -1.0000000000000000 0.0000000000000000
Distance from source: 402616.750 m
Station # 11 AA S0011
original x: 450000.000
original z: 750000.000
Closest estimate found: 0.00000000 m away
in rank 13
in element 5947
in acoustic domain
at xi,gamma coordinates = -1.0000000000000000 -1.0000000000000000
Distance from source: 403112.875 m
Station # 12 AA S0012
original x: 450500.000
original z: 750000.000
Closest estimate found: 0.00000000 m away
in rank 13
in element 5947
in acoustic domain
at xi,gamma coordinates = -1.0000000000000000 0.0000000000000000
Distance from source: 403609.031 m
Station # 13 AA S0013
original x: 451000.000
original z: 750000.000
Closest estimate found: 0.00000000 m away
in rank 13
in element 5946
in acoustic domain
at xi,gamma coordinates = 1.0000000000000000 -1.0000000000000000
Distance from source: 404105.188 m
Station # 14 AA S0014
original x: 451500.000
original z: 750000.000
Closest estimate found: 0.00000000 m away
in rank 13
in element 5946
in acoustic domain
at xi,gamma coordinates = 0.0000000000000000 -1.0000000000000000
Distance from source: 404601.344 m
Station # 15 AA S0015
original x: 452000.000
original z: 750000.000
Closest estimate found: 0.00000000 m away
in rank 13
in element 4199
in acoustic domain
at xi,gamma coordinates = -1.0000000000000000 -1.0000000000000000
Distance from source: 405097.531 m
Station # 16 AA S0016
original x: 452500.000
original z: 750000.000
Closest estimate found: 0.00000000 m away
in rank 13
in element 4199
in acoustic domain
at xi,gamma coordinates = 0.0000000000000000 -1.0000000000000000
Distance from source: 405593.688 m
Station # 17 AA S0017
original x: 453000.000
original z: 750000.000
Closest estimate found: 0.00000000 m away
in rank 13
in element 4196
in acoustic domain
at xi,gamma coordinates = -1.0000000000000000 1.0000000000000000
Distance from source: 406089.906 m
Station # 18 AA S0018
original x: 453500.000
original z: 750000.000
Closest estimate found: 0.00000000 m away
in rank 13
in element 4196
in acoustic domain
at xi,gamma coordinates = -1.0000000000000000 0.0000000000000000
Distance from source: 406586.094 m
Station # 19 AA S0019
original x: 454000.000
original z: 750000.000
Closest estimate found: 0.00000000 m away
in rank 13
in element 4196
in acoustic domain
at xi,gamma coordinates = -1.0000000000000000 -1.0000000000000000
Distance from source: 407082.312 m
Station # 20 AA S0020
original x: 454500.000
original z: 750000.000
Closest estimate found: 0.00000000 m away
in rank 13
in element 15570
in acoustic domain
at xi,gamma coordinates = 0.0000000000000000 -1.0000000000000000
Distance from source: 407578.531 m
Station # 21 AA S0021
original x: 455000.000
original z: 750000.000
Closest estimate found: 0.00000000 m away
in rank 13
in element 15567
in acoustic domain
at xi,gamma coordinates = -1.0000000000000000 1.0000000000000000
Distance from source: 408074.750 m
end of receiver detection
found a total of 21 receivers, this is okay
Preparing timerun:
Preparing timerun constants
Preparing array allocations
arrays for acoustic domains
arrays for elastic domains
done initialization
danielpeter commented
thanks for the outputs! it seems to be a problem with the Stacey absorbing boundary setup becoming blocking when there are partitions without boundary elements.
please try the updated version here on the devel branch - it should work now with PR #1136
gnpang commented
Thank you for the updates. I tried in on our nodes with 16 cores. It is
working fine now.
…On Wed, Nov 10, 2021 at 12:10 AM daniel peter ***@***.***> wrote:
thanks for the outputs! it seems to be a problem with the Stacey absorbing
boundary setup becoming blocking when there are partitions without boundary
elements.
please try the updated version here on the devel branch - it should work
now with PR #1136 <#1136>
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#1134 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AC2FQ4KUL7BKVGMGQZKQEBTULH5DHANCNFSM5HJVQNNQ>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
--
Guanning Pang, Ph. D.
Postdoctoral Scholar
Earth and Atmospheric Sciences
Cornell University