connection to ISAAC server seems to be good, but no visualization
Opened this issue · 24 comments
Dear all,
I am trying to use PIConGPU with ISAAC visualization. My current status is very similar to issue #3410 (part of).
The installation is mostly via spack, and isaac & isaac-server are installed with @develop (1.6.0). Anyway, installation went smoothly.
PIConGPU is on dev branch with tag of 0.7.0-530-g71ec38322 (though it shows 0.8.0-dev during simulation).
The simulation is running on a linux VM with ubuntu 22.04 and with 2x4 GPU cores.
The current situation is, the simulation runs smoothly (myLWFA example, with 4.cfg or 4_isaac.cfg). The png figures looks good; openpmd data looks good; stdout of compiling and of the simulation looks ok.
During compiling I can see:
...
-- ISAAC_RENDERER_OPTIMIZED is enabled
-- Found ISAAC: /home/u07khl00/spack/opt/spack/linux-ubuntu22.04-skylake_avx512/gcc-12.3.0/isaac-develop-52jjftkdf6t3sxsio3ahd2c5sbdjftys/lib/cmake/ISAAC
Optional Dependencies:
openPMD: ON
PNGwriter: ON
ISAAC: ON
...
During the beginning of the simulation, I can see:
...
PIConGPUVerbose INPUT_OUTPUT(32) | ISAAC Init succeded
...
The server is running on the same machine (localhost) in the background. And when the simulation starts (isaac connect to localhost), from the server stdout, I can see:
...
New connection, giving id 1 (control)
Group complete, sending to connected interfaces
...
I am using the tunnel via putty (as mentioned in isaac: tunnel.md), and I open isaac\client\html\visualisation.html.
I click on the observe button and go into the observing page. The message of server stdout shows
...
URIImageConnector: Openend URI JPEG Stream
...
Everything looks find except there is no visualization of the simulation (fields, particles, etc.).
The particle num is correct, the cell num is correct, the time step shows good correspondence to simulation stdout. The dimension is also correct. I also tried clicking the "update minmax interval" button repeatedly, and the max/min values are updated repeatedly with reasonable values.
I tried running simulation and server on two different VMs, but still got no visualization.
In the earlier post #3410, there is no mentioning on how to solve this. It was just fixed somehow.
I'm here seeking help to solve this issue. Thank you in advance.
Hi @onezillion thanks for opening the issue.
Have you activated the data source with the slider so that the B field, E and J will become visible?
I need to retest if we broke something with the current dev branch. We switched with PIConGPU dev fully to native alpaka instead of using the cupla library.
If the slider will not solve the issue I will have a look into it, could you report if the slider start showing something?
@PrometheusPi We need to test ISAAC with the latest dev branch.
@psychocoderHPC Thanks for pinging - I will have a look
Hi @onezillion thanks for opening the issue. Have you activated the data source with the slider so that the B field, E and J will become visible?
I need to retest if we broke something with the current dev branch. We switched with PIConGPU dev fully to native alpaka instead of using the cupla library. If the slider will not solve the issue I will have a look into it, could you report if the slider start showing something?
Thanks for your reply.
I am sorry for my dumb question. What is a slider? How can I interact with it or activate it?
I am attaching the stdout during compiling for your reference.
and in isaac.param, I use
...
using Particle_Seq = VectorAllSpecies;
using Native_Seq = MakeSeq_t<FieldE, FieldB, FieldJ>;
using Density_Seq = deriveField::CreateEligible_t<Particle_Seq, deriveField::derivedAttributes::Density>;
using Fields_Seq = MakeSeq_t<Native_Seq, Density_Seq>;
using VectorFields_Seq = Native_Seq;
...
I use 4_isaac.cfg without modification
Thanks again for your reply.
I mean the slider in the gui under "Volume Weights and Partticle Sizes."
Please do not forget to klick on the update button when you change something.
Hi @onezillion,
I tried to reproduce your issue (using LaserWakfield and 4_isaac.cfg) and could not.
I compiled ISAAC with PIConGPU's latest dev (yes, it says PIConGPU: 0.8.0-dev at startup)
The only issue I had was that I had to use a previously compiled server. However, that software did not change for years (that's why I had issues building it.)
From the log files you shared, I see a few strange things:
- your server states:
New connection, giving id 1 (control)
Group complete, sending to connected interfaces
but your image is only showing simulation ID 0. Do you run two simulations?
- Do you see the connection from your bwoser in the server log, as:
ISAAC Connection from <your IP address> (<your IP address>)!
callback_isaac: LWS_CALLBACK_ESTABLISHED
URIImageConnector: Openend URI JPEG Stream
callback_isaac: LWS_CALLBACK_FILTER_PROTOCOL_CONNECTION
Did you adjust any of the 3 ports that are set by default or did the server or client needed to reroute, e.g. the web port? (because data transfer seems to work)
PS: did you activate moving window? Because your Dimensions look different than the default.
Thank you for your replies. Sorry that I am away for a business trip.
The ID number difference may be due to that the screenshot and stdout are obtained from different simulation runs. But each time, only one simulation is running. I will confirm this later and check the ID number.
I do remember that similar messages have occurred in server stdout:
ISAAC Connection from ()!
callback_isaac: LWS_CALLBACK_ESTABLISHED
……
I will also double confirm this and make screenshot later.
As for the slider, I have been trying changing the max/min limits and playing around the color map before I posted this issue (I did click the update button). I will also check this again.
I have tried both modified and default port settings. The tunelling settings are as suggested by the isaac page. I think there is no firewall between the vm machine and my pc terminal, or at least the default ports are not blocked.
I cannot remember the moving window setting, but I did not change this setting.
Sorry that I am still away for a business trip until this weekend. I will check these suggestions on this weekend.
I am attaching the stdout and screenshots throughout the whole procedure from loading spack modules to establishing the visualization in the following.
I use putty and tunnel connection (localhost)
The issue persists. It is really strange. Maybe it is a machine dependent problem? Is there any system-based setting that I should take care of?
PIConGPU startup
load spack modules
$> source picongpu.profile
export env vars & load pkgs for PIConGPU ......
-- linux-ubuntu22.04-skylake_avx512 / gcc@12.3.0 ----------------
boost@1.83.0 cmake@3.27.7 cuda@12.2.1 glm@develop hdf5@1.14.3 icet@2.1.1 isaac@develop isaac-server@develop jansson@2.14 libjpeg-turbo@3.0.0 libpng@1.6.39 libwebsockets@2.2.1 openmpi@4.1.6 openpmd-api@0.15.2 pngwriter@0.7.0
==> 15 loaded packages
load isaac_server & put it into background
The port settings are all default
$> source isaac_server.profile
Using web_port=2459, tcp_port=2458 and sim_port=2460
lwsts[2130989]: Initial logging level 1031
lwsts[2130989]: Libwebsockets version: 2.2.0 u07khl00@pca1-
lwsts[2130989]: IPV6 not compiled in
lwsts[2130989]: libev support not compiled in
lwsts[2130989]: libuv support not compiled in
lwsts[2130989]: Threads: 1 each 1024 fds
lwsts[2130989]: mem: platform fd map: 8192 bytes
lwsts[2130989]: Compiled with OpenSSL support
lwsts[2130989]: SSL disabled: no LWS_SERVER_OPTION_DO_SSL_GLOBAL_INIT
lwsts[2130989]: Creating Vhost 'default' port 2459, 2 protocols, IPv6 off
lwsts[2130989]: Listening on port 2459
lwsts[2130989]: mem: per-conn: 568 bytes + protocol rx buf
lwsts[2130989]: canonical_hostname = pca1
Running ISAAC Master
Starting insitu plugin listener
Launching WebSocketDataConnector
Launching TCPDataConnector
Launching JPEG_URI_Stream
u07khl00 2136 0.0 0.0 17312 10032 ? Ss Apr08 0:00 /lib/systemd/systemd --user
u07khl00 2137 0.0 0.0 170568 4656 ? S Apr08 0:00 (sd-pam)
u07khl00 835518 0.0 0.0 81400 780 ? Ss Apr11 0:00 gpg-agent --homedir /home/u07khl00/spack/opt/spack/gpg --use-standard-socket --daemon
root 964349 0.0 0.0 17788 10808 ? Ss Apr12 0:00 sshd: u07khl00 [priv]
u07khl00 964447 0.0 0.0 34428 24588 ? S Apr12 3:34 sshd: u07khl00@notty
u07khl00 964448 0.0 0.0 8900 5068 ? Ss Apr12 0:03 -bash
root 2077953 0.0 0.0 17784 11032 ? Ss Apr18 0:00 sshd: u07khl00 [priv]
u07khl00 2078051 0.0 0.0 34504 24556 ? S Apr18 2:44 sshd: u07khl00@notty
u07khl00 2078052 0.0 0.0 8900 5148 ? Ss Apr18 0:00 -bash
root 2130523 0.0 0.0 17792 10872 ? Ss 04:47 0:00 sshd: u07khl00 [priv]
u07khl00 2130624 0.0 0.0 17792 7908 ? S 04:47 0:00 sshd: u07khl00@notty
u07khl00 2130627 0.0 0.0 8900 5128 ? Ss 04:47 0:00 -bash
root 2130828 0.0 0.0 17784 10916 ? Ss 04:50 0:00 sshd: u07khl00 [priv]
u07khl00 2130887 0.0 0.0 17784 7916 ? S 04:51 0:00 sshd: u07khl00@pts/0
u07khl00 2130888 0.0 0.0 9936 6700 pts/0 Ss 04:51 0:00 -bash
u07khl00 2130989 0.0 0.0 178480 5340 pts/0 Sl 04:52 0:00 /home//u07khl00/spack/opt/spack/linux-ubuntu22.04-skylake_avx512/gcc-12.3.0/isaac-server-develop-6fqx5sc3wayrsipstou6utlufvhi2hmn/bin/isaac
u07khl00 2130990 0.0 0.0 10312 2488 pts/0 R+ 04:52 0:00 ps aux
u07khl00 2130991 0.0 0.0 6636 2308 pts/0 S+ 04:52 0:00 grep --color=auto u07khl00
use client "visualisation.html" to connect to isaac server; server stdout:
the message callback_isaac: LWS_CALLBACK_ESTABLISHED is observed.
callback_isaac: LWS_CALLBACK_FILTER_PROTOCOL_CONNECTION
ISAAC Connection from pca1 (172.21.31.1)!
callback_isaac: LWS_CALLBACK_ESTABLISHED
the web page screenshot
I use default example, LWFA, and modify isaac.param
#pragma once
#include "picongpu/particles/particleToGrid/derivedAttributes/DerivedAttributes.def"
namespace picongpu
{
namespace isaacP
{
namespace deriveField = particles::particleToGrid;
using Particle_Seq = VectorAllSpecies;
using Native_Seq = MakeSeq_t<FieldE, FieldB, FieldJ>;
using Density_Seq = deriveField::CreateEligible_t<Particle_Seq, deriveField::derivedAttributes::Density>;
using Fields_Seq = MakeSeq_t<Native_Seq, Density_Seq>;
using VectorFields_Seq = Native_Seq;
} // namespace isaacP
} // namespace picongpu
use 4_isaac.cfg
TBG_wallTime="1:00:00"
TBG_devices_x=1
TBG_devices_y=4
TBG_devices_z=1
TBG_gridSize="192 2048 192"
TBG_steps="2048"
TBG_restartLoop="--checkpoint.restart.loop 10000"
TBG_isaac="--isaac.period 1"
TBG_plugins="!TBG_isaac"
TBG_deviceDist="!TBG_devices_x !TBG_devices_y !TBG_devices_z"
TBG_programParams="-d !TBG_deviceDist \
-g !TBG_gridSize \
-s !TBG_steps \
!TBG_restartLoop \
!TBG_plugins \
--versionOnce"
TBG_tasks="$(( TBG_devices_x * TBG_devices_y * TBG_devices_z ))"
"$TBG_cfgPath"/submitAction.sh
build PIConGPU
pic-build -c"-DPIC_USE_ISAAC=ON -DPIC_USE_openPMD=ON -DPIC_USE_PNGwriter=ON"
The stdout is pretty lengthy, so I attach it as a file
PIConGPU compilation.txt
run PIConGPU
I found that the moving window is OFF by default
tbg -s bash -c etc/picongpu/4_isaac.cfg -t etc/picongpu/bash/mpiexec.tpl $SCRATCH/runs/myLWFA_test1
this time, the ID numbers are the same in the terminal stdout, New connection, giving id 0 (control), and in the web page (screenshot in the following)
$> tbg -s bash -c etc/picongpu/4_isaac.cfg -t etc/picongpu/bash/mpiexec.tpl $SCRATCH/runs/myLWFA_test1
Running program...
export env vars & load pkgs for PIConGPU ......
-- linux-ubuntu22.04-skylake_avx512 / gcc@12.3.0 ----------------
boost@1.83.0 cmake@3.27.7 cuda@12.2.1 glm@develop hdf5@1.14.3 icet@2.1.1 isaac@develop isaac-server@develop jansson@2.14 libjpeg-turbo@3.0.0 libpng@1.6.39 libwebsockets@2.2.1 openmpi@4.1.6 openpmd-api@0.15.2 pngwriter@0.7.0
==> 15 loaded packages
PIConGPU: 0.8.0-dev
Build-Type: Release
Third party:
OS: Linux-5.15.0-101-generic
arch: x86_64
CXX: GNU (12.3.0)
CMake: 3.27.7
CUDA: 12.2.128
mallocMC: 2.6.0
Boost: 1.83.0
MPI:
standard: 3.1
flavor: OpenMPI (4.1.6)
PNGwriter: 0.7.0
openPMD: 0.15.2
PIConGPUVerbose PHYSICS(1) | Sliding Window is OFF
1: -1
2: -1
3: 1
4: -1
5: -1
6: -1
7: -1
8: -1
9: -1
10: -1
11: -1
12: -1
13: -1
14: -1
15: -1
16: -1
17: -1
18: -1
19: -1
20: -1
21: -1
22: -1
23: -1
24: -1
25: -1
26: -1
New connection, giving id 0 (control)
Group complete, sending to connected interfaces
PIConGPUVerbose PHYSICS(1) | used Random Number Generator: RNGProvider3XorMin seed: 42
PIConGPUVerbose PHYSICS(1) | Field solver condition: c * dt <= 1.00229 ? (c * dt = 1)
PIConGPUVerbose PHYSICS(1) | Incident field "GaussianPulse_with_GaussianPulse" numerical dispersion: v_phase = 0.999412 * c
PIConGPUVerbose PHYSICS(1) | Resolving plasma oscillations?
Estimates are based on DensityRatio to BASE_DENSITY of each species
(see: density.param, speciesDefinition.param).
It does not cover other forms of initialization
PIConGPUVerbose PHYSICS(1) | species e: omega_p * dt <= 0.1 ? (omega_p * dt = 0.0247974)
PIConGPUVerbose PHYSICS(1) | macro particles per device: 37748736
PIConGPUVerbose PHYSICS(1) | typical macro particle weighting: 6955.06
PIConGPUVerbose PHYSICS(1) | UNIT_SPEED 2.99792e+08
PIConGPUVerbose PHYSICS(1) | UNIT_TIME 1.39e-16
PIConGPUVerbose PHYSICS(1) | UNIT_LENGTH 4.16712e-08
PIConGPUVerbose PHYSICS(1) | UNIT_MASS 6.33563e-27
PIConGPUVerbose PHYSICS(1) | UNIT_CHARGE 1.11432e-15
PIConGPUVerbose PHYSICS(1) | UNIT_EFIELD 1.22627e+13
PIConGPUVerbose PHYSICS(1) | UNIT_BFIELD 40903.8
PIConGPUVerbose PHYSICS(1) | UNIT_ENERGY 5.69418e-10
PIConGPUVerbose PHYSICS(1) | Resolving Debye length for species "e"?
PIConGPUVerbose PHYSICS(1) | Estimate used momentum variance in 288000 supercells with at least 10 macroparticles each
PIConGPUVerbose PHYSICS(1) | 288000 (100 %) supercells had local Debye length estimate not resolved by a single cell
PIConGPUVerbose PHYSICS(1) | Estimated weighted average temperature 0 keV and corresponding Debye length 0 m.
The grid has 0 cells per average Debye length
initialization time: 20sec 655msec = 20.655 sec
0 % = 0 | time elapsed: 28msec | avg time per step: 0msec
4 % = 102 | time elapsed: 4sec 62msec | avg time per step: 23msec
9 % = 204 | time elapsed: 8sec 111msec | avg time per step: 23msec
14 % = 306 | time elapsed: 12sec 250msec | avg time per step: 24msec
19 % = 408 | time elapsed: 16sec 590msec | avg time per step: 25msec
the web page becomes
use the "observe" button to see the visualization
a message showed up in the terminal URIImageConnector: Openend URI JPEG Stream
I tried the max/min interval button
I played around the sliders, and of course pushed the submit/update button
the web page screenshots
try different slider/colormap settings
just found that I pick the wrong weight option. will redo this test and update later.
I assume the problem is that you installed ISAAC with spack. Due to time constrains we stopped maintaining the spack packages for ISAAC.
You can compile all dependencies with spack but currently you should compile ISAAC by hand.
You can grep all required commands from our CI job:
picongpu/share/ci/install/isaac.sh
Lines 9 to 27 in d18b682
And for the server the install description of ISAAC should work fine if you provide libwebsockets with spack https://github.com/ComputationalRadiationPhysics/isaac/blob/dev/INSTALL.md#the-server
@onezillion Did building the software stack, as @psychocoderHPC recommended, solved our problem?
Hi all,
Sorry for my late response. I recompiled isaac from source for isaac/lib and isaac/server.
All other dependencies are the same as before (spack).
I also git fetch and git pull in picongpu folder.
the pic-build runs smoothly
When opening the client web page and clicking the observe button, the corresponding server messages also show up
But still no visualization:
Tried the sliders, but still no luck:
Since this does not occur commonly, I guess it may be related to our VM machine?
Tried the sliders, but still no luck:
Do you klicked on Update weights after changing the sliders?
Since this does not occur commonly, I guess it may be related to our VM machine?
What do you mean by this? Is it sometimes working?
Which browser do you use? I suggest testing firefox and Chrome if you not already tested these.
There is on option for the server what all images get stored as jpg in the server folder.
Maybe you can try to use this option and change sliders in the webinterface and check if the jpg's has non black images.
In that case this will point to a connection/port issue from the server to the web client.
I do not think that the VM is the issue.
Thank you for your reply.
Do you klicked on
Update weightsafter changing the sliders?
Yes, I did click the update button. I even clicked on all the update buttons that I could find in the page
Since this does not occur commonly, I guess it may be related to our VM machine?
What do you mean by this? Is it sometimes working?
No, I mean: since it seems that this issue only happened to me (I do not see similar case being reported except one, which was quickly solved somehow), I guess that maybe certain settings in our VM machine lead to this issue.
Which browser do you use? I suggest testing firefox and Chrome if you not already tested these.
I am using chrome, also tried firefox and edge when I first tried isaac and found no visualization. That was like 2-3 weeks ago. At that time, I also turned on jpeg output, and all I got was just blank & black images. Anyway, let me try this again with manually compiled isaac later.
Actually, my very first try was with only jpeg output and without server connection and found all are just blank & black images. Then I tried the server connection. Then you know the results.
Dear all,
I tried jpeg output from isaac server but no luck. Here is the screenshot and a jpeg file. I also tried edge, chrome, chrome in privacy and firefox in the screenshot. All other settings are the same as my last shot: isaac from source and other deps from spack. I also did git fetch && git pull for picongpu.
isaac server is loaded by $ISAAC_server_DIR/isaac --dump ~/isaac_plots --url 127.0.0.1 & :
Could it be due to libjpeg-turbo? If that is possible, I will try install libjpeg from source later.
Could it be due to libjpeg-turbo? If that is possible, I will try install libjpeg from source later.
This could be. I suggest try to compile it. I am out of ideas if this is not working :-(
You could also try to change the background color of the simulation, under the simulation image in the web control interface.
This will show if at least the image is transfered correctly or if the rendering is broken.
Changing background color from control page also changes the jpeg output from isaac server.
Does this imply that libjpeg-turbo works well? I totally have no clue now.
Anyway, I will try re-compile libjpeg-turbo from source later.
Does this imply that libjpeg-turbo works well?
I assume because if I remember correctly the background is already rendered into the image on the device.
I will try it asap by my own to see if I get the full chain running by myself.
@onezillion I tried it now by my own on our dev server, here are some snippets how I compiled. I mixed spack and glm compile by hand because I have not tested the latest glm releases.
install
export GLM_ROOT=$HOME/lib/glm/0.9.9.9-dev
export CMAKE_PREFIX_PATH=$GLM_ROOT:$CMAKE_PREFIX_PATH
git clone https://github.com/g-truc/glm.git
cd glm
git checkout 6ad79aae3eb5bf809c30bf1168171e9e55857e45
mkdir build
cd build
cmake ../ -DCMAKE_INSTALL_PREFIX=$GLM_ROOT -DGLM_TEST_ENABLE=OFF
make install
cd ../..
git clone https://github.com/ComputationalRadiationPhysics/isaac.git
cd isaac
mkdir build_isaac
cd build_isaac
export ISAAC_ROOT=$HOME/lib/isaac
export CMAKE_PREFIX_PATH=$ISAAC_ROOT:$CMAKE_PREFIX_PATH
cmake ../lib/ -DCMAKE_INSTALL_PREFIX=$ISAAC_ROOT
make install
cd ../..
spack install icet
spack install jansson
spack install libjpeg-turbo
spack install libwebsockets
compile picongpu
export ISAAC_ROOT=$HOME/lib/isaac
export CMAKE_PREFIX_PATH=$ISAAC_ROOT:$CMAKE_PREFIX_PATH
export GLM_ROOT=$HOME/lib/glm/0.9.9.9-dev
export CMAKE_PREFIX_PATH=$GLM_ROOT:$CMAKE_PREFIX_PATH
# I have problems that icet is not found by cmake therefore I set the prefix path to lib
export CMAKE_PREFIX_PATH=/opt/spack-modules/opt/spack/linux-ubuntu22.04-zen2/gcc-12.2.0/icet-2.1.1-obktzqgpbwaad3qzhpe6xzqxsi3t72ji/lib:$CMAKE_PREFIX_PATH
pic-create ~/workspace/picongpu/share/picongpu/examples/LaserWakefield/ lwfa
cd lwfa
pic-build -b cuda:80
compile server
spack install jansson
spack install libwebsockets
# set openssl include path by hand because of missing include in libwebsockets (I have not investigated the source of the issue)
cmake ../isaac/server/ -DCMAKE_CXX_FLAGS=-I/opt/spack-modules/opt/spack/linux-ubuntu22.04-zen2/gcc-12.2.0/openssl-3.1.3-mqsvskp6tc3zonaqddgvkzmnteaccgm5/include
start server
./isaac
Using web_port=2459, tcp_port=2458 and sim_port=2460
lwsts[1526522]: Initial logging level 1031
lwsts[1526522]: Libwebsockets version: 2.2.0 rwidera@hal8999-
lwsts[1526522]: IPV6 not compiled in
lwsts[1526522]: libev support not compiled in
lwsts[1526522]: libuv support not compiled in
lwsts[1526522]: Threads: 1 each 1024 fds
lwsts[1526522]: mem: platform fd map: 8192 bytes
lwsts[1526522]: Compiled with OpenSSL support
lwsts[1526522]: SSL disabled: no LWS_SERVER_OPTION_DO_SSL_GLOBAL_INIT
lwsts[1526522]: Creating Vhost 'default' port 2459, 2 protocols, IPv6 off
lwsts[1526522]: Listening on port 2459
lwsts[1526522]: mem: per-conn: 568 bytes + protocol rx buf
lwsts[1526522]: canonical_hostname = hal8999start PIConGPU LWFA
./bin/picongpu -d 1 1 1 -g 128 512 128 --periodic 1 0 1 -s 1000 --checkpoint.restart.loop 1000 --isaac.period 1 --isaac.name lwfa
I got images in the client without changing any slider.
@onezillion take care I increased the required glm version for ISAAC to 1.0.0 ComputationalRadiationPhysics/isaac#181
Thanks for the reply. Our VM is down for a few days and may be up next week. I will try the suggestions then.
@onezillion which version of spack do you use?