(Tested with Python 3.12)
Create and activate venv, and install pip packages:
uv venv
source .venv/bin/activate
uv pip install -r requirements.txt
Manually alter requirements.in if needed and update requirements.txt:
uv pip compile requirements.in -o requirements.txt
- Develop using Jupyter:
jupyter-lab - Test dashboard:
panel serve app.ipynb
NB: These are not stable instructions yet!
Configure the token access on your machine if not already done (stores file at ~/.viresclient.ini):
viresclient set_token https://vires.services/ows
viresclient set_default_server https://vires.services/ows
(This runs the CLI provided by SwarmPAL)
Generate a FAC file from Swarm Alpha between two times:
swarmpal fac-single-sat --spacecraft Swarm-A --grade "FAST" --time_start "2024-03-19" --time_end "2024-03-20" --output "test-A.cdf"
Check the latest input product availability:
swarmpal last-available-time "SW_FAST_MAGA_LR_1B"
(This runs the processor continuously to generate new FAC files locally as new data is available, and uploads them via FTP)
Configure the FTP server credentials in a .env file in the root of SwarmPAL-processor:
FTP_SERVER="..."
FTP_USERNAME="..."
FTP_PASSWORD="..."
Run for a specific satellite:
cd tasks
mkdir -p outputs/Sat_A outputs/Sat_B outputs/Sat_C
python fac-fast-processor.py A outputs/Sat_A FAC/TMS/Sat_A
Data is stored locally in ./outputs/Sat_A and uploaded to FAC/TMS/Sat_A on the server. The remote file structure mimics https://swarm-diss.eo.esa.int/#swarm/Level2daily/Latest_baselines/FAC and is currently running as a demonstration uploaded at https://swarmdisc.org/swarmpal-data-test/FAC
Some problems:
- this does not mimic the behaviour of source data (https://swarm-diss.eo.esa.int/#swarm/Fast/Level1b/MAGx_LR) where newer data can supersede old data
- ftp upload should be handled separately and retried in the background when it fails (rather than halting the whole job)
- needs to gracefully handle errors and retry after a few minutes if there is failure
Build the image:
podman build -t swarmpal-processor .
Get a VirES access token from https://vires.services/accounts/tokens/ and store it in a file called .env:
VIRES_TOKEN=.....
To start the panel server:
podman compose up
or equivalently:
podman run --rm -it -p 5006:5006 --env-file .env -d swarmpal-processor /app/start-dashboard.sh