Weird CPU Memory Usage

[suptimq]

I am running nerfstudio on a server with 8 Nvidia A6000 GPUs. I am pretty sure I could use all for training and everything was fine. However, recently, I noticed the training time increases significantly (from 10-ish minutes to 2-ish hours) when I submitted the 3rd training job.

I am using single GPU without masks in the training.

I am wondering if anyone has encountered such behavior before.

The training script:

ns-train nerfacto \
        --vis wandb \
        --experiment-name ""  \
        --max-num-iterations 30000 \
        --pipeline.model.camera-optimizer.mode off \
        nerfstudio-data --data 

[suptimq]

Resolved the issue by setting

export OMP_NUM_THREADS=32
export NUMEXPR_NUM_THREADS=32
export MKL_NUM_THREADS=32

[Mariusmarten]

I had a similar issue when training Nerfacto in a similar setup. Setting the number of threads to 8 as described worked. Thanks.