(1.6.21) Not able to use all available CPU threads in the later part of VEP

[sclan]

The dockerized command and outputs, along with dstat lines (set one entry per 10 minutes during the run) on a 36 threaded computer. With "--vep-forks 45", the observation is ~23 cpu threads are used (it fluctuates) during the first 2 hours. Then it decreased to a single threaded (36 * 3% = 1) process and I wonder why. The VEP runs on the local VEP cache downloaded from Ensembl.

$ perl /opt/vcf2maf.pl --input-vcf file.g.vcf --output-maf file.g.maf --ref-fasta ref.fasta --species homo_sapiens --ncbi-build GRCh37 --tmp-dir /tmp --vep-path /usr/local/bin --vep-data .vep --vep-forks 45

CPU: 1% (36 cores) * Memory: 12617/70235MB * Storage: 36/841GB * Net: 0↓/0↑MBps

STATUS: Running VEP and writing to: /tmp/file.g.vep.vcf
Possible precedence issue with control flow operator at /usr/local/lib/site_perl/5.26.2/Bio/DB/IndexedBase.pm line 805.

CPU: 38% (36 cores) * Memory: 16819/70235MB * Storage: 37/841GB * Net: 0↓/0↑MBps
CPU: 66% (36 cores) * Memory: 17682/70235MB * Storage: 40/841GB * Net: 0↓/0↑MBps
CPU: 65% (36 cores) * Memory: 18342/70235MB * Storage: 43/841GB * Net: 0↓/0↑MBps
CPU: 62% (36 cores) * Memory: 16353/70235MB * Storage: 46/841GB * Net: 0↓/0↑MBps
CPU: 63% (36 cores) * Memory: 19312/70235MB * Storage: 49/841GB * Net: 0↓/0↑MBps
CPU: 64% (36 cores) * Memory: 20737/70235MB * Storage: 52/841GB * Net: 0↓/0↑MBps
CPU: 65% (36 cores) * Memory: 16275/70235MB * Storage: 54/841GB * Net: 0↓/0↑MBps
CPU: 67% (36 cores) * Memory: 18305/70235MB * Storage: 57/841GB * Net: 0↓/0↑MBps
CPU: 65% (36 cores) * Memory: 16429/70235MB * Storage: 60/841GB * Net: 0↓/0↑MBps
CPU: 66% (36 cores) * Memory: 19054/70235MB * Storage: 63/841GB * Net: 0↓/0↑MBps
CPU: 68% (36 cores) * Memory: 20221/70235MB * Storage: 66/841GB * Net: 0↓/0↑MBps
CPU: 69% (36 cores) * Memory: 19972/70235MB * Storage: 69/841GB * Net: 0↓/0↑MBps
CPU: 69% (36 cores) * Memory: 20292/70235MB * Storage: 71/841GB * Net: 0↓/0↑MBps
WARNING: No genotype column for TUMOR in VCF!
WARNING: No genotype column for NORMAL in VCF!
CPU: 51% (36 cores) * Memory: 16459/70235MB * Storage: 73/841GB * Net: 0↓/0↑MBps
CPU: 3% (36 cores) * Memory: 16445/70235MB * Storage: 74/841GB * Net: 0↓/0↑MBps
CPU: 3% (36 cores) * Memory: 16431/70235MB * Storage: 74/841GB * Net: 0↓/0↑MBps
CPU: 3% (36 cores) * Memory: 16419/70235MB * Storage: 75/841GB * Net: 0↓/0↑MBps
CPU: 3% (36 cores) * Memory: 16403/70235MB * Storage: 76/841GB * Net: 0↓/0↑MBps
CPU: 3% (36 cores) * Memory: 16377/70235MB * Storage: 76/841GB * Net: 0↓/0↑MBps
CPU: 3% (36 cores) * Memory: 16404/70235MB * Storage: 77/841GB * Net: 0↓/0↑MBps
CPU: 3% (36 cores) * Memory: 16365/70235MB * Storage: 77/841GB * Net: 0↓/0↑MBps
CPU: 3% (36 cores) * Memory: 16381/70235MB * Storage: 78/841GB * Net: 0↓/0↑MBps
CPU: 3% (36 cores) * Memory: 16387/70235MB * Storage: 78/841GB * Net: 0↓/0↑MBps