LSF profile is broken for newest 7.1.1 (2022-03-07) snakemake version
Closed this issue · 4 comments
Error:
Submitted group job 29137fca-fc8a-5511-88d3-2149b00a8b5e with external jobid '5412741 logs/cluster/group_1/unique/jobid29137fca_07818911-9e6a-460c-a46e-b0f3f0bdb675.out'.
Traceback (most recent call last):
File "/homes/leandro/.config/snakemake/lsf/lsf_status.py", line 201, in <module>
jobid = int(sys.argv[1])
ValueError: invalid literal for int() with base 10: '5412741 logs/cluster/group_1/unique/jobid29137fca_07818911-9e6a-460c-a46e-b0f3f0bdb675.out'
WorkflowError:
Failed to obtain job status. See above for error message.
The reason is that on snakemake 7.1.1 the job id is now quoted. LSF profile works on snakemake 7.1.0 though. Will provide a fix soon
Thanks for pushing this fix. I can confirm that it worked for me using snakemake v7.3.8.
I'm running into an issue where the default memory limit doesn't seem to be getting pulled from the Cookiecutter profile, and I'm not sure if this is again due to the new snakemake version.
In my Cookiecutter profile (copied at bottom), I set 16 GB as the default memory limit, but this setting doesn't seem to get propagated. When I run a basic rule, the rule uses 1000 MB. But explicitly setting the memory limit to 2000 MB for that rule works fine.
(minimal_snakemake) [litichev@node157 ~]$ cat Snakefile
rule make_file:
output:
"new_file.txt"
shell:
"touch {output}"
(minimal_snakemake) [litichev@node157 ~]$ snakemake --profile lsf
Building DAG of jobs...
Using shell: /bin/bash
Provided cluster nodes: 500
Job stats:
job count min threads max threads
--------- ------- ------------- -------------
make_file 1 1 1
total 1 1 1
Select jobs to execute...
[Tue Apr 12 21:24:19 2022]
rule make_file:
output: new_file.txt
jobid: 0
resources: mem_mb=1000, disk_mb=1000, tmpdir=/tmp
touch new_file.txt
Submitted job 0 with external jobid '72762310 logs/cluster/make_file/unique/jobid0_07ecd567-f599-4180-863f-9fa2895a48b3.out'.
[Tue Apr 12 21:24:39 2022]
Finished job 0.
1 of 1 steps (100%) done
Complete log: .snakemake/log/2022-04-12T212418.851163.snakemake.log
(minimal_snakemake) [litichev@node157 ~]$ vi Snakefile
(minimal_snakemake) [litichev@node157 ~]$ cat Snakefile
rule make_file:
output:
"new_file.txt"
resources:
mem_mb=2000
shell:
"touch {output}"
(minimal_snakemake) [litichev@node157 ~]$ snakemake --profile lsf --force
Building DAG of jobs...
Using shell: /bin/bash
Provided cluster nodes: 500
Job stats:
job count min threads max threads
--------- ------- ------------- -------------
make_file 1 1 1
total 1 1 1
Select jobs to execute...
[Tue Apr 12 21:25:11 2022]
rule make_file:
output: new_file.txt
jobid: 0
resources: mem_mb=2000, disk_mb=1000, tmpdir=/tmp
touch new_file.txt
Submitted job 0 with external jobid '72762311 logs/cluster/make_file/unique/jobid0_89c6ce42-fe30-453b-923b-32254835bc3e.out'.
[Tue Apr 12 21:25:31 2022]
Finished job 0.
1 of 1 steps (100%) done
Complete log: .snakemake/log/2022-04-12T212511.483596.snakemake.log
My Cookiecutter profile:
(minimal_snakemake) [litichev@node157 ~]$ cat ~/.config/snakemake/lsf/CookieCutter.py
class CookieCutter:
"""
Cookie Cutter wrapper
"""
@staticmethod
def get_default_threads() -> int:
return int("8")
@staticmethod
def get_default_mem_mb() -> int:
return int("16384")
@staticmethod
def get_log_dir() -> str:
return "logs/cluster"
@staticmethod
def get_default_queue() -> str:
return ""
@staticmethod
def get_lsf_unit_for_limits() -> str:
return "MB"
@staticmethod
def get_unknwn_behaviour() -> str:
return "wait"
@staticmethod
def get_zombi_behaviour() -> str:
return "ignore"
@staticmethod
def get_latency_wait() -> float:
return float("30")
Package versions in my conda environment:
(minimal_snakemake) [litichev@node157 ~]$ conda list
# packages in environment at /home/litichev/miniconda3/envs/minimal_snakemake:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_gnu conda-forge
aioeasywebdav 2.4.0 py39hf3d152e_1001 conda-forge
aiohttp 3.8.1 py39hb9d737c_1 conda-forge
aiosignal 1.2.0 pyhd8ed1ab_0 conda-forge
amply 0.1.4 py_0 conda-forge
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
async-timeout 4.0.2 pyhd8ed1ab_0 conda-forge
attmap 0.13.2 pyhd8ed1ab_0 conda-forge
attrs 21.4.0 pyhd8ed1ab_0 conda-forge
backports 1.0 py_2 conda-forge
backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge
bcrypt 3.2.0 py39hb9d737c_3 conda-forge
boto3 1.21.38 pyhd8ed1ab_0 conda-forge
botocore 1.24.38 pyhd8ed1ab_0 conda-forge
brotlipy 0.7.0 py39hb9d737c_1004 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.18.1 h7f98852_0 conda-forge
ca-certificates 2021.10.8 ha878542_0 conda-forge
cachetools 5.0.0 pyhd8ed1ab_0 conda-forge
certifi 2021.10.8 py39hf3d152e_2 conda-forge
cffi 1.15.0 py39h4bc2ebd_0 conda-forge
charset-normalizer 2.0.12 pyhd8ed1ab_0 conda-forge
coin-or-cbc 2.10.7 h3786ebc_0 conda-forge
coin-or-cgl 0.60.3 he7e83c3_2 conda-forge
coin-or-clp 1.17.6 h256e9bb_3 conda-forge
coin-or-osi 0.108.6 h3b589db_2 conda-forge
coin-or-utils 2.11.6 h573740c_0 conda-forge
coincbc 2.10.7 0_metapackage conda-forge
configargparse 1.5.3 pyhd8ed1ab_0 conda-forge
connection_pool 0.0.3 pyhd3deb0d_0 conda-forge
cryptography 36.0.2 py39hd97740a_1 conda-forge
datrie 0.8.2 py39h3811e60_3 conda-forge
decorator 5.1.1 pyhd8ed1ab_0 conda-forge
defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge
docutils 0.18.1 py39hf3d152e_1 conda-forge
dropbox 11.29.0 pyhd8ed1ab_0 conda-forge
filechunkio 1.8 py_2 conda-forge
filelock 3.6.0 pyhd8ed1ab_0 conda-forge
frozenlist 1.3.0 py39hb9d737c_1 conda-forge
ftputil 5.0.3 pyhd8ed1ab_0 conda-forge
gitdb 4.0.9 pyhd8ed1ab_0 conda-forge
gitpython 3.1.27 pyhd8ed1ab_0 conda-forge
google-api-core 2.5.0 pyhd8ed1ab_0 conda-forge
google-api-python-client 2.44.0 pyhd8ed1ab_0 conda-forge
google-auth 2.6.3 pyh6c4a22f_0 conda-forge
google-auth-httplib2 0.1.0 pyhd8ed1ab_0 conda-forge
google-cloud-core 2.2.2 pyh6c4a22f_0 conda-forge
google-cloud-storage 2.1.0 pyh6c4a22f_0 conda-forge
google-crc32c 1.1.2 py39hb81f231_2 conda-forge
google-resumable-media 2.1.0 pyh6c4a22f_0 conda-forge
googleapis-common-protos 1.56.0 py39hf3d152e_0 conda-forge
grpcio 1.45.0 py39h0f497a6_0 conda-forge
httplib2 0.20.4 pyhd8ed1ab_0 conda-forge
idna 3.3 pyhd8ed1ab_0 conda-forge
importlib-metadata 4.11.3 py39hf3d152e_1 conda-forge
importlib_resources 5.6.0 pyhd8ed1ab_1 conda-forge
iniconfig 1.1.1 pyh9f0ad1d_0 conda-forge
jinja2 3.1.1 pyhd8ed1ab_0 conda-forge
jmespath 1.0.0 pyhd8ed1ab_0 conda-forge
jsonschema 4.4.0 pyhd8ed1ab_0 conda-forge
jupyter_core 4.9.2 py39hf3d152e_0 conda-forge
ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge
libblas 3.9.0 14_linux64_openblas conda-forge
libcblas 3.9.0 14_linux64_openblas conda-forge
libcrc32c 1.1.2 h9c3ff4c_0 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 11.2.0 h1d223b6_15 conda-forge
libgfortran-ng 11.2.0 h69a702a_15 conda-forge
libgfortran5 11.2.0 h5c6108e_15 conda-forge
libgomp 11.2.0 h1d223b6_15 conda-forge
liblapack 3.9.0 14_linux64_openblas conda-forge
liblapacke 3.9.0 14_linux64_openblas conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libopenblas 0.3.20 pthreads_h78a6416_0 conda-forge
libprotobuf 3.20.0 h6239696_0 conda-forge
libsodium 1.0.18 h36c2ea0_1 conda-forge
libstdcxx-ng 11.2.0 he4da1e4_15 conda-forge
libuuid 2.32.1 h7f98852_1000 conda-forge
libzlib 1.2.11 h166bdaf_1014 conda-forge
logmuse 0.2.6 pyh8c360ce_0 conda-forge
markupsafe 2.1.1 py39hb9d737c_1 conda-forge
multidict 6.0.2 py39hb9d737c_1 conda-forge
nbformat 5.3.0 pyhd8ed1ab_0 conda-forge
ncurses 6.3 h27087fc_1 conda-forge
numpy 1.22.3 py39h18676bf_2 conda-forge
oauth2client 4.1.3 py_0 conda-forge
openssl 1.1.1n h166bdaf_0 conda-forge
packaging 21.3 pyhd8ed1ab_0 conda-forge
pandas 1.4.2 py39h1832856_1 conda-forge
paramiko 2.10.3 pyhd8ed1ab_0 conda-forge
peppy 0.31.2 pyhd8ed1ab_2 conda-forge
pip 22.0.4 pyhd8ed1ab_0 conda-forge
plac 1.3.5 pyhd8ed1ab_0 conda-forge
pluggy 1.0.0 py39hf3d152e_3 conda-forge
ply 3.11 py_1 conda-forge
prettytable 3.2.0 pyhd8ed1ab_0 conda-forge
protobuf 3.20.0 py39h5a03fae_4 conda-forge
psutil 5.9.0 py39hb9d737c_1 conda-forge
pulp 2.6.0 py39hf3d152e_1 conda-forge
py 1.11.0 pyh6c4a22f_0 conda-forge
pyasn1 0.4.8 py_0 conda-forge
pyasn1-modules 0.2.7 py_0 conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pygments 2.11.2 pyhd8ed1ab_0 conda-forge
pynacl 1.5.0 py39hb9d737c_1 conda-forge
pyopenssl 22.0.0 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.8 pyhd8ed1ab_0 conda-forge
pyrsistent 0.18.1 py39hb9d737c_1 conda-forge
pysftp 0.2.9 py_1 conda-forge
pysocks 1.7.1 py39hf3d152e_5 conda-forge
pytest 7.1.1 py39hf3d152e_1 conda-forge
python 3.9.12 h9a8a25e_1_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-fastjsonschema 2.15.3 pyhd8ed1ab_0 conda-forge
python-irodsclient 1.1.3 pyhd8ed1ab_0 conda-forge
python_abi 3.9 2_cp39 conda-forge
pytz 2022.1 pyhd8ed1ab_0 conda-forge
pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge
pyyaml 6.0 py39hb9d737c_4 conda-forge
ratelimiter 1.2.0 py_1002 conda-forge
readline 8.1 h46c0cb4_0 conda-forge
requests 2.27.1 pyhd8ed1ab_0 conda-forge
retry 0.9.2 py_0 conda-forge
rsa 4.8 pyhd8ed1ab_0 conda-forge
s3transfer 0.5.2 pyhd8ed1ab_0 conda-forge
setuptools 62.1.0 py39hf3d152e_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
slacker 0.14.0 py_0 conda-forge
smart_open 5.2.1 pyhd8ed1ab_0 conda-forge
smmap 3.0.5 pyh44b312d_0 conda-forge
snakemake 7.3.8 hdfd78af_0 bioconda
snakemake-minimal 7.3.8 pyhdfd78af_0 bioconda
sqlite 3.38.2 h4ff8645_0 conda-forge
stone 3.3.1 pyhd8ed1ab_0 conda-forge
stopit 1.1.2 py_0 conda-forge
tabulate 0.8.9 pyhd8ed1ab_0 conda-forge
tk 8.6.12 h27826a3_0 conda-forge
tomli 2.0.1 pyhd8ed1ab_0 conda-forge
toposort 1.7 pyhd8ed1ab_0 conda-forge
traitlets 5.1.1 pyhd8ed1ab_0 conda-forge
typing-extensions 4.1.1 hd8ed1ab_0 conda-forge
typing_extensions 4.1.1 pyha770c72_0 conda-forge
tzdata 2022a h191b570_0 conda-forge
ubiquerg 0.6.1 pyh9f0ad1d_0 conda-forge
uritemplate 4.1.1 pyhd8ed1ab_0 conda-forge
urllib3 1.26.9 pyhd8ed1ab_0 conda-forge
veracitools 0.1.3 py_0 conda-forge
wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge
wheel 0.37.1 pyhd8ed1ab_0 conda-forge
wrapt 1.14.0 py39hb9d737c_1 conda-forge
xz 5.2.5 h516909a_1 conda-forge
yaml 0.2.5 h7f98852_2 conda-forge
yarl 1.7.2 py39hb9d737c_2 conda-forge
yte 1.2.2 py39hf3d152e_0 conda-forge
zipp 3.8.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.11 h166bdaf_1014 conda-forge
Any thoughts about why the memory limit in the Cookiecutter profile isn't being used by my rule? Thanks very much.
Thanks for the bug report. I tested running your rule without the resources
directive:
rule make_file:
output:
"new_file.txt"
shell:
"touch {output}"
and this is the job script the LSF profile gets:
#!/bin/sh
# properties = {"type": "single", "rule": "make_file", "local": false, "input": [], "output": ["new_file.txt"], "wildcards": {}, "params": {}, "log": [], "threads": 1, "resources": {"mem_mb": 1000, "disk_mb": 1000, "tmpdir": "/hps/nobackup/research/zi/leandro/temp"}, "jobid": 1, "cluster": {}}
cd /hps/nobackup/research/zi/leandro/snakemake_test && /hps/nobackup/research/zi/leandro/miniconda3/bin/python -m snakemake --snakefile '/hps/nobackup/research/zi/leandro/snakemake_test/Snakefile' 'new_file.txt' --allowed-rules 'make_file' --cores 'all' --attempt 1 --force-use-threads --wait-for-files '/hps/nobackup/research/zi/leandro/snakemake_test/.snakemake/tmp.y4ulcv9w' --force --keep-target-files --keep-remote --max-inventory-time 0 --nocolor --notemp --no-hooks --nolock --ignore-incomplete --skip-script-cleanup --conda-frontend 'mamba' --wrapper-prefix 'https://github.com/snakemake/snakemake-wrappers/raw/' --latency-wait 10 --scheduler 'greedy' --scheduler-solver-path '/hps/nobackup/research/zi/leandro/miniconda3/bin' --default-resources 'mem_mb=max(2*input.size_mb, 1000)' 'disk_mb=max(2*input.size_mb, 1000)' 'tmpdir=system_tmpdir' --mode 2 && exit 0 || exit 1
Even though we have not specified a mem_mb
for this rule, snakemake
tells us that this is the resources for this rule in the properties of the job script: "resources": {"mem_mb": 1000, "disk_mb": 1000, "tmpdir": "/hps/nobackup/research/zi/leandro/temp"}
, and this has to be prioritised over the profile's default memory, because usually the user does specify mem_mb
. I think this is the same issue as the default threads: snakemake
has defaults for mem_mb
if the user does not specify it, and there is no way for us to differentiate between the user specifying that a rule needs 1000 MB of RAM or not specifying the mem_mb
directive (and snakemake
implicitly putting 1000 MB of RAM as default). In other words, I think the solution for this is to actually remove default_mem_mb
from the cookie-cutter, and let snakemake
controls the default mem_mb
.
PS: the job script snakemake
produces has the following default resources line: --default-resources 'mem_mb=max(2*input.size_mb, 1000)' 'disk_mb=max(2*input.size_mb, 1000)' 'tmpdir=system_tmpdir'
. We could potentially change the job script so that the default mem_mb
matches LSF profile default mem_mb
, but this would not be enough as we would still need to know if the user explicitly asked for some specific mem_mb
or not, in order to know how much memory to request when bsub
ing the job.