OSError: libbgmg.so after RuntimeWarning
tobiaswolfram opened this issue · 10 comments
Hi, really cool tool! I worked through the tutorial and everything was running fine, but now I tried my own GWAS and after successfully debugging a few issues myself, I eventually failed. When running test1 after a succesful run of fit, I get this log:
INFO:root:__init__(lib_name=src/build/lib/libbgmg.so, context_id=0)
INFO:root:init_log(femshare.test.rep1.log)
INFO:root:log_message(***********************************************************************
* mixer.py: Univariate and Bivariate Causal Mixture for GWAS
* Version 1.2.0
* (c) 2016-2020 Oleksandr Frei, Alexey A. Shadrin, Dominic Holland
* Norwegian Centre for Mental Disorders Research / University of Oslo
* Center for Multimodal Imaging and Genetics / UCSD
* GNU General Public License v3
***********************************************************************
Call:
./mixer.py test1 \
--out femshare.test.rep1 \
--lib src/build/lib/libbgmg.so \
--bim-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.bim \
--ld-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.run4.ld \
--trait1-file femshare.csv.gz \
--load-params-file femshare.fit.rep1.json
)
INFO:root:__init__(lib_name=src/build/lib/libbgmg.so, context_id=0)
INFO:root:log_message(--fit-sequence: ['load', 'inflation']...)
INFO:root:log_message(fit_type==load...)
INFO:root:log_message(fit_type==load: Done, UnivariateParams(_pi: 0.003964433474557695, _sig2_beta: 5.936502354294048e-06, _sig2_zero: 1.050428396730295))
INFO:root:log_message(fit_type==load done (UnivariateParams(_pi: 0.003964433474557695, _sig2_beta: 5.936502354294048e-06, _sig2_zero: 1.050428396730295), {'params': {'pi': 0.003964433474557695, 'sig2_beta': 5.936502354294048e-06, 'sig2_zero': 1.050428396730295}}))
INFO:root:log_message(fit_type==inflation...)
INFO:root:log_message(fit_type==inflation done (UnivariateParams(_pi: 0.003964433474557695, _sig2_beta: 5.936502354294048e-06, _sig2_zero: 1.0352772554264185), AIC: 743816.169366054
BIC: 743826.6273385806
cost: 371907.084683027
cost_df: 1
cost_fast: 371907.08468302735
cost_n: 257293.4
final_simplex: (array([[0.03466927],
[0.03466912]]), array([371907.08468303, 371907.0847105 ]))
fun: 371907.084683027
message: 'Optimization terminated successfully.'
nfev: 45
nit: 19
params: {'pi': 0.003964433474557695, 'sig2_beta': 5.936502354294048e-06, 'sig2_zero': 1.0352772554264185}
status: 0
success: True
x: array([0.03466927])))
INFO:root:log_message(Calculate AIC/BIC w.r.t. infinitesimal model (fast cost function)...)
INFO:root:log_message(fit_type==infinitesimal...)
INFO:root:log_message(fit_type==infinitesimal done (UnivariateParams(_pi: 1.0, _sig2_beta: 2.7956413992854812e-08, _sig2_zero: 1.031896260367893), AIC: 743818.1471324271
BIC: 743839.0630774802
cost: 371907.07356621354
cost_df: 2
cost_fast: 371907.07356621366
cost_n: 257293.4
final_simplex: (array([[-17.39261918, 0.03139814],
[-17.39261919, 0.03139814],
[-17.39261936, 0.03139815]]), array([371907.07356621, 371907.07356621, 371907.07358108]))
fun: 371907.07356621354
message: 'Optimization terminated successfully.'
nfev: 105
nit: 45
params: {'pi': 1.0, 'sig2_beta': 2.7956413992854812e-08, 'sig2_zero': 1.031896260367893}
status: 0
success: True
x: array([-17.39261918, 0.03139814])))
/data/mixer/precimed/mixer/utils.py:894: RuntimeWarning: divide by zero encountered in log10
data_x=-np.log10(np.flip(np.cumsum(np.flip(data_weights[si])))) # step 2
/home/twolfram@ensae.fr/.local/lib/python3.8/site-packages/scipy/interpolate/_interpolate.py:651: RuntimeWarning: invalid value encountered in subtract
slope = (y_hi - y_lo) / (x_hi - x_lo)[:, None]
INFO:root:log_message(Done)
/usr/bin/python3: can't find '__main__' module in 'precimed'
run_mixer_script.sh: 42: --trait1-file: not found
run_mixer_script.sh: 43: --out: not found
run_mixer_script.sh: 44: --extract: not found
run_mixer_script.sh: 45: --bim-file: not found
run_mixer_script.sh: 46: --ld-file: not found
run_mixer_script.sh: 47: --lib: not found
Traceback (most recent call last):
File "precimed/mixer.py", line 18, in <module>
libbgmg = LibBgmg(args.lib)
File "/data/mixer/precimed/mixer/libbgmg.py", line 28, in __init__
self.cdll, self._lib_name = self._load_cdll(lib_name)
File "/data/mixer/precimed/mixer/libbgmg.py", line 464, in _load_cdll
raise OSError(exception_message)
OSError: libbgmg.so: cannot open shared object file: No such file or directory
Failed to load BGMG shared library from `['libbgmg.so', 'libbgmg.so']`. Try to add the location of `libbgmg.so` file into your PATH system variable, or to set BGMG_SHARED_LIBRARY - the specific system variable which may point to `libbgmg.so` file, including the full path.
run_mixer_script.sh: 50: --trait1-file: not found
run_mixer_script.sh: 51: --load-params-file: not found
run_mixer_script.sh: 52: --out: not found
run_mixer_script.sh: 53: --bim-file: not found
run_mixer_script.sh: 54: --ld-file: not found
run_mixer_script.sh: 55: --lib: not found
run_mixer_script.sh: 57: Fit: not found
python3: can't open file 'python/mixer.py': [Errno 2] No such file or directory
run_mixer_script.sh: 59: --trait1-file: not found
run_mixer_script.sh: 60: --trait2-file: not found
run_mixer_script.sh: 61: --trait1-params-file: not found
run_mixer_script.sh: 62: --trait2-params-file: not found
run_mixer_script.sh: 63: --out: not found
run_mixer_script.sh: 64: --extract: not found
run_mixer_script.sh: 65: --bim-file: not found
run_mixer_script.sh: 66: --ld-file: not found
run_mixer_script.sh: 67: --lib: not found
Traceback (most recent call last):
File "precimed/mixer.py", line 18, in <module>
libbgmg = LibBgmg(args.lib)
File "/data/mixer/precimed/mixer/libbgmg.py", line 28, in __init__
self.cdll, self._lib_name = self._load_cdll(lib_name)
File "/data/mixer/precimed/mixer/libbgmg.py", line 464, in _load_cdll
raise OSError(exception_message)
OSError: libbgmg.so: cannot open shared object file: No such file or directory
Failed to load BGMG shared library from `['libbgmg.so', 'libbgmg.so']`. Try to add the location of `libbgmg.so` file into your PATH system variable, or to set BGMG_SHARED_LIBRARY - the specific system variable which may point to `libbgmg.so` file, including the full path.
run_mixer_script.sh: 70: --trait1-file: not found
run_mixer_script.sh: 71: --trait2-file: not found
run_mixer_script.sh: 72: --trait1-params-file: not found
run_mixer_script.sh: 73: --trait2-params-file: not found
run_mixer_script.sh: 74: --out: not found
run_mixer_script.sh: 75: --bim-file: not found
run_mixer_script.sh: 76: --ld-file: not found
run_mixer_script.sh: 77: --lib: not found
python3: can't open file 'python/mixer.py': [Errno 2] No such file or directory
run_mixer_script.sh: 80: --trait1-file: not found
run_mixer_script.sh: 81: --trait2-file: not found
run_mixer_script.sh: 82: --load-params-file: not found
run_mixer_script.sh: 83: --out: not found
run_mixer_script.sh: 84: --bim-file: not found
run_mixer_script.sh: 85: --ld-file: not found
run_mixer_script.sh: 86: --lib: not found
Not quite sure what is happening here, but libbgmg.so is definitely present. I did all the checks you proposed in an earlier issue and nothing out of the ordinary emerged. Thank you for your help!
First indication that something isn't right is /usr/bin/python3: can't find '__main__' module in 'precimed'. Could you paste the entire script you're executing?
Thank you for the fast reply! This is my script. Basically I just tried to write a shell-script around your tutorial in the most idiotic way possible.
#!/bin/bash
TRAIT_1=$1
TRAIT_2=$2
N_1=$3
N_2=$4
python3 sumstats.py csv --sumstats ${TRAIT_1} --out ${TRAIT_1}1.csv --force --auto --head 5 --n-val ${N_1}
python3 sumstats.py zscore --sumstats ${TRAIT_1}1.csv --out ${TRAIT_1}2.csv --force
python3 sumstats.py qc --sumstats ${TRAIT_1}2.csv --exclude-ranges 6:26000000-34000000 --max-or 1e37 --out ${TRAIT_1}.csv --force
gzip ${TRAIT_1}.csv
python3 sumstats.py csv --sumstats ${TRAIT_2} --out ${TRAIT_2}1.csv --force --auto --head 5 --n-val ${N_2}
python3 sumstats.py zscore --sumstats ${TRAIT_2}1.csv --out ${TRAIT_2}2.csv --force
python3 sumstats.py qc --sumstats ${TRAIT_2}2.csv --exclude-ranges 6:26000000-34000000 --max-or 1e37 --out ${TRAIT_2}.csv --force
gzip ${TRAIT_2}.csv
for i in $(seq 1 20); do
SLURM_ARRAY_TASK_ID=$i
python3 precimed/mixer.py fit1
--trait1-file ${TRAIT_1}.csv.gz
--out ${TRAIT_1}.fit.rep${SLURM_ARRAY_TASK_ID}
--extract 1000G_EUR_Phase3_plink/1000G.EUR.QC.prune_maf0p05_rand2M_r2p8.rep${SLURM_ARRAY_TASK_ID}.snps
--bim-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.bim
--ld-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.run4.ld
--lib src/build/lib/libbgmg.so
python3 precimed/mixer.py test1 \
--trait1-file ${TRAIT_1}.csv.gz \
--load-params-file ${TRAIT_1}.fit.rep${SLURM_ARRAY_TASK_ID}.json \
--out ${TRAIT_1}.test.rep${SLURM_ARRAY_TASK_ID} \
--bim-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.bim \
--ld-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.run4.ld \
--lib src/build/lib/libbgmg.so
python3 precimed /mixer.py fit1
--trait1-file ${TRAIT_2}.csv.gz
--out ${TRAIT_2}.fit.rep${SLURM_ARRAY_TASK_ID}
--extract 1000G_EUR_Phase3_plink/1000G.EUR.QC.prune_maf0p05_rand2M_r2p8.rep${SLURM_ARRAY_TASK_ID}.snps
--bim-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.bim
--ld-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.run4.ld
--lib src/build/lib/libbgmg.so
python3 precimed/mixer.py test1
--trait1-file ${TRAIT_2}.csv.gz
--load-params-file ${TRAIT_2}.fit.rep${SLURM_ARRAY_TASK_ID}.json
--out ${TRAIT_2}.test.rep${SLURM_ARRAY_TASK_ID}
--bim-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.bim
--ld-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.run4.ld
--lib src/build/lib/libbgmg.so
python3 python/mixer.py fit2
--trait1-file ${TRAIT_1}.csv.gz
--trait2-file ${TRAIT_2}.csv.gz
--trait1-params-file ${TRAIT_1}.fit.rep${SLURM_ARRAY_TASK_ID}.json
--trait2-params-file ${TRAIT_2}.fit.rep${SLURM_ARRAY_TASK_ID}.json
--out ${TRAIT_1}vs${TRAIT_2}.fit.rep${SLURM_ARRAY_TASK_ID}
--extract 1000G_EUR_Phase3_plink/1000G.EUR.QC.prune_maf0p05_rand2M_r2p8.rep${SLURM_ARRAY_TASK_ID}.snps
--bim-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.bim
--ld-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.run4.ld
--lib src/build/lib/libbgmg.so
python3 precimed/mixer.py test2
--trait1-file ${TRAIT_1}.csv.gz
--trait2-file ${TRAIT_2}.csv.gz
--trait1-params-file ${TRAIT_1}.fit.rep${SLURM_ARRAY_TASK_ID}.json
--trait2-params-file ${TRAIT_2}.fit.rep${SLURM_ARRAY_TASK_ID}.json
--out ${TRAIT_1}vs${TRAIT_2}.fit.rep${SLURM_ARRAY_TASK_ID}
--bim-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.bim
--ld-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.run4.ld
--lib src/build/lib/libbgmg.so
python3 python/mixer.py test2
--trait1-file ${TRAIT_1}.csv.gz
--trait2-file ${TRAIT_2}.csv.gz
--load-params-file ${TRAIT_1}vs${TRAIT_2}.fit.rep${SLURM_ARRAY_TASK_ID}.json
--out ${TRAIT_1}vs${TRAIT_2}.test.rep${SLURM_ARRAY_TASK_ID}
--bim-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.bim
--ld-file 1000G_EUR_Phase3_plink/1000G.EUR.QC.@.run4.ld
--lib src/build/lib/libbgmg.so
done
python precimed/mixer_figures.py combine --json ${TRAIT_1}vs${TRAIT_2}.fit.rep@.json --out ${TRAIT_1}vs${TRAIT_2}.fit
python precimed/mixer_figures.py combine --json ${TRAIT_1}vs${TRAIT_2}.test.rep@.json --out ${TRAIT_1}vs${TRAIT_2}.test
python precimed/mixer_figures.py two --json-fit ${TRAIT_1}vs${TRAIT_2}.fit.json --json-test ${TRAIT_1}vs${TRAIT_2}.test.json --out ${TRAIT_1}vs${TRAIT_2}_out --statistic mean std
Space after precimed ?
# Fit and test the second trait
python3 precimed /mixer.py fit1
Gosh, you are probably right. Let me try it.
No, unfortunately the problem still occurs. It seems like it has to be something that happens inside the first call to test1.
ok, but then it's likely to be a missing backslash after fit1.
python3 precimed /mixer.py fit1 --trait1-file ${TRAIT_2}.csv.gz
For some reason it was not that as well. Even after fixing the backslash issues (sorry for that), it still fails after it has already started to run test1.
Ok, then please execute one command at a time and let me know the exact one that fails, and what error message it produces
Well, it seems like it runs through if I execute it step by step (which it did not before), so I suppose your solution indeed fixed it and it was truly just my lack of ability to properly code a loop that caused this commotion. Thank you so much for your help!
No worries, good it's solved!