Indexing error: piscem index failed to build succesfully ExitStatus(unix_wait_status(132))
lichtobergo opened this issue · 17 comments
Hi,
I was trying to build the spliceu index for the ensembl rat genome.
I used conda to install simpleaf
mamba create -n af -y -c bioconda -c conda-forge simpleaf piscem
Then I downloaded unzipped the rat genome from Ensembl
wget https://ftp.ensembl.org/pub/release-111/fasta/rattus_norvegicus/cdna/Rattus_norvegicus.mRatBN7.2.cdna.all.fa.gz
wget https://ftp.ensembl.org/pub/release-111/gtf/rattus_norvegicus/Rattus_norvegicus.mRatBN7.2.111.gtf.gz
gzip -d Rattus_norvegicus.mRatBN7.2.cdna.all.fa.gz
gzip -d Rattus_norvegicus.mRatBN7.2.111.gtf.gz
Then I ran simpleaf index with these commands
export ALEVIN_FRY_HOME='.'
simpleaf set-paths
ulimit -n 2048
simpleaf index \
--use-piscem \
--output mRatBN7.2.ensembl.v111 \
--fasta Rattus_norvegicus.mRatBN7.2.dna.toplevel.fa \
--gtf Rattus_norvegicus.mRatBN7.2.111.gtf \
--threads 4 \
--ref-type spliced+unspliced
And this is the error I get
2024-03-08T09:11:44.558059Z INFO simpleaf::simpleaf_commands::indexing: preparing to make reference with roers
2024-03-08T09:11:53.538051Z INFO grangers::reader::gtf: Finished parsing the input file. Found 5 comments and 1284445 records.
2024-03-08T09:11:55.266274Z INFO roers: Built the Grangers object for 1284445 records
2024-03-08T09:11:56.059573Z WARN roers: Found missing gene_id and/or gene_name; Imputing. If both missing, will impute using transcript_id; Otherwise, will impute using the existing one.
2024-03-08T09:11:56.090425Z INFO roers: Proceed 526642 exon records from 54993 transcripts
2024-03-08T09:12:14.554211Z INFO roers: Proceed 30562 gene body records
2024-03-08T09:13:31.040646Z INFO roers: Done!
2024-03-08T09:13:31.067558Z INFO simpleaf::simpleaf_commands::indexing: piscem build cmd : /home/haberl/mambaforge/envs/af/bin/piscem build -k 31 -m 19 -o mRatBN7.2.ensembl.v111_AAV/index/piscem_idx -s mRatBN7.2.ensembl.v111_AAV/ref/roers_ref.fa --seed 1 --threads 4
2024-03-08T10:12:52.008871Z ERROR simpleaf::utils::prog_utils: command unsuccessful (signal: 4 (SIGILL) (core dumped)): "/home/haberl/mambaforge/envs/af/bin/piscem" "build" "-k" "31" "-m" "19" "-o" "mRatBN7.2.ensembl.v111_AAV/index/piscem_idx" "-s" "mRatBN7.2.ensembl.v111_AAV/ref/roers_ref.fa" "--seed" "1" "--threads" "4"
2024-03-08T10:12:52.008910Z ERROR simpleaf::utils::prog_utils: stdout :
====
Constructing the compacted reference de Bruijn graph for k = 31.
Enumerating the vertices of the de Bruijn graph.
31-mer enumeration statistics:
Number of sequences: 85555.
Total sequence length: 1228696587.
Total number of 31-mers: 1226129937.
Number of unique 31-mers: 972718483.
Number of solid 31-mers: 972718483.
Enumerated the vertex set of the graph. Time taken = 55.9562 seconds.
Number of vertices: 972718483.
Constructing the minimal perfect hash function (MPHF) over the vertex set.
Total number of k-mers in the set (KMC database): 972718483.
Building the MPHF from the k-mer database ./workdir.noindex/piscem_idx_cfish.cf_V.
Using gamma = 10.
Built the MPHF in memory.
Total MPHF size: 1441 MB. Bits per k-mer: 12.4332.
Allocated hash table buckets for the k-mers. Total size: 579 MB.
Total memory usage by the hash table: 2021 MB. Bits per k-mer: 17.4332.
Constructed the minimal perfect hash function for the vertices. Time taken = 85.2591 seconds.
Computing the DFA states.
Opened reference 1 from mRatBN7.2.ensembl.v111_AAV/ref/roers_ref.fa
Maximum input sequence buffer size used: 4 MB.
Done computing the vertex-states. Time taken = 1337.49 seconds.
Computed the states of the automata. Time taken = 1337.49 seconds.
Extracting the maximal unitigs.
Temporary path file name prefixes: ./workdir.noindex//cuttlefish-path-output-4360066512
Opened reference 1 from mRatBN7.2.ensembl.v111_AAV/ref/roers_ref.fa
["ref_index_builder", "-i", "mRatBN7.2.ensembl.v111_AAV/index/piscem_idx_cfish", "-k", "31", "-m", "19", "--canonical-parsing", "--build-ec-table", "-o", "mRatBN7.2.ensembl.v111_AAV/index/piscem_idx", "-d", "./workdir.noindex", "-t", "4", "--seed", "1"]
====
2024-03-08T10:12:52.016735Z ERROR simpleaf::utils::prog_utils: stderr :
====
2024-03-08T09:13:31.141014Z INFO piscem: starting piscem build
2024-03-08T09:13:31.141061Z INFO piscem: Computing and recording reference signatures...
trimmed polyA tails from 0 records
2024-03-08T09:13:47.407562Z INFO piscem: done.
2024-03-08T09:13:47.407659Z INFO piscem: args = ["cdbg_builder", "--seq", "mRatBN7.2.ensembl.v111_AAV/ref/roers_ref.fa", "-k", "31", "--track-short-seqs", "--poly-N-stretch", "-o", "mRatBN7.2.ensembl.v111_AAV/index/piscem_idx_cfish", "-t", "4", "-f", "3", "-w", "./workdir.noindex"]
****************************************************************************************
Stage 1: 100%
Warning: using counter_max == 1 will cause not storying counters in KMC output file, all counters will be assumed to be 1. This is experimental and is not currently supported in kmc_tools. Will be implemented soon.
Stage 2: 100%
[Building BooPHF] 47.4 % elapsed: 0 min 39 sec remaining: 0 min 43 sec
Completed a pass over the k-mer database.
[Building BooPHF] 94.9 % elapsed: 1 min 21 sec remaining: 0 min 4 sec
Completed a pass over the k-mer database.
[Building BooPHF] 100 % elapsed: 1 min 24 sec remaining: 0 min 0 sec
Closed reference mRatBN7.2.ensembl.v111_AAV/ref/roers_ref.fa.
Processed 85555 sequences. Total reference length: 1231807171 bases.
Closed reference mRatBN7.2.ensembl.v111_AAV/ref/roers_ref.fa.
====
Error: piscem index failed to build succesfully ExitStatus(unix_wait_status(132))
Could not find anything similar in older issues and would appreciate any hint to where I did a mistake.
Thanks in advance!
Hi @lichtobergo,
Indeed, I think we have to do more work to try to propagate up extra information from the underlying program, error messages like this are hard to debug. However, exit code 132 is generally an attempt to execute an illegal instruction. This suggests that, perhaps, the compiled executable contains an instruction not supported on the hardware where the program is being run? Can you share a little bit of information about the environment (CPU, OS version, etc.) where you are running this?
Thanks!
Rob
Thank you, Rob, for your very rapid response. I should’ve included my hardware specs in the original post. Sorry for that. I think this could have to do with it, as I used a quite old Xeon Exxx with 4 cores 4 threads vom around 2015. It runs Ubuntu 22.04 LTS with a 6.2 kernel.
I could try to make the index on my laptop which has an Intel i7-12700H and is running Pop OS 22.04 LTS with a 6.6.10 kernel. Would that help?
Yes. If it works on the other machine, it is evidence of use of an illegal instruction. In that case, maybe we could track it down and determine if it can be / is worth stripping from the conda binary. Alternativel, we could build a binary on the side that doesnt use that instruction (but may be slower). I have some thoughts as to which instructions they might be, if it's indeed the case.
Thanks!
Rob
Dear Rob,
I repeated the exact same commands on my laptop and it finished successfully with the following output
2024-03-11T09:18:28.704504Z INFO simpleaf::simpleaf_commands::indexing: preparing to make reference with roers
2024-03-11T09:18:31.511848Z INFO grangers::reader::gtf: Finished parsing the input file. Found 5 comments and 1284445 records.
2024-03-11T09:18:32.178402Z INFO roers: Built the Grangers object for 1284445 records
2024-03-11T09:18:32.381811Z WARN roers: Found missing gene_id and/or gene_name; Imputing. If both missing, will impute using transcript_id; Otherwise, will impute using the existing one.
2024-03-11T09:18:32.392738Z INFO roers: Proceed 526642 exon records from 54993 transcripts
2024-03-11T09:18:36.098701Z INFO roers: Proceed 30562 gene body records
2024-03-11T09:18:45.641713Z INFO roers: Done!
2024-03-11T09:18:45.661389Z INFO simpleaf::simpleaf_commands::indexing: piscem build cmd : /home/light/mambaforge/envs/simpleaf/bin/piscem build -k 31 -m 19 -o mRatBN7.2.ensembl.v111/index/piscem_idx -s mRatBN7.2.ensembl.v111/ref/roers_ref.fa --seed 1 --threads 10
2024-03-11T09:25:13.484207Z INFO simpleaf::utils::prog_utils: command returned successfully (exit status: 0)
For me that should work. Thank you again! Regarding me this issue can be closed. I don't know if it is worth the effort to track this issue for a soon to be unsupported CPU/architecture.
P.S. Do have a guess which CPU architecture is the required minimum to use simpleaf?
Kind regards,
Michael
Hi Michael,
Thanks for reporting back! It's hard to say without testing and checking exactly what the offending instruction is, but my guess is that it's probably the bit manipulation instructions, which intel introduced first in the haswell arch, and amd fully supported in the excavator arch. I think we actually have an exposed build flag for that, but turning it off would def hurt performance. Let us know if you have any further questions!
Best,
Rob
Hi @rob-p ,
I'm posting this here because I encountered the same error (ExitStatus(unix_wait_status(132)) that might be related to a previous issue. I got this error when trying to build an index using the latest human reference from 10x (https://cf.10xgenomics.com/supp/cell-exp/refdata-gex-GRCh38-2024-A.tar.gz). The command that I'm running looks like this:
ulimit -n 2048
export ALEVIN_FRY_HOME="/home/mkoder/data/alevin_fry_home"
simpleaf set-paths
cd $ALEVIN_FRY_HOME
DATA_DIR="/home/mkoder/data"
REF_DIR="/home/mkoder/reference_genomes/refdata-gex-GRCh38-2024-A"
IDX_DIR="$DATA_DIR/alevin_fry_home/human_2024-A"
simpleaf index \
--output $IDX_DIR \
--fasta $REF_DIR/fasta/genome.fa \
--gtf $REF_DIR/genes/genes.gtf \
--use-piscem \
--rlen 91 \
--threads 6
And this is the exact error message:
And this is the exact error message:Allocated hash table buckets for the k-mers. Total size: 925 MB.
Total memory usage by the hash table: 2814 MB. Bits per k-mer: 15.1983.
Constructed the minimal perfect hash function for the vertices. Time taken = 168.097 seconds.
Computing the DFA states.
Opened reference 1 from /home/mkoder/data/alevin_fry_home/human_2024-A/ref/roers_ref.fa
Maximum input sequence buffer size used: 4 MB.
Done computing the vertex-states. Time taken = 361.109 seconds.
Computed the states of the automata. Time taken = 361.109 seconds.
Extracting the maximal unitigs.
Temporary path file name prefixes: .//cuttlefish-path-output-3615462126
Opened reference 1 from /home/mkoder/data/alevin_fry_home/human_2024-A/ref/roers_ref.fa
["ref_index_builder", "-i", "/home/mkoder/data/alevin_fry_home/human_2024-A/index/piscem_idx_cfish", "-k", "31", "-m", "19", "--canonical-parsing", "--build-ec-table", "-o", "/home/mkoder/data/alevin_fry_home/human_2024-A/index/piscem_idx", "-d", ".", "-t", "6"]
Processed 295650 sequences. Total reference length: 2261910749 bases.
Maximum input sequence buffer size used: 4 MB.
Done writing the compacted graph (in GFA-reduced format). Time taken = 681.216 seconds.
Extracted the maximal unitigs. Time taken = 681.216 seconds.
Maximum temporary disk-usage: 8.9395GB.
Structural information for the de Bruijn graph is written to /home/mkoder/data/alevin_fry_home/human_2024-A/index/piscem_idx_cfish.json.
Constructed the reference compacted de Bruijn graph at /home/mkoder/data/alevin_fry_home/human_2024-A/index/piscem_idx_cfish.
====
2024-07-30T19:52:43.283610Z ERROR simpleaf::utils::prog_utils: stderr :
====
2024-07-30T19:31:08.009639Z INFO piscem: starting piscem build
2024-07-30T19:31:08.009704Z INFO piscem: will use . as the work directory for temporary files.
2024-07-30T19:31:08.009715Z INFO piscem: args = ["cdbg_builder", "--seq", "/home/mkoder/data/alevin_fry_home/human_2024-A/ref/roers_ref.fa", "-k", "31", "--track-short-seqs", "--poly-N-stretch", "-o", "/home/mkoder/data/alevin_fry_home/human_2024-A/index/piscem_idx_cfish", "-t", "6", "-f", "3", "-w", "."]
**********************************************************************************************************************************
Stage 1: 100%
Warning: using counter_max == 1 will cause not storying counters in KMC output file, all counters will be assumed to be 1. This is experimental and is not currently supported in kmc_tools. Will be implemented soon.
Stage 2: 100%
[Building BooPHF] 46.7 % elapsed: 1 min 17 sec remaining: 1 min 28 sec
Completed a pass over the k-mer database.
[Building BooPHF] 93.7 % elapsed: 2 min 36 sec remaining: 0 min 11 sec
Completed a pass over the k-mer database.
[Building BooPHF] 100 % elapsed: 2 min 45 sec remaining: 0 min 0 sec
Closed reference /home/mkoder/data/alevin_fry_home/human_2024-A/ref/roers_ref.fa.
Processed 295650 sequences. Total reference length: 2261910749 bases.
Closed reference /home/mkoder/data/alevin_fry_home/human_2024-A/ref/roers_ref.fa.
directory already exists
====
Error: piscem index failed to build succesfully ExitStatus(unix_wait_status(132))
I'm running Ubuntu 22.04.3 LTS. With the same conda environment, the code finished running successfully on a different system ( Red Hat Enterprise Linux release 8.6 (Ootpa)). It also finishes successfully if I skip the --use-piscem flag.
I would really appreciate your help with this issue. Ideally, I would like to solve this on the Ubuntu system that I'm currently using. If there is any additional information I can provide, please let me know.
Best,
Maruša
Hi @marusakod,
Thanks for posting. Are you using the latest version of piscem? And what version of simpleaf? For a while now, piscem has been the default (i.e. use-piscem has been redundant). I see the above has a message about the directory existing. Can you try writing to a place that doesn't have an index yet (or using the --overwrite flag)?
--Rob
Thanks for the quick response! For the previous run I was using piscem v. 0.6.3 and simpleaf v. 0.15.1. I now tried again with piscem v. 0.10.3 and simpleaf v. 0.17.2 and with the --overwrite flag and I get the exact same error.
Maruša
Thanks, @marusakod!
One more quick check before digging deeper, can you please run
simpleaf refresh-prog-info
And try once more, and then show the result of
simpleaf inspect
If it’s still not working. In that case maybe I can try in a VM matching your specific OS.
Best,
Rob
Thanks @rob-p ,
I tried again without success. This is the output of simpleaf_inspect:
{
"builtin_chemistries": [
"10xv2",
"10xv2-5p",
"10xv3",
"10xv3-5p",
"10xv4-3p",
"custom()"
],
"custom_chem_info": "",
"simpleaf_info": {
"prog_info": {
"alevin_fry": {
"exe_path": "/home/mkoder/miniconda3/envs/simpleaf/bin/alevin-fry",
"version": "0.10.0"
},
"piscem": {
"exe_path": "/home/mkoder/miniconda3/envs/simpleaf/bin/piscem",
"version": "0.10.3"
},
"salmon": {
"exe_path": "/home/mkoder/miniconda3/envs/simpleaf/bin/salmon",
"version": "1.10.3"
}
}
},
"simpleaf_version": "0.17.2"
}
I don't know if this helps, but I tried also on a slightly updated version of Ubuntu (22.04.4) and I'm getting the same error message as the one posted above, except this time it ends with ExitStatus(unix_wait_status(4))
Best,
Maruša
Hi @marusakod,
Thanks; this is useful. I was not yet able to reproduce this natively on our server (using the same dependencies installed from bioconda) --- there the index built successfully. Can you tell me a little bit more about the hardware you're on (e.g. what's the output of cat /proc/cpuinfo)? The next thing I'll probably try is a VM using your exact OS version to see if I can obtain this behavior.
The perplexing thing to me in your output is the "directory already exists" message. I'm not exactly sure where that is coming from or why it shows up if you try to write the index in a completely new place.
--Rob
Small update:
I put together a docker image with Ubuntu 22.04.4, and installed simpleaf via conda. Unfortunately (feels strange to say that in this context), the construction succeeded there as well. I suspect there is either something specific to the hardware on which you're running (in which case the cat /proc/cpuinfo might help suggest differences), or about the filesystem itself. Out of curiosity, before you attempt to build the index, what is the layout of your $DATA_DIR/alevin_fry_home/?
--Rob
Hi @rob-p ,
before I attempt to build the index, my $DATA_DIR/alevin_fry_home/ is empty. I tried again on a different machine with Ubuntu 22.04.4 and it worked. I attached the outputs of cat /proc/cpuinfo for both systems that I used. The directory already exists message is indeed very strange and I'm not sure which directory this is referring to. I tried using different directories for $ALEVIN_FRY_HOME and for building the index and it's always the same.
not_working_sys_info.txt
working_sys_info.txt
Best,
Maruša
Hi @marusakod,
This is very useful. Ok; I have a theory. The non-working system has an AMD Opteron 6376. This is a 12 year old processor. I suspect that the compiler flags used in bioconda when building probably emit instructions that cannot be used on this, rather old, CPU. Overall, we (the devs) should really figure out a policy for "processor" support to determine when a processor is too old to have the necessary capabilities for a tool.
That said, if you really need to run on this machine, the thing to try would be to build a version with the minimal possible instruction set / intrinsics. I can give that a try if you want (build it in my docker image) and strip out all unnecessary flags.
--Rob
Hi @marusakod,
Ok, here is a binary for piscem (the "offending" program) that is explicitly compiled with all of the fancy instructions stripped away and specifically for the opteron architecture. However, it's likely to only work on ubuntu 22.04. Also, because it's not allowed to use BMI2 (bit manipulation instructions), it may be markedly slower than the executable you get from bioconda. That said, hopefully you can use this if you need to process some samples on this machine. Please let me know if you give this a try.
Best,
Rob