phasegenomics/FALCON-Phase

falcon-phase bamfilt: 43456 segmentation fault (core dumped)

jnarayan81 opened this issue · 10 comments

jnarayan81 commented

Falcon-Phase pipeline terminated at falcon-phase bamfilt step with test data-set.

I tried running, it independently
/home/urbe/Tools/FALCON-Phase/bin/falcon-phase bamfilt -f 20 -m 10 -i hic_mapping/test_data.unfiltered.bam -o hic_mapping/test_data.filtered.bam but it return "43456 segmentation fault (core dumped)"

/home/urbe/Tools/FALCON-Phase/bin/falcon-phase bamfilt -f 20 -m 10 -i hic_mapping/test_data.unfiltered.bam -o hic_mapping/test_data.filtered.bam

INFO: RUNNING: /home/urbe/Tools/FALCON-Phase/bin/falcon-phase -f 20 -m 10 -i hic_mapping/test_data.unfiltered.bam -o hic_mapping/test_data.filtered.bam bamfilt 
INFO: detected bam filetype
[1]    43456 segmentation fault (core dumped)  /home/urbe/Tools/FALCON-Phase/bin/falcon-phase bamfilt -f 20 -m 10 -i  -o

Complete logfile:

➜  pipeline git:(master) ✗ ./run.sh 
['000000F_001', '000000F_002', '000000F_003']
['000001F_001', '000001F_002', '000001F_003']
Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 1
Rules claiming more threads will be scaled down.
Job counts:
	count	jobs
	1	aln
	1	bake_mince
	1	bam_filt
	1	binmat
	2	coords
	1	dummy
	1	emit_haplotigs
	2	filter
	1	index_pair
	1	mince_bed
	1	mince_fasta
	2	nucmer
	1	phase
	1	placement
	1	placement_filt
	1	prep_emit
	19
Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807}
Ready jobs (2):
	nucmer
	nucmer
Selected jobs (1):
	nucmer
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

[Tue Aug 21 17:12:11 2018]
Job 17: [info] aligning haplotigs to primary contigs with nucmer




	    /home/urbe/anaconda3/bin/samtools faidx /home/urbe/Tools/FALCON-Phase/test_data/cns_p_ctg.clean.fasta '000000F' > 'test_data.000000F.p.fa'
	    /home/urbe/anaconda3/bin/samtools faidx /home/urbe/Tools/FALCON-Phase/test_data/cns_h_ctg.clean.fasta 000000F_001 000000F_002 000000F_003           > 'test_data.000000F.h.fa'

     	    /home/urbe/Tools/mummer-4.0.0beta2/bin/nucmer --maxmatch -l 100 -c 500 --delta='delta_files/test_data.000000F.delta' -t 2 'test_data.000000F.p.fa' 'test_data.000000F.h.fa'
     
Removing temporary output file test_data.000000F.p.fa.
Removing temporary output file test_data.000000F.h.fa.
[Tue Aug 21 17:12:18 2018]
Finished job 17.
1 of 19 steps (5%) done
Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807}
Ready jobs (2):
	nucmer
	filter
Selected jobs (1):
	nucmer
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

[Tue Aug 21 17:12:18 2018]
Job 18: [info] aligning haplotigs to primary contigs with nucmer




	    /home/urbe/anaconda3/bin/samtools faidx /home/urbe/Tools/FALCON-Phase/test_data/cns_p_ctg.clean.fasta '000001F' > 'test_data.000001F.p.fa'
	    /home/urbe/anaconda3/bin/samtools faidx /home/urbe/Tools/FALCON-Phase/test_data/cns_h_ctg.clean.fasta 000001F_001 000001F_002 000001F_003           > 'test_data.000001F.h.fa'

     	    /home/urbe/Tools/mummer-4.0.0beta2/bin/nucmer --maxmatch -l 100 -c 500 --delta='delta_files/test_data.000001F.delta' -t 2 'test_data.000001F.p.fa' 'test_data.000001F.h.fa'
     
Removing temporary output file test_data.000001F.p.fa.
Removing temporary output file test_data.000001F.h.fa.
[Tue Aug 21 17:12:22 2018]
Finished job 18.
2 of 19 steps (11%) done
Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807}
Ready jobs (2):
	filter
	filter
Selected jobs (1):
	filter
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

[Tue Aug 21 17:12:22 2018]
Job 16: [info] running delta-filter


     	    /home/urbe/Tools/mummer-4.0.0beta2/bin/delta-filter -g delta_files/test_data.000001F.delta > filtered_delta_files/test_data.000001F.delta.filt
     
[Tue Aug 21 17:12:22 2018]
Finished job 16.
3 of 19 steps (16%) done
Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807}
Ready jobs (2):
	filter
	coords
Selected jobs (1):
	coords
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

[Tue Aug 21 17:12:22 2018]
Job 14: [info] running show-coords


     	     /home/urbe/Tools/mummer-4.0.0beta2/bin/show-coords -qTlHL 5000 filtered_delta_files/test_data.000001F.delta.filt > coords_files/test_data.000001F.coords
     
[Tue Aug 21 17:12:22 2018]
Finished job 14.
4 of 19 steps (21%) done
Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807}
Ready jobs (1):
	filter
Selected jobs (1):
	filter
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

[Tue Aug 21 17:12:22 2018]
Job 15: [info] running delta-filter


     	    /home/urbe/Tools/mummer-4.0.0beta2/bin/delta-filter -g delta_files/test_data.000000F.delta > filtered_delta_files/test_data.000000F.delta.filt
     
[Tue Aug 21 17:12:22 2018]
Finished job 15.
5 of 19 steps (26%) done
Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807}
Ready jobs (1):
	coords
Selected jobs (1):
	coords
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

[Tue Aug 21 17:12:22 2018]
Job 13: [info] running show-coords


     	     /home/urbe/Tools/mummer-4.0.0beta2/bin/show-coords -qTlHL 5000 filtered_delta_files/test_data.000000F.delta.filt > coords_files/test_data.000000F.coords
     
[Tue Aug 21 17:12:22 2018]
Finished job 13.
6 of 19 steps (32%) done
Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807}
Ready jobs (1):
	placement
Selected jobs (1):
	placement
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

[Tue Aug 21 17:12:22 2018]
Job 11: [info] making haplotig placement file


     	       cat coords_files/test_data.000000F.coords coords_files/test_data.000001F.coords | sort -k11,11 -k1,1n  > coords_files/test_data.coords.out
     	       /home/urbe/Tools/FALCON-Phase/bin/coords2hp.py coords_files/test_data.coords.out | sort -k6,6 -k8,8n > haplotig_placement_file/test_data.hp.txt
	       
[Tue Aug 21 17:12:23 2018]
Finished job 11.
7 of 19 steps (37%) done
Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807}
Ready jobs (1):
	placement_filt
Selected jobs (1):
	placement_filt
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

[Tue Aug 21 17:12:23 2018]
Job 8: [info] filtering placement file


     		    /home/urbe/Tools/FALCON-Phase/bin/filt_hp.py haplotig_placement_file/test_data.hp.txt > haplotig_placement_file/test_data.filt_hp.txt
     		    
[Tue Aug 21 17:12:23 2018]
Finished job 8.
8 of 19 steps (42%) done
Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807}
Ready jobs (1):
	mince_bed
Selected jobs (1):
	mince_bed
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

[Tue Aug 21 17:12:23 2018]
Job 4: [info] making minced BED files



            awk '{print $1 ":" 0 "-" $4 "	" $6 ":" $8 "-" $9}' haplotig_placement_file/test_data.filt_hp.txt > haplotig_placement_file/test_data.AB_pairs.txt

            cat haplotig_placement_file/test_data.AB_pairs.txt | cut -f1 | perl -lane '@Z = split /-|:/, $_; print join "	", @Z ' | sort -k1,1 -k2,2n > mince/test_data.A_haplotigs.bed
            cat haplotig_placement_file/test_data.AB_pairs.txt | cut -f2 | perl -lane '@Z = split /-|:/, $_; print join "	", @Z ' | sort -k1,1 -k2,2n > mince/test_data.B_haplotigs.bed

            /home/urbe/anaconda3/bin/bedtools merge -i mince/test_data.B_haplotigs.bed > mince/B_haplotigs_merged.bed

            cut -f1,2 /home/urbe/Tools/FALCON-Phase/test_data/cns_p_ctg.clean.fasta.fai | sort -k1,1 > tmp.genome

            # make BED file for collapsed region, exclude haplotigs
            /home/urbe/anaconda3/bin/bedtools complement -i mince/B_haplotigs_merged.bed -g tmp.genome > mince/test_data.collapsed_haplotypes.bed

            rm tmp.genome

     
[Tue Aug 21 17:12:23 2018]
Finished job 4.
9 of 19 steps (47%) done
Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807}
Ready jobs (2):
	prep_emit
	mince_fasta
Selected jobs (1):
	prep_emit
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

[Tue Aug 21 17:12:23 2018]
Job 2: [info] preparing files for haplotig emission


                    cat /home/urbe/Tools/FALCON-Phase/test_data/cns_p_ctg.clean.fasta /home/urbe/Tools/FALCON-Phase/test_data/cns_h_ctg.clean.fasta > test_data.p_h_ctg.fa
                    /home/urbe/anaconda3/bin/samtools faidx test_data.p_h_ctg.fa
                    cat mince/test_data.B_haplotigs.bed mince/test_data.collapsed_haplotypes.bed | sort -k1,1 -k2,2n > mince/test_data.BC.bed
                
[Tue Aug 21 17:12:23 2018]
Finished job 2.
10 of 19 steps (53%) done
Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807}
Ready jobs (1):
	mince_fasta
Selected jobs (1):
	mince_fasta
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

[Tue Aug 21 17:12:23 2018]
Job 9: [info] making individual minced fasta files



            # GENERATE FASTA FILES
            /home/urbe/anaconda3/bin/bedtools getfasta -fi /home/urbe/Tools/FALCON-Phase/test_data/cns_p_ctg.clean.fasta -bed mince/test_data.B_haplotigs.bed > mince/test_data.B_haplotigs.fasta
            /home/urbe/anaconda3/bin/bedtools getfasta -fi /home/urbe/Tools/FALCON-Phase/test_data/cns_p_ctg.clean.fasta -bed mince/test_data.collapsed_haplotypes.bed > mince/test_data.collapsed_haplotypes.fasta
            /home/urbe/anaconda3/bin/bedtools getfasta -fi /home/urbe/Tools/FALCON-Phase/test_data/cns_h_ctg.clean.fasta -bed mince/test_data.A_haplotigs.bed > mince/test_data.A_haplotigs.fasta

     
[Tue Aug 21 17:12:24 2018]
Finished job 9.
11 of 19 steps (58%) done
Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807}
Ready jobs (1):
	bake_mince
Selected jobs (1):
	bake_mince
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

[Tue Aug 21 17:12:24 2018]
Job 5: [info] making minced fasta file


                    cat mince/test_data.A_haplotigs.fasta mince/test_data.B_haplotigs.fasta mince/test_data.collapsed_haplotypes.fasta > mince/test_data.minced.fasta
                  
[Tue Aug 21 17:12:24 2018]
Finished job 5.
12 of 19 steps (63%) done
Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807}
Ready jobs (2):
	aln
	index_pair
Selected jobs (1):
	aln
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

[Tue Aug 21 17:12:24 2018]
Job 12: [info] BWA alignments of HiC read pairs



	    /home/urbe/anaconda3/bin/bwa index mince/test_data.minced.fasta
	    /home/urbe/anaconda3/bin/bwa mem -5 -t 24 mince/test_data.minced.fasta /home/urbe/Tools/FALCON-Phase/test_data/S3HiC_R1.fastq /home/urbe/Tools/FALCON-Phase/test_data/S3HiC_R2.fastq | /home/urbe/anaconda3/bin/samtools view -S -h -b -F 2316 > hic_mapping/test_data.unfiltered.bam

     
[bwa_index] Pack FASTA... 0.54 sec
[bwa_index] Construct BWT for the packed sequence...
[BWTIncCreate] textLength=135356064, availableWord=21523888
[BWTIncConstructFromPacked] 10 iterations done. 35504176 characters processed.
[BWTIncConstructFromPacked] 20 iterations done. 65589904 characters processed.
[BWTIncConstructFromPacked] 30 iterations done. 92326176 characters processed.
[BWTIncConstructFromPacked] 40 iterations done. 116085408 characters processed.
[BWTIncConstructFromPacked] 50 iterations done. 135356064 characters processed.
[bwt_gen] Finished constructing BWT in 50 iterations.
[bwa_index] 34.90 seconds elapse.
[bwa_index] Update BWT... 0.41 sec
[bwa_index] Pack forward-only FASTA... 0.32 sec
[bwa_index] Construct SA from BWT and Occ... 15.76 sec
[main] Version: 0.7.17-r1188
[main] CMD: /home/urbe/anaconda3/bin/bwa index mince/test_data.minced.fasta
[main] Real time: 53.112 sec; CPU: 51.935 sec
[M::bwa_idx_load_from_disk] read 0 ALT contigs
[M::process] read 10000 sequences (800000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 0, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] skip orientation FR as there are not enough pairs
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 10000 reads in 1.290 CPU sec, 0.062 real sec
[main] Version: 0.7.17-r1188
[main] CMD: /home/urbe/anaconda3/bin/bwa mem -5 -t 24 mince/test_data.minced.fasta /home/urbe/Tools/FALCON-Phase/test_data/S3HiC_R1.fastq /home/urbe/Tools/FALCON-Phase/test_data/S3HiC_R2.fastq
[main] Real time: 0.229 sec; CPU: 1.429 sec
[Tue Aug 21 17:13:17 2018]
Finished job 12.
13 of 19 steps (68%) done
Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807}
Ready jobs (2):
	bam_filt
	index_pair
Selected jobs (1):
	index_pair
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

[Tue Aug 21 17:13:17 2018]
Job 6: [info] building haplotig overlap index for falcon phase


     		/home/urbe/anaconda3/bin/samtools faidx mince/test_data.minced.fasta
		/home/urbe/Tools/FALCON-Phase/bin/primary_contig_index.pl haplotig_placement_file/test_data.AB_pairs.txt mince/test_data.minced.fasta.fai > phasing/test_data.ov_index.txt

     
[Tue Aug 21 17:13:18 2018]
Finished job 6.
14 of 19 steps (74%) done
Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807}
Ready jobs (1):
	bam_filt
Selected jobs (1):
	bam_filt
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

[Tue Aug 21 17:13:18 2018]
Job 10: [info] filtering BAM


	/home/urbe/Tools/FALCON-Phase/bin/falcon-phase bamfilt -f 20 -m 10 -i hic_mapping/test_data.unfiltered.bam -o hic_mapping/test_data.filtered.bam
     
INFO: RUNNING: /home/urbe/Tools/FALCON-Phase/bin/falcon-phase -f 20 -m 10 -i hic_mapping/test_data.unfiltered.bam -o hic_mapping/test_data.filtered.bam bamfilt 
INFO: detected bam filetype
Full Traceback (most recent call last):
  File "/home/urbe/anaconda3/lib/python3.6/site-packages/snakemake/executors.py", line 1313, in run_wrapper
    singularity_args, use_singularity, None, jobid, is_shell)
  File "/home/urbe/Tools/FALCON-Phase/pipeline/snakefile", line 218, in __rule_bam_filt
  File "/home/urbe/anaconda3/lib/python3.6/site-packages/snakemake/shell.py", line 133, in __new__
    raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command ' source config.sh; set -eo pipefail ;  
	/home/urbe/Tools/FALCON-Phase/bin/falcon-phase bamfilt -f 20 -m 10 -i hic_mapping/test_data.unfiltered.bam -o hic_mapping/test_data.filtered.bam ' died with <Signals.SIGSEGV: 11>.

    [Tue Aug 21 17:13:18 2018]
    Error in rule bam_filt:
        jobid: 10
        output: hic_mapping/test_data.filtered.bam

Full Traceback (most recent call last):
  File "/home/urbe/anaconda3/lib/python3.6/site-packages/snakemake/executors.py", line 1313, in run_wrapper
    singularity_args, use_singularity, None, jobid, is_shell)
  File "/home/urbe/Tools/FALCON-Phase/pipeline/snakefile", line 218, in __rule_bam_filt
  File "/home/urbe/anaconda3/lib/python3.6/site-packages/snakemake/shell.py", line 133, in __new__
    raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command ' source config.sh; set -eo pipefail ;  
	/home/urbe/Tools/FALCON-Phase/bin/falcon-phase bamfilt -f 20 -m 10 -i hic_mapping/test_data.unfiltered.bam -o hic_mapping/test_data.filtered.bam ' died with <Signals.SIGSEGV: 11>.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/urbe/anaconda3/lib/python3.6/site-packages/snakemake/executors.py", line 346, in _callback
    raise ex
  File "/home/urbe/anaconda3/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/urbe/anaconda3/lib/python3.6/site-packages/snakemake/executors.py", line 1325, in run_wrapper
    show_traceback=True))
snakemake.exceptions.RuleException: CalledProcessError in line 101 of /home/urbe/Tools/FALCON-Phase/pipeline/snakefile:
Command ' source config.sh; set -eo pipefail ;  
	/home/urbe/Tools/FALCON-Phase/bin/falcon-phase bamfilt -f 20 -m 10 -i hic_mapping/test_data.unfiltered.bam -o hic_mapping/test_data.filtered.bam ' died with <Signals.SIGSEGV: 11>.
  File "/home/urbe/Tools/FALCON-Phase/pipeline/snakefile", line 101, in __rule_bam_filt

RuleException:
CalledProcessError in line 101 of /home/urbe/Tools/FALCON-Phase/pipeline/snakefile:
Command ' source config.sh; set -eo pipefail ;  
	/home/urbe/Tools/FALCON-Phase/bin/falcon-phase bamfilt -f 20 -m 10 -i hic_mapping/test_data.unfiltered.bam -o hic_mapping/test_data.filtered.bam ' died with <Signals.SIGSEGV: 11>.
  File "/home/urbe/Tools/FALCON-Phase/pipeline/snakefile", line 101, in __rule_bam_filt
  File "/home/urbe/anaconda3/lib/python3.6/concurrent/futures/thread.py", line 56, in run
Removing output files of failed job bam_filt since they might be corrupted:
hic_mapping/test_data.filtered.bam
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /home/urbe/Tools/FALCON-Phase/pipeline/.snakemake/log/2018-08-21T171211.400331.snakemake.log
unlocking
removing lock
removing lock
removed all locks
zeeev commented

Hi @jnarayan81

Thanks for reporting the issue. I'm looking into it.

zeeev commented

Hi @jnarayan81,

I was not able to reproduce the error. Would you be willing to share your unfiltered bam file with me so I can trouble shoot the issue?

Best,

Zev

jnarayan81 commented

Hi @zeeev

Thanks for your reply.
Please find the gunzipped BAM file (I gunzip it to convert into recommended upload format).

test_data.unfiltered.bam.gz

zeeev commented

Hi @jnarayan81,

It looks like the unfiltered bam is truncated.

~/tools/samtools/samtools view test_data.unfiltered.bam
[W::sam_read1] Parse error at line 1
[main_samview] truncated file.

This suggests there was a problem during read alignment.

  1. Do the reads exist?

  2. Is your bam empty?

  3. I'm not seeing an error in your snakemake but what happens if you run this command?

/home/urbe/anaconda3/bin/bwa mem -5 -t 24 mince/test_data.minced.fasta /home/urbe/Tools/FALCON-Phase/test_data/S3HiC_R1.fastq /home/urbe/Tools/FALCON-Phase/test_data/S3HiC_R2.fastq | /home/urbe/anaconda3/bin/samtools view -S -h -b -F 2316 > hic_mapping/test_data.unfiltered.bam

jnarayan81 commented

Hi @zeeev

Strange, gunzipping BAM file corrupted it. I have attached the BAM file with .txt extension.

Do the reads exist? Yes

Is your bam empty? No

I'm not seeing an error in your snakemake but what happens if you run this command?
It mapped the reads.

➜  pipeline git:(master) ✗ /home/urbe/anaconda3/bin/bwa mem -5 -t 24 mince/test_data.minced.fasta /home/urbe/Tools/FALCON-Phase/test_data/S3HiC_R1.fastq /home/urbe/Tools/FALCON-Phase/test_data/S3HiC_R2.fastq | /home/urbe/anaconda3/bin/samtools view -S -h -b -F 2316 > hic_mapping/test_data.unfiltered.bam
[M::bwa_idx_load_from_disk] read 0 ALT contigs
[M::process] read 10000 sequences (800000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 0, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] skip orientation FR as there are not enough pairs
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 10000 reads in 1.464 CPU sec, 0.067 real sec
[main] Version: 0.7.17-r1188
[main] CMD: /home/urbe/anaconda3/bin/bwa mem -5 -t 24 mince/test_data.minced.fasta /home/urbe/Tools/FALCON-Phase/test_data/S3HiC_R1.fastq /home/urbe/Tools/FALCON-Phase/test_data/S3HiC_R2.fastq
[main] Real time: 1.091 sec; CPU: 1.632 sec

➜  hic_mapping git:(master) ✗ samtools flagstat test_data.unfiltered.bam
1274 + 0 in total (QC-passed reads + QC-failed reads)
0 + 0 secondary
0 + 0 supplementary
0 + 0 duplicates
1274 + 0 mapped (100.00% : N/A)
1274 + 0 paired in sequencing
637 + 0 read1
637 + 0 read2
0 + 0 properly paired (0.00% : N/A)
1274 + 0 with itself and mate mapped
0 + 0 singletons (0.00% : N/A)
140 + 0 with mate mapped to a different chr
0 + 0 with mate mapped to a different chr (mapQ>=5)

test_data.unfiltered.bam.txt

jnarayan81 commented

Is this due to this line in the log file ?

INFO: RUNNING: /home/urbe/Tools/FALCON-Phase/bin/falcon-phase -f 20 -m 10 -i hic_mapping/test_data.unfiltered.bam -o hic_mapping/test_data.filtered.bam bamfilt

It seems to be wrong command !

zeeev commented

Hi @jnarayan81,

Sorry I haven't been able to work on this bug recently. So your last suggestion is not the case, as illogical as that sounds. The binary code isn't printing the command correctly. The pipeline is running the correct command.

Are you running off development or master? If you're on master I'd suggest trying development.

I'll look at the bam you provided soon.

jnarayan81 commented

Hi @zeeev
I am testing on master. Just for your kind update, I have snakemake v5.2.2 installed in my computer !!
conda install snakemake installed the latest version.

I install snakemake v3.6 and rerun it, but now it return following error:

➜  pipeline git:(master) ✗ ./run.sh
['000001F_001', '000001F_002', '000001F_003']
['000000F_001', '000000F_002', '000000F_003']
Provided cores: 1
Rules claiming more threads will be scaled down.
Job counts:
	count	jobs
	1	aln
	1	bake_mince
	1	bam_filt
	1	binmat
	2	coords
	1	dummy
	1	emit_haplotigs
	2	filter
	1	index_pair
	1	mince_bed
	1	mince_fasta
	2	nucmer
	1	phase
	1	placement
	1	placement_filt
	1	prep_emit
	19
Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807}
Ready jobs (2):
	nucmer
	nucmer
Selected jobs (1):
	nucmer
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}
[info] aligning haplotigs to primary contigs with nucmer



	    /home/urbe/anaconda3/bin/samtools faidx /home/urbe/Tools/FALCON-Phase/test_data/cns_p_ctg.clean.fasta '000001F' > 'test_data.000001F.p.fa'
	    /home/urbe/anaconda3/bin/samtools faidx /home/urbe/Tools/FALCON-Phase/test_data/cns_h_ctg.clean.fasta ['000001F_001', '000001F_002', '000001F_003']           > 'test_data.000001F.h.fa'

     	    /home/urbe/Tools/mummer-4.0.0beta2/bin/nucmer --maxmatch -l 100 -c 500 --delta='delta_files/test_data.000001F.delta' -t 2 'test_data.000001F.p.fa' 'test_data.000001F.h.fa'
     
zsh:5: bad pattern: [000001F_001,
Full Traceback (most recent call last):
  File "/home/urbe/anaconda3/lib/python3.5/site-packages/snakemake/executors.py", line 758, in run_wrapper
    version)
  File "/home/urbe/Tools/FALCON-Phase/pipeline/snakefile", line 491, in __rule_nucmer
  File "/home/urbe/anaconda3/lib/python3.5/site-packages/snakemake/shell.py", line 74, in __new__
    raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '


	    /home/urbe/anaconda3/bin/samtools faidx /home/urbe/Tools/FALCON-Phase/test_data/cns_p_ctg.clean.fasta '000001F' > 'test_data.000001F.p.fa'
	    /home/urbe/anaconda3/bin/samtools faidx /home/urbe/Tools/FALCON-Phase/test_data/cns_h_ctg.clean.fasta ['000001F_001', '000001F_002', '000001F_003']           > 'test_data.000001F.h.fa'

     	    /home/urbe/Tools/mummer-4.0.0beta2/bin/nucmer --maxmatch -l 100 -c 500 --delta='delta_files/test_data.000001F.delta' -t 2 'test_data.000001F.p.fa' 'test_data.000001F.h.fa'
     ' returned non-zero exit status 1

Error in job nucmer while creating output files delta_files/test_data.000001F.delta, test_data.000001F.p.fa, test_data.000001F.h.fa.
Full Traceback (most recent call last):
  File "/home/urbe/anaconda3/lib/python3.5/site-packages/snakemake/executors.py", line 758, in run_wrapper
    version)
  File "/home/urbe/Tools/FALCON-Phase/pipeline/snakefile", line 491, in __rule_nucmer
  File "/home/urbe/anaconda3/lib/python3.5/site-packages/snakemake/shell.py", line 74, in __new__
    raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '


	    /home/urbe/anaconda3/bin/samtools faidx /home/urbe/Tools/FALCON-Phase/test_data/cns_p_ctg.clean.fasta '000001F' > 'test_data.000001F.p.fa'
	    /home/urbe/anaconda3/bin/samtools faidx /home/urbe/Tools/FALCON-Phase/test_data/cns_h_ctg.clean.fasta ['000001F_001', '000001F_002', '000001F_003']           > 'test_data.000001F.h.fa'

     	    /home/urbe/Tools/mummer-4.0.0beta2/bin/nucmer --maxmatch -l 100 -c 500 --delta='delta_files/test_data.000001F.delta' -t 2 'test_data.000001F.p.fa' 'test_data.000001F.h.fa'
     ' returned non-zero exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/urbe/anaconda3/lib/python3.5/site-packages/snakemake/executors.py", line 246, in _callback
    raise ex
  File "/home/urbe/anaconda3/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/urbe/anaconda3/lib/python3.5/site-packages/snakemake/executors.py", line 772, in run_wrapper
    show_traceback=True))
snakemake.exceptions.RuleException: CalledProcessError in line 200 of /home/urbe/Tools/FALCON-Phase/pipeline/snakefile:
Command '


	    /home/urbe/anaconda3/bin/samtools faidx /home/urbe/Tools/FALCON-Phase/test_data/cns_p_ctg.clean.fasta '000001F' > 'test_data.000001F.p.fa'
	    /home/urbe/anaconda3/bin/samtools faidx /home/urbe/Tools/FALCON-Phase/test_data/cns_h_ctg.clean.fasta ['000001F_001', '000001F_002', '000001F_003']           > 'test_data.000001F.h.fa'

     	    /home/urbe/Tools/mummer-4.0.0beta2/bin/nucmer --maxmatch -l 100 -c 500 --delta='delta_files/test_data.000001F.delta' -t 2 'test_data.000001F.p.fa' 'test_data.000001F.h.fa'
     ' returned non-zero exit status 1
  File "/home/urbe/Tools/FALCON-Phase/pipeline/snakefile", line 200, in __rule_nucmer

RuleException:
CalledProcessError in line 200 of /home/urbe/Tools/FALCON-Phase/pipeline/snakefile:
Command '


	    /home/urbe/anaconda3/bin/samtools faidx /home/urbe/Tools/FALCON-Phase/test_data/cns_p_ctg.clean.fasta '000001F' > 'test_data.000001F.p.fa'
	    /home/urbe/anaconda3/bin/samtools faidx /home/urbe/Tools/FALCON-Phase/test_data/cns_h_ctg.clean.fasta ['000001F_001', '000001F_002', '000001F_003']           > 'test_data.000001F.h.fa'

     	    /home/urbe/Tools/mummer-4.0.0beta2/bin/nucmer --maxmatch -l 100 -c 500 --delta='delta_files/test_data.000001F.delta' -t 2 'test_data.000001F.p.fa' 'test_data.000001F.h.fa'
     ' returned non-zero exit status 1
  File "/home/urbe/Tools/FALCON-Phase/pipeline/snakefile", line 200, in __rule_nucmer
  File "/home/urbe/anaconda3/lib/python3.5/concurrent/futures/thread.py", line 55, in run
Removing output files of failed job nucmer since they might be corrupted:
test_data.000001F.p.fa
Releasing 1 _cores (now 1).
Releasing 1 _nodes (now 9223372036854775807).
Will exit after finishing currently running jobs.
Exiting because a job execution failed. Look above for error message
unlocking
removing lock
removing lock
removed all locks

I also tried this, but the same error:

➜  pipeline git:(master) ✗ /home/urbe/anaconda3/bin/samtools faidx /home/urbe/Tools/FALCON-Phase/test_data/cns_h_ctg.clean.fasta ['000001F_001', '000001F_002', '000001F_003']           > 'test_data.000001F.h.fa'

zsh: bad pattern: [000001F_001,

error (zsh:5: bad pattern:) appear only when, I downgrade snakemake to v3.6.0, resolved when snakemake is upgraded to v5.2.2

It seems, Snakemake v3.6 is not compatible with Python3.6
Conda download Python

➜  adineta_p_lvl6 conda install snakemake=3.6.1
Solving environment: done

## Package Plan ##

  environment location: /home/urbe/anaconda3

  added / updated specs: 
    - snakemake=3.6.1


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    pywavelets-1.0.0           |   py35h7eb728f_0         6.1 MB  conda-forge
    pyyaml-3.13                |   py35h470a237_1         442 KB  conda-forge
    jupyterlab-0.34.7          |           py35_0        10.0 MB  conda-forge
    widgetsnbextension-3.4.1   |           py35_0         1.7 MB  conda-forge
    ------------------------------------------------------------
                                           Total:        18.2 MB

The following NEW packages will be INSTALLED:

    snakemake:                          3.6.1-py35_0            bioconda   

The following packages will be UPDATED:
.....
.....

The following packages will be DOWNGRADED:

    aiohttp:                            3.4.2-py36h470a237_0    conda-forge --> 3.1.3-py35_0            conda-forge
    anaconda:                           custom-py36hbbc8b67_0               --> custom-py35h104c396_0              
    async-timeout:                      3.0.0-py36_0            conda-forge --> 2.0.1-py35_0            conda-forge
    bz2file:                            0.98-py36_1                         --> 0.98-py35_0                        
    **python:                             3.6.6-h5001a0f_0        conda-forge --> 3.5.5-h5001a0f_2        conda-forge**
jnarayan81 commented

Hi @zeeev
It works on my another cluster, but still wondering why error on my local workstation ? Anyway, thanks for your favor.

I have one more question ! Which long-reads corrector (heterozygous aware) do you prefer/recommend to use?

zeeev commented

Hi @jnarayan81,

Sorry we weren't able figure out the issue on your local computer. Also, just for your information, PacBio is now supporting FALCON-Phase. Support and enhancements will be in their bioconda repository.

I'm not sure I understand your question, maybe @skingan has an Idea?