ZipperBams failed. Error: processed all unmapped reads but there are mapped reads remaining to be read.

[teryyoung]

hello! I'm doing with sequencing data with UMI, I followed #fgbio-best-practise-fastq---consensus-pipeline
When I run fgbio ZipperBams to merge my consensus.ubam and consensus.mapped.bam, there is an exception:
Exception in thread "main" java.lang.IllegalStateException: Error: processed all unmapped reads but there are mapped reads remaining to be read.

and I check my two bam files; they all are queryname sorted, have same number of reads, the difference between them is all of queryname of ubam have suffix "/A". like this:


I found it is caused by fgbio CallConsensusReads...

I wanna know whether this is the problem caused ZipperBams failed, or how to deal with this problem(ZipperBams)?
#my fgbio version is 2.2.2-3a74fd2-SNAPSHOT

[nh13]

Somehow the /A is getting lost between the uBAM and the mapped BAM. I see your filename has picard in it, but we don't use picard in the best-practices pipeline. Can you provide the commands you used to go from the uBAM to the mapped BAM?

[teryyoung]

I used STAR to map the uBAM, so the code looks weird,
1)First I FastqtoBam the raw reads, and picard sorted it;
java -Xmx10g -jar /share/apps/softwares/fgbio/target/scala-2.13/fgbio-2.2.2-3a74fd2-SNAPSHOT.jar FastqToBam \
--input $r1 $r2 \
--read-structures 5M2S+T 5M2S+T \
--sample ${sample} \
--library ${sample} \
--platform-unit BGIT7 \
--output ./0.1.uBAM/${sample}.uBAM
java -Xmx10G -jar /share/apps/softwares/picard-2.26.2/picard.jar SortSam I=0.1.uBAM/${sample}.uBAM \
O=0.1.uBAM/${sample}.picard.sorted.uBAM\
SORT_ORDER=queryname

2)then I extract fastq from uBAM, mapped them using STAR, forgive me that I didn't find ways to mapping one fastq file using STAR.
samtools fastq 0.1.uBAM/${sample}.picard.sorted.uBAM -1 tmp/${sample}.1.fq.bak -2 tmp/${sample}.2.fq.bak

awk 'NR%4==1 {print $0"/1"} NR%4 !=1 {print $0}' tmp/${sample}.1.fq.bak >tmp/${sample}.uBAM_1.fq
awk 'NR%4==1 {print $0"/2"} NR%4 !=1 {print $0}' tmp/${sample}.2.fq.bak >tmp/${sample}.uBAM_2.fq

ref_dir="~/reference_genome/GRC38_p14/hg38_ucsc/STAR_index/"
rm tmp/${sample}/ -rf
/share/apps/softwares/STAR-2.7.10b/bin/Linux_x86_64_static/STAR --twopassMode Basic\
--quantMode TranscriptomeSAM GeneCounts\
--runThreadN 6\
--genomeDir ${ref_dir}\
--alignIntronMin 20\
--alignIntronMax 50000\
--outSAMtype BAM Unsorted\
--outSAMattrRGline ID:"${sample}" SM:"${sample}" PL:BGI\
--outFilterMismatchNmax 2\
--outSJfilterReads Unique --outSAMmultNmax 1\
--outFileNamePrefix ./0.1.uBAM/${sample}/${sample}.\
--outSAMmapqUnique 60\
--outSAMunmapped Within KeepPairs\
--readFilesCommand cat\
--readFilesIn tmp/${sample}.uBAM_1.fq tmp/${sample}.uBAM_2.fq\
--outTmpDir ./tmp/${sample}
3)After first mapping, I sorted result bam file using picard, and ZipperBams;
java -Xmx10G -jar /share/apps/softwares/picard-2.26.2/picard.jar SortSam I=0.1.uBAM/${sample}/${sample}.Aligned.out.bam\
O=0.1.uBAM/${sample}/${sample}.mapped.picard.sorted.BAM\
SORT_ORDER=queryname
java -jar /share/apps/softwares/fgbio/target/scala-2.13/fgbio-2.2.2-3a74fd2-SNAPSHOT.jar ZipperBams \
--unmapped 0.1.uBAM/${sample}.picard.sorted.uBAM \
--input 0.1.uBAM/${sample}/${sample}.mapped.picard.sorted.BAM \
--ref ${ref}/../hg38.p14.fa \
--output 0.1.uBAM/${sample}.umi.mapped.bam

All commands above runs fine, but After I GroupReadsByUmi, CallMolecularConsensusReads, FilterConsensusReads, and mapping them sencond time, ZipperBams have problem;
4)java -jar /share/apps/softwares/fgbio/target/scala-2.13/fgbio-2.2.2-3a74fd2-SNAPSHOT.jar GroupReadsByUmi \
--input=${dir01}/1.umi.mapped.bam \
--output=${dir02}/1.umi.group.BAM \
--strategy=paired \
--min-map-q=10 \
--edits=1 \
--raw-tag=RX \ --family-size-histogram=${dir02}/1.umi.tag_family_histogram.txt`

java -jar /share/apps/softwares/fgbio/target/scala-2.13/fgbio-2.2.2-3a74fd2-SNAPSHOT.jar CallMolecularConsensusReads \
--min-reads=1 \
--min-input-base-quality=20 \
--input=${dir02}/1.umi.group.BAM \
--output=${dir03}/1.consensus.uBAM \
--threads=4 \
--read-group-id=false

java -jar /share/apps/softwares/fgbio/target/scala-2.13/fgbio-2.2.2-3a74fd2-SNAPSHOT.jar FilterConsensusReads \
--input=${dir03}/1.consensus.uBAM \
--output=${dir03}/1.consensus.filter.uBAM \
--ref="${ref}/../hg38.p14.fa" --min-reads=3 \
--max-read-error-rate=0.2 \
--max-base-error-rate=0.2 \
--min-base-quality=30 \
--max-no-call-fraction=0.20

java -Xmx10G -jar /share/apps/softwares/picard-2.26.2/picard.jar SortSam \
I=${dir03}/1.consensus.filter.uBAM\
O=${dir03}/1.consensus.filter.sorted.uBAM\
SORT_ORDER=queryname

samtools fastq ./${dir03}/1.consensus.filter.sorted.uBAM -1 tmp/1.consensus.uBAM_1.fq.bak -2 tmp/1.consensus.uBAM_2.fq.bak
awk 'NR%4 ==1 {print $0"/1"} NR%4 !=1 {print$0}' tmp/1.consensus.uBAM_1.fq.bak >tmp/1.consensus.uBAM_1.fq
awk 'NR%4==1 {print $0"/2"} NR%4!=1 {print$0}' tmp/1.consensus.uBAM_2.fq.bak >tmp/1.consensus.uBAM_2.fq

STAR alignment code is like it above, and then picard sorted;
then ZipperBams;
java -Xmx10G -jar /share/apps/softwares/fgbio/target/scala-2.13/fgbio-2.2.2-3a74fd2-SNAPSHOT.jar ZipperBams \
--unmapped ./${dir03}/1.consensus.filter.sorted.uBAM \
--ref ${ref}/../hg38.p14.fa \
--input ${dir03}/1.consensus.mapped.picard.sorted.bam \
--output ${dir03}/1.consensus.mapped.zipper.bam

Tips: using picard sortBam is because I used to use picard mergeBam to merge unmapped and mapped reads; I know the sort_order need to be the same when using picard mergeBam; (why I didnt use samtools sort is that some BAM files didnt support samtools's sort somehow.)

[teryyoung]

Did you mean that '/A' is necessary for ZipperBams? And that should be retain whatever I run command on it?
now I think my fault is misuse the STAR alignment.