Fetus reference fails to build due to a missing file
volkansevim opened this issue · 0 comments
volkansevim commented
Snakelike fails on rule download_asp because this file on AWS is missing: 'https://md-datasets-cache-zipfiles-prod.s3.eu-west-1.amazonaws.com/mbvhhf8m62-2.zip'
Below is the complete log.
The flag 'directory' used in rule all is only valid for outputs, not inputs.
Building DAG of jobs...
Pulling singularity image docker://satijalab/azimuth-references:vitessce.
Pulling singularity image docker://satijalab/azimuth-references:latest.
Using shell: /usr/bin/bash
Provided cores: 8
Rules claiming more threads will be scaled down.
Job stats:
job count min threads max threads
------------------------------ ------- ------------- -------------
all 1 1 1
download_BBI_individual_organs 14 1 1
download_BBI_metadata 1 1 1
download_BBI_subsampled 1 1 1
download_asp 1 1 1
download_enge 1 1 1
download_lind 1 1 1
export_zarr 1 1 1
make_BBI_query 1 1 1
make_BBI_reference 1 1 1
preprocess_BBI_reference 1 1 1
setup_asp 1 1 1
setup_enge 1 1 1
setup_lind 1 1 1
total 27 1 1
Select jobs to execute...
[Fri Mar 17 18:53:04 2023]
rule download_BBI_individual_organs:
output: data/Stomach_gene_count.RDS
jobid: 18
reason: Missing output files: data/Stomach_gene_count.RDS
wildcards: sample=Stomach
resources: tmpdir=/tmp
Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg
[Fri Mar 17 18:53:04 2023]
rule download_BBI_individual_organs:
output: data/Liver_gene_count.RDS
jobid: 11
reason: Missing output files: data/Liver_gene_count.RDS
wildcards: sample=Liver
resources: tmpdir=/tmp
Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg
[Fri Mar 17 18:53:04 2023]
rule download_BBI_subsampled:
output: data/gene_count_sampled.RDS
jobid: 3
reason: Missing output files: data/gene_count_sampled.RDS
resources: tmpdir=/tmp
Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg
[Fri Mar 17 18:53:04 2023]
rule download_BBI_individual_organs:
output: data/Thymus_gene_count.RDS
jobid: 19
reason: Missing output files: data/Thymus_gene_count.RDS
wildcards: sample=Thymus
resources: tmpdir=/tmp
Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg
[Fri Mar 17 18:53:04 2023]
rule download_BBI_metadata:
output: data/df_cell.RDS, data/df_gene.RDS
jobid: 4
reason: Missing output files: data/df_gene.RDS, data/df_cell.RDS
resources: tmpdir=/tmp
Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg
[Fri Mar 17 18:53:04 2023]
rule download_lind:
output: data/lind_data/counts.tsv
jobid: 23
reason: Missing output files: data/lind_data/counts.tsv
resources: tmpdir=/tmp
Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg
[Fri Mar 17 18:53:04 2023]
rule download_enge:
output: data/enge_data, data/enge_data/enge.tar
jobid: 25
reason: Missing output files: data/enge_data/enge.tar
resources: tmpdir=/tmp
Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg
[Fri Mar 17 18:53:04 2023]
rule download_BBI_individual_organs:
output: data/Heart_gene_count.RDS
jobid: 10
reason: Missing output files: data/Heart_gene_count.RDS
wildcards: sample=Heart
resources: tmpdir=/tmp
Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg
[Fri Mar 17 18:53:06 2023]
Finished job 18.
1 of 27 steps (4%) done
Select jobs to execute...
[Fri Mar 17 18:53:06 2023]
rule download_BBI_individual_organs:
output: data/Lung_gene_count.RDS
jobid: 12
reason: Missing output files: data/Lung_gene_count.RDS
wildcards: sample=Lung
resources: tmpdir=/tmp
Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg
[Fri Mar 17 18:53:08 2023]
Finished job 19.
2 of 27 steps (7%) done
Select jobs to execute...
[Fri Mar 17 18:53:08 2023]
rule download_BBI_individual_organs:
output: data/Pancreas_gene_count.RDS
jobid: 14
reason: Missing output files: data/Pancreas_gene_count.RDS
wildcards: sample=Pancreas
resources: tmpdir=/tmp
Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg
[Fri Mar 17 18:53:11 2023]
Finished job 4.
3 of 27 steps (11%) done
Select jobs to execute...
[Fri Mar 17 18:53:11 2023]
rule download_BBI_individual_organs:
output: data/Cerebrum_gene_count.RDS
jobid: 8
reason: Missing output files: data/Cerebrum_gene_count.RDS
wildcards: sample=Cerebrum
resources: tmpdir=/tmp
Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg
[Fri Mar 17 18:53:21 2023]
Finished job 12.
4 of 27 steps (15%) done
Select jobs to execute...
[Fri Mar 17 18:53:21 2023]
rule download_BBI_individual_organs:
output: data/Adrenal_gene_count.RDS
jobid: 6
reason: Missing output files: data/Adrenal_gene_count.RDS
wildcards: sample=Adrenal
resources: tmpdir=/tmp
Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg
[Fri Mar 17 18:53:30 2023]
Finished job 11.
5 of 27 steps (19%) done
Select jobs to execute...
[Fri Mar 17 18:53:30 2023]
rule download_asp:
output: data/asp_data/counts_filtered.tsv, data/asp_data/md_filtered.tsv
jobid: 21
reason: Missing output files: data/asp_data/md_filtered.tsv, data/asp_data/counts_filtered.tsv
resources: tmpdir=/tmp
Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg
[Fri Mar 17 18:53:31 2023]
Error in rule download_asp:
jobid: 21
output: data/asp_data/counts_filtered.tsv, data/asp_data/md_filtered.tsv
shell:
mkdir -p data/asp_data
cd data/asp_data
wget 'https://md-datasets-cache-zipfiles-prod.s3.eu-west-1.amazonaws.com/mbvhhf8m62-2.zip'
unzip mbvhhf8m62-2.zip
unzip Filtered/'Developmental_heart_filtered_scRNA-seq_and_meta_data.zip'
gunzip share_files/*
mv share_files/all_cells_count_matrix_filtered.tsv counts_filtered.tsv
mv share_files/all_cells_meta_data_filtered.tsv md_filtered.tsv
echo "ASP data downloaded on: $(date)" > ../../logs/download_asp.log
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
[Fri Mar 17 18:53:34 2023]
Finished job 10.
6 of 27 steps (22%) done
[Fri Mar 17 18:53:36 2023]
Finished job 3.
7 of 27 steps (26%) done
[Fri Mar 17 18:53:37 2023]
Finished job 14.
8 of 27 steps (30%) done
[Fri Mar 17 18:53:52 2023]
Finished job 6.
9 of 27 steps (33%) done
[Fri Mar 17 18:55:04 2023]
Finished job 8.
10 of 27 steps (37%) done
[Fri Mar 17 18:55:09 2023]
Finished job 23.
11 of 27 steps (41%) done
[Fri Mar 17 18:55:59 2023]
Finished job 25.
12 of 27 steps (44%) done
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2023-03-17T184850.349820.snakemake.log