hasindu2008/slow5tools

Bad fast5: Attribute Raw/tracked_scaling_scale

nickst85 opened this issue · 9 comments

nickst85 commented

Hi I got the following error messages:
Bad fast5: Attribute Raw/tracked_scaling_scale in /groups/chengrp/dRNAseq/157_-DEX_dRNAseq/157_-DEX/20231031_0244_MC-110944_FAQ79114_c4653c1d/fast5/FAQ79114_c4653c1d_d8ce327a_77.fast5 is unexpected. Please report this with an example FAST5 file at 'https://github.com/hasindu2008/slow5tools/issues' for us to investigate.

Bad fast5: Could not iterate over the read groups in the fast5 file /groups/chengrp/dRNAseq/157_-DEX_dRNAseq/157_-DEX/20231031_0244_MC-110944_FAQ79114_c4653c1d/fast5/FAQ79114_c4653c1d_d8ce327a_77.fast5.

Bad fast5: Could not read contents of the fast5 file '/groups/chengrp/dRNAseq/157_-DEX_dRNAseq/157_-DEX/20231031_0244_MC-110944_FAQ79114_c4653c1d/fast5/FAQ79114_c4653c1d_d8ce327a_77.fast5

The same errors had other fast5 files. Let me know how I can share the fast5 file as it's too large to be attached.

Thanks in advance

hasindu2008 commented

Could you please confirm if you are using the latest version of slow5tools?
I remember that we handled this attribute in a recent version.

nickst85 commented
hasindu2008 commented

0.6 is a very old version. Could you use 1.1?

nickst85 commented
rugilemat commented

Hi,

I seem to be having the same issue (slow5tools 0.6.0) - I know it's an old version but the only way I seem to have been able to successfully install slow5tools on our HPC is through conda (conda install bioconda::slow5tools) as I don't have the permissions to use sudo for the HPC. So would it be possible to update the conda version?

Also I don't know if this is at all relevant but I am trying to run it on the fast5 generated from the RNA flow cells for RNA004 library preps.

This is the code I ran: slow5tools f2s ${fast5} -d ${output}

This is how my error is looking:

[fast5_attribute_itr::ERROR] Bad fast5: Attribute Raw/tracked_scaling_scale in /scratch/prj/ppn_ont_rna/directrna/full_exp/007_CTF_MGL_D1/20240117_2045_1B_PAU73469_e828644c/fast5_all/PAU73469_pass_e828644c_c4f9b990_356.fast5 is unexpected. Please report this with an example FAST5 file at 'https://github.com/hasindu2008/slow5tools/issues' for us to investigate.
[read_fast5::ERROR] Bad fast5: Could not iterate over the read groups in the fast5 file /scratch/prj/ppn_ont_rna/directrna/full_exp/007_CTF_MGL_D1/20240117_2045_1B_PAU73469_e828644c/fast5_all/PAU73469_pass_e828644c_c4f9b990_356.fast5.
[f2s_child_worker::ERROR] Bad fast5: Could not read contents of the fast5 file '/scratch/prj/ppn_ont_rna/directrna/full_exp/007_CTF_MGL_D1/20240117_2045_1B_PAU73469_e828644c/fast5_all/PAU73469_pass_e828644c_c4f9b990_356.fast5'.
[f2s_iop] Child process 2829159 exited with status=1.
hasindu2008 commented

Slowtools 1.1.0 is available on conda https://anaconda.org/bioconda/slow5tools. Could you please try that?

rugilemat commented

Thanks for the quick response! It seems when I try to install slow5tools through conda without specifying the version, it ends up being slow5tools 0.6.0.

I tried conda install bioconda::slow5tools=1.1.0 but it seems to be struggling with the package requirements:

Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: / 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                                                                                                                        

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package libgcc-ng conflicts for:
hdf5[version='>=1.12.2,<1.12.3.0a0'] -> libgcc-ng[version='>=12']
hdf5[version='>=1.12.2,<1.12.3.0a0'] -> libaec[version='>=1.0.6,<2.0a0'] -> libgcc-ng[version='>=10.3.0|>=11.2.0|>=9.4.0|>=13.2.0|>=12.3.0|>=11.4.0|>=7.5.0']
libzlib[version='>=1.2.13,<1.3.0a0'] -> libgcc-ng[version='>=12']
zlib -> libgcc-ng[version='>=10.3.0|>=12|>=7.5.0|>=7.3.0|>=4.9|>=11.2.0|>=7.2.0']
libgcc-ng[version='>=12']

Package libzlib conflicts for:
hdf5[version='>=1.12.2,<1.12.3.0a0'] -> libcurl[version='>=7.83.1,<9.0a0'] -> libzlib[version='>=1.2.11,<1.3.0a0']
libzlib[version='>=1.2.13,<1.3.0a0']
hdf5[version='>=1.12.2,<1.12.3.0a0'] -> libzlib[version='>=1.2.12,<1.3.0a0|>=1.2.13,<1.3.0a0']
zlib -> libzlib[version='1.2.11|1.2.11|1.2.11|1.2.12|1.2.12|1.2.12|1.2.12|1.2.12|1.2.13|1.2.13|1.3',build='h36c2ea0_1012|h166bdaf_2|h166bdaf_3|h166bdaf_4|h166bdaf_4|hd590300_5|hd590300_0|h166bdaf_1|h166bdaf_0|h166bdaf_1014|h36c2ea0_1013']

Package libstdcxx-ng conflicts for:
hdf5[version='>=1.12.2,<1.12.3.0a0'] -> libstdcxx-ng[version='>=12']
hdf5[version='>=1.12.2,<1.12.3.0a0'] -> libaec[version='>=1.0.6,<2.0a0'] -> libstdcxx-ng[version='>=10.3.0|>=9.4.0']
libstdcxx-ng[version='>=12']

Package zlib conflicts for:
hdf5[version='>=1.12.2,<1.12.3.0a0'] -> libcurl[version='>=7.87.0,<9.0a0'] -> zlib[version='>=1.2.12,<1.3.0a0|>=1.2.13,<1.3.0a0']
zlibThe following specifications were found to be incompatible with your system:

  - feature:/linux-64::__glibc==2.31=0
  - feature:|@/linux-64::__glibc==2.31=0
  - hdf5[version='>=1.12.2,<1.12.3.0a0'] -> libgfortran-ng -> __glibc[version='>=2.17']
  - zlib -> libgcc-ng[version='>=10.3.0'] -> __glibc[version='>=2.17']

Your installed version is: 2.31

Note that strict channel priority may have removed packages required for satisfiability.
hiruna72 commented

Hello @rugilemat

Thanks for reporting this.
Until we fix the conda issue can you download the executable to your HPC home directory as follows,

cd ${HOME}
wget https://github.com/hasindu2008/slow5tools/releases/download/v1.1.0/slow5tools-v1.1.0-x86_64-linux-binaries.tar.gz
tar -xvf slow5tools-v1.1.0-x86_64-linux-binaries.tar.gz 
./slow5tools-v1.1.0/slow5tools --version

Now you can set a bash variable to the path of slow5tools executable and use it in your HPC script.

#!/bin/bash

SLOW5TOOLS="${HOME}/slow5tools-v1.1.0/slow5tools"
${SLOW5TOOLS} --version
rugilemat commented

Thanks so much for such a quick response! I am trying it out now and will report back if the issue I've seen above persists.

Edit: it looks like it has worked successfully now. I had to install vbz locally too:

[main] real time = 0.000 sec | CPU time = 0.006 sec | peak RAM = 0.002 GB
slow5tools 1.1.0
[list_all_items] Looking for '*.fast5' files in /scratch/prj/ppn_ont_rna/directrna/full_exp/007_CTF_MGL_D1/20240117_2045_1B_PAU73469_e828644c/fast5_all
[f2s_main] 1237 fast5 files found - took 0.028s
[f2s_main] Just before forking, peak RAM = 0.000 GB
[f2s_iop] 8 proceses will be used.
[f2s_iop] Spawning 8 I/O processes to circumvent HDF hell.
[f2s_child_worker::INFO] Summary - total fast5: 155, bad fast5: 0
[f2s_child_worker::INFO] Summary - total fast5: 155, bad fast5: 0
[f2s_child_worker::INFO] Summary - total fast5: 155, bad fast5: 0
[f2s_child_worker::INFO] Summary - total fast5: 155, bad fast5: 0
[f2s_child_worker::INFO] Summary - total fast5: 152, bad fast5: 0
[f2s_child_worker::INFO] Summary - total fast5: 155, bad fast5: 0
[f2s_child_worker::INFO] Summary - total fast5: 155, bad fast5: 0
[f2s_child_worker::INFO] Summary - total fast5: 155, bad fast5: 0
[f2s_main] Converting 1237 fast5 files took 2002.569s
[f2s_main] Children processes: CPU time = 11280.989 sec | peak RAM = 0.308 GB

[main] cmd: /scratch/users/k19022845/scripts/slow5tools-v1.1.0/slow5tools f2s /scratch/prj/ppn_ont_rna/directrna/full_exp/007_CTF_MGL_D1/20240117_2045_1B_PAU73469_e828644c/fast5_all -d /scratch/prj/ppn_microglia_mod/directrna/full_exp/007_CTF_MGL_D1/blow5
[main] real time = 2002.600 sec | CPU time = 0.014 sec | peak RAM = 0.005 GB
  • 🎉1