I am encountering an ANE evaluation error when running Dorado on macOS.

Question

I am encountering an ANE evaluation error when running Dorado on macOS.

Opened this issue 2 months ago · 12 comments

GanchimegNamuunbayar commented 2 months ago

Issue Report

Please describe the issue:

Please provide a clear and concise description of the issue you are seeing and the result you expect.

Steps to reproduce the issue:

Please list any steps to reproduce the issue.

Run environment:

Dorado version: 0.8.0
Dorado command: dorado basecaller --batchsize 16 sup,4mC_5mC,6mA ~/desktop/AA/pod5/ -r > ~/desktop/AA/bam/calls.bam
Operating system: MacOS Sonoma 14.0
Hardware (CPUs, Memory, GPUs): M2 Max, Memory 64GB
Source data type (e.g., pod5 or fast5 - please note we always recommend converting to pod5 for optimal basecalling performance): multiple pod 5 files
Source data location (on device or networked drive - NFS, etc.): on device
Details about data (flow cell, kit, read lengths, number of reads, total dataset size in MB/GB/TB): totaling 14GB (each 800MB to 4.5GB)
Dataset to reproduce, if applicable (small subset of data to share as a pod5 to reproduce the issue):

Logs

Please provide output trace of dorado (run dorado with -v, or -vv on a small subset)
I was able to run successfully with a single pod5 file of 850MB.
command: dorado basecaller --batchsize 32 sup, 4mC_5mC, 6mA ~/desktop/pod5/ > ~/desktop/calls.bam

Answer 1 · 2024-10-17T14:20:24.000Z

Hi @GanchimegNamuunbayar,
Can you add logs please?

Answer 2 · 2024-10-21T02:28:12.000Z

@HalfPhoton Thank you for your response. I have added logs to the issue report. Could you please reference the updated version.

Answer 3 · 2024-10-21T08:10:35.000Z

You've only added the version information here - The -vv flags must be placed after the sub-command: dorado basecaller -vv to generate detailed logs.

Kind regards,
Rich

Answer 4 · 2024-10-23T01:48:38.000Z

@HalfPhoton Thank you for the details.
I have tried it as you said
command: dorado basecaller -vv --batchsize 16 sup,4mC_5mC,6mA ~/desktop/pod5/ -r > ~/desktop/bam/calls.bam

These are the last lines before the computer shut down:
[2024-10-22 17:42:51.715] [trace] Checking adapter/primer PCR_PSK_rev1
[2024-10-22 17:42:51.715] [trace] Checking adapter/primer PCR_PSK_rev2
[2024-10-22 17:42:51.715] [trace] Checking adapter/primer cDNA_VNP
[2024-10-22 17:42:51.715] [trace] Checking adapter/primer cDNA_SSP
[2024-10-22 17:42:51.715] [trace] Checking adapter/primer PCS110_forward
[2024-10-22 17:42:51.715] [trace] Checking adapter/primer PCS110_reverse
[2024-10-22 17:42:51.715] [trace] Checking adapter/primer RAD

Answer 5 · 2024-10-24T08:56:38.000Z

Could you please write the logs to a file and share them? These last few lines aren't enough for us to see the issue.

dorado basecaller -vv --batchsize 16 sup,4mC_5mC,6mA ~/desktop/pod5/ -r > ~/desktop/bam/calls.bam 2> dorado.trace

Kind regards,
Rich

Answer 6 · 2024-10-24T08:57:16.000Z

Does you issue persist with dorado 0.8.2? It has a number of bug fixes which might be causing your crash.

Answer 7 · 2024-10-29T01:56:33.000Z

@HalfPhoton Thank you for your previous advice. I attempted to install a newer version, but the issue remains unresolved. The command for the logs executes successfully on smaller or fewer files; however, when processing larger datasets (~13GB), the system crashes and restarts due to a panic. I've included the trace file in a Google Drive link for your review, as I couldn't attach it here directly.
Logs

Answer 8 · 2024-11-04T10:04:18.000Z

Hi @GanchimegNamuunbayar,

For some reason I can't view these logs, can you create a ticket on the ONT customer support page please?
We can then get these large log files and help you from there.

Please reference this GitHub issue and the great people at customer support will be able to get this information to me.

Kind regards,
Rich

Answer 9 · 2024-11-04T16:33:11.000Z

@HalfPhoton I've fixed the link for @GanchimegNamuunbayar, logs should be accessible now.

Answer 10 · 2024-11-12T02:23:40.000Z

Hi @HalfPhoton ,
I hope you're doing well! It's been a while since I shared the logs, and I just wanted to check in and see if there have been any updates or if you’ve had a chance to review them. Looking forward to your reply.
Thank you again for your support!

Also thanks to @malton-ont for kindly fixing the link earlier—really appreciate the help!!

Best,
Ganchimeg

Answer 11 · 2024-11-12T09:52:31.000Z

Hi @GanchimegNamuunbayar,

dorado-0.8.3 was released yesterday and contains a couple of fixes for SUP basecalling on Apple silicon - could you give this a try?

Answer 12 · 2024-11-19T01:39:10.000Z

@malton-ont
I tried it, but the error persists. Are there others facing the same problem? Could it be related to something in my environment or command?