`Uncaught (in promise) TypeError: Failed to fetch` is thrown in Chrome when the disk is nearly full

[peaceps]

Hi there,

I see this error again and if it happens it seems to be permanent. Do you have any ideas about this.

Here are the logs I added when the issue orrured (calling ZipWriter.add()):

VM219020:7458 debugggg sendMessage type ack message id 44                      // send ack to worker
VM219020:7510 debugggg onMessage type pull and message id 45                // receive pull from worker
VM219020:7513 debugggg reader.read 524288 false                                  // read data, not done
VM219020:7458 debugggg sendMessage type data message id 45                // send data to worker#1
VM219020:7463 debugggg sendMessage value sent 524288                         // send data to worker#2
VM219020:7478 debugggg sendMessage postMessage 1 45                         // send data to worker#3 after postMessage
/#/page:1 Uncaught (in promise) TypeError: Failed to fetch                       // Error occurred
VM219020:7510 debugggg onMessage type data and message id 46             // receive data from worker
VM219020:7521 debugggg writer.write catch 524912 undefined            // write data but exception thrown by watchClosedStream (sometimes this error catched before the Failed to Fetch error)
VM219020:7532 debugggg error close error undefined                          // close without sending ack to worker, worker is blocked
VM219020:7546 debugggg close onTaskFinished undefined                    // finally add function never return

In successful case, write can succeed and we can see below logs:

VM219020:debugggg sendMessage type ack message id 20
VM219020:debugggg onMessage type data and message id 21
VM219020:7392 debuggggg watchClosedStream write                                // write data succeed from watchClosedStream 
VM219020:7399 debuggggg watchClosedStream writen
VM219020:7524 debugggg writer.write 3971
VM219020:7458 debugggg sendMessage type ack message id 21
VM219020:7510 debugggg onMessage type close and message id undefined
VM219020:7546 debugggg close onTaskFinished null
VM219020:7252 debugggg CodecWorker onTaskFinished 

The browser is chrome v99, zipjs version is 2.7.17, OS version: Debain 10
This issue can be observed on chrome v115, too. I also noticed that the chunk size of writer changed to 16384, much smaller than it in v99

[gildas-lormeau]

Could you try to enable the debugging of "Uncaught exceptions" in the Dev tools? Maybe it would help to identify the cause of the error.

[gildas-lormeau]

BTW, isn't there a stack trace below the error message Uncaught (in promise) TypeError: Failed to fetch and/or in the stack property of the message sent from the worker storing the error details? That would be very useful in order to find the cause.

[peaceps]

Hi,
This error doesn't have any stack. This is how it looks in the console:

If I add breakpoint for uncaught expections we can see the stream became errored:

In this case uint8arrayreader and blobwrite are used.
I tried to replace blobwriter to uint8arraywriter and convert the final data to blob at last, then this error disappears. But the speed of zipping seems becomes slower.
So I guess this error is thrown by the Response class of BlobWriter, but not sure if any error happened in the stream chain.

[gildas-lormeau]

When the breakpoint for uncaught exceptions is hit, what line of code do you see in the "Sources" tab in the dev tools or what is the line of code at line 33778 in your minified script? What is the call stack at that moment (cf "call stack" panel in the devl tools)?

[gildas-lormeau]

BTW, you probably have to add some logs in z-worker-core.js, and build the library in dev mode with npm run build-dev. It's also possible that Chrome cannot hit uncaught exceptions within the web worker. That's why you might need to add some logs in the web worker code, or try to debug the issue in Firefox.

[peaceps]

When the breakpoint for uncaught exceptions is hit, what line of code do you see in the "Sources" tab in the dev tools or what is the line of code at line 33778 in your minified script? What is the call stack at that moment (cf "call stack" panel in the devl tools)?

It's in runWebWorker in index.cjs or lib/core/codec-worker.js

try {
    await writable.getWriter().close();
} catch (_error) {
    // ignored
}

[gildas-lormeau]

This exception is caught and is not the cause of the TypeError: Failed to fetch error. You must pause the execution on uncaught exceptions only.

[gildas-lormeau]

Note that sometimes the debugger does not want to pause the execution on uncaught exception. In that case, you have to pause on all exceptions and identify the last exception just before the TypeError.

I wished I could reproduce this issue though. That would make things much simpler for me.

[peaceps]

I also add onerror callback to the worker, but not called.

It seems to be a little easier to reproduce on computer with a full hardware disk

[gildas-lormeau]

Thank you for the feedback. As I said, this particuler exception is not the cause of the TypeError. You can clearly see that there is a try/catch around the statement. So, this particular exception is caught, it is not an uncaught exception. If this exception was the cause of the error, then that would mean the JS engine is buggy, which is highly unlikely.

Please continue to execute the code until you find the last exception before the TypeError is thrown. I highly suspect this exception to be thrown in the web worker code, not the main JS code.

[peaceps]

Thank you for the feedback. As I said, this particuler exception is not the cause of the TypeError. You can clearly see that there is a try/catch around the statement. So, this particular exception is caught, it is not an uncaught exception. If this exception was the cause of the error, then that would mean the JS engine is buggy, which is highly unlikely.

Please continue to execute the code until you find the last exception before the TypeError is thrown.

Yes I know it's not the cause of TypeError it's just a hint that some writestream got error

[gildas-lormeau]

This exception is normal, that's why it's ignored if it is thrown.

[gildas-lormeau]

Out of curiosity, does the TypeError disappear if you comment this line of code?

[gildas-lormeau]

BTW, as you can see in your screenshots, the stream is already ERRORED (see the error message) when this statement is executed, so it's too late. You have to find why the stream is already ERRORED. The original error that made the stream ERRORED and would not have been caught is (very) probably thrown in the web worker.

I've looked very closely at the web worker code multiple times. I've also added some throw new Error("test") in the code but I can't see where I would have let an unexpected exception be thrown without being handled as it should.

[peaceps]

I add logs in worker. Each time when the error 'Failed to fetch' occurred, the process stopped after the main thread received "data" message from worker and exception caught in watchClosedStream -> await writer.ready
logs starts with 'blob:...' is from worker
logs starts with 'zipjs.js' is from zipjs lib
the log 'Failed to fetch' doesn't start with above prefix, I'm not sure if it's thrown by chrome itself

blob:http://localhost:9000/5febdac7-1aaa-4e45-9673-bbeb725d7cd9:1879 zipjsss worker sendmessage messageid: 689 type: data with length: 16384
zipjs.js:1657 zipjsssssss mainthread onMessage with messageId: 689 and type:data
zipjs.js:1657 zipjsssssss mainthread sendMessage with messageId: 689 and type:ack
blob:http://localhost:9000/5febdac7-1aaa-4e45-9673-bbeb725d7cd9:1787 zipjssssssssssss worker onMessage messageId: 689 type: ack
blob:http://localhost:9000/5febdac7-1aaa-4e45-9673-bbeb725d7cd9:1879 zipjsss worker sendmessage messageid: 690 type: data with length: 16384
zipjs.js:1657 zipjsssssss mainthread onMessage with messageId: 690 and type:data
zipjs.js:1657 zipjsssssss mainthread sendMessage with messageId: 690 and type:ack
:9000/#/:1 Uncaught (in promise) TypeError: Failed to fetch
blob:http://localhost:9000/5febdac7-1aaa-4e45-9673-bbeb725d7cd9:1787 zipjssssssssssss worker onMessage messageId: 690 type: ack
blob:http://localhost:9000/5febdac7-1aaa-4e45-9673-bbeb725d7cd9:1879 zipjsss worker sendmessage messageid: 691 type: data with length: 1725
zipjs.js:1657 zipjsssssss mainthread onMessage with messageId: 691 and type:data
zipjs.js:1656 zipjsssssssss ready write error undefined
zipjs.js:1657 zipjsssssssss sendMessage write error undefined

[gildas-lormeau]

These logs don't help me much. They seem to indicate that the error may occur in the middle of processing and it does not prevent zip.js to work. These logs are also rather obscure as I don't know where they are located, and what is being logged. For example, what does the ready write error undefined log and the one that follows correspond to?

For my part, I tested the demos on a 10-year-old computer with a hard disk. I was able to generate GB of zipped data without any problems.

[peaceps]

Other logs are from onMessage and sendMessage of mainthread and worker

[gildas-lormeau]

Thank you. So that's an error, which is strangely undefined, in the main script. It's really strange because the uncaught error is not supposed to cause this error. I don't really understand what's happening and what could be the direct cause of this undefined error.

You told me that the bug is easier to reproduce on a machine with an hard disk. Are there any other factors that could have some influence on this bug? For example, the number of files in the zip file, the size of the files, their compressibility, the configuration of zip.js, etc.

In fact, I have almost no information about what you do exactly.

[peaceps]

I also can't reproduce it on my machine, it's occasional case, and it happens on different files each time.
If the error appears it most probably appears at the beginning (3rd or 4th file). If it doesn't occur at the begining, the zipping will succeed at last(normally 20 to 30 files).
The file size is average 5-7MB and the final zip will be about 150M. Most files are already zip file and some are text file
We just use the default configs.
I'm also considering chrome bug for this issue

[gildas-lormeau]

Thank you for the info, can you tell me if you can reproduce the bug with this page? https://plnkr.co/edit/ZdAuGz1gBfE6Xf2F?preview

The test page contains the HTML content below.

<!doctype html>
<html>
<head>
  <title>Stress test</title>
  <style>
label {
  display: block;
}
label span {
  display: inline-block;
  min-width: 180px;
}
input[type=number] {
  width: 50px;
}
button,
#output {
  margin-top: 10px;
}
#output {
  font-family: monospace;
}
  </style>
</head>
<body>
  <form>
    <label>
      <span>
        Number of entries:
      </span>
      <input type=number name=entriesCount value=50 min=1>
    </label>
    <label>
      <span>
        Minimum entry size (MB):
      </span>
      <input type=number name=minSize value=5 min=0>
    </label>
    <label>
      <span>
        Maximum entry size (MB):
      </span>
      <input type=number name=maxSize value=7 min=1>
    </label>
    <button type=submit>
      Start test
    </button>
  </form>
  <div id=output></div>
  <script type=module>
import * as zip from "https://unpkg.com/@zip.js/zip.js/index.js";  

const form = document.forms[0];
const { minSize, maxSize, entriesCount } = form;

minSize.oninput = () => {
  if (minSize.valueAsNumber > maxSize.valueAsNumber) {
    maxSize.valueAsNumber = minSize.valueAsNumber;
  }
};
maxSize.oninput = () => {
  if (maxSize.valueAsNumber < minSize.valueAsNumber) {
    minSize.valueAsNumber = maxSize.valueAsNumber;
  }
};
form.onsubmit = event => {
  event.preventDefault();
  output.innerHTML = "Started...";
  test();
};

async function test(indexTest = 0) {
  const blobWriter = new zip.BlobWriter("application/zip");
  const zipWriter = new zip.ZipWriter(blobWriter);
  await Promise.all(new Array(entriesCount.valueAsNumber)
    .fill(getBlob())
    .map((blob, indexEntry) => 
      zipWriter.add("file_" + indexEntry, new zip.BlobReader(blob))
    ));
  await zipWriter.close();
  output.innerHTML = "Test #" + indexTest + ": OK<br>" + output.innerHTML;
  await test(indexTest + 1);
}

function getBlob() {
  const size = minSize.valueAsNumber + Math.random() * (maxSize.valueAsNumber - minSize.valueAsNumber);
  const data = new Float64Array(Math.floor((size * 1024 * 1024) / 8));
  for (let indexData = 0; indexData < data.length; indexData++) {
    data[indexData] = Math.random();
  }
  return new Blob([data]);
}
  </script>
</body>
</html>

It's a "stress test" which tries to reproduce your bug in an infinite loop. Feel free to tweak it if necessary.

[peaceps]

I can reproduce it in this page, with 3.8GB hard disk space available

I also observed a new error message

[peaceps]

After I deleted some files and make hard disk space to 20GB, this error disappears

Finaly this error will still occur when space is low enough since the test page doesn't release the Blobs.

So it seems to be a chrome issue with low hard disk space

[gildas-lormeau]

Thank you for the feedback. You're probably right. However, there is no memory leak with Blobs AFAIK in this test. They should be garbage collected.

I'll try to do some tests in a VM because I would like to know if the exception can be caught or not.

[gildas-lormeau]

I created a VM with Win 10 installed and a fixed disk size of 25GB (~4GB of free space). I confirm that I can easily reproduce the bug in this environment. For the record, it's not related to web workers at all (that was one of my suspicion). It's also interesting to see that as soon as I try to debug the code with breakpoints, the bug totally disappears...

In the end, I doubt the uncaught error is coming from the library because it should at least contain a stack trace with web workers disabled. It should also not be undefined when caught, I guess.

[peaceps]

Thanks for your kind support! I'll close this issue.