Handle model hash mismatches better
tazlin opened this issue · 6 comments
A wide audience of users have reported, at various stages of development, the models being detected as having the incorrect hash.
Is this issue specific enough to ever be closed? I suspect it will sit here forever. There are a large number of causes unrelated to each other.
For example, cyber realistic got stuck in an "invalid hash" error and a download loop a few days ago. On investigation the civitai file had been changed without them actually version bumping or any visible signs this had happened.
A few days before similar error. On investigation I had put an incorrect hash in the model database.
Point taken. I will narrow the scope of this issue to having the hordelib model manager handle this issue more gracefully, ideally by not blocking the worker from running.
Having just watched my worker fail 5 times to download 5.5GB of dreamshaper from Civitai and each time download 5GB, then delete that 5Gb and start again, I'm giving this a quick patch to not delete the damn partial downloaded data, my web browser doesn't do that. It's silly that I'm better off downloading the model manually through the web browser than letting the worker downloader do this.
I suggest you skip the model, and avoid attempting to download or load it that run. If the SHA doesn't match, the worker should not be running it.
Agreed the worker should not be running the model if it fails hash validation and the implemented solution honours that whilst implementing a more robust download solution with resuming and retrying downloads and not deleting the users models which they may well be using in software other than the horde worker (unless we can immediately replace it with another working version).
Agreed. The merged solution looks good.