[Bug] Database corruption & other
Closed this issue · 2 comments
Last update was nice. Though for a while.
DB error was still there, it occurred randomly too.
So i have few suggestions.
-
I normally run with "-u -p -c" flag. But -c flag always creates epub irrespective of whether there was a pending chapter downloaded or not. So even if there was no update, it would download. I suggest make 2 flag "-c" & "-cf" for create if there is any update & force create irrespective if any update respectively.
-
Idk how to solve db error. Idk what you keeping there. Haven't checked. I can only guess that you are indexing all those chapter files in DB. Chapters are already in json format. We can always look into existing file in chapter folder to know what's downloaded whats not. Also while creating epub, we can use chapter files to get all info. Match with downloaded info from base URL. So we can index in run time too. No need of db. Now that chapters are stored in separate json, there are probably duplicate index in db pointing to same file that's making duplicate chapter in epub too(sometimes).
-
Why not make DB json too. that would be easier ti handle manually too.
-
thats a good suggestion.
-
well...
- The reason i'm indexing them is because i didn't want load each json file into memory separately, which could take some time. however, your query has given me an idea. I could create a class, which uses another thread to handle all io operation. the class itself can make sure the files and whats in memory will be in sync.
- Its unlikely for any duplicates to form in chapters table as all new insertion check for id clashes, in this case url
-
it would probably stay TinyDB for now... i would have to rewrite a significant amount of the database code.
you can expect update for 1 soon. 2 will take sometime. 3 maybe.
So as of now i would suggest letting the script run to completion. That way you would have a significantly reduced chances of any corruption.
As we are near the end of the year uni has become busy. so this might take longer that usual.