Bug: Large batch syncing fails

Question

Bug: Large batch syncing fails

Opened this issue 2 years ago · 2 comments

Describe the bug
Batch syncing of a large number of items a a library fails. Syncing a small number of item at a time works fine.

To Reproduce
This is the library I am working with: https://www.zotero.org/groups/4774891/proulxs/library.
On my local machine, I select all items, right click, Cita, Sync citations with Wikidata.

Expected behavior
The number of items selected for a batch sync should not matter.

Screenshots

Environment:

Zotero version: 6.0.13
Cita version: 0.3.3
OS name and version: MacOS 12.2.1

Additional context
List of installed add-ons:
ZotFile
Folder Import for Zotero

Debug output log: Debug Output.txt
Item or collection causing trouble: Exported Items.rdf.zip

Answer 1 · 2022-09-13T09:09:22.000Z

Thanks for the detailed report! I had a quick look to try and localise this - Cita reports that all the items were of an unsupported type [here](https://github.com/diegodlh/zotero-cita/blob/master/src/citations.js#L611) - but from a quick check of your library they are normal journal articles, book sections, et c. - That's triggered by the `else` from this if statement [here](https://github.com/diegodlh/zotero-cita/blob/master/src/citations.js#L524) - that we didn't successfully get the item from its QID - That could be caused by the attempt to get all the items failing [here](https://github.com/diegodlh/zotero-cita/blob/master/src/citations.js#L379), perhaps if the query is too big That's just a quick guess - I will debug this properly soon. I know with reconciling items via to get a QID there was a limit to the query size above which things would break.

Answer 2 · 2022-10-23T22:00:42.000Z

Sorry this has taken me so long! At least on my end I'm getting an out of memory error when I try to sync that many items at once. For each citation we need to download the corresponding item's data from Wikidata, which is about ~1 MB per item. So for 50K citations it's on the order of 5 GB of memory. Do you see significant memory usage on your end too? I then get an error that the item data downloaded from wikidata isn't valid - not sure if this is the real error or as a result of the memory issue.

Does this also run super slowly for you? I'm working to speed this up, but I guess to solve the memory issue things would need to be progressively written to disk.