Script error on imports
war3zlod3r opened this issue · 26 comments
Describe the bug
Seems to happen mostly with TBP imports, had about 20 of them when trying to do a large import
Screenshots/Logs
22-Dec-2019 08:32:30 - WARNING :: mylar.archivefiles.81 : Thread-57 : [EOL while scanning string literal (, line 1)] Error encountered converting import data
22-Dec-2019 08:32:30 - ERROR :: mylar.excepthook.315 : Thread-57 : Uncaught exception: Traceback (most recent call last):
File "C:\Mylar\mylar\logger.py", line 337, in new_run
old_run(*args, **kwargs)
File "C:\Python27\lib\threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "C:\Mylar\mylar\importer.py", line 418, in addComictoDB
moveit.archivefiles(comicid, comlocation, imported)
File "C:\Mylar\mylar\moveit.py", line 83, in archivefiles
ComicName = imported['ComicName']
TypeError: string indices must be integers
Traceback (most recent call last):
File "C:\Mylar\mylar\logger.py", line 337, in new_run
old_run(*args, **kwargs)
File "C:\Python27\lib\threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "C:\Mylar\mylar\importer.py", line 418, in addComictoDB
moveit.archivefiles(comicid, comlocation, imported)
File "C:\Mylar\mylar\moveit.py", line 83, in archivefiles
ComicName = imported['ComicName']
TypeError: string indices must be integers
Environment (please complete the following information):
- OS: Windows 2016 Server
- Branch / commit: Development
Additional information
I get this mainly when presented with the manual intervention option, all files are metatagged and named based on the same convention Mylar is using but still get alot of "no results" or "manual intervention required" sometimes when trying to use manual intervention after clicking select nothing happens or the import page just reloads.
22-Dec-2019 11:13:45 - DEBUG :: mylar.preSearchit.4637 : Thread-15 : [IMPORT][1982] Comparing: hulk - TO - incrediblehulkweekly
22-Dec-2019 11:13:45 - ERROR :: mylar.excepthook.315 : Thread-15 : Uncaught exception: Traceback (most recent call last):
File "C:\Mylar\mylar\logger.py", line 337, in new_run
old_run(*args, **kwargs)
File "C:\Python27\lib\threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "C:\Mylar\mylar\webserve.py", line 4708, in preSearchit
myDB.upsert("searchresults", nVal, cVal)
File "C:\Mylar\mylar\db.py", line 169, in upsert
self.action(query, valueDict.values() + keyDict.values())
File "C:\Mylar\mylar\db.py", line 128, in action
sqlResult = self.connection.execute(query, args)
InterfaceError: Error binding parameter 9 - probably unsupported type.
Traceback (most recent call last):
File "C:\Mylar\mylar\logger.py", line 337, in new_run
old_run(*args, **kwargs)
File "C:\Python27\lib\threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "C:\Mylar\mylar\webserve.py", line 4708, in preSearchit
myDB.upsert("searchresults", nVal, cVal)
File "C:\Mylar\mylar\db.py", line 169, in upsert
self.action(query, valueDict.values() + keyDict.values())
File "C:\Mylar\mylar\db.py", line 128, in action
sqlResult = self.connection.execute(query, args)
sqlite3.InterfaceError: Error binding parameter 9 - probably unsupported type.
another one
We need to see the debug lines leading up to the trace backs.... one line doesn’t give enough context, can you provide the ~50 lines prior?
Mylar won't be able to import TPB/GN's as the importer hasn't been updated to accomodate the new logic that's currently in use in other aspects of the program (file parsing, post-processing, searching, etc). This is on the to-do obviously, but in the meantime, the easiest method to avoid the errors is to just remove the files from the scan folder temporarily so the import can finish the initial scan.
I'm assuming both posted errors are in regards to it being a TPB being scanned in. If I'm wrong however, please paste/pastebin the full debug log leading up to the error, as we need to see the context of what's causing the error - not just the error itself.
Here's the full log
Unfortunately there's nothing but search results in that particular log.
You'll have to keep verbose / debug mode enabled, and then try and re-run the import. Once the error occurs, or it completes (not sure if it completes even with the error), make sure to shut down Mylar so the logs don't rotate. Then pastebin the relevant log so we can track it properly.
Keep in mind that there the 5 log files rotate so that if you're doing a rather large import, you might want to bump up the max_logsize in the config.ini (when editing the ini, make sure Mylar is not running). Something more than the default of 10mb should ensure that you capture the traceback with the relevant logging.
mylar.log
this should be a more useful log
Apparently it's passing back a tuple or something that's not a string, so when it goes to write it to the db it can't as it's not a supported type. Can't quite see where it'd be passing back a value like that - but if we can add in a logging line right before the error, should be able to see what it is that's trying to be written and go from there.
Ok how do I do that?
I can add in a logging line - all you need to do is update to the latest when it's pushed, then run the import again with debug logging enabled and pastebin the leadup log lines to the traceback so that we can get the properly information.
I've got 2 other commits I'm doing atm, will push something within the next 10 minutes for you to try.
mylar.log
Here it goes again also having alot of imports on files that are tagged and named appropriately "All-New Doop" is a good example but they still fail to import with "no results" in the status column can you give any advice about this? Also just wonder what is the expected behavior for importing a series that you already have? Does it replace the existing files or ignore them?
I'm having the same issue. I'll attach a log and a screenshot of the stacktrace...
That's interesting - it's passing a value as a dict when it shouldn't be. I know at one point I was having to return multiple values via the GUI and used the dicts to get around things - but the error you guys are encountering isn't related to the GUI itself.
I tried to reference back and find the point where it was mangling the dict together, but couldn't locate it (at least not yet).
Put up a quick patch to development (00d7107) tho so that it will determine if it's a dict or not prior to adding it, and correct it prior to writing to the db - which should bandaid the problem until I can track down the mangled dict.
@theotherp, thanks for that screenshot - that narrowed it down as I was looking at the wrong field value apparently (thought it was a description character, but obviously wasn't). I'm always interested in new debug tools, would you mind telling me what debugging programl your screenshot was used from as it looks like something that might be useful for us.
I'm using https://www.jetbrains.com/pycharm/ which is awesome. I don't know what you're using right now but I really recommend looking into it - I can't imagine developing python without it.
You can apply for a free license for their whole program suite here: https://www.jetbrains.com/community/opensource/
With your new commit I don't get the error anymore. Now it needs manual intervention but when I click "Select" no popup is opened. The page reloads and I get a JS error in the browser console (jquery.dataTables.min.js:35:390). Should I open a new bug for that?
I've tried pycharm before, but didn't really give it a good go - think I'll have to retry it and see how it goes. Doing everything via shell makes it very easy to miss things.
If you can paste any additional info about the js error, that would help. You can keep everything in here as it's all related (thus far) to the same import aspect of manual intervention. I'll see if I can replicate the error on my side.
I downloaded the unminified version of datatables: 
row.ancells is undefined. I found this very old ticket: https://datatables.net/forums/discussion/24834/getting-javascript-error-when-viewing-details-for-a-row
I tried to find out what's going on but didn't get very far and gave up...
I don't know if it helps but here's a screenshot of the import window:
Any advice you can give on the “no results” issue or information about the expected result if trying to import a series that exists already or has some issues but not all?
There's not much to say - 'no results' just means exactly that. Based on the filename and or metadata within the file (if it exists), Mylar could not find a match on CV and therefore cannot determine what it is. If this happens - you need to make sure the files are named appropriately, as in it needs to have a series title (that coincides with what CV has it named), and an issue number. The issue year and/or series volume, while not required will help to narrow down series that have multiple volumes (if you name it some series 1.cbr, it has no idea what year is was published in or the volume it belongs to unless the file has been metatagged properly).
You also cannot import a series that exists already - in these cases, it will indicate that the series already exists on your watchlist and that you probably need to perform a manual post-processing against that specific location (manual post-processing will only post-process files for series that it determines exist on the watchlist, importing is for series that do not exist on your watchlist). Doesn't matter if you have some issues - if a series exists on your watchlist, it will not be picked up by the importer by design (almost every other download management tool behaves exactly the same way).
@theotherp, thanks for that info - I'm looking at it now, but that old ticket link doesn't make me feel very confident in being able to fix this, aside from possibly upgrading the datatables version to a more recent version. Also, weird how it broke up the series into 2 separate entities - I'm just assuming that they're probably named differently, and didn't have metadata which is why it just file parsed them. Can I ask what unminified version you downloaded? The one included with Mylar doesn't make a reference to that row.anCells line at all from what I've checked at least.
I should've mentioned that I already tried the latest version and it still failed. I'm not sure it's an issue with the library.
I could send you my config, db and the file to import - not sure if that would allow you to reproduce the error. It's not really that important though, I actually rarely use Mylar...
Sure you can send me just the db file - I should be able to reproduce it with just that alone, since the info is all in the dB. You can email it to me, just in case there's sensitive info in there (there's no keys, but just as a security precaution). I'd like to get this bug fixed, since I haven't been able to track it down as of yet, and I'd like the import process to run as smooth as possible.
I just sent you the data.
Now (pretty sure I didn't see that before) I get the error message There are no search results to view for this entry Criminal [680640]. Something is probably wrong..
Ok, this is what happens:
- I execute "Save changes and scan".
- I click the Import button.
- I see one entry with an unknown year, "Not imported status" and the options to import or remove.
- I click Import.
- I see a small popup "There was an error" but not manual intervention option.
- I click "Import" again and get the small bubble "Criminal was successfully imported".
- I see "Select" and click it.
- The page reloads to the same page.
Full info log:
12-Jan-2020 22:17:07 - INFO :: mylar.comicScan.4248 : CP Server Thread-12 : Now updating config...
12-Jan-2020 22:17:07 - INFO :: mylar.comicScan.4251 : CP Server Thread-12 : forcescan is: 0
12-Jan-2020 22:17:07 - INFO :: mylar.libraryScan.47 : LibraryScan : Scanning comic directory: e:\Download\Trials\comics\
12-Jan-2020 22:17:07 - INFO :: mylar.libraryScan.158 : LibraryScan : I have successfully discovered & parsed a total of 1 files....analyzing now
12-Jan-2020 22:17:07 - INFO :: mylar.libraryScan.159 : LibraryScan : I have not been able to determine what 0 files are
12-Jan-2020 22:17:07 - INFO :: mylar.libraryScan.160 : LibraryScan : However, 0 out of the 0 files are in a cbz format, which may contain metadata.
12-Jan-2020 22:17:07 - INFO :: mylar.libraryScan.161 : LibraryScan : [ERRORS] I have encountered 0 file-scanning errors during the scan, but have recorded the necessary information.
12-Jan-2020 22:17:07 - INFO :: mylar.libraryScan.169 : LibraryScan : loading in the watchlist to see if a series is being watched already...
12-Jan-2020 22:17:07 - INFO :: mylar.libraryScan.216 : LibraryScan : Successfully loaded 13 series from your watchlist.
12-Jan-2020 22:17:07 - INFO :: mylar.scanLibrary.643 : LibraryScan : [IMPORT] Parsing/Reading of files completed!
12-Jan-2020 22:17:07 - INFO :: mylar.scanLibrary.644 : LibraryScan : [IMPORT] Attempting to import 1 files into your watchlist.
12-Jan-2020 22:17:07 - INFO :: mylar.scanLibrary.645 : LibraryScan : [IMPORT-BREAKDOWN] Files with ComicIDs successfully extracted: 0
12-Jan-2020 22:17:07 - INFO :: mylar.scanLibrary.646 : LibraryScan : [IMPORT-BREAKDOWN] Files that had to be parsed: 1
12-Jan-2020 22:17:07 - INFO :: mylar.scanLibrary.647 : LibraryScan : [IMPORT-BREAKDOWN] Files that were unable to be parsed: 0
12-Jan-2020 22:17:07 - INFO :: mylar.scanLibrary.648 : LibraryScan : [IMPORT-BREAKDOWN] Files that caused errors during the import: 0
12-Jan-2020 22:17:07 - INFO :: mylar.ThreadcomicScan.4274 : CP Server Thread-12 : Successfully scanned in directory. Enabling the importResults button now.
12-Jan-2020 22:17:31 - INFO :: mylar.dbUpdate.353 : Thread-39 : Update complete
12-Jan-2020 22:17:31 - INFO :: mylar.run.2845 : Thread-41 : [FOLDER-CHECK] Checking folder e:\Download\Trials\comics\ for newly snatched downloads
12-Jan-2020 22:17:31 - INFO :: mylar.traverse_directories.1481 : Thread-41 : there are 1 files.
12-Jan-2020 22:17:31 - INFO :: mylar.Process.409 : Thread-41 : I have located 1 files that I should be able to post-process. Continuing...
12-Jan-2020 22:17:31 - INFO :: mylar.Process.543 : Thread-41 : Now checking: Criminal [116367]
12-Jan-2020 22:17:31 - INFO :: mylar.Process.773 : Thread-41 : [POST-PROCESSING][ISSUE-VERIFY] Found matching issue # 5000 for ComicID: 116367 / IssueID: 710599
12-Jan-2020 22:17:31 - INFO :: mylar.Process.1618 : Thread-41 : [POST-PROCESSING] No matches for Manual Run ... exiting.
12-Jan-2020 22:17:31 - INFO :: mylar.run.2848 : Thread-41 : [FOLDER-CHECK] Finished checking for newly snatched downloads
12-Jan-2020 22:18:06 - INFO :: mylar.preSearchit.4442 : CP Server Thread-17 : [IMPORT][Criminal] Now preparing to import. First I need to determine the highest issue, and possible year(s) of the series.
12-Jan-2020 22:18:09 - INFO :: mylar.preSearchit.4665 : CP Server Thread-17 : [IMPORT] Returning results to Select option - there are 5 possibilities, manual intervention required.
12-Jan-2020 22:18:33 - INFO :: mylar.preSearchit.4363 : CP Server Thread-12 : [IMPORT] There is an import already running. Please wait for it to finish, and then you can resubmit this import.
12-Jan-2020 22:19:21 - WARNING :: mylar.importresults_popup.4790 : CP Server Thread-13 : There are no search results to view for this entry Criminal [875217]. Something is probably wrong.
I’ve been using comictagger GUI to tag and rename the files so I figured it should match up against CV but for some reason they still fail even with the use existing metadata option
@war3zlod3r , what are some of the filenames that you trying to import? Are they TPB's or Graphic Novels ? If they're not matching up to CV, then there might be something different between the datasets that's forcing Mylar to ignore it as a valid result.
@theotherp , I checked out the info you sent (thanks for that too). It looks like you're trying to import a series that you already have on your watchlist (Criminal). So it wouldn't be able to import because it exists already, but it should have said something along the lines of already existing on your watchlist, and/or updated the status to indicate as such. But the error you just noticed about not having any results to view is normal - however, we should add some sort of dialog notification so that people are aware of the no results instead of just flipping back to the main imports page.
I also saw that you tried to do a manual post-processing on it, but it wasn't picked up (so I'm guessing at that point, you attempted the import). In attempting to duplicate, I noticed that the reason why it was rejected as a valid result for post-processing is because of the lack of a volume label or an issue year ([POST-PROCESSING][ISSUE-VERIFY][Lone Volume FAILURE] Volume label of v3 indicates that there is more than one volume for this series, but the one on your watchlist has no volume label set.).
With comics that contain multiple volumes over a variety of years (some overlapping) it becomes necessary to check if a volume label and or issue year is present in the filename so that Mylar can make sure it's the correct series (in case you have more than one volume in your watchlist). In this case, it had neither - so it was rejected (which is the problem with comics & a lack of naming conventions unfortunately). There might be a case to add some kind of inclusion so that if there's only one series on your watchlist and that information isn't available in the filename to allow it, but I'll have to think about a way to include that option.
I also can't duplicate that javascript error, but it might be due to the overlapping of the same items. You might want to do a Flush Imports, located at the top of the Imports Management page, to get rid of the items in your import list as the that one Criminal issue is listed 5 different times (thus why it's causing the multiple entries).
any ideas? Can't seem to come up with any reason on my end why these imports fail but the imports in general seem very inconsistent/unreliable. The issues are named properly and already tagged and if I add the series and then drop in the files to the folder Mylar creates it works great but I would say 8/10 times fails to import the exact same files.