Deptry fails opening ipynb files

[JotaFan]

Describe the bug
When running deptry on a project with jupyter notebooks an error is raised. I think it might have something to do with the encoding on such files.

runing:
deptry . -rt src\requirements.in
will output:

Traceback (most recent call last):
  File "C:\Users\xxxx\.conda\envs\dummy\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\xxxx\.conda\envs\dummy\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\xxxx\.conda\envs\dummy\Scripts\deptry.exe\__main__.py", line 7, in <module>
  File "C:\Users\xxxx\.conda\envs\dummy\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\xxxx\.conda\envs\dummy\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "C:\Users\xxxx\.conda\envs\dummy\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\xxxx\.conda\envs\dummy\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\xxxx\.conda\envs\dummy\lib\site-packages\deptry\cli.py", line 216, in deptry
    Core(
  File "C:\Users\xxxx\.conda\envs\dummy\lib\site-packages\deptry\core.py", line 51, in run
    for mod in ImportParser().get_imported_modules_for_list_of_files(all_python_files)
  File "C:\Users\xxxx\.conda\envs\dummy\lib\site-packages\deptry\import_parser.py", line 24, in get_imported_modules_for_list_of_files
    modules_per_file = [self.get_imported_modules_from_file(file) for file in list_of_files]
  File "C:\Users\xxxx\.conda\envs\dummy\lib\site-packages\deptry\import_parser.py", line 24, in <listcomp>
    modules_per_file = [self.get_imported_modules_from_file(file) for file in list_of_files]
  File "C:\Users\xxxx\.conda\envs\dummy\lib\site-packages\deptry\import_parser.py", line 34, in get_imported_modules_from_file
    modules = self._get_imported_modules_from_ipynb(path_to_file)
  File "C:\Users\xxxx\.conda\envs\dummy\lib\site-packages\deptry\import_parser.py", line 69, in _get_imported_modules_from_ipynb
    imports = NotebookImportExtractor().extract(path_to_ipynb_file)
  File "C:\Users\xxxx\.conda\envs\dummy\lib\site-packages\deptry\notebook_import_extractor.py", line 23, in extract
    notebook = self._read_ipynb_file(path_to_ipynb)
  File "C:\Users\xxxx\.conda\envs\dummy\lib\site-packages\deptry\notebook_import_extractor.py", line 31, in _read_ipynb_file
    notebook: Dict[str, Any] = json.load(f)
  File "C:\Users\xxxx\.conda\envs\dummy\lib\json\__init__.py", line 293, in load
    return loads(fp.read(),
  File "C:\Users\xxxx\.conda\envs\dummy\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 3568: character maps to <undefined>

To Reproduce

Steps to reproduce the behavior:

1. Have a jupyter notebook on the project
2. Run deptry

System :

• OS: Win10
• Language Version: Python 3.8
• Poetry version: Poetry 1.2.2

I installed poetry manually, after getting this error, and got the error again (poetry was not installed with deptry).
This happens in old env, as in new conda env with just deptry installed.

[fpgmaas]

Thanks for raising this issue! This issue is caused by the NotebookImportExtractor not specifying the encoding while reading the file. The PythonImportExtractor has additional functionality that first tries to read the file without specifying the encoding, and if that returns a UnicodeDecodeError, it will use chardet to try and determine the file encoding.

I think we can solve this issue by using this retry-logic also to the NotebookImportExtractor. Probably the best way to do so is by moving the logic to the super-class ImportExtractor.

Will look into this and keep you posted!

[JotaFan]

I am happy to contibute!
I will try to suggest a fix, and open a PR!

[fpgmaas]

Hi @JotaFan! Sorry, I was not aware that you wanted to propose a fix for this yourself. Since I wanted to help out quickly, I already started developing a solution to this issue and created a PR here; #216 I hope that's okay with you. Please feel free to review it and provide feedback.

[fpgmaas]

I see you also forked the repository already before you wrote your comment here, so I am a bit afraid we are doing some duplicate work. I also don't see any methods to reach you in your GitHub profile. If we end up with two MR's, no problem, let's combine them into one and make it a co-authored MR :)

[JotaFan]

I have not started working yet. Thanks for the quick fix. Feel free to merge your PR. I am happy to have raised the issue!

[fpgmaas]

Should be solved in release 0.6.6. Would you be willing to give it a try, and let me know if deptry now runs succesfully on your project?

[JotaFan]

It is solved! It ran smoothly. Thanks

[fpgmaas]

Great, thanks for the feedback!