PyDeps - Unable to render a module that uses a stdlib

Question

PyDeps - Unable to render a module that uses a stdlib

Dholguin-Programmer opened this issue a year ago · 4 comments

Dholguin-Programmer commented a year ago

As the title suggest, I'm having issues using the pydeps utility whenever one of my classes imports a standard python module (os, etc).

Directory Structure

root
     Main.py
     Submodules\
          Class1.py
          Class1Modules.py

Main.py

import os
from root.Submodules.Class1 import SomeClass
from root.Submodules.Class1Modules import some_func

if __name__ == '__main__':
     class_instance = SomeClass()
     some_func(class_instance)
     print(F'finished executing script at {os.getcwd()}')

I noticed that when I run the following command via bash, that it throws some sort of marshalling error.
Command: cd root && pydeps Main.py --show-deps --show-dot --show-cycles --include-missing --pylib-all --no-show -o Main.svg && cd -

Return Code: ValueError: bad marshal data (unknown type code)

Thus far, I've tried playing around with various command line options and have combed through various discussion boards, but haven't seen anything. Wanted to reach out to see if I'm misunderstanding something fundamental and if so, what it is. The only thing I noticed of importance is that when I remove all references to the os module internal to Main.py and omit the --pylib-all option, that I'm getting the graph I expect. The graph looks more or less something like this:


     -------------------------------
     ||                           ||
     \/                           ||
Class1.py ---> Main <---- Class1Modules.py

This isn't specific to the os module and I have seen it with anything that uses the standard set of python modules included. Below are versions of which Python version I have installed as well as the version of pydeps.

Python Version: Python 3.9.14
PyDeps Version: v1.12.2

Answer 1 · 2023-04-27T16:48:03.000Z

Hi @Dholguin-Programmer , and thank you for your interest in pydeps.

If I add the missing (empty) __init__.py files to your example:

c:\srv\tmp\pydeps181> tree .
.
    `-- root
    |-- __init__.py
    |-- Main.py
    `-- Submodules
        |-- __init__.py
        |-- Class1.py
        `-- Class1Modules.py

and run pydeps from the parent folder of root (here /srv/tmp/pydeps181), with the --pylib flag:

c:\srv\tmp\pydeps181> pydeps root --pylib

I get:

Answer 2 · 2023-04-27T17:58:55.000Z

Would you happen to know if relative vs absolute import paths would influence the expected behavior? I ask because my overall project is many levels of submodules that are references and am trying to get the leaf node directory and submodule wise to generate successfully before I attempt to generate from the 'root'.

Also, is there a means I can toggle to try to debug this ValueError ? I tried enabling --debug but I'm not really seeing something that gives me enough output to deduce the actual error.

Answer 3 · 2023-04-27T22:19:28.000Z

From experience it is usually better to let pydeps have at the entire tree and exclude branches using the -x flag.

The ValueError is probably coming from python's modulefinder (or potentially from pydeps/mf27.py line 75, but a traceback would help verify.

You can turn on modulefinder debugging using the --debug-mf flag wich takes an integer specifying the level of logging you want, e.g. --debug-mf=2 (it provides a ton of output).

For debugging pydeps the -LDEBUG flag is more helpful than the --debug flag (not for any particularly good reason..)

Answer 4 · 2023-05-01T13:24:43.000Z

Hey, so running with -LDEBUG, I'm getting more information, but am not sure making much sense of the output, but at least I have some now. Below is what I got when running the follinwg command :

pwd
/home/<some_path>/emulator/
pydeps NetworkVisualizer.py --no-show --include-missing -LDEBUG

pydeps.py:150: DEBUG: Target: {
    "calling_dir": "/home/<some_path>/emulator",
    "calling_fname": "NetworkVisualizer.py",
    "dirname": "/home/<some_path>/emulator",
    "exists": true,
    "fname": "NetworkVisualizer.py",
    "is_dir": false,
    "is_module": false,
    "is_pysource": true,
    "modname": "NetworkVisualizer",
    "modpath": "emulator.NetworkVisualizer",
    "package_root": "/home/<some_path>/emulator",
    "path": "/home/<some_path>/emulator/NetworkVisualizer.py",
    "relpath": "emulator/NetworkVisualizer.py",
    "syspath_dir": "/home/<some_path>",
    "workdir": "/home/<some_path>/emulator"
}

py2depgraph.py:199: INFO: py2dep({
    "calling_dir": "/home/<some_path>/emulator",
    "calling_fname": "NetworkVisualizer.py",
    "dirname": "/home/<some_path>/emulator",
    "exists": true,
    "fname": "NetworkVisualizer.py",
    "is_dir": false,
    "is_module": false,
    "is_pysource": true,
    "modname": "NetworkVisualizer",
    "modpath": "emulator.NetworkVisualizer",
    "package_root": "/home/<some_path>/emulator",
    "path": "/home/<some_path>/emulator/NetworkVisualizer.py",
    "relpath": "emulator/NetworkVisualizer.py",
    "syspath_dir": "/home/<some_path>",
    "workdir": "/home/<some_path>"
})

dummymodule.py:98: DEBUG: dummy-filename: 'NetworkVisualizer.py' (/home/<some_path>/emulator)[module=False, dir=False, file=True]
py2depgraph.py:208: DEBUG: Exclude: ['migrations']
py2depgraph.py:209: DEBUG: KW: {'debug': False, 'no_config': False, 'version': False, 'log': None, 'find_package': False, 'fname': 'NetworkVisualizer.py', 'verbose': 0, 'output': '/home/<some_path>/emulator/emulator_NetworkVisualizer.svg', 'format': 'svg', 'display': None, 'no_show': True, 'show_deps': False, 'show_raw_deps': False, 'deps_out': None, 'show_dot': True, 'dot_out': None, 'nodot': None, 'no_output': False, 'show_cycles': False, 'debug_mf': 0, 'noise_level': 200, 'max_bacon': 2, 'max_module_depth': 0, 'pylib': False, 'pylib_all': False, 'include_missing': True, 'exclude_exact': [], 'only': [], 'externals': False, 'reverse': False, 'rankdir': 'TB', 'cluster': False, 'min_cluster_size': 0, 'max_cluster_size': 0, 'keep_target_cluster': False, 'collapse_target_cluster': False, 'rmprefix': [], 'start_color': 0, 'show': False, 'curdir': '/home/<some_path>/emulator', 'isdir': False, 'dummyname': 'NetworkVisualizer.py'}
py2depgraph.py:220: DEBUG: CURDIR: /home/<some_path>/emulator
dummymodule.py:105: DEBUG: Getting text from 'NetworkVisualizer.py'
py2depgraph.py:221: DEBUG: FNAME: 'NetworkVisualizer.py', CONTENT:

The Modules included (and branched) are as follows:

import tkinter as tk
from tkinter import ttk
import signal
from modules.widgets.notebooks.TabsManager.Class import Class as TabsManager

Taking your advice, I ran this from the root directory of the project. I'll proceed with the -X option and trying to narrow things down, but is there any particular signs I should be looking for? Once I deduce which modules are causing pydeps to barf, I'll post the information here.