ut-parla/Parla.py

Forwarding Libraries Are Too Invasive

Opened this issue · 1 comments

Currently setting attributes on libraries and adding/removing stuff from sys.modules takes up an absolutely awful amount of time when a library lazy-loads its submodules. This is even worse when modules are lazy-imported inside routine calls. To at least some extent lazy-loading is going to be slow until we can fix #10, but based off the profiling data from @dialecticDolt it sounds like packing and unpacking stuff from sys.modules is the biggest performance bottleneck right now.

Proposed fix: stop patching forwarding modules in as attributes on existing modules, and stop swapping stuff into and out of sys.modules when a new module import is detected. Here's a new proposal:

  • sys.modules still contains forwarding modules
  • import operations still return forwarding modules
  • module attribute access to get another module will not return a forwarding module for the corresponding submodule (if this is too much of an ask, we could potentially do some magic inside our __getattribute__ override to specially handle submodules, but I don't think this is actually a problematic change
  • During an import in-progress for a new module, import operations for that module return the module objects specific to the currently active VEC. (I'm not 100% sure what to do about packages that somehow pick up any of their own modules out of sys.modules during setup. This seems like something that's unlikely to hurt us in any big way, but once this is fixed may be worth documenting as its own issue.)