Smart import for Python

Automatically discovers & imports entities, used in the current module.

No magic or monkey patching. Only standard Python functionality.

Before	After
import math from my_project import calc # 100500 other imports def my_code(argument, function=calc): return math.log(function(argument))	import smart_imports smart_imports.all() # no any other imports def my_code(argument, function=calc): return math.log(function(argument))

Before

After

import math
from my_project import calc
# 100500 other imports

def my_code(argument, function=calc):
    return math.log(function(argument))

import smart_imports
smart_imports.all()
# no any other imports

def my_code(argument, function=calc):
    return math.log(function(argument))

MyPy supported.

Summary

Get source code of the module, from which smart_imports.all() has called.
Parse it, find all not initialized variables.
Search imports, suitable for found variables.
Import them.

Library process only modules, from which smart_imports called explicitly.

Main idea

With time every complex project develops own naming convention. If we translate that convention into more formal rules, we will be able to make automatic imports of every entity, knowing only its name.

For example, we will not need to write import math to call math.pi, since our system will understand that math is the module of the standard library.

How it works

Code from the header works in such way:

smart_imports.all() builds AST of the module from which it has called.
Library analyses AST and searches for not initialized variables.
Name of every found variable processed thought chain of rules to determine the correct module (or its attribute) to import. If the rule finds the target module, chain breaks and the next rules will not be processed.
Library load found modules and add imported entities into the global namespace.

Smart Imports searches not initialized variables in every part of code (including new Python syntax).

Automatic importing turns on only for modules, that do explicit call of smart_imports.all().

Moreover, you can use normal imports with Smart Imports at the same time. That helps to integrate Smart Imports step by step.

You can notice, that AST of module builts two times:

when CPython imports module;
when Smart Imports process call of smart_imports.all().

We can build AST once (for that we can add hook into the process of importing modules with help of PEP-0302), but it will make import event slower. I think that it is because at import time CPython builds AST in terms of its internal structures (probably implemented in C). Conversion from them to Python AST cost more than building new AST from scratch.

Smart Imports build AST only once for every module.

Default import rules

Smart Imports can be used without configuration. By default it uses such rules:

By exact match looks for the module with the required name in the folder of the current module.
Checks if the standard library has a module with the required name.
1. By exact match with top-level packages (for example, math ).
2. For sub-packages and modules checks complex names with dots replaced by underscores (for example, os.path will be imported for name os_path).
By exact match looks for installed packages with the required name (for example, requests ).

Performance

Smart Imports does not slow down runtime but increases startup time.

Because of building AST, startup time increased in 1.5-2 times. For small projects it is inconsequential. At the same time, the startup time of large projects depends mostly on architecture and dependencies between modules, than from the time of modules import.

In the future, part of Smart Imports can be rewritten in C — it should eliminate startup delays.

To speed up startup time, results of AST processing can be cached on the file system. That behavior can be turned on in the config. SmartImports invalidates cache when module source code changes.

Also, Smart Imports' work time highly depends on rules and their sequence. You can reduce these costs by modifying configs. For example, you can specify an explicit import path for a name with Rule 4: custom names.

Configuration

The logic of default configuration was already described. It should be enough to work with the standard library.

Default config:

{
    "cache_dir": null,
    "rules": [{"type": "rule_local_modules"},
              {"type": "rule_stdlib"},
              {"type": "rule_predefined_names"},
              {"type": "rule_global_modules"}]
}

If necessary, a more complex config can be put on a file system.

Example of complex config (from my pet project).

At the time of call smart_import.all() library detects a location of config file by searching file smart_imports.json from the current folder up to root. If a file will be found, it will become config for the current module.

You can use multiple config files (place them in different folders).

There are few config parameters now:

{
    // folder to store cached AST
    // if not specified or null, cache will not be used
    "cache_dir": null|"string",

    // list of import rules (see further)
    "rules": []
}

Import rules

A sequence of rules in configs determines the order of their application. The first success rule stops processing and makes import.

Rule 1: predefined names will be often used in the examples below. It required for the correct processing of default names like print.

Rule 1: predefined names

Rule silences import search for predefined names like __file__ and builtins like print.

# config:
# {
#    "rules": [{"type": "rule_predefined_names"}]
# }

import smart_imports

smart_imports.all()

# Smart Imports will not search for module with name __file__
# event if variable is not initialized explicity in code
print(__file__)

Rule 2: local modules

Rule checks if a module with the required name exists in the folder of the current module. If the module found, it will be imported.

# config:
# {
#    "rules": [{"type": "rule_predefined_names"},
#              {"type": "rule_local_modules"}]
# }
#
# project on file sytem:
#
# my_package
# |-- __init__.py
# |-- a.py
# |-- b.py

# b.py
import smart_imports

smart_imports.all()

# module "a" will be found and imported
print(a)

Rule 3: global modules

Rule tries to import the module by name.

# config:
# {
#    "rules": [{"type": "rule_predefined_names"},
#              {"type": "rule_global_modules"}]
# }
#
# install external package
#
# pip install requests

import smart_imports

smart_imports.all()

# module "requests" will be found and imported
print(requests.get('http://example.com'))

Rule 4: custom names

Rule links a name to the specified module and its attribute (optionally).

# config:
# {
#    "rules": [{"type": "rule_predefined_names"},
#              {"type": "rule_custom",
#               "variables": {"my_import_module": {"module": "os.path"},
#                             "my_import_attribute": {"module": "random", "attribute": "seed"}}}]
# }

import smart_imports

smart_imports.all()

# we use modules of the standard library in that example
# but any module can be used
print(my_import_module)
print(my_import_attribute)

Rule 5: standard library

Rule checks if the standard library has a module with the required name. For example math or os.path (which will be imported for the name os_path).

That rule works faster than Rule 3: global modules, since it searches module by predefined list. Lists of modules for every Python version was collected with help of stdlib-list.

# config:
# {
#    "rules": [{"type": "rule_predefined_names"},
#              {"type": "rule_stdlib"}]
# }

import smart_imports

smart_imports.all()

print(math.pi)

Rule 6: import by prefix

Rule imports module by name from the package, which associated with name prefix. It can be helpful when you have a package used in the whole project. For example, you can access modules from package utils with prefix utils_.

# config:
# {
#    "rules": [{"type": "rule_predefined_names"},
#              {"type": "rule_prefix",
#               "prefixes": [{"prefix": "utils_", "module": "my_package.utils"}]}]
# }
#
# project on filesystem
#
# my_package
# |-- __init__.py
# |-- utils
# |-- |-- __init__.py
# |-- |-- a.py
# |-- |-- b.py
# |-- subpackage
# |-- |-- __init__.py
# |-- |-- c.py

# c.py

import smart_imports

smart_imports.all()

print(utils_a)
print(utils_b)

Rule 7: modules from parent package

If you have sub-packages with the same name in different parts of your project (for example, tests or migrations), you can allow for them to search modules by name in parent packages.

# config:
# {
#    "rules": [{"type": "rule_predefined_names"},
#              {"type": "rule_local_modules_from_parent",
#               "suffixes": [".tests"]}]
# }
#
# project on file system:
#
# my_package
# |-- __init__.py
# |-- a.py
# |-- tests
# |-- |-- __init__.py
# |-- |-- b.py

# b.py

import smart_imports

smart_imports.all()

print(a)

Rule 8: modules from namespace

The rule allows for modules from a specified package to import by name modules from another package.

# config:
# {
#    "rules": [{"type": "rule_predefined_names"},
#              {"type": "rule_local_modules_from_namespace",
#               "map": {"my_package.subpackage_1": ["my_package.subpackage_2"]}}]
# }
#
# project on filesystem:
#
# my_package
# |-- __init__.py
# |-- subpackage_1
# |-- |-- __init__.py
# |-- |-- a.py
# |-- subpackage_2
# |-- |-- __init__.py
# |-- |-- b.py

# a.py

import smart_imports

smart_imports.all()

print(b)

How to add custom rule?

Subclass smart_imports.rules.BaseRule.
Implement required logic.
Register rule with method smart_imports.rules.register.
Add rule to config.
???
Profit.

Look into the implementation of current rules, if you need an example.

MyPY

Plugin for integration with MyPy implemented.

MyPy config (mypy.ini) example:

[mypy]
plugins = smart_imports.plugins.mypy

Plans

I love the idea of determining code properties by used names. So, I will try to develop it in the borders of Smart Imports and other projects.

What I planning for Smart Imports:

Continue support and patch it for new versions of Python.
Research usage of type annotations to import automatization.
Try to implement lazy imports.
Implement utilities for automatic config generation and code refactoring.
Rewrite part of code in C, to speedup AST construction.
Implement integrations with popular IDEs.

I open to your suggestions. Feel free to contact me in any way.

ceprio/smart-imports