Pyright & pybind11

Run ./demo.sh to execute this example from start to finish.

Trying to figure out how to package type information together with pybind11 modules.

This example builds and packages a calculator module using pybind. The package has the following structure:

module calculator
- add(a: int, b: int) -> int
- submodule subtract
  - sub(a: int, b: int) -> int

Tested on macOS 13.5.1 (Apple Silicon) and in a Dev Container with Debian bullseye.

Prerequisites

Conda, to manage Python environments and dependencies.
Your system should also meet the minimum requirements for building C++ (make; gcc or clang, etc.)

Building the bindings

Create a new conda environment in .venv using environment.yml, and then activate it.

conda env create --file environment.yml --prefix .venv \
  && conda activate $(realpath .venv)

The environment uses Python 3.8.17, and installs the following packages:

pybind11
CMake, for generating the build system
MyPy, for generating type stubs

Conda also installs the necessary Python/pybind11 headers as well as pybind's CMake input files, which CMake will need.

Generate the build system:

cmake .

CMake will use CMakelists.txt. Your command output should look similar to the following (paths and compiler versions may vary):

-- The CXX compiler identification is AppleClang 14.0.3.14030022
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PythonInterp: ... (found suitable version "3.8.17", minimum required is "3.6")
-- Found PythonLibs: .../.venv/lib/libpython3.8.dylib
-- Found pybind11: .../.venv/lib/python3.8/site-packages/pybind11/include (found version "2.11.1")
-- Configuring done (0.5s)
-- Generating done (0.0s)
-- Build files have been written to: ...

Build the bindings:

make

This creates a dynamic library calculator.*.so in the current directory. Your command output should look similar to the following:

[ 50%] Building CXX object CMakeFiles/calculator.dir/calculator.cpp.o
[100%] Linking CXX shared module calculator.cpython-38-darwin.so
[100%] Built target calculator

It is now possible to import the module from Python:

>>> import calculator
>>> calculator.add(1, 2)
... 3

Generating type stubs

Generate type stubs from the built bindings using MyPy's stubgen command:

stubgen --package calculator --output typings

The stubs will be written to typings/calculator.

Processed 2 modules
Generated files under typings/calculator/

Installing & testing

Create a new conda environment inside example/.venv using example/environment.yml, and then activate it:

conda env create --file example/environment.yml --prefix example/.venv \
  && conda activate $(realpath example/.venv)

This time, the environment installs MyPy, Pyright, and the calculator package we just built.

See what files were installed:

pip show -f calculator

Name: calculator
...
Files:
  ...
  calculator.*.so
  calculator/__init__.pyi
  calculator/py.typed
  calculator/subtract.pyi

Run the example script:

python example/main.py

41 + 1 = 42
43 - 1 = 42

Comparing MyPy and Pyright

The example script contains a typing error:

# def add(a: int, b: int) -> int:
add(41, "1")

Type-check with MyPy:

mypy example/main.py

example/main.py:10: error: Argument 2 to "sub" has incompatible type "str"; expected "int"  [arg-type]
Found 1 error in 1 file (checked 1 source file)

Type-check with Pyright:

pyright example/main.py

.../example/main.py
  .../example/main.py:3:6 - warning: Import "calculator.subtract" could not be resolved from source (reportMissingModuleSource)
  .../example/main.py:10:37 - error: Argument of type "Literal['1']" cannot be assigned to parameter "arg1" of type "int" in function "sub"
    "Literal['1']" is incompatible with "int" (reportGeneralTypeIssues)
1 error, 1 warning, 0 informations

calculator.subtract is a submodule generated by pybind11 which has no corresponding source on the filesystem (the entire calculator package is in a single .so file).