cvnp: A C++ repository from virtuald

cvnp: pybind11 casts and transformers between numpy and OpenCV, possibly with shared memory

Explicit transformers between cv::Mat / cv::Matx and numpy.ndarray, with or without shared memory

Notes:

When going from Python to C++ (nparray_to_mat), the memory is always shared
When going from C++ to Python (mat_to_nparray) , you have to specify whether you want to share memory via the boolean parameter share_memory

    pybind11::array mat_to_nparray(const cv::Mat& m, bool share_memory);
    cv::Mat         nparray_to_mat(pybind11::array& a);

        template<typename _Tp, int _rows, int _cols>
    pybind11::array matx_to_nparray(const cv::Matx<_Tp, _rows, _cols>& m, bool share_memory);
        template<typename _Tp, int _rows, int _cols>
    void            nparray_to_matx(pybind11::array &a, cv::Matx<_Tp, _rows, _cols>& out_matrix);

Warning: be extremely cautious of the lifetime of your Matrixes when using shared memory! For example, the code below is guaranted to be a definitive UB, and a may cause crash much later.

pybind11::array make_array()
{
    cv::Mat m(cv::Size(10, 10), CV_8UC1);               // create a matrix on the stack
    pybind11::array a = cvnp::mat_to_nparray(m, true);  // create a pybind array from it, using
                                                        // shared memory, which is on the stack!
    return a;                                                        
}  // Here be dragons, when closing the scope!
   // m is now out of scope, it is thus freed, 
   // and the returned array directly points to the old address on the stack!

Automatic casts:

Without shared memory

Casts without shared memory between cv::Mat, cv::Matx, cv::Vec and numpy.ndarray
Casts without shared memory for simple types, between cv::Size, cv::Point, cv::Point3 and python tuple

With shared memory

Casts with shared memory between cvnp::Mat_shared, cvnp::Matx_shared, cvnp::Vec_shared and numpy.ndarray

When you want to cast with shared memory, use these wrappers, which can easily be constructed from their OpenCV counterparts. They are defined in cvnp/cvnp_shared_mat.h.

Be sure that your matrixes lifetime if sufficient (do not ever share the memory of a temporary matrix!)

Supported matrix types

Since OpenCV supports a subset of numpy types, here is the table of supported types:

➜ python
>>> import cvnp
>>> cvnp.print_types_synonyms()
  cv_depth   cv_depth_name   np_format   np_format_long
     0          CV_8U           B         np.uint8  
     1          CV_8S           b         np.int8   
     2          CV_16U          H        np.uint16  
     3          CV_16S          h         np.int16  
     4          CV_32S          i         np.int32  
     5          CV_32F          f          float    
     6          CV_64F          d        np.float64

How to use it in your project

Add cvnp to your project. For example:

cd external
git submodule add https://github.com/pthom/cvnp.git

Link it to your python module:

In your python module CMakeLists, add:

add_subdirectory(path/to/cvnp)
target_link_libraries(your_target PRIVATE cvnp)

In your module, include cvnp:

#include "cvnp/cvnp.h"

(Optional) If you want to import the declared functions in your module:

Write this in your main module code:

void pydef_cvnp(pybind11::module& m);

PYBIND11_MODULE(your_module, m)
{
    ....
    ....
    ....
    pydef_cvnp(m);
}

You will get two simple functions:

cvnp.list_types_synonyms()
cvnp.print_types_synonyms()

>>> import cvnp
>>> import pprint
>>> pprint.pprint(cvnp.list_types_synonyms(), indent=2, width=120)
[ {'cv_depth': 0, 'cv_depth_name': 'CV_8U', 'np_format': 'B', 'np_format_long': 'np.uint8'},
  {'cv_depth': 1, 'cv_depth_name': 'CV_8S', 'np_format': 'b', 'np_format_long': 'np.int8'},
  {'cv_depth': 2, 'cv_depth_name': 'CV_16U', 'np_format': 'H', 'np_format_long': 'np.uint16'},
  {'cv_depth': 3, 'cv_depth_name': 'CV_16S', 'np_format': 'h', 'np_format_long': 'np.int16'},
  {'cv_depth': 4, 'cv_depth_name': 'CV_32S', 'np_format': 'i', 'np_format_long': 'np.int32'},
  {'cv_depth': 5, 'cv_depth_name': 'CV_32F', 'np_format': 'f', 'np_format_long': 'float'},
  {'cv_depth': 6, 'cv_depth_name': 'CV_64F', 'np_format': 'd', 'np_format_long': 'np.float64'}]

Shared and non shared matrices - Demo

Demo based on extracts from the tests:

We are using this struct:

// CvNp_TestHelper is a test helper struct
struct CvNp_TestHelper
{
    // m is a *shared* matrix (i.e `cvnp::Mat_shared`)
    cvnp::Mat_shared m = cvnp::Mat_shared(cv::Mat::eye(cv::Size(4, 3), CV_8UC1));
    void SetM(int row, int col, uchar v) { m.Value.at<uchar>(row, col) = v; }

    // m_ns is a standard OpenCV matrix
    cv::Mat m_ns = cv::Mat::eye(cv::Size(4, 3), CV_8UC1);
    void SetM_ns(int row, int col, uchar v) { m_ns.at<uchar>(row, col) = v; }

    // ...
};

Shared matrices

Changes propagate from Python to C++ and from C++ to Python

def test_mat_shared():
    # CvNp_TestHelper is a test helper object
    o = CvNp_TestHelper()
    # o.m is a *shared* matrix i.e `cvnp::Mat_shared` in the object
    assert o.m.shape == (3, 4)

    # From python, change value in the C++ Mat (o.m) and assert that the changes are visible from python and C++
    o.m[0, 0] = 2
    assert o.m[0, 0] == 2

    # Make a python linked copy of the C++ Mat, named m_linked.
    # Values of m_mlinked and the C++ mat should change together
    m_linked = o.m
    m_linked[1, 1] = 3
    assert o.m[1, 1] == 3

    # Ask C++ to change a value in the matrix, at (0,0)
    # and verify that m_linked as well as o.m are impacted
    o.SetM(0, 0, 10)
    o.SetM(2, 3, 15)
    assert m_linked[0, 0] == 10
    assert m_linked[2, 3] == 15
    assert o.m[0, 0] == 10
    assert o.m[2, 3] == 15

Non shared matrices

Changes propagate from C++ to Python, but not the other way.

def test_mat_not_shared():
    # CvNp_TestHelper is a test helper object
    o = CvNp_TestHelper()
    # o.m_ns is a bare `cv::Mat`. Its memory is *not* shared
    assert o.m_ns.shape == (3, 4)

    # From python, change value in the C++ Mat (o.m) and assert that the changes are *not* applied
    o.m_ns[0, 0] = 2
    assert o.m_ns[0, 0] != 2 # No shared memory!

    # Ask C++ to change a value in the matrix, at (0,0) and verify that the change is visible from python
    o.SetM_ns(2, 3, 15)
    assert o.m_ns[2, 3] == 15

Non contiguous matrices are not supported!

From C++

The conversion of non-contiguous matrices from C++ to python will fail. You need to clone them to make them continuous beforehand.

Example:

    cv::Mat m(cv::Size(10, 10), CV_8UC1);
    cv::Mat sub_matrix = m(cv::Rect(3, 0, 3, m.cols));

    TEST_NAME("Try to convert a non continuous Mat to py::array, ensure it throws");
    TEST_ASSERT_THROW(
        cvnp::mat_to_nparray(sub_matrix, share_memory)
    );

    TEST_NAME("Clone the mat, ensure the clone can now be converted to py::array");
    cv::Mat sub_matrix_clone = sub_matrix.clone();
    py::array a = cvnp::mat_to_nparray(sub_matrix_clone, share_memory);
    TEST_ASSERT(a.shape()[0] == 10);

From python

The conversion of non-contiguous arrays from python to python is also not supported and will fail. Please convert you matrix with np.ascontiguousarray(m) beforehand.

Build and test

These steps are only for development and testing of this package, they are not required in order to use it in a different project.

Build

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

mkdir build
cd build

# if you do not have a global install of OpenCV and pybind11
conan install .. --build=missing
# if you do have a global install of OpenCV, but not pybind11
conan install ../conanfile_pybind_only.txt --build=missing

cmake ..
make

Test

In the build dir, run:

cmake --build . --target test

Deep clean

rm -rf build
rm -rf venv
rm -rf .pytest_cache
rm  *.so 
rm *.pyd

Notes

Thanks to Dan Mašek who gave me some inspiration here: https://stackoverflow.com/questions/60949451/how-to-send-a-cvmat-to-python-over-shared-memory

This code is intended to be integrated into your own pip package. As such, no pip tooling is provided.

virtuald/cvnp