Allow calling external cythonized functions from pythranized functions
ashwinvis opened this issue · 16 comments
Maybe I am really naive, and this is not the way it should be done. In some sense this is related to #461.
Background
Here we show some attempts to implement a function which performs ($$\vec{u}. \nabla \vec{u_fft}$$) through a pseudospectral approach. u is a 3D vector in x, y, z and u_fft is its spectral counterpart in k_x, k_y, k_z. The gradient operator $$\nabla \vec{u}$$) is calculated as $$ ifft(1j . \vec{k} (u_fft)) $$. Where ifft means an inverse FFT operation and wavenumber vector k = (k_x, k_y,k_z)
Problem
Enough math. Let's talk code now. Take a look at how we tried to implement this (note: ifft3d is a cythonized function)
- Pythran way: https://github.com/fluiddyn/fluidfft/blob/master/fluidfft/fft3d/dream_pythran.py
- Pythran + Cython way: https://github.com/fluiddyn/fluidfft/blob/master/fluidfft/fft3d/dream_cythran.pyx
We would like to implement the rest of the operations using Pythran. We know the exact input and return types of ifft3d. However, we have upto 5 different implementations of ifft3d depending on which FFT backend is specified / available.
I have tried my best to describe our attempts. If needed I can try to create a simpler test case.
Is it possible for pythran to access a cythonized function?
MMMh sounds very interesting. It is currently not possible but that's in my plan. We need to do two things there:
- allow pythran to accept c function pointer as parameter, e.g. though a
PyCapsule - Put a cython generated function into a capsule
Note that since 357d4d8 it's already possible to put a pythran function in a capsule.
I think @paugier is liklely to be interested in that feature!
Note about the pythran annotation: what should be the pereferd way to describe a capsule containing a function pointer from (int, float) to int?
int(int, float) # cstyle
int -> float -> int # ml style
(int, float) -> int # ml + python style ?For the cython part, a lookup into __pyx_capi__ seems to be enough, great!
Of course, @paugier and me are working together on this. I did read your "Capsule corporation" blog post. Just to be clear, you plan to pass a cython generated function as an argument though a PyCapsule and that needs to be implemented, right?
EDIT: Should the PyCapsule part be done externally, or will pythran takes care of this?
On the pythran annotation question: (personally) I would prefer either the "cstyle" (cython users might like that) or the "ml + python" style. "ml" style looks too verbose
ok, I'll give a try to the cstyle one, that clearly indicates that the fnction must come from C and is not some kind of Python function.
see #746 for a not so tested implementation. Turns out it was relatively simple to implement, thanks to pythran magical typing system :-)
cc @paugier btw
Yay for the implementation. An example / test would be nice. I tried to use it as follows.
- Pythranized a file called
pythran/tests/cython/add2.py
# pythran export add2(
# float[][], float[][], float[][](float[][], float[][]))
def add2(f, g, add):
return add(f, g)- Built the cython extension
pythran/tests/cython/add.pyusingsetup_add.py - Test!
from add import add
from add2 import add2
import numpy as np
a = np.ones([3, 3])
b = a * 3
ans = add(a, b)
ans2 = add2(a, b, add)Obviously I am doing it wrong:
TypeError: Invalid argument type for pythranized function `add2'.
Candidates are:
add2(ndarray<double,2>,ndarray<double,2>,cfun<ndarray<double,2>(ndarray<double,2>, ndarray<double,2>)>)
add2(ndarray<double,2>,numpy_texpr<ndarray<double,2>>,cfun<ndarray<double,2>(ndarray<double,2>, ndarray<double,2>)>)
add2(numpy_texpr<ndarray<double,2>>,ndarray<double,2>,cfun<ndarray<double,2>(ndarray<double,2>, ndarray<double,2>)>)
add2(numpy_texpr<ndarray<double,2>>,numpy_texpr<ndarray<double,2>>,cfun<ndarray<double,2>(ndarray<double,2>, ndarray<double,2>)>)It only accepts capsule as first parameter, so you should#pythran export capsule add2
So now I changed add2.py into
# pythran export capsule add1(float[][], float[][])
def add1(f, g):
return f + g
# pythran export add2(
# float[][], float[][], float[][](float[][], float[][]))
def add2(f, g, add_func):
return add_func(f, g)In [1]: from add2 import add1, add2
In [2]: import numpy as np
In [3]: a = np.ones([3, 3])
In [4]: b = a * 3
In [5]: add2(a, b, add1)
Out[5]:
array([[ 4., 4., 4.],
[ 4., 4., 4.],
[ 4., 4., 4.]])If only add_func could be a Cython function automagically translated into a PyCapsule! We won't need (EDIT: excuse my ignorance!)scipy.LowLevelCallable.
@ashwinvis works out of the box, check the following:
cython file:
cdef api add(int x, int y):
cdef int r;
r = x + y;
return ripython sessions:
>>> import add
>>> add.__pyx_capi__['add']
<capsule object "PyObject *(int, int)" at 0x7f27fe3587e0>
It runs but is buggy. Can you have a look at https://github.com/ashwinvis/cython_capi/tree/master/using_api_keyword?
Clone and simply run inside the directory:
make build
make test
This is what I get by adding 1 and 2
======================================================================
FAIL: test_int (__main__.TestAll)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test.py", line 14, in test_int
np.testing.assert_equal(ans, ans2)
File "/scratch/avmo/opt/pythran/lib/python2.7/site-packages/numpy/testing/utils.py", line 416, in assert_equal
raise AssertionError(msg)
AssertionError:
Items are not equal:
ACTUAL: 3
DESIRED: 94688066438504
There is some hope though. It could be possible that Cython's api keyword could be buggy. See this simple case (which works!): https://github.com/ashwinvis/cython_capi/tree/master/using_pxd
Yeah, you need to make sure the function registered in the capsule actually takes the expected parameters. Feel free to open a bug on cython :-)
@ashwinvis what you wanted to do was actually:
cdef api int add(int x, int y):
cdef int r;
r = x + y;
return rforcing the return type gives you the right signature (here int (int, int))
Pythran part implemented in #746