ABI for unboxed calls from JITs (and cython, etc.)

[steve-s]

Issue #129 is about API design for this feature, but the user-facing part can be tackled separately. HPy should provide ABI to Python interpreters that would give them:

• pointer to "optimized" function that accepts (some) unboxed arguments
• description of the signature of that function

Extension functions are described by HPyMeth C struct. We can just add necessary fields to it. From the API perspective, for the time being, we can let the users to define both entry points manually:.

Current API from proof-of-concept example:

HPyDef_METH(add_ints, "add_ints", HPyFunc_VARARGS)
static HPy add_ints_impl(HPyContext *ctx, HPy self, const HPy *args, size_t nargs)
{
    long a, b;
    if (!HPyArg_Parse(ctx, NULL, args, nargs, "ll", &a, &b))
        return HPy_NULL;
    return HPyLong_FromLong(ctx, a+b);
}

Could become:

HPyDef_METH(add_ints, "add_ints", HPyFunc_VARARGS, {Sig_int64, Sig_int64, Sig_int64})
static HPy add_ints_impl(HPyContext *ctx, HPy self, const HPy *args, size_t nargs)
{
    long a, b;
    if (!HPyArg_Parse(ctx, NULL, args, nargs, "ll", &a, &b))
        return HPy_NULL;
    return HPyLong_FromLong(ctx, add_ints_impl_unboxed(a, b));
}
static int64_t add_ints_impl_unboxed(HPyContext *ctx, HPy self, int64_t a, int64_t b) {
    return a+b;
}

where the macro HPyDef_METH would generate something along these lines:

HPyDef add_ints = {
    .kind=HPyDef_Kind_Meth, 
    .meth={
        .name="add_ints", 
        .impl=(HPyCFunction) add_ints_impl, 
        .unboxed_sig = {Sig_int64, Sig_int64, Sig_int64},
        .unboxed_impl = (HPyCFunction) add_ints_impl_unboxed,
        // ...

Python interpreters that implement HPy can choose to make calls either via .impl with boxed arguments, or via .unboxed_impl. Current, HPy implementation for CPython simply generates trampolines with CPython signature (*) that convert the PyObject* arguments to HPy, fish out HPyContext from a C global variable, and call .impl. Those trampolines are then used to set up the CPython PyModuleDef when loading HPy universal module on CPython. Once there is a support for "optimized" extension functions on CPython, this code in HPy can be updated to translate the HPy ABI to the CPython way of doing this. Current GraalPy implementation should be amenable to this design and I believe that PyPy's too.

(*) This happens as a part of the HPyDef_METH macro and there is a field for such trampoline in HPyMeth, so CPython backend is hard-coded into HPy a little bit, but that is necessary if we want to support CPython "externally" without backing HPy support into CPython core.