Let's take a couple of example files and see what it would take to make
them reusable and move them into the HPy repository for ANY
interpreter host to build upon.
https://foss.heptapod.net/pypy/pypy/-/blob/branch/hpy/pypy/module/_hpy_univ…
According to my IDE, most of the imports in e.g. interp_list.py are not
used.
Imagine that we replaced all of them with:
from interpreter_context.apiset import API, handles, pytypes
W_ListObject would be imported from pytypes.
As an interpreter host …
[View More]vendor, I'd just implement space, API, handles,
pytypes and I could reuse interp_list as-is. The PyPy implementation of the
"interpreter" package would just rename the modules from pypy.module.
_hpy_universal
According to my IDE, these SAME imports would make interp_number and
interp_tuple "just work" as well.
To implement interp_long, I need to inject a new dependency, called
"typesystem" which is a generalization of rffi and lltype.
interp_builder and interp_tracker need only a "Root" object injected.
A couple of the files need Unicode stuff injected.
There is an implicit dependency on cparser which would need to be dealt
with one way or another...using a C Parser seems a bit like overkill given
a) Python has type annotations and b) all of the declarations are
machine-generated anyhow. But if CParser is a hard requirement then I guess
it would become a dependency of the HPyHostAPI package.
My last thought is that perhaps the CPython Universal implementation could
be written in Python instead of C. Of course it would run slower, but
development of it would tend to maintain feature parity with PyPy
"automatically".
Or you could do it in Python for the next year or two and then go back and
optimize it back to C once HPy is finalized.
- Paul
[View Less]
I know that the "object/method calling API" is an open question but in the
meantime is there any reliable (but maybe ugly) way for me to call an
HPyDef_METH in a "Universal" (non-CPython) context?
I'm looking for something like this:
extern int PyWasm_Call_PyMethod(void *meth_handle, void *args, void
*kwargs){
HPyMeth *method = (HPyMeth *)meth_handle;
PyWasmDebug("Calling func %s with %p and %p ", method->name, args,
kwargs);
HPy rc = method->impl(args, kwargs)
}
It's …
[View More]unclear to me whether the thing in HPyMeth->impl has a uniform calling
convention or the heterogeneous one exposed by the objects themselves.
For example:
static HPy do_nothing_impl(HPyContext ctx, HPy self)
Versus
static HPy add_ints_impl(HPyContext ctx, HPy self, HPy *args, HPy_ssize_t
nargs)
Do I have to look at the signature and call each one with the "right"
calling convention?
Or is there a particular CPython library that I could pull in to do the
calling without pulling in the whole interpreter (which would defeat the
purpose of HPy!)?
Paul
[View Less]
For anyone who is interested, a very rough implementation of an extension
ABI based on HPy and WASM is here:
https://github.com/hpyproject/hpy/compare/master...prescod:feature/wasm-pro…
This means that on any platform (operating system or CPU/ISA) where wasmer
works, a single binary extension library can run without porting.
Status and limitations:
There are only about 50 lines of C code other than the generated
stubs/trampolines. Most of it is written with ctypes/Python.
Currently it …
[View More]supports exactly enough APIs to run "pof.c" and no more. Other
modules will probably generate "missing function" errors which are usually
easy to fix with Python code.
It leaks memory like crazy. Handle cleanup is nearly non-existent.
Currently, it is tied specifically to Wasmer-Python and Wasmer-Python
binaries only exist for X86, but in theory it could also be integrated with
pywasm which has only a small dependency on NumPy and no other C
dependencies. Which would mean that these binaries would run (slowly!) even
on Python platforms with no native support for WASM.
For example, many modules don't have Mac ARM builds yet, but with WASM they
would have a slow build available on "day 1", and a faster one when Python
Wasmer is ported and then faster again when native builds are ported.
I'd like to make the abstraction layer that allows it target the "fastest"
WASM engine available and fallback to slower ones as necessary.
I don't know if I will have time to take the prototype to production, but
I'd be more motivated if anyone else pops up who is interested in this
topic and would like to work together. Otherwise I may just document it as
a prototype and put it on a shelf until someone interested in it pops up.
- Paul
[View Less]
I will not be able to contribute to the HPy project but I think what you
are working on is incredibly important. Thank you!
I know it is far from your main goal but has anyone considered whether HPy
could be used as a glue API for WASM components. My thought is that one
could compile a Python extension written in C or Rust to WASM and
distribute it on platforms that you do not have access to a C or Rust
compiler for. This would get Python much closer to the old dream of "write
once, run …
[View More]anywhere".
- Paul Prescod
[View Less]