Mailman 3 December 2020 - hpy-dev

Concrete proposal: Reusable API Glue code
by Paul Prescod Dec. 29, 2020

Dec. 29, 2020

Let's take a couple of example files and see what it would take to make them reusable and move them into the HPy repository for ANY interpreter host to build upon. https://foss.heptapod.net/pypy/pypy/-/blob/branch/hpy/pypy/module/_hpy_univ… According to my IDE, most of the imports in e.g. interp_list.py are not used. Imagine that we replaced all of them with: from interpreter_context.apiset import API, handles, pytypes W_ListObject would be imported from pytypes. As an interpreter host vendor, I'd just implement space, API, handles, pytypes and I could reuse interp_list as-is. The PyPy implementation of the "interpreter" package would just rename the modules from pypy.module. _hpy_universal According to my IDE, these SAME imports would make interp_number and interp_tuple "just work" as well. To implement interp_long, I need to inject a new dependency, called "typesystem" which is a generalization of rffi and lltype. interp_builder and interp_tracker need only a "Root" object injected. A couple of the files need Unicode stuff injected. There is an implicit dependency on cparser which would need to be dealt with one way or another...using a C Parser seems a bit like overkill given a) Python has type annotations and b) all of the declarations are machine-generated anyhow. But if CParser is a hard requirement then I guess it would become a dependency of the HPyHostAPI package. My last thought is that perhaps the CPython Universal implementation could be written in Python instead of C. Of course it would run slower, but development of it would tend to maintain feature parity with PyPy "automatically". Or you could do it in Python for the next year or two and then go back and optimize it back to C once HPy is finalized. - Paul

2 1

Calling functions
by Paul Prescod Dec. 29, 2020

Dec. 29, 2020

I know that the "object/method calling API" is an open question but in the meantime is there any reliable (but maybe ugly) way for me to call an HPyDef_METH in a "Universal" (non-CPython) context? I'm looking for something like this: extern int PyWasm_Call_PyMethod(void *meth_handle, void *args, void *kwargs){ HPyMeth *method = (HPyMeth *)meth_handle; PyWasmDebug("Calling func %s with %p and %p ", method->name, args, kwargs); HPy rc = method->impl(args, kwargs) } It's unclear to me whether the thing in HPyMeth->impl has a uniform calling convention or the heterogeneous one exposed by the objects themselves. For example: static HPy do_nothing_impl(HPyContext ctx, HPy self) Versus static HPy add_ints_impl(HPyContext ctx, HPy self, HPy *args, HPy_ssize_t nargs) Do I have to look at the signature and call each one with the "right" calling convention? Or is there a particular CPython library that I could pull in to do the calling without pulling in the whole interpreter (which would defeat the purpose of HPy!)? Paul

3 10

Ctx object
by Paul Prescod Dec. 28, 2020

Dec. 28, 2020

The docs say: The `ctx` is an opaque "context" argument that stands for the current interpreter. But "pof.c" indirects into the context object: https://github.com/hpyproject/hpy/blob/8e20b89116c2993188157c09a6070a64f8ef… That doesn't seem very opaque to me. - Paul

3 2

WASM <--> HPy
by Paul Prescod Dec. 28, 2020

Dec. 28, 2020

For anyone who is interested, a very rough implementation of an extension ABI based on HPy and WASM is here: https://github.com/hpyproject/hpy/compare/master...prescod:feature/wasm-pro… This means that on any platform (operating system or CPU/ISA) where wasmer works, a single binary extension library can run without porting. Status and limitations: There are only about 50 lines of C code other than the generated stubs/trampolines. Most of it is written with ctypes/Python. Currently it supports exactly enough APIs to run "pof.c" and no more. Other modules will probably generate "missing function" errors which are usually easy to fix with Python code. It leaks memory like crazy. Handle cleanup is nearly non-existent. Currently, it is tied specifically to Wasmer-Python and Wasmer-Python binaries only exist for X86, but in theory it could also be integrated with pywasm which has only a small dependency on NumPy and no other C dependencies. Which would mean that these binaries would run (slowly!) even on Python platforms with no native support for WASM. For example, many modules don't have Mac ARM builds yet, but with WASM they would have a slow build available on "day 1", and a faster one when Python Wasmer is ported and then faster again when native builds are ported. I'd like to make the abstraction layer that allows it target the "fastest" WASM engine available and fallback to slower ones as necessary. I don't know if I will have time to take the prototype to production, but I'd be more motivated if anyone else pops up who is interested in this topic and would like to work together. Otherwise I may just document it as a prototype and put it on a shelf until someone interested in it pops up. - Paul

1 1

HPy and WASM
by Paul Prescod Dec. 15, 2020

Dec. 15, 2020

I will not be able to contribute to the HPy project but I think what you are working on is incredibly important. Thank you! I know it is far from your main goal but has anyone considered whether HPy could be used as a glue API for WASM components. My thought is that one could compile a Python extension written in C or Rust to WASM and distribute it on platforms that you do not have access to a C or Rust compiler for. This would get Python much closer to the old dream of "write once, run anywhere". - Paul Prescod

2 8

Reminder: HPy monthly dev call on Thursday (3 Dec)
by Simon Cross Dec. 2, 2020

Dec. 2, 2020

Hello! The monthly HPy dev call will take place tomorrow -- i.e. Thursday, 3 December at 10am CET (9am UTC). Zoom URL: https://oracle.zoom.us/j/339127822?pwd=YkNZbFgvdXZ4Z3NlcWg1N3BlZ3JTUT09 Meeting ID: 339 127 822 Password: 237729 Everyone is welcome. Yours sincerely, Simon Cross

1 0