Mailman 3 - hpy-dev - python.org

HPy and WASM
by Paul Prescod Dec. 15, 2020

Dec. 15, 2020

I will not be able to contribute to the HPy project but I think what you are working on is incredibly important. Thank you! I know it is far from your main goal but has anyone considered whether HPy could be used as a glue API for WASM components. My thought is that one could compile a Python extension written in C or Rust to WASM and distribute it on platforms that you do not have access to a C or Rust compiler for. This would get Python much closer to the old dream of "write once, run … [View More]

2 8

Reminder: HPy monthly dev call on Thursday (3 Dec)
by Simon Cross Dec. 2, 2020

Dec. 2, 2020

Hello! The monthly HPy dev call will take place tomorrow -- i.e. Thursday, 3 December at 10am CET (9am UTC). Zoom URL: https://oracle.zoom.us/j/339127822?pwd=YkNZbFgvdXZ4Z3NlcWg1N3BlZ3JTUT09 Meeting ID: 339 127 822 Password: 237729 Everyone is welcome. Yours sincerely, Simon Cross

1 0

Calling Python functions from HPy
by Simon Cross Nov. 21, 2020

Nov. 21, 2020

Hello, I've opened a GitHub issue to discuss the design of HPy's support for calling Python functions -- https://github.com/hpyproject/hpy/issues/122. Feedback & comments would be most welcome. TL;DR: The current proposal is to implement: HPy HPy_Call(HPyContext ctx, HPy callable, HPy args, HPy kwargs) // inspired by PyObject_Call and HPy HPy_CallArray(HPyContext ctx, HPy *args, HPy_ssize_t nargs, const char **kwnames, HPy *kwargs) // inspired by our experience with … [View More]

1 0

Meeting November 16 (Monday) at 10:00 UTC
by Victor Stinner Nov. 18, 2020

Nov. 18, 2020

Hi, I would like to discuss evolutions of the C API in CPython. I propose to meet next monday (November 16) at 10:00 on Zoom: Zoom URL: https://oracle.zoom.us/j/339127822?pwd=YkNZbFgvdXZ4Z3NlcWg1N3BlZ3JTUT09 Meeting ID: 339 127 822 Password: 237729 I'm trying to "bend" the C API towards HPy, but I'm facing different issues. Other CPython core devs basically suggest that I leave the C API as it is: "developers will continue to use the old C API since it's not going to move away anytime soon"… [View More]

3 2

Reminder: HPy monthly dev call on Thursday
by Simon Cross Nov. 4, 2020

Nov. 4, 2020

Hello! The monthly HPy dev call will take place on Thursday, 5 November at 10am CEST. Zoom URL: https://oracle.zoom.us/j/339127822?pwd=YkNZbFgvdXZ4Z3NlcWg1N3BlZ3JTUT09 Meeting ID: 339 127 822 Password: 237729 Everyone is welcome. Yours sincerely, Simon Cross

2 2

Interaction between HPy_CAST and HPy_AsPyObject
by Antonio Cuni Oct. 1, 2020

Oct. 1, 2020

Consider the following snippet of code: typedef struct { HPyObject_HEAD long x; long y; } PointObject; void foo(HPyContext ctx, HPy h_point) { PointObject *p1 = HPy_CAST(ctx, PointObject, h_point); PyObjecy *py_point = HPy_AsPyObject(ctx, h_point); // [1] PointObject *p2 = (PointObject*)py_point; ... } [1] Note that it does not need to be a call to HPy_AsPyObject: it might be a legacy method which takes a PyObject *self, or other similar ways It is obvious that … [View More]HPy_CAST and HPy_AsPyObject need to return the very same address. This is straightforward to implement on CPython, but it poses some challenges on PyPy (and probably GraalPython). Things to consider: 1. currently, in PyPy we allocate the PointObject at a non-movable address, but so far the API does not REQUIRE it. I think it would be reasonable to have an implementation in which objects are movable and HPy_CAST pins the memory until the originating handle is closed. OTOH, the only reasonable semantics is that multiple calls to HPy_AsPyObject returns always the same address. 2. HPyObject_HEAD consists of two words which can be used by the implementation as they like. On CPython, it is obviously mapped to PyObject_HEAD, but in PyPy we (usually) don't need these two extra words, so we allocate sizeof(PointObject)-16 and return a pointer to malloc()-16, which works well since nobody is accessing those two words. I think that GraalPython could use a similar approach. 3. On PyPy, PyObject_HEAD is *three words*, because it also contains ob_pypy_link. But, since the code uses *H*PyObject_HEAD, PointObject will contain only 2 extra words. 4. In the real world usage, there will be "pure hpy types" and "legacy hpy types", which uses legacy methods&co. It would be nice if the pure hpy types do NOT have to pay penalties in case they are never casted to PyObject* With this in mind, how do we implement HPy_AsPyObject on PyPy? One easy way is: 1. we allocate sizeof(PointObject)+8 2. we tweak cpyext to find ob_pypy_link at p-8 3. we teach cpyext how to convert W_HPyObject into PyObject* and vice versa. However, this means that we need to always allocate 24 extra bytes for each object, even if nobody ever calls HPy_AsPyObject on it, which looks bad. Moreover, without changes in the API, the pin/unpin implementation of HPy_CAST becomes de facto impossible. So, my proposal is to distinguish between "legacy hpy types" and "pure hpy types". An HPyType_Spec is legacy if: 1. it uses .legacy_slots = ... OR 2. it ses .legacy = true (i.e., you can explicitly mark a type as legacy even if you no longer have any legacy method/slot. This is useful if you pass it to ANOTHER type which expects to be able to cast the PyObject* into the struct). If a type is "legacy", the snippet shown above works as expected; if it's not legacy, it is still possible to call HPy_AsPyObject on it, but then you are no longer allowed to C-cast if to PointObject* (on pypy, this will mean that you will get a "standard" PyObject* which is a proxy to W_HPyObject). Ideally, in that case it would be nice to catch the invalid cast in the debug mode, but I don't think this is possible... too bad. What do you think? ciao, Anto [View Less]

2 2

PyPy hpy branch workflow
by Antonio Cuni Sept. 21, 2020

Sept. 21, 2020

Hi pypy-dev (cc:ing hpy-dev), I have just merged the hpy branch into py3.6. This means that from now, pypy3.6 nightly builds will automatically support hpy, and it will be much easier for interested users to try it out. I propose the following workflow for continuing HPy development on PyPy: 1. the development will continue on the hpy branch, in "update_vendored" steps: i.e., periodically we run the script to update to a newer hpy version and implement all the new features. As soon as we do … [View More]

1 0

PEP 620 -- Hide implementation details from the C API
by Victor Stinner Aug. 6, 2020

Aug. 6, 2020

Hello, I missed the HPy meeting for the 3rd time in a row :-( (I didn't attend it yet.) While i know that it's different approach than HPy, I wanted to share my PEP: https://www.python.org/dev/peps/pep-0620/ It's my concrete proposal to "enhance" the CAPI but limit the breakages. The meat of the PEP is the process to introduce incompatible changes in the C API. It's based on coordination between CPython core devs and maintainers of "popular" extension modules: https://www.python.org/dev/… [View More]

1 0

HPy dev monthly call
by Florian Angerer Aug. 4, 2020

Aug. 4, 2020

Hello, we are planning to have a monthly HPy dev call on every first Thursday at 10a (CET/CEST). The first call will be on April 2nd, 2020 (10a CEST). The meeting will be held on Zoom: Single click URL to join Zoom: https://oracle.zoom.us/j/339127822 Meeting ID: 339 127 822 If anyone wants to join but cannot make it due to the scheduled time, please let us know (by replying to this mailing list or on the #hpy IRC channel). - Florian

2 3

Design ideas for methods and slots
by Antonio Cuni June 12, 2020

June 12, 2020

Hello, this is a long and dense email which contains some design ideas which are needed to progress towards supporting custom types in HPy. So far, I have opened github issues to discuss API design questions, but usually I don't get much feedback, so I'm trying to write an email to see whether I have better luck 😅. I think that it's very important to make the right decisions in this phase, so I'd like to hear opinions. The following text is written using markdown syntax and contains snippets … [View More]of code, so if you prefer you can read with syntax highlighting at this link: https://gist.github.com/antocuni/fd44ee897cc4cf14e874e2e686c60649 To keep compatibility with CPython, we need to generate a trampoline for each method which uses one of HPy calling conventions, e.g.: ```c HPy_DEF_METH_VARARGS(add_ints) static HPy add_ints_impl(HPyContext ctx, HPy self, HPy *args, HPy_ssize_t nargs); ``` After this, `add_ints` is a function of type `HPyMeth` (which is different in the CPython and the Universal cases), and can be put inside an `HPyMethodDef[]`. Morever, we can put old CPython-style methods by using an appropriate flag. Note the `HPy_METH_*` vs `METH_*` flags, and also note that if you specify the wrong variant, things will happily explode: ```c static PyObject *legacy_meth(PyObject *self, PyObject *args); static HPyMethodDef FooMethods[] = { ... {"add_ints", add_ints, HPy_METH_VARARGS, ""}, {"legacy_meth", legacy_meth, METH_VARARGS, ""}, ... } ``` Right now we have a different `HPy_DEF_METH_*` macro for the few calling conventions we support, but to fully support custom types, we need many more, for example `newfunc`, `allocfunc`, `getattrfunc`, etc. which are C typedefs stored in the various `tp_new`, `tp_alloc`, etc. Before you ask: we need trampolines for all these functions to: 1. "Insert" the HPyContext (CPython will call the trampoline without it, but the `_impl` function the user is going to write will need it) 2. Convert `PyObject *` into `HPy` 3. Convert the result The easiest option is to add more macros and create `HPy_DEF_METH_NEWFUNC` & co., but I wonder whether we can do better. To start with, I would like to have only ONE macro. This can be done with a preprocessor trick: ```c #define _HPy_DEF_METH_O(name) ... #define _HPy_DEF_METH_VARARGS(name) ... #define HPy_DEF_METH(kind, name) _HPy_DEF_##kind(name) HPy_DEF_METH(METH_O, foo) HPy_DEF_METH(METH_VARARGS, bar) ``` I think this is already an improvement over the existing situation. The quesion is, what is the best syntax to use? Some random proposals: * `HPy_DEF_METH(METH_O, ...`: I don't like the repetion of METH; moreover, `METH_O` is the *CPython* macro to indicate a legacy function, so you define your meth using `METH_O` but then you need to use `HPy_METH_O` in the method def table. * `HPy_DEF_METH(HPy_METH_O, ...)`: this is a bit verbose, but it's more explicit. * `HPy_DEF(HPy_METH_O, ...)`: is it better? * `HPy_DEFINE(...)` * `HPy_DEF_METH(O, ...)` * `HPyMeth_DEF(...)`: I think I like this better than the current `HPy_DEF_METH`. Keep in mind few facts: * We need to offer also the corresponding `HPy_DECL_METH`, which is needed if you want to put your methods in multiple *.c files. So any syntax we choose, must have a "nice" corresponding syntax for the declaration. * We need to use this syntax also for the various `newfunc`, `allocfunc`, etc. This is alredy a bit weird because `METH_{NOARGS,O,VARARGS}` are flags, while `newfunc` &co. are typedefs; so, for example: - `HPyMeth_DEF(METH_O, ...)` and `HPyMeth_DEF(newfunc, ...)` ? The following proposal tries to solve/improve these issues, and goes a bit deeper than just renaming the macros. To start with, we should have a clear separation between the `PyMethodDef[]` (for legacy functions) and `HPyMethodDef[]`, instead of relying on a flag which can be easily mistaken. For example: ```c static PyMethodDef LegacyMethods[] = { {"legacy1", legacy1, METH_NOARGS, ""}, {"legacy2", legacy2, METH_O, ""}, {NULL, NULL, 0, NULL} }; static HPyMethodDef CoolMethods[] = { {"foo", foo, METH_NOARGS, ""}, {"bar", bar, METH_O, ""}, {NULL, NULL, 0, NULL} }; static HPyModuleDef moduledef = { HPyModuleDef_HEAD_INIT, ... .m_legacy_methods = LegacyMethods, .m_methods = CoolMethods, }; ``` Among the other things, this has advantage of removing the distintion between `METH_O` and `HPy_METH_O`. The original idea of keeping `.m_methods = ...` was to make it as similar as possible to the existing syntax, to make porting easier without having to modify tons of place. However, my experience with ultrajson tells me that what you do in practice is to port methods one by one, so while you are at it moving the definition from one table to the other is not a big deal (plus, if we design our types correctly you would get a nice compilation error if you forget to move the method to the right table). The main disadvantage of this is that `HPyModule_Create` can no longer be an alias for `PyModule_Create` in the CPython ABI mode, but it would need to be a custom function which creates the "real" `PyModuleDef`. But note that we need to implement it anyway for the cpython-universal case, and it's exactly the same sitation we already have for `HPyType_FromSpec`. Once we decide to use our own `HPyModuleDef`, we can become creative, which brings more API design questions. For example, we could use unions to have a more type-safe way of specifying methods: ```c typedef HPy (*HPyMeth_NOARGS)(HPyContext, HPy); typedef HPy (*HPyMeth_O)(HPyContext, HPy, HPy); typedef struct { const char *name; union { void *pfunc; HPyMeth_NOARGS meth_noargs; HPyMeth_O meth_o; }; int flags; } HPyMethodDef; ... static HPyMethodDef Methods[] = { {"foo", .meth_noargs = foo, METH_NOARGS}, {"bar", .meth_o = bar, METH_O}, {NULL, NULL, 0} }; ``` And the same for `HPyType_Spec`/`HPyType_Slot`: ```c typedef HPy (*HPy_newfunc)(HPyContext, HPy, HPy, HPy); typedef HPy (*HPy_allocfunc)(HPyContext, HPy, HPy_ssize_t); typedef struct { int slot; union { void *p; HPy_newfunc tp_new; HPy_allocfunc tp_alloc; // ... }; } HPyType_Slot; static HPyType_Slot slots[] = { {Py_tp_new, .tp_new = Foo_New}, {Py_tp_alloc, .tp_alloc = Foo_Alloc}, {0, NULL}, }; ``` * The biggest advantage is that if you use a function with the wrong signature, you get a nice warning. * The biggest disadvantage is that setting the correct field of the union is obviously not enough: you still need to manually specify the correct flags, but at least it should be "visually evident" if something is wrong (e.g. if you specify `{... .meth_o = foo, METH_NOARGS}`). I don't think there is any way to get a nice compiler error/warning in this case: if you have any idea, please shout :) If we go down this route, I suggest the following syntax for `HPyMeth_DEFINE`. The nice property is that there is a direct correspondence between the first argument of the macro and the `typedef` of the function pointer: ```c typedef HPy (*HPyMeth_NOARGS)(HPyContext, HPy); typedef HPy (*HPyMeth_O)(HPyContext, HPy, HPy); typedef HPy (*HPy_newfunc)(HPyContext, HPy, HPy, HPy); typedef HPy (*HPy_allocfunc)(HPyContext, HPy, HPy_ssize_t); // ... HPyMeth_DEFINE(O, foo) // foo_impl is of type HPyMeth_O HPyMeth_DEFINE(newfunc, Foo_New) // Foo_New_impl is of type HPyMeth_newfunc ``` Another nice property of this scheme is that we can extend it nicely if in the future we want to go towards an "argument-clinic-like" setup. E.g.: ```c // L_LDO means "returns a long and takes a long, a double and an object" typedef long (*HPyMeth_L_LDO)(long, double, HPy); HPyMeth_DEFINE(L_LDO, foo) ... ``` But I think that this email is already too long and too dense, so let's not discuss about "argument-clinic-like" now :) Please let me know what you think! ciao, Anto [View Less]

3 11