Or to put it another way, one cannot generally pass a pointer into a Web Assembly sandbox and have that module manipulate it. One must use handles. Thus "Handle Python API" seemed like a good match.

On Thu, Dec 3, 2020 at 10:15 AM Paul Prescod <paul@prescod.net> wrote:

My understanding is that the HPy project is designed to handle situations where extensions do not have access to the internals of CPython. I think that C code running in a WASM sandbox would fit that pattern.

In fact, C or Rust code running in a WASM sandbox probably wouldn't really know or care whether it was running in CPython, PyPy, Jython, Brython, Skulpt or any other embedding context.

On Thu, Dec 3, 2020 at 12:09 AM Antonio Cuni <anto.cuni@gmail.com> wrote:

...

Hello Paul,

On Thu, Dec 3, 2020 at 1:52 AM Paul Prescod <paul@prescod.net> wrote:

...

I will not be able to contribute to the HPy project but I think what you are working on is incredibly important. Thank you!

thank you :)

I know it is far from your main goal but has anyone considered whether

...

HPy could be used as a glue API for WASM components. My thought is that one could compile a Python extension written in C or Rust to WASM and distribute it on platforms that you do not have access to a C or Rust compiler for. This would get Python much closer to the old dream of "write once, run anywhere".

this sounds like a nice idea generally speaking, but I'm not sure why it is related to HPy. Why couldn't you do the same with the existing C API?

Thanks for responding.

I think it is safest to think about the WASM runtime as "almost" a different process space that you communicate with through "almost" RPC. "Handles" are just integers which the code inside the runtime converts to pointers however it sees fit (cast, lookup table, ...).

In fact you can implement a (slow!) WASM interpreter entirely *in* python without use of ctypes or anything like that:

https://www.reddit.com/r/Python/comments/dxv1of/keynote_david_beazley_this_t...

So yes, if the host interpreter and the extension environment need to share pointers (i.e. both of them need to treat the value as a C pointer) then it probably won't work without some real hackery (e.g. copying objects back and forth to "pretend" you are sharing memory).

I suppose a pure-handles, no-pointers model does not meet your performance goals. One reason the pure-handles model is attractive in many contexts is that it means that an extension cannot crash the Python interpreter, which opens up a universe where extensions do not have to be trusted code. But if the performance cost is too high then I guess that proposal won't go anywhere.

On Mon, Dec 14, 2020 at 2:37 AM Antonio Cuni <anto.cuni@gmail.com> wrote:

Hi Paul, sorry, I forgot to answer this email :(

On Thu, Dec 3, 2020 at 10:08 PM Paul Prescod <paul@prescod.net> wrote:

...

Or to put it another way, one cannot generally pass a pointer into a Web Assembly sandbox and have that module manipulate it. One must use handles. Thus "Handle Python API" seemed like a good match.

I think you are right. However, note that there are cases in which HPy manipulates pointers directly: e.g. HPy_CAST allows you convert a handle into get a pointer to a C struct, which is needed to implement custom types (because your instances are C structs, so you need to get a pointer in order to manipulate). See this issue for a proposal on how it should look like: https://github.com/hpyproject/hpy/issues/83

I don't know much about WASM so I don't know whether this is a hard blocker or if it could be implemented.

ciao, Anto

Antonio: that's good news and yes, it sounds like it would work. Of course I anticipate a shim running inside of the WASM runtime so it can do the allocations based on requests from the parent process.

Do you have any proposal/suggestion for how we could fix/improve the HPy API to allow such a scenario?

If I'm right, with WASM a *truly universal* binary may be possible. Like a Binary that will run on ARM, Intel, Risc-V, CPython, PyPy, in the browser, everywhere!

Maybe it runs faster on Intel and ARM, but it could run everywhere. If David Beazley's work were taken to production it could EVEN run in contexts where the only thing we know is that a Python interpreter exists.

I definitely have too much going on in my life to join your project (as important as I see it as being) but if I could get this concept to prototype/demo, do you think that your team might (just might, not will) adopt it as a real project goal? Personally I think that it might be the kind of thing that would get some really cool attention on HPy and make it "worth it" for package maintainers to accept the pain of migration.

Imagine being able to import NumPy in literally any Python interpreter with enough RAM, even Python interpreters written in Javascript or Python or ToyLang.

I'll have a week off around Christmas where I could work on a prototype of the shim, instead of Advent of Code which was my other plan...

On Mon, Dec 14, 2020 at 9:07 AM Antonio Cuni <anto.cuni@gmail.com> wrote:

On Mon, Dec 14, 2020 at 3:52 PM Paul Prescod <paul@prescod.net> wrote:

...

So yes, if the host interpreter and the extension environment need to share pointers (i.e. both of them need to treat the value as a C pointer) then it probably won't work without some real hackery (e.g. copying objects back and forth to "pretend" you are sharing memory).

I think it is possible to do without sharing pointers between the host interpreters and the extension. Currently the API looks like this:

typedef struct { HPyObject_HEAD double x; double y; } PointObject; static HPyType_Spec point_type_spec = { .basicsize = sizeof(PointObject), .... } HPyDef_METH(Point_bar, "bar", Point_bar_impl, HPyFunc_NOARGS) static HPy Point_bar_impl(HPyContext ctx, HPy self) { PointObject *point = HPy_CAST(ctx, PointObject, self); return HPyLong_FromLong(ctx, point->x + point->y); }

If I understand correctly, in your proposal this C code will be compiled into WASM, so manipulating the PointObject* pointers should be fine and allowed. The host interpreter doesn't know anything about the struct PointObject, and it will never manipulate its content: all it does is to do the equivalent of "malloc(point_type->basicsize)" when it creates the instance, and have a way to return this pointer when you do HPy_CAST.

So, for this to work there must be a way for the host interpreter to allocate memory inside the extension's memory space. Do you think that such a thing has any chance to work in WASM?

I suppose a pure-handles, no-pointers model does not meet your

...

performance goals. One reason the pure-handles model is attractive in many contexts is that it means that an extension cannot crash the Python interpreter, which opens up a universe where extensions do not have to be trusted code. But if the performance cost is too high then I guess that proposal won't go anywhere.

Do you have any proposal/suggestion for how we could fix/improve the HPy API to allow such a scenario?

...

Or to put it another way, one cannot generally pass a pointer into a Web Assembly sandbox and have that module manipulate it. One must use handles. Thus "Handle Python API" seemed like a good match.

On Thu, Dec 3, 2020 at 10:15 AM Paul Prescod <paul@prescod.net> wrote:

My understanding is that the HPy project is designed to handle situations where extensions do not have access to the internals of CPython. I think that C code running in a WASM sandbox would fit that pattern.

In fact, C or Rust code running in a WASM sandbox probably wouldn't really know or care whether it was running in CPython, PyPy, Jython, Brython, Skulpt or any other embedding context.

On Thu, Dec 3, 2020 at 12:09 AM Antonio Cuni <anto.cuni@gmail.com> wrote:

...

Hello Paul,

On Thu, Dec 3, 2020 at 1:52 AM Paul Prescod <paul@prescod.net> wrote:

...

I will not be able to contribute to the HPy project but I think what you are working on is incredibly important. Thank you!

thank you :)

I know it is far from your main goal but has anyone considered whether

...

HPy could be used as a glue API for WASM components. My thought is that one could compile a Python extension written in C or Rust to WASM and distribute it on platforms that you do not have access to a C or Rust compiler for. This would get Python much closer to the old dream of "write once, run anywhere".

this sounds like a nice idea generally speaking, but I'm not sure why it is related to HPy. Why couldn't you do the same with the existing C API?

Thanks for responding.

I think it is safest to think about the WASM runtime as "almost" a different process space that you communicate with through "almost" RPC. "Handles" are just integers which the code inside the runtime converts to pointers however it sees fit (cast, lookup table, ...).

In fact you can implement a (slow!) WASM interpreter entirely *in* python without use of ctypes or anything like that:

https://www.reddit.com/r/Python/comments/dxv1of/keynote_david_beazley_this_t...

So yes, if the host interpreter and the extension environment need to share pointers (i.e. both of them need to treat the value as a C pointer) then it probably won't work without some real hackery (e.g. copying objects back and forth to "pretend" you are sharing memory).

I suppose a pure-handles, no-pointers model does not meet your performance goals. One reason the pure-handles model is attractive in many contexts is that it means that an extension cannot crash the Python interpreter, which opens up a universe where extensions do not have to be trusted code. But if the performance cost is too high then I guess that proposal won't go anywhere.

On Mon, Dec 14, 2020 at 2:37 AM Antonio Cuni <anto.cuni@gmail.com> wrote:

Hi Paul, sorry, I forgot to answer this email :(

On Thu, Dec 3, 2020 at 10:08 PM Paul Prescod <paul@prescod.net> wrote:

...

Or to put it another way, one cannot generally pass a pointer into a Web Assembly sandbox and have that module manipulate it. One must use handles. Thus "Handle Python API" seemed like a good match.

I think you are right. However, note that there are cases in which HPy manipulates pointers directly: e.g. HPy_CAST allows you convert a handle into get a pointer to a C struct, which is needed to implement custom types (because your instances are C structs, so you need to get a pointer in order to manipulate). See this issue for a proposal on how it should look like: https://github.com/hpyproject/hpy/issues/83

I don't know much about WASM so I don't know whether this is a hard blocker or if it could be implemented.

ciao, Anto

Antonio: that's good news and yes, it sounds like it would work. Of course I anticipate a shim running inside of the WASM runtime so it can do the allocations based on requests from the parent process.

Do you have any proposal/suggestion for how we could fix/improve the HPy API to allow such a scenario?

If I'm right, with WASM a *truly universal* binary may be possible. Like a Binary that will run on ARM, Intel, Risc-V, CPython, PyPy, in the browser, everywhere!

Maybe it runs faster on Intel and ARM, but it could run everywhere. If David Beazley's work were taken to production it could EVEN run in contexts where the only thing we know is that a Python interpreter exists.

I definitely have too much going on in my life to join your project (as important as I see it as being) but if I could get this concept to prototype/demo, do you think that your team might (just might, not will) adopt it as a real project goal? Personally I think that it might be the kind of thing that would get some really cool attention on HPy and make it "worth it" for package maintainers to accept the pain of migration.

Imagine being able to import NumPy in literally any Python interpreter with enough RAM, even Python interpreters written in Javascript or Python or ToyLang.

I'll have a week off around Christmas where I could work on a prototype of the shim, instead of Advent of Code which was my other plan...

On Mon, Dec 14, 2020 at 9:07 AM Antonio Cuni <anto.cuni@gmail.com> wrote:

On Mon, Dec 14, 2020 at 3:52 PM Paul Prescod <paul@prescod.net> wrote:

...

So yes, if the host interpreter and the extension environment need to share pointers (i.e. both of them need to treat the value as a C pointer) then it probably won't work without some real hackery (e.g. copying objects back and forth to "pretend" you are sharing memory).

I think it is possible to do without sharing pointers between the host interpreters and the extension. Currently the API looks like this:

typedef struct { HPyObject_HEAD double x; double y; } PointObject; static HPyType_Spec point_type_spec = { .basicsize = sizeof(PointObject), .... } HPyDef_METH(Point_bar, "bar", Point_bar_impl, HPyFunc_NOARGS) static HPy Point_bar_impl(HPyContext ctx, HPy self) { PointObject *point = HPy_CAST(ctx, PointObject, self); return HPyLong_FromLong(ctx, point->x + point->y); }

If I understand correctly, in your proposal this C code will be compiled into WASM, so manipulating the PointObject* pointers should be fine and allowed. The host interpreter doesn't know anything about the struct PointObject, and it will never manipulate its content: all it does is to do the equivalent of "malloc(point_type->basicsize)" when it creates the instance, and have a way to return this pointer when you do HPy_CAST.

So, for this to work there must be a way for the host interpreter to allocate memory inside the extension's memory space. Do you think that such a thing has any chance to work in WASM?

I suppose a pure-handles, no-pointers model does not meet your

...

performance goals. One reason the pure-handles model is attractive in many contexts is that it means that an extension cannot crash the Python interpreter, which opens up a universe where extensions do not have to be trusted code. But if the performance cost is too high then I guess that proposal won't go anywhere.

Do you have any proposal/suggestion for how we could fix/improve the HPy API to allow such a scenario?

...