I will not be able to contribute to the HPy project but I think what you are working on is incredibly important. Thank you!
I know it is far from your main goal but has anyone considered whether HPy could be used as a glue API for WASM components. My thought is that one could compile a Python extension written in C or Rust to WASM and distribute it on platforms that you do not have access to a C or Rust compiler for. This would get Python much closer to the old dream of "write once, run anywhere".
- Paul Prescod
Hello Paul,
On Thu, Dec 3, 2020 at 1:52 AM Paul Prescod paul@prescod.net wrote:
I will not be able to contribute to the HPy project but I think what you are working on is incredibly important. Thank you!
thank you :)
I know it is far from your main goal but has anyone considered whether HPy
could be used as a glue API for WASM components. My thought is that one could compile a Python extension written in C or Rust to WASM and distribute it on platforms that you do not have access to a C or Rust compiler for. This would get Python much closer to the old dream of "write once, run anywhere".
this sounds like a nice idea generally speaking, but I'm not sure why it is related to HPy. Why couldn't you do the same with the existing C API?
My understanding is that the HPy project is designed to handle situations where extensions do not have access to the internals of CPython. I think that C code running in a WASM sandbox would fit that pattern.
In fact, C or Rust code running in a WASM sandbox probably wouldn't really know or care whether it was running in CPython, PyPy, Jython, Brython, Skulpt or any other embedding context.
On Thu, Dec 3, 2020 at 12:09 AM Antonio Cuni anto.cuni@gmail.com wrote:
Hello Paul,
On Thu, Dec 3, 2020 at 1:52 AM Paul Prescod paul@prescod.net wrote:
I will not be able to contribute to the HPy project but I think what you are working on is incredibly important. Thank you!
thank you :)
I know it is far from your main goal but has anyone considered whether HPy
could be used as a glue API for WASM components. My thought is that one could compile a Python extension written in C or Rust to WASM and distribute it on platforms that you do not have access to a C or Rust compiler for. This would get Python much closer to the old dream of "write once, run anywhere".
this sounds like a nice idea generally speaking, but I'm not sure why it is related to HPy. Why couldn't you do the same with the existing C API?
Or to put it another way, one cannot generally pass a pointer into a Web Assembly sandbox and have that module manipulate it. One must use handles. Thus "Handle Python API" seemed like a good match.
On Thu, Dec 3, 2020 at 10:15 AM Paul Prescod paul@prescod.net wrote:
My understanding is that the HPy project is designed to handle situations where extensions do not have access to the internals of CPython. I think that C code running in a WASM sandbox would fit that pattern.
In fact, C or Rust code running in a WASM sandbox probably wouldn't really know or care whether it was running in CPython, PyPy, Jython, Brython, Skulpt or any other embedding context.
On Thu, Dec 3, 2020 at 12:09 AM Antonio Cuni anto.cuni@gmail.com wrote:
Hello Paul,
On Thu, Dec 3, 2020 at 1:52 AM Paul Prescod paul@prescod.net wrote:
I will not be able to contribute to the HPy project but I think what you are working on is incredibly important. Thank you!
thank you :)
I know it is far from your main goal but has anyone considered whether
HPy could be used as a glue API for WASM components. My thought is that one could compile a Python extension written in C or Rust to WASM and distribute it on platforms that you do not have access to a C or Rust compiler for. This would get Python much closer to the old dream of "write once, run anywhere".
this sounds like a nice idea generally speaking, but I'm not sure why it is related to HPy. Why couldn't you do the same with the existing C API?
Hi Paul, sorry, I forgot to answer this email :(
On Thu, Dec 3, 2020 at 10:08 PM Paul Prescod paul@prescod.net wrote:
Or to put it another way, one cannot generally pass a pointer into a Web Assembly sandbox and have that module manipulate it. One must use handles. Thus "Handle Python API" seemed like a good match.
I think you are right. However, note that there are cases in which HPy manipulates pointers directly: e.g. HPy_CAST allows you convert a handle into get a pointer to a C struct, which is needed to implement custom types (because your instances are C structs, so you need to get a pointer in order to manipulate). See this issue for a proposal on how it should look like: https://github.com/hpyproject/hpy/issues/83
I don't know much about WASM so I don't know whether this is a hard blocker or if it could be implemented.
ciao, Anto
Thanks for responding.
I think it is safest to think about the WASM runtime as "almost" a different process space that you communicate with through "almost" RPC. "Handles" are just integers which the code inside the runtime converts to pointers however it sees fit (cast, lookup table, ...).
In fact you can implement a (slow!) WASM interpreter entirely *in* python without use of ctypes or anything like that:
https://www.reddit.com/r/Python/comments/dxv1of/keynote_david_beazley_this_t...
So yes, if the host interpreter and the extension environment need to share pointers (i.e. both of them need to treat the value as a C pointer) then it probably won't work without some real hackery (e.g. copying objects back and forth to "pretend" you are sharing memory).
I suppose a pure-handles, no-pointers model does not meet your performance goals. One reason the pure-handles model is attractive in many contexts is that it means that an extension cannot crash the Python interpreter, which opens up a universe where extensions do not have to be trusted code. But if the performance cost is too high then I guess that proposal won't go anywhere.
On Mon, Dec 14, 2020 at 2:37 AM Antonio Cuni anto.cuni@gmail.com wrote:
Hi Paul, sorry, I forgot to answer this email :(
On Thu, Dec 3, 2020 at 10:08 PM Paul Prescod paul@prescod.net wrote:
Or to put it another way, one cannot generally pass a pointer into a Web Assembly sandbox and have that module manipulate it. One must use handles. Thus "Handle Python API" seemed like a good match.
I think you are right. However, note that there are cases in which HPy manipulates pointers directly: e.g. HPy_CAST allows you convert a handle into get a pointer to a C struct, which is needed to implement custom types (because your instances are C structs, so you need to get a pointer in order to manipulate). See this issue for a proposal on how it should look like: https://github.com/hpyproject/hpy/issues/83
I don't know much about WASM so I don't know whether this is a hard blocker or if it could be implemented.
ciao, Anto
On Mon, Dec 14, 2020 at 3:52 PM Paul Prescod paul@prescod.net wrote:
So yes, if the host interpreter and the extension environment need to share pointers (i.e. both of them need to treat the value as a C pointer) then it probably won't work without some real hackery (e.g. copying objects back and forth to "pretend" you are sharing memory).
I think it is possible to do without sharing pointers between the host interpreters and the extension. Currently the API looks like this:
typedef struct { HPyObject_HEAD double x; double y; } PointObject; static HPyType_Spec point_type_spec = { .basicsize = sizeof(PointObject), .... } HPyDef_METH(Point_bar, "bar", Point_bar_impl, HPyFunc_NOARGS) static HPy Point_bar_impl(HPyContext ctx, HPy self) { PointObject *point = HPy_CAST(ctx, PointObject, self); return HPyLong_FromLong(ctx, point->x + point->y); }
If I understand correctly, in your proposal this C code will be compiled into WASM, so manipulating the PointObject* pointers should be fine and allowed. The host interpreter doesn't know anything about the struct PointObject, and it will never manipulate its content: all it does is to do the equivalent of "malloc(point_type->basicsize)" when it creates the instance, and have a way to return this pointer when you do HPy_CAST.
So, for this to work there must be a way for the host interpreter to allocate memory inside the extension's memory space. Do you think that such a thing has any chance to work in WASM?
I suppose a pure-handles, no-pointers model does not meet your
performance goals. One reason the pure-handles model is attractive in many contexts is that it means that an extension cannot crash the Python interpreter, which opens up a universe where extensions do not have to be trusted code. But if the performance cost is too high then I guess that proposal won't go anywhere.
Do you have any proposal/suggestion for how we could fix/improve the HPy API to allow such a scenario?
Antonio: that's good news and yes, it sounds like it would work. Of course I anticipate a shim running inside of the WASM runtime so it can do the allocations based on requests from the parent process.
Do you have any proposal/suggestion for how we could fix/improve the HPy API to allow such a scenario?
If I'm right, with WASM a *truly universal* binary may be possible. Like a Binary that will run on ARM, Intel, Risc-V, CPython, PyPy, in the browser, everywhere!
Maybe it runs faster on Intel and ARM, but it could run everywhere. If David Beazley's work were taken to production it could EVEN run in contexts where the only thing we know is that a Python interpreter exists.
I definitely have too much going on in my life to join your project (as important as I see it as being) but if I could get this concept to prototype/demo, do you think that your team might (just might, not will) adopt it as a real project goal? Personally I think that it might be the kind of thing that would get some really cool attention on HPy and make it "worth it" for package maintainers to accept the pain of migration.
Imagine being able to import NumPy in literally any Python interpreter with enough RAM, even Python interpreters written in Javascript or Python or ToyLang.
I'll have a week off around Christmas where I could work on a prototype of the shim, instead of Advent of Code which was my other plan...
On Mon, Dec 14, 2020 at 9:07 AM Antonio Cuni anto.cuni@gmail.com wrote:
On Mon, Dec 14, 2020 at 3:52 PM Paul Prescod paul@prescod.net wrote:
So yes, if the host interpreter and the extension environment need to share pointers (i.e. both of them need to treat the value as a C pointer) then it probably won't work without some real hackery (e.g. copying objects back and forth to "pretend" you are sharing memory).
I think it is possible to do without sharing pointers between the host interpreters and the extension. Currently the API looks like this:
typedef struct { HPyObject_HEAD double x; double y; } PointObject; static HPyType_Spec point_type_spec = { .basicsize = sizeof(PointObject), .... } HPyDef_METH(Point_bar, "bar", Point_bar_impl, HPyFunc_NOARGS) static HPy Point_bar_impl(HPyContext ctx, HPy self) { PointObject *point = HPy_CAST(ctx, PointObject, self); return HPyLong_FromLong(ctx, point->x + point->y); }
If I understand correctly, in your proposal this C code will be compiled into WASM, so manipulating the PointObject* pointers should be fine and allowed. The host interpreter doesn't know anything about the struct PointObject, and it will never manipulate its content: all it does is to do the equivalent of "malloc(point_type->basicsize)" when it creates the instance, and have a way to return this pointer when you do HPy_CAST.
So, for this to work there must be a way for the host interpreter to allocate memory inside the extension's memory space. Do you think that such a thing has any chance to work in WASM?
I suppose a pure-handles, no-pointers model does not meet your
performance goals. One reason the pure-handles model is attractive in many contexts is that it means that an extension cannot crash the Python interpreter, which opens up a universe where extensions do not have to be trusted code. But if the performance cost is too high then I guess that proposal won't go anywhere.
Do you have any proposal/suggestion for how we could fix/improve the HPy API to allow such a scenario?
On Mon, Dec 14, 2020 at 8:07 PM Paul Prescod paul@prescod.net wrote:
Antonio: that's good news and yes, it sounds like it would work. Of course I anticipate a shim running inside of the WASM runtime so it can do the allocations based on requests from the parent process.
yes, that's more or less what I had in mind, sounds good.
Do you have any proposal/suggestion for how we could fix/improve the HPy API to allow such a scenario?
If I'm right, with WASM a *truly universal* binary may be possible. Like a Binary that will run on ARM, Intel, Risc-V, CPython, PyPy, in the browser, everywhere!
Maybe it runs faster on Intel and ARM, but it could run everywhere. If David Beazley's work were taken to production it could EVEN run in contexts where the only thing we know is that a Python interpreter exists.
I definitely have too much going on in my life to join your project (as important as I see it as being) but if I could get this concept to prototype/demo, do you think that your team might (just might, not will) adopt it as a real project goal? Personally I think that it might be the kind of thing that would get some really cool attention on HPy and make it "worth it" for package maintainers to accept the pain of migration.
I agree that the idea is cool and that probably would generate some hype. I'm not sure that at this point in time it would be the killer feature to convince e.g. numpy to migrate to HPy, but it would certainly be a welcome plus, in my opinion. Currently, the
Re "adopt it as a real project goal": it is *already* a project goal :): the main goal of HPy is to design an API which is less tied to the underlying implementation to make it possible to experiment with innovative solutions like this one. So, if we can design the API in such a way that makes WASM easier/faster without undermining the other goals (the most important of which is 0 overhead on CPython), I'll be happy to do that.
If your question is whether HPy might want to adopt WASM as an optional backend (or maybe the the default) for its universal ABI, I think it's way too early to say: who knows, maybe in the future, but the point is that such a project could in principle be developed also outside HPy.
I'll have a week off around Christmas where I could work on a prototype of
the shim, instead of Advent of Code which was my other plan...
Having a proof of concept would be really cool and a nice showcase of why HPy is good and important. I don't think we as a team we can commit to maintaining it of course but I'll be happy to mention it in the docs and/or future blog posts or so.
ciao, Anto
participants (2)
-
Antonio Cuni -
Paul Prescod