Consider the following snippet of code:
typedef struct {
HPyObject_HEAD
long x;
long y;
} PointObject;
void foo(HPyContext ctx, HPy h_point)
{
PointObject *p1 = HPy_CAST(ctx, PointObject, h_point);
PyObjecy *py_point = HPy_AsPyObject(ctx, h_point); // [1]
PointObject *p2 = (PointObject*)py_point;
...
}
[1] Note that it does not need to be a call to HPy_AsPyObject: it might be
a legacy method which takes a PyObject *self, or other similar ways
It is obvious that …
[View More]HPy_CAST and HPy_AsPyObject need to return the very same
address. This is straightforward to implement on CPython, but it poses some
challenges on PyPy (and probably GraalPython).
Things to consider:
1. currently, in PyPy we allocate the PointObject at a non-movable address,
but so far the API does not REQUIRE it. I think it would be reasonable to
have an implementation in which objects are movable and HPy_CAST pins the
memory until the originating handle is closed. OTOH, the only reasonable
semantics is that multiple calls to HPy_AsPyObject returns always the same
address.
2. HPyObject_HEAD consists of two words which can be used by the
implementation as they like. On CPython, it is obviously mapped to
PyObject_HEAD, but in PyPy we (usually) don't need these two extra words,
so we allocate sizeof(PointObject)-16 and return a pointer to malloc()-16,
which works well since nobody is accessing those two words. I think that
GraalPython could use a similar approach.
3. On PyPy, PyObject_HEAD is *three words*, because it also contains
ob_pypy_link. But, since the code uses *H*PyObject_HEAD, PointObject will
contain only 2 extra words.
4. In the real world usage, there will be "pure hpy types" and "legacy hpy
types", which uses legacy methods&co. It would be nice if the pure hpy
types do NOT have to pay penalties in case they are never casted to
PyObject*
With this in mind, how do we implement HPy_AsPyObject on PyPy? One easy way
is:
1. we allocate sizeof(PointObject)+8
2. we tweak cpyext to find ob_pypy_link at p-8
3. we teach cpyext how to convert W_HPyObject into PyObject* and vice versa.
However, this means that we need to always allocate 24 extra bytes for each
object, even if nobody ever calls HPy_AsPyObject on it, which looks bad.
Moreover, without changes in the API, the pin/unpin implementation of
HPy_CAST becomes de facto impossible.
So, my proposal is to distinguish between "legacy hpy types" and "pure hpy
types". An HPyType_Spec is legacy if:
1. it uses .legacy_slots = ... OR
2. it ses .legacy = true (i.e., you can explicitly mark a type as legacy
even if you no longer have any legacy method/slot. This is useful if you
pass it to ANOTHER type which expects to be able to cast the PyObject* into
the struct).
If a type is "legacy", the snippet shown above works as expected; if it's
not legacy, it is still possible to call HPy_AsPyObject on it, but then you
are no longer allowed to C-cast if to PointObject* (on pypy, this will mean
that you will get a "standard" PyObject* which is a proxy to W_HPyObject).
Ideally, in that case it would be nice to catch the invalid cast in the
debug mode, but I don't think this is possible... too bad.
What do you think?
ciao,
Anto
[View Less]
Hi pypy-dev (cc:ing hpy-dev),
I have just merged the hpy branch into py3.6. This means that from now,
pypy3.6 nightly builds will automatically support hpy, and it will be much
easier for interested users to try it out.
I propose the following workflow for continuing HPy development on PyPy:
1. the development will continue on the hpy branch, in "update_vendored"
steps: i.e., periodically we run the script to update to a newer hpy
version and implement all the new features. As soon as we do …
[View More]it, new tests
are introduced and they (hopefully :)) will start failing
2. note that it is not necessarily to ./update_vendored.sh to the LATEST
git version: this will probably introduce too many features and will make
it harder/longer to make all tests green again. So I suggest to gradually
./update_vendored.sh in smaller step (you can do it by doing "git checkout
REV" in your main hpy working copy before running update_vendored)
3. Once the tests are green again, we can merge hpy into py3.6. The merge
can happen directly or with a gitlab MR if you want someone to review the
code. Personally, I volunteer to review all hpy-related MR, so feel free to
ping me if you want :)
4. goto 1
At the moment, all tests inside module/_hpy_universal pass, so we are in a
green state: I would like to try hard to keep them green, and merge changes
only after the tests pass: the hpy tests are run automatically on gitlab-ci
whenever you push to the hpy branch, so it should be doable.
A note about nightly builds: as soon as we catch up with hpy git
revision 0a46d31, it will be possible to run "import hpy;
hpy.get_version()". This will be very useful for nightly builds, because it
will tell you exactly which hpy.devel revision to check out in order to be
compatible with your nightly build.
As soon as the API stabilizes we will want to use official version numbers,
but I think that for now it's a reasonable approach.
Please let me know if you have any questions.
ciao,
Anto
[View Less]