Mailman 3 October 2020 - pypy-dev

PyPy Winter Sprint + cpyext
by Stefan Beyer Dec. 21, 2022

Dec. 21, 2022

Hey everyone! Those of you who attended or followed one of the last two winter sprints in Leysin might know me, the others probably won't. I'm (still) a Master's student writing my thesis on PyPy's current issue with cross-heap cycles when using cpyext. The main point is, that they are not collected and stay around as floating garbage, even if they are not reachable any more. Correct me if I'm wrong, but I didn't notice anyone else working on that topic (and/or committing something) since I've picked it up two years ago (yeah, I know, that long ago...). I saw that you would be working on "cpyext performance and completeness" during the current winter sprint this week and thought that this might also concern my thesis. So I thought I'll give you an update. I recently pushed my current (non-optimized, breaking-some-tests, but more or less working) implementation of the "CPython-style" GC-extension for cpyext. It is still in an early stage and not all cases (legacy and non-legacy-finalizers, weakrefs) are handled. However, I picked up some pace during the last couple of weeks and I'm determined to finish this implementation during the following weeks (some quirks with the tests have been haunting me, but I think I figured most of them out by now). After that, I will implement a second alternative implementation (to compare to, mostly for the sake of my thesis), which will take some more time. I also added a couple of fancy test cases (the so called "dot-tests"), so that we can test complex object graphs a little bit easier (and also because it was kind of cool to have another language inside PyPy/RPython, even if it was only for the tests...), with more test cases to come (the current ones are a bit messy). None of the new changes concerning low-latency applications are currently integrated, but that should not be too hard. I guess it won't make much sense for me to join you at this year's winter sprint, especially as I won't be able to get there before Thursday, but I might be able to join you over the IRC channel (or some other form of communication if you like). If there is anything that is worth discussing please let me know! Also feel free to comment on my code, but beware that I might change some things once I try to do some optimizations (probably not many, but at least fix the worst issues) and make it a little bit more readable. You can find my code on the cpyext-gc-cycle branch.<https://bitbucket.org/pypy/pypy/commits/branch/cpyext-gc-cycle> Looking forward to hearing from you! Greetings, Stefan

2 2

Checksums for some old releases changed
by Ivan Pozdeev Oct. 30, 2020

Oct. 30, 2020

When rechecking URLs in installation scripts in Pyenv to fix any invalid ones since the project has moved from BitBucket ( https://github.com/pyenv/pyenv/pull/1692 ) , I noticed that checksums for downloads for some old PyPy releases have changed. See the altered hashes in this diff to see which exactly: https://github.com/pyenv/pyenv/pull/1692/files . Could someone explain why? Were the archives repackaged or something? -- Regards, Ivan

2 1

summary of PyPy progess since the 7.3.2 release
by Matti Picus Oct. 18, 2020

Oct. 18, 2020

We have made some good progress since the 7.3.2 release, it seems our activity is on an uptick. To keep the wave going, here is the status of the various things that have seen work lately with some "call to actions". This is my summary so it probably contains some errors. I will not call out specific contributors, since I would probably miss a few, but a big thanks to all of them. default python2.7 ----------------- The stdlib was updated to 2.7.18 We are still missing some basic cpyext compatability that users have asked for, I have started marking these issues with "cpyext" py3.6 ------- The stdlib was updated to 3.6.12 There are a number of failures around weakref py3.7 ------- The stdlib was updated to 3.7.4 for the release. There are numerous lib-python failures There is a mysterious (to me) crash in the lib-python tests on windows The regex library still needs updating to match upstream Windows ------- There is a win64 branch off default! It appears to have no fewer failures than the default branch, but cpyext and micronumpy are disabled. There is also a win64-py3.6 branch, which translates but has problems running untranslated tests, I think due to some problem with the rsocket module and addresses. hpy_universal ------------- The hpy backend module is active on the py3.6 branch and all its children. It is disabled on 32 bit and fails win64-py3.6 own tests. speed.pypy.org -------------------- We should switch the landing page to show benchmarks on the python3 versions. This is a web-stack/devops task more than a rpython one, so might be a good place to get different contributors involved. pypy + conda ------------ Main packages like scikit-learn, scikit-image, scipy are all available on linux64 + arm64 + macOS on conda. The current situation can be seen by going to https://conda-forge.org/status/, scrolling down 1/2 way to the "pypy migration" section, clicking the purple "in pr" button. "typed-ast" is the biggest blocker, and will not be solved soon. I think it is a false-positive for many of the children packages: they are not really dependent on it. Help here would be looking at the failures of your favorite package and nudging the upstream team to add PyPy to their CI, fix those failures, then fix the conda recipe and get it merged. Any help would be welcome: whether coding, specific hints where things might be going wrong, marketing to get the work out that pypy + conda is usable, help getting sponsors or coders to move the windows branch forward, help with conda feedstock recipes, ... Matti

1 0

Help to contribute
by Andrews Medina Oct. 17, 2020

Oct. 17, 2020

Hi! I'm want to get some time for week to contribute to pypy. I already have sent a few pr's approved in the past with small py3.x lib fixes. Do you think that this is a good way to restart to contribute? There is a branch for 3.8 or there are things to fix in 3.7? Best regards, -- Andrews Medina

2 2

pypy and pyproject.toml
by wlavrijsen＠lbl.gov Oct. 10, 2020

Oct. 10, 2020

Hi, pip 21 is (likely) going to drop the fallback of running `setup.py install` if wheels can not be build. I've been using that fallback b/c wheels are not installed immediately on completion and do not respect the dependency order for build-time, whereas `setup.py install` does. With a "pyproject.toml" file, which pip supports, I can control the build environment again and that works quite nicely for CPython. For PyPy, however, I am unable to make selections based on the PyPy version. I can make selections based on the PyPy Python version, e.g. 2.7, 3.5, etc., but I'm talking about the actual PyPy version (e.g. 7.2.0). Does anyone know how to add a marker based on the PyPy version in a pyproject.toml file? Or, how else to enforce build dependencies for pip in the same way that can be done with the pyproject.toml file? Thanks, Wim -- WLavrijsen(a)lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net

2 2

Pypy on aarch64 (rhe7) has issues with bzip2-libs
by Srinivasa Kempapura Padmanabha Oct. 9, 2020

Oct. 9, 2020

Hi, When I extract the pypy prebuild package as instructions provided in the site. I see that the binary fails to run ./bin/pypy3: error while loading shared libraries: libbz2.so.1.0: cannot open shared object file: No such file or directory We have installed bzip2 1.0.6 but still I see somewhere that the package requires other compatible version of it. /usr/lib64/libbz2.so /usr/lib64/libbz2.so.1 /usr/lib64/libbz2.so.1.0.6 Can you please provide more details on that [A close up of a sign Description automatically generated] Srinivasa Kempapura Padmanabha | Staff Engineer Technology Operations Services Group Email & SFB & Teams: srikem01(a)arm.com<mailto:srikem01@arm.com> Mobile: +91 9980633788 Bagmane World technology Center- SEZ, Citrine Block 11th Floor, Marathahalli ORR, Mahadevapura, Bangalore -560048, India IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

4 9

About PySide2
by Mustafa Ulubas Oct. 7, 2020

Oct. 7, 2020

Hello there, I’m looking for someone who can help me to build-install PySide2 into PyPy. That’d be really helpful and appreciated if there is anyone. Thanks Mustafa

1 0

Interaction between HPy_CAST and HPy_AsPyObject
by Antonio Cuni Oct. 1, 2020

Oct. 1, 2020

Consider the following snippet of code: typedef struct { HPyObject_HEAD long x; long y; } PointObject; void foo(HPyContext ctx, HPy h_point) { PointObject *p1 = HPy_CAST(ctx, PointObject, h_point); PyObjecy *py_point = HPy_AsPyObject(ctx, h_point); // [1] PointObject *p2 = (PointObject*)py_point; ... } [1] Note that it does not need to be a call to HPy_AsPyObject: it might be a legacy method which takes a PyObject *self, or other similar ways It is obvious that HPy_CAST and HPy_AsPyObject need to return the very same address. This is straightforward to implement on CPython, but it poses some challenges on PyPy (and probably GraalPython). Things to consider: 1. currently, in PyPy we allocate the PointObject at a non-movable address, but so far the API does not REQUIRE it. I think it would be reasonable to have an implementation in which objects are movable and HPy_CAST pins the memory until the originating handle is closed. OTOH, the only reasonable semantics is that multiple calls to HPy_AsPyObject returns always the same address. 2. HPyObject_HEAD consists of two words which can be used by the implementation as they like. On CPython, it is obviously mapped to PyObject_HEAD, but in PyPy we (usually) don't need these two extra words, so we allocate sizeof(PointObject)-16 and return a pointer to malloc()-16, which works well since nobody is accessing those two words. I think that GraalPython could use a similar approach. 3. On PyPy, PyObject_HEAD is *three words*, because it also contains ob_pypy_link. But, since the code uses *H*PyObject_HEAD, PointObject will contain only 2 extra words. 4. In the real world usage, there will be "pure hpy types" and "legacy hpy types", which uses legacy methods&co. It would be nice if the pure hpy types do NOT have to pay penalties in case they are never casted to PyObject* With this in mind, how do we implement HPy_AsPyObject on PyPy? One easy way is: 1. we allocate sizeof(PointObject)+8 2. we tweak cpyext to find ob_pypy_link at p-8 3. we teach cpyext how to convert W_HPyObject into PyObject* and vice versa. However, this means that we need to always allocate 24 extra bytes for each object, even if nobody ever calls HPy_AsPyObject on it, which looks bad. Moreover, without changes in the API, the pin/unpin implementation of HPy_CAST becomes de facto impossible. So, my proposal is to distinguish between "legacy hpy types" and "pure hpy types". An HPyType_Spec is legacy if: 1. it uses .legacy_slots = ... OR 2. it ses .legacy = true (i.e., you can explicitly mark a type as legacy even if you no longer have any legacy method/slot. This is useful if you pass it to ANOTHER type which expects to be able to cast the PyObject* into the struct). If a type is "legacy", the snippet shown above works as expected; if it's not legacy, it is still possible to call HPy_AsPyObject on it, but then you are no longer allowed to C-cast if to PointObject* (on pypy, this will mean that you will get a "standard" PyObject* which is a proxy to W_HPyObject). Ideally, in that case it would be nice to catch the invalid cast in the debug mode, but I don't think this is possible... too bad. What do you think? ciao, Anto

2 2

About PySide2
by Mustafa Ulubas Oct. 1, 2020

Oct. 1, 2020

Hi there, Hope you are doing well. I'm currently working on a project which I'm trying to use PyPy in order to speed up our simulation processes. There are a couple of dependencies that our project has including PySide2. I've tried to install PySide2 into PyPy however I couldn't. From what I understand as a result of my investigations, currently, PyPy doesn't support PySide2. First of all, is that true? Does PyPy really not support it? Is there any way to install it? I'll be waiting for your response. Thanks in advance. Mustafa

2 1