Mailman 3 May 2021 - pypy-dev

PyPy Winter Sprint + cpyext
by Stefan Beyer 21 Dec '22

21 Dec '22

Hey everyone! Those of you who attended or followed one of the last two winter sprints in Leysin might know me, the others probably won't. I'm (still) a Master's student writing my thesis on PyPy's current issue with cross-heap cycles when using cpyext. The main point is, that they are not collected and stay around as floating garbage, even if they are not reachable any more. Correct me if I'm wrong, but I didn't notice anyone else working on that topic (and/or committing something) since I've picked it up two years ago (yeah, I know, that long ago...). I saw that you would be working on "cpyext performance and completeness" during the current winter sprint this week and thought that this might also concern my thesis. So I thought I'll give you an update. I recently pushed my current (non-optimized, breaking-some-tests, but more or less working) implementation of the "CPython-style" GC-extension for cpyext. It is still in an early stage and not all cases (legacy and non-legacy-finalizers, weakrefs) are handled. However, I picked up some pace during the last couple of weeks and I'm determined to finish this implementation during the following weeks (some quirks with the tests have been haunting me, but I think I figured most of them out by now). After that, I will implement a second alternative implementation (to compare to, mostly for the sake of my thesis), which will take some more time. I also added a couple of fancy test cases (the so called "dot-tests"), so that we can test complex object graphs a little bit easier (and also because it was kind of cool to have another language inside PyPy/RPython, even if it was only for the tests...), with more test cases to come (the current ones are a bit messy). None of the new changes concerning low-latency applications are currently integrated, but that should not be too hard. I guess it won't make much sense for me to join you at this year's winter sprint, especially as I won't be able to get there before Thursday, but I might be able to join you over the IRC channel (or some other form of communication if you like). If there is anything that is worth discussing please let me know! Also feel free to comment on my code, but beware that I might change some things once I try to do some optimizations (probably not many, but at least fix the worst issues) and make it a little bit more readable. You can find my code on the cpyext-gc-cycle branch.<https://bitbucket.org/pypy/pypy/commits/branch/cpyext-gc-cycle> Looking forward to hearing from you! Greetings, Stefan

2 2

Re: [pypy-dev] Memory leak in 7.3.4
by Brad Kish 31 May '21

31 May '21

Thanks Carl. Appreciate the feedback and suggestions. We will try this out. Brad. Hi Brad, the new json decoder caches values, but not across json.load calls. ie if you don't process a single gigantic message but many tiny messages, the caching of (string) values cannot be the problem. what *could* be the problem is the key caching though, in theory. The key cache is partially persisted across calls. Are there arbitrarily many different keys in your messages? The key cache shouldn't grow without bounds, however. what do you set your max heap size to? To debug further, I would first try to see whether it is indeed the json module or something else. Maybe you could try another json decoder and see whether the problem persists with that? Eg ujson works on PyPy (slowly) or you could even use the pure python builtin one that you get by importing json.decoder.JSONDecoder. Cheers, Carl Friedrich

1 0

seeking pypy gc configuring guideline
by Raihan Rasheed Apurbo 31 May '21

31 May '21

I was trying to run pypy using semispace gc. But as it stands currently I can only compile using incminimark and boehm. What changes I have to make so that I would be able to test pypy with any existing gc implementation? Moreover I want to write my own version of gc and test with pypy. In future. Can anyone point me where do I have to make changes to make my implementation of any arbitrary gc work? So far I was able to compile an edited version of incminimark gc. Any suggestions or link to any perticular documentation would be appreciated. p.s. I ran pypy with the following command and got this error. I can run same command just replacing incminimark in place of semispace. [image: image.png]

3 8

New official IRC channel
by Antonio Cuni 30 May '21

30 May '21

Following the example of many other FOSS projects, the PyPy team has decided to move its official #pypy IRC channel from Freenode to Libera.Chat: irc://irc.libera.chat/hpy The core devs will no longer be present on the Freenode channel, so we recommend to join the new channel as soon as possible. https://www.pypy.org/posts/2021/05/pypy-irc-moves-to-libera-chat.html

1 0

Memory leak in 7.3.4
by Brad Kish 27 May '21

27 May '21

We have a streaming application that processes JSON messages. Our system has been mostly stable running PyPy2.7 6.0 for a couple of years. Recently we tried upgrading to 7.3.4 and saw our processes leak memory until the container was killed. This happened with both PyPy2.7 7.3.4 and PyPy3.7 7.3.4. The only difference was the cpu usage with PyPy3.7 was almost double. :( My question is could this be from the new JSON parser introduced in 7.2? According to https://morepypy.blogspot.com/2019/10/pypys-new-json-parser.html, the parser now caches values not just keys. Is there anyway to disable the value caching? Or other things we can try? Thanks, Brad.

2 2

[External] pypy3 Multiprocessing Socket Issue
by Moulton, Tom 20 May '21

20 May '21

I'm using pypy3.7-v7.3.4-linux64 (Python v3.7.10) but have also tried pypy3.7-v7.3.5rc1-linux64 & pypy-c-jit-102427-77787b8f4c49-linux64. I'm seeing an issue that does NOT occur when using regular Python 3.7 (CPython v3.7.10). My application uses Python's multiprocessing library to take advantage of multiple cores on the CPU and as such I have shared resources that get utilized between these processes (sockets, queues, etc.), arbitrated through appropriate locking mechanisms to address resource contention. The issue seems to occur with respect to the sharing of sockets between these processes. That is, after 30 or more seconds I start to continually receive the following output, with NO stack trace to where it is occurring in my application... Traceback (most recent call last): File "/opt/pypy/lib-python/3/multiprocessing/resource_sharer.py", line 142, in _serve File "/opt/pypy/lib-python/3/multiprocessing/connection.py", line 453, in accept File "/opt/pypy/lib-python/3/multiprocessing/connection.py", line 599, in accept File "/opt/pypy/lib-python/3/socket.py", line 212, in accept OSError: [Errno 24] Too many open files Clearly, under pypy3, my application is exceeding the number of open files supported by the operating system (OS) and I can see this occur during the ~30 second lead up, where if I run "lsof -i" (in another shell), I can see the number of TCP/IP connections increasing steadily (associated with my application) up to the OS limit, which I can only assume is associated with, when one my processes accesses this shared socket resource. Again, this does NOT occur when running my application under regular Python (CPython) and I verify this using the same "lsof -i" method. Thoughts on how I might overcome this issue so I can see if using pypy3 over CPython 3 garners better performance for my specific application? Regards, Tom Moulton -- This communication, including any attachments, may contain Proprietary, Export Controlled, Legally Privileged or other Confidential Information. Any unauthorized review, printing, copying, transmission, dissemination, or use of the information is strictly prohibited. If you are not the intended recipient, please indicate to the sender that you have received this email in error, and delete the copy you received.

2 1

7.3.5 rc3 tarballs are up
by Matti Picus 19 May '21

19 May '21

I have uploaded rc3 candidates to https://downloads.python.org/pypy/ and the checksums can be found in the PR to pypy.org https://608f07a183d23c00083f6115--keen-mestorf-442210.netlify.app/download_… The changes are in the release note. https://doc.pypy.org/en/latest/release-v7.3.5.html: - The new windows 64-bit builds improperly named c-extension modules with the same extension as the 32-bit build (issue 3443) - Use the windows-specific PC/pyconfig.h rather than the posix one - A change to the python 3.7 sysconfig.get_config_var('LIBDIR') was wrong, leading to problems finding libpypy3-c.so for embedded PyPy (issue 3442). - Instantiate distutils.command.install schema for PyPy-specific implementation_lower - Four upstream (CPython) security patches were applied - When assigning the full slice of a list, evaluate the rhs before clearing the list (issue 3440) - On Python2, PyUnicode_Contains accepts bytes as well as unicode. - Update the packaged sqlite3 to 3.35.5 on windows. While not a bugfix, this seems like an easy win. Added in this rc3: ----------------------- - Fix the return type for |_Py_HashDouble| which impacts 64-bit windows - Delay thread-checking logic in greenlets until the thread is actually started (continuation of issue 3441 <https://foss.heptapod.net/pypy/pypy/-/issues/3441>) - Fix for json-specialized dicts (issue 3460 <https://foss.heptapod.net/pypy/pypy/-/issues/3460>) - Specialize |ByteBuffer.setslice| which speeds up binary file reading by a factor of 3 - Test and fix a broken code path in _sqlite3 (issue 3432) Please try them out Matti

1 0

Koka benchmark paper
by Yury V. Zaytsev 17 May '21

17 May '21

Hi folks, You must have already seen the TR by MSR covering the new reference counting garbage collector for Koka? https://www.microsoft.com/en-us/research/publication/perceus-garbage-free-r… I really wonder how warmed up PyPy (including vs. CPython ;-) ) would do on those benchmarks if translated rather close to Koka? -- Sincerely yours, Yury V. Zaytsev

2 3

7.3.5 rc2 tarballs are up.
by Matti Picus 15 May '21

15 May '21

Following the rc1 I have uploaded rc2 candidates to https://downloads.python.org/pypy/ and the checksums can be found in the PR to pypy.org https://608f07a183d23c00083f6115--keen-mestorf-442210.netlify.app/download_… The changes are in the release note. https://doc.pypy.org/en/latest/release-v7.3.5.html: - The new windows 64-bit builds improperly named c-extension modules with the same extension as the 32-bit build (issue 3443) - Use the windows-specific PC/pyconfig.h rather than the posix one - A change to the python 3.7 sysconfig.get_config_var('LIBDIR') was wrong, leading to problems finding libpypy3-c.so for embedded PyPy (issue 3442). - Instantiate distutils.command.install schema for PyPy-specific implementation_lower - Four upstream (CPython) security patches were applied - When assigning the full slice of a list, evaluate the rhs before clearing the list (issue 3440) - On Python2, PyUnicode_Contains accepts bytes as well as unicode. - Update the packaged sqlite3 to 3.35.5 on windows. While not a bugfix, this seems like an easy win. Please try them out Matti

2 2

Re: [pypy-dev] 7.3.5 rc2 tarballs are up.
by Matti Picus 07 May '21

07 May '21

On 7/5/21 11:59 am, Oliver Margetts wrote: > Hi, did this fix https://foss.heptapod.net/pypy/pypy/-/issues/3441 > <https://foss.heptapod.net/pypy/pypy/-/issues/3441> make it into the RC? > > From my testing, it still seems to be a problem. > > Best, > Oliver > Thanks, I missed that but put it in now. It will be part of rc3, which will come out once I figure out what is going wrong with cpyext and win64. Matti

1 0