Mailman 3 September 2021 - pypy-dev

PyPy Winter Sprint + cpyext
by Stefan Beyer Dec. 21, 2022

Dec. 21, 2022

Hey everyone! Those of you who attended or followed one of the last two winter sprints in Leysin might know me, the others probably won't. I'm (still) a Master's student writing my thesis on PyPy's current issue with cross-heap cycles when using cpyext. The main point is, that they are not collected and stay around as floating garbage, even if they are not reachable any more. Correct me if I'm wrong, but I didn't notice anyone else working on that topic (and/or committing something) since I've picked it up two years ago (yeah, I know, that long ago...). I saw that you would be working on "cpyext performance and completeness" during the current winter sprint this week and thought that this might also concern my thesis. So I thought I'll give you an update. I recently pushed my current (non-optimized, breaking-some-tests, but more or less working) implementation of the "CPython-style" GC-extension for cpyext. It is still in an early stage and not all cases (legacy and non-legacy-finalizers, weakrefs) are handled. However, I picked up some pace during the last couple of weeks and I'm determined to finish this implementation during the following weeks (some quirks with the tests have been haunting me, but I think I figured most of them out by now). After that, I will implement a second alternative implementation (to compare to, mostly for the sake of my thesis), which will take some more time. I also added a couple of fancy test cases (the so called "dot-tests"), so that we can test complex object graphs a little bit easier (and also because it was kind of cool to have another language inside PyPy/RPython, even if it was only for the tests...), with more test cases to come (the current ones are a bit messy). None of the new changes concerning low-latency applications are currently integrated, but that should not be too hard. I guess it won't make much sense for me to join you at this year's winter sprint, especially as I won't be able to get there before Thursday, but I might be able to join you over the IRC channel (or some other form of communication if you like). If there is anything that is worth discussing please let me know! Also feel free to comment on my code, but beware that I might change some things once I try to do some optimizations (probably not many, but at least fix the worst issues) and make it a little bit more readable. You can find my code on the cpyext-gc-cycle branch.<https://bitbucket.org/pypy/pypy/commits/branch/cpyext-gc-cycle> Looking forward to hearing from you! Greetings, Stefan

2 2

Contributing to PyPy
by M A Oct. 19, 2021

Oct. 19, 2021

Hi I am interested in contributing to PyPy. I tried to clone the source code repository using hg but saw errors. I then found this git repo: https://github.com/mozillazg/pypy. I was actually able to clone it successfully. Could someone still contribute to PyPy using git instead of hg? If you are curious here are the errors I see when I try to clone the repository using hg: hg clone https://foss.heptapod.net/pypy/pypy destination directory: pypy applying clone bundle from https://cellar-c2.services.clever-cloud.com/heptapod-foss-clonebundles/pypy… adding changesets adding manifests adding file changes transaction abort! rollback completed ** unknown exception encountered, please report by visiting ** https://mercurial-scm.org/wiki/BugTracker ** Python 3.9.7 (default, Sep 3 2021, 04:31:11) [Clang 12.0.5 (clang-1205.0.22.9)] ** Mercurial Distributed SCM (version 5.9.1) ** Extensions loaded: Traceback (most recent call last): File "/opt/homebrew/bin/hg", line 61, in <module> dispatch.run() File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/dispatch.py", line 144, in run status = dispatch(req) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/dispatch.py", line 250, in dispatch status = _rundispatch(req) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/dispatch.py", line 294, in _rundispatch ret = _runcatch(req) or 0 File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/dispatch.py", line 470, in _runcatch return _callcatch(ui, _runcatchfunc) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/dispatch.py", line 480, in _callcatch return scmutil.callcatch(ui, func) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/scmutil.py", line 153, in callcatch return func() File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/dispatch.py", line 460, in _runcatchfunc return _dispatch(req) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/dispatch.py", line 1273, in _dispatch return runcommand( File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/dispatch.py", line 918, in runcommand ret = _runcommand(ui, options, cmd, d) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/dispatch.py", line 1285, in _runcommand return cmdfunc() File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/dispatch.py", line 1271, in <lambda> d = lambda: util.checksignature(func)(ui, *args, **strcmdopt) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/util.py", line 1886, in check return func(*args, **kwargs) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/commands.py", line 1955, in clone r = hg.clone( File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/hg.py", line 942, in clone exchange.pull( File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/exchange.py", line 1668, in pull _maybeapplyclonebundle(pullop) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/exchange.py", line 2779, in _maybeapplyclonebundle if trypullbundlefromurl(repo.ui, repo, url): File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/exchange.py", line 2810, in trypullbundlefromurl bundle2.applybundle(repo, cg, tr, b'clonebundles', url) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/bundle2.py", line 375, in applybundle return processbundle(repo, unbundler, lambda: tr, source=source) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/bundle2.py", line 486, in processbundle processparts(repo, op, unbundler) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/bundle2.py", line 494, in processparts _processpart(op, part) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/bundle2.py", line 568, in _processpart handler(op, part) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/bundle2.py", line 2025, in handlechangegroup ret = _processchangegroup( File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/bundle2.py", line 498, in _processchangegroup ret = cg.apply(op.repo, tr, source, url, **kwargs) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/changegroup.py", line 444, in apply newrevs, newfiles = _addchangegroupfiles( File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/changegroup.py", line 1946, in _addchangegroupfiles added = fl.addgroup( File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/filelog.py", line 168, in addgroup return self._revlog.addgroup( File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/revlog.py", line 2620, in addgroup for data in deltas: File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/revlogutils/rewrite.py", line 783, in filter_delta_issue6528 is_affected = _is_revision_affected_fast_inner( File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/revlogutils/rewrite.py", line 649, in _is_revision_affected_fast_inner return _is_revision_affected_inner( File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/revlogutils/rewrite.py", line 592, in _is_revision_affected_inner has_meta = raw_text.startswith(b'\x01\n') AttributeError: 'memoryview' object has no attribute 'startswith' ** Extensions loaded: Traceback (most recent call last): File "/opt/homebrew/bin/hg", line 61, in <module> dispatch.run() File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/dispatch.py", line 144, in run status = dispatch(req) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/dispatch.py", line 250, in dispatch status = _rundispatch(req) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/dispatch.py", line 294, in _rundispatch ret = _runcatch(req) or 0 File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/dispatch.py", line 470, in _runcatch return _callcatch(ui, _runcatchfunc) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/dispatch.py", line 480, in _callcatch return scmutil.callcatch(ui, func) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/scmutil.py", line 153, in callcatch return func() File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/dispatch.py", line 460, in _runcatchfunc return _dispatch(req) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/dispatch.py", line 1273, in _dispatch return runcommand( File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/dispatch.py", line 918, in runcommand ret = _runcommand(ui, options, cmd, d) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/dispatch.py", line 1285, in _runcommand return cmdfunc() File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/dispatch.py", line 1271, in <lambda> d = lambda: util.checksignature(func)(ui, *args, **strcmdopt) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/util.py", line 1886, in check return func(*args, **kwargs) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/commands.py", line 1955, in clone r = hg.clone( File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/hg.py", line 942, in clone exchange.pull( File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/exchange.py", line 1668, in pull _maybeapplyclonebundle(pullop) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/exchange.py", line 2779, in _maybeapplyclonebundle if trypullbundlefromurl(repo.ui, repo, url): File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/exchange.py", line 2810, in trypullbundlefromurl bundle2.applybundle(repo, cg, tr, b'clonebundles', url) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/bundle2.py", line 375, in applybundle return processbundle(repo, unbundler, lambda: tr, source=source) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/bundle2.py", line 486, in processbundle processparts(repo, op, unbundler) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/bundle2.py", line 494, in processparts _processpart(op, part) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/bundle2.py", line 568, in _processpart handler(op, part) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/bundle2.py", line 2025, in handlechangegroup ret = _processchangegroup( File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/bundle2.py", line 498, in _processchangegroup ret = cg.apply(op.repo, tr, source, url, **kwargs) File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/changegroup.py", line 444, in apply newrevs, newfiles = _addchangegroupfiles( File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/changegroup.py", line 1946, in _addchangegroupfiles added = fl.addgroup( File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/filelog.py", line 168, in addgroup return self._revlog.addgroup( File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/revlog.py", line 2620, in addgroup for data in deltas: File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/revlogutils/rewrite.py", line 783, in filter_delta_issue6528 is_affected = _is_revision_affected_fast_inner( File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/revlogutils/rewrite.py", line 649, in _is_revision_affected_fast_inner return _is_revision_affected_inner( File "/opt/homebrew/Cellar/mercurial/5.9.1/lib/python3.9/site-packages/mercurial/revlogutils/rewrite.py", line 592, in _is_revision_affected_inner has_meta = raw_text.startswith(b'\x01\n') AttributeError: 'memoryview' object has no attribute 'startswith'

2 5

Advice on code location
by M A Sept. 24, 2021

Sept. 24, 2021

Hi I'm trying to use Python to build PyPy. One problem I noticed is the search paths for modules is not the same For PyPy and Python. I want to add code to the build process that will check if PyPy or Python is running. If Python is running, add paths to sys.path that would enable Python to find needed modules. Which file would be a good place to add such code? Thank you.

2 1

Does PyPy memory use benefit from preforking?
by Tin Tvrtković Sept. 24, 2021

Sept. 24, 2021

Hello! A little bit of context: roughly speaking, preforking is a technique where a (supervisor) process is started, the process performs some initialization and then forks off into child worker processes, which it then supervises. It's usually used to make several worker processes share a server TCP socket (which they inherit from the supervisor). In some runtimes preforking can also be used to save memory since the child processes get copy-on-write access to the supervisor memory pages. My understanding is this doesn't actually yield anything on CPython since essentially everything is reference counted and memory pages get copied quickly. PyPy doesn't use reference counting though, so I was wondering if preforking could be used with PyPy for memory saving purposes. All of this is a little low-level for me, and I would appreciate any insight from the resident experts :)

2 1

_init_posix() not ever called?
by M A Sept. 21, 2021

Sept. 21, 2021

Hi I was working in the file lib-python/2.7/distutils/sysconfig_pypy.py, on the function _init_posix(). I placed print() statements in the function to indicate when this function is called. After fully building pypy the print() statements were never called. I then used grep to try to find out where this function is called in the source code. I ran this command: grep "_init_posix()" -r *. After looking at the results it looks like this function is never called by anything. Why do we have it? More importantly can we delete it? Thank you.

2 2

Changes to file not being noticed
by M A Sept. 21, 2021

Sept. 21, 2021

I recently added some functions to a file called _osx_support.py. When I imported the module and tried to use the functions I added, I see an error messages stating the name could not be imported. Would anyone know how to solve this kind of problem? Thank you.

1 0

[PATCH] Correctly set MACOSX_DEPLOYMENT_TARGET on arm, x86, and ppc
by M A Sept. 20, 2021

Sept. 20, 2021

Currently the variable MACOSX_DEPLOYMENT_TARGET was being set to 10.7 without regard to the CPU. This does not work because building a binary that targets the Mac OS 10.7 SDK is not possible on PowerPC. This value also prevents the building of an ARM native version for ARM based Macs. So we decide with architecture to use by looking at the CPU of the computer when on Darwin (Mac OS). I would like some feedback on if this patch looks good or if there a problem I'm not seeing. Thank you. --- lib-python/2.7/distutils/sysconfig_pypy.py | 28 +++++++++++++++------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/lib-python/2.7/distutils/sysconfig_pypy.py b/lib-python/2.7/distutils/sysconfig_pypy.py index ec9f5a31db..61ab3d4f7c 100644 --- a/lib-python/2.7/distutils/sysconfig_pypy.py +++ b/lib-python/2.7/distutils/sysconfig_pypy.py @@ -75,18 +75,28 @@ def _init_posix(): g['VERSION'] = get_python_version() if sys.platform[:6] == "darwin": - import platform - if platform.machine() == 'i386': - if platform.architecture()[0] == '32bit': - arch = 'i386' - else: - arch = 'x86_64' + _, _, _, kernel_string, _, = os.uname() + begin = kernel_string.find("_") + 1 # position of the architecture string + arch = kernel_string[begin:] + arch = arch.lower() + + if "arm64" in arch: + arch = "arm64" + g['MACOSX_DEPLOYMENT_TARGET'] = '11.0' + elif "x86_64" in arch: + arch = "x86_64" + g['MACOSX_DEPLOYMENT_TARGET'] = '10.5' + elif "i386" in arch: + arch = "i386" + g['MACOSX_DEPLOYMENT_TARGET'] = '10.4' + elif "ppc" in arch: + arch = "ppc" + g['MACOSX_DEPLOYMENT_TARGET'] = '10.1' else: - # just a guess - arch = platform.machine() + raise Exception("Error in sysconfig_pypy.py: unknown architecture encountered: " + arch) + g['LDSHARED'] += ' -undefined dynamic_lookup' g['CC'] += ' -arch %s' % (arch,) - g['MACOSX_DEPLOYMENT_TARGET'] = '10.7' # pypy only: give us control over the ABI tag in a wheel name if '__pypy__' in sys.builtin_module_names: -- 2.24.3 (Apple Git-128)

2 3

PyPy v7.3.6rc1 is available for testing
by Matti Picus Sept. 14, 2021

Sept. 14, 2021

The release candidates for pypy v7.3.6rc1 for python2.7, python3.7, and python3.8 are up. This is our first release of 3.8. This release also includes a backend for HPy0.0.2 (may be upgraded to 0.0.3 by the final release). This release has a number of enhancements and speed-ups, as well as the usual bug fixes. The win64 rollout on conda is going well, over 600 packages are already built. Many of the bug fixes in this version were discovered while building those packages. Thanks to all who contributed code, comments, or otherwise support PyPy. The downloads are at https://downloads.python.org/pypy/ and the checksums https://www.pypy.org/checksums.html Please try them out. The directory layout of the 3.8 release has changed, so I hope it still builds c-extension modules on your platform. The release notice https://doc.pypy.org/en/latest/release-v7.3.6.html As always, edits are welcome. Matti

4 4