Mailman 3 July 2021 - pypy-dev

PyPy Winter Sprint + cpyext
by Stefan Beyer Dec. 21, 2022

Dec. 21, 2022

Hey everyone! Those of you who attended or followed one of the last two winter sprints in Leysin might know me, the others probably won't. I'm (still) a Master's student writing my thesis on PyPy's current issue with cross-heap cycles when using cpyext. The main point is, that they are not collected and stay around as floating garbage, even if they are not reachable any more. Correct me if I'm wrong, but I didn't notice anyone else working on that topic (and/or committing something) since I've picked it up two years ago (yeah, I know, that long ago...). I saw that you would be working on "cpyext performance and completeness" during the current winter sprint this week and thought that this might also concern my thesis. So I thought I'll give you an update. I recently pushed my current (non-optimized, breaking-some-tests, but more or less working) implementation of the "CPython-style" GC-extension for cpyext. It is still in an early stage and not all cases (legacy and non-legacy-finalizers, weakrefs) are handled. However, I picked up some pace during the last couple of weeks and I'm determined to finish this implementation during the following weeks (some quirks with the tests have been haunting me, but I think I figured most of them out by now). After that, I will implement a second alternative implementation (to compare to, mostly for the sake of my thesis), which will take some more time. I also added a couple of fancy test cases (the so called "dot-tests"), so that we can test complex object graphs a little bit easier (and also because it was kind of cool to have another language inside PyPy/RPython, even if it was only for the tests...), with more test cases to come (the current ones are a bit messy). None of the new changes concerning low-latency applications are currently integrated, but that should not be too hard. I guess it won't make much sense for me to join you at this year's winter sprint, especially as I won't be able to get there before Thursday, but I might be able to join you over the IRC channel (or some other form of communication if you like). If there is anything that is worth discussing please let me know! Also feel free to comment on my code, but beware that I might change some things once I try to do some optimizations (probably not many, but at least fix the worst issues) and make it a little bit more readable. You can find my code on the cpyext-gc-cycle branch.<https://bitbucket.org/pypy/pypy/commits/branch/cpyext-gc-cycle> Looking forward to hearing from you! Greetings, Stefan

2 2

Python 3.7.10[pypy-7.3.5-final] OperationError in multiprocessing
by Amirouche July 18, 2021

July 18, 2021

I met the following traceback: Traceback (most recent call last): File "/usr/lib/pypy3/lib-python/3/concurrent/futures/process.py", line 102, in _python_exit thread_wakeup.wakeup() File "/usr/lib/pypy3/lib-python/3/concurrent/futures/process.py", line 90, in wakeup self._writer.send_bytes(b"") File "/usr/lib/pypy3/lib-python/3/multiprocessing/connection.py", line 183, in send_bytes self._check_closed() File "/usr/lib/pypy3/lib-python/3/multiprocessing/connection.py", line 136, in _check_closed raise OSError("handle is closed") OSError: handle is closed debug: OperationError: debug: operror-type: OSError debug: operror-value: handle is closed The code is: @pytest.mark.asyncio async def test_peace_search_store(): import multiprocessing from found import pstore from concurrent import futures db = await open() with futures.ProcessPoolExecutor(max_workers=multiprocessing.cpu_count()) as pool: store = pstore.make('test-pstore', (42,), pool) DOC0 = dict( foundationdb=1, okvs=2, database=42, ) DOC1 = dict( sqlite=1, sql=2, database=3 ) DOC2 = dict( spam=42, ) for uid, doc in enumerate((DOC0, DOC1, DOC2)): await found.transactional(db, pstore.index, store, uid, doc) expected = [(0, 1)] out = await found.transactional(db, pstore.search, store, ["foundationdb"], 10) assert out == expected expected = [(2, 42)] out = await found.transactional(db, pstore.search, store, ["spam"], 10) assert out == expected expected = [(0, 42), (1, 3)] out = await found.transactional(db, pstore.search, store, ["database"], 10) assert out == expected Also pstore.search will call the following pool_for_each_par_map: from aiostream import pipe, stream import asyncio async def pool_for_each_par_map(loop, pool, f, p, iterator): zx = stream.iterate(iterator) zx = zx | pipe.map(lambda x: loop.run_in_executor(pool, p, x)) async with zx.stream() as streamer: limit = pool._max_workers unfinished = [] while True: tasks = [] for i in range(limit): try: task = await streamer.__anext__() except StopAsyncIteration: limit = 0 else: tasks.append(task) tasks = tasks + list(unfinished) assert len(tasks) <= pool._max_workers if not tasks: break finished, unfinished = await asyncio.wait( tasks, return_when=asyncio.FIRST_COMPLETED ) for finish in finished: out = finish.result() f(out) limit = len(finished) You can reproduce with the following cli dance: git clone --branch=rework https://github.com/amirouche/asyncio-foundationdb/ cd asyncio-founndationdb make init poetry run make check I am not sure how to proceed I can try nightly builds but which ones? py3.7 oor py3.8? Feedback welcome.

2 3

Autotools integration / How to determine .so filename extension
by Roland Lutz July 13, 2021

July 13, 2021

Hi, I am currently using CPython 2 extensions and code in a project that's mostly written in C. I've been moving the extensions from the CPython API to PyPy/CFFI; but now I'm struggling with integrating the CFFI extensions into the Autotools build process. With CPython, there is the autoconf macro `AM_PATH_PYTHON' for determining the interpreter path and module directories[0]. Is there anything similar for PyPy? In order to build and install the extension modules correctly, I need to find out the appropriate (1) .so filename extension, (2) compiler flags, (3) linker flags, and (4) installation directory. Finding out the flags works the same as it does in CPython 3, >>>> from distutils.ccompiler import new_compiler >>>> from distutils.sysconfig import customize_compiler >>>> c = new_compiler() >>>> customize_compiler(c) >>>> c.compiler_so[1:] ['-pthread', '-DNDEBUG', '-O2', '-fPIC'] >>>> c.linker_so[1:] ['-pthread', '-shared'] but finding out the .so filename >>> import _imp >>> _imp.extension_suffixes()[0] '.cpython-37m-x86_64-linux-gnu.so' doesn't work with PyPy: >>>> import _imp Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named _imp How do I determine the correct filename extension for PyPy extension modules? Roland [0] https://www.gnu.org/software/automake/manual/html_node/Python.html

3 4

Pandas/Numpy in Ubuntu
by Александр Рауд July 12, 2021

July 12, 2021

Hi! I am looking for support to make Pandas working in PyPy3 under Ubuntu. Is there a BKM for this? pypy3 -m pip install pandas fails with: /build_src.py", line 377, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 422, in generate_config_h moredefs, ignored = cocache.check_types(config_cmd, ext, build_dir) File "numpy/core/setup.py", line 48, in check_types out = check_types(*a, **kw) File "numpy/core/setup.py", line 281, in check_types "install {0}-dev|{0}-devel.".format(python)) SystemError: Cannot compile 'Python.h'. Perhaps you need to install pypy-dev|pypy-devel. the python-dev is installed: ~# locate Python.h /usr/include/python2.7/Python.h /usr/include/python3.8/Python.h

4 5

Re: [pypy-dev] Subscribed to mailing list, but don't see how to submit a question
by Armin Rigo July 6, 2021

July 6, 2021

Hi, You write to the mailing list by writing to pypy-dev(a)python.org. Your question: How do you break into running pypy3 code, to stop execution? Like CPython, it's supposed to be ctrl-c or Ctrl-break. Maybe it didn't work on some older versions on windows. If you are using windows, try upgrading to the latest version. A bientôt, Armin >

3 2