Mailman 3 March 2019 - pypy-dev

PyPy Winter Sprint + cpyext
by Stefan Beyer Dec. 21, 2022

Dec. 21, 2022

Hey everyone! Those of you who attended or followed one of the last two winter sprints in Leysin might know me, the others probably won't. I'm (still) a Master's student writing my thesis on PyPy's current issue with cross-heap cycles when using cpyext. The main point is, that they are not collected and stay around as floating garbage, even if they are not reachable any more. Correct me if I'm wrong, but I didn't notice anyone else working on that topic (and/or committing something) since I've picked it up two years ago (yeah, I know, that long ago...). I saw that you would be working on "cpyext performance and completeness" during the current winter sprint this week and thought that this might also concern my thesis. So I thought I'll give you an update. I recently pushed my current (non-optimized, breaking-some-tests, but more or less working) implementation of the "CPython-style" GC-extension for cpyext. It is still in an early stage and not all cases (legacy and non-legacy-finalizers, weakrefs) are handled. However, I picked up some pace during the last couple of weeks and I'm determined to finish this implementation during the following weeks (some quirks with the tests have been haunting me, but I think I figured most of them out by now). After that, I will implement a second alternative implementation (to compare to, mostly for the sake of my thesis), which will take some more time. I also added a couple of fancy test cases (the so called "dot-tests"), so that we can test complex object graphs a little bit easier (and also because it was kind of cool to have another language inside PyPy/RPython, even if it was only for the tests...), with more test cases to come (the current ones are a bit messy). None of the new changes concerning low-latency applications are currently integrated, but that should not be too hard. I guess it won't make much sense for me to join you at this year's winter sprint, especially as I won't be able to get there before Thursday, but I might be able to join you over the IRC channel (or some other form of communication if you like). If there is anything that is worth discussing please let me know! Also feel free to comment on my code, but beware that I might change some things once I try to do some optimizations (probably not many, but at least fix the worst issues) and make it a little bit more readable. You can find my code on the cpyext-gc-cycle branch.<https://bitbucket.org/pypy/pypy/commits/branch/cpyext-gc-cycle> Looking forward to hearing from you! Greetings, Stefan

2 2

Continual Increase In Memory Utilization Using PyPy 6.0 (python 2.7) -- Help!?
by Robert Whitcher March 28, 2019

March 28, 2019

So I have a process that use PyPy and pymongo in a loop. It does basically the same thing every loop, which query a table in via pymongo and do a few non-save calculations and then wait and loop again The RSS of the process continually increased (the PYPY_GC_MAX is set pretty high). So I hooked in the GC stats output per: http://doc.pypy.org/en/latest/gc_info.html I also assure that gc.collect() was called at least every 3 minutes. What I see is that... The memory while high is fair constant for a long time: 2019-03-27 00:04:10.033-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 144244736 ... 2019-03-27 01:01:46.841-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 144420864 2019-03-27 01:02:36.943-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 144269312 Then it decides (an the exact per-loop behavior is the same each time) to chew up much more memory: 2019-03-27 01:04:17.184-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 145469440 2019-03-27 01:05:07.305-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 158175232 2019-03-27 01:05:57.401-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 173191168 2019-03-27 01:06:47.490-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 196943872 2019-03-27 01:07:37.575-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 205406208 2019-03-27 01:08:27.659-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 254562304 2019-03-27 01:09:17.770-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 256020480 2019-03-27 01:10:07.866-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 289779712 That's 140 MB .... Where is all that memory going... What's more is that the PyPy GC stats do not show anything different: Here are the GC stats from GC-Complete when we were at *144MB*: 2019-03-26 23:55:49.127-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 140632064 2019-03-26 23:55:49.133-0600 [-] main_thread(29621)log (async_worker_process.py:308): DBG0: Total memory consumed: GC used: 56.8MB (peak: 69.6MB) in arenas: 39.3MB rawmalloced: 14.5MB nursery: 3.0MB raw assembler used: 521.6kB ----------------------------- Total: 57.4MB Total memory allocated: GC allocated: 63.0MB (peak: 71.2MB) in arenas: 43.9MB rawmalloced: 22.7MB nursery: 3.0MB raw assembler allocated: 1.0MB ----------------------------- Total: 64.0MB Here are the GC stats from GC-Complete when we are at *285MB*: 2019-03-27 01:42:41.751-0600 [-] main_thread(29621)log (async_worker_process.py:304): INFO_FLUSH: RSS: 285147136 2019-03-27 01:42:41.751-0600 [-] main_thread(29621)log (async_worker_process.py:308): DBG0: Total memory consumed: GC used: 57.5MB (peak: 69.6MB) in arenas: 39.9MB rawmalloced: 14.6MB nursery: 3.0MB raw assembler used: 1.5MB ----------------------------- Total: 58.9MB Total memory allocated: GC allocated: 63.1MB (peak: 71.2MB) in arenas: 43.9MB rawmalloced: 22.7MB nursery: 3.0MB raw assembler allocated: 2.0MB ----------------------------- Total: 65.1MB How is this possible? I am measuring RSS with: def get_rss_mem_usage(): ''' Get the RSS memory usage in bytes @return: memory size in bytes; -1 if error occurs ''' try: process = psutil.Process(os.getpid()) return process.get_memory_info().rss except: return -1 And cross referencing with "ps -orss -p <pid>" and the RSS values reported are correct.... I cannot figure out where to go from here with this as it appears that PyPy is leaking this memory somehow... And I have no idea howto proceed from here... I end up having memory problems and getting Memory Warnings for a process that just loops and queries via pymongo Pymongo version is 3.7.1 This is: Python 2.7.13 (ab0b9caf307db6592905a80b8faffd69b39005b8, Apr 30 2018, 08:21:35) [PyPy 6.0.0 with GCC 7.2.0]

3 3

[CfP] DLS 2019 - 15th Dynamic Languages Symposium, co-located with SPLASH 2019 Call for Papers
by Carl Friedrich Bolz-Tereick March 23, 2019

March 23, 2019

DLS 2019 - 15th Dynamic Languages Symposium Co-located with SPLASH 2019, October 22, Athens, Greece https://conf.researchr.org/home/dls-2019 Follow us @dynlangsym Dynamic Languages play a fundamental role in today’s world of software, from the perspective of research and practice. Languages such as JavaScript, R, and Python are vehicles for cutting edge research as well as building widely used products and computational tools. The 15th Dynamic Languages Symposium (DLS) at SPLASH 2019 is the premier forum for researchers and practitioners to share research and experience on all aspects on dynamic languages. DLS 2019 invites high quality papers reporting original research and experience related to the design, implementation, and applications of dynamic languages. Areas of interest are generally empirical studies, language design, implementation, and runtimes, which includes but is not limited to: - innovative language features - innovative implementation techniques - innovative applications - development environments and tools - experience reports and case studies - domain-oriented programming - late binding, dynamic composition, and run-time adaptation - reflection and meta-programming - software evolution - language symbiosis and multi-paradigm languages - dynamic optimization - interpretation, just-in-time and ahead-of-time compilation - soft/optional/gradual typing - hardware support - educational approaches and perspectives - semantics of dynamic languages - frameworks and languages for the Cloud and the IoT Submission Details Submissions must neither be previously published nor under review at other events. DLS 2019 uses a single-blind, two-phase reviewing process. Papers are assumed to be in one of the following categories: Research Papers: describe work that advances the current state of the art Experience Papers: describe insights gained from substantive practical applications that should be of a broad interest Dynamic Pearls: describe a known idea in an appealing way to remind the community and capture a reader’s interest The program committee will evaluate each paper based on its relevance, significance, clarity, and originality. The paper category needs to be indicated during submission, and papers are judged accordingly. Papers are to be submitted electronically at https://dls19.hotcrp.com/ in PDF format. Submissions must be in the ACM SIGPLAN conference acmart format, 10 point font, and should not exceed 12 pages. Please see full details in the instructions for authors available at: https://conf.researchr.org/home/dls-2019#Instructions-for-Authors DLS 2019 will run a two-phase reviewing process to help authors make their final papers the best that they can be. Accepted papers will be published in the ACM Digital Library and will be freely available for one month, starting two weeks before the event. Important Deadlines Abstract Submission: May 29, 2019 Paper Submission: June 5, 2019 First Phase Notification: July 3, 2019 Final Notifications: August 14, 2019 Camera Ready: August 28, 2019 All deadlines are 23:59 AoE (UTC-12h). AUTHORS TAKE NOTE: The official publication date is the date the proceedings are made available in the ACM Digital Library. This date may be up to two weeks prior to the first day of your conference. The official publication date affects the deadline for any patent filings related to published work. Program CommitteeAlexandre Bergel, University of Chile Carl Friedrich Bolz-Tereick, Heinrich-Heine-Universität Düsseldorf Chris Seaton, Oracle Labs David Chisnall, Microsoft Research Elisa Gonzalez Boix, Vrije Universiteit Brussel Gregor Richards, University of Waterloo Guido Chari, Czech Technical University Hannes Payer, Google James Noble, Victoria University of Wellington Jeremy Singer, University of Glasgow Joe Gibbs Politz, University of California San Diego Juan Fumero, The University of Manchester Julien Ponge, Red Hat Mandana Vaziri, IBM Research Manuel Serrano, Inria Marc Feeley, Université de Montréal Mark Marron, Microsoft Research Na Meng, Virginia Tech Nick Papoulias, Université Grenoble Alpes Richard Roberts, Victoria University of Wellington Sam Tobin-Hochstadt, Indiana University Sarah Mount, Aston University Sébastien Doeraene, École polytechnique fédérale de Lausanne William Cook, University of Texas at Austin Program Chair Stefan Marr, University of Kent, United Kingdom

1 0

Some questions about using rpython
by Matthew Francis-Landau March 23, 2019

March 23, 2019

Hello, I am considering using rpython to optimize some interpreter like code and I have a few questions that I couldn't quite figure out from pypy's source. 1. Is it possible to call something written in rpython from "normal python" (either running on cPython or PyPy)? Ideally, I would like something where objects could be shared between the two environments. Can I compile something using rpython and then turn that into a C compiled python module that can share objects with the interpreter. 2. How much control can I have over the tracing compiler, eg is it possible to forcing something to be traced the first time it is run? 3. If I am inside of a trace, can I add my own guards or conditionally aborting trace and returning to an interpreted version of the code. I am going to have some code like: if self.method is None: # this branch is only taken at most once for this object # Abort the trace??? self.method = build_method() # marked as an elidable method? self.method(....) 4. How can I use threads from inside rpython? What considerations do I need to make when interacting with the JitDriver. What sorts of atomic operations are available when interfacing with my data structures. What considerations would I need to make to work with rpython's GC. Can I release "normal python's" GIL when writing threaded code from rpython? Also, is software transactional memory usable by things that are not PyPy? It looks like there where a few branches that haven't been merged. 5. Inside of a trace, I would like to be able to look at mutable objects (namely the entire current stack frame) from an elidable methods and have the result be cached without a need for any guards. This is something that I know will be safe due to how the program was written, but not something I would expect rpython to be able to deduce from the context of a trace. How can I annotate a method to be able to do this? Thanks, Matthew -- Matthew Francis-Landau

1 0

Downloads for release v7.1 of pypy2.7 and pypy3.6 are on bitbucket
by Matti Picus March 22, 2019

March 22, 2019

1 1

PyPy as micropython Replacement
by Mark Melvin March 19, 2019

March 19, 2019

Hi All, I was wondering if PyPy would be a good candidate to create an embedded distribution of Python similar to micropython. Do you think it would be feasible to create an installation (stripped of things like Tkinter and other unneeded modules) that could execute with as little as say 2-4 MB of heap on an ARM7 running Linux? Thanks, Mark

2 2

pypy configured Cloud host
by Sunil Guglani March 18, 2019

March 18, 2019

Hi, I am facing issues in installing/configuring pypy with other libraries..Will you pls suggest any cloud hosting service provider, which gives pypy preconfigured environment? Thanks, Sunil

1 0

Pypy friendly to get pointer enclosed in python object ?
by Stuart Axon March 17, 2019

March 17, 2019

PyCairo has a struct like ` typedef struct { | PyObject_HEAD | | | cairo_t *ctx; | | | PyObject *base; /* base object used to create context, or NULL */ | | } PycairoContext; ` And a #define that you can use to get pointer to ctx: ` #define PycairoContext_GET(obj) (((PycairoContext *)(obj))->ctx)` How can I get hold of the pointer *ctx in Pypy ? I had a bit of a look at CFFI, but couldn't find anything using structures with PyObject_HEAD in them. S++

1 1

Updating packages.pypy.org
by Matti Picus March 7, 2019

March 7, 2019

I have updated the repo behind http://packages.pypy.org (forked to https://github.com/pypy/pypy.packages): - use the latest pypy3.6 (v7.1 from two days ago), which has the utf8 improvements - PyPI changed their interface, so I had to manually create a download.csv with the top 1000 packages from the last 12 months - updated the docker image to use utf-8 as the filesystemencoding so we don't get strange ascii encoding errors when reading files with unicode With these changes, we successfully can install almost all the packages. OpenCV, Tensorflow and apache-beam, which all require complicated install procedures that seem to be beyond the ability of a simple "pip install", fail, as do some python2-only packages that are still in the top 1000. Next steps: - Release 7.1 once the newmemoryview-app-level branch lands (it allows creating memoryviews of ctypes structures, passing more tests on numpy's latest version) - Rerun the repo with an official build - Update the web site.

1 0