Mailman 3 March 2016 - pypy-dev

Interest in GSoC project: UTF-8 internal unicode storage
by Piotr Jurkiewicz 08 Mar '16

08 Mar '16

Hi PyPy devs, my name is Piotr Jurkiewicz and I am a first-year PhD student at the AGH University of Science and Technology, Kraków, Poland. I am writing this email to make sure that PyPy is going to participate in GSoC 2016, since I am interested in one of the proposed projects: Optimized Unicode Representation Below is a list of my ideas and plan for the project. (I use Python 2 nomenclature, that is unicode strings are `unicode` objects and bytes strings are `str` objects.) 1. Store all unicode objects contents internally as UTF-8. This would reduce size of stored contents and allow external libraries, which expect UTF-8, to process contents directly in the memory (for example using various regexp libraries to search unicode string). 2. Unify interning caches for str and unicode. This would allow unicode objects and corresponding utf8-encoded-str objects to share the same interned buffer. For example unicode object u'koń' would share interned buffer with str 'ko\xc5\x84'. This would make unicode.encode('utf-8') basically no op. As UTF-8 becomes dominant encoding for any data exchange, including web (86%) [1], more and more data coming out from Python scripts needs to be UTF-8 encoded. Therefore, it is important to make this operation as cheap as possible. It would speed up str.decode('utf-8') significantly too, although it wouldn't make it no op. String still would need to be checked if it is a correct UTF-8 string when transforming to unicode object. But we can get rid of additional allocation, copying string contents and storing it twice, in CONST_STR_CACHE and CONST_UNICODE_CACHE. 3. Indexing of codepoints positions, what would allow O(1) random access and slicing. The idea is simple: alongside contents of each interned unicode object, store an array of unsigned integers. These integers will be positions (in bytes), counting from the beginning of the buffer, at which each next 64-codepoint-long 'pages' start. Random access would be as follows: page_num, byte_in_page = divmod(codepoint_pos, 64) page_start_byte = index[page_num] exact_byte = seek_forward(buffer[page_start_byte], byte_in_page) return buffer[exact_byte] Using 64-byte long pages, like in the example above, would allow O(1) random access, with constant terms of: - one cache access in cases of only-ASCII texts (indexes for such unicode objects will not be created and maintained) - three cache accesses in cases of texts consisting of ASCII mixed with two-byte characters (Latin, Greek, Cyrillic, Hebrew, Arabic alphabets) - four or five cache accesses in cases of texts consisting mostly of three- and four- byte characters (all above assuming 64-byte long CPU cache lines) Memory overhead associated with storing index array would be in range 0 - 6.25%. (or 0 - 12.5% if unicode objects longer than 2^32 codepoints will be allowed) (assuming that the index array consists of integers of smallest possible type which can store buffer_bytes_len - 1) 4. Fast codepoints counting/seeking with branchless algorithm [2]. When unicode object is interned, we are sure that it is a correct UTF-8 string. Therefore, there is no need for correctness checking when seeking, so a branchless algorithm can be used. [1]: http://w3techs.com/technologies/details/en-utf8/all/all [2]: http://blogs.perl.org/users/nick_wellnhofer/2015/04/branchless-utf-8-length… All of these changes can be introduced one at a time, what would improve tracking of performance changes and debugging of eventual errors. After completing the project I plan to write a paper describing speedup method of random access unicode access based on indexing, as this method has a potential for being used in other language interpreters which have immutable and/or interned unicode strings. Note that similar index can be created for graphemes as well, so this method can be used in languages which provide grapheme-based interface (like Perl 6). Please share your thoughts about these ideas. Cheers, Piotr

7 13

Release 5.0.0
by Matti Picus 07 Mar '16

07 Mar '16

Pre-release bundles are up on the buildbot, http://buildbot.pypy.org/nightly/release-5.x please test them out. There are still a few last touches pending, but it would be nice to have some preliminary indication whether the bundles work in real-life work loads and bugs that we claim we fixed since 4.0.1 actually do not reappear. Also the release notice is up at https://bitbucket.org/pypy/pypy/src/default/pypy/doc/release-5.0.0.rst Any help with it would be appreciated Matti

4 7

Making Pyrasite work with PyPy
by Tin Tvrtković 06 Mar '16

06 Mar '16

Hello, in case you haven't heard of it, Pyrasite (https://github.com/lmacken/pyrasite) is a tool for injecting code into running Python processes. Personally I have found it invaluable for forensics on services running in production and have successfully solved memory leaks, connection leaks and deadlocks with it. One of the payloads provided will open a remote REPL right in a running process, without the process having *any* preparation logic in it. I think this is extremely powerful and makes Python catch and up even surpass Java (which has automatic stack trace dumping on SIGQUIT and useful tools like JConsole and VisualVM that can connect to running processes, again by default with no setup in the process) for these kinds of things. Anyway, Pyrasite uses gdb under the hood; gdb will attach to a running process and inject the following: gdb_cmds = [ 'PyGILState_Ensure()', 'PyRun_SimpleString("' 'import sys; sys.path.insert(0, \\"%s\\"); ' 'sys.path.insert(0, \\"%s\\"); ' 'exec(open(\\"%s\\").read())")' % (os.path.dirname(filename), os.path.abspath(os.path.join(os.path.dirname(__file__), '..')), filename), 'PyGILState_Release($1)', ] If I change the Py* functions to PyPy* (PyRun_SimpleString to PyPyRun_SimpleString), this seems to work just fine on PyPy too. This is great, and now I'd like to contribute back to Pyrasite and get PyPy support in there. It'd be great if Pyrasite could automatically detect if the underlying process is CPython or PyPy, so since my experience working on the C level is very basic, I'm asking you, the PyPy devs, if there's a good way of detecting a process is PyPy given its PID and gdb's ability of attaching to a process and doing gdb things. Worst case scenario, gdb supports "info functions", which is how I found the PyPy functions in the first place, but is there a better way? I apologize if this is off-topic for PyPy-dev.

2 1

CFP: ICOOOLPS'16: Workshop on Implementation, Compilation, Optimization of OO Languages, Programs and Systems
by Edd Barrett 03 Mar '16

03 Mar '16

May be of interest to some of the members on this list: Call for Papers: ICOOOLPS’16 ============================ 11th Workshop on Implementation, Compilation, Optimization of OO Languages, Programs and Systems Co-located with ECOOP July 18, 2016, Rome, Italy URL: http://2016.ecoop.org/track/ICOOOLPS-2016 Twitter: @ICOOOLPS The ICOOOLPS workshop series brings together researchers and practitioners working in the field of language implementation and optimization. The goal of the workshop is to discuss emerging problems and research directions as well as new solutions to classic performance challenges. The topics of interest for the workshop include techniques for the implementation and optimization of a wide range of languages including but not limited to object-oriented ones. Furthermore, meta-compilation techniques or language-agnostic approaches are welcome, too. A non-exclusive list of topics follows: - implementation and optimization of fundamental languages features (from automatic memory management to zero-overhead metaprogramming) - runtime systems technology (libraries, virtual machines) - static, adaptive, and speculative optimizations and compiler techniques - meta-compilation techniques and language-agnostic approaches for the efficient implementation of languages - compilers (intermediate representations, offline and online optimizations,...) - empirical studies on language usage, benchmark design, and benchmarking methodology - resource-sensitive systems (real-time, low power, mobile, cloud) - studies on design choices and tradeoffs (dynamic vs. static compilation, heuristics vs. programmer input,...) - tooling support, debuggability and observability of languages as well as their implementations ### Workshop Format and Submissions This workshop welcomes the presentation and discussion of new ideas and emerging problems that give a chance for interaction and exchange. More mature work is welcome as part of a mini-conference format, too. We aim to interleave interactive brainstorming and demonstration sessions between the formal presentations to foster an active exchange of ideas. The workshop papers will be published either in the ACM DL or in the Dagstuhl LIPIcs ECOOP Workshop proceedings. Until further notice, please use the ACM SIGPLAN template with a 10pt font size: http://www.sigplan.org/Resources/Author/ - position and work-in-progress paper: 1-4 pages - technical paper: max. 10 pages - demos and posters: 1-page abstract For the submission, please use the HotCRP system: http://ssw.jku.at/icooolps/ ### Important Dates - abstract submission: April 11, 2016 - paper submission: April 15, 2016 - notification: May 13, 2016 - all deadlines: Anywhere on Earth (AoE), i.e., GMT/UTC−12:00 hour - workshop: July 18th, 2016 ### Program Committee Edd Barrett, King’s College London, UK Clement Bera, Inria Lille, France Maxime Chevalier-Boisvert, Université de Montréal, Canada Tim Felgentreff, Hasso Plattner Institute, Germany Roland Ducournau, LIRMM, Université de Montpellier, France Elisa Gonzalez Boix, Vrije Universiteit Brussel, Belgium David Gregg, Trinity College Dublin, Ireland Matthias Grimmer, Johannes Kepler University Linz, Austria Michael Haupt, Oracle, Germany Richard Jones, University of Kent, UK Tomas Kalibera, Northeastern University, USA Hidehiko Masuhara, Tokyo Institute of Technology, Japan Tiark Rompf, Purdue University, USA Jennifer B. Sartor, Ghent University, Belgium Sam Tobin-Hochstadt, Indiana University, USA ### Workshop Organizers Stefan Marr, Johannes Kepler University Linz, Austria Eric Jul, University of Oslo, Norway For questions or concerns, please mail to stefan.marr at jku.at or contact us via https://twitter.com/icooolps. -- Best Regards Edd Barrett http://www.theunixzoo.co.uk

1 0

GSoC 2016
by Richard Plangger 02 Mar '16

02 Mar '16

Hi, I was wondering who applied as a sub org to python last year? The registration for new sub orgs is open until March 7th. (https://wiki.python.org/moin/SummerOfCode/2016#Sub-orgs) As we discussed on the sprint I will try to attract some students tomorrow at the university in Vienna. Of course I'm also willing to mentor if there is a good proposal. Cheers, Richard

2 1