Mailman 3 July 2017 - pypy-dev

About gurobipy
by Luis José Novoa 31 May '18

31 May '18

Hi All, My name is Luis Novoa. Im working on a large scale optimization problem using python/gurobipy for my implementation. I was wondering if any of you in this list has used gurobipy with pypy, or if it is even possible at this time. Thanks in advance, Luis.

9 9

PhD Positions in programming language runtimes (Python)
by David Bremner 02 May '18

02 May '18

1 Position I: Python on OMR =========================== The candidate will explore the benefits and challenges of using OMR [1] components (e.g. JIT, GC) in Python [2,3] runtimes. This may involve extending OMR in novel ways to better support Python, or vice-versa. If questions like how to balance fast startup time for scripting applications against good performance for compute-intensive workloads sound interesting, this might an opportunity for you. 2 Position II: Python and PGAS ============================== The candidate will implement support for parallel programming with Python, using the PGAS [4] (parallel global address space) model. The PGAS model can significantly simplify the implementation of parallel algorithms on distributed and shared memory computers. Challenges include enabling parallelism while dealing with Python's Global Interpreter Lock (GIL), and implementing distributed garbage collection efficiently. 3 Qualifications ================ Normally applicants to the Computer Science PhD program at UNB require a Master's degree in computer science or a closely related area. This project will require substantial hands-on and experimental work, and being comfortable with Python and C++. Familiarity with Linux and Open Source Software development would be a plus. For position I, exposure to programming language concepts like GC, JIT, and bytecode would be helpful. For position II, experience with parallel programming frameworks and/or languages such as MPI, OpenMP, X10 is an asset. 4 Environment ============= This project is supported by the IBM Centre for Advanced Studies (Atlantic) at the University of New Brunswick [5]. We have a lively group of students, postdocs, and research assistants working on related problems. Students work closely with both UNB researchers and technical experts from IBM, gaining valuable perspective and experience. 5 Support ========= These positions are "fully funded"; the exact funding package is subject to finalization but will be consistent with other fully funded students in Computer Science at UNB. You can find some information about expenses on the UNB web site [6]. 6 Contact ========= Send your resumé and unofficial transcripts by email to Stephen MacKay <smackay3(a)unb.ca> Please specify which position(s) you are interested in. [1]: [http://www.eclipse.org/omr/] [2]: [http://www.python.org] [3]: [http://www.pypy.org] [4]: [http://www.pgas.org/] [5]: [http://www.unb.ca/research/casatlantic/] [6]: [http://www.unb.ca/fredericton/studentservices/international/new_internation…]

1 1

Pypy3 lxml support - please help
by Michal Moiseev 30 Jul '17

30 Jul '17

Hi, In our firm we are developing on python 3.4, and from performance issues, we have been trying to see if we we can run our code with pypy3, I couldn't seem to fine a solution for the lxml package, the pypy implementation - lxml-cffi is only for python 2.x, and when installing lxml, it is also seems to work only for python 2.x, and the following error is printed when trying to import: from lxml import etree File "src/lxml/xmlschema.pxi", line 22, in init lxml.etree (src/lxml/lxml.etree.c:231895) File "src/lxml/xpath.pxi", line 420, in lxml.etree.XPath.__init__ (src/lxml/lxml.etree.c:173999) File "src/lxml/xpath.pxi", line 150, in lxml.etree._XPathEvaluatorBase.set_context (src/lxml/lxml.etree.c:170398) File "src/lxml/xpath.pxi", line 66, in lxml.etree._XPathContext.set_context (src/lxml/lxml.etree.c:169176) File "src/lxml/extensions.pxi", line 255, in lxml.etree._BaseContext.registerLocalFunctions (src/lxml/lxml.etree.c:160400) AttributeError: 'dict' object has no attribute 'iteritems' thank you! --

2 1

Change function closure cell contents
by Tin Tvrtković 27 Jul '17

27 Jul '17

Hello! Here's a very simple example (Python 3): >>>> class C: .... def t(self): .... return __class__ .... >>>> C.t.__closure__[0] <cell at 0x0000000109eca0c0: type object at 0x00007f7f82c225b0> My question is, what's the easiest/most natural way of changing the contents of this closure cell on PyPy? On CPython there's a way using ctypes. This is regarding https://github.com/python-attrs/attrs/issues/102, but the issue boils down to this. Cheers!

2 2

lib_pypy/_marshal.py looks out of date for Python 3
by Rocky Bernstein 26 Jul '17

26 Jul '17

A while back, in working on a cross-version python decompiler, I used PyPy's lib_pypy/_marshal.py. Recently in doing work in writing a cross-platform a Python bytecode assember, I noticed what is in the py3.5 branch is a bit out of date for Python 3. Specifically these types: TYPE_REF = 'r' # Since 3.4 TYPE_ASCII = 'a' # since 3.4 TYPE_ASCII_INTERNED = 'A' # since 3.4 TYPE_SMALL_TUPLE = ')' # since 3.4 TYPE_SHORT_ASCII = 'z' # since 3.4 TYPE_SHORT_ASCII_INTERNED = 'Z' # since 3.4 In xdis, the cross-version marshal/unmarshal and opcode routines, I've started adding some of these. Is lib_pypy/_marshal.py used? If so, shouldn't this be corrected?

2 1

Need to rebuild wheels for every pypy minor version
by Daniele Rolando 25 Jul '17

25 Jul '17

Hi guys. Right now pypy wheels names include both the major and minor pypy version in them: e.g. uWSGI-2.0.14.0.*pp257*-pypy_41-linux_x86_64.whl This means that if we want to upgrade pypy from 5.7.1 to 5.8 we'd need to rebuild all our wheels and this is not scalable since there are new pypy releases every 3/4 months. Wouldn't it be enough to only include the major version in the wheel name? Are minor pypy versions really incompatible between them? Thanks, Daniele

4 3

Re: [pypy-dev] Need to rebuild wheels for every pypy minor version
by Nathaniel Smith 25 Jul '17

25 Jul '17

On Jul 25, 2017 12:16 PM, "Antonio Cuni" <anto.cuni(a)gmail.com> wrote: Hi, note that this is not because of PyPy: it's the wheel package which chooses what to include in the wheel filename: https://bitbucket.org/pypa/wheel/src/5d49f8cf18679d1bc6f3e1e414a5df 3a1e492644/wheel/pep425tags.py?at=default&fileviewer=file- view-default#pep425tags.py-39 Well, probably it is most correct to say that this happens because PyPy has not come up with a coherent strategy for ABI compatibility and communicated that to wheel/pip :-). In the mean time, if you want to live dangerously, you can always rename your wheel something like 'foo-py2-pypy_41-linux_x86_64.whl', and then pip will happily install it on newer pypys. Whether it will work though is hard to say... -n

2 1

okay to rename cppyy -> _cppyy
by wlavrijsen＠lbl.gov 21 Jul '17

21 Jul '17

Hi, any objections to renaming cppyy into _cppyy? I want to be able to do a straight 'pip install cppyy' and then use it w/o further gymnastics (this works today for CPython), but then I can't have 'cppyy' be a built-in module. (You can pip install PyPy-cppyy-backend, but then you'd still have to deal with LD_LIBRARY_PATH and certain cppyy features are PyPy version dependent even as they need not be as they are pure Python.) The pip-installed cppyy will still use the built-in _cppyy for the PyPy specific parts (low-level manipulations etc.). I'm also moving the cppyy documentation out of the pypy documentation and placing it on its own (http://cppyy.readthedocs.io/), given that the CPython side of things now works, too. Yes, no, conditional? Thanks, Wim -- WLavrijsen(a)lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net

6 13

Link to paper
by Maciej Fijalkowski 21 Jul '17

21 Jul '17

Hi Wim There was a complaint on reddit, that, quoting: "The link to the pyhpc paper does not work for me, does anyone have a mirror? Would love to read it." can you fix the link please? Cheers, fijal

2 1

Hide implementation details of the CPython C API
by Victor Stinner 20 Jul '17

20 Jul '17

Hi, I'm not sure that PyPy developers are following the python-ideas mailing list, so here is a copy of my email. I really want your feedback to know if my idea is stupid and if it's feasible. You have a much better experience with C API compatibility than me ;-) LWN already wrote an article on the python-ideas discussion: https://lwn.net/Articles/727973/ or if you are not subscribed: https://lwn.net/SubscriberLink/727973/9b2004c4fe039833/ My PEP draft: https://github.com/haypo/misc/blob/master/python/pep_c_api.rst -- Hi, This is the first draft of a big (?) project to prepare CPython to be able to "modernize" its implementation. Proposed changes should allow to make CPython more efficient in the future. The optimizations themself are out of the scope of the PEP, but some examples are listed to explain why these changes are needed. For the background, see also my talk at the previous Python Language Summit at Pycon US, Portland OR: "Keeping Python competitive" https://lwn.net/Articles/723752/#723949 "Python performance", slides (PDF): https://github.com/haypo/conf/raw/master/2017-PyconUS/summit.pdf Since this is really the first draft, I didn't assign a PEP number to it yet. I prefer to wait for a first feedback round. Victor PEP: xxx Title: Hide implementation details of the C API Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com>, Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 31-May-2017 Abstract ======== Modify the C API to remove implementation details. Add an opt-in option to compile C extensions to get the old full API with implementation details. The modified C API allows to more easily experiment new optimizations: * Indirect Reference Counting * Remove Reference Counting, New Garbage Collector * Remove the GIL * Tagged pointers Reference counting may be emulated in a future implementation for backward compatibility. Rationale ========= History of CPython forks ------------------------ Last 10 years, CPython was forked multiple times to attempt different CPython enhancements: * Unladen Swallow: add a JIT compiler based on LLVM * Pyston: add a JIT compiler based on LLVM (CPython 2.7 fork) * Pyjion: add a JIT compiler based on Microsoft CLR * Gilectomy: remove the Global Interpreter Lock nicknamed "GIL" * etc. Sadly, none is this project has been merged back into CPython. Unladen Swallow lost its funding from Google, Pyston lost its funding from Dropbox, Pyjion is developed in the limited spare time of two Microsoft employees. One hard technically issue which blocked these projects to really unleash their power is the C API of CPython. Many old technical choices of CPython are hardcoded in this API: * reference counting * garbage collector * C structures like PyObject which contains headers for reference counting and the garbage collector * specific memory allocators * etc. PyPy ---- PyPy uses more efficient structures and use a more efficient garbage collector without reference counting. Thanks to that (but also many other optimizations), PyPy succeeded to run Python code up to 5x faster than CPython. Plan made of multiple small steps ================================= Step 1: split Include/ into subdirectories ------------------------------------------ Split the ``Include/`` directory of CPython: * ``python`` API: ``Include/Python.h`` remains the default C API * ``core`` API: ``Include/core/Python.h`` is a new C API designed for building Python * ``stable`` API: ``Include/stable/Python.h`` is the stable ABI Expect declarations to be duplicated on purpose: ``#include`` should be not used to include files from a different API to prevent mistakes. In the past, too many functions were exposed *by mistake*, especially symbols exported to the stable ABI by mistake. At this point, ``Include/Python.h`` is not changed at all: zero risk of backward incompatibility. The ``core`` API is the most complete API exposing *all* implementation details and use macros for best performances. XXX should we abandon the stable ABI? Never really used by anyone. Step 2: Add an opt-in API option to tools building packages ----------------------------------------------------------- Modify Python packaging tools (distutils, setuptools, flit, pip, etc.) to add an opt-in option to choose the API: ``python``, ``core`` or ``stable``. For example, debuggers like ``vmprof`` need need the ``core`` API to get a full access to implementation details. XXX handle backward compatibility for packaging tools. Step 3: first pass of implementation detail removal --------------------------------------------------- Modify the ``python`` API: * Add a new ``API`` subdirectory in the Python source code which will "implement" the Python C API * Replace macros with functions. The implementation of new functions will be written in the ``API/`` directory. For example, Py_INCREF() becomes the function ``void Py_INCREF(PyObject *op)`` and its implementation will be written in the ``API`` directory. * Slowly remove more and more implementation details from this API. Modifications of these API should be driven by tests of popular third party packages like: * Django with database drivers * numpy * scipy * Pillow * lxml * etc. Compilation errors on these extensions are expected. This step should help to draw a line for the backward incompatible change. Goal: remove a few implementation details but don't break numpy and lxml. Step 4 ------ Switch the default API to the new restricted ``python`` API. Help third party projects to patch their code: don't break the "Python world". Step 5 ------ Continue Step 3: remove even more implementation details. Long-term goal to complete this PEP: Remove *all* implementation details, remove all structures and macros. Alternative: keep core as the default API ========================================= A smoother transition would be to not touch the existing API but work on a new API which would only be used as an opt-in option. Similar plan used by Gilectomy: opt-in option to get best performances. It would be at least two Python binaries per Python version: default compatible version, and a new faster but incompatible version. Idea: implementation of the C API supporting old Python versions? ================================================================= Open questions. Q: Would it be possible to design an external library which would work on Python 2.7, Python 3.4-3.6, and the future Python 3.7? Q: Should such library be linked to libpythonX.Y? Or even to a pythonX.Y binary which wasn't built with shared library? Q: Would it be easy to use it? How would it be downloaded and installed to build extensions? Collaboration with PyPy, IronPython, Jython and MicroPython =========================================================== XXX to be done Enhancements becoming possible thanks to a new C API ==================================================== Indirect Reference Counting --------------------------- * Replace ``Py_ssize_t ob_refcnt;`` (integer) with ``Py_ssize_t *ob_refcnt;`` (pointer to an integer). * Same change for GC headers? * Store all reference counters in a separated memory block (or maybe multiple memory blocks) Expected advantage: smaller memory footprint when using fork() on UNIX which is implemented with Copy-On-Write on physical memory pages. See also `Dismissing Python Garbage Collection at Instagram <https://engineering.instagram.com/dismissing-python-garbage-collection-at-i…>`_. Remove Reference Counting, New Garbage Collector ------------------------------------------------ If the new C API hides well all implementation details, it becomes possible to change fundamental features like how CPython tracks the lifetime of an object. * Remove ``Py_ssize_t ob_refcnt;`` from the PyObject structure * Replace the current XXX garbage collector with a new tracing garbage collector * Use new macros to define a variable storing an object and to set the value of an object * Reimplement Py_INCREF() and Py_DECREF() on top of that using a hash table: object => reference counter. XXX PyPy is only partially successful on that project, cpyext remains very slow. XXX Would it require an opt-in option to really limit backward compatibility? Remove the GIL -------------- * Don't remove the GIL, but replace the GIL with smaller locks * Builtin mutable types: list, set, dict * Modules * Classes * etc. Backward compatibility: * Keep the GIL Tagged pointers --------------- https://en.wikipedia.org/wiki/Tagged_pointer Common optimization, especially used for "small integers". Current C API doesn't allow to implement tagged pointers. Tagged pointers are used in MicroPython to reduce the memory footprint. Note: ARM64 was recently extended its address space to 48 bits, causing issue in LuaJIT: `47 bit address space restriction on ARM64 <https://github.com/LuaJIT/LuaJIT/issues/49>`_. Misc ideas ---------- * Software Transactional Memory? See `PyPy STM <http://doc.pypy.org/en/latest/stm.html>`_ Idea: Multiple Python binaries ============================== Instead of a single ``python3.7``, providing two or more binaries, as PyPy does, would allow to experiment more easily changes without breaking the backward compatibility. For example, ``python3.7`` would remain the default binary with reference counting and the current garbage collector, whereas ``fastpython3.7`` would not use reference counting and a new garbage collector. It would allow to more quickly "break the backward compatibility" and make it even more explicit than only prepared C extensions will be compatible with the new ``fastpython3.7``. cffi ==== XXX Long term goal: "more cffi, less libpython". Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

1 0