Mailman 3 April 2014 - Python-Dev

Python "2migr8"
by Steve Dower 15 Apr '14

15 Apr '14

Just in case there's anyone out there who isn't yet sick of discussing how to proceed with Python 2.7, I have some more inputs to contribute. To put it up front, I'm totally against "CPython 2.8" ever becoming a real thing. Anything that comes out should be seen as a migration path, not an upgrade path. I'll also admit I'm not heavily invested in working on it myself, but I had a number of conversations during PyCon (as well as being at the language summit) that puts me in a position to share the ideas and concerns that have been raised. The main trigger was a conversation I had with two employees of a very large bank that has about 3000 Python users (not developers - mostly financial analysts) and 16 million lines of code running on 2.7. They are keen to migrate to 3.x but cannot afford to stop work entirely while their code is updated. (There was much more to the conversation than I'm relating here - I'm keeping to the directly relevant bits.) In describing the approach they'd like to take, they made me realise that there is definitely a place for a Python that is different but mostly compatible with 2.7, in a way that 2.7.x could not be. For the sake of having a name, I'll refer to this as "Python 2migr8" (pronounced "to migrate" :) ). The two important components of Python 2migr8 would be the ability to disable 2.7-only features, and to do so on a module-by-module basis. My best idea so far would be to have a magic comment (to ensure 2.7 compatibility better than a "from __future__ ...") near the top of the file that marks that file as "must straddle 2.7 and 3.3". Adding this comment causes (for example) the parser to treat "except x, y" as a syntax error in this file, forces "from __future__ import ...", hides "dict.iterkeys", undefines "basestring", etc., but only for this file. (I haven't thought through all the possibilities or implications - Eric Snow said he was going to sprint on this today/tomorrow, so he'll soon have a better idea just what can be done.) In effect, 2migr8 would be the version that *only* supports "single-source" files. This allows large code bases to progressively migrate modules from 2.x to single-source while continuing to run against Python 2.7. As files are updated, they are marked as being single-source. When all files have this marker, it should be possible to flip the switch and run with Python 3.3 or later. You could also think of this as enabling "-3 --warnings-as-errors" for individual modules, though since the user has already opted in to 2migr8, it isn't unreasonable to make more significant changes, like having dict.keys returning a list that warns if it is mutated. This sort of warning can only really be done by changing the interpreter - static analysis just can't catch everything - and only when users accept a potential performance hit and low probability of breakage when they move to 2migr8 (followed by a not-quite-as-low probability of breaking when they eventually move from 2migr8 to 3.x, but it's still better than guaranteed breakage). As a fork, it would also be possible to bundle the modules that have been backported, and possibly also to disallow importing deprecated stdlib modules when 2.7 functionality is disabled. As I said, I haven't thought through all the possibilities, but the general idea is to take 2.7 and *remove* features so it becomes easier to migrate. Where does python-dev come in? Obviously this is where a fork like this would have to start - there has been such strong and public opposition to any significant changes like this that you'd be hard pressed to find someone willing to start and promote it from outside. There is also a good opportunity to make a start and directly invite those using it to contribute the rules or warnings that they need - the 3000 Python "users" I mentioned earlier are backed by a team of true developers who are more than capable of contributing, and this would be a great opportunity to directly invite them. However unfair and incorrect it may be, there is a perception in some businesses that open-source projects do not want contributions from them. I invited more than one business to have someone join python-dev and get involved during PyCon, and I heard that others did the same - it may not be at the level of employing a core developer full time, but it's the starting point that some companies will need to be able to become comfortable with employing a core dev. I'm not pretending to have a full plan on how this will work. I was privileged to have some private conversations during PyCon that are directly relevant, so I'm bringing it here to promote the discussion. Thanks to everyone I had a chance to chat to, and to everyone generally for a great PyCon. Cheers, Steve

14 23

Appeal for reviews
by Nikolaus Rath 15 Apr '14

15 Apr '14

Hello, I've accumulated a number of patches in the issue tracker that are waiting for someone to review/commit/reject them. I'm eager to make corrections as necessary, I just need someone to look the work that I've done so far: * http://bugs.python.org/issue20951 (SSLSocket.send() returns 0 for non-blocking socket) * http://bugs.python.org/issue1738 (filecmp.dircmp does exact match only) * http://bugs.python.org/issue7776 (http.client.HTTPConnection tunneling is broken) * http://bugs.python.org/issue20177 (Derby #8: Convert 28 sites to Argument Clinic across 2 files) * http://bugs.python.org/issue19414 (iter(ordered_dict) yields keys not in dict in some circumstances) * http://bugs.python.org/issue20578 (BufferedIOBase.readinto1 is missing) * http://bugs.python.org/issue21057 (TextIOWrapper does not support reading bytearrays or memoryviews) I realize that core developer time is scarce, so I have started to only work on patches after I've confirmed that someone is available and interested to review them. However, it would be great if some people could find time to look at the issues above as well. Having your contributions just languish in the bugtracker is really dispiriting... I *want* to contribute, and I can't believe that Python is the one open-source project that is not in need of more manpower. But for some reason it seems really hard to actually find something to do.. Best, Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.«

12 26

Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)
by Skip Montanaro 15 Apr '14

15 Apr '14

Eric wrote: > Perhaps not so much "a very real advantage" as "less of a > distraction". It's still significantly slower than 2.7. :) I'm still confused. I peeked in /usr/bin/hg. The only "system" modules it imports directly are os and system (maybe I'm using an ancient version?). After that, it imports its own lazy import module. This suggests to me that Mercurial's import slowness is mostly in its own code (I counted 104 Python modules and six shared objects in its mercurial package, which isn't going to be affected (much?) by freezing the Python standard modules. I'm not trying to be difficult here. I thought that way back in the day a huge amount of work was done to remove needless filesystem activity, and zip packaging has been around for quite awhile. As an experiment, I ran hg pull as /usr/bin/python -v /usr/bin/hg pull in my cpython repo then looked at the -v output. Summarizing the output I saw the following: 30 imports (0 dlopens) Python banner printed 86 imports (18 dlopens) adding changesets message 5 imports (2 dlopens) adding manifests message 1 import (0 dlopens) adding file changes message 7 imports (3 dlopens) added ... changesets message 4 imports (0 dlopens) run 'hg update' message (I missed a "searching" message in there somewhere.) That's a total of 133 imports, 23 of which were satisfied by loading an extension module. The imports break down as follows: 51 imports (4 dlopens) from the mercurial package 5 imports from the hgext package 7 imports from the encodings package Everything else is imported from the top level, and at a glance appears to be all Python stdlib stuff. The key period of execution looks to be between the printing of the Python banner and the printing of the adding changesets message. I see 46 imports (2 dlopens) from the mercurial or hgext packages. That leaves 40 stdlib imports, of which 16 were satisfied by dlopen. As a final check, I saved all the stdlib import statements from the -v output (77 statements) to a file and timed its execution: % time /usr/bin/python ~/tmp/pyimp.py real 0m0.162s user 0m0.034s sys 0m0.010s It doesn't take much time to import every stdlib module that Mercurial needs. I don't know how much slower all this import machinery is in 3.x (and can't test at work, as we don't have a copy laying about). It would probably have to be 3x or more slower for it to have much visible impact on even simple hg commands. I find it hard to believe that freezing the stdlib is going to lower the barrier enough for the Mercurial folks, if, in fact, import slowness is their main reason for not moving to 3.x. Skip

8 14

[numpy wishlist] PyMem_*Calloc
by Nathaniel Smith 15 Apr '14

15 Apr '14

Hi all, The new tracemalloc infrastructure in python 3.4 is super-interesting to numerical folks, because we really like memory profiling. Numerical programs allocate a lot of memory, and sometimes it's not clear which operations allocate memory (some numpy operations return views of the original array without allocating anything; others return copies). So people actually use memory tracking tools[1], even though traditionally these have been pretty hacky (i.e., just checking RSS before and after each line is executed), and numpy has even grown its own little tracemalloc-like infrastructure [2], but it only works for numpy data. BUT, we also really like calloc(). One of the basic array creation routines in numpy is numpy.zeros(), which returns an array full of -- you guessed it -- zeros. For pretty much all the data types numpy supports, the value zero is represented by the bytestring consisting of all zeros. So numpy.zeros() usually uses calloc() to allocate its memory. calloc() is more awesome than malloc()+memset() for two reasons. First, calloc() for larger allocations is usually implemented using clever VM tricks, so that it doesn't actually allocate any memory up front, it just creates a COW mapping of the system zero page and then does the actual allocation one page at a time as different entries are written to. This means that in the somewhat common case where you allocate a large array full of zeros, and then only set a few scattered entries to non-zero values, you can end up using much much less memory than otherwise. It's entirely possible for this to make the difference between being able to run an analysis versus not. memset() forces the whole amount of RAM to be committed immediately. Secondly, even if you *are* going to touch all the memory, then calloc() is still faster than malloc()+memset(). The reason is that for large allocations, malloc() usually does a calloc() no matter what -- when you get a new page from the kernel, the kernel has to make sure you can't see random bits of other processes's memory, so it unconditionally zeros out the page before you get to see it. calloc() knows this, so it doesn't bother zeroing it again. malloc()+memset(), by contrast, zeros the page twice, producing twice as much memory traffic, which is huge. SO, we'd like to route our allocations through PyMem_* in order to let tracemalloc "see" them, but because there is no PyMem_*Calloc, doing this would force us to give up on the calloc() optimizations. The obvious solution is to add a PyMem_*Calloc to the API. Would this be possible? Unfortunately it would require adding a new field to the PyMemAllocator struct, which would be an ABI/API break; PyMemAllocator is exposed directly in the C API and passed by value: https://docs.python.org/3.5/c-api/memory.html#c.PyMemAllocator (Too bad we didn't notice this a few months ago before 3.4 was released :-(.) I guess we could just rename the struct in 3.5, to force people to update their code. (I guess there aren't too many people who would have to update their code.) Thoughts? -n [1] http://scikit-learn.org/stable/developers/performance.html#memory-usage-pro… [2] https://github.com/numpy/numpy/pull/309 -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org

5 5

Re: [Python-Dev] [Python-checkins] cpython (3.1): disallow a negative idx parameter
by Zachary Ware 14 Apr '14

14 Apr '14

On Mon, Apr 14, 2014 at 10:52 AM, benjamin.peterson <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/4bd1fb0f4f44 > changeset: 90256:4bd1fb0f4f44 > branch: 3.1 > parent: 90235:a8facac493ef > user: Benjamin Peterson <benjamin(a)python.org> > date: Mon Apr 14 11:45:21 2014 -0400 > summary: > disallow a negative idx parameter > > files: > Modules/_json.c | 9 +++++---- > 1 files changed, 5 insertions(+), 4 deletions(-) > > > diff --git a/Modules/_json.c b/Modules/_json.c > --- a/Modules/_json.c > +++ b/Modules/_json.c > @@ -902,10 +902,11 @@ > PyObject *res; > Py_UNICODE *str = PyUnicode_AS_UNICODE(pystr); > Py_ssize_t length = PyUnicode_GET_SIZE(pystr); > - if (idx < 0) > - /* Compatibility with Python version. */ > - idx += length; > - if (idx < 0 || idx >= length) { > + if (idx < 0) { > + PyErr_SetString(PyExc_ValueError, "idx canont be negative"); s/canont/cannot/ Also in the comparable 3.2 commit, but not the 3.3+ merge. Regards, -- Zach

2 1

death to 2.7; long live 2.7
by Benjamin Peterson 14 Apr '14

14 Apr '14

This email is to share idea that has been bouncing around in my head for a while about 2.7 releases. Guido's last email containing notes from the language summit made me think it's time to propose it. We'll keep doing what we're currently doing for another year, making normal bug fix releases with installers. After that, we _won't_ close 2.7 to normal bug fixes as is currently implied by the release schedule. Instead dealing 2.7 will just be completely optional for core developers. (The much anticipated vendor support arrives at this point.) Releases will continue at a measured (6-12 months) pace. These releases will be source only (unless someone steps up to make installers). Planning-on-making-2.7-releases-'til-the-cows-come-home-ly yours, Benjamin

12 21

your next regular installment of python 2.7
by Benjamin Peterson 13 Apr '14

13 Apr '14

Yep, it's almost that time of the year again. My proposed schedule for 2.7.7 is May 12th, 2.7.7rc1 May 26th, 2.7.7 final That means if you want your 2.7 improvement to see the light of day before November, the next few weeks are the time to do it. (PEP 446, I'm looking at you.) Also, I doubt anyone cares, but I'm going to do the last security release of python 3.1 concurrently. That will, of course, be a source-only release. Regards, Benjamin

1 0

Fwd: Jython Report
by Michael Foord 13 Apr '14

13 Apr '14

Below is the Jython "status update" report on Jython I received from Jim Baker and summarised in the Language Summit. It comes with one addendum from Frank: Jim's list is fantastic - the one bit I'd like to add to the list: Jython now supports a buffer protocol that parallels CPython's C API buffer protocol. This provided the basis for support of buffer() and memoryview(). The work was done with Jython3 in mind and will be a huge boost to that eventual effort. Begin forwarded message: > From: Jim Baker <jbaker(a)zyasoft.com> > Subject: Re: Jython Report > Date: 7 April 2014 06:42:51 BST > To: Michael Foord <michael(a)voidspace.org.uk> > Cc: Frank Wierzbicki <fwierzbicki(a)gmail.com> > > Recent changes to trunk (last 6 months) > > * Recently tagged a soft beta 2! > * Java 7 JVM is now the minimum version, which gives a larger base of functionality to work with (such as using Java 7's AutoCloseable to imply corresponding context manager support in using Python code) > * Enable mixing Python and Java types in the bases of a class when using a metaclass > * Added support for buffer, memoryview, although not complete yet with respect to Java integration > * Console and encoding support, such as unicodedata/idna updates > * Many, many small fixes > > About to be in trunk, to support beta 3 > > * socket-reboot reimplements socket/select/ssl on top of Netty 4, a popular event loop networking framework for the JVM (used by a large number of performant projects in Java space and originally part of JBoss). There was no ssl support before, but now socket and especially select semantics are much closer to CPython as well (basically close to the Windows socket model). > * socket-reboot in turn enables requests and thereby pip. A branch of pip currently works, actually modifying an upstream vendor lib (html5lib) so that it doesn't use isolated UTF-16 surrogates in literals, since this is not actually legal unicode, nor does it work in Jython's UTF-16 based representation. Ironically this usage is to detect such illegal use in input streams. > * Relative star imports, which seems to impact a number of interesting projects. > * Performance tuning of sre. Jython has a port of CPython's sre, however our use of UTF-16 requires expansion into an array of codepoints. Currently this is done on demand, which can potentially add another O(n) factor in evaluating regexes. A pull request we will apply memoizes. In the future, we will rewrite the logic in sre so that it does next/prev, much like JRuby currently does for similar encoding issues. > > Related work > > * Other PyPA tooling including virtualenv and wheel needs more diagnosis to see why they currently fail on Jython, but our hope is that this is minor. > * New project jythontools by a number of Jython developers (including Frank and Jim). This includes a number of projects that will help evolve Jython, but outside the usual release schedule and the usual problem of being in core (such as eventual deprecation): > - Clamp - precise integration with Java, enabling such capabilities as Java directly importing Python modules without explicitly initializing the Jython runtime or using object factories. Future work will enable Java annotation integration, as decorators. Integrates with setuptools; future integration as well with Maven via Aether. > - Jiffy - provide a CFFI backend for Jython. Right now it is pure vaporware, but cursory examination of cffi.backend_ctypes suggests that it should be straightforward and of modest effort to provide a similar backend by using JFFI, which Jython and JRuby both use to access native runtime services (such as Posix API) as part of the Java native runtime project. > * The Patois project has been started to collect examples for cross-implementation support, as seen in surrogate support, but it will be a good question to get that really going, vs just talking about it. > * JyNI - simply adding this jar to the classpath enables C extension API support. Note that this project has been licensed by its developer (not a Jython committer) under an LGPL license. > > Release schedule > > * Complete beta 2 > * Beta 3 is forthcoming, likely in 2 weeks > * For beta 4, need to perform a comprehensive bug triage - what will be in, not in for 2.7.0 > * EuroPython sprint to finalize a release candidate for 2.7.0? > > Future > > * Mostly around performance, Java integration, and of course the usual bug fixes > * Python bytecode compiler remains important, including for support targeting Android and removing restriction on getting too large a method for the JVM > * More hooks for Java integration, including managing generated bytecode > * Integrating Zippy could provide for PyPy-like performance, but requires Graal JVM > * Supporting invokedynamic is a more realistic solution, but without the use of annotations (eg turn off Python frames) is going to be limited (maybe 2x?) based on earlier analysis > > Jython 3.x? > > * This comes up periodically, and it would be super nice for us to complete this support. At the very least it would make unicode strings and bytestrings correspond directly to how they are represented in Java, so that will be a nice cleanup. > * Release schedule: we will get there at some point! > > > On Sun, Apr 6, 2014 at 5:20 PM, Jim Baker <jbaker(a)zyasoft.com> wrote: > Michael, > > I was thinking about this very topic this morning! Will send you the latest status in the next 24h, specifically our work to support pypa (setuptools, pip, virtualenv, wheel) and related tooling. > > - Jim > > > On Sun, Apr 6, 2014 at 11:30 AM, Michael Foord <michael(a)voidspace.org.uk> wrote: > Hey guys, > > Would you be able to write up a brief report on the current and future status of Jython, for me to read out at the Python language summit on Wednesday? (Unless someone who works on Jython will be there - but as far as I know that isn't the case.) > > All the best, > > Michael Foord > > -- > http://www.voidspace.org.uk/ > > > May you do good and not evil > May you find forgiveness for yourself and forgive others > May you share freely, never taking more than you give. > -- the sqlite blessing > http://www.sqlite.org/different.html > > > > > > > > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

2 1

Windows installers and OpenSSL
by Paul Moore 13 Apr '14

13 Apr '14

Given the OpenSSL vulnerability and the fact that we bundle OpenSSL with the Windows installers (1.0.1e in Python 3.4.0) should we be releasing updated installers? Paul

8 8

python-3.4.0
by Greg Mildenstein 13 Apr '14

13 Apr '14

Hi, I'm running windows 8.1 64 bit. I was using 'python-3.3.1.amd64' but have uninstalled it and installed the above version. However, when I try running 'PyScripter', I get an error stating, 'ERROR - unable to open python, it will now exit'. Is there something I'm doing wrong? Thanks Greg Mildenstein

2 1