Mailman 3 November 2014 - Python-checkins

peps: Add PEP 481 - Migrate Some Supporting Repositories to Git and Github
by donald.stufft 29 Nov '14

29 Nov '14

https://hg.python.org/peps/rev/f394fde562b2 changeset: 5626:f394fde562b2 user: Donald Stufft <donald(a)stufft.io> date: Sat Nov 29 18:12:06 2014 -0500 summary: Add PEP 481 - Migrate Some Supporting Repositories to Git and Github files: pep-0481.txt | 284 +++++++++++++++++++++++++++++++++++++++ 1 files changed, 284 insertions(+), 0 deletions(-) diff --git a/pep-0481.txt b/pep-0481.txt new file mode 100644 --- /dev/null +++ b/pep-0481.txt @@ -0,0 +1,284 @@ +PEP: 481 +Title: Migrate Some Supporting Repositories to Git and Github +Version: $Revision$ +Last-Modified: $Date$ +Author: Donald Stufft <donald(a)stufft.io> +Status: Draft +Type: Process +Content-Type: text/x-rst +Created: 29-Nov-2014 +Post-History: 29-Nov-2014 + + +Abstract +======== + +This PEP proposes migrating to Git and Github for certain supporting +repositories (such as the repository for Python Enhancement Proposals) in a way +that is more accessible to new contributors, and easier to manage for core +developers. This is offered as an alternative to PEP 474 which aims to achieve +the same overall benefits but while continuing to use the Mercurial DVCS and +without relying on a commerical entity. + +In particular this PEP proposes changes to the following repositories: + +* https://hg.python.org/devguide/ +* https://hg.python.org/devinabox/ +* https://hg.python.org/peps/ + + +This PEP does not propose any changes to the core development workflow for +CPython itself. + + +Rationale +========= + +As PEP 474 mentions, there are currently a number of repositories hosted on +hg.python.org which are not directly used for the development of CPython but +instead are supporting or ancillary repositories. These supporting repositories +do not typically have complex workflows or often branches at all other than the +primary integration branch. This simplicity makes them very good targets for +the "Pull Request" workflow that is commonly found on sites like Github. + +However where PEP 474 wants to continue to use Mercurial and wishes to use an +OSS and self-hosted and therefore restricts itself to only those solutions this +PEP expands the scope of that to include migrating to Git and using Github. + +The existing method of contributing to these repositories generally includes +generating a patch and either uploading them to bugs.python.org or emailing +them to peps(a)python.org. This process is unfriendly towards non-comitter +contributors as well as making the process harder than it needs to be for +comitters to accept the patches sent by users. In addition to the benefits +in the pull request workflow itself, this style of workflow also enables +non techincal contributors, especially those who do not know their way around +the DVCS of choice, to contribute using the web based editor. On the committer +side the Pull Requests enable them to tell, before merging, whether or not +a particular Pull Request will break anything. It also enables them to do a +simple "push button" merge which does not require them to check out the +changes locally. Another such feature that is useful in particular for docs, +is the ability to view a "prose" diff. This Github specific feature enables +a committer to view a diff of the rendered output which will hide things like +reformatting a paragraph and show you what the actual "meat" of the change +actually is. + + +Why Git? +-------- + +Looking at the variety of DVCS which are available today it becomes fairly +clear that git has gotten the vast mindshare of people who are currently using +it. The Open Hub (Previously Ohloh) statistics [#openhub-stats]_ show that +currently 37% of the repositories Open Hub is indexing is using git which is +second only to SVN (which has 48%) while Mercurial has just 2% of the indexed +repositories (beating only bazaar which has 1%). In additon to the Open Hub +statistics a look at the top 100 projects on PyPI (ordered by total download +counts) shows us that within the Python space itself there is a majority of +projects using git: + +=== ========= ========== ====== === ==== +Git Mercurial Subversion Bazaar CVS None +=== ========= ========== ====== === ==== +62 22 7 4 1 1 +=== ========= ========== ====== === ==== + + +Chosing a DVCS which has the larger mindshare will make it more likely that any +particular person who has experience with DVCS at all will be able to +meaningfully use the DVCS that we have chosen without having to learn a new +tool. + +In addition to simply making it more likely that any individual will already +know how to use git, the number of projects and people using it means that the +resources for learning the tool are likely to be more fully fleshed out and +when you run into problems the liklihood that someone else had that problem +and posted a question and recieved an answer is also far likelier. + +Thirdly by using a more popular tool you also increase your options for tooling +*around* the DVCS itself. Looking at the various options for hosting +repositories it's extremely rare to find a hosting solution (whether OSS or +commerical) that supports Mercurial but does not support Git, on the flip side +there are a number of tools which support Git but do not support Mercurial. +Therefore the popularity of git increases the flexibility of our options going +into the future for what toolchain these projects use. + +Also by moving to the more popular DVCS we increase the likelhood that the +knowledge that the person has learned in contributing to these support +repositories will transfer to projects outside of the immediate CPython project +such as to the larger Python community which is primarily using Git hosted on +Github. + +In previous years there was concern about how well supported git was on Windows +in comparison to Mercurial. However git has grown to support Windows as a first +class citizen. In addition to that, for Windows users who are not well aquanted +with the Windows command line there are GUI options as well. + +On a techincal level git and Mercurial are fairly similar, however the git +branching model is signifcantly better than Mercurial "Named Branches" for +non-comitter contributors. Mercurial does have a "Bookmarks" extension however +this isn't quite as good as git's branching model. All bookmarks live in the +same namespace so it requires individual users to ensure that they namespace +the branchnames themselves lest the risk collision. It also is an extension +which requires new users to first discover they need an extension at all and +then figure out what they need to do in order to enable that extension. Since +it is an extension it also means that in general support for them outside of +Mercurial core is going to be less than 100% in comparison to git where the +feature is built in and core to using git at all. Finally users who are not +used to Mercurial are unlikely to discover bookmarks on their own, instead they +will likely attempt to use Mercurial's "Named Branches" which, given the fact +they live "forever", are not often what a project wants their contributors to +use. + + +Why Github? +----------- + +There are a number of software projects or web services which offer +functionality similar to that of Github. These range from commerical web +services such as a Bitbucket to self-hosted OSS solutions such as Kallithea or +Gitlab. This PEP proposes that we move these repositories to Github. + +There are two primary reasons for selecting Github: Popularity and +Quality/Polish. + +Github is currently the most popular hosted repository hosting according to +Alexa where it currently has a global rank of 121. Much like for Git itself by +choosing the most popular tool we gain benefits in increasing the likelhood +that a new contributor will have already experienced the toolchain, the quality +and availablity of the help, more and better tooling being built around it, and +the knowledge transfer to other projects. A look again at the top 100 projects +by download counts on PyPI shows the following hosting locations: + +====== ========= =========== ========= =========== ========== +GitHub BitBucket Google Code Launchpad SourceForge Other/Self +====== ========= =========== ========= =========== ========== +62 18 6 4 3 7 +====== ========= =========== ========= =========== ========== + +In addition to all of those reasons, Github also has the benefit that while +many of the options have similar features when you look at them in a feature +matrix the Github version of each of those features tend to work better and be +far more polished. This is hard to quantify objectively however it is a fairly +common sentiment if you go around and ask people who are using these services +often. + +Finally a reason to choose a web service at all over something that is +self-hosted is to be able to more efficiently use volunteer time and donated +resources. Every additional service hosted on the PSF infrastruture by the +PSF infrastructure team further spreads out the amount of time that the +volunteers on that team have to spend and uses some chunk of resources that +could potentionally be used for something where there is no free or affordable +hosted solution available. + +One concern that people do have with using a hosted service is that there is a +lack of control and that at some point in the future the service may no longer +be suitable. It is the opinion of this PEP that Github does not currently and +has not in the past engaged in any attempts to lock people into their platform +and that if at some point in the future Github is no longer suitable for one +reason or another than at that point we can look at migrating away from Github +onto a different solution. In other words, we'll cross that bridge if and when +we come to it. + + +Example: Scientific Python +-------------------------- + +One of the key ideas behind the move to both git and Github is that a feature +of a DVCS, the repository hosting, and the workflow used is the social network +and size of the community using said tools. We can see this is true by looking +at an example from a sub-community of the Python community: The Scientific +Python community. They have already migrated most of the key pieces of the +SciPy stack onto Github using the Pull Request based workflow starting with +IPython and as more projects moved over it became a natural default for new +projects. + +They claim to have seen a great benefit from this move, where it enables casual +contributors to easily move between different projects within their +sub-community without having to learn a special, bespoke workflow and a +different toolchain for each project. They've found that when people can use +their limited time on actually contributing instead of learning the different +tools and workflows that not only do they contribute more to one project, that +they also expand out and contribute to other projects. This move is also +attributed to making it commonplace for members of that community to go so far +as publishing their research and educational materials on Github as well. + +This showcases the real power behind moving to a highly popular toolchain and +workflow, as each variance introduces yet another hurdle for new and casual +contributors to get past and it makes the time spent learning that workflow +less reusable with other projects. + + +Migration +========= + +Through the use of hg-git [#hg-git]_ we can easily convert a Mercurial +repository to a Git repository by simply pushing the Mercurial repository to +the Git repository. People who wish to continue to use Mercurual locally can +then use hg-git going into the future using the new Github URL, however they +will need to re-clone their repositories as using Git as the server seems to +trigger a one time change of the changeset ids. + +As none of the selected repositories have any tags, branches, or bookmarks +other than the ``default`` branch the migration will simply map the ``default`` +branch in Mercurial to the ``master`` branch in git. + +In addition since none of the selected projects have any great need of a +complex bug tracker, they will also migrate their issue handling to using the +GitHub issues. + +In addition to the migration of the repository hosting itself there are a +number of locations for each particular repository which will require updating. +The bulk of these will simply be changing commands from the hg equivilant to +the git equivilant. + +In particular this will include: + +* Updating www.python.org to generate PEPs using a git clone and link to + Github. +* Updating docs.python.org to pull from Github instead of hg.python.org for the + devguide. +* Enabling the ability to send an email to python-checkins(a)python.org for each + push. +* Enabling the ability to send an IRC message to #python-dev on Freenode for + each push. +* Migrate any issues for these projects to their respective bug tracker on + Github. + +This will restore these repositories to similar functionality as they currently +have. In addition to this the migration will also include enabling testing for +each pull request using Travis CI [#travisci]_ where possible to ensure that +a new PR does not break the ability to render the documentation or PEPs. + + +User Access +=========== + +Moving to Github would involve adding an additional user account that will need +to be managed, however it also offers finer grained control, allowing the +ability to grant someone access to only one particular repository instead of +the coarser grained ACLs available on hg.python.org. + + +References +========== + +.. [#openhub-stats] `Open Hub Statistics <https://www.openhub.net/repositories/compare>` +.. [#hg-git] `hg-git <https://hg-git.github.io/>` +.. [#travisci] `Travis CI <https://travis-ci.org/>` + + +Copyright +========= + +This document has been placed in the public domain. + + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: -- Repository URL: https://hg.python.org/peps

1 0

cpython (merge 3.4 -> default): Close issue #22895: fix test failure introduced by the fix for issue #22462.
by antoine.pitrou 29 Nov '14

29 Nov '14

https://hg.python.org/cpython/rev/4990157343c6 changeset: 93656:4990157343c6 parent: 93654:bd97eab25c70 parent: 93655:e4b986350feb user: Antoine Pitrou <solipsis(a)pitrou.net> date: Sat Nov 29 15:56:38 2014 +0100 summary: Close issue #22895: fix test failure introduced by the fix for issue #22462. files: Lib/test/test_pyexpat.py | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_pyexpat.py b/Lib/test/test_pyexpat.py --- a/Lib/test/test_pyexpat.py +++ b/Lib/test/test_pyexpat.py @@ -3,6 +3,7 @@ from io import BytesIO import os +import sysconfig import unittest import traceback @@ -444,7 +445,8 @@ "pyexpat.c", "StartElement") self.check_traceback_entry(entries[2], "test_pyexpat.py", "StartElementHandler") - self.assertIn('call_with_frame("StartElement"', entries[1][3]) + if sysconfig.is_python_build(): + self.assertIn('call_with_frame("StartElement"', entries[1][3]) # Test Current* members: -- Repository URL: https://hg.python.org/cpython

1 0

cpython (3.4): Close issue #22895: fix test failure introduced by the fix for issue #22462.
by antoine.pitrou 29 Nov '14

29 Nov '14

https://hg.python.org/cpython/rev/e4b986350feb changeset: 93655:e4b986350feb branch: 3.4 parent: 93653:21257f916668 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Sat Nov 29 15:56:07 2014 +0100 summary: Close issue #22895: fix test failure introduced by the fix for issue #22462. files: Lib/test/test_pyexpat.py | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_pyexpat.py b/Lib/test/test_pyexpat.py --- a/Lib/test/test_pyexpat.py +++ b/Lib/test/test_pyexpat.py @@ -3,6 +3,7 @@ from io import BytesIO import os +import sysconfig import unittest import traceback @@ -444,7 +445,8 @@ "pyexpat.c", "StartElement") self.check_traceback_entry(entries[2], "test_pyexpat.py", "StartElementHandler") - self.assertIn('call_with_frame("StartElement"', entries[1][3]) + if sysconfig.is_python_build(): + self.assertIn('call_with_frame("StartElement"', entries[1][3]) # Test Current* members: -- Repository URL: https://hg.python.org/cpython

1 0

Daily reference leaks (bd97eab25c70): sum=3
by solipsis＠pitrou.net 29 Nov '14

29 Nov '14

results for bd97eab25c70 on branch "default" -------------------------------------------- test_functools leaked [0, 0, 3] memory blocks, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogWR_tSy', '-x']

1 0

peps: Two new sections (more motivation, rejected @allow_implicit_stop).
by guido.van.rossum 29 Nov '14

29 Nov '14

https://hg.python.org/peps/rev/2cb6c16d7d2a changeset: 5625:2cb6c16d7d2a user: Guido van Rossum <guido(a)python.org> date: Fri Nov 28 20:11:11 2014 -0800 summary: Two new sections (more motivation, rejected @allow_implicit_stop). files: pep-0479.txt | 56 ++++++++++++++++++++++++++++++++++++++++ 1 files changed, 56 insertions(+), 0 deletions(-) diff --git a/pep-0479.txt b/pep-0479.txt --- a/pep-0479.txt +++ b/pep-0479.txt @@ -100,6 +100,16 @@ terminate a generator: the proper way is ``return``, not ``raise StopIteration``. +As an added bonus, the above changes bring generator functions much +more in line with regular functions. If you wish to take a piece of +code presented as a generator and turn it into something else, you +can usually do this fairly simply, by replacing every ``yield`` with +a call to ``print()`` or ``list.append()``; however, if there are any +bare ``next()`` calls in the code, you have to be aware of them. If +the code was originally written without relying on ``StopIteration`` +terminating the function, the transformation would be that much +easier. + Background information ====================== @@ -433,6 +443,7 @@ bubble out would still be potentially wrong, depending on the use made of the distinction between the two exception types. + Converting the exception inside next() -------------------------------------- @@ -449,6 +460,45 @@ would help this; however, all code would still need to be rewritten.) +Sub-proposal: decorator to explicitly request current behaviour +--------------------------------------------------------------- + +Nick Coghlan suggested [13]_ that the situations where the current +behaviour is desired could be supported by means of a decorator:: + + from itertools import allow_implicit_stop + + @allow_implicit_stop + def my_generator(): + ... + yield next(it) + ... + +Which would be semantically equivalent to:: + + def my_generator(): + try: + ... + yield next(it) + ... + except StopIteration + return + +but be faster, as it could be implemented by simply permitting the +``StopIteration`` to bubble up directly. + +Single-source Python 2/3 code would also benefit in a 3.7+ world, +since libraries like six and python-future could just define their own +version of "allow_implicit_stop" that referred to the new builtin in +3.5+, and was implemented as an identity function in other versions. + +However, due to the implementation complexities required, the ongoing +compatibility issues created, the subtlety of the decorator's effect, +and the fact that it would encourage the "quick-fix" solution of just +slapping the decorator onto all generators instead of properly fixing +the code in question, this sub-proposal has been rejected. [14]_ + + Criticism ========= @@ -516,6 +566,12 @@ .. [12] Post from Mark Shannon with alternate proposal (https://mail.python.org/pipermail/python-dev/2014-November/137129.html) +.. [13] Idea from Nick Coghlan + (https://mail.python.org/pipermail/python-dev/2014-November/137201.html) + +.. [14] Rejection by GvR + (https://mail.python.org/pipermail/python-dev/2014-November/137243.html) + Copyright ========= -- Repository URL: https://hg.python.org/peps

1 0

peps: Elaborate transaction() manager example (Chris A).
by guido.van.rossum 29 Nov '14

29 Nov '14

https://hg.python.org/peps/rev/4803b0829533 changeset: 5624:4803b0829533 user: Guido van Rossum <guido(a)python.org> date: Fri Nov 28 17:26:33 2014 -0800 summary: Elaborate transaction() manager example (Chris A). files: pep-0479.txt | 31 ++++++++++++++++++++++--------- 1 files changed, 22 insertions(+), 9 deletions(-) diff --git a/pep-0479.txt b/pep-0479.txt --- a/pep-0479.txt +++ b/pep-0479.txt @@ -44,26 +44,39 @@ existent". Unfortunately while intentional use is rare, it is easy to stumble on these cases by accident:: + import contextlib + @contextlib.contextmanager def transaction(): - begin() + print('begin') try: yield from do_it() except: - rollback() + print('rollback') raise else: - commit() - + print('commit') + def do_it(): - initial_preparations() - yield - finishing_touches() + print('Refactored preparations') + yield # Body of with-statement is executed here + print('Refactored finalization') + + def gene(): + for i in range(2): + with transaction(): + yield i + # return + raise StopIteration # This is wrong + print('Should not be reached') + + for i in gene(): + print('main: i =', i) Here factoring out ``do_it`` into a subgenerator has introduced a subtle bug: if the wrapped block raises ``StopIteration``, under the -current behavior ``do_it`` will fail but report success by returning -normally, causing the failed transaction to be committed! Similarly +current behavior this exception will be swallowed by the context +manager; and, worse, the finalization is silently skipped! Similarly problematic behavior occurs when an ``asyncio`` coroutine raises ``StopIteration``, causing it to terminate silently. -- Repository URL: https://hg.python.org/peps

1 0

cpython (merge 3.4 -> default): Issue #16113: Also remove test_case_sha3_224_huge
by ned.deily 28 Nov '14

28 Nov '14

https://hg.python.org/cpython/rev/bd97eab25c70 changeset: 93654:bd97eab25c70 parent: 93652:70163e18da87 parent: 93653:21257f916668 user: Ned Deily <nad(a)acm.org> date: Fri Nov 28 15:22:15 2014 -0800 summary: Issue #16113: Also remove test_case_sha3_224_huge files: Lib/test/test_hashlib.py | 6 ------ 1 files changed, 0 insertions(+), 6 deletions(-) diff --git a/Lib/test/test_hashlib.py b/Lib/test/test_hashlib.py --- a/Lib/test/test_hashlib.py +++ b/Lib/test/test_hashlib.py @@ -373,12 +373,6 @@ "e718483d0ce769644e2e42c7bc15b4638e1f98b13b2044285632a803afa973eb"+ "de0ff244877ea60a4cb0432ce577c31beb009c5c2c49aa2e4eadb217ad8cc09b") - @unittest.skipIf(sys.maxsize < _4G + 5, 'test cannot run on 32-bit systems') - @bigmemtest(size=_4G + 5, memuse=1, dry_run=False) - def test_case_sha3_224_huge(self, size): - self.check('sha3_224', b'A'*size, - '58ef60057c9dddb6a87477e9ace5a26f0d9db01881cf9b10a9f8c224') - def test_gil(self): # Check things work fine with an input larger than the size required # for multithreaded operation (which is hardwired to 2048). -- Repository URL: https://hg.python.org/cpython

1 0

cpython (3.4): Issue #16113: Also remove test_case_sha3_224_huge
by ned.deily 28 Nov '14

28 Nov '14

https://hg.python.org/cpython/rev/21257f916668 changeset: 93653:21257f916668 branch: 3.4 parent: 93647:737355f61ba2 user: Ned Deily <nad(a)acm.org> date: Fri Nov 28 15:21:12 2014 -0800 summary: Issue #16113: Also remove test_case_sha3_224_huge files: Lib/test/test_hashlib.py | 6 ------ 1 files changed, 0 insertions(+), 6 deletions(-) diff --git a/Lib/test/test_hashlib.py b/Lib/test/test_hashlib.py --- a/Lib/test/test_hashlib.py +++ b/Lib/test/test_hashlib.py @@ -373,12 +373,6 @@ "e718483d0ce769644e2e42c7bc15b4638e1f98b13b2044285632a803afa973eb"+ "de0ff244877ea60a4cb0432ce577c31beb009c5c2c49aa2e4eadb217ad8cc09b") - @unittest.skipIf(sys.maxsize < _4G + 5, 'test cannot run on 32-bit systems') - @bigmemtest(size=_4G + 5, memuse=1, dry_run=False) - def test_case_sha3_224_huge(self, size): - self.check('sha3_224', b'A'*size, - '58ef60057c9dddb6a87477e9ace5a26f0d9db01881cf9b10a9f8c224') - def test_gil(self): # Check things work fine with an input larger than the size required # for multithreaded operation (which is hardwired to 2048). -- Repository URL: https://hg.python.org/cpython

1 0

cpython: Minor code cleanup.
by raymond.hettinger 28 Nov '14

28 Nov '14

https://hg.python.org/cpython/rev/70163e18da87 changeset: 93652:70163e18da87 user: Raymond Hettinger <python(a)rcn.com> date: Fri Nov 28 14:52:14 2014 -0800 summary: Minor code cleanup. files: Lib/xml/etree/ElementPath.py | 5 +---- 1 files changed, 1 insertions(+), 4 deletions(-) diff --git a/Lib/xml/etree/ElementPath.py b/Lib/xml/etree/ElementPath.py --- a/Lib/xml/etree/ElementPath.py +++ b/Lib/xml/etree/ElementPath.py @@ -295,10 +295,7 @@ # Find first matching object. def find(elem, path, namespaces=None): - try: - return next(iterfind(elem, path, namespaces)) - except StopIteration: - return None + return next(iterfind(elem, path, namespaces), None) ## # Find all matching objects. -- Repository URL: https://hg.python.org/cpython

1 0

cpython: Issue #22685, asyncio: resume_reading() must also be called in
by victor.stinner 28 Nov '14

28 Nov '14

https://hg.python.org/cpython/rev/276515d2ceed changeset: 93651:276515d2ceed user: Victor Stinner <victor.stinner(a)gmail.com> date: Fri Nov 28 22:42:06 2014 +0100 summary: Issue #22685, asyncio: resume_reading() must also be called in test_pause_reading() files: Lib/test/test_asyncio/test_subprocess.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -196,6 +196,7 @@ self.assertEqual(stdout, b'x' * size) self.assertTrue(transport.pause_reading.called) + self.assertTrue(transport.resume_reading.called) if sys.platform != 'win32': -- Repository URL: https://hg.python.org/cpython

1 0