Mailman 3 April 2014 - Python-Dev

PEP 466: Network security enhancements for Python 2.7.7
by Nick Coghlan April 19, 2014

April 19, 2014

I've reworded the PEP to make it clear it is now just about backporting a specific set of enhancements to 2.7.7, as well as switching to updating to new OpenSSL feature releases in the binary installers. The idea of an open ended backport policy is now listed as a rejected variant. I believe that change addresses Guido's main remaining concern, so I think this version is ready for pronouncement. Regards, Nick. ======================================== PEP: 466 Title: Network Security Enhancements for Python 2.7.7 Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan(a)gmail.com>, Status: Draft Type: Informational Content-Type: text/x-rst Created: 23-Mar-2014 Post-History: 23-Mar-2014, 24-Mar-2014, 25-Mar-2014, 26-Mar-2014, 16-Apr-2014 Abstract ======== Most CPython tracker issues are classified as errors in behaviour or proposed enhancements. Most patches to fix behavioural errors are applied to all active maintenance branches. Enhancement patches are restricted to the default branch that becomes the next Python version. This cadence works reasonably well during Python's normal 18-24 month feature release cycle, which is still applicable to the Python 3 series. However, the age of the standard library in Python 2 has now reached a point where it is sufficiently far behind the state of the art in network security protocols for it to be causing real problems in use cases where upgrading to Python 3 in the near term may not be feasible. In recognition of the additional practical considerations that have arisen during the 4+ year maintenance cycle for Python 2.7, this PEP allows a critical set of network security related features to be backported from Python 3.4 to the upcoming Python 2.7.7 maintenance release. While this PEP does not make any changes to the core development team's handling of security-fix-only branches that are no longer in active maintenance, it *does* recommend that commercial redistributors providing extended support periods for the Python standard library either backport these features to their supported versions, or else explicitly disclaim support for the use of older versions in roles that involve connecting directly to the public internet. New security related features in Python 2.7.7 ============================================= Under this proposal, the following features will be backported from Python 3.4 to the upcoming Python 2.7.7 maintenance release: * in the ``os`` module: * persistent file descriptor for ``os.urandom()``. * in the ``hmac`` module: * constant time comparison function (``hmac.compare_digest()``). * in the ``hashlib`` module: * password hashing function (``hashlib.pbkdf2_hmac()``). * details of hash algorithm availability (``hashlib.algorithms_guaranteed`` and ``hashlib.algorithms_available``). * in the ``ssl`` module: * this module is almost entirely synchronised with its Python 3 counterpart, bringing TLSv1.x settings, SSLContext manipulation, Server Name Indication, access to platform certificate stores, standard library support for peer hostname validation and more to the Python 2 series. * the only ``ssl`` module features *not* backported under this policy are the ``ssl.RAND_*`` functions that provide access to OpenSSL's random number generation capabilities - use ``os.urandom()`` instead. As a general change in maintenance policy, permission is also granted to upgrade to newer feature releases of OpenSSL when preparing the binary installers for new maintenance releases of Python 2.7. This PEP does NOT propose a general exception for backporting new features to Python 2.7 - every new feature proposed for backporting will still need to be justified independently. In particular, it will need to be explained why relying on and independently updated backport on the Python Package Index instead is not an acceptable solution. Backwards compatibility considerations ====================================== As in the Python 3 series, the backported ``ssl.create_default_context()`` API is granted a backwards compatibility exemption that permits the protocol, options, cipher and other settings of the created SSL context to be updated in maintenance releases to use higher default security settings. This allows them to appropriately balance compatibility and security at the time of the maintenance release, rather than at the time of the original feature release. This PEP does *not* grant any other exemptions to the usual backwards compatibility policy for maintenance releases. Instead, by explicitly encouraging the use of feature based checks, it is designed to make it easier to write more secure cross-version compatible Python software, while still limiting the risk of breaking currently working software when upgrading to a new Python 2.7 maintenance release. In all cases where this proposal allows new features to be backported to the Python 2.7 release series, it is possible to write cross-version compatible code that operates by "feature detection" (for example, checking for particular attributes in a module), without needing to explicitly check the Python version. It is then up to library and framework code to provide an appropriate warning and fallback behaviour if a desired feature is found to be missing. While some especially security sensitive software MAY fail outright if a desired security feature is unavailable, most software SHOULD instead emit a warning and continue operating using a slightly degraded security configuration. The backported APIs allow library and application code to perform the following actions after detecting the presence of a relevant network security related feature: * explicitly opt in to more secure settings (to allow the use of enhanced security features in older maintenance releases of Python with less secure default behaviour) * explicitly opt in to less secure settings (to allow the use of newer Python feature releases in lower security environments) * determine the default setting for the feature (this MAY require explicit Python version checks to determine the Python feature release, but DOES NOT require checking for a specific maintenance release) Security related changes to other modules (such as higher level networking libraries and data format processing libraries) will continue to be made available as backports and new modules on the Python Package Index, as independent distribution remains the preferred approach to handling software that must continue to evolve to handle changing development requirements independently of the Python 2 standard library. Refer to the `Motivation and Rationale`_ section for a review of the characteristics that make the secure networking infrastructure worthy of special consideration. OpenSSL compatibility --------------------- Under this proposal, OpenSSL may be upgraded to more recent feature releases in Python 2.7 maintenance releases. On Linux and most other POSIX systems, the specific version of OpenSSL used already varies, as CPython dynamically links to the system provided OpenSSL library by default. For the Windows binary installers, the ``_ssl`` and ``_hashlib`` modules are statically linked with OpenSSL and the associated symbols are not exported. Marc-Andre Lemburg indicates that updating to newer OpenSSL releases in the ``egenix-pyopenssl`` binaries has not resulted in any reported compatibility issues [3]_ The Mac OS X binary installers historically followed the same policy as other POSIX installations and dynamically linked to the Apple provided OpenSSL libraries. However, Apple has now ceased updating these cross-platform libraries, instead requiring that even cross-platform developers adopt Mac OS X specific interfaces to access up to date security infrastructure on their platform. Accordingly, and independently of this PEP, the Mac OS X binary installers were already going to be switched to statically linker newer versions of OpenSSL [4]_ Other Considerations ==================== Maintainability --------------- A number of developers, including Alex Gaynor and Donald Stufft, have expressed interest in carrying out the feature backports covered by this policy, and assisting with any additional maintenance burdens that arise in the Python 2 series as a result. Steve Dower and Brian Curtin have offered to help with the creation of the Windows installers, allowing Martin von Löwis the opportunity to step back from the task of maintaining the 2.7 Windows installer. This PEP is primarily about establishing the consensus needed to allow them to carry out this work. For other core developers, this policy change shouldn't impose any additional effort beyond potentially reviewing the resulting patches for those developers specifically interested in the affected modules. Security releases ----------------- This PEP does not propose any changes to the handling of security releases - those will continue to be source only releases that include only critical security fixes. However, the recommendations for library and application developers are deliberately designed to accommodate commercial redistributors that choose to apply these changes to additional Python release series that are either in security fix only mode, or have been declared "end of life" by the core development team. Whether or not redistributors choose to exercise that option will be up to the individual redistributor. Integration testing ------------------- Third party integration testing services should offer users the ability to test against multiple Python 2.7 maintenance releases (at least 2.7.6 and 2.7.7+), to ensure that libraries, frameworks and applications can still test their handling of the legacy security infrastructure correctly (either failing or degrading gracefully, depending on the security sensitivity of the software), even after the features covered in this proposal have been backported to the Python 2.7 series. Handling lower security environments with low risk tolerance ------------------------------------------------------------ For better or for worse (mostly worse), there are some environments where the risk of latent security defects is more tolerated than even a slightly increased risk of regressions in maintenance releases. This proposal largely excludes these environments from consideration where the modules covered by the exemption are concerned - this approach is entirely inappropriate for software connected to the public internet, and defence in depth security principles suggest that it is not appropriate for most private networks either. Downstream redistributors may still choose to cater to such environments, but they will need to handle the process of downgrading the security related modules and doing the associated regression testing themselves. The main CPython continuous integration infrastructure will not cover this scenario. Motivation and Rationale ======================== The creation of this PEP was prompted primarily by the aging SSL support in the Python 2 series. As of March 2014, the Python 2.7 SSL module is approaching four years of age, and the SSL support in the still popular Python 2.6 release had its feature set locked six years ago. These are simply too old to provide a foundation that can be recommended in good conscience for secure networking software that operates over the public internet, especially in an era where it is becoming quite clearly evident that advanced persistent security threats are even more widespread and more indiscriminate in their targeting than had previously been understood. While they represented reasonable security infrastructure in their time, the state of the art has moved on, and we need to investigate mechanisms for effectively providing more up to date network security infrastructure for users that, for whatever reason, are not currently in a position to migrate to Python 3. While the use of the system OpenSSL installation addresses many of these concerns on Linux platforms, it doesn't address all of them (in particular, it is still difficult for sotware to explicitly require some higher level security settings). The standard library support can be bypassed by using a third party library like PyOpenSSL or Pycurl, but this still results in a security problem, as these can be difficult dependencies to deploy, and many users will remain unaware that they might want them. Rather than explaining to potentially naive users how to obtain and use these libraries, it seems better to just fix the included batteries. In the case of the binary installers for Windows and Mac OS X that are published on python.org, the version of OpenSSL used is entirely within the control of the Python core development team, but is currently limited to OpenSSL maintenance releases for the version initially shipped with the corresponding Python feature release. With increased popularity comes increased responsibility, and this proposal aims to acknowledge the fact that Python's popularity and adoption is at a sufficiently high level that some of our design and policy decisions have significant implications beyond the Python development community. As one example, the Python 2 ``ssl`` module does not support the Server Name Indication standard. While it is possible to obtain SNI support by using the third party ``requests`` client library, actually doing so currently requires using not only ``requests`` and its embedded dependencies, but also half a dozen or more additional libraries. The lack of support in the Python 2 series thus serves as an impediment to making effective use of SNI on servers, as Python 2 clients will frequently fail to handle it correctly. Another more critical example is the lack of SSL hostname matching in the Python 2 standard library - it is currently necessary to rely on a third party library, such as ``requests`` or ``backports.ssl_match_hostname`` to obtain that functionality in Python 2. The Python 2 series also remains more vulnerable to remote timing attacks on security sensitive comparisons than the Python 3 series, as it lacks a standard library equivalent to the timing attack resistant ``hmac.compare_digest()`` function. While appropriate secure comparison functions can be implemented in third party extensions, many users don't even consider the issue and use ordinary equality comparisons instead - while a standard library solution doesn't automatically fix that problem, it *does* make the barrier to resolution much lower once the problem is pointed out. Python 2.7 represents the only long term maintenance release the core development team has provided, and it is natural that there will be things that worked over a historically shorter maintenance lifespan that don't work over this longer support period. In the specific case of the problem described in this PEP, the simplest available solution is to acknowledge that long term maintenance of network security related modules *requires* the ability to add new features, even while retaining backwards compatibility for existing interfaces. For those familiar with it, it is worth comparing the approach described in this PEP with Red Hat's handling of its long term open source support commitments: it isn't the RHEL 6.0 release itself that receives 10 years worth of support, but the overall RHEL 6 *series*. The individual RHEL 6.x point releases within the series then receive a wide variety of new features, including security enhancements, all while meeting strict backwards compatibility guarantees for existing software. The proposal covered in this PEP brings our approach to long term maintenance more into line with this precedent - we retain our strict backwards compatibility requirements, but make an exception to the restriction against adding new features. To date, downstream redistributors have respected our upstream policy of "no new features in Python maintenance releases". This PEP explicitly accepts that a more nuanced policy is appropriate in the case of network security related features, and the specific change it describes is deliberately designed such that it is potentially suitable for Red Hat Enterprise Linux and its downstream derivatives. Rejected alternative: just advise developers to migrate to Python 3 ------------------------------------------------------------------- This alternative represents the status quo. Unfortunately, it has proven to be unworkable in practice, as the backwards compatibility implications mean that this is a non-trivial migration process for large applications and integration projects. While the tools for migration have evolved to a point where it is possible to migrate even large applications opportunistically and incrementally (rather than all at once) by updating code to run in the large common subset of Python 2 and Python 3, using the most recent technology often isn't a priority in commercial environments. Previously, this was considered an acceptable harm, as while it was an unfortunate problem for the affected developers to have to face, it was seen as an issue between them and their management chain to make the case for infrastructure modernisation, and this case would become naturally more compelling as the Python 3 series evolved. However, now that we're fully aware of the impact the limitations of the Python 2 standard library may be having on the evolution of internet security standards, I no longer believe that it is reasonable to expect platform and application developers to resolve all of the latent defects in an application's Unicode correctness solely in order to gain access to the network security enhancements already available in Python 3. While Ubuntu (and to some extent Debian as well) are committed to porting all default system services and scripts to Python 3, and to removing Python 2 from its default distribution images (but not from its archives), this is a mammoth task and won't be completed for the Ubuntu 14.04 LTS release (at least for the desktop image - it may be achieved for the mobile and server images). Fedora has even more work to do to migrate, and it will take a non-trivial amount of time to migrate the relevant infrastructure components. While Red Hat are also actively working to make it easier for users to use more recent versions of Python on our stable platforms, it's going to take time for those efforts to start having an impact on end users' choice of version, and any such changes also don't benefit the core platform infrastructure that runs in the integrated system Python by necessity. The OpenStack migration to Python 3 is also still in its infancy, and even though that's a project with an extensive and relatively robust automated test suite, it's still large enough that it is going to take quite some time to migrate fully to a Python 2/3 compatible code base. And that's just three of the highest profile open source projects that make heavy use of Python. Given the likely existence of large amounts of legacy code that lacks the kind of automated regression test suite needed to help support a migration from Python 2 to Python 3, there are likely to be many cases where reimplementation (perhaps even in Python 3) proves easier than migration. The key point of this PEP is that those situations affect more people than just the developers and users of the affected application: the existence of clients and servers with outdated network security infrastructure becomes something that developers of secure networked services need to take into account as part of their security design, and that's a problem that inhibits the adoption of better security standards. As Terry Reedy noted, if we try to persist with the status quo, the likely outcome is that commercial redistributors will attempt to do something like this on behalf of their customers *anyway*, but in a potentially inconsistent and ad hoc manner. By drawing the scope definition process into the upstream project we are in a better position to influence the approach taken to address the situation and to help ensure some consistency across redistributors. The problem is real, so *something* needs to change, and this PEP describes my preferred approach to addressing the situation. Rejected alternative: create and release Python 2.8 --------------------------------------------------- With sufficient corporate support, it likely *would* be possible to create and release Python 2.8 (it's highly unlikely such a project would garner enough interest to be achievable with only volunteers). However, this wouldn't actually solve the problem, as the aim is to provide a *relatively low impact* way to incorporate enhanced security features into integrated products and deployments that make use of Python 2. Upgrading to a new Python feature release would mean both more work for the core development team, as well as a more disruptive update that most potential end users would likely just skip entirely. Attempting to create a Python 2.8 release would also bring in suggestions to backport many additional features from Python 3 (such as ``tracemalloc`` and the improved coroutine support), making the migration from Python 2.7 to this hypothetical 2.8 release even riskier and more disruptive. This is not a recommended approach, as it would involve substantial additional work for a result that is actually less effective in achieving the original aim (which is to eliminate the current widespread use of the aging network security infrastructure in the Python 2 series). Furthermore, while I can't make any commitments to actually addressing this issue on Red Hat platforms, I *can* categorically rule out the idea of a Python 2.8 being of any use to me in even attempting to get it addressed. Rejected alternative: distribute the security enhancements via PyPI ------------------------------------------------------------------- While this initially appears to be an attractive and easier to manage approach, it actually suffers from several significant problems. Firstly, this is complex, low level, cross-platform code that integrates with the underlying operating system across a variety of POSIX platforms (including Mac OS X) and Windows. The CPython BuildBot fleet is already set up to handle continuous integration in that context, but most of the freely available continuous integration services just offer Linux, and perhaps paid access to Windows. Those services work reasonably well for software that largely runs on the abstraction layers offered by Python and other dynamic languages, as well as the more comprehensive abstraction offered by the JVM, but won't suffice for the kind of code involved here. The OpenSSL dependency for the network security support also qualifies as the kind of "complex binary dependency" that isn't yet handled well by the ``pip`` based software distribution ecosystem. Relying on a third party binary dependency also creates potential compatibility problems for ``pip`` when running on other interpreters like ``PyPy``. Another practical problem with the idea is the fact that ``pip`` itself relies on the ``ssl`` support in the standard library (with some additional support from a bundled copy of ``requests``, which in turn bundles ``backport.ssl_match_hostname``), and hence would require any replacement module to also be bundled within ``pip``. This wouldn't pose any insurmountable difficulties (it's just another dependency to vendor), but it *would* mean yet another copy of OpenSSL to keep up to date. This approach also has the same flaw as all other "improve security by renaming things" approaches: they completely miss the users who most need help, and raise significant barriers against being able to encourage users to do the right thing when their infrastructure supports it (since "use this other module" is a much higher impact change than "turn on this higher security setting"). Deprecating the aging SSL infrastructure in the standard library in favour of an external module would be even more user hostile than accepting the slightly increased risk of regressions associated with upgrading it in place. Last, but certainly not least, this approach suffers from the same problem as the idea of doing a Python 2.8 release: likely not solving the actual problem. Commercial redistributors of Python are set up to redistribute *Python*, and a pre-existing set of additional packages. Getting new packages added to the pre-existing set *can* be done, but means approaching each and every redistributor and asking them to update their repackaging process accordingly. By contrast, the approach described in this PEP would require redistributors to deliberately *opt out* of the security enhancements by deliberately downgrading the provided network security infrastructure, which most of them are unlikely to do. Rejected variant: provide a "legacy SSL infrastructure" branch -------------------------------------------------------------- Earlier versions of this PEP included the concept of a ``2.7-legacy-ssl`` branch that preserved the exact feature set of the Python 2.7.6 network security infrastructure. In my opinion, anyone that actually wants this is almost certainly making a mistake, and if they insist they really do want it in their specific situation, they're welcome to either make it themselves or arrange for a downstream redistributor to make it for them. If they are made publicly available, any such rebuilds should be referred to as "Python 2.7 with Legacy SSL" to clearly distinguish them from the official Python 2.7 releases that include more up to date network security infrastructure. After the first Python 2.7 maintenance release that implements this PEP, it would also be appropriate to refer to Python 2.7.6 and earlier releases as "Python 2.7 with Legacy SSL". Rejected variant: synchronise particular modules entirely with Python 3 ----------------------------------------------------------------------- Earlier versions of this PEP suggested synchronising the ``hmac``, ``hashlib`` and ``ssl`` modules entirely with their Python 3 counterparts. This approach proved too vague to build a compelling case for the exception, and has thus been replaced by the current more explicit proposal. Rejected variant: open ended backport policy -------------------------------------------- Earlier versions of this PEP suggested a general policy change related to future Python 3 enhancements that impact the general security of the internet. That approach created unnecessary uncertainty, so it has been simplified to propose backport a specific concrete set of changes. Future feature backport proposals can refer back to this PEP as precedent, but it will still be necessary to make a specific case for each feature addition to the Python 2.7 long term support release. Disclosure of Interest ====================== The author of this PEP currently works for Red Hat on test automation tools. If this proposal is accepted, I will be strongly encouraging Red Hat to take advantage of the resulting opportunity to help improve the overall security of the Python ecosystem. However, I do not speak for Red Hat in this matter, and cannot make any commitments on Red Hat's behalf. Acknowledgements ================ Thanks to Christian Heimes and other for their efforts in greatly improving Python's SSL support in the Python 3 series, and a variety of members of the Python community for helping me to better understand the implications of the default settings we provide in our SSL modules, and the impact that tolerating the use of SSL infrastructure that was defined in 2010 (Python 2.7) or even 2008 (Python 2.6) potentially has for the security of the web as a whole. Thanks to Donald Stufft and Alex Gaynor for identifying a more limited set of essential security features that allowed the proposal to be made more fine-grained than backporting entire modules from Python 3.4 [7,8]_. Christian and Donald also provided valuable feedback on a preliminary draft of this proposal. Thanks also to participants in the python-dev mailing list threads [1,2,5,6]_, as well as the various folks I discussed this issue with at PyCon 2014 in Montreal. References ========== .. [1] PEP 466 discussion (round 1) (https://mail.python.org/pipermail/python-dev/2014-March/133334.html) .. [2] PEP 466 discussion (round 2) (https://mail.python.org/pipermail/python-dev/2014-March/133389.html) .. [3] Marc-Andre Lemburg's OpenSSL feedback for Windows (https://mail.python.org/pipermail/python-dev/2014-March/133438.html) .. [4] Ned Deily's OpenSSL feedback for Mac OS X (https://mail.python.org/pipermail/python-dev/2014-March/133347.html) .. [5] PEP 466 discussion (round 3) (https://mail.python.org/pipermail/python-dev/2014-March/133442.html) .. [6] PEP 466 discussion (round 4) (https://mail.python.org/pipermail/python-dev/2014-March/133472.html) .. [7] Donald Stufft's recommended set of backported features (https://mail.python.org/pipermail/python-dev/2014-March/133500.html) .. [8] Alex Gaynor's recommended set of backported features (https://mail.python.org/pipermail/python-dev/2014-March/133503.html) Copyright ========= This document has been placed in the public domain. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 2

Re: [Python-Dev] devguide: Add note about Kushal Das' privs
by Antoine Pitrou April 19, 2014

April 19, 2014

On Mon, 14 Apr 2014 23:18:42 +0200 (CEST) brett.cannon <python-checkins(a)python.org> wrote: > http://hg.python.org/devguide/rev/c14c8a195fec > changeset: 686:c14c8a195fec > user: Brett Cannon <brett(a)python.org> > date: Mon Apr 14 17:18:37 2014 -0400 > summary: > Add note about Kushal Das' privs I have no objection to Kushal getting commit rights, but is there a public record of the discussion leading to this decision? I can't find anything on either python-dev or python-committers (perhaps my search skills are failing me). Also, welcome on board, Kushal! Regards Antoine.

5 4

Summary of Python tracker Issues
by Python tracker April 18, 2014

April 18, 2014

ACTIVITY SUMMARY (2014-04-11 - 2014-04-18) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4579 (+12) closed 28493 (+88) total 33072 (+100) Open issues with patches: 2118 Issues opened (68) ================== #17160: test_urllib2net fails http://bugs.python.org/issue17160 reopened by ddvento(a)ucar.edu #17861: put opcode information in one place http://bugs.python.org/issue17861 reopened by twouters #21201: Uninformative error message in multiprocessing.Manager() http://bugs.python.org/issue21201 opened by wojtekwalczak #21202: Naming a file` io.py` causes cryptic error message http://bugs.python.org/issue21202 opened by madison.may #21204: multiprocessing example does not work on Windows http://bugs.python.org/issue21204 opened by jmaki #21205: Unable to make decorated generator object to inherit generator http://bugs.python.org/issue21205 opened by SzieberthAdam #21207: urandom persistent fd - not re-openned after fd close http://bugs.python.org/issue21207 opened by kwirk #21211: pkgutil.find_loader() raises ImportError instead of returning http://bugs.python.org/issue21211 opened by eric.snow #21213: Memory bomb by incorrect custom serializer to json.dumps http://bugs.python.org/issue21213 opened by saaj #21216: getaddrinfo is wrongly considered thread safe on linux http://bugs.python.org/issue21216 opened by Julien.Palard #21217: inspect.getsourcelines finds wrong lines when lambda used argu http://bugs.python.org/issue21217 opened by ballingt #21220: Enhance obmalloc allocation strategy http://bugs.python.org/issue21220 opened by kristjan.jonsson #21221: Minor struct_time documentation bug http://bugs.python.org/issue21221 opened by lurchman #21224: BaseHTTPRequestHandler, update the protocol version to http 1. http://bugs.python.org/issue21224 opened by matrixise #21225: io.py: Improve docstrings for classes http://bugs.python.org/issue21225 opened by akuchling #21226: PyImport_ExecCodeModuleObject not setting module attributes http://bugs.python.org/issue21226 opened by trevor3 #21227: Decimal class error messages for integer division aren't good http://bugs.python.org/issue21227 opened by leewz #21228: Missing enumeration of HTTPResponse Objects methods of urllib. http://bugs.python.org/issue21228 opened by EvensF #21230: imghdr does not accept adobe photoshop mime type http://bugs.python.org/issue21230 opened by faiz #21231: Issue a python 3 warning when old style classes are defined. http://bugs.python.org/issue21231 opened by alex #21232: Use of '1' instead of 'True' as 'splitlines' argument in diffl http://bugs.python.org/issue21232 opened by jayanthkoushik #21233: Add *Calloc functions to CPython memory allocation API http://bugs.python.org/issue21233 opened by njs #21235: importlib's spec module create algorithm is not exposed http://bugs.python.org/issue21235 opened by aronacher #21236: patch to use cabinet.lib instead of fci.lib (fixes build with http://bugs.python.org/issue21236 opened by mst #21237: Update Python 2/3 porting HOWTO's suggestion for dealing with http://bugs.python.org/issue21237 opened by brett.cannon #21240: Add an abstactmethod directive to the Python ReST domain http://bugs.python.org/issue21240 opened by eric.snow #21242: Generalize configure check for working Python executable http://bugs.python.org/issue21242 opened by brett.cannon #21243: Auto-generate exceptions.c from a Python file http://bugs.python.org/issue21243 opened by brett.cannon #21247: test_asyncio: test_subprocess_send_signal hangs on Fedora buil http://bugs.python.org/issue21247 opened by opoplawski #21249: removing pythonXY.zip from sys.path results in additional test http://bugs.python.org/issue21249 opened by doko #21250: sqlite3 doesn't have unit tests for 'insert or [algorithm]' fu http://bugs.python.org/issue21250 opened by Alex.Lord #21251: Standard library trace module crashes with exception http://bugs.python.org/issue21251 opened by mkolman #21252: Lib/asyncio/events.py has tons of docstrings which are just "X http://bugs.python.org/issue21252 opened by alex #21253: Difflib.compare() crashes when sequences contain little or no http://bugs.python.org/issue21253 opened by nnja #21254: PropertyMock refuses to raise AttributeErrror as a side effect http://bugs.python.org/issue21254 opened by michael.foord #21255: Attaching a PropertyMock records calls http://bugs.python.org/issue21255 opened by michael.foord #21256: Sort keyword arguments in mock _format_call_signature http://bugs.python.org/issue21256 opened by michael.foord #21257: Document parse_headers function of http.client http://bugs.python.org/issue21257 opened by orsenthil #21258: Add __iter__ support for mock_open http://bugs.python.org/issue21258 opened by michael.foord #21259: replace "except: pass" by "except Exception: pass" http://bugs.python.org/issue21259 opened by matrixise #21261: Teach IDLE to Autocomplete dictionary keys http://bugs.python.org/issue21261 opened by rhettinger #21263: test_gdb failures on os x 10.9.2 http://bugs.python.org/issue21263 opened by sam.kimbrel #21264: test_compileall fails to build in the installed location http://bugs.python.org/issue21264 opened by doko #21265: ConfigParser allows "get(*, raw=True), but no corresponding "s http://bugs.python.org/issue21265 opened by metagriffin #21266: test_zipfile fails to run in the installed location http://bugs.python.org/issue21266 opened by doko #21269: Provide args and kwargs attributes on mock call objects http://bugs.python.org/issue21269 opened by michael.foord #21270: unittest.mock.call object has inherited count method http://bugs.python.org/issue21270 opened by michael.foord #21271: reset_mock needs parameters to also reset return_value and sid http://bugs.python.org/issue21271 opened by michael.foord #21272: use _sysconfigdata to itinialize distutils.sysconfig http://bugs.python.org/issue21272 opened by doko #21273: don't defined socket constants which are not implemented for G http://bugs.python.org/issue21273 opened by doko #21277: don't try to link _ctypes with a ffi_convenience library http://bugs.python.org/issue21277 opened by doko #21278: Running the test suite with -v makes the test_ctypes and the t http://bugs.python.org/issue21278 opened by ddvento(a)ucar.edu #21279: str.translate documentation incomplete http://bugs.python.org/issue21279 opened by bgailer #21280: shutil.make_archive() with default "root_dir" parameter raises http://bugs.python.org/issue21280 opened by vadmium #21281: DEBUGGING: Simultaneous stopping of all threads on breakpoint http://bugs.python.org/issue21281 opened by azyr #21282: setup.py: More informative error msg for modules which built b http://bugs.python.org/issue21282 opened by Lukas.Vacek #21284: IDLE reformat tests fail in presence of non-default FormatPara http://bugs.python.org/issue21284 opened by rhettinger #21287: Better support for AF_PACKET on opensolaris (illumos) http://bugs.python.org/issue21287 opened by igor.pashev #21288: hashlib.pbkdf2_hmac Hash Constructor http://bugs.python.org/issue21288 opened by aronacher #21289: make.bat not building documentation http://bugs.python.org/issue21289 opened by dsawyer #21291: subprocess Popen objects are not thread safe w.r.t. wait() and http://bugs.python.org/issue21291 opened by gregory.p.smith #21292: C API in debug fails http://bugs.python.org/issue21292 opened by Banger #21293: Remove "capsule hack" from object.c? http://bugs.python.org/issue21293 opened by larry #21295: Python 3.4 gives wrong col_offset for Call nodes returned from http://bugs.python.org/issue21295 opened by Aivar.Annamaa #21296: smtplib Sends Commands in Lower-Case http://bugs.python.org/issue21296 opened by luiji #21297: skipinitialspace in the csv module only skips spaces, not "whi http://bugs.python.org/issue21297 opened by Daniel.Andersson #21298: Cheese shop registration stopped working for me recently http://bugs.python.org/issue21298 opened by tlevine #21300: Docs (incorrectly) suggest email.policy.default is the default http://bugs.python.org/issue21300 opened by valhallasw Most recent 15 issues with no replies (15) ========================================== #21300: Docs (incorrectly) suggest email.policy.default is the default http://bugs.python.org/issue21300 #21298: Cheese shop registration stopped working for me recently http://bugs.python.org/issue21298 #21297: skipinitialspace in the csv module only skips spaces, not "whi http://bugs.python.org/issue21297 #21296: smtplib Sends Commands in Lower-Case http://bugs.python.org/issue21296 #21293: Remove "capsule hack" from object.c? http://bugs.python.org/issue21293 #21287: Better support for AF_PACKET on opensolaris (illumos) http://bugs.python.org/issue21287 #21284: IDLE reformat tests fail in presence of non-default FormatPara http://bugs.python.org/issue21284 #21282: setup.py: More informative error msg for modules which built b http://bugs.python.org/issue21282 #21281: DEBUGGING: Simultaneous stopping of all threads on breakpoint http://bugs.python.org/issue21281 #21280: shutil.make_archive() with default "root_dir" parameter raises http://bugs.python.org/issue21280 #21278: Running the test suite with -v makes the test_ctypes and the t http://bugs.python.org/issue21278 #21271: reset_mock needs parameters to also reset return_value and sid http://bugs.python.org/issue21271 #21270: unittest.mock.call object has inherited count method http://bugs.python.org/issue21270 #21269: Provide args and kwargs attributes on mock call objects http://bugs.python.org/issue21269 #21266: test_zipfile fails to run in the installed location http://bugs.python.org/issue21266 Most recent 15 issues waiting for review (15) ============================================= #21300: Docs (incorrectly) suggest email.policy.default is the default http://bugs.python.org/issue21300 #21293: Remove "capsule hack" from object.c? http://bugs.python.org/issue21293 #21291: subprocess Popen objects are not thread safe w.r.t. wait() and http://bugs.python.org/issue21291 #21289: make.bat not building documentation http://bugs.python.org/issue21289 #21287: Better support for AF_PACKET on opensolaris (illumos) http://bugs.python.org/issue21287 #21279: str.translate documentation incomplete http://bugs.python.org/issue21279 #21273: don't defined socket constants which are not implemented for G http://bugs.python.org/issue21273 #21272: use _sysconfigdata to itinialize distutils.sysconfig http://bugs.python.org/issue21272 #21265: ConfigParser allows "get(*, raw=True), but no corresponding "s http://bugs.python.org/issue21265 #21259: replace "except: pass" by "except Exception: pass" http://bugs.python.org/issue21259 #21258: Add __iter__ support for mock_open http://bugs.python.org/issue21258 #21256: Sort keyword arguments in mock _format_call_signature http://bugs.python.org/issue21256 #21249: removing pythonXY.zip from sys.path results in additional test http://bugs.python.org/issue21249 #21240: Add an abstactmethod directive to the Python ReST domain http://bugs.python.org/issue21240 #21233: Add *Calloc functions to CPython memory allocation API http://bugs.python.org/issue21233 Top 10 most discussed issues (10) ================================= #12916: Add inspect.splitdoc http://bugs.python.org/issue12916 21 msgs #21233: Add *Calloc functions to CPython memory allocation API http://bugs.python.org/issue21233 20 msgs #20438: inspect: Deprecate getfullargspec? http://bugs.python.org/issue20438 14 msgs #21199: Python on 64-bit Windows uses signed 32-bit type for read leng http://bugs.python.org/issue21199 14 msgs #21259: replace "except: pass" by "except Exception: pass" http://bugs.python.org/issue21259 14 msgs #21227: Decimal class error messages for integer division aren't good http://bugs.python.org/issue21227 11 msgs #20434: Fix error handler of _PyString_Resize() on allocation failure http://bugs.python.org/issue20434 10 msgs #20578: BufferedIOBase.readinto1 is missing http://bugs.python.org/issue20578 10 msgs #17861: put opcode information in one place http://bugs.python.org/issue17861 9 msgs #21220: Enhance obmalloc allocation strategy http://bugs.python.org/issue21220 9 msgs Issues closed (81) ================== #5420: Queue deprecation warning patch http://bugs.python.org/issue5420 closed by rhettinger #6490: os.popen documentation is probably wrong http://bugs.python.org/issue6490 closed by akuchling #6727: ImportError when package is symlinked on Windows http://bugs.python.org/issue6727 closed by r.david.murray #8060: PEP 3101 string formatting missing engineering presentation ty http://bugs.python.org/issue8060 closed by eric.smith #8931: '#' has no effect with 'c' type http://bugs.python.org/issue8931 closed by eric.smith #10224: Build 3.x documentation using python3.x http://bugs.python.org/issue10224 closed by georg.brandl #10481: subprocess PIPEs are byte streams http://bugs.python.org/issue10481 closed by akuchling #10983: Errors in http.client.HTTPConnection class (python3) http://bugs.python.org/issue10983 closed by orsenthil #11316: RFC822 header parsing API inconsistencies between httplib.HTTP http://bugs.python.org/issue11316 closed by r.david.murray #12546: builtin __format__ methods cannot fill with \x00 char http://bugs.python.org/issue12546 closed by eric.smith #13244: WebSocket schemes in urllib.parse http://bugs.python.org/issue13244 closed by orsenthil #14216: ImportError: No module named binascii http://bugs.python.org/issue14216 closed by akuchling #15234: avoid runtime library path for extensions found in system dire http://bugs.python.org/issue15234 closed by doko #15370: test_runpy should include namespace package tests http://bugs.python.org/issue15370 closed by akuchling #15840: Ambiguity with regard to the effect of accessing a closed IOBa http://bugs.python.org/issue15840 closed by akuchling #15916: change doctest DocTestSuite not to raise ValueError if no docs http://bugs.python.org/issue15916 closed by r.david.murray #16236: Doc/Makefile should have $PYTHON=python2 http://bugs.python.org/issue16236 closed by berker.peksag #17009: "Thread Programming With Python" should be removed http://bugs.python.org/issue17009 closed by r.david.murray #17498: error responses from server are masked in smtplib when server http://bugs.python.org/issue17498 closed by r.david.murray #17660: mock.patch could whitelist builtins to not need create=True http://bugs.python.org/issue17660 closed by python-dev #17826: Setting a side_effect on mock from create_autospec doesn't wor http://bugs.python.org/issue17826 closed by michael.foord #18321: Multivolume support in tarfile module http://bugs.python.org/issue18321 closed by lars.gustaebel #18566: In unittest.TestCase docs for setUp() and tearDown() don't men http://bugs.python.org/issue18566 closed by terry.reedy #18628: Better index entry for encoding declarations http://bugs.python.org/issue18628 closed by r.david.murray #18650: intermittent test_pydoc failure on 3.4.0a1 http://bugs.python.org/issue18650 closed by ned.deily #20103: Documentation of itertools.accumulate is confused http://bugs.python.org/issue20103 closed by akuchling #20307: Android's failure to expose SYS_* system call constants causes http://bugs.python.org/issue20307 closed by gregory.p.smith #20480: Add ipaddress property to get name of reverse DNS PTR record http://bugs.python.org/issue20480 closed by eric.smith #20624: Clarify recommendation to inherit from Exception http://bugs.python.org/issue20624 closed by mark.dickinson #20636: Better repr for tkinter widgets http://bugs.python.org/issue20636 closed by serhiy.storchaka #20701: warning in compileall.rst http://bugs.python.org/issue20701 closed by Arfrever #20874: Tutorial section on starting python is out of date http://bugs.python.org/issue20874 closed by r.david.murray #20904: HAVE_PY_SET_53BIT_PRECISION for m68k http://bugs.python.org/issue20904 closed by python-dev #20956: tokenize module claims tokenize.tokenize returns namedtuple, b http://bugs.python.org/issue20956 closed by python-dev #20968: mock.MagicMock does not mock __truediv__ http://bugs.python.org/issue20968 closed by michael.foord #21028: ElementTree objects should support all the same methods as Ele http://bugs.python.org/issue21028 closed by rhettinger #21164: print unicode in Windows console http://bugs.python.org/issue21164 closed by terry.reedy #21169: getpass.getpass() fails with non-ASCII characters in prompt http://bugs.python.org/issue21169 closed by serhiy.storchaka #21170: Incorrect signature for unittest.TestResult.startTestRun(), .s http://bugs.python.org/issue21170 closed by terry.reedy #21171: Outdated usage str.encode('rot-13') in rot13 codec http://bugs.python.org/issue21171 closed by benjamin.peterson #21177: ValueError: byte must be in range(0, 256) http://bugs.python.org/issue21177 closed by pitrou #21188: Broken link http://bugs.python.org/issue21188 closed by loewis #21193: pow(a, b, c) should not raise TypeError when b is negative and http://bugs.python.org/issue21193 closed by mark.dickinson #21194: json.dumps with ensure_ascii=False doesn't escape control char http://bugs.python.org/issue21194 closed by ned.deily #21197: venv does not create lib64 directory and appropriate symlinks http://bugs.python.org/issue21197 closed by python-dev #21203: logging configurators ignoring documented options http://bugs.python.org/issue21203 closed by python-dev #21206: Provide legit HTTPS certificate for http://bugs.python.org/, r http://bugs.python.org/issue21206 closed by ned.deily #21208: Change default behavior of arguments with type bool when optio http://bugs.python.org/issue21208 closed by r.david.murray #21209: q.put(some_tuple) fails when PYTHONASYNCIODEBUG=1 http://bugs.python.org/issue21209 closed by python-dev #21210: Warnings in Doc/library/json.rst http://bugs.python.org/issue21210 closed by python-dev #21212: Documentation of octal representation http://bugs.python.org/issue21212 closed by python-dev #21214: PEP8 doesn't verifies last line. http://bugs.python.org/issue21214 closed by mark.dickinson #21215: build-deps instructions for Ubuntu http://bugs.python.org/issue21215 closed by r.david.murray #21218: Test failure for test_ssl.test_default_ecdh_curve on OS X http://bugs.python.org/issue21218 closed by ned.deily #21219: WARNING: Inline literal start-string without end-string. http://bugs.python.org/issue21219 closed by orsenthil #21222: Mock create_autospec with name argument fails http://bugs.python.org/issue21222 closed by python-dev #21223: fix test_site/test_startup_imports when some of the extensions http://bugs.python.org/issue21223 closed by doko #21229: Path used for HTTP PUT request doesn't match the description http://bugs.python.org/issue21229 closed by python-dev #21234: __contains__ and friends should check "is" for all elements fi http://bugs.python.org/issue21234 closed by rhettinger #21238: unittest.mock.Mock should not allow you to use non-existent as http://bugs.python.org/issue21238 closed by python-dev #21239: unittest.mock.patch.stopall intermittently doesn't work when t http://bugs.python.org/issue21239 closed by python-dev #21241: Variable name with number causes interactive console to crash http://bugs.python.org/issue21241 closed by ned.deily #21244: distutils fails to build C extensions with XCode 5.1 and OS X http://bugs.python.org/issue21244 closed by ned.deily #21245: Logging Logger.exception documentation http://bugs.python.org/issue21245 closed by python-dev #21246: test_ssl handshake failure http://bugs.python.org/issue21246 closed by pitrou #21248: BROWSER env var described inaccurately in webbrowser docs http://bugs.python.org/issue21248 closed by python-dev #21260: python malloc mach_vm_map failed http://bugs.python.org/issue21260 closed by ned.deily #21262: assert_not_called method for mocks http://bugs.python.org/issue21262 closed by python-dev #21267: mktime_tz may return wrong result for past dates before Python http://bugs.python.org/issue21267 closed by akira #21268: Update pydoc module docstring http://bugs.python.org/issue21268 closed by eric.araujo #21274: define PATH_MAX for GNU/Hurd in Python/pythonrun.c http://bugs.python.org/issue21274 closed by doko #21275: fix a socket test on KFreeBSD http://bugs.python.org/issue21275 closed by doko #21276: don't define USE_XATTRS on kfreebsd and the Hurd http://bugs.python.org/issue21276 closed by doko #21283: A escape character is used when a REGEXP is an argument of "st http://bugs.python.org/issue21283 closed by eric.smith #21285: refactor and fix curses configure checks http://bugs.python.org/issue21285 closed by doko #21286: Refcounting information missing in docs for Python 3.4 and abo http://bugs.python.org/issue21286 closed by python-dev #21290: imaplib.error when importing email package http://bugs.python.org/issue21290 closed by r.david.murray #21294: len wrong help http://bugs.python.org/issue21294 closed by python-dev #21299: Spam http://bugs.python.org/issue21299 closed by berker.peksag #984870: curses: getmaxyx() breaks when the window shrinks http://bugs.python.org/issue984870 closed by akuchling #1521196: smtplib login fails with aol smtp server http://bugs.python.org/issue1521196 closed by r.david.murray

1 0

Re: [Python-Dev] [Python-checkins] cpython (merge 3.4 -> default): Merge 3.4
by Benjamin Peterson April 17, 2014

April 17, 2014

On Thu, Apr 17, 2014, at 8:55, matthias.klose wrote: > http://hg.python.org/cpython/rev/1d1aefd00f07 > changeset: 90382:1d1aefd00f07 > parent: 90380:517de1983677 > parent: 90381:1a00e04a233d > user: doko(a)ubuntu.com > date: Thu Apr 17 17:55:03 2014 +0200 > summary: > Merge 3.4 > > files: > Misc/NEWS | 6 ++++++ > setup.py | 15 +++++++++++++-- > 2 files changed, 19 insertions(+), 2 deletions(-) > > > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -244,8 +244,14 @@ > Build > ----- > > +<<<<<<< local > - Issue #17861: Tools/scripts/generate_opcode_h.py automatically > regenerates > Include/opcode.h from Lib/opcode.py if the later gets any change. > +======= > +- Issue #15234: For BerkelyDB and Sqlite, only add the found library and > + include directories if they aren't already being searched. This avoids > + an explicit runtime library dependency. > +>>>>>>> other You might want to actually resolve this conflict.

1 0

is the concept of 'reference ownership' no long applicable in Python 3.4?
by Jianfeng Mao April 17, 2014

April 17, 2014

Hi, I noticed the following changes in the C API manuals from 3.3.5 (and earlier versions) to 3.4. I don't know if these changes are deliberate and imply that we C extension developers no longer need to care about 'reference ownership' because of some improvements in 3.4. Could anyone clarify it? Thanks, Jianfeng --- 3.4 C API reference manual ---- PyObject<https://docs.python.org/3/c-api/structures.html#c.PyObject>* PySequence_GetItem(PyObject<https://docs.python.org/3/c-api/structures.html#c.PyObject> *o, Py_ssize_t i)¶<https://docs.python.org/3/c-api/sequence.html#c.PySequence_GetItem> Return the ith element of o, or NULL on failure. This is the equivalent of the Python expression o[i]. PyObject<https://docs.python.org/3.4/c-api/structures.html#c.PyObject>* PyList_GetItem(PyObject<https://docs.python.org/3.4/c-api/structures.html#c.PyObject> *list, Py_ssize_t index) Return the object at position index in the list pointed to by list. The position must be positive, indexing from the end of the list is not supported. If index is out of bounds, return NULL and set an IndexError<https://docs.python.org/3.4/library/exceptions.html#IndexError> exception. --- 3.3.5 C API reference manual --- PyObject<https://docs.python.org/3.3/c-api/structures.html#PyObject>* PySequence_GetItem(PyObject<https://docs.python.org/3.3/c-api/structures.html#PyObject> *o, Py_ssize_t i)¶<https://docs.python.org/3.3/c-api/sequence.html#PySequence_GetItem> Return value: New reference. Return the ith element of o, or NULL on failure. This is the equivalent of the Python expression o[i]. PyObject<https://docs.python.org/3.3/c-api/structures.html#PyObject>* PyList_GetItem(PyObject<https://docs.python.org/3.3/c-api/structures.html#PyObject> *list, Py_ssize_t index) Return value: Borrowed reference. Return the object at position index in the list pointed to by list. The position must be positive, indexing from the end of the list is not supported. If index is out of bounds, return NULL and set an IndexError<https://docs.python.org/3.3/library/exceptions.html#IndexError> exception.

2 2

[numpy wishlist] PyMem_*Calloc
by Julian Taylor April 17, 2014

April 17, 2014

Hi, In NumPy what we want is the tracing, not the exchangeable allocators. I don't think it is a good idea for the core of a whole stack of C-extension based modules to replace the default allocator or allowing other modules to replace the allocator NumPy uses. I think it would be more useful if Python provides functions to register memory allocations and frees and the tracemalloc module registers handlers for these register functions. If no trace allocation tracer is registered the functions just return immediately. That way the tracemalloc can be used with arbitrary allocators as long as they register their allocations with Python. For example a hugepage allocator, which you would not want to use that as the default allocator for all python objects, but you may still want to trace its usage: my_hugetlb_alloc(size) p = mmap('hugepagefs', ..., MAP_HUGETLB); PyMem_Register_Alloc(p, size, __func__, __line__); return p my_hugetlb_free(p); PyMem_Register_Free(p, __func__, __line__); munmap(p, ...); normally the registers are nops, but if tracemalloc did register tracers the memory is tracked, e.g. tracemodule does this on start(): tracercontext.register_alloc = trace_alloc tracercontext.register_free = trace_free tracercontext.data = mycontext PyMem_SetTracer(&tracercontext) Regards, Julian Taylor

3 3

Language Summit notes
by Taavi Burns April 17, 2014

April 17, 2014

I was the guy at the back taking pictures. I also took notes! :) Language Summit --------------- Intros ~~~~~~ Everyone starts by doing a 15 second intro. People are good and keep it short, and we make it through all ~40 people. People start introduce themselves saying "module X is my fault". Guido just says, "It's my fault." Release Management ~~~~~~~~~~~~~~~~~~ Larry Hastings talks about release versioning! How to use branches for the 3.5 RC process. When RCs start, should default be locked down, should it become 3.5.1, or should it become 3.6? Or something else? After a bit of discussion, people asked to move the decision to email. PyPy ~~~~ PyPy update: 7 iterations of stm! (compared to "The JIT took 5?") STM is 25-50% slower on a single-threaded workload, but shows nearly linear speedups with more cores on non-conflicting workloads. Brett Cannon: "So it's only HOW much faster than CPython?" It's not sure if there will be multiple PyPy releases or everyone takes the hit and STM is default. STM fundraiser II is active now. PyPy oving to 2.7.6 stdlib soon. A bunch of effort has gone to supporting CFFI porting in the community which aids porting of packages to PyPy. Something about Python 3 support that I didn't quite catch. Alex Gaynor says that most people who look to PyPy are looking for Python 2 right now. IronPython ~~~~~~~~~~ IronPython is active again! New contributors! Python 3 support! What can the PSF do to help IronPython? IronPython should ask for support if it would help. Jython ~~~~~~ Nobody from the team was present, but they sent a report which Michael Foord talked through with the group. Jython moving to Java 7 minimum runtime. Added buffer protocol (memoryview etc), required for Python 3 support. PyPA tooling? CFFI backend is coming! EuroPython sprint in July. Looking forward to Python 3 because of string types vs JVM. But it's slow (work?). Packaging ~~~~~~~~~ pip will suddenly need to become more user-friendly, now that it's bundled with 3.4. Instead of being used by people who need it, it's going to be presented to people who don't know what it is. Nick Coghlan: "PyPI's security model: it didn't have one." Richard Jones says the new PyPI code is nearly ready. It has some rough bits (OAuth, OpenID) but they're rough anyway. Replacing old crufty code with nice code; then moving forward with shiny new features. Oh, and wheels! Want to be at the point where Windows and OSX users can download the binary distro from python.org, and ``pip install django`` and have it work. Numpy is apparently still a sore point; conda would still be a better way for most people. Largely two kinds of packages: those that are self-contained (may include c-extensions though), and those that depend on system libraries (e.g. link to libmysql). Packaging has to make both of those work, and if it's made easier to make distro packages (i.e. RedHat packages, Debian packages, etc), it should be easier to build conda packages. Nick doesn't want to cross the line where python packaging (PyPI) does the distro side of things. Currently, "we do c extensions, but not linking to C libraries". Also, "metadata 2" has been put off by ~12 months, but is becoming the last thing standing before dropping setup.py and having wheels for all? Bits of metdata 2 have been pushed out as extensions or "decide later" bits, which has been instructive in terms of "how far can we get". Build chain in stdlib: bad idea, release schedules are different. Need a build system that supports all the verisons in the wild. Setuptools is now version-independent distutils. Oh, and Apple screwed up python, building it with a compiler that's not in xcode. ``pkg_resources`` has a dependency on ``setuptools`` which is WRONG, needs to be the other way around. But ``pkg_resources`` does what it does very well. Technically pip itself doesn't ship with 3.4, but rather ensurepip. This is so that new versions of pip can be brought in in point releases, and for security updates. Windows and OSX installers run ensurepip. ``virtualenv`` installs pip by default at this point, too. Someone asks, "Can we do that for SSL?" Everyone laughs. Disussion about packaging continues. Glyph asks if the PSF could fund a usability study on installing Python. People generally seem to think it's a good idea. Go to `packaging.python.org <http://packaging.python.org/en/latest/>`__! Documentation! Quite a few sections are already filled in: please contribute! There is no "one installer" that has everything you need for 2.7 right now. Nick: "C development is stupidly complicated." Nature of the old guard of Python programmers: you were already a C developer, and just wanted Python to help make some things easier. Nick got into Python for DSP testing. The demographics have changed. How do we change the docs and ecosystem to avoid the assumption that Python programmers already know how to program in C? "We have the wrong people maintaining the docs." "A distro is a pre-packaged group of binaries." Linux distros do it for Linux; On OSX you have macports, fink, and homebrew (OSX isn't itself a distro, except that it is because they do ship with *some* libraries, which are often broken); and on Windows there's Enthought, Activestate, and cygwin. Need SEO to get Windows users searching for "installers" to find "packages". Want to turn PyPI into "write metadata, upload source, PyPI does the rest", i.e. building c-extensions. Compiling c-extensions is particularly painful on Windows becuase Python 2.7 is build with Visual Studio 2008. OSX has similar issues where the system Python isn't compiled with what's in XCode. Ways to get the python.org Python packages to include the required bits: 2.7 is missing pip, 3.4 ships with it. If you want a GUI…then maybe you should just get Enthought or ActiveState? Python(x,y) is another option on Windows. Lunch ~~~~~ There was food! Pyston ~~~~~~ Pyston: Python on LLVM, coming out of Dropbox. Rebuilding a complete VM targetted for Python. Pro: optimize for Python, most custom you can get. Drawback: most work you can do. A different kind of JIT from PyPy: method-at-a-time. "Directly" from Python to machine code. Generate a fastpath for the types that you're seeing to avoid dynamic overheads. `speed.pypy.org <speed.pypy.org>`__ and `speed.python.org <speed.python.org>`__: suggested for pyston to help move it along. Benchmarks aren't just to show how good you are, they're a guide to where you can improve. Unladen Swallow did great work compiling a list of less synthetic benchmarks. Pyston memory usage is higher (includes jit etc), object size is smaller (inferred slots), function calls are bigger (than CPython). There's an LLVM IR interpreter for pre-JIT execution. Looking to add a "quick, IR to machine code" skipping the expensive bits. [Taavi: sounds like Firefox's `collection of OdinMonkey, IonMonkey, etc <https://blog.mozilla.org/javascript/2014/01/23/the-monkeys-in-2013/>`__, where JS is JITted very quickly to not-particularly-efficient machine code, and re-JITted more expensively as the hot codepaths are found.] Optional static type checking ----------------------------- Presentation by Jukka Lehtosalo on Mypy's static type checking, making use of type annotations in Python 3: - Faster than unit tests - Compatible with existing VMs (no runtime checks, just compilation checks; errors are warnings at runtime) - Documentation - Tools (IDEs) - Mypy: import typing; then you get static type checks! All valid programs, using Python 3 type annotations. - Can always fall back: from typing import Dict, Any; d = Dict[str, Any]; d[x] = 'y'; d[x] += 2 - `www.mypy-lang.org <http://www.mypy-lang.org>`__ - See also `https://github.com/google/pytypedecl <www.mypy-lang.org>`__ Michael Foord: mock and autospec; having function return annotations would be nice so an autospec'd mock can return more autospec'd mocks PEP8 said "don't use annotations", but that was only intended for stdlib (so that people were free to play with annotations in their own code). It was said that stdlib ought to be a good example of how to write Python code. But it isn't (right now, at least). Education --------- No docs on python.org for teachers (links). Mostly just "teach yourself" tutorials. Want things more like lesson plans. Having a new doc section for "absolute beginners" (Jessica McKellar volunteers to write it!) would be helpful; beginners often download the other python (2/3) from the tutorial they're trying to do. AP exams are starting to allow Python, but it's 10% of the AP CS exams. Jessica's working on making it Real. People on the APCS board already want it, just need the numbers to show that it's the right choice to teach to students. "The packaging and install programs, oh god." New users are good at contributing to docs. StackOverflow gets refd'd in bug tickets "see this StackOverflow post for why this is a problem/confusing". Selena Deckelmann talked about having a list of mitigations for teachers to help students using Windows and OSX. Lots of non-obvious "gotchas". Lots of outreach things. Women In Tech Linux kernel thing received 144 patches before selecting anyone (because applicants had to submit a patch to apply in the first place), and ended up adding an extra 1-2 people beyhond their planned 7 acceptees. Read the docs website! "Write the docs" conference! Yay! Learning to use the tooling currently required for contribution is a high bar. Being able to submit doc changes without a pull request might help. `bugs.python.org <http://bugs.python.org>`__: Selena filed one for a doc bug. But there is no "docs" type tag. Is a dead (404) link an "enhancement"? "Bug"? Should github issues be used for python.org website bugs? We have too many bug trackers? One for bugs.pycon.org, for pypi, for website, and a bugtracker for the bugtracker… Richard Jones is no longer working on Roundup, and reduced pypi from 3 bug trackers to 2?!? Too many places to report problems, and it's not clear how to even find them all. Terminology and vocabulary: if there are no words that seem to have anything to do with their problem, new bug sumitters will walk away. Being able to report a doc bug without specifying a type (because they're more worried about using the wrong type) might help. Jessica suggesting ("OPW"s: outreach programmes?) to talk about how to fill the pipeline. India and UK have re-done syllabi to include programming. India has put Python in explicitly, but there's pushback from parents and teachers because they don't know what this "strange language" is. In the UK, there's no language mandate, but children 7 and over are to be taught programming. The biggest problem is that the teachers are used to teaching spreadsheets and word processing, not programming. Raspberry PIs are common, £25 version includes minecraft. Kids usually start with Scratch and move to Python. Selena says that some American parents are worried because they don't know if their kids could get jobs [Taavi: loud surprise from audience]. Our industry knows the demand, but apparently it's not "common knowledge". Bringing parents and kids together in workshops, meeting people who have tech jobs and businesspeople who want to hire for tech jobs could help educate parents who don't realise there really are jobs… Pain points; why Mercurial isn't on Python 3 -------------------------------------------- Currently supports 2.4, 2.5, 2.6, 2.7. 2.4 sucks, but there's nothing much extra in 2.5 to make dropping 2.4 worth it. Mercurial will stop supporting 2.7 when RHEL7 is end-of-life. "We'll all be dead, I'll be on a differnet project." -- Matt Mackall Mercurial is bytestrings everywhere (intentionally), which isn't nearly as nice in Python 3 at this time. Most important benchmark: startup time. "hello world" in 2.7: 0.017s. In 3.4, 0.043s. git is 10x faster (Perl. Or bash.). Estimate of 1 person full time for 1.5 years to get mercurial running on Python 3, at half the speed of Python 2. It's not a pressing issue for anyone (that Matt knows of). People ask about Python 3 support once every few months. Matt's asking for 2.8. "I want a path forward that's not end-of-life." Looking for new features, bugfixes? Guido: "We can keep 2.7 alive for longer. … It's reasonable. 2.8 is [a can of worms]." 3/4 of pypi downloads are for 2.7 (biased sample, but of millions a month) Glyph: 2.8 would give an opportunity to be more aggressive, e.g. raise warnings on implicit string coercion. Guido disagrees (people will stay away from loud warnings that break their code). Nick: there are missing warnings for "-3", but they could be added in 2.7. Guido: What's going to be in 2.8? Tres: All the backports from pypi. Guido: Can we prevent people from adding new features along with the backports? The biggest difference for 2 =< 3 is str/bytes. Guido: There is not going to be a 2.8. Another good reason for a 2.8: Windows. 2.7 on Windows is compiled with Visual Studio 2008, and changing compiler is (apparently) a real problem. The total amount of churn any 2.8 release would cause is too much. (packaging, porting, etc) Even if it's a no-op from 2.7.6 to 2.8.0. 3.5 --- Kevin Modzelewski: Don't do 3.5, call it 4.0, then you can tell 2.x people they're TWO major versions behind! :) Guido: Can we get the PSF to issue a press release that 2.7 support will continue? (Matt wants to know that support is not ending.) Bring up a list of things that would be useful in a 2.8, so we can see what to do in 3.5. Glyph might start the list. Someone should also look through existing PEPs, because stuff might already be done. Guido asked about 2.4ish =< 2.7-single-source. "modernize" was mentioned, but it's abandoned. [Taavi: modernize and one other library were mentioned at the Python 3 panel discussion at the Montréal Python meetup on April 14th. Video will probably show up `on YouTube <https://www.youtube.com/user/MontrealPython/playlists>`__ as MP-45 in a few weeks?] Talk of the PSF paying experienced contractors to do certain bits of work. OpenStack is looking to do single-source Python 2/3. Other helpful things: instead of saying "print: invalid syntax", say something like "you need parentheses in Python 3". Adding AI to error messaging so that it can provide helpful messages would be great! Tooling to help port 2.x to 2.7+3-single-source would be great. Action items ------------ - Update pep to declare that 2.7 is not dead, and getting new maintenance updates. - Glyph will tell us what to do in 3.5. -- taa /*eof*/

7 7

Timing breakdown of Py_InitializeEx_Private()
by Brett Cannon April 16, 2014

April 16, 2014

To finish my timing work I decided to see where Py_InitializeEx_Private() spends its time. The following is a breakdown measured in microseconds running using -E: INIT: setlocale: 11 envvar: 2 random init: 2 interp creation: 15 thread creation: 6 GIL: 10 _Py_ReadyTypes(): 930 more types: 44 builtins: 141 exceptions: 287 sys: 258 _PyImport_Init: 15 import hooks: 4 _PyWarnings_Init(): 15 ENTERING import_init(): PyImport_ImportFrozenModule(_frozen_importlib): 1186 interp->importlib: 6 PyInit_imp(): 15 _imp: 3 importlib._install(): 876 _PyImportZip_Init(): 130 _PyFaulthandler_Init(): 13 time: 3 ENTERING initfsencoding(): codec lookup: 2179 signals: 120 tracemalloc: 7 __main__: 13 initstdio(): 1568 warnings module: 4 initsite(): 9552

7 12

New mailing list for workflow/workflow infrastructure discussion/tasks
by R. David Murray April 16, 2014

April 16, 2014

Based on a number of conversations at PyCon, we've created a new mailing list: https://mail.python.org/mailman/listinfo/core-workflow The purpose of this list is to facilitate the conversations and coordinate the work that needs to happen to improve our development workflow. Nick's PEP is one piece of this conversation, but there are many other aspects to it as well. Here is the list description: This is a place to discuss and work on improvements to the CPython core development workflow and the infrastructure that supports that workflow. This includes changes to the roundup interface and functionality, rietveld, mercurial, buildbots, and any other infrastructure we may add. It also includes discussing how we use these tools, and most importantly how we use these tools to integrate the community beyond the core developers into the workflow that gets patches committed to the python repository. This means that it also includes discussions of the process of bringing in new contributors, including how we use the core-mentorship list, as well as how we organize ticket triage, and how we make use of external resources such as openhatch. Discussions of documentation and how we organize and maintain the documentation are also appropriate. Anyone who is interested helping with this, or who wants to keep up with the evolution of our thoughts on these topics, are invited to sign up to the mailing list. We will of course check in with python-dev and/or python-committers as appropriate. --David

2 1

Re: [Python-Dev] cpython (2.7): do not generate pipe names in the temporary dir
by Antoine Pitrou April 16, 2014

April 16, 2014

On Mon, 14 Apr 2014 18:24:44 +0200 (CEST) benjamin.peterson <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/ea677e26dbeb > changeset: 90269:ea677e26dbeb > branch: 2.7 > parent: 90261:c095ff9b0e84 > user: Benjamin Peterson <benjamin(a)python.org> > date: Mon Apr 14 12:24:37 2014 -0400 > summary: > do not generate pipe names in the temporary dir > > files: > Lib/multiprocessing/connection.py | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > > diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py > --- a/Lib/multiprocessing/connection.py > +++ b/Lib/multiprocessing/connection.py > @@ -90,7 +90,7 @@ > return tempfile.mktemp(prefix='listener-', dir=get_temp_dir()) > elif family == 'AF_PIPE': > return tempfile.mktemp(prefix=r'\\.\pipe\pyc-%d-%d-' % > - (os.getpid(), _mmap_counter.next())) > + (os.getpid(), _mmap_counter.next()), dir="") This is some Windows-specific code, right? Did you witness problems with this? Your commit message doesn't mention an issue number. Regards Antoine.

2 2