Mailman 3 February 2012 - Python-Dev

PEP 413 updated (with improved rationale and simpler stdlib versioning scheme)
by Nick Coghlan 26 Feb '12

26 Feb '12

After working through some additional scenarios (primarily the question of handling security fixes), I have simplified the proposed versioning scheme in PEP 413. New version included below, or you can read the nicely formatted version: http://www.python.org/dev/peps/pep-0413/ ========================== PEP: 413 Title: Faster evolution of the Python Standard Library Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan(a)gmail.com> Status: Draft Type: Process Content-Type: text/x-rst Created: 2012-02-24 Post-History: 2012-02-24, 2012-02-25 Resolution: TBD Abstract ======== This PEP proposes the adoption of a separate versioning scheme for the standard library (distinct from, but coupled to, the existing language versioning scheme) that allows accelerated releases of the Python standard library, while maintaining (or even slowing down) the current rate of change in the core language definition. Like PEP 407, it aims to adjust the current balance between measured change that allows the broader community time to adapt and being able to keep pace with external influences that evolve more rapidly than the current release cycle can handle (this problem is particularly notable for standard library elements that relate to web technologies). However, it's more conservative in its aims than PEP 407, seeking to restrict the increased pace of development to builtin and standard library interfaces, without affecting the rate of change for other elements such as the language syntax and version numbering as well as the CPython binary API and bytecode format. Rationale ========= To quote the PEP 407 abstract: Finding a release cycle for an open-source project is a delicate exercise in managing mutually contradicting constraints: developer manpower, availability of release management volunteers, ease of maintenance for users and third-party packagers, quick availability of new features (and behavioural changes), availability of bug fixes without pulling in new features or behavioural changes. The current release cycle errs on the conservative side. It is adequate for people who value stability over reactivity. This PEP is an attempt to keep the stability that has become a Python trademark, while offering a more fluid release of features, by introducing the notion of long-term support versions. I agree with the PEP 407 authors that the current release cycle of the *standard library* is too slow to effectively cope with the pace of change in some key programming areas (specifically, web protocols and related technologies, including databases, templating and serialisation formats). However, I have written this competing PEP because I believe that the approach proposed in PEP 407 of offering full, potentially binary incompatible releases of CPython every 6 months places too great a burden on the wider Python ecosystem. Under the current CPython release cycle, distributors of key binary extensions will often support Python releases even after the CPython branches enter "security fix only" mode (for example, Twisted currently ships binaries for 2.5, 2.6 and 2.7, NumPy and SciPy suport those 3 along with 3.1 and 3.2, PyGame adds a 2.4 binary release, wxPython provides both 32-bit and 64-bit binaries for 2.6 and 2.7, etc). If CPython were to triple (or more) its rate of releases, the developers of those libraries (many of which are even more resource starved than CPython) would face an unpalatable choice: either adopt the faster release cycle themselves (up to 18 simultaneous binary releases for PyGame!), drop older Python versions more quickly, or else tell their users to stick to the CPython LTS releases (thus defeating the entire point of speeding up the CPython release cycle in the first place). Similarly, many support tools for Python (e.g. syntax highlighters) can take quite some time to catch up with language level changes. At a cultural level, the Python community is also accustomed to a certain meaning for Python version numbers - they're linked to deprecation periods, support periods, all sorts of things. PEP 407 proposes that collective knowledge all be swept aside, without offering a compelling rationale for why such a course of action is actually *necessary* (aside from, perhaps, making the lives of the CPython core developers a little easier at the expense of everyone else). However, if we go back to the primary rationale for increasing the pace of change (i.e. more timely support for web protocols and related technologies), we can note that those only require *standard library* changes. That means many (perhaps even most) of the negative effects on the wider community can be avoided by explicitly limiting which parts of CPython are affected by the new release cycle, and allowing other parts to evolve at their current, more sedate, pace. Proposal ======== This PEP proposes the introduction of a new kind of CPython release: "standard library releases". As with PEP 407, this will give CPython 3 kinds of release: * Language release: "x.y.0" * Maintenance release: "x.y.z" (where z > 0) * Standard library release: "x.y (xy.z)" (where z > 0) Under this scheme, an unqualified version reference (such as "3.3") would always refer to the most recent corresponding language or maintenance release. It will never be used without qualification to refer to a standard library release (at least, not by python-dev - obviously, we can only set an example, not force the rest of the Python ecosystem to go along with it). Language releases will continue as they are now, as new versions of the Python language definition, along with a new version of the CPython interpreter and the Python standard library. Accordingly, a language release may contain any and all of the following changes: * new language syntax * new standard library changes (see below) * new deprecation warnings * removal of previously deprecated features * changes to the emitted bytecode * changes to the AST * any other significant changes to the compilation toolchain * changes to the core interpreter eval loop * binary incompatible changes to the C ABI (although the PEP 384 stable ABI must still be preserved) * bug fixes Maintenance releases will also continue as they do today, being strictly limited to bug fixes for the corresponding language release. No new features or radical internal changes are permitted. The new standard library releases will occur in parallel with each maintenance release and will be qualified with a new version identifier documenting the standard library version. Standard library releases may include the following changes: * new features in pure Python modules * new features in C extension modules (subject to PEP 399 compatibility requirements) * new features in language builtins (provided the C ABI remains unaffected) * bug fixes from the corresponding maintenance release Standard library version identifiers are constructed by combining the major and minor version numbers for the Python language release into a single two digit number and then appending a sequential standard library version identifier. Release Cycle ------------- When maintenance releases are created, *two* new versions of Python would actually be published on python.org (using the first 3.3 maintenance release, planned for February 2013 as an example):: 3.3.1 # Maintenance release 3.3 (33.1) # Standard library release A further 6 months later, the next 3.3 maintenance release would again be accompanied by a new standard library release:: 3.3.2 # Maintenance release 3.3 (33.2) # Standard library release Again, the standard library release would be binary compatible with the previous language release, merely offering additional features at the Python level. Finally, 18 months after the release of 3.3, a new language release would be made around the same time as the final 3.3 maintenance and standard library releases:: 3.3.3 # Maintenance release 3.3 (33.3) # Standard library release 3.4.0 # Language release The 3.4 release cycle would then follow a similar pattern to that for 3.3:: 3.4.1 # Maintenance release 3.4 (34.1) # Standard library release 3.4.2 # Maintenance release 3.4 (34.2) # Standard library release 3.4.3 # Maintenance release 3.4 (34.3) # Standard library release 3.5.0 # Language release Programmatic Version Identification ----------------------------------- To expose the new version details programmatically, this PEP proposes the addition of a new ``sys.stdlib_info`` attribute that records the new standard library version above and beyond the underlying interpreter version. Using the initial Python 3.3 release as an example:: sys.stdlib_info(python=33, version=0, releaselevel='final', serial=0) This information would also be included in the ``sys.version`` string:: Python 3.3.0 (33.0, default, Feb 17 2012, 23:03:41) [GCC 4.6.1] Security Fixes and Other "Out of Cycle" Releases ------------------------------------------------ For maintenance releases the process of handling out-of-cycle releases (for example, to fix a security issue or resolve a critical bug in a new release), remains the same as it is now: the minor version number is incremented and a new release is made incorporating the required bug fixes, as well as any other bug fixes that have been committed since the previous release. For standard library releases, the process is essentially the same, but the corresponding "What's New?" document may require some tidying up for the release (as the standard library release may incorporate new features, not just bug fixes). User Scenarios ============== The versioning scheme proposed above is based on a number of user scenarios that are likely to be encountered if this scheme is adopted. In each case, the scenario is described for both the status quo (i.e. slow release cycle) the versioning scheme in this PEP and the free wheeling minor version number scheme proposed in PEP 407. To give away the ending, the point of using a separate version number is that for almost all scenarios, the important number is the *language* version, not the standard library version. Most users won't even need to care that the standard library version number exists. In the two identified cases where it matters, providing it as a separate number is actually clearer and more explicit than embedding the two different kinds of number into a single sequence and then tagging some of the numbers in the unified sequence as special. Novice user, downloading Python from python.org in March 2013 ------------------------------------------------------------- **Status quo:** must choose between 3.3 and 2.7 **This PEP:** must choose between 3.3 (33.1), 3.3 and 2.7. **PEP 407:** must choose between 3.4, 3.3 (LTS) and 2.7. **Verdict:** explaining the meaning of a Long Term Support release is about as complicated as explaining the meaning of the proposed standard library release version numbers. I call this a tie. Novice user, attempting to judge currency of third party documentation ---------------------------------------------------------------------- **Status quo:** minor version differences indicate 18-24 months of language evolution **This PEP:** same as status quo for language core, standard library version numbers indicate 6 months of standard library evolution. **PEP 407:** minor version differences indicate 18-24 months of language evolution up to 3.3, then 6 months of language evolution thereafter. **Verdict:** Since language changes and deprecations can have a much bigger effect on the accuracy of third party documentation than the addition of new features to the standard library, I'm calling this a win for the scheme in this PEP. Novice user, looking for an extension module binary release ----------------------------------------------------------- **Status quo:** look for the binary corresponding to the Python version you are running. **This PEP:** same as status quo. **PEP 407 (full releases):** same as status quo, but corresponding binary version is more likely to be missing (or, if it does exist, has to be found amongst a much larger list of alternatives). **PEP 407 (ABI updates limited to LTS releases):** all binary release pages will need to tell users that Python 3.3, 3.4 and 3.5 all need the 3.3 binary. **Verdict:** I call this a clear win for the scheme in this PEP. Absolutely nothing changes from the current situation, since the standard library version is actually irrelevant in this case (only binary extension compatibility is important). Extension module author, deciding whether or not to make a binary release ------------------------------------------------------------------------- **Status quo:** unless using the PEP 384 stable ABI, a new binary release is needed every time the minor version number changes. **This PEP:** same as status quo. **PEP 407 (full releases):** same as status quo, but becomes a far more frequent occurrence. **PEP 407 (ABI updates limited to LTS releases):** before deciding, must first look up whether the new release is an LTS release or an interim release. If it is an LTS release, then a new build is necessary. **Verdict:** I call this another clear win for the scheme in this PEP. As with the end user facing side of this problem, the standard library version is actually irrelevant in this case. Moving that information out to a separate number avoids creating unnecessary confusion. Python developer, deciding priority of eliminating a Deprecation Warning ------------------------------------------------------------------------ **Status quo:** code that triggers deprecation warnings is not guaranteed to run on a version of Python with a higher minor version number. **This PEP:** same as status quo **PEP 407:** unclear, as the PEP doesn't currently spell this out. Assuming the deprecation cycle is linked to LTS releases, then upgrading to a non-LTS release is safe but upgrading to the next LTS release may require avoiding the deprecated construct. **Verdict:** another clear win for the scheme in this PEP since, once again, the standard library version is irrelevant in this scenario. Alternative interpreter implementor, updating with new features --------------------------------------------------------------- **Status quo:** new Python versions arrive infrequently, but are a mish-mash of standard library updates and core language definition and interpreter changes. **This PEP:** standard library updates, which are easier to integrate, are made available more frequently in a form that is clearly and explicitly compatible with the previous version of the language definition. This means that, once an alternative implementation catches up to Python 3.3, they should have a much easier time incorporating standard library features as they happen (especially pure Python changes), leaving minor version number updates as the only task that requires updates to their core compilation and execution components. **PEP 407 (full releases):** same as status quo, but becomes a far more frequent occurrence. **PEP 407 (language updates limited to LTS releases):** unclear, as the PEP doesn't currently spell out a specific development strategy. Assuming a 3.3 compatibility branch is adopted (as proposed in this PEP), then the outcome would be much the same, but the version number signalling would be slightly less clear (since you would have to check to see if a particular release was an LTS release or not). **Verdict:** while not as clear cut as some previous scenarios, I'm still calling this one in favour of the scheme in this PEP. Explicit is better than implicit, and the scheme in this PEP makes a clear split between the two different kinds of update rather than adding a separate "LTS" tag to an otherwise ordinary release number. Tagging a particular version as being special is great for communicating with version control systems and associated automated tools, but it's a lousy way to communicate information to other humans. Python developer, deciding their minimum version dependency ----------------------------------------------------------- **Status quo:** look for "version added" or "version changed" markers in the documentation, check against ``sys.version_info`` **This PEP:** look for "version added" or "version changed" markers in the documentation. If written as a bare Python version, such as "3.3", check against ``sys.version_info``. If qualified with a standard library version, such as "3.3 (33.1)", check against ``sys.stdlib_info``. **PEP 407:** same as status quo **Verdict:** the scheme in this PEP actually allows third party libraries to be more explicit about their rate of adoption of standard library features. More conservative projects will likely pin their dependency to the language version and avoid features added in the standard library releases. Faster moving projects could instead declare their dependency on a particular standard library version. However, since PEP 407 does have the advantage of preserving the status quo, I'm calling this one for PEP 407 (albeit with a slim margin). Python developers, attempting to reproduce a tracker issue ---------------------------------------------------------- **Status quo:** if not already provided, ask the reporter which version of Python they're using. This is often done by asking for the first two lines displayed by the interactive prompt or the value of ``sys.version``. **This PEP:** same as the status quo (as ``sys.version`` will be updated to also include the standard library version), but may be needed on additional occasions (where the user knew enough to state their Python version, but that proved to be insufficient to reproduce the fault). **PEP 407:** same as the status quo **Verdict:** another marginal win for PEP 407. The new standard library version *is* an extra piece of information that users may need to pass back to developers when reporting issues with Python libraries (or Python itself, on our own tracker). However, by including it in ``sys.version``, many fault reports will already include it, and it is easy to request if needed. CPython release managers, handling a security fix ------------------------------------------------- **Status quo:** create a new maintenance release incorporating the security fix and any other bug fixes under source control. Also create source releases for any branches open solely for security fixes. **This PEP:** same as the status quo for maintenance branches. Also create a new standard library release (potentially incorporating new features along with the security fix). For security branches, create source releases for both the former maintenance branch and the standard library update branch. **PEP 407:** same as the status quo for maintenance and security branches, but handling security fixes for non-LTS releases is currently an open question. **Verdict:** until PEP 407 is updated to actually address this scenario, a clear win for this PEP. Effects ======= Effect on development cycle --------------------------- Similar to PEP 407, this PEP will break up the delivery of new features into more discrete chunks. Instead of a whole raft of changes landing all at once in a language release, each language release will be limited to 6 months worth of standard library changes, as well as any changes associated with new syntax. Effect on workflow ------------------ This PEP proposes the creation of a single additional branch for use in the normal workflow. After the release of 3.3, the following branches would be in use:: 2.7 # Maintenance branch, no change 3.3 # Maintenance branch, as for 3.2 3.3-compat # New branch, backwards compatible changes default # Language changes, standard library updates that depend on them When working on a new feature, developers will need to decide whether or not it is an acceptable change for a standard library release. If so, then it should be checked in on ``3.3-compat`` and then merged to ``default``. Otherwise it should be checked in directly to ``default``. The "version added" and "version changed" markers for any changes made on the ``3.3-compat`` branch would need to be flagged with both the language version and the standard library version. For example: "3.3 (33.1)". Any changes made directly on the ``default`` branch would just be flagged with "3.4" as usual. The ``3.3-compat`` branch would be closed to normal development at the same time as the ``3.3`` maintenance branch. The ``3.3-compat`` branch would remain open for security fixes for the same period of time as the ``3.3`` maintenance branch. Effect on bugfix cycle ---------------------- The effect on the bug fix workflow is essentially the same as that on the workflow for new features - there is one additional branch to pass through before the change reaches the ``default`` branch. If critical bugs are found in a maintenance release, then new maintenance and standard library releases will be created to resolve the problem. The final part of the version number will be incremented for both the language version and the standard library version. If critical bugs are found in a standard library release that do not affect the associated maintenance release, then only a new standard library release will be created and only the standard library's version number will be incremented. Note that in these circumstances, the standard library release *may* include additional features, rather than just containing the bug fix. It is assumed that anyone that cares about receiving *only* bug fixes without any new features mixed in will already be relying strictly on the maintenance releases rather than using the new standard library releases. Effect on the community ----------------------- PEP 407 has this to say about the effects on the community: People who value stability can just synchronize on the LTS releases which, with the proposed figures, would give a similar support cycle (both in duration and in stability). I believe this statement is just plain wrong. Life isn't that simple. Instead, developers of third party modules and frameworks will come under pressure to support the full pace of the new release cycle with binary updates, teachers and book authors will receive complaints that they're only covering an "old" version of Python ("You're only using 3.3, the latest is 3.5!"), etc. As the minor version number starts climbing 3 times faster than it has in the past, I believe perceptions of language stability would also fall (whether such opinions were justified or not). I believe isolating the increased pace of change to the standard library, and clearly delineating it with a separate version number will greatly reassure the rest of the community that no, we're not suddenly asking them to triple their own rate of development. Instead, we're merely going to ship standard library updates for the next language release in 6-monthly installments rather than delaying them all until the next language definition update, even those changes that are backwards compatible with the previously released version of Python. The community benefits listed in PEP 407 are equally applicable to this PEP, at least as far as the standard library is concerned: People who value reactivity and access to new features (without taking the risk to install alpha versions or Mercurial snapshots) would get much more value from the new release cycle than currently. People who want to contribute new features or improvements would be more motivated to do so, knowing that their contributions will be more quickly available to normal users. If the faster release cycle encourages more people to focus on contributing to the standard library rather than proposing changes to the language definition, I don't see that as a bad thing. Handling News Updates ===================== What's New? ----------- The "What's New" documents would be split out into separate documents for standard library releases and language releases. So, during the 3.3 release cycle, we would see: * What's New in Python 3.3? * What's New in the Python Standard Library 33.1? * What's New in the Python Standard Library 33.2? * What's New in the Python Standard Library 33.3? And then finally, we would see the next language release: * What's New in Python 3.4? For the benefit of users that ignore standard library releases, the 3.4 What's New would link back to the What's New documents for each of the standard library releases in the 3.3 series. NEWS ---- Merge conflicts on the NEWS file are already a hassle. Since this PEP proposes introduction of an additional branch into the normal workflow, resolving this becomes even more critical. While Mercurial phases may help to some degree, it would be good to eliminate the problem entirely. One suggestion from Barry Warsaw is to adopt a non-conflicting separate-files-per-change approach, similar to that used by Twisted [2_]. Given that the current manually updated NEWS file will be used for the 3.3.0 release, one possible layout for such an approach might look like:: Misc/ NEWS # Now autogenerated from news_entries news_entries/ 3.3/ NEWS # Original 3.3 NEWS file maint.1/ # Maintenance branch changes core/ <news entries> builtins/ <news entries> extensions/ <news entries> library/ <news entries> documentation/ <news entries> tests/ <news entries> compat.1/ # Compatibility branch changes builtins/ <news entries> extensions/ <news entries> library/ <news entries> documentation/ <news entries> tests/ <news entries> # Add maint.2, compat.2 etc as releases are made 3.4/ core/ <news entries> builtins/ <news entries> extensions/ <news entries> library/ <news entries> documentation/ <news entries> tests/ <news entries> # Add maint.1, compat.1 etc as releases are made Putting the version information in the directory heirarchy isn't strictly necessary (since the NEWS file generator could figure out from the version history), but does make it easier for *humans* to keep the different versions in order. Other benefits of reduced version coupling ========================================== Slowing down the language release cycle --------------------------------------- The current release cycle is a compromise between the desire for stability in the core language definition and C extension ABI, and the desire to get new features (most notably standard library updates) into user's hands more quickly. With the standard library release cycle decoupled (to some degree) from that of the core language definition, it provides an opportunity to actually *slow down* the rate of change in the language definition. The language moratorium for Python 3.2 effectively slowed that cycle down to *more than 3 years* (3.1: June 2009, 3.3: August 2012) without causing any major problems or complaints. The NEWS file management scheme described above is actually designed to allow us the flexibility to slow down language releases at the same time as standard library releases become more frequent. As a simple example, if a full two years was allowed between 3.3 and 3.4, the 3.3 release cycle would end up looking like:: 3.2.4 # Maintenance release 3.3.0 # Language release 3.3.1 # Maintenance release 3.3 (33.1) # Standard library release 3.3.2 # Maintenance release 3.3 (33.2) # Standard library release 3.3.3 # Maintenance release 3.3 (33.3) # Standard library release 3.3.4 # Maintenance release 3.3 (33.4) # Standard library release 3.4.0 # Language release The elegance of the proposed branch structure and NEWS entry layout is that this decision wouldn't really need to be made until shortly before the planned 3.4 release date. At that point, the decision could be made to postpone the 3.4 release and keep the ``3.3`` and ``3.3-compat`` branches open after the 3.3.3 maintenance release and the 3.3 (33.3) standard library release, thus adding another standard library release to the cycle. The choice between another standard library release or a full language release would then be available every 6 months after that. Further increasing the pace of standard library development ----------------------------------------------------------- As noted in the previous section, one benefit of the scheme proposed in this PEP is that it largely decouples the language release cycle from the standard library release cycle. The standard library could be updated every 3 months, or even once a month, without having any flow on effects on the language version numbering or the perceived stability of the core language. While that pace of development isn't practical as long as the binary installer creation for Windows and Mac OS X involves several manual steps (including manual testing) and for as long as we don't have separate "<branch>-release" trees that only receive versions that have been marked as good by the stable buildbots, it's still a useful criterion to keep in mind when considering proposed new versioning schemes: what if we eventually want to make standard library releases even *faster* than every 6 months? If the practical issues were ever resolved, then the separate standard library versioning scheme in this PEP could handle it. The tagged version number approach proposed in PEP 407 could not (at least, not without a lot of user confusion and uncertainty). Other Questions =============== Why not a date-based versioning scheme? --------------------------------------- Earlier versions of this PEP proposed a date-based versioning scheme for the standard library. However, such a scheme made it very difficult to handle out-of-cycle releases to fix security issues and other critical bugs in standard library releases, as it required the following steps: 1. Change the release version number to the date of the current month. 2. Update the What's New, NEWS and documentation to refer to the new release number. 3. Make the new release. With the sequential scheme now proposed, such releases should at most require a little tidying up of the What's New document before making the release. Why isn't PEP 384 enough? ------------------------- PEP 384 introduced the notion of a "Stable ABI" for CPython, a limited subset of the full C ABI that is guaranteed to remain stable. Extensions built against the stable ABI should be able to support all subsequent Python versions with the same binary. This will help new projects to avoid coupling their C extension modules too closely to a specific version of CPython. For existing modules, however, migrating to the stable ABI can involve quite a lot of work (especially for extension modules that define a lot of classes). With limited development resources available, any time spent on such a change is time that could otherwise have been spent working on features that offer more direct benefits to end users. There are also other benefits to separate versioning (as described above) that are not directly related to the question of binary compatibility with third party C extensions. Why no binary compatible additions to the C ABI in standard library releases? ----------------------------------------------------------------------------- There's a case to be made that *additions* to the CPython C ABI could reasonably be permitted in standard library releases. This would give C extension authors the same freedom as any other package or module author to depend either on a particular language version or on a standard library version. The PEP currently associates the interpreter version with the language version, and therefore limits major interpreter changes (including C ABI additions) to the language releases. An alternative, internally consistent, approach would be to link the interpreter version with the standard library version, with only changes that may affect backwards compatibility limited to language releases. Under such a scheme, the following changes would be acceptable in standard library releases: * Standard library updates * new features in pure Python modules * new features in C extension modules (subject to PEP 399 compatibility requirements) * new features in language builtins * Interpreter implementation updates * binary compatible additions to the C ABI * changes to the compilation toolchain that do not affect the AST or alter the bytecode magic number * changes to the core interpreter eval loop * bug fixes from the corresponding maintenance release And the following changes would be acceptable in language releases: * new language syntax * any updates acceptable in a standard library release * new deprecation warnings * removal of previously deprecated features * changes to the AST * changes to the emitted bytecode that require altering the magic number * binary incompatible changes to the C ABI (although the PEP 384 stable ABI must still be preserved) While such an approach could probably be made to work, there does not appear to be a compelling justification for it, and the approach currently described in the PEP is simpler and easier to explain. Why not separate out the standard library entirely? --------------------------------------------------- A concept that is occasionally discussed is the idea of making the standard library truly independent from the CPython reference implementation. My personal opinion is that actually making such a change would involve a lot of work for next to no pay-off. CPython without the standard library is useless (the build chain won't even run, let alone the test suite). You also can't create a standalone pure Python standard library either, because too many "standard library modules" are actually tightly linked in to the internal details of their respective interpreters (for example, the builtins, ``weakref``, ``gc``, ``sys``, ``inspect``, ``ast``). Creating a separate CPython development branch that is kept compatible with the previous language release, and making releases from that branch that are identified with a separate standard library version number should provide most of the benefits of a separate standard library repository with only a fraction of the pain. Acknowledgements ================ Thanks go to the PEP 407 authors for starting this discussion, as well as to those authors and Larry Hastings for initial discussions of the proposal made in this PEP. References ========== .. [1] PEP 407: New release cycle and introducing long-term support versions http://www.python.org/dev/peps/pep-0407/ .. [2] Twisted's "topfiles" approach to NEWS generation http://twistedmatrix.com/trac/wiki/ReviewProcess#Newsfiles Copyright ========= This document has been placed in the public domain. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

[RELEASED] Release candidates for Python 2.6.8, 2.7.3, 3.1.5, and 3.2.3
by Benjamin Peterson 25 Feb '12

25 Feb '12

We're pleased to announce the immediate availability of release candidates for Python 2.6.8, 2.7.3, 3.1.5, and 3.2.3 . The main impetus for these releases is fixing a security issue in Python's hash based types, dict and set, as described below. Python 2.7.3 and 3.2.3 include the security patch and the normal set of bug fixes. Since Python 2.6 and 3.1 are maintained only for security issues, 2.6.8 and 3.1.5 contain only various security patches. The security issue exploits Python's dict and set implementations. Carefully crafted input can lead to extremely long computation times and denials of service. [1] Python dict and set types use hash tables to provide amortized constant time operations. Hash tables require a well-distributed hash function to spread data evenly across the hash table. The security issue is that an attacker could compute thousands of keys with colliding hashes; this causes quadratic algorithmic complexity when the hash table is constructed. To alleviate the problem, the new releases add randomization to the hashing of Python's string types (bytes/str in Python 3 and str/unicode in Python 2), datetime.date, and datetime.datetime. This prevents an attacker from computing colliding keys of these types without access to the Python process. Hash randomization causes the iteration order of dicts and sets to be unpredictable and differ across Python runs. Python has never guaranteed iteration order of keys in a dict or set, and applications are advised to never rely on it. Historically, dict iteration order has not changed very often across releases and has always remained consistent between successive executions of Python. Thus, some existing applications may be relying on dict or set ordering. Because of this and the fact that many Python applications which don't accept untrusted input are not vulnerable to this attack, in all stable Python releases mentioned here, HASH RANDOMIZATION IS DISABLED BY DEFAULT. There are two ways to enable it. The -R commandline option can be passed to the python executable. It can also be enabled by setting an environmental variable PYTHONHASHSEED to "random". (Other values are accepted, too; pass -h to python for complete description.) More details about the issue and the patch can be found in the oCERT advisory [1] and the Python bug tracker [2]. These releases are releases candidates and thus not recommended for production use. Please test your applications and libraries with them, and report any bugs you encounter. We are especially interested in any buggy behavior observed using hash randomization. Excepting major calamity, final versions should appear after several weeks. Downloads are at http://python.org/download/releases/2.6.8/ http://python.org/download/releases/2.7.3/ http://python.org/download/releases/3.1.5/ http://python.org/download/releases/3.2.3/ Please test these candidates and report bugs to http://bugs.python.org/ With regards, The Python release team Barry Warsaw (2.6), Georg Brandl (3.2), Benjamin Peterson (2.7 and 3.1) [1] http://www.ocert.org/advisories/ocert-2011-003.html [2] http://bugs.python.org/issue13703

1 0

PEP 413: Faster evolution of the Python Standard Library
by Nick Coghlan 25 Feb '12

25 Feb '12

To allow the PEP 407 authors to focus on making the case for doing full CPython releases every 6 months (including language spec updates), I've created PEP 413 as a competing proposal. It leaves the current language versioning (and rate of change) alone, while adding an additional date based standard library version number and adopting a new development workflow that will allow "standard library" releases to be made alongside each new maintenance release. PEP text is below, or you can read it online: http://www.python.org/dev/peps/pep-0413/ Cheers, Nick. PEP: 413 Title: Faster evolution of the Python Standard Library Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan(a)gmail.com> Status: Draft Type: Process Content-Type: text/x-rst Created: 2012-02-24 Post-History: 2012-02-24 Resolution: TBD Abstract ======== This PEP proposes the adoption of a new date-based versioning scheme for the standard library (distinct from, but coupled to, the existing language versioning scheme) that allows accelerated releases of the Python standard library, while maintaining (or even slowing down) the current rate of change in the core language definition. Like PEP 407, it aims to adjust the current balance between measured change that allows the broader community time to adapt and being able to keep pace with external influences that evolve more rapidly than the current release cycle can handle (this problem is particularly notable for standard library elements that relate to web technologies). However, it's more conservative in its aims than PEP 407, seeking to restrict the increased pace of development to builtin and standard library interfaces, without affecting the rate of change for other elements such as the language syntax and version numbering as well as the CPython binary API and bytecode format. Rationale ========= To quote the PEP 407 abstract: Finding a release cycle for an open-source project is a delicate exercise in managing mutually contradicting constraints: developer manpower, availability of release management volunteers, ease of maintenance for users and third-party packagers, quick availability of new features (and behavioural changes), availability of bug fixes without pulling in new features or behavioural changes. The current release cycle errs on the conservative side. It is adequate for people who value stability over reactivity. This PEP is an attempt to keep the stability that has become a Python trademark, while offering a more fluid release of features, by introducing the notion of long-term support versions. I agree with the PEP 407 authors that the current release cycle of the *standard library* is too slow to effectively cope with the pace of change in some key programming areas (specifically, web protocols and related technologies, including databases, templating and serialisation formats). However, I have written this competing PEP because I believe that the approach proposed in PEP 407 of offering full, potentially binary incompatible releases of CPython every 6 months places too great a burden on the wider Python ecosystem. Under the current CPython release cycle, distributors of key binary extensions will often support Python releases even after the CPython branches enter "security fix only" mode (for example, Twisted currently ships binaries for 2.5, 2.6 and 2.7, NumPy and SciPy suport those 3 along with 3.1 and 3.2, PyGame adds a 2.4 binary release, wxPython provides both 32-bit and 64-bit binaries for 2.6 and 2.7, etc). If CPython were to triple (or more) its rate of releases, the developers of those libraries (many of which are even more resource starved than CPython) would face an unpalatable choice: either adopt the faster release cycle themselves (up to 18 simultaneous binary releases for PyGame!), drop older Python versions more quickly, or else tell their users to stick to the CPython LTS releases (thus defeating the entire point of speeding up the CPython release cycle in the first place). Similarly, many support tools for Python (e.g. syntax highlighters) can take quite some time to catch up with language level changes. At a cultural level, the Python community is also accustomed to a certain meaning for Python version numbers - they're linked to deprecation periods, support periods, all sorts of things. PEP 407 proposes that collective knowledge all be swept aside, without offering a compelling rationale for why such a course of action is actually *necessary* (aside from, perhaps, making the lives of the CPython core developers a little easier at the expense of everyone else). But, if we go back to the primary rationale for increasing the pace of change (i.e. more timely support for web protocols and related technologies), we can note that those only require *standard library* changes. That means many (perhaps even most) of the negative effects on the wider community can be avoided by explicitly limiting which parts of CPython are affected by the new release cycle, and allowing other parts to evolve at their current, more sedate, pace. Proposal ======== This PEP proposes the addition of a new ``sys.stdlib_info`` attribute that records a date based standard library version above and beyond the underlying interpreter version:: sys.stdlib_info(year=2012, month=8, micro=0, releaselevel='final', serial=0) This information would also be included in the ``sys.version`` string:: Python 3.3.0 (12.08.0, default:c1a07c8092f7+, Feb 17 2012, 23:03:41) [GCC 4.6.1] When maintenance releases are created, *two* new versions of Python would actually be published on python.org (using the first 3.3 maintenance release, planned for February 2013 as an example):: 3.3.1 + 12.08.1 # Maintenance release 3.3.1 + 13.02.0 # Standard library release A standard library release would just be the corresponding maintenance release, with the following additional, backwards compatible changes: * new features in pure Python modules * new features in C extension modules (subject to PEP 399 compatibility requirements) * new features in language builtins (provided the C ABI remains unaffected) A further 6 months later, the next 3.3 maintenance release would again be accompanied by a new standard library release:: 3.3.2 + 12.08.2 # Maintenance release 3.3.2 + 13.08.1 # Standard library release Again, the standard library release would be binary compatible with the previous language release, merely offering additional features at the Python level. Finally, 18 months after the release of 3.3, a new language release would be made around the same time as the final 3.3 maintenance release: 3.3.3 + 12.08.3 # Maintenance release 3.4.0 + 14.02.0 # Language release Language releases would then contain all the features that are not permitted in standard library releases: * new language syntax * new deprecation warnings * removal of previously deprecated features * changes to the emitted bytecode * changes to the AST * any other significant changes to the compilation toolchain * changes to the C ABI The 3.4 release cycle would then follow a similar pattern to that for 3.3:: 3.4.1 + 14.02.1 # Maintenance release 3.4.1 + 14.08.0 # Standard library release 3.4.2 + 14.02.2 # Maintenance release 3.4.2 + 15.02.0 # Standard library release 3.4.3 + 14.02.3 # Maintenance release 3.5.0 + 15.08.0 # Language release Effects ======= Effect on development cycle --------------------------- Similar to PEP 407, this PEP will break up the delivery of new features into more discrete chunks. Instead of whole raft of changes landing all at once in a language release, each language release will be limited to 6 months worth of standard library changes, as well as any changes associated with new syntax. Effect on workflow ------------------ This PEP proposes the creation of a single additional branch for use in the normal workflow. After the release of 3.3, the following branches would be in use:: 2.7 # Maintenance branch, no change 3.3 # Maintenance branch, as for 3.2 3.3-compat # New branch, backwards compatible changes default # Language changes, standard library updates that depend on them When working on a new feature, developers will need to decide whether or not it is an acceptable change for a standard library release. If so, then it should be checked in on ``3.3-compat`` and then merged to ``default``. Otherwise it should be checked in directly to ``default``. Effect on bugfix cycle ---------------------- The effect on the bug fix cycle is essentially the same as that on the workflow for new features - there is one additional branch to pass through before the change reaches default branch. Effect on the community ----------------------- PEP 407 has this to say about the effects on the community: People who value stability can just synchronize on the LTS releases which, with the proposed figures, would give a similar support cycle (both in duration and in stability). I believe this statement is just plain wrong. Life isn't that simple. Instead, developers of third party modules and frameworks will come under pressure to support the full pace of the new release cycle with binary updates, teachers and book authors will receive complaints that they're only covering an "old" version of Python ("You're only using 3.3, the latest is 3.5!"), etc. As the minor version number starts climbing 3 times faster than it has in the past, I believe perceptions of language stability would also fall (whether such opinions were justified or not). I believe isolating the increased pace of change to the standard library, and clearly delineating it with a separate date-based version number will greatly reassure the rest of the community that no, we're not suddenly asking them to triple their own rate of development. Instead, we're merely going to ship standard library updates for the next language release in three 6-monthly installments rather than delaying them all, even those that are backwards compatible with the previously released version of Python. The community benefits list in PEP 407 are equally applicable to this PEP, at least as far as the standard library is concerned: People who value reactivity and access to new features (without taking the risk to install alpha versions or Mercurial snapshots) would get much more value from the new release cycle than currently. People who want to contribute new features or improvements would be more motivated to do so, knowing that their contributions will be more quickly available to normal users. If the faster release cycle encourages more people to focus on contributing to the standard library rather than proposing changes to the language definition, I don't see that as a bad thing. Handling News Updates ===================== What's New? ----------- The "What's New" documents would be split out into separate documents for standard library releases and language releases. If the major version number only continues to increase once every decade or so, resolving the eventual numbering conflict can be safely deemed somebody elses problem :) NEWS ---- Merge conflicts on the NEWS file is already a hassle. Since this PEP proposes introduction of an additional branch into the normal workflow, resolving this becomes even more critical. While Mercurial phases will help to some degree, it would be good to eliminate the problem entirely. One suggestion from Barry Warsaw is to adopt a non-conflicting separate-files-per-change approach, similar to that used by Twisted [2_]. For this PEP, one possible layout for such an approach (adopted following the release of 3.3.0+12.8.0 using the existing NEWS process) might look like:: Misc/ lang_news/ 3.3.1/ <files for core language changes> 3.4.0/ <files for core language changes> stdlib_news/ 12.08.1/ builtins/ <files for builtin changes> extensions/ <files for extension module changes> library/ <files for pure Python module changes> documentation/ <files for documentation changes> tests/ <files for testing changes> 13.02.0/ builtins/ <files for builtin changes> extensions/ <files for extension module changes> library/ <files for pure Python module changes> documentation/ <files for documentation changes> tests/ <files for testing changes> NEWS # Now autogenerated from lang_news and stdlib_news Putting the version information in the directory heirarchy isn't strictly necessary (since the NEWS file generator could figure out from the version history), but does make it easy for *humans* to keep the different versions in order. Why isn't PEP 384 enough? ========================= PEP 384 introduced the notion of a "Stable ABI" for CPython, a limited subset of the full C ABI that is guaranteed to remain stable. Extensions built against the stable ABI should be able to support all subsequent Python versions with the same binary. This will help new projects to avoid coupling their C extension modules too closely to a specific version of CPython. For existing modules, however, migrating to the stable ABI can involve quite a lot of work (especially for extension modules that define a lot of classes). With limited development resources available, any time spent on such a change is time that could otherwise have been spent working on features that are offer more direct benefits to end users. Why not separate out the standard library entirely? =================================================== Because it's a lot of work for next to no pay-off. CPython without the standard library is useless (the build chain won't even finish). You can't create a standalone pure Python standard library, because too many "modules" are actually tightly linked in to the internal details of the respective interpreters (e.g. ``weakref``, ``gc``, ``sys``). Creating a separate development branch that is kept compatible with the previous feature release should provide most of the benefits of a separate standard library repository with only a fraction of the pain. Acknowledgements ================ Thanks go to the PEP 407 authors for starting this discussion, as well as to those authors and Larry Hastings for initial discussions of the proposal made in this PEP. References ========== .. [1] PEP 407: New release cycle and introducing long-term support versions http://www.python.org/dev/peps/pep-0407/ .. [2] Twisted's "topfiles" approach to NEWS generation http://twistedmatrix.com/trac/wiki/ReviewProcess#Newsfiles Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

8 16

Summary of Python tracker Issues
by Python tracker 24 Feb '12

24 Feb '12

ACTIVITY SUMMARY (2012-02-17 - 2012-02-24) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 3277 (+20) closed 22611 (+44) total 25888 (+64) Open issues with patches: 1406 Issues opened (46) ================== #13637: binascii.a2b_* functions could accept unicode strings http://bugs.python.org/issue13637 reopened by r.david.murray #13641: decoding functions in the base64 module could accept unicode s http://bugs.python.org/issue13641 reopened by r.david.murray #14044: IncompleteRead error with urllib2 or urllib.request -- fine wi http://bugs.python.org/issue14044 opened by Alex Quinn #14046: argparse: assertion failure if optional argument has square/ro http://bugs.python.org/issue14046 opened by oxplot #14049: execfile() fails on files that use global variables inside fun http://bugs.python.org/issue14049 opened by techtonik #14050: Tutorial, list.sort() and items comparability http://bugs.python.org/issue14050 opened by sandro.tosi #14055: Implement __sizeof__ for etree Element http://bugs.python.org/issue14055 opened by loewis #14056: Misc doc changes for tarfile http://bugs.python.org/issue14056 opened by eric.araujo #14057: Speedup sysconfig startup http://bugs.python.org/issue14057 opened by haypo #14059: Implement multiprocessing.Barrier http://bugs.python.org/issue14059 opened by anacrolix #14060: Implement a CSP-style channel http://bugs.python.org/issue14060 opened by anacrolix #14061: Clean up archiving code in shutil http://bugs.python.org/issue14061 opened by eric.araujo #14062: UTF-8 Email Subject problem http://bugs.python.org/issue14062 opened by msladek #14065: Element should support cyclic GC http://bugs.python.org/issue14065 opened by loewis #14067: Avoid more stat() calls in importlib http://bugs.python.org/issue14067 opened by pitrou #14069: In extensions (?...) the lookbehind assertion cannot choose be http://bugs.python.org/issue14069 opened by py.user #14070: Idea: Add a flag to reload from source, e.g. reload(module, ig http://bugs.python.org/issue14070 opened by timClicks #14071: allow more than one hash seed per process (move _Py_HashSecret http://bugs.python.org/issue14071 opened by gregory.p.smith #14072: urlparse on tel: URI-s misses the scheme in some cases http://bugs.python.org/issue14072 opened by ivan_herman #14074: argparse does not allow nargs>1 for positional arguments but d http://bugs.python.org/issue14074 opened by tshepang #14075: argparse: unused method? http://bugs.python.org/issue14075 opened by tshepang #14076: sqlite3 module ignores placeholers in CREATE TRIGGER code http://bugs.python.org/issue14076 opened by GuGu #14078: Add 'sourceline' property to xml.etree Elements http://bugs.python.org/issue14078 opened by leonov #14080: Sporadic test_imp failure http://bugs.python.org/issue14080 opened by pitrou #14081: Allow "maxsplit" argument to str.split() to be passed as a key http://bugs.python.org/issue14081 opened by ncoghlan #14082: shutil doesn't support extended attributes http://bugs.python.org/issue14082 opened by pitrou #14085: PyUnicode_WRITE: "comparison is always true" warnings http://bugs.python.org/issue14085 opened by skrah #14087: multiprocessing.Condition.wait_for missing http://bugs.python.org/issue14087 opened by sbt #14088: sys.executable generating canonical path http://bugs.python.org/issue14088 opened by alvesjnr #14089: Patch to increase fractions lib test coverage http://bugs.python.org/issue14089 opened by Oleg.Plakhotnyuk #14092: __name__ inconsistently applied in class definition http://bugs.python.org/issue14092 opened by eukreign #14093: Mercurial version information not appearing in Windows builds http://bugs.python.org/issue14093 opened by vinay.sajip #14094: nt.realpath() should use GetFinalPathNameByHandle() when avail http://bugs.python.org/issue14094 opened by haypo #14095: type_new() removes __qualname__ from the input dictionary http://bugs.python.org/issue14095 opened by haypo #14097: Improve the "introduction" page of the tutorial http://bugs.python.org/issue14097 opened by ezio.melotti #14098: provide public C-API for reading/setting sys.exc_info() http://bugs.python.org/issue14098 opened by scoder #14099: ZipFile.open() should not reopen the underlying file http://bugs.python.org/issue14099 opened by kasal #14100: Add a missing info to PEP 393 + link from whatsnew 3.3 http://bugs.python.org/issue14100 opened by tshepang #14101: example function in tertools.count docstring is misindented http://bugs.python.org/issue14101 opened by zbysz #14102: argparse: add ability to create a man page http://bugs.python.org/issue14102 opened by Daniel.Walsh #14103: argparse: add ability to create a bash_completion script http://bugs.python.org/issue14103 opened by Daniel.Walsh #14104: Implement time.monotonic() on Mac OS X http://bugs.python.org/issue14104 opened by haypo #14105: Breakpoints in debug lost if line is inserted; IDLE http://bugs.python.org/issue14105 opened by ltaylor934 #14106: Distutils manifest: recursive-(include|exclude) matches suffix http://bugs.python.org/issue14106 opened by nadeem.vawda #14107: Debian bigmem buildbot hanging in test_bigmem http://bugs.python.org/issue14107 opened by nadeem.vawda #14073: allow per-thread atexit() http://bugs.python.org/issue14073 opened by tarek Most recent 15 issues with no replies (15) ========================================== #14107: Debian bigmem buildbot hanging in test_bigmem http://bugs.python.org/issue14107 #14106: Distutils manifest: recursive-(include|exclude) matches suffix http://bugs.python.org/issue14106 #14105: Breakpoints in debug lost if line is inserted; IDLE http://bugs.python.org/issue14105 #14104: Implement time.monotonic() on Mac OS X http://bugs.python.org/issue14104 #14094: nt.realpath() should use GetFinalPathNameByHandle() when avail http://bugs.python.org/issue14094 #14089: Patch to increase fractions lib test coverage http://bugs.python.org/issue14089 #14082: shutil doesn't support extended attributes http://bugs.python.org/issue14082 #14078: Add 'sourceline' property to xml.etree Elements http://bugs.python.org/issue14078 #14076: sqlite3 module ignores placeholers in CREATE TRIGGER code http://bugs.python.org/issue14076 #14074: argparse does not allow nargs>1 for positional arguments but d http://bugs.python.org/issue14074 #14072: urlparse on tel: URI-s misses the scheme in some cases http://bugs.python.org/issue14072 #14069: In extensions (?...) the lookbehind assertion cannot choose be http://bugs.python.org/issue14069 #14065: Element should support cyclic GC http://bugs.python.org/issue14065 #14062: UTF-8 Email Subject problem http://bugs.python.org/issue14062 #14055: Implement __sizeof__ for etree Element http://bugs.python.org/issue14055 Most recent 15 issues waiting for review (15) ============================================= #14106: Distutils manifest: recursive-(include|exclude) matches suffix http://bugs.python.org/issue14106 #14101: example function in tertools.count docstring is misindented http://bugs.python.org/issue14101 #14100: Add a missing info to PEP 393 + link from whatsnew 3.3 http://bugs.python.org/issue14100 #14099: ZipFile.open() should not reopen the underlying file http://bugs.python.org/issue14099 #14098: provide public C-API for reading/setting sys.exc_info() http://bugs.python.org/issue14098 #14097: Improve the "introduction" page of the tutorial http://bugs.python.org/issue14097 #14095: type_new() removes __qualname__ from the input dictionary http://bugs.python.org/issue14095 #14093: Mercurial version information not appearing in Windows builds http://bugs.python.org/issue14093 #14089: Patch to increase fractions lib test coverage http://bugs.python.org/issue14089 #14088: sys.executable generating canonical path http://bugs.python.org/issue14088 #14087: multiprocessing.Condition.wait_for missing http://bugs.python.org/issue14087 #14085: PyUnicode_WRITE: "comparison is always true" warnings http://bugs.python.org/issue14085 #14081: Allow "maxsplit" argument to str.split() to be passed as a key http://bugs.python.org/issue14081 #14078: Add 'sourceline' property to xml.etree Elements http://bugs.python.org/issue14078 #14075: argparse: unused method? http://bugs.python.org/issue14075 Top 10 most discussed issues (10) ================================= #13703: Hash collision security issue http://bugs.python.org/issue13703 25 msgs #14080: Sporadic test_imp failure http://bugs.python.org/issue14080 13 msgs #6884: Impossible to include file in sdist that starts with 'build' o http://bugs.python.org/issue6884 11 msgs #13641: decoding functions in the base64 module could accept unicode s http://bugs.python.org/issue13641 11 msgs #13405: Add DTrace probes http://bugs.python.org/issue13405 9 msgs #14073: allow per-thread atexit() http://bugs.python.org/issue14073 9 msgs #13447: Add tests for some scripts in Tools/scripts http://bugs.python.org/issue13447 8 msgs #13873: SIGBUS in test_big_buffer() of test_zlib on Debian bigmem buil http://bugs.python.org/issue13873 8 msgs #14088: sys.executable generating canonical path http://bugs.python.org/issue14088 8 msgs #2377: Replace __import__ w/ importlib.__import__ http://bugs.python.org/issue2377 7 msgs Issues closed (45) ================== #1659: Tests needing network flag? http://bugs.python.org/issue1659 closed by eric.araujo #6039: cygwin compilers should not check compiler versions http://bugs.python.org/issue6039 closed by eric.araujo #6807: No such file or directory: 'msisupport.dll' in msi.py http://bugs.python.org/issue6807 closed by loewis #7813: Bug in command-line module launcher http://bugs.python.org/issue7813 closed by eric.araujo #7966: mhlib does not emit deprecation warning http://bugs.python.org/issue7966 closed by eric.araujo #8033: sqlite: broken long integer handling for arguments to user-def http://bugs.python.org/issue8033 closed by python-dev #9691: sdist includes files that are not in MANIFEST.in http://bugs.python.org/issue9691 closed by eric.araujo #10580: Minor grammar change in Python’s MSI installer http://bugs.python.org/issue10580 closed by loewis #11689: sqlite: Incorrect unit test fails to detect failure http://bugs.python.org/issue11689 closed by python-dev #12406: msi.py needs updating for Python 3.3 http://bugs.python.org/issue12406 closed by loewis #12627: Implement PEP 394: The "python" Command on Unix-Like Systems http://bugs.python.org/issue12627 closed by ned.deily #12702: shutil.copytree() should use os.lutimes() to copy the metadata http://bugs.python.org/issue12702 closed by petri.lehtinen #12817: test_multiprocessing: io.BytesIO() requires bytearray buffers http://bugs.python.org/issue12817 closed by skrah #13909: Ordering of free variables in dis is dependent on dict orderin http://bugs.python.org/issue13909 closed by Mark.Shannon #13974: packaging: test for set_platform() http://bugs.python.org/issue13974 closed by eric.araujo #13978: OSError exception in multiprocessing module when using os.remo http://bugs.python.org/issue13978 closed by neologix #14001: CVE-2012-0845 Python v2.7.2 / v3.2.2 (SimpleXMLRPCServer): DoS http://bugs.python.org/issue14001 closed by neologix #14004: Distutils filelist selects too many files on Windows http://bugs.python.org/issue14004 closed by eric.araujo #14005: IDLE Crash when running/saving a file http://bugs.python.org/issue14005 closed by ned.deily #14020: Improve HTMLParser doc http://bugs.python.org/issue14020 closed by ezio.melotti #14023: bytes implied to be mutable http://bugs.python.org/issue14023 closed by terry.reedy #14038: Packaging test support code raises exception http://bugs.python.org/issue14038 closed by eric.araujo #14040: Deprecate some of the module file formats http://bugs.python.org/issue14040 closed by pitrou #14043: Speed-up importlib's _FileFinder http://bugs.python.org/issue14043 closed by pitrou #14045: In regex pattern long unicode character isn't recognized by re http://bugs.python.org/issue14045 closed by loewis #14047: UTF-8 Email Header http://bugs.python.org/issue14047 closed by eric.araujo #14048: calendar bug related to September 2-14, 1752 http://bugs.python.org/issue14048 closed by mark.dickinson #14051: Cannot set attributes on staticmethod http://bugs.python.org/issue14051 closed by python-dev #14052: importlib mixes up '.' and os.getcwd() http://bugs.python.org/issue14052 closed by brett.cannon #14053: Make patchcheck work with MQ http://bugs.python.org/issue14053 closed by nadeem.vawda #14054: test_importlib failures under Windows http://bugs.python.org/issue14054 closed by brett.cannon #14058: test_sys has started failing http://bugs.python.org/issue14058 closed by vinay.sajip #14063: test_importlib failure on Mac OS X http://bugs.python.org/issue14063 closed by pitrou #14064: collections module imported twice in urllib/parse.py http://bugs.python.org/issue14064 closed by benjamin.peterson #14066: Crash in imputil.imp.find_module when a *directory* exists wit http://bugs.python.org/issue14066 closed by dmalcolm #14068: problem with re split http://bugs.python.org/issue14068 closed by ezio.melotti #14077: sporadic test_multiprocessing failure http://bugs.python.org/issue14077 closed by neologix #14079: Problems with recent test_subprocess changes http://bugs.python.org/issue14079 closed by vinay.sajip #14083: Use local timezone offset by default in datetime.timezone http://bugs.python.org/issue14083 closed by ncoghlan #14084: test_imp resource leak http://bugs.python.org/issue14084 closed by pitrou #14086: str(KeyError("Foo")) Unexpected Result http://bugs.python.org/issue14086 closed by vencabot_teppoo #14090: Bus error on test_big_buffer() of test_zlib, buildbot AMD64 de http://bugs.python.org/issue14090 closed by nadeem.vawda #14091: python subprocess hangs if script called from another director http://bugs.python.org/issue14091 closed by Massimo.Paladin #14096: IDLE quits unexpectedly when some keys are pressed http://bugs.python.org/issue14096 closed by guxianminer #1173134: improvement of the script adaptation for the win32 platform http://bugs.python.org/issue1173134 closed by eric.araujo

1 0

Re: [Python-Dev] cpython: Issue #13706: Fix format(int, "n") for locale with non-ASCII thousands separator
by Antoine Pitrou 24 Feb '12

24 Feb '12

On Fri, 24 Feb 2012 00:49:31 +0100 victor.stinner <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/f89e2f4cda88 > changeset: 75231:f89e2f4cda88 > user: Victor Stinner <victor.stinner(a)haypocalc.com> > date: Fri Feb 24 00:37:51 2012 +0100 > summary: > Issue #13706: Fix format(int, "n") for locale with non-ASCII thousands separator Can you please check the buildbots after you commit something large? This commit broke compilation under Windows. cheers Antoine.

2 1

Re: [Python-Dev] New shared-keys dictionary implementation (issue13903)
by "Martin v. Löwis" 24 Feb '12

24 Feb '12

> Unfortunately it seems to be the norm in CPython to publish almost > everything in header files that get included in "Python.h". In many cases, this is purely for historic reasons. In many additional cases, it's to support fast access macros, at least in the interpreter itself, but then also in extension modules. I agree that moving the structures into the implementation is fine, as long as there are sufficient access functions (for dictionaries, there are plenty, of course). Regards, Martin

1 0

Re: [Python-Dev] [Python-checkins] cpython (3.2): logging: Added locking in flush() and close() handler methods. Thanks to Fayaz
by Gregory P. Smith 24 Feb '12

24 Feb '12

On Thu, Feb 23, 2012 at 12:04 PM, vinay.sajip <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/b2adcd90e656 > changeset: 75211:b2adcd90e656 > branch: 3.2 > parent: 75200:85d08a1ba74e > user: Vinay Sajip <vinay_sajip(a)yahoo.co.uk> > date: Thu Feb 23 19:45:52 2012 +0000 > summary: > logging: Added locking in flush() and close() handler methods. Thanks to Fayaz Yusuf Khan for the suggestion. > > files: > Lib/logging/__init__.py | 24 +++++++++------- > Lib/logging/handlers.py | 40 +++++++++++++++------------- > 2 files changed, 35 insertions(+), 29 deletions(-) > > > diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py > --- a/Lib/logging/__init__.py > +++ b/Lib/logging/__init__.py > @@ -1,4 +1,4 @@ > -# Copyright 2001-2010 by Vinay Sajip. All Rights Reserved. > +# Copyright 2001-2012 by Vinay Sajip. All Rights Reserved. > # > # Permission to use, copy, modify, and distribute this software and its > # documentation for any purpose and without fee is hereby granted, > @@ -16,9 +16,9 @@ > > """ > Logging package for Python. Based on PEP 282 and comments thereto in > -comp.lang.python, and influenced by Apache's log4j system. > +comp.lang.python. > > -Copyright (C) 2001-2011 Vinay Sajip. All Rights Reserved. > +Copyright (C) 2001-2012 Vinay Sajip. All Rights Reserved. > > To use, simply 'import logging' and log away! > """ > @@ -917,8 +917,9 @@ > """ > Flushes the stream. > """ > - if self.stream and hasattr(self.stream, "flush"): > - self.stream.flush() > + with self.lock: > + if self.stream and hasattr(self.stream, "flush"): > + self.stream.flush() I don't know if anyone actually builds Python without thread support anymore, but if so, self.lock will be set to None and these "with self.lock"s will fail. Perhaps change lock = None to self.lock = some dummy duck typed lock that supports use as a context manager and acquire/release calls? > > def emit(self, record): > """ > @@ -969,12 +970,13 @@ > """ > Closes the stream. > """ > - if self.stream: > - self.flush() > - if hasattr(self.stream, "close"): > - self.stream.close() > - StreamHandler.close(self) > - self.stream = None > + with self.lock: > + if self.stream: > + self.flush() > + if hasattr(self.stream, "close"): > + self.stream.close() > + StreamHandler.close(self) > + self.stream = None > > def _open(self): > """ > diff --git a/Lib/logging/handlers.py b/Lib/logging/handlers.py > --- a/Lib/logging/handlers.py > +++ b/Lib/logging/handlers.py > @@ -1,4 +1,4 @@ > -# Copyright 2001-2010 by Vinay Sajip. All Rights Reserved. > +# Copyright 2001-2012 by Vinay Sajip. All Rights Reserved. > # > # Permission to use, copy, modify, and distribute this software and its > # documentation for any purpose and without fee is hereby granted, > @@ -16,10 +16,9 @@ > > """ > Additional handlers for the logging package for Python. The core package is > -based on PEP 282 and comments thereto in comp.lang.python, and influenced by > -Apache's log4j system. > +based on PEP 282 and comments thereto in comp.lang.python. > > -Copyright (C) 2001-2010 Vinay Sajip. All Rights Reserved. > +Copyright (C) 2001-2012 Vinay Sajip. All Rights Reserved. > > To use, simply 'import logging.handlers' and log away! > """ > @@ -554,10 +553,11 @@ > """ > Closes the socket. > """ > - if self.sock: > - self.sock.close() > - self.sock = None > - logging.Handler.close(self) > + with self.lock: > + if self.sock: > + self.sock.close() > + self.sock = None > + logging.Handler.close(self) > > class DatagramHandler(SocketHandler): > """ > @@ -752,9 +752,10 @@ > """ > Closes the socket. > """ > - if self.unixsocket: > - self.socket.close() > - logging.Handler.close(self) > + with self.lock: > + if self.unixsocket: > + self.socket.close() > + logging.Handler.close(self) > > def mapPriority(self, levelName): > """ > @@ -1095,7 +1096,8 @@ > > This version just zaps the buffer to empty. > """ > - self.buffer = [] > + with self.lock: > + self.buffer = [] > > def close(self): > """ > @@ -1145,18 +1147,20 @@ > > The record buffer is also cleared by this operation. > """ > - if self.target: > - for record in self.buffer: > - self.target.handle(record) > - self.buffer = [] > + with self.lock: > + if self.target: > + for record in self.buffer: > + self.target.handle(record) > + self.buffer = [] > > def close(self): > """ > Flush, set the target to None and lose the buffer. > """ > self.flush() > - self.target = None > - BufferingHandler.close(self) > + with self.lock: > + self.target = None > + BufferingHandler.close(self) > > > class QueueHandler(logging.Handler): > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > http://mail.python.org/mailman/listinfo/python-checkins >

1 1

requirements for moving __import__ over to importlib?
by Brett Cannon 24 Feb '12

24 Feb '12

I'm going to start this off with the caveat that hg.python.org/sandbox/bcannon#bootstrap_importlib is not completely at feature parity, but getting there shouldn't be hard. There is a FAILING file that has a list of the tests that are not passing because importlib bootstrapping and a comment as to why (I think) they are failing. But no switch would ever happen until the test suite passes. Anyway, to start this conversation I'm going to open with why I think removing most of the C code in Python/import.c and replacing it with importlib/_bootstrap.py is a positive thing. One is maintainability. Antoine mentioned how if change occurs everyone is going to have to be able to fix code in importlib, and that's the point! I don't know about the rest of you but I find Python code easier to work with than C code (and if you don't you might be subscribed to the wrong mailing list =). I would assume the ability to make changes or to fix bugs will be a lot easier with importlib than import.c. So maintainability should be easier when it comes to imports. Two is APIs. PEP 302 introduced this idea of an API for objects that can perform imports so that people can control it, enhance it, introspect it, etc. But as it stands right now, import.c implements none of PEP 302 for any built-in import mechanism. This mostly stems from positive thing #1 I just mentioned. but since I was able to do this code from scratch I was able to design for (and extend) PEP 302 compliance in order to make sure the entire import system was exposed cleanly. This means it is much easier now to write a custom importer for quirky syntax, a different storage mechanism, etc. Third is multi-VM support. IronPython, Jython, and PyPy have all said they would love importlib to become the default import implementation so that all VMs have the same implementation. Some people have even said they will use importlib regardless of what CPython does simply to ease their coding burden, but obviously that still leads to the possibility of subtle semantic differences that would go away if all VMs used the same implementation. So switching would lead to one less possible semantic difference between the various VMs. So, that is the positives. What are the negatives? Performance, of course. Now I'm going to be upfront and say I really did not want to have this performance conversation now as I have done *NO* profiling or analysis of the algorithms used in importlib in order to tune performance (e.g. the function that handles case-sensitivity, which is on the critical path for importing source code, has a platform check which could go away if I instead had platform-specific versions of the function that were assigned to a global variable at startup). I also know that people have a bad habit of latching on to micro-benchmark numbers, especially for something like import which involves startup or can easily be measured. I mean I wrote importlib.test.benchmark to help measure performance changes in any algorithmic changes I might make, but it isn't a real-world benchmark like what Unladen Swallow gave us (e.g. the two start-up benchmarks that use real-world apps -- hg and bzr -- aren't available on Python 3 so only normal_startup and nosite_startup can be used ATM). IOW I really do not look forward to someone saying "importlib is so much slower at importing a module containing ``pass``" when (a) that never happens, and (b) most programs do not spend their time importing but instead doing interesting work. For instance, right now importlib does ``python -c "import decimal"`` (which, BTW, is the largest module in the stdlib) 25% slower on my machine with a pydebug build (a non-debug build would probably be in my favor as I have more Python objects being used in importlib and thus more sanity checks). But if you do something (very) slightly more interesting like ``python -m calendar`` where is a slight amount of work then importlib is currently only 16% slower. So it all depends on how we measure (as usual). So, if there is going to be some baseline performance target I need to hit to make people happy I would prefer to know what that (real-world) benchmark is and what the performance target is going to be on a non-debug build. And if people are not worried about the performance then I'm happy with that as well. =)

11 43

Re: [Python-Dev] [Python-checkins] peps: Switch back to named functions, since the Ellipsis version degenerated badly
by Jim Jewett 23 Feb '12

23 Feb '12

On Wed, Feb 22, 2012 at 10:22 AM, nick.coghlan <python-checkins(a)python.org> wrote: > + in x = weakref.ref(target, report_destruction) > + def report_destruction(obj): > print("{} is being destroyed".format(obj)) > +If the repetition of the name seems especially annoying, then a throwaway > +name like ``f`` can be used instead:: > + in x = weakref.ref(target, f) > + def f(obj): > + print("{} is being destroyed".format(obj)) I still feel that the helper function (or class) is subordinate, and should be indented. Thinking of "in ..." as a decorator helps, but makes it seem that the helper function is the important part (which it sometimes is...) I understand that adding a colon and indent has its own problems, but ... I'm not certain this is better, and I am certain that the desire for indentation is strong enough to at least justify discussion in the PEP. -jJ

1 0

Re: [Python-Dev] cpython: Refactor importlib to make it easier to re-implement in C.
by Antoine Pitrou 23 Feb '12

23 Feb '12

On Thu, 23 Feb 2012 16:18:19 +0100 brett.cannon <python-checkins(a)python.org> wrote: > def _sanity_check(name, package, level): > """Verify arguments are "sane".""" > + if not hasattr(name, 'rpartition'): > + raise TypeError("module name must be str, not {}".format(type(name))) Why don't you simply use isinstance()? (bytes objects also have rpartition()) Regards Antoine.

2 1