Mailman 3 September 2013 - Python-Dev

PEP 1, PEP Purpose and Guidelines
by barry＠zope.com May 18, 2021

May 18, 2021

It has been a while since I posted a copy of PEP 1 to the mailing lists and newsgroups. I've recently done some updating of a few sections, so in the interest of gaining wider community participation in the Python development process, I'm posting the latest revision of PEP 1 here. A version of the PEP is always available on-line at http://www.python.org/peps/pep-0001.html Enjoy, -Barry -------------------- snip snip -------------------- PEP: 1 Title: PEP Purpose and Guidelines Version: … [View More]$Revision: 1.36 $ Last-Modified: $Date: 2002/07/29 18:34:59 $ Author: Barry A. Warsaw, Jeremy Hylton Status: Active Type: Informational Created: 13-Jun-2000 Post-History: 21-Mar-2001, 29-Jul-2002 What is a PEP? PEP stands for Python Enhancement Proposal. A PEP is a design document providing information to the Python community, or describing a new feature for Python. The PEP should provide a concise technical specification of the feature and a rationale for the feature. We intend PEPs to be the primary mechanisms for proposing new features, for collecting community input on an issue, and for documenting the design decisions that have gone into Python. The PEP author is responsible for building consensus within the community and documenting dissenting opinions. Because the PEPs are maintained as plain text files under CVS control, their revision history is the historical record of the feature proposal[1]. Kinds of PEPs There are two kinds of PEPs. A standards track PEP describes a new feature or implementation for Python. An informational PEP describes a Python design issue, or provides general guidelines or information to the Python community, but does not propose a new feature. Informational PEPs do not necessarily represent a Python community consensus or recommendation, so users and implementors are free to ignore informational PEPs or follow their advice. PEP Work Flow The PEP editor, Barry Warsaw <peps(a)python.org>, assigns numbers for each PEP and changes its status. The PEP process begins with a new idea for Python. It is highly recommended that a single PEP contain a single key proposal or new idea. The more focussed the PEP, the more successfully it tends to be. The PEP editor reserves the right to reject PEP proposals if they appear too unfocussed or too broad. If in doubt, split your PEP into several well-focussed ones. Each PEP must have a champion -- someone who writes the PEP using the style and format described below, shepherds the discussions in the appropriate forums, and attempts to build community consensus around the idea. The PEP champion (a.k.a. Author) should first attempt to ascertain whether the idea is PEP-able. Small enhancements or patches often don't need a PEP and can be injected into the Python development work flow with a patch submission to the SourceForge patch manager[2] or feature request tracker[3]. The PEP champion then emails the PEP editor <peps(a)python.org> with a proposed title and a rough, but fleshed out, draft of the PEP. This draft must be written in PEP style as described below. If the PEP editor approves, he will assign the PEP a number, label it as standards track or informational, give it status 'draft', and create and check-in the initial draft of the PEP. The PEP editor will not unreasonably deny a PEP. Reasons for denying PEP status include duplication of effort, being technically unsound, not providing proper motivation or addressing backwards compatibility, or not in keeping with the Python philosophy. The BDFL (Benevolent Dictator for Life, Guido van Rossum) can be consulted during the approval phase, and is the final arbitrator of the draft's PEP-ability. If a pre-PEP is rejected, the author may elect to take the pre-PEP to the comp.lang.python newsgroup (a.k.a. python-list(a)python.org mailing list) to help flesh it out, gain feedback and consensus from the community at large, and improve the PEP for re-submission. The author of the PEP is then responsible for posting the PEP to the community forums, and marshaling community support for it. As updates are necessary, the PEP author can check in new versions if they have CVS commit permissions, or can email new PEP versions to the PEP editor for committing. Standards track PEPs consists of two parts, a design document and a reference implementation. The PEP should be reviewed and accepted before a reference implementation is begun, unless a reference implementation will aid people in studying the PEP. Standards Track PEPs must include an implementation - in the form of code, patch, or URL to same - before it can be considered Final. PEP authors are responsible for collecting community feedback on a PEP before submitting it for review. A PEP that has not been discussed on python-list(a)python.org and/or python-dev(a)python.org will not be accepted. However, wherever possible, long open-ended discussions on public mailing lists should be avoided. Strategies to keep the discussions efficient include, setting up a separate SIG mailing list for the topic, having the PEP author accept private comments in the early design phases, etc. PEP authors should use their discretion here. Once the authors have completed a PEP, they must inform the PEP editor that it is ready for review. PEPs are reviewed by the BDFL and his chosen consultants, who may accept or reject a PEP or send it back to the author(s) for revision. Once a PEP has been accepted, the reference implementation must be completed. When the reference implementation is complete and accepted by the BDFL, the status will be changed to `Final.' A PEP can also be assigned status `Deferred.' The PEP author or editor can assign the PEP this status when no progress is being made on the PEP. Once a PEP is deferred, the PEP editor can re-assign it to draft status. A PEP can also be `Rejected'. Perhaps after all is said and done it was not a good idea. It is still important to have a record of this fact. PEPs can also be replaced by a different PEP, rendering the original obsolete. This is intended for Informational PEPs, where version 2 of an API can replace version 1. PEP work flow is as follows: Draft -> Accepted -> Final -> Replaced ^ +----> Rejected v Deferred Some informational PEPs may also have a status of `Active' if they are never meant to be completed. E.g. PEP 1. What belongs in a successful PEP? Each PEP should have the following parts: 1. Preamble -- RFC822 style headers containing meta-data about the PEP, including the PEP number, a short descriptive title (limited to a maximum of 44 characters), the names, and optionally the contact info for each author, etc. 2. Abstract -- a short (~200 word) description of the technical issue being addressed. 3. Copyright/public domain -- Each PEP must either be explicitly labelled as placed in the public domain (see this PEP as an example) or licensed under the Open Publication License[4]. 4. Specification -- The technical specification should describe the syntax and semantics of any new language feature. The specification should be detailed enough to allow competing, interoperable implementations for any of the current Python platforms (CPython, JPython, Python .NET). 5. Motivation -- The motivation is critical for PEPs that want to change the Python language. It should clearly explain why the existing language specification is inadequate to address the problem that the PEP solves. PEP submissions without sufficient motivation may be rejected outright. 6. Rationale -- The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work, e.g. how the feature is supported in other languages. The rationale should provide evidence of consensus within the community and discuss important objections or concerns raised during discussion. 7. Backwards Compatibility -- All PEPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. The PEP must explain how the author proposes to deal with these incompatibilities. PEP submissions without a sufficient backwards compatibility treatise may be rejected outright. 8. Reference Implementation -- The reference implementation must be completed before any PEP is given status 'Final,' but it need not be completed before the PEP is accepted. It is better to finish the specification and rationale first and reach consensus on it before writing code. The final implementation must include test code and documentation appropriate for either the Python language reference or the standard library reference. PEP Template PEPs are written in plain ASCII text, and should adhere to a rigid style. There is a Python script that parses this style and converts the plain text PEP to HTML for viewing on the web[5]. PEP 9 contains a boilerplate[7] template you can use to get started writing your PEP. Each PEP must begin with an RFC822 style header preamble. The headers must appear in the following order. Headers marked with `*' are optional and are described below. All other headers are required. PEP: <pep number> Title: <pep title> Version: <cvs version string> Last-Modified: <cvs date string> Author: <list of authors' real names and optionally, email addrs> * Discussions-To: <email address> Status: <Draft | Active | Accepted | Deferred | Final | Replaced> Type: <Informational | Standards Track> * Requires: <pep numbers> Created: <date created on, in dd-mmm-yyyy format> * Python-Version: <version number> Post-History: <dates of postings to python-list and python-dev> * Replaces: <pep number> * Replaced-By: <pep number> The Author: header lists the names and optionally, the email addresses of all the authors/owners of the PEP. The format of the author entry should be address(a)dom.ain (Random J. User) if the email address is included, and just Random J. User if the address is not given. If there are multiple authors, each should be on a separate line following RFC 822 continuation line conventions. Note that personal email addresses in PEPs will be obscured as a defense against spam harvesters. Standards track PEPs must have a Python-Version: header which indicates the version of Python that the feature will be released with. Informational PEPs do not need a Python-Version: header. While a PEP is in private discussions (usually during the initial Draft phase), a Discussions-To: header will indicate the mailing list or URL where the PEP is being discussed. No Discussions-To: header is necessary if the PEP is being discussed privately with the author, or on the python-list or python-dev email mailing lists. Note that email addresses in the Discussions-To: header will not be obscured. Created: records the date that the PEP was assigned a number, while Post-History: is used to record the dates of when new versions of the PEP are posted to python-list and/or python-dev. Both headers should be in dd-mmm-yyyy format, e.g. 14-Aug-2001. PEPs may have a Requires: header, indicating the PEP numbers that this PEP depends on. PEPs may also have a Replaced-By: header indicating that a PEP has been rendered obsolete by a later document; the value is the number of the PEP that replaces the current document. The newer PEP must have a Replaces: header containing the number of the PEP that it rendered obsolete. PEP Formatting Requirements PEP headings must begin in column zero and the initial letter of each word must be capitalized as in book titles. Acronyms should be in all capitals. The body of each section must be indented 4 spaces. Code samples inside body sections should be indented a further 4 spaces, and other indentation can be used as required to make the text readable. You must use two blank lines between the last line of a section's body and the next section heading. You must adhere to the Emacs convention of adding two spaces at the end of every sentence. You should fill your paragraphs to column 70, but under no circumstances should your lines extend past column 79. If your code samples spill over column 79, you should rewrite them. Tab characters must never appear in the document at all. A PEP should include the standard Emacs stanza included by example at the bottom of this PEP. A PEP must contain a Copyright section, and it is strongly recommended to put the PEP in the public domain. When referencing an external web page in the body of a PEP, you should include the title of the page in the text, with a footnote reference to the URL. Do not include the URL in the body text of the PEP. E.g. Refer to the Python Language web site [1] for more details. ... [1] http://www.python.org When referring to another PEP, include the PEP number in the body text, such as "PEP 1". The title may optionally appear. Add a footnote reference that includes the PEP's title and author. It may optionally include the explicit URL on a separate line, but only in the References section. Note that the pep2html.py script will calculate URLs automatically, e.g.: ... Refer to PEP 1 [7] for more information about PEP style ... References [7] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton http://www.python.org/peps/pep-0001.html If you decide to provide an explicit URL for a PEP, please use this as the URL template: http://www.python.org/peps/pep-xxxx.html PEP numbers in URLs must be padded with zeros from the left, so as to be exactly 4 characters wide, however PEP numbers in text are never padded. Reporting PEP Bugs, or Submitting PEP Updates How you report a bug, or submit a PEP update depends on several factors, such as the maturity of the PEP, the preferences of the PEP author, and the nature of your comments. For the early draft stages of the PEP, it's probably best to send your comments and changes directly to the PEP author. For more mature, or finished PEPs you may want to submit corrections to the SourceForge bug manager[6] or better yet, the SourceForge patch manager[2] so that your changes don't get lost. If the PEP author is a SF developer, assign the bug/patch to him, otherwise assign it to the PEP editor. When in doubt about where to send your changes, please check first with the PEP author and/or PEP editor. PEP authors who are also SF committers, can update the PEPs themselves by using "cvs commit" to commit their changes. Remember to also push the formatted PEP text out to the web by doing the following: % python pep2html.py -i NUM where NUM is the number of the PEP you want to push out. See % python pep2html.py --help for details. Transferring PEP Ownership It occasionally becomes necessary to transfer ownership of PEPs to a new champion. In general, we'd like to retain the original author as a co-author of the transferred PEP, but that's really up to the original author. A good reason to transfer ownership is because the original author no longer has the time or interest in updating it or following through with the PEP process, or has fallen off the face of the 'net (i.e. is unreachable or not responding to email). A bad reason to transfer ownership is because you don't agree with the direction of the PEP. We try to build consensus around a PEP, but if that's not possible, you can always submit a competing PEP. If you are interested assuming ownership of a PEP, send a message asking to take over, addressed to both the original author and the PEP editor <peps(a)python.org>. If the original author doesn't respond to email in a timely manner, the PEP editor will make a unilateral decision (it's not like such decisions can be reversed. :). References and Footnotes [1] This historical record is available by the normal CVS commands for retrieving older revisions. For those without direct access to the CVS tree, you can browse the current and past PEP revisions via the SourceForge web site at http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi/python/nondist/peps/?cvsroot=… [2] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [3] http://sourceforge.net/tracker/?atid=355470&group_id=5470&func=browse [4] http://www.opencontent.org/openpub/ [5] The script referred to here is pep2html.py, which lives in the same directory in the CVS tree as the PEPs themselves. Try "pep2html.py --help" for details. The URL for viewing PEPs on the web is http://www.python.org/peps/ [6] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [7] PEP 9, Sample PEP Template http://www.python.org/peps/pep-0009.html Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: [View Less]

8 14

More compact dictionaries with faster iteration
by Raymond Hettinger Jan. 3, 2015

Jan. 3, 2015

The current memory layout for dictionaries is unnecessarily inefficient. It has a sparse table of 24-byte entries containing the hash value, key pointer, and value pointer. Instead, the 24-byte entries should be stored in a dense table referenced by a sparse table of indices. For example, the dictionary: d = {'timmy': 'red', 'barry': 'green', 'guido': 'blue'} is currently stored as: entries = [['--', '--', '--'], [-8522787127447073495, 'barry', 'green'], … [View More] ['--', '--', '--'], ['--', '--', '--'], ['--', '--', '--'], [-9092791511155847987, 'timmy', 'red'], ['--', '--', '--'], [-6480567542315338377, 'guido', 'blue']] Instead, the data should be organized as follows: indices = [None, 1, None, None, None, 0, None, 2] entries = [[-9092791511155847987, 'timmy', 'red'], [-8522787127447073495, 'barry', 'green'], [-6480567542315338377, 'guido', 'blue']] Only the data layout needs to change. The hash table algorithms would stay the same. All of the current optimizations would be kept, including key-sharing dicts and custom lookup functions for string-only dicts. There is no change to the hash functions, the table search order, or collision statistics. The memory savings are significant (from 30% to 95% compression depending on the how full the table is). Small dicts (size 0, 1, or 2) get the most benefit. For a sparse table of size t with n entries, the sizes are: curr_size = 24 * t new_size = 24 * n + sizeof(index) * t In the above timmy/barry/guido example, the current size is 192 bytes (eight 24-byte entries) and the new size is 80 bytes (three 24-byte entries plus eight 1-byte indices). That gives 58% compression. Note, the sizeof(index) can be as small as a single byte for small dicts, two bytes for bigger dicts and up to sizeof(Py_ssize_t) for huge dict. In addition to space savings, the new memory layout makes iteration faster. Currently, keys(), values, and items() loop over the sparse table, skipping-over free slots in the hash table. Now, keys/values/items can loop directly over the dense table, using fewer memory accesses. Another benefit is that resizing is faster and touches fewer pieces of memory. Currently, every hash/key/value entry is moved or copied during a resize. In the new layout, only the indices are updated. For the most part, the hash/key/value entries never move (except for an occasional swap to fill a hole left by a deletion). With the reduced memory footprint, we can also expect better cache utilization. For those wanting to experiment with the design, there is a pure Python proof-of-concept here: http://code.activestate.com/recipes/578375 YMMV: Keep in mind that the above size statics assume a build with 64-bit Py_ssize_t and 64-bit pointers. The space savings percentages are a bit different on other builds. Also, note that in many applications, the size of the data dominates the size of the container (i.e. the weight of a bucket of water is mostly the water, not the bucket). Raymond [View Less]

27 70

Semi-official read-only Github mirror of the CPython Mercurial repository
by Eli Bendersky Oct. 15, 2014

Oct. 15, 2014

Hi all, https://github.com/python/cpython is now live as a semi-official, *read only* Github mirror of the CPython Mercurial repository. Let me know if you have any problems/concerns. I still haven't decided how often to update it (considering either just N times a day, or maybe use a Hg hook for batching). Suggestions are welcome. The methodology I used to create it is via hg-fast-export. I also tried to pack and gc the git repo as much as possible before the initial Github push - it went … [View More]

4 8

Reviving restricted mode?
by Guido van Rossum Aug. 14, 2014

Aug. 14, 2014

I've received some enthusiastic emails from someone who wants to revive restricted mode. He started out with a bunch of patches to the CPython runtime using ctypes, which he attached to an App Engine bug: http://code.google.com/p/googleappengine/issues/detail?id=671 Based on his code (the file secure.py is all you need, included in secure.tar.gz) it seems he believes the only security leaks are __subclasses__, gi_frame and gi_code. (I have since convinced him that if we add "restricted" … [View More]

19 35

CLA link from bugs.python.org
by Tim Delaney Feb. 16, 2014

Feb. 16, 2014

It appears there's no obvious link from bugs.python.org to the contributor agreement - you need to go via the unintuitive link Foundation -> Contribution Forms (and from what I've read, you're prompted when you add a patch to the tracker). I'd suggest that if the "Contributor Form Received" field is "No" in user details, there be a link to http://www.python.org/psf/contrib/. Tim Delaney

3 4

cffi in stdlib
by Maciej Fijalkowski Dec. 19, 2013

Dec. 19, 2013

Hello. I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib. This is a project Armin Rigo has been working for a while, with some input from other developers. It seems that the main reason why people would prefer ctypes over cffi these days is "because it's included in stdlib", which is not generally the reason I would like to hear. Our calls to not use C extensions and to use an FFI instead has seen very limited success with ctypes and quite a lot more … [View More]since cffi got released. The API is fairly stable right now with minor changes going in and it'll definitely stablize until Python 3.4 release. Notable projects using it: * pypycore - gevent main loop ported to cffi * pgsql2cffi * sdl-cffi bindings * tls-cffi bindings * lxml-cffi port * pyzmq * cairo-cffi * a bunch of others So relatively a lot given that the project is not even a year old (it got 0.1 release in June). As per documentation, the advantages over ctypes: * The goal is to call C code from Python. You should be able to do so without learning a 3rd language: every alternative requires you to learn their own language (Cython, SWIG) or API (ctypes). So we tried to assume that you know Python and C and minimize the extra bits of API that you need to learn. * Keep all the Python-related logic in Python so that you don’t need to write much C code (unlike CPython native C extensions). * Work either at the level of the ABI (Application Binary Interface) or the API (Application Programming Interface). Usually, C libraries have a specified C API but often not an ABI (e.g. they may document a “struct” as having at least these fields, but maybe more). (ctypes works at the ABI level, whereas Cython and native C extensions work at the API level.) * We try to be complete. For now some C99 constructs are not supported, but all C89 should be, including macros (and including macro “abuses”, which you can manually wrap in saner-looking C functions). * We attempt to support both PyPy and CPython, with a reasonable path for other Python implementations like IronPython and Jython. * Note that this project is not about embedding executable C code in Python, unlike Weave. This is about calling existing C libraries from Python. so among other things, making a cffi extension gives you the same level of security as writing C (and unlike ctypes) and brings quite a bit more flexibility (API vs ABI issue) that let's you wrap arbitrary libraries, even those full of macros. Cheers, fijal .. [1]: http://cffi.readthedocs.org/en/release-0.5/ [View Less]

22 75

PEP 455: TransformDict
by Antoine Pitrou Oct. 30, 2013

Oct. 30, 2013

Hello, Following the python-dev discussion, I've written a PEP to recap the proposal and the various arguments. It's inlined below, and it will probably appear soon at http://www.python.org/dev/peps/pep-0455/, too. Regards Antoine. PEP: 455 Title: Adding a key-transforming dictionary to collections Version: $Revision$ Last-Modified: $Date$ Author: Antoine Pitrou <solipsis(a)pitrou.net> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 13-Sep-2013 Python-Version: 3.… [View More]4 Post-History: Abstract ======== This PEP proposes a new data structure for the ``collections`` module, called "TransformDict" in this PEP. This structure is a mutable mapping which transforms the key using a given function when doing a lookup, but retains the original key when reading. Rationale ========= Numerous specialized versions of this pattern exist. The most common is a case-insensitive case-preserving dict, i.e. a dict-like container which matches keys in a case-insensitive fashion but retains the original casing. It is a very common need in network programming, as many protocols feature some arrays of "key / value" properties in their messages, where the keys are textual strings whose casing isn't relevant. Another common request is an identity dict, where keys are matched according to their respective id()s instead of normal matching. Both are instances of a more general pattern, where a given transformation function is applied to keys when looking them up: that function being ``str.lower`` in the former example and the built-in ``id`` function in the latter. (it can be said that the pattern *projects* keys from the user-visible set onto the internal lookup set, hence this PEP's title) Semantics ========= TransformDict is a ``MutableMapping`` implementation: it faithfully implements the well-known API of mutable mappings, as ``dict`` itself and other dict-like classes in the standard library. Therefore, this PEP won't rehash the semantics of most TransformDict methods. The transformation function needn't be bijective, it can be strictly surjective as in the case-insensitive example:: >>> d = TransformDict(str.lower) >>> d['SomeKey'] = 5 >>> d['somekey'] 5 >>> d['SOMEKEY'] 5 TransformDict retains the first key used when creating an entry:: >>> d = TransformDict(str.lower) >>> d['SomeKey'] = 1 >>> d['somekey'] = 2 >>> list(d.items()) [('SomeKey', 2)] The original keys needn't be hashable, as long as the transformation function returns a hashable one:: >>> d = TransformDict(id) >>> l = [None] >>> d[l] = 5 >>> l in d True Constructor ----------- As shown in the example aboves, creating a TransformDict requires passing the key transformation function as the first argument (much like creating a ``defaultdict`` requires passing the factory function as first argument). The constructor also takes other optional arguments which can be used to initialize the TransformDict with certain key-value pairs. Those optional arguments are the same as in the ``dict`` and ``defaultdict`` constructors:: >>> d = TransformDict(str.lower, [('Foo': 1)], Bar=2) >>> sorted(d.items()) [('Bar', 2), ('Foo', 1)] Alternative proposals and questions =================================== Retaining the last original key ------------------------------- Most python-dev respondents found retaining the first user-supplied key more intuitive than retaining the last. Also, it matches the dict object's own behaviour when using different but equal keys:: >>> d = {} >>> d[1] = 'hello' >>> d[1.0] = 'world' >>> d {1: 'world'} Furthermore, explicitly retaining the last key in a first-key-retaining scheme is still possible using the following approach:: d.pop(key, None) d[key] = value while the converse (retaining the first key in a last-key-retaining scheme) doesn't look possible without rewriting part of the container's code. Using an encoder / decoder pair ------------------------------- Using a function pair isn't necessary, since the original key is retained by the container. Moreover, an encoder / decoder pair would require the transformation to be bijective, which prevents important use cases like case-insensitive matching. Providing a transformation function for values ---------------------------------------------- Dictionary values are not used for lookup, their semantics are totally irrelevant to the container's operation. Therefore, there is no point in having both an "original" and a "transformed" value: the transformed value wouldn't be used for anything. Providing a specialized container, not generic ---------------------------------------------- It was asked why we would provide the generic TransformDict construct rather than a specialized case-insensitive dict variant. The answer is that it's nearly as cheap (code-wise and performance-wise) to provide the generic construct, and it can fill more use cases. Implementation ============== A patch for the collections module is tracked on the bug tracker at http://bugs.python.org/issue18986. Existing work ============= Case-insensitive dicts are a popular request: * http://twistedmatrix.com/documents/current/api/twisted.python.util.Insensit… * https://mail.python.org/pipermail/python-list/2013-May/647243.html * https://mail.python.org/pipermail/python-list/2005-April/296208.html * https://mail.python.org/pipermail/python-list/2004-June/241748.html * http://bugs.python.org/msg197376 * http://stackoverflow.com/a/2082169 * http://stackoverflow.com/a/3296782 * http://code.activestate.com/recipes/66315-case-insensitive-dictionary/ * https://gist.github.com/babakness/3901174 * http://www.wikier.org/blog/key-insensitive-dictionary-in-python * http://en.sharejs.com/python/14534 * http://www.voidspace.org.uk/python/archive.shtml#caseless Identity dicts have been requested too: * https://mail.python.org/pipermail/python-ideas/2010-May/007235.html * http://www.gossamer-threads.com/lists/python/python/209527 Python's own pickle module uses identity lookups for object memoization: http://hg.python.org/cpython/file/0e70bf1f32a3/Lib/pickle.py#l234 Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: [View Less]

22 101

compile python 3.3 with bz2 support
by Isml Oct. 22, 2013

Oct. 22, 2013

hi, everyone: I want to compile python 3.3 with bz2 support on RedHat 5.5 but fail to do that. Here is how I do it: 1、download bzip2 and compile it(make、make -f Makefile_libbz2_so、make install) 2、chang to python 3.3 source directory : ./configure --with-bz2=/usr/local/include 3、make 4、make install after installation complete, I test it： [root@localhost Python-3.3.0]# python3 -c "import bz2" Traceback (most recent call last): File "<string>", line 1, … [View More]

3 2

Re: [Python-Dev] cpython (2.7): Add fake buildbottouch target.
by Antoine Pitrou Oct. 2, 2013

Oct. 2, 2013

On Mon, 30 Sep 2013 16:18:57 +0200 (CEST) martin.v.loewis <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/c1a294bbb4fa > changeset: 85882:c1a294bbb4fa > branch: 2.7 > parent: 85877:dd55d54b2a15 > user: Martin v. Löwis <martin(a)v.loewis.de> > date: Mon Sep 30 16:18:31 2013 +0200 > summary: > Add fake buildbottouch target. "make touch" does exist in 2.7, so was this new target needed? Regards Antoine.

2 1

Python for new users
by Nick Efford Oct. 1, 2013

Oct. 1, 2013

In the 'PEP453 ready for pronouncement' thread, Donald said >> Because reality is that new users are still likely to be using Python 2.7. >> Python 3 is just now starting to be really usable, however there's a huge >> corpus of existing tutorials, course work, books etc for Python 2.7. As >> Python 3 becomes more usable that existing corpus of material will be >> ported over to Python 3 but in the interim there is still a pretty large >> hurdle for new users … [View More]to get over. And Guido replied > Based on my day-to-day experience this is still very true. (And yes, I'm > slowly turning the tide. But it will take a long time and I am committed to > giving users the choice.) Widely-used and linked web resources tend to persist for a very long time, so we shouldn't use the prevalence of Python 2 resources as a reason for excessive caution. The key question is how much good material is available based on Python 3 - and this has improved significantly over the past couple of years. The classic "How to Think Like a Computer Scientist" has an excellent Python 3 version available at http://interactivepython.org, for example. Things are changing with print media, too. Pragmatic Programmers are about to publish the 2nd edition of "Practical Programming", based on Python 3. Most of the major academic publishers have released Python 3 books in the last 12 months. The tide is definitely turning, perhaps has already turned. Encouraging the continued use of 2.7 for existing programmers is entirely justifiable, but for *newcomers* to programming I think it is now much harder to justify. A stronger case could have been made a couple of years ago, when many important packages were not yet available for Python 3, but things have changed. Even big frameworks like Django are now usable with Python 3. If we aren't yet past the point where package availability shouldn't be regarded as an adoption barrier by beginners, we are surely very close. I've been teaching Python as a first language to university students for many years now, initially with Python 2 and for the last few years with Python 3. In my experience, they encounter fewer problems with Python 3 (just as Guido intended, no doubt :) The one stumbling block in the past has been package availability for project work, but I don't expect that to a problem this year. All of the web and GUI development work that they'll be doing with me, for example, will be done entirely in Python 3. Nick [View Less]

2 1

Re: [Python-Dev] peps: document requirements
by Antoine Pitrou Sept. 30, 2013

Sept. 30, 2013

On Mon, 30 Sep 2013 17:08:00 +0200 (CEST) christian.heimes <python-checkins(a)python.org> wrote: > + > +* It should return ``0`` for zero length input. (Note: This can be handled as > + special case, too.) What is this required for? The main requirement is that equal stuff hashes equal, but there's no reason for zero-length input to hash to zero, AFAIK. Regards Antoine.

2 1

Getting Tulip (PEP 3156) into the 3.4 stdlib, marked provisional, named asyncio
by Guido van Rossum Sept. 30, 2013

Sept. 30, 2013

I've been looking at my progress with Tulip and the 3.4 release schedule (PEP 429) and it looks like I will have to do some kind of sprint to get it into the release in time for beta 1, which is planned for Nov 24. Ideally I'd get it into alpha 4, which is scheduled for Oct 20 -- that's in about three weeks, and probably too tight. Even Nov 24 is aggressive, because the PEP (PEP 3156) hasn't even been discussed formally, let alone accepted! Fortunately I'm pretty happy with most of the APIs … [View More]

10 17

PEP 453 (pip bootstrapping) ready for pronouncement?
by Nick Coghlan Sept. 30, 2013

Sept. 30, 2013

With the last round of updates, I believe PEP 453 is ready for Martin's pronouncement. HTML: http://www.python.org/dev/peps/pep-0453/ Major diffs: http://hg.python.org/peps/rev/b2993450b32a Many of the updates are just clarifications in response to questions, but the actual significant changes are: - the proposed module name is now "ensurepip" (thanks Antoine for that suggestion) - distros are granted considerable latitude to ensure the distro pip package is used with the system Python … [View More]installation, so long as they ensure virtual environments continue to work as expected - the two open questions regarding uninstallation and tweaking the directory layout on Windows are decided in favour of the approaches that were already documented in the PEP (since we hadn't received any objections to them in earlier discussions) Regards, Nick. ===================== PEP: 453 Title: Explicit bootstrapping of pip in Python installations Version: $Revision$ Last-Modified: $Date$ Author: Donald Stufft <donald(a)stufft.io>, Nick Coghlan <ncoghlan(a)gmail.com> BDFL-Delegate: Martin von Löwis Status: Draft Type: Process Content-Type: text/x-rst Created: 10-Aug-2013 Post-History: 30-Aug-2013, 15-Sep-2013, 18-Sep-2013, 19-Sep-2013, 23-Sep-2013 Abstract ======== This PEP proposes that the `pip`_ package manager be made available by default when installing CPython and when creating virtual environments using the standard library's ``venv`` module via the ``pyvenv`` command line utility). To clearly demarcate development responsibilities, and to avoid inadvertently downgrading ``pip`` when updating CPython, the proposed mechanism to achieve this is to include an explicit `pip`_ bootstrapping mechanism in the standard library that is invoked automatically by the CPython installers provided on python.org. The PEP also strongly recommends that CPython redistributors and other Python implementations ensure that ``pip`` is available by default, or at the very least, explicitly document the fact that it is not included. Proposal ======== This PEP proposes the inclusion of an ``ensurepip`` bootstrapping module in Python 3.4, as well as in the next maintenance releases of Python 3.3 and 2.7. This PEP does *not* propose making pip (or any dependencies) directly available as part of the standard library. Instead, pip will be a bundled application provided along with CPython for the convenience of Python users, but subject to its own development life cycle and able to be upgraded independently of the core interpreter and standard library. Rationale ========= Currently, on systems without a platform package manager and repository, installing a third-party Python package into a freshly installed Python requires first identifying an appropriate package manager and then installing it. Even on systems that *do* have a platform package manager, it is unlikely to include every package that is available on the Python Package Index, and even when a desired third-party package is available, the correct name in the platform package manager may not be clear. This means that, to work effectively with the Python Package Index ecosystem, users must know which package manager to install, where to get it, and how to install it. The effect of this is that third-party Python projects are currently required to choose from a variety of undesirable alternatives: * Assume the user already has a suitable cross-platform package manager installed. * Duplicate the instructions and tell their users how to install the package manager. * Completely forgo the use of dependencies to ease installation concerns for their users. All of these available options have significant drawbacks. If a project simply assumes a user already has the tooling then beginning users may get a confusing error message when the installation command doesn't work. Some operating systems may ease this pain by providing a global hook that looks for commands that don't exist and suggest an OS package they can install to make the command work, but that only works on systems with platform package managers (such as major Linux distributions). No such assistance is available for Windows and Mac OS X users. The challenges of dealing with this problem are a regular feature of feedback the core Python developers receive from professional educators and others introducing new users to Python. If a project chooses to duplicate the installation instructions and tell their users how to install the package manager before telling them how to install their own project then whenever these instructions need updates they need updating by every project that has duplicated them. This is particular problematic when there are multiple competing installation tools available, and different projects recommend different tools. This specific problem can be partially alleviated by strongly promoting ``pip`` as the default installer and recommending that other projects reference `pip's own bootstrapping instructions <http://www.pip-installer.org/en/latest/installing.html>`__ rather than duplicating them. However the user experience created by this approach still isn't good (especially on Windows, where downloading and running the ``get-pip.py`` bootstrap script with the default OS configuration is significantly more painful than downloading and running a binary executable or installer). The situation becomes even more complicated when multiple Python versions are involved (for example, parallel installations of Python 2 and Python 3), since that makes it harder to create and maintain good platform specific ``pip`` installers independently of the CPython installers. The projects that have decided to forgo dependencies altogether are forced to either duplicate the efforts of other projects by inventing their own solutions to problems or are required to simply include the other projects in their own source trees. Both of these options present their own problems either in duplicating maintenance work across the ecosystem or potentially leaving users vulnerable to security issues because the included code or duplicated efforts are not automatically updated when upstream releases a new version. By providing a cross-platform package manager by default it will be easier for users trying to install these third-party packages as well as easier for the people distributing them as they should now be able to safely assume that most users will have the appropriate installation tools available. This is expected to become more important in the future as the Wheel_ package format (deliberately) does not have a built in "installer" in the form of ``setup.py`` so users wishing to install from a wheel file will want an installer even in the simplest cases. Reducing the burden of actually installing a third-party package should also decrease the pressure to add every useful module to the standard library. This will allow additions to the standard library to focus more on why Python should have a particular tool out of the box, and why it is reasonable for that package to adopt the standard library's 18-24 month feature release cycle, instead of using the general difficulty of installing third-party packages as justification for inclusion. Providing a standard installation system also helps with bootstrapping alternate build and installer systems, such as ``setuptools``, ``zc.buildout`` and the ``hashdist``/``conda`` combination that is aimed specifically at the scientific community. So long as ``pip install <tool>`` works, then a standard Python-specific installer provides a reasonably secure, cross platform mechanism to get access to these utilities. Why pip? -------- ``pip`` has been chosen as the preferred default installer, as it addresses several design and user experience issues with its predecessor ``easy_install`` (these issues can't readily be fixed in ``easy_install`` itself due to backwards compatibility concerns). ``pip`` is also well suited to working within the bounds of a single Python runtime installation (including associated virtual environments), which is a desirable feature for a tool bundled with CPython. Other tools like ``zc.buildout`` and ``conda`` are more ambitious in their aims (and hence substantially better than ``pip`` at handling external binary dependencies), so it makes sense for the Python ecosystem to treat them more like platform package managers to inter operate with rather than as the default cross-platform installation tool. This relationship is similar to that between ``pip`` and platform package management systems like ``apt`` and ``yum`` (which are also designed to handle arbitrary binary dependencies). Explicit bootstrapping mechanism ================================ An additional module called ``ensurepip`` will be added to the standard library whose purpose is to install pip and any of its dependencies into the appropriate location (most commonly site-packages). It will expose a callable named ``bootstrap()`` as well as offer direct execution via ``python -m ensurepip``. The bootstrap will *not* contact PyPI, but instead rely on a private copy of pip stored inside the standard library. Accordingly, only options related to the installation location will be supported (``--user``, ``--root``, etc). It is considered desirable that users be strongly encouraged to use the latest available version of ``pip``, in order to take advantage of the ongoing efforts to improve the security of the PyPI based ecosystem, as well as benefiting from the efforts to improve the speed, reliability and flexibility of that ecosystem. In order to satisfy this goal of providing the most recent version of ``pip`` by default, the private copy of ``pip`` will be updated in CPython maintenance releases, which should align well with the 6-month cycle used for new ``pip`` releases. Security considerations ----------------------- The design in this PEP has been deliberately chosen to avoid making any significant changes to the trust model of the CPython installers for end users that do not subsequently make use of ``pip``. The installers will contain all the components of a fully functioning version of Python, including the ``pip`` installer. The installation process will *not* require network access, and will *not* rely on trusting the security of the network connection established between ``pip`` and the Python package index. Only users that choose to use ``pip`` directly will need to pay attention to any PyPI related security considerations. Implementation strategy ----------------------- To ensure there is no need for network access when installing Python or creating virtual environments, the ``ensurepip`` module will, as an implementation detail, include a complete private copy of pip and its dependencies which will be used to extract pip and install it into the target environment. It is important to stress that this private copy of pip is *only* an implementation detail and it should *not* be relied on or assumed to exist beyond the public capabilities exposed through the ``ensurepip`` module (and indirectly through ``venv``). There is not yet a reference ``ensurepip`` implementation. The existing ``get-pip.py`` bootstrap script demonstrates an earlier variation of the general concept, but the standard library version would take advantage of the improved distribution capabilities offered by the CPython installers to include private copies of ``pip`` and ``setuptools`` as wheel files (rather than as embedded base64 encoded data), and would not try to contact PyPI (instead installing directly from the private wheel files. Rather than including separate code to handle the bootstrapping, the ``ensurepip`` module will manipulate sys.path appropriately to allow the wheel files to be used to install themselves, either into the current Python installation or into a virtual environment (as determined by the options passed to the bootstrap command). It is proposed that the implementation be carried out in five separate steps (all steps after the first are independent of each other and can be carried out in any order): * the first step would add the ``ensurepip`` module and the private copies of the most recently released versions of pip and setuptools, and update the "Installing Python Modules" documentation. This change would be applied to Python 2.7, 3.3 and 3.4. * the Windows installer would be updated to offer the new ``pip`` installation option for Python 2.7.6, 3.3.3 and 3.4.0. * the Mac OS X installer would be updated to offer the new ``pip`` installation option for Python 2.7.6, 3.3.3 and 3.4.0. * the ``venv`` module and ``pyvenv`` command would be updated to make use of ``ensurepip`` in Python 3.4+ * the PATH handling and ``sysconfig`` directory layout on Windows would be updated for Python 3.4+ Proposed CLI ------------ The proposed CLI is based on a subset of the existing ``pip install`` options:: Usage: python -m ensurepip [options] General Options: -h, --help Show help. -v, --verbose Give more output. Option is additive, and can be used up to 3 times. -V, --version Show the pip version that would be extracted and exit. -q, --quiet Give less output. Installation Options: -U, --upgrade Upgrade pip and dependencies, even if already installed --user Install using the user scheme. --root <dir> Install everything relative to this alternate root directory. In most cases, end users won't need to use this CLI directly, as ``pip`` should have been installed automatically when installing Python or when creating a virtual environment. Users that want to retrieve the latest version from PyPI, or otherwise need more flexibility, should invoke the extracted ``pip`` appropriately. Proposed module API ------------------- The proposed ``ensurepip`` module API consists of the following two functions:: def version(): """ Returns a string specifying the bundled version of pip. """ def bootstrap(root=None, upgrade=False, user=False, verbosity=0): """ Bootstrap pip into the current Python installation (or the given root directory). """ Invocation from the CPython installers -------------------------------------- The CPython Windows and Mac OS X installers will each gain a new option: * Install pip (the default Python package management utility)? This option will be checked by default. If the option is checked, then the installer will invoke the following command with the just installed Python:: python -m ensurepip --upgrade This ensures that, by default, installing or updating CPython will ensure that the installed version of pip is at least as recent as the one included with that version of CPython. If a newer version of pip has already been installed then ``python -m ensurepip --upgrade`` will simply return without doing anything. Installing from source ---------------------- While the prebuilt binary installers will be updated to run ``python -m ensurepip`` by default, no such change will be made to the ``make install`` and ``make altinstall`` commands of the source distribution. ``ensurepip`` itself (including the private copy of ``pip`` and its dependencies) will still be installed normally (as it is a regular part of the standard library), only the implicit installation of pip and its dependencies will be skipped. Keeping the pip bootstrapping as a separate step for ``make``-based installations should minimize the changes CPython redistributors need to make to their build processes. Avoiding the layer of indirection through ``make`` for the ``ensurepip`` invocation avoids any challenges associated with determining where to install the extracted ``pip``. Changes to virtual environments ------------------------------- Python 3.3 included a standard library approach to virtual Python environments through the ``venv`` module. Since its release it has become clear that very few users have been willing to use this feature directly, in part due to the lack of an installer present by default inside of the virtual environment. They have instead opted to continue using the ``virtualenv`` package which *does* include pip installed by default. To make the ``venv`` more useful to users it will be modified to issue the pip bootstrap by default inside of the new environment while creating it. This will allow people the same convenience inside of the virtual environment as this PEP provides outside of it as well as bringing the ``venv`` module closer to feature parity with the external ``virtualenv`` package, making it a more suitable replacement. To handle cases where a user does not wish to have pip bootstrapped into their virtual environment a ``--without-pip`` option will be added. The ``venv.EnvBuilder`` and ``venv.create`` APIs will be updated to accept one new parameter: ``with_pip`` (defaulting to ``False``). The new default for the module API is chosen for backwards compatibility with the current behaviour (as it is assumed that most invocation of the ``venv`` module happens through third part tools that likely will not want ``pip`` installed without explicitly requesting it), while the default for the command line interface is chosen to try to ensure ``pip`` is available in most virtual environments without additional action on the part of the end user. This particular change will be made only for Python 3.4 and later versions. The third-party ``virtualenv`` project will still be needed to obtain a consistent cross-version experience in Python 3.3 and 2.7. Documentation ------------- The "Installing Python Modules" section of the standard library documentation will be updated to recommend the use of the bootstrapped `pip` installer. It will give a brief description of the most common commands and options, but delegate to the externally maintained ``pip`` documentation for the full details. The existing content of the module installation guide will be retained, but under a new "Invoking distutils directly" subsection. Bundling CA certificates with CPython ------------------------------------- The ``ensurepip`` implementation will include the ``pip`` CA bundle along with the rest of ``pip``. This means CPython effectively includes a CA bundle that is used solely by ``pip`` after it has been extracted. This is considered preferable to relying solely on the system certificate stores, as it ensures that ``pip`` will behave the same across all supported versions of Python, even those prior to Python 3.4 that cannot access the system certificate store on Windows. Automatic installation of setuptools ------------------------------------ ``pip`` currently depends on ``setuptools`` to handle metadata generation during the build process, along with some other features. While work is ongoing to reduce or eliminate this dependency, it is not clear if that work will be complete for pip 1.5 (which is the version likely to be current when Python 3.4.0 is released). This PEP proposes that, if pip still requires it as a dependency, ``ensurepip`` will include a private copy of ``setuptools`` (in addition to the private copy of ``ensurepip``). ``python -m ensurepip`` will then install the private copy in addition to installing ``pip`` itself. However, this behavior is officially considered an implementation detail. Other projects which explicitly require ``setuptools`` must still provide an appropriate dependency declaration, rather than assuming ``setuptools`` will always be installed alongside ``pip``. Once pip is able to run ``pip install --upgrade pip`` without needing ``setuptools`` installed first, then the private copy of ``setuptools`` will be removed from ``ensurepip`` in subsequent CPython releases. Updating the private copy of pip -------------------------------- In order to keep up with evolutions in packaging as well as providing users with as recent version a possible the ``ensurepip`` module will be regularly updated to the latest versions of everything it bootstraps. After each new ``pip`` release, and again during the preparation for any release of Python (including feature releases), a script, provided as part of this PEP, will be run to ensure the private copies stored in the CPython source repository have been updated to the latest versions. Updating the ensurepip module API and CLI ----------------------------------------- Like ``venv`` and ``pyvenv``, the ``ensurepip`` module API and CLI will be governed by the normal rules for the standard library: no new features are permitted in maintenance releases. However, the embedded components may be updated as noted above, so the extracted ``pip`` may offer additional functionality in maintenance releases. Feature addition in maintenance releases ======================================== Adding a new module to the standard library in Python 2.7 and 3.3 maintenance releases breaks the usual policy of "no new features in maintenance releases". It is being proposed in this case as the current bootstrapping issues for the third-party Python package ecosystem greatly affects the experience of new users, especially on Python 2 where many Python 3 standard library improvements are available as backports on PyPI, but are not included in the Python 2 standard library. By updating Python 2.7, 3.3 and 3.4 to easily bootstrap the PyPI ecosystem, this change should aid the vast majority of current Python users, rather than only those with the freedom to adopt Python 3.4 as soon as it is released. Uninstallation ============== No changes are proposed to the uninstallation process by this PEP. The bootstrapped pip will be installed the same way as any other pip installed packages, and will be handled in the same way as any other post-install additions to the Python environment. At least on Windows, that means the bootstrapped files will be left behind after uninstallation, since those files won't be associated with the Python MSI installer. While the case can be made for the CPython installers clearing out these directories automatically, changing that behaviour is considered outside the scope of this PEP. Script Execution on Windows =========================== While the Windows installer was updated in Python 3.3 to optionally make ``python`` available on the PATH, no such change was made to include the Scripts directory. Independently of this PEP, a proposal has also been made to rename the ``Tools\Scripts`` subdirectory to ``bin`` in order to improve consistency with the typical script installation directory names on \*nix systems. Accordingly, in addition to adding the option to extract and install ``pip`` during installation, this PEP proposes that the Windows installer (and ``sysconfig``) in Python 3.4 and later be updated to: - install scripts to PythonXY\bin rather than PythonXY\Tools\Scripts - add PythonXY\bin to the Windows PATH (in addition to PythonXY) when the PATH modification option is enabled during installation For Python 2.7 and 3.3, it is proposed that the only change be the one to bootstrap ``pip`` by default. This means that, for Python 3.3, the most reliable way to invoke pip on Windows (without tinkering manually with PATH) will actually be ``py -m pip`` (or ``py -3 -m pip`` to select the Python 3 version if both Python 2 and 3 are installed) rather than simply calling ``pip``. For Python 2.7 and 3.2, the most reliable mechanism will be to install the standalone Python launcher for Windows and then use ``py -m pip`` as noted above. Adding the scripts directory to the system PATH would mean that ``pip`` works reliably in the "only one Python installation on the system PATH" case, with ``py -m pip``, ``pipX``, or ``pipX.Y`` needed only to select a non-default version in the parallel installation case (and outside a virtual environment). This change should also make the ``pyvenv`` command substantially easier to invoke on Windows, along with all scripts installed by ``pip``, ``easy_install`` and similar tools. While the script invocations on recent versions of Python will run through the Python launcher for Windows, this shouldn't cause any issues, as long as the Python files in the Scripts directory correctly specify a Python version in their shebang line or have an adjacent Windows executable (as ``easy_install`` and ``pip`` do). Recommendations for Downstream Distributors =========================================== A common source of Python installations are through downstream distributors such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX package managers [#homebrew]_, or Python-specific tools [#conda]_. In order to provide a consistent, user-friendly experience to all users of Python regardless of how they attained Python this PEP recommends and asks that downstream distributors: * Ensure that whenever Python is installed pip is also installed. * This may take the form of separate packages with dependencies on each other so that installing the Python package installs the pip package and installing the pip package installs the Python package. * Another reasonable way to implement this is to package pip separately but ensure that there is some sort of global hook that will recommend installing the separate pip package when a user executes ``pip`` without it being installed. Systems that choose this option should ensure that the ``pyvenv`` command still installs pip into the virtual environment by default, but may modify the ``ensurepip`` module in the system Python installation to redirect to the platform provided mechanism when installing ``pip`` globally. * Do not remove the bundled copy of pip. * This is required for installation of pip into a virtual environment by the ``venv`` module. * This is similar to the existing ``virtualenv`` package for which many downstream distributors have already made exception to the common "debundling" policy. * This does mean that if ``pip`` needs to be updated due to a security issue, so does the private copy in the ``ensurepip`` bootstrap module * However, altering the private copy of pip to remove the embedded CA certificate bundle and rely on the system CA bundle instead is a reasonable change. * Migrate build systems to utilize `pip`_ and `Wheel`_ instead of directly using ``setup.py``. * This will ensure that downstream packages can more easily utilize the new metadata formats which may not have a ``setup.py``. * Ensure that all features of this PEP continue to work with any modifications made to the redistributed version of Python. * Checking the version of pip that will be bootstrapped using ``python -m ensurepip --version`` or ``ensurepip.version()``. * Installation of pip into a global or virtual python environment using ``python -m ensurepip`` or ``ensurepip.bootstrap()``. * ``pip install --upgrade pip`` in a global installation should not affect any already created virtual environments (but is permitted to affect future virtual environments, even though it will not do so when using the upstream version of ``ensurepip``). * ``pip install --upgrade pip`` in a virtual environment should not affect the global installation. In the event that a Python redistributor chooses *not* to follow these recommendations, we request that they explicitly document this fact and provide their users with suitable guidance on translating upstream ``pip`` based installation instructions into something appropriate for the platform. Other Python implementations are also encouraged to follow these guidelines where applicable. Policies & Governance ===================== The maintainers of the bootstrapped software and the CPython core team will work together in order to address the needs of both. The bootstrapped software will still remain external to CPython and this PEP does not include CPython subsuming the development responsibilities or design decisions of the bootstrapped software. This PEP aims to decrease the burden on end users wanting to use third-party packages and the decisions inside it are pragmatic ones that represent the trust that the Python community has already placed in the Python Packaging Authority as the authors and maintainers of ``pip``, ``setuptools``, PyPI, ``virtualenv`` and other related projects. Backwards Compatibility ----------------------- The public API and CLI of the ``ensurepip`` module itself will fall under the typical backwards compatibility policy of Python for its standard library. The externally developed software that this PEP bundles does not. Most importantly, this means that the bootstrapped version of pip may gain new features in CPython maintenance releases, and pip continues to operate on its own 6 month release cycle rather than CPython's 18-24 month cycle. Security Releases ----------------- Any security update that affects the ``ensurepip`` module will be shared prior to release with the Python Security Response Team (security(a)python.org). The PSRT will then decide if the reported issue warrants a security release of CPython with an updated private copy of ``pip``. Appendix: Rejected Proposals ============================ Automatically contacting PyPI when bootstrapping pip ---------------------------------------------------- Earlier versions of this PEP called the bootstrapping module ``getpip`` and defaulted to downloading and installing ``pip`` from PyPI, with the private copy used only as a fallback option or when explicitly requested. This resulted in several complex edge cases, along with difficulties in defining a clean API and CLI for the bootstrap module. It also significantly altered the default trust model for the binary installers published on python.org, as end users would need to explicitly *opt-out* of trusting the security of the PyPI ecosystem (rather than opting in to it by explicitly invoking ``pip`` following installation). As a result, the PEP was simplified to the current design, where the bootstrapping *always* uses the private copy of ``pip``. Contacting PyPI is now always an explicit separate step, with direct access to the full pip interface. Implicit bootstrap ------------------ `PEP439`_, the predecessor for this PEP, proposes its own solution. Its solution involves shipping a fake ``pip`` command that when executed would implicitly bootstrap and install pip if it does not already exist. This has been rejected because it is too "magical". It hides from the end user when exactly the pip command will be installed or that it is being installed at all. It also does not provide any recommendations or considerations towards downstream packagers who wish to manage the globally installed pip through the mechanisms typical for their system. The implicit bootstrap mechanism also ran into possible permissions issues, if a user inadvertently attempted to bootstrap pip without write access to the appropriate installation directories. Including pip directly in the standard library ---------------------------------------------- Similar to this PEP is the proposal of just including pip in the standard library. This would ensure that Python always includes pip and fixes all of the end user facing problems with not having pip present by default. This has been rejected because we've learned, through the inclusion and history of ``distutils`` in the standard library, that losing the ability to update the packaging tools independently can leave the tooling in a state of constant limbo. Making it unable to ever reasonably evolve in a time frame that actually affects users as any new features will not be available to the general population for *years*. Allowing the packaging tools to progress separately from the Python release and adoption schedules allows the improvements to be used by *all* members of the Python community and not just those able to live on the bleeding edge of Python releases. There have also been issues in the past with the "dual maintenance" problem if a project continues to be maintained externally while *also* having a fork maintained in the standard library. Since external maintenance of ``pip`` will always be needed to support earlier Python versions, the proposed bootstrapping mechanism will becoming the explicit responsibility of the CPython core developers (assisted by the pip developers), while pip issues reported to the CPython tracker will be migrated to the pip issue tracker. There will no doubt still be some user confusion over which tracker to use, but hopefully less than has been seen historically when including complete public copies of third-party projects in the standard library. The approach described in this PEP also avoids some technical issues related to handling CPython maintenance updates when pip has been independently updated to a more recent version. The proposed pip-based bootstrapping mechanism handles that automatically, since pip and the system installer never get into a fight about who owns the pip installation (it is always managed through pip, either directly, or indirectly via the ``ensurepip`` bootstrap module). Finally, the separate bootstrapping step means it also easy to avoid installing ``pip`` at all if end users so desire. This is often the case if integrators are using system packages to handle installation of components written in multiple languages using a common set of tools. Defaulting to --user installation --------------------------------- Some consideration was given to bootstrapping pip into the per-user site-packages directory by default. However, this behavior would be surprising (as it differs from the default behavior of pip itself) and is also not currently considered reliable (there are some edge cases which are not handled correctly when pip is installed into the user site-packages directory rather than the system site-packages). .. _Wheel: http://www.python.org/dev/peps/pep-0427/ .. _pip: http://www.pip-installer.org .. _setuptools: https://pypi.python.org/pypi/setuptools .. _PEP439: http://www.python.org/dev/peps/pep-0439/ References ========== .. [1] Discussion thread 1 (distutils-sig) (https://mail.python.org/pipermail/distutils-sig/2013-August/022529.html) .. [2] Discussion thread 2 (distutils-sig) (https://mail.python.org/pipermail/distutils-sig/2013-September/022702.html) .. [3] Discussion thread 3 (python-dev) (https://mail.python.org/pipermail/python-dev/2013-September/128723.html) .. [4] Discussion thread 4 (python-dev) (https://mail.python.org/pipermail/python-dev/2013-September/128780.html) .. [#ubuntu] `Ubuntu <http://www.ubuntu.com/>` .. [#debian] `Debian <http://www.debian.org>` .. [#fedora] `Fedora <https://fedoraproject.org/>` .. [#homebrew] `Homebrew <http://brew.sh/>` .. [#conda] `Conda <http://www.continuum.io/blog/conda>` Copyright ========= This document has been placed in the public domain. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia [View Less]

20 68

Re: [Python-Dev] Why not support user defined operator overloading ?
by 张佩佩 Sept. 30, 2013

Sept. 30, 2013

On 2013/9/30 8:53 Greg Ewing wrote: > It does need to know the operator's precedence and >associativity, though, which means either declaring >it somewhere, or having some kind of fixed rule I suggest all user defined operator are at lowest priority. Regards peipei

3 2

Re: [Python-Dev] cpython (merge 3.3 -> default): merge from 3.3
by Antoine Pitrou Sept. 30, 2013

Sept. 30, 2013

On Thu, 12 Sep 2013 07:57:25 +0200 (CEST) senthil.kumaran <python-checkins(a)python.org> wrote: > > +<<<<<<< local > Optional argument random is a 0-argument function returning a > random float in [0.0, 1.0); if it is the default None, the > standard random.random will be used. > +======= > + Optional arg random is a 0-argument function returning a random > + float in [0.0, 1.0); by default, the … [View More]

2 1

Re: [Python-Dev] [Python-checkins] cpython (merge 3.3 -> default): Fix http.server's request handling case on trailing '/'.
by Eric V. Smith Sept. 30, 2013

Sept. 30, 2013

On 9/13/2013 3:22 AM, senthil.kumaran wrote: > http://hg.python.org/cpython/rev/b85c9d2a5227 > changeset: 85672:b85c9d2a5227 > parent: 85668:66ec8431032d > parent: 85671:1fcccbbe15e2 > user: Senthil Kumaran <senthil(a)uthcode.com> > date: Fri Sep 13 00:22:45 2013 -0700 > summary: > Fix http.server's request handling case on trailing '/'. > > Patch contributed by Vajrasky Kok. Addresses Issue #17324 > + trailing_slash = … [View More]

2 1

Why not support user defined operator overloading ?
by 张佩佩 Sept. 30, 2013

Sept. 30, 2013

Hello: As far as I know, there is not a language support user defined operator overloading. Python3 can overloading belowed operators. - negated + unchanged - minus + add * multiplication / division // true division % remainder ** power (Do I miss something ?) If we can overloading these operators, why we can't overloading other operators? (like .* often used in matrix, U in set operation) Regards! Peipei

6 6

Completed: Google Summer of Code 2013 IDLE Unit Testing and Improvements
by Todd V. Rovito Sept. 30, 2013

Sept. 30, 2013

In April 2013 we challenged students to not spend their summer idle around the swimming pool but rather spend their time on IDLE and make a difference. IDLE is Python’s Integrated Development Environment (IDE) that is shipped with each Python release (http://en.wikipedia.org/wiki/IDLE_(Python)). Since IDLE ships with Python it is often the first IDE a new Python programmer uses. The Python development community wants to make IDLE an awesome experience especially for people that are learning … [View More]

1 0

Re: [Python-Dev] Why not support user defined operator overloading ?
by 张佩佩 Sept. 30, 2013

Sept. 30, 2013

Hello: I agree with Steven D'Aprano. Here is an example: class A(object): def __init__(self, value): self.value = value def add(self, other): return self.value + other.value __magic_method__ = {'+':add} a1 = A(1) a2 = A(2) We only need a macro expand a1 + a2 to a1.__magic__method__['+'](a, b) the later can be execute on Python. Regards peipei

1 0

[RELEASED] Python 3.4.0a3
by Larry Hastings Sept. 30, 2013

Sept. 30, 2013

On behalf of the Python development team, I'm pleased to announce the third alpha release of Python 3.4. This is a preview release, and its use is not recommended for production settings. Python 3.4 includes a range of improvements of the 3.x series, including hundreds of small improvements and bug fixes. Major new features and changes in the 3.4 release series so far include: * PEP 435, a standardized "enum" module * PEP 442, improved semantics for object finalization * PEP 443, adding … [View More]

1 0

2.6.9rc1
by Barry Warsaw Sept. 29, 2013

Sept. 29, 2013

Just a quick reminder that I intend to cut 2.6.9rc1 tomorrow. Here are the open issues still on my list: - 16040 - nntplib: unlimited readline() from connection This one is waiting for a patch, but TBH if it doesn't make it into 2.6.9 I won't care too much. I also don't mind reviewing and applying a patch between 2.6.9rc1 and 2.6.9 final. - 16041 - poplib: unlimited readline() from connection This one has a patch for 2.7 which does not apply cleanly to 2.6. I'm of a similar … [View More]

2 1

PEP 453 updated with improved rationale
by Nick Coghlan Sept. 29, 2013

Sept. 29, 2013

PEP 453 has been updated in response to the previous discussion thread. Updated version is online at http://www.python.org/dev/peps/pep-0453/ Relevant diffs can be found at: http://hg.python.org/peps/log/bdd50f92ee73/pep-0453.txt The bulk of the updates can be seen in the following two diffs: http://hg.python.org/peps/rev/58f33b8eb68d http://hg.python.org/peps/rev/438e3fa95c88 The underlying technical proposal itself hasn't actually changed from the previous iteration: we still … [View More]

1 0

Best practice for documentation for std lib
by Steven D'Aprano Sept. 28, 2013

Sept. 28, 2013

Hi all, I have a question about how I should manage documentation for the statistics module for Python 3.4. At the moment, I have extensive docstrings in the module itself. I don't believe anyone has flagged that as "too much information" in a code review, so I'm going to assume that large docstrings will be acceptable. However, I have been asked to ensure that there is a separate statistics.rst file for documentation. I don't want to have to maintain the documentation in two places, … [View More]

26 61

toolbar
by Kevin Ngugi Sept. 28, 2013

Sept. 28, 2013

Hi, I just downloaded Python 3.3 top teach myself how to program, I am new to programming, but the guide I am using requires me to access the toolbar, which I cannot seem to find. How do I find it and have it displayed on the interface? I tried v 3.1 but its still absent. Thank you.

4 3

PEP 453 (pip bootstrap) TL;DR and Usage Scenario
by anatoly techtonik Sept. 28, 2013

Sept. 28, 2013

On Mon, Sep 23, 2013 at 2:15 PM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > With the last round of updates, I believe PEP 453 is ready for > Martin's pronouncement. > > HTML: http://www.python.org/dev/peps/pep-0453/ > Major diffs: http://hg.python.org/peps/rev/b2993450b32a I'd enjoy concise PEP texts, but it is extremely hard to write one, so is it possible to at least add a TL;DR chapter that just shows the concept in 15 seconds without all the gory details? What is the … [View More]

1 0