Mailman 3 September 2009 - Python-Dev

PEP 1, PEP Purpose and Guidelines
by barry＠zope.com May 18, 2021

May 18, 2021

It has been a while since I posted a copy of PEP 1 to the mailing lists and newsgroups. I've recently done some updating of a few sections, so in the interest of gaining wider community participation in the Python development process, I'm posting the latest revision of PEP 1 here. A version of the PEP is always available on-line at http://www.python.org/peps/pep-0001.html Enjoy, -Barry -------------------- snip snip -------------------- PEP: 1 Title: PEP Purpose and Guidelines Version: … [View More]$Revision: 1.36 $ Last-Modified: $Date: 2002/07/29 18:34:59 $ Author: Barry A. Warsaw, Jeremy Hylton Status: Active Type: Informational Created: 13-Jun-2000 Post-History: 21-Mar-2001, 29-Jul-2002 What is a PEP? PEP stands for Python Enhancement Proposal. A PEP is a design document providing information to the Python community, or describing a new feature for Python. The PEP should provide a concise technical specification of the feature and a rationale for the feature. We intend PEPs to be the primary mechanisms for proposing new features, for collecting community input on an issue, and for documenting the design decisions that have gone into Python. The PEP author is responsible for building consensus within the community and documenting dissenting opinions. Because the PEPs are maintained as plain text files under CVS control, their revision history is the historical record of the feature proposal[1]. Kinds of PEPs There are two kinds of PEPs. A standards track PEP describes a new feature or implementation for Python. An informational PEP describes a Python design issue, or provides general guidelines or information to the Python community, but does not propose a new feature. Informational PEPs do not necessarily represent a Python community consensus or recommendation, so users and implementors are free to ignore informational PEPs or follow their advice. PEP Work Flow The PEP editor, Barry Warsaw <peps(a)python.org>, assigns numbers for each PEP and changes its status. The PEP process begins with a new idea for Python. It is highly recommended that a single PEP contain a single key proposal or new idea. The more focussed the PEP, the more successfully it tends to be. The PEP editor reserves the right to reject PEP proposals if they appear too unfocussed or too broad. If in doubt, split your PEP into several well-focussed ones. Each PEP must have a champion -- someone who writes the PEP using the style and format described below, shepherds the discussions in the appropriate forums, and attempts to build community consensus around the idea. The PEP champion (a.k.a. Author) should first attempt to ascertain whether the idea is PEP-able. Small enhancements or patches often don't need a PEP and can be injected into the Python development work flow with a patch submission to the SourceForge patch manager[2] or feature request tracker[3]. The PEP champion then emails the PEP editor <peps(a)python.org> with a proposed title and a rough, but fleshed out, draft of the PEP. This draft must be written in PEP style as described below. If the PEP editor approves, he will assign the PEP a number, label it as standards track or informational, give it status 'draft', and create and check-in the initial draft of the PEP. The PEP editor will not unreasonably deny a PEP. Reasons for denying PEP status include duplication of effort, being technically unsound, not providing proper motivation or addressing backwards compatibility, or not in keeping with the Python philosophy. The BDFL (Benevolent Dictator for Life, Guido van Rossum) can be consulted during the approval phase, and is the final arbitrator of the draft's PEP-ability. If a pre-PEP is rejected, the author may elect to take the pre-PEP to the comp.lang.python newsgroup (a.k.a. python-list(a)python.org mailing list) to help flesh it out, gain feedback and consensus from the community at large, and improve the PEP for re-submission. The author of the PEP is then responsible for posting the PEP to the community forums, and marshaling community support for it. As updates are necessary, the PEP author can check in new versions if they have CVS commit permissions, or can email new PEP versions to the PEP editor for committing. Standards track PEPs consists of two parts, a design document and a reference implementation. The PEP should be reviewed and accepted before a reference implementation is begun, unless a reference implementation will aid people in studying the PEP. Standards Track PEPs must include an implementation - in the form of code, patch, or URL to same - before it can be considered Final. PEP authors are responsible for collecting community feedback on a PEP before submitting it for review. A PEP that has not been discussed on python-list(a)python.org and/or python-dev(a)python.org will not be accepted. However, wherever possible, long open-ended discussions on public mailing lists should be avoided. Strategies to keep the discussions efficient include, setting up a separate SIG mailing list for the topic, having the PEP author accept private comments in the early design phases, etc. PEP authors should use their discretion here. Once the authors have completed a PEP, they must inform the PEP editor that it is ready for review. PEPs are reviewed by the BDFL and his chosen consultants, who may accept or reject a PEP or send it back to the author(s) for revision. Once a PEP has been accepted, the reference implementation must be completed. When the reference implementation is complete and accepted by the BDFL, the status will be changed to `Final.' A PEP can also be assigned status `Deferred.' The PEP author or editor can assign the PEP this status when no progress is being made on the PEP. Once a PEP is deferred, the PEP editor can re-assign it to draft status. A PEP can also be `Rejected'. Perhaps after all is said and done it was not a good idea. It is still important to have a record of this fact. PEPs can also be replaced by a different PEP, rendering the original obsolete. This is intended for Informational PEPs, where version 2 of an API can replace version 1. PEP work flow is as follows: Draft -> Accepted -> Final -> Replaced ^ +----> Rejected v Deferred Some informational PEPs may also have a status of `Active' if they are never meant to be completed. E.g. PEP 1. What belongs in a successful PEP? Each PEP should have the following parts: 1. Preamble -- RFC822 style headers containing meta-data about the PEP, including the PEP number, a short descriptive title (limited to a maximum of 44 characters), the names, and optionally the contact info for each author, etc. 2. Abstract -- a short (~200 word) description of the technical issue being addressed. 3. Copyright/public domain -- Each PEP must either be explicitly labelled as placed in the public domain (see this PEP as an example) or licensed under the Open Publication License[4]. 4. Specification -- The technical specification should describe the syntax and semantics of any new language feature. The specification should be detailed enough to allow competing, interoperable implementations for any of the current Python platforms (CPython, JPython, Python .NET). 5. Motivation -- The motivation is critical for PEPs that want to change the Python language. It should clearly explain why the existing language specification is inadequate to address the problem that the PEP solves. PEP submissions without sufficient motivation may be rejected outright. 6. Rationale -- The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work, e.g. how the feature is supported in other languages. The rationale should provide evidence of consensus within the community and discuss important objections or concerns raised during discussion. 7. Backwards Compatibility -- All PEPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. The PEP must explain how the author proposes to deal with these incompatibilities. PEP submissions without a sufficient backwards compatibility treatise may be rejected outright. 8. Reference Implementation -- The reference implementation must be completed before any PEP is given status 'Final,' but it need not be completed before the PEP is accepted. It is better to finish the specification and rationale first and reach consensus on it before writing code. The final implementation must include test code and documentation appropriate for either the Python language reference or the standard library reference. PEP Template PEPs are written in plain ASCII text, and should adhere to a rigid style. There is a Python script that parses this style and converts the plain text PEP to HTML for viewing on the web[5]. PEP 9 contains a boilerplate[7] template you can use to get started writing your PEP. Each PEP must begin with an RFC822 style header preamble. The headers must appear in the following order. Headers marked with `*' are optional and are described below. All other headers are required. PEP: <pep number> Title: <pep title> Version: <cvs version string> Last-Modified: <cvs date string> Author: <list of authors' real names and optionally, email addrs> * Discussions-To: <email address> Status: <Draft | Active | Accepted | Deferred | Final | Replaced> Type: <Informational | Standards Track> * Requires: <pep numbers> Created: <date created on, in dd-mmm-yyyy format> * Python-Version: <version number> Post-History: <dates of postings to python-list and python-dev> * Replaces: <pep number> * Replaced-By: <pep number> The Author: header lists the names and optionally, the email addresses of all the authors/owners of the PEP. The format of the author entry should be address(a)dom.ain (Random J. User) if the email address is included, and just Random J. User if the address is not given. If there are multiple authors, each should be on a separate line following RFC 822 continuation line conventions. Note that personal email addresses in PEPs will be obscured as a defense against spam harvesters. Standards track PEPs must have a Python-Version: header which indicates the version of Python that the feature will be released with. Informational PEPs do not need a Python-Version: header. While a PEP is in private discussions (usually during the initial Draft phase), a Discussions-To: header will indicate the mailing list or URL where the PEP is being discussed. No Discussions-To: header is necessary if the PEP is being discussed privately with the author, or on the python-list or python-dev email mailing lists. Note that email addresses in the Discussions-To: header will not be obscured. Created: records the date that the PEP was assigned a number, while Post-History: is used to record the dates of when new versions of the PEP are posted to python-list and/or python-dev. Both headers should be in dd-mmm-yyyy format, e.g. 14-Aug-2001. PEPs may have a Requires: header, indicating the PEP numbers that this PEP depends on. PEPs may also have a Replaced-By: header indicating that a PEP has been rendered obsolete by a later document; the value is the number of the PEP that replaces the current document. The newer PEP must have a Replaces: header containing the number of the PEP that it rendered obsolete. PEP Formatting Requirements PEP headings must begin in column zero and the initial letter of each word must be capitalized as in book titles. Acronyms should be in all capitals. The body of each section must be indented 4 spaces. Code samples inside body sections should be indented a further 4 spaces, and other indentation can be used as required to make the text readable. You must use two blank lines between the last line of a section's body and the next section heading. You must adhere to the Emacs convention of adding two spaces at the end of every sentence. You should fill your paragraphs to column 70, but under no circumstances should your lines extend past column 79. If your code samples spill over column 79, you should rewrite them. Tab characters must never appear in the document at all. A PEP should include the standard Emacs stanza included by example at the bottom of this PEP. A PEP must contain a Copyright section, and it is strongly recommended to put the PEP in the public domain. When referencing an external web page in the body of a PEP, you should include the title of the page in the text, with a footnote reference to the URL. Do not include the URL in the body text of the PEP. E.g. Refer to the Python Language web site [1] for more details. ... [1] http://www.python.org When referring to another PEP, include the PEP number in the body text, such as "PEP 1". The title may optionally appear. Add a footnote reference that includes the PEP's title and author. It may optionally include the explicit URL on a separate line, but only in the References section. Note that the pep2html.py script will calculate URLs automatically, e.g.: ... Refer to PEP 1 [7] for more information about PEP style ... References [7] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton http://www.python.org/peps/pep-0001.html If you decide to provide an explicit URL for a PEP, please use this as the URL template: http://www.python.org/peps/pep-xxxx.html PEP numbers in URLs must be padded with zeros from the left, so as to be exactly 4 characters wide, however PEP numbers in text are never padded. Reporting PEP Bugs, or Submitting PEP Updates How you report a bug, or submit a PEP update depends on several factors, such as the maturity of the PEP, the preferences of the PEP author, and the nature of your comments. For the early draft stages of the PEP, it's probably best to send your comments and changes directly to the PEP author. For more mature, or finished PEPs you may want to submit corrections to the SourceForge bug manager[6] or better yet, the SourceForge patch manager[2] so that your changes don't get lost. If the PEP author is a SF developer, assign the bug/patch to him, otherwise assign it to the PEP editor. When in doubt about where to send your changes, please check first with the PEP author and/or PEP editor. PEP authors who are also SF committers, can update the PEPs themselves by using "cvs commit" to commit their changes. Remember to also push the formatted PEP text out to the web by doing the following: % python pep2html.py -i NUM where NUM is the number of the PEP you want to push out. See % python pep2html.py --help for details. Transferring PEP Ownership It occasionally becomes necessary to transfer ownership of PEPs to a new champion. In general, we'd like to retain the original author as a co-author of the transferred PEP, but that's really up to the original author. A good reason to transfer ownership is because the original author no longer has the time or interest in updating it or following through with the PEP process, or has fallen off the face of the 'net (i.e. is unreachable or not responding to email). A bad reason to transfer ownership is because you don't agree with the direction of the PEP. We try to build consensus around a PEP, but if that's not possible, you can always submit a competing PEP. If you are interested assuming ownership of a PEP, send a message asking to take over, addressed to both the original author and the PEP editor <peps(a)python.org>. If the original author doesn't respond to email in a timely manner, the PEP editor will make a unilateral decision (it's not like such decisions can be reversed. :). References and Footnotes [1] This historical record is available by the normal CVS commands for retrieving older revisions. For those without direct access to the CVS tree, you can browse the current and past PEP revisions via the SourceForge web site at http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi/python/nondist/peps/?cvsroot=… [2] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [3] http://sourceforge.net/tracker/?atid=355470&group_id=5470&func=browse [4] http://www.opencontent.org/openpub/ [5] The script referred to here is pep2html.py, which lives in the same directory in the CVS tree as the PEPs themselves. Try "pep2html.py --help" for details. The URL for viewing PEPs on the web is http://www.python.org/peps/ [6] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [7] PEP 9, Sample PEP Template http://www.python.org/peps/pep-0009.html Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: [View Less]

8 14

Reviving restricted mode?
by Guido van Rossum Aug. 14, 2014

Aug. 14, 2014

I've received some enthusiastic emails from someone who wants to revive restricted mode. He started out with a bunch of patches to the CPython runtime using ctypes, which he attached to an App Engine bug: http://code.google.com/p/googleappengine/issues/detail?id=671 Based on his code (the file secure.py is all you need, included in secure.tar.gz) it seems he believes the only security leaks are __subclasses__, gi_frame and gi_code. (I have since convinced him that if we add "restricted" … [View More]

19 35

PEP 3145 (With Contents)
by Eric Pruitt Dec. 26, 2012

Dec. 26, 2012

Alright, I will re-submit with the contents pasted. I never use double backquotes as I think them rather ugly; that is the work of an editor or some automated program in the chain. Plus, it also messed up my line formatting and now I have lines with one word on them... Anyway, the contents of PEP 3145: PEP: 3145 Title: Asynchronous I/O For subprocess.Popen Author: (James) Eric Pruitt, Charles R. McCreary, Josiah Carlson Type: Standards Track Content-Type: text/plain Created: 04-Aug-2009 Python-… [View More]Version: 3.2 Abstract: In its present form, the subprocess.Popen implementation is prone to dead-locking and blocking of the parent Python script while waiting on data from the child process. Motivation: A search for "python asynchronous subprocess" will turn up numerous accounts of people wanting to execute a child process and communicate with it from time to time reading only the data that is available instead of blocking to wait for the program to produce data [1] [2] [3]. The current behavior of the subprocess module is that when a user sends or receives data via the stdin, stderr and stdout file objects, dead locks are common and documented [4] [5]. While communicate can be used to alleviate some of the buffering issues, it will still cause the parent process to block while attempting to read data when none is available to be read from the child process. Rationale: There is a documented need for asynchronous, non-blocking functionality in subprocess.Popen [6] [7] [2] [3]. Inclusion of the code would improve the utility of the Python standard library that can be used on Unix based and Windows builds of Python. Practically every I/O object in Python has a file-like wrapper of some sort. Sockets already act as such and for strings there is StringIO. Popen can be made to act like a file by simply using the methods attached the the subprocess.Popen.stderr, stdout and stdin file-like objects. But when using the read and write methods of those options, you do not have the benefit of asynchronous I/O. In the proposed solution the wrapper wraps the asynchronous methods to mimic a file object. Reference Implementation: I have been maintaining a Google Code repository that contains all of my changes including tests and documentation [9] as well as blog detailing the problems I have come across in the development process [10]. I have been working on implementing non-blocking asynchronous I/O in the subprocess.Popen module as well as a wrapper class for subprocess.Popen that makes it so that an executed process can take the place of a file by duplicating all of the methods and attributes that file objects have. There are two base functions that have been added to the subprocess.Popen class: Popen.send and Popen._recv, each with two separate implementations, one for Windows and one for Unix based systems. The Windows implementation uses ctypes to access the functions needed to control pipes in the kernel 32 DLL in an asynchronous manner. On Unix based systems, the Python interface for file control serves the same purpose. The different implementations of Popen.send and Popen._recv have identical arguments to make code that uses these functions work across multiple platforms. When calling the Popen._recv function, it requires the pipe name be passed as an argument so there exists the Popen.recv function that passes selects stdout as the pipe for Popen._recv by default. Popen.recv_err selects stderr as the pipe by default. "Popen.recv" and "Popen.recv_err" are much easier to read and understand than "Popen._recv('stdout' ..." and "Popen._recv('stderr' ..." respectively. Since the Popen._recv function does not wait on data to be produced before returning a value, it may return empty bytes. Popen.asyncread handles this issue by returning all data read over a given time interval. The ProcessIOWrapper class uses the asyncread and asyncwrite functions to allow a process to act like a file so that there are no blocking issues that can arise from using the stdout and stdin file objects produced from a subprocess.Popen call. References: [1] [ python-Feature Requests-1191964 ] asynchronous Subprocess http://mail.python.org/pipermail/python-bugs-list/2006-December/ 036524.html [2] Daily Life in an Ivory Basement : /feb-07/problems-with-subprocess http://ivory.idyll.org/blog/feb-07/problems-with-subprocess [3] How can I run an external command asynchronously from Python? - Stack Overflow http://stackoverflow.com/questions/636561/how-can-i-run-an-external- command-asynchronously-from-python [4] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation http://docs.python.org/library/subprocess.html#subprocess.Popen.wait [5] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation http://docs.python.org/library/subprocess.html#subprocess.Popen.kill [6] Issue 1191964: asynchronous Subprocess - Python tracker http://bugs.python.org/issue1191964 [7] Module to allow Asynchronous subprocess use on Windows and Posix platforms - ActiveState Code http://code.activestate.com/recipes/440554/ [8] subprocess.rst - subprocdev - Project Hosting on Google Code http://code.google.com/p/subprocdev/source/browse/doc/subprocess.rst?spec=s… [9] subprocdev - Project Hosting on Google Code http://code.google.com/p/subprocdev [10] Python Subprocess Dev http://subdev.blogspot.com/ Copyright: This P.E.P. is licensed under the Open Publication License; http://www.opencontent.org/openpub/. On Tue, Sep 8, 2009 at 22:56, Benjamin Peterson <benjamin(a)python.org> wrote: > 2009/9/7 Eric Pruitt <eric.pruitt(a)gmail.com>: >> Hello all, >> >> I have been working on adding asynchronous I/O to the Python >> subprocess module as part of my Google Summer of Code project. Now >> that I have finished documenting and pruning the code, I present PEP >> 3145 for its inclusion into the Python core code. Any and all feedback >> on the PEP (http://www.python.org/dev/peps/pep-3145/) is appreciated. > > Hi Eric, > One of the reasons you're not getting many response is that you've not > pasted the contents of the PEP in this message. That makes it really > easy for people to comment on various sections. > > BTW, it seems like you were trying to use reST formatting with the > text PEP layout. Double backquotes only mean something in reST. > > > -- > Regards, > Benjamin > [View Less]

10 26

Problems with hex-conversion functions
by Ender Wiggin Oct. 2, 2010

Oct. 2, 2010

Hello everyone. I see several problems with the two hex-conversion function pairs that Python offers: 1. binascii.hexlify and binascii.unhexlify 2. bytes.fromhex and bytes.hex Problem #1: bytes.hex is not implemented, although it was specified in PEP 358. This means there is no symmetrical function to accompany bytes.fromhex. Problem #2: Both pairs perform the same function, although The Zen Of Python suggests that "There should be one-- and preferably only one --obvious way to do it." I do … [View More]not understand why PEP 358 specified the bytes function pair although it mentioned the binascii pair... Problem #3: bytes.fromhex may receive spaces in the input string, although binascii.unhexlify may not. I see no good reason for these two functions to have different features. Problem #4: binascii.unhexlify may receive both input types: strings or bytes, whereas bytes.fromhex raises an exception when given a bytes parameter. Again there is no reason for these functions to be different. Problem #5: binascii.hexlify returns a bytes type - although ideally, converting to hex should always return string types and converting from hex should always return bytes. IMO there is no meaning of bytes as an output of hexlify, since the output is a representation of other bytes. This is also the suggested behavior of bytes.hex in PEP 358 Problems #4 and #5 call for a decision about the input and output of the functions being discussed: Option A : Strict input and output unhexlify (and bytes.fromhex) may only receives string and may only return bytes hexlify (and bytes.hex) may only receives bytes and may only return strings Option B : Robust input and strict output unhexlify (and bytes.fromhex) may receive bytes and strings and may only return bytes hexlify (and bytes.hex) may receive bytes or strings and may only return strings Of course we may also consider a third option, which will allow the return type of all functions to be robust (perhaps specified in a keyword argument), but as I wrote in the description of problem #5, I see no sense in that. Note that PEP 3137 describes: "... the more strict definitions of encoding and decoding in Python 3000: encoding always takes a Unicode string and returns a bytes sequence, and decoding always takes a bytes sequence and returns a Unicode string." - suggesting option A. To repeat problems #4 and #5, the current behavior does not match any option: * The return type of binascii.hexlify should be string, and this is not the current behavior. As for the input: * Option A is not the current behavior because binascii.unhexlify may receive both input types. * Option B is not the current behavior because bytes.fromhex does not allow bytes as input. To fix these issues, three changes should be applied: 1. Deprecate bytes.fromhex. This fixes the following problems: #4 (go with option B and remove the function that does not allow bytes input) #2 (the binascii functions will be the only way to "do it") #1 (bytes.hex should not be implemented) 2. In order to keep the functionality that bytes.fromhex has over unhexlify, the latter function should be able to handle spaces in its input (fix #3) 3. binascii.hexlify should return string as its return type (fix #5) [View Less]

5 7

PEP 382: Namespace Packages
by "Martin v. Löwis" Nov. 5, 2009

Nov. 5, 2009

I propose the following PEP for inclusion to Python 3.1. Please comment. Regards, Martin Abstract ======== Namespace packages are a mechanism for splitting a single Python package across multiple directories on disk. In current Python versions, an algorithm to compute the packages __path__ must be formulated. With the enhancement proposed here, the import machinery itself will construct the list of directories that make up the package. Terminology =========== Within this PEP, the term … [View More]package refers to Python packages as defined by Python's import statement. The term distribution refers to separately installable sets of Python modules as stored in the Python package index, and installed by distutils or setuptools. The term vendor package refers to groups of files installed by an operating system's packaging mechanism (e.g. Debian or Redhat packages install on Linux systems). The term portion refers to a set of files in a single directory (possibly stored in a zip file) that contribute to a namespace package. Namespace packages today ======================== Python currently provides the pkgutil.extend_path to denote a package as a namespace package. The recommended way of using it is to put:: from pkgutil import extend_path __path__ = extend_path(__path__, __name__) int the package's ``__init__.py``. Every distribution needs to provide the same contents in its ``__init__.py``, so that extend_path is invoked independent of which portion of the package gets imported first. As a consequence, the package's ``__init__.py`` cannot practically define any names as it depends on the order of the package fragments on sys.path which portion is imported first. As a special feature, extend_path reads files named ``*.pkg`` which allow to declare additional portions. setuptools provides a similar function pkg_resources.declare_namespace that is used in the form:: import pkg_resources pkg_resources.declare_namespace(__name__) In the portion's __init__.py, no assignment to __path__ is necessary, as declare_namespace modifies the package __path__ through sys.modules. As a special feature, declare_namespace also supports zip files, and registers the package name internally so that future additions to sys.path by setuptools can properly add additional portions to each package. setuptools allows declaring namespace packages in a distribution's setup.py, so that distribution developers don't need to put the magic __path__ modification into __init__.py themselves. Rationale ========= The current imperative approach to namespace packages has lead to multiple slightly-incompatible mechanisms for providing namespace packages. For example, pkgutil supports ``*.pkg`` files; setuptools doesn't. Likewise, setuptools supports inspecting zip files, and supports adding portions to its _namespace_packages variable, whereas pkgutil doesn't. In addition, the current approach causes problems for system vendors. Vendor packages typically must not provide overlapping files, and an attempt to install a vendor package that has a file already on disk will fail or cause unpredictable behavior. As vendors might chose to package distributions such that they will end up all in a single directory for the namespace package, all portions would contribute conflicting __init__.py files. Specification ============= Rather than using an imperative mechanism for importing packages, a declarative approach is proposed here, as an extension to the existing ``*.pkg`` mechanism. The import statement is extended so that it directly considers ``*.pkg`` files during import; a directory is considered a package if it either contains a file named __init__.py, or a file whose name ends with ".pkg". In addition, the format of the ``*.pkg`` file is extended: a line with the single character ``*`` indicates that the entire sys.path will be searched for portions of the namespace package at the time the namespace packages is imported. Importing a package will immediately compute the package's __path__; the ``*.pkg`` files are not considered anymore after the initial import. If a ``*.pkg`` package contains an asterisk, this asterisk is prepended to the package's __path__ to indicate that the package is a namespace package (and that thus further extensions to sys.path might also want to extend __path__). At most one such asterisk gets prepended to the path. extend_path will be extended to recognize namespace packages according to this PEP, and avoid adding directories twice to __path__. No other change to the importing mechanism is made; searching modules (including __init__.py) will continue to stop at the first module encountered. Discussion ========== With the addition of ``*.pkg`` files to the import mechanism, namespace packages can stop filling out the namespace package's __init__.py. As a consequence, extend_path and declare_namespace become obsolete. It is recommended that distributions put a file <distribution>.pkg into their namespace packages, with a single asterisk. This allows vendor packages to install multiple portions of namespace package into a single directory, with no risk of overlapping files. Namespace packages can start providing non-trivial __init__.py implementations; to do so, it is recommended that a single distribution provides a portion with just the namespace package's __init__.py (and potentially other modules that belong to the namespace package proper). The mechanism is mostly compatible with the existing namespace mechanisms. extend_path will be adjusted to this specification; any other mechanism might cause portions to get added twice to __path__. Copyright ========= This document has been placed in the public domain. [View Less]

31 110

transitioning from % to {} formatting
by Steven Bethard Oct. 10, 2009

Oct. 10, 2009

There's a lot of code already out there (in the standard library and other places) that uses %-style formatting, when in Python 3.0 we should be encouraging {}-style formatting. We should really provide some sort of transition plan. Consider an example from the logging docs: logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s") We'd like to support both this style as well as the following style: logging.Formatter("{asctime} - {name} - {levelname} - {message}") Perhaps we'… [View More]

26 131

PEP 389: argparse - new command line parsing module
by Steven Bethard Oct. 9, 2009

Oct. 9, 2009

On Sun, Sep 27, 2009 at 8:41 PM, Benjamin Peterson <benjamin(a)python.org> wrote: > 2009/9/27 Steven Bethard <steven.bethard(a)gmail.com>: >> The first release where any real deprecation message would show up is >> Python 3.4, more than 3 years away. If you think 3 years isn't long >> enough for people to be over the Python 3 transition, let's stick in >> another version in there and make it 4.5 years. > > So, why even bother deprecating it if nobody … [View More]

27 72

please consider changing --enable-unicode default to ucs4
by Zooko O'Whielacronx Oct. 8, 2009

Oct. 8, 2009

Dear Pythonistas: This issue causes serious problems. Users occasionally get binaries built for a compatible Linux and Python version but with a different UCS2-vs-UCS4 setting, and those users get mysterious memory corruption errors which are hard to diagnose. It is possible that these situations also open up security vulnerabilities. A couple such instances are documented on http://bugs.python.org/setuptools/issue78, but you can find more by googling. I would like to get this problem fixed!… [View More] In order to help address this issue I sampled what UCS size is used by python executables in the wild. I instrumented a few buildslaves that are contributed by various people to the Tahoe-LAFS project to print out their platform, python version, and sys.maxunicode. The full results are appended below. maxunicode: 1114111 means that python executable was configured with --enable-unicode=ucs4, and maxunicode: 65535 means that python executable was configured with --enable-unicode=ucs2 or just with --enable-unicode . The only incompatibilities that I found are because some packagers have deliberately set UCS4 configuration and other packagers have left the default setting. In the three cases where someone configured python with UCS2, one of the three is certainly an accident (a custom-built python executable on an Ubuntu server) and the other two just use the default instead of specifically configuring ucs2 in their configurations of Python and I suspect that they don't know the difference and that it was an accident that they built a Python incompatible with other distributions of their operating system. In sum, while it would be good to add the unicode setting to the platform's ABI (as discussed in setuptools ticket #78), it would also be good to make the default value be UCS4 instead of UCS2. This would fix all three of the potential incompatibilities that I found (listed below), and once we have proper inclusion of the unicode setting in the ABI in order to prevent the memory corruption, defaulting to UCS4 would increase the likelihood that a binary built on one distribution would be usable on another. I'm sure that someone can come up with a reason why UCS2 is better than UCS4, but I'm also sure that the benefits of compatibility outweigh any benefits of UCS2 encoding, and that the widespread use of UCS4 demonstrates that there is nothing fatally wrong with it, and that people who really value UCS2 encoding more than compatibility can choose that for themselves by explicitly setting UCS2. Let me restate that I am not suggesting taking away anyone's options, only making the setting for people who don't specify default to the compatible option. Hm, I guess that means that it should default to UCS2 on Windows and Mac and to UCS4 on Linux and Solaris. Regards, Zooko Ubuntu 6.10 "edgy" i386: python: 2.4.4c1 (#2, Mar 7 2008, 03:03:38) [GCC 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5)], maxunicode: 1114111 Ubuntu 7.04 "feisty": python: 2.5.1 (r251:54863, Jul 31 2008, 22:53:39) [GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)], maxunicode: 1114111 Ubuntu 7.10 "gutsy" i386: python: 2.5.1 (r251:54863, Jul 31 2008, 23:17:40) [GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)], maxunicode: 1114111 Ubuntu 8.04 "hardy" amd64: python: 2.5.2 (r252:60911, Jul 22 2009, 15:33:10) [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)], maxunicode: 1114111 Ubuntu 8.04 "hardy" i386: *custom* python: 2.6 (r26:66714, Oct 2 2008, 13:40:28) [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)], maxunicode: 65535 Ubuntu 8.04 "hardy" i386: python: 2.5.2 (r252:60911, Jul 22 2009, 15:35:03) [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)], maxunicode: 1114111 Ubuntu 9.04 "jaunty" amd64: *custom* python: 2.6.2 (release26-maint, Apr 19 2009, 01:58:18) [GCC 4.3.3], maxunicode: 1114111 Debian 4.0 "etch" i386: python: 2.4.4 (#2, Oct 22 2008, 19:52:44) [GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)], maxunicode: 1114111 Debian 5.0 "lenny" i386: python: 2.5.2 (r252:60911, Jan 4 2009, 17:40:26) [GCC 4.3.2], maxunicode: 1114111 Debian 5.0 "lenny" amd64: python: 2.5.2 (r252:60911, Jan 4 2009, 21:59:32) [GCC 4.3.2], maxunicode: 1114111 Debian 5.0 "lenny" armv5tel: python: 2.5.2 (r252:60911, Jan 5 2009, 02:00:00) [GCC 4.3.2], maxunicode: 1114111 Debian unstable "squeeze/sid" i386: python: 2.5.4 (r254:67916, Feb 17 2009, 20:16:45) [GCC 4.3.3], maxunicode: 1114111 Fedora 11 "leonidas" amd64: python: 2.6 (r26:66714, Jul 4 2009, 17:37:13) [GCC 4.4.0 20090506 (Red Hat 4.4.0-4)], maxunicode: 1114111 ArchLinux: python: 2.6.2 (r262:71600, Jul 20 2009, 02:23:30) [GCC 4.4.0 20090630 (prerelease)], maxunicode: 65535 NetBSD 4: python: 2.5.2 (r252:60911, Mar 20 2009, 14:00:07) [GCC 4.1.2 20060628 prerelease (NetBSD nb2 20060711)], maxunicode: 65535 OpenSolaris SunOS-5.11-i86pc-i386-32bit: python: 2.4.4 (#1, Mar 10 2009, 09:35:36) [C], maxunicode: 65535 Nexenta NCP1 SunOS-5.11-i86pc-i386-32bit: python: 2.4.3 (#2, May 3 2006, 19:12:42) [GCC 4.0.3 (GNU_OpenSolaris 4.0.3-1nexenta4)], maxunicode: 1114111 Mac OS 10.6 "snow leopard" i386: python: 2.6.1 (r261:67515, Jul 7 2009, 23:51:51) [GCC 4.2.1 (Apple Inc. build 5646)], maxunicode: 65535 Mac OS 10.5 "leopard" i386: python: 2.5.1 (r251:54863, Feb 6 2009, 19:02:12) [GCC 4.0.1 (Apple Inc. build 5465)], maxunicode: 65535 Mac OS 10.4 "tiger" *custom* python: 2.5.4 (release25-maint:72153M, Apr 30 2009, 12:28:20) [GCC 4.0.1 (Apple Computer, Inc. build 5367)], maxunicode: 65535 Cygwin CYGWIN_NT-5.1-1.5.25-0.156-4-2-i686-32bit-WindowsPE: python: 2.5.2 (r252:60911, Dec 2 2008, 09:26:14) [GCC 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)], maxunicode: 65535 Windows: python: 2.6.2 (r262:71600, Apr 21 2009, 15:05:37) [MSC v.1500 32 bit (Intel)], maxunicode: 65535 Windows: python: 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)], maxunicode: 65535 [View Less]

7 13

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4
by M.-A. Lemburg Oct. 8, 2009

Oct. 8, 2009

Zooko O'Whielacronx wrote: > Folks: > > I'm sorry, I think I didn't make my concern clear. My users, and lots > of other users, are having a problem with incompatibility between > Python binary extension modules. One way to improve the situation > would be if the Python devs would use their "bully pulpit" -- their > unique position as a source respected by all Linux distributions -- > and say "We recommend that Linux distributions use UCS4 for > compatibility with … [View More]one another". This would not abrogate anyone's > ability to choose their preferred setting nor, as far as I can tell, > would it interfere with the ongoing development of Python. -1 Please note that we did not choose to ship Python as UCS4 binary on Linux - the Linux distributions did. The Python default is UCS2 for a good reason: it's a good trade-off between memory consumption, functionality and performance. As already mentioned, I also don't understand how the changing the Python default on Linux would help your users in any way - if you let distutils compile your extensions, it's automatically going to use the right Unicode setting for you (as well as your users). Unfortunately, this automatic support doesn't help you when shipping e.g. setuptools eggs, but this is a tool problem, not one of Python: setuptools completely ignores the fact that there are two ways to build Python. I'd suggest you ask the tool maintainers to adjust their tools to support the Python Unicode option. > Here are the details: > > I'm the maintainer of several Python packages. I work hard to make it > easy for users, even users who don't know anything about Python, to > use my software. There have been many pain points in this process and > I've spent a lot of time on it for about three years now working on > packaging, including the tools such as setuptools and distutils and > the new "distribute" tool. Python packaging has been improving during > these years -- things are looking up. > > One of the remaining pain points is that I can distribute binaries of > my Python extension modules for Windows or Mac, but if I distribute a > binary Python extension module on Linux, then if the user has a > different UCS2/UCS4 setting then they won't be able to use the > extension module. The current de facto standard for Linux is UCS4 -- > it is used by Debian, Ubuntu, Fedora, RHEL, OpenSuSE, etc. etc.. The > vast majority of Linux users in practice have UCS4, and most binary > Python modules are compiled for UCS4. > > That means that a few folks will get left out. Those folks, from my > experience, are people who built their python executable themselves > without specifying an override for the default, and the smaller Linux > distributions who insist on doing whatever upstream Python devs > recommend instead of doing whatever the other Linux distros are doing. > One of the data points that I reported was a Python interpreter that > was built locally on an Ubuntu server. Since the person building it > didn't know to override the default setting of --enable-unicode, he > ended up with a Python interpreter built for UCS2, even though all the > Python extension modules shipped by Ubuntu were built with UCS4. People building their own Python version will usually also build their own extensions, so I don't really believe that the above scenario is very common. Also note that Python will complain loudly when you try to load a UCS2 extension in a UCS4 build and vice-versa. We've made sure that any extension using the Python Unicode C API has to be built for the same UCS version of Python. This is done by using different names for the C APIs at the C level. > These are not isolated incidents. The following google searches > suggest that a number of people spend time trying to figure out why > Python extension modules fail on their linux systems: > > http://www.google.com/search?q=PyUnicodeUCS4_FromUnicode+undefined+symbol > http://www.google.com/search?q=+PyUnicodeUCS2_FromUnicode+undefined+symbol > http://www.google.com/search?q=_PyUnicodeUCS2_AsDefaultEncodedString+undefi… Perhaps we should add a FAQ entry for these linker errors (which are caused by the mentioned C API changes to prevent mixing UCS version) ?! Here's a quick way to determine you Python Unicode build type: python -c "import sys;print((sys.maxunicode<66000)and'UCS2'or'UCS4')" Perhaps we should include this info as well as an 32/64-bit indicator and the processor type in the Python startup line: # python Python 2.6 (r26:66714, Feb 3 2009, 20:49:49, UCS4, 64-bit, x86_64) [GCC 4.3.2 [gcc-4_3-branch revision 141291]] on linux2 Type "help", "copyright", "credits" or "license" for more information. This would help users find the right binaries to install as extension. > Another data point is the Mandriva Linux distribution. It is probably > much smaller than Debian, Ubuntu, or RedHat, but it is still one of > the major, well-known distributions. I requested of the Python > maintainer for Mandriva, Michael Scherer, that they switch from UCS2 > to UCS4 in order to reduce compatibility problems like these. His > answer as I understood it was that it is best to follow the > recommendations of the upstream Python devs by using the default > setting instead of choosing a setting for himself. Which is IMHO what all Linux distributions should have done. Distributions should really not be put in charge of upstream coding design decisions. Regards, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 28 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ [View Less]

8 14

Announcing PEP 3136
by Hatem Nassrat Oct. 7, 2009

Oct. 7, 2009

Tue Jul 3 10:14:17 CEST 2007, Guido van Rossum wrote: > On 6/30/07, Matt Chisholm <matt-python at theory.org> wrote: > > I've created and submitted a new PEP proposing support for labels in > > Python's break and continue statements. Georg Brandl has graciously > > added it to the PEP list as PEP 3136: > > > > http://www.python.org/dev/peps/pep-3136/ > > I think this is a good summary of various proposals that have been > floated in the past, … [View More]

7 7

Re: [Python-Dev] Distutils ML wrap-up: setup.cfg new format
by P.J. Eby Oct. 7, 2009

Oct. 7, 2009

At 07:00 PM 9/23/2009 +0200, Tarek Ziadé wrote: >While it's great to have Philipp being part of our distutils design >discussions, >for his experience, I am not concerned of not having him in this "internal >consensus" since Setuptools is not maintained anymore. > >He said some months ago, he would work on a brand new setuptools >version with zero >dependency to distutils. But it's still vaporware (from his own >words), and the previous version is unmaintained for a … [View More]year, so it was >forked. Here's what actually happened, if anyone cares. Tarek and friends announced a fork of setuptools. I reviewed the work and saw that -- for the most part -- I was happy with it, and opined as how I might be willing to bless the the "package inquisition" team as official maintainers of the 0.6 branch of setuptools, so that I could work on the fun bits I've long planned for 0.7, but never felt free to start on while there was so much still needing to be done on 0.6. However, just as I mentioned this, and suggested an option for what I could do that would be helpful to his Distribute 0.7 project as well as various other tools (e.g. implementing some of Jim Fulton's long-requested features for better modularization of setuptools), Tarek accused me of somehow trying to undermine his plans. In addition, it appears Tarek was also offended by my earlier statement that there were only a few people in the Python community who had *already* earned my implicit trust to not only hack on setuptools unsupervised, but also to take over its *future* direction and BDFL-ship. (For example, Jim Fulton and Ian Bicking.) Tarek, however, appears to have taken this to mean that I personally thought he was an incompetent programmer or something (when I actually had no opinion one way or the other), and ever since he has taken to levelling potshots like the above at me on a semi-regular basis. I've tried to ignore this and play nice, because he is actually working on this stuff and I am not. But it's hard for me to actually give any help in practice, if Tarek is too busy projecting hidden plots onto everything I say and do. If you read Tarek's distutils-sig posts, it appears my already-existing trust in Ian and Jim was not only a personal insult to Tarek, but also a plot to ensure that nobody with any time to do so would ever work on setuptools, just as my excitement about working on setuptools again was a plot to steal thunder from his fork. All I want is for good stuff to happen for setuptools users and Python users in general, so I don't think all the suspicion and backbiting is merited. I certainly don't appreciate it, and I would like it to stop. It also isn't even relevant to the thread, since my lack of work on setuptools says exactly zero about the merits or lack thereof of Tarek's proposals for the distutils! Hell, I *support* the bulk of Tarek's setup.cfg proposal, and don't even object to him Pronouncing it or cutting off the discussion! My only issue on Python-Dev was his inaccurate implication that it was a SIG consensus rather than a pronouncement on it. There is and was no need for any of this to get personal, and I have continually strived to keep my posts here and distutils-sig civil, even when I didn't feel like being civil in response to Tarek's jabs. I have in fact bent over backwards to be *nice* to Tarek, because he seemed so damn sensitive about everything. Apparently, however, this does not actually help things. :-( [View Less]

7 8

Python 2.6.3rc1 available
by Barry Warsaw Oct. 4, 2009

Oct. 4, 2009

The first (and hopefully last) release candidate for Python 2.6.3 is now available via http://www.python.org/download/releases/2.6.3/ Source releases and Windows binaries are currently available, and Mac OS X binaries should be forthcoming. Nearly 100 bugs have been fixed since 2.6.2. Barring any unforeseen problems, we will make the final 2.6.3 release this Friday, October 2nd. Please give this release candidate a spin and let us know if you encounter any show stopping problems. Enjoy, -Barry

4 3

PEP 3144 review.
by Peter Moody Oct. 3, 2009

Oct. 3, 2009

Folks, Guido, I believe PEP 3144 is ready for your review. When you get a chance, can you take a look/make a pronouncement? Cheers, /peter

35 266

Python 2.6.3
by Barry Warsaw Oct. 2, 2009

Oct. 2, 2009

I had previously wanted to release Python 2.6.3 over the summer, but for various personal reasons, the summer was just too insane. I'd like to reschedule a 2.6.3 release, shooting for final release on 25- September. We should probably do a release candidate, so I'd like to make that on 23-September. Does anybody have objections to that schedule? If not, I'll try to spend some time over the next few days looking at outstanding bugs, and marking release blockers, etc. -Barry

14 35

Python logging and 1.5.2 compatibility
by Vinay Sajip Oct. 1, 2009

Oct. 1, 2009

I'm planning to "officially" drop support for Python 1.5.2 in the logging package. When the logging package was introduced in Python 2.3, many Linux distros were shipping 1.5.2 as the system's Python, even though 2.2 had been out for a while. So it seemed important to support 1.5.2 for those sysadmins who wanted to use logging with their system Python. The Linux landscape has changed a bit since then. Most Linux distros ship with much more recent versions of Python, and so I no longer see 1.5.… [View More]

4 4

operator precedence of __eq__, __ne__, etc, if both object have implementations
by Chris Withers Sept. 30, 2009

Sept. 30, 2009

Hi All, I didn't see any docs on this: http://docs.python.org/reference/datamodel.html?highlight=__eq__#object.__e… Where are the specifications on what happens if two objects are compared and both have implementations of __eq__? Which __eq__ is called? What happens if the first one called returns False? Is the second one called? What is one implements __eq__ and the other __ne__? If I've missed something, please point me in the right direction. To all those about to tell me to go read … [View More]

16 37

Distutils ML wrap-up: setup.cfg new format
by Tarek Ziadé Sept. 30, 2009

Sept. 30, 2009

Hello Here's a wrapup of the Distutils-SIG discussion we had on the "static metadata" topic. I realize that it's a good thing to send in. python-dev such wrapup on distutils design decisions, to keep everyone informed and get more feedback when required. I will try to do it for every upcoming changes that are not going in a PEP process (when it's not a 'big' change). The rate of such mails should not be very high. (around 1 mail in python-dev for +150 mails in distutils-SIG) If you feel … [View More]that what we are about to change in distutils is wrong, you can go ahead and help us by participating in Distutils-ML, so we keep one and only one media for these discussions. The four sentences summary for people in a hurry: Getting metadata of a distribution that is not. installed means running its setup.py. This means. downloading the whole archive, and running. third party code on your system. To avoid it, we are adding a [setup] section in. setup.cfg where you can express the package metadata statically. Conditional sections, specific to some system. can be added to add some specific fields in [setup]. At the end, you will be able to get metadata fields without running any third-party code, and possibly get only the distribution setup.cfg for this. Now the long version. Rational ======== Today, if you want to list all the metadata (PEP 314) of a. distribution that is not installed, you need to use it's. setup.py CLI. So, basically, you download it, and run:: $ python setup.py --name Foo $ python setup.py --version 1.2 Where `name` and `version` are metadata fields. That's OK but as. soon as the developers add more code in setup.py, this feature might break or (worse) might do unwanted things on the target. system. Why should we run third-party code just to get its metadata ? So we worked on a way to express those metadata statically, by pushing them in `setup.cfg`. That's not hard, since all. the metadata fields are static values. Adding a setup section in setup.cfg =================================== So the first thing we want to introduce is a [setup] section, that may contain any field from the metadata:: [setup] name: Foo version: 1.2 Distutils will look for them, and will use them. Of course setup.py is still required, and in case an option that is. a metadata field is passed to setup(), it will override one. located in setup.cfg. PEP 341 is coming up ==================== Remember the last Pycon Summit ? We said that we would introduce a new metadata field to describe requirements.. That's basically what PEP 341 is about, and we are still. working on it. Basically, we will be able to write things like:: requires: Twisted == 8.2.0 What takes us so long is that adding requirements like this in the metadata requires more things: - the ability to compare versions (this was described in. PEP 386 but not finished yet) - the ability to make these requirements vary depending on. the target system And the later makes our "setup.cfg" new [setup] section. obsolete as soon as this new metadata field is added in. Python. So we need more that what I've shown in the previous section Context-dependant sections ========================== The second change we are going to introduce is context-dependant sections in setup.cfg. A context dependant section is a section with a condition that is. used only if the execution environment meets its condition. Here's an example:: [setup] name: Foo version: 1.3 [setup:sys_platform == 'win32'] requires: pywin32 requires: bar > 1.0 [setup:os_machine == '64bits'] requires: some_package [setup:python_version == '2.4' or python_version == '2.5'] requires: some_package Every [setup:condition] section will be added in [setup] only if the condition is met. The micro-language behind this is the simplest possible: it compares only strings, with usual string operators, and with the ability to combine expressions. It makes it also easy to understand to non-pythoneers (os packagers). The pseudo-grammar is (I don't know how to write those but you'll get it hopefully):: comp: '<'|'>'|'=='|'>='|'<='|'<>'|'!='|'in'|'not' 'in' comparison: expr (comp_op expr)* expr: STRING test: or_test or_test: and_test ('or' and_test)* and_test: not_test ('and' not_test)* not_test: 'not' not_test | comparison where STRING belongs to any of those: - python_version = '%s.%s' % (sys.version_info[0], sys.version_info[1]) - os_name = os.name - sys_platform = sys.platform - os_sysname = os.uname()[0]. - os_nodename = os.uname()[1] - os_release = os.uname()[2]. - os_version = os.uname()[3].. - os_machine = os.uname()[4]. - a free string, like '2.4', or 'win32' Distutils will provide a function that is able to read the metadata of a distribution, given a setup.cfg file, for the target environment:: >>> from distutils.util import local_metadata >>> local_metadata('setup.cfg') <DistributionMetadata instance> Meaning that a Vanilla Python will be able to read the metadata of a package without running any code. That's what distutils will use when setup.py is run. That's what any package manager will be able to use to work. with distributions. Limitations =========== If a distribution is unable for obscure reasons (or does not wish) to set all metadata fields in setup.cfg, that's fine :. the fields will be set to UNKNOWN when `local_metadata` is called. When setup.py is run, options passed to setup() will complete those. When a package manager gets 'UNKOWN' values, it knows it might need to do further processing by running setup.py. Why this change is good ===================== Once the requires metadata is added in PEP 341, being able to query the metadata for a distribution can be done without doing anything else than reading a static file and interpreting its conditions in a restricted and secured manner. So there's no need to run third-party code anymore. Possible use cases: - setup.cfg is published on PyPI. Packages managers like easy_install or pip will be able to list all the distributions needed and their versions for installing a distribution without having to download anything ! - OS packagers will be able to list the dependencies of a distribution without having to browse Python code. And, again, this new [setup] section is backward-compatible, and will be a good complement to the work done in PEP 376. Thanks for reading so far. This code will be added in 2.7 and 3.2. Tarek -- Tarek Ziadé | http://ziade.org | オープンソースはすごい! [View Less]

15 40

test_thread tracebacks
by skip＠pobox.com Sept. 30, 2009

Sept. 30, 2009

It's been awhile since I rebuilt Python and ran the test suite. This evening I noticed this on my Mac (OS X 10.5): test_thread Unhandled exception in thread started by <bound method ThreadRunningTests.task of <test.test_thread.ThreadRunningTests testMethod=test_stack_size>> Traceback (most recent call last): File "/Users/skip/src/python/trunk/Lib/test/test_thread.py", line 51, in task self.done_mutex.release() thread.error: release unlocked lock … [View More]

3 4

Re: [Python-Dev] transitioning from % to {} formatting
by Steven Bethard Sept. 30, 2009

Sept. 30, 2009

On Tue, Sep 29, 2009 at 5:54 PM, Benjamin Peterson <benjamin(a)python.org> wrote: > * Provide a flag to Formatter which controls whether new or old > formatting is used. Emit a warning when it's not true. So then the transition strategy is something like: version N: Add formatting flag which uses {}-style formatting on True and %-style formatting on False, which defaults to False version N + 1: Deprecate False value for formatting flag (all code should start specifying flag=True) … [View More]

1 0

PEP 389: argparse - new command line parsing module
by Steven Bethard Sept. 30, 2009

Sept. 30, 2009

Below is a new PEP based on discussions from the stdlib-sig, which proposes to add the argparse module to the standard library in Python 2.7 and 3.2. Looking forward to your feedback! Steve http://www.python.org/dev/peps/pep-0389/ ---------------------------------------------------------------------- PEP: 389 Title: argparse - new command line parsing module Version: $Revision: 75097 $ Last-Modified: $Date: 2009-09-27 12:42:40 -0700 (Sun, 27 Sep 2009) $ Author: Steven Bethard <steven.… [View More]bethard(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 25-Sep-2009 Python-Version: 2.7 and 3.2 Post-History: Abstract ======== This PEP proposes inclusion of the argparse [1]_ module in the Python standard library in Python 2.7 and 3.2. Motivation ========== The argparse module is a command line parsing library which provides more functionality than the existing command line parsing modules in the standard library, getopt [2]_ and optparse [3]_. It includes support for positional arguments (not just options), subcommands, required options, options syntaxes like "/f" and "+rgb", zero-or-more and one-or-more style arguments, and many other features the other two lack. The argparse module is also already a popular third-party replacement for these modules. It is used in projects like IPython (the Scipy Python shell) [4]_, is included in Debian testing and unstable [5]_, and since 2007 has had various requests for its inclusion in the standard library [6]_ [7]_ [8]_. This popularity suggests it may be a valuable addition to the Python libraries. Why aren't getopt and optparse enough? ====================================== One argument against adding argparse is that thare are "already two different option parsing modules in the standard library" [9]_. The following is a list of features provided by argparse but not present in getopt or optparse: * While it is true there are two *option* parsing libraries, there are no full command line parsing libraries -- both getopt and optparse support only options and have no support for positional arguments. The argparse module handles both, and as a result, is able to generate better help messages, avoiding redundancies like the ``usage=`` string usually required by optparse. * The argparse module values practicality over purity. Thus, argparse allows required options and customization of which characters are used to identify options, while optparse explicitly states "the phrase 'required option' is self-contradictory" and that the option syntaxes ``-pf``, ``-file``, ``+f``, ``+rgb``, ``/f`` and ``/file`` "are not supported by optparse, and they never will be". * The argparse module allows options to accept a variable number of arguments using ``nargs='?'``, ``nargs='*'`` or ``nargs='+'``. The optparse module provides an untested recipe for some part of this functionality [10]_ but admits that "things get hairy when you want an option to take a variable number of arguments." * The argparse module supports subcommands, where a main command line parser dispatches to other command line parsers depending on the command line arguments. This is a common pattern in command line interfaces, e.g. ``svn co`` and ``svn up``. Why isn't the functionality just being added to optparse? ========================================================= Clearly all the above features offer improvements over what is available through optparse. A reasonable question then is why these features are not simply provided as patches to optparse, instead of introducing an entirely new module. In fact, the original development of argparse intended to do just that, but because of various fairly constraining design decisions of optparse, this wasn't really possible. Some of the problems included: * The optparse module exposes the internals of its parsing algorithm. In particular, ``parser.largs`` and ``parser.rargs`` are guaranteed to be available to callbacks [11]_. This makes it extremely difficult to improve the parsing algorithm as was necessary in argparse for proper handling of positional arguments and variable length arguments. For example, ``nargs='+'`` in argparse is matched using regular expressions and thus has no notion of things like ``parser.largs``. * The optparse extension APIs are extremely complex. For example, just to use a simple custom string-to-object conversion function, you have to subclass ``Option``, hack class attributes, and then specify your custom option type to the parser, like this:: class MyOption(Option): TYPES = Option.TYPES + ("mytype",) TYPE_CHECKER = copy(Option.TYPE_CHECKER) TYPE_CHECKER["mytype"] = check_mytype parser = optparse.OptionParser(option_class=MyOption) parser.add_option("-m", type="mytype") For comparison, argparse simply allows conversion functions to be used as ``type=`` arguments directly, e.g.:: parser = argparse.ArgumentParser() parser.add_option("-m", type=check_mytype) But given the baroque customization APIs of optparse, it is unclear how such a feature should interact with those APIs, and it is quite possible that introducing the simple argparse API would break existing custom Option code. * Both optparse and argparse parse command line arguments and assign them as attributes to an object returned by ``parse_args``. However, the optparse module guarantees that the ``take_action`` method of custom actions will always be passed a ``values`` object which provides an ``ensure_value`` method [12]_, while the argparse module allows attributes to be assigned to any object, e.g.:: foo_object = ... parser.parse_args(namespace=foo_object) foo_object.some_attribute_parsed_from_command_line Modifying optparse to allow any object to be passed in would be difficult because simply passing the ``foo_object`` around instead of a ``Values`` instance will break existing custom actions that depend on the ``ensure_value`` method. Because of issues like these, which made it unreasonably difficult for argparse to stay compatible with the optparse APIs, argparse was developed as an independent module. Given these issues, merging all the argparse features into optparse with no backwards incompatibilities seems unlikely. Deprecation of getopt and optparse ================================== There is still some debate over the best way (if at all) to encourage users to move from getopt and optparse to their replacement, argparse. The current recommendation of this PEP is the following conservative deprecation strategy: * Python 3.2, Python 2.7 and any later Python 2.X releases -- PendingDeprecation warnings, which by default are not displayed, and documentation notes directing users of getopt and optparse to argparse. * Python 3.3 -- Same as above * Python 3.4 -- Deprecation warnings for getopt and optparse, which by default *are* displayed. Though this is slower than the usual deprecation process, it seems prudent to avoid producing any casually visible Deprecation warnings until Python 3.X has had some additional time to attract developers. Open Issues =========== The argparse module supports Python from 2.3 up through 3.2 and as a result relies on traditional ``%(foo)s`` style string formatting. It has been suggested that it might be better to use the new style ``{foo}`` string formatting [13]_. This seems like a good idea, but would break backwards compatibility for existing argparse-based scripts unless we can come up with a way to reasonably support both syntaxes. References ========== .. [1] argparse (http://code.google.com/p/argparse/) .. [2] getopt (http://docs.python.org/library/getopt.html) .. [3] optparse (http://docs.python.org/library/optparse.html) .. [4] argparse in IPython (http://mail.scipy.org/pipermail/ipython-dev/2009-April/005102.html) .. [5] argparse in Debian (http://packages.debian.org/search?keywords=python-argparse) .. [6] 2007-01-03 request for argparse in the standard library (http://mail.python.org/pipermail/python-list/2007-January/592646.html) .. [7] 2009-06-09 request for argparse in the standard library (http://bugs.python.org/issue6247) .. [8] 2009-09-10 request for argparse in the standard library (http://mail.python.org/pipermail/stdlib-sig/2009-September/000342.html) .. [9] Fredrik Lundh response to [6]_ (http://mail.python.org/pipermail/python-list/2007-January/592675.html) .. [10] optparse variable args (http://docs.python.org/library/optparse.html#callback-example-6-variable-ar…) .. [11] parser.largs and parser.rargs (http://docs.python.org/library/optparse.html#how-callbacks-are-called) .. [12] take_action values argument (http://docs.python.org/library/optparse.html#adding-new-actions) .. [13] use {}-formatting instead of %-formatting (http://bugs.python.org/msg89279) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: [View Less]

20 32

Python 2.7 Mac universal builds seem broken on trunk
by Barry Scott Sept. 30, 2009

Sept. 30, 2009

I'm working with http://svn.python.org/projects/python/trunk on Mac OS X 10.6.1 using Apples xcode gcc 4.2.1. When I run the following commands: ./configure --enable-framework --with-universal-archs=32-bit | tee build.config.log make clean all | tee build.make.log I end up with a x86_64 Python image. No matter what I use for archs its always the same. I would expect to see -arch arg to GCC but it is not there. export CFLAG="-arch i386" did not work either. Am I doing something … [View More]

5 9

bug 3890
by Barry Warsaw Sept. 30, 2009

Sept. 30, 2009

I'd like to include the patch for bug 3890 in Python 2.6.3. There is a patch by Amaury that has gotten the blessing of Bill Janssen, however the patch has no test. Is there anybody who can come up with a test in the next few hours so we can get this into 2.6.3rc1? If not, is the fix important enough to include without the test? -Barry

1 0

Fuzziness in io module specs
by Pascal Chambon Sept. 29, 2009

Sept. 29, 2009

Hello everyone I'm currently working on a reimplementation of io.FileIO, which would allow cross-platform file range locking and all kinds of other safety features ; however I'm slightly stuck due to some specification fuzziness in the IO docs. CF http://bugs.python.org/issue6939 The main points that annoy me at the moment : - it is unclear what truncate() methods do with the file pointer, and even if the current implementation simply moves it to the truncation point, it's very … [View More]

14 45

IO module precisions and exception hierarchy
by Pascal Chambon Sept. 29, 2009

Sept. 29, 2009

Found in current io PEP : Q: Do we want to mandate in the specification that switching between reading and writing on a read-write object implies a .flush()? Or is that an implementation convenience that users should not rely on? -> it seems that the only important matter is : file pointer positions and bytes/characters read should always be the ones that the user expects, as if there were no buffering. So flushing or not may stay a non-mandatory behaviour, as long as the buffered … [View More]streams ensures this data integrity. Eg. If a user opens a file in r/w mode, writes two bytes in it (which stay buffered), and then reads 2 bytes, the two bytes read should be those on range [2:4] of course, even though the file pointer would, due to python buffering, still be at index 0. Q from me : What happens in read/write text files, when overwriting a three-bytes character with a single-byte character ? Or at the contrary, when a single chinese character overrides 3 ASCII characters in an UTF8 file ? Is there any system designed to avoid this data corruption ? Or should TextIO classes forbid read+write streams ? IO Exceptions : Currently, the situation is kind of fuzzy around EnvironmentError subclasses. * OSError represents errors notified by the OS via errno.h error codes (as mirrored in the python "errno" module). errno.h errors (less than 125 error codes) seem to represent the whole of *nix system errors. However, Windows has many more system errors (15000+). So windows errors, when they can't be mapped to one of the errno errors are raises as "WindowsError" instances (a subclass of OSError), with the special attribute "winerror" indicating that win32 error code. * IOError are "errors raised because of I/O problems", but they use errno codes, like OSError. Thus, at the moment IOErrors rather have the semantic of "particular case of OSError", and it's kind of confusing to have them remain in their own separate tree... Furthermore, OSErrors are often used where IOErrors would perfectly fit, eg. in low level I/O functions of the OS module. Since OSErrors and IOErrors are slightly mixed up when we deal with IO operations, maybe the easiest way to make it clearer would be to push to their limits already existing designs. - the os module should only raise OSErrors, whatever the os operation involved (maybe it's already the case in CPython, isn't it ?) - the io module should only raise IOErrors and its subclasses, so that davs can easily take measures depending on the cause of the io failure (except 1 OSError exception, it's already the case in _fileio) - other modules refering to i/o might maybe keep their current (fuzzy) behaviour, since they're more platform specific, and should in the end be replaced by a crossplatform solution (at least I'd love it to happen) Until there, there would be no real benefits for the user, compared to catching EnvironmentErrors as most probably do. But the sweet thing would be to offer a concise but meaningfull IOError hierarchy, so that we can easily handle most specific errors gracefully (having a disk full is not the same level of gravity as simply having another process locking your target file). Here is a very rough beginning of IOError hierarchy. I'd liek to have people's opinion on the relevance of these, as well as on what other exceptions should be distinguished from basic IOErrors. IOError +-InvalidStreamError (eg. we try to write on a stream opened in readonly mode) +-LockingError +-PermissionError (mostly *nix chmod stuffs) +-FileNotFoundError +-DiskFullError +-MaxFileSizeError (maybe hard to implement, happens when we exceed 4Gb on fat32 and stuffs...) +-InvalidFileNameError (filepath max lengths, or "? / : " characters in a windows file name...) Regards, Pascal [View Less]

7 16

Re: [Python-Dev] PEP 389: argparse - new command line parsing module
by Steven Bethard Sept. 28, 2009

Sept. 28, 2009

On Sun, Sep 27, 2009 at 8:49 PM, Benjamin Peterson <benjamin(a)python.org> wrote: > 2009/9/27 Steven Bethard <steven.bethard(a)gmail.com>: >> On Sun, Sep 27, 2009 at 8:41 PM, Benjamin Peterson <benjamin(a)python.org> wrote: >>> 2009/9/27 Steven Bethard <steven.bethard(a)gmail.com>: >>>> The first release where any real deprecation message would show up is >>>> Python 3.4, more than 3 years away. If you think 3 years isn't long &… [View More]

1 0