Mailman 3 August 2009 - Python-Dev

PEP 1, PEP Purpose and Guidelines
by barry＠zope.com May 18, 2021

May 18, 2021

It has been a while since I posted a copy of PEP 1 to the mailing lists and newsgroups. I've recently done some updating of a few sections, so in the interest of gaining wider community participation in the Python development process, I'm posting the latest revision of PEP 1 here. A version of the PEP is always available on-line at http://www.python.org/peps/pep-0001.html Enjoy, -Barry -------------------- snip snip -------------------- PEP: 1 Title: PEP Purpose and Guidelines Version: $Revision: 1.36 $ Last-Modified: $Date: 2002/07/29 18:34:59 $ Author: Barry A. Warsaw, Jeremy Hylton Status: Active Type: Informational Created: 13-Jun-2000 Post-History: 21-Mar-2001, 29-Jul-2002 What is a PEP? PEP stands for Python Enhancement Proposal. A PEP is a design document providing information to the Python community, or describing a new feature for Python. The PEP should provide a concise technical specification of the feature and a rationale for the feature. We intend PEPs to be the primary mechanisms for proposing new features, for collecting community input on an issue, and for documenting the design decisions that have gone into Python. The PEP author is responsible for building consensus within the community and documenting dissenting opinions. Because the PEPs are maintained as plain text files under CVS control, their revision history is the historical record of the feature proposal[1]. Kinds of PEPs There are two kinds of PEPs. A standards track PEP describes a new feature or implementation for Python. An informational PEP describes a Python design issue, or provides general guidelines or information to the Python community, but does not propose a new feature. Informational PEPs do not necessarily represent a Python community consensus or recommendation, so users and implementors are free to ignore informational PEPs or follow their advice. PEP Work Flow The PEP editor, Barry Warsaw <peps(a)python.org>, assigns numbers for each PEP and changes its status. The PEP process begins with a new idea for Python. It is highly recommended that a single PEP contain a single key proposal or new idea. The more focussed the PEP, the more successfully it tends to be. The PEP editor reserves the right to reject PEP proposals if they appear too unfocussed or too broad. If in doubt, split your PEP into several well-focussed ones. Each PEP must have a champion -- someone who writes the PEP using the style and format described below, shepherds the discussions in the appropriate forums, and attempts to build community consensus around the idea. The PEP champion (a.k.a. Author) should first attempt to ascertain whether the idea is PEP-able. Small enhancements or patches often don't need a PEP and can be injected into the Python development work flow with a patch submission to the SourceForge patch manager[2] or feature request tracker[3]. The PEP champion then emails the PEP editor <peps(a)python.org> with a proposed title and a rough, but fleshed out, draft of the PEP. This draft must be written in PEP style as described below. If the PEP editor approves, he will assign the PEP a number, label it as standards track or informational, give it status 'draft', and create and check-in the initial draft of the PEP. The PEP editor will not unreasonably deny a PEP. Reasons for denying PEP status include duplication of effort, being technically unsound, not providing proper motivation or addressing backwards compatibility, or not in keeping with the Python philosophy. The BDFL (Benevolent Dictator for Life, Guido van Rossum) can be consulted during the approval phase, and is the final arbitrator of the draft's PEP-ability. If a pre-PEP is rejected, the author may elect to take the pre-PEP to the comp.lang.python newsgroup (a.k.a. python-list(a)python.org mailing list) to help flesh it out, gain feedback and consensus from the community at large, and improve the PEP for re-submission. The author of the PEP is then responsible for posting the PEP to the community forums, and marshaling community support for it. As updates are necessary, the PEP author can check in new versions if they have CVS commit permissions, or can email new PEP versions to the PEP editor for committing. Standards track PEPs consists of two parts, a design document and a reference implementation. The PEP should be reviewed and accepted before a reference implementation is begun, unless a reference implementation will aid people in studying the PEP. Standards Track PEPs must include an implementation - in the form of code, patch, or URL to same - before it can be considered Final. PEP authors are responsible for collecting community feedback on a PEP before submitting it for review. A PEP that has not been discussed on python-list(a)python.org and/or python-dev(a)python.org will not be accepted. However, wherever possible, long open-ended discussions on public mailing lists should be avoided. Strategies to keep the discussions efficient include, setting up a separate SIG mailing list for the topic, having the PEP author accept private comments in the early design phases, etc. PEP authors should use their discretion here. Once the authors have completed a PEP, they must inform the PEP editor that it is ready for review. PEPs are reviewed by the BDFL and his chosen consultants, who may accept or reject a PEP or send it back to the author(s) for revision. Once a PEP has been accepted, the reference implementation must be completed. When the reference implementation is complete and accepted by the BDFL, the status will be changed to `Final.' A PEP can also be assigned status `Deferred.' The PEP author or editor can assign the PEP this status when no progress is being made on the PEP. Once a PEP is deferred, the PEP editor can re-assign it to draft status. A PEP can also be `Rejected'. Perhaps after all is said and done it was not a good idea. It is still important to have a record of this fact. PEPs can also be replaced by a different PEP, rendering the original obsolete. This is intended for Informational PEPs, where version 2 of an API can replace version 1. PEP work flow is as follows: Draft -> Accepted -> Final -> Replaced ^ +----> Rejected v Deferred Some informational PEPs may also have a status of `Active' if they are never meant to be completed. E.g. PEP 1. What belongs in a successful PEP? Each PEP should have the following parts: 1. Preamble -- RFC822 style headers containing meta-data about the PEP, including the PEP number, a short descriptive title (limited to a maximum of 44 characters), the names, and optionally the contact info for each author, etc. 2. Abstract -- a short (~200 word) description of the technical issue being addressed. 3. Copyright/public domain -- Each PEP must either be explicitly labelled as placed in the public domain (see this PEP as an example) or licensed under the Open Publication License[4]. 4. Specification -- The technical specification should describe the syntax and semantics of any new language feature. The specification should be detailed enough to allow competing, interoperable implementations for any of the current Python platforms (CPython, JPython, Python .NET). 5. Motivation -- The motivation is critical for PEPs that want to change the Python language. It should clearly explain why the existing language specification is inadequate to address the problem that the PEP solves. PEP submissions without sufficient motivation may be rejected outright. 6. Rationale -- The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work, e.g. how the feature is supported in other languages. The rationale should provide evidence of consensus within the community and discuss important objections or concerns raised during discussion. 7. Backwards Compatibility -- All PEPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. The PEP must explain how the author proposes to deal with these incompatibilities. PEP submissions without a sufficient backwards compatibility treatise may be rejected outright. 8. Reference Implementation -- The reference implementation must be completed before any PEP is given status 'Final,' but it need not be completed before the PEP is accepted. It is better to finish the specification and rationale first and reach consensus on it before writing code. The final implementation must include test code and documentation appropriate for either the Python language reference or the standard library reference. PEP Template PEPs are written in plain ASCII text, and should adhere to a rigid style. There is a Python script that parses this style and converts the plain text PEP to HTML for viewing on the web[5]. PEP 9 contains a boilerplate[7] template you can use to get started writing your PEP. Each PEP must begin with an RFC822 style header preamble. The headers must appear in the following order. Headers marked with `*' are optional and are described below. All other headers are required. PEP: <pep number> Title: <pep title> Version: <cvs version string> Last-Modified: <cvs date string> Author: <list of authors' real names and optionally, email addrs> * Discussions-To: <email address> Status: <Draft | Active | Accepted | Deferred | Final | Replaced> Type: <Informational | Standards Track> * Requires: <pep numbers> Created: <date created on, in dd-mmm-yyyy format> * Python-Version: <version number> Post-History: <dates of postings to python-list and python-dev> * Replaces: <pep number> * Replaced-By: <pep number> The Author: header lists the names and optionally, the email addresses of all the authors/owners of the PEP. The format of the author entry should be address(a)dom.ain (Random J. User) if the email address is included, and just Random J. User if the address is not given. If there are multiple authors, each should be on a separate line following RFC 822 continuation line conventions. Note that personal email addresses in PEPs will be obscured as a defense against spam harvesters. Standards track PEPs must have a Python-Version: header which indicates the version of Python that the feature will be released with. Informational PEPs do not need a Python-Version: header. While a PEP is in private discussions (usually during the initial Draft phase), a Discussions-To: header will indicate the mailing list or URL where the PEP is being discussed. No Discussions-To: header is necessary if the PEP is being discussed privately with the author, or on the python-list or python-dev email mailing lists. Note that email addresses in the Discussions-To: header will not be obscured. Created: records the date that the PEP was assigned a number, while Post-History: is used to record the dates of when new versions of the PEP are posted to python-list and/or python-dev. Both headers should be in dd-mmm-yyyy format, e.g. 14-Aug-2001. PEPs may have a Requires: header, indicating the PEP numbers that this PEP depends on. PEPs may also have a Replaced-By: header indicating that a PEP has been rendered obsolete by a later document; the value is the number of the PEP that replaces the current document. The newer PEP must have a Replaces: header containing the number of the PEP that it rendered obsolete. PEP Formatting Requirements PEP headings must begin in column zero and the initial letter of each word must be capitalized as in book titles. Acronyms should be in all capitals. The body of each section must be indented 4 spaces. Code samples inside body sections should be indented a further 4 spaces, and other indentation can be used as required to make the text readable. You must use two blank lines between the last line of a section's body and the next section heading. You must adhere to the Emacs convention of adding two spaces at the end of every sentence. You should fill your paragraphs to column 70, but under no circumstances should your lines extend past column 79. If your code samples spill over column 79, you should rewrite them. Tab characters must never appear in the document at all. A PEP should include the standard Emacs stanza included by example at the bottom of this PEP. A PEP must contain a Copyright section, and it is strongly recommended to put the PEP in the public domain. When referencing an external web page in the body of a PEP, you should include the title of the page in the text, with a footnote reference to the URL. Do not include the URL in the body text of the PEP. E.g. Refer to the Python Language web site [1] for more details. ... [1] http://www.python.org When referring to another PEP, include the PEP number in the body text, such as "PEP 1". The title may optionally appear. Add a footnote reference that includes the PEP's title and author. It may optionally include the explicit URL on a separate line, but only in the References section. Note that the pep2html.py script will calculate URLs automatically, e.g.: ... Refer to PEP 1 [7] for more information about PEP style ... References [7] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton http://www.python.org/peps/pep-0001.html If you decide to provide an explicit URL for a PEP, please use this as the URL template: http://www.python.org/peps/pep-xxxx.html PEP numbers in URLs must be padded with zeros from the left, so as to be exactly 4 characters wide, however PEP numbers in text are never padded. Reporting PEP Bugs, or Submitting PEP Updates How you report a bug, or submit a PEP update depends on several factors, such as the maturity of the PEP, the preferences of the PEP author, and the nature of your comments. For the early draft stages of the PEP, it's probably best to send your comments and changes directly to the PEP author. For more mature, or finished PEPs you may want to submit corrections to the SourceForge bug manager[6] or better yet, the SourceForge patch manager[2] so that your changes don't get lost. If the PEP author is a SF developer, assign the bug/patch to him, otherwise assign it to the PEP editor. When in doubt about where to send your changes, please check first with the PEP author and/or PEP editor. PEP authors who are also SF committers, can update the PEPs themselves by using "cvs commit" to commit their changes. Remember to also push the formatted PEP text out to the web by doing the following: % python pep2html.py -i NUM where NUM is the number of the PEP you want to push out. See % python pep2html.py --help for details. Transferring PEP Ownership It occasionally becomes necessary to transfer ownership of PEPs to a new champion. In general, we'd like to retain the original author as a co-author of the transferred PEP, but that's really up to the original author. A good reason to transfer ownership is because the original author no longer has the time or interest in updating it or following through with the PEP process, or has fallen off the face of the 'net (i.e. is unreachable or not responding to email). A bad reason to transfer ownership is because you don't agree with the direction of the PEP. We try to build consensus around a PEP, but if that's not possible, you can always submit a competing PEP. If you are interested assuming ownership of a PEP, send a message asking to take over, addressed to both the original author and the PEP editor <peps(a)python.org>. If the original author doesn't respond to email in a timely manner, the PEP editor will make a unilateral decision (it's not like such decisions can be reversed. :). References and Footnotes [1] This historical record is available by the normal CVS commands for retrieving older revisions. For those without direct access to the CVS tree, you can browse the current and past PEP revisions via the SourceForge web site at http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi/python/nondist/peps/?cvsroot=… [2] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [3] http://sourceforge.net/tracker/?atid=355470&group_id=5470&func=browse [4] http://www.opencontent.org/openpub/ [5] The script referred to here is pep2html.py, which lives in the same directory in the CVS tree as the PEPs themselves. Try "pep2html.py --help" for details. The URL for viewing PEPs on the web is http://www.python.org/peps/ [6] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [7] PEP 9, Sample PEP Template http://www.python.org/peps/pep-0009.html Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:

8 14

Reviving restricted mode?
by Guido van Rossum Aug. 14, 2014

Aug. 14, 2014

I've received some enthusiastic emails from someone who wants to revive restricted mode. He started out with a bunch of patches to the CPython runtime using ctypes, which he attached to an App Engine bug: http://code.google.com/p/googleappengine/issues/detail?id=671 Based on his code (the file secure.py is all you need, included in secure.tar.gz) it seems he believes the only security leaks are __subclasses__, gi_frame and gi_code. (I have since convinced him that if we add "restricted" guards to these attributes, he doesn't need the functions added to sys.) I don't recall the exploits that Samuele once posted that caused the death of rexec.py -- does anyone recall, or have a pointer to the threads? -- --Guido van Rossum (home page: http://www.python.org/~guido/)

19 35

PEP 382: Namespace Packages
by "Martin v. Löwis" Nov. 4, 2009

Nov. 4, 2009

I propose the following PEP for inclusion to Python 3.1. Please comment. Regards, Martin Abstract ======== Namespace packages are a mechanism for splitting a single Python package across multiple directories on disk. In current Python versions, an algorithm to compute the packages __path__ must be formulated. With the enhancement proposed here, the import machinery itself will construct the list of directories that make up the package. Terminology =========== Within this PEP, the term package refers to Python packages as defined by Python's import statement. The term distribution refers to separately installable sets of Python modules as stored in the Python package index, and installed by distutils or setuptools. The term vendor package refers to groups of files installed by an operating system's packaging mechanism (e.g. Debian or Redhat packages install on Linux systems). The term portion refers to a set of files in a single directory (possibly stored in a zip file) that contribute to a namespace package. Namespace packages today ======================== Python currently provides the pkgutil.extend_path to denote a package as a namespace package. The recommended way of using it is to put:: from pkgutil import extend_path __path__ = extend_path(__path__, __name__) int the package's ``__init__.py``. Every distribution needs to provide the same contents in its ``__init__.py``, so that extend_path is invoked independent of which portion of the package gets imported first. As a consequence, the package's ``__init__.py`` cannot practically define any names as it depends on the order of the package fragments on sys.path which portion is imported first. As a special feature, extend_path reads files named ``*.pkg`` which allow to declare additional portions. setuptools provides a similar function pkg_resources.declare_namespace that is used in the form:: import pkg_resources pkg_resources.declare_namespace(__name__) In the portion's __init__.py, no assignment to __path__ is necessary, as declare_namespace modifies the package __path__ through sys.modules. As a special feature, declare_namespace also supports zip files, and registers the package name internally so that future additions to sys.path by setuptools can properly add additional portions to each package. setuptools allows declaring namespace packages in a distribution's setup.py, so that distribution developers don't need to put the magic __path__ modification into __init__.py themselves. Rationale ========= The current imperative approach to namespace packages has lead to multiple slightly-incompatible mechanisms for providing namespace packages. For example, pkgutil supports ``*.pkg`` files; setuptools doesn't. Likewise, setuptools supports inspecting zip files, and supports adding portions to its _namespace_packages variable, whereas pkgutil doesn't. In addition, the current approach causes problems for system vendors. Vendor packages typically must not provide overlapping files, and an attempt to install a vendor package that has a file already on disk will fail or cause unpredictable behavior. As vendors might chose to package distributions such that they will end up all in a single directory for the namespace package, all portions would contribute conflicting __init__.py files. Specification ============= Rather than using an imperative mechanism for importing packages, a declarative approach is proposed here, as an extension to the existing ``*.pkg`` mechanism. The import statement is extended so that it directly considers ``*.pkg`` files during import; a directory is considered a package if it either contains a file named __init__.py, or a file whose name ends with ".pkg". In addition, the format of the ``*.pkg`` file is extended: a line with the single character ``*`` indicates that the entire sys.path will be searched for portions of the namespace package at the time the namespace packages is imported. Importing a package will immediately compute the package's __path__; the ``*.pkg`` files are not considered anymore after the initial import. If a ``*.pkg`` package contains an asterisk, this asterisk is prepended to the package's __path__ to indicate that the package is a namespace package (and that thus further extensions to sys.path might also want to extend __path__). At most one such asterisk gets prepended to the path. extend_path will be extended to recognize namespace packages according to this PEP, and avoid adding directories twice to __path__. No other change to the importing mechanism is made; searching modules (including __init__.py) will continue to stop at the first module encountered. Discussion ========== With the addition of ``*.pkg`` files to the import mechanism, namespace packages can stop filling out the namespace package's __init__.py. As a consequence, extend_path and declare_namespace become obsolete. It is recommended that distributions put a file <distribution>.pkg into their namespace packages, with a single asterisk. This allows vendor packages to install multiple portions of namespace package into a single directory, with no risk of overlapping files. Namespace packages can start providing non-trivial __init__.py implementations; to do so, it is recommended that a single distribution provides a portion with just the namespace package's __init__.py (and potentially other modules that belong to the namespace package proper). The mechanism is mostly compatible with the existing namespace mechanisms. extend_path will be adjusted to this specification; any other mechanism might cause portions to get added twice to __path__. Copyright ========= This document has been placed in the public domain.

31 110

PEP 3144: IP Address Manipulation Library for the Python Standard Library
by Peter Moody Sept. 27, 2009

Sept. 27, 2009

Howdy folks, I have a first draft of a PEP for including an IP address manipulation library in the python stdlib. It seems like there are a lot of really smart folks with some, ahem, strong ideas about what an IP address module should and shouldn't be so I wanted to solicit your input on this pep. the pep can be found here: http://www.python.org/dev/peps/pep-3144/ the code can be found here: http://ipaddr-py.googlecode.com/svn/branches/2.0.x/ Please let me know if you have any comments (some already coming :) Cheers, /peter

22 76

how to debug httplib slowness
by Chris Withers Sept. 17, 2009

Sept. 17, 2009

Hi All, I'd like to work on this issue: http://bugs.python.org/issue2576 Specifically, in my case, while IE can download a 150Mb file from a local server in about 3 seconds, httplib takes over 20 minutes! However, I'm kinda stumped on where to start with debugging the difference. I've tried upping the buffer size as suggested in the issue, but it's had no effect... Any ideas? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

7 19

pthreads, fork, import, and execvp
by Rotem Yaari Sept. 16, 2009

Sept. 16, 2009

Hello everyone! We have been encountering several deadlocks in a threaded Python application which calls subprocess.Popen (i.e. fork()) in some of its threads. This has occurred on Python 2.4.1 on a 2.4.27 Linux kernel. Preliminary analysis of the hang shows that the child process blocks upon entering the execvp function, in which the import_lock is acquired due to the following line: def _ execvpe(file, args, env=None): from errno import ENOENT, ENOTDIR ... It is known that when forking from a pthreaded application, acquisition attempts on locks which were already locked by other threads while fork() was called will deadlock. Due to these oddities we were wondering if it would be better to extract the above import line from the execvpe call, to prevent lock acquisition attempts in such cases. Another workaround could be re-assigning a new lock to import_lock (such a thing is done with the global interpreter lock) at PyOS_AfterFork or pthread_atfork. We'd appreciate any opinions you might have on the subject. Thanks in advance, Yair and Rotem

12 29

compiling python2.5 on linux under wine
by Luke Kenneth Casson Leighton Sept. 8, 2009

Sept. 8, 2009

hey, has anyone investigated compiling python2.5 using winegcc, under wine? i'm presently working my way through it, just for kicks, and was wondering if anyone would like to pitch in or stare at the mess under a microscope. it's not as crazed as it sounds. cross-compiling python2.5 for win32 with mingw32 is an absolute miserable bitch of a job that goes horribly wrong when you actually try to use the minimalist compiler to do any real work. so i figured that it would be easier to get python compiled using wine. i _have_ got some success - a python script and a python.exe.so (which is winegcc's friendly way of telling you you have something that stands a chance of working) as well as a libpython25.dll.so. what i _don't_ yet have is an _md5.dll (or should it be _md5.lib?) i.e. the standard modules are a bit... iffy. the _winreg.o is compiled; the _md5.o is compiled; the winreg.lib is not. whoops. plus, it's necessary to enable nt_dl.c which is in PC/ _not_ in Modules/. one of the key issues that's a bit of a bitch is that python is compiled up for win32 with a hard-coded pyconfig.h which someone went to a _lot_ of trouble to create by hand instead of using autoconf. oh - and it uses visualstudio so there's not even a Makefile. ignoring that for the time-being was what allowed me to get as far as actually having a python interpreter (with no c-based modules). so there's a whole _stack_ of stuff that needs dragging kicking and screaming into the 21st century. there _is_ a reason why i want to do this. actually, there's two. firstly, i sure as shit do _not_ want to buy, download, install _or_ run visual studio. i flat-out refuse to run an MS os and visual studio runs like a dog under wine. secondly, i want a python25.lib which i can use to cross-compile modules for poor windows users _despite_ sticking to my principles and keeping my integrity as a free software developer. thirdly i'd like to cross-compile pywebkitgtk for win32 fourthly i'd like to compile and link applications to the extremely successful and well wicked MSHTML.DLL... in the _wine_ project :) not the one in windows (!) i want to experiment with DOM model manipulation - from python - similar to the OLPC HulaHop project - _but_ i want to compile or cross-compile everything from linux, not windows (see 1 above) fifthly i'd like to see COM (DCOM) working and pywin32 compiled and useable under wine, even if it means having to get a license to use dcom98 and oleauth.lib and oleauth.h etc. and all the developer files needed to link DCOM applications under windows. actually what i'd _really_ like to see is FreeDCE's DCOM work actually damn well finished, it's only been eight years since wez committed the first versions of the IDL and header files, and it's only been over fifteen years since microsoft began its world domination using COM and DCOM. ... but that's another story :) so that's ... five reasons not two. if anyone would like to collaborate on a crazed project with someone who can't count, i'm happy to make available what i've got up to so far, on github.org. l.

6 12

Mercurial migration: help needed
by "Martin v. Löwis" Sept. 6, 2009

Sept. 6, 2009

This is a repost from two weeks ago. It didn't get much feedback last time. I still keep trying, reposting to python-list also this time. In this thread, I'd like to collect things that ought to be done but where Dirkjan has indicated that he would prefer if somebody else did it. Item 1 ------ The first item is build identification. If you want to work on this, please either provide a patch (for trunk and/or py3k), or (if you are a committer) create a subversion branch. It seems that Barry and I agree that for the maintenance branches, sys.subversion should be frozen, so we need actually two sets of patches: one that removes sys.subversion entirely, and the other that freezes the branch to the respective one, and freezes the subversion revision to None. The patch should consider what Dirkjan proposes as the branching strategy: clones to separate 2.x and 3.x, as well as for features, and branches with the clones for releases and maintenance (see the PEP for details). Anybody working on this should have good knowledge of the Python source code, Mercurial, and either autoconf or Visual Studio (preferably both). Item 2 ------ The second item is line conversion hooks. Dj Gilcrease has posted a solution which he considers a hack himself. Mark Hammond has also volunteered, but it seems some volunteer needs to be "in charge", keeping track of a proposed solution until everybody agrees that it is a good solution. It may be that two solutions are necessary: a short-term one, that operates as a hook and has limitations, and a long-term one, that improves the hook system of Mercurial to implement the proper functionality (which then might get shipped with Mercurial in a cross-platform manner). Regards, Martin

15 77

default of returning None hurts performance?
by Gregory P. Smith Sept. 4, 2009

Sept. 4, 2009

food for thought as noticed by a coworker who has been profiling some hot code to optimize a library... If a function does not have a return statement we return None. Ironically this makes the foo2 function below faster than the bar2 function at least as measured using bytecode size: Python 2.6.2 (r262:71600, Jul 24 2009, 17:29:21) [GCC 4.2.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import dis >>> def foo(x): ... y = x() ... return y ... >>> def foo2(x): ... return x() ... >>> def bar(x): ... y = x() ... >>> def bar2(x): ... x() ... >>> dis.dis(foo) 2 0 LOAD_FAST 0 (x) 3 CALL_FUNCTION 0 6 STORE_FAST 1 (y) 3 9 LOAD_FAST 1 (y) 12 RETURN_VALUE >>> dis.dis(foo2) 2 0 LOAD_FAST 0 (x) 3 CALL_FUNCTION 0 6 RETURN_VALUE >>> dis.dis(bar) 2 0 LOAD_FAST 0 (x) 3 CALL_FUNCTION 0 6 STORE_FAST 1 (y) 9 LOAD_CONST 0 (None) 12 RETURN_VALUE >>> dis.dis(bar2) 2 0 LOAD_FAST 0 (x) 3 CALL_FUNCTION 0 6 POP_TOP 7 LOAD_CONST 0 (None) 10 RETURN_VALUE

14 24

Setting up a buildbot
by Paul Moore Sept. 2, 2009

Sept. 2, 2009

I've just had a look on python.org, but couldn't immediately see a pointer to instructions on what the process is to set up a buildbot. There's a not on setting things up for pybots, but nothing on the core buildbot setup. The reason I'm asking is that I'm thinking of seeing if I could set up a Windows buildbot of some sort, to offer extra coverage. It's early days, yet, but I wonder if someone could answer a few questions for me: - Is there any documentation on how to set up a buildbot? If so, can someone give me a pointer? - What configurations would be most useful? (I've got a 64-bit PC, so I can theoretically set up 32 or 64 bit VMs with VMWare, and with my shiny new MSDN subscription, I can set up whatever OS is most useful). - Is it possible to set up the pull/build/test side of the process separately, before linking it into the full buildbot farm? That would let me try things out on my own, and iron out any configuration glitches before dumping it on the world. Thanks for any pointers. It's early days yet, so it may be a while before I have anything properly set up, but I'd like to see what I can do. Paul.

3 4

Re: [Python-Dev] Support for Encrypted Zip as python scripts
by Brett Cannon Sept. 1, 2009

Sept. 1, 2009

There is a standard for encrypting entire zip files. And I was looking at the zip docs the other day and zipfile can already decrypt but not encrypt (assuming my memory is accurate; doing this from my phone on vacation). On Aug 23, 2009 2:10 PM, "Guido van Rossum" <guido(a)python.org> wrote: On Sun, Aug 23, 2009 at 9:09 AM, Shashank Singh< shashank.sunny.singh(a)gmail.com> wrote: > There is an... MvL already asked for a patch so I suppose that means he thinks it's useful. Personally I've never encountered an encrypted zipfile, so I just have questions: is there a standard encryption algorithm? What is encrypted? The entire file or individual members? How are you supposed to give the password? Also, I suppose there could be (US) export problems with the code, so it would have to be optional (and we might not be able to build it into binaries we distribute from python.org). -- --Guido van Rossum (home page: http://www.python.org/~guido/) _______________________________________________ Python-Dev mailing list Python-Dev(a)python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org

7 19

how important is setting co_filename for a module being imported to what __file__ is set to?
by Brett Cannon Sept. 1, 2009

Sept. 1, 2009

I am going through and running the entire test suite using importlib to ferret out incompatibilities. I have found a bunch, although all rather minor (raising a different exception typically; not even sure they are worth backporting as anyone reliant on the old exceptions might get a nasty surprise in the next micro release), and now I am down to my last failing test suite: test_import. Ignoring the execution bit problem (http://bugs.python.org/issue6526 but I have no clue why this is happening), I am bumping up against TestPycRewriting.test_incorrect_code_name. Turns out that import resets co_filename on a code object to __file__ before exec'ing it to create a module's namespace in order to ignore the file name passed into compile() for the filename argument. Now I can't change co_filename from Python as it's a read-only attribute and thus can't match this functionality in importlib w/o creating some custom code to allow me to specify the co_filename somewhere (marshal.loads() or some new function). My question is how important is this functionality? Do I really need to go through and add an argument to marshal.loads or some new function just to set co_filename to something that someone explicitly set in a .pyc file? Or I can let this go and have this be the one place where builtins.__import__ and importlib.__import__ differ and just not worry about it? -Brett

10 30

3to2 0.1 alpha 1 released
by Joe Amenta Sept. 1, 2009

Sept. 1, 2009

Hello all, I have released the first alpha version of 3to2 after finishing it for my Google Summer of Code 2009(tm) project. You can get the tarball for this release at http://bitbucket.org/amentajo/lib3to2/downloads/3to2_0.1-alpha1.tar.gz. This requires python 2.7, because it requires a newer version of 2to3 than what comes with 2.6. Release notes are in the RELEASE file. Development happens at http://bitbucket.org/amentajo/lib3to2/, and the source code for this release lives at http://bitbucket.org/amentajo/3to2-01-alpha-1. Report bugs at http://bitbucket.org/amentajo/lib3to2/issues/, please. Additional notes and comments can (for now) be found at http://www.startcodon.com/wordpress/?cat=4. --Joe

9 19

Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is set to?
by P.J. Eby Aug. 31, 2009

Aug. 31, 2009

At 02:57 PM 8/31/2009 -0700, Brett Cannon wrote: >Ignoring that 'new' is not in Python 3.x (luckily 'types' is), I want >a proper solution that doesn't require reconstructing every code >object that I happen to import. Practicality beats purity. ;-) (Especially if it allows importlib to run on older Pythons.) Also, surely you're not worried about *performance* here?

2 1

Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is set to?
by P.J. Eby Aug. 31, 2009

Aug. 31, 2009

At 01:39 PM 8/31/2009 -0700, Brett Cannon wrote: >On Mon, Aug 31, 2009 at 12:59, Brett Cannon<brett(a)python.org> wrote: > > On Mon, Aug 31, 2009 at 12:27, Antoine Pitrou<solipsis(a)pitrou.net> wrote: > >> Brett Cannon <brett <at> python.org> writes: > >>> > >>> I will plan to take this approach then; > >>> http://bugs.python.org/issue6811 will track all of this. Since this is > >>> a 3.2 thing I am not going to rush to implement this. > >> > >> I still don't understand what the point is of this complicated > approach (adding > >> an argument to marshal.load()) compared to the simple and obvious approach > >> (making co_filename mutable). > > > > If we add the argument to marshal.load* we can eventually drop the > > file location string from marshal data entirely by requiring people to > > specify the filename to use when the code object is created. Making > > co_filename mutable simply doesn't allow for this case unless we > > decide a default value should be used instead. > > > >I should also mention that I am +0 on the marshal.load* change. I >could be convinced to try to pursue a mutable co_filenme direction, >but considering the BDFL likes the marshal.load* approach and it opens >up the possibility of compacting the marshal format I am leaning >towards sticking with this initial direction. Why not just try the code I posted earlier, that doesn't need a mutable attribute OR an API change?

2 1

Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is set to?
by P.J. Eby Aug. 31, 2009

Aug. 31, 2009

At 09:33 AM 8/31/2009 -0700, Guido van Rossum wrote: >Of course, tracking down all the code objects in the return value of >marshal.load*() might be a bit tricky -- API-wise I still think that >making it an argument to marshal.load*() might be simpler. Also it >would preserve the purity of code objects. Or maybe we could just do something like this: from new import code def with_changed_filename(code_ob, filename): def remap(ob): if not isinstance(ob, code): return ob return code( ob.co_argcount, ob.co_nlocals, ob.co_stacksize, ob.co_flags, ob.co_code, map(remap, ob.co_consts), ob.co_names, ob.co_varnames, filename, ob.co_name, ob.co_firstlineno, ob.co_lnotab, ob.co_freevars, ob.co_cellvars ) return remap(code_ob) Granted, this takes a bit more memory than an in-place modification, but it's immediately usable and at least works wherever new.code is available. (I've not tested the above, so it may not work. I seem to recall the last time I wrote something like this there was something tricky about handling co_freevars and co_cellvars; I think you may need to omit them if empty, or convert them to None, or from None to an empty tuple or some such rigamarole. And a 3.x version is left as an exercise for the reader. ;-) )

1 0

deleting setdefaultencoding iin site.py is evil
by Chris Withers Aug. 31, 2009

Aug. 31, 2009

Hi All, Would anyone object if I removed the deletion of of sys.setdefaultencoding in site.py? I'm guessing "yes!" so thought I'd state my reasons now: This deletion appears to be pretty flimsy; reload(sys) and you have it back. Which is lucky, because I need it after it's been deleted... Why? Well, because you can no longer put sitecustomize.py in a project-specific location (http://bugs.python.org/issue1734860) and because for some projects the only way I can deal with encoded strings sensibly is to use setdefaultencoding, in my case at the start of a script generated by zc.buildout's zc.recipe.egg (I *know* all the encodings in this project are utf-8, but I don't want to go playing whack-a-mole with whatever modules this rather large project uses that haven't been made properly unicode aware). Yes, it needs to be used as early as possible, and the docs should say this, but deleting it seems to be petty in terms of stopping its use when sitecustomize.py is too early and too system-wide and spraying .decode('utf-8')'s all over a code base made up of a load of eggs managed by buildout simply isn't feasible... Thoughts? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

9 27

Excluding the current path from module search path?
by Chris Withers Aug. 31, 2009

Aug. 31, 2009

Hi All, I'm being bitten by this issue: http://bugs.python.org/issue1734860 I'm not sure I agree with Daniel's closing of it so thought I'd ask here... Am I right in thinking that the general idea is that "the current working directory at the time of invoking a script or interpreter ends up on the python path" or should I be thinking "the directory that a script exists in should end up on the python path"? If the latter, then what happens in the case of just starting up an interpreter? If neither, then how come when I have two .py files in a directory, I can import one as a module from the other? In any case, as a parting comment, http://bugs.python.org/issue1232023 seems to have been committed with no tests and the only documentation being a one liner in the NEWS.txt file. Was there other discussion of this? (Incidentally, export PYTHONPATH= or its Windows equivalent circumvents whatever the patch was trying to achieve, so the change doesn't seem to make sense anyway...) cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

5 7

Fast Implementation for ZIP decryption
by Shashank Singh Aug. 31, 2009

Aug. 31, 2009

Hi All, I have implemented the simple zip decryption in C (yes, the much loathed weak password based classical PKWARE encryption, which incidentally is the only one currently supported in python) . It performs fast, as one would expect, as compared to the current all-python implementation. Does it sound worthy enough to create a patch for and integrate into python itself? -- Regards Shashank Singh Senior Undergraduate, Department of Computer Science and Engineering Indian Institute of Technology Bombay shashank.sunny.singh(a)gmail.com http://www.cse.iitb.ac.in/~shashanksingh

9 10

timed_command.py: consider for inclusion in std. library?
by Charles Waldman Aug. 28, 2009

Aug. 28, 2009

Here's a module "timed_command" I wrote a while ago and is generally useful and might be a good addition to the standard library. It is like commands.getstatusoutput but lets you run a command with an optional timeout. Useful for systems programming where a sub-process might hang. Only works on POSIX, but could perhaps be modified to run on other platforms (I don't have the knowledge of Windows to do this). If you would like to add this to the library, I relinquish all rights to it. Here's a link to the source repository: http://repo.mwt2.org/viewvc/Python/timed_command.py?view=markup

2 1

deprecated methods on array objects
by Jake McGuire Aug. 28, 2009

Aug. 28, 2009

The python documentation says that the read() and write() methods on array objects have been deprecated since 1.5.1. I assume this is because their semantics are almost the exact opposite of read() and write() on a file-like object; array.read() reads data from a file into the array and array.write() writes data from the array to a file. This causes fatal confusion in code that checks for the existence of read() and write() to determine whether an object is file-like. Code such as httplib. What is the timeline for removing these methods from array? It has been 11 years now. -jake

2 1

Re: [Python-Dev] functools.compose to chain functions together
by Peter Harris Aug. 28, 2009

Aug. 28, 2009

I am personally indifferent to this, even though I had in mind in PEP309, that compose would probably end up in there too. On the one hand, some people will keep on expecting it to be there. The ones that care about it will not be confused: they'll expect compose(f,g)(x) to be f(g(x)) as is proposed. It can't do any significant harm. On the other hand it's not likely to be used even as often as partial, which I always wanted mostly to make anonymous callables for Tkinter, not because of any ivory-tower functional programming bias. And the most common use case of compose() is covered by a one-liner that really doesn't need to be in the standard library. I'll say +0, with the + because if new Python programmers run across compose() in the docs, and aren't familiar with the idea, they can follow a link from there to Wikipedia, and maybe it will give them an idea we haven't thought of for something cool to do with it. Peter Harris

1 0

Summary of Python tracker Issues
by Python tracker Aug. 28, 2009

Aug. 28, 2009

ACTIVITY SUMMARY (08/21/09 - 08/28/09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2370 open (+23) / 16247 closed (+15) / 18617 total (+38) Open issues with patches: 935 Average duration of open issues: 657 days. Median duration of open issues: 410 days. Open Issues Breakdown open 2339 (+23) pending 30 ( +0) Issues Created Or Reopened (39) _______________________________ add conversion table to time module docs 08/22/09 http://bugs.python.org/issue5365 reopened cvrebert implement new setuid-related calls and a standard way to drop al 08/21/09 CLOSED http://bugs.python.org/issue6758 created solinym zipfile.ZipExtFile.read() is missing universal newline support 08/22/09 http://bugs.python.org/issue6759 created ryles patch patch to subprocess docs to better explain Popen's 'args' argume 08/22/09 http://bugs.python.org/issue6760 created cvrebert patch Class calling 08/22/09 http://bugs.python.org/issue6761 created onlyme strange string representation of xrange in print 08/22/09 CLOSED http://bugs.python.org/issue6762 created mintaka Crash on mac os x leopard in mimetypes.guess_type (or PyObject_M 08/22/09 http://bugs.python.org/issue6763 created santagada os.path.join should call os.path.normpath on result 08/22/09 CLOSED http://bugs.python.org/issue6764 created michael.foord easy math.log, log10 inconsistency 08/23/09 CLOSED http://bugs.python.org/issue6765 created steve21 Cannot modify dictionaries inside dictionaries using Managers fr 08/23/09 http://bugs.python.org/issue6766 created carlosdf Python as zip package 08/23/09 http://bugs.python.org/issue6767 created Joe asyncore file_wrapper leaking file descriptors? 08/23/09 http://bugs.python.org/issue6768 created keysers in xmlrpclib.py: NameError: global name 'HTTPSConnection' is not 08/23/09 http://bugs.python.org/issue6769 created ivank PDF download links of docs for 3.1.1 are broken 08/24/09 CLOSED http://bugs.python.org/issue6770 created Radiant documentation/implementation error 08/24/09 http://bugs.python.org/issue6771 created steve21 Missing alias utf-8 in Standard Encodings list. 08/24/09 CLOSED http://bugs.python.org/issue6772 created mintaka subprocess issue on Win 7 x64 08/24/09 http://bugs.python.org/issue6773 created tesla socket.shudown documentation: on some platforms, closing one hal 08/24/09 http://bugs.python.org/issue6774 created nicdumz patch readme: correct python.org/community/lists url 08/24/09 CLOSED http://bugs.python.org/issue6775 created nicdumz patch A few tests are failing when zlib is not supported 08/24/09 CLOSED http://bugs.python.org/issue6776 created nicdumz Python 2.6 tutorial still recommends using Exception.message att 08/24/09 http://bugs.python.org/issue6777 created cito False positives given through bisect module (binary search) 08/24/09 CLOSED http://bugs.python.org/issue6778 created kaashif NotImplementedError on for statement 08/24/09 CLOSED http://bugs.python.org/issue6779 created sbq startswith error message is incomplete 08/24/09 http://bugs.python.org/issue6780 created srid even exponentiation of negative numbers 08/25/09 CLOSED http://bugs.python.org/issue6781 created benlbroussard Scoping of variables in closures 08/25/09 CLOSED http://bugs.python.org/issue6782 created bitfort Add a builtin method to 'int' for base/radix conversion 08/25/09 http://bugs.python.org/issue6783 created ubershmekel byte/unicode pickle incompatibilities between python2 and python 08/26/09 http://bugs.python.org/issue6784 created RonnyPfannschmidt IncompleteRead / BadStatus when parsing http://peakoil.mobi 08/26/09 http://bugs.python.org/issue6785 created kmoon readline and zero based indexing 08/26/09 http://bugs.python.org/issue6786 created purpleidea thread docs contain an incorrect link in a reference to the 'exi 08/27/09 CLOSED http://bugs.python.org/issue6787 created r.david.murray codecs.open on Win32 does not force binary mode 08/27/09 http://bugs.python.org/issue6788 created EnigmaCurry ftplib storelines does not honor strings returned in fp.readline 08/27/09 http://bugs.python.org/issue6789 created aymanhs httplib and array do not play together well 08/27/09 http://bugs.python.org/issue6790 created jakemcguire easy httplib read status memory usage 08/28/09 http://bugs.python.org/issue6791 created m.sucajtys patch Distutils-based installer does not detect 64bit versions of Pyth 08/28/09 http://bugs.python.org/issue6792 created erluk decimal.py: div_nearest ==> _div_nearest 08/28/09 CLOSED http://bugs.python.org/issue6793 created skrah 26backport decimal.py: incorrect results in NaN comparisons 08/28/09 CLOSED http://bugs.python.org/issue6794 created skrah decimal.py: minor issues && usability 08/28/09 http://bugs.python.org/issue6795 created skrah Issues Now Closed (26) ______________________ Extension module build fails for MinGW: missing vcvarsall.bat 483 days http://bugs.python.org/issue2698 hagen Allow buffering for HTTPResponse 6 days http://bugs.python.org/issue4879 gregory.p.smith patch, patch let unittest.assertRaises() return the exception object caught 75 days http://bugs.python.org/issue6275 michael.foord patch, patch, easy, needs review Add "path" to the xmrlpc dispatcher method 22 days http://bugs.python.org/issue6654 krisvale buffer c-api: memoryview object documentation 20 days http://bugs.python.org/issue6659 benjamin.peterson Desire python.org documentation link to user contribution wiki ( 18 days http://bugs.python.org/issue6660 keenethery Place the term "delete" within the documentation for os.remove() 14 days http://bugs.python.org/issue6677 mcow ValueError in 21.17.4.2. SocketServer.UDPServer Example 7 days http://bugs.python.org/issue6718 georg.brandl Inconsistency in Documentation: "Name Spaces" vs "Namespaces" 6 days http://bugs.python.org/issue6725 georg.brandl dict.fromkeys() should not cross reference mutable value by defa 6 days http://bugs.python.org/issue6730 georg.brandl implement new setuid-related calls and a standard way to drop al 1 days http://bugs.python.org/issue6758 exarkun strange string representation of xrange in print 0 days http://bugs.python.org/issue6762 rhettinger os.path.join should call os.path.normpath on result 0 days http://bugs.python.org/issue6764 michael.foord easy math.log, log10 inconsistency 1 days http://bugs.python.org/issue6765 tim_one PDF download links of docs for 3.1.1 are broken 1 days http://bugs.python.org/issue6770 benjamin.peterson Missing alias utf-8 in Standard Encodings list. 0 days http://bugs.python.org/issue6772 georg.brandl readme: correct python.org/community/lists url 0 days http://bugs.python.org/issue6775 georg.brandl patch A few tests are failing when zlib is not supported 0 days http://bugs.python.org/issue6776 ezio.melotti False positives given through bisect module (binary search) 0 days http://bugs.python.org/issue6778 r.david.murray NotImplementedError on for statement 0 days http://bugs.python.org/issue6779 r.david.murray even exponentiation of negative numbers 0 days http://bugs.python.org/issue6781 marketdickinson Scoping of variables in closures 0 days http://bugs.python.org/issue6782 benjamin.peterson thread docs contain an incorrect link in a reference to the 'exi 1 days http://bugs.python.org/issue6787 georg.brandl decimal.py: div_nearest ==> _div_nearest 0 days http://bugs.python.org/issue6793 marketdickinson 26backport decimal.py: incorrect results in NaN comparisons 0 days http://bugs.python.org/issue6794 marketdickinson Wide-character curses 2357 days http://bugs.python.org/issue700921 gpolo Top Issues Most Discussed (10) ______________________________ 9 byte/unicode pickle incompatibilities between python2 and pytho 2 days open http://bugs.python.org/issue6784 9 zipfile.ZipExtFile.read() is missing universal newline support 6 days open http://bugs.python.org/issue6759 9 implement new setuid-related calls and a standard way to drop a 1 days closed http://bugs.python.org/issue6758 7 Patch: new method get_wch for ncurses bindings: accept wide cha 7 days open http://bugs.python.org/issue6755 6 Support for encrypted zipfiles when interpreting zipfile as scr 7 days open http://bugs.python.org/issue6749 6 (curses) addstr() takes str in Python 3 8 days open http://bugs.python.org/issue6745 5 expose setresuid 42 days open http://bugs.python.org/issue6508 4 Add a builtin method to 'int' for base/radix conversion 3 days open http://bugs.python.org/issue6783 4 Python as zip package 5 days open http://bugs.python.org/issue6767 4 math.log, log10 inconsistency 1 days closed http://bugs.python.org/issue6765

1 0

Adding a C Module to python source distribution
by Shashank Singh Aug. 28, 2009

Aug. 28, 2009

Hi All, I am trying to add a module written in c to python source on Win32 using VC++ 9 Pro. I went through the available documentation but there doesn't seem to be any clear instruction on how to do that. Basically I opened pcbuild.sln in vc++, added the c file (xxx.c) to Modules/ directory. Building the solution after that works fine: xxx.c is compiled (no errors, no warnings) and the python executable gets created. But I am not able to import the module defined in xxx.c using that executable. Do I need to register this module some place else too (setup.py?) ? Any hints and pointers will be appreciated :) --shashank

2 1

quick PEP 387 comments
by Brett Cannon Aug. 28, 2009

Aug. 28, 2009

Is the PEP considering all non-private APIs public even if they are not documented? If so we might want to be up front about that and say so to make sure we are all very careful about making all non-essential APIs private (assuming this PEP gets accepted). And we might want to say that all code in 'test' sub-packages are not subject to backwards compatibility unless documented. I have a ton of support code in importlib.test that I do not want to have to maintain for public consumption as they are meant solely for testing purposes by me. If you read the PEP it would suggest that all modules in test are subject to the PEP's compatibility policy which is obviously absurd. -Brett

3 2