Python-Dev July 2009

python-dev@python.org

94 participants
90 discussions

by barry＠zope.com

It has been a while since I posted a copy of PEP 1 to the mailing lists and newsgroups. I've recently done some updating of a few sections, so in the interest of gaining wider community participation in the Python development process, I'm posting the latest revision of PEP 1 here. A version of the PEP is always available on-line at http://www.python.org/peps/pep-0001.html Enjoy, -Barry -------------------- snip snip -------------------- PEP: 1 Title: PEP Purpose and Guidelines Version: $Revision: 1.36 $ Last-Modified: $Date: 2002/07/29 18:34:59 $ Author: Barry A. Warsaw, Jeremy Hylton Status: Active Type: Informational Created: 13-Jun-2000 Post-History: 21-Mar-2001, 29-Jul-2002 What is a PEP? PEP stands for Python Enhancement Proposal. A PEP is a design document providing information to the Python community, or describing a new feature for Python. The PEP should provide a concise technical specification of the feature and a rationale for the feature. We intend PEPs to be the primary mechanisms for proposing new features, for collecting community input on an issue, and for documenting the design decisions that have gone into Python. The PEP author is responsible for building consensus within the community and documenting dissenting opinions. Because the PEPs are maintained as plain text files under CVS control, their revision history is the historical record of the feature proposal[1]. Kinds of PEPs There are two kinds of PEPs. A standards track PEP describes a new feature or implementation for Python. An informational PEP describes a Python design issue, or provides general guidelines or information to the Python community, but does not propose a new feature. Informational PEPs do not necessarily represent a Python community consensus or recommendation, so users and implementors are free to ignore informational PEPs or follow their advice. PEP Work Flow The PEP editor, Barry Warsaw <peps(a)python.org>, assigns numbers for each PEP and changes its status. The PEP process begins with a new idea for Python. It is highly recommended that a single PEP contain a single key proposal or new idea. The more focussed the PEP, the more successfully it tends to be. The PEP editor reserves the right to reject PEP proposals if they appear too unfocussed or too broad. If in doubt, split your PEP into several well-focussed ones. Each PEP must have a champion -- someone who writes the PEP using the style and format described below, shepherds the discussions in the appropriate forums, and attempts to build community consensus around the idea. The PEP champion (a.k.a. Author) should first attempt to ascertain whether the idea is PEP-able. Small enhancements or patches often don't need a PEP and can be injected into the Python development work flow with a patch submission to the SourceForge patch manager[2] or feature request tracker[3]. The PEP champion then emails the PEP editor <peps(a)python.org> with a proposed title and a rough, but fleshed out, draft of the PEP. This draft must be written in PEP style as described below. If the PEP editor approves, he will assign the PEP a number, label it as standards track or informational, give it status 'draft', and create and check-in the initial draft of the PEP. The PEP editor will not unreasonably deny a PEP. Reasons for denying PEP status include duplication of effort, being technically unsound, not providing proper motivation or addressing backwards compatibility, or not in keeping with the Python philosophy. The BDFL (Benevolent Dictator for Life, Guido van Rossum) can be consulted during the approval phase, and is the final arbitrator of the draft's PEP-ability. If a pre-PEP is rejected, the author may elect to take the pre-PEP to the comp.lang.python newsgroup (a.k.a. python-list(a)python.org mailing list) to help flesh it out, gain feedback and consensus from the community at large, and improve the PEP for re-submission. The author of the PEP is then responsible for posting the PEP to the community forums, and marshaling community support for it. As updates are necessary, the PEP author can check in new versions if they have CVS commit permissions, or can email new PEP versions to the PEP editor for committing. Standards track PEPs consists of two parts, a design document and a reference implementation. The PEP should be reviewed and accepted before a reference implementation is begun, unless a reference implementation will aid people in studying the PEP. Standards Track PEPs must include an implementation - in the form of code, patch, or URL to same - before it can be considered Final. PEP authors are responsible for collecting community feedback on a PEP before submitting it for review. A PEP that has not been discussed on python-list(a)python.org and/or python-dev(a)python.org will not be accepted. However, wherever possible, long open-ended discussions on public mailing lists should be avoided. Strategies to keep the discussions efficient include, setting up a separate SIG mailing list for the topic, having the PEP author accept private comments in the early design phases, etc. PEP authors should use their discretion here. Once the authors have completed a PEP, they must inform the PEP editor that it is ready for review. PEPs are reviewed by the BDFL and his chosen consultants, who may accept or reject a PEP or send it back to the author(s) for revision. Once a PEP has been accepted, the reference implementation must be completed. When the reference implementation is complete and accepted by the BDFL, the status will be changed to `Final.' A PEP can also be assigned status `Deferred.' The PEP author or editor can assign the PEP this status when no progress is being made on the PEP. Once a PEP is deferred, the PEP editor can re-assign it to draft status. A PEP can also be `Rejected'. Perhaps after all is said and done it was not a good idea. It is still important to have a record of this fact. PEPs can also be replaced by a different PEP, rendering the original obsolete. This is intended for Informational PEPs, where version 2 of an API can replace version 1. PEP work flow is as follows: Draft -> Accepted -> Final -> Replaced ^ +----> Rejected v Deferred Some informational PEPs may also have a status of `Active' if they are never meant to be completed. E.g. PEP 1. What belongs in a successful PEP? Each PEP should have the following parts: 1. Preamble -- RFC822 style headers containing meta-data about the PEP, including the PEP number, a short descriptive title (limited to a maximum of 44 characters), the names, and optionally the contact info for each author, etc. 2. Abstract -- a short (~200 word) description of the technical issue being addressed. 3. Copyright/public domain -- Each PEP must either be explicitly labelled as placed in the public domain (see this PEP as an example) or licensed under the Open Publication License[4]. 4. Specification -- The technical specification should describe the syntax and semantics of any new language feature. The specification should be detailed enough to allow competing, interoperable implementations for any of the current Python platforms (CPython, JPython, Python .NET). 5. Motivation -- The motivation is critical for PEPs that want to change the Python language. It should clearly explain why the existing language specification is inadequate to address the problem that the PEP solves. PEP submissions without sufficient motivation may be rejected outright. 6. Rationale -- The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work, e.g. how the feature is supported in other languages. The rationale should provide evidence of consensus within the community and discuss important objections or concerns raised during discussion. 7. Backwards Compatibility -- All PEPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. The PEP must explain how the author proposes to deal with these incompatibilities. PEP submissions without a sufficient backwards compatibility treatise may be rejected outright. 8. Reference Implementation -- The reference implementation must be completed before any PEP is given status 'Final,' but it need not be completed before the PEP is accepted. It is better to finish the specification and rationale first and reach consensus on it before writing code. The final implementation must include test code and documentation appropriate for either the Python language reference or the standard library reference. PEP Template PEPs are written in plain ASCII text, and should adhere to a rigid style. There is a Python script that parses this style and converts the plain text PEP to HTML for viewing on the web[5]. PEP 9 contains a boilerplate[7] template you can use to get started writing your PEP. Each PEP must begin with an RFC822 style header preamble. The headers must appear in the following order. Headers marked with `*' are optional and are described below. All other headers are required. PEP: <pep number> Title: <pep title> Version: <cvs version string> Last-Modified: <cvs date string> Author: <list of authors' real names and optionally, email addrs> * Discussions-To: <email address> Status: <Draft | Active | Accepted | Deferred | Final | Replaced> Type: <Informational | Standards Track> * Requires: <pep numbers> Created: <date created on, in dd-mmm-yyyy format> * Python-Version: <version number> Post-History: <dates of postings to python-list and python-dev> * Replaces: <pep number> * Replaced-By: <pep number> The Author: header lists the names and optionally, the email addresses of all the authors/owners of the PEP. The format of the author entry should be address(a)dom.ain (Random J. User) if the email address is included, and just Random J. User if the address is not given. If there are multiple authors, each should be on a separate line following RFC 822 continuation line conventions. Note that personal email addresses in PEPs will be obscured as a defense against spam harvesters. Standards track PEPs must have a Python-Version: header which indicates the version of Python that the feature will be released with. Informational PEPs do not need a Python-Version: header. While a PEP is in private discussions (usually during the initial Draft phase), a Discussions-To: header will indicate the mailing list or URL where the PEP is being discussed. No Discussions-To: header is necessary if the PEP is being discussed privately with the author, or on the python-list or python-dev email mailing lists. Note that email addresses in the Discussions-To: header will not be obscured. Created: records the date that the PEP was assigned a number, while Post-History: is used to record the dates of when new versions of the PEP are posted to python-list and/or python-dev. Both headers should be in dd-mmm-yyyy format, e.g. 14-Aug-2001. PEPs may have a Requires: header, indicating the PEP numbers that this PEP depends on. PEPs may also have a Replaced-By: header indicating that a PEP has been rendered obsolete by a later document; the value is the number of the PEP that replaces the current document. The newer PEP must have a Replaces: header containing the number of the PEP that it rendered obsolete. PEP Formatting Requirements PEP headings must begin in column zero and the initial letter of each word must be capitalized as in book titles. Acronyms should be in all capitals. The body of each section must be indented 4 spaces. Code samples inside body sections should be indented a further 4 spaces, and other indentation can be used as required to make the text readable. You must use two blank lines between the last line of a section's body and the next section heading. You must adhere to the Emacs convention of adding two spaces at the end of every sentence. You should fill your paragraphs to column 70, but under no circumstances should your lines extend past column 79. If your code samples spill over column 79, you should rewrite them. Tab characters must never appear in the document at all. A PEP should include the standard Emacs stanza included by example at the bottom of this PEP. A PEP must contain a Copyright section, and it is strongly recommended to put the PEP in the public domain. When referencing an external web page in the body of a PEP, you should include the title of the page in the text, with a footnote reference to the URL. Do not include the URL in the body text of the PEP. E.g. Refer to the Python Language web site [1] for more details. ... [1] http://www.python.org When referring to another PEP, include the PEP number in the body text, such as "PEP 1". The title may optionally appear. Add a footnote reference that includes the PEP's title and author. It may optionally include the explicit URL on a separate line, but only in the References section. Note that the pep2html.py script will calculate URLs automatically, e.g.: ... Refer to PEP 1 [7] for more information about PEP style ... References [7] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton http://www.python.org/peps/pep-0001.html If you decide to provide an explicit URL for a PEP, please use this as the URL template: http://www.python.org/peps/pep-xxxx.html PEP numbers in URLs must be padded with zeros from the left, so as to be exactly 4 characters wide, however PEP numbers in text are never padded. Reporting PEP Bugs, or Submitting PEP Updates How you report a bug, or submit a PEP update depends on several factors, such as the maturity of the PEP, the preferences of the PEP author, and the nature of your comments. For the early draft stages of the PEP, it's probably best to send your comments and changes directly to the PEP author. For more mature, or finished PEPs you may want to submit corrections to the SourceForge bug manager[6] or better yet, the SourceForge patch manager[2] so that your changes don't get lost. If the PEP author is a SF developer, assign the bug/patch to him, otherwise assign it to the PEP editor. When in doubt about where to send your changes, please check first with the PEP author and/or PEP editor. PEP authors who are also SF committers, can update the PEPs themselves by using "cvs commit" to commit their changes. Remember to also push the formatted PEP text out to the web by doing the following: % python pep2html.py -i NUM where NUM is the number of the PEP you want to push out. See % python pep2html.py --help for details. Transferring PEP Ownership It occasionally becomes necessary to transfer ownership of PEPs to a new champion. In general, we'd like to retain the original author as a co-author of the transferred PEP, but that's really up to the original author. A good reason to transfer ownership is because the original author no longer has the time or interest in updating it or following through with the PEP process, or has fallen off the face of the 'net (i.e. is unreachable or not responding to email). A bad reason to transfer ownership is because you don't agree with the direction of the PEP. We try to build consensus around a PEP, but if that's not possible, you can always submit a competing PEP. If you are interested assuming ownership of a PEP, send a message asking to take over, addressed to both the original author and the PEP editor <peps(a)python.org>. If the original author doesn't respond to email in a timely manner, the PEP editor will make a unilateral decision (it's not like such decisions can be reversed. :). References and Footnotes [1] This historical record is available by the normal CVS commands for retrieving older revisions. For those without direct access to the CVS tree, you can browse the current and past PEP revisions via the SourceForge web site at http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi/python/nondist/peps/?cvsroot=… [2] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [3] http://sourceforge.net/tracker/?atid=355470&group_id=5470&func=browse [4] http://www.opencontent.org/openpub/ [5] The script referred to here is pep2html.py, which lives in the same directory in the CVS tree as the PEPs themselves. Try "pep2html.py --help" for details. The URL for viewing PEPs on the web is http://www.python.org/peps/ [6] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [7] PEP 9, Sample PEP Template http://www.python.org/peps/pep-0009.html Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:

8 months, 1 week

Reviving restricted mode?

by Guido van Rossum

I've received some enthusiastic emails from someone who wants to revive restricted mode. He started out with a bunch of patches to the CPython runtime using ctypes, which he attached to an App Engine bug: http://code.google.com/p/googleappengine/issues/detail?id=671 Based on his code (the file secure.py is all you need, included in secure.tar.gz) it seems he believes the only security leaks are __subclasses__, gi_frame and gi_code. (I have since convinced him that if we add "restricted" guards to these attributes, he doesn't need the functions added to sys.) I don't recall the exploits that Samuele once posted that caused the death of rexec.py -- does anyone recall, or have a pointer to the threads? -- --Guido van Rossum (home page: http://www.python.org/~guido/)

7 years, 5 months

PEP 382: Namespace Packages

by "Martin v. Löwis"

I propose the following PEP for inclusion to Python 3.1. Please comment. Regards, Martin Abstract ======== Namespace packages are a mechanism for splitting a single Python package across multiple directories on disk. In current Python versions, an algorithm to compute the packages __path__ must be formulated. With the enhancement proposed here, the import machinery itself will construct the list of directories that make up the package. Terminology =========== Within this PEP, the term package refers to Python packages as defined by Python's import statement. The term distribution refers to separately installable sets of Python modules as stored in the Python package index, and installed by distutils or setuptools. The term vendor package refers to groups of files installed by an operating system's packaging mechanism (e.g. Debian or Redhat packages install on Linux systems). The term portion refers to a set of files in a single directory (possibly stored in a zip file) that contribute to a namespace package. Namespace packages today ======================== Python currently provides the pkgutil.extend_path to denote a package as a namespace package. The recommended way of using it is to put:: from pkgutil import extend_path __path__ = extend_path(__path__, __name__) int the package's ``__init__.py``. Every distribution needs to provide the same contents in its ``__init__.py``, so that extend_path is invoked independent of which portion of the package gets imported first. As a consequence, the package's ``__init__.py`` cannot practically define any names as it depends on the order of the package fragments on sys.path which portion is imported first. As a special feature, extend_path reads files named ``*.pkg`` which allow to declare additional portions. setuptools provides a similar function pkg_resources.declare_namespace that is used in the form:: import pkg_resources pkg_resources.declare_namespace(__name__) In the portion's __init__.py, no assignment to __path__ is necessary, as declare_namespace modifies the package __path__ through sys.modules. As a special feature, declare_namespace also supports zip files, and registers the package name internally so that future additions to sys.path by setuptools can properly add additional portions to each package. setuptools allows declaring namespace packages in a distribution's setup.py, so that distribution developers don't need to put the magic __path__ modification into __init__.py themselves. Rationale ========= The current imperative approach to namespace packages has lead to multiple slightly-incompatible mechanisms for providing namespace packages. For example, pkgutil supports ``*.pkg`` files; setuptools doesn't. Likewise, setuptools supports inspecting zip files, and supports adding portions to its _namespace_packages variable, whereas pkgutil doesn't. In addition, the current approach causes problems for system vendors. Vendor packages typically must not provide overlapping files, and an attempt to install a vendor package that has a file already on disk will fail or cause unpredictable behavior. As vendors might chose to package distributions such that they will end up all in a single directory for the namespace package, all portions would contribute conflicting __init__.py files. Specification ============= Rather than using an imperative mechanism for importing packages, a declarative approach is proposed here, as an extension to the existing ``*.pkg`` mechanism. The import statement is extended so that it directly considers ``*.pkg`` files during import; a directory is considered a package if it either contains a file named __init__.py, or a file whose name ends with ".pkg". In addition, the format of the ``*.pkg`` file is extended: a line with the single character ``*`` indicates that the entire sys.path will be searched for portions of the namespace package at the time the namespace packages is imported. Importing a package will immediately compute the package's __path__; the ``*.pkg`` files are not considered anymore after the initial import. If a ``*.pkg`` package contains an asterisk, this asterisk is prepended to the package's __path__ to indicate that the package is a namespace package (and that thus further extensions to sys.path might also want to extend __path__). At most one such asterisk gets prepended to the path. extend_path will be extended to recognize namespace packages according to this PEP, and avoid adding directories twice to __path__. No other change to the importing mechanism is made; searching modules (including __init__.py) will continue to stop at the first module encountered. Discussion ========== With the addition of ``*.pkg`` files to the import mechanism, namespace packages can stop filling out the namespace package's __init__.py. As a consequence, extend_path and declare_namespace become obsolete. It is recommended that distributions put a file <distribution>.pkg into their namespace packages, with a single asterisk. This allows vendor packages to install multiple portions of namespace package into a single directory, with no risk of overlapping files. Namespace packages can start providing non-trivial __init__.py implementations; to do so, it is recommended that a single distribution provides a portion with just the namespace package's __init__.py (and potentially other modules that belong to the namespace package proper). The mechanism is mostly compatible with the existing namespace mechanisms. extend_path will be adjusted to this specification; any other mechanism might cause portions to get added twice to __path__. Copyright ========= This document has been placed in the public domain.

12 years, 2 months

pthreads, fork, import, and execvp

by Rotem Yaari

Hello everyone! We have been encountering several deadlocks in a threaded Python application which calls subprocess.Popen (i.e. fork()) in some of its threads. This has occurred on Python 2.4.1 on a 2.4.27 Linux kernel. Preliminary analysis of the hang shows that the child process blocks upon entering the execvp function, in which the import_lock is acquired due to the following line: def _ execvpe(file, args, env=None): from errno import ENOENT, ENOTDIR ... It is known that when forking from a pthreaded application, acquisition attempts on locks which were already locked by other threads while fork() was called will deadlock. Due to these oddities we were wondering if it would be better to extract the above import line from the execvpe call, to prevent lock acquisition attempts in such cases. Another workaround could be re-assigning a new lock to import_lock (such a thing is done with the global interpreter lock) at PyOS_AfterFork or pthread_atfork. We'd appreciate any opinions you might have on the subject. Thanks in advance, Yair and Rotem

12 years, 4 months

compiling python2.5 on linux under wine

by Luke Kenneth Casson Leighton

hey, has anyone investigated compiling python2.5 using winegcc, under wine? i'm presently working my way through it, just for kicks, and was wondering if anyone would like to pitch in or stare at the mess under a microscope. it's not as crazed as it sounds. cross-compiling python2.5 for win32 with mingw32 is an absolute miserable bitch of a job that goes horribly wrong when you actually try to use the minimalist compiler to do any real work. so i figured that it would be easier to get python compiled using wine. i _have_ got some success - a python script and a python.exe.so (which is winegcc's friendly way of telling you you have something that stands a chance of working) as well as a libpython25.dll.so. what i _don't_ yet have is an _md5.dll (or should it be _md5.lib?) i.e. the standard modules are a bit... iffy. the _winreg.o is compiled; the _md5.o is compiled; the winreg.lib is not. whoops. plus, it's necessary to enable nt_dl.c which is in PC/ _not_ in Modules/. one of the key issues that's a bit of a bitch is that python is compiled up for win32 with a hard-coded pyconfig.h which someone went to a _lot_ of trouble to create by hand instead of using autoconf. oh - and it uses visualstudio so there's not even a Makefile. ignoring that for the time-being was what allowed me to get as far as actually having a python interpreter (with no c-based modules). so there's a whole _stack_ of stuff that needs dragging kicking and screaming into the 21st century. there _is_ a reason why i want to do this. actually, there's two. firstly, i sure as shit do _not_ want to buy, download, install _or_ run visual studio. i flat-out refuse to run an MS os and visual studio runs like a dog under wine. secondly, i want a python25.lib which i can use to cross-compile modules for poor windows users _despite_ sticking to my principles and keeping my integrity as a free software developer. thirdly i'd like to cross-compile pywebkitgtk for win32 fourthly i'd like to compile and link applications to the extremely successful and well wicked MSHTML.DLL... in the _wine_ project :) not the one in windows (!) i want to experiment with DOM model manipulation - from python - similar to the OLPC HulaHop project - _but_ i want to compile or cross-compile everything from linux, not windows (see 1 above) fifthly i'd like to see COM (DCOM) working and pywin32 compiled and useable under wine, even if it means having to get a license to use dcom98 and oleauth.lib and oleauth.h etc. and all the developer files needed to link DCOM applications under windows. actually what i'd _really_ like to see is FreeDCE's DCOM work actually damn well finished, it's only been eight years since wez committed the first versions of the IDL and header files, and it's only been over fifteen years since microsoft began its world domination using COM and DCOM. ... but that's another story :) so that's ... five reasons not two. if anyone would like to collaborate on a crazed project with someone who can't count, i'm happy to make available what i've got up to so far, on github.org. l.

12 years, 4 months

standard library mimetypes module pathologically broken?

by Jacob Rus

Hi all, In an attempt to figure out some twisted.web code, I was reading through the Python Standard Library’s mimetypes module today, and was shocked at the poor quality of the code. I wonder how the mimetypes code made it into the standard library, and whether anyone has ever bothered to read it or update it: it is an embarrassment. Much of the code is redundant, portions fail to execute, control flow is routed through a horribly confusing mess of spaghetti, and most of the complexity has no clear benefit as far as I can tell. I probably should drop the subject and get back to work, but as a good citizen, it’s hard to just ignore this sort of thing. mimetypes.py stores its types in a pair of dictionaries, one for "strict" use, and the other for "non-standard types". It creates the strict dictionary by default out of apache's mime.types file, and then overrides the entries it finds with a set of exceptions. Then it creates the non-standard dictionary, which is set to match if the strict parameter is set to False when guessing types. Just in this basic design, and in the list of types in the file, there are several problems: * Various apache mime types files are read, if found, but the ordering of the files is such that older versions of apache are sometimes read after newer ones, overriding updated mime types with out-of-date versions if multiple versions of apache are installed on the system. * The vast majority of types declared in mimetypes.py are duplicates of types already declared by Apache. In a few cases this is to change the apache default (make an exception, that is), but in most cases the mime type and extension are completely identical. This huge number of redundant types makes the file substantially harder to follow. No comments are provided to explain why various sets of exceptions are made to Apache's default mime types, and in several cases mimetypes.py seems to just be out of date as compared to recent versions of Apache, for instance not knowing about the 'text/troff' type which was registered in January 2006 in RFC 4263. * The 'non-standard' type dictionary is nearly useless, because all of the types it declares are already in apache's mime.types file, meaning that types are, as far as I can tell trying to follow ugly program flow, *never* drawn from the non-strict dictionary, except in the improbable situation where the mimetypes module is initialized with a custom set of apache-mime.types–like files, which does not include those 'non-standard' types. I personally cannot see a use case for initializing the module with a custom set of mime types, but then leaving the very few types included as non-strict to the defaults: this seems like a fragile and pathological use case. Given this, I don’t see any benefit to dragging the 'strict' parameter along all the way through the code, and would advise getting rid of it altogether. Does anyone know of any code that uses the mimetypes module with strict set to False, where the non-strict code path ever *actually* is executed? But though these problems, which affect actual use of the code and are therefore probably most important, are significant, they really pale in comparison to the awful quality of implementation. I'll try to briefly outline my understanding of how code flows in mimetypes.py, and what the problems are. I haven't stepped through the code in a debugger, this is just from reading it, so I apologize in advance if I get something wrong. This is, however, some of the worst code I’ve seen in the standard library or anywhere else. * It defines __all__: I didn’t even realize __all__ could be used for single-file modules (w/o submodules), but it definitely shouldn’t be here. This specific __all__ oddly does not include all of the documented variables and functions in the mimetypes class. It’s not clear why someone calling import * here wouldn’t want the bits not included. * It creates a _default_mime_types() function which declares a bunch of global variables, and then immediately calls _default_mime_types() below the definition. There is literally no difference in result between this and just putting those variables at the top level of the file, so I have no idea why this function exists, except to make the code more confusing. * It allows command line usage: I don’t think this is necessary for a part of the standard library like this. There are better tools for finding mime types from the command line which ship with most operating systems. * Its API is pretty poorly designed. It offers 6 functions when about 3 are needed, and it takes a couple reads-through of the code to figure out exactly what any of them are supposed to do. * The operation is crazy: It defines a MimeTypes class which actually stores the type mappings, but this class is designed to be a singleton. The way that such a design is enforced is through the use of the module-global 'init' function, which makes an instance of the class, and then maps all of the functions in the module global namespace to instance methods. But confusingly, all such functions are also defined independently of the init function, with definitions such as: def guess_type(url, strict=True): if not inited: init() return guess_type(url, strict) I’d be amazed if anyone could guess what that code was trying to do. I did a double-take when I saw it. Of course, that return call is only ever reached the first time this function is called, if init() has not happened yet. This was all presumably done for lazy initialization, so that the type information would only be loaded when needed. Needless to say, there are more pythonic ways to accomplish such a goal. Oh, also, the other good one here is that it means that someone who writes `from mimetypes import guess_types` gets something different than someone who writes: `import mimetypes; guess_types = mimetypes.guess_types`. In the former case, this wrapper function is saved as guess_type, which each time just calls the (changed after init()) mimetypes.guess_types function. This caused a performance nightmare before March of this year, when there was no check for `if not inited` before running init() (amazing!?). * Because the type datastore is set up to be a singleton, any time init() is called in one section of code, it resets any types which have been added manually: this means that if init() is called by different pieces of code in the same python program, they will interfere with each-others’ type databases, and break each-other. This is extremely fragile and, in my opinion, crazy. It is hard for me to imagine any use case that would benefit from this ability to clobber custom type mappings, and I very much doubt that any code calling the mimetypes module realizes that the contract of the API is so flimsy by definition. In practice, I would not advise consumers of this API to ever call init() manually, or to ever add custom mime type mappings, because they are setting themselves up for hard-to-track bugs down the line. * The 'inited' flag is a documented part of the interface, in the standard library documentation. I cannot imagine any reason to set this flag manually: setting it to false when it was true will have no effect, because the top-level functions have already been replaced by instance methods of the 'db' MimeTypes instance. Setting it to true when it was false will make the code just break outright. * In python 3, this has been changed a bit. There’s still an inited flag, and it still in the docs, but now awful code from above has been changed slightly, to: def guess_type(url, strict=True): if _db is None: init() return _db.guess_type(url, strict) Which is still embarrassingly confusing. On the upside, the inited flag now does literally nothing, but remains defined, and in the docs. * The 'types_map' and 'common_types' (for 'strict' and 'common' types, respectively) dictionaries are also a documented part of the interface. When init() is called, a new MimeTypes instance makes a (different) types_map which is a tuple of two dictionaries, for 'strict' and 'common' types. Then this instance reads the apache mime.types files and adds the types to its pair of self.types_map dictionaries, and then after that looks at the global types_map and common_types dictionaries and adds *those* types to its self.types_map. Then at the end it replaces the global types_map with self.types_map[True] and replaces common_types with self.types_map[False]. Unfortunately, while changing these dictionaries will have an effect on the operation of the library, it will not update the types_map_inv mapping, so inverse lookups will not behave as the changer expects. If these dictionaries are going to remain documented, the documentation should be clear to describe them as read only to avoid very confusing bugs. * Speaking of these dictionaries, .copy() is called on those two and a few other inside MimeTypes.__init__(), which happens every time the global init() function is called, but then init() puts the copies back in the global namespace, meaning that the original is discarded. Basically the only reason for the .copy() is to make sure that the correct updates are applied to the apache mimetype defaults, but the code will gladly re-read all of the apache files even after its mapped types are already in these dictionaries, essentially making re-initializing a (very expensive) no-op. All we’re doing is a lot of unnecessary extra disk reads and memory allocations and deallocations. The only time this has any effect is when a non-singleton MimeTypes instance is created, as in the read_mime_types function. * And that read_mime_types function is a doozy. It tries to open a filename, spits back None if there’s an IOError (instead of raising the exception as it should), and then creates a new MimeTypes instance (remember, this is identical to the singleton MimeTypes instance because it starts itself from that one’s mappings), adds any new types it finds in the file with that name, and then returns the 'strict' types_map from it. I’m not sure whether any sane user of this API would expect it to return the existing type mappings *plus* the extra ones in the provided filename, but I really can’t imagine this function ever being particularly useful: it requires you are reading mime types in apache format, but not the apache mime type files you already looked at, and then the only way to find out what new mappings were defined is to take the difference of the default mappings with the result of the function. * The code itself, on a line-by-line basis, is unpythonic and unnecessarily verbose, confusing, and slow. The code should be rewritten to use python 2.3–2.6 features: even leaving its functionality identical it could be cut to about half the number of lines, and made clearer. In case the above doesn’t make this clear: this code is extremely confusing. Trying to read it has caused all the people around me to look up as I shout "what the fuck??!" at the screen every few minutes, as each new revelation gives another surprise. I’m not convinced that I completely understand what the code does, because it has been quite effectively obfuscated, but I understand enough to want to throw the whole thing out, and start essentially from scratch. So the question is, what should be done about this? I’d like to hear how people use the mimetypes module, and how they expect it to work, to figure out the sanest possible mostly-backwards-compatible replacement which could be dropped in (ideally this would just allow the use of default mimetypes and rip out the ability to alter the default datastore: or is there some easy way to change this away from a singleton without breaking code which calls these methods?), and then extend that replacement to support a somewhat saner model for anyone who actually wants to extend the set of mappings. My guess is that replacement code could actually fix subtle bugs in existing uses of this module, by people who had a sane expectation of how it was supposed to work. At the very least, the parts about figuring out exactly which exceptions to Apache’s set of default types are useful would be a good idea, and I’d maybe even recommend including an up-to-date copy of Apache’s mime.types file in the Python Standard Library, and then only overriding its definitions for future versions of Apache (and then overriding the combination of both of those with further exceptions deemed useful for python, with comments explaining why each exception), so that we’re not bothering to look up horribly out-of-date types in multiple locations from Apache 1, 1.2, 1.3, etc. I’d also recommend making the API for overriding definitions be the same as the code used to declare the default overrides, because as it is there are three ways do define types: a) in a mime.types formatted file, b) in a python dictionary that gets initialized with a confusing bit of code, and c) through the add_type function. Does anyone else have thoughts about this, or maybe some good (it had better be *really* good) explanations why this code is the way it is? I'd be happy to try to rewrite it, but I think I’d need a bit of help figuring out how to make the rewrite backwards-compatible. Note: someone else has had fun with this module: <http://lucumr.pocoo.org/2009/3/1/the-1000-speedup-or-the-stdlib-sucks> <http://lucumr.pocoo.org/2009/7/24/singletons-and-their-problems-in-python> Cheers, Jacob Rus

12 years, 5 months

expy: an expressway to extend Python

by Yingjie Lan

Hi All, This is to announce the initial release of expy 0.1.0. More details at http://expy.sourceforge.net/ Thanks! Yingjie

12 years, 5 months

REVIEW: PyArg_ParseTuple with "s" format and NUL: Bogus TypeError detail string.

by Sean Reifschneider

Please review this, I'm worried that there are cases where convertitem() is returning a string that really should be overridden by the argument "help string". However, I'm worried that this change will get rid of useful messages (via the format "; help string"), when there otherwise wouldn't be. PyArg_ParseTuple when handling a string format "s", raises a TypeError when the passed string contains a NUL. However, the TypeError doesn't contain useful information. For example: syslog.syslog('hello\0there') TypeError: [priority,] message string This seems to be a thinko in Python/getargs.c at line 331: msg = convertitem(PyTuple_GET_ITEM(args, i), &format, p_va, flags, levels, msgbuf, sizeof(msgbuf), &freelist); if (msg) { seterror(i+1, msg, levels, fname, message); <<< Line 331 return cleanreturn(0, freelist); } This also applies to Python 3 trunk in line 390. I think that's supposed to be "msg" instead of "message" in the last argument. If I change it, I get: >>> import syslog; syslog.syslog('hello\0there') Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: must be string without null bytes, not str I think it's safe to change "message" to "msg" because: "message" is the "help" string "[priority,] message string", but "msg" contains much more useful information. "msg" is in the "fall back if the last argument doesn't contain useful information" argument position, but "msg" is never NULL. "message" only is NULL if the format string doesn't contain ";". In every case I can find in the code, convertitem() is returning a much more useful string than the help string. Or perhaps it should do something like: if (msg) { seterror(i+1, msg, levels, fname, '%s (%s)' % ( msg, message )); Pardon my mixed C+Python, but you get the idea. Thoughts? Thanks, Sean -- [...] Premature optimization is the root of all evil. -- Donald Knuth Sean Reifschneider, Member of Technical Staff <jafo(a)tummy.com> tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability

12 years, 5 months

Summary of Python tracker Issues

by Python tracker

ACTIVITY SUMMARY (07/24/09 - 07/31/09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2292 open (+33) / 16143 closed (+18) / 18435 total (+51) Open issues with patches: 899 Average duration of open issues: 660 days. Median duration of open issues: 414 days. Open Issues Breakdown open 2261 (+32) pending 30 ( +0) Issues Created Or Reopened (52) _______________________________ string.lowercase/uppercase/letters not affected by locale change 07/25/09 CLOSED http://bugs.python.org/issue6525 reopened ezio.melotti inserting None into sys.modules does not raise ImportError with 07/24/09 http://bugs.python.org/issue6563 created brett.cannon Error in Sec. 8.4 of Tutorial 07/24/09 CLOSED http://bugs.python.org/issue6564 created gmarsden improper use of __setitem__ in ElementTree for Python 3.1 07/24/09 http://bugs.python.org/issue6565 created aroberge json.dumps converts None to "null" (not null) 07/24/09 http://bugs.python.org/issue6566 created srid Make inf be almost equal to inf 07/25/09 http://bugs.python.org/issue6567 reopened lucaspmelo unittest test discovery improvements 07/24/09 http://bugs.python.org/issue6568 created michael.foord unittest document bug (random.shuffle sequence) 07/25/09 CLOSED http://bugs.python.org/issue6569 created hsmtkk patch Documentation Clarity 07/25/09 http://bugs.python.org/issue6570 created StMark Help index 07/25/09 CLOSED http://bugs.python.org/issue6571 created cjw Manas Thapliyal wants to chat 07/25/09 CLOSED http://bugs.python.org/issue6572 created gravitywarrior1 set union method ignores arguments appearing after the original 07/25/09 CLOSED http://bugs.python.org/issue6573 created ssmout patch List the __future__ features in a table 07/26/09 http://bugs.python.org/issue6574 created ezio.melotti patch, easy Can't download docs 07/26/09 CLOSED http://bugs.python.org/issue6575 created zuo re docs: wrong link targets 07/26/09 CLOSED http://bugs.python.org/issue6576 created zuo Links wrongly targeting to builtin functions' instead of module 07/26/09 CLOSED http://bugs.python.org/issue6577 created zuo 2 problems with 'Docs for other versions' section on left HTML d 07/26/09 CLOSED http://bugs.python.org/issue6578 created zuo No update about automatic numbering of fields in format strings 07/26/09 http://bugs.python.org/issue6579 reopened georg.brandl No deprecation warning for list comprehension leak conflict 07/26/09 http://bugs.python.org/issue6580 created mrbax inspect.py sys._getframe patch for Python 2.6 07/26/09 CLOSED http://bugs.python.org/issue6581 created michael.foord patch, patch, needs review test_telnetlib doesn't test Telnet.write 07/26/09 http://bugs.python.org/issue6582 created jackdied easy 2to3 fails to fix test.test_support 07/27/09 http://bugs.python.org/issue6583 created joe.amenta gzip module has no custom exception 07/27/09 http://bugs.python.org/issue6584 created srid configure.in forces specific autoconf version 07/27/09 CLOSED http://bugs.python.org/issue6585 created cavallo71 Documentation of os.write and os.read are inaccurate. 07/27/09 CLOSED http://bugs.python.org/issue6586 created cliechti interrupts during long writes cause connection corruption with S 07/28/09 http://bugs.python.org/issue6587 created janssen insert cookies into cookie jar - cookielib.py 07/28/09 http://bugs.python.org/issue6588 created jondebonis patch smtpd.SMTPServer can cause asyncore.loop to enter infinite event 07/28/09 http://bugs.python.org/issue6589 created cmcginty Py3K can't set the browser's character encoding! (CGI) 07/28/09 CLOSED http://bugs.python.org/issue6590 created [PhysiC] add reference to fcntl.ioctl in the socket module 07/28/09 CLOSED http://bugs.python.org/issue6591 created gagenellina patch otparse: processing variable number of short option arguments 07/28/09 http://bugs.python.org/issue6592 created tanvalley Documentation: gettext install link 07/28/09 CLOSED http://bugs.python.org/issue6593 created efrerich json C serializer performance tied to structure depth on some sy 07/28/09 http://bugs.python.org/issue6594 created swalker Make Decimal constructor accept all unicode decimal digits in in 07/29/09 http://bugs.python.org/issue6595 created marketdickinson patch urllib2 bug on CentOS 07/29/09 http://bugs.python.org/issue6596 created rk3dov Deprecate iterable.next in Python > 2.6.x when called with -3 op 07/29/09 CLOSED http://bugs.python.org/issue6597 created mattrussell calling email.utils.make_msgid frequently has a non-trivial prob 07/30/09 http://bugs.python.org/issue6598 created mwh 2to3 test_print_function_option fails on Windows 07/30/09 http://bugs.python.org/issue6599 created abbeyj patch MemoryError in AiX 64-bit build - PyMem_MALLOC failed 07/30/09 http://bugs.python.org/issue6600 created srid Fractions do not support other Fractions as numerators or denomi 07/30/09 CLOSED http://bugs.python.org/issue6601 created honeyman BaseHTTPServer log_message should log to sys.stdout 07/30/09 http://bugs.python.org/issue6602 created tomkm Compilation error if configuref --with-computed-gotos 07/30/09 http://bugs.python.org/issue6603 created wiget test_distutils subtest test_get_exe_bytes fails depending on exe 07/30/09 http://bugs.python.org/issue6604 created matejcik patch smtplib.SMTP.sendmail() rejected after quit(),connect() sequence 07/30/09 http://bugs.python.org/issue6605 created cmcginty csv.Sniffer.sniff on data with doublequotes doesn't set up the d 07/30/09 http://bugs.python.org/issue6606 created jtate patch asctime causing python to crash 07/30/09 CLOSED http://bugs.python.org/issue6607 created AmirHabibi asctime causing python to crash 07/30/09 http://bugs.python.org/issue6608 created AmirHabibi zipfile: WindowsError [267] The directory name is invalid 07/31/09 http://bugs.python.org/issue6609 created srid Subprocess descriptor debacle 07/31/09 http://bugs.python.org/issue6610 created christian.heimes easy, needs review HTMLParser cannot deal with mixture of arbitrary data and charac 07/31/09 http://bugs.python.org/issue6611 created liudongmiao(a)gmail.com 'import site' fails when called from an unlinked directory 07/31/09 http://bugs.python.org/issue6612 created labrat ctypes.PyThreadState_SetAsyncExc does not work on linux 64bit ma 07/31/09 http://bugs.python.org/issue6613 created coutinhotiago Issues Now Closed (40) ______________________ The ValueError raised by failing to unpack sequence should have 412 days http://bugs.python.org/issue3071 jackdied patch telnetlib process_rawq buffer handling is confused 168 days http://bugs.python.org/issue5188 jackdied patch Subprocess.getstatusoutput Fails Executing 'dir' Command on Wind 98 days http://bugs.python.org/issue5808 sorin spurious space after opening parenthesis when auto-completing 78 days http://bugs.python.org/issue5992 pitrou Document and slightly simplify lnotab tracing 75 days http://bugs.python.org/issue6042 alexandre.vassalotti patch, needs review read_until 63 days http://bugs.python.org/issue6106 jackdied Make logging configuration files easier to use 63 days http://bugs.python.org/issue6136 vsajip logging.basicConfig(level='DEBUG', ... and setLevel("DEBUG") res 24 days http://bugs.python.org/issue6314 vsajip patch trunk does not build with --enable-unicode=ucs4 37 days http://bugs.python.org/issue6330 eric.smith easy nb_divide missing in docs 35 days http://bugs.python.org/issue6336 georg.brandl TimedRotatingFileHandler permission denied rename failure on Win 35 days http://bugs.python.org/issue6343 vsajip zipfile: Invalid argument when opening zero-sized files 11 days http://bugs.python.org/issue6511 amaury.forgeotdarc string.lowercase/uppercase/letters not affected by locale change 0 days http://bugs.python.org/issue6525 georg.brandl cmath documentation misleading: suggests existence of real() an 6 days http://bugs.python.org/issue6548 marketdickinson patch Regex '\d' should not match unicode category 'No'. 4 days http://bugs.python.org/issue6561 marketdickinson patch, needs review Error in Sec. 8.4 of Tutorial 0 days http://bugs.python.org/issue6564 georg.brandl unittest document bug (random.shuffle sequence) 0 days http://bugs.python.org/issue6569 benjamin.peterson patch Help index 0 days http://bugs.python.org/issue6571 georg.brandl Manas Thapliyal wants to chat 0 days http://bugs.python.org/issue6572 benjamin.peterson set union method ignores arguments appearing after the original 2 days http://bugs.python.org/issue6573 rhettinger patch Can't download docs 0 days http://bugs.python.org/issue6575 georg.brandl re docs: wrong link targets 0 days http://bugs.python.org/issue6576 georg.brandl Links wrongly targeting to builtin functions' instead of module 0 days http://bugs.python.org/issue6577 georg.brandl 2 problems with 'Docs for other versions' section on left HTML d 0 days http://bugs.python.org/issue6578 georg.brandl inspect.py sys._getframe patch for Python 2.6 0 days http://bugs.python.org/issue6581 michael.foord patch, patch, needs review configure.in forces specific autoconf version 1 days http://bugs.python.org/issue6585 r.david.murray Documentation of os.write and os.read are inaccurate. 2 days http://bugs.python.org/issue6586 georg.brandl Py3K can't set the browser's character encoding! (CGI) 0 days http://bugs.python.org/issue6590 [PhysiC] add reference to fcntl.ioctl in the socket module 1 days http://bugs.python.org/issue6591 georg.brandl patch Documentation: gettext install link 1 days http://bugs.python.org/issue6593 georg.brandl Deprecate iterable.next in Python > 2.6.x when called with -3 op 1 days http://bugs.python.org/issue6597 brett.cannon Fractions do not support other Fractions as numerators or denomi 0 days http://bugs.python.org/issue6601 marketdickinson asctime causing python to crash 0 days http://bugs.python.org/issue6607 alexandre.vassalotti Hang using files named prn.txt, etc 2812 days http://bugs.python.org/issue481171 ezio.melotti Allow set swig include dirs in setup.py 2108 days http://bugs.python.org/issue828336 wiget patch sendmsg() and recvmsg() for C socket module 1546 days http://bugs.python.org/issue1194378 ezio.melotti patch telnetlib expect() and read_until() do not time out properly 1346 days http://bugs.python.org/issue1360221 jackdied patch urllib/urllib2: HTTPS over (Squid) Proxy fails 1268 days http://bugs.python.org/issue1424152 orsenthil patch urllib2+https+proxy not working 1231 days http://bugs.python.org/issue1448934 orsenthil Telnetlib dosn't accept u'only ascii' 714 days http://bugs.python.org/issue1772794 jackdied Top Issues Most Discussed (10) ______________________________ 28 base64 module ignores non-alphabet characters 1212 days open http://bugs.python.org/issue1466065 8 asyncore incorrect failure when connection is refused and using 9 days open http://bugs.python.org/issue6550 8 Regexp 2.7 (modifications to current re 2.2.2) 472 days open http://bugs.python.org/issue2636 7 MemoryError in AiX 64-bit build - PyMem_MALLOC failed 2 days open http://bugs.python.org/issue6600 7 Make inf be almost equal to inf 6 days open http://bugs.python.org/issue6567 7 OverflowError in RLock.acquire() 7 days open http://bugs.python.org/issue6562 6 csv.Sniffer.sniff on data with doublequotes doesn't set up the 1 days open http://bugs.python.org/issue6606 6 No update about automatic numbering of fields in format strings 5 days open http://bugs.python.org/issue6579 6 Regex '\d' should not match unicode category 'No'. 4 days closed http://bugs.python.org/issue6561 6 email.FeedParser.BufferedSubFile improperly handles "\r\n" 804 days open http://bugs.python.org/issue1721862

12 years, 5 months

Re: [Python-Dev] Implementing File Modes

by Eric Pruitt

> On Tue, 28 Jul 2009 04:06:45 am Eric Pruitt wrote: > > I am implementing the file wrapper using changes to subprocess.Popen > > that also make it asynchronous and non-blocking so implementing "r+" > > should be trivial to do. How about handling stderr? I have the > > following ideas: leave out support for reading from stderr, make it > > so that there is an optional additional argument like "outputstderr = > > False", create another function that toggles / sets whether stderr or > > stdout is returned or mix the two outputs. > > Leaving it out is always an option. > > As I see it, fundamentally you can either read from stdout and sterr as > two different streams, or you can interleave (mix) them. To me, that > suggests three functions: > > ProcessIOWrapper() # read from stdout (or write to stdin etc.) > ProcessIOWrapperStdErr() # read/write from stderr > ProcessIOWrapper2() # read from mixed stdout and stderr > > I don't like a function to toggle between one and the other: that smacks > of relying on a global setting in a bad way. I suppose you could add an > optional argument to ProcessIOWrapper() to select between stdout, > stderr, or both together. > > > > -- > Steven D'Aprano How would having a toggle function rely on a global setting? Each class would simply have its own member variable like "self.readsstderr." It's a moot point though as I've decided to use separate functions as you suggested. With separate functions, the user doesn't have to worry about modifying the mode keyword if stderr is needed and as an added bonus, it also makes the code a lot more readable. Eric

12 years, 5 months

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev July 2009