Python-Dev August 2003

python-dev@python.org

119 participants
135 discussions

PEP 1, PEP Purpose and Guidelines
by barry＠zope.com 18 May '21

18 May '21

It has been a while since I posted a copy of PEP 1 to the mailing lists and newsgroups. I've recently done some updating of a few sections, so in the interest of gaining wider community participation in the Python development process, I'm posting the latest revision of PEP 1 here. A version of the PEP is always available on-line at http://www.python.org/peps/pep-0001.html Enjoy, -Barry -------------------- snip snip -------------------- PEP: 1 Title: PEP Purpose and Guidelines Version: $Revision: 1.36 $ Last-Modified: $Date: 2002/07/29 18:34:59 $ Author: Barry A. Warsaw, Jeremy Hylton Status: Active Type: Informational Created: 13-Jun-2000 Post-History: 21-Mar-2001, 29-Jul-2002 What is a PEP? PEP stands for Python Enhancement Proposal. A PEP is a design document providing information to the Python community, or describing a new feature for Python. The PEP should provide a concise technical specification of the feature and a rationale for the feature. We intend PEPs to be the primary mechanisms for proposing new features, for collecting community input on an issue, and for documenting the design decisions that have gone into Python. The PEP author is responsible for building consensus within the community and documenting dissenting opinions. Because the PEPs are maintained as plain text files under CVS control, their revision history is the historical record of the feature proposal[1]. Kinds of PEPs There are two kinds of PEPs. A standards track PEP describes a new feature or implementation for Python. An informational PEP describes a Python design issue, or provides general guidelines or information to the Python community, but does not propose a new feature. Informational PEPs do not necessarily represent a Python community consensus or recommendation, so users and implementors are free to ignore informational PEPs or follow their advice. PEP Work Flow The PEP editor, Barry Warsaw <peps(a)python.org>, assigns numbers for each PEP and changes its status. The PEP process begins with a new idea for Python. It is highly recommended that a single PEP contain a single key proposal or new idea. The more focussed the PEP, the more successfully it tends to be. The PEP editor reserves the right to reject PEP proposals if they appear too unfocussed or too broad. If in doubt, split your PEP into several well-focussed ones. Each PEP must have a champion -- someone who writes the PEP using the style and format described below, shepherds the discussions in the appropriate forums, and attempts to build community consensus around the idea. The PEP champion (a.k.a. Author) should first attempt to ascertain whether the idea is PEP-able. Small enhancements or patches often don't need a PEP and can be injected into the Python development work flow with a patch submission to the SourceForge patch manager[2] or feature request tracker[3]. The PEP champion then emails the PEP editor <peps(a)python.org> with a proposed title and a rough, but fleshed out, draft of the PEP. This draft must be written in PEP style as described below. If the PEP editor approves, he will assign the PEP a number, label it as standards track or informational, give it status 'draft', and create and check-in the initial draft of the PEP. The PEP editor will not unreasonably deny a PEP. Reasons for denying PEP status include duplication of effort, being technically unsound, not providing proper motivation or addressing backwards compatibility, or not in keeping with the Python philosophy. The BDFL (Benevolent Dictator for Life, Guido van Rossum) can be consulted during the approval phase, and is the final arbitrator of the draft's PEP-ability. If a pre-PEP is rejected, the author may elect to take the pre-PEP to the comp.lang.python newsgroup (a.k.a. python-list(a)python.org mailing list) to help flesh it out, gain feedback and consensus from the community at large, and improve the PEP for re-submission. The author of the PEP is then responsible for posting the PEP to the community forums, and marshaling community support for it. As updates are necessary, the PEP author can check in new versions if they have CVS commit permissions, or can email new PEP versions to the PEP editor for committing. Standards track PEPs consists of two parts, a design document and a reference implementation. The PEP should be reviewed and accepted before a reference implementation is begun, unless a reference implementation will aid people in studying the PEP. Standards Track PEPs must include an implementation - in the form of code, patch, or URL to same - before it can be considered Final. PEP authors are responsible for collecting community feedback on a PEP before submitting it for review. A PEP that has not been discussed on python-list(a)python.org and/or python-dev(a)python.org will not be accepted. However, wherever possible, long open-ended discussions on public mailing lists should be avoided. Strategies to keep the discussions efficient include, setting up a separate SIG mailing list for the topic, having the PEP author accept private comments in the early design phases, etc. PEP authors should use their discretion here. Once the authors have completed a PEP, they must inform the PEP editor that it is ready for review. PEPs are reviewed by the BDFL and his chosen consultants, who may accept or reject a PEP or send it back to the author(s) for revision. Once a PEP has been accepted, the reference implementation must be completed. When the reference implementation is complete and accepted by the BDFL, the status will be changed to `Final.' A PEP can also be assigned status `Deferred.' The PEP author or editor can assign the PEP this status when no progress is being made on the PEP. Once a PEP is deferred, the PEP editor can re-assign it to draft status. A PEP can also be `Rejected'. Perhaps after all is said and done it was not a good idea. It is still important to have a record of this fact. PEPs can also be replaced by a different PEP, rendering the original obsolete. This is intended for Informational PEPs, where version 2 of an API can replace version 1. PEP work flow is as follows: Draft -> Accepted -> Final -> Replaced ^ +----> Rejected v Deferred Some informational PEPs may also have a status of `Active' if they are never meant to be completed. E.g. PEP 1. What belongs in a successful PEP? Each PEP should have the following parts: 1. Preamble -- RFC822 style headers containing meta-data about the PEP, including the PEP number, a short descriptive title (limited to a maximum of 44 characters), the names, and optionally the contact info for each author, etc. 2. Abstract -- a short (~200 word) description of the technical issue being addressed. 3. Copyright/public domain -- Each PEP must either be explicitly labelled as placed in the public domain (see this PEP as an example) or licensed under the Open Publication License[4]. 4. Specification -- The technical specification should describe the syntax and semantics of any new language feature. The specification should be detailed enough to allow competing, interoperable implementations for any of the current Python platforms (CPython, JPython, Python .NET). 5. Motivation -- The motivation is critical for PEPs that want to change the Python language. It should clearly explain why the existing language specification is inadequate to address the problem that the PEP solves. PEP submissions without sufficient motivation may be rejected outright. 6. Rationale -- The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work, e.g. how the feature is supported in other languages. The rationale should provide evidence of consensus within the community and discuss important objections or concerns raised during discussion. 7. Backwards Compatibility -- All PEPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. The PEP must explain how the author proposes to deal with these incompatibilities. PEP submissions without a sufficient backwards compatibility treatise may be rejected outright. 8. Reference Implementation -- The reference implementation must be completed before any PEP is given status 'Final,' but it need not be completed before the PEP is accepted. It is better to finish the specification and rationale first and reach consensus on it before writing code. The final implementation must include test code and documentation appropriate for either the Python language reference or the standard library reference. PEP Template PEPs are written in plain ASCII text, and should adhere to a rigid style. There is a Python script that parses this style and converts the plain text PEP to HTML for viewing on the web[5]. PEP 9 contains a boilerplate[7] template you can use to get started writing your PEP. Each PEP must begin with an RFC822 style header preamble. The headers must appear in the following order. Headers marked with `*' are optional and are described below. All other headers are required. PEP: <pep number> Title: <pep title> Version: <cvs version string> Last-Modified: <cvs date string> Author: <list of authors' real names and optionally, email addrs> * Discussions-To: <email address> Status: <Draft | Active | Accepted | Deferred | Final | Replaced> Type: <Informational | Standards Track> * Requires: <pep numbers> Created: <date created on, in dd-mmm-yyyy format> * Python-Version: <version number> Post-History: <dates of postings to python-list and python-dev> * Replaces: <pep number> * Replaced-By: <pep number> The Author: header lists the names and optionally, the email addresses of all the authors/owners of the PEP. The format of the author entry should be address(a)dom.ain (Random J. User) if the email address is included, and just Random J. User if the address is not given. If there are multiple authors, each should be on a separate line following RFC 822 continuation line conventions. Note that personal email addresses in PEPs will be obscured as a defense against spam harvesters. Standards track PEPs must have a Python-Version: header which indicates the version of Python that the feature will be released with. Informational PEPs do not need a Python-Version: header. While a PEP is in private discussions (usually during the initial Draft phase), a Discussions-To: header will indicate the mailing list or URL where the PEP is being discussed. No Discussions-To: header is necessary if the PEP is being discussed privately with the author, or on the python-list or python-dev email mailing lists. Note that email addresses in the Discussions-To: header will not be obscured. Created: records the date that the PEP was assigned a number, while Post-History: is used to record the dates of when new versions of the PEP are posted to python-list and/or python-dev. Both headers should be in dd-mmm-yyyy format, e.g. 14-Aug-2001. PEPs may have a Requires: header, indicating the PEP numbers that this PEP depends on. PEPs may also have a Replaced-By: header indicating that a PEP has been rendered obsolete by a later document; the value is the number of the PEP that replaces the current document. The newer PEP must have a Replaces: header containing the number of the PEP that it rendered obsolete. PEP Formatting Requirements PEP headings must begin in column zero and the initial letter of each word must be capitalized as in book titles. Acronyms should be in all capitals. The body of each section must be indented 4 spaces. Code samples inside body sections should be indented a further 4 spaces, and other indentation can be used as required to make the text readable. You must use two blank lines between the last line of a section's body and the next section heading. You must adhere to the Emacs convention of adding two spaces at the end of every sentence. You should fill your paragraphs to column 70, but under no circumstances should your lines extend past column 79. If your code samples spill over column 79, you should rewrite them. Tab characters must never appear in the document at all. A PEP should include the standard Emacs stanza included by example at the bottom of this PEP. A PEP must contain a Copyright section, and it is strongly recommended to put the PEP in the public domain. When referencing an external web page in the body of a PEP, you should include the title of the page in the text, with a footnote reference to the URL. Do not include the URL in the body text of the PEP. E.g. Refer to the Python Language web site [1] for more details. ... [1] http://www.python.org When referring to another PEP, include the PEP number in the body text, such as "PEP 1". The title may optionally appear. Add a footnote reference that includes the PEP's title and author. It may optionally include the explicit URL on a separate line, but only in the References section. Note that the pep2html.py script will calculate URLs automatically, e.g.: ... Refer to PEP 1 [7] for more information about PEP style ... References [7] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton http://www.python.org/peps/pep-0001.html If you decide to provide an explicit URL for a PEP, please use this as the URL template: http://www.python.org/peps/pep-xxxx.html PEP numbers in URLs must be padded with zeros from the left, so as to be exactly 4 characters wide, however PEP numbers in text are never padded. Reporting PEP Bugs, or Submitting PEP Updates How you report a bug, or submit a PEP update depends on several factors, such as the maturity of the PEP, the preferences of the PEP author, and the nature of your comments. For the early draft stages of the PEP, it's probably best to send your comments and changes directly to the PEP author. For more mature, or finished PEPs you may want to submit corrections to the SourceForge bug manager[6] or better yet, the SourceForge patch manager[2] so that your changes don't get lost. If the PEP author is a SF developer, assign the bug/patch to him, otherwise assign it to the PEP editor. When in doubt about where to send your changes, please check first with the PEP author and/or PEP editor. PEP authors who are also SF committers, can update the PEPs themselves by using "cvs commit" to commit their changes. Remember to also push the formatted PEP text out to the web by doing the following: % python pep2html.py -i NUM where NUM is the number of the PEP you want to push out. See % python pep2html.py --help for details. Transferring PEP Ownership It occasionally becomes necessary to transfer ownership of PEPs to a new champion. In general, we'd like to retain the original author as a co-author of the transferred PEP, but that's really up to the original author. A good reason to transfer ownership is because the original author no longer has the time or interest in updating it or following through with the PEP process, or has fallen off the face of the 'net (i.e. is unreachable or not responding to email). A bad reason to transfer ownership is because you don't agree with the direction of the PEP. We try to build consensus around a PEP, but if that's not possible, you can always submit a competing PEP. If you are interested assuming ownership of a PEP, send a message asking to take over, addressed to both the original author and the PEP editor <peps(a)python.org>. If the original author doesn't respond to email in a timely manner, the PEP editor will make a unilateral decision (it's not like such decisions can be reversed. :). References and Footnotes [1] This historical record is available by the normal CVS commands for retrieving older revisions. For those without direct access to the CVS tree, you can browse the current and past PEP revisions via the SourceForge web site at http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi/python/nondist/peps/?cvsroot=… [2] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [3] http://sourceforge.net/tracker/?atid=355470&group_id=5470&func=browse [4] http://www.opencontent.org/openpub/ [5] The script referred to here is pep2html.py, which lives in the same directory in the CVS tree as the PEPs themselves. Try "pep2html.py --help" for details. The URL for viewing PEPs on the web is http://www.python.org/peps/ [6] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [7] PEP 9, Sample PEP Template http://www.python.org/peps/pep-0009.html Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:

8 14

RE: [Python-Dev] Those import related syntax errors again...
by Samuele Pedroni 23 Feb '05

23 Feb '05

Hi. [Mark Hammond] > The point isn't about my suffering as such. The point is more that > python-dev owns a tiny amount of the code out there, and I don't believe we > should put Python's users through this. > > Sure - I would be happy to "upgrade" all the win32all code, no problem. I > am also happy to live in the bleeding edge and take some pain that will > cause. > > The issue is simply the user base, and giving Python a reputation of not > being able to painlessly upgrade even dot revisions. I agree with all this. [As I imagined explicit syntax did not catch up and would require lot of discussions.] [GvR] > > Another way is to use special rules > > (similar to those for class defs), e.g. having > > > > <frag> > > y=3 > > def f(): > > exec "y=2" > > def g(): > > return y > > return g() > > > > print f() > > </frag> > > > > # print 3. > > > > Is that confusing for users? maybe they will more naturally expect 2 > > as outcome (given nested scopes). > > This seems the best compromise to me. It will lead to the least > broken code, because this is the behavior that we had before nested > scopes! It is also quite easy to implement given the current > implementation, I believe. > > Maybe we could introduce a warning rather than an error for this > situation though, because even if this behavior is clearly documented, > it will still be confusing to some, so it is better if we outlaw it in > some future version. > Yes this can be easy to implement but more confusing situations can arise: <frag> y=3 def f(): y=9 exec "y=2" def g(): return y return y,g() print f() </frag> What should this print? the situation leads not to a canonical solution as class def scopes. or <frag> def f(): from foo import * def g(): return y return g() print f() </frag> [Mark Hammond] > > This probably won't be a very popular suggestion, but how about pulling > > nested scopes (I assume they are at the root of the problem) > > until this can be solved cleanly? > > Agreed. While I think nested scopes are kinda cool, I have lived without > them, and really without missing them, for years. At the moment the cure > appears worse then the symptoms in at least a few cases. If nothing else, > it compromises the elegant simplicity of Python that drew me here in the > first place! > > Assuming that people really _do_ want this feature, IMO the bar should be > raised so there are _zero_ backward compatibility issues. I don't say anything about pulling nested scopes (I don't think my opinion can change things in this respect) but I should insist that without explicit syntax IMO raising the bar has a too high impl cost (both performance and complexity) or creates confusion. [Andrew Kuchling] > >Assuming that people really _do_ want this feature, IMO the bar should be > >raised so there are _zero_ backward compatibility issues. > > Even at the cost of additional implementation complexity? At the cost > of having to learn "scopes are nested, unless you do these two things > in which case they're not"? > > Let's not waffle. If nested scopes are worth doing, they're worth > breaking code. Either leave exec and from..import illegal, or back > out nested scopes, or think of some better solution, but let's not > introduce complicated backward compatibility hacks. IMO breaking code would be ok if we issue warnings today and implement nested scopes issuing errors tomorrow. But this is simply a statement about principles and raised impression. IMO import * in an inner scope should end up being an error, not sure about 'exec's. We will need a final BDFL statement. regards, Samuele Pedroni.

18 78

Windows installer and .pyc files mtime
by Guido van Rossum 16 Sep '03

16 Sep '03

I was watching file modification times on my Windows box (strange hobby, I know :-), and I noticed that after a fresh install of Python, the .pyc files seem to be written when the first code that imports the corresponding module runs, rather than all of the .pyc files being compiled at once by the installer. Wasn't there code in the installer that precompiles all modules? I know the Unix install does this, and I vaguely remember that the Windows installer did this too -- or was it only the Win32all installer??? If there's code to do that in the Windows installer now, it seems it's not working. If there isn't such code, perhaps there should be? --Guido van Rossum (home page: http://www.python.org/~guido/)

6 11

pre-PEP: Resource-Release Support for Generators
by Samuele Pedroni 16 Sep '03

16 Sep '03

this is something we discussed with Guido, and also Moshe Zadka at Europython. Guido thought it seems reasonable enough, if the details can be nailed. I have written it down so the idea doesn't get lost, for the moment is more a matter of whether it can get a number, and then it can go dormant for a while. - * - PEP: XXX Title: Resource-Release Support for Generators Version: $Revision$ Last-Modified: $Date$ Author: Samuele Pedroni <pedronis(a)python.org> Status: Draft Type: Standards Track Content-Type: text/plain Created: 25-Aug-2003 Python-Version: 2.4 Post-History: Abstract Generators allow for natural coding and abstraction of traversal over data. Currently if external resources needing proper timely release are involved, generators are unfortunately not adequate. The typical idiom for timely release is not supported, a yield statement is not allowed in the try clause of a try-finally statement inside a generator. The finally clause execution cannot be either guaranteed or enforced. This PEP proposes that generators support a close method and destruction semantics, such that the restriction can be lifted, expanding the applicability of generators. Rationale Python generators allow for natural coding of many data traversal scenarios Their instantiation produces iterators, i.e. first-class objects abstracting traversal (with all the advantages of first- classness). In this respect they match in power and offer some advantages over the approach using iterator methods taking a (smalltalkish) block. On the other hand, given current limitations (no yield allowed in a try clause of a try-finally inside a generator) the latter approach seems better suited at encapsulating not only traversal but also exception handling and proper resource acquisition and release. Let's consider an example (for simplicity, files in read-mode are used): def all_lines(index_path): for path in file(index_path,"r"): for line in file(path.strip(),"r"): yield line this is short and to the point, but the try-finally for timely closing of the files cannot be added. (While instead of a path, a file, whose closing then would be responsibility of the caller, could be passed in as argument, the same is not applicable for the files opened depending on the contents of the index). If we want timely release, we have to sacrifice the simplicity and directness of the generator-only approach: (e.g.) class AllLines: def __init__(self,index_path): self.index_path = index_path self.index = None self.document = None def __iter__(self): self.index = file(self.index_path,"r") for path in self.index: self.document = file(path.strip(),"r") for line in self.document: yield line self.document.close() self.document = None def close(self): if self.index: self.index.close() if self.document: self.document.close() to be used as: all_lines = AllLines("index.txt") try: for line in all_lines: ... finally: all_lines.close() The more convoluted solution implementing timely release, seems to offer a precious hint. What we have done is encapsulating our traversal in an object (iterator) with a close method. This PEP proposes that generators should grow such a close method with such semantics that the example could be rewritten as: def all_lines(index_path): index = file(index_path,"r") try: for path in file(index_path,"r"): document = file(path.strip(),"r") try: for line in document: yield line finally: document.close() finally: index.close() all = all_lines("index.txt") try: for line in all: ... finally: all.close() PEP 255 [1] disallows yield inside a try clause of a try-finally statement, because the execution of the finally clause cannot be guaranteed as required by try-finally semantics. The semantics of the proposed close method should be such, that while the finally clause execution still cannot be guaranteed, it can be enforced when required. The semantics of generator destruction on the other hand should be extended in order to implement a best-effort policy for the general case. This strikes as a reasonable compromise, the resulting global behavior being similar to that of files and closing. Possible Semantics A close() method should be implemented for generator objects. 1) If a generator is already terminated, close should be a no-op. Otherwise: (two alternative solutions) (Return Semantics) The generator should be resumed, generator execution should continue as if the instruction at re-entry point is a return. Consequently finally clauses surrounding the re-entry point would be executed, in the case of a then allowed try-yield-finally pattern. Issues: is it important to be able to distinguish forced termination by close, normal termination, exception propagation from generator or generator-called code? In the normal case it seems not, finally clauses should be there to work the same in all these cases, still this semantics could make such a distinction hard. Except-clauses, like by a normal return, are not executed, such clauses in legacy generators expect to be executed for exceptions raised by the generator or by code called from it. Not executing them in the close case seems correct. OR (Exception Semantics) The generator should be resumed and execution should continue as if a special-purpose exception (e.g. CloseGenerator) has been raised at re-entry point. Close implementation should consume and not propagate further this exception. Issues: should StopIteration be reused for this purpose? Probably not. We would like close to be a harmless operation for legacy generators, which could contain code catching StopIteration to deal with other generators/iterators. In general, with exception semantics, it is unclear what to do if the generator does not terminate or we do not receive the special exception propagated back. Other different exceptions should probably be propagated, but consider this possible legacy generator code: try: ... yield ... ... except: # or except Exception:, etc raise Exception("boom") If close is invoked with the generator suspended after the yield, the except clause would catch our special purpose exception, so we would get a different exception propagated back, which in this case ought to be reasonably consumed and ignored but in general should be propagated, but separating these scenarios seem hard. The exception approach has the advantage to let the generator distinguish between termination cases and have more control. On the other hand clear-cut semantics seem harder to define. 2) Generator destruction should invoke close method behavior. Remarks If this proposal is accepted, it should become common practice to document whether a generator acquires resources, so that its close method ought to be called. If a generator is no longer used, calling close should be harmless. On the other hand, in the typical scenario the code that instantiated the generator should call close if required by it, generic code dealing with iterators/generators instantiated elsewhere should typically not be littered with close calls. The rare case of code that has acquired ownership of and need to properly deal with all of iterators, generators and generators acquiring resources that need timely release, is easily solved: if hasattr(iterator,'close'): iterator.close() Open Issues Definitive semantics ought to be chosen, implementation issues should be explored. Alternative Ideas The idea that the yield placement limitation should be removed and that generator destruction should trigger execution of finally clauses has been proposed more than once. Alone it cannot guarantee that timely release of resources acquired by a generator can be enforced. PEP 288 [2] proposes a more general solution, allowing custom exception passing to generators. References [1] PEP 255 Simple Generators http://www.python.org/peps/pep-0255.html [2] PEP 288 Generators Attributes and Exceptions http://www.python.org/peps/pep-0288.html Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:

7 14

Be Honest about LC_NUMERIC [REPOST]
by Christian Reis 10 Sep '03

10 Sep '03

So, in an attempt to garner comments (now that we have 2.3 off the chopping block) I'm reposting my PEP proposal (with minor updates). Comments would be appreciated, of course (nudges Barry slightly after him getting me to write this on my only free Sunday in months ;) PEP: XXX Title: Be Honest about LC_NUMERIC (to the C library) Version: $Revision: 1.9 $ Last-Modified: $Date: 2002/08/26 16:29:31 $ Author: Christian R. Reis <kiko at async.com.br> Status: Draft Type: Standards Track Content-Type: text/plain <pep-xxxx.html> Created: 19-July-2003 Post-History: ------------------------------------------------------------------------ Abstract Support in Python for the LC_NUMERIC locale category is currently implemented only in Python-space, which causes inconsistent behavior and thread-safety issues for applications that use extension modules and libraries implemented in C. This document proposes a plan for removing this inconsistency by providing and using substitute locale-agnostic functions as necessary. Introduction Python currently provides generic localization services through the locale module, which among other things allows localizing the display and conversion process of numeric types. Locale categories, such as LC_TIME and LC_COLLATE, allow configuring precisely what aspects of the application are to be localized. The LC_NUMERIC category specifies formatting for non-monetary numeric information, such as the decimal separator in float and fixed-precision numbers. Localization of the LC_NUMERIC category is currently implemented in only in Python-space; the C libraries are unaware of the application's LC_NUMERIC setting. This is done to avoid changing the behavior of certain low-level functions that are used by the Python parser and related code [2]. However, this presents a problem for extension modules that wrap C libraries; applications that use these extension modules will inconsistently display and convert numeric values. James Henstridge, the author of PyGTK [3], has additionally pointed out that the setlocale() function also presents thread-safety issues, since a thread may call the C library setlocale() outside of the GIL, and cause Python to function incorrectly. Rationale The inconsistency between Python and C library localization for LC_NUMERIC is a problem for any localized application using C extensions. The exact nature of the problem will vary depending on the application, but it will most likely occur when parsing or formatting a numeric value. Example Problem The initial problem that motivated this PEP is related to the GtkSpinButton [4] widget in the GTK+ UI toolkit, wrapped by PyGTK. The widget can be set to numeric mode, and when this occurs, characters typed into it are evaluated as a number. Because LC_NUMERIC is not set in libc, float values are displayed incorrectly, and it is impossible to enter values using the localized decimal separator (for instance, `,' for the Brazilian locale pt_BR). This small example demonstrates reduced usability for localized applications using this toolkit when coded in Python. Proposal Martin v. Löwis commented on the initial constraints for an acceptable solution to the problem on python-dev: - LC_NUMERIC can be set at the C library level without breaking the parser. - float() and str() stay locale-unaware. The following seems to be the current practice: - locale-aware str() and float() [XXX: atof(), currently?] stay in the locale module. An analysis of the Python source suggests that the following functions currently depend on LC_NUMERIC being set to the C locale: - Python/compile.c:parsenumber() - Python/marshal.c:r_object() - Objects/complexobject.c:complex_to_buf() - Objects/complexobject.c:complex_subtype_from_string() - Objects/floatobject.c:PyFloat_FromString() - Objects/floatobject.c:format_float() - Modules/stropmodule.c:strop_atof() - Modules/cPickle.c:load_float() [XXX: still need to check if any other occurrences exist] The proposed approach is to implement LC_NUMERIC-agnostic functions for converting from (strtod()/atof()) and to (snprintf()) float formats, using these functions where the formatting should not vary according to the user-specified locale. This change should also solve the aforementioned thread-safety problems. Potential Code Contributions This problem was initially reported as a problem in the GTK+ libraries [5]; since then it has been correctly diagnosed as an inconsistency in Python's implementation. However, in a fortunate coincidence, the glib library implements a number of LC_NUMERIC-agnostic functions (for an example, see [6]) for reasons similar to those presented in this paper. In the same GTK+ problem report, Havoc Pennington has suggested that the glib authors would be willing to contribute this code to the PSF, which would simplify implementation of this PEP considerably. [I'm checking if Alex Larsson is willing to sign the PSF contributor agreement [7] to make sure the code is safe to integrate; XXX: what would be necessary to sign here?] Risks There may be cross-platform issues with the provided locale-agnostic functions. This needs to be tested further. Martin has pointed out potential copyright problems with the contributed code. I believe we will have no problems in this area as members of the GTK+ and glib teams have said they are fine with relicensing the code. Code An implementation is being developed by Gustavo Carneiro <gjc at inescporto.pt>. It is currently attached to Sourceforge.net bug 744665 [8] [XXX: The SF.net tracker is horrible 8(] References [1] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton http://www.python.org/peps/pep-0001.html [2] Python locale documentation for embedding, http://www.python.org/doc/current/lib/embedding-locale.html [3] PyGTK homepage, http://www.daa.com.au/~james/pygtk/ [4] GtkSpinButton screenshot (demonstrating problem), http://www.async.com.br/~kiko/spin.png [5] GNOME bug report, http://bugzilla.gnome.org/show_bug.cgi?id=114132 [6] Code submission of g_ascii_strtod and g_ascii_dtostr (later renamed g_ascii_formatd) by Alex Larsson, http://mail.gnome.org/archives/gtk-devel-list/2001-October/msg00114.html [7] PSF Contributor Agreement, http://www.python.org/psf/psf-contributor-agreement.html [8] Python bug report, http://www.python.org/sf/774665 Copyright This document has been placed in the public domain. Take care, -- Christian Reis, Senior Engineer, Async Open Source, Brazil. http://async.com.br/~kiko/ | [+55 16] 261 2331 | NMFL

9 32

readline.clear_history()
by Phillip J. Eby 05 Sep '03

05 Sep '03

I've been working on a set of Python tools that use readline, but want to keep history separate between different interaction modes. Unfortunately, this really needs to be able to access readline's clear_history(), as read_history_file() leaves existing history intact. I'd be happy to whip up a patch to add this (as readline.clear_history()), but I was wondering if perhaps the reason it's not currently exported by the readline module is a compatibility issue for older readline implementations that are officially supported. Thanks.

7 15

Weekly Python Bug/Patch Summary
by Skip Montanaro 31 Aug '03

31 Aug '03

Bug/Patch Summary ----------------- 470 open / 4073 total bugs (+3) 187 open / 2342 total patches (+4) New Bugs -------- Cannot retrieve name of super object (2003-04-28) http://python.org/sf/729103 cygwin builds do not embed (2003-08-24) http://python.org/sf/794140 double symlinking corrupts sys.path[0] (2003-08-24) http://python.org/sf/794291 email bug with message/rfc822 (2003-08-24) http://python.org/sf/794458 email module param parsing bug (2003-08-24) http://python.org/sf/794466 Named groups limitation in sre (2003-08-25) http://python.org/sf/794819 email.Message param parsing problem II (2003-08-25) http://python.org/sf/795081 Documentation (2003-08-26) http://python.org/sf/795649 sdist ignores scripts argument in setup (2003-08-27) http://python.org/sf/796042 Parser wart (2003-08-27) http://python.org/sf/796116 ntpath.expanduser() is still wrong (2003-08-27) http://python.org/sf/796219 locale.setlocale(locale.LC_ALL, "de") raises exception (2003-08-29) http://python.org/sf/797447 csv.DictWriter -- inconsistent doc & code (2003-08-30) http://python.org/sf/797844 Small problems with the csv module's documentation (2003-08-30) http://python.org/sf/797853 select module doesn't allow any iterable. (2003-08-31) http://python.org/sf/798046 IDLE / PyOS_InputHook (2003-08-31) http://python.org/sf/798058 New Patches ----------- startup file compiler flags (2003-08-24) http://python.org/sf/794400 __file__ attribute missing from dynamicly loaded module (2003-08-25) http://python.org/sf/794826 license inconsistency (2003-08-26) http://python.org/sf/795531 CGIHTTPServer fix (2003-08-28) http://python.org/sf/796772 decode message attachments in email.Message (2003-08-28) http://python.org/sf/796908 Windows installer changes for 2.3.1 (2003-08-28) http://python.org/sf/796919 Closed Bugs ----------- An extended definition of "non-overlapping" would save time. (2003-05-04) http://python.org/sf/732120 urllib: open_https generates wrong Authorize header (2003-07-23) http://python.org/sf/776542 Bug in "Build and C API Changes" section of what's new doc? (2003-07-30) http://python.org/sf/780231 ImportError: cannot import name __all__ (2003-08-04) http://python.org/sf/782752 Fatal Python error: unknown scope (2003-08-06) http://python.org/sf/784075 zlib monotonic test -- false assumptions or bug in zlib.lib (2003-08-08) http://python.org/sf/785222 compiler.compileFile fails on csv.py (2003-08-13) http://python.org/sf/788011 Cannot decode/encode anything! (2003-08-13) http://python.org/sf/788457 curses.ascii.unctrl broken (2003-08-17) http://python.org/sf/790356 Tutorial: mutable objects may be keys (2003-08-19) http://python.org/sf/791397 Re: Bugs item #765456 (2003-08-19) http://python.org/sf/791445 Misleading [i:j:k] slicing description (2003-08-21) http://python.org/sf/792656 runnig thread multiple times (2003-08-23) http://python.org/sf/793687 Section 13.1 HTMLParser documentation error (2003-08-23) http://python.org/sf/793702 using itertools.izip to mutate tuples (2003-08-23) http://python.org/sf/793826 Closed Patches -------------- Port tests to unittest (Part 2) (2003-05-13) http://python.org/sf/736962 test_largefile cleanup patch (2003-08-11) http://python.org/sf/786591

1 0

Berkeley breakage
by Tim Peters 30 Aug '03

30 Aug '03

Over in the spambayes project, we get reports of database corruption from people using Sleepycat bsddb. The most recent comment on a bug report is revealing: http://sf.net/tracker/?func=detail&atid=498103&aid=788051&group_id=61702 """ Date: 2003-08-18 04:09 Sender: fufsource i did some experimenting with various bsddb3 versions: - with db-3.3.11 the python segfaults and core is dumped - with db-3.2.9 the database is corrupted - with db-4.1.25 everything works as it should (no db corruption) """ spambayes makes elementary use of a Berkeley DB, just accessing via the dict interface -- inserts, deletes and lookups, but no cursors, no transactions, nothing "fancy". I don't have time to dig into this, but assuming the report is correct, how can we "encourage" Unix weenies to use db-4.1.25? (On Windows, db-4.1.25 is shipped with the installer.) If the problems with older versions are so severe, maybe the Python wrapper should do a version check and refuse to run if it finds an old version?

5 6

RE: [Spambayes] Question (or possibly a bug report)
by Meyer, Tony 30 Aug '03

30 Aug '03

> So when the marshalled representation of 0.001 is loaded under > "german" LC_NUMERIC here, we get back exactly 0.0. I'm not > sure why. When I call "marshal.dumps(0.1)" from AsyncDialog (or anywhere in the Outlook code) I get "f\x030.0", which fits with what you have. > So the obvious <wink> answers are: (Glad you posted this - I was wading through the progress of marshalling (PyOS_snprintf etc) and getting rapidly lost). > 1. When LC_NUMERIC is "german", MS C's atof() stops at the first > period it sees. This is the case: """ #include <locale.h> #include <stdio.h> #include <stdlib.h> int main() { float f; setlocale(LC_NUMERIC, "german"); f = atof("0.1"); printf("%f\n", f); } """ Gives me with gcc version 3.2 20020927 (prerelease): 0.100000 Gives me with Microsoft C++ Builder (I don't have Visual C++ handy, but I suppose it would be the same): 0,00000 The help file for Builder does say that this is the correct behaviour - it will stop when it finds an unrecognised character - here '.' is unrecognised (because we are in German), so it stops. Does this then mean that this is a Python bug? Or because Python tells us not to change the c locale and we (Outlook) are, it's our fault/problem? Presumably what we'll have to do for a solution is just what Mark is doing now - find the correct place to put a call that (re)sets the c locale to English. =Tony Meyer

7 15

pyconfig.h installation issue
by Tom Emerson 30 Aug '03

30 Aug '03

Greetings, I am building Python 2.3 in an heterogenous environment where I want all of the platform-independent code to be shared and the platform specific code kept segregated. In particular I have a structure like /foo/bar/python/include /foo/bar/python/lib /foo/bar/python/man for the platform independent files, and /foo/bar/python/RH72-gcc-3.2 /foo/bar/python/sol-CC ... for platform-specific directories and files (bin/, include/, and lib/). Getting this to work is easily accomplished with --prefix and --exec-prefix when running configure on each platform: --prefix=/foo/bar/python --exec-prefix=/foo/bar/python/RH72-gcc-3.2 But... These builds are put under source control and are read-only once they are checked in. Hence the problem: If /foo/bar/python/include and its contents are read-only then following installs fail to put pyconfig.h (a platform-dependent file) into the platform-specific include directory because the $(INSTALL_DATA) in the 'inclinstall' target fails and the final line in that target (to install pyconfig.h) never runs. Clear as mud? The main thing I want to do is avoid reinstalling the shared files after the first time, since I'm building this on many different platforms. My proposed solution to this is to move the $(INSTALL_DATA) for pyconfig.h from the bottom of the inclinstall target to before the loop that installs the shared headers. Then that step will succeed and it doesn't matter whether or not the other INSTALL_DATA calls fail. Make sense? This begs another install target, which just installs the items that end up in the exec-prefix directories. Then on each platform I can just install the platform-specific code. Thoughts? Thanks in advance, tree -- Tom Emerson Basis Technology Corp. Software Architect http://www.basistech.com "Beware the lollipop of mediocrity: lick it once and you suck forever"

2 1

Jump to page:

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev August 2003