Python-Dev August 2003

python-dev@python.org

119 participants
134 discussions

RE: [Python-Dev] Those import related syntax errors again...

by Samuele Pedroni

Hi. [Mark Hammond] > The point isn't about my suffering as such. The point is more that > python-dev owns a tiny amount of the code out there, and I don't believe we > should put Python's users through this. > > Sure - I would be happy to "upgrade" all the win32all code, no problem. I > am also happy to live in the bleeding edge and take some pain that will > cause. > > The issue is simply the user base, and giving Python a reputation of not > being able to painlessly upgrade even dot revisions. I agree with all this. [As I imagined explicit syntax did not catch up and would require lot of discussions.] [GvR] > > Another way is to use special rules > > (similar to those for class defs), e.g. having > > > > <frag> > > y=3 > > def f(): > > exec "y=2" > > def g(): > > return y > > return g() > > > > print f() > > </frag> > > > > # print 3. > > > > Is that confusing for users? maybe they will more naturally expect 2 > > as outcome (given nested scopes). > > This seems the best compromise to me. It will lead to the least > broken code, because this is the behavior that we had before nested > scopes! It is also quite easy to implement given the current > implementation, I believe. > > Maybe we could introduce a warning rather than an error for this > situation though, because even if this behavior is clearly documented, > it will still be confusing to some, so it is better if we outlaw it in > some future version. > Yes this can be easy to implement but more confusing situations can arise: <frag> y=3 def f(): y=9 exec "y=2" def g(): return y return y,g() print f() </frag> What should this print? the situation leads not to a canonical solution as class def scopes. or <frag> def f(): from foo import * def g(): return y return g() print f() </frag> [Mark Hammond] > > This probably won't be a very popular suggestion, but how about pulling > > nested scopes (I assume they are at the root of the problem) > > until this can be solved cleanly? > > Agreed. While I think nested scopes are kinda cool, I have lived without > them, and really without missing them, for years. At the moment the cure > appears worse then the symptoms in at least a few cases. If nothing else, > it compromises the elegant simplicity of Python that drew me here in the > first place! > > Assuming that people really _do_ want this feature, IMO the bar should be > raised so there are _zero_ backward compatibility issues. I don't say anything about pulling nested scopes (I don't think my opinion can change things in this respect) but I should insist that without explicit syntax IMO raising the bar has a too high impl cost (both performance and complexity) or creates confusion. [Andrew Kuchling] > >Assuming that people really _do_ want this feature, IMO the bar should be > >raised so there are _zero_ backward compatibility issues. > > Even at the cost of additional implementation complexity? At the cost > of having to learn "scopes are nested, unless you do these two things > in which case they're not"? > > Let's not waffle. If nested scopes are worth doing, they're worth > breaking code. Either leave exec and from..import illegal, or back > out nested scopes, or think of some better solution, but let's not > introduce complicated backward compatibility hacks. IMO breaking code would be ok if we issue warnings today and implement nested scopes issuing errors tomorrow. But this is simply a statement about principles and raised impression. IMO import * in an inner scope should end up being an error, not sure about 'exec's. We will need a final BDFL statement. regards, Samuele Pedroni.

15 years, 7 months

Windows installer and .pyc files mtime

by Guido van Rossum

I was watching file modification times on my Windows box (strange hobby, I know :-), and I noticed that after a fresh install of Python, the .pyc files seem to be written when the first code that imports the corresponding module runs, rather than all of the .pyc files being compiled at once by the installer. Wasn't there code in the installer that precompiles all modules? I know the Unix install does this, and I vaguely remember that the Windows installer did this too -- or was it only the Win32all installer??? If there's code to do that in the Windows installer now, it seems it's not working. If there isn't such code, perhaps there should be? --Guido van Rossum (home page: http://www.python.org/~guido/)

17 years

pre-PEP: Resource-Release Support for Generators

by Samuele Pedroni

this is something we discussed with Guido, and also Moshe Zadka at Europython. Guido thought it seems reasonable enough, if the details can be nailed. I have written it down so the idea doesn't get lost, for the moment is more a matter of whether it can get a number, and then it can go dormant for a while. - * - PEP: XXX Title: Resource-Release Support for Generators Version: $Revision$ Last-Modified: $Date$ Author: Samuele Pedroni <pedronis(a)python.org> Status: Draft Type: Standards Track Content-Type: text/plain Created: 25-Aug-2003 Python-Version: 2.4 Post-History: Abstract Generators allow for natural coding and abstraction of traversal over data. Currently if external resources needing proper timely release are involved, generators are unfortunately not adequate. The typical idiom for timely release is not supported, a yield statement is not allowed in the try clause of a try-finally statement inside a generator. The finally clause execution cannot be either guaranteed or enforced. This PEP proposes that generators support a close method and destruction semantics, such that the restriction can be lifted, expanding the applicability of generators. Rationale Python generators allow for natural coding of many data traversal scenarios Their instantiation produces iterators, i.e. first-class objects abstracting traversal (with all the advantages of first- classness). In this respect they match in power and offer some advantages over the approach using iterator methods taking a (smalltalkish) block. On the other hand, given current limitations (no yield allowed in a try clause of a try-finally inside a generator) the latter approach seems better suited at encapsulating not only traversal but also exception handling and proper resource acquisition and release. Let's consider an example (for simplicity, files in read-mode are used): def all_lines(index_path): for path in file(index_path,"r"): for line in file(path.strip(),"r"): yield line this is short and to the point, but the try-finally for timely closing of the files cannot be added. (While instead of a path, a file, whose closing then would be responsibility of the caller, could be passed in as argument, the same is not applicable for the files opened depending on the contents of the index). If we want timely release, we have to sacrifice the simplicity and directness of the generator-only approach: (e.g.) class AllLines: def __init__(self,index_path): self.index_path = index_path self.index = None self.document = None def __iter__(self): self.index = file(self.index_path,"r") for path in self.index: self.document = file(path.strip(),"r") for line in self.document: yield line self.document.close() self.document = None def close(self): if self.index: self.index.close() if self.document: self.document.close() to be used as: all_lines = AllLines("index.txt") try: for line in all_lines: ... finally: all_lines.close() The more convoluted solution implementing timely release, seems to offer a precious hint. What we have done is encapsulating our traversal in an object (iterator) with a close method. This PEP proposes that generators should grow such a close method with such semantics that the example could be rewritten as: def all_lines(index_path): index = file(index_path,"r") try: for path in file(index_path,"r"): document = file(path.strip(),"r") try: for line in document: yield line finally: document.close() finally: index.close() all = all_lines("index.txt") try: for line in all: ... finally: all.close() PEP 255 [1] disallows yield inside a try clause of a try-finally statement, because the execution of the finally clause cannot be guaranteed as required by try-finally semantics. The semantics of the proposed close method should be such, that while the finally clause execution still cannot be guaranteed, it can be enforced when required. The semantics of generator destruction on the other hand should be extended in order to implement a best-effort policy for the general case. This strikes as a reasonable compromise, the resulting global behavior being similar to that of files and closing. Possible Semantics A close() method should be implemented for generator objects. 1) If a generator is already terminated, close should be a no-op. Otherwise: (two alternative solutions) (Return Semantics) The generator should be resumed, generator execution should continue as if the instruction at re-entry point is a return. Consequently finally clauses surrounding the re-entry point would be executed, in the case of a then allowed try-yield-finally pattern. Issues: is it important to be able to distinguish forced termination by close, normal termination, exception propagation from generator or generator-called code? In the normal case it seems not, finally clauses should be there to work the same in all these cases, still this semantics could make such a distinction hard. Except-clauses, like by a normal return, are not executed, such clauses in legacy generators expect to be executed for exceptions raised by the generator or by code called from it. Not executing them in the close case seems correct. OR (Exception Semantics) The generator should be resumed and execution should continue as if a special-purpose exception (e.g. CloseGenerator) has been raised at re-entry point. Close implementation should consume and not propagate further this exception. Issues: should StopIteration be reused for this purpose? Probably not. We would like close to be a harmless operation for legacy generators, which could contain code catching StopIteration to deal with other generators/iterators. In general, with exception semantics, it is unclear what to do if the generator does not terminate or we do not receive the special exception propagated back. Other different exceptions should probably be propagated, but consider this possible legacy generator code: try: ... yield ... ... except: # or except Exception:, etc raise Exception("boom") If close is invoked with the generator suspended after the yield, the except clause would catch our special purpose exception, so we would get a different exception propagated back, which in this case ought to be reasonably consumed and ignored but in general should be propagated, but separating these scenarios seem hard. The exception approach has the advantage to let the generator distinguish between termination cases and have more control. On the other hand clear-cut semantics seem harder to define. 2) Generator destruction should invoke close method behavior. Remarks If this proposal is accepted, it should become common practice to document whether a generator acquires resources, so that its close method ought to be called. If a generator is no longer used, calling close should be harmless. On the other hand, in the typical scenario the code that instantiated the generator should call close if required by it, generic code dealing with iterators/generators instantiated elsewhere should typically not be littered with close calls. The rare case of code that has acquired ownership of and need to properly deal with all of iterators, generators and generators acquiring resources that need timely release, is easily solved: if hasattr(iterator,'close'): iterator.close() Open Issues Definitive semantics ought to be chosen, implementation issues should be explored. Alternative Ideas The idea that the yield placement limitation should be removed and that generator destruction should trigger execution of finally clauses has been proposed more than once. Alone it cannot guarantee that timely release of resources acquired by a generator can be enforced. PEP 288 [2] proposes a more general solution, allowing custom exception passing to generators. References [1] PEP 255 Simple Generators http://www.python.org/peps/pep-0255.html [2] PEP 288 Generators Attributes and Exceptions http://www.python.org/peps/pep-0288.html Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:

17 years

Be Honest about LC_NUMERIC [REPOST]

by Christian Reis

So, in an attempt to garner comments (now that we have 2.3 off the chopping block) I'm reposting my PEP proposal (with minor updates). Comments would be appreciated, of course (nudges Barry slightly after him getting me to write this on my only free Sunday in months ;) PEP: XXX Title: Be Honest about LC_NUMERIC (to the C library) Version: $Revision: 1.9 $ Last-Modified: $Date: 2002/08/26 16:29:31 $ Author: Christian R. Reis <kiko at async.com.br> Status: Draft Type: Standards Track Content-Type: text/plain <pep-xxxx.html> Created: 19-July-2003 Post-History: ------------------------------------------------------------------------ Abstract Support in Python for the LC_NUMERIC locale category is currently implemented only in Python-space, which causes inconsistent behavior and thread-safety issues for applications that use extension modules and libraries implemented in C. This document proposes a plan for removing this inconsistency by providing and using substitute locale-agnostic functions as necessary. Introduction Python currently provides generic localization services through the locale module, which among other things allows localizing the display and conversion process of numeric types. Locale categories, such as LC_TIME and LC_COLLATE, allow configuring precisely what aspects of the application are to be localized. The LC_NUMERIC category specifies formatting for non-monetary numeric information, such as the decimal separator in float and fixed-precision numbers. Localization of the LC_NUMERIC category is currently implemented in only in Python-space; the C libraries are unaware of the application's LC_NUMERIC setting. This is done to avoid changing the behavior of certain low-level functions that are used by the Python parser and related code [2]. However, this presents a problem for extension modules that wrap C libraries; applications that use these extension modules will inconsistently display and convert numeric values. James Henstridge, the author of PyGTK [3], has additionally pointed out that the setlocale() function also presents thread-safety issues, since a thread may call the C library setlocale() outside of the GIL, and cause Python to function incorrectly. Rationale The inconsistency between Python and C library localization for LC_NUMERIC is a problem for any localized application using C extensions. The exact nature of the problem will vary depending on the application, but it will most likely occur when parsing or formatting a numeric value. Example Problem The initial problem that motivated this PEP is related to the GtkSpinButton [4] widget in the GTK+ UI toolkit, wrapped by PyGTK. The widget can be set to numeric mode, and when this occurs, characters typed into it are evaluated as a number. Because LC_NUMERIC is not set in libc, float values are displayed incorrectly, and it is impossible to enter values using the localized decimal separator (for instance, `,' for the Brazilian locale pt_BR). This small example demonstrates reduced usability for localized applications using this toolkit when coded in Python. Proposal Martin v. Löwis commented on the initial constraints for an acceptable solution to the problem on python-dev: - LC_NUMERIC can be set at the C library level without breaking the parser. - float() and str() stay locale-unaware. The following seems to be the current practice: - locale-aware str() and float() [XXX: atof(), currently?] stay in the locale module. An analysis of the Python source suggests that the following functions currently depend on LC_NUMERIC being set to the C locale: - Python/compile.c:parsenumber() - Python/marshal.c:r_object() - Objects/complexobject.c:complex_to_buf() - Objects/complexobject.c:complex_subtype_from_string() - Objects/floatobject.c:PyFloat_FromString() - Objects/floatobject.c:format_float() - Modules/stropmodule.c:strop_atof() - Modules/cPickle.c:load_float() [XXX: still need to check if any other occurrences exist] The proposed approach is to implement LC_NUMERIC-agnostic functions for converting from (strtod()/atof()) and to (snprintf()) float formats, using these functions where the formatting should not vary according to the user-specified locale. This change should also solve the aforementioned thread-safety problems. Potential Code Contributions This problem was initially reported as a problem in the GTK+ libraries [5]; since then it has been correctly diagnosed as an inconsistency in Python's implementation. However, in a fortunate coincidence, the glib library implements a number of LC_NUMERIC-agnostic functions (for an example, see [6]) for reasons similar to those presented in this paper. In the same GTK+ problem report, Havoc Pennington has suggested that the glib authors would be willing to contribute this code to the PSF, which would simplify implementation of this PEP considerably. [I'm checking if Alex Larsson is willing to sign the PSF contributor agreement [7] to make sure the code is safe to integrate; XXX: what would be necessary to sign here?] Risks There may be cross-platform issues with the provided locale-agnostic functions. This needs to be tested further. Martin has pointed out potential copyright problems with the contributed code. I believe we will have no problems in this area as members of the GTK+ and glib teams have said they are fine with relicensing the code. Code An implementation is being developed by Gustavo Carneiro <gjc at inescporto.pt>. It is currently attached to Sourceforge.net bug 744665 [8] [XXX: The SF.net tracker is horrible 8(] References [1] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton http://www.python.org/peps/pep-0001.html [2] Python locale documentation for embedding, http://www.python.org/doc/current/lib/embedding-locale.html [3] PyGTK homepage, http://www.daa.com.au/~james/pygtk/ [4] GtkSpinButton screenshot (demonstrating problem), http://www.async.com.br/~kiko/spin.png [5] GNOME bug report, http://bugzilla.gnome.org/show_bug.cgi?id=114132 [6] Code submission of g_ascii_strtod and g_ascii_dtostr (later renamed g_ascii_formatd) by Alex Larsson, http://mail.gnome.org/archives/gtk-devel-list/2001-October/msg00114.html [7] PSF Contributor Agreement, http://www.python.org/psf/psf-contributor-agreement.html [8] Python bug report, http://www.python.org/sf/774665 Copyright This document has been placed in the public domain. Take care, -- Christian Reis, Senior Engineer, Async Open Source, Brazil. http://async.com.br/~kiko/ | [+55 16] 261 2331 | NMFL

17 years

readline.clear_history()

by Phillip J. Eby

I've been working on a set of Python tools that use readline, but want to keep history separate between different interaction modes. Unfortunately, this really needs to be able to access readline's clear_history(), as read_history_file() leaves existing history intact. I'd be happy to whip up a patch to add this (as readline.clear_history()), but I was wondering if perhaps the reason it's not currently exported by the readline module is a compatibility issue for older readline implementations that are officially supported. Thanks.

17 years

Weekly Python Bug/Patch Summary

by Skip Montanaro

Bug/Patch Summary ----------------- 470 open / 4073 total bugs (+3) 187 open / 2342 total patches (+4) New Bugs -------- Cannot retrieve name of super object (2003-04-28) http://python.org/sf/729103 cygwin builds do not embed (2003-08-24) http://python.org/sf/794140 double symlinking corrupts sys.path[0] (2003-08-24) http://python.org/sf/794291 email bug with message/rfc822 (2003-08-24) http://python.org/sf/794458 email module param parsing bug (2003-08-24) http://python.org/sf/794466 Named groups limitation in sre (2003-08-25) http://python.org/sf/794819 email.Message param parsing problem II (2003-08-25) http://python.org/sf/795081 Documentation (2003-08-26) http://python.org/sf/795649 sdist ignores scripts argument in setup (2003-08-27) http://python.org/sf/796042 Parser wart (2003-08-27) http://python.org/sf/796116 ntpath.expanduser() is still wrong (2003-08-27) http://python.org/sf/796219 locale.setlocale(locale.LC_ALL, "de") raises exception (2003-08-29) http://python.org/sf/797447 csv.DictWriter -- inconsistent doc & code (2003-08-30) http://python.org/sf/797844 Small problems with the csv module's documentation (2003-08-30) http://python.org/sf/797853 select module doesn't allow any iterable. (2003-08-31) http://python.org/sf/798046 IDLE / PyOS_InputHook (2003-08-31) http://python.org/sf/798058 New Patches ----------- startup file compiler flags (2003-08-24) http://python.org/sf/794400 __file__ attribute missing from dynamicly loaded module (2003-08-25) http://python.org/sf/794826 license inconsistency (2003-08-26) http://python.org/sf/795531 CGIHTTPServer fix (2003-08-28) http://python.org/sf/796772 decode message attachments in email.Message (2003-08-28) http://python.org/sf/796908 Windows installer changes for 2.3.1 (2003-08-28) http://python.org/sf/796919 Closed Bugs ----------- An extended definition of "non-overlapping" would save time. (2003-05-04) http://python.org/sf/732120 urllib: open_https generates wrong Authorize header (2003-07-23) http://python.org/sf/776542 Bug in "Build and C API Changes" section of what's new doc? (2003-07-30) http://python.org/sf/780231 ImportError: cannot import name __all__ (2003-08-04) http://python.org/sf/782752 Fatal Python error: unknown scope (2003-08-06) http://python.org/sf/784075 zlib monotonic test -- false assumptions or bug in zlib.lib (2003-08-08) http://python.org/sf/785222 compiler.compileFile fails on csv.py (2003-08-13) http://python.org/sf/788011 Cannot decode/encode anything! (2003-08-13) http://python.org/sf/788457 curses.ascii.unctrl broken (2003-08-17) http://python.org/sf/790356 Tutorial: mutable objects may be keys (2003-08-19) http://python.org/sf/791397 Re: Bugs item #765456 (2003-08-19) http://python.org/sf/791445 Misleading [i:j:k] slicing description (2003-08-21) http://python.org/sf/792656 runnig thread multiple times (2003-08-23) http://python.org/sf/793687 Section 13.1 HTMLParser documentation error (2003-08-23) http://python.org/sf/793702 using itertools.izip to mutate tuples (2003-08-23) http://python.org/sf/793826 Closed Patches -------------- Port tests to unittest (Part 2) (2003-05-13) http://python.org/sf/736962 test_largefile cleanup patch (2003-08-11) http://python.org/sf/786591

17 years, 1 month

Berkeley breakage

by Tim Peters

Over in the spambayes project, we get reports of database corruption from people using Sleepycat bsddb. The most recent comment on a bug report is revealing: http://sf.net/tracker/?func=detail&atid=498103&aid=788051&group_id=61702 """ Date: 2003-08-18 04:09 Sender: fufsource i did some experimenting with various bsddb3 versions: - with db-3.3.11 the python segfaults and core is dumped - with db-3.2.9 the database is corrupted - with db-4.1.25 everything works as it should (no db corruption) """ spambayes makes elementary use of a Berkeley DB, just accessing via the dict interface -- inserts, deletes and lookups, but no cursors, no transactions, nothing "fancy". I don't have time to dig into this, but assuming the report is correct, how can we "encourage" Unix weenies to use db-4.1.25? (On Windows, db-4.1.25 is shipped with the installer.) If the problems with older versions are so severe, maybe the Python wrapper should do a version check and refuse to run if it finds an old version?

17 years, 1 month

RE: [Spambayes] Question (or possibly a bug report)

by Meyer, Tony

> So when the marshalled representation of 0.001 is loaded under > "german" LC_NUMERIC here, we get back exactly 0.0. I'm not > sure why. When I call "marshal.dumps(0.1)" from AsyncDialog (or anywhere in the Outlook code) I get "f\x030.0", which fits with what you have. > So the obvious <wink> answers are: (Glad you posted this - I was wading through the progress of marshalling (PyOS_snprintf etc) and getting rapidly lost). > 1. When LC_NUMERIC is "german", MS C's atof() stops at the first > period it sees. This is the case: """ #include <locale.h> #include <stdio.h> #include <stdlib.h> int main() { float f; setlocale(LC_NUMERIC, "german"); f = atof("0.1"); printf("%f\n", f); } """ Gives me with gcc version 3.2 20020927 (prerelease): 0.100000 Gives me with Microsoft C++ Builder (I don't have Visual C++ handy, but I suppose it would be the same): 0,00000 The help file for Builder does say that this is the correct behaviour - it will stop when it finds an unrecognised character - here '.' is unrecognised (because we are in German), so it stops. Does this then mean that this is a Python bug? Or because Python tells us not to change the c locale and we (Outlook) are, it's our fault/problem? Presumably what we'll have to do for a solution is just what Mark is doing now - find the correct place to put a call that (re)sets the c locale to English. =Tony Meyer

17 years, 1 month

pyconfig.h installation issue

by Tom Emerson

Greetings, I am building Python 2.3 in an heterogenous environment where I want all of the platform-independent code to be shared and the platform specific code kept segregated. In particular I have a structure like /foo/bar/python/include /foo/bar/python/lib /foo/bar/python/man for the platform independent files, and /foo/bar/python/RH72-gcc-3.2 /foo/bar/python/sol-CC ... for platform-specific directories and files (bin/, include/, and lib/). Getting this to work is easily accomplished with --prefix and --exec-prefix when running configure on each platform: --prefix=/foo/bar/python --exec-prefix=/foo/bar/python/RH72-gcc-3.2 But... These builds are put under source control and are read-only once they are checked in. Hence the problem: If /foo/bar/python/include and its contents are read-only then following installs fail to put pyconfig.h (a platform-dependent file) into the platform-specific include directory because the $(INSTALL_DATA) in the 'inclinstall' target fails and the final line in that target (to install pyconfig.h) never runs. Clear as mud? The main thing I want to do is avoid reinstalling the shared files after the first time, since I'm building this on many different platforms. My proposed solution to this is to move the $(INSTALL_DATA) for pyconfig.h from the bottom of the inclinstall target to before the loop that installs the shared headers. Then that step will succeed and it doesn't matter whether or not the other INSTALL_DATA calls fail. Make sense? This begs another install target, which just installs the items that end up in the exec-prefix directories. Then on each platform I can just install the platform-specific code. Thoughts? Thanks in advance, tree -- Tom Emerson Basis Technology Corp. Software Architect http://www.basistech.com "Beware the lollipop of mediocrity: lick it once and you suck forever"

17 years, 1 month

python 2.3 bug patch

by Harald Koenig

Hi, compiling python-2.3 on various platforms I needed the attached patch. the first two patches to Makefile.pre.in and Modules/makesetup are needed if source dir and compile dir are not the same. Modules/resource.c patch fixes a problem for Solaris 2.5.1 which doesn't define _SC_PAGE_SIZE but _SC_PAGESIZE in unistd.h. other than those small patches, python-2.3 compiled fine on the following systems using either gcc-2.95.3 or gcc-3.3: alpha.OSF1-V4.0 hp9000s700.HP-UX-B.10.20 hp9000s700.HP-UX-B.11.00 i386-linux.RedHat-6.2 i386-linux.RedHat-7.1 i386-linux.suse62 i386-linux.suse72 iris4d.IRIX64-6.5 rs6000.AIX-4.3 sun4.SunOS-5.5.1 sun4.SunOS-5.8 Harald Koenig -- "I hope to die ___ _____ before I *have* to use Microsoft Word.", 0--,| /OOOOOOO\ Donald E. Knuth, 02-Oct-2001 in Tuebingen. <_/ / /OOOOOOOOOOO\ \ \/OOOOOOOOOOOOOOO\ \ OOOOOOOOOOOOOOOOO|// Harald Koenig \/\/\/\/\/\/\/\/\/ science+computing ag // / \\ \ koenig(a)science-computing.de ^^^^^ ^^^^^

17 years, 1 month

Jump to page:

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev August 2003