Mailman 3 September 2003 - Python-Dev

Windows installer and .pyc files mtime
by Guido van Rossum Sept. 16, 2003

Sept. 16, 2003

I was watching file modification times on my Windows box (strange hobby, I know :-), and I noticed that after a fresh install of Python, the .pyc files seem to be written when the first code that imports the corresponding module runs, rather than all of the .pyc files being compiled at once by the installer. Wasn't there code in the installer that precompiles all modules? I know the Unix install does this, and I vaguely remember that the Windows installer did this too -- or was it only the Win32all installer??? If there's code to do that in the Windows installer now, it seems it's not working. If there isn't such code, perhaps there should be? --Guido van Rossum (home page: http://www.python.org/~guido/)

6 11

Re: [Python-Dev] Making python C-API thread safe (try 2)
by Harri Pesonen Sept. 16, 2003

Sept. 16, 2003

Phillip J. Eby wrote: > At 08:47 PM 9/11/03 +0300, Harri Pesonen wrote: > >> But my basic message is this: Python needs to be made thread safe. >> Making the individual interpreters thread safe is trivial, and >> benefits many people, and is a necessary first step; > > > It's far from trivial - you're talking about invalidating every piece > of C code written for Python over a multi-year people by dozens upon > dozens of extension authors. The change is trivial in Python C API. I already said that it would break everything outside the Python distribution, but the change in other applications is also trivial. > It doesn't benefit many people: only those using isolated interpreters > embedded in a multithreaded C program. I don't know how many people are writing threads in Python, either. I guess that not so many. In my case I only need a thread safe interpreter, I don't create threads in Python code. So just having what I described would be enough for me: no need for global interpreter lock, and Python would be really multithreading. It would benefit many people, I'm sure. > >> making threads within interpreter thread safe is possible as well, at >> least if you leave something for the developer, as you should, as you >> do in every other programming language as well. > > > You misunderstand. Those "critical sections" are for the most part in > Python's C code, not in the Python script. Yes, I'm aware of the None problem at least (only one instance of it). Please enlighten me about the other critical sections? Object allocation/freeing? > I'm guessing you haven't done much writing of C extensions for Python > (or Python core C), or else you'd realize why trying to make > INCREF/DECREF threadsafe would absolutely decimate performance. > Reference count updates happen *way* too often in normal code flow. > I also knew that already. But how else can you do it? Of course, changing Python to not have a single None would help a lot. Or, perhaps it could have a single None, but in case of None, the reference count would have no meaning, it would never be deallocated, because it would be checked in code. Maybe it does it already, I don't know. I'm also wondering why this problem has not been addressed before? If I had the power to change Python, this would be the first thing I did. Harri

6 6

datetime issues
by Gustavo Niemeyer Sept. 16, 2003

Sept. 16, 2003

Greetings! Is there any reason for not having a single keyword parameter for every function accepting a tzinfo instance? I've noticed that some require 'tz', and others 'tzinfo'. Is it too late to fix that? Also, is there any further work going on to improve the datetime module? I'm not suggesting we should mirror mx.DateTime functionalities, but two features I miss from mx.DateTime is the DateFrom(), which does its best to parse a given string, and the RelativeDateTime() functionality. This has probably been raised in the past, so if that's the case, I'm sorry. Marc-Andre, what's your opinion about reusing code from mx.DateTime into datetime? Anyway, reusing code or not, I'll probably put sometime on it in the future, if this looks interesting to everyone (after I finish my current python pendencies, like SRE's recursivity removal). -- Gustavo Niemeyer http://niemeyer.net

5 9

pre-PEP: Resource-Release Support for Generators
by Samuele Pedroni Sept. 16, 2003

Sept. 16, 2003

this is something we discussed with Guido, and also Moshe Zadka at Europython. Guido thought it seems reasonable enough, if the details can be nailed. I have written it down so the idea doesn't get lost, for the moment is more a matter of whether it can get a number, and then it can go dormant for a while. - * - PEP: XXX Title: Resource-Release Support for Generators Version: $Revision$ Last-Modified: $Date$ Author: Samuele Pedroni <pedronis(a)python.org> Status: Draft Type: Standards Track Content-Type: text/plain Created: 25-Aug-2003 Python-Version: 2.4 Post-History: Abstract Generators allow for natural coding and abstraction of traversal over data. Currently if external resources needing proper timely release are involved, generators are unfortunately not adequate. The typical idiom for timely release is not supported, a yield statement is not allowed in the try clause of a try-finally statement inside a generator. The finally clause execution cannot be either guaranteed or enforced. This PEP proposes that generators support a close method and destruction semantics, such that the restriction can be lifted, expanding the applicability of generators. Rationale Python generators allow for natural coding of many data traversal scenarios Their instantiation produces iterators, i.e. first-class objects abstracting traversal (with all the advantages of first- classness). In this respect they match in power and offer some advantages over the approach using iterator methods taking a (smalltalkish) block. On the other hand, given current limitations (no yield allowed in a try clause of a try-finally inside a generator) the latter approach seems better suited at encapsulating not only traversal but also exception handling and proper resource acquisition and release. Let's consider an example (for simplicity, files in read-mode are used): def all_lines(index_path): for path in file(index_path,"r"): for line in file(path.strip(),"r"): yield line this is short and to the point, but the try-finally for timely closing of the files cannot be added. (While instead of a path, a file, whose closing then would be responsibility of the caller, could be passed in as argument, the same is not applicable for the files opened depending on the contents of the index). If we want timely release, we have to sacrifice the simplicity and directness of the generator-only approach: (e.g.) class AllLines: def __init__(self,index_path): self.index_path = index_path self.index = None self.document = None def __iter__(self): self.index = file(self.index_path,"r") for path in self.index: self.document = file(path.strip(),"r") for line in self.document: yield line self.document.close() self.document = None def close(self): if self.index: self.index.close() if self.document: self.document.close() to be used as: all_lines = AllLines("index.txt") try: for line in all_lines: ... finally: all_lines.close() The more convoluted solution implementing timely release, seems to offer a precious hint. What we have done is encapsulating our traversal in an object (iterator) with a close method. This PEP proposes that generators should grow such a close method with such semantics that the example could be rewritten as: def all_lines(index_path): index = file(index_path,"r") try: for path in file(index_path,"r"): document = file(path.strip(),"r") try: for line in document: yield line finally: document.close() finally: index.close() all = all_lines("index.txt") try: for line in all: ... finally: all.close() PEP 255 [1] disallows yield inside a try clause of a try-finally statement, because the execution of the finally clause cannot be guaranteed as required by try-finally semantics. The semantics of the proposed close method should be such, that while the finally clause execution still cannot be guaranteed, it can be enforced when required. The semantics of generator destruction on the other hand should be extended in order to implement a best-effort policy for the general case. This strikes as a reasonable compromise, the resulting global behavior being similar to that of files and closing. Possible Semantics A close() method should be implemented for generator objects. 1) If a generator is already terminated, close should be a no-op. Otherwise: (two alternative solutions) (Return Semantics) The generator should be resumed, generator execution should continue as if the instruction at re-entry point is a return. Consequently finally clauses surrounding the re-entry point would be executed, in the case of a then allowed try-yield-finally pattern. Issues: is it important to be able to distinguish forced termination by close, normal termination, exception propagation from generator or generator-called code? In the normal case it seems not, finally clauses should be there to work the same in all these cases, still this semantics could make such a distinction hard. Except-clauses, like by a normal return, are not executed, such clauses in legacy generators expect to be executed for exceptions raised by the generator or by code called from it. Not executing them in the close case seems correct. OR (Exception Semantics) The generator should be resumed and execution should continue as if a special-purpose exception (e.g. CloseGenerator) has been raised at re-entry point. Close implementation should consume and not propagate further this exception. Issues: should StopIteration be reused for this purpose? Probably not. We would like close to be a harmless operation for legacy generators, which could contain code catching StopIteration to deal with other generators/iterators. In general, with exception semantics, it is unclear what to do if the generator does not terminate or we do not receive the special exception propagated back. Other different exceptions should probably be propagated, but consider this possible legacy generator code: try: ... yield ... ... except: # or except Exception:, etc raise Exception("boom") If close is invoked with the generator suspended after the yield, the except clause would catch our special purpose exception, so we would get a different exception propagated back, which in this case ought to be reasonably consumed and ignored but in general should be propagated, but separating these scenarios seem hard. The exception approach has the advantage to let the generator distinguish between termination cases and have more control. On the other hand clear-cut semantics seem harder to define. 2) Generator destruction should invoke close method behavior. Remarks If this proposal is accepted, it should become common practice to document whether a generator acquires resources, so that its close method ought to be called. If a generator is no longer used, calling close should be harmless. On the other hand, in the typical scenario the code that instantiated the generator should call close if required by it, generic code dealing with iterators/generators instantiated elsewhere should typically not be littered with close calls. The rare case of code that has acquired ownership of and need to properly deal with all of iterators, generators and generators acquiring resources that need timely release, is easily solved: if hasattr(iterator,'close'): iterator.close() Open Issues Definitive semantics ought to be chosen, implementation issues should be explored. Alternative Ideas The idea that the yield placement limitation should be removed and that generator destruction should trigger execution of finally clauses has been proposed more than once. Alone it cannot guarantee that timely release of resources acquired by a generator can be enforced. PEP 288 [2] proposes a more general solution, allowing custom exception passing to generators. References [1] PEP 255 Simple Generators http://www.python.org/peps/pep-0255.html [2] PEP 288 Generators Attributes and Exceptions http://www.python.org/peps/pep-0288.html Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:

7 14

RE: [Python-Dev] python/dist/src/Lib/bsddb __init__.py,1.5,1.6
by Michael Chermside Sept. 15, 2003

Sept. 15, 2003

[Jeremy] > Isn't this just the sort of little feature that was the subject of > recent discussion on the dangers of backporting. It would mean that > someone writing an app against Python 2.3 could never be sure whether > the feature existed. In practice, that would mean developers using > 2.3.x would only find out about problems after deployment when they bump > into a user with the original 2.3. > > I'm not sure how convincing I find this argument, but it's got some > merit. [Martin] > There is no proper balance. If you believe in the policy "no new > features", this policy is absolute - it would be pointless to accept > some new features, but reject others on grounds of the policy. I don't really understand this absolutist position. Suppose (hypothetical example made extreme for retorical purposes) we discover that there was a HUGE bug in release 2.3. Using a variable named "Q" as the argument of a for loop causes the interpreter to seg-fault. This is CLEARLY a bad thing... capital one-letter variable names are uncommon, but not THAT uncommon! So I suppose that 2.3.1 would need to include a fix for this bug. But THEN anyone writing code for 2.3.x could never be SURE whether the bug existed. If they used a capital Q in a for loop, their code would eventually be run by somebody with 2.3.0, causing a crash. The only way to be safe would be to test their code on 2.3.0. How is this "bug fix" any different from a "new feature"? I understand the argument "avoid new features in maintanence releases because there we value stability FAR more than we value features". I understand the argument "avoid new features in maintanence releases because features should be added on the main trunk to encourage the (eventual) use of new versions of Python". But I don't understand the argument "PROHIBIT new features in maintanence releases because then programs written under the maintanence release might not run under the original release", because to me it seems to be logically equivalent to "prohibit maintanence releases". [Martin] > This policy is mutually exclusive with "new features are fine as long > as they don't break anything". You can get opinions, but you cannot > get consensus. If "consensus" is taken to mean "universal agreement", then I'd agree, but if we mean "general agreement or accord" (ie, everyone gets heard then NEARLY everyone agrees and the naysayers decide they can live with it), then we might have a shot. I'd certainly say (and I think most here would agree) that even ONE seriously intended firmly held argument against a particular new feature (in a maint. release) would make me think VERY carefully before allowing that feature. -- Michael Chermside

2 1

refleak status
by Michael Hudson Sept. 15, 2003

Sept. 15, 2003

After a week or so of frantically searching for these things, I sort of forgot about the reference leak hunt I started in Python's tests. A week ago I ran my hacked regrtest.py again, made some notes, tried & failed to send them to python-dev, went on holiday, came back, digged a little more and am trying to send again... real leaks i think i've fixed (am syncing up to CVS now, will test overnight). test_class leaked [66, 66, 66, 66] references test_hotshot leaked [111, 111, 111, 111] references real leaks i cant fix: test_compile leaked [45, 45, 45, 45] references lambda a,a:0 worst offender. confused. red herrings: test_codeccallbacks leaked [2, 2, 2, 2] references callback registry test_mutants leaked [-1489, 1520, -755, -183] references randomized test suspected bugs in _Py_RefTotal tracking: test_descr leaked [2, 2, 2, 2] references something to do with resurrection test_gc leaked [18, 18, 18, 18] references test_trashcan ... not a real leak, it seems suspected red herrings: test_cpickle leaked [6, 3, 2, 1] references test_logging leaked [82, 82, 82, 82] references (bet this is registry stuff of one sort or another) test_threadedtempfile leaked [-74, 3, 3, 161] references test_unicode leaked [7, 7, 7, 7] references registries again apparent refleaks that still need digging: test_minidom leaked [1, 1, 1, 1] references test_new leaked [2, 2, 2, 2] references test_pkg leaked [8, 8, 9, 9] references test_pkgimport leaked [5, 6, 6, 6] references test_sax leaked [1762, 1762, 1762, 1762] references Wouldn't like to claim anything about the following tests: 6 tests failed: test_dircache test_doctest test_getopt test_multifile test_warnings test_zipimport 27 tests skipped: test_al test_bsddb test_bsddb3 test_cd test_cl test_curses test_dl test_email_codecs test_gdbm test_gl test_imgfile test_largefile test_linuxaudiodev test_locale test_mpz test_nis test_normalization test_ossaudiodev test_pep277 test_poll test_socket_ssl test_socketserver test_sunaudiodev test_timeout test_urllibnet test_winreg test_winsound Those skips are all expected on darwin. (test_threaded_import deadlocks, so add that to the list) Should investigate why the failing tests fail when run repeatedly and check that they're not hiding ref leaks. Cheers, mwh

1 0

SF #805304: bug or feature?
by Raymond Hettinger Sept. 14, 2003

Sept. 14, 2003

Fred reported that super objects don't respond well to language constructs that bypass attribute lookup. For example, supobj.__setitem__(name,value) works if __setitem__() is defined in the target, but supobj[name]=value will raise a TypeError. www.python.org/sf/805304 Once the fix is approved, there is a question of whether it should be backported. The case against it is that some programs written under Py2.3.1 or Py2.2.4 won't run on Py2.3.0 or Py2.2.3 and the patch can be viewed as an API expansion. The other point of view is that super objects should have always behaved this way and that the patch just fixes buggy behavior. Raymond

2 1

Weekly Python Bug/Patch Summary
by Skip Montanaro Sept. 14, 2003

Sept. 14, 2003

Bug/Patch Summary ----------------- 475 open / 4113 total bugs (+10) 193 open / 2371 total patches (+8) New Bugs -------- cgi.CGIHTTPRequestHandler remembers QUERY_STRING (2003-09-07) http://python.org/sf/801992 Mode argument of dumbdbm does not work (2003-09-07) http://python.org/sf/802128 tkFont may reuse font names (2003-09-08) http://python.org/sf/802310 Bogus use of Tkinter.PhotoImage can give sig11 (2003-09-08) http://python.org/sf/802417 uu.decode prints to stderr (2003-09-09) http://python.org/sf/803413 sgmllib doesn't support hex or Unicode character references (2003-09-09) http://python.org/sf/803422 plat-mac/applesingle.py needs cosmetic changes (2003-09-09) http://python.org/sf/803498 urllib/urllib2(?) timeouts (2003-09-10) http://python.org/sf/803634 Crypto terminology for crypto hash function (2003-09-10) http://python.org/sf/804113 invalid use of setlocale (2003-09-11) http://python.org/sf/804543 PyUnicode_FromEncodedObject (2003-09-12) http://python.org/sf/805015 Inappropriate error received using socket timeout (2003-09-12) http://python.org/sf/805194 _tkinter compilation fails (2003-09-12) http://python.org/sf/805200 super instances don't support item assignment (2003-09-12) http://python.org/sf/805304 IDLE can't save UTF-8 files on Windows (2003-09-13) http://python.org/sf/805728 Minor Python Intro update (2003-09-13) http://python.org/sf/805788 New Patches ----------- Suggested change in how ntpath.expanduser works (2003-09-07) http://python.org/sf/802159 better parser error message for non-EOL following line cont. (2003-09-07) http://python.org/sf/802188 [_ssl.c] SSL_write() called with -1 as size (2003-09-10) http://python.org/sf/803998 A ForwardingHandler for logging (2003-09-10) http://python.org/sf/804180 unittest's main don't handle TestSuite in command line (2003-09-10) http://python.org/sf/804212 Lib/email/Encoders.py iso-2022 is 7bit (2003-09-11) http://python.org/sf/804885 fix --without-threads (2003-09-13) http://python.org/sf/805604 PTH related fixes (2003-09-13) http://python.org/sf/805613 make test_fcntl 64bit clean (2003-09-13) http://python.org/sf/805626 Fix linking with shared libpython (2003-09-13) http://python.org/sf/805678 Highlight builtins (2003-09-13) http://python.org/sf/805830 Closed Bugs ----------- Add docs for 'basestring' (2002-05-24) http://python.org/sf/560286 htmllib.HTMLParser.anchorlist problem (2003-03-28) http://python.org/sf/711632 2 problems with IDLE (2003-07-21) http://python.org/sf/775061 RESET_ERROR is not defined(logging module) (2003-08-21) http://python.org/sf/792649 select module doesn't allow any iterable. (2003-08-31) http://python.org/sf/798046 test_dumbdbm failing (2003-09-04) http://python.org/sf/800824 Making "|" directive from REs a bit clearer (2003-09-04) http://python.org/sf/800899 Bad RE in scanf example (2003-09-05) http://python.org/sf/801306 Closed Patches -------------- Allow os.access to handle Unicode file name (2003-08-17) http://python.org/sf/790000 implement htmllib.HTMLParser.reset (2003-08-22) http://python.org/sf/793021 zipimport 64-bit fix (2003-09-06) http://python.org/sf/801821

1 0

To You as a Nicer Person
by Dr. Newell Sept. 13, 2003

Sept. 13, 2003

Dear python-dev: You may ask yourself > How to make people like you and respect you > How to win friends > How to let your conduct help your health, work, job, career, success, relationships, spirit, mind, well-being, ... > How to make your life smoother and happier > How to do whatever you like without being unpleasant to other people > How to develop good conduct in your children or students > How to make the world peaceful and better You can find all the answers to these questions, and much more, in this great handbook: " Complete Conduct Principles for the 21st Century " by Dr. John Newton It is the best educational GIFT idea for children, friends, relatives, classmates, students, parents, teachers, other educators, ..., particularly at this special time. BENEFITS to each individual reader: Many! -- such as for health, work, job, career, success, self-improvement, education, relationships, spirit, mind, well-being, and much more -- almost all the areas that are important to you in the 21st century. People around you will benefit, too. (Please see the preface of the book for details.) EVERYONE may find this handbook useful and helpful, regardless of age (from children to oldsters), occupation, rank, status, gender, religious belief, race, nationality, country, or region. If you are a parent or a teacher, you can learn how to develop good conduct in your children or students from this handbook. Please advise your children or students to read the book. It will result in great benefits for both you and them. This book is a must for EVERYONE to be better prepared for personal conduct for the rest of the 21st century. The book's content is obvious from its title. The complete useful conduct principles cover not only what we should do, but also what we should not do -- especially those faults people make often and easily. This timely, unique, and very important handbook is designed to suit most people, and is self-contained and user-friendly. The book was also praised as "a compendium of concisely expressed, practical, informative, pertinent, workable advice" by Michael J. Carson, a professional book reviewer. This book is significantly different and better than competitive works. "Unlike most books of this subject, it is NOT a religious book, nor is a collection of old conduct rules." Some of its innovative contents may help solve problems that the Western culture cannot. The book's merit and importance have been recognized and praised by many experts, elected public officials, and world leaders. As a result of popular demand, the book has been a top "Featured Item" in some top on-line bookstores in some subjects, such as "Personal Practical Guides" in "Nonfiction" and "Reference", "Conduct" in "Health, Mind & Body", "Life skills guides", "Ethics & Moral Philosophy", ... "The book will also be effective for violence prevention for the whole society." said some experts. How to make the world peaceful and better --- You can find the solution in the book. Let's work together to make the world peaceful and better! The author, John Newton, holds a Ph.D. from MIT, and does researches at Harvard. His long-term research on "The personal conduct in the human society of the 21st century" resulted in this book. Before the human beings went into the 21st century, the compassionate, merciful, courageous and farsighted Dr. Newton had issued a number of warning predictions, some of which have already been proved in the new century. The book is published by NCWO, headquartered beside Harvard University and MIT, two leading institutes of new knowledge and literature. The publisher has a deep commitment to publishing only the best. NCWO is an educational, non-profit, non-partisan, and honorary organization; it endeavors to make the 21st century nicer than ever before. To accomplish its mission, NCWO is proud to introduce this book. The Web site of NCWO has been chosen as one of "Top Non Profit Sites Chosen by Type Non Profit Editors". The book is available in two types of binding: Hardcover (ISBN 0967370574; case bound, Smyth sewn; with dust jacket) and Paperback (ISBN 0967370582; perfect bound). Both editions are unabridged, and are printed on 60 lb, natural, acid-free, excellent and healthful paper. You can get the book from many fine on-line bookstores and traditional bookstores. For your convenience, (if you wish for an Internet link; otherwise you may skip this section) I herewith provide you with a link directly to the book page of each edition in Half.com by eBay, a popular on-line discount mall: for paperback: http://half.ebay.com/cat/buy/prod.cgi?cpid=2425993 for hardcover: http://half.ebay.com/cat/buy/prod.cgi?cpid=2425992 Please forward this e-mail to people you know -- children, friends, relatives, classmates, students, parents, teachers, other educators, ..., because they can benefit from it, too. This can be a wonderful kindness you provide to them! Sincerely yours, (python-dev, best wishes to you!) Christopher Newell, Ph.D. Cambridge, Massachusetts, USA P.S. Some educational units, ranging from the level of nation or state to individual school or university, have ordered the book as textbook, reference book, gift to students, or as an active action to prevent school violence, to improve education and to benefit students, teachers & parents. To have more people benefit from the book, please consider suggesting to the schools -- your children attend, or you yourself attend, have attended before, teach at, or serve -- that the book be used for fundraising for the schools. The book is an ideal fundraising tool. For example, it may used as a premium or a re-sale product for the fundraising. The successful fundraising will significantly help school education. Better yet, each supporter and his/her family will benefit from the book. Suggesting to the parent-teacher associations (organizations) (PTA/PTO) of the schools is also a good idea.

1 0

PyUnicode_FromEncodedObject
by David Abrahams Sept. 12, 2003

Sept. 12, 2003

I'd post a SF bug but it keeps insisting I log in, even after I have already. There's a bug, either in the code or docs for PyUnicode_FromEncodedObject. The docs read: "Unicode objects are passed back as-is with incremented refcount. Note: These cannot be decoded; passing a non-NULL value for encoding will result in a TypeError." 'tain't so; the following shows that the error is unconditional. if (obj == NULL) { PyErr_BadInternalCall(); return NULL; } #if 0 /* For b/w compatibility we also accept Unicode objects provided that no encodings is given and then redirect to PyObject_Unicode() which then applies the additional logic for Unicode subclasses. NOTE: This API should really only be used for object which represent *encoded* Unicode ! */ if (PyUnicode_Check(obj)) { if (encoding) { PyErr_SetString(PyExc_TypeError, "decoding Unicode is not supported"); return NULL; } return PyObject_Unicode(obj); } #else if (PyUnicode_Check(obj)) { PyErr_SetString(PyExc_TypeError, "decoding Unicode is not supported"); return NULL; } #endif -- Dave Abrahams Boost Consulting www.boost-consulting.com

2 4