Mailman 3 October 2004 - Python-Dev

PEP 1, PEP Purpose and Guidelines
by barry＠zope.com 18 May '21

18 May '21

It has been a while since I posted a copy of PEP 1 to the mailing lists and newsgroups. I've recently done some updating of a few sections, so in the interest of gaining wider community participation in the Python development process, I'm posting the latest revision of PEP 1 here. A version of the PEP is always available on-line at http://www.python.org/peps/pep-0001.html Enjoy, -Barry -------------------- snip snip -------------------- PEP: 1 Title: PEP Purpose and Guidelines Version: $Revision: 1.36 $ Last-Modified: $Date: 2002/07/29 18:34:59 $ Author: Barry A. Warsaw, Jeremy Hylton Status: Active Type: Informational Created: 13-Jun-2000 Post-History: 21-Mar-2001, 29-Jul-2002 What is a PEP? PEP stands for Python Enhancement Proposal. A PEP is a design document providing information to the Python community, or describing a new feature for Python. The PEP should provide a concise technical specification of the feature and a rationale for the feature. We intend PEPs to be the primary mechanisms for proposing new features, for collecting community input on an issue, and for documenting the design decisions that have gone into Python. The PEP author is responsible for building consensus within the community and documenting dissenting opinions. Because the PEPs are maintained as plain text files under CVS control, their revision history is the historical record of the feature proposal[1]. Kinds of PEPs There are two kinds of PEPs. A standards track PEP describes a new feature or implementation for Python. An informational PEP describes a Python design issue, or provides general guidelines or information to the Python community, but does not propose a new feature. Informational PEPs do not necessarily represent a Python community consensus or recommendation, so users and implementors are free to ignore informational PEPs or follow their advice. PEP Work Flow The PEP editor, Barry Warsaw <peps(a)python.org>, assigns numbers for each PEP and changes its status. The PEP process begins with a new idea for Python. It is highly recommended that a single PEP contain a single key proposal or new idea. The more focussed the PEP, the more successfully it tends to be. The PEP editor reserves the right to reject PEP proposals if they appear too unfocussed or too broad. If in doubt, split your PEP into several well-focussed ones. Each PEP must have a champion -- someone who writes the PEP using the style and format described below, shepherds the discussions in the appropriate forums, and attempts to build community consensus around the idea. The PEP champion (a.k.a. Author) should first attempt to ascertain whether the idea is PEP-able. Small enhancements or patches often don't need a PEP and can be injected into the Python development work flow with a patch submission to the SourceForge patch manager[2] or feature request tracker[3]. The PEP champion then emails the PEP editor <peps(a)python.org> with a proposed title and a rough, but fleshed out, draft of the PEP. This draft must be written in PEP style as described below. If the PEP editor approves, he will assign the PEP a number, label it as standards track or informational, give it status 'draft', and create and check-in the initial draft of the PEP. The PEP editor will not unreasonably deny a PEP. Reasons for denying PEP status include duplication of effort, being technically unsound, not providing proper motivation or addressing backwards compatibility, or not in keeping with the Python philosophy. The BDFL (Benevolent Dictator for Life, Guido van Rossum) can be consulted during the approval phase, and is the final arbitrator of the draft's PEP-ability. If a pre-PEP is rejected, the author may elect to take the pre-PEP to the comp.lang.python newsgroup (a.k.a. python-list(a)python.org mailing list) to help flesh it out, gain feedback and consensus from the community at large, and improve the PEP for re-submission. The author of the PEP is then responsible for posting the PEP to the community forums, and marshaling community support for it. As updates are necessary, the PEP author can check in new versions if they have CVS commit permissions, or can email new PEP versions to the PEP editor for committing. Standards track PEPs consists of two parts, a design document and a reference implementation. The PEP should be reviewed and accepted before a reference implementation is begun, unless a reference implementation will aid people in studying the PEP. Standards Track PEPs must include an implementation - in the form of code, patch, or URL to same - before it can be considered Final. PEP authors are responsible for collecting community feedback on a PEP before submitting it for review. A PEP that has not been discussed on python-list(a)python.org and/or python-dev(a)python.org will not be accepted. However, wherever possible, long open-ended discussions on public mailing lists should be avoided. Strategies to keep the discussions efficient include, setting up a separate SIG mailing list for the topic, having the PEP author accept private comments in the early design phases, etc. PEP authors should use their discretion here. Once the authors have completed a PEP, they must inform the PEP editor that it is ready for review. PEPs are reviewed by the BDFL and his chosen consultants, who may accept or reject a PEP or send it back to the author(s) for revision. Once a PEP has been accepted, the reference implementation must be completed. When the reference implementation is complete and accepted by the BDFL, the status will be changed to `Final.' A PEP can also be assigned status `Deferred.' The PEP author or editor can assign the PEP this status when no progress is being made on the PEP. Once a PEP is deferred, the PEP editor can re-assign it to draft status. A PEP can also be `Rejected'. Perhaps after all is said and done it was not a good idea. It is still important to have a record of this fact. PEPs can also be replaced by a different PEP, rendering the original obsolete. This is intended for Informational PEPs, where version 2 of an API can replace version 1. PEP work flow is as follows: Draft -> Accepted -> Final -> Replaced ^ +----> Rejected v Deferred Some informational PEPs may also have a status of `Active' if they are never meant to be completed. E.g. PEP 1. What belongs in a successful PEP? Each PEP should have the following parts: 1. Preamble -- RFC822 style headers containing meta-data about the PEP, including the PEP number, a short descriptive title (limited to a maximum of 44 characters), the names, and optionally the contact info for each author, etc. 2. Abstract -- a short (~200 word) description of the technical issue being addressed. 3. Copyright/public domain -- Each PEP must either be explicitly labelled as placed in the public domain (see this PEP as an example) or licensed under the Open Publication License[4]. 4. Specification -- The technical specification should describe the syntax and semantics of any new language feature. The specification should be detailed enough to allow competing, interoperable implementations for any of the current Python platforms (CPython, JPython, Python .NET). 5. Motivation -- The motivation is critical for PEPs that want to change the Python language. It should clearly explain why the existing language specification is inadequate to address the problem that the PEP solves. PEP submissions without sufficient motivation may be rejected outright. 6. Rationale -- The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work, e.g. how the feature is supported in other languages. The rationale should provide evidence of consensus within the community and discuss important objections or concerns raised during discussion. 7. Backwards Compatibility -- All PEPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. The PEP must explain how the author proposes to deal with these incompatibilities. PEP submissions without a sufficient backwards compatibility treatise may be rejected outright. 8. Reference Implementation -- The reference implementation must be completed before any PEP is given status 'Final,' but it need not be completed before the PEP is accepted. It is better to finish the specification and rationale first and reach consensus on it before writing code. The final implementation must include test code and documentation appropriate for either the Python language reference or the standard library reference. PEP Template PEPs are written in plain ASCII text, and should adhere to a rigid style. There is a Python script that parses this style and converts the plain text PEP to HTML for viewing on the web[5]. PEP 9 contains a boilerplate[7] template you can use to get started writing your PEP. Each PEP must begin with an RFC822 style header preamble. The headers must appear in the following order. Headers marked with `*' are optional and are described below. All other headers are required. PEP: <pep number> Title: <pep title> Version: <cvs version string> Last-Modified: <cvs date string> Author: <list of authors' real names and optionally, email addrs> * Discussions-To: <email address> Status: <Draft | Active | Accepted | Deferred | Final | Replaced> Type: <Informational | Standards Track> * Requires: <pep numbers> Created: <date created on, in dd-mmm-yyyy format> * Python-Version: <version number> Post-History: <dates of postings to python-list and python-dev> * Replaces: <pep number> * Replaced-By: <pep number> The Author: header lists the names and optionally, the email addresses of all the authors/owners of the PEP. The format of the author entry should be address(a)dom.ain (Random J. User) if the email address is included, and just Random J. User if the address is not given. If there are multiple authors, each should be on a separate line following RFC 822 continuation line conventions. Note that personal email addresses in PEPs will be obscured as a defense against spam harvesters. Standards track PEPs must have a Python-Version: header which indicates the version of Python that the feature will be released with. Informational PEPs do not need a Python-Version: header. While a PEP is in private discussions (usually during the initial Draft phase), a Discussions-To: header will indicate the mailing list or URL where the PEP is being discussed. No Discussions-To: header is necessary if the PEP is being discussed privately with the author, or on the python-list or python-dev email mailing lists. Note that email addresses in the Discussions-To: header will not be obscured. Created: records the date that the PEP was assigned a number, while Post-History: is used to record the dates of when new versions of the PEP are posted to python-list and/or python-dev. Both headers should be in dd-mmm-yyyy format, e.g. 14-Aug-2001. PEPs may have a Requires: header, indicating the PEP numbers that this PEP depends on. PEPs may also have a Replaced-By: header indicating that a PEP has been rendered obsolete by a later document; the value is the number of the PEP that replaces the current document. The newer PEP must have a Replaces: header containing the number of the PEP that it rendered obsolete. PEP Formatting Requirements PEP headings must begin in column zero and the initial letter of each word must be capitalized as in book titles. Acronyms should be in all capitals. The body of each section must be indented 4 spaces. Code samples inside body sections should be indented a further 4 spaces, and other indentation can be used as required to make the text readable. You must use two blank lines between the last line of a section's body and the next section heading. You must adhere to the Emacs convention of adding two spaces at the end of every sentence. You should fill your paragraphs to column 70, but under no circumstances should your lines extend past column 79. If your code samples spill over column 79, you should rewrite them. Tab characters must never appear in the document at all. A PEP should include the standard Emacs stanza included by example at the bottom of this PEP. A PEP must contain a Copyright section, and it is strongly recommended to put the PEP in the public domain. When referencing an external web page in the body of a PEP, you should include the title of the page in the text, with a footnote reference to the URL. Do not include the URL in the body text of the PEP. E.g. Refer to the Python Language web site [1] for more details. ... [1] http://www.python.org When referring to another PEP, include the PEP number in the body text, such as "PEP 1". The title may optionally appear. Add a footnote reference that includes the PEP's title and author. It may optionally include the explicit URL on a separate line, but only in the References section. Note that the pep2html.py script will calculate URLs automatically, e.g.: ... Refer to PEP 1 [7] for more information about PEP style ... References [7] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton http://www.python.org/peps/pep-0001.html If you decide to provide an explicit URL for a PEP, please use this as the URL template: http://www.python.org/peps/pep-xxxx.html PEP numbers in URLs must be padded with zeros from the left, so as to be exactly 4 characters wide, however PEP numbers in text are never padded. Reporting PEP Bugs, or Submitting PEP Updates How you report a bug, or submit a PEP update depends on several factors, such as the maturity of the PEP, the preferences of the PEP author, and the nature of your comments. For the early draft stages of the PEP, it's probably best to send your comments and changes directly to the PEP author. For more mature, or finished PEPs you may want to submit corrections to the SourceForge bug manager[6] or better yet, the SourceForge patch manager[2] so that your changes don't get lost. If the PEP author is a SF developer, assign the bug/patch to him, otherwise assign it to the PEP editor. When in doubt about where to send your changes, please check first with the PEP author and/or PEP editor. PEP authors who are also SF committers, can update the PEPs themselves by using "cvs commit" to commit their changes. Remember to also push the formatted PEP text out to the web by doing the following: % python pep2html.py -i NUM where NUM is the number of the PEP you want to push out. See % python pep2html.py --help for details. Transferring PEP Ownership It occasionally becomes necessary to transfer ownership of PEPs to a new champion. In general, we'd like to retain the original author as a co-author of the transferred PEP, but that's really up to the original author. A good reason to transfer ownership is because the original author no longer has the time or interest in updating it or following through with the PEP process, or has fallen off the face of the 'net (i.e. is unreachable or not responding to email). A bad reason to transfer ownership is because you don't agree with the direction of the PEP. We try to build consensus around a PEP, but if that's not possible, you can always submit a competing PEP. If you are interested assuming ownership of a PEP, send a message asking to take over, addressed to both the original author and the PEP editor <peps(a)python.org>. If the original author doesn't respond to email in a timely manner, the PEP editor will make a unilateral decision (it's not like such decisions can be reversed. :). References and Footnotes [1] This historical record is available by the normal CVS commands for retrieving older revisions. For those without direct access to the CVS tree, you can browse the current and past PEP revisions via the SourceForge web site at http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi/python/nondist/peps/?cvsroot=… [2] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [3] http://sourceforge.net/tracker/?atid=355470&group_id=5470&func=browse [4] http://www.opencontent.org/openpub/ [5] The script referred to here is pep2html.py, which lives in the same directory in the CVS tree as the PEPs themselves. Try "pep2html.py --help" for details. The URL for viewing PEPs on the web is http://www.python.org/peps/ [6] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [7] PEP 9, Sample PEP Template http://www.python.org/peps/pep-0009.html Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:

8 14

unicode inconsistency?
by Neil Schemenauer 09 Mar '05

09 Mar '05

Perhaps this is more approprate for python-list but I looks like a bug to me. Example code: class A: def __str__(self): return u'\u1234' '%s' % u'\u1234' # this works '%s' % A() # this doesn't work It will work if 'A' subclasses from 'unicode' but should not be necessary, IMHO. Any reason why this shouldn't be fixed? Neil

6 15

RE: [Python-Dev] Those import related syntax errors again...
by Samuele Pedroni 23 Feb '05

23 Feb '05

Hi. [Mark Hammond] > The point isn't about my suffering as such. The point is more that > python-dev owns a tiny amount of the code out there, and I don't believe we > should put Python's users through this. > > Sure - I would be happy to "upgrade" all the win32all code, no problem. I > am also happy to live in the bleeding edge and take some pain that will > cause. > > The issue is simply the user base, and giving Python a reputation of not > being able to painlessly upgrade even dot revisions. I agree with all this. [As I imagined explicit syntax did not catch up and would require lot of discussions.] [GvR] > > Another way is to use special rules > > (similar to those for class defs), e.g. having > > > > <frag> > > y=3 > > def f(): > > exec "y=2" > > def g(): > > return y > > return g() > > > > print f() > > </frag> > > > > # print 3. > > > > Is that confusing for users? maybe they will more naturally expect 2 > > as outcome (given nested scopes). > > This seems the best compromise to me. It will lead to the least > broken code, because this is the behavior that we had before nested > scopes! It is also quite easy to implement given the current > implementation, I believe. > > Maybe we could introduce a warning rather than an error for this > situation though, because even if this behavior is clearly documented, > it will still be confusing to some, so it is better if we outlaw it in > some future version. > Yes this can be easy to implement but more confusing situations can arise: <frag> y=3 def f(): y=9 exec "y=2" def g(): return y return y,g() print f() </frag> What should this print? the situation leads not to a canonical solution as class def scopes. or <frag> def f(): from foo import * def g(): return y return g() print f() </frag> [Mark Hammond] > > This probably won't be a very popular suggestion, but how about pulling > > nested scopes (I assume they are at the root of the problem) > > until this can be solved cleanly? > > Agreed. While I think nested scopes are kinda cool, I have lived without > them, and really without missing them, for years. At the moment the cure > appears worse then the symptoms in at least a few cases. If nothing else, > it compromises the elegant simplicity of Python that drew me here in the > first place! > > Assuming that people really _do_ want this feature, IMO the bar should be > raised so there are _zero_ backward compatibility issues. I don't say anything about pulling nested scopes (I don't think my opinion can change things in this respect) but I should insist that without explicit syntax IMO raising the bar has a too high impl cost (both performance and complexity) or creates confusion. [Andrew Kuchling] > >Assuming that people really _do_ want this feature, IMO the bar should be > >raised so there are _zero_ backward compatibility issues. > > Even at the cost of additional implementation complexity? At the cost > of having to learn "scopes are nested, unless you do these two things > in which case they're not"? > > Let's not waffle. If nested scopes are worth doing, they're worth > breaking code. Either leave exec and from..import illegal, or back > out nested scopes, or think of some better solution, but let's not > introduce complicated backward compatibility hacks. IMO breaking code would be ok if we issue warnings today and implement nested scopes issuing errors tomorrow. But this is simply a statement about principles and raised impression. IMO import * in an inner scope should end up being an error, not sure about 'exec's. We will need a final BDFL statement. regards, Samuele Pedroni.

18 78

mimetypes and _winreg
by Mike Brown 02 Feb '05

02 Feb '05

I thought it would be nice to try to improve the mimetypes module by having it, on Windows, query the Registry to get the mapping of filename extensions to media types, since the mimetypes code currently just blindly checks posix-specific paths for httpd-style mapping files. However, it seems that the way to get mappings from the Windows registry is excessively slow in Python. I'm told that the reason has to do with the limited subset of APIs that are exposed in the _winreg module. I think it is that EnumKey(key, index) is querying for the entire list of subkeys for the given key every time you call it. Or something. Whatever the situation is, the code I tried below is way slower than I think it ought to be. Does anyone have any suggestions (besides "write it in C")? Could _winreg possibly be improved to provide an iterator or better interface to get the subkeys? (or certain ones? There are a lot of keys under HKEY_CLASSES_ROOT, and I only need the ones that start with a period). Should I file this as a feature request? Thanks -Mike from _winreg import HKEY_CLASSES_ROOT, OpenKey, EnumKey, QueryValueEx i = 0 typemap = {} try: while 1: subkeyname = EnumKey(HKEY_CLASSES_ROOT, i) try: subkey = OpenKey(HKEY_CLASSES_ROOT, subkeyname) if subkeyname[:1] == '.': data = QueryValueEx(subkey, 'Content Type')[0] print subkeyname, '=', data typemap[subkeyname] = data # data will be unicode except EnvironmentError, WindowsError: pass i += 1 except WindowsError: pass

3 4

Minor cookielib changes for 2.4b1
by John J Lee 13 Nov '04

13 Nov '04

Would anyone care to check this cookielib.py patch in before 2.4b1? http://python.org/sf/1028908 Quote from above: The patch contains uncontroversial changes to cookielib and associated modules, documentation and tests. Would be unfortunate not to have these in 2.4.0. [...SNIP...] John

4 5

Bug [ 959379 ] Implicit close() should check for errors
by Peter Astrand 07 Nov '04

07 Nov '04

I'd like to resolve bug 959379. Here's a naive fix. Comments? diff -u -r2.192 fileobject.c --- Objects/fileobject.c 11 Jun 2004 04:49:03 -0000 2.192 +++ Objects/fileobject.c 24 Oct 2004 20:58:49 -0000 @@ -304,7 +304,8 @@ PyObject_ClearWeakRefs((PyObject *) f); if (f->f_fp != NULL && f->f_close != NULL) { Py_BEGIN_ALLOW_THREADS - (*f->f_close)(f->f_fp); + if ((*f->f_close)(f->f_fp) != 0) + perror("close failed"); Py_END_ALLOW_THREADS } PyMem_Free(f->f_setbuf); /Peter Åstrand <astrand(a)lysator.liu.se>

4 8

Tests broken on Windows
by Tim Peters 01 Nov '04

01 Nov '04

Current test_shutil is apparently trying to do something that can't work on Windows, so it fails: test_shutil test test_shutil failed -- Traceback (most recent call last): File "C:\Code\python\lib\test\test_shutil.py", line 28, in test_on_error shutil.rmtree(TESTFN, onerror=self.check_args_to_onerror) File "C:\Code\python\lib\shutil.py", line 168, in rmtree onerror(os.rmdir, path, sys.exc_info()) File "C:\Code\python\lib\test\test_shutil.py", line 36, in check_args_to_onerror self.assertEqual(func, os.remove) AssertionError: <built-in function rmdir> != <built-in function remove> 1 test failed: test_shutil After that, it leaves behind temp directories that cause other tests to fail. Sorry, I've got other problems on my stack, and can't make time for this one.

4 10

Windows: PyBanner048.gif damage repaired
by Tim Peters 31 Oct '04

31 Oct '04

Lib\email\test\data\PyBanner048.gif wasn't marked binary in CVS. It is now. If you're on Windows, delete your copy, then cvs up to refetch an unmangled copy.

1 0

Pickle breakage with reduce of recursive structures (was: AssertionError in pickle's memoize function)
by Dima Dorfman 31 Oct '04

31 Oct '04

[Followups to python-dev, please.] [Michael Hohn] > > > under python 2.2, the pickle/unpickle sequence incorrectly restores > > > a larger data structure I have. > > > > > > Under Python 2.3, these structures now give an explicit exception from > > > Pickle.memoize(): > > > assert id(obj) not in self.memo > > > [Tim Peters] > > Assertions are never expected to fail, so "something impossible > > happened" when they do fail. [Michael Hohn] > Here is a code sample that shows the problem I ran into: Summary for the OP: This is a bug in Python. Using cPickle won't help, but if you don't subclass builtin container types others than list, dict, and tuple, using pickle protocol 2 should work. The rest of this message is for python-dev. The simplest breaking case is: t = type('t', (list,), {}) obj = t() obj.append(obj) pickle.dumps(obj) [infinite recursion] The subclass causes save_reduce to be used instead of save_list. For proto < 2, copy_reg._reduce_ex returns (_reconstructor, list(a)), and the args--list(a)--cycle back through obj. Initially it looks like this should be okay, but the args are saved before obj is memoized, and obj can't be memoized until REDUCE can be executed with the args--and there's the cycle. It is even more obviously impossible from the unpickler's perspective because it has to call _reconstructor([obj]) to create obj! There are two separate problems: 1. Any __reduce__ implementation that returns args that cycle back through the object it tried to reduce hasn't done its job. As described above, _reduce_ex is one such implementation. reduce_2 avoids this by using the listitems and dictitems parameters. Since that's a pickler-side feature it can be used in _reduce_ex too. The basetype(obj) hook (documented in PEP 307) would remain for immutable bases; it doesn't work for containers, but user-defined containers already have to implement their own reduce functions. POC patch: http://www.trit.org/~dima/home/reduce_ex.diff At least the set and deque types also have this problem. 2. The pickle implementations don't detect reduction cycles. Pickling an instance of this obviously broken class causes an infinite recursion: class evil(object): def __reduce__(self): return evil, (self,) It's easy to detect this case. POC patch for the pickle module: http://www.trit.org/~dima/home/redcycle.diff BTW, the failed assert the OP is seeing happens when the cycle goes through another object: t = type('t', (list,), {}) obj = t() d = {'obj': obj} obj.append(d) pickle.dumps(obj) [AssertionError] cPickle has the same problem, but it lacks the assert, so it writes garbage instead: new = cPickle.loads(cPickle.dumps(obj)) new[0]['obj'] is new -> False # wrong obj[0]['obj'] is obj -> True # right This makes the reduction cycle check (#2 above) more than just cosmetic since if cPickle had that assert (it should) it would've been a crash. Right now it's garbage output instead, which is arguably worse. Formally complete versions of the above patches will be on SF tomorrow unless someone suggests better alternatives. Dima.

2 1

Weekly Python Patch/Bug Summary
by Kurt B. Kaiser 30 Oct '04

30 Oct '04

Patch / Bug Summary ___________________ Patches : 243 open ( +3) / 2678 closed (+17) / 2921 total (+20) Bugs : 789 open (+17) / 4545 closed (+17) / 5334 total (+34) RFE : 156 open ( +1) / 131 closed ( +0) / 287 total ( +1) New / Reopened Patches ______________________ Fix various x86_64 build issues (2004-10-19) http://python.org/sf/1050475 opened by Bob Halley make distutils.core.run_setup re-entrant (2004-10-20) http://python.org/sf/1051216 opened by Bob Ippolito xml.dom missing API docs (bugs 1010196, 1013525) (2004-10-21) http://python.org/sf/1051321 opened by Mike Brown locale.getdefaultlocale does not return tuple in some OS (2004-10-21) http://python.org/sf/1051395 opened by Jiwon Seo Syntax error in Misc/NEWS (2004-10-22) CLOSED http://python.org/sf/1051866 opened by Felix Wiemann plistlib: add plst resource functionality, fix bugs (2004-10-22) CLOSED http://python.org/sf/1052399 opened by Bob Ippolito urllib2: better import ftplib and gopherlib etc late (2004-10-24) http://python.org/sf/1053150 opened by Robert Kiendl nntplib: add support for NNTP over SSL (2004-10-24) http://python.org/sf/1053365 opened by Reinhard Speyerer ftplib: add support for MDTM command (2004-10-24) http://python.org/sf/1053369 opened by Reinhard Speyerer Decimal speed up (2004-10-24) CLOSED http://python.org/sf/1053375 opened by Facundo Batista mingw compile (2004-10-25) http://python.org/sf/1053879 opened by Matthias Kramm fix inplace string concat hash error (2004-10-25) CLOSED http://python.org/sf/1054197 opened by Bob Halley automatic XMLRPC boxcarring via multicall for xmlrpclib (2004-10-26) http://python.org/sf/1054434 opened by Eric Ries Fix for Unix macro expansion in citetitle (2004-10-26) http://python.org/sf/1054715 opened by Richard Brodie bdist_deb - Debian packager (2004-10-26) http://python.org/sf/1054967 opened by Geoffrey T. Dairiki CGIHTTPServer can't redirect (2004-10-27) http://python.org/sf/1055159 opened by Thomas Guettler cgi.py does not correctly handle fields with ';' (2004-10-27) http://python.org/sf/1055234 opened by Vova typo in comment (unicodeobject.h) (2004-10-28) http://python.org/sf/1056231 opened by Baris Metin python / pythonw replacement in C (2004-10-29) http://python.org/sf/1056561 opened by Michiel de Hoon No ValueError for safe_substitute() (2004-10-29) http://python.org/sf/1056967 opened by Barry A. Warsaw Patches Closed ______________ __main__ for whichdb.py (2004-10-01) http://python.org/sf/1038388 closed by rhettinger Bug fixes and cleanup for decimal.py (2004-10-17) http://python.org/sf/1048728 closed by rhettinger tarfile.py: patch for bug #1017553 (2004-10-10) http://python.org/sf/1043972 closed by akuchling replace() method in unicode objects (2003-02-05) http://python.org/sf/680863 closed by nnorwitz Remove __file__ after running $PYTHONSTARTUP (2003-04-11) http://python.org/sf/719777 closed by nnorwitz sre fixes for lastindex and minimizing repeats+assertions (2003-03-31) http://python.org/sf/712900 closed by nnorwitz Some bug fixes for regular ex code. (2003-04-14) http://python.org/sf/720991 closed by nnorwitz optimize eval_frame (2003-12-21) http://python.org/sf/864059 closed by nnorwitz Implementation for PEP 318 ([as classmethod] version) (2004-04-08) http://python.org/sf/932100 closed by nnorwitz Add FreeBSD* to configure.in for --enable-shared (2004-10-11) http://python.org/sf/1044395 closed by perky New patch for Python 2.3 spec file. (2004-07-23) http://python.org/sf/996316 closed by jafo Syntax error in Misc/NEWS (2004-10-21) http://python.org/sf/1051866 closed by bcannon plistlib: add plst resource functionality, fix bugs (2004-10-22) http://python.org/sf/1052399 closed by jvr Decimal speed up (2004-10-24) http://python.org/sf/1053375 closed by rhettinger fix inplace string concat hash error (2004-10-25) http://python.org/sf/1054197 closed by rhettinger traceback module caches sources invalid (2003-05-14) http://python.org/sf/737473 closed by perky do not add directory of sys.argv[0] into sys.path (2004-05-02) http://python.org/sf/946373 closed by wrobell New / Reopened Bugs ___________________ 2.4b1 Win32 installer eliminates 2.3 assoc (2004-10-20) http://python.org/sf/1050486 opened by Mike C. Fletcher reindent.py strips execute permission on Unix (2004-10-20) http://python.org/sf/1050828 opened by David Ripton incorrect traceback filename from pyc (2004-10-21) http://python.org/sf/1051638 opened by Kevin Quick HTMLParser doesn't treat endtags in <script> tags as CDATA (2004-10-21) http://python.org/sf/1051840 opened by Luke Bradley Appending non-empty element to a non-empty list yields None (2004-10-22) CLOSED http://python.org/sf/1052036 opened by Peer Janssen Seting defaultencoding through an environment variable (2004-10-22) http://python.org/sf/1052098 opened by Diedrich Vorberg page on python api down (2004-10-22) CLOSED http://python.org/sf/1052274 opened by mperpick pdb runcall should accept keyword arguments (2004-10-22) CLOSED http://python.org/sf/1052503 opened by Leonardo Rochael Almeida filelist.findall should use os.lstat (2004-10-23) http://python.org/sf/1052827 opened by Geoffrey T. Dairiki time module returns wrong timezone under windows (2004-10-25) http://python.org/sf/1053539 reopened by panjunyong time module returns wrong timezone under windows (2004-10-25) http://python.org/sf/1053539 opened by panjunyong Unjustified SyntaxWarning (2004-10-25) CLOSED http://python.org/sf/1053604 opened by Markus Gritsch Why is this expression = False? (2004-10-25) CLOSED http://python.org/sf/1053629 opened by Alexander Rødseth PyOS_InputHook not called in subprocess (2004-10-25) http://python.org/sf/1053687 opened by Michiel de Hoon Segfault in tuple_of_constants (2004-10-25) CLOSED http://python.org/sf/1053819 opened by Duncan Grisby help() nor in tutorial index (2004-10-25) CLOSED http://python.org/sf/1053956 opened by Guido van Rossum Python doesn't exit with proper resultcode on SIGINT (2004-10-25) http://python.org/sf/1054041 opened by James Y Knight serious string hashing error in 2.4b1 (2004-10-25) CLOSED http://python.org/sf/1054139 opened by Bob Halley RE '*.?' cores if len of found string exceeds 10000 (2004-10-26) http://python.org/sf/1054564 opened by Rob New failure in debug build (2004-10-26) http://python.org/sf/1054615 opened by Tim Peters difflib HtmlDiff() extra space on inserted 1 character lines (2004-10-26) http://python.org/sf/1054821 opened by Dan Gass Python may contain NFC/NFKC bug per Unicode PRI #29 (2004-10-26) http://python.org/sf/1054943 opened by Rick McGowan pdb.set_trace should go up one more frame (2004-10-27) http://python.org/sf/1055168 opened by Johannes Gijsbers bdist_wininst broken for pure Python distributions (2004-10-27) CLOSED http://python.org/sf/1055540 opened by Thomas Heller weakref callback vs gc vs threads (2004-10-27) http://python.org/sf/1055820 opened by Tim Peters HTMLParser not compliant to XHTML spec (2004-10-27) http://python.org/sf/1055864 opened by Luke Bradley dir() should only return strings (2004-10-28) http://python.org/sf/1056293 opened by Skip Montanaro subprocess fails for args="...", executable="..." (2004-10-28) http://python.org/sf/1056441 opened by Russell Owen Lambda in list comprehension (2004-10-28) CLOSED http://python.org/sf/1056483 opened by Doug Blank minor error in os.access (2004-10-29) http://python.org/sf/1056498 opened by edward ambiguity in os.tmpfile (2004-10-29) CLOSED http://python.org/sf/1056502 opened by edward ambiguity in os.tmpfile (2004-10-29) http://python.org/sf/1056515 opened by edward subprocess works poorly on Windows with Python 2.3 (2004-10-29) http://python.org/sf/1057048 opened by Russell Owen subprocess on Windows 2: unexpected failure (2004-10-29) http://python.org/sf/1057052 opened by Russell Owen subprocess on Windows: extra windows (2004-10-29) http://python.org/sf/1057061 opened by Russell Owen Bugs Closed ___________ relative on-disk SYSTEM id handling is incorrect (2004-02-22) http://python.org/sf/902037 closed by rhettinger filemode() in tarfile.py makes wrong file mode strings (2004-08-27) http://python.org/sf/1017553 closed by akuchling test_subprocess 2.4b1 fails on FreeBSD 5.2 (2004-10-17) http://python.org/sf/1048808 closed by astrand No command line args when script run without python.exe (2004-09-16) http://python.org/sf/1029047 closed by facundobatista Appending non-empty element to a non-empty list yields None (2004-10-22) http://python.org/sf/1052036 closed by peerjanssen page on python api down (2004-10-22) http://python.org/sf/1052274 closed by loewis pdb runcall should accept keyword arguments (2004-10-22) http://python.org/sf/1052503 closed by rhettinger call arg of lambda not updating (2004-10-17) http://python.org/sf/1048870 closed by rhettinger time module returns wrong timezone under windows (2004-10-25) http://python.org/sf/1053539 closed by tim_one Unjustified SyntaxWarning (2004-10-25) http://python.org/sf/1053604 closed by rhettinger Why is this expression = False? (2004-10-25) http://python.org/sf/1053629 closed by doerwalter Segfault in tuple_of_constants (2004-10-25) http://python.org/sf/1053819 closed by rhettinger help() nor in tutorial index (2004-10-25) http://python.org/sf/1053956 closed by rhettinger serious string hashing error in 2.4b1 (2004-10-25) http://python.org/sf/1054139 closed by rhettinger strptime doesn't work with %U (2004-10-12) http://python.org/sf/1045381 closed by bcannon bdist_wininst broken for pure Python distributions (2004-10-28) http://python.org/sf/1055540 closed by mhammond Lambda in list comprehension (2004-10-28) http://python.org/sf/1056483 closed by rhettinger ambiguity in os.tmpfile (2004-10-28) http://python.org/sf/1056502 closed by facundobatista New / Reopened RFE __________________ Attempt to run all atexit handlers (2004-10-22) http://python.org/sf/1052242 opened by Greg Chapman Seting defaultencoding through an environment variable (2004-10-22) http://python.org/sf/1052098 reopened by loewis RFE Closed __________ Seting defaultencoding through an environment variable (2004-10-22) http://python.org/sf/1052098 closed by loewis

1 0

Re: [Python-Dev] bsddb todo for someone - auto-upgrade db format
by Gregory P. Smith 29 Oct '04

29 Oct '04

On Thu, Oct 21, 2004 at 12:49:19PM +0400, Oleg Broytmann wrote: > > BTW, just installing is not enough, even when it is come with Python > distribution. Installing a newer version of BerkeleyDB breaks older > databases due to incompatible file formats. fwiw its possible to fix this in the bsddb code. someone willing to do the exception catching database backing-up + auto db_upgrade() work should submit a patch. (i added backing-up as its rude to auto-convert someones database for them to a version their old software can no longer read) -g

4 5

Re: weakref callback vs gc vs threads
by Neil Schemenauer 29 Oct '04

29 Oct '04

[Taking this to email. Carrying out discussions via the SF bug tracker sucks.] Comment By: Tim Peters (tim_one) > [Neil] > > I had to change _PyWeakref_ClearRef() since it was also > > clearing the weakref list of the trash object. > > That was really its *purpose*. If a trash weakref with a > callback isn't removed from the referent's list of weakrefs, > then the callback will trigger when PyObject_ClearWeakRefs() > is invoked on the referent. The purpose of > _PyWeakref_ClearRef() was to ensure that the callback never > triggers. But it's okay of the callback triggers, as long as the callback doesn't reference trash. > > Now it just sets wr_object to Py_None. > > That won't stop the callback from triggering. It also means > (see earlier comment) that PyObject_ClearWeakRefs() will > never removed the weakref from the list either, although I'm > not sure that does real harm. I'm trying to figure out PyObject_ClearWeakRefs() right now. > > I also made some serious simplifications to gcmodule by > > just treating trash weakref objects with callbacks the same > > as objects with __del__ methods (i.e. move them to the > > finalizers list and then do the transitive closure of that set). > > Does that mean they can end up in gc.garbage too? If so, I > don't think that's sellable. I think so. That can be easily changed though. What we can't do is invoke those callbacks. Neil

3 6

RE: [Python-Dev] TRUNK UNFROZEN (release is done)
by Michael Chermside 29 Oct '04

29 Oct '04

Johannes writes: > I finally decided to stop postponing doing this. I'll be running a bug day on > November 7. I'll be in our office with some coworkers: any Dutch Pythoneers are > welcome to join us for some 'real life' Python hacking! Just a thought: should we consider doing a "patch review day" someday instead of a bug day? At a time like this we might be dividing patches into categories like "apply for 2.4", "apply for 2.5", "reject", "needs work", and "contraversial". -- Michael Chermside

2 1

TRUNK UNFROZEN (release is done)
by Anthony Baxter 29 Oct '04

29 Oct '04

The first beta is out, so the trunk is unfrozen, and available for checkins. Now that we're in beta, we shouldn't see any new features or behaviour changing fixes going into the trunk, unless it's been seen and agreed to on python-dev. I currently plan for a second beta in either 2 or 3 weeks, and then, assuming all goes well, a release candidate a couple of weeks after that. Anthony (apologies in the pause in getting the release email out - an unexpectedly hectic weekend popped up without notice)

10 21

RE: logging needs better documentation
by Vinay Sajip＠RedDove 29 Oct '04

29 Oct '04

Hi Neal, I quite agree that the documentation for the logging package can be improved. As others have said in response to your post, you can definitely help by indicating more specifically where you find the documentation lacking, with a patch if possible. For example, your initial post about the length prefix for pickles is documented in the docstring for SocketHandler.makePickle(): "Pickles the record in binary format with a length prefix, and returns it ready for transmission across the socket." I agree that the documentation does not mention specifically that the length is encoded as four bytes, or that it is packed using struct.pack(), or exactly how you unpack it. I also agree that more examples would be helpful. I will endeavour to improve the situation insofar as time allows. Patches and specific suggestions from users, especially new users like you, will be a definite help to me. Best Regards, Vinay Sajip

2 1

Re: Pre-existing bug of the millennium (so far)
by Jim Fulton 29 Oct '04

29 Oct '04

Tim Peters wrote: ... > Anyone have a bright idea? It's remarkable how long we've managed to go > without noticing that everything is disastrously broken here <0.9 wink>. Sure. Clearing cyclic trash can call Python code. If there are weakrefs to any of the cyclic trash, then those wekrefs can be used to resurrect the objects. Therefore, *before* clearing cyclic trash, we need to remove any weakrefs. If any of the weakrefs being removed have callbacks, then we need to save the callbacks and call them *after* all of the weakrefs have been cleared. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

3 3

RE: [Python-Dev] logging needs better documentation
by Chermside, Michael 29 Oct '04

29 Oct '04

Neal Becker writes: > There is only a single example in the logging module documentation. Please > add more examples! You are completely right... I am sure that those docs need to be improved. Would you be willing to write up some improved docs? If so, submit them to the patch manager on SourceForge, and we'll incorporate them into the next release possible (probably Python 2.5). If not, then it may have to wait until someone else has time to address it. -- Michael Chermside This email may contain confidential or privileged information. If you believe you have received the message in error, please notify the sender and delete the message without copying or disclosing it.

5 6

IPv6 bug in 2.3.4??
by David G Mills 28 Oct '04

28 Oct '04

Hey there, I've noticed a potential bug and wanted to see if anyone else can replicate it, or has also seen it. When creating a socket with getaddrinfo, with the address family set to AF_UNSPEC and the host set to localhost, the socket binds to IPv4 instead of IPv6. When using the FQD of the machine it binds to IPv6. I've tested the same code on two different machines, and the one running python 2.3.3 works as it should, but the python 2.3.4 doesn't. Anyone seen this? Can you replicate the error? Any help would be appreciated. Oh yeah, I'm running on FC2 on the 2.3.4 machine and redhat 8 on the 2.3.3 machine. Cheers, David.

3 5

logging needs better documentation
by Neal D. Becker 28 Oct '04

28 Oct '04

There is only a single example in the logging module documentation. Please add more examples! I'm trying to use DatagramHandler. The documentation on this is not adequate. There is a confusing mention of makeLogRecord. No example. I finally stumbled across this: data, address = sock.recvfrom (8192) rec = logging.makeLogRecord (pickle.loads(data[4:])) I can't find _anywhere_ a mention that you need to leave off the first 4 bytes in order for unpickle to work.

2 1

SQLite module for Python 2.5
by Gerhard Haering 28 Oct '04

28 Oct '04

Hi python-dev-elopers, Last December, we had a short thread discussing the integration of PySQLite into Python 2.4. At the time, I was against inclusion, because I thought PySQLite was not ripe for it, mostly because I thought the API was not stable. Now, I have started writing a new PySQLite module, which has the following key features: - Uses iterator-style SQLite 3.x API: sqlite3_compile, sqlite3_step() etc. This way, it is possible to use prepared statements, and for large resultsets, it requires less memory, because the whole resultset isn't fetched into memory at once any longer. - Completely incompatible with the SQLite 0.x/1.x API: I'm free to create a much better API now. - "In the face of ambiguity, refuse the temptation to guess." - PySQLite 1.x tries to "guess" which Python type to convert to. It's pretty good at it, because it queries the column type information. This works for, I'd say 90 % of all cases at least. But as soon as you use anything fancy like functions, aggregates or expressions in SQL, the _typeless_ nature of SQLite breaks through and it will tell us nothing about the declared column type (of course, because the data is not coming from a database column). So I decided to change the default behaviour and make PySQLite typeless by default, too. Everything will be returned as a Unicode string (the default might be user-configurable per connection). Unless, unless of course the user explicitly activates the "guess-mode" ;-) But to do so, she must read the docs then she will be aware of the fact that it only works in 90 % of all cases. So why am I bothering you about this? I think that a simple embedded relational database would be a good thing to have in Python by default. And as Python 2.5 won't happen anytime soon, there's plenty of time for developing it, getting it stable, and integrating it. Especially those of you that have used PySQLite in the past, do you have any suggestions that would make the rewrite a better candidate for inclusion into Python? One problem I see is that even the new PySQLite will grow and try to wrap much of the SQLite API that are not directly related to the DB-API. If such a thing is too complicated/big for the standard library, then maybe it would be better to produce a much simpler PySQLite, especially for the Python standard library that leaves all the fancy stuff out. My codename would be "embsql". So, what would you like to see? "import sqlite", "import embsql", or "pypi.install('pysqlite')" ? -- Gerhard

21 47

Changing pymalloc behaviour for long running processes
by Evan Jones 28 Oct '04

28 Oct '04

I know that this has been discussed a bit in the past, but I was hoping that some Python gurus could shed some light on this issue, and maybe let me know if there are any plans for solving this problem. I know a hack that might work, but there must be a better way to solve this problem. The short version of the problem is that obmalloc.c never frees memory. This is a great strategy if the application runs for a short time then quits, or if it has fairly constant memory usage. However, applications with very dynamic memory needs and that run for a long time do not perform well because Python hangs on to the peak amount of memory required, even if that memory is only required for a tiny fraction of the run time. With my application, I have a python process which occupy 1 GB of RAM for ~20 hours, even though it only uses that 1 GB for about 5 minutes. This is a problem that needs to be addressed, as it negatively impacts the performance of Python when manipulating very large data sets. In fact, I found a mailing list post where the poster was looking for a workaround for this issue, but I can't find it now. Some posts to various lists [1] have stated that this is not a real problem because virtual memory takes care of it. This is fair if you are talking about a couple megabytes. In my case, I'm talking about ~700 MB of wasted RAM, which is a problem. First, this is wasting space which could be used for disk cache, which would improve the performance of my system. Second, when the system decides to swap out the pages that haven't been used for a while, they are dirty and must be written to swap. If Python ever wants to use them again, they will be brought it from swap. This is much worse than informing the system that the pages can be discarded, and allocating them again later. In fact, the other native object types (ints, lists) seem to realize that holding on to a huge amount of memory indefinitely is a bad strategy, because they explicitly limit the size of their free lists. So why is this not a good idea for other types? Does anyone else see this as a problem? Does anyone think this is not a problem? Proposal: - Python's memory allocator should occasionally free memory if the memory usage has been relatively constant, and has been well below the amount of memory allocated. This will incur additional overhead to free the memory, and additional overhead to reallocate it if the memory is needed again quickly. However, it will make Python co-operate nicely with other processes, and a clever implementation should be able to reduce the overhead. Problem: - I do not completely understand Python's memory allocator, but from what I see, it will not easily support this. Gross Hack: I've been playing with the fact that the "collect" function in the gc module already gets called occasionally. Whenever it gets called for a level 2 collection, I've hacked it to call a cleanup function in obmalloc.c. This function goes through the free pool list, reorganizes it to decrease memory fragmentation and decides based on metrics collected from the last run if it should free some memory. It currently works fine, except that it will permit the arena vector to grow indefinitely, which is also bad for a long running process. It is also bad because these cleanups are relatively slow as they touch every free page that is currently allocated, so I'm trying to figure out a way to integrate them more cleanly into the allocator itself. This also requires that nothing call the allocation functions while this is happening. I believe that this is reasonable, considering that it is getting called from the cyclical garbage collector, but I don't know enough about Python internals to figure that out. Eventually, I hope to do some benchmarks and figure out if this is actually a reasonable strategy. However, I was hoping to get some feedback before I waste too much time on this. Evan Jones [1] http://groups.google.com/groups?selm=mailman.1053801468.4243.python- list%40python.org -- Evan Jones: http://evanjones.ca/ "Computers are useless. They can only give answers" - Pablo Picasso

7 14

To run all exit handlers or not?
by Skip Montanaro 28 Oct '04

28 Oct '04

The current behavior of the atexit module is that if any of the exit handlers raises an exception the remaining handlers are not run. Greg Chapman posted a bug report about this: http://www.python.org/sf/1052242 Greg proposed catching any exceptions and continuing so that all exit handlers at least have a chance to run and Raymond agrees with him. I attached a patch to the ticket to add a flag to determine the behavior on the principle that atexit has been around long enough that someone out there probably relies on the early exit behavior. This is the old Python chestnut of using a flag to preserve existing behavior as the default while allowing users to set the flag to get the new behavior. I'm happy to go either way, but thought perhaps a quick poll of the troops might be in order, hence this note. Skip

2 2

Pre-existing bug of the millennium (so far)
by Tim Peters 28 Oct '04

28 Oct '04

This one came up while working on ZODB: weakref callback vs. gc vs. threads http://www.python.org/sf/1055820 Short course: in the presence of weakrefs, cyclic gc is still hosed (it turns out that neither threads nor weakref callbacks are necessary to get hosed). temp2a.py there demonstrates there's a problem, but in an unclear way (hundreds of objects, hundreds of weakrefs and weakref callbacks (all via WeakValueDictionary internals), 3 threads). OTOH, there's nothing clever or tricky about it. Sooner or later, it just fails (an accessible instance of a user-defined class gets its __dict__ cleared "by magic"). temp2b.py reduces it to 2 objects and 1 thread. This is contrived, but is deterministic. temp2c.py boosts it to 3 objects, and is a nightmare: it shows that the problem can occur during a gc collection that doesn't see *any* objects having a weakref with a callback. There is a weakref with a callback here, but it's attached to an object in an older generation, and collection of a younger generation triggers that callback indirectly. Because this is such a nasty case (no amount of analysis of the objects in the generation being collected can deduce that it's possible for a weakref callback to run), there are extensive comments and an ASCII-art diagram in the file. Even worse, temp2d.py shows we can get in trouble even if there's never a weakref with a callback. It's enough to have one weakref (without a callback), and one object with a __del__ method. Offhand, I don't have a plausible solution that will work. The elegant <wink> analysis in gc_weakref.txt missed what should have been obvious even then: cyclic trash is still potentially reachable via "global" weakrefs, so if any Python code whatsoever can run while gc is breaking cycles (whether via __del__ or via wr callback), global weakrefs can resurrect cyclic trash. That suggests some Draconian approaches. Anyone have a bright idea? It's remarkable how long we've managed to go without noticing that everything is disastrously broken here <0.9 wink>.

1 0

Re: Python-Dev Digest, Vol 15, Issue 46
by Luis P Caamano 28 Oct '04

28 Oct '04

On Tue, 19 Oct 2004 12:02:14 +0200 (CEST), Evan Jones <ejones(a)uwaterloo.ca> wrote: > Subject: [Python-Dev] Changing pymalloc behaviour for long running > processes > [ snip ] > > The short version of the problem is that obmalloc.c never frees memory. > This is a great strategy if the application runs for a short time then > quits, or if it has fairly constant memory usage. However, applications > with very dynamic memory needs and that run for a long time do not > perform well because Python hangs on to the peak amount of memory > required, even if that memory is only required for a tiny fraction of > the run time. With my application, I have a python process which occupy > 1 GB of RAM for ~20 hours, even though it only uses that 1 GB for about > 5 minutes. This is a problem that needs to be addressed, as it > negatively impacts the performance of Python when manipulating very > large data sets. In fact, I found a mailing list post where the poster > was looking for a workaround for this issue, but I can't find it now. > > Some posts to various lists [1] have stated that this is not a real > problem because virtual memory takes care of it. This is fair if you > are talking about a couple megabytes. In my case, I'm talking about > ~700 MB of wasted RAM, which is a problem. First, this is wasting space > which could be used for disk cache, which would improve the performance > of my system. Second, when the system decides to swap out the pages > that haven't been used for a while, they are dirty and must be written > to swap. If Python ever wants to use them again, they will be brought > it from swap. This is much worse than informing the system that the > pages can be discarded, and allocating them again later. In fact, the > other native object types (ints, lists) seem to realize that holding on > to a huge amount of memory indefinitely is a bad strategy, because they > explicitly limit the size of their free lists. So why is this not a > good idea for other types? > > Does anyone else see this as a problem? > This is such a big problem for us that we had to rewrite some of our daemons to fork request handlers so that the memory would be freed. That's the only way we've found to deal with it, and it seems, that's the preferred python way of doing things, using processes, IPC, fork, etc. instead of threads. In order to be able to release memory, the interpreter has to allocate memory in chunks bigger than the minimum that can be returned to the OS, e.g., in Linux that'd be 256bytes (iirc), so that libc's malloc would use mmap to allocate that chunk. Otherwise, if the memory was obtained with brk, then in most virtually all OSes and malloc implementations, it won't be returned to the OS even if the interpreter frees the memory. For example, consider the following code in the interactive interpreter: for i in range(10000000): pass That run will create a lot of little integer objects and the virtual memory size of the interpreter will quickly grow to 155MB and then drop to 117MB. The 117MB left are all those little integer objects that are not in use any more that the interpreter would reuse as needed. When the system needs memory, it will page out the pages where these objects have been allocated to swap. In our application, paging to swap is extremely bad because sometimes we're running the OS booted from the net without swap. The daemon has to loop over list of 20 to 40 thousand items at a time and it quickly grows to 60mb on the first run and then continues to grow from there. When something else needs memory, it tries to swap and then crashes. In the example above, the difference between 155MB and 117MB is 37MB, which I assume is the size of the list object returned by 'range()' which contains the references to the integers. The list goes away when the interpreter finishes running the loop and because it was already known how big it was going to be, it was allocated as a big chunk using mmap (my speculation). As a result, that memory was given back to the OS and the virtual memory size of the interpreter went down from 155MB to 117MB. Regards, -- Luis P Caamano Atlanta, GA USA PS I rarely post to python-dev, this is probably the first time, so please let me take this opportunity to thank all the python developers for all your efforts, such a great language, and great tool. My respect and admiration to all of you.

4 4

adding Py{String|Unicode}_{Lower|Upper} and fixing CreateProcess in _subprocess.pyd and PyWin32
by Trent Mick 28 Oct '04

28 Oct '04

There is a subtlety in CreateProcess in the Win32 API in that if one specifies an environment (via the lpEnvironment argument), the environment strings (A) must be sorted alphabetically and (B) that sort must be case-insensitive. See the Remarks section on: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/ba… If this is not done, then surprises can happen with the use of {Get|Set}EnvironmentVariable in the created process: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/ba… Neither _subprocess.pyd (supporting the new subprocess.py module on Windows) nor PyWin32's CreateProcess binding do this. I haven't done so yet, but I should be able to put together a test case for subprocess.py for this. We came across such a surprise when using my process.py module that uses this PyWin32 code (which it looks like _subprocess.c borrowed). Fixing (A) is easy with a "PyList_Sort(keys)" and some other minor changes to _subprocess.c::getenvironment() -- and to win32process.i::CreateEnvironmentString() in PyWin32. However, I'd like some guidance on the best way to case-insensitively sort a Python list in C code to fix (B). The best thing I see would be to expose PyString_Lower/PyUnicode_Lower and/or PyString_Upper/PyUnicode_Upper so they can be used to .lower()/.upper() the given environment mapping keys for sorting. Does that sound reasonable? Is there some problem to this approach that anyone can see? Trent -- Trent Mick trentm(a)activestate.com

6 6

RE: [Python-Dev] On when making modifications to the CVS code
by Batista, Facundo 27 Oct '04

27 Oct '04

[Tim Delaney] #- I think those three platforms are sufficiently #- representative of Python #- users, so if it works on them, and the code looks good to a #- reviewer, it #- should be committed. It's not exactly a large patch after all ... Do you want to take a look at it? ;) #- What's the bug number? I've got a FreeBSD (5.2.1) virtual #- machine sitting #- around I could try it on (tomorrow - bed time now ;). The bug is 1050828. Thanks! . Facundo

1 0

On when making modifications to the CVS code
by Batista, Facundo 27 Oct '04

27 Oct '04

People: I have these doubts from a while ago, and while a learned a lot about this through Raymond Hettinger, I still have some loose ends. Don't know if there's an official position or it's just developer common sense (which I still don't have), but I didn't find an article/PEP about this. Such paper exists? For now, I'll ask you about a specific issue: there's this bug open about the reindent.py tool, which has an issue about the reindented code file's metadata (more specific: permissions). So I came up with a solution (small patch, three lines), which leaves the reindented file with the same permissions that the original one. My problem is that I can not decide to commit those changes. We're in beta, and don't know if the changes will be tested enough before the final release, in enough platforms (for now, it's tested on Linux, Win2k and MacOS X). And if I don't commit these now, when? Thank you all! . Facundo

2 1

[Python-Dev] Re: [Python-checkins] python/dist/src/Python compile.c, 2.330, 2.331
by Guido van Rossum 26 Oct '04

26 Oct '04

Haven't seen the bug report, but you do realize that comparing code objects has other applications, and this pretty much kills that. On Sat, 23 Oct 2004 17:10:09 -0700, rhettinger(a)users.sourceforge.net <rhettinger(a)users.sourceforge.net> wrote: > Update of /cvsroot/python/python/dist/src/Python > In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv30133/Python > > Modified Files: > compile.c > Log Message: > SF bug #1048870: call arg of lambda not updating > > Index: compile.c > =================================================================== > RCS file: /cvsroot/python/python/dist/src/Python/compile.c,v > retrieving revision 2.330 > retrieving revision 2.331 > diff -u -d -r2.330 -r2.331 > --- compile.c 29 Sep 2004 23:54:08 -0000 2.330 > +++ compile.c 24 Oct 2004 00:10:05 -0000 2.331 > @@ -261,6 +261,8 @@ > if (cmp) return (cmp<0)?-1:1; > cmp = co->co_flags - cp->co_flags; > if (cmp) return (cmp<0)?-1:1; > + cmp = co->co_firstlineno - cp->co_firstlineno; > + if (cmp) return (cmp<0)?-1:1; > cmp = PyObject_Compare(co->co_code, cp->co_code); > if (cmp) return cmp; > cmp = PyObject_Compare(co->co_consts, cp->co_consts); > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > http://mail.python.org/mailman/listinfo/python-checkins > -- --Guido van Rossum (home page: http://www.python.org/~guido/)

6 10

python with database connectivity
by Sandeep Gohad 25 Oct '04

25 Oct '04

Hello I am little new with the python.... If I want to develope any database related application,with options like add,save,modify,delete,previous,last,next,first for maintaining Employee's records.I want to use .mdb file or d-base database as a backend. In VB we can use ADO or DAO, In java we use JDBC, so for python please guide me for developing any basic application.... Regards Sandeep Yahoo! India Matrimony: Find your life partneronline.

2 1

Paid Research Project on Stackless 3.1
by Christian Tismer 24 Oct '04

24 Oct '04

(re-sent and modified, after I recognized that my hardware-clock is broken, need a new note-buck) Dear community, I would love to publish Stackless 3.1, of course. Also I know that there is some inherent bug in it. This is the state of the art sine four months. I am currently in a very tight project and have no time to dig into this problem. BUT IT IS URGENT! I'm seeking for a person who would take the job to find the buglet. He would need to debug and nail down a commercial application, which I cannot make public. (S)He would need to sign an NDA with me. The success payment would be $500, minimum. If the problem shows up to be very hard (to some undefined definition of very hard, to be negotiated), it can be increased to $1000. If my app works afterwards, Stackless 3.1 is just fine and can go out to the public.. If it doesn't work, no payment happens. The identified problem needs to be documented by a reproducible test case. If somebody is interested, please contact me privately. And be aware, this is really no easy stuff. You need to be a real hardcore system hacker with many years of experience. (Armin, Bob, Jeff, Lutz, Richard, Stefan, Stephan?) Here is the CVS path to the dev trunk: CVSROOT=:pserver:anonymous@stackless.com:/home/cvs cvs co slpdev/src/2.3/dev The cheapest complete solution wins. Hurry up :-) Sincerely -- chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Carmerstr. 2 : *Starship* http://starship.python.net/ 10623 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 31 86 04 18 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

1 0

Re: [ python-Bugs-1048808 ] test_subprocess 2.4b1 fails on FreeBSD 5.2
by Peter Astrand 23 Oct '04

23 Oct '04

Comments? Is it safe or not to assume that by the time the Python has started, only fds 0,1,2 are open? /Peter > I think this is the core of the problem. The test_close_fds > test works like this: > > All file descriptors in the forked child (except 0,1,2) are > closed. Then the Python binary is executed via execvp(). A > small test program is passed to the Python binary via the -c > command line option. If the OS and subprocess module works > correctly, we can be sure of that by the time of the > execve() system call, only file descriptors (0,1,2) are open > (well, the errpipe as well, but let's leave that out for > now). But, by the time the Python binary starts executing > the small program, all sorts of things may have happened. > I'm not really sure we can trust Python not to open files > during startup. For example, if we have a PYTHONSTARTUP > file, that open file will have a file descriptor, perhaps 3. > > On one hand, this bug could indicate a bug in the Python > interpreter itself: perhaps a file descriptor leak. On the > other hand, this test might be a bit too unsafe. > > So probably, this test should be removed.

4 4

Adding a developer
by A.M. Kuchling 23 Oct '04

23 Oct '04

Could someone please add Lars Gustaebel (SF ID "gustaebel") as a developer? He's responsible for writing and maintaining tarfile.py, but his bugfixes usually sit around in the tracker until someone gets around to looking at them, which is inefficient. Thanks! --amk

7 14

ConfigParser shootout, preliminary entry
by Michael Chermside 21 Oct '04

21 Oct '04

A few weeks ago, the suggestion was made on Python-Dev that it might be time to consider replacing the ConfigParser module and that we should hold a "shootout" (ie ask for implementations and see what we get). Since then I've been playing around with this... not the parsing part (which so far I have completely ignored) but the programmer interface. There needs to be a well-thought-out data model for the information stored, and the user interface needs to be very easy to use, yet not so "magical" that it becomes difficult to understand. I have put together what I think is probably my best proposal. It is based on a superset of ini config files and java .property files. There is a convenient access mechanism ("config.my_app.some_value") as well as more general approaches ("config.values['my_app.serviceByPort.80']"). I have tried to consider issues like unicode (I permit fairly lenient mixing of unicode and str), and unit testing ("... call config.clear_all() in the tearDown() method of any unittests that use the config module..."). I have even considered carefully what to leave OUT (converting to non-string data types, interpolating values, things like that). I think that I am now at the point where I could really use some input from others. So I'd like to invite people to review my design and send me your suggestions. I'm not expecting this as a *useful* module yet (it doesn't yet parse files!), but it seemed like a good stage at which to ask for feedback. I'm attaching two files, config.py and configTest.py, and they are also available from these urls: http://www.mcherm.com/publish/2004-10-17/config.py http://www.mcherm.com/publish/2004-10-17/configTest.py Thanks in advance for reviewing this. -- Michael Chermside

10 22

audiotest.au and possible copyright issues?
by Mihai Ibanescu 21 Oct '04

21 Oct '04

Hello, Here's an interesting bug: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=136654 Is anyone aware of copyright issues on the Monty Python audio file? Thanks, Misa

8 8

Patches
by Oleg Broytmann 21 Oct '04

21 Oct '04

Hello! I submitted a number of patches to the SF tracker. Few of them were accepted, but most of them still are unreviewd by core team. What can I do to help? (A polite way to say "how can I push them?" :) I have updated them to Python 2.4b1. Actually, only one of them required minor changes (different line numbers). But I must admit that the patches are not a simple bug fix, so it is probably too late to apply them for 2.4. They can wait for 2.5... but I am afraid they will be just forgotten. The patches are: http://python.org/sf/754022 This is the biggest and the oldest. It hangs in the tracker since Python 2.2. In short, it makes the webbrowser.py runs through _tryorder list of browsers and trie to run every browser until one started successfully. Currently webbrowser.py tries to run a browser and if it fails - stops trying. Assigned to Fred Drake. http://python.org/sf/784089 A program to scan python files and list those require -*- coding -*- directive. Reviewed by Marc-Andre Lemburg and Andrew Kuchling. http://python.org/sf/821862 Makes ftplib.py a bit more RFC959-compliant. The RFC requires the FTP protocol to be run not over TCP but over telnet. For most people there is no difference but there is a subtle different for those who use non-ASCII encodings: chr(255) - a special character in telnet - requires a special handling. My patch adds a toggle that allows to turn telnet on (actually, it only doubles chr(255) in command stream, but it is one of two things that are required for FTP-over-telnet). By default it is off to preserve backward compatibility. http://python.org/sf/1038388 The smallest and the simplest of all. 3 lines patch that adds __main__ to whichdb.py, so one can run it on the command line: $ whichdb.py *.db Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd(a)phd.pp.ru Programmers don't die, they just GOSUB without RETURN.

8 12

What are the goals for ConfigParser 3000?
by David Wilson 20 Oct '04

20 Oct '04

On Sun, Oct 17, 2004 at 03:38:11PM -0700, Michael Chermside wrote: > My apologies, my previous email was intended to go to python-list, not > python-dev. I didn't intend to bother this list with it until it was > at a point where I thought it was a contender for inclusion. Sometimes > I really wish there was an "Unsend" option for email. I think it raised a point though - there were a few wildly different ideas about what ConfigParser's replacement should be. Would it be wise to discuss first what ConfigParser is *supposed* to be, and where it fits in peoples' applications? If this hits a chord with you, pump your reply button and discuss! If you look at the current ConfigParser, you might find the following: - It is a simple storage method for text data that categorises sets of keys with sets of values. - Its backend format is simple and fast for other tools to parse (even for on-request applications, like web scripts). - For almost any complex data, it becomes immediately less useful, but with a small amount of external wrapping, can also be used for more complex configuration structures. - It is *not* a means for declaring objects or doing sums: that is what the Python syntax itself is for. My problems with the current ConfigParser are as follows: - The interface is unfriendly and error-prone. Examples: - read(): raises no exception if a file is missing. This is against Python culture, and the unsurprising name causes IMHO surprising behaviour, especially for the standard library. - readfp(): 'more than one way to do it'. File-like objects are such a fundamental part of Python that I think read() should go away entirely. This would mean that a programmer is explicitly reminded of the errors that can occur when opening his configuration file as he explicitly types the file() factory expression himself. - Again, for getting values there are a few methods which essentially repeat themselves - getint(), getfloat(), etc. What is wrong with int(obj.get(...))? - get() shares the same method name as the infinitely popular dict.get() method, yet does not support a <default> argument. - String interpolation! There is a friendly string syntax as a standard library feature these days, and less friendly interpolation has always existed in the language itself. This should go, and be made more explicit for the programmer. In my opinion we should differenciate at an early stage between complex processing tasks (Python itself) and storing random snippets of configuration data. In short, removing the functions that really shouldn't be there and modernizing the code, would be a good starting place for a replacement. Only then we can start talking about ConfigParser-NG. Thanks, David. -- Now that my house has burned down I have a much better view of the moon.

3 2

Python 2.4b1 on win32 installer bug?
by Alex A. Naanou 20 Oct '04

20 Oct '04

here it is.. actually quite simple, the option not to register extensions simply unregisters them.... thanks. -- Alex.

3 6

Paid Research Project on Stackless 3.1
by Christian Tismer 20 Oct '04

20 Oct '04

Dear community, I would love to publish Stackless 3.1, of course. Also I know that there is some inherent bug in it. I'm seeking a person who would take the job to find the buglet. He would need to debug and nail down a commercial application, which I cannot make public. (S)He would need to sign an NDA with me. The success payment would be $500. If the problem shows up to be very hard (to some undefined definition of very hard, to be negotiated), it can be increased to $1000. If my app works afterwards, Stackless 3.1 is just fine. If it doesn't work, no payment happens. The identified problem needs to be documented by a reprodicuble test case. If somebody is interested, please contact me privately. And be aware, this is really no easy stuff. You need to be a real hardcore system hacker with many years of experience. Here is the CVS path to the dev trunk: CVSROOT=:pserver:anonymous@stackless.com:/home/cvs cvs co slpdev/src/2.3/dev The cheapest complete solution wins. Hurry up :-) Sincerely -- chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Carmerstr. 2 : *Starship* http://starship.python.net/ 10623 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 31 86 04 18 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

1 0

Re: subprocess - updated popen5 module
by Peter Astrand 20 Oct '04

20 Oct '04

[From Mail to python-announce list] > I'd like to give you an updated status report of the popen5 project. Since > my last post at 2004-06-04, this has happened: ... > With these changes, the subprocess module now feels quite complete. In the > near future, I will focus on getting the module included in the standard > library. I've recieved very positive feedback on my module. Many users are also asking me if this module will be included in the standard library. That is, of course, my wish as well. So, can the subprocess module be accepted? If not, what needs to be done? /Peter Åstrand <astrand(a)lysator.liu.se>

14 48

Weekly Python Patch/Bug Summary
by Kurt B. Kaiser 20 Oct '04

20 Oct '04

Patch / Bug Summary ___________________ Patches : 240 open ( +0) / 2661 closed ( +6) / 2901 total ( +6) Bugs : 772 open ( +6) / 4528 closed (+14) / 5300 total (+20) RFE : 155 open ( +0) / 131 closed ( +0) / 286 total ( +0) New / Reopened Patches ______________________ typo in liblocale.tex (2004-10-15) CLOSED http://python.org/sf/1047577 opened by George Yoshida fileinput.py fixed clobbering behavior and added encodings (2004-10-15) http://python.org/sf/1048075 opened by Chris Connett subprocess documentation (2004-10-16) CLOSED http://python.org/sf/1048341 opened by Fredrik Lundh Bug fixes and cleanup for decimal.py (2004-10-17) http://python.org/sf/1048728 opened by Neal Norwitz adding bool support to xdrlib.py (2004-10-18) http://python.org/sf/1049151 opened by Lars Gustäbel PyOS_InputHook inconsistency on Windows (2004-10-19) http://python.org/sf/1049855 opened by Michiel de Hoon Patches Closed ______________ Bad test for HAVE_UINTPTR_T in PC/pyconfig.h (2004-09-01) http://python.org/sf/1020042 closed by loewis typo in liblocale.tex (2004-10-15) http://python.org/sf/1047577 closed by nnorwitz Buffer overwrite in PyUnicode_AsWideChar (2004-10-14) http://python.org/sf/1047269 closed by lemburg subprocess documentation (2004-10-16) http://python.org/sf/1048341 closed by effbot Implementation for PEP 318 (Guido's version) (2004-03-31) http://python.org/sf/926860 closed by nnorwitz rfc822.parsedate returns a tuple (2003-04-01) http://python.org/sf/713599 closed by nnorwitz New / Reopened Bugs ___________________ Turtle.py hangs Idle (2004-10-15) http://python.org/sf/1047540 opened by Taro Ogawa Turtle.py hangs Idle (2004-10-15) CLOSED http://python.org/sf/1047549 opened by Taro Ogawa stat documentation is incorrect (2004-10-16) CLOSED http://python.org/sf/1048234 opened by libber ator Memory leaks? (2004-10-17) http://python.org/sf/1048495 opened by Roman Mamedov SystemError with deque (2004-10-17) CLOSED http://python.org/sf/1048498 opened by George Yoshida Add pythonX.dll to KnownDlls (2004-10-17) CLOSED http://python.org/sf/1048681 opened by Miki Tebeka Doc typo (2004-10-17) CLOSED http://python.org/sf/1048756 opened by Scott David Daniels test_subprocess 2.4b1 fails on FreeBSD 5.2 (2004-10-17) http://python.org/sf/1048808 opened by Shalabh Chaturvedi C-k does not work for curses.textpad.Textbox (2004-10-17) CLOSED http://python.org/sf/1048816 opened by Stefan Heimann Only "Overwrite" mode possible with curses.textpad.Textbox (2004-10-17) http://python.org/sf/1048820 opened by Stefan Heimann unexpected floating point exponent behaviour (2004-10-17) CLOSED http://python.org/sf/1048864 opened by Ruwen Hahn codecs.StreamReader.readlines() broken (2004-10-17) CLOSED http://python.org/sf/1048865 opened by Thomas Waldmann call arg of lambda not updating (2004-10-17) http://python.org/sf/1048870 reopened by kquick call arg of lambda not updating (2004-10-17) http://python.org/sf/1048870 opened by Kevin Quick shutil.copytree copies stat of files, but not of dirs (2004-10-18) http://python.org/sf/1048878 opened by Thomas Waldmann Registry not updated by 2.4b1 installer (2004-10-17) CLOSED http://python.org/sf/1048900 opened by B Sizer strptime doesn't work with %U (2004-10-12) http://python.org/sf/1045381 reopened by bcannon os.rmtree error handling was always broken (2004-10-17) http://python.org/sf/1048941 opened by Guido van Rossum Install-time module compilation fails with 2.4b1 (2004-10-18) http://python.org/sf/1049003 opened by Tony Meyer Solaris: EINTR exception in select/socket calls in telnetlib (2004-10-18) http://python.org/sf/1049450 opened by Mike Brauwerman smeared title when installing (2004-10-18) http://python.org/sf/1049615 opened by Jim Jewett test_socket PORT conflict with boa-constructor (2004-10-19) http://python.org/sf/1049816 opened by Simon Dahlbacka zlib module documentation wrong version (2004-10-19) CLOSED http://python.org/sf/1049826 opened by Wolfgang Langner rfc822.parseaddr is broken, breaks sendmail call in smtplib (2004-10-19) http://python.org/sf/1050268 opened by Scott Dossey Bugs Closed ___________ Turtle.py hangs Idle (2004-10-15) http://python.org/sf/1047549 closed by tso Error in the given example (2004-10-14) http://python.org/sf/1046800 closed by facundobatista stat documentation is incorrect (2004-10-16) http://python.org/sf/1048234 closed by jlgijsbers Docs for os, popen2 omit list argument to popen2() et al (2004-03-01) http://python.org/sf/907457 closed by jlgijsbers Unable to install Docs - Linux (2004-10-11) http://python.org/sf/1044872 closed by jlgijsbers distutils.util.get_platform() should include sys.version[:3] (2004-09-27) http://python.org/sf/1035703 closed by etrepum SystemError with deque (2004-10-16) http://python.org/sf/1048498 closed by rhettinger textReader: reading after close is a seg fault (2004-09-30) http://python.org/sf/1038013 closed by nnorwitz Add pythonX.dll to KnownDlls (2004-10-17) http://python.org/sf/1048681 closed by loewis Doc typo (2004-10-17) http://python.org/sf/1048756 closed by nnorwitz C-k does not work for curses.textpad.Textbox (2004-10-17) http://python.org/sf/1048816 closed by akuchling unexpected floating point exponent behaviour (2004-10-17) http://python.org/sf/1048864 closed by tim_one codecs.StreamReader.readlines() broken (2004-10-18) http://python.org/sf/1048865 closed by perky call arg of lambda not updating (2004-10-17) http://python.org/sf/1048870 closed by rhettinger Registry not updated by 2.4b1 installer (2004-10-18) http://python.org/sf/1048900 closed by loewis strptime doesn't work with %U (2004-10-12) http://python.org/sf/1045381 closed by bcannon zlib module documentation wrong version (2004-10-19) http://python.org/sf/1049826 closed by akuchling warning '"_POSIX_C_SOURCE" redefined' when include Python.h (2004-10-13) http://python.org/sf/1045893 closed by loewis

1 0

Fwd: [Python-Dev] Python 2.4b1 on win32 installer bug?
by Alex A. Naanou 19 Oct '04

19 Oct '04

Martin wrote: > Also, upgrading from 2.4a3 (with extensions installed) to 2.4b1 > (with no extensions) would have the same effect - there is simply > no support for restoring older versions of the extensions when > uninstalling (and upgrading from 2.4a3 to 2.4b1 is actually the > same as uninstalling 2.4a3 first). this appears to be exactly the case with me, I run both the 2.3 and 2.4 branches on the same box... and I upgraded py2.4a3 to py2.4b1. the funny thing is that when I installed Py2.4a3 some time back all went well. this time the installer restored the last setup options (except the setup dir... I do not use %systemdrive%\python24\) and removed ALL file associations, including for Py2.3.4, and messed up the overall python registry configuration! (which is quite odd....). -- Alex.

1 1

Re: [Python-Dev] Changing pymalloc behaviour for long running
by Luis P Caamano 19 Oct '04

19 Oct '04

> "Martin v. L?wis" <martin(a)v.loewis.de> wrote: > > Evan Jones wrote: > > Some posts to various lists [1] have stated that this is not a real > > problem because virtual memory takes care of it. This is fair if you > > are talking about a couple megabytes. In my case, I'm talking about > > ~700 MB of wasted RAM, which is a problem. > > This is not true. The RAM is not wasted. As you explain later, the > pages will be swapped out to swap space, making the RAM available > again for other tasks. > > > First, this is wasting space > > which could be used for disk cache, which would improve the performance > > of my system. > > And indeed, this is what the operating system does for you: free the > memory (by swapping it out), then using the memory for disk cache, thus > improving performance of your system. Your statements above assume that swapping is OK but that's not necessarily true. Some of our daemons run on systems with no swap or extremely slow swap. Either way it's bad if the system has to swap out 50-100 MBs of "unused" memory. Now, that's a waste. -- Luis P Caamano Atlanta, GA USA

1 0

Re: python/dist/src/Doc/lib libzlib.tex,1.28,1.29
by Fredrik Lundh 19 Oct '04

19 Oct '04

akuchling(a)users.sourceforge.net wrote: > Index: libzlib.tex > =================================================================== > RCS file: /cvsroot/python/python/dist/src/Doc/lib/libzlib.tex,v > retrieving revision 1.28 > retrieving revision 1.29 > diff -u -d -r1.28 -r1.29 > --- libzlib.tex 21 Jun 2003 14:15:25 -0000 1.28 > +++ libzlib.tex 19 Oct 2004 18:52:49 -0000 1.29 > @@ -9,8 +9,8 @@ > For applications that require data compression, the functions in this > module allow compression and decompression, using the zlib library. > The zlib library has its own home page at > -\url{http://www.gzip.org/zlib/}. Version 1.1.3 is the > -most recent version as of September 2000; use a later version if one > +\url{http://www.gzip.org/zlib/}. Version 1.2.1 is the > +most recent version as of October 2004; use a later version if one > is available. There are known incompatibilities between the Python > module and earlier versions of the zlib library. that's somewhat misleading, isn't it? version 1.2.1 is of course the most recent version right now, but zlib works just fine with 1.1.3 or later. I suggest rephrasing the "known incompatibilities" part to recommend 1.1.4 or later (there's a known security problem with 1.1.3; see http://www.gzip.org/zlib/advisory-2002-03-11.txt for details). </F>

1 0

import w/options
by Neal D. Becker 19 Oct '04

19 Oct '04

I have an application where it would be useful to set RTLD_GLOBAL on dlopen when import is used. Currently, python doesn't provide this option. Why not allow an optional argument to import that would allow this?

4 4

alternate pty module available
by J Raynor 18 Oct '04

18 Oct '04

See the following threads for some background: http://mail.python.org/pipermail/python-dev/2004-September/049085.html http://mail.python.org/pipermail/python-dev/2004-September/049092.html I've created a python module called altpty that uses code from Openssh to implement openpty() and forkpty() functions. They should work across more platforms than the functions from the standard pty module. The module can be downloaded from: http://home.mn.rr.com/raynorj/altpty.tar.gz Installation instructions are in the INSTALL file. I believe you'll need python 2.2 or better to build the module. There's still some cleanup work that could be done, but the module works, and I wanted to make it available and see if there was any real interest in it before I put too much effort into getting everything just so. Send comments or bug reports to raynorj AT mn.rr.com.

2 1

[Python-Dev] RE: [Python-checkins] python/dist/src/Lib _strptime.py, 1.35, 1.36
by Tony Meyer 18 Oct '04

18 Oct '04

> Index: _strptime.py [...] > + # For some reason when Dec 31 falls on a Monday the > week of the year is > + # off by a week; verified on both OS X and Solaris. > + elif weekday == 0 and week_of_year_start == 6 and > week_of_year >= 52: > + week_of_year += 1 Is this right? At the moment, if I ask for Monday in the 52nd week of 2008, I get the 5th of January 2009: >>> time.strptime("2008 52 1", "%Y %U %w") (2009, 1, 5, 0, 0, 0, 0, 371, -1) For reference, Monday in the previous week is the 22nd of December: >>> time.strptime("2008 51 1", "%Y %U %w") (2008, 12, 22, 0, 0, 0, 0, 357, -1) Wouldn't December 29 2008 be the right answer? This is the result without that if: >>> time.strptime("2008 52 1", "%Y %U %w") (2008, 12, 29, 0, 0, 0, 0, 364, -1) This is with WinXP SP1. =Tony Meyer

3 3

Adding support for %U and %W for strptime in b2
by Brett C. 18 Oct '04

18 Oct '04

http://www.python.org/sf/1045381 is a bug report about how strptime does nothing with %U and %W directives (week of the year with differences in whether Sunday or Monday starts the week). I just finished getting the algorithm to calculate the date given the year, either of %U or %W and the day of the week. Now the question is whether this is a bug fix or a feature implementation. If it is a bug fix I can check it right in. The only reason it wasn't in since the beginning is I didn't think of the day of the week being all you needed to figure out the date with %W or %U. If it is considered a feature, though, I will just add a note that %U and %W consume values but do nothing with them. So, what do you all think? -Brett

3 2

RE: [Python-Dev] ConfigParser patches
by Daniel 'Dang' Griffith 17 Oct '04

17 Oct '04

At 11:43 PM 10/2/2004, python-dev-request(a)python.org wrote: > Date: Sat, 2 Oct 2004 21:14:07 -0400 > From: "Raymond Hettinger" <python(a)rcn.com> > Subject: RE: [Python-Dev] ConfigParser patches > To: <guido(a)python.org> > Cc: python-dev(a)python.org > Message-ID: <000801c4a8e6$6217fc20$e841fea9@oemcomputer> > Content-Type: text/plain; charset="us-ascii" > > > I was in fact thinking of the -m proposal when I wrote that... > > The timing does suck. > > > It's so easy to define an alias or use a one-line shell script for > > invoking Python with a full pathname that I'm really not sure I like > > the -m idea, > > My understanding was that it wasn't about a full pathname to python, it > was about searching sys.path for the darned script so us poor Windows > guys don't have to change directories a million times a day (no aliases > or shebangs for us either). > > I have a big pile of batch files just to invoke timeit, texchecker, > profile, etc. It bites the big one. Am I missing something? I run python scripts all the time from the command line in Windows. Modify your PATHEXT environment variable: PATHEXT=.PY;.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH Then put your script directory into your PATH: PATH=C:\Python23\lib;%PATH% The first time you run timeit (or whatever.py), Windows will prompt you to associate the .py extension with a program. Pick your favorite python.exe. You can also do this manually before running your script. From then on, you're set: C:\>timeit -s "a=range(20)" "a.append(1)" "a.pop()" 1000000 loops, best of 3: 1.13 usec per loop --dang p.s. You can put .pyc in your path too, but then you'll get the compiled version even if the uncompiled version is newer. But you can always compile them.

2 1

RE: ConfigParser shootout, preliminary entry
by Michael Chermside 17 Oct '04

17 Oct '04

My apologies, my previous email was intended to go to python-list, not python-dev. I didn't intend to bother this list with it until it was at a point where I thought it was a contender for inclusion. Sometimes I really wish there was an "Unsend" option for email. -- Michael Chermside

1 0

RELEASED Python 2.4, beta 1
by Anthony Baxter 17 Oct '04

17 Oct '04

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On behalf of the Python development team and the Python community, I'm happy to announce the first beta of Python 2.4. Python 2.4b1 is an beta release. We'd greatly appreciate it if you could download it, kick the tires and let us know of any problems you find, but it is probably not suitable for production usage. ~ http://www.python.org/2.4 Notable changes in beta1 include a new subprocess module to provide a more convenient interface to spawning processes and a new command line argument -m modulename to run a module from the library. See either the highlights, the What's New in Python 2.4, or the detailed NEWS file -- all available from the Python 2.4 webpage. This is the first beta release of Python 2.4. Python 2.4 is now in the beta part of the release cycle - there should be few or no new features, merely bug fixes. There will be at least one more beta in a couple of weeks, followed by a release candidate and then the final release. Please log any problems you have with this release in the SourceForge bug tracker (noting that you're using 2.4b1): ~ http://sourceforge.net/bugs/?group_id=5470 Enjoy the new release, Anthony Anthony Baxter anthony(a)python.org Python Release Manager (on behalf of the entire python-dev team) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.6 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFBcoEMDt3F8mpFyBYRAhJhAKCCclb5ytuljQIGvIz9dIV2LmL0KQCgl5DX 357fPses//NSjyK+7B10YPg= =lzWC -----END PGP SIGNATURE-----

1 0

Reducing core dependencies for Python on OS X / distutils nit
by Bob Ippolito 16 Oct '04

16 Oct '04

Remove CoreServices / CoreFoundation dependencies in core http://python.org/sf/1035255 It came to my attention a couple weeks ago that Python unnecessarily links to a few frameworks on OS X, so I generated a patch that removed these dependencies. I would hope that this is uncontroversial. Presumably Jack has been really busy lately so I understand that this patch didn't make it into b1, but I would like to get it rolled into CVS by *somebody* after CVS is reopened so the final of Python 2.4 doesn't link to a bunch of frameworks for no reason, which potentially (not benchmarked) increases memory usage and startup time because the resultant Python.framework and python interpreter no longer have any explicit dependencies at all (just libSystem, which is linked in by default). -- distutils.util.get_platform() should include sys.version[:3] http://python.org/sf/1035703 On a side note, when I was playing with Python 2.4, I noticed that distutils doesn't create build directories that have the major Python version encoded in them, only the operating system information. This doesn't really make much sense, because extensions and bytecode aren't typically compatible between major versions! Nobody commented on this, but I'm willing to write a patch to create build directories with sys.version[:3] in them if it will be accepted... -bob

2 2

Wither docs for the subprocess module?
by Raymond Hettinger 16 Oct '04

16 Oct '04

Is anyone working on documenting the module for the library reference? Raymond

2 2

python-dev Summary for 2004-09-16 through 2004-09-30 [draft]
by Brett C 15 Oct '04

15 Oct '04

So we were all rather quiet in the last half of September. The whole summary fits on two sheets of 8.5x11 (normally it is over 10 and I have hit over 20 when I was summarizing *everything*). Going to send this out no earlier than Friday night so send in corrections by then. ---------------------------------- ===================== Summary Announcements ===================== Wow. This must have been the easiest summary I have ever done. Why can't they all be like this? I didn't even skip that much! ========= Summaries ========= ------------------------------------------ Assume nothing when mutability is possible ------------------------------------------ Tim Peters discovered a new way to create an infinite list thanks to generator expressions. But what really came out of this whole discussion came about when someone else came up with an example that exposed a bug in list.extend(). The first thing was that "you can't assume anything about a mutable object after potentially calling back into Python." Basically you can't assume the state of any mutable object was not changed if you execute Python code from C. While it might seem handy to store state while in a loop for instance, you can't count on things not change by the time you get control back so you just have to do it the hard way and get state all over again. Second was that you need to be careful when dealing with iterators. If you mutate an iterator while iterating you don't have a guarantee it won't explode in your face. Unless you explicitly support it, document it, and take care to protect against it then just don't assume you can mutate an iterator while using it. Contributing threads: - `A cute new way to get an infinite loop <>`__ - `More data points <>`__ ---------------------------- The less licenses the better ---------------------------- The idea of copying some code from OpenSSH_ for better pty handling was proposed. This was frowned upon since that becomes one more legal issue to keep track of. Minimizing the licenses that Python must keep track of and make sure to comply with, no matter how friendly, is a good thing. .. _OpenSSH: http://www.openssh.com/ Contributing threads: - `using openssh's pty code <>`__ ------------------------------------------------------------------------ Trying to deal with the exception hierarchy and a backwards-friendly way ------------------------------------------------------------------------- Nick Coghlan came up with the idea of having a tuple that contained all of the exceptions you normally would not want to catch in a blanket 'except' statement; KeyboardInterrupt, MemoryError, SystemExit, etc.). This tuple was proposed to live in sys.special_exceptions with the intended usage of:: try: pass # stuff... except sys.special_exceptions: raise # exceptions that you would not want to catch should keep propogating up the call chain except: pass # if you reach here the exception should not be a *huge* deal Obviously the best solution is to just clean up the exception hierarchy, but that breaks backwards-compatibility. But this idea seemed to lose steam. Contributing threads: - `Proposing a sys.special_exceptions tuple <>`__ =============== Skipped Threads =============== - Decimal, copyright and license - Planning to drop gzip compression for future releases. - built on beer? - Noam's open regex requests - Socket/Asyncore bug needs attention - open('/dev/null').read() -> MemoryError - Finding the module from PyTypeObject? - Odd compile errors for bad genexps - Running a module as a script

13 21

About the Wise installer on Windows
by Alexandre Parenteau 15 Oct '04

15 Oct '04

Hi, Again forgive me if the answer is somewhere, but I could not find it. I am looking for a way to build and make an installer out of Python on a Win x86_64 platform which is not supported yet by the Python team. My plan is to compile Python for this platform (several developers on the list told me it should be straightforward), and make an installer out of the Python binaries in order to be able to distribute the Python binaries internally. I downloaded the evaluation copy of Wise Installer System 9, but it complains that the installer file (which I assume is PCBuild\Python20.wse) is corrupted. I need to establish exactly the version of the installer to use, if we need to acquire a license for the Wise installer. - Could someone give me some pointer about the exact version of the Wise Installer to use in order to be able to replicate the same installer that the Python team is distributing on Windows? - The ISS files (Inno) do not seem up to date. Is anybody having a ISS version of the installer which works for 2.3.4? Thanks, alex.

6 10

Re: python/dist/src/Objects unicodeobject.c, 2.228, 2.229
by Fredrik Lundh 15 Oct '04

15 Oct '04

> Applied patch for [ 1047269 ] Buffer overwrite in PyUnicode_AsWideChar. > > Python 2.3.x candidate. why bother? the unicode object you're copying to holds size+1 characters, so all the code does is copying an extra NULL character... completely harmless. </F>

2 2

Patch wanted
by Scott David Daniels 15 Oct '04

15 Oct '04

Ray Hettinger's fix 1.29 to PC/pyconfig.h: Revision 1.29 - (view) (download) (annotate) - [select for diffs] CVS Tags: r24a2, r24a3 Changes since 1.28: +4 -2 lines Restore compilation on MSVC++ 6.0 =================================== /* Atleast VC 7.1 has them. If some compiler does not provide them, #ifdef appropriately .*/ #define HAVE_UINTPTR_T 1 #define HAVE_INTPTR_T 1 =================================== /* VC 7.1 has them and VC 6.0 does not. VC 6.0 has a version number of 1200. If some compiler does not provide them, modify the #if appropriately. */ #if _MSC_VER != 1200 #define HAVE_UINTPTR_T 1 #define HAVE_INTPTR_T 1 #endif =================================== Tests the wrong thing. His test presumes every non VC 7.1 PC compiler has those types defined. For non-MSC compilers, _MSC_VER is undefined, and the expression #if _MSC_VER != 1200 becomes: #if 0 != 1200 (Not what was intended, I presume). Even: #if _MSC_VER >= 1200 would be preferable (better failure, will presumably work with VC 7.2), but I believe the correct test should be: #if HAVE_STD_INT as I inexpertly explained in patch 1020042. This affects any C compilers on Windows that don't have the uintptr_t and intptr_t types (those without the newer include file stdint.h) except VC 6.0. Such compilers cannot compile any file that contains #include "Python.h" I think this needs to be fixed before 2.4 goes out. -- -- Scott David Daniels Scott.Daniels(a)Acm.Org

2 3

Weekly Python Patch/Bug Summary
by Kurt B. Kaiser 15 Oct '04

15 Oct '04

Patch / Bug Summary ___________________ Patches : 240 open ( -1) / 2655 closed (+15) / 2895 total (+14) Bugs : 766 open ( +0) / 4514 closed (+22) / 5280 total (+22) RFE : 155 open ( +1) / 131 closed ( +0) / 286 total ( +1) New / Reopened Patches ______________________ True/False instead of 1/0 in libstdtypes.tex (2004-10-06) CLOSED http://python.org/sf/1041364 opened by Gerrit Holl bbetter document popenX 'cmd' argument (2004-10-07) CLOSED http://python.org/sf/1042705 opened by Jeff Epler broken TeX markup (2004-10-08) CLOSED http://python.org/sf/1042969 opened by George Yoshida Decimal.py enhancements (2004-10-08) CLOSED http://python.org/sf/1043218 opened by Facundo Batista Enhancing '-m' to support packages (2004-10-09) http://python.org/sf/1043356 opened by Nick Coghlan External storage protocol for large email messages (2004-10-09) http://python.org/sf/1043706 opened by Barry A. Warsaw tarfile: add extractall() method (2004-10-10) http://python.org/sf/1043890 opened by Lars Gustäbel tarfile.py: patch for bug #1017553 (2004-10-10) http://python.org/sf/1043972 opened by Lars Gustäbel Bug #1011380 alternate fix - locking is noop without threads (2004-10-11) CLOSED http://python.org/sf/1044089 opened by Nick Coghlan Add FreeBSD* to configure.in for --enable-shared (2004-10-11) http://python.org/sf/1044395 opened by James William Pye improved configure.in output (2004-10-12) http://python.org/sf/1045620 opened by Wummel Remove unncessary NLST from ftp transfers (2004-10-12) http://python.org/sf/1045783 opened by Chris Cogdon Consistent retrieval of Python version. (2004-10-14) http://python.org/sf/1046831 opened by uiltje Buffer overwrite in PyUnicode_AsWideChar (2004-10-14) http://python.org/sf/1047269 opened by Thomas Heller Patches Closed ______________ -m option to run a module as a script (2004-09-27) http://python.org/sf/1035498 closed by rhettinger True/False instead of 1/0 in libstdtypes.tex (2004-10-06) http://python.org/sf/1041364 closed by rhettinger pdb enhancements and bug fixes (2004-02-12) http://python.org/sf/896011 closed by jlgijsbers bbetter document popenX 'cmd' argument (2004-10-08) http://python.org/sf/1042705 closed by jlgijsbers popen2.popen2() 'cmd' list syntax docstring (2004-02-01) http://python.org/sf/888380 closed by jlgijsbers broken TeX markup (2004-10-08) http://python.org/sf/1042969 closed by jlgijsbers Decimal.py enhancements (2004-10-08) http://python.org/sf/1043218 closed by rhettinger replacing rfc822.formatdate (2004-08-09) http://python.org/sf/1005857 closed by anthonybaxter email/__init__.py: Parse headersonly (2004-03-20) http://python.org/sf/920037 closed by bwarsaw Bad behavior of email.Charset.Charset when locale is tr_TR (2003-12-29) http://python.org/sf/866982 closed by bwarsaw mailbox / fromline matching (2002-02-22) http://python.org/sf/521478 closed by bwarsaw Bug #1011380 alternate fix - locking is noop without threads (2004-10-10) http://python.org/sf/1044089 closed by tim_one Logger file objects (2004-08-02) http://python.org/sf/1001864 closed by vsajip Signal fixes for BSD systems (2004-06-18) http://python.org/sf/975056 closed by anthonybaxter Allow ldshared to be overridden by environment var (2004-07-01) http://python.org/sf/983206 closed by anthonybaxter New / Reopened Bugs ___________________ copy.deepcopy exceptions.TypeError: 'NoneType' object ... (2004-10-06) CLOSED http://python.org/sf/1041325 opened by Arthur Lutz SimpleXMLRPCServer example is broken (2004-10-06) CLOSED http://python.org/sf/1041501 opened by Kent Johnson Thread management corrupts heap (2004-10-06) CLOSED http://python.org/sf/1041645 opened by benson margulies Patch (2004-10-07) CLOSED http://python.org/sf/1042179 opened by Michiel de Hoon Lib/compiler chokes on certain genexps (2004-10-07) CLOSED http://python.org/sf/1042238 opened by Michael Hudson mimetypes.guess_extension('text/plain') = '.ksh' ??? (2004-10-08) http://python.org/sf/1043134 opened by Robert Kiendl Uninterruptable loop (2004-10-09) http://python.org/sf/1043446 opened by Jp Calderone compile error with stlport (2004-10-10) http://python.org/sf/1044299 opened by Debug Can't raise "C API version mismatch" warning (2004-10-11) http://python.org/sf/1044382 opened by Frank Sonnenburg docs for Py_UNICODE are wrong (2004-10-11) http://python.org/sf/1044479 opened by Thomas Heller Unable to install Docs - Linux (2004-10-11) http://python.org/sf/1044872 opened by Colin J. Williams strptime doesn't work with %U (2004-10-12) http://python.org/sf/1045381 opened by Sebastien JUST Problems with os.system and ulimit -f (2004-10-12) http://python.org/sf/1045509 opened by Tomasz Kowaltowski test_subprocess vs Windows (2004-10-12) CLOSED http://python.org/sf/1045748 opened by Tim Peters warning '"_POSIX_C_SOURCE" redefined' when include Python.h (2004-10-13) http://python.org/sf/1045893 opened by Eugene Sizikov urllib2: better import ftplib and gopherlib etc late (2004-10-13) http://python.org/sf/1046077 opened by Robert Kiendl HTMLParser fix to accept mailformed tag attributes (2004-10-13) http://python.org/sf/1046092 opened by Vsevolod Novikov improving distutils swig support (2004-10-14) CLOSED http://python.org/sf/1046404 opened by Anthony Baxter improving distutils swig support (2) (2004-10-14) CLOSED http://python.org/sf/1046644 opened by Lars Immisch difflib.HtmlDiff doc errors (EASY) (2004-10-13) CLOSED http://python.org/sf/1046690 opened by Dan Gass Error in the given example (2004-10-14) http://python.org/sf/1046800 opened by Vishnu httplib index out of range (2004-10-14) CLOSED http://python.org/sf/1046855 opened by Grzegorz Makarewicz improving distutils swig support - documentation (2004-10-14) http://python.org/sf/1046945 opened by Lars Immisch cgitb failures (2004-10-14) http://python.org/sf/1047397 opened by Robin Becker Bugs Closed ___________ copy.deepcopy exceptions.TypeError: 'NoneType' object ... (2004-10-06) http://python.org/sf/1041325 closed by arthur_lutz SimpleXMLRPCServer example is broken (2004-10-06) http://python.org/sf/1041501 closed by akuchling Thread management corrupts heap (2004-10-06) http://python.org/sf/1041645 closed by tim_one PyOS_InputHook broken (2004-09-19) http://python.org/sf/1030629 closed by mwh Patch (2004-10-07) http://python.org/sf/1042179 closed by mwh Lib/compiler chokes on certain genexps (2004-10-07) http://python.org/sf/1042238 closed by mwh Enhancements to pdb.py when invoked as script (2003-06-09) http://python.org/sf/751124 closed by jlgijsbers shutils.rmtree() uses excessive amounts of memory (2004-09-09) http://python.org/sf/1025127 closed by jlgijsbers Conflicting descriptions of application order of decorators (2004-09-21) http://python.org/sf/1031897 closed by akuchling x, y in curses window object documentation (2004-09-04) http://python.org/sf/1022311 closed by akuchling Confusing description of strict option for email.Parser (2004-09-22) http://python.org/sf/1032960 closed by bwarsaw email package uses \n to rebuild content of a message (2004-06-01) http://python.org/sf/964433 closed by bwarsaw smtpd.py docstring says wrong default class (2004-08-16) http://python.org/sf/1010102 closed by bwarsaw email: mishandles Content-Transfer-Enc. for text/* messages (2004-05-09) http://python.org/sf/950747 closed by bwarsaw Email message croaks the new email pkg parser (2004-09-19) http://python.org/sf/1030941 closed by bwarsaw Make GIL acquire/release functions noops if PyEval_Init (2004-08-18) http://python.org/sf/1011380 closed by tim_one test_subprocess vs Windows (2004-10-12) http://python.org/sf/1045748 closed by tim_one BSD restartable signals not correctly disabled (2004-06-09) http://python.org/sf/969574 closed by anthonybaxter Solaris likes sys/loadavg.h (2004-06-22) http://python.org/sf/977343 closed by anthonybaxter improving distutils swig support (2004-10-14) http://python.org/sf/1046404 closed by anthonybaxter improving distutils swig support (2) (2004-10-14) http://python.org/sf/1046644 closed by anthonybaxter difflib.HtmlDiff doc errors (EASY) (2004-10-13) http://python.org/sf/1046690 closed by tim_one httplib index out of range (2004-10-14) http://python.org/sf/1046855 closed by rhettinger

1 0

PyPy Vilnius Sprint 15-23 nov 2004
by hpk＠trillke.net 14 Oct '04

14 Oct '04

Hi Pythonistas and interested developers, PyPy, the python-in-python implementation, is steadily moving on. The next coding sprint will take place in Vilnius, Lithunia, from 15th to 23rd of November, 2004 and is organized by the nice Programmers of Vilnius (POV) company. See http://codespeak.net/pypy/index.cgi?doc for more in-depth information about PyPy. Again, we will be heading towards a first generated C version of our already pretty compliant Python interpreter and types implementation. Last time, before the EuroPython 2004 conference, we actually had a similar goal (a PyPy/C-version) but discovered we had to largely refactor the basic model for attribute accesses. We are now closely mirroring the marvelous "descriptor"-mechanism of CPython. If you are interested to participate in our fun and somewhat mind-altering python sprint event then please subscribe at http://codespeak.net/moin/pypy/moin.cgi/VilniusSprintAttendants and look around for more information. You'll find that most of the core PyPy developers are already determined to come. There are also many areas that need attention so that we should have tasks suited for different levels of expertise. At http://codespeak.net/moin/pypy/moin.cgi/VilniusSprintTasks you'll find our sprint planning task list which will probably grow in the next weeks. Note that our EU funding efforts are at the final stage now. In the next weeks, quite likely before the Vilnius sprint, we _hope_ to get a "go!" from the european commission. One side effect would be that coders - probably restricted to european citizens - may generally apply for getting travel and accomodation costs refunded for PyPy sprints. This would reduce the barrier of entry to the question if you like to spend your time with a pypy sprint. However, we probably need some time to work out the details after when we get more information from the EU. If you have any questions don't hesitate to contact pypy-sprint(a)codespeak.net or one of us personally. cheers & a bientot, Holger Krekel, Armin Rigo

1 0

Cyclic GC issues
by Jason Evans 14 Oct '04

14 Oct '04

Since the spring of 2003, I have been developing Crux, which is a computer program for phylogenetic inferencing (bioinformatics) research. In March of 2004, I switched Crux to using Python from having used a different embeddable interpreter. For the most part, I have been very happy with Python, but Python's garbage collector has been a major source of frustration. Below, I describe my trials and tribulations with Python's GC. I also offer some suggestions for changes to Python; if any of the proposed changes receive favorable feedback, I am willing to help develop patches to Python. Naturally, if I am somehow abusing Python, and there are better ways to do things, I'd be happy to hear how to improve Crux. The important aspect of Crux is that it deals with trees. These trees are unrooted (there is no up or down), and multifurcating (nodes can have an arbitrary number of neighboring nodes). Thus, the trees are self-referential, and without the cyclic GC capabilities of Python, there would be little hope of making these trees integrate well with Python. Following is a diagram that illustrates the linkage between various objects for a simple tree. Crux exposes all of the components of the trees as Python objects. All lines in the diagram represent bi-directional references (except for the T-->N line). Every object refers to the tree object; those lines are left out in order to reduce clutter. T: Tree N N N: Node \ / E: Edge R R R: Ring \ / E E \ / R---------R / \ / \ / \ / \ | \ / | | \ / | | T--->N | | | | \ | / \ | / \----R----/ | E | R | N At the C (not Python object) level, the R-E-R construct is actually a set of structures that are allocated/deallocated as a single unit. Edges are *always* connected to two rings, so there's no point in allocating these separately. Also, lone ring objects actually are rings with one member; they refer to themselves (prev and next pointers). That should be enough information to understand the problems I encountered. 1) I originally had lone rings refer to themselves (ob_refcnt started out at 3; 2 self-references and one reference held by the associated edge). This didn't work. It appears that the cyclic GC does not actually calculate the number of live references to objects (references that are reachable by traversing all objects accessible from the root set); instead it assumes that if tp_clear() doesn't drop the reference count to a number that equals the number of references from live objects, there must still be references from live objects. Unfortunately, visit()ing self does not work, so there is no way to convince Python that all references are from unreachable objects. Working around this in Crux requires a lot of extra reference counting complexity, because there are three different cases for reference counts, depending on how many members there are in a ring (1, 2, or 3+ members). 2) This issue is really just a different manifestation of issue (1). At the C (not Python object) level, each node only actually stores a pointer to a single member of the associated ring. Given a single ring member, it is possible to traverse the ring and reach all other ring members. As mentioned in issue (1), the cyclic GC expects tp_traverse() to call visit() once for each reference held. It is not enough for a node to visit() one ring member; it must visit() all ring members, in order for the GC to count how many references are from unreachable objects, versus reachable from the root set. In summary, issues (1) and (2) are due to how the cyclic GC does the "marking" phase of mark/sweep GC. My expectation of mark/sweep GC is that it should be sufficient to assure that all objects reachable from the root set are visit()ed at least once; it should not be important how many times each unreachable object is visit()ed. I don't have a deep enough understanding of the Python interpreter to give a detailed suggestion for improvement. I suspect that determining the root set is not an easy operation; if this is the case, then I think we're stuck with the current design. If determining the root set *is* easy (or even possible), then I would suggest using a standard mark/sweep collector, where unreachable objects are scheduled for destruction. tp_traverse(), tp_clear(), and tp_dealloc() would retain the same structure; the only difference would be the logic that determines which objects can be destroyed. 3) A strange thing can happen when tp_clear() is called. Suppose that an edge object is being cleared, and it drops its references to the associated rings. If ob_refcnt of one of the rings drops to 0 as a result, Python will tp_dealloc() the ring *right* *now*, without ever calling tp_clear() on the ring. That means that I have to keep track of whether tp_clear() has been called on objects, if it is at all important that tp_clear() be called, so that I can manually do so in tp_dealloc(), if necessary. It is in my opinion reasonable to have cleanup code in tp_clear(), with the assumption that it will be called precisely once, but Python makes me do extra work to make sure that this happens. This should be pretty easy to change. A single bit per object is needed to keep track of whether tp_clear() has been called. I think this only needs to be done for object types that support cyclic GC. 4) There is no way to control the order in which objects are tp_dealloc()ed. This is a problem for the R-E-R construct, since at a low level, these objects are always linked together. What I would like to do is defer tp_dealloc() on the edge until after both rings have been deleted. Instead, I am forced to use a reference-counted deletion function. Not calling self->ob_type->tp_free() on the edge in tp_dealloc() until later is not a reasonable option, because this defers deletion of the edge until a later round of garbage collection. This could be addressed in the Python interpreter by paying heed to the return value of tp_dealloc(). If the return value is non-zero, move the object to the end of the list of objects to be destroyed, so that destruction is tried later. This allows the module to impose its own destruction ordering. I look forward to feedback. Thank you, Jason Evans

7 12

[Python-Dev] Re: python/dist/src/Doc/whatsnew whatsnew24.tex, 1.108, 1.109
by Fredrik Lundh 14 Oct '04

14 Oct '04

> +\begin{seealso} > +\seepep{324}{subprocess - New process module}{Written and implemented by Peter Astrand, with > assistance from Fredrik Lundh and others.} > +\end{seealso} I don't know how to add ISO-8859-1 characters to a Latex document, but in HTML notation, Peter's last name should be Åstrand. cheers /F

4 3

Python on Windows 64bits?
by Alexandre Parenteau 14 Oct '04

14 Oct '04

Sorry if I'm asking something that has been answered before, but I could not find a clear answer anywhere. Is there a project page for Windows 64bits? Is it already on CVS? My company tries to assess if we can plan on using python on Windows 64 bits. We want to do it natively (i.e. no 32b emulation), because we have some python extensions we want to use in the process. Thanks! alex.

4 6

Inno-Setup for Python 2.3.4
by A.B., Khalid 13 Oct '04

13 Oct '04

[Alexandre Parenteau] >... >- The ISS files (Inno) do not seem up to date.Is anybody having a ISS >version of the installer which works for 2.3.4? Hello Alex, I have been working on and off on an Inno-Setup installer for the MinGW compiled Python I've been working on. The reason being I have Win98 so the Msi.py approach does not work. The script is still not finished, nor is it identical to setup script found in the Python 2.3.4 sources. Moreover, it does not make any registry entries yet, which I am thinking and hoping of doing either through a Python script (which runs after install is finished and before uninstall is started) or an external DLL so as to be able to revert changes. But even in its current unfinished form, the script works for me in that it gathers all of the needed Python 2.3.4 files into one package and offers to install Python and Python's extensions, and uninstalls the files as needed. All the regrtest tests pass when Python 2.3.4 is installed from that script. One can run Python's scripts from the DOS Shell and if file association is urgently needed then hints from the Python.iss found in the original Python source distribution can be of immediate help. Get it here (3.49 KB): http://jove.prohosting.com/iwave/ipython/Inno-Python234.zip Regards Khalid PS: I have pymingw.py which is enclosed in the zip file where I have python23.dll and friends. It will be picked up by the script from there and placed in site-packages for use when uninstalling Python. _________________________________________________________________ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.com/

1 0

2.4b1 release, this Friday (AU time) -- imminent TRUNK FREEZING
by Anthony Baxter 13 Oct '04

13 Oct '04

Ok, I'm going to be cutting the Python release Friday (Australian time). This means that I'm declaring the trunk frozen for the release in about 24 hours. Lets say at 18:00 hours UTC on Thursday 14th (that's 14:00 on the US east coast, if my memory is correct, and about 04:00 where here in Australia). No-one except for the usual team (Fred, Martin, myself) should check into the trunk after that. As per usual, I'll probably keep the trunk frozen for a time after the release, in case of brown-paper-bag issues. Note that this is the last chance you have to get those behaviour- changing bugs^Wfixes introduced, so get to it! After b1, it will become _much_ harder to get new fixes introduced. And if there's a pending bug or patch that needs some lovin', add a comment to it or mail me - I think I've responded to, or fixed, all the ones I've been told about, but if I missed one, tell me now. Anthony -- Anthony Baxter <anthony(a)interlink.com.au> It's never too late to have a happy childhood.

2 1

Re: [Python-checkins] python/dist/src/Lib subprocess.py, NONE, 1.1
by Thomas Heller 13 Oct '04

13 Oct '04

effbot(a)users.sourceforge.net writes: > Update of /cvsroot/python/python/dist/src/Lib > In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv24287/Lib > > Added Files: > subprocess.py > Log Message: > Added Peter Astrand's subprocess module. I suggest to remove the imports of the pywin32 stuff, it's not needed and confuses tools like freeze or py2exe. if mswindows: import threading import msvcrt try: from _subprocess import * class STARTUPINFO: dwFlags = 0 hStdInput = None hStdOutput = None hStdError = None class pywintypes: error = IOError except ImportError: import pywintypes from win32api import GetStdHandle, STD_INPUT_HANDLE, \ STD_OUTPUT_HANDLE, STD_ERROR_HANDLE from win32api import GetCurrentProcess, DuplicateHandle, \ GetModuleFileName, GetVersion from win32con import DUPLICATE_SAME_ACCESS from win32pipe import CreatePipe from win32process import CreateProcess, STARTUPINFO, \ GetExitCodeProcess, STARTF_USESTDHANDLES, \ CREATE_NEW_CONSOLE from win32event import WaitForSingleObject, INFINITE, WAIT_OBJECT_0

7 14

Re: [Python-checkins] python/dist/src/Lib/distutils/command __init__.py, 1.19, 1.20 install.py, 1.70, 1.71 checkdep.py, 1.1, NONE
by Fred L. Drake, Jr. 13 Oct '04

13 Oct '04

On Wednesday 13 October 2004 08:35 am, Anthony Baxter wrote: > Backing out the basic dependency checking (from pycon sprint). > This support was only a first cut, and doesn't deserve to be in > a released version (where we have to support it in an ongoing > manner) Thanks, Anthony! -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>

1 0

test_cwd of test_subprocess.py fails on Win98
by Khalid A. B. 13 Oct '04

13 Oct '04

Hello. The test fails because it seems to be comparing "C:\WINDOWS\TEMP" with "C:\WINDOWS\Temp" and even though they are the same thing in Win98, they are of cource not equal. Details follow: $ python -i Python 2.4a3 (#56, Oct 13 2004, 02:29:05) [GCC 3.4.1 (mingw special)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>>import os, sys, subprocess >>>tmpdir = os.getenv("TEMP", "/tmp") >>>tmpdir = os.path.realpath(tmpdir) >>>tmpdir 'C:\\WINDOWS\\TEMP' >>>p = subprocess.Popen([sys.executable, "-c", ... 'import sys,os;' \ ... 'sys.stdout.write(os.getcwd())'], ... stdout=subprocess.PIPE, ... cwd=tmpdir) >>>subProcessTmpDir = p.stdout.read() >>>subProcessTmpDir 'C:\\WINDOWS\\Temp' >>>tmpdir == subProcessTmpDir False >>> Although I don't know if this will work in other versions of Windows, here is what fixed it for me: Index: python/dist/src/Lib/test/test_subprocess.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/test/test_subprocess.py,v retrieving revision 1.4 diff -u -d -r1.4 test_subprocess.py --- python/dist/src/Lib/test/test_subprocess.py 12 Oct 2004 22:29:54 -0000 1.4 +++ python/dist/src/Lib/test/test_subprocess.py 13 Oct 2004 02:27:13 -0000 @@ -192,9 +192,16 @@ def test_cwd(self): tmpdir = os.getenv("TEMP", "/tmp") tmpdir = os.path.realpath(tmpdir) - p = subprocess.Popen([sys.executable, "-c", - 'import sys,os;' \ - 'sys.stdout.write(os.getcwd())'], + commandStr = [sys.executable, "-c"] + if mswindows: + tmpdir = tmpdir.upper() + commandStr.append('import sys,os;' \ + 'sys.stdout.write(os.getcwd().upper())') + else: + commandStr.append('import sys,os;' \ + 'sys.stdout.write(os.getcwd())') + + p = subprocess.Popen(commandStr, stdout=subprocess.PIPE, cwd=tmpdir) self.assertEqual(p.stdout.read(), tmpdir) _________________________________________________________________ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.com/

2 1

test_cwd of test_subprocess.py fails on Win98
by Khalid A. B. 13 Oct '04

13 Oct '04

Hello On Win98 test_cwd of test_subprocess.py tries to assert that "C:\WINDOWS\TEMP" is equal to "C:\WINDOWS\Temp" but since "TEMP" != "Temp" it fails, even when they are the same directory. Details follow: $ python -i Python 2.4a3 (#56, Oct 13 2004, 02:29:05) [GCC 3.4.1 (mingw special)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>>import os, sys, subprocess >>>tmpdir = os.getenv("TEMP", "/tmp") >>>tmpdir = os.path.realpath(tmpdir) >>>tmpdir 'C:\\WINDOWS\\TEMP' >>>p = subprocess.Popen([sys.executable, "-c", ... 'import sys,os;' \ ... 'sys.stdout.write(os.getcwd())'], ... stdout=subprocess.PIPE, ... cwd=tmpdir) >>>subProcessTmpDir = p.stdout.read() >>>subProcessTmpDir 'C:\\WINDOWS\\Temp' >>>tmpdir == subProcessTmpDir False >>> Here is what fixed it for me: Index: python/dist/src/Lib/test/test_subprocess.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/test/test_subprocess.py,v retrieving revision 1.4 diff -u -d -r1.4 test_subprocess.py --- python/dist/src/Lib/test/test_subprocess.py 12 Oct 2004 22:29:54 -0000 1.4 +++ python/dist/src/Lib/test/test_subprocess.py 13 Oct 2004 02:27:13 -0000 @@ -192,9 +192,16 @@ def test_cwd(self): tmpdir = os.getenv("TEMP", "/tmp") tmpdir = os.path.realpath(tmpdir) - p = subprocess.Popen([sys.executable, "-c", - 'import sys,os;' \ - 'sys.stdout.write(os.getcwd())'], + commandStr = [sys.executable, "-c"] + if mswindows: + tmpdir = tmpdir.upper() + commandStr.append('import sys,os;' \ + 'sys.stdout.write(os.getcwd().upper())') + else: + commandStr.append('import sys,os;' \ + 'sys.stdout.write(os.getcwd())') + + p = subprocess.Popen(commandStr, stdout=subprocess.PIPE, cwd=tmpdir) self.assertEqual(p.stdout.read(), tmpdir) _________________________________________________________________ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.com/

1 0

test_pep263.py is "-kb" in CVS
by Michael Hudson 12 Oct '04

12 Oct '04

As subject. Is this deliberate? It breaks test_compiler, though that's possibly a bug because test_compiler should be using universal newlines modes when opening the file, but still... Cheers, mwh -- The word "Fascism" has now no meaning except in so far as it signifies 'something not desirable'. -- George Orwell in "Politics and the English Language"

6 8

subprocess module test failure when --with-pydebug
by Barry Warsaw 12 Oct '04

12 Oct '04

The new subprocess module test is not compatible with a Python built --with-pydebug because the [xxxx refs] output breaks the comparison. I've recently been thinking that it would be nice if that [xxxx refs] output could be suppressed with an envar or a command line switch. FWIW, I generally build our embedding product with a --with-pydebug built Python and that output breaks our unit tests too. I work around this by suppressing the stderr comparison in our testing framework. Once we decide to do it, and what to wrap that call in, it's likely a one or two line change to pythonrun.c. Does one of the existing envars make sense to use? Barring any change here, we need to at least make sure the subprocess test doesn't fail for a debug build. -Barry

1 0

Re: [Python-Dev] test_pep263.py is "-kb" in CVS
by Guido van Rossum 12 Oct '04

12 Oct '04

In any case, the necessity of -kb for this file (if it is still necessary) should be documented in a comment inside the file, to prevent a repeat of this discussion three years from now. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

1 0

RE: [Python-Dev] scropion
by Michael Chermside 12 Oct '04

12 Oct '04

> [Paul introduces himself] > My question is for performance: isn't binary code faster than p code? > So If I can vote, I would vote for c or c++ in the numerics functions and > data transfer areas. > even assembler. The quicker the better. Hello, and welcome. Thanks for your input on what kinds of things you value in Python. (The Python developers, who use the python-dev list) DO pay attention to such opinion (although don't expect sudden changes... you are one in a sea of users). You are not alone in caring about performance -- it's been a popular topic lately. I'll comment briefly on your specific suggestions. Binary code is faster than "p code" usually, but in some cases the "p code" is more ammenable to automated optimization. That is why certain kinds of code can run FASTER in java than in C. Python code too will often run faster than the equivalent C code... not because of a powerful just-in-time optimizer, but because the Python code is written using better algorithms and data structures than one would bother with in C. But you specifically mention numerics functions and data transfer... two areas where well-written C code is likely to be much better performing. We certainly share the goal of better performance (so long as it doesn't come at the cost of other things like usability and maintainability). If you know of some specific areas where we could improve, please submit a patch (through SourceForge's patch manager) -- we'd love to have the help! Finally, you mention the possibility of dropping into assembler. Unfortunately, this is sure to be rejected because it conflicts with an important design principle -- Python is designed to be VERY portable, and should compile on just about any system with a standards-compliant C compiler. -- Michael Chermside

1 0

Re: [Python-Dev] Re: Re: subprocess - updated popen5 module
by Neil Hodgson 11 Oct '04

11 Oct '04

Fredrik Lundh: > Thomas Heller wrote: > > > How many _xxxx.pyd windows specific modules do we need? > > as many as it takes to make Python an excellent tool. it's not like one > more entry in the makefile, and 20k in the binary distribution will cause > problem for anyone. Requiring windows interface modules to be written in C rather than Python reduces the number of Python users that can add, modify and fix these modules. It also increases the amount of effort involved in working on these modules. Neil

6 7

compiler bug #1042238 is blocking the 2.4b1 release
by Anthony Baxter 11 Oct '04

11 Oct '04

Anyone with knowledge of Lib/compiler is urged to examine www.python.org/sf/1042238. It's breaking on some genexprs, in particular in Lib/_strptime.py The failing code, distilled down, is: import compiler a="""def foo(*x): return x foo((a for b in [[1,2,3],[4,5,6]] for a in b),'Z')""" compiler.compile(a,'<string>', 'exec') It's the combination of the genexps and the function call with another argument. This bug is bad enough (IMHO) to block 2.4b1. Please, if you know about this, have a look at the failure. Anthony -- Anthony Baxter <anthony(a)interlink.com.au> It's never too late to have a happy childhood.

2 1

Patch to make GIL API calls noops in single threaded applications
by Gustavo J. A. M. Carneiro 10 Oct '04

10 Oct '04

Can someone review my patch[1]? It makes GIL API calls noops in single threaded applications, i.e. if PyEval_InitThreads hasn't been called. It's a very simple patch, but it makes a lot of difference if you want to make a Python extension module thread-aware without incurring an extra (unnecessary) performance penalty in single-threaded programs. Without this in Python, each extension library has to keep track of thread state (enabled or not), and one extension library doesn't work well with another without some explicit synchronization between the two. Please, we should fix this before Python 2.4 final. Regards. [1] https://sourceforge.net/tracker/? func=detail&aid=1011380&group_id=5470&atid=105470 -- Gustavo J. A. M. Carneiro <gjc(a)inescporto.pt> <gustavo(a)users.sourceforge.net> The universe is always one step beyond logic

1 0

assert failure on obmalloc
by Jeremy Hylton 10 Oct '04

10 Oct '04

Failure running the test suite today with -u compiler enabled on Windows XP. test_logging Assertion failed: bp != NULL, file \code\python\dist\src\Objects\obmalloc.c, line 604 The debugger says the error is here: msvcr71d.dll!_assert(const char * expr=0x1e22bcc0, const char * filename=0x1e22bc94, unsigned int lineno=604) Line 306 C python24_d.dll!PyObject_Malloc(unsigned int nbytes=100) Line 604 + 0x1b C python24_d.dll!_PyObject_DebugMalloc(unsigned int nbytes=84) Line 1014 + 0x9 C python24_d.dll!PyThreadState_New(_is * interp=0x00951028) Line 136 + 0x7 C python24_d.dll!PyGILState_Ensure() Line 430 + 0xc C python24_d.dll!t_bootstrap(void * boot_raw=0x02801d48) Line 431 + 0x5 C python24_d.dll!bootstrap(void * call=0x04f0d264) Line 166 + 0x7 C msvcr71d.dll!_threadstart(void * ptd=0x026a2320) Line 196 + 0xd C I've been seeing this sort of error on-and-off for at least a year with my Python 2.3 install. It's the usual reason my spambayes popproxy dies. I can't recell seeing it before on Windows or while running the test suite. Jeremy

3 3

Guidelines for Logging Usage
by Michael P. Dubner 10 Oct '04

10 Oct '04

Hello, I beg your pardon in advance for my English, which isn't my native language. I'm posting PEProposal for discussion. Last part is reasoning for posting it in this hot time. I'm ready to implement it all by myself... :-) -- Best regards, Michael Dubner (dubnerm(a)mindless.com) Brainbench MVP/HTML+JavaScript (http://www.brainbench.com) PS: Sorry for my English PEP: XXX Title: Guidelines for Logging Usage Version: $Revision: $ Last-Modified: $Date: $ Author: Michael P. Dubner <dubnerm(a)mindless.com> Status: Draft Type: Standards Track Content-Type: text/plain Created: 02-Oct-2004 Python-Version: 2.5 Post-History: Abstract This PEP defines guidelines for using logging system (PEP 282 [1]) in standard library. Implementing this PEP will simplify development of daemon applications. As a downside this PEP requires to slightly modify (however in backportable way) large number of standard modules. After implementing this PEP one can use following filtering scheme:: logging.getLogger('stdlib.BaseHTTPServer').setLevel(logging.FATAL) Rationale There are couple of situations when output to stdout or stderr is very incomfortable. - Daemon application where framework doesn't allows to redirect standard output to some file, but assumes use of some other way of logging. Examples are syslog under *nix'es and EventLog under WinNT+. - GUI application which want to output every new log entry in separate popup window (i.e. fading OSD). Also sometimes application want to filter output enties based on it's source or severity. This requirement can't be implemented using simple redirection. At last sometimes output need to be marked with time of event, which can be acqured with logging system with ease. Proposal Every module usable for daemon and GUI applications should be rewritten to use logging system instead of 'print' or 'sys.stdout.write'. There should be code like this included in the beginning of every modified module:: import logging _log = logging.getLogger('stdlib.<module-name>') Prefix of 'stdlib.' must be used by all modules included in standard library distributed along with Python, and only by such modules (unverifiable). Using of "_log" is intentional as we don't want to auto-export it. For modules that using log only in one class logger can be created inside class definition as following:: class XXX: __log = logging.getLogger('stdlib.<module-name>') Then this class can create access methods to log to this private logger. So print's and "sys.std{out|err}.write" should be replaced with "_log.{debug|info}" and "traceback.print_exception" with "_log.exception" or sometimes "_log.debug('...',exc_info=1)". Module List Here is (possibly incomplete) list of modules to be reworked: - asyncore (dispatcher.log, dispatcher.log_info) - BaseHTTPServer (BaseHTTPRequestHandler.log_request, BaseHTTPRequestHandler.log_error, BaseHTTPRequestHandler.log_message) - cgi (possibly - is cgi.log used by somebody?) - ftplib (if FTP.debugging) - gopherlib (get_directory) - httplib (HTTPResponse, HTTPConnection) - ihooks (_Verbose) - imaplib (IMAP4._mesg) - mhlib (MH.error) - nntplib (NNTP) - pipes (Template.makepipeline) - pkgutil (extend_path) - platform (_syscmd_ver) - poplib (if POP3._debugging) - profile (if Profile.verbose) - robotparser (_debug) - smtplib (if SGMLParser.verbose) - shlex (if shlex.debug) - smtpd (SMTPChannel/PureProxy where print >> DEBUGSTREAM) - smtplib (if SMTP.debuglevel) - SocketServer (BaseServer.handle_error) - telnetlib (if Telnet.debuglevel) - threading? (_Verbose._note, Thread.__bootstrap) - timeit (Timer.print_exc) - trace - uu (decode) Additionaly there are couple of modules with commented debug output or modules where debug output should be added. For example: - urllib At last possibly some modules should be extended to provide more debug information. Doubtful Modules Here should be placed modules that community will propose for addition to module list and modules that community say should be removed from module list. - tabnanny (check) Guidelines for Logging Usage Also we can provide some recommendation to authors of library modules so they all follow same format of naming loggers. I propose that non-standard library modules should use logger named after their full names, so module "spam" in sub-package "junk" of package "dummy" will be named "dummy.junk.spam" and, of cause, "__init__" module of same package will have logger "dummy.junk". Implementation Schedule Proposal As one can see whole bunch of changes required to fullfil this proposal is rather large. I propose to delay these changes until after 2.4 release, and change only modules critical for server applications: - BaseHTTPServer - ftplib - httplib - poplib - smtpd - smtplib - SocketServer This can be done rather fast and reliable. References [1] PEP 282, A Logging System, Vinay Sajip, Trent Mick http://www.python.org/peps/pep-0282.html Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:

7 8

Toward Python's future article
by daniel narf 08 Oct '04

08 Oct '04

Hi i am sure most of you have read the article of Andrew Kuchling about focusing more in the standart library than language newFeatures/tweaking and probably i as many others would like to know what the python hackers/developers think about this proposal.Maybe this post is out of place but oh well.. i am personaly very interested in improving the stdlib which is very messy in my opinion right now. the article(several comments): http://www.amk.ca/diary/archives/cat_python.html#003382

6 7

HAVE_USABLE_WCHAR_T
by Thomas Heller 08 Oct '04

08 Oct '04

The Include/unicodeobject.h file says (line 103): /* If the compiler provides a wchar_t type we try to support it through the interface functions PyUnicode_FromWideChar() and PyUnicode_AsWideChar(). */ This isn't true - grepping the CVS sources for this symbol shows that it is used in these ways: - When defined together with the WANT_WCTYPE_FUNCTIONS symbol, the compiler's wctype.h functions are used instead of the ones supplied with Python. Include/unicodeobject.h, line 294. - When defined together with MS_WINDOWS, it makes available mbcs_enocde and mbcs_decode functions (in Modules/_codecsmodule.c), plus the PyUnicode_DecodeMBCS and PyUnicode_AsMBCSString functions in Objects/unicodeobject.c. - Contrary to the comment at the top of this message, the PyUnicode_FromWideChar and PyUnicode_AsWideChar functions are compiled when HAVE_WCHAR_H is defined. The HAVE_USABLE_WCHAR_T symbol is only used to determine whether memcpy is used, or the unicode characters are copied one by one. - Finally, again when defined together with MS_WINDOWS, it sets the filesystem encoding to mbcs. So, it seems that the HAVE_USABLE_WCHAR_T symbol doesn't play any role for the extension programmer *at all*. The preprocessor flag that plays a role for extensions seem to be HAVE_WCHAR_H since they mark whether the PyUnicode_FromWideChar and PyUnicode_AsWideChar are available or not. This has caused me quite some confusion, and so I suggest the comment above in the Include/unicodeobject.h file should be fixed. Finally, the docs also seem to get it wrong (although I haven't followed that in detail). Can't reach python.org at the moment, but Python C/api manual, section 7.3.2, unicode objects says: Py_UNICODE This type represents a 16-bit unsigned storage type which is used by Python internally as basis for holding Unicode ordinals. On platforms where wchar_t is available and also has 16-bits, Py_UNICODE is a typedef alias for wchar_t to enhance native platform compatibility. On all other platforms, Py_UNICODE is a typedef alias for unsigned short. Isn't the size 32 bits for wide unicode builds? Please, please fix this - unicode is already complicated enough even without this confusion! Thomas

2 1

Python Bug Day this month?
by Jeff Epler 08 Oct '04

08 Oct '04

I thought the Python Bug Day was going to return this month, and I was hoping to spare some of my time to help out (on one previous bug day I reviewed some doc patches), but I haven't seen anything about it. Jeff

2 1

Version # for PSF license?
by A.M. Kuchling 07 Oct '04

07 Oct '04

DOAP (http://usefulinc.com/doap) is yet another schema for recording data about software projects. I'd like to add support for it to PyPI; however, before doing that a URL needs to be allocated for the PSF license in the listing at http://usefulinc.com/doap/licenses . Question: does the PSF license have a version number of its own or do we have to say 'PSF license for Python 2.2' (2.3, 2.4, etc)? --amk

1 0

Re: [Python-checkins] python/dist/src/Doc/tut tut.tex, 1.253, 1.254
by Michael Hudson 07 Oct '04

07 Oct '04

rhettinger(a)users.sourceforge.net writes: > Update of /cvsroot/python/python/dist/src/Doc/tut > In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv27294/Doc/tut > > Modified Files: > tut.tex > Log Message: > SF patch #1035498: -m option to run a module as a script > (Contributed by Nick Coghlan.) Does anyone know how to edit Misc/python.man to reflect this change? Heck, does anyone know enough Nroff or whatever to check that python.man is even vaguely up to date? I see it mentions -E, so it can't be *that* stale... Cheers, mwh -- <xyld> what proportion of Swedes speak english? <bea> xyld: all those with teeth

2 1

Status of undocumented distutils features in 2.4?
by Phillip J. Eby 07 Oct '04

07 Oct '04

I was looking up something in the distutils source today, and I noticed that there are some new, undocumented, and not quite complete features that were added earlier this year. Specifically, the 'requires' and 'provides' arguments to 'setup()', and the 'checkdep' command. Should these be removed for 2.4? It doesn't seem likely they'd be able to be fleshed out before the beta, especially since as far as I can recall they haven't been discussed on the Distutils-SIG yet, and some of the features don't seem to be in alignment with the relevant PEPs (e.g. PEP 262's definition of "requires" and "provides"). And, the actual current implementation of the dependency checking is (per the author's comments) "kinda hacky".

3 2

Generator expression in _strptime.py TimeRE.__init__
by Nick Coghlan 06 Oct '04

06 Oct '04

This is currently breaking the tests for me, as [:] doesn't work for generator expressions. Either TimeRE.__init__ needs to go back to using a list comp, or it needs to use list() in __seqToRE instead of [:]. Cheers, Nick. -- Nick Coghlan Brisbane, Australia

2 1

Weekly Python Patch/Bug Summary
by Kurt B. Kaiser 06 Oct '04

06 Oct '04

Patch / Bug Summary ___________________ Patches : 241 open ( +6) / 2640 closed ( +3) / 2881 total ( +9) Bugs : 766 open ( -2) / 4492 closed (+12) / 5258 total (+10) RFE : 154 open ( +2) / 131 closed ( +0) / 285 total ( +2) New / Reopened Patches ______________________ ftplib PASV error bug (2004-09-30) http://python.org/sf/1037516 opened by Tim Nelson urllib2 HTTP digest authentication fix (2004-09-30) http://python.org/sf/1037974 opened by Mathieu Fenniak Provide tuple of "special" exceptions (2004-10-01) http://python.org/sf/1038256 opened by Nick Coghlan __main__ for whichdb.py (2004-10-01) http://python.org/sf/1038388 opened by Oleg Broytmann Fix struct.pack on 64-bit archs (broken on 2.*) (2004-10-02) http://python.org/sf/1038854 opened by Stefan Grundmann pydoc method documentation lookup enhancement (2004-10-02) http://python.org/sf/1038909 opened by Alexander Schmolck OS X: Can't use #!/usr/bin/pythonw (2004-10-02) http://python.org/sf/1038911 opened by Nathan Gray repair typos (2004-10-02) CLOSED http://python.org/sf/1038917 opened by George Yoshida SimpleXMLRPCServer optional allow_none / encoding arguments (2004-10-02) http://python.org/sf/1039083 opened by Guillermo M. Narvaja Patches Closed ______________ repair typos (2004-10-02) http://python.org/sf/1038917 closed by rhettinger doctest: allow custom matchers for testing if got=want (2004-04-10) http://python.org/sf/932935 closed by edloper doctest: Add Tester params to DocTestSuite (2004-04-10) http://python.org/sf/932932 closed by edloper New / Reopened Bugs ___________________ len(str(x)) in decimal.py is not Python 2.3 compatible (2004-09-30) CLOSED http://python.org/sf/1037373 opened by Nick Coghlan textReader: reading after close is a seg fault (2004-09-30) http://python.org/sf/1038013 opened by Jyrki Alakuijala silent installation on windows: no drive in registry values (2004-10-01) CLOSED http://python.org/sf/1038410 opened by Matteo Gallivanoni Python 2.3+ socket._fileobject handles EAGAIN with data loss (2004-10-01) http://python.org/sf/1038591 opened by Jon Nelson bad link in Tkinter docs (2004-10-01) http://python.org/sf/1038693 opened by Brett Cannon Typo "signiture" in HTML generated by DocXMLRPCServer (2004-10-02) CLOSED http://python.org/sf/1038935 opened by Malte Helmert apply.__doc__ says "Deprecated since release 2.3" (2004-10-02) CLOSED http://python.org/sf/1039092 opened by Jp Calderone time zone tests fail on Windows (2004-10-03) CLOSED http://python.org/sf/1039270 opened by George Yoshida os.times() is bogus (2004-10-04) http://python.org/sf/1040026 opened by Guido van Rossum Missing documentation on How To Link With libpython (2004-10-05) http://python.org/sf/1040439 opened by Sir Raorn Bugs Closed ___________ Odd behavior with unicode.translate on OSX. (2004-09-22) http://python.org/sf/1032615 closed by doerwalter Can't inherit slots from new-style classes implemented in C (2004-09-24) http://python.org/sf/1034178 closed by philthompson10 len(str(x)) in decimal.py is not Python 2.3 compatible (2004-09-30) http://python.org/sf/1037373 closed by ncoghlan printf() in dynload_shlib.c should be PySys_WriteStderr (2004-09-29) http://python.org/sf/1036752 closed by loewis hex() and oct() documentation is incorrect (2004-09-27) http://python.org/sf/1035279 closed by rhettinger silent installation on windows: no drive in registry values (2004-10-01) http://python.org/sf/1038410 closed by theller Typo "signiture" in HTML generated by DocXMLRPCServer (2004-10-02) http://python.org/sf/1038935 closed by bcannon apply.__doc__ says "Deprecated since release 2.3" (2004-10-02) http://python.org/sf/1039092 closed by rhettinger Message.get_filename() problem (2003-12-04) http://python.org/sf/854102 closed by bwarsaw time zone tests fail on Windows (2004-10-02) http://python.org/sf/1039270 closed by bcannon Case sensitivity bug in ConfigParser (2004-08-27) http://python.org/sf/1017864 closed by goodger ConfigParser behavior change (2004-07-24) http://python.org/sf/997050 closed by goodger New / Reopened RFE __________________ win32 silent installation: allow MAINDIR on command line (2004-10-04) http://python.org/sf/1039857 opened by Matteo Gallivanoni os.run function for convinience and robustness (2004-10-04) http://python.org/sf/1040267 opened by Alexander Schmolck

1 0

Re: [Python-Dev] proposed struct module format code addition
by Josiah Carlson 06 Oct '04

06 Oct '04

If the proposed struct module post seemed to have been a dupe, it is because it sat in a moderator queue for a few days. Sorry about that. - Josiah

1 0

Re: [Python-Dev] Guidelines for Logging Usage
by Michael P. Dubner 05 Oct '04

05 Oct '04

Garth T Kidd wrote: >Strikes me that not much work could let someone keep all of their >existing print statements more or less intact and use the logging >module properly, too: > > import logging > > class LoggingHandle: > def __init__(self, wrappedmethod): > self.wrappedmethod = wrappedmethod > def write(self, msg): > msg = msg.rstrip() > if msg: > self.wrappedmethod(msg.rstrip()) > > logging.basicConfig() > log = logging.getLogger('whatever') > stderr = LoggingHandle(log.error) > stdout = LoggingHandle(log.info) > > print >> stdout, "This is informational." # logged as INFO > print >> stderr, "This is an error." # guess. > >Add some basic infrastructure to logging, and you're done. The >Extremely Brave could even alter sys.stderr and sys.stdout. > > Main purpose of logging module is not redirection, but filtering capability. -- Best regards, Michael Dubner (dubnerm(a)mindless.com) Brainbench MVP/HTML+JavaScript (http://www.brainbench.com) PS: Sorry for my English

2 1

Guidelines for Logging Usage
by Vinay Sajip＠RedDove 05 Oct '04

05 Oct '04

[Michael Dubner] > Main purpose of logging module is not redirection, but filtering capability. I don't agree. In general, both the redirection capability (targeting events at a particular audience) and the filtering capability (verbosity control) are important. Regards, Vinay Sajip

2 1

ConfigParser patches
by David Goodger 05 Oct '04

05 Oct '04

I have patches up on SF for two ConfigParser bugs, which I'd like to check in ASAP. Any objections? http://www.python.org/sf/1017864: Case sensitivity bug in ConfigParser ====================================================================== This is a simple bug relating to default values passed in as dictionaries, illustrated here: >>> import ConfigParser >>> cp = ConfigParser.ConfigParser({"foo": "Bar"}) >>> cp.get("DEFAULT", "Foo") 'Bar' >>> cp = ConfigParser.ConfigParser({"FOO": "Bar"}) # <-- note CAPS >>> cp.get("DEFAULT", "FOO") # same for "foo", "Foo", etc. Traceback (most recent call last): ... ConfigParser.NoOptionError: No option 'foo' in section: 'DEFAULT' The patch applies ConfigParser.optionxform to keys of defaults when supplied, to be consistent with the handling of keys of config file entries and runtime-set options. A test patch is also included. http://www.python.org/sf/997050: ConfigParser behavior change ============================================================= This may be more controversial. In Python 2.3 and earlier, ConfigParser implicitly allowed non-string values to be set. http://python.org/sf/810843 asked for clarification, and in rev 1.65 an explicit type check was added that raises TypeError. The problem is that this breaks Docutils code, and I suspect it will break other code as well, which we won't hear about until after Python 2.4 final is released. Setting non-string values worked just fine for application-internal use, as long as interpolation is turned off (RawConfigParser is used or ConfigParser with "raw" parameter set) and config files aren't written. A new SafeConfigParser class was added in Python 2.3: New applications should prefer this version if they don't need to be compatible with older versions of Python. My solution is to add the new string-only restriction to the SafeConfigParser class, leave existing ConfigParser & RawConfigParser behavior alone, and document the conditions under which non-string values work. Code, doc, and test patches included. What say you? -- David Goodger <http://python.net/~goodger>

22 54

grant applications
by Paul F. Dubois 05 Oct '04

05 Oct '04

Dear devs, You'll be pleased to learn that the PSF grants process brought in sixty-four proposals. I think that is a stunning success, and the wide variety of these proposals is a testimony to the breadth and strength of the community. We should seek some branding sponsorship in the future so we can accept more of these. I'm thinking along the likes of the "Kraft Cottage Cheese Grant". (:-> Picture of smiling Guido enjoying the product at his terminal.

1 0

Re: '-m' option (was RE: [Python-Dev] ConfigParser patches)
by Nick Coghlan 04 Oct '04

04 Oct '04

Quoting Paramjit Oberoi <psoberoi(a)gmail.com>: > With a slightly longer script, and a little work in the installer, > these could be written as: > > Installed: runpy.py profile demo.py > Prompt After: runpy.py -i profile demo.py > Alt install: runpy2.4.py profile demo.py > Build dir: ./runpy.py profile demo.py > - or - ./scripts/runpy.py profile demo.py Indeed, it may be possible to get those to *run*, but the problem is that the semantics of the following two commands are likely to differ: python <interpreter-options> runpy.py module <script-args> python runpy.py <interpreter-options> module <script-args And getting the semantics close enough that the remaining differences don't matter is likely to be a pain. Certainly, you won't be able to use execfile() any more, because it doesn't support passing compiler flags. And the script doesn't have to get much more complicated before it becomes _harder_ to understand and maintain than the C code required to implement '-m' (which is really pretty straightforward when it is limited to top-level modules). Cheers, Nick. -- Nick Coghlan Brisbane, Australia

2 1

new member
by B H, Vijaya Kumar (Vijaya) 04 Oct '04

04 Oct '04

Hi all My name is Vijay I am new member of this group I have been working on python from last 1 year , i love this language for its simpicity , i also worked on jython which i used in few of my automation tasks Regards Vijaya Kumar B.H.

1 0

proposed struct module format code addition
by Josiah Carlson 04 Oct '04

04 Oct '04

Good day everyone, I have produced a patch against the latest CVS to add support for two new formatting characters in the struct module. It is currently an RFE, which I include a link to at the end of this post. Please read the email before you respond to it. Generally, the struct module is for packing and unpacking of binary data. It includes support to pack and unpack the c types: byte, char, short, long, long long, char[], *, and certain variants of those (signed/unsigned, big/little endian, etc.) Purpose ------- I had proposed two new formatting characters, 'g' and 'G' (for biGint or lonG int). There was one primary purpose, to offer users the opportunity to specify their own integer lengths (very useful for cryptography, and real-world applications that involve non-standard sized integers). Current solutions involve shifting, masking, and multiple passes over data. There is a secondary purpose, and that is that future n-byte integers (like 16-byte/128-bit integers as supported by SSE2) are already taken care of. It also places packing and unpacking of these larger integers in the same module as packing and packing of other integers, floats, etc. This makes documentation easy. Functionality-wise, it merely uses the two C functions _PyLong_FromByteArray() and _PyLong_ToByteArray(), with a few lines to handle interfacing with the pack and unpack functions in the struct module. An example of use is as follows: >>> struct.pack('>3g', -1) '\xff\xff\xff' >>> struct.pack('>3g', 2**23-1) '\x7f\xff\xff' >>> struct.pack('>3g', 2**23) Traceback (most recent call last): File "<stdin>", line 1, in ? OverflowError: long too big to convert >>> struct.pack('>3G', 2**23) '\x80\x00\x00' It follows the struct module standard 'lowercase for signed, uppercase for unsigned'. Arguments --------- There seem to be a few arguments against its inclusion into structmodule.c... Argument: The size specifier is variable, so you must know the size/magnitude of the thing you are (un)packing before you (un)pack it. My Response: All use cases I have for this particular mechanism involve not using 'variable' sized structs, but fixed structs with integers of non-standard byte-widths. Specifically, I have a project in which I use some 3 and 5 byte unsigned integers. One of my (un)pack format specifiers is '>H3G3G', and another is '>3G5G' (I have others, but these are the most concise). Certainly this does not fit the pickle/cPickle long (un)packing use-case, but that problem relies on truely variable long integer lengths, of which this specifier does not seek to solve. Really, the proposed 'g' and 'G' format specifiers are only as variable as the previously existing 's' format specifier. Argument: The new specifiers are not standard C types. My Response: Certainly they are not standard C types, but they are flexible enough to subsume all current integer C type specifiers. The point was to allow a user to have the option of specifying their own integer lengths. This supports use cases involving certain kinds of large dataset processing (my use case, which I may discuss after we release) and cryptography, specifically in the case of PKC... while 1: blk = get_block() iblk = struct.unpack('>128G', blk)[0] uiblk = pow(iblk, power, modulous) write_block(struct.pack('>128G', uiblk)) The 'p' format specifier is also not a standard C type, and yet it is included in struct, specifically because it is useful. Argument: You can already do the same thing with: pickle.encode_long(long_int) pickle.decode_long(packed_long) and some likely soon-to-be included additions to the binascii module. My Response: That is not the same. Nontrivial problems require multiple passes over your data with multiple calls. A simple: struct.unpack('H3G3G', st) becomes: pickle.decode_long(st[:2]) #or an equivalent struct call pickle.decode_long(st[2:5]) pickle.decode_long(st[5:8]) And has no endian or sign options, or requires the status quo using of masks and shifts to get the job done. As previously stated, one point of the module is to reduce the amount of bit shifting and masking required. Argument: We could just document a method for packing/unpacking these kinds of things in the struct module, if this really is where people would look for such a thing. My Response: I am not disputing that there are other methods of doing this, I am saying that the struct module includes a framework and documentation location that can include this particular modification with little issue, which is far better than any other proposed location for equivalent functionality. Note that functionality equivalent to pickle.encode/decode_long is NOT what this proposed enhancement is for. Argument: The struct module has a steep learning curve already, and this new format specifier doesn't help it. My Response: I can't see how a new format specifier would necessarily make the learning curve any more difficult, if it was even difficult in the first place. Why am I even posting --------------------- Raymond has threatened to close this RFE due to the fact that only I have been posting to state that I would find such an addition useful. If you believe this functionality is useful, or even if you think that I am full of it, tell us: http://python.org/sf/1023290 - Josiah

9 13

introducing
by Lars Blockken 03 Oct '04

03 Oct '04

Hello, I'm Lars and i'm knew for python so i hope you can answer my questions in the future. Greetz Lars

2 1

proposed struct module format code addition
by Josiah Carlson 03 Oct '04

03 Oct '04

Good day everyone, I have produced a patch against the latest CVS to add support for two new formatting characters in the struct module. It is currently an RFE, which I include a link to at the end of this post. Please read the email before you respond to it. Generally, the struct module is for packing and unpacking of binary data. It includes support to pack and unpack the c types: byte, char, short, long, long long, char[], *, and certain variants of those (signed/unsigned, big/little endian, etc.) Purpose ------- I had proposed two new formatting characters, 'g' and 'G' (for biGint or lonG int). There was one primary purpose, to offer users the opportunity to specify their own integer lengths (very useful for cryptography, and real-world applications that involve non-standard sized integers). Current solutions involve shifting, masking, and multiple passes over data. There is a secondary purpose, and that is that future n-byte integers (like 16-byte/128-bit integers as supported by SSE2) are already taken care of. It also places packing and unpacking of these larger integers in the same module as packing and packing of other integers, floats, etc. This makes documentation easy. Functionality-wise, it merely uses the two C functions _PyLong_FromByteArray() and _PyLong_ToByteArray(), with a few lines to handle interfacing with the pack and unpack functions in the struct module. An example of use is as follows: >>> struct.pack('>3g', -1) '\xff\xff\xff' >>> struct.pack('>3g', 2**23-1) '\x7f\xff\xff' >>> struct.pack('>3g', 2**23) Traceback (most recent call last): File "<stdin>", line 1, in ? OverflowError: long too big to convert >>> struct.pack('>3G', 2**23) '\x80\x00\x00' It follows the struct module standard 'lowercase for signed, uppercase for unsigned'. Arguments --------- There seem to be a few arguments against its inclusion into structmodule.c... Argument: The size specifier is variable, so you must know the size/magnitude of the thing you are (un)packing before you (un)pack it. My Response: All use cases I have for this particular mechanism involve not using 'variable' sized structs, but fixed structs with integers of non-standard byte-widths. Specifically, I have a project in which I use some 3 and 5 byte unsigned integers. One of my (un)pack format specifiers is '>H3G3G', and another is '>3G5G' (I have others, but these are the most concise). Certainly this does not fit the pickle/cPickle long (un)packing use-case, but that problem relies on truely variable long integer lengths, of which this specifier does not seek to solve. Really, the proposed 'g' and 'G' format specifiers are only as variable as the previously existing 's' format specifier. Argument: The new specifiers are not standard C types. My Response: Certainly they are not standard C types, but they are flexible enough to subsume all current integer C type specifiers. The point was to allow a user to have the option of specifying their own integer lengths. This supports use cases involving certain kinds of large dataset processing (my use case, which I may discuss after we release) and cryptography, specifically in the case of PKC... while 1: blk = get_block() iblk = struct.unpack('>128G', blk)[0] uiblk = pow(iblk, power, modulous) write_block(struct.pack('>128G', uiblk)) The 'p' format specifier is also not a standard C type, and yet it is included in struct, specifically because it is useful. Argument: You can already do the same thing with: pickle.encode_long(long_int) pickle.decode_long(packed_long) and some likely soon-to-be included additions to the binascii module. My Response: That is not the same. Nontrivial problems require multiple passes over your data with multiple calls. A simple: struct.unpack('H3G3G', st) becomes: pickle.decode_long(st[:2]) #or an equivalent struct call pickle.decode_long(st[2:5]) pickle.decode_long(st[5:8]) And has no endian or sign options, or requires the status quo using of masks and shifts to get the job done. As previously stated, one point of the module is to reduce the amount of bit shifting and masking required. Argument: We could just document a method for packing/unpacking these kinds of things in the struct module, if this really is where people would look for such a thing. My Response: I am not disputing that there are other methods of doing this, I am saying that the struct module includes a framework and documentation location that can include this particular modification with little issue, which is far better than any other proposed location for equivalent functionality. Note that functionality equivalent to pickle.encode/decode_long is NOT what this proposed enhancement is for. Argument: The struct module has a steep learning curve already, and this new format specifier doesn't help it. My Response: I can't see how a new format specifier would necessarily make the learning curve any more difficult, if it was even difficult in the first place. Why am I even posting --------------------- Raymond has threatened to close this RFE due to the fact that only I have been posting to state that I would find such an addition useful. If you believe this functionality is useful, or even if you think that I am full of it, tell us: http://python.org/sf/1023290 - Josiah

1 0

[OT] stats on type-inferencing atomic types in local variables in the stdlib
by Brett C. 02 Oct '04

02 Oct '04

My thesis (which, for those who don't know, was to come up with a way to do type inferencing in the compiler without requiring any semantic changes; basically type inferencing atomic types assigned to local variables) is now far enough long that I have the algorithm done and I can generate statistics on what opcodes are called with the most common types that I can specifically infer (can also do static type checking on occasion; only triggers were actual unit tests making sure TypeError was raised for certain things like ``~4.2`` and such). Thought some of you might get a kick out of this since the numbers are rather blatent for certain opcodes and methods. To read the stats, the number to the left is the number of times the opcode was compiled (not executed!) with the specific type(s) known for the opcode (if it took two args, then both types are listed; order was considered irrelevant). Now they are listed as integers, so here is the conversion:: Basestring 4 IntegralType 8 FloatType 16 ImagType 32 DictType 64 ListType 128 TupleType 256 For the things named "meth_<something>" that is the method being called immediately on the type. Now realize these numbers are only for opcodes where I could definitely infer the type; ones where it could be more than one type, regardless if those possibilities were very specific, I just ignored it and did not include in the stats. I also tweaked some opcodes knowing how they are more often used. So, for instance, BINARY_MODULO checks specifically for the case of when the left side is a basestring and then just doesn't worry about the other args. Other ones I just didn't bother with all the args since it was not interesting to me in terms of deciding what type-specific opcodes I want to come up with. Anyway, here are the numbers on Lib sans Lib/test (129,814 lines according to SLOCCount) for the ones above 100:: (101, ('BINARY_MULTIPLY', (8, 4))), (106, ('BINARY_SUBSCR', 128)), (118, ('GET_ITER', 128)), (124, ('BINARY_MODULO', None)), (195, ('meth_join', 4)), (204, ('BINARY_ADD', (8, 8))), (331, ('BINARY_ADD', (4, 4))), (513, ('BINARY_LSHIFT', (8, 8))), (840, ('meth_append', 128)), (1270, ('PRINT_ITEM', 4)), (1916, ('BINARY_MODULO', 4)), (12302, ('STORE_SUBSCR', 64))] We sure like our dictionaries (for those that don't know, dictionaries are created by making an empty dict and then basically doing an indivual assignment for each value). We also seem to love to use string interpolation, and printing stuff. Using list.append is also popular. Now the BINARY_LSHIFT is rather interesting, and that ties into the whole issue of how much I can actually infer; since binary work tends to be with all constants I can infer it really easily and so its frequency is rather high. Its actual frequency of use, though, compared to other things probably is not high, though. Plus I doubt Plone, for instance, uses ``<<`` very often so I suspect the opcode will get weeded out when I incorporate stats from the other apps I am taking stats from. As for the stuff I cut out, the surprising thing from those numbers was how few mathematical expressions could be inferred. I checked my numbers with grep and there really is only 3 times where a float constant is divided by a float constant (and they are all in colorsys). I was not expecting that at all. Guess global variables or object attributes tend to have them or I just can't infer the values. Either way I just wasn't expecting that. Anyway, as I said I just thought some people might find this interesting. Don't read into this too much since I am just using these numbers as guidelines for type-specific opcodes to write for use as a quantifiable measurement of the usefulness of type inferencing like this. -Brett P.S.: anyone who is *really* interested I can send you the full stats for the apps I have run my modified version of compile.c against.

2 2

python-dev Summary for 2004-09-01 through 2004-09-15 [draft]
by Brett C. 01 Oct '04

01 Oct '04

===================== Summary Announcements ===================== Python 2.4a3 has been released. Go to http://www.python.org/2.4/ and give it twirl. Sorry for this summary being so short, but school has started back up again so I am in the middle of suddenly having to switch back into homework mode after spending the summer just having a 9:00-17:00 job. And since it is a new school year I am going to abuse this space and say that anyone in San Luis Obispo, including students, should join the `SLO Meetup`_ coming up on October 14. .. _SLO Meetup: http://python.meetup.com/95/ ========= Summaries ========= -------------------- Movement in PEP Land -------------------- `PEP 334`_ (Simple Coroutines via SuspendIteration) came into existence. `PEP 328`_ (Relative Imports) got some discussion on postponing making imports absolute instead of the relative/absolute semantics they have now. As it stands it looks like the changeover might get pushed off. `PEP 292`_ (Simpler String Substitutions) seems to finally be done and locked down. `PEP 335`_ (Overloadable Boolean Operators) came into existence. .. _PEP 334: http://www.python.org/peps/pep-0334.html .. _PEP 328: http://www.python.org/peps/pep-0328.html .. _PEP 292: http://www.python.org/peps/pep-0292.html .. _PEP 335: http://www.python.org/peps/pep-0335.html Contributing threads: - `PEP 334 - Simple Coroutines via SuspendIteration <>`__ - `PEP 328 - Relative Imports <>`__ - `Re: Alternative Implementation for PEP 292:Simple String Substitutions <>`__ - `ANN: PEP 335: Overloadable Boolean Operators <>`__ ------------------------------------------------------ __str__, __unicode__, and how to have them play nicely ------------------------------------------------------ Did you know that __str__ methods are allowed to return Unicode objects? Well, it turns out they can, but that str() (which calls PyObject_Str()) automatically tries to convert the value returned by __str__ into ASCII. Basically __str__ shouldn't return Unicode if you can help it and you should use __unicode__ instead and reserve __str__ to return str objects only. Contributing threads: - `unicode and __str__ <>`__ ---------------------- Backporting C APIs bad ---------------------- Somebody (*cough* Guido *cough*) asked if the datetime C API could be backported to 2.3 . The argument was that the only person who would probably use it is the person who asked for it, the author of cx_Oracle. Well, pretty much everyone spoke up against this. The argument went that adding an API would just be bad since there would suddenly be a point in the 2.3 releases where backwards compatibility was broken. People brought up the point of 2.2 where in 2.2.1 booleans were added and how that has caused compatibility headaches for some people. In the end the API was not backported. Contributing threads: - `Re: [Python-checkins] python/dist/src/Modules threadmodule.c, 2.56, 2.56.8.1 <>`__ ---------------------------------------------------------------------- Got to love race conditions thanks to the filesystem and external apps ---------------------------------------------------------------------- Tim Peters found a race condition one can have on Windows if you have an app that uses the low-level hooks into the filesystem. If you create a file, delete it, and then try to delete the directory the directory deletion will fail since the file is not deleted yet. What can happen is an indexing program can still be indexing the file before the filesystem is allowed to delete it and thus the directory is not truly empty when the directory deletion is executed. Fun stuff. Contributing threads: - `Coernic Desktop Search versus shutil.rmtree <>`__ ------------------------------- Python 2.4a3 is out the door!!! ------------------------------- Go to http://www.python.org/2.4/ , download it (using the bz2 version if possible so as to save on bandwidth), and run it against your code, run the test suite, put it on your head and sell yourself to an art gallery, etc. Contributing threads: - `RELEASED Python 2.4, alpha 3 <>`__ ---------------------------- Cleaning the Exception House ---------------------------- The idea of reorganizing the exceptions hierarchy came up again (see http://www.python.org/dev/summary/2004-08-01_2004-08-15.html#an-exception-i… for a previous discussion on this). This time, the idea of separating the hierarchy into exceptions one would like to catch with a bare 'except' and those that you wouldn't was brought up. The idea is that some exceptions, such as MemoryError, one does not want to catch in a blanket statement usually. Chances of recovering from that kind of exception is low and should only be caught if you know what you are doing. So tweaking the exception hierarchy so that exceptions that were not near-catastrophic could inherit from an exception class that people could catch so as to allow the proper exceptions to propagate to the top-level without issue. Tim Peters even went as far as to suggest deprecating bare 'except' statements. This would force people to be explicit about what they want to catch, whether it be all "safe" exceptions or *all* exceptions. As it stands now no officially decision has been made for Python 3000 since that is about the only place this could happen. Contributing threads: - `Dangerous exceptions <>`__ --------------------------------------------------------------- Making decorators not look like decorators to the outside world --------------------------------------------------------------- Raymond Hettinger pointed out that a decorator does not, by default, look like the function that it is fiddling with (if that is the intent). Since most decorators will most likely be a wrapper function some things need to be set in the wrapper in order not to mask things in the wrapped function (doc string, argument parameters, etc.). So Raymond pointed out some things one can do. This also led to the suggestion of having a common name used to store a reference back to the wrapped function. There was also the mention that a decorator-oriented module in the stdlib will probably materialize in Python 2.5 . For now, though, stick recipes either in the Python Cookbook or in the Python wiki at http://www.python.org/cgi-bin/moinmoin/PythonDecoratorLibrary . Contributing threads: - `decorator support <>`__ =============== Skipped Threads =============== - random.py still broken wrt. urandom - random.py fixage - Re: [Python-checkins] python/dist/src/Lib/test test_compiler.py, 1.5, 1.6 test_decimal.py, 1.13, 1.14 - assert failure on obmalloc - Re: [Python-checkins] python/dist/src/Modules socketmodule.c, 1.304, 1.305 - Install-on-first-use vs. optional extensions - Console vs. GUI applications - Adding status code constants to httplib - PEP 292: method names - PEP 265 - Sorting dicts by value - httplib is not v6 compatible, is this going to be fixed? - OT: Unicode history - --with-tsc compile fails - tempfile.TemporaryFile on Windows NT - PyExc_UnicodeDecodeError - urllib.urlopen() vs IDNs, percent-encoded hosts, ':' - tabs in httplib.py and test_httplib.py There is now a vimrc file in the Misc directory that sets things up to follow PEPs 7 & 8 - strawman decision: @decorator won't change - unicode inconsistency?

1 1

Proposing a sys.special_exceptions tuple
by Nick Coghlan 01 Oct '04

01 Oct '04

I spent some time the other day looking at the use of bare except statements in the standard library. Many of them seemed to fall into the category of 'need to catch anything user code is likely to throw, but shouldn't be masking SystemExit, StopIteration, KeyboardInterrupt, MemoryError, etc'. Changing them to "except Exception:" doesn't help, since all of the above still fit into that category (Tim posted a message recently about rearranging the Exception heirarchy to fix this. Backwards compatibility woes pretty much killed the discussion though). However, another possibility occurred to me: try: # Do stuff except sys.special_exceptions: raise except: # Deal with all the mundane stuff With an appropriately defined tuple, that makes it easy for people to "do the right thing" with regards to critical exceptions. Such a tuple could also be useful for invoking isinstance() and issubclass(). Who knows? If something like this caught on, it might some day be possible to kill a Python script with a single press of Ctrl-C };> Cheers, Nick. -- Nick Coghlan Brisbane, Australia

6 9

Running a module as a script
by Nick Coghlan 01 Oct '04

01 Oct '04

Patch # 1035498 attempts to implement the semantics suggested by Ilya and Anthony and co. "python -m module" Means: - find the source file for the relevant module (using the standard locations for module import) - run the located script as __main__ (note that containing packages are NOT imported first - it's as if the relevant module was executed directly from the command line) - as with '-c' anything before the option is an argument to the interpreter, anything after is an argument to the script The allowed modules are those whose associated source file meet the normal rules for a command line script. I believe that means .py and .pyc files only (e.g. "python -m profile" works, but "python -m hotshot" does not). Special import hooks (such as zipimport) almost certainly won't work (since I don't believe they work with the current command line script mechanism). Cheers, Nick. -- Nick Coghlan Brisbane, Australia

4 7

Another test_compiler mystery
by Tim Peters 01 Oct '04

01 Oct '04

I've noticed several times now, in both debug and release builds, that if I run regrtest.py with -uall, *sometimes* it just stops after running test_compiler: $ python_d regrtest.py -uall test_grammar test_opcodes ... test_compare test_compile test_compiler $ There's no indication of error, it just ends. It's not consistent. Happened once when I was running with -v, and test_compiler's output ended here: ... compiling C:\Code\python\lib\test\test_operator.py compiling C:\Code\python\lib\test\test_optparse.py compiling C:\Code\python\lib\test\test_os.py compiling C:\Code\python\lib\test\test_ossaudiodev.py compiling C:\Code\python\lib\test\test_parser.py In particular, there's no Ran M tests in Ns output, so it doesn't look like unittest (let alone regrtest) ever got control back. Hmm. os.listdir() is in sorted order on NTFS, so test_compiler should be chewing over a lot more files after test_parser.py. *This* I could blame on a blown C stack -- although I'd expect a much nastier symptom then than just premature termination. Anyone else?

23 50