Mailman 3 August 2002 - Python-Dev

Re: [ python-Patches-590682 ] New codecs: html, asciihtml
by Oren Tirosh 05 Aug '02

05 Aug '02

(I'm moving this to python-dev) On Sun, Aug 04, 2002 at 08:54:05AM -0700, noreply(a)sourceforge.net wrote: > >Comment By: Martin v. Löwis (loewis) > Date: 2002-08-04 17:54 > > I'm in favour of exposing this via a search functions, for > generated codec names, on top of PEP 293 (I would not like > your codec to compete with the alternative mechanism). My > dislike for the current patch also comes from the fact that > it singles-out ASCII, which the search function would not. I find PEP 293 too complex while my solution is, admittedly, too simplistic. Some of my reservations about PEP 293: It overloads the meaning of the error handling argument in an unintuitive way. It gets to the point where it's much more than just error handling - it's actually extending the functionality of the codec. Why implement yet another name-based registry? There must be a simpler way to do it. Generating an exception for each character that isn't handled by simple lookup probably adds quite a lot of overhead. What are the use cases? Maybe a simple extension to charmap would be enough for all the practical cases? > In anycase, I'd encourage you to contribute to the progress > of PEP 293 first - this has been an issue for several years > now, and I would be sorry if it would fail. Me too. But if you really don't want it to be rejected you should try to find a way to make it simpler. > While you are waiting for PEP 293 to complete, please do > consider cleaning up htmlentitydefs to provide mappings from > and to Unicode characters. No problem. The question is whether anyone depends on its current form. My proposed changes: 1. Use all lowercase entity names as keys. 2. Map "entityname" to u"\uXXXX" (currently it's mapped to "&#nnnn;") In its current form I find htmlentitydefs.py pretty useless. Names in the input in arbitrary case will not match the MixedCase keys in the entitydefs dictionary and the decimal character reference isn't really more useful than the named entity reference. Oren

4 5

Single- vs. Multi-pass iterability
by David Abrahams 05 Aug '02

05 Aug '02

I keep running into the problem that there is no reliable way to introspect about whether a type supports multi-pass iterability (in the sense that an input stream might support only a single pass, but a list supports multiple passes). I suppose you could check for __getitem__, but that wouldn't cover linked lists, for example. Can anyone channel Guido's intent for me? Is this an oversight or a deliberate design decision? Is there an interface for checking multi-pass-ability that I've missed? TIA, Dave

23 141

Re: [Python-Dev] seeing off SET_LINENO
by Michael Hudson 05 Aug '02

05 Aug '02

Tim Peters <tim.one(a)comcast.net> writes: > [Michael Hudson] > > I've found another annoying problem. I'm not really expecting someone > > here to sovle it for me, but writing it down might help me think > > clearly. > > > > This is about the function epilogues that always get generated. I.e: [snip example] > > You can see here that the epilogue gets associated with line 3, > > whereas it shouldn't really be associated with any line at all. > > It has to be associated with some line >= 3, as c_lnotab isn't capable of > expressing anything other than that. Yes. > It *could* associate it with "line 4", though, if the compiler were > changed to pump out another c_lntab entry at the epilogue. That > would be better than saying the time is charged to line 3, since it > isn't on line 3 then. I'd be happy to trade away total insanity for > partial insanity <wink>. This would be bad if you had def f(): print 1 def g(): print 2 Anyway, I think I've found a way to get around this (see the patch). > It stops on the "if a:" for me twice today, and I doubt that's any less > confusing. If it were set to line 4 instead, an unaltered pdb would > presumably show a blank line (whatever) after the function body, and an > altered pdb could be taught that "the last line" c_lnotab claims exists is > really devoted to exit code not associated with any source-file line. Yes. I didn't really like the idea of heavily hacking pdb, as I don't understand it. Cheers, M. -- 39. Re graphics: A picture is worth 10K words - but only those to describe the picture. Hardly any sets of 10K words can be adequately described with pictures. -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html

1 0

New encoding error in debug build
by Tim Peters 04 Aug '02

04 Aug '02

This assert near the end of get_coding_spec() in tokenizer.c triggers when running test_heapq in a debug build: assert(strlen(r) >= strlen(q)); It's a very new failure. Note that heapq.py begins with the line # -*- coding: Latin-1 -*- I assume that's relevant, but Latin-1 is way beyond my personal experience with strange character sets <wink>.

2 1

Re: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib
by David Goodger 04 Aug '02

04 Aug '02

[Guido] >>> I was originally going to make it an unnamed string >>> literal -- maybe that's better? [David] >> In PEP 258 I call those "Additional Docstrings": [Guido] > Ah, I thought there had to be something like that. :-) > > Do you also recognize this if there are comments between? Or blank > lines? E.g. > > def f(): > """ > foo > """ > > # blah > > """ > bar > """ We haven't gotten that far get. I see no problems with blank lines, but comments may block recognition, unless we choose to ignore them. On the other hand, comments themselves may be used in some circumstances; HappyDoc recognizes comments *before* a def/class statement if there's no docstring. There's still much to be thought out. -- David Goodger <goodger(a)users.sourceforge.net> Open-source projects: - Python Docutils: http://docutils.sourceforge.net/ (includes reStructuredText: http://docutils.sf.net/rst.html) - The Go Tools Project: http://gotools.sourceforge.net/

1 0

Weekly Python Bug/Patch Summary
by Skip Montanaro 04 Aug '02

04 Aug '02

Bug/Patch Summary ----------------- 272 open / 2715 total bugs (+8) 131 open / 1632 total patches (-12) New Bugs -------- Invalid mmap crashes Python interpreter (2002-07-24) http://python.org/sf/585792 re.finditer (2002-07-24) http://python.org/sf/585882 macfs.FSSpec fails for "new" files (2002-07-24) http://python.org/sf/585923 Two corrects for weakref docs (2002-07-25) http://python.org/sf/586583 -S hides standard dynamic modules (2002-07-25) http://python.org/sf/586680 site-packages & build-dir python (2002-07-25) http://python.org/sf/586700 email package does not work with mailbox (2002-07-26) http://python.org/sf/586899 Empty genindex.html pages (2002-07-26) http://python.org/sf/586926 pydoc -w fails with path specified (2002-07-26) http://python.org/sf/586931 references to email package (2002-07-26) http://python.org/sf/586937 OSA Python integration (2002-07-26) http://python.org/sf/586998 IBCarbon module (2002-07-26) http://python.org/sf/587011 ur'\u' not handled properly (2002-07-26) http://python.org/sf/587087 python-mode and nested indents (2002-07-26) http://python.org/sf/587239 imaplib: prefix-quoted strings (2002-07-30) http://python.org/sf/588711 python should obey the FHS (2002-07-30) http://python.org/sf/588756 add main to py_pycompile (2002-07-30) http://python.org/sf/588768 unittest.py, better error message (2002-07-30) http://python.org/sf/588825 Memory leakage in SAX with exception (2002-07-31) http://python.org/sf/589149 socket.py wrapper needs a class (2002-07-31) http://python.org/sf/589262 shared libpython & dependant libraries (2002-07-31) http://python.org/sf/589422 standard include paths on command line (2002-07-31) http://python.org/sf/589427 "".split() ignores maxsplit arg (2002-08-01) http://python.org/sf/589965 PyMapping_Keys unexported in dll (2002-08-02) http://python.org/sf/590207 preconvert AppleSingle resource files (2002-08-02) http://python.org/sf/590456 New Patches ----------- PEP 282 Implementation (2002-07-07) http://python.org/sf/578494 Adds Galeon support to webbrowser.py (2002-07-24) http://python.org/sf/585913 galeon support in webbrowser (2002-07-25) http://python.org/sf/586437 Better token-related error messages (2002-07-25) http://python.org/sf/586561 alternative SET_LINENO killer (2002-07-29) http://python.org/sf/587993 Cygwin _hotshot patch (2002-07-30) http://python.org/sf/588561 _locale library patch (2002-07-30) http://python.org/sf/588564 LDFLAGS support for build_ext.py (2002-07-30) http://python.org/sf/588809 Mindless editing, DL_EXPORT/IMPORT (2002-07-31) http://python.org/sf/588982 tempfile.py rewrite (2002-08-01) http://python.org/sf/589982 types.BoolType (2002-08-02) http://python.org/sf/590119 os._execvpe security fix (2002-08-02) http://python.org/sf/590294 py2texi.el update (2002-08-02) http://python.org/sf/590352 db4 include not found (2002-08-02) http://python.org/sf/590377 Add popen2 like functionality to pty.py. (2002-08-03) http://python.org/sf/590513 New codecs: html, asciihtml (2002-08-03) http://python.org/sf/590682 Closed Bugs ----------- Need user-centered info for Windows users. (2000-11-27) http://python.org/sf/223599 profiling with xml parsing asserts (2002-03-25) http://python.org/sf/534864 Compile error _sre.c on Cray T3E (2002-05-19) http://python.org/sf/558153 DL_EXPORT on VC7 broken (2002-05-20) http://python.org/sf/558488 crash on gethostbyaddr (2002-06-07) http://python.org/sf/565747 asynchat module undocumented (2002-06-12) http://python.org/sf/568134 socket module htonl/ntohl bug (2002-06-12) http://python.org/sf/568322 .PYO files not imported unless -OO used (2002-06-18) http://python.org/sf/570640 CGIHTTPServer flushes read-only file. (2002-06-18) http://python.org/sf/570678 Negative __len__ provokes SystemError (2002-06-30) http://python.org/sf/575773 Sig11 in cPickle (stack overflow) (2002-07-01) http://python.org/sf/576084 Infinite recursion in Pickle (2002-07-02) http://python.org/sf/576419 GC Changes not mentioned in What's New (2002-07-12) http://python.org/sf/580462 pty.spawn - wrong error caught (2002-07-15) http://python.org/sf/581698 os.getlogin() fails (2002-07-21) http://python.org/sf/584566 Closed Patches -------------- timestamp function for time module (2001-08-17) http://python.org/sf/452232 Unambiguous import for encodings (2001-09-06) http://python.org/sf/459381 no '_d' ending for mingw32 (2001-09-18) http://python.org/sf/462754 HTML version of the Idle "documentation" (2001-10-12) http://python.org/sf/470607 whichdb unittest (2002-04-09) http://python.org/sf/541694 s/Copyright/License/ in bdist_rpm.py (2002-04-13) http://python.org/sf/543498 merging sorted sequences (2002-04-15) http://python.org/sf/544113 Read/Write buffers from buffer() (2002-04-30) http://python.org/sf/550551 Better description in "python -h" for -u (2002-05-06) http://python.org/sf/552812 Cygwin make install patch (2002-05-08) http://python.org/sf/553702 __va_copy patches (2002-05-10) http://python.org/sf/554716 Ebcdic compliancy in stringobject source (2002-05-19) http://python.org/sf/557946 README additions for Cray T3E (2002-05-28) http://python.org/sf/561724 Fix bug in encodings.search_function (2002-06-20) http://python.org/sf/571603 Executable .pyc-files with hashbang (2002-06-23) http://python.org/sf/572796 Changing owner of symlinks (2002-06-25) http://python.org/sf/573770 Make python-mode.el use jython (2002-06-27) http://python.org/sf/574747 list.extend docstring fix (2002-06-27) http://python.org/sf/574867 SSL release GIL (2002-06-30) http://python.org/sf/575827 Extend PyErr_SetFromWindowsErr (2002-07-02) http://python.org/sf/576458 Remove PyArg_Parse() and METH_OLDARGS (2002-07-03) http://python.org/sf/577031 Merge xrange() into slice() (2002-07-05) http://python.org/sf/577875 fix for problems with test_longexp (2002-07-06) http://python.org/sf/578297 less restrictive HTML comments (2002-07-12) http://python.org/sf/580670 Canvas "select_item" always returns None (2002-07-14) http://python.org/sf/581396 info reader bug (2002-07-14) http://python.org/sf/581414 fix to pty.spawn error on Linux (2002-07-15) http://python.org/sf/581705 get python to link on OSF1 (Dec Unix) (2002-07-20) http://python.org/sf/584245

1 0

split('') revisited
by Andrew Koenig 03 Aug '02

03 Aug '02

Back in February, there was a thread in comp.lang.python (and, I think, also on Python-Dev) that asked whether the following behavior: >>> 'abcde'.split('') Traceback (most recent call last): File "<stdin>", line 1, in ? ValueError: empty separator was a bug or a feature. The prevailing opinion at the time seemed to be that there was not a sensible, unique way of defining this operation, so rejecting it was a feature. That answer didn't bother me particularly at the time, but since then I have learned a new fact (or perhaps an old fact that I didn't notice at the time) that has changed my mind: Section 4.2.4 of the library reference says that the 'split' method of a regular expression object is defined as Identical to the split() function, using the compiled pattern. This claim does not appear to be correct: >>> import re >>> re.compile('').split('abcde') ['abcde'] This result differs from the result of using the string split method. In other words, the documentation doesn't match the actual behavior, so the status quo is broken. It seems to me that there are four reasonable courses of action: 1) Do nothing -- the problem is too trivial to worry about. 2) Change string split (and its documentation) to match regexp split. 3) Change regexp split (and its documentation) to match string split. 4) Change both string split and regexp split to do something else :-) My first impulse was to argue that (4) is right, and that the behavior should be as follows >>> 'abcde'.split('') ['a', 'b', 'c', 'd', 'e'] >>> import re >>> re.compile('').split('abcde') ['a', 'b', 'c', 'd', 'e'] When this discussion came up last time, I think there was an objection that s.split('') was ambiguous: What argument is there in favor of 'abcde'.split('') being ['a', 'b', 'c', 'd', 'e'] instead of, say, ['', 'a', 'b', 'c', 'd', 'e', ''] or, for that matter, ['', 'a', '', 'b', '', 'c', '', 'd', '', 'e', '']? I made the counterargument that one could disambiguate by adding the rule that no element of the result could be equal to the delimiter. Therefore, if s is a string, s.split('') cannot contain any empty strings. However, looking at the behavior of regular expression splitting more closely, I become more confused. Can someone explain the following behavior to me? >>> re.compile('a|(x?)').split('abracadabra') ['', None, 'br', None, 'c', None, 'd', None, 'br', None, '']

4 5

Sorting
by Tim Peters 03 Aug '02

03 Aug '02

An enormous amount of research has been done on sorting since the last time I wrote a sort for Python. Major developments have been in two areas: 1. Adaptive sorting. Sorting algorithms are usually tested on random data, but in real life you almost never see random data. Python's sort tries to catch some common cases of near-order via special- casing. The literature has since defined more than 15 formal measures of disorder, and developed algorithms provably optimal in the face of one or more of them. But this is O() optimality, and theoreticians aren't much concerned about how big the constant factor is. Some researchers are up front about this, and toward the end of one paper with "practical" in its title, the author was overjoyed to report that an implementation was only twice as slow as a naive quicksort <wink>. 2. Pushing the worst-case number of comparisons closer to the information-theoretic limit (ceiling(log2(N!))). I don't care much about #2 -- in experiments conducted when it was new, I measured the # of comparisons our samplesort hybrid did on random inputs, and it was never more than 2% over the theoretical lower bound, and typically closer. As N grows large, the expected case provably converges to the theoretical lower bound. There remains a vanishly small chance for a bad case, but nobody has reported one, and at the time I gave up trying to construct one. Back on Earth, among Python users the most frequent complaint I've heard is that list.sort() isn't stable. Alex is always quick to trot out the appropriate DSU (Decorate Sort Undecorate) pattern then, but the extra memory burden for that can be major (a new 2-tuple per list element costs about 32 bytes, then 4 more bytes for a pointer to it in a list, and 12 more bytes that don't go away to hold each non-small index). After reading all those papers, I couldn't resist taking a crack at a new algorithm that might be practical, and have something you might call a non-recursive adaptive stable natural mergesort / binary insertion sort hybrid. In playing with it so far, it has two bad aspects compared to our samplesort hybrid: + It may require temp memory, up to 2*N bytes worst case (one pointer each for no more than half the array elements). + It gets *some* benefit for arrays with many equal elements, but not nearly as much as I was able to hack samplesort to get. In effect, paritioning is very good at moving equal elements close to each other quickly, but merging leaves them spread across any number of runs. This is especially irksome because we're sticking to Py_LT for comparisons, so can't even detect a==b without comparing a and b twice (and then it's a deduction from that not a < b and not b < a). Given the relatively huge cost of comparisons, it's a timing disaster to do that (compare twice) unless it falls out naturally. It was fairly natural to do so in samplesort, but not at all in this sort. It also has good aspects: + It's stable (items that compare equal retain their relative order, so, e.g., if you sort first on zip code, and a second time on name, people with the same name still appear in order of increasing zip code; this is important in apps that, e.g., refine the results of queries based on user input). + The code is much simpler than samplesort's (but I think I can fix that <wink>). + It gets benefit out of more kinds of patterns, and without lumpy special-casing (a natural mergesort has to identify ascending and descending runs regardless, and then the algorithm builds on just that). + Despite that I haven't micro-optimized it, in the random case it's almost as fast as the samplesort hybrid. In fact, it might have been a bit faster had I run tests yesterday (the samplesort hybrid got sped up by 1-2% last night). This one surprised me the most, because at the time I wrote the samplesort hybrid, I tried several ways of coding mergesorts and couldn't make it as fast. + It has no bad cases (O(N log N) is worst case; N-1 compares is best). Here are some typical timings, taken from Python's sortperf.py, over identical lists of floats: Key: *sort: random data \sort: descending data /sort: ascending data 3sort: ascending data but with 3 random exchanges ~sort: many duplicates =sort: all equal !sort: worst case scenario That last one was a worst case for the last quicksort Python had before it grew the samplesort, and it was a very bad case for that. By sheer coincidence, turns out it's an exceptionally good case for the experimental sort: samplesort i 2**i *sort \sort /sort 3sort ~sort =sort !sort 15 32768 0.13 0.01 0.01 0.10 0.04 0.01 0.11 16 65536 0.24 0.02 0.02 0.23 0.08 0.02 0.24 17 131072 0.54 0.05 0.04 0.49 0.18 0.04 0.53 18 262144 1.18 0.09 0.09 1.08 0.37 0.09 1.16 19 524288 2.58 0.19 0.18 2.34 0.76 0.17 2.52 20 1048576 5.58 0.37 0.36 5.12 1.54 0.35 5.46 timsort 15 32768 0.16 0.01 0.02 0.05 0.14 0.01 0.02 16 65536 0.24 0.02 0.02 0.06 0.19 0.02 0.04 17 131072 0.55 0.04 0.04 0.13 0.42 0.04 0.09 18 262144 1.19 0.09 0.09 0.25 0.91 0.09 0.18 19 524288 2.60 0.18 0.18 0.46 1.97 0.18 0.37 20 1048576 5.61 0.37 0.35 1.00 4.26 0.35 0.74 If it weren't for the ~sort column, I'd seriously suggest replacing the samplesort with this. 2*N extra bytes isn't as bad as it might sound, given that, in the absence of massive object duplication, each list element consumes at least 12 bytes (type pointer, refcount and value) + 4 bytes for the list pointer. Add 'em all up and that's a 13% worst-case temp memory overhead.

9 32

Re: [Python-Dev] seeing off SET_LINENO
by Michael Hudson 03 Aug '02

03 Aug '02

Guido van Rossum <guido(a)python.org> writes: > What's the next step? I haven't had time to review your code. Do you > want to check it in without further review, or do you want to wait > until someone can give it a serious look? (Tim's on vacation this > week so it might be a while.) I've found another annoying problem. I'm not really expecting someone here to sovle it for me, but writing it down might help me think clearly. This is about the function epilogues that always get generated. I.e: >>> def f(): ... if a: ... print 1 ... >>> import dis >>> dis.dis(f) 2 0 LOAD_GLOBAL 0 (a) 3 JUMP_IF_FALSE 9 (to 15) 6 POP_TOP 3 7 LOAD_CONST 1 (1) 10 PRINT_ITEM 11 PRINT_NEWLINE 12 JUMP_FORWARD 1 (to 16) >> 15 POP_TOP >> 16 LOAD_CONST 0 (None) 19 RETURN_VALUE You can see here that the epilogue gets associated with line 3, whereas it shouldn't really be associated with any line at all. For why this is a problem: $ cat t.py a = 0 def f(): if a: print 1 >>> pdb.runcall(t.f) > /home/mwh/src/sf/python/dist/src/build/t.py(3)f() -> if a: (Pdb) s > /home/mwh/src/sf/python/dist/src/build/t.py(4)f() -> print 1 (Pdb) --Return-- > /home/mwh/src/sf/python/dist/src/build/t.py(4)f()->None -> print 1 (Pdb) The debugger stopping on the "print 1" is confusing. There's an "obvious" solution to this: check it we're less than 4 bytes from the end of the code string and don't do anything if we are. This would be easy, except that for some bonkers reason, we support arbitrary buffer objects for code strings! (see _PyCode_GETCODEPTR in Include/compile.h -- though at least you can't create a code object with an array code string from python, the getreadbuffer failing will cause the interpreter to unceremoniously crash and burn). I guess I can store the length somewhere -- _PyCode_GETCODEPTR returns this, more by accident than design I suspect -- or call bf_getsegcount(frame->f_code->co_code, &length) or something. Does anyone actually *use* this feature? I see Guido checked it in and the patch was written by Greg Stein. Anyone remember motivations from the time? Cheers, M. -- In general, I'd recommend injecting LSD directly into your temples, Syd-Barret-style, before mucking with Motif's resource framework. The former has far lower odds of leading directly to terminal insanity. -- Dan Martinez

3 2

Re: [Python-Dev] seeing off SET_LINENO
by Michael Hudson 03 Aug '02

03 Aug '02

Tim Peters <tim.one(a)comcast.net> writes: > I've been telling people at Zope Corp that getting rid of SET_LINENO would > speed pystone (which is said to be a good predictor of Zope performance) by > at least 7%. If you can fudge up a test showing that, your performance work > will be complete. It's about 5%: $ ../../build/python pystone.py Pystone(1.1) time for 100000 passes = 8.11 This machine benchmarks at 12330.5 pystones/second $ ../../build/python pystone.py Pystone(1.1) time for 100000 passes = 7.69 This machine benchmarks at 13003.9 pystones/second I can run the vanilla pystone whilst compiling or something if you like :) As I said, my patched Python is much more variable in pystone than before. I'm going to try invoking the Cache Effect Demon on this one, unless someone can come up with a real explanation. > [Guido] > > ... > > What's the next step? I haven't had time to review your code. Do you > > want to check it in without further review, or do you want to wait > > until someone can give it a serious look? (Tim's on vacation this > > week so it might be a while.) > > I'm really not the best person for this, since, e.g., I never use the > debugger, so couldn't personally care less if it stopped working <0.9 wink>. > > The patch set looks very complete, so I'd encourage a checkin if nobody > objects. > > I have one objection, but it's kind of vague: Michael, you're taking too > much delight in how obscure this is! It's the old boys club effect: I worked damn hard to get to the point of understanding this stuff, so everyone else should bloody well have to too! > Two examples: > > + int instr_ub = -1, instr_lb = 0; /* for tracing */ > > It takes a lot of effort to reverse-engineer that the line number has > changed if and only if > > not instr_lb <= current_bytecode_offset < instr_ub > > -- or at least to reverse-engineer that this is what you believe <wink>. > Paste the above in as a comment and save the next person the pain. I got > hung up the first 5 minutes guessing that "lb" and "ub" referred to "lower > byte" and "upper byte". Ah, OK. Actually, taking the tracing code out of line makes me feel less uneasy about adding hundred+ line comments explaining what's going on. > The other example: > > + /* I (mwh) will gladly buy anyone a beer who > + can tell me off the top of their head why > + the exception for POP_TOP is needed... */ > > That's not going to be amusing two years from now when your unstated > reasoning is no longer true, and this code breaks. Then someone will have > to guess what you thought you meant by this comment, whether your reasoning > was correct at the time, and what may have changed to invalidate it. Rather > than tease, just explain why POP_TOP must be an exception. If you don't > know why, I'll buy *you* a beer <wink>. All I can say is that I'd been driven insane by co_lnotab and Python/compile.c when I wrote that comment <wink>. Cheers, M. -- I'm okay with intellegent buildings, I'm okay with non-sentient buildings. I have serious reservations about stupid buildings. -- Dan Sheppard, ucam.chat (from Owen Dunn's summary of the year)

2 1