Mailman 3 July 2000 - Python-Dev

Bookstand at LA Python conference
by Guido van Rossum July 29, 2000

July 29, 2000

The next Python conference will be in Long Beach (Los Angeles). We're looking for a bookstore to set up a bookstand like we had at the last conference. Does anybody have a suggestion? See http://www.python9.com for conference info! --Guido van Rossum (home page: http://www.pythonlabs.com/~guido/)

1 0

Re: [Python-checkins] CVS: python/dist/src/Lib SimpleHTTPServer.py,1.10,1.11
by Fredrik Lundh July 29, 2000

July 29, 2000

is it worth replacing a couple of lines of obvious code with a reference to a little-used module? and if I want to, how do I tweak the block size? </F> > *** SimpleHTTPServer.py 2000/05/21 16:25:29 1.10 > --- SimpleHTTPServer.py 2000/07/29 05:15:56 1.11 > *************** > *** 16,19 **** > --- 16,20 ---- > import urllib > import cgi > + import shutil > from StringIO import StringIO > > *************** > *** 152,161 **** > > """ > ! > ! BLOCKSIZE = 8192 > ! while 1: > ! data = source.read(BLOCKSIZE) > ! if not data: break > ! outputfile.write(data) > > def guess_type(self, path): > --- 153,157 ---- > > """ > ! shutil.copyfileobj(source, outputfile) > > def guess_type(self, path): > > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > http://www.python.org/mailman/listinfo/python-checkins

3 5

Re: [Python-Dev] extended slicing for lists
by Michael Hudson July 29, 2000

July 29, 2000

Michael Hudson <mwh21(a)cam.ac.uk> writes: > The patch is here, btw: > > http://sourceforge.net/patch/?func=detailpatch&patch_id=100998&group_id=5470 And has now been updated to support assignments & deletions. I'm happy with deletions, but assignments seem to be a bit counter-intuitive - for example both of Guido's examples: >>> l = range(10) >>> l[2:10:2] = [0]*5 >>> l [0, 1, 0, 3, 0, 5, 0, 7, 0, 9] >>> l = range(10) >>> l[4:-1:-1] = range(5) >>> l [4, 3, 2, 1, 0, 5, 6, 7, 8, 9] are wrong! >>> l[2:10:2] = [0]*5 Traceback (most recent call last): File "<stdin>", line 1, in ? ValueError: attempt to assign list of size 5 to extended slice of size 4 >>> l[4:-1:-1] = range(5) Traceback (most recent call last): File "<stdin>", line 1, in ? ValueError: attempt to assign list of size 5 to extended slice of size 0 So I'd really appreciate it if people could play around with the patch and see if (a) they can break it (b) they like the way assignments behave (I am already hooked on the indexing & deletion). now-all-we-need-is-a-slice-expression-for-sorting-ly y'rs M. -- 59. In English every word can be verbed. Would that it were so in our programming languages. -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html

1 0

extended slicing for lists
by Michael Hudson July 29, 2000

July 29, 2000

Michael Hudson <mwh21(a)cam.ac.uk> writes: > Guido van Rossum <guido(a)beopen.com> writes: > > > > Good idea. Are lists and tuples going to support seq[a:b:c] anytime > > > soon? > > > > It's on my wish list, but can't be done for 2.0. > > This would involve the list type implementing the mapping API wouldn't > it? This would seem to be a little on the silly side. Never mind; > here's a quick-hack patch that "seems to work" for lists: [snip] I've now put this (or a slightly less buggy version) onto sourceforge. Guido said (after I bothered him in private email) that he'd expect that: >>> l = range(10) >>> l[2:10:2] = [0]*5 >>> l [0, 1, 0, 3, 0, 5, 0, 7, 0, 9] >>> And also: >>> l = range(10) >>> l[4:-1:-1] = range(5) >>> l [4, 3, 2, 1, 0, 5, 6, 7, 8, 9] >>> (and presumably del l[a:b:c]) Now I'd expect *that* as well, but I was wondering what should happen if the lengths of the lists don't match up ... just bail? That would be easiest, but it doesn't quite gel well with current behaviour for simple slices. Presuming an error, what about these (assuming d is a list of the wrong length): l[a:b:-1] = d l[a:b:1] = d l[a:b:] = d I think these should probably be errors too, but I'd welcome any opinions people have. Cheers, M. -- A witty saying proves nothing. -- Voltaire

2 1

Fire, may I present accellerant. Accellerant, this is fire.
by Mitchell Morris July 29, 2000

July 29, 2000

In general I try to keep my head down. I recently needed some support for regression testing as part of a stealth Python introduction-cum-advocacy effort. I wound up using Steve Purcell's pyunit testing framework, and thought it was a shame that it wasn't part of the standard distribution. To that end ... PEP: eric-the-half-a-pep Title: Guido-blessed unit/regression test framework Version: 0 Target of Opportunity: mmorris(a)mindspring.com (Mitchell Morris) Python-Version: 2.0 Status: pre-pre-pre-Draft Introduction This proto-PEP advocates that there should be, in the standard library, a unit/regression test framework. This Guido-blessed framework would provide test semantics both useful and standardized. In addition to being a useful structure in it's own right, it would provide an obvious direction for development projects to prepare and use their own regression test policies. Why Should I Care? Software developers have known for quite some time that the majority of our time is spent identifying and correcting errors. Various suggestions for reducing this burden have been explored and most of them have been rejected. One thing that seems to provide some benefit is writing test plans and running them against the software product. This doesn't necessarily eliminate bugs, but it does allow us to find some of them and insure they don't reoccur as we correct later ones. One variation of this process is to extend the test plan as other defects are identified and corrected. Part of the correction would be updating the test suite to include a test that would have identified and exercised the defect. In this way the test plan grows to verify that correct behavior increases monotonically. This kind of test plan also allows very aggressive refactoring, in that the developer can make sweeping changes to module internals with great confidence that things will either (a) not break elsewhere, or (b) break loudly when the unit tests fail. Given the (relative) success of test plans and Python's general friendliness to rapid development and refactoring, there should be a standard tool in all distributions to encourage this dynamic. But ... Smalltalk and Java are Evil! Kent Beck has recently generated a lot of press by writing about his development methodology which he calls "Extreme Programming"[1]. Aside from the unfortunate connotations to sports which would still be completely anonymous without ESPN2, there are a number of useful tools emerging from the XP camp. One of the prime tenets is that developers write their classes and their unit tests at the same time. According to XP, these unit tests have to run at 100% for the code to be acceptable for integration. The design of the testing framework generated by XP is fairly simple: individual tests check the correct functionality of a single quantum of testable behavior, usually public behavior. These individual tests can be aggregated into suites, which are also runnable. Suites can be aggregated as well. Tools are provided to execute these tests and suites in both text and GUI formats. Kent Beck and Erich Gamma wrote a paper called "Test Infected"[2] to demonstrate how the code and tests are developed in tandem. This paper, written as a script for a pair programming session, shows how the various pieces of the JUnit implementation work together. (JUnit is the XP testing tool for Java.) We've Already Got One, Thanks. The current distribution has a regression testing framework, in Lib/test/regrtest.py, which addresses some of these same issues. The obvious question is why not use the tool which is already there? The current tool presents several issues that make this a less-than-exemplary solution: it doesn't allow for graphical manipulation, it separates test and expected result into separate files stored in separate locations, and it tries to generate the tests as well as run them. The objections to regrtest.py then boil down to the it's philosophy. We've Got More Than We Can Stand, Thanks. Really. Also for our consideration is doctest.py, contributed by the inestimable Tim Peters. doctest.py presents a simple straightforward means to embed the individual tests into the production code itself as commentary. This suggests that the separation of test and result presents a cognitive dissonance in other people besides just me. doctest.py doesn't provide any mechanism, however, for aggregating several modules into a single test run, nor does it provide a GUI. Once again, additional effort could obviously provide the missing functionality. On the other hand, doctest.py isn't currently part of the standard distribution. What's All This Then? The point of this proposal is Steve Purcell's pyunit package[3]. This is a Python implementation of XP's xUnit specification. It provides a means for running individual tests, aggregating individual tests into larger suites, running those tests in both text and GUI modes, and keeps the tests and expected results together in a single source module. You Have A Point? Yes: pyunit should be part of the standard library. pyunit documentation should be included in the distribution. New development should be encouraged to use pyunit. We should all rub pyunit all over our bodies as often as we can. Even if it isn't pyunit, Python should have a documented standard component for unit and regression testing, and we should encourage new development to use it, and we should demand that core development include it. References 1 -- XP Wiki <URL:http://c2.com/cgi/wiki?ExtremeProgramming> see also: <URL:http://www.xprogramming.com/> 2 -- Beck, Kent and Gamma, Erich, "Test Infected" <URL:http://members.pingnet.ch/gamma/junit.htm> 3 -- pyunit project at Source Forge <URL:http://pyunit.sourceforge.net/> pyromanically y'rs +Mitchell

1 0

Python License News
by Guido van Rossum July 29, 2000

July 29, 2000

Bob Weiner (BeOpen's CTO, my boss) told me that he's very close to reaching a compromise with CNRI about the new Python license. There's only one remaining issue before Richard Stallman is willing to declare the license GPL-compatible. We expect that this wrinkle will be worked out with CNRI's president Bob Kahn on Monday. Tim Peters knows more, you just have to ask the right way. :-) Gotta run, --Guido van Rossum (home page: http://www.pythonlabs.com/~guido/)

3 2

further optimising the micro-optimisations for cache locality (fwd)
by Vladimir.Marangozov＠inrialpes.fr July 28, 2000

July 28, 2000

FYI. It would be interesting to collect some feedback on these ideas for popular combos. Who knows... Forwarded message: > From morton(a)dennisinter.com Fri Jul 28 20:48:29 2000 > From: morton(a)dennisinter.com (Damien Morton) > To: <Vladimir.Marangozov(a)inrialpes.fr> > Subject: further optimising the micro-optimisations for cache locality > Date: Fri, 28 Jul 2000 14:34:29 -0400 > Message-ID: <NDBBLENPLCPAHKODFHNGOEADEBAA.morton(a)dennisinter.com> > MIME-Version: 1.0 > Content-Type: text/plain; > charset="iso-8859-1" > Content-Transfer-Encoding: 7bit > X-Priority: 3 (Normal) > X-MSMail-Priority: Normal > X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) > X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 > Importance: Normal > > Folded version: > > case BINARY_ADD: > w = POP(); > v = POP(); > x = PyNumber_Add(v, w); > goto end_binop; > > case BINARY_MULTIPLY: > w = POP(); > v = POP(); > x = PyNumber_Multiply(v, w); > goto end_binop; > > ... > end_binop: > Py_DECREF(v); > Py_DECREF(w); > PUSH(x); > if (x != NULL) continue; > break; > > Further folded version: > > > > case BINARY_POWER: > f = PyNumber_Add; > goto do_binop; > > case BINARY_MULTIPLY: > f = PyNumber_Multiply; > goto do_binop; > > ... > do_binop: > w = POP(); > v = POP(); > x = f(v, w); > Py_DECREF(v); > Py_DECREF(w); > PUSH(x); > if (x != NULL) continue; > break; > > Further Further folded version: > (i havent looked at the mainloop code, but can guess its structure) > > i assume 'op' is the opcode we are swtching on > > // the function to call for this op > (void *)() op_func[] = { ..., PyNumber_Add, PyNumber_Multiply, ... }; > > // the kind of op this func describes > unsigned char op_type[] = { ..., DO_BINOP1, DO_BINOP2, DO_UNOP, ... }; > > // these two tables will be cached because of the frequency the are accessed > // ive used a table of pointers and a table of bytes to reduce the memory > required > // because the tables are small, locality within and between the tables isnt > important > // might be an idea to force the tables into contiguous memory somehow > // i could also have used a table of structs, but alignment would increase > memory usage > > unsigned char op; > > switch(op_type[op]) { > case DO_BINOP1: > w = POP(); > v = POP(); > x = op_func[op](v, w); > Py_DECREF(v); > Py_DECREF(w); > PUSH(x); > if (x != NULL) continue; > break; > case DO_BINOP2: > w = POP(); > v = POP(); > x = op_func[op](v, w, Py_None); > Py_DECREF(v); > Py_DECREF(w); > PUSH(x); > if (x != NULL) continue; > break; > case DO_UNOP: > v = POP(); > x = op_func[op](v); > Py_DECREF(v); > PUSH(x); > if (x != NULL) continue; > break; > > ===================== original message ================================= > Guido van Rossum wrote: > > > > > [me, on micro-optimizing the main loop] > > > > Fair enough (for one particular CPU at least). > > Since we're at it, it's worth mentioning another conclusion we came across > at the time: the cache effects in the main loop are significant -- it is > more important to try keeping at best the main loop small enough, so that > those effects are minimized. > > An experiment I did at the time which gave some delta-speedup: > I folded most of the UNOP & BINOP opcodes since they differ only by the > functon they call and they duplicate most of the opcode body. Like this: > > Original: > case BINARY_POWER: > w = POP(); > v = POP(); > x = PyNumber_Power(v, w, Py_None); > Py_DECREF(v); > Py_DECREF(w); > PUSH(x); > if (x != NULL) continue; > break; > > case BINARY_MULTIPLY: > w = POP(); > v = POP(); > x = PyNumber_Multiply(v, w); > Py_DECREF(v); > Py_DECREF(w); > PUSH(x); > if (x != NULL) continue; > break; > ... > > > Folded version: > > case BINARY_POWER: > w = POP(); > v = POP(); > x = PyNumber_Power(v, w, Py_None); > goto end_binop; > > case BINARY_MULTIPLY: > w = POP(); > v = POP(); > x = PyNumber_Multiply(v, w); > goto end_binop; > > ... > end_binop: > Py_DECREF(v); > Py_DECREF(w); > PUSH(x); > if (x != NULL) continue; > break; > > > This reduced the code size of ceval.c, which resulted in less cache effects > and gave more speed, despite the additional jumps. It possibly results in > less page-faults too, although this is unlikely. > > Which makes me think that, if we want to do something about cache effects, > it is probably not a bad idea to just "reorder" the bytecodes in the big > switch by decreasing frequency (we have some stats about this -- I believe > Skip and MAL have discussed the opcodes' frequency and the charts lie > somewhere in the archives). I remember Marc-Andre had done something in > this direction and reported some perf improvements too. Since reordering > the opcodes doesn't really hurt, if I'm about to do something with the > main loop, it'll be only this. > > > > > > Micro-optimization doesn't worth the effort. As to the 2-byte arg limit, > > > I think I'd be happy if the compiler raises an exception if this limit > > > is exceeded. > > > > That would be a good first step indeed! > > I'll try to have a look, if someone else is not more reactive than me. > > -- > Vladimir MARANGOZOV | Vladimir.Marangozov(a)inrialpes.fr > http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252 > -- Vladimir MARANGOZOV | Vladimir.Marangozov(a)inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252

2 2

build problems under MSVC 5.0
by Fredrik Lundh July 28, 2000

July 28, 2000

Python\thread_nt.h(185) : error C2065: 'INT_PTR' : undeclared identifier Python\thread_nt.h(185) : error C2146: syntax error : missing ';' before identifier 'rv' Python\thread_nt.h(185) : error C2065: 'rv' : undeclared identifier Python\thread_nt.h(186) : error C2143: syntax error : missing ';' before 'type' Python\thread_nt.h(195) : error C2065: 'success' : undeclared identifier in MSVC 5.0, _beginthread returns an "unsigned long" (this is also how it's documented in MSDN). I suggest changing the declaration in thread_nt.h to: #if _MSC_VER >= 1200 INT_PTR rv; #else unsigned long rv; #endif on the other hand, the same docs claim that it returns -1 on errors. doesn't exactly look like an unsigned long to me, but that's another story... </F>

2 3

RE: [Python-Dev] UTF-16 code point comparison
by Bill Tutt July 28, 2000

July 28, 2000

> From: Tim Peters [mailto:tim_one@email.msn.com] > [Tim] > > ... Don't know how long it will take this half of the world to > > realize it, but UCS-4 is inevitable. > > [Bill Tutt] > > On new systems perhaps, but important existing systems (Win32, > > and probably Java) are stuck with that bad decision and have to > > use UTF-16 for backward compatability purposes. > Somehow that doesn't strike me as a good reason for Python to mimic them > <wink>. So don't. If you think UTF-16 is yet another bad engineering decision, then take the hit now of making Python's unicode support natively UCS-4 so we don't have a backward compatability problem when the next Unicode or ISO 10646 revision comes out. Just realize and accept the cost of doing so. (constant conversions for a nice big chunk of your users.) > > Surrogates aren't as far out as you might think. (The next rev of > > the Unicode spec) > But indeed, that's the *point*: they exhausted their 64K space in just a > few years. Now the same experts say that adding 4 bits to the range will > suffice for all time; I don't buy it; they picked 4 bits because that's what > the surrogate mechanism was defined earlier to support. I don't think the experts are saying the extra 4 bits will suffice for all time, but it should certainly suffice until we meet aliens form a different planet. :) > > That's certainly sooner than Win32 going away. :) > I hope it stays around forever -- it's a great object lesson in what > optimizing for yesterday's hardware can buy you <wink>. Heh. A dev manager from Excel made the exact same comment to me just yesterday. :) Bill

1 0

Re: [Patch #100740] gzip: readline(), readlines() patch
by Andrew Kuchling July 28, 2000

July 28, 2000

On Fri, Jul 28, 2000 at 03:25:36AM -0700, noreply(a)sourceforge.net wrote: >Patch #100740 has been updated. >gvanrossum wrote: >Almost right. The docs for file objects specifically imply that a >negative arg to readline() or a zero arg to readlines() is the same as >no args; your defaults are sys.maxint so an explicit -1 or 0 won't do >the right thing. (This is important for wrapper classes that want to Which docs say this? I looked at Doc/lib/libstdtypes.tex to get the semantics: \begin{methoddesc}[file]{readline}{\optional{size}} ... If the \var{size} argument is present and non-negative, it is a maximum byte count (including the trailing newline) and an incomplete line may be returned. ... \end{methoddesc} \begin{methoddesc}[file]{readlines}{\optional{sizehint}} ... If the optional \var{sizehint} argument is present, instead of reading up to \EOF{}, whole lines totalling approximately \var{sizehint} bytes (possibly after rounding up to an internal buffer size) are read. \end{methoddesc} Neither text mentions the special cases for -1 and 0 in each function. Fixing the patch to get this right is trivial, but I think the docs need to be updated to mention this. --amk

2 1