Mailman 3 September 2002 - Python-Dev

Patch to make cgi.FieldStorage iterate over its keys
by Hamish Lawson 11 Sep '02

11 Sep '02

Below is a patch to make cgi.FieldStorage iterate over its keys, allowing it to behave like any other dictionary in this kind of construct: form = cgi.FieldStorage() for key in form: do something ... Hamish Lawson --- Compare: (<)E:\Python22\Lib\cgi.py (34894 bytes) with: (>)E:\temp\cgi.py (34955 bytes) 524a524,526 > def __iter__(self): > return iter(self.keys()) >

2 1

Re: [Python-checkins] python/dist/src/Modules _hotshot.c,1.26,1.27
by Michael Hudson 11 Sep '02

11 Sep '02

"Raymond Hettinger" <python(a)rcn.com> writes: > Houston, we have a problem: > > C:\py23\Modules\_hotshot.c(891) : error C2198: 'pack_lineno_tdelta' : too few actual parameters > C:\py23\Modules\_hotshot.c(892) : error C2059: syntax error : ')' Yikes! Fixed. I was sure I checked everything before checking in... Cheers, M. -- Imagine if every Thursday your shoes exploded if you tied them the usual way. This happens to us all the time with computers, and nobody thinks of complaining. -- Jeff Raskin

1 0

Re: raw headers in rfc822.Message
by Michael Chermside 11 Sep '02

11 Sep '02

Raymond Hettinger writes: > FAQs and pointers be darned; there is only one way to > developerhood and that is through the school of hard knocks. > > [Terrific course catalog for said school elided] Raymond, I had to laugh at your "course catalog"... very funny but true, all of it very true. It made it into my long-term list of bookmarks not to lose. But it's worth noting that, although the school of hard knocks may be required by life itself (and the nature of programming), anytime that we can just add a link to a web page and perhaps allow someone to skip a course, it's a win all around. Anyway, thanks for bringing some humor into my morning. -- Michael Chermside

2 1

utf-8 issue thread question
by Brett Cannon 11 Sep '02

11 Sep '02

So here is the summary question for this thread: what exactly is a surrogate? I think I get it (from reading a l18n email from MAL on the l18n list), but I am not confident enough to stick in the summary as of yet. The following is my current rough summary explanation for what a surrogate is. Can someone please correct it as needed? """ In Unicode, a surrogate is when you encode from a higher bit total encoding (such as utf-16) into a smaller bit total encoding by representing the character as several more bit chunks (such as two utf-8 chunks). The following line is an example: >>> u'\ud800'.encode('utf-8') == '\xed\xa0\x80' Notice how the initial Unicode character ends up being encoded as three characters in utf-8. """ Also, anyone know of some good Unicode tutorials, explanations, etc. on the web, in book form, whatever? Most of the threads that I don't totally comprehend are Unicode related and I would like to minimize my brain-dead questions to a minimum. Don't want my reputation to go down the drain. =) -Brett

4 4

Re: raw headers in rfc822.Message
by Michael Chermside 11 Sep '02

11 Sep '02

Zack Weinberg writes: > You've generated this patch backward, and in a format which makes it > useless to us. Please regenerate it with diff -c or diff -u (either > is acceptable) and put the newer file _second_ on the command line: > diff -u OLD_FILE NEW_FILE. It wasn't all that long ago that I submitted my first patch (of documentation, not code) to SourceForge. It took me > 20 minutes of careful web searching to figure out the desired way of submitting files and the correct way to generate that. And I still wasn't 100% sure I was generating the diff in the correct direction. Couldn't Zack's comment be added to the directions found at https://sourceforge.net/tracker/?func=add&group_id=5470&atid=305470 so that anyone submitting a patch would see how to do it. (But of course that wouldn't have helped THIS person, who didn't use sourceforge... :-( ) -- Michael Chermside

8 9

Python-dev summary for 2002-08-15 - 2002-09-01
by Brett Cannon 10 Sep '02

10 Sep '02

Yes, with Michael's permission, I am attempting to start up the Python-dev summaries again. Below is my attempt at summarizing the last half of August. It's longer then normal summaries, but that is because I bothered to include discussions on threads that were not directly relating to the Python core but are interesting nonetheless (e.g., the whole spambayes thread). I am posting to Python-dev first before posting to c.l.py, c.l.py.a (also lwn.net and probably Slashdot) because I want to get the general okay from the list that I have done a good enough of a job to send this out; I don't want to have a summary that represents the going-ons here without the general populace (or just the BDFL since he can overrule =) being okay with it. I am also curious as to whether I should go into more or less detail, leave out the summaries that do not directly pertain to the Python core, etc. So please read the summary and let me know if you are okay with it. If so I will try to do semi-monthly summaries from now on. Oh, and I am on vacation right now and will be doing a lot of travelling in the next two months, so I can't guarantee summaries will be this quick to come out for a while. I will do them, though, even if they are a week late. =) Oh, and if I do get the okay to do this, expect a lot of dumb questions from me in the future in terms of clarifying things. Just remember, it is for the good of the Python community. =) ======================================= This is a summary of traffic on the python-dev mailing list between August 16, 2002 and September 1, 2002 (exclusive). It is intended to inform the wider Python community of ongoing developments. To comment, just post to python-list(a)python.org or comp.lang.python in the usual way. Give your posting a meaningful subject line, and if it's about a PEP, include the PEP number (e.g. Subject: PEP 201 - Lockstep iteration) All python-dev members are interested in seeing ideas discussed by the community, so don't hesitate to take a stance on a PEP if you have an opinion. This is the first summary written by Brett Cannon. Summaries are archived no where at the moment. =) They will be, though, so stay tuned for the URL in future summaries. Posting distribution (with apologies to mbm, but thanks to mwh for the code) Number of articles in summary: 585 80 | [|] | [|] | [|] | [|] | [|] [|] 60 | [|] [|] [|] | [|] [|] [|] | [|] [|] [|] | [|] [|] [|] | [|] [|] [|] [|] 40 | [|] [|] [|] [|] [|] | [|] [|] [|] [|] [|] [|] [|] | [|] [|] [|] [|] [|] [|] [|] [|] | [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] | [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] 20 | [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] | [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] | [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] | [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] | [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] [|] 0 +-071-025-012-042-063-084-030-021-039-009-047-027-033-041-036-005 Fri 16| Sun 18| Tue 20| Thu 22| Sat 24| Mon 26| Wed 28| Fri 30| Sat 17 Mon 19 Wed 21 Fri 23 Sun 25 Tue 27 Thu 29 Sat 31 ================ Type Categories ================ This VERY long thread was sparked by Andrew Koenig asking if a discussion of making type categories more explicit had ever occured (Andrew meant for category to mean "the set of all types that implement a particular marker interface"). As Andrew later pointed out, he was asking about "a way of making notions such as 'file-like object' more formal and/or automatic". The discussion quickly started using the term interface to mean defining a way to specify that an object implemented certain methods (think of it in terms of Java's 'implements' mechanism). Once that was out of the way, the discussion took off. Zope's implementation was pointed out (http://cvs.zope.org/Zope3/lib/python/Interface/) very quickly. PEP 245 (Python Interface Syntax) was also brought to the attention of the list. The idea of using inheritance to handle interfaces was brought up. Guido said that he hasn't "given up the hope that inheritance and interfaces could use the same mechanisms. But Jim Fulton, based on years of experience in Zope, claims they really should be different" in terms of how interfaces should be handled in objects. Jeremy Hylton tried to channel Jim's opinion by pointing out that "We'd like to use interfaces to make fairly strong claims. If a class A implements an interface I, then we should be able to use an instance of A anywhere that an I is needed." But "the inheritance mechanism is too general" because if a class A implements interface I and then a class B, which does not implement I, subclasses class A we end up with a class B that claims it has a certain interface which it doesn't actually have. Guido understood the point, but still thought inheritence could be used "if there was a way to "shut off" inheritance as far as isinstance() (or issubclass())" is concerned. Guido asked the simple question, "Why do keep arguing for inheritance? (a) the need to deny inheritance from an interface, while essential, is relatively rare IMO, and in *most* cases the inheritance rules work just fine; (b) having two separate but similar mechanisms makes the language larger." Samuele Pedroni asked that any implementation "allow also for refering to anonymous super-interfaces of an interface in terms of the interface plus a subset of its signatures, also e.g. FileLike and just 'write'. [that means an interface can be thought to correspond to a set of (tag,signature) tuples, where tag identifies the interface, and one can also just consider subsets of it]". The thread has finally seemed to have stopped (for now) with Guido saying he is mulling the whole thing in the back of his head. This is a very sticky topic because of the number of design decisions required and how it might change the way people program in Python. There was also a partial sub-thread in this whole discussion about multimethods; basically a way to do overloading of methods based on parameter signature. Most of the discussion was over syntax and such and how to handle resolution order. It then seemed to go to the wayside when the main part of the thread took over again. ============================== type categories -- an example ============================== This thread was starteed when Andrew Koenig said that the reason he brought up his type category question was because he wanted a way so as to be able to identify members of a type easily. He now had an example in a program he was writing where what the type of the argument was varied and thus what needed to be done to the data changed accordingly. Jermey Hylton suggested the isinstance(obj, type(re.compile(''))) idiom. Andrew asked if this was guaranteed to work, which Jeremy said no. I asked why this was not guaranteed, and Frederick Lundh said because re.compile() is a factory fxn and it is possible that a future version could return a different object based on the pattern. =============================================== Python build trouble with the new gcc/binutils =============================================== Andrew Koenig said that he couldn't compile Python using the newest gcc (this was the day after the latest release hit servers). With help from Zack Weinberg of Code Sourcery (who also recently rewrote the tempfile module), the problem was tracked down to binutils 2.13. being the culprit and was not Python's fault. =================================== Last call: mortal interned strings =================================== The patch python.org/sf/576101 removes the default immortality of interned strings. I believe it was in early August (possibly spilled over from late July) when Oren Tirosh proposed the idea and wrote the above mentioned patch. There had been some discussion over whether any 3rd party code was reliant upon interned strings being immortal; none was found (MacPython was reliant upon it, but since it is under Python core control it was considered a moot point since it could be changed). It has been checked in. With the patch the way to make a string immortal is to call PyString_InternImmortal(); no code in the core uses this function. ===================================== PEP 218 (sets); moving set.py to Lib ===================================== Thanks to Greg Wilson (for writing the PEP), Alex Martelli (for writing the module initially), and Guido (for refactoring Alex's code) the stdlib has now gained a sets module. It has both the notion of mutable and immutable sets (the latter used when you have a set of sets). There was discussion about how sets should print (sorted or not; unsorted is default but option is there to print sorted) and what operators should be overloaded for working on sets (| and & were chosen). The module is a beautiful chunk of code and I highly recommend reading its source. =========================================== A few lessons from the tempfile.py rewrite =========================================== Zack Weinberg, after rewriting the tempfile module, brought up three points: 1) Lack of dummy threads, 2) lack of a pthreads_once equivalent, and 3) lack of a way to skip tests from unittest.py via some built-in method. Guido responded accordingly: 1) since some code uses the idiom of trying to import thread and catching the exception if it fails, Guido said he would be willing to accept a dummy_thread.py that would allow: try: import thread as _thread except ImportError: import dummy_thread as _thread to work. No word on whether this is being written at the moment. 2) Guido said the method was, in his opinion, overkill. He said to "be Pythonic, live dangerously, accept the risk that a ^C can screw you. It can anyway. :-)". And as for 3) Guido deferred Zack to the PyUnit list and Steve Purcell since Python just tracks Steve's code (pyunit.sf.net). Guido's suggestion was to stick code that was reliant on some other code in a separate testing suite that is only run when the reliant code is available. =========================== Standard datetime objects? =========================== Kevin Jacobs asked what stage the new datetime object was at. Guido said it is in python/nondist/sandbox/datetime/ in CVS which also has comments pointing to a wiki containing the current work on it. Fred L. Drake, Jr. is working on the C re-implementation and Guido expects a checkin at any moment (hasn't happened as of this writing). =================== PEP 269 versus 283 =================== Jonathan Riehl noticed that PEP 283 said PEP 269 was dead; not good considering he was close to having a patch for PEP 269 (pgen module to interface with the C version). Guido said he will revive the PEP. The patch has since been put on SF at python.org/sf/599331 . ============================== What is a backport candidate? ============================== Since Python 2.2 is going to be around for a long time, the question was brought up of what constitutes code that should be backported. Guido made the following three points: 1) code trivial to backport should always be backported 2) code patcheing 2.3 code should obviously not be backported 3) 2.2 code requires changes to use patch, but applies; gradients of this exist. So please, when submitting patches, mention whether you think the patch should be backported to the 2.2 tree and any possible dependencies it might have in a backport. ================================= python/nondist/sandbox/spambayes ================================= In response to Paul Graham's spam filter written using Baye's Rule (Slashdot post on it is at http://developers.slashdot.org/article.pl?sid=02/08/16/1428238&tid=156), a thread spawned around this checkin of code that followed that paper's suggestions. This thread quickly jumped into discussions on data structures, Baye's Rule, and a whole lot of talk about spam. Very interesting if spam filtering interests you. Tim Peters has been leading the drive on this chunk of code (and thanks to his illness that befelled him in late August which he has subsequently gotten over he had a few days of major hacking on it; Tim showed he is a performance stats whore <wink>). A very cool quote came out of this thread from Eric S. Raymond when discussing the spam filter he has been working on: "This is actually the first new program I've coded in C (rather than Python) in a good four years or so". ==================== Parsing vs. lexing. ==================== In response to a question by Aahz about what the differences were between a lexer, parser, and tokenizer, Eric Raymond posted a good overview of the differences. Guido later commented in an email mentioning SPARK and about how Python's lexer (pgen) works and why he wrote it. He also made some other comments on lexers. Jeremy Hylton pointed out a "neat new paper about an old algorithm for recursive descent parsers with backtracking and unlimited lookahead" by Bryan Ford at http://www.brynosaurus.com/pub.html . Alex Martelli pointed out that this discussion reminded him of "a long-ago interview with Borland's techies" in which they said they were able to make Borland PASCAL fit on a floppy while MS PASCAL took multiple floppies. Their trick was "we just did everything by the Dragon Book -- except that the parser is a hand-written recursive descent parser [Aho &c being adamant defenders of Yacc & the like], which buys us a lot". Someone named Noah also emailed a discussion on lexers and parsers pulling in Finite State Machines, Push Down Autonoma, and Turing Machines in his discussion. Martin Sj?n says that Haskell's pattern matching and lazy evaluation makes lexers easy (even a Recursive-Descent parser), but unfortunately Haskell does not play with other languages nicely. Haskell is where Python got it's list comprehension idea. ========================================= [Python-Dev] Fw: Security hole in rexec? ========================================= It was brought to the attention of the list that deleting __builtins__ allowed a compromise in rexec. Guido pointed out that python.org/sf/577530 reports this. He also said don't trust rexec. A patch is going to be submitted to document the view that rexec is really not that safe. ================= A `cogen' module ================= Francois Pinard asked about Cartesian products using the new sets module. Guido didn't think people would in general need it. Francois quickly started this thread of discussing a cogen module to generate Cartesian products and other ways of operating on sets. ================= Mersenne Twister ================= Raymond Hettinger volunteered to implement the Merseene Twister algorithm (one in Python exists at www.math.keio.ac.jp/~matumoto/emt.html). While discussing to implement in C or Python, Guido noticed that random.Random re-implements whrandom. Guido then came up with the idea of writing a base random class that is subclassed where .random() can be implemented; Tim Peters agreed and suggested more methods to subclass. ================================= New PEP Format: reStructuredText ================================= David Goodger and Barry Warsaw have now gotten reST as a usable syntax for PEPs. Read the PEPs on the subject to learn more: - PEP 12 -- Sample reStructuredText PEP Template (http://www.python.org/peps/pep-0012.html) - PEP 258 -- Docutils Design Specification (http://www.python.org/peps/pep-0258.html) - PEP 287 -- reStructuredText Docstring Format (http://www.python.org/peps/pep-0287.html) ==================================== tiny optimization in ceval mainloop ==================================== Jeremy Hylton noticed that in ceval that their is a test of whether the ticker was 0 or if things_to_do was set to true (explanation of the ticker, checkinterval, and the GIL follow this paragraph). Jeremy wondered if we could just drop the ticker to 0 when things_to_do is true. Jack Janssen, though, pointed out that clearing it is not guaranteed since there may be an interrupt routine when "we fiddle things_to_do". Skip Montanaro then pointed out that since neither ticker nor things_to_do is fiddled with unless the GIL is held that instead of causing each thread to execute this test that they could be made globals instead; he did a patch that implements this (python.org/sf/602191). Guido then said that if there wasn't a decent speed improvement, then no patch would be checked in. He then changed his mind when it was pointed out that it actually simplified the code. Skip tested anyway, though, and there is a speed improvement. This also brought up whether the default value of 10 for checkinterval was reasonable. It was then agreed to be bumped up to 100. Jack ran some code and said he noticed a definite improvement. Python's version of threading is not like in C. There is something called the GIL (Global Interpreter Lock) which any thread wishing to execute Python code or play with Python objects must hold. This means that when you have Python threads running (using the thread or threading module) they are usually all waiting in line to get the GIL. Now for Python to decide when to release the GIL for another thread to grab it, it uses the ticker. This variable counts down to zero by being decremented every time a Python opcode is executed (originally defaulted to 10, now defaulted to 100). The ticker's starting value after each release of the GIL is what sys.checkinterval() sets. To get a better understanding of therading under Python I recommend reading Aahz's tutorials on threading.

13 31

Codecs lookup order
by pinard＠iro.umontreal.ca 10 Sep '02

10 Sep '02

Hi, people. Happily playing with codecs (using Python 2.2.1), I found out that one should be careful about _not_ naming a module after the encoding name, when closely following the documentation in the Library Reference manual. Here is what I guess is happening. `codecs.register()' appends the search function from the new codec module at end of existing search functions. `codecs.lookup()' tries the search functions in the same order in which they were declared. Consequently, `encodings.lookup()' is tried first. If the encoding does not exist in the cache, `encodings.lookup()' tries to import a module by the name of the encoding, slightly transformed, and will indeed import the new user codec module, because that module has the name of the encoding, and is on the module search path. But now, `encodings.lookup()' expects a `getregentry' function in that module, does not find it, and raises a CodecRegistryError, not leaving a chance to subsequent codec search functions to be used. On the user side, a mere renaming the user module holding the new codec solves the problem. I'm not sure what should best be done. The documentation might be modified to explain the limitation, so other users do not trip up on it. `encoding.lookup()' might merely return None in case `getregentry' is not defined in the imported module, or else, it could make sure that it imports modules exclusively from within the `encodings' package. The best and simplest might be to lookup the code search functions in reverse order of their registration. `encoding.lookup()' would be called last instead of first. It would be easier for the user to override an encoding bundled with the Python distribution, if there is a need to do so. Because the Python Library Reference does not specify yet in which order codec search functions are tried, the order is not frozen yet and it might be easier to change it. -- François Pinard http://www.iro.umontreal.ca/~pinard

2 3

Subsecond time stamps
by loewis＠informatik.hu-berlin.de 10 Sep '02

10 Sep '02

A number of systems provide subsecond time stamp resolution for files. In particular: - NFS v3 has nanosecond time stamps. - Solaris 9 has nanosecond time stamps in stat(2), and microsecond time stamps in utimes(2). In addition, they have microsecond time stamps on ufs. It appears that other Unices have also extended stat(2), as does OS X. - NTFS has 100ns resolution for time stamps. I'd like to expose atleast the stat extensions to Python. Adding new fields to stat_result is easy enough, but there are a number of alternatives: A. Add an additional field to hold the nanoseconds, i.e. st_mtimensec, st_atimensec, st_ctimensec. This is the BSD Posix extension. B. Follow the Unix API (Solaris and others). They define a struct timespec_t { time_t tv_sec; unsigned long tv_nsec; }; and fields st_mtim, st_ctim, st_atim of timespec_t. For compatibility, they #define st_mtime st_mtim.tv_sec So to get at the seconds, you can write either st_mtim.tv_sec, or st_mtime. For the nanoseconds, you write st_mtim.tv_nsec. This requires to add a new type. C. Make st_mtime a floating point number. This won't offer nanosecond resolution, as C doubles are not dense enough. What do you think? Regards, Martin

9 17

Re: RE: [Python-Dev] The first trustworthy <wink> GBayes results
by Paul Graham 10 Sep '02

10 Sep '02

Don't count words multiple times, and you'll probably get fewer false positives. That's the main reason I don't do it-- because it magnifies the effect of some random word like water happening to have a big spam probability. (Incidentally, why so high? In my db it's only 0.3930784.) --pg --Tim Peters wrote: > FYI. After cleaning the blatant spam identified by the classifier out of my > ham corpus, and replacing it with new random msgs from Barry's corpus, the > reported false positive rate fell to about 0.2% (averaging 8 per each batch > of 4000 ham test messages). This seems remarkable given that it's ignoring > headers, and just splitting the raw text on whitespace in total ignorance of > HTML & MIME etc. > > 'FREE' (all caps) moved into the ranks of best spam indicators. The false > negative rate got reduced by a small amount, but I doubt it's a > statistically significant reduction (I'll compute that stuff later; I'm > looking for Big Things now). > > Some of these false positives are almost certainly spam, and at least one is > almost certainly a virus: these are msgs that are 100% base64-encoded, or > maximally obfuscated quoted-printable. That could almost certainly be fixed > by, e.g., decoding encoded text. > > The other false positives seem harder to deal with: > > + Brief HMTL msgs from newbies. I doubt the headers will help these > get through, as they're generally first-time posters, and aren't > replies to earlier msgs. There's little positive content, while > all elements of raw HTML have high "it's spam" probability. > > Example: > > """ > --------------=_4D4800B7C99C4331D7B8 > Content-Description: filename="text1.txt" > Content-Type: text/plain; charset=ISO-8859-1 > Content-Transfer-Encoding: quoted-printable > > Is there a version of Python with Prolog Extension?? > Where can I find it if there is? > > Thanks, > Luis. > > P.S. Could you please reply to the sender too. > > > --------------=_4D4800B7C99C4331D7B8 > Content-Description: filename="text1.html" > Content-Type: text/html > Content-Transfer-Encoding: quoted-printable > > <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN"> > <HTML> > <HEAD> > <TITLE>Prolog Extension</TITLE> > <META NAME=3D"GENERATOR" CONTENT=3D"StarOffice/5.1 (Linux)"> > <META NAME=3D"CREATED" CONTENT=3D"19991127;12040200"> > <META NAME=3D"CHANGEDBY" CONTENT=3D"Luis Cortes"> > <META NAME=3D"CHANGED" CONTENT=3D"19991127;12044700"> > </HEAD> > <BODY> > <PRE>Is there a version of Python with Prolog Extension?? > Where can I find it if there is? > > Thanks, > Luis. > > P.S. Could you please reply to the sender too.</PRE> > </BODY> > </HTML> > > --------------=_4D4800B7C99C4331D7B8--""" > """ > > Here's how it got scored: > > prob = 0.999958816093 > prob('<META') = 0.957529 > prob('<META') = 0.957529 > prob('<META') = 0.957529 > prob('<BODY>') = 0.979284 > prob('Prolog') = 0.01 > prob('<HEAD>') = 0.97989 > prob('Thanks,') = 0.0337316 > prob('Prolog') = 0.01 > prob('Python') = 0.01 > prob('NAME=3D"GENERATOR"') = 0.99 > prob('<HTML>') = 0.99 > prob('</HTML>') = 0.989494 > prob('</BODY>') = 0.987429 > prob('Thanks,') = 0.0337316 > prob('Python') = 0.01 > > Note that '<META' gets penalized 3 times. More on that later. > > + Msgs talking *about* HTML, and including HTML in examples. This one > may be troublesome, but there are mercifully few of them. > > + Brief msgs with obnoxious employer-generated signatures. Example: > > """ > Hi there, > > I am looking for you recommendations on training courses available in the UK > on Python. Can you help? > > Thanks, > > Vickie Mills > IS Training Analyst > > Tel: 0131 245 1127 > Fax: 0131 245 1550 > E-mail: vickie_mills(a)standardlife.com > > For more information on Standard Life, visit our website > http://www.standardlife.com/ The Standard Life Assurance Company, Standard > Life House, 30 Lothian Road, Edinburgh EH1 2DH, is registered in Scotland > (No SZ4) and regulated by the Personal Investment Authority. Tel: 0131 225 > 2552 - calls may be recorded or monitored. This confidential e-mail is for > the addressee only. If received in error, do not retain/copy/disclose it > without our consent and please return it to us. We virus scan all e-mails > but are not responsible for any damage caused by a virus or alteration by a > third party after it is sent. > """ > > The scoring: > > prob = 0.98654879055 > prob('our') = 0.928936 > prob('sent.') = 0.939891 > prob('Tel:') = 0.0620155 > prob('Thanks,') = 0.0337316 > prob('received') = 0.940256 > prob('Tel:') = 0.0620155 > prob('Hi') = 0.0533333 > prob('help?') = 0.01 > prob('Personal') = 0.970976 > prob('regulated') = 0.99 > prob('Road,') = 0.01 > prob('Training') = 0.99 > prob('e-mails') = 0.987542 > prob('Python.') = 0.01 > prob('Investment') = 0.99 > > The brief human-written part is fine, but the longer boilerplate sig is > indistinguishable from spam. > > + The occassional non-Python conference announcement(!). These are > long, so I'll skip an example. In effect, it's automated bulk email > trying to sell you a conference, so is prone to use the language and > artifacts of advertising. Here's typical scoring, for the TOOLS > Europe '99 conference announcement: > > prob = 0.983583974285 > prob('THE') = 0.983584 > prob('Object') = 0.01 > prob('Bell') = 0.01 > prob('Object-Oriented') = 0.01 > prob('**************************************************************') = > 0.99 > prob('Bertrand') = 0.01 > prob('Rational') = 0.01 > prob('object-oriented') = 0.01 > prob('CONTACT') = 0.99 > prob('**************************************************************') = > 0.99 > prob('innovative') = 0.99 > prob('**************************************************************') = > 0.99 > prob('Olivier') = 0.01 > prob('VISIT') = 0.99 > prob('OUR') = 0.99 > > Note the repeated penalty for the lines of asterisks. That segues into the > next one: > > + Artifacts of that the algorithm counts multiples instances of "a word" > multiple times. These are baffling at first sight! The two clearest > examples: > > """ > > > Can you create and use new files with dbhash.open()? > > > > Yes. But if I run db_dump on these files, it says "unexpected file type > > or format", regardless which db_dump version I use (2.0.77, 3.0.55, > > 3.1.17) > > > > It may be that db_dump isn't compatible with version 1.85 databse files. I > can't remember. I seem to recall that there was an option to build 1.85 > versions of db_dump and db_load. Check the configure options for > BerkeleyDB to find out. (Also, while you are there, make sure that > BerkeleyDB was built the same on both of your platforms...) > > > > > > > Try running db_verify (one of the utilities built > > > when you compiled DB) on the file and see what it tells you. > > > > There is no db_verify among my Berkeley DB utilities. > > There should have been a bunch of them built when you compiled DB. I've got > these: >

5 6

64-bit process optimization 1
by ledwith＠cas.org 10 Sep '02

10 Sep '02

Hello, This is my first post to Python-Dev. As requested by the list manager I am supplying a brief personal introduction before getting to the topic of this message: I am a Senior Research Scientist at <A http://www.cas.org>CAS</A>, a branch of the <A http://www.acs.org>American Chemical Society</A>. I have used Python as my programming language of choice for the last four years. I typically work with large collections of text documents performing analyses of text, computer indexing of text, and information retrieval. I use Python as (1) a general purpose programming language, and (2) a high-level programming language to invoke high-performance C and C++ modules (including Numeric). If I examine my programs by data structures, I would find that they contain mostly: 1. Very large dictionaries using tuples and strings as keys. Guido's essay on <a href=http://www.python.org/doc/essays/graphs.html>Implementing Graphs</A> was the inspiration for my using dictionaries to create very large directed acyclic graphs. 2. Specialized C++ objects to represent inverted lists. 3. Numeric objects for representing vectors and tables of floating point values. My primary computing platforms are four dedicated Sun servers, containing 30 processors, 88GB of RAM and 2TB of DASD. Most of the programs I write require between 1 hour and 27 days to complete. (Obviously, I am an atypical Python user!) During the last three months, I have been forced to migrate from 32-bit python processes to 64-bit processes due to the large number of data points I am analyzing within a single program run. It is my experiences while migrating from 32-bit to 64-bit code that prompted this message. It is with some trepidation that as the subject of my first posting I am suggesting that Python 2.3 should use a different layout of all Python objects than is defined in Python 2.2.1. Specifically, I have found that changing lines 63-74 of Include/object.h from: #ifdef Py_TRACE_REFS #define PyObject_HEAD \ struct _object *_ob_next, *_ob_prev; \ int ob_refcnt; \ struct _typeobject *ob_type; #define PyObject_HEAD_INIT(type) 0, 0, 1, type, #else /* !Py_TRACE_REFS */ #define PyObject_HEAD \ int ob_refcnt; \ struct _typeobject *ob_type; #define PyObject_HEAD_INIT(type) 1, type, #endif /* !Py_TRACE_REFS */ to: #ifdef Py_TRACE_REFS #define PyObject_HEAD \ struct _object *_ob_next, *_ob_prev; \ struct _typeobject *ob_type; \ int ob_refcnt; #define PyObject_HEAD_INIT(type) 0, 0, type, 1, #else /* !Py_TRACE_REFS */ #define PyObject_HEAD \ struct _typeobject *ob_type; \ int ob_refcnt; #define PyObject_HEAD_INIT(type) type, 1, #endif /* !Py_TRACE_REFS */ significantly improved the performance of my 64-bit processes. Basically, I have just changed the order of the items in PyObject and PyVarObject to avoid gas due to an "int" being a 4-byte long and aligned types, while "long" and pointers are 8-byte long and aligned types (on 64-bit platforms that conform to the LP64 guideline). For the ILP32 guideline, such as Intel x86 and AMD CPUs, this should have no effect. On the Sun platform on which I live, the changes work for both ILP32 and LP64. For the very large programs I run, the modification saved me 40% execution time. This was probably due to the increased number of Python objects that would fit into the L2 cache, so I don't believe that others would necessarily see as large as a difference with this coding change. Please consider this change for inclusion in the upcoming Python release. - Bob

7 7

Python-Dev September 2002