Mailman 3 May 2003 - Python-Dev

Re: [Python-Dev] [development doc updates]
by Barry Warsaw May 16, 2003

May 16, 2003

On Fri, 2003-05-16 at 11:35, Fred L. Drake wrote: > The development version of the documentation has been updated: > > http://www.python.org/dev/doc/devel/ > > Various updates, including Jim Fulton's efforts on updating the Extending & > Embedding manual. I think this one's gonna make my Python quotes file: "So, if you want to define a new object type, you need to create a new type object." :) -Barry

3 4

test_urllibnet failing on Windows
by Tim Peters May 16, 2003

May 16, 2003

I'm not familiar with this test. In a release build: """ C:\Code\python\PCbuild>python ../lib/test/test_urllibnet.py testURLread (__main__.URLTimeoutTest) ... ok test_bad_address (__main__.urlopenNetworkTests) ... ok test_basic (__main__.urlopenNetworkTests) ... ok test_fileno (__main__.urlopenNetworkTests) ... ERROR test_geturl (__main__.urlopenNetworkTests) ... ok test_info (__main__.urlopenNetworkTests) ... ok test_readlines (__main__.urlopenNetworkTests) ... ok test_basic (__main__.urlretrieveNetworkTests) ... ok test_header (__main__.urlretrieveNetworkTests) ... ok test_specified_path (__main__.urlretrieveNetworkTests) ... ok ====================================================================== ERROR: test_fileno (__main__.urlopenNetworkTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_urllibnet.py", line 91, in test_fileno FILE = os.fdopen(fd) OSError: (0, 'Error') ---------------------------------------------------------------------- Ran 10 tests in 7.081s FAILED (errors=1) Traceback (most recent call last): File "../lib/test/test_urllibnet.py", line 149, in ? test_main() File "../lib/test/test_urllibnet.py", line 146, in test_main urlretrieveNetworkTests) File "C:\Code\python\lib\test\test_support.py", line 259, in run_unittest run_suite(suite, testclass) File "C:\Code\python\lib\test\test_support.py", line 247, in run_suite raise TestFailed(err) test.test_support.TestFailed: Traceback (most recent call last): File "../lib/test/test_urllibnet.py", line 91, in test_fileno FILE = os.fdopen(fd) OSError: (0, 'Error') """ In a debug build: """ C:\Code\python\PCbuild>python_d ../lib/test/test_urllibnet.py testURLread (__main__.URLTimeoutTest) ... ok test_bad_address (__main__.urlopenNetworkTests) ... ok test_basic (__main__.urlopenNetworkTests) ... ok test_fileno (__main__.urlopenNetworkTests) ... """ and there it dies with an assertion error in the bowels of Microsoft's fdopen.c. That's called by Python's posix_fdopen, here: fp = fdopen(fd, mode); At this point, fd is 436. MS's fdopen is unhappy because only 32 handles actually exist at this point, and 436 is bigger than that. In the release build, the MS assert doesn't (of course) trigger; instead, that 436 >= 32 causes MS's fdopen to return NULL.

3 3

codeop: small details (Q); commit priv request
by Samuele Pedroni May 16, 2003

May 16, 2003

1) Python 2.3b1 (#40, Apr 25 2003, 19:06:24) [MSC v.1200 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import codeop >>> codeop.compile_command("",symbol="eval") Traceback (most recent call last): File "<stdin>", line 1, in ? File "s:\transit\py23\lib\codeop.py", line 129, in compile_command return _maybe_compile(_compile, source, filename, symbol) File "s:\transit\py23\lib\codeop.py", line 106, in _maybe_compile raise SyntaxError, err1 File "<input>", line 1 pass ^ SyntaxError: invalid syntax the error is basically an artifact of the logic that enforces: compile_command("",symbol="single") === compile_command("pass",symbol="single") (this makes typing enter immediately after the prompt at a simulated shell a nop as expected) I would expect compile_command("",symbol="eval") to return None, i.e. to simply signal an incomplete expression (that is what would happen if the code for "eval" case would avoid the cited logic). 2) symbol = "exec" is silently accepted but the documentation intentionally only refers to "exec" and "single" as valid values for symbol. Maybe a ValueError should be raised. Context: I was working on improving Jython codeop compatibility with CPython codeop. Btw, as considered here by Guido http://sourceforge.net/tracker/index.php?func=detail&aid=645404&group_id=54… I would ask to have commit privileges for CPython regards

4 5

Simple dicts
by Tim Peters May 15, 2003

May 15, 2003

Behind the scenes, Damien Morton has been playing with radically different designs for "small" dicts. This reminded me that, in previous lives, I always used chaining ("linked lists") for collision resolution in hash tables. I don't have time to pursue this now, but it would make a nice experiment for someone who wants to try a non-trivial but still-simple and interesting project. It may even pay off, but that shouldn't be a motivation <wink>. Design notes: PyDictEntry would grow a new PyDictEntry *next; slot, boosting it from 12 to 16 bytes on a 32-bit box. This wasn't reasonable when Python was designed, but pymalloc allocates 16-byte chunks with virtually no wasted space. PyDictObject would loose the ma_fill and ma_table and ma_smalltable members, and gain a pointer to a variable-size vector of pointers PyDictEntry **first; For a hash table with 2**n slots, this vector holds 2**n pointers, memset to NULL initially. The hash chain for an object with hash code h starts at first[h & ma_mask], and is linked together via PyDictEntry.next pointers. Hash chains per hash code are independent. There's no use for the "dummy" state. There's no *logical* need for the "unused" state either, although a micro-optimizer may want to retain that in some form. Memory use isn't nearly as bad as it may first appear -- to the contrary, it's probably better on average! Assuming a 32-bit box: Current: tables are normally 1/3 to 2/3 full. If there are N active objects in the table, at 1/3 full the table contains 3*N PyDictEntries, and at 2/3 full it contains 1.5*N PyDictEntries, for a total of (multiplying by 12 bytes per PyDictEntry) 18*N to 36*N bytes. Chaining: assuming tables are still 1/3 to 2/3 full. At 1/3 full there are 3*N first pointers and at 2/3 full there are 1.5*N first pointers, for a total of 6*N to 12*N bytes for first pointers. Independent of load factor, 16*N bytes are consumed by the larger PyDictEntry structs. Adding, that's 22*N to 28*N bytes. This relies on pymalloc's tiny wastage when allocating 16-byte chunks (under 1%). The worst case is worse than the current scheme, and the best case is better. The average is probably better. Note that "full" is a misnomer here. A chained table with 2**i slots can actually hold any number of objects, even if i==0; on average, each hash chain contains N/2**i PyDictEntry structs. Note that a small load factor is less important with chained resolution than with open addressing, because collisions at different hash codes can't interfere with each other (IOW, an object in slot #i never slows access to an object in the slot #j collision list, whenever i != j; "breathing room" to ease cross-hashcode collision pressure isn't needed; primary collisions are all that exist). Collision resolution code: Just a list walk. For example, lookdict_string could be, in its entirety: static dictentry * lookdict_string(dictobject *mp, PyObject *key, register long hash) { dictentry *p = mp->first[hash & mp->ma_mask]; if (PyString_CheckExact(key)) { for (; p != NULL; p = p->next) { if (p->me_key == key || (ep->me_hash == hash && PyString_Eq(ep->me_key, key))) return p; } return NULL; } mp->ma_lookup = lookdict; return lookdict(mp, key, hash); } Resizing: Probably much faster. The vector of first pointers can be realloc'ed, and sometimes benefit from the platform malloc extending it in-place. No other memory allocation operation is needed on a resize. Instead about half the PyDictEntry structs will need to move to "the other half" of the table (the structs themselves don't get copied or moved; they just get relinked via their next pointers). Copying: Probably slower, due to needing a PyObject_Malloc() call for each key/value pair. Building a dict up: Probably slower, again due to repeated PyObject_Malloc() calls. Referencing a dict: Probably a wash, although because the code can be so much simpler compilers may do a better job of optimizing it, and no tests are needed to distinguish among three kinds of states. Out-of-cache dicts are killers either way. Also see next point. Optimizations: The cool algorithmic thing about chaining is that self-organizing gimmicks (like swap-toward-front (or move-to-front) on reference) are easy to code and run fast (again, the dictentry structs don't move, you just need to fiddle a few next pointers). When collision chains can collide, dynamic table reorganization is so complicated and expensive that nobody has even thought about trying it in Python. When they can't collide, it's simple. Note too that since the memory burden per unused slot falls from 12 to 4 bytes, sparser tables are less painful to contemplate. Small dicts: There's no gimmick here to favor them.

4 4

Re: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib warnings.py,1.19,1.20
by Charles G Waldman May 15, 2003

May 15, 2003

GvR> only reason I can come up with is that for dotted names, the dot would GvR> have to be escaped on the command line, and escaping something on the GvR> command line is painful because \ is also a shell escape character, so GvR> you'd have to escape the escape. I'm afraid I must be missing something terribly obvious here, but why would you need to escape a dot on a command line? None of the shells I'm familiar with treat dot as a metacharacter. Isn't `?' the standard shell metacharacter for "any character"? Filename patterns on the shell command line are "glob patterns", not RE's. But, like I said, I'm probably missing something. I think I'll go back into the shadows to lurk some more now.... Charles

5 6

Re: [Python-checkins] python/dist/src/Lib warnings.py,1.19,1.20
by Guido van Rossum May 15, 2003

May 15, 2003

> Modified Files: > warnings.py > Log Message: > defer re module imports to help improve interpreter startup Are you sure that's going to help? "import warnings" callse _processoptions() and makes a few calls to filterwarnings() which brings in the re module anyway... --Guido van Rossum (home page: http://www.python.org/~guido/)

3 5

sf.net/708007: expectlib.py telnetlib.py split
by Luke Kenneth Casson Leighton May 15, 2003

May 15, 2003

[i am not on the python-dev list but i check the archives, please cc me] approximately two years ago i needed the functionality outlined in the present telnetlib.py for several other remote protocols, most notably commands (including ssh and bash) and also HTTP. i figure that this functionality should be more than invaluable to other python developers. for example even the existing python libraries such as the ftp client ftplib.py, which makes extensive use of regular expressions to parse commands, could possibly benefit from rewrites using the "new" expectlib.py. also i believe it's the sort of thing that the twisted crowd should already have invented, and they're mad if they haven't already got something similar. this message is therefore just a polite ping to the regular python developers that the above referenced patch appears not to have yet been looked at or assigned to anybody. that having been said (perhaps with unintended implications of criticism that the regular python developers are slackers, which i most certainly am NOT saying!): the expectlib.py / telnetlib.py split is not exactly a top priority - just the sort of thing that one python fanatic would classify as "nice to have". l. -- -- expecting email to be received and understood is a bit like picking up the telephone and immediately dialing without checking for a dial-tone; speaking immediately without listening for either an answer or ring-tone; hanging up immediately and then expecting someone to call you (and to be able to call you). -- every day, people send out email expecting it to be received without being tampered with, read by other people, delayed or simply - without prejudice but lots of incompetence - destroyed. -- please therefore treat email more like you would a CB radio to communicate across the world (via relaying stations): ask and expect people to confirm receipt; send nothing that you don't mind everyone in the world knowing about...

1 1

MS VC 7 offer
by Guido van Rossum May 15, 2003

May 15, 2003

A month ago at Python UK in Oxford (which was colocated with C and C++ standardization meetings as well as a general C and C++ users conference) I met with some folks from Microsoft's VC development team, including the project lead, Nick Hodapp. I told Nick that Python for Windows was still built using VC 6. He pointed out that the actual compilers (not the GUI) from VC 7 are freely downloadable. More recently, Nick sent me an email offering to donate copies of VC 7 to the "key developers". I count Tim, myself and Mark Hammond among the key developers. Is there anyone else who would count themselves among those? I presume he's offering the pro version, which has a real optimizer, unlike the "standard" version that was kindly donated by Bjorn Pettersen. I can see advantages and disadvantages of moving to VC 7; I'm sure the VC 7 compiler is more standard-compliant and generates faster code, but a disadvantage is that you can't apparently link binaries built with VC 6 to a program built with VC 7, meaning that 3rd party extensions will have to be recompiled with VC 7 as well. I have no idea how many projects this will affect (don't worry about Zope Corp :-). Maybe we should try to include those 3rd party developers in the deal. (I think Robin Dunn would be affected, wxPython has a Windows distribution.) If you think this is a bad idea or if you would like to qualify for a compiler donation, please follow up! --Guido van Rossum (home page: http://www.python.org/~guido/)

27 85

Re: sf.net/708007: expectlib.py telnetlib.py split
by Noah Spurrier May 14, 2003

May 14, 2003

Tell me more about expectlib.py. Is it pty based? I ask because I wonder if it's like my pexpect module: http://pexpect.sourceforge.net/ Yours, Noah

2 1

inplace multiply problem was a bug that has been fixed
by Paul Dubois May 14, 2003

May 14, 2003

My question about inplace multiply was answered by Todd Miller: it was a bug in Python that is now fixed. I upgraded and my problem went away.

1 0