Mailman 3 January 2004 - Python-Dev

bad exec* prototypes check - obsolete?
by Skip Montanaro Jan. 17, 2004

Jan. 17, 2004

In configure.in there is a check for bad exec* prototypes: if test "$have_prototypes" = yes; then bad_prototypes=no AC_MSG_CHECKING(for bad exec* prototypes) AC_TRY_COMPILE([#include <unistd.h>], [char **t;execve("@",t,t);], , AC_DEFINE(BAD_EXEC_PROTOTYPES, 1, [Define if your <unistd.h> contains bad prototypes for exec*() (as it does on SGI IRIX 4.x)]) bad_prototypes=yes ) AC_MSG_RESULT($bad_prototypes) fi Since we assume a proper ANSI C compiler at this point, can we get rid of this test? Skip

2 2

RE: [Python-Dev] Accumulation module
by Raymond Hettinger Jan. 17, 2004

Jan. 17, 2004

> > * Is there a way to compute the standard deviation without multiple > > passes over the data (one to compute the mean and a second to sum the > > squares of the differences from the mean? [Tim] > The textbook formula does it in one pass, by accumulating running totals > of > the elements and of their squares. It's numerically atrocious, though, > and > should never be used (it can even end up delivering a negative result!). > Knuth gives a much better recurrence formula for this, doing one-pass > computation of variance in a numerically stable way. I found it. There is a suggestion that the formula would be better off if the recurrence sums are accumulated along with a cumulative error term that tracks the loss of significance when adding a small adjustment to a larger sum. The textbook exercise indicates that that technique is the most useful in the presence of crummy rounding. Do you think there is any value in using the error term tracking? def stddev(data): # Knuth -- Seminumerical Algorithms 4.2.2 p.216 it = iter(data) m = float(it.next()) # new running mean s = 0.0 # running sum((x-mean)**2) k = 1 # number of items dm = ds = 0 # carried forward error term for x in it: k += 1 adjm = (x-m)/k - dm newm = m + adjm dm = (newm - m) - adjm adjs = (x-m)*(x-newm) - ds news = s + adjs ds = (news - s) - adjs s = news m = newm return (s / (k-1)) ** 0.5 # sample standard deviation Without the error term tracking, the inner loop simplifies to: k += 1 newm = m+(x-m)/k s += (x-m)*(x-newm) m = newm Raymond Hettinger

2 1

anydbm test failures using Berkeley DB 4.2.52 on Solaris 8
by Skip Montanaro Jan. 17, 2004

Jan. 17, 2004

Trying to test some of my obsolete platform excisions on more than just Mac OS X, I came across these test failures this evening on Solaris 8 when Berkeley DB 4.2.52 was installed: bash-2.03$ ./python Lib/test/test_anydbm.py test_anydbm_creation (__main__.AnyDBMTestCase) ... ok test_anydbm_keys (__main__.AnyDBMTestCase) ... ERROR test_anydbm_modification (__main__.AnyDBMTestCase) ... ERROR test_anydbm_read (__main__.AnyDBMTestCase) ... ERROR ====================================================================== ERROR: test_anydbm_keys (__main__.AnyDBMTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "Lib/test/test_anydbm.py", line 58, in test_anydbm_keys self.init_db() File "Lib/test/test_anydbm.py", line 69, in init_db f = anydbm.open(_fname, 'n') File "/export/home/skip/src/python/Lib/anydbm.py", line 83, in open return mod.open(file, flag, mode) File "/export/home/skip/src/python/Lib/dbhash.py", line 16, in open return bsddb.hashopen(file, flag, mode) File "/export/home/skip/src/python/Lib/bsddb/__init__.py", line 293, in hashopen d.open(file, db.DB_HASH, flags, mode) DBInvalidArgError: (22, 'Invalid argument -- DB_TRUNCATE illegal with locking specified') ====================================================================== ERROR: test_anydbm_modification (__main__.AnyDBMTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "Lib/test/test_anydbm.py", line 45, in test_anydbm_modification self.init_db() File "Lib/test/test_anydbm.py", line 69, in init_db f = anydbm.open(_fname, 'n') File "/export/home/skip/src/python/Lib/anydbm.py", line 83, in open return mod.open(file, flag, mode) File "/export/home/skip/src/python/Lib/dbhash.py", line 16, in open return bsddb.hashopen(file, flag, mode) File "/export/home/skip/src/python/Lib/bsddb/__init__.py", line 293, in hashopen d.open(file, db.DB_HASH, flags, mode) DBInvalidArgError: (22, 'Invalid argument -- DB_TRUNCATE illegal with locking specified') ====================================================================== ERROR: test_anydbm_read (__main__.AnyDBMTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "Lib/test/test_anydbm.py", line 52, in test_anydbm_read self.init_db() File "Lib/test/test_anydbm.py", line 69, in init_db f = anydbm.open(_fname, 'n') File "/export/home/skip/src/python/Lib/anydbm.py", line 83, in open return mod.open(file, flag, mode) File "/export/home/skip/src/python/Lib/dbhash.py", line 16, in open return bsddb.hashopen(file, flag, mode) File "/export/home/skip/src/python/Lib/bsddb/__init__.py", line 293, in hashopen d.open(file, db.DB_HASH, flags, mode) DBInvalidArgError: (22, 'Invalid argument -- DB_TRUNCATE illegal with locking specified') ---------------------------------------------------------------------- Ran 4 tests in 0.102s FAILED (errors=3) Traceback (most recent call last): File "Lib/test/test_anydbm.py", line 95, in ? test_main() File "Lib/test/test_anydbm.py", line 90, in test_main test_support.run_unittest(AnyDBMTestCase) File "/export/home/skip/src/python/Lib/test/test_support.py", line 290, in run_unittest run_suite(suite, testclass) File "/export/home/skip/src/python/Lib/test/test_support.py", line 274, in run_suite raise TestFailed(msg) test.test_support.TestFailed: errors occurred in __main__.AnyDBMTestCase I don't recall seeing these before, but this is the first time I've built from CVS on Solaris 8 w/ BDB 4.2.52. The same tests work fine with Python 2.3.3+Berkeley DB 4.2.52 (Lib/test/test_anydbm.py is the same in 2.3.3 and CVS). I'm going to make a wild ass guess and say these aren't related to what I'm doing. Greg, have you seen this before? Skip

1 0

No more releases of MacPython-OS9?
by Jack Jansen Jan. 17, 2004

Jan. 17, 2004

[I'm cc-ing python-dev to get some feedback on this issue from a wider community too, let me know if the crowd there don't want this]. There is a serious problem with distributing future versions of MacPython-OS9. While the 2.3.3 installer has been sitting on my machine for the last three weeks I cannot distribute it due to licensing issues with Installer VISE from Mindvision. Mindvision's licensing scheme is a subscription-based model: you pay a yearly fee based on the number of copies you intend to distribute and whether you're a commercial/noncommercial operation. Built installers continue to work when you don't renew your subscription, but you cannot create new installers. For many years they have had a program whereby shareware and freeware developers could apply for a free license, and MacPython-OS9 installers have been created with Installer VISE for many many years. And, I must say, with great satisfaction on my side: even though MacPython installers have been very complex with 68K versus PowerPC, Classic versus Carbon execution model, inclusion of various optional bits and other issues all handled fairly easily. However, Mindvision has discontinued the free licensing program. And while there is some leeway for renewing existing freeware licenses apparently I should have applied for that before last october, when the previous license expired (apparently Mindvision thinks that people create installers every couple of days?). So now I'm looking at three choices: 1. Stop distributing MacPython-OS9. I was planning to continue the 2.3.X series for as long as I had access to compatible hardware, but as a community service (personally I couldn't be bothered with OS9 anymore:-). 2. Buy an Installer VISE license. If I read <http://www.mindvision.com/pricing.asp> correctly this would cost $500 per year, probably for the next 2 years. This will only happen if someone else comes up with the money: I don't have it... 3. Switch to a different installer. Given the complexity of the MacPython distribution (even though it is a lot simpler nowadays, with only Carbon/PPC supported) I don't want to invest time in this. So: someone else will have to step in and create an installer. I'd like to hear what people think the best course of action is, especially from people who have a current interest in MacPython-OS9 (and/or are willing to donate time or money), -- Jack Jansen, <Jack.Jansen(a)cwi.nl>, http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman

11 13

Python 2.4 and MSVC 7.1 - extension compatibility
by Paul Moore Jan. 16, 2004

Jan. 16, 2004

One of the issues with moving to VC7 for the Python Windows build is the fact that extensions (or embedding apps) built with MSVC 6 "won't work" with it. But I've never actually managed to provoke a crash (not that I've tried *very* hard, mind). And thinking about the issue a little, I wonder whether there is a real problem here at all... A quick check of the DLLs in my Windows\System32 directory shows a lot which use MSVCRT. Including OLEAUT32, which is one of the key OLE/COM DLLs. So, presumably, this means that DLLs built with MSVCRT *can* be used by applications built with VC7. And a Python extension is just a DLL, with an unusual extension. So, the question is - is there really any problem with Python 2.4, built with MSVC7, using an extension DLL built with MSVC6? The areas where I can see a potential issue are where CRT objects are shared between the extension, and the Python runtime: 1. Memory allocated by the extension CRT, and freed by Python 2. FILE* objects passed from one CRT to the other 3. Any others? Memory allocation "should" be OK, if extension writers follow the rules in the Python/C API manual, section 9.1 - "To avoid memory corruption, extension writers should never try to operate on Python objects with the functions exported by the C library: malloc(), calloc(), realloc() and free()". And no-one ever writes extensions which violate the rules :-) There are a series of APIs in the "Very High Level Layer" (PyRun_* and friends), and some in the PyMarshal series, which pass FILE* objects about. I wouldn't expect these to be common in extensions. So, a question - would it be worth documenting which APIs are *not* supported in extensions built with an incompatible CRT to the one Python was built with? And if this was done, would it be OK to state "if you don't use these APIs, you don't have to build your extension or embedding application with the same CRT as that of Python"? If it is deemed acceptable to make these sort of guarantees, I'll volunteer to search out the relevant APIs, and draft a section for the Python/C API manual. But I don't want to spend the time and effort, if it's still not possible to say that it's safe to use MSVCRT in those cases. Paul. -- This signature intentionally left blank

2 2

Bug #878572
by Barry Warsaw Jan. 16, 2004

Jan. 16, 2004

The bug and attached patch improves the error messages when pwd.getpwnam() and friends raises a KeyError. I expect the change to be uncontroversial for Python 2.4 and I believe it should be uncontroversial for Python 2.3, since it affects only the error string. I'll commit these changes in the next few days unless objections are raised. -Barry

1 0

Improve your cup size NATURALLLY! Clinically verified! dbmxbwj mkr r i
by Kerry Mcneil Jan. 15, 2004

Jan. 15, 2004

1 0

PyCon Reminders (proposal submission ends tonight!)
by Guido van Rossum Jan. 15, 2004

Jan. 15, 2004

Proposal Submission Deadline: Tonight! -------------------------------------- Tonight at midnight (in *some* timezone :-) is the last time to submit PyCon proposals. http://pycon.org/dc2004/cfp/ Early Bird Registration ----------------------- Deadline for Early Bird Registration: January 31. *** Speakers, don't forget to register!!!!!! *** http://pycon.org/dc2004/register/ Sprint pre-announcement ----------------------- Come a few days early and participate in a coding sprint. Sprints are free (courtesy of the PSF!), and are held from Saturday March 20 through Tuesday March 23 (i.e. the four days leading up to the conference). http://pycon.org/dc2004 . Volunteers ---------- We're looking for volunteers to help run PyCon. http://mail.python.org/mailman/listinfo/pycon-organizers About PyCon ----------- PyCon DC 2004 will be held March 24-26, 2004 in Washington, D.C. Keynote speaker is Mitch Kapor of the Open Source Applications Foundation (http://www.osafoundation.org/). Don't miss any PyCon announcements! Subscribe to http://mail.python.org/mailman/listinfo/pycon-announce You can discuss PyCon with other interested people by subscribing to http://mail.python.org/mailman/listinfo/pycon-interest The central resource for PyCon DC 2004 is http://www.pycon.org/ Pictures from last year's PyCon: http://www.python.org/cgi-bin/moinmoin/PyConPhotos I'm looking forward to seeing you all in DC in March!!! --Guido van Rossum (home page: http://www.python.org/~guido/) Open Source Awards: http://builder.com.com/5100-6375_14-5136758.html

1 0

Re: [Python-Dev] collections module
by Josiah Carlson Jan. 14, 2004

Jan. 14, 2004

> None of those can compete with my proposed C implementation > which pre-allocates around 50 cells at a time and does it's > read/writes through array lookups. It avoids all the expensive > resizing of list and uses C ints instead of PyInts. That's pretty crazy. If I had a say, I'd be +1 just because I've seen so many people using their own {append, pop(0)} queues before. - Josiah P.S. I know knuth had it first, but one never knows how well read anyone else is. It is better to err on the side of giving too much information, than too little.

13 50

RE: [Python-Dev] collections module
by Michael Chermside Jan. 13, 2004

Jan. 13, 2004

[Guido:] > My own ideal solution [for queues] would be a subtle hack to the list > implementation to make pop(0) and insert(0, x) go a lot faster -- this > can be done by adding one or two extra fields to the list head and > allowing empty space at the front of the list as well as at the end. [Tim:] > That Python's builtin list calls realloc() on every append is pretty gross, > but it's also a tradeoff, saving the bytes in the list header that would be > required to remember how many allocated-but-unused slots exist. If you've > got a million tiny lists (and some apps do), burning an additional 8MB for > something your app doesn't really need isn't attractive (on a 32-bit box > today, the list header is 16 bytes; pymalloc aligns to 8-byte boundaries, so > the next possible size is 24 bytes). [Guido:] > Just in case nobody had thought of it before, couldn't the realloc() > call be avoided when roundupsize(oldsize) == roundupsize(newsize)??? [Tim:] > I had thought of it before, but never acted on it <wink>. roundupsize() has > gotten more expensive over the years too, and it may vary across boxes > whether it's cheaper to call that a second time than it is to make the > system realloc deduce that nothing needs to be done. roundupsize() is > certainly cheaper than Microsoft's realloc() in the do-nothing case. > > Note that there's an endcase glitch here when oldsize==0 and newsize==1. > roundupsize() returns 8 for both, but it doesn't follow that ob_item isn't > NULL. realloc(p, n) special-cases the snot out of p==NULL to avoid crashing > then. [...] > Note in passing that roundupsize() is expensive *because* lists don't > remember the amount of overallocated space too: roundupsize() has to > arrange to deliver the same result for many consecutive inputs I'm sure this is just pointing out the obvious, but *IF* the list header were increased, we could add TWO fields... on for allocated_size, and another for left_side_slots. According to Tim, this would still "only" bump list objects up to 24 bytes, but would allow: [1] O(1) performance for pop(0) [2] No need to call realloc() on every append(), just on "true" resizes. I guess I'm proposing the following (all ideas previously mentioned): [1] pop(0) would increment ob_item and left_side_slots. [2] insert(0,x) would decrement ob_item and left_side_slots *IF* left_side_slots were > 0. Lists should still grow at the tail for efficiency, but we can re-use space at the head left by a pop(0). [3] append(0,x) (and other functions that insert) would compare ob_size and allocated_size (do we need left_side_slots in here too?) to see whether new space is needed at the end. If so, it would EITHER call realloc() for more space, OR it would memmove so left_side_slots became 0. I'm still not quite sure what rule to use to determine which to do (although for MOST lists, left_side_slots would be 0, so there wouldn't be a question). So, to recap advantages and disadvantages: lists would still grow just as they do now. Lists used as Queues by calling append() and pop(0) would be time efficient so long as we avoid calling memmove for each and every append() (that's why we don't want to ALWAYS memmove when left_side_slots > 0). Lists on which pop(0) was never called (that's most of 'em) would have NO degradation of space used, and IMPROVED speed in cases (like Windows) where a realloc() noop takes noticable time. And (this seems the most significant drawback) the size of a list would increase by 8 bytes, even for very short lists. I wouldn't want to make the change without trying out the performance in some "real applications", but it seems like there are some potentially useful ideas floating around here. -- Michael Chermside

2 1