Mailman 3 April 2001 - Python-Dev

Showstopper
by Tim Peters April 16, 2001

April 16, 2001

We've got a showstopper bug involving gc and dicts. It's not brand new, but was probably introduced when we fiddled PyDict_Next() to stop the dict resizing problems plaguing Jeremy. Cut to the chase: 1. dict_items() is called. The dict has 22 of 32 slots filled. 2. PyList_New(22) creates the result list. 3. dict_items() loops through the dict looking for active entries. It does *not* use PyDict_Next, but a loop of the form for (i = 0, j = 0; i < mp->ma_size; i++) { 4. When it finds an active entry, it calls PyTuple_New(2) to create the entry's (key, value) result tuple. 5. At the end, PyTuple_New() calls PyObject_GC_Init(), which calls _PyGC_Insert(). 6. _PyGC_Insert() decides it's time for a collection. 7. The dict dict_items() is iterating over (remember step #1 <wink>?) is one of the objects gc traverses. gc dict traversal *does* use PyDict_Next. 8. 22 of 32 slots filled is exactly at the dict resize point, so the PyDict_Next() invoked by gc traversal grows the dict. 9. So, back to step #1, dict_item()'s for (i = 0, j = 0; i < mp->ma_size; i++) { loop now goes up to 64 instead of the original 32, and, because of the internal dict reshuffling, *can* (depending on the exact data content of the dict) see values again that it already saw before the dict got rearranged. 10. As a result, the later PyList_SetItem(v, j, item); inside the loop can eventually pass values for j too large for the list. 11. PyList_SetItem() sets a "list assignment index out of range" error string, but nobody pays ttention to that, and dict_items() proceeds to trigger the error multiple times. 12. The value returned by dict_items() is wrong, containing some duplicate (key, value) pairs, and missing others. 13. The eval loop goes around a few more times, until it finally hits a bytecode that notices the pending exception. Then (I finally got lucky here!) IndexError finally pops up on a line where an IndexError doesn't make sense. I have a test case that reliably triggers this bug, but was unable to whittle it below 100+ Kb of code + data files. So trust me about the details above (they're clear enough!).

4 14

ANNOUNCE: A Python 2.1 release candidate!
by Guido van Rossum April 16, 2001

April 16, 2001

Python 2.1 is almost ready. We've built a release candidate -- a version that should become the final version, barring any last-minute showstopper bugs (which will of course be fixed before releasing the final version). Thanks to all who helped! Please download the release candidate and use it on your favorite platform. For more info: http://www.python.org/2.1/ There's not much new since 2.1b2: we mostly fixed bugs and added documentation. Some things that *did* change visibly: - Ping added an interactive help browser to pydoc. (Very cool! Try it!) - Eric Raymond extended the pstats module with a simple interactive statistics browser, invoked when the module is run as a script. - An updated python-mode.el version 4.0 which integrates Ken Manheimer's pdbtrack.el. This makes debugging Python code via pdb much nicer in XEmacs and Emacs. When stepping through your program with pdb, in either the shell window or the *Python* window, the source file and line will be tracked by an arrow. - After a public outcry, assignment to __debug__ is no longer illegal; instead, a warning is issued. It will become illegal in 2.2. - New port: SCO Unixware 7, by Billy G. Allie. - Updated the RISCOS port. We expect to release the final version on Tuesday morning, April 17. Enjoy! --Guido van Rossum (home page: http://www.python.org/~guido/)

6 8

2nd release candidate?
by Guido van Rossum April 15, 2001

April 15, 2001

I'd like to issue a 2nd release candidate late tonight, and then change *nothing* between 2.1c2 and the final release Tuesday. The only thing I still need to change before making the 2nd release candidate is to rip out Moshe's EGD patch for the socket module, which has too many problems IMO. --Guido van Rossum (home page: http://www.python.org/~guido/)

1 0

2.1c1 on BSDI
by Thomas Wouters April 15, 2001

April 15, 2001

For the record :) Python 2.1c1 performs as expected on BSDI 4.1 and 4.0.1. That is, there are some crashes, but those were there in 2.0 and 1.5.2 as well, for the most part, and are test-specific errors in general. We've been using 2.1b2 (actually slightly newer) on development machines, where it is used for various tools, and I haven't encountered many bugs yet. BSDI 4.0.1 still needs to disable threads, but that's a platform bug, not a Python one. -- Thomas Wouters <thomas(a)xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

1 0

test_fcntl on Solaris 8
by Guido van Rossum April 15, 2001

April 15, 2001

While testing Python 2.1 on SF's Solaris 8 box, I noticed that it hangs forever in test_fcntl, on this line: rv = fcntl.fcntl(f.fileno(), FCNTL.F_SETLKW, lockdata) Why could this be? Could it be that the NFS lock server is stuck? But I could also note that this test is pretty silly. It has all this platform-specific cruft to do a locking operation the hard way, while the fcntl module supplies two operations (flock() and lockf()) that could be used instead! (Unfortunately, using lockf() I get the same behavior -- not surprising, since the C code does the same thing that the Python code was doing. :-( ) Should I update the test, or declare the machine broken? (I *think* I recall that the tests succeeded yesterday.) --Guido van Rossum (home page: http://www.python.org/~guido/)

3 4

make clean and make clobber semantics
by Guido van Rossum April 15, 2001

April 15, 2001

I just noticed for the first time that the semantics of "make clean" and "make clobber" have changed. "make clean" used to remove the .so files too, AFAIK, but no longer does so. "make clean" used to leave the configuration files lying around, but now seems to remove at least some of them. One of the consequences of all this is that, after "make clean", another "make" doesn't rebuild the extensions, because setup.py finds that the .so files are still there and decides not to rebuild the missing .o's. Another consequence is that after "make clobber" you have to rerun the configure script (or say "make recheck"). This takes almost as long as the rest of the build... In other words, "make clean" doesn't go far enough, and "make clobber" goes too far, for what I seem to want most often (recompile every C source file). Can someone suggest a fix? --Guido van Rossum (home page: http://www.python.org/~guido/)

3 6

Re: [Python-Dev] s1 == (sf % (s1 / sf))? A bad idea?
by Dinu Gherman April 15, 2001

April 15, 2001

> Examples > > Here is an example of an interactive session exhibiting the > expected behaviour of this feature. > > >>> "12345 John Doe" / "%5d %8s" > (12345, 'John Doe') > >>> "12 34 56 7.890" / "%d %d %d %f" > (12, 34, 56, 7.8899999999999997) > >>> "12345 John Doe, Foo Bar" / "%(num)d %(n)s, %(f)s %(b)s" > {'n': 'John Doe', 'f': 'Foo', 'b': 'Bar', 'num': 12345} > >>> "1 2" / "%d %d %d" > Traceback (innermost last): > File "<stdin>", line 1, in ? > TypeError: not all arguments filled Kind of late to jump in, but this is the nature of this list. I'd like to support Peter's proposal for having *some* kind of inverse mechanism to string formatting using '%'. Now, that doesn't mean anything, of course, but no matter what the syn- tax would look like: I'd prefer having that feature over not having it and I'll give an example below. Reminding you of a thread I triggered a while ago (that went slightly astray) which was, kind of, received with suspicion, I notice that it matches quite nicely with Peter's (more ge- neral) idea. Here's the thread's summary: Grouping function for string module? http://mail.python.org/pipermail/python-list/1999-September/011875.html Combining this I'd like to see something like the following (again, maybe with a different syntax): >>> "1010000011110101" / "%4s%4s%4s%4s" ('1010', '0000', '1111', '0101') >>> "10100000111101" / "%4s%4s%4s%4s" ('1010', '0000', '1111', '01') or even: >>> "1010000011110101" / ("%4s" * 4) ('1010', '0000', '1111', '0101') ;-) Regards and Happy Easter (will be away for a week)! Dinu -- Dinu C. Gherman ReportLab Consultant - http://www.reportlab.com ................................................................ "The only possible values [for quality] are 'excellent' and 'in- sanely excellent', depending on whether lives are at stake or not. Otherwise you don't enjoy your work, you don't work well, and the project goes down the drain." (Kent Beck, "Extreme Programming Explained")

2 1

RE: [Python-Dev] Cygwin Python 2.1c1 test_threadedtempfile problem
by Tim Peters April 15, 2001

April 15, 2001

[Michael Hudson] > Maybe modules should be removed from sys.modules when they fail to be > imported due to an exception? [Guido] > This has been suggested before, but *in general* this is not a good > idea. For example, you want to debug the remains of the failed > module. Ya, I've heard you say something like that before, but haven't understood what it meant. That is, I've never had, or been able to picture, a case where having a module object in an incomplete and unknown state is actually of use. OTOH, I've certainly burned my share of time recovering from that importing a broken module only fails the first time. It's like Python only raised an exception the first time somebody tried to open a particular non-existent file for reading, but the second time it returned a file object that may or may not fail in use, and may or may not work correctly when it doesn't fail, depending on what you do with it. > However, the test suite can easily guard against this, e.g. by > inserting "import thread" before "import threading" whereever it > occurs. So how come a failed attempt to import thread doesn't leave a bogus module object in sys.modules["thread"] too <0.9 wink>? This is obscure stuff. Is "debugging the remains" of sufficient use to make up for the conceptual complications?

3 2

Python 2.1 RC - PyChecker
by Neal Norwitz April 15, 2001

April 15, 2001

I've run the CVS version of PyChecker (http://pychecker.sourceforge.net) against Python 2.1 RC1. Below are the real or possible problems I found. Some of the "not used" could be real errors, most are not. All "uses named arguments" can be safely ignored. "No global"s are definitely problems AFAICT (except 1 which can't be reached). Neal -- MimeWriter.py:108 Function (addheader) uses named arguments MimeWriter.py:117 Function (startbody) uses named arguments StringIO.py:187 Local variable (here) not used UserString.py:137 Base class (UserString.UserString) __init__() not called aifc.py:181 Local variable (math) not used asynchat.py:99 No method (collect_incoming_data) found asynchat.py:112 No method (found_terminator) found (2 methods documented, but should add method and raise exception?) asyncore.py:155 Local variable (l) not used audiodev.py:214 No global (BUFFERSIZE) found bdb.py:220 Local variable (bp) not used cgi.py:743 Base class (UserDict.UserDict) __init__() not called cgi.py:843 Local variable (traceback) not used chunk.py:109 No attribute (chunk_size) found (should be chunksize) fileinput.py:292 Function (input) uses named arguments getpass.py:29 Local variable (getpass) not used gopherlib.py:172 Local variable (port) not used gzip.py:276 Local variable (orig_size) not used ihooks.py:494 Function (import_it) uses named arguments ihooks.py:498 Function (import_it) uses named arguments imaplib.py:1019 No global (j) found locale.py:273 No global (l) found (can't be reach, but could remove last else and always raise error) mailbox.py:21 No attribute (stop) found mailbox.py:23 No attribute (start) found mailbox.py:29 No method (_search_start) found mailbox.py:34 No method (_search_end) found (_search_*() used in subclasses, add method and raise exception?) mailbox.py:106 No method (_isrealfromline) found mailbox.py:260 Local variable (time) not used mhlib.py:651 Local variable (messages) not used netrc.py:13 No global (file) found (should be filename) nturl2path.py:45 Local variable (string) not used pstats.py:188 Local variable (std_list) not used pyclbr.py:206 Local variable (imports) not used pydoc.py:170 Base class (exceptions.Exception) __init__() not called rfc822.py:607 Local variable (expectaddrspec) not used robotparser.py:234 Local variable (sys) not used sgmllib.py:178 No global (SGMLParserError) found (should be SGMLParseError?) shlex.py:99 Local variable (tok) not used smtpd.py:312 No global (UnimplementedError) found smtpd.py:375 Local variable (paths) not used sndhdr.py:87 Local variable (hdr_size) not used sndhdr.py:134 Local variable (style) not used sre_parse.py:286 Local variable (here) not used threading.py:547 Local variable (random) not used threading.py:611 Local variable (time) not used token.py:85 Local variable (string) not used unittest.py:675 Local variable (opts) not used urllib.py:1147 Local variable (x) not used urllib2.py:450 No global (error_302_dict) found (needs req.?) urllib2.py:630 No attribute (parent) found urllib2.py:1053 No global (OpenerDirectory) found urlparse.py:58 Local variable (path) not used webbrowser.py:77 No global (ret) found

1 0

Re: Problem with SSL and socketmodule on Debian Potato?
by Mark Favas April 15, 2001

April 15, 2001

[Guido] > [Moshe] >> Correct on both. My "worse" is: it would print a warning about using >> an insecure method on systems with /dev/random but without an EGD, >> even though it is secure. > And indeed it does when I tried it on SF's Solaris 8 box, which has > OpenSSL installed and /dev/random. Interesting - there's no /dev/random on my Solaris 8 boxes... > Even worse (in my view), the error message is as soon as the socket > module is imported! This is bad, because most uses of socket aren't >interested in its SSl capabilities! Quite agree - I've got OpenSSL on my Tru64 box (no /dev/random either) and it's annoying to get this warning whenever I "import socket". If the OpenSSL bits don't themselves warn about insecure methods, I'd prefer if Python didn't take it upon itself to nag <wink>. -- Mark Favas - m.favas(a)per.dem.csiro.au CSIRO, Private Bag No 5, Wembley, Western Australia 6913, AUSTRALIA

1 0