Mailman 3 August 2003 - Python-Dev

pirate 0.01 alpha!
by Michal Wallace 19 Aug '03

19 Aug '03

Annoucing pirate 0.01 alpha! http://pirate.tangentcode.com/ """ Right now, pirate supports a good deal of python. The two biggest things that are still missing are classes (because parrot classes aren't quite done yet) and exec/eval/import (because those things require a compiler -- and without classes, pirate can't compile itself). None of the builtin functions exist, either, and the types are borrowed from perl. :) BUT: there's still a lot you can do! For example, parrot is able to run this generator-based microtheads demo and correctly run over 50 small test cases covering much of the language. As a special bonus, it currently runs dog slow! The microthreads example runs about 8-9 times slower on pirate than on regular python. But that's not parrot's fault: the generated code is completely unoptimized. """ Comments, bug reports, patches, flames, etc welcome. :) Sincerely, Michal J Wallace Sabren Enterprises, Inc. ------------------------------------- contact: michal(a)sabren.com hosting: http://www.cornerhost.com/ my site: http://www.withoutane.com/ --------------------------------------

2 2

Re: [Python-Dev] file read-ahead with Mac end-of-line
by Andrew McNamara 19 Aug '03

19 Aug '03

>Then that person needs to specify '\r' as EOL (and open the file as >binary). There's no one-size fits all and yes, you normally do need to >quote string fields with embedded newlines with CSV. The csv module, as defined in 2.3, takes an iterable as the source of input lines. It doesn't make sense to then attempt additional EOL processing within the csv module - the current EOL processing in the csv module is a hang-over from it's past life as a stand-alone module that worked with 1.5.2, etc. I would propose that we remove most of the EOL processing from the csv module, making it simply ignore any terminal whitespace characters. If someone has EOL conventions that aren't catered to by the Python universal newline code, they can supply an alternate iterable source of lines. This should make the csv module behave as other parts of python behave (less nasty surprises). -- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/

1 0

Found a bug!
by Barry Warsaw 19 Aug '03

19 Aug '03

Well, I was debugging a Mailman user's problem when I hit what I think is a bug in mimelib/email pkg. I don't yet know if this is related to the MM bug because I can't get as far through the code, but you don't care about that. :) I believe mimelib is not handling RFC 2231 headers such as the following correctly: Content-Disposition: inline; filename*0="This%20is%20even%20more%20"; filename*1="%2A%2A%2Afun%2A%2A%2A%20"; filename*2="is it not.pdf" Try creating a message with a header like this and doing a msg.get_filename() on it -- you'll get a TypeError. The bug is that get_filename() isn't prepared for a missing language parameter. The fix is probably simple, and I'll try to resist doing too much refactoring <wink>. I'll check in a test case and a patch and backport it to the Python 2.3.1 and 2.2.4 branches. -Barry

1 1

Insane change to md5sum.py in 2.3
by Tim Peters 18 Aug '03

18 Aug '03

The 2.3 version of the Tools script md5sum.py opens files in text mode by default. This is plain crazy -- even having an option to open a file in text mode is nuts for an md5 program. 2.2's version was much simpler and didn't have this deadly (well, it's deadly on Windows ...) flaw. I would like to remove the -t and -b switches altogether, except than anyone else who learned the hard way that they *have* to pass -b to the 2.3 version on Windows to get the right answer would get screwed. So I'd like to switch the default to binary mode instead. Objections?

2 1

Re: hook for standalone executable
by Gordon McMillan 18 Aug '03

18 Aug '03

[Skip Montanaro] > I don't think zipimport supports importing from zip files > with appended stuff. I suspect it needs to know the last N > bytes form the zip length. Don't know about zipimport, but zip files are designed to be opened from the back. [Neil Schemenauer] > I suppose some executable formats do not like extra data > appended at the end. It's an ELF / COFF trick. [Neil Schemenauer again] > I mean that Unix users are less likely to want a stand-alone > executable instead of a Python script. Also, stand-alone > executables are usually intended for end users with little or > no Python development skills. Almost all of those people are > using windows. It may be a surprise to the developer community, but there are a lot of Unix users who are not developers and do not have root priveleges on their machines. These people want to install to ~/bin and don't particularly care what the install looks like. In this country it's usually analysts, statisticians and scientists. In other countries where MS is not so popular, it's anybody. [in general] The trick being discussed really doesn't cut the mustard on *nix anyway, because shared libs will be involved. -- Gordon http://www.mcmillan-inc.com/

2 1

PEP 311 Simplified Global Interpreter Lock Acquisition for Extensions
by Harri Pasanen 18 Aug '03

18 Aug '03

Hi, Just prior to my vacation I ran into a bit of trouble with my Python embedding, as much to my dismay my C++ program that had been working fine under Linux deadlocked under Windows. Apparently the underlying C threading API's were behaving in subtly different ways. After boiling down from my original embedded Python app which features omniORB, omniORBpy, and bunch of other goodies, I got down to the following testcase: #include <Python.h> class PythonLocker { public: PythonLocker () { _tstate = PyGILState_Ensure (); } ~PythonLocker() { PyGILState_Release(_tstate); } } private: PyGILState_STATE _tstate; }; int main (int argc, const char **argv) { Py_Initialize(); PyEval_InitThreads(); PyEval_ReleaseLock (); // This is the problem { PythonLocker pl; PyObject* usermod = PyImport_ImportModule ("echo"); // The previous line deadlocks on win32 } return 0; } The above works fine on Linux, but deadlocks on Win32. Tracing the execution on Win32 showed that it hanged in thread_nt.h: DWORD EnterNonRecursiveMutex(PNRMUTEX mutex, BOOL wait) { ... ret = InterlockedIncrement(&mutex->owned) ? /* Some thread owns the mutex, let's wait... */ WaitForSingleObject(mutex->hevent, INFINITE) : WAIT_OBJECT_0 ; The gotcha being that when entering above, mutex->owned was == -2, so it incorrectly entered the wait. I had a private discussion with Mark about this, and he quickly pointed out that I should not call PyEval_ReleaseLock(). Although this is not exactly clear from the PEP, which says "Apart from the existing, standard Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros, it is assumed that no additional thread state API functions will be used by the extension." I did not think PyEval_ReleaseLock() would qualify for the above, but somehow it does, under win32 at least. Now if I modify thread_nt.h, simple mindedly by changing the above to: ret = InterlockedIncrement(&mutex->owned) > 0 ? /* Some thread owns the mutex, let's wait... */ WaitForSingleObject(mutex->hevent, INFINITE) : WAIT_OBJECT_0 ; everything works for me on Win32 as well, including python regression tests. But I haven't wrestled with the code in question in detail to know what gremlins that change might unleash elsewhere. Without modifying Python, I can make my testcase working with the following change to my test program: { Py_Initialize(); PyEval_InitThreads(); { PythonLocker init_pl }; // release the lock // PyEval_ReleaseLock (); { // this block could potentially be called from another thread PythonLocker pl; PyObject* usermod = PyImport_ImportModule ("echo"); // The previous line deadlocks on win32 } In the C API world this is a bit more convoluted looking, as I need to call PyGILState_Ensure(), and PyGILState_Restore, even if after PyEval_InitThreads() I already know I have the lock. *** Sorry for rambling, but this is in the hope that someone else can avoid banging their heads against this as I have. And it is a bit unclear to me also if this kind of a platform difference between Linux and Win32 is a bug, or a feature. Ideally the Python C API should be bug-compatible. So possibly wrong usage should cause all platforms to fail. Btw., the new PyGILState_* functions do not appear to be part of the Python/C API documentation, there is only the PEP. Anyway, thanks to Mark for this definite improvement on the state of things. Harri

1 0

refleak hunting fun!
by Michael Hudson 18 Aug '03

18 Aug '03

Prompted by finding my own refcounting leaks in test_descr, I'm now looking to spread the misery :-) I've hacked regrtest (hacks attached) to do some very crude monitoring of sys.gettotalrefcount(). Here's the output of a recent run: test_exceptions leaked 17 references test_types leaked 22 references test_class leaked 66 references test_codeccallbacks leaked 1107 references test_coercion leaked 14 references test_compile leaked 69 references test_complex leaked 2 references test_cookie leaked 14 references test_copy leaked 82 references test_cpickle leaked 17 references test_descr leaked 1 references test_gc leaked 18 references test_global leaked 14 references test_hotshot leaked 111 references test_logging leaked 82 references test_minidom leaked 1 references test_mutants leaked -452 references test_new leaked 2 references test_optparse leaked 408 references test_os leaked 24 references test_pkg leaked 7 references test_pkgimport leaked 6 references test_regex leaked 14 references test_rotor leaked 14 references test_sax leaked 1762 references test_scope leaked 14 references test_socket leaked 1140 references test_sundry leaked 42 references test_threadedtempfile leaked 81 references test_threading leaked -24 references test_time leaked -24 references test_unicode leaked 29 references test_urlparse leaked -70 references test_xreadline leaked 14 references test_grammar leaked 14 references *some* of these are bogus -- caches in particular play merry hell with trying to instrument this kind of thing -- and indeed I've already found causes for some of the bogosities above but as my hacked regrtest runs each test five times, runs are rraatthheerr ssllooww. Some however, seem to be real. test_sax, test_socket and test_codeccallbacks seem to be the most alarming. The TrackRef class Tim posted a link to is likely to be indispensable in hunting these. Some tests are a bit unhappy at being run repeatedly -- notably test_threaded_import (which deadlocks) and all the doctest tests spew stuff to stdout. You'll want to add an '-x test_threaded_import' to your command line. I'd like to do some more work in aid of finding out exactly which test cases in unittest-based are leaking the references. Finding out which of the flat-file style tests in test_types is leaking is not going to be fun... (seems to be leaking a list of integers, oh dear hope that's not my fault again :-/). Cheers, mwh -- We've had a lot of problems going from glibc 2.0 to glibc 2.1. People claim binary compatibility. Except for functions they don't like. -- Peter Van Eynde, comp.lang.lisp

7 20

file read-ahead with Mac end-of-line
by Skip Montanaro 18 Aug '03

18 Aug '03

A bug was reported against the csv module, claiming (rightly so) that the csv module was not properly parsing files which use Mac line endings. I tracked the problem down to an apparent defiency in readahead_get_line_skip() in fileobject.c. It believes that only \n can terminate a line. The patch below fixes my csv module problem, but I wonder if it's the correct fix. Suppose you're using Mac line endings and encounter a \n before a \r? This function will return a too-short line. (Of course, it would without the patch as well.) I don't know how (or if) this should work with universal newline support. We expect files to be opened in binary mode, so I don't know if universal newline support applies. In short, does this look like the correct patch, closer to the correct behavior than the current setup, or no improvement at all? Thx, Skip cvs diff fileobject.c Index: fileobject.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Objects/fileobject.c,v retrieving revision 2.179 diff -c -r2.179 fileobject.c *** fileobject.c 18 May 2003 12:56:25 -0000 2.179 --- fileobject.c 18 Aug 2003 03:13:38 -0000 *************** *** 1803,1810 **** return (PyStringObject *) PyString_FromStringAndSize(NULL, skip); bufptr = memchr(f->f_bufptr, '\n', len); if (bufptr != NULL) { ! bufptr++; /* Count the '\n' */ len = bufptr - f->f_bufptr; s = (PyStringObject *) PyString_FromStringAndSize(NULL, skip+len); --- 1803,1812 ---- return (PyStringObject *) PyString_FromStringAndSize(NULL, skip); bufptr = memchr(f->f_bufptr, '\n', len); + if (bufptr == NULL) + bufptr = memchr(f->f_bufptr, '\r', len); if (bufptr != NULL) { ! bufptr++; /* Count the '\n' or '\r' */ len = bufptr - f->f_bufptr; s = (PyStringObject *) PyString_FromStringAndSize(NULL, skip+len);

3 8

PyMapping_Check Bug?
by Skip Montanaro 18 Aug '03

18 Aug '03

Passing along a fragment from the psycopg list where Federico Di Gregorio (fog(a)initd.org) wrote: .... because on python 2.3 PyMapping_Check apperently return True for every new-type instance.... Is this true? If so, is it a bug? Skip

3 3

RE: [spambayes-dev] World domination
by Meyer, Tony 18 Aug '03

18 Aug '03

> front page as the 7th "most active" project (of 67,000) at SF > last week. There have been at least 4,300 downloads of the > OL addin from SF. #6 now...maybe one fewer beating today? ;) Interestingly, the number of downloads fell quite a lot over the last couple of days, but the ranking went up. Maybe it was a slow day for everyone... > Congratulations! I wish I could claim more credit for > myself, but I've had little to do with it since last year. Well, to be fair, as nice as the Outlook plug-in is, I doubt many people would be recommending it if it didn't actually do the business of correctly classifying mail. You can have as much of that credit as you can get before Gary et. al. realise that there is credit to be taken and come for their share... :) > Nevertheless, beatings will continue until it's #1 <wink>. I don't know if I've ever seen the sf page without "Compiere ERP + CRM Business Solution", "Gaim", and "phpMyAdmin" in the top five. Pushing them out will be quite a feat, but then there's bound to be a surge with the next release of the plug-in, so maybe that would do it. =Tony Meyer

1 0