On Mon, 20 Dec 2004 15:03:13 -0500, Barry Warsaw wrote:

On Mon, 2004-12-20 at 12:48, Raymond Hettinger wrote:

...

Perhaps a rather quick Py2.4.1 would be in order.

+1

Nothing wrong with an incremental release, but none of these sound like critical bugs to me. Jeremy

On Mon, 20 Dec 2004 18:25:39 -0500, Aahz wrote:

On Mon, Dec 20, 2004, Jeremy Hylton wrote:

...

On Mon, 20 Dec 2004 15:03:13 -0500, Barry Warsaw wrote:

...

On Mon, 2004-12-20 at 12:48, Raymond Hettinger wrote:

...

Perhaps a rather quick Py2.4.1 would be in order.

+1

Nothing wrong with an incremental release, but none of these sound like critical bugs to me.

You don't think a blowup in marshal is critical? Mind expanding on that?

An undocumented extension to marshal causes a segfault. It's certainly a bug worth fixing. It doesn't sound like a critical bug to me. Jeremy

Hi Tim, On Mon, Dec 20, 2004 at 07:26:25PM -0500, Tim Peters wrote:

Still, it's a new-in-2.4 gimmick, and no pre-2.4 code could be using it. I suppose Armin found a use for it in 2.4, but I'm still scratching my head. If ZODB doesn't already depend on it, how useful can it be? QED

Some code in the 'py' lib used to use marshal to send simple objects between the main process and a subprocess. We ran into trouble when we extended the idea to a subprocess that would actually run via ssh on a remote machine, and the remote machine's Python version didn't match the local one. The obvious quick fix was to set the 'version' argument to 0 and pretend that it would be fine forever. Now that we essentially can't use this trick any more because of the 2.4.0 bug, we reverted to repr/eval, which is quite slower (and actually not guaranteed to work across Python versions either: string escapes sometimes change). We avoid to use cPickle because we want to be sure that only "simple enough" objects are sent over this way -- essentially nothing that depends on a precise Python module to be installed and identical on both machines. Armin

[Armin Rigo]

Some code in the 'py' lib used to use marshal to send simple objects between the main process and a subprocess. We ran into trouble when we extended the idea to a subprocess that would actually run via ssh on a remote machine, and the remote machine's Python version didn't match the local one. The obvious quick fix was to set the 'version' argument to 0 and pretend that it would be fine forever.

Yes, I believed you had *some* use for it <wink>.

Now that we essentially can't use this trick any more because of the 2.4.0 bug,

Well, you can still use 2.3.4 -- or wait for 2.4.1 -- or use your patched 2.4. Or use the stock 2.4, but set up a "marshal server" running 2.3.4 <heh>.

we reverted to repr/eval, which is quite slower (and actually not guaranteed to work across Python versions either: string escapes sometimes change).

Really? The precise rules str's __repr__ uses for which escapes to produce certainly change, but I don't recall any case outside Unicodeland where a new string escape was ever introduced. So, e.g., current Python str.__repr__() produces '\n' for a newline, while long-ago Pythons produced '\012', but all versions of Python *accept* either form of escape. The biggest change of this kind was moving from octal escapes to hex escapes in Python 2.1, but hex escapes have always been accepted -- repr() just didn't produce them before 2.1.

We avoid to use cPickle because we want to be sure that only "simple enough" objects are sent over this way -- essentially nothing that depends on a precise Python module to be installed and identical on both machines.

It's possible (but irksome) to subclass pickle.py's Pickler class, and override its save_global() and save_inst() methods. For example, replace them with one-liners that just raise an exception. Then any attempt to pickle an object requiring a specific module will raise that exception. But if you're worried about speed, going thru pickle.py is significantly slower than going thru repr().

Raymond Hettinger wrote:

...

Perhaps a rather quick Py2.4.1 would be in order.

Ideally, it should include other important fixes: [...] * Fix for off-by-one bug in urllib.URLopener.retrieve http://www.python.org/sf/810023 (assigned to me)

Is http://www.python.org/sf/1062060 perhaps of similar importance? (fix for urllib.urlretrieve silently truncating download)

That seems reasonable to me. There is no point in having the error pass silently. Raymond

Irmen de Jong wrote:

Also, I'm not sure how a test-case should be constructed for this patch? Can the Python regression test download stuff as part of a test? Or is there some other way to make a testcase for this.

Hmm - perhaps start a server on the local machine at the start of the test, and tear it down at the end? you've then got full control of that server and can make it do whatever you want. Or replace the socket objects with mock objects? Tim Delaney

On Tue, Dec 21, 2004 at 08:00:22PM +0100, Irmen de Jong wrote:

Thanks for those suggestions. Let's see what I can concoct.

Never made (or studied in detail) a python regression test case before so now is a good time :)

You might be especially interested in 'class urlopen_HttpTests' in test_urllib.py and all the Mock* classes in test_urllib2.py. BTW, don't hesitate to ask for help in the bug/patch. I'm happy to help, though I might be a bit slow to answer. Cheers, Johannes