[Python-bugs-list] [ python-Bugs-496873 ] structseqs unpicklable
noreply@sourceforge.net
noreply@sourceforge.net
Tue, 05 Mar 2002 05:29:35 -0800
Bugs item #496873, was opened at 2001-12-26 21:18
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=496873&group_id=5470
Category: Type/class unification
Group: Python 2.2.1 candidate
>Status: Closed
>Resolution: Fixed
Priority: 5
Submitted By: bixentxo (bixentxo)
Assigned to: Michael Hudson (mwh)
Summary: structseqs unpicklable
Initial Comment:
The following code produces and endless loop
on python 2.2 ( ok on python 2.1 ). The
file being saved by cPickle.dump grows
indefinetely.
-----------------------------
import time, cPickle
created = time.localtime()
fd = open('/tmp/bla.pickle','w')
cPickle.dump(created,fd)
[...endless loop... ]
----------------------------------------------------------------------
>Comment By: Michael Hudson (mwh)
Date: 2002-03-05 13:29
Message:
Logged In: YES
user_id=6656
Checked in a very-slightly different patch & simple test case.
Yes, that's what I thought too. I wonder why that wasn't
done originally? Time to dig into archives, I guess.
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2002-03-04 23:14
Message:
Logged In: YES
user_id=6380
The patch looks OK to me.
I wonder if the solution to the varying size isn't to fix
the size, and simply supply -1 for the fields a platform
doesn't have?
----------------------------------------------------------------------
Comment By: Michael Hudson (mwh)
Date: 2002-03-03 16:48
Message:
Logged In: YES
user_id=6656
You seem to have divined my intent, but I've changed the
summary to be clearer.
Found a spare hour or so; can you find the time to review
the attached patch? I would really like to get this into
2.2.1 -- I think having pickle.dumps(time.localtime())
break is a bit gratuitous.
I haven't written a proper test-suite yet, but hand tests
suggest it does the job for os.stat_result, os.statvfs_result
& time.struct_time both for cPickle and pickle.
There's still a gremlin in here; os.stat_result can have a
varying number of fields & so a varying number of constructor
arguments across platforms which will mean stat_results
pickled on one platform may not unpickle on another. Not
sure what to do about that.
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2002-03-01 22:50
Message:
Logged In: YES
user_id=6380
MWH, I don't understand your comment.
I've already checked in a bugfix for this (long ago). The
references are below in my comment of 12/27/01. Did you copy
those into 2.2.1 already?
I'm not sure I have time to come up with a way to make these
types actually picklable...
----------------------------------------------------------------------
Comment By: Michael Hudson (mwh)
Date: 2002-03-01 08:46
Message:
Logged In: YES
user_id=6656
I think Guido is the only person who knows how to fix this.
Anyone else who does is encouraged to jump in, of course!
----------------------------------------------------------------------
Comment By: Simo Salminen (frangen)
Date: 2002-01-31 13:13
Message:
Logged In: YES
user_id=291461
os.stat results cant be pickled either.
>>> pickle.dump(os.stat('.'), open('foo', 'w'))
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/opt/local/lang/python2.2/lib/python2.2/pickle.py",
line 969, in dump
Pickler(file, bin).dump(object)
File "/opt/local/lang/python2.2/lib/python2.2/pickle.py",
line 115, in dump
self.save(object)
File "/opt/local/lang/python2.2/lib/python2.2/pickle.py",
line 185, in save
tup = reduce()
File "/opt/local/lang/python2.2/lib/python2.2/copy_reg.py",
line 56, in _reduce
state = base(self)
TypeError: constructor takes exactly 13 arguments (10 given)
----------------------------------------------------------------------
Comment By: bixentxo (bixentxo)
Date: 2001-12-27 20:31
Message:
Logged In: YES
user_id=395354
As for me, it is not a problem anymore
as I found the 'tuple' solution straight away.
It is not that I needed pickling struct_time
is that I had no reason why not doing it.
Only that it DID break my code.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2001-12-27 18:02
Message:
Logged In: YES
user_id=31435
Re "does anybody need to pickle time structs?", I think a
better question is "does anybody pickle time.localtime()
results?". The existence of the bug report suggests yes,
and the OP is right that this worked in 2.1 (and 2.0,
and ...).
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2001-12-27 16:32
Message:
Logged In: YES
user_id=6380
Actually, the bug was in copy_reg._reduce: it was possible
that this would not make any progress, returning an instance
of the same type that it was passed, thus causing the
infinite recursion. I've checked in a fix that raises an
exception when this situation is detected.
(copy_reg.py:1.10)
Now, the question is, does anybody need to pickle time
structs? I would recommend converting them into a tuple
before pickling: pickle.dumps(tuple(time.localtime())) works
fine.
The statvfs issue was an unrelated bug; I've checked in a
fix for that too. (posixmodule.c:2.217)
Both are 2.2.1 candidates (I only noted this in the
posixmodule.c CVS checkin message by mistake).
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2001-12-27 14:06
Message:
Logged In: YES
user_id=6380
It gets in an infinite loop with pickle too, so I think this
is definitely a bug in the magical type used for localtime.
The stacktrack looks like this:
File "/usr/local/lib/python2.2/pickle.py", line 370, in
save_tuple
save(element)
File "/usr/local/lib/python2.2/pickle.py", line 215, in
save
self.save_reduce(callable, arg_tup, state)
File "/usr/local/lib/python2.2/pickle.py", line 241, in
save_reduce
save(arg_tup)
File "/usr/local/lib/python2.2/pickle.py", line 221, in
save
f(self, object)
repeated an infinite number of times.
Note that we should look at the other objects that use
structseq.c/h too. pickle.dumps(os.stat("/")) gives
TypeError: constructor takes exactly 13 arguments (10 given)
pickle.dumps(os.statvfs("/")) gives
pickle.PicklingError: Can't pickle <type
'posix.statvfs_result'>: it's not the same object as
posix.statvfs_result
Looks like a variety of bugs. :-(
----------------------------------------------------------------------
Comment By: Michael Hudson (mwh)
Date: 2001-12-27 11:50
Message:
Logged In: YES
user_id=6656
Another factoid: if you back out the change that renamed the
"struct_time" type to "time.struct_time", the crash goes
away, to be replaced with
cPickle.PicklingError: Can't pickle <type 'struct_time'>:
it's not found as __builtin__.struct_time
so I suspect this bug has been lurking for some time.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2001-12-27 06:06
Message:
Logged In: YES
user_id=31435
Boosted priority, assigned to Guido.
A stack fault is faster to produce via the one-liner
cPickle.dumps(time.localtime())
There's unbounded recursion.
----------------------------------------------------------------------
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=496873&group_id=5470