[Python-bugs-list] [ python-Bugs-496873 ] structseqs unpicklable

noreply@sourceforge.net noreply@sourceforge.net
Tue, 05 Mar 2002 05:32:12 -0800


Bugs item #496873, was opened at 2001-12-26 16:18
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=496873&group_id=5470

Category: Type/class unification
Group: Python 2.2.1 candidate
Status: Closed
Resolution: Fixed
Priority: 5
Submitted By: bixentxo (bixentxo)
Assigned to: Michael Hudson (mwh)
Summary: structseqs unpicklable

Initial Comment:
The following code produces and endless loop
on python 2.2 ( ok on python 2.1 ). The
file being saved by cPickle.dump grows
indefinetely.

-----------------------------
import time, cPickle

created = time.localtime()

fd = open('/tmp/bla.pickle','w')
cPickle.dump(created,fd)
[...endless loop... ]

----------------------------------------------------------------------

>Comment By: Guido van Rossum (gvanrossum)
Date: 2002-03-05 08:32

Message:
Logged In: YES 
user_id=6380

Thanks!  I think the change to fixed size would have to be a
2.3 feature.

I guess it wasn't done that way because nobody realized it
had any advantage.

----------------------------------------------------------------------

Comment By: Michael Hudson (mwh)
Date: 2002-03-05 08:29

Message:
Logged In: YES 
user_id=6656

Checked in a very-slightly different patch & simple test case.

Yes, that's what I thought too.  I wonder why that wasn't
done originally?  Time to dig into archives, I guess.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2002-03-04 18:14

Message:
Logged In: YES 
user_id=6380

The patch looks OK to me.

I wonder if the solution to the varying size isn't to fix
the size, and simply supply -1 for the fields a platform
doesn't have?


----------------------------------------------------------------------

Comment By: Michael Hudson (mwh)
Date: 2002-03-03 11:48

Message:
Logged In: YES 
user_id=6656

You seem to have divined my intent, but I've changed the 
summary to be clearer.

Found a spare hour or so; can you find the time to review 
the attached patch?  I would really like to get this into 
2.2.1 -- I think having pickle.dumps(time.localtime())
break is a bit gratuitous.

I haven't written a proper test-suite yet, but hand tests
suggest it does the job for os.stat_result, os.statvfs_result
& time.struct_time both for cPickle and pickle.

There's still a gremlin in here; os.stat_result can have a
varying number of fields & so a varying number of constructor
arguments across platforms which will mean stat_results
pickled on one platform may not unpickle on another.  Not 
sure what to do about that.


----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2002-03-01 17:50

Message:
Logged In: YES 
user_id=6380

MWH, I don't understand your comment.

I've already checked in a bugfix for this (long ago). The
references are below in my comment of 12/27/01. Did you copy
those into 2.2.1 already?

I'm not sure I have time to come up with a way to make these
types actually picklable...

----------------------------------------------------------------------

Comment By: Michael Hudson (mwh)
Date: 2002-03-01 03:46

Message:
Logged In: YES 
user_id=6656

I think Guido is the only person who knows how to fix this.

Anyone else who does is encouraged to jump in, of course!

----------------------------------------------------------------------

Comment By: Simo Salminen (frangen)
Date: 2002-01-31 08:13

Message:
Logged In: YES 
user_id=291461

os.stat results cant be pickled either.

>>> pickle.dump(os.stat('.'), open('foo', 'w'))
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/opt/local/lang/python2.2/lib/python2.2/pickle.py", 
line 969, in dump
    Pickler(file, bin).dump(object)
  File "/opt/local/lang/python2.2/lib/python2.2/pickle.py", 
line 115, in dump
    self.save(object)
  File "/opt/local/lang/python2.2/lib/python2.2/pickle.py", 
line 185, in save
    tup = reduce()
  
File "/opt/local/lang/python2.2/lib/python2.2/copy_reg.py", 
line 56, in _reduce
    state = base(self)
TypeError: constructor takes exactly 13 arguments (10 given)


----------------------------------------------------------------------

Comment By: bixentxo (bixentxo)
Date: 2001-12-27 15:31

Message:
Logged In: YES 
user_id=395354

As for me, it is not a problem anymore
as I found the 'tuple' solution straight away.

It is not that I needed pickling struct_time
is that I had no reason why not doing it.

Only that it DID break my code.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2001-12-27 13:02

Message:
Logged In: YES 
user_id=31435

Re "does anybody need to pickle time structs?", I think a 
better question is "does anybody pickle time.localtime() 
results?".  The existence of the bug report suggests yes, 
and the OP is right that this worked in 2.1 (and 2.0, 
and ...).

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-12-27 11:32

Message:
Logged In: YES 
user_id=6380

Actually, the bug was in copy_reg._reduce: it was possible
that this would not make any progress, returning an instance
of the same type that it was passed, thus causing the
infinite recursion. I've checked in a fix that raises an
exception when this situation is detected.
(copy_reg.py:1.10)

Now, the question is, does anybody need to pickle time
structs? I would recommend converting them into a tuple
before pickling: pickle.dumps(tuple(time.localtime())) works
fine.

The statvfs issue was an unrelated bug; I've checked in a
fix for that too. (posixmodule.c:2.217)

Both are 2.2.1 candidates (I only noted this in the
posixmodule.c CVS checkin message by mistake).

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-12-27 09:06

Message:
Logged In: YES 
user_id=6380

It gets in an infinite loop with pickle too, so I think this
is definitely a bug in the magical type used for localtime.
The stacktrack looks like this:

  File "/usr/local/lib/python2.2/pickle.py", line 370, in
save_tuple
    save(element)
  File "/usr/local/lib/python2.2/pickle.py", line 215, in
save
    self.save_reduce(callable, arg_tup, state)
  File "/usr/local/lib/python2.2/pickle.py", line 241, in
save_reduce
    save(arg_tup)
  File "/usr/local/lib/python2.2/pickle.py", line 221, in
save
    f(self, object)

repeated an infinite number of times.

Note that we should look at the other objects that use
structseq.c/h too. pickle.dumps(os.stat("/")) gives

TypeError: constructor takes exactly 13 arguments (10 given)

pickle.dumps(os.statvfs("/")) gives

pickle.PicklingError: Can't pickle <type
'posix.statvfs_result'>: it's not the same object as
posix.statvfs_result

Looks like a variety of bugs. :-(



----------------------------------------------------------------------

Comment By: Michael Hudson (mwh)
Date: 2001-12-27 06:50

Message:
Logged In: YES 
user_id=6656

Another factoid: if you back out the change that renamed the 
"struct_time" type to "time.struct_time", the crash goes 
away, to be replaced with 

cPickle.PicklingError: Can't pickle <type 'struct_time'>: 
it's not found as __builtin__.struct_time

so I suspect this bug has been lurking for some time.


----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2001-12-27 01:06

Message:
Logged In: YES 
user_id=31435

Boosted priority, assigned to Guido.

A stack fault is faster to produce via the one-liner

cPickle.dumps(time.localtime())

There's unbounded recursion.

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=496873&group_id=5470