[Twisted-Python] pickle apparently doesn't go with curry ...
Twisted Gurus:
I'm using the "curry" recipe from the Python Cookbook
(pretty much verbatim), and it appears to work fine in my
code, but cPickle isn't happy about it:
Traceback (most recent call last):
File "/usr/local/bin/mktap", line 31, in ?
run()
File "/usr/local/lib/python2.2/site-packages/twisted/scripts/mktap.py", line 180, in run
a.save()
File "/usr/local/lib/python2.2/site-packages/twisted/internet/app.py", line 508, in save
dumpFunc(self, f)
File "/usr/local/lib/python2.2/site-packages/twisted/internet/app.py", line 490, in dumpFunc
_dump(obj, file, 1)
cPickle.PicklingError: Can't pickle
On Tuesday 12 November 2002 11:06 am, Steve Waterbury wrote:
Twisted Gurus:
Not yet, but I'd like to be one day. ;-)
I'm using the "curry" recipe from the Python Cookbook (pretty much verbatim), and it appears to work fine in my code, but cPickle isn't happy about it:
The short answer is that Python doesn't currently pickle nested class and fucntion objects, i.e. those not defined at the top level of the module. This may change in the future, and there was just some discussion about this on the python-dev list last week (including some clever hacks that might work for you as well) Start here and browse the archive by thread: http://mail.python.org/pipermail/python-dev/2002-November/029874.html You might also find some useful information in an article I just wrote about Python's pickling capabilities: http://www.ibm.com/developerworks/library/l-pypers.html Hope that helps. -- Patrick K. O'Brien Orbtech http://www.orbtech.com/web/pobrien ----------------------------------------------- "Your source for Python programming expertise." -----------------------------------------------
"Patrick K. O'Brien"
On Tuesday 12 November 2002 11:06 am, Steve Waterbury wrote:
Twisted Gurus:
Not yet, but I'd like to be one day. ;-)
I'm using the "curry" recipe from the Python Cookbook (pretty much verbatim), and it appears to work fine in my code, but cPickle isn't happy about it:
The short answer is that Python doesn't currently pickle nested class and fucntion objects, i.e. those not defined at the top level of the module.
Unless something's changed since I last looked, you can't pickle functions at all. And this is a particularly hopeless case. How did you expect it to work at all? Think like an implementer! Cheers, M. -- Darned confusing, unless you have that magic ingredient coffee, of which I can pay you Tuesday for a couple pounds of extra-special grind today. -- John Mitchell, 11 Jan 1999
On Tue, Nov 12, 2002 at 07:20:12PM +0000, Michael Hudson wrote:
"Patrick K. O'Brien"
writes: On Tuesday 12 November 2002 11:06 am, Steve Waterbury wrote:
Twisted Gurus:
Not yet, but I'd like to be one day. ;-)
I'm using the "curry" recipe from the Python Cookbook (pretty much verbatim), and it appears to work fine in my code, but cPickle isn't happy about it:
The short answer is that Python doesn't currently pickle nested class and fucntion objects, i.e. those not defined at the top level of the module.
Unless something's changed since I last looked, you can't pickle functions at all.
def foo(): ... pass ... import cPickle cPickle.dump(foo, open('testpickle', 'w')) bar = cPickle.load(open('testpickle'))
I'd imagine this would work in most versions of python, given that people have been pickling objects for countless generations.
jml@mumak.net (Jonathan M. Lange) writes:
def foo(): =2E.. pass =2E.. import cPickle cPickle.dump(foo, open('testpickle', 'w')) bar =3D cPickle.load(open('testpickle'))
I'd imagine this would work in most versions of python, given that people have been pickling objects for countless generations.
Oops, my mistake. But note that whats been pickled is a reference to __main__.foo, not the function object itself: $ python2 Python 2.2.1 (#1, Apr 9 2002, 13:10:27) [GCC 2.96 20000731 (Red Hat Linux 7.1 2.96-98)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
def foo(): ... pass ... import pickle pickle.dump(foo, open("/tmp/t", "w"))
$ python2 Python 2.2.1 (#1, Apr 9 2002, 13:10:27) [GCC 2.96 20000731 (Red Hat Linux 7.1 2.96-98)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import pickle foo = 1 pickle.load(open("/tmp/t")) 1
so def f(): def g(): return 1 return g pickle.dump(f(), ...) still isn't going to work... (without fiddling). Cheers, M.
On Wednesday 13 November 2002 04:50 am, Michael Hudson wrote:
jml@mumak.net (Jonathan M. Lange) writes:
def foo():
=2E.. pass =2E..
import cPickle cPickle.dump(foo, open('testpickle', 'w')) bar =3D cPickle.load(open('testpickle'))
I'd imagine this would work in most versions of python, given that people have been pickling objects for countless generations.
Oops, my mistake. But note that whats been pickled is a reference to __main__.foo, not the function object itself:
That's true about all code-like objects. The code itself doesn't get pickled, only the state. The code is retrieved by the unpickler via the references to the location of the code - the package name, module name, class name, function name, etc. -- Patrick K. O'Brien Orbtech http://www.orbtech.com/web/pobrien ----------------------------------------------- "Your source for Python programming expertise." -----------------------------------------------
Michael Hudson wrote:
"Patrick K. O'Brien"
writes: On Tuesday 12 November 2002 11:06 am, Steve Waterbury wrote:
Twisted Gurus:
Not yet, but I'd like to be one day. ;-)
I'm using the "curry" recipe from the Python Cookbook (pretty much verbatim), and it appears to work fine in my code, but cPickle isn't happy about it:
The short answer is that Python doesn't currently pickle nested class and fucntion objects, i.e. those not defined at the top level of the module.
Unless something's changed since I last looked, you can't pickle functions at all.
And this is a particularly hopeless case. How did you expect it to work at all? Think like an implementer!
Gee, thanks for the encouragement ... I got it to work, and now I'm not gonna tell you how ... nyah nyah! :^) Cheers, -- Steve.
Steve Waterbury
Michael Hudson wrote:
And this is a particularly hopeless case. How did you expect it to work at all? Think like an implementer!
Gee, thanks for the encouragement ... I got it to work, and now I'm not gonna tell you how ... nyah nyah! :^)
Sorry, I wasn't *trying* to be offensive. But if one understands how pickling works, one knows not to try things like this... Cheers, M. -- If you don't use emacs, you're a pathetic, mewling, masochistic weakling and I can't be bothered to convert you. -- Ron Echeverri
On Tuesday 12 November 2002 01:20 pm, Michael Hudson wrote:
"Patrick K. O'Brien"
writes: The short answer is that Python doesn't currently pickle nested class and fucntion objects, i.e. those not defined at the top level of the module.
Unless something's changed since I last looked, you can't pickle functions at all.
Sure you can:
import pickle pickle.dumps
p = pickle.dumps(pickle.dumps) p 'cpickle\ndumps\np0\n.' dir <built-in function dir> p = pickle.dumps(dir) p 'c__builtin__\ndir\np0\n.'
-- Patrick K. O'Brien Orbtech http://www.orbtech.com/web/pobrien ----------------------------------------------- "Your source for Python programming expertise." -----------------------------------------------
I have a stupid device that's talking to a Twisted web server that demands HTTP headers to be capitalized Like-This, however Twisted only capitalizes the first character.. On line 567 in twisted/protocol/http.py (Twisted 1.0.0) what we have right now is: l.append("%s: %s\r\n" % (name.capitalize(), value)) # content-length turns into Content-length But what my stupid device wants to see is: l.append("%s: %s\r\n" % ('-'.join(map(lambda n:n.capitalize(),name.split('-'))), value)) # content-length turns into Content-Length Any chance someone could patch that up? -bob
Bob Ippolito wrote:
But what my stupid device wants to see is: l.append("%s: %s\r\n" % ('-'.join(map(lambda n:n.capitalize(),name.split('-'))), value)) # content-length turns into Content-Length
Unrelated yet related, string.capwords() would perhaps be cleaner. string.capwords(value, '-') lurkingly, carlos
On Tue, Nov 12, 2002 at 04:03:24PM -0500, Bob Ippolito wrote:
I have a stupid device that's talking to a Twisted web server that demands HTTP headers to be capitalized Like-This, however Twisted only capitalizes the first character.. [...] Any chance someone could patch that up?
Doubtful. http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2 "Field names are case-insensitive" Jeff
On Tuesday, Nov 12, 2002, at 16:19 America/New_York, Jeff Epler wrote:
On Tue, Nov 12, 2002 at 04:03:24PM -0500, Bob Ippolito wrote:
I have a stupid device that's talking to a Twisted web server that demands HTTP headers to be capitalized Like-This, however Twisted only capitalizes the first character.. [...] Any chance someone could patch that up?
Doubtful.
http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2 "Field names are case-insensitive"
Well, it's already doing capitalize(), might as well capitalize it the way that everything else does. It certainly wouldn't break anything to send it with string.capwords(name, '-') instead of name.capitalize(). -bob
On Tue, 12 Nov 2002 16:40:00 -0500, Bob Ippolito
Well, it's already doing capitalize(), might as well capitalize it the way that everything else does. It certainly wouldn't break anything to send it with string.capwords(name, '-') instead of name.capitalize().
If I recall correctly, it's doing capitalize() because it was *originally* just sending the headers lowercase, and somebody else had a broken device and/or script that was expecting headers to be formatted exactly as they are now (in particular, Cache-control). If people are really encountering broken devices like this on a regular basis (and they seem to be) perhaps it would be best to make the header-formatting code by default do all-lowercase, but be easily pluggable? This would have the advantage of eliminating a bunch of unnecessary string allocations during HTTP request processing. Another option would be to have the headers decided on a case-by-case basis, by looking up in a hashtable. This would let us have a little popularity contest to determine how existing servers send their "case-insensitive" headers and record each decision... -- | <`'> | Glyph Lefkowitz: Traveling Sorcerer | | < _/ > | Lead Developer, the Twisted project | | < ___/ > | http://www.twistedmatrix.com |
On Tue, 12 Nov 2002 12:06:07 -0500, Steve Waterbury
Any ideas?
Well, others have already told you why this doesn't work :-) but if I may suggest something... No offense to the very bright folks who brought us the Python Cookbook, because I find much of it very valuable, but some of their recepies are overly complicated with bizarre side effects that are really not obvious until you've been using them for a while. Persistence is a particularly dark corner, both with pickle and without. Two of my least favorite recipies in this regard are "Borg" and "curry". While interesting from a technical perspective on Python's expressiveness, they're rarely expressing what you want :). For example, to translate the curry example you gave to a slightly-more-verbose but much-more-straightforward python idiom: class curry: def __init__(self, func, *create_time_args, **create_time_kwds): self.func = func self.create_time_args = create_time_args self.create_time_kwds = create_time_kwds def __call__(self, *call_time_args, **call_time_kwds): args = self.create_time_args + call_time_args kwds = self.create_time_kwds.copy() kwds.update(call_time_kwds) return self.func(*args, **kwds) The cost of a much more straightforward interaction with other Python tools -- not just for persistence, but for example, for debugging -- comes to exactly 1 additional line of code. -- | <`'> | Glyph Lefkowitz: Traveling Sorcerer | | < _/ > | Lead Developer, the Twisted project | | < ___/ > | http://www.twistedmatrix.com |
participants (8)
-
Bob Ippolito
-
Carlos Eberhardt
-
Glyph Lefkowitz
-
Jeff Epler
-
jml@mumak.net
-
Michael Hudson
-
Patrick K. O'Brien
-
Steve Waterbury