Why can't we pickle module objects?
import pickle pickle.dumps(pickle.dumps) 'cpickle\ndumps\np0\n.' pickle.dumps(pickle) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Runtimes\Python\lib\pickle.py", line 1374, in dumps Pickler(file, protocol).dump(obj) File "C:\Runtimes\Python\lib\pickle.py", line 224, in dump self.save(obj) File "C:\Runtimes\Python\lib\pickle.py", line 306, in save rv = reduce(self.proto) File "C:\Runtimes\Python\lib\copy_reg.py", line 70, in _reduce_ex raise TypeError, "can't pickle %s objects" % base.__name__ TypeError: can't pickle module objects
I know that you can't, but why not? You can pickle class objects and function objects in modules, and their static path is stored so that when you unpickle it, the correct module is imported and the function is retrieved from that module. It seems odd an inconsistent that you can't pickle the module object itself; can't it just store itself as a name, and have loads() import it and return the resultant module object? This isn't entirely of academic interest; I'm working with some code which is meant to pickle/unpickle arbitrary things, and occasionally it blows up when i accidentally pass in a module. It's always possible to work around the TypeErrors by just changing the stuff I pass in for serializing from the module to the exact function/class i want, but it seems like needless pain. Is there any good reason we don't let people pickle module objects using the same technique that we use to pickle classes and top-level functions? -Haoyi -Haoyi
I don't think there is any particularly *good* reason, although it would probably require some changes to the unpickling code specifically to support this case (currently when given the name 'foo.bar' it tries to import foo but never bar, IIRC). A bad reason might be that enabling this would make some people think that pickling a module would somehow store the code. But pickling classes and functions already has that disadvantage, so I don't think it's the real reason. Maybe you should just try your hands at a patch? Possibly you'd end up discovering the real reason. :-) --Guido On Mon, Jul 8, 2013 at 8:36 PM, Haoyi Li <haoyi.sg@gmail.com> wrote:
import pickle pickle.dumps(pickle.dumps) 'cpickle\ndumps\np0\n.' pickle.dumps(pickle) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Runtimes\Python\lib\pickle.py", line 1374, in dumps Pickler(file, protocol).dump(obj) File "C:\Runtimes\Python\lib\pickle.py", line 224, in dump self.save(obj) File "C:\Runtimes\Python\lib\pickle.py", line 306, in save rv = reduce(self.proto) File "C:\Runtimes\Python\lib\copy_reg.py", line 70, in _reduce_ex raise TypeError, "can't pickle %s objects" % base.__name__ TypeError: can't pickle module objects
I know that you can't, but why not? You can pickle class objects and function objects in modules, and their static path is stored so that when you unpickle it, the correct module is imported and the function is retrieved from that module. It seems odd an inconsistent that you can't pickle the module object itself; can't it just store itself as a name, and have loads() import it and return the resultant module object?
This isn't entirely of academic interest; I'm working with some code which is meant to pickle/unpickle arbitrary things, and occasionally it blows up when i accidentally pass in a module. It's always possible to work around the TypeErrors by just changing the stuff I pass in for serializing from the module to the exact function/class i want, but it seems like needless pain.
Is there any good reason we don't let people pickle module objects using the same technique that we use to pickle classes and top-level functions?
-Haoyi
-Haoyi
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- --Guido van Rossum (python.org/~guido)
You can work around it fairly easily with copyreg:
import sys, copyreg, pickle def reduce_mod(m): ... assert sys.modules[m.__name__] is m ... return rebuild_mod, (m.__name__,) ... def rebuild_mod(name): ... __import__(name) ... return sys.modules[name] ... copyreg.pickle(type(sys), reduce_mod) pickle.loads(pickle.dumps(sys)) <module 'sys' (built-in)>
-- Richard
You can work around it fairly easily with copyreg:
I actually just realized that after i downloading the python repo and was poking around how to proceed (actually, no, i found a way manually monkeypatching pickle.Pickler.dispatch, but i guess it's close enough). Regardless, *should* it work with modules? I'm in the mood to try and make a (first) contribution to python, and this seems like a feasible (not too hard) patch, If a patch is desirable/potentially desirable. I guess the (hypothetical) patch would cover all versions {2.7, 3.3} X {pickle, cPickle}? -Haoyi -Haoyi On Tue, Jul 9, 2013 at 5:22 PM, Richard Oudkerk <shibturn@gmail.com> wrote:
You can work around it fairly easily with copyreg:
import sys, copyreg, pickle def reduce_mod(m): ... assert sys.modules[m.__name__] is m ... return rebuild_mod, (m.__name__,) ... def rebuild_mod(name): ... __import__(name) ... return sys.modules[name] ... copyreg.pickle(type(sys), reduce_mod) pickle.loads(pickle.dumps(sys)**) <module 'sys' (built-in)>
-- Richard
______________________________**_________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/**mailman/listinfo/python-ideas<http://mail.python.org/mailman/listinfo/python-ideas>
On Tue, Jul 9, 2013 at 1:20 PM, Haoyi Li <haoyi.sg@gmail.com> wrote:
[...] I guess the (hypothetical) patch would cover all versions {2.7, 3.3} X {pickle, cPickle}?
It would be for 3.4 only as 2.7 and 3.3 are in feature freeze. Personally I stopped enjoying pickling classes because of the pains in renaming the source after saving the pickle file, so my opinion would be to leave pickle alone. Yuval
As I said I don't see much harm in it. On Tue, Jul 9, 2013 at 5:24 AM, Yuval Greenfield <ubershmekel@gmail.com> wrote:
On Tue, Jul 9, 2013 at 1:20 PM, Haoyi Li <haoyi.sg@gmail.com> wrote:
[...] I guess the (hypothetical) patch would cover all versions {2.7, 3.3} X {pickle, cPickle}?
It would be for 3.4 only as 2.7 and 3.3 are in feature freeze.
Personally I stopped enjoying pickling classes because of the pains in renaming the source after saving the pickle file, so my opinion would be to leave pickle alone.
Yuval
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- --Guido van Rossum (python.org/~guido)
participants (4)
-
Guido van Rossum -
Haoyi Li -
Richard Oudkerk -
Yuval Greenfield