[Python-Dev] Module renaming and pickle mechanisms

M.-A. Lemburg mal at egenix.com
Sat May 17 11:05:55 CEST 2008


I'd like to bring a potential problem to attention that is caused
by the recent module renaming approach:

Object serialization protocols like e.g. pickle usually store the
complete module path to the object class together with the object.

They access this module path by looking at the __module__ attribute
of the object classes.

With the renaming, all objects which use classes from the renamed
modules will now refer to the renamed modules in their serialized
form, e.g. queue.Queue instead of Queue.Queue (just to name one
example).

While this is nice for forward compatibility, it causes rather serious
problems for making object serialization backwards compatible, since
the older Python versions can no longer unserialize objects due
to missing modules.

This can happen in client-server setups where e.g. the server
uses Python 2.6 and the clients some other Python version (e.g.
Python 2.5).

It can also happen in storage setups where Python
objects are stored using e.g. pickle, ZODB being a prominent
example. As soon as a Python 2.6 application starts writing to
such storages, Python 2.5 and lower versions will no longer be
able to read back all the data.

Now, I think there's a way to solve this puzzle:

Instead of renaming the modules (e.g. Queue -> queue), we leave
the code in the existing modules and packages and instead add
the new module names and package structure with pointers and
redirects to the existing 2.5 modules.

Code can (and probably should) still be changed to try to import
the new module name. In cases where backwards compatibility is
needed, this can also be done using

try:
     import newname
except ImportError:
     import oldname

Later on, when porting applications to 3.0, the 2to3 script can
then apply the final renaming in the source code.

Example:

queue.py:
---------
import sys, Queue
sys.modules[__name__] = Queue

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 17 2008)
 >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
 >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
 >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
            Registered at Amtsgericht Duesseldorf: HRB 46611


More information about the Python-Dev mailing list