[MATRIX-SIG] Changing shelve...

[Hinsen Konrad]

   Actually, the latest pickle and cPickle provide a generalized way
   of handling new types without subclassing.  This involves a protocol
   for "reducing" custom types to standard types and works very well
   for many applications.

But this won't be standard before Python 1.5. OK, that's soon enough
I hope...

   The major difficulty is handling arrays that are soooo big, that it
   isn't good enough to marshal their data to a portable string format in
   memory.  I'd guess that many people have arrays that are small enough
   that they can afford to marshal the array to a string.  In such cases
   the current reduce mechanism can work quite well.

I agree that most arrays will be small in practice. But that won't
help the people who do have large arrays, e.g. myself ;-)

   Hm....What about a special picklable type that would wrap:

     - A void pointer,
     - An object that contains the void pointer,
     - a size, and
     - a type code.

   So, an array's __reduce__ method would contruct one of these special
   objects and the picking machinery would be prepared to pickle and
   unpickle the object efficiently *and* portably.
   This last idea takes advantage of an assumption that we 
   want to pickle a block of memory that contains objects of
   some constant known C type.

Unfortunately the array data space does not have to be contiguous.
But it would be possible to turn an array into a *sequence* of
such "binary data" objects. That sounds like a good idea...

   I think this last idea can work.  Anybody want to volunteer
   to help me make it work? (I have so little time these days. :-()

I happen to suffer from exactly the same problem :-(

-------------------------------------------------------------------------------
Konrad Hinsen                          | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire   | Tel.: +33-76.88.99.28
Institut de Biologie Structurale       | Fax:  +33-76.88.54.94
41, Ave. des Martyrs                   | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France         | Nederlands/Francais
-------------------------------------------------------------------------------

_______________
MATRIX-SIG  - SIG on Matrix Math for Python

send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________