[MATRIX-SIG] Changing shelve...

[Jim Fulton]

Janko Hauser wrote:
> 
> As far as I understand pickle is rather slow,

cPickle is orders of magnitude faster than pickle

> not very memory conserving

Hm, well, currently for arrays, you would have to marshal to a string
in memory, temporarily doubling your memory usage.

> and at the end not portable between different applications.

I'ts very portable, assuming that the applications are all written
in Python. :-)  Pickles are certainly portable from machine to 
machine.

> Wouldn't it be good to have a simple routine to save raw
> Numpy-arrays in a portable (machine and application) way.

Yes.
 
> I think I can build something like that on top of the netcdf
> module. The main drawback of this data format is that it is not easy
> to append to a file. But I don't know if HDF is more flexible in this
> regard. PDB from llnl can this but other applications can't read this
> without extensions.
> 
> Besides, also pickle can't append, right?

Wrong. In fact, I suppose if you wanted to write a pickle file
that only contained a single array, you could write the file in
multiple pickles, avoiding the large memory usage.

For that matter, if you were writing simple objects (like arrays) 
to a pickle file, it would be easy to write pickling and unpickling
routines for other non-python applications.

I imagine that when we commercialize our pickle-based OODBMS, we'll
provide some picklers and unpicklers for other languages.

Jim

_______________
MATRIX-SIG  - SIG on Matrix Math for Python

send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________