[SciPy-user] Fast saving/loading of huge matrices

[Gael Varoquaux]

On Thu, Apr 19, 2007 at 02:19:46PM -0500, Ryan Krauss wrote:
> The cPickle approach seems fast, I just wish there was some way to
> make the files smaller.  Is there a good way to do this that doesn't
> slow down the read time too much?

The problem with pickle is that the compatibility is not garantied from
one version of modules to another. I would use pytables for everything
else than temporary storage.

Another good thing about pytables is that the file is standard and can be
read by many other programs. I made the decision for our lab to
standardise on hdf5, even without knowing that it was one of the fastest
IO useable with scipy.

Thanks Cárabos for pytables !

Gaël