[SciPy-user] Fast saving/loading of huge matrices

[Francesc Altet]

El dj 19 de 04 del 2007 a les 18:01 -0500, en/na Ryan Krauss va
escriure:
> I have a very similar question.  Pytables clearly has much more
> capability than I need and the documentation is a bit intimidating.  I
> have tests that involve multiple channels of data that I need to
> store.  Can you give a simple example of using pytables to store 3
> seperate Nx1 vectors in the same file and easily retreive the
> individual channels.  The cPickle equivalent would be something like:
> 
> v1=rand(1000,)
> v2=rand(1000,)
> mydict={'v1':v1,'v2':v2}
> 
> and then dump mydict to a pickle file.  How would I do this samething
> in pytables?

In a former message, I've shown how easily is to save a recarray with
pytables. You can achieve what you want by getting used to recarrays
(which are very powerful beasts for doing many numerical tasks). There
are many convenient ways of creating recarrays, and for your problem,
rec.fromarrays is best:

>>> v1=numpy.random.rand(10,)
>>> v2=numpy.random.rand(10,)
>>> ra=numpy.rec.fromarrays([v1,v2], dtype=[('v1', 'f8'), ('v2',
'f8')]) 
# save 'ra' in pytables
>>> tra=f.createTable('/', 'ra2', ra)
# read the entire 'ra' from disk
>>> tra[:]
array([(0.33512633154470473, 0.065904821918977619),
       (0.53020917640437315, 0.025182584316907786),
       (0.8336930367762152, 0.78541751699681861),
       (0.36623947675706092, 0.67927796809305641),
       (0.464207127024309, 0.85144582476536301),
       (0.012377388362621145, 0.4211020211753902),
       (0.3012702551957076, 0.71677896535796437),
       (0.38805504723782103, 0.48775066039074322),
       (0.92300245732715691, 0.52952648422581394),
       (0.1549704417226937, 0.070114112948997387)],
      dtype=[('v1', '<f8'), ('v2', '<f8')])

HTH,

-- 
Francesc Altet    |  Be careful about using the following code --
Carabos Coop. V.  |  I've only proven that it works, 
www.carabos.com   |  I haven't tested it. -- Donald Knuth