[SciPy-user] storing numpy arrays

[Robert Kern]

On Thu, Mar 12, 2009 at 18:14, Dav Clark <dav at alum.mit.edu> wrote:
> On Mar 12, 2009, at 3:57 PM, Robert Kern wrote:
>
>> On Thu, Mar 12, 2009 at 17:50, Michael Hearne <mhearne at usgs.gov>
>> wrote:
>>> Conor - Thank you for your suggestion.  I am in fact already using
>>> pytables in another part of my application.  However, using
>>> PyTables for
>>> this portion is not a desirable option.
>>>
>>> Perhaps I should rephrase the question:
>>> What is the the most efficient way to serialize a numpy array into a
>>> structure that _could_ be written to a disk, or stuffed into a
>>> database,
>>> etc?
>>
>> Use numpy.lib.format.{read,write}_array() to use the NPY format, which
>> should be the most robust and efficient way to do what you want. I
>> have used this to store numpy arrays as MySQL BLOBs before, and it
>> worked quite well.
>
> It sounds like you are trying to avoid writing a file in order to get
> your blob?  I don't know if you could provide a "file-like" object
> straight to your database for write_array.

No you can't. Just use cStringIO.StringIO() to collect the string.
This is standard idiom.

> You could certainly use
> the dumps function, which returns a pickled array (which is not as
> efficient), or maybe do something clever storing the array.data (a
> python "buffer"), array.dtype.str and array.shape attributes?

The NPY format is just such a clever storage.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco