[Numpy-discussion] saving incrementally numpy arrays

[Juan Fiol]

Hi, thanks for all the answers. I am checking how to use pytables now, though I probably prefer to do it without further dependencies. I tried opening the file as 'append' and then pickle the array (because looking to the numpy.save it looked like what they did), but to retrieve the data then I have to load multiple times and concatenate (numpy.c_[]). I did not tried Robert suggestion yet, but it will probably happen the same and that is what Keith is seeing (though I may be wrong too).
If I do not find a suitable solution with only numpy I'll learn how to use pytables. Thanks and Best regards,
Juan

--- On Tue, 8/11/09, Keith Goodman <kwgoodman at gmail.com> wrote:

> From: Keith Goodman <kwgoodman at gmail.com>
> Subject: Re: [Numpy-discussion] saving incrementally numpy arrays
> To: "Discussion of Numerical Python" <numpy-discussion at scipy.org>
> Date: Tuesday, August 11, 2009, 7:46 PM
> On Tue, Aug 11, 2009 at 11:05 AM,
> Robert Kern<robert.kern at gmail.com>
> wrote:
> > On Mon, Aug 10, 2009 at 22:29, Juan Fiol<fiolj at yahoo.com>
> wrote:
> >> Hi, I am creating numpy arrays in chunks and I
> want to save the chunks while my program creates them. I
> tried to use numpy.save but it failed (because it is not
> intended to append data). I'd like to know what is, in your
> opinion, the best way to go. I will put a few thousands
> every time but building up a file of several Gbytes. I do
> not want to put into memory
> >> all previous data each time. Also I cannot wait
> until the program finishes, I must save partial results
> periodically. Thanks, any help will be appreciated
> >
> > As others mentioned, PyTables is an excellent,
> complete solution. If
> > you still want to write your own, then you can pass an
> open file
> > object to numpy.save() in order to append. Just open
> it with the mode
> > 'a+b' and seek to the end.
> >
> >  f = open('myfile.npy', 'a+b')
> >  f.seek(0, 2)
> >  numpy.save(f, chunk)
> >  f.close()
> 
> That looks nice. What am I doing wrong?
> 
> >> x = np.array([1,2,3])
> >> y = np.array([4,5,6])
> >>
> >> f = open('myfile.npy', 'a+b')
> >> np.save(f, x)
> >> f.seek(0, 2)
> >> np.save(f, y)
> >> f.close()
> >>
> >> xy = np.load('myfile.npy')
> >> xy
>    array([1, 2, 3])
> 
> I was expecting something like array([1, 2, 3, 4, 5, 6]).
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>