[Numpy-discussion] np.savez not multi-processing safe, alternatives?

[Wes McKinney]

I have a process that stores a number of sets of 3 arrays output which can
either be stored as a few .npy files or an .npz file with the same keys in
each file (let's say, writing roughly 10,000 npz files, all containing the
same keys 'a', 'b', 'c'). If I run multiple processes on the same machine
(desirable, since they heavily database-IO-bound), over a period of hours
some of the npz-writes will collide and fail due to the use of tempfile and
tempfile.gettempdir() (either one of the .npy subfiles will be locked for
writing or will get os.remove'd while the zip file is being written).

So my question-- recommendations for a way around this, or possible to
change the savez function to make it less likely to happen? (I am on Win32)

Thanks,
Wes
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090330/c08607d5/attachment.html>