[PYTHON MATRIX-SIG] Re: Saving HUGE arrays

[Konrad Hinsen]

> > How about a stream-like protocol, i.e. implementing read() and write()?
> > That could be done for non-contiguous arrays, and still be efficient.
> 
> This makes some sense, except that it wouldn't (really) help for the
> situation where you have a HUGE array (HUGE == too big to copy in
> virtual memory) and you need to write it to disk.  If the array is
> contiguous, a single write() system call could write the entire thing
> to disk.

One solution would be to add some nice stream protocol to Python.
Files would be one kind of stream, memory blocks another one.
Then file.write(memory_block) should do the following:

1) The file object asks the memory block object for the next
   chunk of data (however large), imposing a maximum (which
   would actually be infinite in case of files).

2) The memory block object provides a pointer and length indicator for
   such a block and remembers where to continue, or an end-of-stream
   indication when there is no more data.

3) The file object writes the data to the disk file.

4) Goto 1.

Such a protocol would transfer data in the largest blocks that both
objects can handle.
  
> Next try (though this is getting complicated :-( ): how about a
> "sequence-of-buffers" interface.  E.g.:

I am not too fond of solutions that just try to work around one
special problem. But of course it's better than raising exceptions
for arrays.

-- 
-------------------------------------------------------------------------------
Konrad Hinsen                          | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire   | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale       | Fax:  +33-4.76.88.54.94
41, av. des Martyrs                    | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France         | Nederlands/Francais
-------------------------------------------------------------------------------

=================
MATRIX-SIG  - SIG on Matrix Math for Python

send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================