[PYTHON MATRIX-SIG] More on Less Copying

[Konrad Hinsen]

> > > You are adressing an important topic that has largely been neglected
> > > so far: file I/O, both in binary and in text form, suitable for
> > > interchanging data with non-Python programs
> 
> This problem can be easily solved with an extension module rather than by 
> adding file io methods to array objects.  For many reasons I think this is 

It's fine with me to have I/O in a separate module or even several of
them. That seems like a minor technical detail. However, it would be
nice to have a way to create a non-initialized array at the Python
level.

> the right solution.  In only the most extreme cases will the temporary 
> double memory currently required for an array on file input be a problem. 

I couldn't disagree more. I run into that problem every day!  I have a
30 MB array in a file created by pickle, which I have to read
frequently. It takes a few minutes and the noise made by the disk
isn't pleasant at all.

>  Notice that this memory is used only for a very short time.  In general, 

That makes no difference once you need to access swap space.

> if your memory is so tight that you can't afford this brief hit on a single 
> array, then you're really pushing the limits of where NumPy is applicable 
> (vs. hand-crafted C).

Not at all. My arrays take almost all of physical memory, but there is
no problem during the calculations. Only I/O is a pain.

> I really want to keep the basic array objects as simple as possible with 
> the hopes that one day they can be added to the core python distribution. 

That makes sense, and I see no problem in moving I/O to an external module.
But we still need a good solution...
-- 
-------------------------------------------------------------------------------
Konrad Hinsen                          | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire   | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale       | Fax:  +33-4.76.88.54.94
41, av. des Martyrs                    | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France         | Nederlands/Francais
-------------------------------------------------------------------------------

_______________
MATRIX-SIG  - SIG on Matrix Math for Python

send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________