[Numpy-discussion] iterate over multiple arrays

[Olivier Delalleau]

I agree with Robert, don't use locals(). I should have added a disclaimer
"this is very hackish and probably not a good idea", sorry ;)  (interesting
read: http://stackoverflow.com/questions/1450275/modifying-locals-in-python)

>From what you said I think what you really want is f to work in-place. This
also has the advantage of minimizing memory allocations.
If you can't directly modify f, you can also do:

def f_inplace(x):
  x[:] = f(x)

then just call map(f_inplace, [a, b, c, d]). However note that there will be
some memory temporarily allocated to store the result of f(x) (so it is not
as optimal as ensuring f directly works in-place).

In addition to the dictionary workaround mentioned by Robert, if it is not
practical to have all your variables of interest into a single dictionary,
you can instead declare your variables as one-element lists, or use a class
with a single field:

1. a = [numpy.array(...)]
    a[0] = f(a[0])

2. class ArrayHolder(object):
        def __init__(self, arr):
             self.arr = arr

   a = ArrayHolder(numpy.array(...))
   a.arr = f(a.arr)

But of course it is not as convenient to write a[0] or a.arr instead of just
a.

-=- Olivier

2011/9/13 Robert Kern <robert.kern at gmail.com>

> On Tue, Sep 13, 2011 at 01:53, David Froger <david.froger at gmail.com>
> wrote:
> >
> > Thank you Olivier and Robert for your replies!
> >
> > Some remarks about the dictionnary solution:
> >
> > from numpy import *
> >
> > def f(arr):
> >     return arr + 100.
> >
> > arrs = {}
> > arrs['a'] = array( [1,1,1] )
> > arrs['b'] = array( [2,2,2] )
> > arrs['c'] = array( [3,3,3] )
> > arrs['d'] = array( [4,4,4] )
> >
> > for key,value in arrs.iteritems():
> >    arrs[key] = f(value)
> >
> > 1. about the memory
> > Memory is first allocated with the array functions:
> >    arrs['a'] = array( [1,1,1] )
> >    arrs['b'] = array( [2,2,2] )
> >    arrs['c'] = array( [3,3,3] )
> >    arrs['d'] = array( [4,4,4] )
> >
> > Are there others memory allocations with this assignemnt:
> >    arrs[key] = f(value)
> > or is the already allocated memory used to store the result of f(value)?
> >
> > In other words, if I have N arrays of the same shape, each of them
> costing
> > nbytes of memory, does it use N*nbytes memory, or 2*N*bytes?
>
> Temporarily, yes, for all of the variations mentioned. When the
> expression "f(value)" is evaluated, both the result array and the
> input array will exist simultaneously in memory. Once the assignment
> happens, the original input array will be destroyed and free up the
> memory. There is no difference memory-wise between assigning into a
> dictionary or assigning to a variable name.
>
> Sometimes, you can write your f() such that you just need to do
>
>  f(value)
>
> and have the value object modified in-place. In that case, there is no
> need to reassign the result to a variable or dictionary key.
>
> > I think this is well documented on the web and I can find it....
> >
> > 2. about individual array
> > The problem is that now, if one want to use a individual array, one have
> now to
> > use:
> >    arrs['a']
> > instead of just:
> >    a
> > So I'm sometime tempted to use locals() instead of arrs...
>
> Seriously, don't. It makes your code worse, not better. It's also
> unreliable. The locals() dictionary is meant to be read-only (and even
> then for debugger tooling and the like, not regular code), and this is
> sometimes enforced. If you want to use variable names instead of
> dictionaries, use them, but write out each assignment statement.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>   -- Umberto Eco
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110913/aac3ef6c/attachment.html>