[Numpy-discussion] loop through values in a array and find maximum as looping

Olivier Delalleau shish at keba.be
Tue Dec 6 23:07:53 EST 2011


I *think* it may work better if you replace the last 3 lines in your loop
by:

            a=all_TSFC[0]
            if len(all_TSFC) > 1:
                N.maximum(a, TSFC, out=a)

Not 100% sure that would work though, as I'm not entirely confident I
understand your code.

-=- Olivier

2011/12/6 questions anon <questions.anon at gmail.com>

> Something fancier I think,
> I am able to compare the result with my previous method so I can easily
> see I am doing something wrong.
> see code below:
>
>
> all_TSFC=[]
> for (path, dirs, files) in os.walk(MainFolder):
>     for dir in dirs:
>         print dir
>     path=path+'/'
>     for ncfile in files:
>         if ncfile[-3:]=='.nc':
>             print "dealing with ncfiles:", ncfile
>             ncfile=os.path.join(path,ncfile)
>             ncfile=Dataset(ncfile, 'r+', 'NETCDF4')
>             TSFC=ncfile.variables['T_SFC'][:]
>             fillvalue=ncfile.variables['T_SFC']._FillValue
>             TSFC=MA.masked_values(TSFC, fillvalue)
>             ncfile.close()
>             all_TSFC.append(TSFC)
>             a=TSFC[0]
>             for b in TSFC[1:]:
>                 N.maximum(a,b,out=a)
>
> big_array=N.ma.concatenate(all_TSFC)
> Max=big_array.max(axis=0)
> print "max is", Max,"a is", a
>
>
> On Wed, Dec 7, 2011 at 2:34 PM, Olivier Delalleau <shish at keba.be> wrote:
>
>> Is 'a' a regular numpy array or something fancier?
>>
>>
>> -=- Olivier
>>
>> 2011/12/6 questions anon <questions.anon at gmail.com>
>>
>>> thanks again my only problem though is that the out=a in the loop does
>>> not seem to replace my a= outside the loop so my final a is whatever I
>>> started with for a.
>>> Not sure what I am doing wrong whether it is something with the loop or
>>> with the command.
>>>
>>> On Wed, Dec 7, 2011 at 1:44 PM, <josef.pktd at gmail.com> wrote:
>>>
>>>> On Tue, Dec 6, 2011 at 9:36 PM, Olivier Delalleau <shish at keba.be>
>>>> wrote:
>>>> > The "out=a" keyword will ensure your first array will keep being
>>>> updated. So
>>>> > you can do something like:
>>>> >
>>>> > a = my_list_of_arrays[0]
>>>> > for b in my_list_of_arrays[1:]:
>>>> >   numpy.maximum(a, b, out=a)
>>>>
>>>> I didn't think of the out argument which makes it more efficient, but
>>>> in my example I used Python's reduce which takes an iterable and not
>>>> one huge array.
>>>>
>>>> Josef
>>>>
>>>>
>>>> >
>>>> > -=- Olivier
>>>> >
>>>> > 2011/12/6 questions anon <questions.anon at gmail.com>
>>>> >>
>>>> >> thanks for all of your help, that does look appropriate but I am not
>>>> sure
>>>> >> how to loop it over thousands of files.
>>>> >> I need to keep the first array to compare with but replace any
>>>> greater
>>>> >> values as I loop through each array comparing back to the same
>>>> array. does
>>>> >> that make sense?
>>>> >>
>>>> >>
>>>> >> On Wed, Dec 7, 2011 at 1:12 PM, Olivier Delalleau <shish at keba.be>
>>>> wrote:
>>>> >>>
>>>> >>> Thanks, I didn't know you could specify the out array :)
>>>> >>>
>>>> >>> (to the OP: my initial suggestion, although probably not very
>>>> efficient,
>>>> >>> seems to work with 2D arrays too, so I have no idea why it didn't
>>>> work for
>>>> >>> you -- but Nathaniel's one seems to be the ideal one anyway).
>>>> >>>
>>>> >>> -=- Olivier
>>>> >>>
>>>> >>>
>>>> >>> 2011/12/6 Nathaniel Smith <njs at pobox.com>
>>>> >>>>
>>>> >>>> I think you want
>>>> >>>>   np.maximum(a, b, out=a)
>>>> >>>>
>>>> >>>> - Nathaniel
>>>> >>>>
>>>> >>>> On Dec 6, 2011 9:04 PM, "questions anon" <questions.anon at gmail.com
>>>> >
>>>> >>>> wrote:
>>>> >>>>>
>>>> >>>>> thanks for responding Josef but that is not really what I am
>>>> looking
>>>> >>>>> for, I have a multidimensional array and if the next array has
>>>> any values
>>>> >>>>> greater than what is in my first array I want to replace them.
>>>> The data are
>>>> >>>>> contained in netcdf files.
>>>> >>>>> I can achieve what I want if I combine all of my arrays using
>>>> numpy
>>>> >>>>> concatenate and then using the command numpy.max(myarray, axis=0)
>>>> but
>>>> >>>>> because I have so many arrays I end up with a memory error so I
>>>> need to find
>>>> >>>>> a way to get the maximum while looping.
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> On Wed, Dec 7, 2011 at 12:36 PM, <josef.pktd at gmail.com> wrote:
>>>> >>>>>>
>>>> >>>>>> On Tue, Dec 6, 2011 at 7:55 PM, Olivier Delalleau <shish at keba.be
>>>> >
>>>> >>>>>> wrote:
>>>> >>>>>> > It may not be the most efficient way to do this, but you can
>>>> do:
>>>> >>>>>> > mask = b > a
>>>> >>>>>> > a[mask] = b[mask]
>>>> >>>>>> >
>>>> >>>>>> > -=- Olivier
>>>> >>>>>> >
>>>> >>>>>> > 2011/12/6 questions anon <questions.anon at gmail.com>
>>>> >>>>>> >>
>>>> >>>>>> >> I would like to produce an array with the maximum values out
>>>> of
>>>> >>>>>> >> many
>>>> >>>>>> >> (10000s) of arrays.
>>>> >>>>>> >> I need to loop through many multidimentional arrays and if a
>>>> value
>>>> >>>>>> >> is
>>>> >>>>>> >> larger (in the same place as the previous array) then I would
>>>> like
>>>> >>>>>> >> that
>>>> >>>>>> >> value to replace it.
>>>> >>>>>> >>
>>>> >>>>>> >> e.g.
>>>> >>>>>> >> a=[1,1,2,2
>>>> >>>>>> >> 11,2,2
>>>> >>>>>> >> 1,1,2,2]
>>>> >>>>>> >> b=[1,1,3,2
>>>> >>>>>> >> 2,1,0,0
>>>> >>>>>> >> 1,1,2,0]
>>>> >>>>>> >>
>>>> >>>>>> >> where b>a replace with value in b, so the new a should be :
>>>> >>>>>> >>
>>>> >>>>>> >> a=[1,1,3,2]
>>>> >>>>>> >> 2,1,2,2
>>>> >>>>>> >> 1,1,2,2]
>>>> >>>>>> >>
>>>> >>>>>> >> and then keep looping through many arrays and replace whenever
>>>> >>>>>> >> value is
>>>> >>>>>> >> larger.
>>>> >>>>>> >>
>>>> >>>>>> >> I have tried numpy.putmask but that results in
>>>> >>>>>> >> TypeError: putmask() argument 1 must be numpy.ndarray, not
>>>> list
>>>> >>>>>> >> Any other ideas? Thanks
>>>> >>>>>>
>>>> >>>>>> if I understand correctly it's a minimum.reduce
>>>> >>>>>>
>>>> >>>>>> numpy
>>>> >>>>>>
>>>> >>>>>> >>> a = np.concatenate((np.arange(5)[::-1],
>>>> >>>>>> >>> np.arange(5)))*np.ones((4,3,1))
>>>> >>>>>> >>> np.minimum.reduce(a, axis=2)
>>>> >>>>>> array([[ 0.,  0.,  0.],
>>>> >>>>>>       [ 0.,  0.,  0.],
>>>> >>>>>>       [ 0.,  0.,  0.],
>>>> >>>>>>       [ 0.,  0.,  0.]])
>>>> >>>>>> >>> a.T.shape
>>>> >>>>>> (10, 3, 4)
>>>> >>>>>>
>>>> >>>>>> python with iterable
>>>> >>>>>>
>>>> >>>>>> >>> reduce(np.maximum, a.T)
>>>> >>>>>> array([[ 4.,  4.,  4.,  4.],
>>>> >>>>>>       [ 4.,  4.,  4.,  4.],
>>>> >>>>>>       [ 4.,  4.,  4.,  4.]])
>>>> >>>>>> >>> reduce(np.minimum, a.T)
>>>> >>>>>> array([[ 0.,  0.,  0.,  0.],
>>>> >>>>>>       [ 0.,  0.,  0.,  0.],
>>>> >>>>>>       [ 0.,  0.,  0.,  0.]])
>>>> >>>>>>
>>>> >>>>>> Josef
>>>> >>>>>>
>>>> >>>>>> >>
>>>> >>>>>> >> _______________________________________________
>>>> >>>>>> >> NumPy-Discussion mailing list
>>>> >>>>>> >> NumPy-Discussion at scipy.org
>>>> >>>>>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>> >>>>>> >>
>>>> >>>>>> >
>>>> >>>>>> >
>>>> >>>>>> > _______________________________________________
>>>> >>>>>> > NumPy-Discussion mailing list
>>>> >>>>>> > NumPy-Discussion at scipy.org
>>>> >>>>>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>> >>>>>> >
>>>> >>>>>> _______________________________________________
>>>> >>>>>> NumPy-Discussion mailing list
>>>> >>>>>> NumPy-Discussion at scipy.org
>>>> >>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> _______________________________________________
>>>> >>>>> NumPy-Discussion mailing list
>>>> >>>>> NumPy-Discussion at scipy.org
>>>> >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>> >>>>>
>>>> >>>>
>>>> >>>> _______________________________________________
>>>> >>>> NumPy-Discussion mailing list
>>>> >>>> NumPy-Discussion at scipy.org
>>>> >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>> >>>>
>>>> >>>
>>>> >>>
>>>> >>> _______________________________________________
>>>> >>> NumPy-Discussion mailing list
>>>> >>> NumPy-Discussion at scipy.org
>>>> >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>> >>>
>>>> >>
>>>> >>
>>>> >> _______________________________________________
>>>> >> NumPy-Discussion mailing list
>>>> >> NumPy-Discussion at scipy.org
>>>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>> >>
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > NumPy-Discussion mailing list
>>>> > NumPy-Discussion at scipy.org
>>>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>> >
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>
>>>
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20111206/a8e3a525/attachment.html>


More information about the NumPy-Discussion mailing list