I *think* it may work better if you replace the last 3 lines in your loop by:

            a=all_TSFC[0]
            if len(all_TSFC) > 1:
                N.maximum(a, TSFC, out=a)

Not 100% sure that would work though, as I'm not entirely confident I understand your code.

-=- Olivier

2011/12/6 questions anon <questions.anon@gmail.com>
Something fancier I think,
I am able to compare the result with my previous method so I can easily see I am doing something wrong.
see code below:


all_TSFC=[]
for (path, dirs, files) in os.walk(MainFolder):
    for dir in dirs:
        print dir
    path=path+'/'
    for ncfile in files:
        if ncfile[-3:]=='.nc':
            print "dealing with ncfiles:", ncfile
            ncfile=os.path.join(path,ncfile)
            ncfile=Dataset(ncfile, 'r+', 'NETCDF4')
            TSFC=ncfile.variables['T_SFC'][:]
            fillvalue=ncfile.variables['T_SFC']._FillValue
            TSFC=MA.masked_values(TSFC, fillvalue)
            ncfile.close()
            all_TSFC.append(TSFC)
            a=TSFC[0]
            for b in TSFC[1:]:
                N.maximum(a,b,out=a)

big_array=N.ma.concatenate(all_TSFC)
Max=big_array.max(axis=0)
print "max is", Max,"a is", a


On Wed, Dec 7, 2011 at 2:34 PM, Olivier Delalleau <shish@keba.be> wrote:
Is 'a' a regular numpy array or something fancier?


-=- Olivier

2011/12/6 questions anon <questions.anon@gmail.com>
thanks again my only problem though is that the out=a in the loop does not seem to replace my a= outside the loop so my final a is whatever I started with for a.
Not sure what I am doing wrong whether it is something with the loop or with the command.

On Wed, Dec 7, 2011 at 1:44 PM, <josef.pktd@gmail.com> wrote:
On Tue, Dec 6, 2011 at 9:36 PM, Olivier Delalleau <shish@keba.be> wrote:
> The "out=a" keyword will ensure your first array will keep being updated. So
> you can do something like:
>
> a = my_list_of_arrays[0]
> for b in my_list_of_arrays[1:]:
>   numpy.maximum(a, b, out=a)

I didn't think of the out argument which makes it more efficient, but
in my example I used Python's reduce which takes an iterable and not
one huge array.

Josef


>
> -=- Olivier
>
> 2011/12/6 questions anon <questions.anon@gmail.com>
>>
>> thanks for all of your help, that does look appropriate but I am not sure
>> how to loop it over thousands of files.
>> I need to keep the first array to compare with but replace any greater
>> values as I loop through each array comparing back to the same array. does
>> that make sense?
>>
>>
>> On Wed, Dec 7, 2011 at 1:12 PM, Olivier Delalleau <shish@keba.be> wrote:
>>>
>>> Thanks, I didn't know you could specify the out array :)
>>>
>>> (to the OP: my initial suggestion, although probably not very efficient,
>>> seems to work with 2D arrays too, so I have no idea why it didn't work for
>>> you -- but Nathaniel's one seems to be the ideal one anyway).
>>>
>>> -=- Olivier
>>>
>>>
>>> 2011/12/6 Nathaniel Smith <njs@pobox.com>
>>>>
>>>> I think you want
>>>>   np.maximum(a, b, out=a)
>>>>
>>>> - Nathaniel
>>>>
>>>> On Dec 6, 2011 9:04 PM, "questions anon" <questions.anon@gmail.com>
>>>> wrote:
>>>>>
>>>>> thanks for responding Josef but that is not really what I am looking
>>>>> for, I have a multidimensional array and if the next array has any values
>>>>> greater than what is in my first array I want to replace them. The data are
>>>>> contained in netcdf files.
>>>>> I can achieve what I want if I combine all of my arrays using numpy
>>>>> concatenate and then using the command numpy.max(myarray, axis=0) but
>>>>> because I have so many arrays I end up with a memory error so I need to find
>>>>> a way to get the maximum while looping.
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Dec 7, 2011 at 12:36 PM, <josef.pktd@gmail.com> wrote:
>>>>>>
>>>>>> On Tue, Dec 6, 2011 at 7:55 PM, Olivier Delalleau <shish@keba.be>
>>>>>> wrote:
>>>>>> > It may not be the most efficient way to do this, but you can do:
>>>>>> > mask = b > a
>>>>>> > a[mask] = b[mask]
>>>>>> >
>>>>>> > -=- Olivier
>>>>>> >
>>>>>> > 2011/12/6 questions anon <questions.anon@gmail.com>
>>>>>> >>
>>>>>> >> I would like to produce an array with the maximum values out of
>>>>>> >> many
>>>>>> >> (10000s) of arrays.
>>>>>> >> I need to loop through many multidimentional arrays and if a value
>>>>>> >> is
>>>>>> >> larger (in the same place as the previous array) then I would like
>>>>>> >> that
>>>>>> >> value to replace it.
>>>>>> >>
>>>>>> >> e.g.
>>>>>> >> a=[1,1,2,2
>>>>>> >> 11,2,2
>>>>>> >> 1,1,2,2]
>>>>>> >> b=[1,1,3,2
>>>>>> >> 2,1,0,0
>>>>>> >> 1,1,2,0]
>>>>>> >>
>>>>>> >> where b>a replace with value in b, so the new a should be :
>>>>>> >>
>>>>>> >> a=[1,1,3,2]
>>>>>> >> 2,1,2,2
>>>>>> >> 1,1,2,2]
>>>>>> >>
>>>>>> >> and then keep looping through many arrays and replace whenever
>>>>>> >> value is
>>>>>> >> larger.
>>>>>> >>
>>>>>> >> I have tried numpy.putmask but that results in
>>>>>> >> TypeError: putmask() argument 1 must be numpy.ndarray, not list
>>>>>> >> Any other ideas? Thanks
>>>>>>
>>>>>> if I understand correctly it's a minimum.reduce
>>>>>>
>>>>>> numpy
>>>>>>
>>>>>> >>> a = np.concatenate((np.arange(5)[::-1],
>>>>>> >>> np.arange(5)))*np.ones((4,3,1))
>>>>>> >>> np.minimum.reduce(a, axis=2)
>>>>>> array([[ 0.,  0.,  0.],
>>>>>>       [ 0.,  0.,  0.],
>>>>>>       [ 0.,  0.,  0.],
>>>>>>       [ 0.,  0.,  0.]])
>>>>>> >>> a.T.shape
>>>>>> (10, 3, 4)
>>>>>>
>>>>>> python with iterable
>>>>>>
>>>>>> >>> reduce(np.maximum, a.T)
>>>>>> array([[ 4.,  4.,  4.,  4.],
>>>>>>       [ 4.,  4.,  4.,  4.],
>>>>>>       [ 4.,  4.,  4.,  4.]])
>>>>>> >>> reduce(np.minimum, a.T)
>>>>>> array([[ 0.,  0.,  0.,  0.],
>>>>>>       [ 0.,  0.,  0.,  0.],
>>>>>>       [ 0.,  0.,  0.,  0.]])
>>>>>>
>>>>>> Josef
>>>>>>
>>>>>> >>
>>>>>> >> _______________________________________________
>>>>>> >> NumPy-Discussion mailing list
>>>>>> >> NumPy-Discussion@scipy.org
>>>>>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>>> >>
>>>>>> >
>>>>>> >
>>>>>> > _______________________________________________
>>>>>> > NumPy-Discussion mailing list
>>>>>> > NumPy-Discussion@scipy.org
>>>>>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>>> >
>>>>>> _______________________________________________
>>>>>> NumPy-Discussion mailing list
>>>>>> NumPy-Discussion@scipy.org
>>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> NumPy-Discussion mailing list
>>>>> NumPy-Discussion@scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>>
>>>>
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion@scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>
>>>
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion@scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion



_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion



_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion