I *think* it may work better if you replace the last 3 lines in your loop by:<br><br> a=all_TSFC[0]<br> if len(all_TSFC) > 1:<br> N.maximum(a, TSFC, out=a)<br><br>Not 100% sure that would work though, as I'm not entirely confident I understand your code.<br>
<br>-=- Olivier<br><br><div class="gmail_quote">2011/12/6 questions anon <span dir="ltr"><<a href="mailto:questions.anon@gmail.com">questions.anon@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Something fancier I think, <br>I am able to compare the result with my previous method so I can easily see I am doing something wrong.<br>see code below:<br><br><br>all_TSFC=[]<br>for (path, dirs, files) in os.walk(MainFolder):<br>
for dir in dirs:<br> print dir<br> path=path+'/'<br> for ncfile in files:<br> if ncfile[-3:]=='.nc':<br> print "dealing with ncfiles:", ncfile<br> ncfile=os.path.join(path,ncfile)<br>
ncfile=Dataset(ncfile, 'r+', 'NETCDF4')<br> TSFC=ncfile.variables['T_SFC'][:]<br> fillvalue=ncfile.variables['T_SFC']._FillValue<br> TSFC=MA.masked_values(TSFC, fillvalue)<br>
ncfile.close()<br> all_TSFC.append(TSFC)<br> a=TSFC[0]<br> for b in TSFC[1:]:<br> N.maximum(a,b,out=a)<br><br>big_array=N.ma.concatenate(all_TSFC)<br>Max=big_array.max(axis=0)<br>
print "max is", Max,"a is", a<div class="HOEnZb"><div class="h5"><br><br><div class="gmail_quote">On Wed, Dec 7, 2011 at 2:34 PM, Olivier Delalleau <span dir="ltr"><<a href="mailto:shish@keba.be" target="_blank">shish@keba.be</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Is 'a' a regular numpy array or something fancier?<div><div>
<br><br>-=- Olivier<br><br><div class="gmail_quote">2011/12/6 questions anon <span dir="ltr"><<a href="mailto:questions.anon@gmail.com" target="_blank">questions.anon@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">thanks again my only problem though is that the out=a in the loop does not seem to replace my a= outside the loop so my final a is whatever I started with for a. <br>
Not sure what I am doing wrong whether it is something with the loop or with the command. <br><div><div>
<br><div class="gmail_quote">On Wed, Dec 7, 2011 at 1:44 PM, <span dir="ltr"><<a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>On Tue, Dec 6, 2011 at 9:36 PM, Olivier Delalleau <<a href="mailto:shish@keba.be" target="_blank">shish@keba.be</a>> wrote:<br>
> The "out=a" keyword will ensure your first array will keep being updated. So<br>
> you can do something like:<br>
><br>
> a = my_list_of_arrays[0]<br>
> for b in my_list_of_arrays[1:]:<br>
> numpy.maximum(a, b, out=a)<br>
<br>
</div>I didn't think of the out argument which makes it more efficient, but<br>
in my example I used Python's reduce which takes an iterable and not<br>
one huge array.<br>
<br>
Josef<br>
<div><div><br>
<br>
><br>
> -=- Olivier<br>
><br>
> 2011/12/6 questions anon <<a href="mailto:questions.anon@gmail.com" target="_blank">questions.anon@gmail.com</a>><br>
>><br>
>> thanks for all of your help, that does look appropriate but I am not sure<br>
>> how to loop it over thousands of files.<br>
>> I need to keep the first array to compare with but replace any greater<br>
>> values as I loop through each array comparing back to the same array. does<br>
>> that make sense?<br>
>><br>
>><br>
>> On Wed, Dec 7, 2011 at 1:12 PM, Olivier Delalleau <<a href="mailto:shish@keba.be" target="_blank">shish@keba.be</a>> wrote:<br>
>>><br>
>>> Thanks, I didn't know you could specify the out array :)<br>
>>><br>
>>> (to the OP: my initial suggestion, although probably not very efficient,<br>
>>> seems to work with 2D arrays too, so I have no idea why it didn't work for<br>
>>> you -- but Nathaniel's one seems to be the ideal one anyway).<br>
>>><br>
>>> -=- Olivier<br>
>>><br>
>>><br>
>>> 2011/12/6 Nathaniel Smith <<a href="mailto:njs@pobox.com" target="_blank">njs@pobox.com</a>><br>
>>>><br>
>>>> I think you want<br>
>>>> np.maximum(a, b, out=a)<br>
>>>><br>
>>>> - Nathaniel<br>
>>>><br>
>>>> On Dec 6, 2011 9:04 PM, "questions anon" <<a href="mailto:questions.anon@gmail.com" target="_blank">questions.anon@gmail.com</a>><br>
>>>> wrote:<br>
>>>>><br>
>>>>> thanks for responding Josef but that is not really what I am looking<br>
>>>>> for, I have a multidimensional array and if the next array has any values<br>
>>>>> greater than what is in my first array I want to replace them. The data are<br>
>>>>> contained in netcdf files.<br>
>>>>> I can achieve what I want if I combine all of my arrays using numpy<br>
>>>>> concatenate and then using the command numpy.max(myarray, axis=0) but<br>
>>>>> because I have so many arrays I end up with a memory error so I need to find<br>
>>>>> a way to get the maximum while looping.<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> On Wed, Dec 7, 2011 at 12:36 PM, <<a href="mailto:josef.pktd@gmail.com" target="_blank">josef.pktd@gmail.com</a>> wrote:<br>
>>>>>><br>
>>>>>> On Tue, Dec 6, 2011 at 7:55 PM, Olivier Delalleau <<a href="mailto:shish@keba.be" target="_blank">shish@keba.be</a>><br>
>>>>>> wrote:<br>
>>>>>> > It may not be the most efficient way to do this, but you can do:<br>
>>>>>> > mask = b > a<br>
>>>>>> > a[mask] = b[mask]<br>
>>>>>> ><br>
>>>>>> > -=- Olivier<br>
>>>>>> ><br>
>>>>>> > 2011/12/6 questions anon <<a href="mailto:questions.anon@gmail.com" target="_blank">questions.anon@gmail.com</a>><br>
>>>>>> >><br>
>>>>>> >> I would like to produce an array with the maximum values out of<br>
>>>>>> >> many<br>
>>>>>> >> (10000s) of arrays.<br>
>>>>>> >> I need to loop through many multidimentional arrays and if a value<br>
>>>>>> >> is<br>
>>>>>> >> larger (in the same place as the previous array) then I would like<br>
>>>>>> >> that<br>
>>>>>> >> value to replace it.<br>
>>>>>> >><br>
>>>>>> >> e.g.<br>
>>>>>> >> a=[1,1,2,2<br>
>>>>>> >> 11,2,2<br>
>>>>>> >> 1,1,2,2]<br>
>>>>>> >> b=[1,1,3,2<br>
>>>>>> >> 2,1,0,0<br>
>>>>>> >> 1,1,2,0]<br>
>>>>>> >><br>
>>>>>> >> where b>a replace with value in b, so the new a should be :<br>
>>>>>> >><br>
>>>>>> >> a=[1,1,3,2]<br>
>>>>>> >> 2,1,2,2<br>
>>>>>> >> 1,1,2,2]<br>
>>>>>> >><br>
>>>>>> >> and then keep looping through many arrays and replace whenever<br>
>>>>>> >> value is<br>
>>>>>> >> larger.<br>
>>>>>> >><br>
>>>>>> >> I have tried numpy.putmask but that results in<br>
>>>>>> >> TypeError: putmask() argument 1 must be numpy.ndarray, not list<br>
>>>>>> >> Any other ideas? Thanks<br>
>>>>>><br>
>>>>>> if I understand correctly it's a minimum.reduce<br>
>>>>>><br>
>>>>>> numpy<br>
>>>>>><br>
>>>>>> >>> a = np.concatenate((np.arange(5)[::-1],<br>
>>>>>> >>> np.arange(5)))*np.ones((4,3,1))<br>
>>>>>> >>> np.minimum.reduce(a, axis=2)<br>
>>>>>> array([[ 0., 0., 0.],<br>
>>>>>> [ 0., 0., 0.],<br>
>>>>>> [ 0., 0., 0.],<br>
>>>>>> [ 0., 0., 0.]])<br>
>>>>>> >>> a.T.shape<br>
>>>>>> (10, 3, 4)<br>
>>>>>><br>
>>>>>> python with iterable<br>
>>>>>><br>
>>>>>> >>> reduce(np.maximum, a.T)<br>
>>>>>> array([[ 4., 4., 4., 4.],<br>
>>>>>> [ 4., 4., 4., 4.],<br>
>>>>>> [ 4., 4., 4., 4.]])<br>
>>>>>> >>> reduce(np.minimum, a.T)<br>
>>>>>> array([[ 0., 0., 0., 0.],<br>
>>>>>> [ 0., 0., 0., 0.],<br>
>>>>>> [ 0., 0., 0., 0.]])<br>
>>>>>><br>
>>>>>> Josef<br>
>>>>>><br>
>>>>>> >><br>
>>>>>> >> _______________________________________________<br>
>>>>>> >> NumPy-Discussion mailing list<br>
>>>>>> >> <a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a><br>
>>>>>> >> <a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
>>>>>> >><br>
>>>>>> ><br>
>>>>>> ><br>
>>>>>> > _______________________________________________<br>
>>>>>> > NumPy-Discussion mailing list<br>
>>>>>> > <a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a><br>
>>>>>> > <a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
>>>>>> ><br>
>>>>>> _______________________________________________<br>
>>>>>> NumPy-Discussion mailing list<br>
>>>>>> <a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a><br>
>>>>>> <a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> _______________________________________________<br>
>>>>> NumPy-Discussion mailing list<br>
>>>>> <a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a><br>
>>>>> <a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
>>>>><br>
>>>><br>
>>>> _______________________________________________<br>
>>>> NumPy-Discussion mailing list<br>
>>>> <a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a><br>
>>>> <a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
>>>><br>
>>><br>
>>><br>
>>> _______________________________________________<br>
>>> NumPy-Discussion mailing list<br>
>>> <a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a><br>
>>> <a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
>>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> NumPy-Discussion mailing list<br>
>> <a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a><br>
>> <a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
>><br>
><br>
><br>
> _______________________________________________<br>
> NumPy-Discussion mailing list<br>
> <a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a><br>
> <a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
><br>
_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
</div></div></blockquote></div><br>
</div></div><br>_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
<br></blockquote></div><br>
</div></div><br>_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org" target="_blank">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
<br></blockquote></div><br>
</div></div><br>_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
<br></blockquote></div><br>