sorry the 'all_TSFC' is for my other check of maximum using concatenate and N.max, I know that works so I am comparing it to this method. The only reason I need another method is for memory error issues. I like the code I have written so far as it makes sense to me. I can't get the extra examples I have been given to work and that is most likely because I don't understand them, these are the errors I get : Traceback (most recent call last): File "d:\plot_summarystats\test_plot_remove_memoryerror_max.py", line 46, in <module> N.maximum(a,TSFC,out=a) ValueError: non-broadcastable output operand with shape (106,193) doesn't match the broadcast shape (721,106,193) and Traceback (most recent call last): File "d:\plot_summarystats\test_plot_remove_memoryerror_max.py", line 45, in <module> if not instance(a, N.ndarray): NameError: name 'instance' is not defined On Wed, Dec 7, 2011 at 3:07 PM, Olivier Delalleau <shish@keba.be> wrote:
I *think* it may work better if you replace the last 3 lines in your loop by:
a=all_TSFC[0] if len(all_TSFC) > 1: N.maximum(a, TSFC, out=a)
Not 100% sure that would work though, as I'm not entirely confident I understand your code.
-=- Olivier
2011/12/6 questions anon <questions.anon@gmail.com>
Something fancier I think, I am able to compare the result with my previous method so I can easily see I am doing something wrong. see code below:
all_TSFC=[] for (path, dirs, files) in os.walk(MainFolder): for dir in dirs: print dir path=path+'/' for ncfile in files: if ncfile[-3:]=='.nc': print "dealing with ncfiles:", ncfile ncfile=os.path.join(path,ncfile) ncfile=Dataset(ncfile, 'r+', 'NETCDF4') TSFC=ncfile.variables['T_SFC'][:] fillvalue=ncfile.variables['T_SFC']._FillValue TSFC=MA.masked_values(TSFC, fillvalue) ncfile.close() all_TSFC.append(TSFC) a=TSFC[0] for b in TSFC[1:]: N.maximum(a,b,out=a)
big_array=N.ma.concatenate(all_TSFC) Max=big_array.max(axis=0) print "max is", Max,"a is", a
On Wed, Dec 7, 2011 at 2:34 PM, Olivier Delalleau <shish@keba.be> wrote:
Is 'a' a regular numpy array or something fancier?
-=- Olivier
2011/12/6 questions anon <questions.anon@gmail.com>
thanks again my only problem though is that the out=a in the loop does not seem to replace my a= outside the loop so my final a is whatever I started with for a. Not sure what I am doing wrong whether it is something with the loop or with the command.
On Wed, Dec 7, 2011 at 1:44 PM, <josef.pktd@gmail.com> wrote:
On Tue, Dec 6, 2011 at 9:36 PM, Olivier Delalleau <shish@keba.be> wrote:
The "out=a" keyword will ensure your first array will keep being updated. So you can do something like:
a = my_list_of_arrays[0] for b in my_list_of_arrays[1:]: numpy.maximum(a, b, out=a)
I didn't think of the out argument which makes it more efficient, but in my example I used Python's reduce which takes an iterable and not one huge array.
Josef
-=- Olivier
2011/12/6 questions anon <questions.anon@gmail.com> > > thanks for all of your help, that does look appropriate but I am
> how to loop it over thousands of files. > I need to keep the first array to compare with but replace any greater > values as I loop through each array comparing back to the same array. does > that make sense? > > > On Wed, Dec 7, 2011 at 1:12 PM, Olivier Delalleau <shish@keba.be> wrote: >> >> Thanks, I didn't know you could specify the out array :) >> >> (to the OP: my initial suggestion, although probably not very efficient, >> seems to work with 2D arrays too, so I have no idea why it didn't work for >> you -- but Nathaniel's one seems to be the ideal one anyway). >> >> -=- Olivier >> >> >> 2011/12/6 Nathaniel Smith <njs@pobox.com> >>> >>> I think you want >>> np.maximum(a, b, out=a) >>> >>> - Nathaniel >>> >>> On Dec 6, 2011 9:04 PM, "questions anon" < questions.anon@gmail.com> >>> wrote: >>>> >>>> thanks for responding Josef but that is not really what I am looking >>>> for, I have a multidimensional array and if the next array has any values >>>> greater than what is in my first array I want to replace them. The data are >>>> contained in netcdf files. >>>> I can achieve what I want if I combine all of my arrays using numpy >>>> concatenate and then using the command numpy.max(myarray, axis=0) but >>>> because I have so many arrays I end up with a memory error so I need to find >>>> a way to get the maximum while looping. >>>> >>>> >>>> >>>> On Wed, Dec 7, 2011 at 12:36 PM, <josef.pktd@gmail.com> wrote: >>>>> >>>>> On Tue, Dec 6, 2011 at 7:55 PM, Olivier Delalleau < shish@keba.be> >>>>> wrote: >>>>> > It may not be the most efficient way to do this, but you can do: >>>>> > mask = b > a >>>>> > a[mask] = b[mask] >>>>> > >>>>> > -=- Olivier >>>>> > >>>>> > 2011/12/6 questions anon <questions.anon@gmail.com> >>>>> >> >>>>> >> I would like to produce an array with the maximum values out of >>>>> >> many >>>>> >> (10000s) of arrays. >>>>> >> I need to loop through many multidimentional arrays and if a value >>>>> >> is >>>>> >> larger (in the same place as the previous array) then I would like >>>>> >> that >>>>> >> value to replace it. >>>>> >> >>>>> >> e.g. >>>>> >> a=[1,1,2,2 >>>>> >> 11,2,2 >>>>> >> 1,1,2,2] >>>>> >> b=[1,1,3,2 >>>>> >> 2,1,0,0 >>>>> >> 1,1,2,0] >>>>> >> >>>>> >> where b>a replace with value in b, so the new a should be : >>>>> >> >>>>> >> a=[1,1,3,2] >>>>> >> 2,1,2,2 >>>>> >> 1,1,2,2] >>>>> >> >>>>> >> and then keep looping through many arrays and replace whenever >>>>> >> value is >>>>> >> larger. >>>>> >> >>>>> >> I have tried numpy.putmask but that results in >>>>> >> TypeError: putmask() argument 1 must be numpy.ndarray, not
not sure list
>>>>> >> Any other ideas? Thanks >>>>> >>>>> if I understand correctly it's a minimum.reduce >>>>> >>>>> numpy >>>>> >>>>> >>> a = np.concatenate((np.arange(5)[::-1], >>>>> >>> np.arange(5)))*np.ones((4,3,1)) >>>>> >>> np.minimum.reduce(a, axis=2) >>>>> array([[ 0., 0., 0.], >>>>> [ 0., 0., 0.], >>>>> [ 0., 0., 0.], >>>>> [ 0., 0., 0.]]) >>>>> >>> a.T.shape >>>>> (10, 3, 4) >>>>> >>>>> python with iterable >>>>> >>>>> >>> reduce(np.maximum, a.T) >>>>> array([[ 4., 4., 4., 4.], >>>>> [ 4., 4., 4., 4.], >>>>> [ 4., 4., 4., 4.]]) >>>>> >>> reduce(np.minimum, a.T) >>>>> array([[ 0., 0., 0., 0.], >>>>> [ 0., 0., 0., 0.], >>>>> [ 0., 0., 0., 0.]]) >>>>> >>>>> Josef >>>>> >>>>> >> >>>>> >> _______________________________________________ >>>>> >> NumPy-Discussion mailing list >>>>> >> NumPy-Discussion@scipy.org >>>>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >> >>>>> > >>>>> > >>>>> > _______________________________________________ >>>>> > NumPy-Discussion mailing list >>>>> > NumPy-Discussion@scipy.org >>>>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> > >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion@scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>>> >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion@scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion@scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion@scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion >
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion