Mailman 3 July 2008 - NumPy-Discussion

1.1.1rc2 tagged
by Jarrod Millman 25 Jul '08

25 Jul '08

Hello, The 1.1.1rc2 is now available: http://svn.scipy.org/svn/numpy/tags/1.1.1rc2 The source tarball is here: http://cirl.berkeley.edu/numpy/numpy-1.1.1rc2.tar.gz Here is the universal Mac binary: http://cirl.berkeley.edu/numpy/numpy-1.1.1rc2-py2.5-macosx10.5.dmg David Cournapeau will be creating a 1.1.1rc2 Windows binary in next few days. Please test this release ASAP and let us know if there are any problems. If there are no show stoppers, this will likely become the 1.1.1 release. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/

3 2

Change of SVD output
by Arnar Flatberg 25 Jul '08

25 Jul '08

Hi In a recent thread there was an error in how a matrix is reconstructed from its SVD decomposition. I apologize if this is just an old and settled issue and I am just adding noise, but I got bitten by numpy's unfamiliar output myself a long time ago and I see others get confused as well. So what is the issue? Let the svd decomposition of X be USV: U, S, V = np.linalg.svd(X), Then U has X's left singular vectors in its columns, S contains the singular values, and V has X's right singular vectors in its *rows*. The reconstruction of X is (matrix notation) will be: U*diag(S)*V (not U*diag(S)*V.T). All other high-level software (Matlab, Scilab, Mathematica, R, etc), outputs the right singular vectors columnwise, that is V = [v1, v2, v3, ... vn], where vn is a column (eigenvector) in V, thus the reconstruction would be U*diag(S)*V.T. Also, as far as I know, most linear algebra textbooks operate with eigenvectors consistent as column vectors in explanations of the SVD. I think numpy's svd should do so too. I know lapack's dgesdd returns V.T (or conjugate), and specify that in its documentation so this is a true interface of the library, but I still think its wrong and its just too confusing for any beginner who usually has experience in other software, such as matlab, prior to numpy. Also, ,as I am typing here, I realize that changing the output would break lots of stuff, and pass silently through many tests as the shape of V is similar (if full_matrices=0). Oh well, I guess that proposal is off the table? Perhaps some *stronger* hints in the documentation are needed. Arnar PS: In the docs at http://www.scipy.org/NumPy_for_Matlab_Users , the svd equivalents have wrong notation, this is not helping :-). I didnt manage to change it, perhaps some other may be so kind?

2 1

exponentiation q.
by Gideon Simpson 25 Jul '08

25 Jul '08

How does python (or numpy/scipy) do exponentiation? If I do x**p, where p is some positive integer, will it compute x*x*...*x (p times), or will it use logarithms? -gideon

3 3

Re: [Numpy-discussion] Numpy-discussion Digest, Vol 22, Issue 109
by Frank Lagor 25 Jul '08

25 Jul '08

Thanks so much for your help on the '*' confusion. It makes sense now. Thanks, Frank On Fri, Jul 25, 2008 at 3:57 PM, <numpy-discussion-request(a)scipy.org> wrote: > Send Numpy-discussion mailing list submissions to > numpy-discussion(a)scipy.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://projects.scipy.org/mailman/listinfo/numpy-discussion > or, via email, send a message with subject or body 'help' to > numpy-discussion-request(a)scipy.org > > You can reach the person managing the list at > numpy-discussion-owner(a)scipy.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Numpy-discussion digest..." > > Today's Topics: > > 1. Re: [Cdat-discussion] Arrays containing NaNs (Charles Doutriaux) > 2. numpy-1.1.1rc2 Mac binary - Please Test. (Christopher Burns) > 3. Re: numpy-1.1.1rc2 Mac binary - Please Test. (Alan McIntyre) > 4. curious problem with SVD (Frank Lagor) > 5. Re: curious problem with SVD (Keith Goodman) > 6. Re: curious problem with SVD (Keith Goodman) > 7. Re: curious problem with SVD (Robert Kern) > 8. Re: curious problem with SVD (Arnar Flatberg) > > > ---------- Forwarded message ---------- > From: Charles Doutriaux <doutriaux1(a)llnl.gov> > To: Pierre GM <pgmdevlist(a)gmail.com> > Date: Fri, 25 Jul 2008 11:43:41 -0700 > Subject: Re: [Numpy-discussion] [Cdat-discussion] Arrays containing NaNs > Hi Pierre, > > Thanks for the answer, I'm ccing cdat's discussion list. > > It makes sense, that's also the way we develop things here NEVER assume > what the user is going to do with the data BUT give the user the necessary > tools to do what you're assuming he/she wants to do (as simple as possible) > > Thanks again for the answer. > > C. > > > Pierre GM wrote: > >> Oh, I guess this one's for me... >> >> On Thursday 01 January 1970 04:21:03 Charles Doutriaux wrote: >> >> >> >>> Basically it was suggested to automarically mask NaN (and Inf ?) when >>> creating ma. >>> I'm sure you already thought of this on this list and was curious to >>> know why you decided not to do it. >>> >>> >> >> Because it's always best to let the user decide what to do with his/her >> data and not impose anything ? >> >> Masking a point doesn't necessarily mean that the point is invalid (in the >> sense of NaNs/Infs), just that it doesn't satisfy some particular condition. >> In that sense, masks act as selecting tools. >> >> By forcing invalid data to be masked at the creation of an array, you run >> the risk to tamper with the (potential) physical meaning of the mask you >> have given as input, and/or miss the fact that some data are actually >> invalid when you don't expect it to be. >> >> Let's take an example: I want to analyze sea surface temperatures at the >> world scale. The data comes as a regular 2D ndarray, with NaNs for missing >> or invalid data. In a first step, I create a masked array of this data, >> filtering out the land masses by a predefined geographical mask. The >> remaining NaNs in the masked array indicate areas where the sensor failed... >> It's an important information I would probably have missed by masking all >> the NaNs at first... >> >> >> As Eric F. suggested, you can use numpy.ma.masked_invalid to create a >> masked array with NaNs/Infs filtered out: >> >> >> >>> import numpy as np,. numpy.ma as ma >>>>> x = np.array([1,2,None,4], dtype=float) >>>>> x >>>>> >>>>> >>>> array([ 1., 2., NaN, 4.]) >> >> >>> mx = ma.masked_invalid(x) >>>>> mx >>>>> >>>>> >>>> masked_array(data = [1.0 2.0 -- 4.0], >> mask = [False False True False], >> fill_value=1e+20) >> >> Note that the underlying data still has NaNs/Infs: >> >> >>> mx._data >>>>> >>>>> >>>> array([ 1., 2., NaN, 4.]) >> >> You can also use the ma.fix_invalid function: it creates a mask where the >> data is not finite (NaNs/Infs), and set the corresponding points to >> fill_value. >> >> >>> mx = ma.fix_invalid(x, fill_value=999) >>>>> mx >>>>> >>>>> >>>> masked_array(data = [1.0 2.0 -- 4.0], >> mask = [False False True False], >> fill_value=1e+20) >> >> >>> mx._data >>>>> >>>>> >>>> array([ 1., 2., 999., 4.]) >> >> >> The advantage of the second approach is that you no longer have NaNs/Infs >> in the underlying data, which speeds things up during computation. The >> obvious disadvantage is that you no longer know where the data was >> invalid... >> >> >> > > > > > ---------- Forwarded message ---------- > From: "Christopher Burns" <cburns(a)berkeley.edu> > To: "Discussion of Numerical Python" <numpy-discussion(a)scipy.org> > Date: Fri, 25 Jul 2008 11:48:13 -0700 > Subject: [Numpy-discussion] numpy-1.1.1rc2 Mac binary - Please Test. > Reminder, please test the Mac installer for rc2 so we have time to fix any > bugs before the release next week. > > Also, I committed my build script to the trunk/tools/osxbuild. bdist_mpkg > 0.4.3 is required. > > Thank you, > Chris > > On Thu, Jul 24, 2008 at 11:03 AM, Jarrod Millman <millman(a)berkeley.edu> > wrote: > >> Hello, >> >> The 1.1.1rc2 is now available: >> http://svn.scipy.org/svn/numpy/tags/1.1.1rc2 >> >> The source tarball is here: >> http://cirl.berkeley.edu/numpy/numpy-1.1.1rc2.tar.gz >> >> Here is the universal Mac binary: >> http://cirl.berkeley.edu/numpy/numpy-1.1.1rc2-py2.5-macosx10.5.dmg >> >> David Cournapeau will be creating a 1.1.1rc2 Windows binary in next few >> days. >> >> Please test this release ASAP and let us know if there are any >> problems. If there are no show stoppers, this will likely become the >> 1.1.1 release. >> >> Thanks, >> >> -- >> Jarrod Millman >> Computational Infrastructure for Research Labs >> 10 Giannini Hall, UC Berkeley >> phone: 510.643.4014 >> http://cirl.berkeley.edu/ >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion(a)scipy.org >> http://projects.scipy.org/mailman/listinfo/numpy-discussion >> > > > > ---------- Forwarded message ---------- > From: "Alan McIntyre" <alan.mcintyre(a)gmail.com> > To: "Discussion of Numerical Python" <numpy-discussion(a)scipy.org> > Date: Fri, 25 Jul 2008 15:00:42 -0400 > Subject: Re: [Numpy-discussion] numpy-1.1.1rc2 Mac binary - Please Test. > On Fri, Jul 25, 2008 at 2:48 PM, Christopher Burns <cburns(a)berkeley.edu> > wrote: > > Reminder, please test the Mac installer for rc2 so we have time to fix > any > > bugs before the release next week. > > I just tried it; it installs with no problems and tests run with no > failures. > > > > ---------- Forwarded message ---------- > From: "Frank Lagor" <dfranci(a)seas.upenn.edu> > To: numpy-discussion(a)scipy.org > Date: Fri, 25 Jul 2008 15:32:56 -0400 > Subject: [Numpy-discussion] curious problem with SVD > Perhaps I do not understand something properly, if so could someone please > explain the behavior I notice with numpy.linalg.svd when acting on arrays. > It gives the incorrect answer, but works fine with matrices. My numpy is > 1.1.0. > > >>> R = n.array([[3.6,.35],[.35,1.8]]) > >>> V,D,W = n.linalg.svd(R) > >>> V*n.diag(D)*W.transpose() > array([[ 3.5410365 , 0. ], > [ 0. , 1.67537611]]) > >>> R = n.matrix([[3.6,.35],[.35,1.8]]) > >>> V,D,W = n.linalg.svd(R) > >>> V*n.diag(D)*W.transpose() > matrix([[ 3.6 , 0.35], > [ 0.35, 1.8 ]]) > > Thanks in advance, > Frank > -- > Frank Lagor > Ph.D. Candidate > Mechanical Engineering and Applied Mechanics > University of Pennsylvania > > > ---------- Forwarded message ---------- > From: "Keith Goodman" <kwgoodman(a)gmail.com> > To: "Discussion of Numerical Python" <numpy-discussion(a)scipy.org> > Date: Fri, 25 Jul 2008 12:36:28 -0700 > Subject: Re: [Numpy-discussion] curious problem with SVD > On Fri, Jul 25, 2008 at 12:32 PM, Frank Lagor <dfranci(a)seas.upenn.edu> > wrote: > > Perhaps I do not understand something properly, if so could someone > please > > explain the behavior I notice with numpy.linalg.svd when acting on > arrays. > > It gives the incorrect answer, but works fine with matrices. My numpy is > > 1.1.0. > > > >>>> R = n.array([[3.6,.35],[.35,1.8]]) > >>>> V,D,W = n.linalg.svd(R) > >>>> V*n.diag(D)*W.transpose() > > array([[ 3.5410365 , 0. ], > > [ 0. , 1.67537611]]) > >>>> R = n.matrix([[3.6,.35],[.35,1.8]]) > >>>> V,D,W = n.linalg.svd(R) > >>>> V*n.diag(D)*W.transpose() > > matrix([[ 3.6 , 0.35], > > [ 0.35, 1.8 ]]) > > '*' does element-by-element multiplication for arrays but matrix > multiplication for matrices. > > > > ---------- Forwarded message ---------- > From: "Keith Goodman" <kwgoodman(a)gmail.com> > To: "Discussion of Numerical Python" <numpy-discussion(a)scipy.org> > Date: Fri, 25 Jul 2008 12:39:23 -0700 > Subject: Re: [Numpy-discussion] curious problem with SVD > On Fri, Jul 25, 2008 at 12:36 PM, Keith Goodman <kwgoodman(a)gmail.com> > wrote: > > On Fri, Jul 25, 2008 at 12:32 PM, Frank Lagor <dfranci(a)seas.upenn.edu> > wrote: > >> Perhaps I do not understand something properly, if so could someone > please > >> explain the behavior I notice with numpy.linalg.svd when acting on > arrays. > >> It gives the incorrect answer, but works fine with matrices. My numpy > is > >> 1.1.0. > >> > >>>>> R = n.array([[3.6,.35],[.35,1.8]]) > >>>>> V,D,W = n.linalg.svd(R) > >>>>> V*n.diag(D)*W.transpose() > >> array([[ 3.5410365 , 0. ], > >> [ 0. , 1.67537611]]) > >>>>> R = n.matrix([[3.6,.35],[.35,1.8]]) > >>>>> V,D,W = n.linalg.svd(R) > >>>>> V*n.diag(D)*W.transpose() > >> matrix([[ 3.6 , 0.35], > >> [ 0.35, 1.8 ]]) > > > > '*' does element-by-element multiplication for arrays but matrix > > multiplication for matrices. > > As a check (for the array case): > > >> n.dot(V, n.dot(n.diag(D), W.transpose())) # That's hard to read! > > array([[ 3.6 , 0.35], > [ 0.35, 1.8 ]]) > > > > ---------- Forwarded message ---------- > From: "Robert Kern" <robert.kern(a)gmail.com> > To: "Discussion of Numerical Python" <numpy-discussion(a)scipy.org> > Date: Fri, 25 Jul 2008 14:50:24 -0500 > Subject: Re: [Numpy-discussion] curious problem with SVD > On Fri, Jul 25, 2008 at 14:32, Frank Lagor <dfranci(a)seas.upenn.edu> wrote: > > Perhaps I do not understand something properly, if so could someone > please > > explain the behavior I notice with numpy.linalg.svd when acting on > arrays. > > It gives the incorrect answer, but works fine with matrices. My numpy is > > 1.1.0. > > > >>>> R = n.array([[3.6,.35],[.35,1.8]]) > >>>> V,D,W = n.linalg.svd(R) > >>>> V*n.diag(D)*W.transpose() > > For regular arrays, * is element-wise multiplication, not matrix > multiplication. For matrix objects, * is matrix multiplication. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > > > > ---------- Forwarded message ---------- > From: "Arnar Flatberg" <arnar.flatberg(a)gmail.com> > To: "Discussion of Numerical Python" <numpy-discussion(a)scipy.org> > Date: Fri, 25 Jul 2008 21:53:09 +0200 > Subject: Re: [Numpy-discussion] curious problem with SVD > > On Fri, Jul 25, 2008 at 9:39 PM, Keith Goodman <kwgoodman(a)gmail.com> > wrote: > >> On Fri, Jul 25, 2008 at 12:36 PM, Keith Goodman <kwgoodman(a)gmail.com> >> wrote: >> > On Fri, Jul 25, 2008 at 12:32 PM, Frank Lagor <dfranci(a)seas.upenn.edu> >> wrote: >> >> Perhaps I do not understand something properly, if so could someone >> please >> >> explain the behavior I notice with numpy.linalg.svd when acting on >> arrays. >> >> It gives the incorrect answer, but works fine with matrices. My numpy >> is >> >> > '*' does element-by-element multiplication for arrays but matrix >> > multiplication for mat >> >> n.dot(V, n.dot(n.diag(D), W.transpose())) # That's hard to read! > > > Just two small points: > > 1.) Broadcasting may be easier on the eye ... well, atleast when you have > gotten used to it > Then the above is np.dot(V*D, W) > > 2.) Also, note that the right hand side eigenvectors in numpy's svd routine > is ordered by rows! > Yes, I know this is confusing as it is different from just about any other > linear algebra software out there, but the documentation is clear. It is > also a little inconsistent with eig and eigh, some more experienced user can > probably answer on why it is like that? > > Arnar > > > >> <http://projects.scipy.org/mailman/listinfo/numpy-discussion> >> > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion(a)scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -- Frank Lagor Ph.D. Candidate Mechanical Engineering and Applied Mechanics University of Pennsylvania

1 0

curious problem with SVD
by Frank Lagor 25 Jul '08

25 Jul '08

Perhaps I do not understand something properly, if so could someone please explain the behavior I notice with numpy.linalg.svd when acting on arrays. It gives the incorrect answer, but works fine with matrices. My numpy is 1.1.0. >>> R = n.array([[3.6,.35],[.35,1.8]]) >>> V,D,W = n.linalg.svd(R) >>> V*n.diag(D)*W.transpose() array([[ 3.5410365 , 0. ], [ 0. , 1.67537611]]) >>> R = n.matrix([[3.6,.35],[.35,1.8]]) >>> V,D,W = n.linalg.svd(R) >>> V*n.diag(D)*W.transpose() matrix([[ 3.6 , 0.35], [ 0.35, 1.8 ]]) Thanks in advance, Frank -- Frank Lagor Ph.D. Candidate Mechanical Engineering and Applied Mechanics University of Pennsylvania

4 4

Re: [Numpy-discussion] [Cdat-discussion] Arrays containing NaNs
by Charles Doutriaux 25 Jul '08

25 Jul '08

Hi All, I'm sending a copy of this reply here because i think we could get some good answer. Basically it was suggested to automarically mask NaN (and Inf ?) when creating ma. I'm sure you already thought of this on this list and was curious to know why you decided not to do it. Just so I can relay it to our list (sending to both list came back flagged as spam...) C. Hi Stephane, This is a good suggestion, I'm ccing the numpy list on this. Because I'm wondering if it wouldn't be a better fit to do it directly at the numpy.ma level. I'm sure they already thought about this (and 'inf' values as well) and if they don't do it , there's probably some good reason we didn't think of yet. So before i go ahead and do it in MV2 I'd like to know the reason why it's not in numpy.ma, they are probably valid for MVs too. C. Stephane Raynaud wrote: > Hi, > > how about automatically (or at least optionally) masking all NaN > values when creating a MV array? > > On Thu, Jul 24, 2008 at 11:43 PM, Arthur M. Greene > <amg(a)iri.columbia.edu <mailto:amg@iri.columbia.edu>> wrote: > > Yup, this works. Thanks! > > I guess it's time for me to dig deeper into numpy syntax and > functions, now that CDAT is using the numpy core for array > management... > > Best, > > Arthur > > > Charles Doutriaux wrote: > > Seems right to me, > > Except that the syntax might scare a bit the new users :) > > C. > > Andrew.Dawson(a)uea.ac.uk <mailto:Andrew.Dawson@uea.ac.uk> wrote: > > Hi, > > I'm not sure if what I am about to suggest is a good idea > or not, perhaps Charles will correct me if this is a bad > idea for any reason. > > Lets say you have a cdms variable called U with NaNs as > the missing > value. First we can replace the NaNs with 1e20: > > U.data[numpy.where(numpy.isnan(U.data))] = 1e20 > > And remember to set the missing value of the variable > appropriately: > > U.setMissing(1e20) > > I hope that helps, Andrew > > > > Hi Arthur, > > If i remember correctly the way i used to do it was: > a= MV2.greater(data,1.) b=MV2.less_equal(data,1) > c=MV2.logical_and(a,b) # Nan are the only one left > data=MV2.masked_where(c,data) > > BUT I believe numpy now has way to deal with nan I > believe it is numpy.nan_to_num But it replaces with 0 > so it may not be what you > want > > C. > > > Arthur M. Greene wrote: > > A typical netcdf file is opened, and the single > variable extracted: > > > fpr=cdms.open('prTS2p1_SEA_allmos.cdf') > pr0=fpr('prcp') type(pr0) > > <class 'cdms2.tvariable.TransientVariable'> > > Masked values (indicating ocean in this case) show > up here as NaNs. > > > pr0[0,-15:-5,0] > > prcp array([NaN NaN NaN NaN NaN NaN 0.37745094 > 0.3460784 0.21960783 0.19117641]) > > So far this is all consistent. A map of the first > time step shows the proper land-ocean boundaries, > reasonable-looking values, and so on. But there > doesn't seem to be any way to mask > this array, so, e.g., an 'xy' average can be > computed (it > comes out all nans). NaN is not equal to anything > -- even > itself -- so there does not seem to be any > condition, among the > MV.masked_xxx options, that can be applied as a > test. Also, it > does not seem possible to compute seasonal averages, > anomalies, etc. -- they also produce just NaNs. > > The workaround I've come up with -- for now -- is > to first generate a new array of identical shape, > filled with 1.0E+20. One test I've found that can > detect NaNs is numpy.isnan: > > > isnan(pr0[0,0,0]) > > True > > So it is _possible_ to tediously loop through > every value in the old array, testing with isnan, > then copying to the new array if the test fails. > Then the axes have to be reset... > > isnan does not accept array arguments, so one > cannot do, e.g., > > prmasked=MV.masked_where(isnan(pr0),pr0) > > The element-by-element conversion is quite slow. > (I'm still waiting for it to complete, in fact). > Any suggestions for dealing with NaN-infested data > objects? > > Thanks! > > AMG > > P.S. This is 5.0.0.beta, RHEL4. > > > *^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~* > Arthur M. Greene, Ph.D. > The International Research Institute for Climate and Society > The Earth Institute, Columbia University, Lamont Campus > Monell Building, 61 Route 9W, Palisades, NY 10964-8000 USA > amg*at*iri-dot-columbia\dot\edu | http://iri.columbia.edu > *^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~* > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's > challenge > Build the coolest Linux based applications with Moblin SDK & win > great prizes > Grand prize is a trip for two to an Open Source event anywhere in > the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > <http://moblin-contest.org/redirect.php?banner_id=100&url=/> > _______________________________________________ > Cdat-discussion mailing list > Cdat-discussion(a)lists.sourceforge.net > <mailto:Cdat-discussion@lists.sourceforge.net> > https://lists.sourceforge.net/lists/listinfo/cdat-discussion > > > > > -- > Stephane Raynaud > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http:// moblin-contest.org/redirect.php?banner_id=100&url=/ > ------------------------------------------------------------------------ > > _______________________________________________ > Cdat-discussion mailing list > Cdat-discussion(a)lists.sourceforge.net > https:// lists.sourceforge.net/lists/listinfo/cdat-discussion >

2 2

Re: [Numpy-discussion] [Cdat-discussion] Arrays containing NaNs
by Charles Doutriaux 25 Jul '08

25 Jul '08

Hi Stephane, This is a good suggestion, I'm ccing the numpy list on this. Because I'm wondering if it wouldn't be a better fit to do it directly at the numpy.ma level. I'm sure they already thought about this (and 'inf' values as well) and if they don't do it , there's probably some good reason we didn't think of yet. So before i go ahead and do it in MV2 I'd like to know the reason why it's not in numpy.ma, they are probably valid for MVs too. C. Stephane Raynaud wrote: > Hi, > > how about automatically (or at least optionally) masking all NaN > values when creating a MV array? > > On Thu, Jul 24, 2008 at 11:43 PM, Arthur M. Greene > <amg(a)iri.columbia.edu <mailto:amg@iri.columbia.edu>> wrote: > > Yup, this works. Thanks! > > I guess it's time for me to dig deeper into numpy syntax and > functions, now that CDAT is using the numpy core for array > management... > > Best, > > Arthur > > > Charles Doutriaux wrote: > > Seems right to me, > > Except that the syntax might scare a bit the new users :) > > C. > > Andrew.Dawson(a)uea.ac.uk <mailto:Andrew.Dawson@uea.ac.uk> wrote: > > Hi, > > I'm not sure if what I am about to suggest is a good idea > or not, perhaps Charles will correct me if this is a bad > idea for any reason. > > Lets say you have a cdms variable called U with NaNs as > the missing > value. First we can replace the NaNs with 1e20: > > U.data[numpy.where(numpy.isnan(U.data))] = 1e20 > > And remember to set the missing value of the variable > appropriately: > > U.setMissing(1e20) > > I hope that helps, Andrew > > > > Hi Arthur, > > If i remember correctly the way i used to do it was: > a= MV2.greater(data,1.) b=MV2.less_equal(data,1) > c=MV2.logical_and(a,b) # Nan are the only one left > data=MV2.masked_where(c,data) > > BUT I believe numpy now has way to deal with nan I > believe it is numpy.nan_to_num But it replaces with 0 > so it may not be what you > want > > C. > > > Arthur M. Greene wrote: > > A typical netcdf file is opened, and the single > variable extracted: > > > fpr=cdms.open('prTS2p1_SEA_allmos.cdf') > pr0=fpr('prcp') type(pr0) > > <class 'cdms2.tvariable.TransientVariable'> > > Masked values (indicating ocean in this case) show > up here as NaNs. > > > pr0[0,-15:-5,0] > > prcp array([NaN NaN NaN NaN NaN NaN 0.37745094 > 0.3460784 0.21960783 0.19117641]) > > So far this is all consistent. A map of the first > time step shows the proper land-ocean boundaries, > reasonable-looking values, and so on. But there > doesn't seem to be any way to mask > this array, so, e.g., an 'xy' average can be > computed (it > comes out all nans). NaN is not equal to anything > -- even > itself -- so there does not seem to be any > condition, among the > MV.masked_xxx options, that can be applied as a > test. Also, it > does not seem possible to compute seasonal averages, > anomalies, etc. -- they also produce just NaNs. > > The workaround I've come up with -- for now -- is > to first generate a new array of identical shape, > filled with 1.0E+20. One test I've found that can > detect NaNs is numpy.isnan: > > > isnan(pr0[0,0,0]) > > True > > So it is _possible_ to tediously loop through > every value in the old array, testing with isnan, > then copying to the new array if the test fails. > Then the axes have to be reset... > > isnan does not accept array arguments, so one > cannot do, e.g., > > prmasked=MV.masked_where(isnan(pr0),pr0) > > The element-by-element conversion is quite slow. > (I'm still waiting for it to complete, in fact). > Any suggestions for dealing with NaN-infested data > objects? > > Thanks! > > AMG > > P.S. This is 5.0.0.beta, RHEL4. > > > *^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~* > Arthur M. Greene, Ph.D. > The International Research Institute for Climate and Society > The Earth Institute, Columbia University, Lamont Campus > Monell Building, 61 Route 9W, Palisades, NY 10964-8000 USA > amg*at*iri-dot-columbia\dot\edu | http://iri.columbia.edu > *^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~*^*~* > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's > challenge > Build the coolest Linux based applications with Moblin SDK & win > great prizes > Grand prize is a trip for two to an Open Source event anywhere in > the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > <http://moblin-contest.org/redirect.php?banner_id=100&url=/> > _______________________________________________ > Cdat-discussion mailing list > Cdat-discussion(a)lists.sourceforge.net > <mailto:Cdat-discussion@lists.sourceforge.net> > https://lists.sourceforge.net/lists/listinfo/cdat-discussion > > > > > -- > Stephane Raynaud > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http:// moblin-contest.org/redirect.php?banner_id=100&url=/ > ------------------------------------------------------------------------ > > _______________________________________________ > Cdat-discussion mailing list > Cdat-discussion(a)lists.sourceforge.net > https:// lists.sourceforge.net/lists/listinfo/cdat-discussion >

3 5

Putting blas/lapack/atlas code + tools somewhere in svn.scipy.org ?
by David Cournapeau 25 Jul '08

25 Jul '08

Hi, I would like to know if it would be possible at all to put blas/lapack/atlas code + various scripts to build them for windows in an automated way somewhere in svn.scipy.org, a bit like what svn.python.org does for external dependencies ? The rationale is that to build atlas, you need to build it on different machines for different archs, and having a repeatable build by several people would be a big plus (I also hope that someone would step in to support windows at some point :) ). As python devs do for tk and co I think, I don't want to track the changes in those external packages, just some releases to be in sync together with build scripts: I don't think it should take too much space. cheers, David

2 2

Documenting `zipf`
by Stéfan van der Walt 24 Jul '08

24 Jul '08

Hi, Does anybody know how Zipf's law or how Zipfian distributions work, and how they relate to NumPy's `np.random.zipf`? I'm afraid I can't make head or tail of these results: In [106]: np.random.zipf(2, size=(10)) Out[106]: array([ 1, 1, 1, 29, 1, 1, 1, 1, 1, 2]) (8x1, 1x2, 1x29) In [107]: np.random.zipf(2, size=(10)) Out[107]: array([75, 1, 1, 3, 1, 1, 1, 1, 1, 4]) (7x1, 1x3, 1x4, 1x75) In [108]: np.random.zipf(2, size=(10)) Out[108]: array([ 6, 17, 2, 1, 1, 2, 1, 20, 1, 2]) (4x1, 3x2, 1x6, 1x17, 1x20) Thanks! Stéfan

2 1

import numpy fails with multiarray.so: undefined symbol: PyUnicodeUCS2_FromUnicode
by G 24 Jul '08

24 Jul '08

Hello, I have installed the svn version of numpy. I have deleted all previous versions of and files related to numpy prior to installing. I also have tried reinstalling python from source. Regardless, when I try import numpy, I get the following: Python 2.5.2 (r252:60911, Jul 23 2008, 23:54:29) [GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. import numpy Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python2.5/site-packages/numpy/__init__.py", line 93, in <module> import add_newdocs File "/usr/local/lib/python2.5/site-packages/numpy/add_newdocs.py", line 9, in <module> from lib import add_newdoc File "/usr/local/lib/python2.5/site-packages/numpy/lib/__init__.py", line 4, in <module> from type_check import * File "/usr/local/lib/python2.5/site-packages/numpy/lib/type_check.py", line 8, in <module> import numpy.core.numeric as _nx File "/usr/local/lib/python2.5/site-packages/numpy/core/__init__.py", line 5, in <module> import multiarray ImportError: /usr/local/lib/python2.5/site-packages/numpy/core/multiarray.so: undefined symbol: PyUnicodeUCS2_FromUnicode I also tried compiling python using ./configure --enable-unicode=ucs4 with no luck. I had python and numpy all working well but then some file got corrupted so I was forced to reinstall, and I have not yet been able to get things working again after two days of attempts. Thank you, G

4 4