From sebastian.walter at gmail.com Wed Sep 1 04:07:05 2010 From: sebastian.walter at gmail.com (Sebastian Walter) Date: Wed, 1 Sep 2010 10:07:05 +0200 Subject: [Numpy-discussion] inversion of large matrices In-Reply-To: References: <54848FA5-AD74-46E9-AE0B-3189FB40C104@cs.toronto.edu> <20A7F4F6-3F2A-47CF-AACE-8484AB92A0E3@cs.toronto.edu> Message-ID: is it really the covariance matrix you want to invert? Or do you want to compute something like x^T C^{-1} x, where x is an array of size N and C an array of size (N,N)? It would also be interesting to know how the covariance matrix gets computed and what its condition number is, at least approximately. On Wed, Sep 1, 2010 at 1:58 AM, Charles R Harris wrote: > > > On Tue, Aug 31, 2010 at 4:52 PM, Dan Elliott > wrote: >> >> David Warde-Farley cs.toronto.edu> writes: >> > On 2010-08-30, at 10:36 PM, Charles R Harris wrote: >> > I think he means that if he needs both the determinant and to solve the >> > system, it might be more efficient to do >> > the SVD, obtain the determinant from the diagonal values, and obtain the >> > solution by multiplying by U D^-1 V^T? >> >> Thank you, that is what I meant. ?Poorly worded on my part. >> >> In particular, I am writing code to invert a very large covariance matrix. >> ?I >> think David has some good information in another post in this thread. >> > > Where did the covariance array come from? It may be the case that you can > use a much smaller one, for instance in PCA of images. > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From ralf.gommers at googlemail.com Wed Sep 1 09:56:28 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 1 Sep 2010 21:56:28 +0800 Subject: [Numpy-discussion] Where is the dev version of numpydoc? In-Reply-To: References: Message-ID: On Wed, Sep 1, 2010 at 11:31 AM, John Salvatier wrote: > Hello, > > I would like to update my numpydoc so it works with sphinx 1.0, but I am > not sure where the dev version is; can someone point me in the right > direction? > > In numpy trunk, under doc/sphinxext/. That works *only* with sphinx 1.0 now, which is perhaps the reason there's no new release on pypi (but that's just my guess). Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Sep 1 11:46:53 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 1 Sep 2010 09:46:53 -0600 Subject: [Numpy-discussion] Github migration? In-Reply-To: References: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> Message-ID: On Tue, Aug 31, 2010 at 2:56 PM, Jason McCampbell wrote: > Hi Chuck (and anyone else interested), > > I updated the refactoring page on the NumPy developer wiki (seems to be > down or I'd paste in the link). It certainly isn't complete, but there are > a lot more details about the data structures and memory handling and an > outline of some additional topics that needs to be filled in. > > Thanks Jason. How much of the core library can be used without any reference counting? I was originally thinking that the base ufuncs would just be functions accepting a pointer and a descriptor and handling memory allocations and such would be at a higher level. That is to say, the object oriented aspects of numpy would be removed from the bottom layers where they just get in the way. Also, since many of the public macros expect the old type structures, what is going to happen with them? They are really part of the API, but a particularly troublesome part for going forward. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From kackvogel at googlemail.com Wed Sep 1 12:07:21 2010 From: kackvogel at googlemail.com (Babak Ahmadi) Date: Wed, 1 Sep 2010 18:07:21 +0200 Subject: [Numpy-discussion] Parsing PyArrays Message-ID: Hi, I'm having a little Problem that might be trivial. I'm trying to use a nupmy array within a c++ method and having some trouble. The PyArray is not parsed correctly and PyArg_ParseTuple returns 0. Appreciate any help. #include "Python.h" #include void fromMatrix(PyObject *args){ _import_array(); PyArrayObject* A; PyArrayObject* B; if (!PyArg_ParseTuple(args, "O!O!", &PyArray_Type,&A, &PyArray_Type,&B)) return; int m = PyArray_DIM(A,0); int n = PyArray_DIM(A,1); .... and so on. -------------- next part -------------- An HTML attachment was scrubbed... URL: From perry at stsci.edu Wed Sep 1 15:36:12 2010 From: perry at stsci.edu (Perry Greenfield) Date: Wed, 1 Sep 2010 15:36:12 -0400 Subject: [Numpy-discussion] Developer Job Openings at Space Telescope Science Institute Message-ID: <63BE5DE9-4882-4B2E-B03A-F28349DB39C0@stsci.edu> We are advertising for two different positions at the Space Telescope Science Institute (located on the Johns Hopkins University Campus in Baltimore, Md). STScI is seeking Senior Systems Software Engineers to develop applications to calibrate and analyze data from the Hubble and the James Webb Space Telescopes. **************************************************************************** First position: Description The developer will work with other members of the Science Software Branch and instrument scientists to develop applications to calibrate and analyze data from the Hubble Space Telescope and its successor, the James Webb Space Telescope, as well as other astronomy-related projects. The individual will be developing open source applications primarily in Python and C, using open source tools developed at STScI, such as PyRAF and PyFITS, and elsewhere. Some projects may involve developing software applications or libraries as part of a team, or leading a team. Requirements Candidates should have experience writing applications to calibrate, reduce, and analyze scientific data, preferably astronomical data. A background in astronomy and experience with astronomical data reduction is highly desirable. Candidates should have experience writing large programs in a compiled language as well as experience with an interpreted language such as IDL, Matlab, or Python. Experience using array manipulations facilities such as are available in IDL, Matlab, numpy/numarray, or APL is a plus. Experience using software engineering tools such as debuggers, CVS or subversion, and bug trackers is strongly desired. Strong analytical, problem-solving, planning, and organizational skills are needed, and excellent written and verbal communication skills are essential. Prior experience in developing medium or large projects sufficient to demonstrate the specified knowledge, skills and abilities is required. Qualified candidates should possess a Bachelor's degree in a science- related field such as Physics, Astronomy, or Mathematics. A Master's or Ph.D degree is desirable. Substitution of additional relevant education or experience for the stated qualifications may be considered. Apply through the following link: https://www.ultirecruit.com/SPA1004/jobboard/JobDetails.aspx?__ID=*6D48E0EFCC47915A **************************************************************************** Second position: Description The developer will work with other members of the Science Software Branch to help in enhancing and maintaining our Python-based framework for developing astronomical data analysis and calibration applications. STScI has pioneered in the generation of tools for using Python for scientific analysis and programming through its development of PyRAF, numarray, PyFITS, and contributions to other Python Open Source projects. The individual being sought will help STScI maintain its leadership in this area by developing leading-edge capabilities by enhancing existing tools such as PyRAF and PyFITS, contributing to scipy, numpy, and matplotlib, and developing new libraries to meet the needs of future astronomical processing. Some projects may involve developing software tools as part of a team, or leading a team. Work will also require working with an external community on Open Source software projects. Requirements Candidates should be experienced with systems-level programming, preferably with C or C++ and familiar with variances in processor and operating system architectures (preferably Linux, OS X, and MS Windows) with regard to file systems, memory, data types and efficiency, as well as modern software development techniques including Object-Oriented design and programming. Experience with Python and writing C extensions for Python is highly desirable. A working knowledge of any of the following would be a plus: parsers, code generation, numerical techniques, image processing and data analysis, web and network protocols, or parallel processing. Experience using software engineering tools such as debuggers, version control systems (e.g., subversion), and bug trackers is strongly desired. Strong analytical, problem-solving, planning, and organizational skills are needed, and excellent written and verbal communication skills are essential. Prior experience in developing medium or large projects sufficient to demonstrate the specified knowledge, skills and abilities is required. Qualified candidates should possess a Bachelor's Degree in Computer Science, Physics, Math, or technically related field. Master's degree preferred. Substitution of additional relevant education or experience for the stated qualifications may be considered. Apply through the following link: https://www.ultirecruit.com/SPA1004/jobboard/JobDetails.aspx?__ID=*85D01A9E3BE42CFD **************************************************************************** STScI offers an excellent benefits package, tuition reimbursement, competitive salaries, and a stimulating work environment. Interested candidates are requested to complete an on-line application, attach a resume in the "Resume Upload Section." Please include job #10-0083 in the filename. Resumes received by October 15, 2010 will receive full consideration. Committed to the benefits of diversity, we strongly encourage qualified women and minority candidates to apply. EOE/AA/M/F/D/V. From michael.s.gilbert at gmail.com Wed Sep 1 16:26:59 2010 From: michael.s.gilbert at gmail.com (Michael Gilbert) Date: Wed, 1 Sep 2010 16:26:59 -0400 Subject: [Numpy-discussion] Unexpected float96 precision loss Message-ID: <20100901162659.b80f6faf.michael.s.gilbert@gmail.com> Hi, I've been using numpy's float96 class lately, and I've run into some strange precision errors. See example below: >>> import numpy >>> numpy.version.version '1.5.0' >>> sys.version '3.1.2 (release31-maint, Jul 8 2010, 01:16:48) \n[GCC 4.4.4]' >>> x = numpy.array( [0.01] , numpy.float32 ) >>> y = numpy.array( [0.0001] , numpy.float32 ) >>> x[0]*x[0] - y[0] 0.0 >>> x = numpy.array( [0.01] , numpy.float64 ) >>> y = numpy.array( [0.0001] , numpy.float64 ) >>> x[0]*x[0] - y[0] 0.0 >>> x = numpy.array( [0.01] , numpy.float96 ) >>> y = numpy.array( [0.0001] , numpy.float96 ) >>> x[0]*x[0] - y[0] -6.286572655403010329e-22 I would expect the float96 calculation to also produce 0.0 exactly as found in the float32 and float64 examples. Why isn't this the case? Slightly off-topic: why was the float128 class dropped? Thanks in advance for any thoughts/feedback, Mike From charlesr.harris at gmail.com Wed Sep 1 17:13:07 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 1 Sep 2010 15:13:07 -0600 Subject: [Numpy-discussion] Unexpected float96 precision loss In-Reply-To: <20100901162659.b80f6faf.michael.s.gilbert@gmail.com> References: <20100901162659.b80f6faf.michael.s.gilbert@gmail.com> Message-ID: On Wed, Sep 1, 2010 at 2:26 PM, Michael Gilbert wrote: > Hi, > > I've been using numpy's float96 class lately, and I've run into some > strange precision errors. See example below: > > >>> import numpy > >>> numpy.version.version > '1.5.0' > >>> sys.version > '3.1.2 (release31-maint, Jul 8 2010, 01:16:48) \n[GCC 4.4.4]' > >>> x = numpy.array( [0.01] , numpy.float32 ) > >>> y = numpy.array( [0.0001] , numpy.float32 ) > >>> x[0]*x[0] - y[0] > 0.0 > >>> x = numpy.array( [0.01] , numpy.float64 ) > >>> y = numpy.array( [0.0001] , numpy.float64 ) > >>> x[0]*x[0] - y[0] > 0.0 > >>> x = numpy.array( [0.01] , numpy.float96 ) > >>> y = numpy.array( [0.0001] , numpy.float96 ) > >>> x[0]*x[0] - y[0] > -6.286572655403010329e-22 > > I would expect the float96 calculation to also produce 0.0 exactly as > found in the float32 and float64 examples. Why isn't this the case? > > None of the numbers is exactly represented in ieee floating format, so what you are seeing is rounding error. Note that the first two zeros are only accurate to about 7 and 16 digits respectively, whereas fot float 96 is accurate to about 19 digits. Slightly off-topic: why was the float128 class dropped? > > It wasn't, but you won't see it on a 32 bit system because of how the gcc compiler treats long doubles for alignment reasons. For common intel hardware/os, float96, and float128 are the same precision, just stored differently. In general the long precision formats are not portable, so watch out. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Wed Sep 1 17:15:22 2010 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 1 Sep 2010 21:15:22 +0000 (UTC) Subject: [Numpy-discussion] Unexpected float96 precision loss References: <20100901162659.b80f6faf.michael.s.gilbert@gmail.com> Message-ID: Wed, 01 Sep 2010 16:26:59 -0400, Michael Gilbert wrote: > I've been using numpy's float96 class lately, and I've run into some > strange precision errors. [clip] > >>> x = numpy.array( [0.01] , numpy.float96 ) [clip] > I would expect the float96 calculation to also produce 0.0 exactly as > found in the float32 and float64 examples. Why isn't this the case? (i) It is not possible to write long double literals in Python. "float96(0.0001)" means in fact "float96(float64(0.0001))" (ii) It is not possible to represent numbers 10^-r, r > 1 exactly in base-2 floating point. So if you write "float96(0.0001)", the result is not the float96 number closest to 0.0001, but the 96-bit representation of the 64-bit number closest to 0.0001. Indeed, >>> float96(0.0001), float96(1.0)/1000 (0.00010000000000000000479, 0.00099999999999999999996) -- Pauli Virtanen From michael.s.gilbert at gmail.com Wed Sep 1 17:30:08 2010 From: michael.s.gilbert at gmail.com (Michael Gilbert) Date: Wed, 1 Sep 2010 17:30:08 -0400 Subject: [Numpy-discussion] Unexpected float96 precision loss In-Reply-To: References: <20100901162659.b80f6faf.michael.s.gilbert@gmail.com> Message-ID: <20100901173008.96868da3.michael.s.gilbert@gmail.com> On Wed, 1 Sep 2010 21:15:22 +0000 (UTC), Pauli Virtanen wrote: > Wed, 01 Sep 2010 16:26:59 -0400, Michael Gilbert wrote: > > I've been using numpy's float96 class lately, and I've run into some > > strange precision errors. > [clip] > > >>> x = numpy.array( [0.01] , numpy.float96 ) > [clip] > > I would expect the float96 calculation to also produce 0.0 exactly as > > found in the float32 and float64 examples. Why isn't this the case? > > (i) It is not possible to write long double literals in Python. > "float96(0.0001)" means in fact "float96(float64(0.0001))" > > (ii) It is not possible to represent numbers 10^-r, r > 1 exactly > in base-2 floating point. > > So if you write "float96(0.0001)", the result is not the float96 number > closest to 0.0001, but the 96-bit representation of the 64-bit number > closest to 0.0001. Indeed, > > >>> float96(0.0001), float96(1.0)/1000 > (0.00010000000000000000479, 0.00099999999999999999996) Interesting. float96( '0.0001' ) also seems to evaluate to the first result. I assume that it also does a float64( '0.0001' ) conversion first. I understand that you can't change how python passes in floats, but wouldn't it be better to exactly handle strings since those can be converted exactly, which is what the user wants/expects? Thanks so much for the quick responses. Best wishes, Mike From thomas.robitaille at gmail.com Wed Sep 1 17:54:26 2010 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Wed, 1 Sep 2010 17:54:26 -0400 Subject: [Numpy-discussion] Array slices and number of dimensions Message-ID: Hi, I'm trying to extract sub-sections of a multidimensional array while keeping the number of dimensions the same. If I just select a specific element along a given direction, then the number of dimensions goes down by one: >>> import numpy as np >>> a = np.zeros((10,10,10)) >>> a.shape (10, 10, 10) >>> a[0,:,:].shape (10, 10) This makes sense to me. If I want to retain the initial number of dimensions, I can do >>> a[[0],:,:].shape (1, 10, 10) However, if I try and do this along two directions, I do get a reduction in the number of dimensions: >>> a[[0],:,[5]].shape (1, 10) I'm wondering if this is normal, or is a bug? In fact, I can get what I want by doing: >>> a[[0],:,:][:,:,[5]].shape (1, 10, 1) so I can get around the issue, but just wanted to check whether the issue with a[[0],:,[5]] is a bug? Thanks, Tom From warren.weckesser at enthought.com Wed Sep 1 17:59:53 2010 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Wed, 01 Sep 2010 16:59:53 -0500 Subject: [Numpy-discussion] Array slices and number of dimensions In-Reply-To: References: Message-ID: <4C7ECCD9.5020605@enthought.com> Thomas Robitaille wrote: > Hi, > > I'm trying to extract sub-sections of a multidimensional array while keeping the number of dimensions the same. If I just select a specific element along a given direction, then the number of dimensions goes down by one: > > > > In fact, I can get what I want by doing: > > >>>> a[[0],:,:][:,:,[5]].shape >>>> > (1, 10, 1) > > so I can get around the issue You can also use "trivial" slices: In [2]: a = np.zeros((10,10,10)) In [3]: a.shape Out[3]: (10, 10, 10) In [4]: a[0:1, :, 5:6].shape Out[4]: (1, 10, 1) Warren > , but just wanted to check whether the issue with a[[0],:,[5]] is a bug? > > Thanks, > > Tom > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From pav at iki.fi Wed Sep 1 18:05:07 2010 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 1 Sep 2010 22:05:07 +0000 (UTC) Subject: [Numpy-discussion] Array slices and number of dimensions References: Message-ID: Wed, 01 Sep 2010 17:54:26 -0400, Thomas Robitaille wrote: > >>> a[[0],:,:].shape > (1, 10, 10) [clip] > so I can get around the issue, but just wanted to check whether the > issue with a[[0],:,[5]] is a bug? No. The syntax does not mean what you think it means, see http://docs.scipy.org/doc/numpy/user/basics.indexing.html#indexing-multi-dimensional-arrays http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#advanced-indexing -- Pauli Virtanen From aarchiba at physics.mcgill.ca Wed Sep 1 18:07:34 2010 From: aarchiba at physics.mcgill.ca (Anne Archibald) Date: Wed, 1 Sep 2010 18:07:34 -0400 Subject: [Numpy-discussion] Array slices and number of dimensions In-Reply-To: References: Message-ID: On 1 September 2010 17:54, Thomas Robitaille wrote: > Hi, > > I'm trying to extract sub-sections of a multidimensional array while keeping the number of dimensions the same. If I just select a specific element along a given direction, then the number of dimensions goes down by one: > >>>> import numpy as np >>>> a = np.zeros((10,10,10)) >>>> a.shape > (10, 10, 10) >>>> a[0,:,:].shape > (10, 10) > > This makes sense to me. If I want to retain the initial number of dimensions, I can do > >>>> a[[0],:,:].shape > (1, 10, 10) > > However, if I try and do this along two directions, I do get a reduction in the number of dimensions: > >>>> a[[0],:,[5]].shape > (1, 10) > > I'm wondering if this is normal, or is a bug? In fact, I can get what I want by doing: > >>>> a[[0],:,:][:,:,[5]].shape > (1, 10, 1) > > so I can get around the issue, but just wanted to check whether the issue with a[[0],:,[5]] is a bug? No, it's not a bug. The key problem is that supplying lists does not extract a slice - it uses fancy indexing. This implies, among other things, that the data must be copied. When you supply two lists, that means something very different in fancy indexing. When you are supplying arrays in all index slots, what you get back has the same shape as the arrays you put in; so if you supply one-dimensional lists, like A[[1,2,3],[1,4,5],[7,6,2]] what you get is [A[1,1,7], A[2,4,6], A[3,5,2]] When you supply slices in some slots, what you get is complicated, and maybe not well-defined. In particular, I think the fancy-indexing dimensions always wind up at the front, and any slice dimensions are left at the end. In short, fancy indexing is not the way to go with your problem. I generally use np.newaxis: a[7,np.newaxis,:,8,np.newaxis] but you can also use slices of length one: a[7:8, :, 8:9] Anne > Thanks, > > Tom > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ndbecker2 at gmail.com Wed Sep 1 18:24:16 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 01 Sep 2010 18:24:16 -0400 Subject: [Numpy-discussion] kron produces F-contiguous? Message-ID: It seems if I call kron with 2 C-contiguous arrays, it returns an F- contiguous array. Any reason for this (it's not what I wanted)? From dalcinl at gmail.com Wed Sep 1 19:51:22 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 1 Sep 2010 20:51:22 -0300 Subject: [Numpy-discussion] kron produces F-contiguous? In-Reply-To: References: Message-ID: On 1 September 2010 19:24, Neal Becker wrote: > It seems if I call kron with 2 C-contiguous arrays, it returns an F- > contiguous array. ?Any reason for this (it's not what I wanted)? > Try numpy.linalg.inv ... -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From macdonald at maths.ox.ac.uk Wed Sep 1 21:06:14 2010 From: macdonald at maths.ox.ac.uk (Colin Macdonald) Date: Thu, 02 Sep 2010 02:06:14 +0100 Subject: [Numpy-discussion] Unexpected float96 precision loss In-Reply-To: <20100901173008.96868da3.michael.s.gilbert@gmail.com> References: <20100901162659.b80f6faf.michael.s.gilbert@gmail.com> <20100901173008.96868da3.michael.s.gilbert@gmail.com> Message-ID: <4C7EF886.805@maths.ox.ac.uk> On 09/01/10 22:30, Michael Gilbert wrote: > Interesting. float96( '0.0001' ) also seems to evaluate to the first > result. I assume that it also does a float64( '0.0001' ) conversion > first. I understand that you can't change how python passes in floats, > but wouldn't it be better to exactly handle strings since those can be > converted exactly, which is what the user wants/expects? I posted about this a week or two ago as well. Mean meaning to file bugs, but busy :( IIRC, the suggestion at the time was I might be able to use something like C's sscanf to do the string-to-float96 conversion properly. I haven't yet looked into it further, please let me know if you sort out any way to do it... cheers, Colin From charlesr.harris at gmail.com Wed Sep 1 21:03:00 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 1 Sep 2010 19:03:00 -0600 Subject: [Numpy-discussion] Unexpected float96 precision loss In-Reply-To: <20100901173008.96868da3.michael.s.gilbert@gmail.com> References: <20100901162659.b80f6faf.michael.s.gilbert@gmail.com> <20100901173008.96868da3.michael.s.gilbert@gmail.com> Message-ID: On Wed, Sep 1, 2010 at 3:30 PM, Michael Gilbert wrote: > On Wed, 1 Sep 2010 21:15:22 +0000 (UTC), Pauli Virtanen wrote: > > Wed, 01 Sep 2010 16:26:59 -0400, Michael Gilbert wrote: > > > I've been using numpy's float96 class lately, and I've run into some > > > strange precision errors. > > [clip] > > > >>> x = numpy.array( [0.01] , numpy.float96 ) > > [clip] > > > I would expect the float96 calculation to also produce 0.0 exactly as > > > found in the float32 and float64 examples. Why isn't this the case? > > > > (i) It is not possible to write long double literals in Python. > > "float96(0.0001)" means in fact "float96(float64(0.0001))" > > > > (ii) It is not possible to represent numbers 10^-r, r > 1 exactly > > in base-2 floating point. > > > > So if you write "float96(0.0001)", the result is not the float96 number > > closest to 0.0001, but the 96-bit representation of the 64-bit number > > closest to 0.0001. Indeed, > > > > >>> float96(0.0001), float96(1.0)/1000 > > (0.00010000000000000000479, 0.00099999999999999999996) > > Interesting. float96( '0.0001' ) also seems to evaluate to the first > result. I assume that it also does a float64( '0.0001' ) conversion > first. I understand that you can't change how python passes in floats, > but wouldn't it be better to exactly handle strings since those can be > converted exactly, which is what the user wants/expects? > > Well, yes. But then we would need to write our own routines for the conversions... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Wed Sep 1 21:44:56 2010 From: cournape at gmail.com (David Cournapeau) Date: Thu, 2 Sep 2010 10:44:56 +0900 Subject: [Numpy-discussion] Unexpected float96 precision loss In-Reply-To: References: <20100901162659.b80f6faf.michael.s.gilbert@gmail.com> <20100901173008.96868da3.michael.s.gilbert@gmail.com> Message-ID: On Thu, Sep 2, 2010 at 10:03 AM, Charles R Harris wrote: > > > On Wed, Sep 1, 2010 at 3:30 PM, Michael Gilbert > wrote: >> >> On Wed, 1 Sep 2010 21:15:22 +0000 (UTC), Pauli Virtanen wrote: >> > Wed, 01 Sep 2010 16:26:59 -0400, Michael Gilbert wrote: >> > > I've been using numpy's float96 class lately, and I've run into some >> > > strange precision errors. >> > [clip] >> > > ? >>> x = numpy.array( [0.01] , numpy.float96 ) >> > [clip] >> > > I would expect the float96 calculation to also produce 0.0 exactly as >> > > found in the float32 and float64 examples. ?Why isn't this the case? >> > >> > (i) It is not possible to write long double literals in Python. >> > ? ? "float96(0.0001)" means in fact "float96(float64(0.0001))" >> > >> > (ii) It is not possible to represent numbers 10^-r, r > 1 exactly >> > ? ? ?in base-2 floating point. >> > >> > So if you write "float96(0.0001)", the result is not the float96 number >> > closest to 0.0001, but the 96-bit representation of the 64-bit number >> > closest to 0.0001. Indeed, >> > >> > >>> float96(0.0001), float96(1.0)/1000 >> > (0.00010000000000000000479, 0.00099999999999999999996) >> >> Interesting. ?float96( '0.0001' ) also seems to evaluate to the first >> result. I assume that it also does a float64( '0.0001' ) conversion >> first. I understand that you can't change how python passes in floats, >> but wouldn't it be better to exactly handle strings since those can be >> converted exactly, which is what the user wants/expects? >> > > Well, yes. But then we would need to write our own routines for the > conversions... I think that it is needed at some point, though. There are quite a few bugs related to this kind of issues in NumPy, cheers, David From charlesr.harris at gmail.com Wed Sep 1 22:07:16 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 1 Sep 2010 20:07:16 -0600 Subject: [Numpy-discussion] Github migration? In-Reply-To: References: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> Message-ID: Hi Jason, On Tue, Aug 31, 2010 at 2:56 PM, Jason McCampbell wrote: > Hi Chuck (and anyone else interested), > > I updated the refactoring page on the NumPy developer wiki (seems to be > down or I'd paste in the link). It certainly isn't complete, but there are > a lot more details about the data structures and memory handling and an > outline of some additional topics that needs to be filled in. > > I note that there are some C++ style comments in the code which will cause errors on some platforms, so I hope you are planning on removing them at some point. Also, if (yes) foo; is very bad style. There is a lot of that in old code like that that still needs to be cleaned up, but I also see some in the new code. It would be best to get it right to start with. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Thu Sep 2 03:56:03 2010 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 2 Sep 2010 09:56:03 +0200 Subject: [Numpy-discussion] Equivalent Matlab function Message-ID: Hi, I'm looking for a Numpy equivalent of convmtx (http://www.mathworks.in/access/helpdesk/help/toolbox/signal/convmtx.html). Is there something inside Numpy directly? or perhaps Scipy? Matthieu -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher From ndbecker2 at gmail.com Thu Sep 2 08:27:15 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 02 Sep 2010 08:27:15 -0400 Subject: [Numpy-discussion] kron produces F-contiguous? References: Message-ID: Lisandro Dalcin wrote: > On 1 September 2010 19:24, Neal Becker wrote: >> It seems if I call kron with 2 C-contiguous arrays, it returns an F- >> contiguous array. Any reason for this (it's not what I wanted)? >> > > Try numpy.linalg.inv ... > > I don't understand. What has linalg.inv got to do with kronecker product? From pav at iki.fi Thu Sep 2 09:04:44 2010 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 2 Sep 2010 13:04:44 +0000 (UTC) Subject: [Numpy-discussion] kron produces F-contiguous? References: Message-ID: Wed, 01 Sep 2010 18:24:16 -0400, Neal Becker wrote: > It seems if I call kron with 2 C-contiguous arrays, it returns an F- > contiguous array. Any reason for this (it's not what I wanted)? Implementation detail. I don't think we have or want to have a policy of C-contiguous return values -- if you need C-contiguity, you'll need to ensure it by using `ascontiguousarray`. Especially when we later on implement memory access pattern optimizations for ufuncs, such assumptions will break down even more often. -- Pauli Virtanen From josef.pktd at gmail.com Thu Sep 2 10:19:31 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 2 Sep 2010 10:19:31 -0400 Subject: [Numpy-discussion] Equivalent Matlab function In-Reply-To: References: Message-ID: On Thu, Sep 2, 2010 at 3:56 AM, Matthieu Brucher wrote: > Hi, > > I'm looking for a Numpy equivalent of convmtx > (http://www.mathworks.in/access/helpdesk/help/toolbox/signal/convmtx.html). > Is there something inside Numpy directly? or perhaps Scipy? I haven't seen it in numpy or scipy, but I have seen something similar in a PDE or FEM package. Using the toeplitz hint from the matlab help, the following seems to work, at least in the examples >>> from scipy import linalg >>> h = np.array([1,3,1]) >>> nc = 10; nr = nc-len(h)+1 >>> linalg.toeplitz(np.r_[[1],np.zeros(nr-1)], np.r_[h,np.zeros(nc-len(h))]) array([[ 1., 3., 1., 0., 0., 0., 0., 0., 0., 0.], [ 0., 1., 3., 1., 0., 0., 0., 0., 0., 0.], [ 0., 0., 1., 3., 1., 0., 0., 0., 0., 0.], [ 0., 0., 0., 1., 3., 1., 0., 0., 0., 0.], [ 0., 0., 0., 0., 1., 3., 1., 0., 0., 0.], [ 0., 0., 0., 0., 0., 1., 3., 1., 0., 0.], [ 0., 0., 0., 0., 0., 0., 1., 3., 1., 0.], [ 0., 0., 0., 0., 0., 0., 0., 1., 3., 1.]]) >>> h = np.array([1,2,3,2,1]) >>> nc = 10; nr = nc-len(h)+1 >>> linalg.toeplitz(np.r_[[1],np.zeros(nr-1)], np.r_[h,np.zeros(nc-len(h))]) array([[ 1., 2., 3., 2., 1., 0., 0., 0., 0., 0.], [ 0., 1., 2., 3., 2., 1., 0., 0., 0., 0.], [ 0., 0., 1., 2., 3., 2., 1., 0., 0., 0.], [ 0., 0., 0., 1., 2., 3., 2., 1., 0., 0.], [ 0., 0., 0., 0., 1., 2., 3., 2., 1., 0.], [ 0., 0., 0., 0., 0., 1., 2., 3., 2., 1.]]) maybe worth adding to the other special matrix improvements that Warren did. Josef > > Matthieu > -- > Information System Engineer, Ph.D. > Blog: http://matt.eifelle.com > LinkedIn: http://www.linkedin.com/in/matthieubrucher > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Thu Sep 2 10:27:18 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 2 Sep 2010 08:27:18 -0600 Subject: [Numpy-discussion] Github migration? In-Reply-To: References: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> Message-ID: On Mon, Aug 30, 2010 at 3:08 PM, Pauli Virtanen wrote: > Mon, 23 Aug 2010 21:15:55 +0000, Pauli Virtanen wrote: > [clip] > > in the history to have the wrong content -- so to be sure, we have to do > > a brute-force comparison of the tree against SVN for each commit. The > > particular bug here was fixed in the conversion tool, but better safe > > than sorry. > > Also this is now done. The repository at > > http://github.com/numpy/numpy_svn_test4 > > is essentially fine. > > You can grab it via > > git clone git://github.com/numpy/numpy_svn_test4.git > > The deleted SVN branches can also be obtained: > > git fetch origin refs/svn/*:refs/remotes/svn/* > > They are not pulled by default. > > *** > > svn-all-fast-export still has some bugs, so revisions that in SVN are > r2478, r2491, r2534, r2602, r2603, r4113, r4179, r4180 contain the wrong > tree in Git. The errors are > > - Wrong branch point for branches/arraykit (an abandoned and deleted > branch that was never merged back to trunk) > > - The commits r4113, r4179, r4180 contain changes in wrong order > (the subsequent commits are OK). > > I'd be willing just to ignore these. Such small errors should not matter. > > *** > > So as far as I see, we would be ready to go with this. > > Hi Pauli, I gave it a quick spin and it looks good so far. The cloning was really fast, I like that ;) Is there any way to test out commiting? I didn't have permissions to push to the repository. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmccampbell at enthought.com Thu Sep 2 10:45:20 2010 From: jmccampbell at enthought.com (Jason McCampbell) Date: Thu, 2 Sep 2010 09:45:20 -0500 Subject: [Numpy-discussion] Github migration? In-Reply-To: References: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> Message-ID: On Wed, Sep 1, 2010 at 10:46 AM, Charles R Harris wrote: > > > On Tue, Aug 31, 2010 at 2:56 PM, Jason McCampbell < > jmccampbell at enthought.com> wrote: > >> Hi Chuck (and anyone else interested), >> >> I updated the refactoring page on the NumPy developer wiki (seems to be >> down or I'd paste in the link). It certainly isn't complete, but there are >> a lot more details about the data structures and memory handling and an >> outline of some additional topics that needs to be filled in. >> >> > Thanks Jason. How much of the core library can be used without any > reference counting? I was originally thinking that the base ufuncs would > just be functions accepting a pointer and a descriptor and handling memory > allocations and such would be at a higher level. That is to say, the object > oriented aspects of numpy would be removed from the bottom layers where they > just get in the way. > Hi Chuck. Unfortunately pretty much all of the main core object are reference counted. We had hoped to avoid this, but the issue is that many of the objects reference each other. For example, some functions create a "new" array, but that array may just be a view of another array. The same is true for the descriptor objects. One option was to push all memory management up into the interface, but that had the effect of requiring quite a few callbacks which makes the core a lot harder to use from standard C/C++ application. > Also, since many of the public macros expect the old type structures, what > is going to happen with them? They are really part of the API, but a > particularly troublesome part for going forward. > Are there any specific macros that are a particular problem? I do agree, I dislike macros in general and some have been simplified, but largely they are similar to what was there before. > > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmccampbell at enthought.com Thu Sep 2 10:51:14 2010 From: jmccampbell at enthought.com (Jason McCampbell) Date: Thu, 2 Sep 2010 09:51:14 -0500 Subject: [Numpy-discussion] Github migration? In-Reply-To: References: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> Message-ID: On Wed, Sep 1, 2010 at 9:07 PM, Charles R Harris wrote: > > Hi Jason, > > On Tue, Aug 31, 2010 at 2:56 PM, Jason McCampbell < > jmccampbell at enthought.com> wrote: > >> Hi Chuck (and anyone else interested), >> >> I updated the refactoring page on the NumPy developer wiki (seems to be >> down or I'd paste in the link). It certainly isn't complete, but there are >> a lot more details about the data structures and memory handling and an >> outline of some additional topics that needs to be filled in. >> >> > I note that there are some C++ style comments in the code which will cause > errors on some platforms, so I hope you are planning on removing them at > some point. Also, > Mostly the C++ comments are there for specific things we need to fix before it's complete (easier to search for). Likely a few are attributable to "muscle memory" in my fingers as well, but all will be removed as we button it up. > > if (yes) foo; > > is very bad style. There is a lot of that in old code like that that still > needs to be cleaned up, but I also see some in the new code. It would be > best to get it right to start with. > Agreed. In the code I have edited I typically re-write it as "if (NULL != yes) foo;" but a lot of code has been copied in wholesale and we haven't always updated that code. > > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Sep 2 11:25:20 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 2 Sep 2010 09:25:20 -0600 Subject: [Numpy-discussion] Github migration? In-Reply-To: References: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> Message-ID: On Thu, Sep 2, 2010 at 8:51 AM, Jason McCampbell wrote: > > > On Wed, Sep 1, 2010 at 9:07 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> Hi Jason, >> >> On Tue, Aug 31, 2010 at 2:56 PM, Jason McCampbell < >> jmccampbell at enthought.com> wrote: >> >>> Hi Chuck (and anyone else interested), >>> >>> I updated the refactoring page on the NumPy developer wiki (seems to be >>> down or I'd paste in the link). It certainly isn't complete, but there are >>> a lot more details about the data structures and memory handling and an >>> outline of some additional topics that needs to be filled in. >>> >>> >> I note that there are some C++ style comments in the code which will cause >> errors on some platforms, so I hope you are planning on removing them at >> some point. Also, >> > > Mostly the C++ comments are there for specific things we need to fix before > it's complete (easier to search for). Likely a few are attributable to > "muscle memory" in my fingers as well, but all will be removed as we button > it up. > > >> >> if (yes) foo; >> >> is very bad style. There is a lot of that in old code like that that still >> needs to be cleaned up, but I also see some in the new code. It would be >> best to get it right to start with. >> > > Agreed. In the code I have edited I typically re-write it as "if (NULL != > yes) foo;" but a lot of code has been copied in wholesale and we haven't > always updated that code. > > I mean it is bad style to have foo on the same line as the if. I think this happens because folks start off wanting to save a bit of vertical space and a couple of keystrokes, but in the long run it tends to make the code harder to read. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Thu Sep 2 12:06:23 2010 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 02 Sep 2010 09:06:23 -0700 Subject: [Numpy-discussion] Unexpected float96 precision loss In-Reply-To: References: <20100901162659.b80f6faf.michael.s.gilbert@gmail.com> <20100901173008.96868da3.michael.s.gilbert@gmail.com> Message-ID: <4C7FCB7F.2060307@noaa.gov> Charles R Harris wrote: >>> So if you write "float96(0.0001)", the result is not the float96 number >>> closest to 0.0001, but the 96-bit representation of the 64-bit number >>> closest to 0.0001. ... >> but wouldn't it be better to exactly handle strings since those can be >> converted exactly, which is what the user wants/expects? >> > Well, yes. But then we would need to write our own routines for the > conversions... The mechanism used in fromstring() and fromfile() (when used for text parsing) looks like it's designed to support this -- i.e. the actual string parsing is done be a function provided by the datatype. I haven't looked to see if there is anything special there for float96. Does the clib for a compiler that provides a float64 also provide an atof() function that supports it? Its seems that it should. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From jmccampbell at enthought.com Thu Sep 2 12:55:23 2010 From: jmccampbell at enthought.com (Jason McCampbell) Date: Thu, 2 Sep 2010 11:55:23 -0500 Subject: [Numpy-discussion] Github migration? In-Reply-To: References: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> Message-ID: On Thu, Sep 2, 2010 at 10:25 AM, Charles R Harris wrote: > > > On Thu, Sep 2, 2010 at 8:51 AM, Jason McCampbell < > jmccampbell at enthought.com> wrote: > >> >> >> On Wed, Sep 1, 2010 at 9:07 PM, Charles R Harris < >> charlesr.harris at gmail.com> wrote: >> >>> >>> Hi Jason, >>> >>> On Tue, Aug 31, 2010 at 2:56 PM, Jason McCampbell < >>> jmccampbell at enthought.com> wrote: >>> >>>> Hi Chuck (and anyone else interested), >>>> >>>> I updated the refactoring page on the NumPy developer wiki (seems to be >>>> down or I'd paste in the link). It certainly isn't complete, but there are >>>> a lot more details about the data structures and memory handling and an >>>> outline of some additional topics that needs to be filled in. >>>> >>>> >>> I note that there are some C++ style comments in the code which will >>> cause errors on some platforms, so I hope you are planning on removing them >>> at some point. Also, >>> >> >> Mostly the C++ comments are there for specific things we need to fix >> before it's complete (easier to search for). Likely a few are attributable >> to "muscle memory" in my fingers as well, but all will be removed as we >> button it up. >> >> >>> >>> if (yes) foo; >>> >>> is very bad style. There is a lot of that in old code like that that >>> still needs to be cleaned up, but I also see some in the new code. It would >>> be best to get it right to start with. >>> >> >> Agreed. In the code I have edited I typically re-write it as "if (NULL != >> yes) foo;" but a lot of code has been copied in wholesale and we haven't >> always updated that code. >> >> > > I mean it is bad style to have foo on the same line as the if. I think this > happens because folks start off wanting to save a bit of vertical space and > a couple of keystrokes, but in the long run it tends to make the code harder > to read. > Oh, that's interesting. I don't generally have an objection to 'foo' on the same line for simple statement as, like you said, it saves a lot of vertical space and the lack of curly's is more of an issue with: if (yes) foo; I try to avoid the C-ism of default conversion of pointers to a bool comparison. Jason -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Thu Sep 2 13:05:26 2010 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 2 Sep 2010 19:05:26 +0200 Subject: [Numpy-discussion] Equivalent Matlab function In-Reply-To: References: Message-ID: Thanks Joseph, I'll wrap this inside my code ;) Matthieu 2010/9/2 : > On Thu, Sep 2, 2010 at 3:56 AM, Matthieu Brucher > wrote: >> Hi, >> >> I'm looking for a Numpy equivalent of convmtx >> (http://www.mathworks.in/access/helpdesk/help/toolbox/signal/convmtx.html). >> Is there something inside Numpy directly? or perhaps Scipy? > > I haven't seen it in numpy or scipy, but I have seen something similar > in a PDE or FEM package. > > Using the toeplitz hint from the matlab help, the following seems to > work, at least in the examples > >>>> from scipy import linalg > >>>> h = np.array([1,3,1]) >>>> nc = 10; nr = nc-len(h)+1 >>>> linalg.toeplitz(np.r_[[1],np.zeros(nr-1)], np.r_[h,np.zeros(nc-len(h))]) > array([[ 1., ?3., ?1., ?0., ?0., ?0., ?0., ?0., ?0., ?0.], > ? ? ? [ 0., ?1., ?3., ?1., ?0., ?0., ?0., ?0., ?0., ?0.], > ? ? ? [ 0., ?0., ?1., ?3., ?1., ?0., ?0., ?0., ?0., ?0.], > ? ? ? [ 0., ?0., ?0., ?1., ?3., ?1., ?0., ?0., ?0., ?0.], > ? ? ? [ 0., ?0., ?0., ?0., ?1., ?3., ?1., ?0., ?0., ?0.], > ? ? ? [ 0., ?0., ?0., ?0., ?0., ?1., ?3., ?1., ?0., ?0.], > ? ? ? [ 0., ?0., ?0., ?0., ?0., ?0., ?1., ?3., ?1., ?0.], > ? ? ? [ 0., ?0., ?0., ?0., ?0., ?0., ?0., ?1., ?3., ?1.]]) > >>>> h = np.array([1,2,3,2,1]) >>>> nc = 10; nr = nc-len(h)+1 >>>> linalg.toeplitz(np.r_[[1],np.zeros(nr-1)], np.r_[h,np.zeros(nc-len(h))]) > array([[ 1., ?2., ?3., ?2., ?1., ?0., ?0., ?0., ?0., ?0.], > ? ? ? [ 0., ?1., ?2., ?3., ?2., ?1., ?0., ?0., ?0., ?0.], > ? ? ? [ 0., ?0., ?1., ?2., ?3., ?2., ?1., ?0., ?0., ?0.], > ? ? ? [ 0., ?0., ?0., ?1., ?2., ?3., ?2., ?1., ?0., ?0.], > ? ? ? [ 0., ?0., ?0., ?0., ?1., ?2., ?3., ?2., ?1., ?0.], > ? ? ? [ 0., ?0., ?0., ?0., ?0., ?1., ?2., ?3., ?2., ?1.]]) > > maybe worth adding to the other special matrix improvements that Warren did. > > Josef > >> >> Matthieu >> -- >> Information System Engineer, Ph.D. >> Blog: http://matt.eifelle.com >> LinkedIn: http://www.linkedin.com/in/matthieubrucher >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher From macdonald at maths.ox.ac.uk Thu Sep 2 13:13:23 2010 From: macdonald at maths.ox.ac.uk (Colin Macdonald) Date: Thu, 02 Sep 2010 18:13:23 +0100 Subject: [Numpy-discussion] Unexpected float96 precision loss In-Reply-To: <4C7FCB7F.2060307@noaa.gov> References: <20100901162659.b80f6faf.michael.s.gilbert@gmail.com> <20100901173008.96868da3.michael.s.gilbert@gmail.com> <4C7FCB7F.2060307@noaa.gov> Message-ID: <4C7FDB33.4050804@maths.ox.ac.uk> On 09/02/10 17:06, Christopher Barker wrote: > Does the clib for a compiler that provides a float64 also provide an > atof() function that supports it? Its seems that it should. I think so, for example in C I can do: fscanf(fp, "%Lf %Lf %Lf", &x, &y, &z); where x,y,z are "long doubles". The equivalent for regular doubles would be: fscanf(fp, "%lf %lf %lf", &x, &y, &z); From charlesr.harris at gmail.com Thu Sep 2 13:11:35 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 2 Sep 2010 11:11:35 -0600 Subject: [Numpy-discussion] Github migration? In-Reply-To: References: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> Message-ID: On Thu, Sep 2, 2010 at 10:55 AM, Jason McCampbell wrote: > > > On Thu, Sep 2, 2010 at 10:25 AM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> On Thu, Sep 2, 2010 at 8:51 AM, Jason McCampbell < >> jmccampbell at enthought.com> wrote: >> >>> >>> >>> On Wed, Sep 1, 2010 at 9:07 PM, Charles R Harris < >>> charlesr.harris at gmail.com> wrote: >>> >>>> >>>> Hi Jason, >>>> >>>> On Tue, Aug 31, 2010 at 2:56 PM, Jason McCampbell < >>>> jmccampbell at enthought.com> wrote: >>>> >>>>> Hi Chuck (and anyone else interested), >>>>> >>>>> I updated the refactoring page on the NumPy developer wiki (seems to be >>>>> down or I'd paste in the link). It certainly isn't complete, but there are >>>>> a lot more details about the data structures and memory handling and an >>>>> outline of some additional topics that needs to be filled in. >>>>> >>>>> >>>> I note that there are some C++ style comments in the code which will >>>> cause errors on some platforms, so I hope you are planning on removing them >>>> at some point. Also, >>>> >>> >>> Mostly the C++ comments are there for specific things we need to fix >>> before it's complete (easier to search for). Likely a few are attributable >>> to "muscle memory" in my fingers as well, but all will be removed as we >>> button it up. >>> >>> >>>> >>>> if (yes) foo; >>>> >>>> is very bad style. There is a lot of that in old code like that that >>>> still needs to be cleaned up, but I also see some in the new code. It would >>>> be best to get it right to start with. >>>> >>> >>> Agreed. In the code I have edited I typically re-write it as "if (NULL >>> != yes) foo;" but a lot of code has been copied in wholesale and we haven't >>> always updated that code. >>> >>> >> >> I mean it is bad style to have foo on the same line as the if. I think >> this happens because folks start off wanting to save a bit of vertical space >> and a couple of keystrokes, but in the long run it tends to make the code >> harder to read. >> > > Oh, that's interesting. I don't generally have an objection to 'foo' on > the same line for simple statement as, like you said, it saves a lot of > vertical space and the lack of curly's is more of an issue with: > if (yes) > foo; > > It's pretty universally a no-no, you won't find many (if any) style guides that permit it. Where they do differ is whether a single statement needs the curly braces and even there, if one block has multiple statements requiring the curlies, then all blocks get them. I think curly braces for all the blocks is reasonable as it adds a bit more regularity to the style. In any case, if you adhere to this standard I won't have to spend time fixing things up after you commit to mainline. Another no-no that I haven't noticed yet is to put more than one statement on a line. > I try to avoid the C-ism of default conversion of pointers to a bool > comparison. > > Yeah, that too ;) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Thu Sep 2 13:22:30 2010 From: ben.root at ou.edu (Benjamin Root) Date: Thu, 2 Sep 2010 12:22:30 -0500 Subject: [Numpy-discussion] Github migration? In-Reply-To: References: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> Message-ID: On Thu, Sep 2, 2010 at 12:11 PM, Charles R Harris wrote: > > > On Thu, Sep 2, 2010 at 10:55 AM, Jason McCampbell < > jmccampbell at enthought.com> wrote: > >> >> >> On Thu, Sep 2, 2010 at 10:25 AM, Charles R Harris < >> charlesr.harris at gmail.com> wrote: >> >>> >>> >>> On Thu, Sep 2, 2010 at 8:51 AM, Jason McCampbell < >>> jmccampbell at enthought.com> wrote: >>> >>>> >>>> >>>> On Wed, Sep 1, 2010 at 9:07 PM, Charles R Harris < >>>> charlesr.harris at gmail.com> wrote: >>>> >>>>> >>>>> Hi Jason, >>>>> >>>>> On Tue, Aug 31, 2010 at 2:56 PM, Jason McCampbell < >>>>> jmccampbell at enthought.com> wrote: >>>>> >>>>>> Hi Chuck (and anyone else interested), >>>>>> >>>>>> I updated the refactoring page on the NumPy developer wiki (seems to >>>>>> be down or I'd paste in the link). It certainly isn't complete, but there >>>>>> are a lot more details about the data structures and memory handling and an >>>>>> outline of some additional topics that needs to be filled in. >>>>>> >>>>>> >>>>> I note that there are some C++ style comments in the code which will >>>>> cause errors on some platforms, so I hope you are planning on removing them >>>>> at some point. Also, >>>>> >>>> >>>> Mostly the C++ comments are there for specific things we need to fix >>>> before it's complete (easier to search for). Likely a few are attributable >>>> to "muscle memory" in my fingers as well, but all will be removed as we >>>> button it up. >>>> >>>> >>>>> >>>>> if (yes) foo; >>>>> >>>>> is very bad style. There is a lot of that in old code like that that >>>>> still needs to be cleaned up, but I also see some in the new code. It would >>>>> be best to get it right to start with. >>>>> >>>> >>>> Agreed. In the code I have edited I typically re-write it as "if (NULL >>>> != yes) foo;" but a lot of code has been copied in wholesale and we haven't >>>> always updated that code. >>>> >>>> >>> >>> I mean it is bad style to have foo on the same line as the if. I think >>> this happens because folks start off wanting to save a bit of vertical space >>> and a couple of keystrokes, but in the long run it tends to make the code >>> harder to read. >>> >> >> Oh, that's interesting. I don't generally have an objection to 'foo' on >> the same line for simple statement as, like you said, it saves a lot of >> vertical space and the lack of curly's is more of an issue with: >> if (yes) >> foo; >> >> > It's pretty universally a no-no, you won't find many (if any) style guides > that permit it. Where they do differ is whether a single statement needs the > curly braces and even there, if one block has multiple statements requiring > the curlies, then all blocks get them. I think curly braces for all the > blocks is reasonable as it adds a bit more regularity to the style. In any > case, if you adhere to this standard I won't have to spend time fixing > things up after you commit to mainline. > > Another no-no that I haven't noticed yet is to put more than one statement > on a line. > > > >> I try to avoid the C-ism of default conversion of pointers to a bool >> comparison. >> >> > Yeah, that too ;) > > Chuck > > I don't know if you guys are familiar with 'indent'? I use it a lot to convert code that I get from other people into a style that I personally like. It was specifically designed for C, but it can handle C++ code. It can do things like force curly braces, or make all single statement ifs/while/for blocks be curly-less. It can enforce line widths and break up comments accordingly. I bet it can even convert C++ style comments into C style. The man page is quite long with many options to choose from. I have come across some other programs that were like 'indent' but designed for C++, but I can not remember the name of them. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Thu Sep 2 13:43:36 2010 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 2 Sep 2010 17:43:36 +0000 (UTC) Subject: [Numpy-discussion] Unexpected float96 precision loss References: <20100901162659.b80f6faf.michael.s.gilbert@gmail.com> <20100901173008.96868da3.michael.s.gilbert@gmail.com> <4C7FCB7F.2060307@noaa.gov> <4C7FDB33.4050804@maths.ox.ac.uk> Message-ID: Thu, 02 Sep 2010 18:13:23 +0100, Colin Macdonald wrote: > On 09/02/10 17:06, Christopher Barker wrote: >> Does the clib for a compiler that provides a float64 also provide an >> atof() function that supports it? Its seems that it should. > > I think so, for example in C I can do: > > fscanf(fp, "%Lf %Lf %Lf", &x, &y, &z); > > where x,y,z are "long doubles". The only question here is if that is portable, i.e., in the C89 standard. IIRC long doubles and the corresponding sscanf codes are in C89, so they should be safe. We'll just need to add long double versions of NumPyOS_ascii_strtod and NumPyOS_ftolf that call sscanf with the correct format string in the end. -- Pauli Virtanen From pav at iki.fi Thu Sep 2 13:51:19 2010 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 2 Sep 2010 17:51:19 +0000 (UTC) Subject: [Numpy-discussion] Github migration? References: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> Message-ID: Thu, 02 Sep 2010 08:27:18 -0600, Charles R Harris wrote: [clip] > Hi Pauli, I gave it a quick spin and it looks good so far. The cloning > was really fast, I like that ;) Is there any way to test out commiting? > I didn't have permissions to push to the repository. You should have push permissions now. Feel free to play around, the test repo will be dropped later on in any case. In Github you may need to go to your Dashboard, and from the right hand side select "Switch context", to see the correct push address. Pauli From Chris.Barker at noaa.gov Thu Sep 2 14:09:46 2010 From: Chris.Barker at noaa.gov (Chris Barker) Date: Thu, 02 Sep 2010 11:09:46 -0700 Subject: [Numpy-discussion] Unexpected float96 precision loss In-Reply-To: References: <20100901162659.b80f6faf.michael.s.gilbert@gmail.com> <20100901173008.96868da3.michael.s.gilbert@gmail.com> <4C7FCB7F.2060307@noaa.gov> <4C7FDB33.4050804@maths.ox.ac.uk> Message-ID: <4C7FE86A.7070208@noaa.gov> Pauli Virtanen wrote: > We'll just need to add long double versions of NumPyOS_ascii_strtod and > NumPyOS_ftolf that call sscanf with the correct format string in the end. hmm -- if you're going to do that, maybe we could re-factor and use sscanf everywhere instead of ato*() -- the ato* functions are a nightmare, and fromfile() and fromstring are riddled with bugs because of it. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From charlesr.harris at gmail.com Thu Sep 2 14:22:30 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 2 Sep 2010 12:22:30 -0600 Subject: [Numpy-discussion] Github migration? In-Reply-To: References: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> Message-ID: On Thu, Sep 2, 2010 at 11:51 AM, Pauli Virtanen wrote: > Thu, 02 Sep 2010 08:27:18 -0600, Charles R Harris wrote: > [clip] > > Hi Pauli, I gave it a quick spin and it looks good so far. The cloning > > was really fast, I like that ;) Is there any way to test out commiting? > > I didn't have permissions to push to the repository. > > You should have push permissions now. Feel free to play around, the test > repo will be dropped later on in any case. > > In Github you may need to go to your Dashboard, and from the right hand > side select "Switch context", to see the correct push address. > > I do have a question of how the maintenance branch shows up in the github network tool, that is, I expected to see the different release branches coming off the master branch whereas the all seem to be points along mainteance. However, that may just be an artifact of the network tool and I am not understanding what it is showing. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Thu Sep 2 15:16:31 2010 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 2 Sep 2010 19:16:31 +0000 (UTC) Subject: [Numpy-discussion] Github migration? References: <08D681AB-4D6E-44B3-80BD-75AC43313261@enthought.com> Message-ID: Thu, 02 Sep 2010 12:22:30 -0600, Charles R Harris wrote: [clip] > I do have a question of how the maintenance branch shows up in the > github network tool, that is, I expected to see the different release > branches coming off the master branch whereas the all seem to be points > along mainteance. However, that may just be an artifact of the network > tool and I am not understanding what it is showing. It looks correct to me -- I don't understand what the problem you are seeing is. (Anyway, it sounds Github-related.) What the network tool shows to me is that there is a master branch, and the release branches maintenance/1.x.y branch off it at some point in the history. For instance, maintenance/1.5.x branches off at July 7th 2010. Earlier in the history (e.g. around Feb 18th 2009) in the master branch there's more branching and merging. These are from topic branches, such as what branches/fix_float_format in SVN that got later on merged to trunk. Github seems to show these in different colors, but in reality they are just a part of the master branch history. So all in all, it looks correct to me. -- Pauli Virtanen From jkington at wisc.edu Thu Sep 2 15:50:08 2010 From: jkington at wisc.edu (Joe Kington) Date: Thu, 02 Sep 2010 14:50:08 -0500 Subject: [Numpy-discussion] Histogram does not preserve subclasses of ndarray (e.g. masked arrays) Message-ID: Hi all, I just wanted to check if this would be considered a bug. numpy.histogram does not appear to preserve subclasses of ndarrays (e.g. masked arrays). This leads to considerable problems when working with masked arrays. (As per this Stack Overflow question ) E.g. import numpy as np x = np.arange(100) x = np.ma.masked_where(x > 30, x) counts, bin_edges = np.histogram(x) yields: counts --> array([10, 10, 10, 10, 10, 10, 10, 10, 10, 10]) bin_edges --> array([ 0. , 9.9, 19.8, 29.7, 39.6, 49.5, 59.4, 69.3, 79.2, 89.1, 99. ]) I would have expected histogram to ignore the masked portion of the data. Is this a bug, or expected behavior? I'll open a bug report, if it's not expected behavior... This would appear to be easily fixed by using asanyarray rather than asarray within histogram. E.g. this diff for numpy/lib/function_base.py Index: function_base.py =================================================================== --- function_base.py (revision 8604) +++ function_base.py (working copy) @@ -132,9 +132,9 @@ """ - a = asarray(a) + a = asanyarray(a) if weights is not None: - weights = asarray(weights) + weights = asanyarray(weights) if np.any(weights.shape != a.shape): raise ValueError( 'weights should have the same shape as a.') @@ -156,7 +156,7 @@ mx += 0.5 bins = linspace(mn, mx, bins+1, endpoint=True) else: - bins = asarray(bins) + bins = asanyarray(bins) if (np.diff(bins) < 0).any(): raise AttributeError( 'bins must increase monotonically.') Thanks! -Joe -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Thu Sep 2 16:47:57 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 02 Sep 2010 15:47:57 -0500 Subject: [Numpy-discussion] Histogram does not preserve subclasses of ndarray (e.g. masked arrays) In-Reply-To: References: Message-ID: <4C800D7D.7@gmail.com> On 09/02/2010 02:50 PM, Joe Kington wrote: > Hi all, > > I just wanted to check if this would be considered a bug. > > numpy.histogram does not appear to preserve subclasses of ndarrays > (e.g. masked arrays). This leads to considerable problems when > working with masked arrays. (As per this Stack Overflow question > ) > > E.g. > > import numpy as np > x = np.arange(100) > x = np.ma.masked_where(x > 30, x) > > counts, bin_edges = np.histogram(x) > > yields: > counts --> array([10, 10, 10, 10, 10, 10, 10, 10, 10, 10]) > bin_edges --> array([ 0. , 9.9, 19.8, 29.7, 39.6, 49.5, 59.4, > 69.3, 79.2, 89.1, 99. ]) > > I would have expected histogram to ignore the masked portion of the > data. Is this a bug, or expected behavior? I'll open a bug report, > if it's not expected behavior... > > This would appear to be easily fixed by using asanyarray rather than > asarray within histogram. E.g. this diff for numpy/lib/function_base.py > Index: function_base.py > =================================================================== > --- function_base.py (revision 8604) > +++ function_base.py (working copy) > @@ -132,9 +132,9 @@ > > """ > > - a = asarray(a) > + a = asanyarray(a) > if weights is not None: > - weights = asarray(weights) > + weights = asanyarray(weights) > if np.any(weights.shape != a.shape): > raise ValueError( > 'weights should have the same shape as a.') > @@ -156,7 +156,7 @@ > mx += 0.5 > bins = linspace(mn, mx, bins+1, endpoint=True) > else: > - bins = asarray(bins) > + bins = asanyarray(bins) > if (np.diff(bins) < 0).any(): > raise AttributeError( > 'bins must increase monotonically.') > > Thanks! > -Joe > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion I would not call it a bug as this a known 'feature' of functions that use np.asarray(). You are welcome to file a enhancement bug but there are some issues that need to be addressed. Typical questions that come to mind are: 1) Should a user be warned that the input is a masked array? 2) Should histogram count the number of masked values? 3) What is the expected output when normed=True? 4) What type of array should be the weights and bin arguments? 5) What is the dimensions of the weight and bin arguments since it only needs to have the number of bins? 6) If the input array is masked should the weight and bins arguments also be masked arrays when applicable? If so, what happens if the masks are in different locations between arrays? Regards Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From neilcrighton at gmail.com Thu Sep 2 18:02:07 2010 From: neilcrighton at gmail.com (Neil Crighton) Date: Thu, 2 Sep 2010 22:02:07 +0000 (UTC) Subject: [Numpy-discussion] Boolean arrays References: Message-ID: > > Ideally, I would like in1d to always be the right answer to this problem. It > should be easy to put in an if statement to switch to a kern_in()-type function > in the case of large ar1 but small ar2. I will do some timing tests and make a > patch. > I uploaded a timing test and a patch to arraysetops.py here: http://projects.scipy.org/numpy/ticket/1603 The new in1d() uses the kern_in algorithm when it's faster, and the existing algorithm otherwise. The speedup compared to the old in1d() for cases with very large ar1 and small ar2 can be up to 10x on my laptop. If someone with commit access could take a look and and apply it if ok, that would be great. Thanks, Neil From josef.pktd at gmail.com Thu Sep 2 18:31:14 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 2 Sep 2010 18:31:14 -0400 Subject: [Numpy-discussion] Histogram does not preserve subclasses of ndarray (e.g. masked arrays) In-Reply-To: References: Message-ID: On Thu, Sep 2, 2010 at 3:50 PM, Joe Kington wrote: > Hi all, > > I just wanted to check if this would be considered a bug. > > numpy.histogram does not appear to preserve subclasses of ndarrays (e.g. > masked arrays).? This leads to considerable problems when working with > masked arrays. (As per this Stack Overflow question) > > E.g. > > import numpy as np > x = np.arange(100) > x = np.ma.masked_where(x > 30, x) > > counts, bin_edges = np.histogram(x) > > yields: > counts --> array([10, 10, 10, 10, 10, 10, 10, 10, 10, 10]) > bin_edges --> array([? 0. ,?? 9.9,? 19.8,? 29.7,? 39.6,? 49.5,? 59.4, > 69.3,? 79.2, 89.1,? 99. ]) > > I would have expected histogram to ignore the masked portion of the data. > Is this a bug, or expected behavior?? I'll open a bug report, if it's not > expected behavior... If you want to ignore masked data it's just on extra function call histogram(m_arr.compressed()) I don't think the fact that this makes an extra copy will be relevant, because I guess full masked array handling inside histogram will be a lot more expensive. Using asanyarray would also allow matrices in and other subtypes that might not be handled correctly by the histogram calculations. For anything else besides dropping masked observations, it would be necessary to figure out what the masked array definition of a histogram is, as Bruce pointed out. (Another interesting question would be if histogram handles nans correctly, searchsorted ???) Josef > > This would appear to be easily fixed by using asanyarray rather than asarray > within histogram.? E.g. this diff for numpy/lib/function_base.py > Index: function_base.py > =================================================================== > --- function_base.py??? (revision 8604) > +++ function_base.py??? (working copy) > @@ -132,9 +132,9 @@ > > ???? """ > > -??? a = asarray(a) > +??? a = asanyarray(a) > ???? if weights is not None: > -??????? weights = asarray(weights) > +??????? weights = asanyarray(weights) > ???????? if np.any(weights.shape != a.shape): > ???????????? raise ValueError( > ???????????????????? 'weights should have the same shape as a.') > @@ -156,7 +156,7 @@ > ???????????? mx += 0.5 > ???????? bins = linspace(mn, mx, bins+1, endpoint=True) > ???? else: > -??????? bins = asarray(bins) > +??????? bins = asanyarray(bins) > ???????? if (np.diff(bins) < 0).any(): > ???????????? raise AttributeError( > ???????????????????? 'bins must increase monotonically.') > > Thanks! > -Joe > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From jkington at wisc.edu Thu Sep 2 19:33:47 2010 From: jkington at wisc.edu (Joe Kington) Date: Thu, 02 Sep 2010 18:33:47 -0500 Subject: [Numpy-discussion] Histogram does not preserve subclasses of ndarray (e.g. masked arrays) In-Reply-To: References: Message-ID: On Thu, Sep 2, 2010 at 5:31 PM, wrote: > On Thu, Sep 2, 2010 at 3:50 PM, Joe Kington wrote: > > Hi all, > > > > I just wanted to check if this would be considered a bug. > > > > numpy.histogram does not appear to preserve subclasses of ndarrays (e.g. > > masked arrays). This leads to considerable problems when working with > > masked arrays. (As per this Stack Overflow question) > > > > E.g. > > > > import numpy as np > > x = np.arange(100) > > x = np.ma.masked_where(x > 30, x) > > > > counts, bin_edges = np.histogram(x) > > > > yields: > > counts --> array([10, 10, 10, 10, 10, 10, 10, 10, 10, 10]) > > bin_edges --> array([ 0. , 9.9, 19.8, 29.7, 39.6, 49.5, 59.4, > > 69.3, 79.2, 89.1, 99. ]) > > > > I would have expected histogram to ignore the masked portion of the data. > > Is this a bug, or expected behavior? I'll open a bug report, if it's not > > expected behavior... > > If you want to ignore masked data it's just on extra function call > > histogram(m_arr.compressed()) > > I don't think the fact that this makes an extra copy will be relevant, > because I guess full masked array handling inside histogram will be a > lot more expensive. > > Using asanyarray would also allow matrices in and other subtypes that > might not be handled correctly by the histogram calculations. > > For anything else besides dropping masked observations, it would be > necessary to figure out what the masked array definition of a > histogram is, as Bruce pointed out. > > (Another interesting question would be if histogram handles nans > correctly, searchsorted ???) > > Josef > Good points all around. I'll skip the enhancement request. Sorry for the noise! Thanks! -Joe > > > > > This would appear to be easily fixed by using asanyarray rather than > asarray > > within histogram. E.g. this diff for numpy/lib/function_base.py > > Index: function_base.py > > =================================================================== > > --- function_base.py (revision 8604) > > +++ function_base.py (working copy) > > @@ -132,9 +132,9 @@ > > > > """ > > > > - a = asarray(a) > > + a = asanyarray(a) > > if weights is not None: > > - weights = asarray(weights) > > + weights = asanyarray(weights) > > if np.any(weights.shape != a.shape): > > raise ValueError( > > 'weights should have the same shape as a.') > > @@ -156,7 +156,7 @@ > > mx += 0.5 > > bins = linspace(mn, mx, bins+1, endpoint=True) > > else: > > - bins = asarray(bins) > > + bins = asanyarray(bins) > > if (np.diff(bins) < 0).any(): > > raise AttributeError( > > 'bins must increase monotonically.') > > > > Thanks! > > -Joe > > > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From asmund.hjulstad at gmail.com Fri Sep 3 01:53:00 2010 From: asmund.hjulstad at gmail.com (=?ISO-8859-1?Q?=C5smund_Hjulstad?=) Date: Fri, 3 Sep 2010 08:53:00 +0300 Subject: [Numpy-discussion] PyGTK breaks fortran extension??? Message-ID: Hi, all, I have a f2py wrapped fortran extension, compiled using gcc-mingw32 (v.4.5.0), numpy 1.5, Python 2.7, where I am experiencing the strangest behaviour. It appears that loading pygtk breaks my fortran extension. The fortran code has initialization code (it calculates reference state for a combination of fluids). The initialization appearantly has some sanity checks, verifying the results. This initialization fails if gtk is loaded first. import refprop from gtk import _gtk # if this line is commented out the next line works as expected ierr,herr = refprop.setup0(2,"METHANE.FLD|ETHANE.FLD","HMX.BNC","DEF") I initially started suspecting matplotlib, but as the problem only appeared when loading a gtk backend, I narrowed it down to this particular import. The following imports break it in just the same way (with, I suspect, the gtk import still being the culprit). import matplotlib matplotlib.use('GTK') import matplotlib.pyplot I have little idea how to debug this further, and probably will just choose to use a different backend. Has anybody else encountered something like this? Is is possible that msvcrt.dll has something to do with this? I haven't been able to get my library to link to msvcr90, I guess I have to do something with a manifest, but I haven't grasped it yet. Before using the gtk backend of matplotlib, everything worked OK, so I thought it wasn't a problem. I have a VS2008PRO DVD on the way in the mail, but was hoping to be able to fix this with mingw. Best regards, ?smund Hjulstad, asmund at hjulstad.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri Sep 3 04:22:20 2010 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 3 Sep 2010 08:22:20 +0000 (UTC) Subject: [Numpy-discussion] PyGTK breaks fortran extension??? References: Message-ID: Fri, 03 Sep 2010 08:53:00 +0300, ?smund Hjulstad wrote: > I have a f2py wrapped fortran extension, compiled using gcc-mingw32 > (v.4.5.0), numpy 1.5, Python 2.7, where I am experiencing the strangest > behaviour. It appears that loading pygtk breaks my fortran extension. Be aware that "import gtk" sets the current locale. You can reset it with "import locale; locale.setlocale(locale.LC_ALL, 'C')". Depending on the platform, the locale might for example alter how your Fortran runtime parses floating point numbers (i.e., whether the decimal separator is "," or "."). For instance, in some old versions of gfortran this *was* a problem -- changing the C locale would break the Fortran runtime, making it stop the program with some obscure assertion. > The fortran code has initialization code (it calculates reference state > for a combination of fluids). The initialization appearantly has some > sanity checks, verifying the results. This initialization fails if gtk > is loaded first. How does it fail? Knowing this could help debugging. [clip] > Is is possible that msvcrt.dll has something to do with this? I haven't > been able to get my library to link to msvcr90, I guess I have to do > something with a manifest, but I haven't grasped it yet. Before using > the gtk backend of matplotlib, everything worked OK, so I thought it > wasn't a problem. I have a VS2008PRO DVD on the way in the mail, but was > hoping to be able to fix this with mingw. Mixing different runtimes could maybe also be a problem. I'm not an expert on this, however. -- Pauli Virtanen From rpmuller at gmail.com Fri Sep 3 07:02:47 2010 From: rpmuller at gmail.com (Rick Muller) Date: Fri, 3 Sep 2010 05:02:47 -0600 Subject: [Numpy-discussion] Help with the tensordot function? Message-ID: Can someone help me replace a slow expression with a faster one based on tensordot? I've read the documentation and I'm still confused. I have two matrices b and d. b is n x m and d is m x m. I want to replace the expression >>> bdb = zeros(n,'d') >>> for i in xrange(n): >>> bdb[i,:] = dot(b[i,:],dot(d,b[i,:]) with something that doesn't have the for loop and thus is a bit faster. The first step is >>> bd = dot(b,d) However, following this with >>> bdb = dot(bd,b.T) doesn't work, since it yields a n x n matrix instead of an n x 1 vector. Reading the notes on tensordot makes me think it's the function to use, but I'm having trouble grokking the axes argument. Can anyone help? Thanks in advance! -- Rick Muller rpmuller at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rpmuller at gmail.com Fri Sep 3 07:48:31 2010 From: rpmuller at gmail.com (Rick Muller) Date: Fri, 3 Sep 2010 05:48:31 -0600 Subject: [Numpy-discussion] Simplified question on tensordot Message-ID: Sorry for the rapid repost, but I thought of a much easier way to ask the question I asked a few minutes ago. I have two matrices, A and B, both of which are n x m. n is big (~10,000), and m is small (~10). I want to take the product AB such that I get a length-n vector, as in: >>> AB = zeros(n,'d') >>> for i in xrange(n): >>> AB[i] = dot(A[i,:],B[i,:]) only I don't want the for-loop. The various things I've tried with dot and tensordot inevitably give me a n x n matrix, which isn't what I want. Can anyone help me find an easier way to do this? -- Rick Muller rpmuller at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri Sep 3 07:57:22 2010 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 3 Sep 2010 11:57:22 +0000 (UTC) Subject: [Numpy-discussion] Simplified question on tensordot References: Message-ID: Fri, 03 Sep 2010 05:48:31 -0600, Rick Muller wrote: [clip] > I want to take the product AB such that I get a length-n vector, as in: > > >>> AB = zeros(n,'d') > >>> for i in xrange(n): > >>> AB[i] = dot(A[i,:],B[i,:]) >>> AB = np.sum(A*B, axis=1) It does create an intermediate (n, d) matrix, however. If this is a problem because of memory issues, you can try numexpr [1], >>> numexpr.evaluate("sum(A*B, axis=1)") .. [1] http://code.google.com/p/numexpr/ -- Pauli Virtanen From josef.pktd at gmail.com Fri Sep 3 07:59:24 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 3 Sep 2010 07:59:24 -0400 Subject: [Numpy-discussion] Simplified question on tensordot In-Reply-To: References: Message-ID: On Fri, Sep 3, 2010 at 7:48 AM, Rick Muller wrote: > Sorry for the rapid repost, but I thought of a much easier way to ask the > question I asked a few minutes ago. > > I have two matrices, A and B, both of which are n x m. n is big (~10,000), > and m is small (~10). > > I want to take the product AB such that I get a length-n vector, as in: > >>>> AB = zeros(n,'d') >>>> for i in xrange(n): >>>>??? AB[i] = dot(A[i,:],B[i,:]) Isn't this just (A*B).sum(1) ? Josef > > only I don't want the for-loop. The various things I've tried with dot and > tensordot inevitably give me a n x n matrix, which isn't what I want. Can > anyone help me find an easier way to do this? > > -- > Rick Muller > rpmuller at gmail.com > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From rpmuller at gmail.com Fri Sep 3 08:28:19 2010 From: rpmuller at gmail.com (Rick Muller) Date: Fri, 3 Sep 2010 06:28:19 -0600 Subject: [Numpy-discussion] Simplified question on tensordot In-Reply-To: References: Message-ID: Josef and Pauli, Wow, you guys rock! I'm amazed you could pull that out so quickly. I thank you, and PyQuante thanks you (hopefully this will make for faster density functional theory grids). Rick On Fri, Sep 3, 2010 at 5:59 AM, wrote: > On Fri, Sep 3, 2010 at 7:48 AM, Rick Muller wrote: > > Sorry for the rapid repost, but I thought of a much easier way to ask the > > question I asked a few minutes ago. > > > > I have two matrices, A and B, both of which are n x m. n is big > (~10,000), > > and m is small (~10). > > > > I want to take the product AB such that I get a length-n vector, as in: > > > >>>> AB = zeros(n,'d') > >>>> for i in xrange(n): > >>>> AB[i] = dot(A[i,:],B[i,:]) > > Isn't this just (A*B).sum(1) ? > > Josef > > > > > only I don't want the for-loop. The various things I've tried with dot > and > > tensordot inevitably give me a n x n matrix, which isn't what I want. Can > > anyone help me find an easier way to do this? > > > > -- > > Rick Muller > > rpmuller at gmail.com > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Rick Muller rpmuller at gmail.com 505-750-7557 -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.c.cherel at gmail.com Fri Sep 3 09:32:02 2010 From: guillaume.c.cherel at gmail.com (=?ISO-8859-1?Q?Guillaume_Ch=E9rel?=) Date: Fri, 03 Sep 2010 15:32:02 +0200 Subject: [Numpy-discussion] recarray to csv Message-ID: <4C80F8D2.5020606@gmail.com> Hello, I'd like to know if there is a convenient routine to write recarrays into cvs files, with the first line of the file being the name of the fields. Thanks, Guillaume From pgmdevlist at gmail.com Fri Sep 3 09:35:20 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 3 Sep 2010 15:35:20 +0200 Subject: [Numpy-discussion] recarray to csv In-Reply-To: <4C80F8D2.5020606@gmail.com> References: <4C80F8D2.5020606@gmail.com> Message-ID: <422D1E15-7910-433E-BA29-2E6B7358C080@gmail.com> On Sep 3, 2010, at 3:32 PM, Guillaume Ch?rel wrote: > Hello, > > I'd like to know if there is a convenient routine to write recarrays > into cvs files, with the first line of the file being the name of the > fields. matplotlib.mlab.rec2csv From guillaume.c.cherel at gmail.com Fri Sep 3 09:44:30 2010 From: guillaume.c.cherel at gmail.com (=?ISO-8859-1?Q?Guillaume_Ch=E9rel?=) Date: Fri, 03 Sep 2010 15:44:30 +0200 Subject: [Numpy-discussion] recarray to csv In-Reply-To: <422D1E15-7910-433E-BA29-2E6B7358C080@gmail.com> References: <4C80F8D2.5020606@gmail.com> <422D1E15-7910-433E-BA29-2E6B7358C080@gmail.com> Message-ID: <4C80FBBE.9070602@gmail.com> Great, Thank you. I also found out about csv2rec. I've been missing these two a lot. Le 03/09/2010 15:35, Pierre GM a ?crit : > On Sep 3, 2010, at 3:32 PM, Guillaume Ch?rel wrote: > >> Hello, >> >> I'd like to know if there is a convenient routine to write recarrays >> into cvs files, with the first line of the file being the name of the >> fields. > matplotlib.mlab.rec2csv > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From ben.root at ou.edu Fri Sep 3 09:50:04 2010 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 3 Sep 2010 08:50:04 -0500 Subject: [Numpy-discussion] recarray to csv In-Reply-To: <422D1E15-7910-433E-BA29-2E6B7358C080@gmail.com> References: <4C80F8D2.5020606@gmail.com> <422D1E15-7910-433E-BA29-2E6B7358C080@gmail.com> Message-ID: On Fri, Sep 3, 2010 at 8:35 AM, Pierre GM wrote: > > On Sep 3, 2010, at 3:32 PM, Guillaume Ch?rel wrote: > > > Hello, > > > > I'd like to know if there is a convenient routine to write recarrays > > into cvs files, with the first line of the file being the name of the > > fields. > > matplotlib.mlab.rec2csv > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > Why is this function in matplotlib? Wouldn't it be more useful in numpy? Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From erin.sheldon at gmail.com Fri Sep 3 09:56:25 2010 From: erin.sheldon at gmail.com (Erin Sheldon) Date: Fri, 03 Sep 2010 09:56:25 -0400 Subject: [Numpy-discussion] recarray to csv In-Reply-To: <4C80F8D2.5020606@gmail.com> References: <4C80F8D2.5020606@gmail.com> Message-ID: <1283521258-sup-1263@theshire> Excerpts from Guillaume Ch?rel's message of Fri Sep 03 09:32:02 -0400 2010: > Hello, > > I'd like to know if there is a convenient routine to write recarrays > into cvs files, with the first line of the file being the name of the > fields. > > Thanks, > Guillaume Yes, you can do this with the recfile package: http://pypi.python.org/pypi/recfile/0.40 http://code.google.com/p/recfile/ import numpy import recfile rec = numpy.zeros(3, dtype=[('x','f4'),('y','i8')]) rec['x'] = [0,1,2] rec['y'] = [0,1,2] recf = recfile.Open('filename.csv', 'w', delim=',') names = rec.dtype.names header = ','.join(names) recf.fobj.write(header+'\n') recf.write(rec) recf.close() Erin Scott Sheldon Brookhaven National Laboratory From jdh2358 at gmail.com Fri Sep 3 10:09:17 2010 From: jdh2358 at gmail.com (John Hunter) Date: Fri, 3 Sep 2010 09:09:17 -0500 Subject: [Numpy-discussion] recarray to csv In-Reply-To: <4C80FBBE.9070602@gmail.com> References: <4C80F8D2.5020606@gmail.com> <422D1E15-7910-433E-BA29-2E6B7358C080@gmail.com> <4C80FBBE.9070602@gmail.com> Message-ID: 2010/9/3 Guillaume Ch?rel : > ?Great, Thank you. I also found out about csv2rec. I've been missing > these two a lot. Some other handy rec functions in mlab http://matplotlib.sourceforge.net/examples/misc/rec_groupby_demo.html http://matplotlib.sourceforge.net/examples/misc/rec_join_demo.html JDH From jdh2358 at gmail.com Fri Sep 3 10:19:39 2010 From: jdh2358 at gmail.com (John Hunter) Date: Fri, 3 Sep 2010 09:19:39 -0500 Subject: [Numpy-discussion] recarray to csv In-Reply-To: References: <4C80F8D2.5020606@gmail.com> <422D1E15-7910-433E-BA29-2E6B7358C080@gmail.com> Message-ID: On Fri, Sep 3, 2010 at 8:50 AM, Benjamin Root wrote: > Why is this function in matplotlib?? Wouldn't it be more useful in numpy? I tend to add stuff I write to matplotlib. mlab was initially a repository of matlab-like functions that were not available in numpy (load, save, linspace, psd, cohere, polyfit, polyval, prctile, ...). I've always encouraged numpy developers to harvest what they want and move them into numpy, and many of these functions have been moved. Once they make it into stable numpy, we deprecate them and eventually remove them from mlab. Many of the rec functions have been ported to numpy in numpy.lib.recfunctions. There are some differences, particularly in the csv2rec (mpl handles date parsing) and I rely heavily on all these functions, so I have not ported all my code to use numpy yet. We should start the process of deprecating the ones that have been ported and have API and functional compatibility. JDH From rpmuller at gmail.com Fri Sep 3 12:39:13 2010 From: rpmuller at gmail.com (Rick Muller) Date: Fri, 3 Sep 2010 10:39:13 -0600 Subject: [Numpy-discussion] Cutoff small values of a vector to stop underflow Message-ID: There just *has* to be a better way of doing this. I want to cut off small values of a vector, and I'm currently doing something like: >>> for i in xrange(n): >>> if abs(A[i]) < tol: A[i] = 0 Which is slow, since A can be really long. Is there a way to write a Ufunc that would do something like this more directly? The naive things I've tried didn't work, e.g. >>> def cut(a,tol=1e-10): >>> if less(absolute(a),tol): >>> a = 0 >>> return -- Rick Muller rpmuller at gmail.com 505-750-7557 -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Fri Sep 3 12:42:22 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 3 Sep 2010 09:42:22 -0700 Subject: [Numpy-discussion] Cutoff small values of a vector to stop underflow In-Reply-To: References: Message-ID: On Fri, Sep 3, 2010 at 9:39 AM, Rick Muller wrote: > There just *has* to be a better way of doing this. I want to cut off small > values of a vector, and I'm currently doing something like: > >>>> for i in xrange(n): >>>>??? if abs(A[i]) < tol: A[i] = 0 > > Which is slow, since A can be really long. A[np.absolute(A) < tol] = 0 From rpmuller at gmail.com Fri Sep 3 12:49:21 2010 From: rpmuller at gmail.com (Rick Muller) Date: Fri, 3 Sep 2010 10:49:21 -0600 Subject: [Numpy-discussion] Cutoff small values of a vector to stop underflow In-Reply-To: References: Message-ID: Sweet! Guess I need to learn more about numpy indexing: this is pretty powerful. On Fri, Sep 3, 2010 at 10:42 AM, Keith Goodman wrote: > On Fri, Sep 3, 2010 at 9:39 AM, Rick Muller wrote: > > There just *has* to be a better way of doing this. I want to cut off > small > > values of a vector, and I'm currently doing something like: > > > >>>> for i in xrange(n): > >>>> if abs(A[i]) < tol: A[i] = 0 > > > > Which is slow, since A can be really long. > > A[np.absolute(A) < tol] = 0 > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Rick Muller rpmuller at gmail.com 505-750-7557 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Fri Sep 3 16:50:16 2010 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 3 Sep 2010 15:50:16 -0500 Subject: [Numpy-discussion] ndarray**maskedArray with broadcasting bug Message-ID: Here is a fun one... import numpy as np a_2d = np.random.random((3, 5)) b_1d = np.random.random(5) b_2d = np.vstack((b_1d, b_1d, b_1d)) a_ma_2d = np.ma.masked_array(a_2d, mask=(numpy.random.random((3, 5)) < 0.25)) b_ma_1d = np.ma.masked_array(b_1d, mask=(numpy.random.random(5) < 0.25)) b_ma_2d = np.ma.masked_array(b_2d, mask=(numpy.random.random((3, 5)) < 0.25)) # a**b without broadcasting (works) print a_2d**b_2d # a**b with broadcasting (works) print a_2d**b_1d # a_ma**b_ma without broadcasting (works) print a_ma_2d**b_ma_2d # a_ma**b_ma with broadcasting (works) print a_ma_2d**b_ma_1d # a**b_ma without broadcasting (works) print a_2d**b_ma_2d # a**b_ma with broadcasting (FAILS) print a_2d**b_ma_1d Traceback (most recent call last): File "", line 1, in File "/home/bvr/Programs/numpy/numpy/ma/core.py", line 3697, in __rpow__ return power(other, self) File "/home/bvr/Programs/numpy/numpy/ma/core.py", line 6043, in power m |= invalid ValueError: invalid return array shape # Now, test broadcasting of the other side of the operator c = np.random.random(5) c_ma = np.ma.masked_array(c, mask=(np.random.random(10) < 0.25)) # c**b_ma with broadcasting (works) print c**b_ma_2d # c_ma**b with broadcasting (FAILS) print c_ma**b_2d Traceback (most recent call last): File "", line 1, in File "/home/bvr/Programs/numpy/numpy/ma/core.py", line 3693, in __pow__ return power(self, other) File "/home/bvr/Programs/numpy/numpy/ma/core.py", line 6043, in power m |= invalid ValueError: invalid return array shape So, this fails if we broadcast the masked array portion of a power expression and the other value is a ndarray. Note, the same also occurs if we broadcast a masked array in order to raise it to powers indicated by an ndarray. Took me the longest time to nail this one down... Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsalvati at u.washington.edu Fri Sep 3 22:35:39 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Fri, 3 Sep 2010 19:35:39 -0700 Subject: [Numpy-discussion] Add function for creating recarray from database query? Message-ID: Hello, I recently had to get data from a mysql database into a recarray. The result was not very long but nontrivial to figure out: def recarray_from_db(db, command): """ executes a command and turns the results into a numpy recarray (record array)""" cursor = db.cursor() cursor.execute(command) column_dtypes = [(col[0], _convert(col[1])) for col in cursor.description] return np.fromiter((tuple (row) for row in cursor), dtype=column_dtypes, count = cursor.rowcount) _type_conversions = {decimal.Decimal : float} def _convert(type): try : return _type_conversions[type] except: return type It uses only the Python database API. Would something like this be a useful addition to numpy (maybe have it take a cursor object instead of a connection object)? I also found the following function useful for results with timedate columns, since numpy does not yet have a datetime dtype. def time_column(date, column_name): """ makes a string for calculating a time as (decimal) number of days since a date for a MySQL column. This is because the numpy.datetime datatype is not well developed.""" return "TIME_TO_SEC(timediff(" + column_name + ",'" + str(datetime.strptime(date, date_format)) + "'))/(60*60*24) as " + column_name date_format = '%Y-%m-%d' Best Regards, John -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Sat Sep 4 08:38:35 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Sat, 4 Sep 2010 14:38:35 +0200 Subject: [Numpy-discussion] Add function for creating recarray from database query? In-Reply-To: References: Message-ID: <71F951DC-9E16-4FC4-A7E7-364AA1BDA4EE@gmail.com> On Sep 4, 2010, at 4:35 AM, John Salvatier wrote: > Hello, > > I recently had to get data from a mysql database into a recarray. The result was not very long but nontrivial to figure out: > > def recarray_from_db(db, command): > """ executes a command and turns the results into a numpy recarray (record array)""" > cursor = db.cursor() > cursor.execute(command) > > column_dtypes = [(col[0], _convert(col[1])) for col in cursor.description] > > return np.fromiter((tuple (row) for row in cursor), dtype=column_dtypes, count = cursor.rowcount) > > _type_conversions = {decimal.Decimal : float} > def _convert(type): > try : > return _type_conversions[type] > except: > return type > > > It uses only the Python database API. Would something like this be a useful addition to numpy (maybe have it take a cursor object instead of a connection object)? > > I also found the following function useful for results with timedate columns, since numpy does not yet have a datetime dtype. > > def time_column(date, column_name): > """ makes a string for calculating a time as (decimal) number of days since a date for a MySQL column. This is because the numpy.datetime datatype is not well developed.""" > return "TIME_TO_SEC(timediff(" + column_name + ",'" + str(datetime.strptime(date, date_format)) + "'))/(60*60*24) as " + column_name > date_format = '%Y-%m-%d' FAIW, I have some pieces of code to connect ndarrays, MaskedArays and TimeSeries to sql3 databases (read from and write to). http://projects.scipy.org/scikits/browser/branches/pierregm/hydroclimpy/scikits/hydroclimpy/io/sqlite.py From eadrogue at gmx.net Sun Sep 5 15:12:34 2010 From: eadrogue at gmx.net (Ernest =?iso-8859-1?Q?Adrogu=E9?=) Date: Sun, 5 Sep 2010 21:12:34 +0200 Subject: [Numpy-discussion] test if two arrays share the same data Message-ID: <20100905191234.GA19392@doriath.local> Hi, How can it be done? id() doesn't do it: In [238]: a= np.arange(5) In [239]: id(a) == id(a[:]) Out[239]: False Any ideas? Ernest From punchagan at gmail.com Sun Sep 5 15:23:56 2010 From: punchagan at gmail.com (Puneeth) Date: Mon, 6 Sep 2010 00:53:56 +0530 Subject: [Numpy-discussion] test if two arrays share the same data In-Reply-To: <20100905191234.GA19392@doriath.local> References: <20100905191234.GA19392@doriath.local> Message-ID: 2010/9/6 Ernest Adrogu? : > Hi, > > How can it be done? > > id() doesn't do it: > > In [238]: a= np.arange(5) > > In [239]: id(a) == id(a[:]) > Out[239]: False > > Any ideas? Checking for a[:].base and a[:].size works. I'm not sure if that is a good way to do it. (a[:].base is a and a[:].size == a) -- Puneeth From gael.varoquaux at normalesup.org Sun Sep 5 15:25:17 2010 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 5 Sep 2010 21:25:17 +0200 Subject: [Numpy-discussion] test if two arrays share the same data In-Reply-To: <20100905191234.GA19392@doriath.local> References: <20100905191234.GA19392@doriath.local> Message-ID: <20100905192517.GC27337@phare.normalesup.org> On Sun, Sep 05, 2010 at 09:12:34PM +0200, Ernest Adrogu? wrote: > Hi, > How can it be done? np.may_share_memory From eadrogue at gmx.net Sun Sep 5 16:42:32 2010 From: eadrogue at gmx.net (Ernest =?iso-8859-1?Q?Adrogu=E9?=) Date: Sun, 5 Sep 2010 22:42:32 +0200 Subject: [Numpy-discussion] test if two arrays share the same data In-Reply-To: <20100905192517.GC27337@phare.normalesup.org> References: <20100905191234.GA19392@doriath.local> <20100905192517.GC27337@phare.normalesup.org> Message-ID: <20100905204232.GA19492@doriath.local> 5/09/10 @ 21:25 (+0200), thus spake Gael Varoquaux: > On Sun, Sep 05, 2010 at 09:12:34PM +0200, Ernest Adrogu? wrote: > > Hi, > > > How can it be done? > > np.may_share_memory Thanks Gael and Puneeth. I think the .base attribute is enough for what I want. Bye :) > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From robert.kern at gmail.com Sun Sep 5 16:59:34 2010 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 5 Sep 2010 15:59:34 -0500 Subject: [Numpy-discussion] test if two arrays share the same data In-Reply-To: <20100905204232.GA19492@doriath.local> References: <20100905191234.GA19392@doriath.local> <20100905192517.GC27337@phare.normalesup.org> <20100905204232.GA19492@doriath.local> Message-ID: 2010/9/5 Ernest Adrogu? : > ?5/09/10 @ 21:25 (+0200), thus spake Gael Varoquaux: >> On Sun, Sep 05, 2010 at 09:12:34PM +0200, Ernest Adrogu? wrote: >> > Hi, >> >> > How can it be done? >> >> np.may_share_memory > > Thanks Gael and Puneeth. > I think the .base attribute is enough for what I want. No, you really want may_share_memory(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From eadrogue at gmx.net Sun Sep 5 17:14:26 2010 From: eadrogue at gmx.net (Ernest =?iso-8859-1?Q?Adrogu=E9?=) Date: Sun, 5 Sep 2010 23:14:26 +0200 Subject: [Numpy-discussion] test if two arrays share the same data In-Reply-To: References: <20100905191234.GA19392@doriath.local> <20100905192517.GC27337@phare.normalesup.org> <20100905204232.GA19492@doriath.local> Message-ID: <20100905211426.GA19570@doriath.local> 5/09/10 @ 15:59 (-0500), thus spake Robert Kern: > 2010/9/5 Ernest Adrogu? : > > ?5/09/10 @ 21:25 (+0200), thus spake Gael Varoquaux: > >> On Sun, Sep 05, 2010 at 09:12:34PM +0200, Ernest Adrogu? wrote: > >> > Hi, > >> > >> > How can it be done? > >> > >> np.may_share_memory > > > > Thanks Gael and Puneeth. > > I think the .base attribute is enough for what I want. > > No, you really want may_share_memory(). Ok, I trust you :) Ernest From aarchiba at physics.mcgill.ca Sun Sep 5 17:53:02 2010 From: aarchiba at physics.mcgill.ca (Anne Archibald) Date: Sun, 5 Sep 2010 17:53:02 -0400 Subject: [Numpy-discussion] test if two arrays share the same data In-Reply-To: <20100905211426.GA19570@doriath.local> References: <20100905191234.GA19392@doriath.local> <20100905192517.GC27337@phare.normalesup.org> <20100905204232.GA19492@doriath.local> <20100905211426.GA19570@doriath.local> Message-ID: 2010/9/5 Ernest Adrogu? : > ?5/09/10 @ 15:59 (-0500), thus spake Robert Kern: >> 2010/9/5 Ernest Adrogu? : >> > ?5/09/10 @ 21:25 (+0200), thus spake Gael Varoquaux: >> >> On Sun, Sep 05, 2010 at 09:12:34PM +0200, Ernest Adrogu? wrote: >> >> > Hi, >> >> >> >> > How can it be done? >> >> >> >> np.may_share_memory >> > >> > Thanks Gael and Puneeth. >> > I think the .base attribute is enough for what I want. >> >> No, you really want may_share_memory(). > > Ok, I trust you :) Just to elaborate, one of the problems is that .base gives only the direct ancestor of the current array: In [2]: a = np.arange(10) In [3]: b = a[:] In [4]: c = b[:] In [5]: c.base Out[5]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) In [6]: c.base is b Out[6]: True In [7]: c.base is a Out[7]: False To check whether the arrays use the same memory arena, you have to walk the chain all the way back to the original base array. may_share_memory does this, and checks whether the arrays use overlapping ranges of memory as well. (It's "may" because you can have things like one being the even elements and the other being the odd elements, but checking for this is highly nontrivial, so may_share_memory just returns True.) Anne P.S. if you're thinking that this definition of base can cause memory leaks, then yes, you're right, but they are leaks only of the array descriptor objects, not the underlying memory. Still, you can exhaust memory doing things like: a = np.arange(10) while True: a = a[:] So don't do that. -A > Ernest > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From jfr.maurel at gmail.com Mon Sep 6 02:27:52 2010 From: jfr.maurel at gmail.com (=?ISO-8859-1?Q?Jean-Fran=E7ois_MAUREL?=) Date: Mon, 06 Sep 2010 08:27:52 +0200 Subject: [Numpy-discussion] dll load failed when loading numpy in embedded python Message-ID: <4C8489E8.6000901@gmail.com> Hi everyone, I get the error: 'DLL load failed' when loading numpy in embedded python. I could not find a solution in the archive. Any help appreciated. I have written a test as follows (borrowed from an other post): #include #include #include int load(){ PyObject *errobj, *errdata, *errtraceback; char* code = "import numpy\n"; Py_Initialize(); PyObject *m = PyImport_AddModule("__main__"); PyObject* d = PyModule_GetDict( m ); PyObject * resu = PyRun_String( code, Py_file_input, d, d); PyErr_Fetch (&errobj, &errdata, &errtraceback); if (errdata != NULL){ PyObject *s = PyObject_Str(errdata); char * c = PyString_AS_STRING(s); printf("%s\n",c); Py_DECREF(s); }else{ printf("no errors.\n"); } Py_XDECREF(errobj); Py_XDECREF(errdata); Py_XDECREF(errtraceback); Py_Finalize(); return 0; } int main(int argc, char **argv){ load(); return 0; } this is build using: gcc -I/openSource/python26/include -ggdb -c t.c -oc.o -Wall gcc c.o C:\openSource\python26\libs\libpython26.a -otest.exe -Wall configuration is: windows vista mingw gcc 4.5.0 python 2.6 numpy 1.5 -- Jean-Fran?ois MAUREL From cournape at gmail.com Mon Sep 6 02:45:08 2010 From: cournape at gmail.com (David Cournapeau) Date: Mon, 6 Sep 2010 15:45:08 +0900 Subject: [Numpy-discussion] dll load failed when loading numpy in embedded python In-Reply-To: <4C8489E8.6000901@gmail.com> References: <4C8489E8.6000901@gmail.com> Message-ID: On Mon, Sep 6, 2010 at 3:27 PM, Jean-Fran?ois MAUREL wrote: > ?Hi everyone, > > I get the error: 'DLL load failed' when loading numpy in embedded > python. I could not find a solution in the archive. > Any help appreciated. Could you get a more detailed log (looking into one of myriad logging system in windows) ? It should say why the dll fails loading (missing function, file not found, etc...). cheers, David From cgohlke at uci.edu Mon Sep 6 03:07:50 2010 From: cgohlke at uci.edu (Christoph Gohlke) Date: Mon, 06 Sep 2010 00:07:50 -0700 Subject: [Numpy-discussion] dll load failed when loading numpy in embedded python In-Reply-To: <4C8489E8.6000901@gmail.com> References: <4C8489E8.6000901@gmail.com> Message-ID: <4C849346.7080704@uci.edu> On 9/5/2010 11:27 PM, Jean-Fran?ois MAUREL wrote: > Hi everyone, > > I get the error: 'DLL load failed' when loading numpy in embedded > python. I could not find a solution in the archive. > Any help appreciated. > > I have written a test as follows (borrowed from an other post): > #include > #include > #include > > int load(){ > PyObject *errobj, *errdata, *errtraceback; > char* code = "import numpy\n"; > Py_Initialize(); > PyObject *m = PyImport_AddModule("__main__"); > PyObject* d = PyModule_GetDict( m ); > PyObject * resu = PyRun_String( code, Py_file_input, d, d); > PyErr_Fetch (&errobj,&errdata,&errtraceback); > if (errdata != NULL){ > PyObject *s = PyObject_Str(errdata); > char * c = PyString_AS_STRING(s); > printf("%s\n",c); > Py_DECREF(s); > }else{ > printf("no errors.\n"); > } > Py_XDECREF(errobj); > Py_XDECREF(errdata); > Py_XDECREF(errtraceback); > Py_Finalize(); > return 0; > } > > int main(int argc, char **argv){ > load(); > return 0; > } > > this is build using: > gcc -I/openSource/python26/include -ggdb -c t.c -oc.o -Wall > gcc c.o C:\openSource\python26\libs\libpython26.a -otest.exe -Wall > > configuration is: > windows vista > mingw gcc 4.5.0 > python 2.6 > numpy 1.5 > Your test program worked for me. Try opening the file test.exe with Dependecy Walker and see which DLLs are found or missing. Christoph From jfr.maurel at gmail.com Mon Sep 6 03:24:01 2010 From: jfr.maurel at gmail.com (=?UTF-8?B?SmVhbi1GcmFuw6dvaXMgTUFVUkVM?=) Date: Mon, 06 Sep 2010 09:24:01 +0200 Subject: [Numpy-discussion] dll load failed when loading numpy in embedded python In-Reply-To: References: <4C8489E8.6000901@gmail.com> Message-ID: <4C849711.7020001@gmail.com> Le 06/09/2010 08:45, David Cournapeau a ?crit : > Could you get a more detailed log (looking into one of myriad logging > system in windows) ? It should say why the dll fails loading (missing > function, file not found, etc...). > > cheers, Thank you for your quick answer. Unfortunately no success looking in windows log files. Any idea where to look more specifically? Regards -- Jean-Fran?ois MAUREL From jfr.maurel at gmail.com Mon Sep 6 03:26:49 2010 From: jfr.maurel at gmail.com (=?ISO-8859-1?Q?Jean-Fran=E7ois_MAUREL?=) Date: Mon, 06 Sep 2010 09:26:49 +0200 Subject: [Numpy-discussion] dll load failed when loading numpy in embedded python In-Reply-To: <4C849346.7080704@uci.edu> References: <4C8489E8.6000901@gmail.com> <4C849346.7080704@uci.edu> Message-ID: <4C8497B9.1030201@gmail.com> Le 06/09/2010 09:07, Christoph Gohlke a ?crit : > > Your test program worked for me. Try opening the file test.exe with > Dependecy Walker and see which DLLs > are found or missing. > > Christoph Thank you, i will try this one regards -- Jean-Fran?ois MAUREL From seb.haase at gmail.com Mon Sep 6 04:41:38 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Mon, 6 Sep 2010 10:41:38 +0200 Subject: [Numpy-discussion] [NumPy-Tickets] [NumPy] #1348: CHM of Numpy Reference Guide (development version) is outdated In-Reply-To: <026.1e3e7fd3ceafc1904fb425dc705853a1@scipy.org> References: <017.cd10f20f6093b255b6cf43fa66e3a0d7@scipy.org> <026.1e3e7fd3ceafc1904fb425dc705853a1@scipy.org> Message-ID: Hi, is there an URL of the weekly built CHM documentation file ? Thanks, Sebastian Haase On Sun, Sep 5, 2010 at 6:39 PM, wrote: > #1348: CHM of Numpy Reference Guide (development version) is outdated > ----------------------------+----------------------------------------------- > ?Reporter: ?petar ? ? ? ? ?| ? ? ? Owner: ?pv > ? ? ?Type: ?defect ? ? ? ? | ? ? ?Status: ?closed > ?Priority: ?normal ? ? ? ? | ? Milestone: > ?Component: ?Documentation ?| ? ? Version: > Resolution: ?fixed ? ? ? ? ?| ? ?Keywords: > ----------------------------+----------------------------------------------- > Changes (by pv): > > ?* status: ?new => closed > ?* resolution: ?=> fixed > > > Comment: > > ?You need to have installed http://sphinx.pocoo.org/ version >= 1.0.1, > ?Scipy, and matplotlib. > > ?Anyway, I set up scripts for rebuilding the CHM weekly, so from now on it > ?should stay up to date. > From asmund.hjulstad at gmail.com Mon Sep 6 05:16:54 2010 From: asmund.hjulstad at gmail.com (=?ISO-8859-1?Q?=C5smund_Hjulstad?=) Date: Mon, 6 Sep 2010 12:16:54 +0300 Subject: [Numpy-discussion] PyGTK breaks fortran extension??? In-Reply-To: References: Message-ID: 2010/9/3 Pauli Virtanen > > Fri, 03 Sep 2010 08:53:00 +0300, ?smund Hjulstad wrote: > > I have a f2py wrapped fortran extension, compiled using gcc-mingw32 > > (v.4.5.0), numpy 1.5, Python 2.7, where I am experiencing the strangest > > behaviour. It appears that loading pygtk breaks my fortran extension. > > Be aware that "import gtk" sets the current locale. > > You can reset it with > "import locale; locale.setlocale(locale.LC_ALL, 'C')". > > Depending on the platform, the locale might for example alter how your > Fortran runtime parses floating point numbers (i.e., whether the decimal > separator is "," or "."). For instance, in some old versions of gfortran > this *was* a problem -- changing the C locale would break the Fortran > runtime, making it stop the program with some obscure assertion. Thanks for the suggestion, I was not thinking about this at all, but still.. The problem still persists. (I'm running gfortran 4.5.0 on windows xp.), even after resetting the locale after the gtk import, like this import gtk import locale locale.setlocale(locale.LC_ALL,'C') import refprop ierr,herr = refprop.setup0(2,"METHANE.FLD|ETHANE.FLD","HMX.BNC","DEF") > > > The fortran code has initialization code (it calculates reference state > > for a combination of fluids). The initialization appearantly has some > > sanity checks, verifying the results. This initialization fails if gtk > > is loaded first. > > How does it fail? Knowing this could help debugging The specific error message is [SETREF error 119] convergence failure in calculating reference state for compon ent #? 2:? [SATP error 144] iteration for saturation state did not converge; P = ?0.10133 MPa. From jfr.maurel at gmail.com Mon Sep 6 06:19:35 2010 From: jfr.maurel at gmail.com (=?ISO-8859-1?Q?Jean-Fran=E7ois_MAUREL?=) Date: Mon, 06 Sep 2010 12:19:35 +0200 Subject: [Numpy-discussion] dll load failed when loading numpy in embedded python In-Reply-To: <4C8497B9.1030201@gmail.com> References: <4C8489E8.6000901@gmail.com> <4C849346.7080704@uci.edu> <4C8497B9.1030201@gmail.com> Message-ID: <4C84C037.8020607@gmail.com> Le 06/09/2010 09:26, Jean-Fran?ois MAUREL a ?crit : > Le 06/09/2010 09:07, Christoph Gohlke a ?crit : >> Your test program worked for me. Try opening the file test.exe with >> Dependecy Walker and see which DLLs >> are found or missing. >> >> Christoph > Thank you, i will try this one > > regards > problem remains. dependencywalker only found missing IEShims.dll which I added in the path but the error is still there. I only have a warning now from dependencywalker on ieframe.dll. @Cristoph: What was your configuration when executing the test successfully (os, compiler)? Regards -- Jean-Fran?ois MAUREL From pav at iki.fi Mon Sep 6 06:24:44 2010 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 6 Sep 2010 10:24:44 +0000 (UTC) Subject: [Numpy-discussion] [NumPy-Tickets] [NumPy] #1348: CHM of Numpy Reference Guide (development version) is outdated References: <017.cd10f20f6093b255b6cf43fa66e3a0d7@scipy.org> <026.1e3e7fd3ceafc1904fb425dc705853a1@scipy.org> Message-ID: Mon, 06 Sep 2010 10:41:38 +0200, Sebastian Haase wrote: > > is there an URL of the weekly built CHM documentation file ? It's the one linked from http://docs.scipy.org/doc/ -- Pauli Virtanen From jfr.maurel at gmail.com Mon Sep 6 07:27:43 2010 From: jfr.maurel at gmail.com (=?ISO-8859-1?Q?Jean-Fran=E7ois_MAUREL?=) Date: Mon, 06 Sep 2010 13:27:43 +0200 Subject: [Numpy-discussion] dll load failed when loading numpy in embedded python In-Reply-To: <4C84C037.8020607@gmail.com> References: <4C8489E8.6000901@gmail.com> <4C849346.7080704@uci.edu> <4C8497B9.1030201@gmail.com> <4C84C037.8020607@gmail.com> Message-ID: <4C84D02F.2080107@gmail.com> Le 06/09/2010 12:19, Jean-Fran?ois MAUREL a ?crit : > Le 06/09/2010 09:26, Jean-Fran?ois MAUREL a ?crit : >> Le 06/09/2010 09:07, Christoph Gohlke a ?crit : >>> Your test program worked for me. Try opening the file test.exe with >>> Dependecy Walker and see which DLLs >>> are found or missing. >>> >>> Christoph >> Thank you, i will try this one >> >> regards >> > using profiling in dependencyWalker I found this: LoadLibraryExA("C:\openSource\python26\lib\site-packages\numpy\core\multiarray.pyd", 0x00000000, LOAD_WITH_ALTERED_SEARCH_PATH) called from "PYTHON26.DLL" at address 0x1E02D9BD. *Loaded "MULTIARRAY.PYD" at address 0x69B80000. Successfully hooked module.* Unloaded "MULTIARRAY.PYD" at address 0x69B80000. *LoadLibraryExA("C:\openSource\python26\lib\site-packages\numpy\core\multiarray.pyd", 0x00000000, LOAD_WITH_ALTERED_SEARCH_PATH) returned NULL. Error: Le module spcifi est introuvable (126). * any ideas? regards -- Jean-Fran?ois MAUREL From seb.haase at gmail.com Mon Sep 6 07:38:00 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Mon, 6 Sep 2010 13:38:00 +0200 Subject: [Numpy-discussion] [NumPy-Tickets] [NumPy] #1348: CHM of Numpy Reference Guide (development version) is outdated In-Reply-To: References: <017.cd10f20f6093b255b6cf43fa66e3a0d7@scipy.org> <026.1e3e7fd3ceafc1904fb425dc705853a1@scipy.org> Message-ID: On Mon, Sep 6, 2010 at 12:24 PM, Pauli Virtanen wrote: > Mon, 06 Sep 2010 10:41:38 +0200, Sebastian Haase wrote: >> >> is there an URL of the weekly built CHM documentation file ? > > It's the one linked from http://docs.scipy.org/doc/ > Hi Pauli, Thanks for the info. that page could say: Latest: (development versions, weekly generated) instead of just Latest: (development versions) Just a quick idea - take it, or leave it ... ;-) - Sebastian From guillaume.c.cherel at gmail.com Mon Sep 6 08:35:35 2010 From: guillaume.c.cherel at gmail.com (=?ISO-8859-1?Q?Guillaume_Ch=E9rel?=) Date: Mon, 06 Sep 2010 14:35:35 +0200 Subject: [Numpy-discussion] saving masked array to a text file Message-ID: <4C84E017.5050804@gmail.com> Hello, Is there a function like savetxt that can deal with missing values in masked arrays? I'd like to control the way the missing values are written to the file. Thanks, Guillaume From jfr.maurel at gmail.com Mon Sep 6 10:21:50 2010 From: jfr.maurel at gmail.com (=?ISO-8859-1?Q?Jean-Fran=E7ois_MAUREL?=) Date: Mon, 06 Sep 2010 16:21:50 +0200 Subject: [Numpy-discussion] dll load failed when loading numpy in embedded python In-Reply-To: <4C8489E8.6000901@gmail.com> References: <4C8489E8.6000901@gmail.com> Message-ID: <4C84F8FE.6080808@gmail.com> Le 06/09/2010 08:27, Jean-Fran?ois MAUREL a ?crit : > Hi everyone, > > I get the error: 'DLL load failed' when loading numpy in embedded > python. I could not find a solution in the archive. > Any help appreciated. > > I have written a test as follows (borrowed from an other post): > #include > #include > #include > > int load(){ > PyObject *errobj, *errdata, *errtraceback; > char* code = "import numpy\n"; > Py_Initialize(); > PyObject *m = PyImport_AddModule("__main__"); > PyObject* d = PyModule_GetDict( m ); > PyObject * resu = PyRun_String( code, Py_file_input, d, d); > PyErr_Fetch (&errobj,&errdata,&errtraceback); > if (errdata != NULL){ > PyObject *s = PyObject_Str(errdata); > char * c = PyString_AS_STRING(s); > printf("%s\n",c); > Py_DECREF(s); > }else{ > printf("no errors.\n"); > } > Py_XDECREF(errobj); > Py_XDECREF(errdata); > Py_XDECREF(errtraceback); > Py_Finalize(); > return 0; > } > > int main(int argc, char **argv){ > load(); > return 0; > } > > this is build using: > gcc -I/openSource/python26/include -ggdb -c t.c -oc.o -Wall > gcc c.o C:\openSource\python26\libs\libpython26.a -otest.exe -Wall > > configuration is: > windows vista > mingw gcc 4.5.0 > python 2.6 > numpy 1.5 > For the record I intalled python 2.6.6 instead of an old 2.6.1 I had and it works Regards -- Jean-Fran?ois MAUREL From lists at informa.tiker.net Mon Sep 6 15:03:50 2010 From: lists at informa.tiker.net (Andreas Kloeckner) Date: Mon, 06 Sep 2010 15:03:50 -0400 Subject: [Numpy-discussion] Pthreads ATLAS Message-ID: <87eid6zobt.fsf@grizzly.selfip.net> Hi all, I've just tried to set up a Python 3 test environment for me to begin porting PyCUDA and PyOpenCL to Python 3, but numpy installation managed to foil my. I'm on Debian, and I believe that Debian purposefully compiles ATLAS without pthreads support. Whenever I build numpy 1.5, I encounter this error: Python 3.1.2 (release31-maint, Jul 8 2010, 09:18:08) [GCC 4.4.4] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy Traceback (most recent call last): File "", line 1, in File "/home/andreas/src3/lib/python3.1/site-packages/numpy/__init__.py", line 136, in from . import add_newdocs File "/home/andreas/src3/lib/python3.1/site-packages/numpy/add_newdocs.py", line 9, in from numpy.lib import add_newdoc File "/home/andreas/src3/lib/python3.1/site-packages/numpy/lib/__init__.py", line 13, in from .polynomial import * File "/home/andreas/src3/lib/python3.1/site-packages/numpy/lib/polynomial.py", line 17, in from numpy.linalg import eigvals, lstsq File "/home/andreas/src3/lib/python3.1/site-packages/numpy/linalg/__init__.py", line 48, in from .linalg import * File "/home/andreas/src3/lib/python3.1/site-packages/numpy/linalg/linalg.py", line 23, in from numpy.linalg import lapack_lite ImportError: /home/andreas/src3/lib/python3.1/site-packages/numpy/linalg/lapack_lite.so: undefined symbol: ATL_dptsyrk I gather that "pt" in the symbol name means that this is a pthreads-accelerated version of dsyrk. How do I convey to numpy that I don't have (nor want) Pthreads-accelerated ATLAS bits? Failing that, how do I tell it to just not use ATLAS? I tried looking for information at - numpy-1.5.0/doc/source/user/install.rst - http://docs.scipy.org/doc/ - http://new.scipy.org/building/linux.html - http://www.scipy.org/Installing_SciPy/Linux but nothing seemed to apply, and Google only gave me an instance of Skipper Seabold asking the same question (apparently when building his own Atlas) only to reply with a somewhat opaque "I figured it out" message. Thanks in advance, Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From xscript at gmx.net Mon Sep 6 15:17:37 2010 From: xscript at gmx.net (=?UTF-8?B?TGx1w61z?=) Date: Mon, 06 Sep 2010 21:17:37 +0200 Subject: [Numpy-discussion] DataArray fixes Message-ID: <86wrqywujy.wl%lluis@ginnungagap.pc.ac.upc.edu> I started writing some fixes for DataArray, but some others need discussion on whether they should be implemented as I describe. To this extent, please see issues at http://github.com/fperez/datarray/issues and comment on them, so that I'll know if I should implement these. Thanks, Lluis -- "And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer." -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth From Luke.Pinner at environment.gov.au Mon Sep 6 19:34:39 2010 From: Luke.Pinner at environment.gov.au (Pinner, Luke) Date: Tue, 7 Sep 2010 09:34:39 +1000 Subject: [Numpy-discussion] saving masked array to a text file In-Reply-To: <4C84E017.5050804@gmail.com> References: <4C84E017.5050804@gmail.com> Message-ID: <64102B2FB850C345B04D09C82018F45F056E4BF1@ACT01VEXCCL04.internal.govt> How about the masked aray filled method? eg numpy.savetxt(somefilename, somemaskedarray.filled(somevalue)) Luke -----Original Message----- From: Guillaume Ch?rel Sent: Monday, 6 September 2010 10:36 PM To: Discussion of Numerical Python Subject: [Numpy-discussion] saving masked array to a text file Hello, Is there a function like savetxt that can deal with missing values in masked arrays? I'd like to control the way the missing values are written to the file. Thanks, Guillaume ------ If you have received this transmission in error please notify us immediately by return e-mail and delete all copies. If this e-mail or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the e-mail or attachments. Please consider the environment before printing this email. ------ From dalcinl at gmail.com Mon Sep 6 19:40:56 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 6 Sep 2010 20:40:56 -0300 Subject: [Numpy-discussion] kron produces F-contiguous? In-Reply-To: References: Message-ID: On 2 September 2010 09:27, Neal Becker wrote: > Lisandro Dalcin wrote: > >> On 1 September 2010 19:24, Neal Becker wrote: >>> It seems if I call kron with 2 C-contiguous arrays, it returns an F- >>> contiguous array. ?Any reason for this (it's not what I wanted)? >>> >> >> Try numpy.linalg.inv ... >> >> > > I don't understand. ?What has linalg.inv got to do with kronecker product? > Sorry, incomplete mail... numpy.linalg.inv behave exactly like that. There are many functions that return F arrays for C inputs. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From cournape at gmail.com Tue Sep 7 02:17:14 2010 From: cournape at gmail.com (David Cournapeau) Date: Tue, 7 Sep 2010 15:17:14 +0900 Subject: [Numpy-discussion] Pthreads ATLAS In-Reply-To: <87eid6zobt.fsf@grizzly.selfip.net> References: <87eid6zobt.fsf@grizzly.selfip.net> Message-ID: On Tue, Sep 7, 2010 at 4:03 AM, Andreas Kloeckner wrote: > Hi all, > > I've just tried to set up a Python 3 test environment for me to begin > porting PyCUDA and PyOpenCL to Python 3, but numpy installation managed > to foil my. I'm on Debian, and I believe that Debian purposefully > compiles ATLAS without pthreads support. Whenever I build numpy 1.5, I > encounter this error: What does ldd tells you (ldd /home/andreas/src3/lib/python3.1/site-packages/numpy/linalg/lapack_lite.so) ? Also, do you have customized site.cfg ? cheers, David From guillaume.c.cherel at gmail.com Tue Sep 7 04:21:22 2010 From: guillaume.c.cherel at gmail.com (=?ISO-8859-1?Q?Guillaume_Ch=E9rel?=) Date: Tue, 07 Sep 2010 10:21:22 +0200 Subject: [Numpy-discussion] saving masked array to a text file In-Reply-To: <64102B2FB850C345B04D09C82018F45F056E4BF1@ACT01VEXCCL04.internal.govt> References: <4C84E017.5050804@gmail.com> <64102B2FB850C345B04D09C82018F45F056E4BF1@ACT01VEXCCL04.internal.govt> Message-ID: <4C85F602.9080306@gmail.com> It works only if I can find a value belonging to the dtype of the array that is meaningless in the context of my data. For example, if it is only composed of positive integers, then I can fill the missing values with -1. But in my case, I would like to write a string (empty string, actually) to the output file where values are missing. I can't do that with the ma.filled(value) method if my masked array dtype is an integer. What I would need is an equivalent of the genfromtxt function for writing. guillaume Le 07/09/2010 01:34, Pinner, Luke a ?crit : > How about the masked aray filled method? > eg numpy.savetxt(somefilename, somemaskedarray.filled(somevalue)) > > Luke > > -----Original Message----- > From: Guillaume Ch?rel > Sent: Monday, 6 September 2010 10:36 PM > To: Discussion of Numerical Python > Subject: [Numpy-discussion] saving masked array to a text file > > > Hello, > Is there a function like savetxt that can deal with missing values in > masked arrays? I'd like to control the way the missing values are > written to the file. > Thanks, > Guillaume > > > ------ > If you have received this transmission in error please notify us immediately by return e-mail and delete all copies. If this e-mail or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the e-mail or attachments. > > > > Please consider the environment before printing this email. > > ------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From pav at iki.fi Tue Sep 7 04:45:19 2010 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 7 Sep 2010 08:45:19 +0000 (UTC) Subject: [Numpy-discussion] Pthreads ATLAS References: <87eid6zobt.fsf@grizzly.selfip.net> Message-ID: Mon, 06 Sep 2010 15:03:50 -0400, Andreas Kloeckner wrote: [clip] > I gather that "pt" in the symbol name means that this is a > pthreads-accelerated version of dsyrk. How do I convey to numpy that I > don't have (nor want) Pthreads-accelerated ATLAS bits? Failing that, how > do I tell it to just not use ATLAS? > > I tried looking for information at > > - numpy-1.5.0/doc/source/user/install.rst - http://docs.scipy.org/doc/ > - http://new.scipy.org/building/linux.html - > http://www.scipy.org/Installing_SciPy/Linux BLAS=None LAPACK=None python setup.py build or BLAS=/path/to/libblas.so LAPACK=/path/to/liblapack.so ... may work. This'll probably even make it to the documentation one day... -- Pauli Virtanen From amenity at enthought.com Tue Sep 7 14:26:04 2010 From: amenity at enthought.com (Amenity Applewhite) Date: Tue, 7 Sep 2010 13:26:04 -0500 Subject: [Numpy-discussion] EPD Webinar: How to solve ODEs with SciPy References: Message-ID: <7586F6F9-0CFF-471D-9FB6-94253826705C@enthought.com> Enthought Python Distribution Webinar September 10 This Friday,Warren Weckesser will host the first of three webinars in a series on solving differential equations in Python. We will take a close look at the two tools available for solving ordinary differential equations in SciPy: the "odeint" function and the "ode" class. Two examples will be discussed: (1) the famous Lorenz equations that exhibit chaos, and (2) the Gray-Scott reaction-diffusion equations in 1D, from which we will obtain a system of ordinary differential equations by using the "Method of Lines". The demonstrations also include 3D plots and animation using Matplotlib. Enthought Python Distribution Webinar How do I...solve differential equations with Python? Part I: SciPy Tools Friday, September 10: 1pm CST/6pm UTC Wait list (for non EPD subscribers): email:amenity at enthought.com Early in 2011, Warren will host Part II: boundary value problems, and in the spring he'll follow up with a third installment to the series. Have a fantastic September, The Enthought Team From doutriaux1 at llnl.gov Tue Sep 7 15:12:58 2010 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Tue, 07 Sep 2010 12:12:58 -0700 Subject: [Numpy-discussion] distutils Message-ID: <4C868EBA.9020103@llnl.gov> Hi, I'm using distutils to build extensions written in C. I noticed that lately (it seems to be python 2.7 related) whenever I touch 1 C file, ALL the C files are rebuilt. Since I have a lot of C code, it takes a lot of time for nothing. Any idea why this is happening? Do I need to set something new in my setup.py ? Thx, C. From ondrej at certik.cz Tue Sep 7 15:23:11 2010 From: ondrej at certik.cz (Ondrej Certik) Date: Tue, 7 Sep 2010 12:23:11 -0700 Subject: [Numpy-discussion] Help with the tensordot function? In-Reply-To: References: Message-ID: Hi Rick! On Fri, Sep 3, 2010 at 4:02 AM, Rick Muller wrote: > Can someone help me replace a slow expression with a faster one based on > tensordot? I've read the documentation and I'm still confused. > > I have two matrices b and d. b is n x m and d is m x m. I want to replace > the expression > >>>> bdb = zeros(n,'d') >>>> for i in xrange(n): >>>>? ?? bdb[i,:] = dot(b[i,:],dot(d,b[i,:]) I am first trying to reproduce this --- the above is missing one ")" and also dot() seems to produce a number, but you are assigning it to bdb[i,:], also you declare bdb as a 1D array. So I tried this: http://gist.github.com/568879/ > > with something that doesn't have the for loop and thus is a bit faster. > > The first step is > >>>> bd = dot(b,d) > > However, following this with > >>>> bdb = dot(bd,b.T) > > doesn't work, since it yields a n x n matrix instead of an n x 1 vector. > Reading the notes on tensordot makes me think it's the function to use, but > I'm having trouble grokking the axes argument. Can anyone help? In the above gist, I did the following: bd = dot(b, d) bdb = diag(dot(bd, b.T)) print bdb which printed the same as: for i in xrange(n): bdb[i] = dot(b[i,:], dot(d, b[i, :])) print bdb but I think that this is not what you want, is it? I think you want to do something like: b * d * b.T but since b is a (n, m) matrix, the result is a matrix, not a vector, right? Ondrej From lxander.m at gmail.com Tue Sep 7 16:01:31 2010 From: lxander.m at gmail.com (Alexander Michael) Date: Tue, 7 Sep 2010 13:01:31 -0700 Subject: [Numpy-discussion] Inverting argsort(a, axis=0) to obtain column-wise ranks Message-ID: Calculating ranks by inverting the results of an argsort is straightforward and fast for 1D arrays: indices = argsort(a1) ranks = zeros_like(indices) ranks[indices] = arange(len(indices)) I was wondering if there was an equally pithy way to do this for multiple data samples stored column-wise in a 2D array. That is, is there a trick to invert the results of argsort(a2, axis=0) without iterating (in python) over the columns? Thanks, Alex From zachary.pincus at yale.edu Tue Sep 7 16:06:38 2010 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Tue, 7 Sep 2010 16:06:38 -0400 Subject: [Numpy-discussion] Inverting argsort(a, axis=0) to obtain column-wise ranks In-Reply-To: References: Message-ID: <222CE988-7159-418F-9859-585005E529A0@yale.edu> > indices = argsort(a1) > ranks = zeros_like(indices) > ranks[indices] = arange(len(indices)) Doesn't answer your original question directly, but I only recently learned from this list that the following does the same as the above: ranks = a1.argsort().argsort() Will wonders never cease... So does ranks=a2.argsort(axis=0).argsort(axis=0) then do the trick? Zach From friedrichromstedt at gmail.com Tue Sep 7 16:08:31 2010 From: friedrichromstedt at gmail.com (Friedrich Romstedt) Date: Tue, 7 Sep 2010 22:08:31 +0200 Subject: [Numpy-discussion] clip() with None as argument Message-ID: I just came across a problem with the intention to specify unset boundaries given to numpy.clip() or array.clip(): a.clip(1e-10, None) a.clip(None, -1e-10) When doing this, the returned array is dtype=numpy.object, seemingly None gets converted to a numpy.asarray(None, dtype=numpy.object), and this is then used for the comparison. Since (*any number* > None) is true, and (*any number* < None) is false, it works or not, depending on the usecase, despite one is still left with a dtype=numpy.object array. Precisely, using None as the lower boundary semi-works, while using None as the upper boundary doesn't match expectations at all. One can work around this, sure, but in my opinion it would be handy anyway to have clip() interpret None as argument as \pm \infty. Maybe I'm just missing the appropriate function, though. Friedrich From ondrej at certik.cz Tue Sep 7 16:09:32 2010 From: ondrej at certik.cz (Ondrej Certik) Date: Tue, 7 Sep 2010 13:09:32 -0700 Subject: [Numpy-discussion] Help with the tensordot function? In-Reply-To: References: Message-ID: On Tue, Sep 7, 2010 at 12:23 PM, Ondrej Certik wrote: > Hi Rick! > > On Fri, Sep 3, 2010 at 4:02 AM, Rick Muller wrote: >> Can someone help me replace a slow expression with a faster one based on >> tensordot? I've read the documentation and I'm still confused. >> >> I have two matrices b and d. b is n x m and d is m x m. I want to replace >> the expression >> >>>>> bdb = zeros(n,'d') >>>>> for i in xrange(n): >>>>>? ?? bdb[i,:] = dot(b[i,:],dot(d,b[i,:]) > > I am first trying to reproduce this --- the above is missing one ")" > and also dot() seems to produce a number, but you are assigning it to > bdb[i,:], also you declare bdb as a 1D array. So I tried this: > > http://gist.github.com/568879/ > >> >> with something that doesn't have the for loop and thus is a bit faster. >> >> The first step is >> >>>>> bd = dot(b,d) >> >> However, following this with >> >>>>> bdb = dot(bd,b.T) >> >> doesn't work, since it yields a n x n matrix instead of an n x 1 vector. >> Reading the notes on tensordot makes me think it's the function to use, but >> I'm having trouble grokking the axes argument. Can anyone help? > > In the above gist, I did the following: > > > bd = dot(b, d) > bdb = diag(dot(bd, b.T)) > print bdb > > > which printed the same as: > > > for i in xrange(n): > ? ?bdb[i] = dot(b[i,:], dot(d, b[i, :])) > print bdb > > > but I think that this is not what you want, is it? I think you want to > do something like: > > b * d * b.T > > but since b is a (n, m) matrix, the result is a matrix, not a vector, right? Ah, I just noticed you got this resolved in the other thread. Ondrej From friedrichromstedt at gmail.com Tue Sep 7 16:12:34 2010 From: friedrichromstedt at gmail.com (Friedrich Romstedt) Date: Tue, 7 Sep 2010 22:12:34 +0200 Subject: [Numpy-discussion] clip() with None as argument In-Reply-To: References: Message-ID: Ah, no need to answer, I do this myself: Friedrich, would you please use numpy.inf and -numpy.inf. Thanks, and sorry for the noise, Friedrich From rpmuller at gmail.com Tue Sep 7 16:14:11 2010 From: rpmuller at gmail.com (Rick Muller) Date: Tue, 7 Sep 2010 14:14:11 -0600 Subject: [Numpy-discussion] Help with the tensordot function? In-Reply-To: References: Message-ID: Ondrej, I was confused an unclear in my original question. I subsequently posted a followup, titled "Simplified question on tensordot' where I explained myself a lot better, and got some really good help. So, thank you very much for looking into this issue, but I believe that this has been resolved. Thanks! Rick http://mail.scipy.org/pipermail/numpy-discussion/2010-September/052584.html On Tue, Sep 7, 2010 at 1:23 PM, Ondrej Certik wrote: > Hi Rick! > > On Fri, Sep 3, 2010 at 4:02 AM, Rick Muller wrote: > > Can someone help me replace a slow expression with a faster one based on > > tensordot? I've read the documentation and I'm still confused. > > > > I have two matrices b and d. b is n x m and d is m x m. I want to replace > > the expression > > > >>>> bdb = zeros(n,'d') > >>>> for i in xrange(n): > >>>> bdb[i,:] = dot(b[i,:],dot(d,b[i,:]) > > I am first trying to reproduce this --- the above is missing one ")" > and also dot() seems to produce a number, but you are assigning it to > bdb[i,:], also you declare bdb as a 1D array. So I tried this: > > http://gist.github.com/568879/ > > > > > with something that doesn't have the for loop and thus is a bit faster. > > > > The first step is > > > >>>> bd = dot(b,d) > > > > However, following this with > > > >>>> bdb = dot(bd,b.T) > > > > doesn't work, since it yields a n x n matrix instead of an n x 1 vector. > > Reading the notes on tensordot makes me think it's the function to use, > but > > I'm having trouble grokking the axes argument. Can anyone help? > > In the above gist, I did the following: > > > bd = dot(b, d) > bdb = diag(dot(bd, b.T)) > print bdb > > > which printed the same as: > > > for i in xrange(n): > bdb[i] = dot(b[i,:], dot(d, b[i, :])) > print bdb > > > but I think that this is not what you want, is it? I think you want to > do something like: > > b * d * b.T > > but since b is a (n, m) matrix, the result is a matrix, not a vector, > right? > > Ondrej > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Rick Muller rpmuller at gmail.com 505-750-7557 -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Sep 7 16:22:45 2010 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 7 Sep 2010 15:22:45 -0500 Subject: [Numpy-discussion] clip() with None as argument In-Reply-To: References: Message-ID: On Tue, Sep 7, 2010 at 15:12, Friedrich Romstedt wrote: > Ah, no need to answer, I do this myself: > > Friedrich, would you please use numpy.inf and -numpy.inf. But if you have an integer array, you will run into the same problem. The result will be upcast to float. I think we would accept a patch that interprets None to be the appropriate extreme bound given the input datatype. This will be tricky, though, since it must be done in C (PyArray_Clip in numpy/core/src/multiarray/calculation.c). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From ben.root at ou.edu Tue Sep 7 17:41:01 2010 From: ben.root at ou.edu (Benjamin Root) Date: Tue, 7 Sep 2010 16:41:01 -0500 Subject: [Numpy-discussion] ndarray**maskedArray with broadcasting bug In-Reply-To: References: Message-ID: On Fri, Sep 3, 2010 at 3:50 PM, Benjamin Root wrote: > Here is a fun one... > > import numpy as np > > a_2d = np.random.random((3, 5)) > b_1d = np.random.random(5) > b_2d = np.vstack((b_1d, b_1d, b_1d)) > > a_ma_2d = np.ma.masked_array(a_2d, mask=(numpy.random.random((3, 5)) < > 0.25)) > b_ma_1d = np.ma.masked_array(b_1d, mask=(numpy.random.random(5) < 0.25)) > b_ma_2d = np.ma.masked_array(b_2d, mask=(numpy.random.random((3, 5)) < > 0.25)) > > # a**b without broadcasting (works) > print a_2d**b_2d > > # a**b with broadcasting (works) > print a_2d**b_1d > > # a_ma**b_ma without broadcasting (works) > print a_ma_2d**b_ma_2d > > # a_ma**b_ma with broadcasting (works) > print a_ma_2d**b_ma_1d > > # a**b_ma without broadcasting (works) > print a_2d**b_ma_2d > > # a**b_ma with broadcasting (FAILS) > print a_2d**b_ma_1d > Traceback (most recent call last): > File "", line 1, in > File "/home/bvr/Programs/numpy/numpy/ma/core.py", line 3697, in __rpow__ > return power(other, self) > File "/home/bvr/Programs/numpy/numpy/ma/core.py", line 6043, in power > m |= invalid > ValueError: invalid return array shape > > # Now, test broadcasting of the other side of the operator > c = np.random.random(5) > c_ma = np.ma.masked_array(c, mask=(np.random.random(10) < 0.25)) > > # c**b_ma with broadcasting (works) > print c**b_ma_2d > > # c_ma**b with broadcasting (FAILS) > print c_ma**b_2d > Traceback (most recent call last): > File "", line 1, in > File "/home/bvr/Programs/numpy/numpy/ma/core.py", line 3693, in __pow__ > return power(self, other) > File "/home/bvr/Programs/numpy/numpy/ma/core.py", line 6043, in power > m |= invalid > ValueError: invalid return array shape > > > So, this fails if we broadcast the masked array portion of a power > expression and the other value is a ndarray. > Note, the same also occurs if we broadcast a masked array in order to raise > it to powers indicated by an ndarray. > > Took me the longest time to nail this one down... > Ben Root > Btw, I have filed a bug report on this: http://projects.scipy.org/numpy/ticket/1606#comment:1 Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Wed Sep 8 03:50:50 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 8 Sep 2010 00:50:50 -0700 Subject: [Numpy-discussion] DataArray fixes In-Reply-To: <86wrqywujy.wl%lluis@ginnungagap.pc.ac.upc.edu> References: <86wrqywujy.wl%lluis@ginnungagap.pc.ac.upc.edu> Message-ID: Hi Lluis, On Mon, Sep 6, 2010 at 12:17 PM, Llu?s wrote: > I started writing some fixes for DataArray, but some others need discussion on > whether they should be implemented as I describe. > > To this extent, please see issues at http://github.com/fperez/datarray/issues > and comment on them, so that I'll know if I should implement these. I'm afraid I'm completely bandwidth-saturated for the next 2-3 weeks, but at the same time I have no intention whatsoever in becoming the bottleneck on all of this. Ultimately I don't see datarray as *my* code, but rather as code we as a group will refine to make numpy better. Keith Goodman has been also helping out, but I'm wondering if it would be good if I add a few more people with commit rights to that repo, so that development can move forward? Ultimately it's the people who do the coding who matter, and right now I'm too busy on the ipython side to do anything on this front at all. Just let me know what you and the others (Keith, Skipper, Rob, etc) think and I'm more than happy to open the doors to make the process easier. Once numpy settles in its github home, I hope we'll just move this repo from my personal hosting over to the numpy organization one, but I want to do that after numpy makes the transition to give them a chance to 'settle in'. Cheers, f From ruidc at yahoo.com Wed Sep 8 05:42:51 2010 From: ruidc at yahoo.com (RuiDC) Date: Wed, 8 Sep 2010 02:42:51 -0700 (PDT) Subject: [Numpy-discussion] ValueError: objects are not aligned Message-ID: <29641661.post@talk.nabble.com> I'm getting this error, which must be a simple error in getting the result from the cursor in the right shape, how do I get the cursor into the array as a single dimension?: import numpy as np import sqlite3 conn = sqlite3.connect("simpledb") cursor1 = conn.execute("select LogReturn from tblReturns limit 3") rf1 = np.asarray(cursor1.fetchall(), dtype=np.float64) print (rf1) print (np.corrcoef([rf1, rf1])) result: [[-0.00641625] [-0.00498411] [-0.0038015 ]] Traceback (most recent call last): File "C:\Documents and Settings\rui\workspace\numpytest\src\numpytest.py", line 38, in print (np.corrcoef([rf1, rf1])) File "C:\Python26\lib\site-packages\numpy\lib\function_base.py", line 2003, in corrcoef c = cov(x, y, rowvar, bias, ddof) File "C:\Python26\lib\site-packages\numpy\lib\function_base.py", line 1953, in cov return (dot(X, X.T.conj()) / fact).squeeze() -- View this message in context: http://old.nabble.com/ValueError%3A-objects-are-not-aligned-tp29641661p29641661.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From j.reid at mail.cryst.bbk.ac.uk Wed Sep 8 06:38:24 2010 From: j.reid at mail.cryst.bbk.ac.uk (John Reid) Date: Wed, 08 Sep 2010 11:38:24 +0100 Subject: [Numpy-discussion] new warning in 1.5.0: divide by zero encountered in log Message-ID: Hi, I recently upgraded to numpy 1.5.0 and now I get warnings when I take the logarithm of 0. In [5]: np.log(0.) Warning: divide by zero encountered in log Out[5]: -inf I want to evaluate x * log(x) where its value is defined as 0 when x=0 so I have the following function: def safe_x_log_x(x): "@return: x log(x) but replaces -inf with 0." l = np.log(x) result = x * l result[np.isneginf(l)] = 0. return result Is there a better way of doing this that won't give me all these warnings? Can I turn the warnings off? Thanks, John. From josef.pktd at gmail.com Wed Sep 8 07:09:46 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 8 Sep 2010 07:09:46 -0400 Subject: [Numpy-discussion] ValueError: objects are not aligned In-Reply-To: <29641661.post@talk.nabble.com> References: <29641661.post@talk.nabble.com> Message-ID: On Wed, Sep 8, 2010 at 5:42 AM, RuiDC wrote: > > I'm getting this error, which must be a simple error in getting the result > from the cursor in the right shape, how do I get the cursor into the array > as a single dimension?: > > ? ?import numpy as np > ? ?import sqlite3 > ? ?conn = sqlite3.connect("simpledb") > ? ?cursor1 = conn.execute("select LogReturn from tblReturns limit 3") > ? ?rf1 = np.asarray(cursor1.fetchall(), dtype=np.float64) > ? ?print (rf1) > ? ?print (np.corrcoef([rf1, rf1])) numpy.corrcoef(x, y=None, rowvar=1, bias=0) try np.corrcoef(rf1, rf1, rowvar=0) no [ ] or stacking rf1 rf1 into a single array first Josef > > result: > [[-0.00641625] > ?[-0.00498411] > ?[-0.0038015 ]] > Traceback (most recent call last): > ?File "C:\Documents and Settings\rui\workspace\numpytest\src\numpytest.py", > line 38, in > ? ?print (np.corrcoef([rf1, rf1])) > ?File "C:\Python26\lib\site-packages\numpy\lib\function_base.py", line > 2003, in corrcoef > ? ?c = cov(x, y, rowvar, bias, ddof) > ?File "C:\Python26\lib\site-packages\numpy\lib\function_base.py", line > 1953, in cov > ? ?return (dot(X, X.T.conj()) / fact).squeeze() > -- > View this message in context: http://old.nabble.com/ValueError%3A-objects-are-not-aligned-tp29641661p29641661.html > Sent from the Numpy-discussion mailing list archive at Nabble.com. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ndbecker2 at gmail.com Wed Sep 8 07:44:20 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 08 Sep 2010 07:44:20 -0400 Subject: [Numpy-discussion] vectorize doesn't like functools.partial Message-ID: If I try to use vectorize on the result of functools.partial, I seem to get: ValueError: failed to determine the number of arguments for Anything I can do about it? From xscript at gmx.net Wed Sep 8 08:00:01 2010 From: xscript at gmx.net (=?UTF-8?B?TGx1w61z?=) Date: Wed, 08 Sep 2010 14:00:01 +0200 Subject: [Numpy-discussion] DataArray fixes In-Reply-To: References: <86wrqywujy.wl%lluis@ginnungagap.pc.ac.upc.edu> Message-ID: <8662yge98e.wl%lluis@ginnungagap.pc.ac.upc.edu> Fernando Perez writes: > Just let me know what you and the others (Keith, Skipper, Rob, etc) > think and I'm more than happy to open the doors to make the process > easier. Well, the pull requests I posted to your repository are for fairly trivial fixes (but some still need documentation and regression test updates). But some of the other features I posted on the issue tracker might need some previous discussion (I can recall the 'axis' vs 'axes' issue from the top of my head). Lluis -- "And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer." -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth From josef.pktd at gmail.com Wed Sep 8 08:07:41 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 8 Sep 2010 08:07:41 -0400 Subject: [Numpy-discussion] vectorize doesn't like functools.partial In-Reply-To: References: Message-ID: On Wed, Sep 8, 2010 at 7:44 AM, Neal Becker wrote: > If I ?try to use vectorize on the result of functools.partial, I seem to > get: > > ValueError: failed to determine the number of arguments for > > > Anything I can do about it? Set .nin (attribute of vectorized function, I think) directly with number of arguments (int) Josef > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ruidc at yahoo.com Wed Sep 8 08:08:20 2010 From: ruidc at yahoo.com (Rui DaCosta) Date: Wed, 8 Sep 2010 05:08:20 -0700 (PDT) Subject: [Numpy-discussion] ValueError: objects are not aligned In-Reply-To: References: <29641661.post@talk.nabble.com> Message-ID: <642649.38327.qm@web31008.mail.mud.yahoo.com> Thanks for your assistance, I was following this example: http://www.scipy.org/Numpy_Example_List#head-779283aaa3cc20a3786ad33c2ee1fee9d68a4a53 I intend to use different arrays once I have resolved this problem. Trying your suggestion: ... print (rf1) print (np.corrcoef(rf1, rf1, rowvar=0)) results in: [[-0.00641625] [-0.00498411] [-0.0038015 ]] [[ 1. 1.] [ 1. 1.]] which should have been a 3x3 matrix as for "stacking rf1 rf1 into a single array first" how would I do this? Regards, Rui ________________________________ From: "josef.pktd at gmail.com" To: Discussion of Numerical Python Sent: Wed, September 8, 2010 1:09:46 PM Subject: Re: [Numpy-discussion] ValueError: objects are not aligned On Wed, Sep 8, 2010 at 5:42 AM, RuiDC wrote: > > I'm getting this error, which must be a simple error in getting the result > from the cursor in the right shape, how do I get the cursor into the array > as a single dimension?: > > import numpy as np > import sqlite3 > conn = sqlite3.connect("simpledb") > cursor1 = conn.execute("select LogReturn from tblReturns limit 3") > rf1 = np.asarray(cursor1.fetchall(), dtype=np.float64) > print (rf1) > print (np.corrcoef([rf1, rf1])) numpy.corrcoef(x, y=None, rowvar=1, bias=0) try np.corrcoef(rf1, rf1, rowvar=0) no [ ] or stacking rf1 rf1 into a single array first Josef > > result: > [[-0.00641625] > [-0.00498411] > [-0.0038015 ]] > Traceback (most recent call last): > File "C:\Documents and Settings\rui\workspace\numpytest\src\numpytest.py", > line 38, in > print (np.corrcoef([rf1, rf1])) > File "C:\Python26\lib\site-packages\numpy\lib\function_base.py", line > 2003, in corrcoef > c = cov(x, y, rowvar, bias, ddof) > File "C:\Python26\lib\site-packages\numpy\lib\function_base.py", line > 1953, in cov > return (dot(X, X.T.conj()) / fact).squeeze() > -- > View this message in context: >http://old.nabble.com/ValueError%3A-objects-are-not-aligned-tp29641661p29641661.html > > Sent from the Numpy-discussion mailing list archive at Nabble.com. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Sep 8 08:34:42 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 8 Sep 2010 08:34:42 -0400 Subject: [Numpy-discussion] ValueError: objects are not aligned In-Reply-To: <642649.38327.qm@web31008.mail.mud.yahoo.com> References: <29641661.post@talk.nabble.com> <642649.38327.qm@web31008.mail.mud.yahoo.com> Message-ID: On Wed, Sep 8, 2010 at 8:08 AM, Rui DaCosta wrote: > Thanks for your assistance, > I was following this example: > http://www.scipy.org/Numpy_Example_List#head-779283aaa3cc20a3786ad33c2ee1fee9d68a4a53 I didn't know, If arrays are 1d, then numpy does the stacking for you., but it doesn't work with column-vectors or 2d arrays. >>> np.corrcoef([np.arange(3), np.arange(3)], rowvar=1) array([[ 1., 1.], [ 1., 1.]]) >>> np.corrcoef([np.arange(3), np.arange(3)], rowvar=0) array([[ NaN, NaN, NaN], [ NaN, NaN, NaN], [ NaN, NaN, NaN]]) > > I intend to use different arrays once I have resolved this problem. > Trying your suggestion: > ... > print (rf1) > print (np.corrcoef(rf1, rf1, rowvar=0)) > results in: > [[-0.00641625] > [-0.00498411] > [-0.0038015 ]] > [[ 1. 1.] > [ 1. 1.]] > > which should have been a 3x3 matrix I'm not sure what you want here, you only have three values variance would also be zero, and corrcoef wouldn't be defined if you only have 1 observation per variable >>> np.corrcoef(np.arange(3)[:,None]) array([[ NaN, NaN, NaN], [ NaN, NaN, NaN], [ NaN, NaN, NaN]]) >>> np.corrcoef(np.column_stack([np.arange(5), np.arange(5)**2, np.arange(5)**3]), rowvar=0) array([[ 1. , 0.9589266 , 0.90588235], [ 0.9589266 , 1. , 0.98713033], [ 0.90588235, 0.98713033, 1. ]]) examples are in docstring for np.cov > as for "stacking rf1 rf1 into a single array first" how would I do this? >>> np.column_stack([np.arange(3)[:,None], np.arange(3)[:,None]]) array([[0, 0], [1, 1], [2, 2]]) >>> np.column_stack([np.arange(3)[:,None], np.arange(3)]) #doesn't required 2d array([[0, 0], [1, 1], [2, 2]]) >>> np.c_[np.arange(3)[:,None], np.arange(3)[:,None]] array([[0, 0], [1, 1], [2, 2]]) >>> np.hstack([np.arange(3)[:,None], np.arange(3)[:,None]]) array([[0, 0], [1, 1], [2, 2]]) Josef > Regards, > Rui > ________________________________ > From: "josef.pktd at gmail.com" > To: Discussion of Numerical Python > Sent: Wed, September 8, 2010 1:09:46 PM > Subject: Re: [Numpy-discussion] ValueError: objects are not aligned > > On Wed, Sep 8, 2010 at 5:42 AM, RuiDC wrote: >> >> I'm getting this error, which must be a simple error in getting the result >> from the cursor in the right shape, how do I get the cursor into the array >> as a single dimension?: >> >> ? ?import numpy as np >> ? ?import sqlite3 >> ? ?conn = sqlite3.connect("simpledb") >> ? ?cursor1 = conn.execute("select LogReturn from tblReturns limit 3") >> ? ?rf1 = np.asarray(cursor1.fetchall(), dtype=np.float64) >> ? ?print (rf1) >> ? ?print (np.corrcoef([rf1, rf1])) > > numpy.corrcoef(x, y=None, rowvar=1, bias=0) > > try > np.corrcoef(rf1, rf1, rowvar=0)? no [ ] > or > stacking rf1 rf1 into a single array first > > Josef > >> >> result: >> [[-0.00641625] >> ?[-0.00498411] >> ?[-0.0038015 ]] >> Traceback (most recent call last): >> ?File "C:\Documents and >> Settings\rui\workspace\numpytest\src\numpytest.py", >> line 38, in >> ? ?print (np.corrcoef([rf1, rf1])) >> ?File "C:\Python26\lib\site-packages\numpy\lib\function_base.py", line >> 2003, in corrcoef >> ? ?c = cov(x, y, rowvar, bias, ddof) >> ?File "C:\Python26\lib\site-packages\numpy\lib\function_base.py", line >> 1953, in cov >> ? ?return (dot(X, X.T.conj()) / fact).squeeze() >> -- >> View this message in context: >> http://old.nabble.com/ValueError%3A-objects-are-not-aligned-tp29641661p29641661.html >> Sent from the Numpy-discussion mailing list archive at Nabble.com. >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From alan.isaac at gmail.com Wed Sep 8 08:35:47 2010 From: alan.isaac at gmail.com (Alan G Isaac) Date: Wed, 08 Sep 2010 08:35:47 -0400 Subject: [Numpy-discussion] new warning in 1.5.0: divide by zero encountered in log In-Reply-To: References: Message-ID: <4C878323.2080709@gmail.com> On 9/8/2010 6:38 AM, John Reid wrote: > def safe_x_log_x(x): > "@return: x log(x) but replaces -inf with 0." > l = np.log(x) > result = x * l > result[np.isneginf(l)] = 0. > return result Assuming x is known to contain nonnegative floats: def safe_x_log_x(x): x = np.asarray(x) l = np.zeros_like(x) l[x>0] = np.log(x[x>0]) return x * l hth, Alan Isaac From charlesr.harris at gmail.com Wed Sep 8 08:54:25 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 8 Sep 2010 06:54:25 -0600 Subject: [Numpy-discussion] new warning in 1.5.0: divide by zero encountered in log In-Reply-To: References: Message-ID: On Wed, Sep 8, 2010 at 4:38 AM, John Reid wrote: > Hi, > > I recently upgraded to numpy 1.5.0 and now I get warnings when I take > the logarithm of 0. > > In [5]: np.log(0.) > Warning: divide by zero encountered in log > Out[5]: -inf > > > I want to evaluate x * log(x) where its value is defined as 0 when x=0 > so I have the following function: > > def safe_x_log_x(x): > "@return: x log(x) but replaces -inf with 0." > l = np.log(x) > result = x * l > result[np.isneginf(l)] = 0. > return result > > > Is there a better way of doing this that won't give me all these > warnings? Can I turn the warnings off? > > In [1]: log(0) Warning: divide by zero encountered in log Out[1]: -inf In [2]: olderr = np.seterr(divide='ignore') In [3]: log(0) Out[3]: -inf Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ruidc at yahoo.com Wed Sep 8 09:21:55 2010 From: ruidc at yahoo.com (Rui DaCosta) Date: Wed, 8 Sep 2010 06:21:55 -0700 (PDT) Subject: [Numpy-discussion] ValueError: objects are not aligned In-Reply-To: References: <29641661.post@talk.nabble.com> <642649.38327.qm@web31008.mail.mud.yahoo.com> Message-ID: <650099.57405.qm@web31007.mail.mud.yahoo.com> Thanks again, I was trying to follow http://www.scipy.org/Numpy_Example_List#head-779283aaa3cc20a3786ad33c2ee1fee9d68a4a53 but with the key difference being, that instead of the data for the three arrays being constructed from literals, that i'd be sourcing from a relational database. The data is quite simple, in that I have 199 recorded days for 8 different variables, but am just trying to do a proof-of-concept first with three, when I had this problem. The functionality I am seeking to reproduce is in Excel as a 3x3 grid: =CORREL(rf1,rf1) =CORREL(rf1,rf2) =CORREL(rf1,rf3) =CORREL(rf2,rf1) =CORREL(rf2,rf2) =CORREL(rf2,rf3) =CORREL(rf3,rf1) =CORREL(rf3,rf2) =CORREL(rf3,rf3) I tried to simplify and isolate the problem, and hence the confusion. So In sourcing from the DB, I have 3 different cursors (taking a subset of 3 of the 8), each 199 rows in length. From reading the lists and blogs, the way to put these into numpy seemed to be: rf1 = np.asarray(cursor1.fetchall(), dtype=np.float64) rf2 = np.asarray(cursor2.fetchall(), dtype=np.float64) rf3 = np.asarray(cursor3.fetchall(), dtype=np.float64) Then to try and produce a correlation matrix: print (np.corrcoef([rf1, rf2, rf3])) But this is where I get the original error. From looking at column_stack, this doesn't seem like what I want to achieve, or am I looking at this the wrong way somehow?. Thanks again, Rui ________________________________ From: "josef.pktd at gmail.com" To: Discussion of Numerical Python Sent: Wed, September 8, 2010 2:34:42 PM Subject: Re: [Numpy-discussion] ValueError: objects are not aligned On Wed, Sep 8, 2010 at 8:08 AM, Rui DaCosta wrote: > Thanks for your assistance, > I was following this example: >http://www.scipy.org/Numpy_Example_List#head-779283aaa3cc20a3786ad33c2ee1fee9d68a4a53 >3 I didn't know, If arrays are 1d, then numpy does the stacking for you., but it doesn't work with column-vectors or 2d arrays. >>> np.corrcoef([np.arange(3), np.arange(3)], rowvar=1) array([[ 1., 1.], [ 1., 1.]]) >>> np.corrcoef([np.arange(3), np.arange(3)], rowvar=0) array([[ NaN, NaN, NaN], [ NaN, NaN, NaN], [ NaN, NaN, NaN]]) > > I intend to use different arrays once I have resolved this problem. > Trying your suggestion: > ... > print (rf1) > print (np.corrcoef(rf1, rf1, rowvar=0)) > results in: > [[-0.00641625] > [-0.00498411] > [-0.0038015 ]] > [[ 1. 1.] > [ 1. 1.]] > > which should have been a 3x3 matrix I'm not sure what you want here, you only have three values variance would also be zero, and corrcoef wouldn't be defined if you only have 1 observation per variable >>> np.corrcoef(np.arange(3)[:,None]) array([[ NaN, NaN, NaN], [ NaN, NaN, NaN], [ NaN, NaN, NaN]]) >>> np.corrcoef(np.column_stack([np.arange(5), np.arange(5)**2, np.arange(5)**3]), >>>rowvar=0) array([[ 1. , 0.9589266 , 0.90588235], [ 0.9589266 , 1. , 0.98713033], [ 0.90588235, 0.98713033, 1. ]]) examples are in docstring for np.cov > as for "stacking rf1 rf1 into a single array first" how would I do this? >>> np.column_stack([np.arange(3)[:,None], np.arange(3)[:,None]]) array([[0, 0], [1, 1], [2, 2]]) >>> np.column_stack([np.arange(3)[:,None], np.arange(3)]) #doesn't required 2d array([[0, 0], [1, 1], [2, 2]]) >>> np.c_[np.arange(3)[:,None], np.arange(3)[:,None]] array([[0, 0], [1, 1], [2, 2]]) >>> np.hstack([np.arange(3)[:,None], np.arange(3)[:,None]]) array([[0, 0], [1, 1], [2, 2]]) Josef > Regards, > Rui > ________________________________ > From: "josef.pktd at gmail.com" > To: Discussion of Numerical Python > Sent: Wed, September 8, 2010 1:09:46 PM > Subject: Re: [Numpy-discussion] ValueError: objects are not aligned > > On Wed, Sep 8, 2010 at 5:42 AM, RuiDC wrote: >> >> I'm getting this error, which must be a simple error in getting the result >> from the cursor in the right shape, how do I get the cursor into the array >> as a single dimension?: >> >> import numpy as np >> import sqlite3 >> conn = sqlite3.connect("simpledb") >> cursor1 = conn.execute("select LogReturn from tblReturns limit 3") >> rf1 = np.asarray(cursor1.fetchall(), dtype=np.float64) >> print (rf1) >> print (np.corrcoef([rf1, rf1])) > > numpy.corrcoef(x, y=None, rowvar=1, bias=0) > > try > np.corrcoef(rf1, rf1, rowvar=0) no [ ] > or > stacking rf1 rf1 into a single array first > > Josef > >> >> result: >> [[-0.00641625] >> [-0.00498411] >> [-0.0038015 ]] >> Traceback (most recent call last): >> File "C:\Documents and >> Settings\rui\workspace\numpytest\src\numpytest.py", >> line 38, in >> print (np.corrcoef([rf1, rf1])) >> File "C:\Python26\lib\site-packages\numpy\lib\function_base.py", line >> 2003, in corrcoef >> c = cov(x, y, rowvar, bias, ddof) >> File "C:\Python26\lib\site-packages\numpy\lib\function_base.py", line >> 1953, in cov >> return (dot(X, X.T.conj()) / fact).squeeze() >> -- >> View this message in context: >>http://old.nabble.com/ValueError%3A-objects-are-not-aligned-tp29641661p29641661.html >>l >> Sent from the Numpy-discussion mailing list archive at Nabble.com. >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Sep 8 09:30:54 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 8 Sep 2010 09:30:54 -0400 Subject: [Numpy-discussion] ValueError: objects are not aligned In-Reply-To: <650099.57405.qm@web31007.mail.mud.yahoo.com> References: <29641661.post@talk.nabble.com> <642649.38327.qm@web31008.mail.mud.yahoo.com> <650099.57405.qm@web31007.mail.mud.yahoo.com> Message-ID: On Wed, Sep 8, 2010 at 9:21 AM, Rui DaCosta wrote: > Thanks again, > > I was trying to > follow?http://www.scipy.org/Numpy_Example_List#head-779283aaa3cc20a3786ad33c2ee1fee9d68a4a53 > but with the key difference being, that instead of the data for the three > arrays being constructed from literals, that i'd be sourcing from a > relational database. The data is quite simple, in that I have 199 recorded > days for 8 different variables, but am just trying to do a proof-of-concept > first with three, when I had this problem. > The functionality I am seeking to reproduce is in Excel as a 3x3 grid: > =CORREL(rf1,rf1) > =CORREL(rf1,rf2) > =CORREL(rf1,rf3) > =CORREL(rf2,rf1) > =CORREL(rf2,rf2) > =CORREL(rf2,rf3) > =CORREL(rf3,rf1) > =CORREL(rf3,rf2) > =CORREL(rf3,rf3) > I tried to simplify and isolate the problem, and hence the confusion. > So In sourcing from the DB, I have 3 different cursors (taking a subset of 3 > of the 8), each 199 rows in length. > From reading the lists and blogs, the way to put these into numpy seemed to > be: > rf1 = np.asarray(cursor1.fetchall(), dtype=np.float64) > rf2 = np.asarray(cursor2.fetchall(), dtype=np.float64) > rf3 = np.asarray(cursor3.fetchall(), dtype=np.float64) > Then to try and produce a correlation matrix: > print (np.corrcoef([rf1, rf2, rf3])) > But this is where I get the original error. > From looking at column_stack, this doesn't seem like what I want to achieve, > or am I looking at this the wrong way somehow?. column_stack and rowvar=0 should work, but it might be easier to get rid of the extra dimension that the cursor gives you e.g. rf1 = np.asarray(cursor1.fetchall(), dtype=np.float64).ravel() depends on whether you want to work with 2d or 1d data arrays Josef > Thanks again, > Rui > > ________________________________ > From: "josef.pktd at gmail.com" > To: Discussion of Numerical Python > Sent: Wed, September 8, 2010 2:34:42 PM > Subject: Re: [Numpy-discussion] ValueError: objects are not aligned > > On Wed, Sep 8, 2010 at 8:08 AM, Rui DaCosta wrote: >> Thanks for your assistance, >> I was following this example: >> >> http://www.scipy.org/Numpy_Example_List#head-779283aaa3cc20a3786ad33c2ee1fee9d68a4a53 > > I didn't know, > If arrays are 1d, then numpy does the stacking for you., but it > doesn't work with column-vectors or 2d arrays. >>>> np.corrcoef([np.arange(3), np.arange(3)], rowvar=1) > array([[ 1.,? 1.], > ? ? ? [ 1.,? 1.]]) >>>> np.corrcoef([np.arange(3), np.arange(3)], rowvar=0) > array([[ NaN,? NaN,? NaN], > ? ? ? [ NaN,? NaN,? NaN], > ? ? ? [ NaN,? NaN,? NaN]]) >> >> I intend to use different arrays once I have resolved this problem. >> Trying your suggestion: >> ... >> print (rf1) >> print (np.corrcoef(rf1, rf1, rowvar=0)) >> results in: >> [[-0.00641625] >> [-0.00498411] >> [-0.0038015 ]] >> [[ 1. 1.] >> [ 1. 1.]] >> >> which should have been a 3x3 matrix > > I'm not sure what you want here, you only have three values > variance would also be zero, and corrcoef wouldn't be defined if you > only have 1 observation per variable > >>>> np.corrcoef(np.arange(3)[:,None]) > array([[ NaN,? NaN,? NaN], > ? ? ? [ NaN,? NaN,? NaN], > ? ? ? [ NaN,? NaN,? NaN]]) > >>>> np.corrcoef(np.column_stack([np.arange(5), np.arange(5)**2, >>>> np.arange(5)**3]), rowvar=0) > array([[ 1.? ? ? ? ,? 0.9589266 ,? 0.90588235], > ? ? ? [ 0.9589266 ,? 1.? ? ? ? ,? 0.98713033], > ? ? ? [ 0.90588235,? 0.98713033,? 1.? ? ? ? ]]) > > examples are in docstring for np.cov > > > > >> as for "stacking rf1 rf1 into a single array first" how would I do this? > >>>> np.column_stack([np.arange(3)[:,None], np.arange(3)[:,None]]) > array([[0, 0], > ? ? ? [1, 1], > ? ? ? [2, 2]]) >>>> np.column_stack([np.arange(3)[:,None], np.arange(3)])? #doesn't required >>>> 2d > array([[0, 0], > ? ? ? [1, 1], > ? ? ? [2, 2]]) > >>>> np.c_[np.arange(3)[:,None], np.arange(3)[:,None]] > array([[0, 0], > ? ? ? [1, 1], > ? ? ? [2, 2]]) >>>> np.hstack([np.arange(3)[:,None], np.arange(3)[:,None]]) > array([[0, 0], > ? ? ? [1, 1], > ? ? ? [2, 2]]) > > > Josef > >> Regards, >> Rui >> ________________________________ >> From: "josef.pktd at gmail.com" >> To: Discussion of Numerical Python >> Sent: Wed, September 8, 2010 1:09:46 PM >> Subject: Re: [Numpy-discussion] ValueError: objects are not aligned >> >> On Wed, Sep 8, 2010 at 5:42 AM, RuiDC wrote: >>> >>> I'm getting this error, which must be a simple error in getting the >>> result >>> from the cursor in the right shape, how do I get the cursor into the >>> array >>> as a single dimension?: >>> >>> ? ?import numpy as np >>> ? ?import sqlite3 >>> ? ?conn = sqlite3.connect("simpledb") >>> ? ?cursor1 = conn.execute("select LogReturn from tblReturns limit 3") >>> ? ?rf1 = np.asarray(cursor1.fetchall(), dtype=np.float64) >>> ? ?print (rf1) >>> ? ?print (np.corrcoef([rf1, rf1])) >> >> numpy.corrcoef(x, y=None, rowvar=1, bias=0) >> >> try >> np.corrcoef(rf1, rf1, rowvar=0)? no [ ] >> or >> stacking rf1 rf1 into a single array first >> >> Josef >> >>> >>> result: >>> [[-0.00641625] >>> ?[-0.00498411] >>> ?[-0.0038015 ]] >>> Traceback (most recent call last): >>> ?File "C:\Documents and >>> Settings\rui\workspace\numpytest\src\numpytest.py", >>> line 38, in >>> ? ?print (np.corrcoef([rf1, rf1])) >>> ?File "C:\Python26\lib\site-packages\numpy\lib\function_base.py", line >>> 2003, in corrcoef >>> ? ?c = cov(x, y, rowvar, bias, ddof) >>> ?File "C:\Python26\lib\site-packages\numpy\lib\function_base.py", line >>> 1953, in cov >>> ? ?return (dot(X, X.T.conj()) / fact).squeeze() >>> -- >>> View this message in context: >>> >>> http://old.nabble.com/ValueError%3A-objects-are-not-aligned-tp29641661p29641661.html >>> Sent from the Numpy-discussion mailing list archive at Nabble.com. >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From bsouthey at gmail.com Wed Sep 8 09:39:56 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Wed, 08 Sep 2010 08:39:56 -0500 Subject: [Numpy-discussion] ValueError: objects are not aligned In-Reply-To: References: <29641661.post@talk.nabble.com> <642649.38327.qm@web31008.mail.mud.yahoo.com> <650099.57405.qm@web31007.mail.mud.yahoo.com> Message-ID: <4C87922C.4020405@gmail.com> On 09/08/2010 08:30 AM, josef.pktd at gmail.com wrote: > On Wed, Sep 8, 2010 at 9:21 AM, Rui DaCosta wrote: >> Thanks again, >> >> I was trying to >> follow http://www.scipy.org/Numpy_Example_List#head-779283aaa3cc20a3786ad33c2ee1fee9d68a4a53 >> but with the key difference being, that instead of the data for the three >> arrays being constructed from literals, that i'd be sourcing from a >> relational database. The data is quite simple, in that I have 199 recorded >> days for 8 different variables, but am just trying to do a proof-of-concept >> first with three, when I had this problem. >> The functionality I am seeking to reproduce is in Excel as a 3x3 grid: >> =CORREL(rf1,rf1) >> =CORREL(rf1,rf2) >> =CORREL(rf1,rf3) >> =CORREL(rf2,rf1) >> =CORREL(rf2,rf2) >> =CORREL(rf2,rf3) >> =CORREL(rf3,rf1) >> =CORREL(rf3,rf2) >> =CORREL(rf3,rf3) >> I tried to simplify and isolate the problem, and hence the confusion. >> So In sourcing from the DB, I have 3 different cursors (taking a subset of 3 >> of the 8), each 199 rows in length. >> From reading the lists and blogs, the way to put these into numpy seemed to >> be: >> rf1 = np.asarray(cursor1.fetchall(), dtype=np.float64) >> rf2 = np.asarray(cursor2.fetchall(), dtype=np.float64) >> rf3 = np.asarray(cursor3.fetchall(), dtype=np.float64) >> Then to try and produce a correlation matrix: >> print (np.corrcoef([rf1, rf2, rf3])) >> But this is where I get the original error. >> From looking at column_stack, this doesn't seem like what I want to achieve, >> or am I looking at this the wrong way somehow?. > column_stack and rowvar=0 should work, but it might be easier to get > rid of the extra dimension that the cursor gives you > > e.g. > rf1 = np.asarray(cursor1.fetchall(), dtype=np.float64).ravel() > > depends on whether you want to work with 2d or 1d data arrays > > Josef > >> Thanks again, >> Rui >> >> ________________________________ >> From: "josef.pktd at gmail.com" >> To: Discussion of Numerical Python >> Sent: Wed, September 8, 2010 2:34:42 PM >> Subject: Re: [Numpy-discussion] ValueError: objects are not aligned >> >> On Wed, Sep 8, 2010 at 8:08 AM, Rui DaCosta wrote: >>> Thanks for your assistance, >>> I was following this example: >>> >>> http://www.scipy.org/Numpy_Example_List#head-779283aaa3cc20a3786ad33c2ee1fee9d68a4a53 >> I didn't know, >> If arrays are 1d, then numpy does the stacking for you., but it >> doesn't work with column-vectors or 2d arrays. >>>>> np.corrcoef([np.arange(3), np.arange(3)], rowvar=1) >> array([[ 1., 1.], >> [ 1., 1.]]) >>>>> np.corrcoef([np.arange(3), np.arange(3)], rowvar=0) >> array([[ NaN, NaN, NaN], >> [ NaN, NaN, NaN], >> [ NaN, NaN, NaN]]) >>> I intend to use different arrays once I have resolved this problem. >>> Trying your suggestion: >>> ... >>> print (rf1) >>> print (np.corrcoef(rf1, rf1, rowvar=0)) >>> results in: >>> [[-0.00641625] >>> [-0.00498411] >>> [-0.0038015 ]] >>> [[ 1. 1.] >>> [ 1. 1.]] >>> >>> which should have been a 3x3 matrix >> I'm not sure what you want here, you only have three values >> variance would also be zero, and corrcoef wouldn't be defined if you >> only have 1 observation per variable >> >>>>> np.corrcoef(np.arange(3)[:,None]) >> array([[ NaN, NaN, NaN], >> [ NaN, NaN, NaN], >> [ NaN, NaN, NaN]]) >> >>>>> np.corrcoef(np.column_stack([np.arange(5), np.arange(5)**2, >>>>> np.arange(5)**3]), rowvar=0) >> array([[ 1. , 0.9589266 , 0.90588235], >> [ 0.9589266 , 1. , 0.98713033], >> [ 0.90588235, 0.98713033, 1. ]]) >> >> examples are in docstring for np.cov >> >> >> >> >>> as for "stacking rf1 rf1 into a single array first" how would I do this? >>>>> np.column_stack([np.arange(3)[:,None], np.arange(3)[:,None]]) >> array([[0, 0], >> [1, 1], >> [2, 2]]) >>>>> np.column_stack([np.arange(3)[:,None], np.arange(3)]) #doesn't required >>>>> 2d >> array([[0, 0], >> [1, 1], >> [2, 2]]) >> >>>>> np.c_[np.arange(3)[:,None], np.arange(3)[:,None]] >> array([[0, 0], >> [1, 1], >> [2, 2]]) >>>>> np.hstack([np.arange(3)[:,None], np.arange(3)[:,None]]) >> array([[0, 0], >> [1, 1], >> [2, 2]]) >> >> >> Josef >> >>> Regards, >>> Rui >>> ________________________________ >>> From: "josef.pktd at gmail.com" >>> To: Discussion of Numerical Python >>> Sent: Wed, September 8, 2010 1:09:46 PM >>> Subject: Re: [Numpy-discussion] ValueError: objects are not aligned >>> >>> On Wed, Sep 8, 2010 at 5:42 AM, RuiDC wrote: >>>> I'm getting this error, which must be a simple error in getting the >>>> result >>>> from the cursor in the right shape, how do I get the cursor into the >>>> array >>>> as a single dimension?: >>>> >>>> import numpy as np >>>> import sqlite3 >>>> conn = sqlite3.connect("simpledb") >>>> cursor1 = conn.execute("select LogReturn from tblReturns limit 3") >>>> rf1 = np.asarray(cursor1.fetchall(), dtype=np.float64) >>>> print (rf1) >>>> print (np.corrcoef([rf1, rf1])) >>> numpy.corrcoef(x, y=None, rowvar=1, bias=0) >>> >>> try >>> np.corrcoef(rf1, rf1, rowvar=0) no [ ] >>> or >>> stacking rf1 rf1 into a single array first >>> >>> Josef >>> >>>> result: >>>> [[-0.00641625] >>>> [-0.00498411] >>>> [-0.0038015 ]] >>>> Traceback (most recent call last): >>>> File "C:\Documents and >>>> Settings\rui\workspace\numpytest\src\numpytest.py", >>>> line 38, in >>>> print (np.corrcoef([rf1, rf1])) >>>> File "C:\Python26\lib\site-packages\numpy\lib\function_base.py", line >>>> 2003, in corrcoef >>>> c = cov(x, y, rowvar, bias, ddof) >>>> File "C:\Python26\lib\site-packages\numpy\lib\function_base.py", line >>>> 1953, in cov >>>> return (dot(X, X.T.conj()) / fact).squeeze() >>>> -- >>>> View this message in context: >>>> >>>> http://old.nabble.com/ValueError%3A-objects-are-not-aligned-tp29641661p29641661.html >>>> Sent from the Numpy-discussion mailing list archive at Nabble.com. >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion Your errors suggest that you are not getting what expect from the database query. Please check the dtype and shape of your arrays created from the database query. Then check the contents to ensure the values make sense especially for unexpected values. You should be able to find the mean and variance of each of your arrays. Of course, providing the actual data from the query and/or the arrays is also a plus. Bruce From ruidc at yahoo.com Wed Sep 8 10:51:30 2010 From: ruidc at yahoo.com (Rui DaCosta) Date: Wed, 8 Sep 2010 07:51:30 -0700 (PDT) Subject: [Numpy-discussion] ValueError: objects are not aligned In-Reply-To: References: <29641661.post@talk.nabble.com> <642649.38327.qm@web31008.mail.mud.yahoo.com> <650099.57405.qm@web31007.mail.mud.yahoo.com> Message-ID: <352501.49369.qm@web31008.mail.mud.yahoo.com> rf1 = np.asarray(cursor1.fetchall(), dtype=np.float64).ravel() did the trick, 1d is perfect Thankyou very much! Rui ________________________________ From: "josef.pktd at gmail.com" To: Discussion of Numerical Python Sent: Wed, September 8, 2010 3:30:54 PM Subject: Re: [Numpy-discussion] ValueError: objects are not aligned On Wed, Sep 8, 2010 at 9:21 AM, Rui DaCosta wrote: > Thanks again, > > I was trying to >follow http://www.scipy.org/Numpy_Example_List#head-779283aaa3cc20a3786ad33c2ee1fee9d68a4a53 >3 > but with the key difference being, that instead of the data for the three > arrays being constructed from literals, that i'd be sourcing from a > relational database. The data is quite simple, in that I have 199 recorded > days for 8 different variables, but am just trying to do a proof-of-concept > first with three, when I had this problem. > The functionality I am seeking to reproduce is in Excel as a 3x3 grid: > =CORREL(rf1,rf1) > =CORREL(rf1,rf2) > =CORREL(rf1,rf3) > =CORREL(rf2,rf1) > =CORREL(rf2,rf2) > =CORREL(rf2,rf3) > =CORREL(rf3,rf1) > =CORREL(rf3,rf2) > =CORREL(rf3,rf3) > I tried to simplify and isolate the problem, and hence the confusion. > So In sourcing from the DB, I have 3 different cursors (taking a subset of 3 > of the 8), each 199 rows in length. > From reading the lists and blogs, the way to put these into numpy seemed to > be: > rf1 = np.asarray(cursor1.fetchall(), dtype=np.float64) > rf2 = np.asarray(cursor2.fetchall(), dtype=np.float64) > rf3 = np.asarray(cursor3.fetchall(), dtype=np.float64) > Then to try and produce a correlation matrix: > print (np.corrcoef([rf1, rf2, rf3])) > But this is where I get the original error. > From looking at column_stack, this doesn't seem like what I want to achieve, > or am I looking at this the wrong way somehow?. column_stack and rowvar=0 should work, but it might be easier to get rid of the extra dimension that the cursor gives you e.g. rf1 = np.asarray(cursor1.fetchall(), dtype=np.float64).ravel() depends on whether you want to work with 2d or 1d data arrays Josef > Thanks again, > Rui > > ________________________________ > From: "josef.pktd at gmail.com" > To: Discussion of Numerical Python > Sent: Wed, September 8, 2010 2:34:42 PM > Subject: Re: [Numpy-discussion] ValueError: objects are not aligned > > On Wed, Sep 8, 2010 at 8:08 AM, Rui DaCosta wrote: >> Thanks for your assistance, >> I was following this example: >> >>http://www.scipy.org/Numpy_Example_List#head-779283aaa3cc20a3786ad33c2ee1fee9d68a4a53 >>3 > > I didn't know, > If arrays are 1d, then numpy does the stacking for you., but it > doesn't work with column-vectors or 2d arrays. >>>> np.corrcoef([np.arange(3), np.arange(3)], rowvar=1) > array([[ 1., 1.], > [ 1., 1.]]) >>>> np.corrcoef([np.arange(3), np.arange(3)], rowvar=0) > array([[ NaN, NaN, NaN], > [ NaN, NaN, NaN], > [ NaN, NaN, NaN]]) >> >> I intend to use different arrays once I have resolved this problem. >> Trying your suggestion: >> ... >> print (rf1) >> print (np.corrcoef(rf1, rf1, rowvar=0)) >> results in: >> [[-0.00641625] >> [-0.00498411] >> [-0.0038015 ]] >> [[ 1. 1.] >> [ 1. 1.]] >> >> which should have been a 3x3 matrix > > I'm not sure what you want here, you only have three values > variance would also be zero, and corrcoef wouldn't be defined if you > only have 1 observation per variable > >>>> np.corrcoef(np.arange(3)[:,None]) > array([[ NaN, NaN, NaN], > [ NaN, NaN, NaN], > [ NaN, NaN, NaN]]) > >>>> np.corrcoef(np.column_stack([np.arange(5), np.arange(5)**2, >>>> np.arange(5)**3]), rowvar=0) > array([[ 1. , 0.9589266 , 0.90588235], > [ 0.9589266 , 1. , 0.98713033], > [ 0.90588235, 0.98713033, 1. ]]) > > examples are in docstring for np.cov > > > > >> as for "stacking rf1 rf1 into a single array first" how would I do this? > >>>> np.column_stack([np.arange(3)[:,None], np.arange(3)[:,None]]) > array([[0, 0], > [1, 1], > [2, 2]]) >>>> np.column_stack([np.arange(3)[:,None], np.arange(3)]) #doesn't required >>>> 2d > array([[0, 0], > [1, 1], > [2, 2]]) > >>>> np.c_[np.arange(3)[:,None], np.arange(3)[:,None]] > array([[0, 0], > [1, 1], > [2, 2]]) >>>> np.hstack([np.arange(3)[:,None], np.arange(3)[:,None]]) > array([[0, 0], > [1, 1], > [2, 2]]) > > > Josef > >> Regards, >> Rui >> ________________________________ >> From: "josef.pktd at gmail.com" >> To: Discussion of Numerical Python >> Sent: Wed, September 8, 2010 1:09:46 PM >> Subject: Re: [Numpy-discussion] ValueError: objects are not aligned >> >> On Wed, Sep 8, 2010 at 5:42 AM, RuiDC wrote: >>> >>> I'm getting this error, which must be a simple error in getting the >>> result >>> from the cursor in the right shape, how do I get the cursor into the >>> array >>> as a single dimension?: >>> >>> import numpy as np >>> import sqlite3 >>> conn = sqlite3.connect("simpledb") >>> cursor1 = conn.execute("select LogReturn from tblReturns limit 3") >>> rf1 = np.asarray(cursor1.fetchall(), dtype=np.float64) >>> print (rf1) >>> print (np.corrcoef([rf1, rf1])) >> >> numpy.corrcoef(x, y=None, rowvar=1, bias=0) >> >> try >> np.corrcoef(rf1, rf1, rowvar=0) no [ ] >> or >> stacking rf1 rf1 into a single array first >> >> Josef >> >>> >>> result: >>> [[-0.00641625] >>> [-0.00498411] >>> [-0.0038015 ]] >>> Traceback (most recent call last): >>> File "C:\Documents and >>> Settings\rui\workspace\numpytest\src\numpytest.py", >>> line 38, in >>> print (np.corrcoef([rf1, rf1])) >>> File "C:\Python26\lib\site-packages\numpy\lib\function_base.py", line >>> 2003, in corrcoef >>> c = cov(x, y, rowvar, bias, ddof) >>> File "C:\Python26\lib\site-packages\numpy\lib\function_base.py", line >>> 1953, in cov >>> return (dot(X, X.T.conj()) / fact).squeeze() >>> -- >>> View this message in context: >>> >>>http://old.nabble.com/ValueError%3A-objects-are-not-aligned-tp29641661p29641661.html >>>l >>> Sent from the Numpy-discussion mailing list archive at Nabble.com. >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Wed Sep 8 10:53:20 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 08 Sep 2010 10:53:20 -0400 Subject: [Numpy-discussion] vectorize doesn't like functools.partial References: Message-ID: josef.pktd at gmail.com wrote: > On Wed, Sep 8, 2010 at 7:44 AM, Neal Becker wrote: >> If I try to use vectorize on the result of functools.partial, I seem to >> get: >> >> ValueError: failed to determine the number of arguments for >> >> >> Anything I can do about it? > > Set .nin (attribute of vectorized function, I think) directly with > number of arguments (int) > > Josef > >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> Not sure what you mean. Here is a demo of 2 problems of interop of numpy and functools.partial. Tried both vectorize and frompyfunc. vectorize gives: ValueError: failed to determine the number of arguments for frompyfunc gives: TypeError: H() got multiple values for keyword argument 'iir' Note that doc for frompyfunc says: "Takes an arbitrary Python function..." import numpy as np class iir (object): def H (z): return 1 def H (iir, f): z = complex (cos (2*pi*f/fs), sin (2*pi*f/fs)) return iir.H (z) f = np.linspace (0, 10e6, 1000) #bug1 from functools import partial the_iir = iir() p = partial (H, iir=the_iir) resp_pll = np.vectorize (p)(f) #bug2 resp_pll = np.frompyfunc (p, 1, 1)(f) From josef.pktd at gmail.com Wed Sep 8 11:12:37 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 8 Sep 2010 11:12:37 -0400 Subject: [Numpy-discussion] vectorize doesn't like functools.partial In-Reply-To: References: Message-ID: On Wed, Sep 8, 2010 at 10:53 AM, Neal Becker wrote: > josef.pktd at gmail.com wrote: > >> On Wed, Sep 8, 2010 at 7:44 AM, Neal Becker wrote: >>> If I ?try to use vectorize on the result of functools.partial, I seem to >>> get: >>> >>> ValueError: failed to determine the number of arguments for >>> >>> >>> Anything I can do about it? >> >> Set .nin (attribute of vectorized function, I think) directly with >> number of arguments (int) >> >> Josef >> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> > Not sure what you mean. ?Here is a demo of 2 problems of interop of numpy > and functools.partial. ?Tried both vectorize and frompyfunc. ?vectorize > gives: > ValueError: failed to determine the number of arguments for > > > frompyfunc gives: > TypeError: H() got multiple values for keyword argument 'iir' > > Note that doc for frompyfunc says: > "Takes an arbitrary Python function..." > > import numpy as np > > class iir (object): > ? ?def H (z): > ? ? ? ?return 1 > > def H (iir, f): > ? ?z = complex (cos (2*pi*f/fs), sin (2*pi*f/fs)) > ? ?return iir.H (z) > > f = np.linspace (0, 10e6, 1000) > #bug1 > from functools import partial > the_iir = iir() > p = partial (H, iir=the_iir) > resp_pll = np.vectorize (p)(f) > #bug2 > resp_pll = np.frompyfunc (p, 1, 1)(f) looks like a limitation of vectorize (fails in constructor), and nin cannot be included in the constructor this should be a bug report or enhancement ticket a work around, after fixing other problems in your code example: p = partial(H, iir=the_iir) fun = np.vectorize(lambda x: p(x)) >>> fun.nin 1 >>> fun(5) array(1) >>> fun(f)[:10] array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) Josef > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ndbecker2 at gmail.com Wed Sep 8 11:17:17 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 08 Sep 2010 11:17:17 -0400 Subject: [Numpy-discussion] vectorize doesn't like functools.partial References: Message-ID: josef.pktd at gmail.com wrote: > On Wed, Sep 8, 2010 at 10:53 AM, Neal Becker wrote: >> josef.pktd at gmail.com wrote: >> >>> On Wed, Sep 8, 2010 at 7:44 AM, Neal Becker wrote: >>>> If I try to use vectorize on the result of functools.partial, I seem >>>> to get: >>>> >>>> ValueError: failed to determine the number of arguments for >>>> >>>> >>>> Anything I can do about it? >>> >>> Set .nin (attribute of vectorized function, I think) directly with >>> number of arguments (int) >>> >>> Josef >>> >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >> Not sure what you mean. Here is a demo of 2 problems of interop of numpy >> and functools.partial. Tried both vectorize and frompyfunc. vectorize >> gives: >> ValueError: failed to determine the number of arguments for >> >> >> frompyfunc gives: >> TypeError: H() got multiple values for keyword argument 'iir' >> >> Note that doc for frompyfunc says: >> "Takes an arbitrary Python function..." >> >> import numpy as np >> >> class iir (object): >> def H (z): >> return 1 >> >> def H (iir, f): >> z = complex (cos (2*pi*f/fs), sin (2*pi*f/fs)) >> return iir.H (z) >> >> f = np.linspace (0, 10e6, 1000) >> #bug1 >> from functools import partial >> the_iir = iir() >> p = partial (H, iir=the_iir) >> resp_pll = np.vectorize (p)(f) >> #bug2 >> resp_pll = np.frompyfunc (p, 1, 1)(f) > > looks like a limitation of vectorize (fails in constructor), and nin > cannot be included in the constructor > > this should be a bug report or enhancement ticket > > a work around, after fixing other problems in your code example: > > p = partial(H, iir=the_iir) > fun = np.vectorize(lambda x: p(x)) > >>>> fun.nin > 1 >>>> fun(5) > array(1) >>>> fun(f)[:10] > array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) > > Josef Yes, I found that using lamda there is no need for partial at all. fun = np.vectorize (lambda f: H(the_iir, f)) From michael.s.gilbert at gmail.com Wed Sep 8 11:26:11 2010 From: michael.s.gilbert at gmail.com (Michael Gilbert) Date: Wed, 8 Sep 2010 11:26:11 -0400 Subject: [Numpy-discussion] IEEE 754-2008 decimal floating point support Message-ID: <20100908112611.40fd69d1.michael.s.gilbert@gmail.com> Hi, Are there any plans to add support for decimal floating point arithmetic, as defined in the 2008 revision of the IEEE 754 standard [0], in numpy? Thanks for any info. Best wishes, Mike [0] http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4610935&tag=1 From robert.kern at gmail.com Wed Sep 8 11:34:41 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 8 Sep 2010 10:34:41 -0500 Subject: [Numpy-discussion] IEEE 754-2008 decimal floating point support In-Reply-To: <20100908112611.40fd69d1.michael.s.gilbert@gmail.com> References: <20100908112611.40fd69d1.michael.s.gilbert@gmail.com> Message-ID: On Wed, Sep 8, 2010 at 10:26, Michael Gilbert wrote: > Hi, > > Are there any plans to add support for decimal floating point > arithmetic, as defined in the 2008 revision of the IEEE 754 standard > [0], in numpy? No, there are no plans. Although IEEE 754-2008 defines the format and semantics of such numbers, they do not provide an implementation. If someone wishes to make a C implementation and integrate that into numpy as a new dtype, I think that would be welcome. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From charlesr.harris at gmail.com Wed Sep 8 11:43:56 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 8 Sep 2010 09:43:56 -0600 Subject: [Numpy-discussion] IEEE 754-2008 decimal floating point support In-Reply-To: <20100908112611.40fd69d1.michael.s.gilbert@gmail.com> References: <20100908112611.40fd69d1.michael.s.gilbert@gmail.com> Message-ID: On Wed, Sep 8, 2010 at 9:26 AM, Michael Gilbert wrote: > Hi, > > Are there any plans to add support for decimal floating point > arithmetic, as defined in the 2008 revision of the IEEE 754 standard > [0], in numpy? > > Not at the moment. There is currently no hardware or C support and adding new types to numpy isn't trivial. You can get some limited Decimal functionality by using python classes and object arrays, for instance the Decimal class in the python decimal module, but the performance isn't great. What is your particular interest in decimal support? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.s.gilbert at gmail.com Wed Sep 8 11:46:13 2010 From: michael.s.gilbert at gmail.com (Michael Gilbert) Date: Wed, 8 Sep 2010 11:46:13 -0400 Subject: [Numpy-discussion] IEEE 754-2008 decimal floating point support In-Reply-To: References: <20100908112611.40fd69d1.michael.s.gilbert@gmail.com> Message-ID: <20100908114613.89c3e2fb.michael.s.gilbert@gmail.com> On Wed, 8 Sep 2010 09:43:56 -0600, Charles R Harris wrote: > On Wed, Sep 8, 2010 at 9:26 AM, Michael Gilbert > wrote: > > > Hi, > > > > Are there any plans to add support for decimal floating point > > arithmetic, as defined in the 2008 revision of the IEEE 754 standard > > [0], in numpy? > > > > > Not at the moment. There is currently no hardware or C support and adding > new types to numpy isn't trivial. You can get some limited Decimal > functionality by using python classes and object arrays, for instance the > Decimal class in the python decimal module, but the performance isn't great. > > What is your particular interest in decimal support? Primarily avoiding catastrophic cancellation when subtracting large values. I was planning to use the decimal class, but was curious whether support for the IEEE standard was coming any time soon. Thanks, Mike From cournape at gmail.com Wed Sep 8 11:56:36 2010 From: cournape at gmail.com (David Cournapeau) Date: Thu, 9 Sep 2010 00:56:36 +0900 Subject: [Numpy-discussion] IEEE 754-2008 decimal floating point support In-Reply-To: References: <20100908112611.40fd69d1.michael.s.gilbert@gmail.com> Message-ID: On Thu, Sep 9, 2010 at 12:43 AM, Charles R Harris wrote: > > > On Wed, Sep 8, 2010 at 9:26 AM, Michael Gilbert > wrote: >> >> Hi, >> >> Are there any plans to add support for decimal floating point >> arithmetic, as defined in the 2008 revision of the IEEE 754 standard >> [0], in numpy? >> > > Not at the moment. There is currently no hardware Strictly speaking, there is hardware support on some high end hardware (power6, but I have no idea how this is available through common languages, like C or Fortran. I guess this would be compiler specific. A software implementation would be much more interesting at this stage, as I guess it will takes years before support comes to commodity CPUs. cheers, David From charlesr.harris at gmail.com Wed Sep 8 12:23:28 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 8 Sep 2010 10:23:28 -0600 Subject: [Numpy-discussion] IEEE 754-2008 decimal floating point support In-Reply-To: <20100908114613.89c3e2fb.michael.s.gilbert@gmail.com> References: <20100908112611.40fd69d1.michael.s.gilbert@gmail.com> <20100908114613.89c3e2fb.michael.s.gilbert@gmail.com> Message-ID: On Wed, Sep 8, 2010 at 9:46 AM, Michael Gilbert wrote: > On Wed, 8 Sep 2010 09:43:56 -0600, Charles R Harris wrote: > > On Wed, Sep 8, 2010 at 9:26 AM, Michael Gilbert < > michael.s.gilbert at gmail.com > > > wrote: > > > > > Hi, > > > > > > Are there any plans to add support for decimal floating point > > > arithmetic, as defined in the 2008 revision of the IEEE 754 standard > > > [0], in numpy? > > > > > > > > Not at the moment. There is currently no hardware or C support and adding > > new types to numpy isn't trivial. You can get some limited Decimal > > functionality by using python classes and object arrays, for instance the > > Decimal class in the python decimal module, but the performance isn't > great. > > > > What is your particular interest in decimal support? > > Primarily avoiding catastrophic cancellation when subtracting > large values. I was planning to use the decimal class, but was > curious whether support for the IEEE standard was coming any time soon. > > If you just need more precision, mpmath has better performance than the Decimal class. Also, it might be possible to avoid the loss of precision by changing the computation, but I don't know the details of what you are doing. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From eadrogue at gmx.net Wed Sep 8 15:21:49 2010 From: eadrogue at gmx.net (Ernest =?iso-8859-1?Q?Adrogu=E9?=) Date: Wed, 8 Sep 2010 21:21:49 +0200 Subject: [Numpy-discussion] indexing with booleans without making a copy? Message-ID: <20100908192149.GA11838@doriath.local> Hello, I have a sorted, flat array: In [139]: a =np.array([0,1,2,2,2,3]) Basically, I want views of the areas where there are repeated numbers (since the array is sorted, they will be contiguous). But, of course, to find the numbers that are repeated I have to use comparison operations that return boolean arrays, so I suppose the problem is converting the boolean array into a slice. This is what I have come up with: In [146]: np.flatnonzero(a==2) Out[146]: array([2, 3, 4]) In [147]: b=np.flatnonzero(a==2) In [148]: b.min() Out[148]: 2 In [149]: b.max() Out[149]: 4 In [150]: a[b.min():b.max()+1] Out[150]: array([2, 2, 2]) If you know a more straightforward way of doing this, I'll be glad to know... Cheers :) Ernest From aarchiba at physics.mcgill.ca Wed Sep 8 15:35:18 2010 From: aarchiba at physics.mcgill.ca (Anne Archibald) Date: Wed, 8 Sep 2010 15:35:18 -0400 Subject: [Numpy-discussion] indexing with booleans without making a copy? In-Reply-To: <20100908192149.GA11838@doriath.local> References: <20100908192149.GA11838@doriath.local> Message-ID: 2010/9/8 Ernest Adrogu? : > I have a sorted, flat array: > > In [139]: a =np.array([0,1,2,2,2,3]) > > Basically, I want views of the areas where there > are repeated numbers (since the array is sorted, they > will be contiguous). > > But, of course, to find the numbers that are repeated > I have to use comparison operations that return > boolean arrays, so I suppose the problem is converting > the boolean array into a slice. Well, you're going to have to do some allocation, but how's this? Use unique1d to get an array of unique values. Then use searchsorted twice, once to find the leftmost end of each hunk, and once to find the rightmost end of each hunk. Anne From josef.pktd at gmail.com Wed Sep 8 15:36:31 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 8 Sep 2010 15:36:31 -0400 Subject: [Numpy-discussion] indexing with booleans without making a copy? In-Reply-To: <20100908192149.GA11838@doriath.local> References: <20100908192149.GA11838@doriath.local> Message-ID: 2010/9/8 Ernest Adrogu? : > Hello, > > I have a sorted, flat array: > > In [139]: a =np.array([0,1,2,2,2,3]) > > Basically, I want views of the areas where there > are repeated numbers (since the array is sorted, they > will be contiguous). > > But, of course, to find the numbers that are repeated > I have to use comparison operations that return > boolean arrays, so I suppose the problem is converting > the boolean array into a slice. > > This is what I have come up with: > > In [146]: np.flatnonzero(a==2) > Out[146]: array([2, 3, 4]) > > In [147]: b=np.flatnonzero(a==2) > > In [148]: b.min() > Out[148]: 2 > > In [149]: b.max() > Out[149]: 4 > > In [150]: a[b.min():b.max()+1] > Out[150]: array([2, 2, 2]) > > If you know a more straightforward way of doing this, > I'll be glad to know... np.nonzero(np.diff(a)) add boundaries index zero and end make two arrays one with starting points, one with endpoints. Josef > > Cheers :) > Ernest > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ceball at gmail.com Wed Sep 8 15:42:25 2010 From: ceball at gmail.com (Chris Ball) Date: Wed, 8 Sep 2010 19:42:25 +0000 (UTC) Subject: [Numpy-discussion] clip() with None as argument References: Message-ID: Robert Kern gmail.com> writes: > > On Tue, Sep 7, 2010 at 15:12, Friedrich Romstedt > gmail.com> wrote: > > Ah, no need to answer, I do this myself: > > > > Friedrich, would you please use numpy.inf and -numpy.inf. > > But if you have an integer array, you will run into the same problem. > The result will be upcast to float. I think we would accept a patch > that interprets None to be the appropriate extreme bound given the > input datatype. This will be tricky, though, since it must be done in > C (PyArray_Clip in numpy/core/src/multiarray/calculation.c). I've been a bit confused about what numpy's clip is supposed to support. The documentation (e.g. http://docs.scipy.org/doc/numpy/reference/generated/numpy.clip.html) doesn't make any mention of supporting only one of min or max, but I remember a message some time back from Travis Oliphant that seemed to suggest it does (http://thread.gmane.org/gmane.comp.python.numeric.general/17844/focus=17877). Right now, I'm surprised by two aspect's of clip's behavior, both demonstrated below... $ python Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41) [GCC 4.4.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.__version__ '2.0.0.dev8698' >>> a = numpy.array([1,2,3,4,5]) >>> a.clip(2,None) array([2, 2, 2, 2, 2], dtype=object) I'm not sure why the returned array has a dtype of object (although that can be avoided by using clip's "out" argument), and also why the values are not like this: >>> numpy.maximum(a,2) array([2, 2, 3, 4, 5]) I've been using minimum() and maximum() instead of clip() to have "clip upper" and "clip lower" functions, so it's not really a problem for me, I'm just curious (and thought maybe this could be useful for others searching in the future). Thanks, Chris From michael.s.gilbert at gmail.com Wed Sep 8 15:44:02 2010 From: michael.s.gilbert at gmail.com (Michael Gilbert) Date: Wed, 8 Sep 2010 15:44:02 -0400 Subject: [Numpy-discussion] IEEE 754-2008 decimal floating point support In-Reply-To: References: <20100908112611.40fd69d1.michael.s.gilbert@gmail.com> <20100908114613.89c3e2fb.michael.s.gilbert@gmail.com> Message-ID: On Wed, Sep 8, 2010 at 12:23 PM, Charles R Harris wrote: > > > On Wed, Sep 8, 2010 at 9:46 AM, Michael Gilbert > wrote: >> >> On Wed, 8 Sep 2010 09:43:56 -0600, Charles R Harris wrote: >> > On Wed, Sep 8, 2010 at 9:26 AM, Michael Gilbert >> > > > > wrote: >> > >> > > Hi, >> > > >> > > Are there any plans to add support for decimal floating point >> > > arithmetic, as defined in the 2008 revision of the IEEE 754 standard >> > > [0], in numpy? >> > > >> > > >> > Not at the moment. There is currently no hardware or C support and >> > adding >> > new types to numpy isn't trivial. You can get some limited Decimal >> > functionality by using python classes and object arrays, for instance >> > the >> > Decimal class in the python decimal module, but the performance isn't >> > great. >> > >> > What is your particular interest in decimal support? >> >> Primarily avoiding catastrophic cancellation when subtracting >> large values. ?I was planning to use the decimal class, but was >> curious whether support for the IEEE standard was coming any time soon. >> > > If you just need more precision, mpmath has better performance than the > Decimal class. Also, it might be possible to avoid the loss of precision by > changing the computation, but I don't know the details of what you are > doing. Just wanted to say that numpy object arrays + decimal solved all of my problems, which were all caused by the disconnect between decimal and binary representation of floating point numbers. It would be really nice to have IEEE decimal floating point since python's decimal module is a bit slow, but I can see thats going to be a very long process. Thanks again for your help! Mike From eadrogue at gmx.net Wed Sep 8 16:00:38 2010 From: eadrogue at gmx.net (Ernest =?iso-8859-1?Q?Adrogu=E9?=) Date: Wed, 8 Sep 2010 22:00:38 +0200 Subject: [Numpy-discussion] indexing with booleans without making a copy? In-Reply-To: References: <20100908192149.GA11838@doriath.local> Message-ID: <20100908200038.GA11927@doriath.local> 8/09/10 @ 15:35 (-0400), thus spake Anne Archibald: > 2010/9/8 Ernest Adrogu? : > > I have a sorted, flat array: > > > > In [139]: a =np.array([0,1,2,2,2,3]) > > > > Basically, I want views of the areas where there > > are repeated numbers (since the array is sorted, they > > will be contiguous). > > > > But, of course, to find the numbers that are repeated > > I have to use comparison operations that return > > boolean arrays, so I suppose the problem is converting > > the boolean array into a slice. > > Well, you're going to have to do some allocation, but how's this? Use > unique1d to get an array of unique values. Then use searchsorted > twice, once to find the leftmost end of each hunk, and once to find > the rightmost end of each hunk. I like it. Thanks :) > > Anne > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From robert.kern at gmail.com Wed Sep 8 16:04:17 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 8 Sep 2010 15:04:17 -0500 Subject: [Numpy-discussion] IEEE 754-2008 decimal floating point support In-Reply-To: References: <20100908112611.40fd69d1.michael.s.gilbert@gmail.com> <20100908114613.89c3e2fb.michael.s.gilbert@gmail.com> Message-ID: On Wed, Sep 8, 2010 at 14:44, Michael Gilbert wrote: > On Wed, Sep 8, 2010 at 12:23 PM, Charles R Harris wrote: >> >> >> On Wed, Sep 8, 2010 at 9:46 AM, Michael Gilbert >> wrote: >>> >>> On Wed, 8 Sep 2010 09:43:56 -0600, Charles R Harris wrote: >>> > On Wed, Sep 8, 2010 at 9:26 AM, Michael Gilbert >>> > >> > > wrote: >>> > >>> > > Hi, >>> > > >>> > > Are there any plans to add support for decimal floating point >>> > > arithmetic, as defined in the 2008 revision of the IEEE 754 standard >>> > > [0], in numpy? >>> > > >>> > > >>> > Not at the moment. There is currently no hardware or C support and >>> > adding >>> > new types to numpy isn't trivial. You can get some limited Decimal >>> > functionality by using python classes and object arrays, for instance >>> > the >>> > Decimal class in the python decimal module, but the performance isn't >>> > great. >>> > >>> > What is your particular interest in decimal support? >>> >>> Primarily avoiding catastrophic cancellation when subtracting >>> large values. ?I was planning to use the decimal class, but was >>> curious whether support for the IEEE standard was coming any time soon. >>> >> >> If you just need more precision, mpmath has better performance than the >> Decimal class. Also, it might be possible to avoid the loss of precision by >> changing the computation, but I don't know the details of what you are >> doing. > > Just wanted to say that numpy object arrays + decimal solved all of my > problems, which were all caused by the disconnect between decimal and > binary representation of floating point numbers. Are you sure? Unless if I'm failing to think through this properly, catastrophic cancellation for large numbers is an intrinsic property of fixed-precision floating point regardless of the base. decimal and mpmath both help with that problem because they have arbitrary precision. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From michael.s.gilbert at gmail.com Wed Sep 8 16:10:20 2010 From: michael.s.gilbert at gmail.com (Michael Gilbert) Date: Wed, 8 Sep 2010 16:10:20 -0400 Subject: [Numpy-discussion] IEEE 754-2008 decimal floating point support In-Reply-To: References: <20100908112611.40fd69d1.michael.s.gilbert@gmail.com> <20100908114613.89c3e2fb.michael.s.gilbert@gmail.com> Message-ID: <20100908161020.65e4892c.michael.s.gilbert@gmail.com> On Wed, 8 Sep 2010 15:04:17 -0500, Robert Kern wrote: > On Wed, Sep 8, 2010 at 14:44, Michael Gilbert > wrote: > > On Wed, Sep 8, 2010 at 12:23 PM, Charles R Harris wrote: > >> > >> > >> On Wed, Sep 8, 2010 at 9:46 AM, Michael Gilbert > >> wrote: > >>> > >>> On Wed, 8 Sep 2010 09:43:56 -0600, Charles R Harris wrote: > >>> > On Wed, Sep 8, 2010 at 9:26 AM, Michael Gilbert > >>> > >>> > > wrote: > >>> > > >>> > > Hi, > >>> > > > >>> > > Are there any plans to add support for decimal floating point > >>> > > arithmetic, as defined in the 2008 revision of the IEEE 754 standard > >>> > > [0], in numpy? > >>> > > > >>> > > > >>> > Not at the moment. There is currently no hardware or C support and > >>> > adding > >>> > new types to numpy isn't trivial. You can get some limited Decimal > >>> > functionality by using python classes and object arrays, for instance > >>> > the > >>> > Decimal class in the python decimal module, but the performance isn't > >>> > great. > >>> > > >>> > What is your particular interest in decimal support? > >>> > >>> Primarily avoiding catastrophic cancellation when subtracting > >>> large values. ?I was planning to use the decimal class, but was > >>> curious whether support for the IEEE standard was coming any time soon. > >>> > >> > >> If you just need more precision, mpmath has better performance than the > >> Decimal class. Also, it might be possible to avoid the loss of precision by > >> changing the computation, but I don't know the details of what you are > >> doing. > > > > Just wanted to say that numpy object arrays + decimal solved all of my > > problems, which were all caused by the disconnect between decimal and > > binary representation of floating point numbers. > > Are you sure? Unless if I'm failing to think through this properly, > catastrophic cancellation for large numbers is an intrinsic property > of fixed-precision floating point regardless of the base. decimal and > mpmath both help with that problem because they have arbitrary > precision. Here is an example: >>> 0.3/3.0 - 0.1 -1.3877787807814457e-17 >>> mpmath.mpf( '0.3' )/mpmath.mpf( '3.0' ) - mpmath.mpf( '0.1' ) mpf('-1.3877787807814457e-17') >>> decimal.Decimal( '0.3' )/decimal.Decimal( '3.0' ) - decimal.Decimal ( '0.1' ) Decimal("0.0") Decimal solves the problem; whereas mpmath doesn't. Mike From morph at debian.org Wed Sep 8 16:20:30 2010 From: morph at debian.org (Sandro Tosi) Date: Wed, 8 Sep 2010 22:20:30 +0200 Subject: [Numpy-discussion] IEEE 754-2008 decimal floating point support In-Reply-To: <20100908161020.65e4892c.michael.s.gilbert@gmail.com> References: <20100908112611.40fd69d1.michael.s.gilbert@gmail.com> <20100908114613.89c3e2fb.michael.s.gilbert@gmail.com> <20100908161020.65e4892c.michael.s.gilbert@gmail.com> Message-ID: On Wed, Sep 8, 2010 at 22:10, Michael Gilbert wrote: > Here is an example: > > ? >>> 0.3/3.0 - 0.1 > ? -1.3877787807814457e-17 > > ? >>> mpmath.mpf( '0.3' )/mpmath.mpf( '3.0' ) - mpmath.mpf( '0.1' ) > ? mpf('-1.3877787807814457e-17') > > ? >>> decimal.Decimal( '0.3' )/decimal.Decimal( '3.0' ) - decimal.Decimal ( '0.1' ) > ? Decimal("0.0") > > Decimal solves the problem; whereas mpmath doesn't. you can change mpmath precision up to an arbitrary high value: In [4]: mpmath.mp.prec = 100 In [5]: mpmath.mpf( '0.3' )/mpmath.mpf( '3.0' ) - mpmath.mpf( '0.1' ) Out[5]: mpf('0.0') Regards. -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi From michael.s.gilbert at gmail.com Wed Sep 8 16:28:26 2010 From: michael.s.gilbert at gmail.com (Michael Gilbert) Date: Wed, 8 Sep 2010 16:28:26 -0400 Subject: [Numpy-discussion] IEEE 754-2008 decimal floating point support In-Reply-To: References: <20100908112611.40fd69d1.michael.s.gilbert@gmail.com> <20100908114613.89c3e2fb.michael.s.gilbert@gmail.com> <20100908161020.65e4892c.michael.s.gilbert@gmail.com> Message-ID: <20100908162826.1a7152cf.michael.s.gilbert@gmail.com> On Wed, 8 Sep 2010 22:20:30 +0200, Sandro Tosi wrote: > On Wed, Sep 8, 2010 at 22:10, Michael Gilbert > wrote: > > Here is an example: > > > > ? >>> 0.3/3.0 - 0.1 > > ? -1.3877787807814457e-17 > > > > ? >>> mpmath.mpf( '0.3' )/mpmath.mpf( '3.0' ) - mpmath.mpf( '0.1' ) > > ? mpf('-1.3877787807814457e-17') > > > > ? >>> decimal.Decimal( '0.3' )/decimal.Decimal( '3.0' ) - decimal.Decimal ( '0.1' ) > > ? Decimal("0.0") > > > > Decimal solves the problem; whereas mpmath doesn't. > > you can change mpmath precision up to an arbitrary high value: > > In [4]: mpmath.mp.prec = 100 > > In [5]: mpmath.mpf( '0.3' )/mpmath.mpf( '3.0' ) - mpmath.mpf( '0.1' ) > Out[5]: mpf('0.0') Thanks. I already knew that. I tried prec = 500, which did of course get the error down quite a bit to begin with, but even so, it continued to grow and grow. So decimal wins out. Mike From robert.kern at gmail.com Wed Sep 8 16:30:25 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 8 Sep 2010 15:30:25 -0500 Subject: [Numpy-discussion] clip() with None as argument In-Reply-To: References: Message-ID: On Wed, Sep 8, 2010 at 14:42, Chris Ball wrote: > Robert Kern gmail.com> writes: > >> >> On Tue, Sep 7, 2010 at 15:12, Friedrich Romstedt >> gmail.com> wrote: >> > Ah, no need to answer, I do this myself: >> > >> > Friedrich, would you please use numpy.inf and -numpy.inf. >> >> But if you have an integer array, you will run into the same problem. >> The result will be upcast to float. I think we would accept a patch >> that interprets None to be the appropriate extreme bound given the >> input datatype. This will be tricky, though, since it must be done in >> C (PyArray_Clip in numpy/core/src/multiarray/calculation.c). > > I've been a bit confused about what numpy's clip is > supposed to support. The documentation > (e.g. http://docs.scipy.org/doc/numpy/reference/generated/numpy.clip.html) > doesn't make any mention of supporting only one of min or max, but I > remember a message some time back from Travis Oliphant that seemed to > suggest it does > (http://thread.gmane.org/gmane.comp.python.numeric.general/17844/focus=17877). > > Right now, I'm surprised by two aspect's of clip's behavior, both > demonstrated below... > > $ python > Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41) > [GCC 4.4.3] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> import numpy >>>> numpy.__version__ > '2.0.0.dev8698' >>>> a = numpy.array([1,2,3,4,5]) >>>> a.clip(2,None) > array([2, 2, 2, 2, 2], dtype=object) > > I'm not sure why the returned array has a dtype of object The usual type-matching semantics. a.clip() tries to find a common type that fits `a`, 2 and None. Since None only fits in an object dtype, that's what you get. > (although that can be > avoided by using clip's "out" argument), and also > why the values are not like this: > >>>> numpy.maximum(a,2) > array([2, 2, 3, 4, 5]) clip(a, low, high) boils down to this in the C code itself (or at least the code path that it goes down when the input array needs to be upcast): maximum(minimum(a, high), low) Since all integer objects compare > None based on the default Python 2.x implementation of comparison across types, you would just get the low value out of that. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From danelliottster at gmail.com Wed Sep 8 16:32:56 2010 From: danelliottster at gmail.com (Daniel Elliott) Date: Wed, 8 Sep 2010 20:32:56 +0000 (UTC) Subject: [Numpy-discussion] inversion of large matrices References: <54848FA5-AD74-46E9-AE0B-3189FB40C104@cs.toronto.edu> <20A7F4F6-3F2A-47CF-AACE-8484AB92A0E3@cs.toronto.edu> Message-ID: Wow, this is great! Thanks for all the great questions. Sebastian Walter gmail.com> writes: > is it really the covariance matrix you want to invert? Or do you want > to compute something like > x^T C^{-1} x, > where x is an array of size N and C an array of size (N,N)? Yes, this is what I am computing. I am computing the pdf of a very high- dimensional multivariate normal. Is there a specialized method to compute this? > It would also be interesting to know how the covariance matrix gets computed > and what its condition number is, at least approximately. Chuck and Sebastian are on to me. I should look into computing this using the small covariance matrix. I suppose I could even postpone evaluation of this covariance matrix by passing in the data matrix instead of the covariance matrix. What do you guys think? As for the condition number, the matrix will, initially, be ill-conditioned because the dimension of the data will far outnumber the number of data points. However, I am writing this code on the way to writing code that fits a mixture of Gaussians but can use any number of covariance regularization techniques to "fix" the covariance matrices. Basically, I am writing code that does this: http://www.cs.colostate.edu/~dane/papers/ANNIE2010.pdf. I am guessing you math and stats guys might shake your head at my clumsy, computer science ways... :) However, in addition to the two, simple regularization techniques described in that paper, I also want to be able to do things like mixture of factor analyzers and mixture of principal component analyzers and so on by simply supplying different regularization techniques. FYI, I am currently adding this to PyMix. Does this seem like a good package to extend? > On Wed, Sep 1, 2010 at 1:58 AM, Charles R Harris > gmail.com> wrote: > > On Tue, Aug 31, 2010 at 4:52 PM, Dan Elliott gmail.com> > > wrote: > >> David Warde-Farley cs.toronto.edu> writes: > >> > On 2010-08-30, at 10:36 PM, Charles R Harris wrote: > >> > I think he means that if he needs both the determinant and to solve the > >> > system, it might be more efficient to do > >> > the SVD, obtain the determinant from the diagonal values, and obtain the > >> > solution by multiplying by U D^-1 V^T? > >> Thank you, that is what I meant. ?Poorly worded on my part. > >> > >> In particular, I am writing code to invert a very large covariance matrix. > >> ?I > >> think David has some good information in another post in this thread. > >> > > > > Where did the covariance array come from? It may be the case that you can > > use a much smaller one, for instance in PCA of images. > > > > Chuck From robert.kern at gmail.com Wed Sep 8 16:33:29 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 8 Sep 2010 15:33:29 -0500 Subject: [Numpy-discussion] IEEE 754-2008 decimal floating point support In-Reply-To: <20100908161020.65e4892c.michael.s.gilbert@gmail.com> References: <20100908112611.40fd69d1.michael.s.gilbert@gmail.com> <20100908114613.89c3e2fb.michael.s.gilbert@gmail.com> <20100908161020.65e4892c.michael.s.gilbert@gmail.com> Message-ID: On Wed, Sep 8, 2010 at 15:10, Michael Gilbert wrote: > On Wed, 8 Sep 2010 15:04:17 -0500, Robert Kern wrote: >> On Wed, Sep 8, 2010 at 14:44, Michael Gilbert >> wrote: >> > Just wanted to say that numpy object arrays + decimal solved all of my >> > problems, which were all caused by the disconnect between decimal and >> > binary representation of floating point numbers. >> >> Are you sure? Unless if I'm failing to think through this properly, >> catastrophic cancellation for large numbers is an intrinsic property >> of fixed-precision floating point regardless of the base. decimal and >> mpmath both help with that problem because they have arbitrary >> precision. > > Here is an example: > > ? >>> 0.3/3.0 - 0.1 > ? -1.3877787807814457e-17 > > ? >>> mpmath.mpf( '0.3' )/mpmath.mpf( '3.0' ) - mpmath.mpf( '0.1' ) > ? mpf('-1.3877787807814457e-17') > > ? >>> decimal.Decimal( '0.3' )/decimal.Decimal( '3.0' ) - decimal.Decimal ( '0.1' ) > ? Decimal("0.0") > > Decimal solves the problem; whereas mpmath doesn't. Okay, that's not an example of catastrophic cancellation, just a representation issue. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From aarchiba at physics.mcgill.ca Wed Sep 8 16:50:52 2010 From: aarchiba at physics.mcgill.ca (Anne Archibald) Date: Wed, 8 Sep 2010 16:50:52 -0400 Subject: [Numpy-discussion] IEEE 754-2008 decimal floating point support In-Reply-To: References: <20100908112611.40fd69d1.michael.s.gilbert@gmail.com> <20100908114613.89c3e2fb.michael.s.gilbert@gmail.com> <20100908161020.65e4892c.michael.s.gilbert@gmail.com> Message-ID: On 8 September 2010 16:33, Robert Kern wrote: > On Wed, Sep 8, 2010 at 15:10, Michael Gilbert > wrote: >> On Wed, 8 Sep 2010 15:04:17 -0500, Robert Kern wrote: >>> On Wed, Sep 8, 2010 at 14:44, Michael Gilbert >>> wrote: > >>> > Just wanted to say that numpy object arrays + decimal solved all of my >>> > problems, which were all caused by the disconnect between decimal and >>> > binary representation of floating point numbers. >>> >>> Are you sure? Unless if I'm failing to think through this properly, >>> catastrophic cancellation for large numbers is an intrinsic property >>> of fixed-precision floating point regardless of the base. decimal and >>> mpmath both help with that problem because they have arbitrary >>> precision. >> >> Here is an example: >> >> >>> 0.3/3.0 - 0.1 >> -1.3877787807814457e-17 >> >> >>> mpmath.mpf( '0.3' )/mpmath.mpf( '3.0' ) - mpmath.mpf( '0.1' ) >> mpf('-1.3877787807814457e-17') >> >> >>> decimal.Decimal( '0.3' )/decimal.Decimal( '3.0' ) - decimal.Decimal ( '0.1' ) >> Decimal("0.0") >> >> Decimal solves the problem; whereas mpmath doesn't. > > Okay, that's not an example of catastrophic cancellation, just a > representation issue. Indeed - and as I understand it the motivation for decimal numbers is not that they suffer less from roundoff than binary, but because the round-off they suffer from is better suited to (for example) financial applications, where representing exact decimal numbers can be important. If your problem is the fact of roundoff error, try using as your test case, taking the square root of two and squaring it again. This will suffer from the same sort of roundoff problems in decimal as binary. Anne > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From michael.s.gilbert at gmail.com Wed Sep 8 17:35:37 2010 From: michael.s.gilbert at gmail.com (Michael Gilbert) Date: Wed, 8 Sep 2010 17:35:37 -0400 Subject: [Numpy-discussion] IEEE 754-2008 decimal floating point support In-Reply-To: References: <20100908112611.40fd69d1.michael.s.gilbert@gmail.com> <20100908114613.89c3e2fb.michael.s.gilbert@gmail.com> Message-ID: <20100908173537.92c85a58.michael.s.gilbert@gmail.com> On Wed, 8 Sep 2010 15:44:02 -0400, Michael Gilbert wrote: > On Wed, Sep 8, 2010 at 12:23 PM, Charles R Harris wrote: > > > > > > On Wed, Sep 8, 2010 at 9:46 AM, Michael Gilbert > > wrote: > >> > >> On Wed, 8 Sep 2010 09:43:56 -0600, Charles R Harris wrote: > >> > On Wed, Sep 8, 2010 at 9:26 AM, Michael Gilbert > >> > >> > > wrote: > >> > > >> > > Hi, > >> > > > >> > > Are there any plans to add support for decimal floating point > >> > > arithmetic, as defined in the 2008 revision of the IEEE 754 standard > >> > > [0], in numpy? > >> > > > >> > > > >> > Not at the moment. There is currently no hardware or C support and > >> > adding > >> > new types to numpy isn't trivial. You can get some limited Decimal > >> > functionality by using python classes and object arrays, for instance > >> > the > >> > Decimal class in the python decimal module, but the performance isn't > >> > great. > >> > > >> > What is your particular interest in decimal support? > >> > >> Primarily avoiding catastrophic cancellation when subtracting > >> large values. ?I was planning to use the decimal class, but was > >> curious whether support for the IEEE standard was coming any time soon. > >> > > > > If you just need more precision, mpmath has better performance than the > > Decimal class. Also, it might be possible to avoid the loss of precision by > > changing the computation, but I don't know the details of what you are > > doing. > > Just wanted to say that numpy object arrays + decimal solved all of my > problems, which were all caused by the disconnect between decimal and > binary representation of floating point numbers. It would be really > nice to have IEEE decimal floating point since python's decimal module > is a bit slow, but I can see thats going to be a very long process. FYI, gcc 4.2 introduced initial support for decimal floats. However, various macros like DEC32_MAX, and printf support for dd, df, and dl is not yet implemented making it rather hard to work with. Mike From michael.s.gilbert at gmail.com Wed Sep 8 19:13:51 2010 From: michael.s.gilbert at gmail.com (Michael Gilbert) Date: Wed, 8 Sep 2010 19:13:51 -0400 Subject: [Numpy-discussion] IEEE 754-2008 decimal floating point support In-Reply-To: <20100908173537.92c85a58.michael.s.gilbert@gmail.com> References: <20100908112611.40fd69d1.michael.s.gilbert@gmail.com> <20100908114613.89c3e2fb.michael.s.gilbert@gmail.com> <20100908173537.92c85a58.michael.s.gilbert@gmail.com> Message-ID: On Wed, Sep 8, 2010 at 5:35 PM, Michael Gilbert wrote: > On Wed, 8 Sep 2010 15:44:02 -0400, Michael Gilbert wrote: >> On Wed, Sep 8, 2010 at 12:23 PM, Charles R Harris wrote: >> > >> > >> > On Wed, Sep 8, 2010 at 9:46 AM, Michael Gilbert >> > wrote: >> >> >> >> On Wed, 8 Sep 2010 09:43:56 -0600, Charles R Harris wrote: >> >> > On Wed, Sep 8, 2010 at 9:26 AM, Michael Gilbert >> >> > > >> > > wrote: >> >> > >> >> > > Hi, >> >> > > >> >> > > Are there any plans to add support for decimal floating point >> >> > > arithmetic, as defined in the 2008 revision of the IEEE 754 standard >> >> > > [0], in numpy? >> >> > > >> >> > > >> >> > Not at the moment. There is currently no hardware or C support and >> >> > adding >> >> > new types to numpy isn't trivial. You can get some limited Decimal >> >> > functionality by using python classes and object arrays, for instance >> >> > the >> >> > Decimal class in the python decimal module, but the performance isn't >> >> > great. >> >> > >> >> > What is your particular interest in decimal support? >> >> >> >> Primarily avoiding catastrophic cancellation when subtracting >> >> large values. ?I was planning to use the decimal class, but was >> >> curious whether support for the IEEE standard was coming any time soon. >> >> >> > >> > If you just need more precision, mpmath has better performance than the >> > Decimal class. Also, it might be possible to avoid the loss of precision by >> > changing the computation, but I don't know the details of what you are >> > doing. >> >> Just wanted to say that numpy object arrays + decimal solved all of my >> problems, which were all caused by the disconnect between decimal and >> binary representation of floating point numbers. ?It would be really >> nice to have IEEE decimal floating point since python's decimal module >> is a bit slow, but I can see thats going to be a very long process. > > FYI, gcc 4.2 introduced initial support for decimal floats. ?However, > various macros like DEC32_MAX, and printf support for dd, df, and dl is > not yet implemented making it rather hard to work with. A reference would probably be useful: http://gcc.gnu.org/onlinedocs/gcc-4.2.4/gcc/Decimal-Float.html As with all gcc documentation, its rather terse :/ Best wishes, Mike From ceball at gmail.com Thu Sep 9 06:05:09 2010 From: ceball at gmail.com (Chris Ball) Date: Thu, 9 Sep 2010 10:05:09 +0000 (UTC) Subject: [Numpy-discussion] clip() with None as argument References: Message-ID: Robert Kern gmail.com> writes: > > On Wed, Sep 8, 2010 at 14:42, Chris Ball gmail.com> wrote: > > Robert Kern gmail.com> writes: > > ... > >>>> a = numpy.array([1,2,3,4,5]) > >>>> a.clip(2,None) > > array([2, 2, 2, 2, 2], dtype=object) > > > > I'm not sure why the returned array has a dtype of object > > The usual type-matching semantics. a.clip() tries to find a common > type that fits `a`, 2 and None. Since None only fits in an object > dtype, that's what you get. ... > > and also why the values are not like this: > >>>> numpy.maximum(a,2) > > array([2, 2, 3, 4, 5]) > > clip(a, low, high) boils down to this in the C code itself (or at > least the code path that it goes down when the input array needs to be > upcast): > > maximum(minimum(a, high), low) > > Since all integer objects compare > None based on the default Python > 2.x implementation of comparison across types, you would just get the > low value out of that. Thanks for explaining that. It all makes sense if clip() requires both bounds to be passed, which is what the documentation indicates, so there is not really a problem. I was just confused because of the post and associated commit from Travis a while back implying that he'd altered clip() to support only one bound ("Allow clip method to have either min or max passed in", 4566). Chris From robert.kern at gmail.com Thu Sep 9 09:44:28 2010 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 9 Sep 2010 08:44:28 -0500 Subject: [Numpy-discussion] clip() with None as argument In-Reply-To: References: Message-ID: On Thu, Sep 9, 2010 at 05:05, Chris Ball wrote: > Robert Kern gmail.com> writes: >> >> On Wed, Sep 8, 2010 at 14:42, Chris Ball gmail.com> wrote: >> > Robert Kern gmail.com> writes: >> > > ... >> >>>> a = numpy.array([1,2,3,4,5]) >> >>>> a.clip(2,None) >> > array([2, 2, 2, 2, 2], dtype=object) >> > >> > I'm not sure why the returned array has a dtype of object >> >> The usual type-matching semantics. a.clip() tries to find a common >> type that fits `a`, 2 and None. Since None only fits in an object >> dtype, that's what you get. > ... >> > and also why the values are not like this: >> >>>> numpy.maximum(a,2) >> > array([2, 2, 3, 4, 5]) >> >> clip(a, low, high) boils down to this in the C code itself (or at >> least the code path that it goes down when the input array needs to be >> upcast): >> >> ? maximum(minimum(a, high), low) >> >> Since all integer objects compare > None based on the default Python >> 2.x implementation of comparison across types, you would just get the >> low value out of that. > > Thanks for explaining that. It all makes sense if clip() requires both bounds to > be passed, which is what the documentation indicates, so there is not really a > problem. I was just confused because of the post and associated commit from > Travis a while back implying that he'd altered clip() to support only one bound > ("Allow clip method to have either min or max passed in", 4566). I believe that the clip function was overhauled sometime after that. The enhancement probably got lost then. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From doutriaux1 at llnl.gov Thu Sep 9 09:59:46 2010 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Thu, 09 Sep 2010 06:59:46 -0700 Subject: [Numpy-discussion] Fwd: distutils Message-ID: <4C88E852.1000807@llnl.gov> -------- Original Message -------- Subject: [Numpy-discussion] distutils Date: Tue, 7 Sep 2010 12:12:58 -0700 From: Charles Doutriaux Reply-To: Discussion of Numerical Python To: Discussion of Numerical Python Hi, I'm using distutils to build extensions written in C. I noticed that lately (it seems to be python 2.7 related) whenever I touch 1 C file, ALL the C files are rebuilt. Since I have a lot of C code, it takes a lot of time for nothing. Any idea why this is happening? Do I need to set something new in my setup.py ? Thx, C. _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://*mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Sep 9 10:22:51 2010 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 9 Sep 2010 09:22:51 -0500 Subject: [Numpy-discussion] distutils In-Reply-To: <4C868EBA.9020103@llnl.gov> References: <4C868EBA.9020103@llnl.gov> Message-ID: On Tue, Sep 7, 2010 at 14:12, Charles Doutriaux wrote: > ?Hi, > > I'm using distutils to build extensions written in C. > > I noticed that lately (it seems to be python 2.7 related) whenever I > touch 1 C file, ALL the C files are rebuilt. > Since I have a lot of C code, it takes a lot of time for nothing. > > Any idea why this is happening? Not without an example that demonstrates the problem, no. Are you using numpy.distutils? If you are just using regular distutils, you may wish to ask on the distutils-sig list or python-list, instead. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From numpy-discussion at maubp.freeserve.co.uk Thu Sep 9 12:05:52 2010 From: numpy-discussion at maubp.freeserve.co.uk (Peter) Date: Thu, 9 Sep 2010 17:05:52 +0100 Subject: [Numpy-discussion] Silent DeprecationWarnings under Python 2.7 onwards In-Reply-To: References: Message-ID: Hi all, NumPy currently makes extensive use of the DeprecationWarning class to alert users when some feature is going to be withdrawn. However, as of Python 2.7, the DeprecationWarning is silent by default, see: http://docs.python.org/library/warnings.html#updating-code-for-new-versions-of-python This makes sense to me for deprecation warnings from Python itself or the standard library - after all, Python 2.7 is the last of the Python 2.x series. However, I believe that a silent DeprecationWarning is bad news for third party libraries like NumPy where many end users are coders and we want them to see these warnings by default. Is anyone else concerned about this? A typical NumPy user (on Python 2.7+ or Python 3.2+) may never see the warnings (because they have to deliberately turn them on), the first they'll ?know about it is when they upgrade to a new release and their code suddenly stops working. Potentially NumPy would have to introduce its own NumPy specific DeprecationWarning warning class, and use that. Peter From sturla at molden.no Thu Sep 9 12:18:29 2010 From: sturla at molden.no (Sturla Molden) Date: Thu, 9 Sep 2010 18:18:29 +0200 Subject: [Numpy-discussion] inversion of large matrices In-Reply-To: References: <54848FA5-AD74-46E9-AE0B-3189FB40C104@cs.toronto.edu> <20A7F4F6-3F2A-47CF-AACE-8484AB92A0E3@cs.toronto.edu> Message-ID: <8eb37d6c43beda8bb89a0afab604e0d6.squirrel@webmail.uio.no> > Yes, this is what I am computing. I am computing the pdf of a very high- > dimensional multivariate normal. Is there a specialized method to compute > this? If you use cho_solve and cho_factor from scipy.linalg, you can proceed like this: cx = X - m sqmahal = (cx*cho_solve(cho_factor(S),cx.T).T).sum(axis=1) where X is the data (n x p), m is mean (1 x p) and S is the covariance (p x p). We could do this twice as fast using a hypothetical method tri_solve that calls LAPACK subroutine DTRTRS. cx = X - m sqmahal = (tri_solve(cho_factor(S),cx.T).T**2).sum(axis=1) It would be twice as fast because cho_solve does 50% redundant work in the calculation of Mahalanobis distances, as we can skip the backsubstitution. You can e.g. get DTRTRS from Intel MKL, ACML (free download from AMD) or GotoBLAS. Just downloading ACML and using ctypes (or f2py or Cython) to get DTRTRS is easy. If you do this you probably also want to expose LAPACK's Cholesky subroutine DPOTRS from ACML or Intel MKL, to make sure you get everything as fast as possible (that would also avoid the dependency on SciPy). I.e. call DPOTRS on S, compute CX, call DTRTRS, then take the sum of squares along the first axis (of CX that DTRTRS modified inplace). That is the fastest way to compute Mahalanobis distances I can think of... Make sure you use an optimized LAPACK library: ACML, Intel MKL and GotoBLAS are the only one worth looking at. ATLAS is much slower, and forget about Netlib's reference implementations of BLAS and LAPACK. I hope the SciPy dev team can be persuaded to include a wrapper for DTRTRS in the future. It is after all extremely useful for Mahalanobis distances, and thus for any use of linear models in statistics. Sturla Molden From pav at iki.fi Thu Sep 9 12:29:15 2010 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 9 Sep 2010 16:29:15 +0000 (UTC) Subject: [Numpy-discussion] inversion of large matrices References: <54848FA5-AD74-46E9-AE0B-3189FB40C104@cs.toronto.edu> <20A7F4F6-3F2A-47CF-AACE-8484AB92A0E3@cs.toronto.edu> <8eb37d6c43beda8bb89a0afab604e0d6.squirrel@webmail.uio.no> Message-ID: Thu, 09 Sep 2010 18:18:29 +0200, Sturla Molden wrote: [clip] > I hope the SciPy dev team can be persuaded to include a wrapper for > DTRTRS in the future. It is after all extremely useful for Mahalanobis > distances, and thus for any use of linear models in statistics. I don't see reasons why not to include such a wrapper, so little persuasion is needed. The only thing is that someone should spend some time implementing this suggestion (and I probably won't -- I don't really need that feature myself, and there are many other things that need to be done). -- Pauli Virtanen From lxander.m at gmail.com Thu Sep 9 16:59:42 2010 From: lxander.m at gmail.com (Alexander Michael) Date: Thu, 9 Sep 2010 13:59:42 -0700 Subject: [Numpy-discussion] Inverting argsort(a, axis=0) to obtain column-wise ranks In-Reply-To: <222CE988-7159-418F-9859-585005E529A0@yale.edu> References: <222CE988-7159-418F-9859-585005E529A0@yale.edu> Message-ID: Clever and concise (and expect that it works), but isn't this less efficient? Sorting is O(n*log(n)), while the code I gave is O(n). Using argsort has the potential to use less memory, though. On Tuesday, September 7, 2010, Zachary Pincus wrote: >> indices = argsort(a1) >> ranks = zeros_like(indices) >> ranks[indices] = arange(len(indices)) > > Doesn't answer your original question directly, but I only recently > learned from this list that the following does the same as the above: > ranks = a1.argsort().argsort() > Will wonders never cease... > > So does ranks=a2.argsort(axis=0).argsort(axis=0) then do the trick? > > Zach > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From lxander.m at gmail.com Thu Sep 9 17:05:49 2010 From: lxander.m at gmail.com (Alexander Michael) Date: Thu, 9 Sep 2010 14:05:49 -0700 Subject: [Numpy-discussion] Inverting argsort(a, axis=0) to obtain column-wise ranks In-Reply-To: <222CE988-7159-418F-9859-585005E529A0@yale.edu> References: <222CE988-7159-418F-9859-585005E529A0@yale.edu> Message-ID: Clever and concise (and expect that it works), but isn't this less efficient? Sorting is O(n*log(n)), while the code I gave is O(n). Using argsort has the potential to use less memory, though. On Tuesday, September 7, 2010, Zachary Pincus wrote: >> indices = argsort(a1) >> ranks = zeros_like(indices) >> ranks[indices] = arange(len(indices)) > > Doesn't answer your original question directly, but I only recently > learned from this list that the following does the same as the above: > ranks = a1.argsort().argsort() > Will wonders never cease... > > So does ranks=a2.argsort(axis=0).argsort(axis=0) then do the trick? > > Zach > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Thu Sep 9 17:25:54 2010 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 9 Sep 2010 16:25:54 -0500 Subject: [Numpy-discussion] Inverting argsort(a, axis=0) to obtain column-wise ranks In-Reply-To: References: <222CE988-7159-418F-9859-585005E529A0@yale.edu> Message-ID: On Thu, Sep 9, 2010 at 15:59, Alexander Michael wrote: > Clever and concise (and expect that it works), but isn't this less > efficient? Sorting is O(n*log(n)), while the code I gave is O(n). > Using argsort has the potential to use less memory, though. No, the code you gave is also O(n*log(n)) because it uses an argsort(). Using two argsort()s won't change the O() complexity. O() often tells you very little. Time it with typical values and find out. Memory use is often the bottleneck. And sometimes, the real performance differences just don't matter. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From cpblpublic+numpy at gmail.com Thu Sep 9 22:22:05 2010 From: cpblpublic+numpy at gmail.com (cpblpublic) Date: Thu, 9 Sep 2010 19:22:05 -0700 Subject: [Numpy-discussion] weighted mean; weighted standard error of the mean (sem) In-Reply-To: References: Message-ID: I am looking for some reaally basic statistical tools. I have some sample data, some sample weights for those measurements, and I want to calculate a mean and a standard error of the mean. Here are obvious places to look: numpy scipy.stats statsmodels It seems to me that numpy's "mean" and "average" functions have their names backwards. That is, often a mean is defined more generally than average, and includes the possibility of weighting, but in this case it is "average" that has a weights argument. Can these functions be merged/renamed/deprecated in the future? It's clear to me that "mean" should allow for weights. None of these modules, above, offer standard error of the mean which incorporates weights. scipy.stats.sem() doesn't, and that's the closest thing. numpy's "var" doesn't allow weights. There aren't any weighted variances in the above modules. Again, are there favoured codes for these functions? Can they be incorporated appropriately in the future? Most immediately, I'd love to get code for weighted sem. I'll write it otherwise, but it might be crude and dumb... Thanks! Chris Barrington-Leigh UBC -------------- next part -------------- An HTML attachment was scrubbed... URL: From erin.sheldon at gmail.com Thu Sep 9 22:38:18 2010 From: erin.sheldon at gmail.com (Erin Sheldon) Date: Thu, 09 Sep 2010 22:38:18 -0400 Subject: [Numpy-discussion] weighted mean; weighted standard error of the mean (sem) In-Reply-To: References: Message-ID: <1284086101-sup-6447@theshire> Excerpts from cpblpublic's message of Thu Sep 09 22:22:05 -0400 2010: > I am looking for some reaally basic statistical tools. I have some > sample data, some sample weights for those measurements, and I want to > calculate a mean and a standard error of the mean. > > Here are obvious places to look: > > numpy > scipy.stats > statsmodels > > It seems to me that numpy's "mean" and "average" functions have their > names backwards. That is, often a mean is defined more generally than > average, and includes the possibility of weighting, but in this case > it is "average" that has a weights argument. Can these functions be > merged/renamed/deprecated in the future? It's clear to me that "mean" > should allow for weights. > > None of these modules, above, offer standard error of the mean which > incorporates weights. scipy.stats.sem() doesn't, and that's the closest > thing. numpy's "var" doesn't allow weights. > There aren't any weighted variances in the above modules. > > Again, are there favoured codes for these functions? Can they be > incorporated appropriately in the future? > > Most immediately, I'd love to get code for weighted sem. I'll write it > otherwise, but it might be crude and dumb... > Thanks! > Chris Barrington-Leigh > UBC This code below should do what you want. It is part of the stat sub-package of esutil http://code.google.com/p/esutil/ Hope this helps, Erin Scott Sheldon Brookhaven National Laboratory def wmom(arrin, weights_in, inputmean=None, calcerr=False, sdev=False): """ NAME: wmom() PURPOSE: Calculate the weighted mean, error, and optionally standard deviation of an input array. By default error is calculated assuming the weights are 1/err^2, but if you send calcerr=True this assumption is dropped and the error is determined from the weighted scatter. CALLING SEQUENCE: wmean,werr = wmom(arr, weights, inputmean=None, calcerr=False, sdev=False) INPUTS: arr: A numpy array or a sequence that can be converted. weights: A set of weights for each elements in array. OPTIONAL INPUTS: inputmean: An input mean value, around which them mean is calculated. calcerr=False: Calculate the weighted error. By default the error is calculated as 1/sqrt( weights.sum() ). If calcerr=True it is calculated as sqrt( (w**2 * (arr-mean)**2).sum() )/weights.sum() sdev=False: If True, also return the weighted standard deviation as a third element in the tuple. OUTPUTS: wmean, werr: A tuple of the weighted mean and error. If sdev=True the tuple will also contain sdev: wmean,werr,wsdev REVISION HISTORY: Converted from IDL: 2006-10-23. Erin Sheldon, NYU """ # no copy made if they are already arrays arr = numpy.array(arrin, ndmin=1, copy=False) # Weights is forced to be type double. All resulting calculations # will also be double weights = numpy.array(weights_in, ndmin=1, dtype='f8', copy=False) wtot = weights.sum() # user has input a mean value if inputmean is None: wmean = ( weights*arr ).sum()/wtot else: wmean=float(inputmean) # how should error be calculated? if calcerr: werr2 = ( weights**2 * (arr-wmean)**2 ).sum() werr = numpy.sqrt( werr2 )/wtot else: werr = 1.0/numpy.sqrt(wtot) # should output include the weighted standard deviation? if sdev: wvar = ( weights*(arr-wmean)**2 ).sum()/wtot wsdev = numpy.sqrt(wvar) return wmean,werr,wsdev else: return wmean,werr From josef.pktd at gmail.com Thu Sep 9 22:39:25 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 9 Sep 2010 22:39:25 -0400 Subject: [Numpy-discussion] weighted mean; weighted standard error of the mean (sem) In-Reply-To: References: Message-ID: On Thu, Sep 9, 2010 at 10:22 PM, cpblpublic wrote: > > > I am looking for some reaally basic statistical tools. I have some > sample data, some sample weights for those measurements, and I want to > calculate a mean and a standard error of the mean. > > Here are obvious places to look: > > numpy > scipy.stats > statsmodels > > It seems to me that numpy's "mean" and "average" functions have their > names backwards. That is, often a mean is defined more generally than > average, and includes the possibility of weighting, but in this case > it is "average" that has a weights argument. Can these functions be > merged/renamed/deprecated in the future? ?It's clear to me that "mean" > should allow for weights. I think of weighted mean and weighted average, pretty much as synonyms, changing names would break backwards compatibility without any real benefit, in my opinion. > > None of these modules, above, offer standard error of the mean which > incorporates weights. scipy.stats.sem() doesn't, and that's the closest > thing. numpy's "var" doesn't allow weights. > There aren't any weighted variances in the above modules. for weighted statistics, I usually refer to ticket 604 http://projects.scipy.org/scipy/attachment/ticket/604/Statistics.py but I didn't see weighted sem in it > > Again, are there favoured codes for these functions? Can they be > incorporated appropriately in the future? > > Most immediately, I'd love to get code for weighted sem. I'll write it > otherwise, but it might be crude and dumb... just a thought, I still have to check the details: Estimating statsmodels.WLS with just a constant should give all the result statistics on the mean, e.g. bse for variance of constant, t() for t-statistic Josef > > Thanks! > Chris Barrington-Leigh > UBC > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From lpc at cmu.edu Thu Sep 9 22:50:11 2010 From: lpc at cmu.edu (Luis Pedro Coelho) Date: Thu, 9 Sep 2010 22:50:11 -0400 Subject: [Numpy-discussion] ANNOUNCE: mahotas 0.5 Message-ID: <20100909225011.4b0ba037@oakeshott> Hello everyone, My numpy based image processing toolbox has just had a new release: 0.5 New features are: Distance transform bwperim() freeimage interface [borrowed and improved from scikits.image] zernike moment computation There were some fixes to the namespace (in particular, functions that were previously available as mahotas.bbox.bbox are now just mahotas.bbox), which is good, but might break code. Sorry, but the new system is just so much better. Many small fixes and performance improvements all around. You can get it from pypi: http://pypi.python.org/pypi/mahotas (including through easy_install mahotas or pip install mahotas). Bleeding edge is at: http://github.com/luispedro/mahotas API Documentation at http://packages.python.org/mahotas/ (very complete, but rather ugly; will fix uglyness). Mailing-List (for Python-based computer vision related subjects in general including this package): http://groups.google.com/group/pythonvision -- Luis Pedro Coelho | Carnegie Mellon University | http://luispedro.org From kwgoodman at gmail.com Thu Sep 9 23:07:49 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 9 Sep 2010 20:07:49 -0700 Subject: [Numpy-discussion] weighted mean; weighted standard error of the mean (sem) In-Reply-To: References: Message-ID: On Thu, Sep 9, 2010 at 7:22 PM, cpblpublic wrote: > I am looking for some reaally basic statistical tools. I have some > sample data, some sample weights for those measurements, and I want to > calculate a mean and a standard error of the mean. How about using a bootstrap? Array and weights: >> a = np.arange(100) >> w = np.random.rand(100) >> w = w / w.sum() Initialize: >> n = 1000 >> ma = np.zeros(n) Save mean of each bootstrap sample: >> for i in range(n): ....: idx = np.random.randint(0, 100, 100) ....: ma[i] = np.dot(a[idx], w[idx]) ....: ....: Error in mean: >> ma.std() 3.854023384833674 Sanity check: >> np.dot(w, a) 49.231127299096954 >> ma.mean() 49.111478821225127 Hmm...should w[idx] be renormalized to sum to one in each bootstrap sample? From kwgoodman at gmail.com Thu Sep 9 23:32:07 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 9 Sep 2010 20:32:07 -0700 Subject: [Numpy-discussion] weighted mean; weighted standard error of the mean (sem) In-Reply-To: References: Message-ID: On Thu, Sep 9, 2010 at 8:07 PM, Keith Goodman wrote: > On Thu, Sep 9, 2010 at 7:22 PM, cpblpublic wrote: >> I am looking for some reaally basic statistical tools. I have some >> sample data, some sample weights for those measurements, and I want to >> calculate a mean and a standard error of the mean. > > How about using a bootstrap? > > Array and weights: > >>> a = np.arange(100) >>> w = np.random.rand(100) >>> w = w / w.sum() > > Initialize: > >>> n = 1000 >>> ma = np.zeros(n) > > Save mean of each bootstrap sample: > >>> for i in range(n): > ? ....: ? ? idx = np.random.randint(0, 100, 100) > ? ....: ? ? ma[i] = np.dot(a[idx], w[idx]) > ? ....: > ? ....: > > Error in mean: > >>> ma.std() > ? 3.854023384833674 > > Sanity check: > >>> np.dot(w, a) > ? 49.231127299096954 >>> ma.mean() > ? 49.111478821225127 > > Hmm...should w[idx] be renormalized to sum to one in each bootstrap sample? Or perhaps there is no uncertainty about the weights, in which case: >> for i in range(n): ....: idx = np.random.randint(0, 100, 100) ....: ma[i] = np.dot(a[idx], w) ....: ....: >> ma.std() 3.2548815339711115 From josef.pktd at gmail.com Thu Sep 9 23:44:06 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 9 Sep 2010 23:44:06 -0400 Subject: [Numpy-discussion] weighted mean; weighted standard error of the mean (sem) In-Reply-To: References: Message-ID: On Thu, Sep 9, 2010 at 11:32 PM, Keith Goodman wrote: > On Thu, Sep 9, 2010 at 8:07 PM, Keith Goodman wrote: >> On Thu, Sep 9, 2010 at 7:22 PM, cpblpublic wrote: >>> I am looking for some reaally basic statistical tools. I have some >>> sample data, some sample weights for those measurements, and I want to >>> calculate a mean and a standard error of the mean. >> >> How about using a bootstrap? >> >> Array and weights: >> >>>> a = np.arange(100) >>>> w = np.random.rand(100) >>>> w = w / w.sum() >> >> Initialize: >> >>>> n = 1000 >>>> ma = np.zeros(n) >> >> Save mean of each bootstrap sample: >> >>>> for i in range(n): >> ? ....: ? ? idx = np.random.randint(0, 100, 100) >> ? ....: ? ? ma[i] = np.dot(a[idx], w[idx]) >> ? ....: >> ? ....: >> >> Error in mean: >> >>>> ma.std() >> ? 3.854023384833674 >> >> Sanity check: >> >>>> np.dot(w, a) >> ? 49.231127299096954 >>>> ma.mean() >> ? 49.111478821225127 >> >> Hmm...should w[idx] be renormalized to sum to one in each bootstrap sample? > > Or perhaps there is no uncertainty about the weights, in which case: > >>> for i in range(n): > ? ....: ? ? idx = np.random.randint(0, 100, 100) > ? ....: ? ? ma[i] = np.dot(a[idx], w) > ? ....: > ? ....: >>> ma.std() > ? 3.2548815339711115 or maybe `w` reflects an underlying sampling scheme and you should sample in the bootstrap according to w ? if weighted average is a sum of linear functions of (normal) distributed random variables, it still depends on whether the individual observations have the same or different variances, e.g. http://en.wikipedia.org/wiki/Weighted_mean#Statistical_properties What I can't figure out is whether if you assume simga_i = sigma for all observation i, do we use the weighted or the unweighted variance to get an estimate of sigma. And I'm not able to replicate with simple calculations what statsmodels.WLS gives me. ??? Josef > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From lxander.m at gmail.com Thu Sep 9 23:53:07 2010 From: lxander.m at gmail.com (Alexander Michael) Date: Thu, 9 Sep 2010 20:53:07 -0700 Subject: [Numpy-discussion] Inverting argsort(a, axis=0) to obtain column-wise ranks In-Reply-To: References: <222CE988-7159-418F-9859-585005E529A0@yale.edu> Message-ID: I was, of course, just thinking of the incremental work of inverting the initial argsort, but you are completely correct in pointing out that the overall complexity is O(n*log(n)) either way. As it turns out, both approaches run in the same amount of time for my problem. Thanks, Alex On Thursday, September 9, 2010, Robert Kern wrote: > On Thu, Sep 9, 2010 at 15:59, Alexander Michael wrote: >> Clever and concise (and expect that it works), but isn't this less >> efficient? Sorting is O(n*log(n)), while the code I gave is O(n). >> Using argsort has the potential to use less memory, though. > > No, the code you gave is also O(n*log(n)) because it uses an > argsort(). Using two argsort()s won't change the O() complexity. O() > often tells you very little. Time it with typical values and find out. > Memory use is often the bottleneck. And sometimes, the real > performance differences just don't matter. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ? -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From kwgoodman at gmail.com Fri Sep 10 00:13:42 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 9 Sep 2010 21:13:42 -0700 Subject: [Numpy-discussion] weighted mean; weighted standard error of the mean (sem) In-Reply-To: References: Message-ID: On Thu, Sep 9, 2010 at 8:44 PM, wrote: > On Thu, Sep 9, 2010 at 11:32 PM, Keith Goodman wrote: >> On Thu, Sep 9, 2010 at 8:07 PM, Keith Goodman wrote: >>> On Thu, Sep 9, 2010 at 7:22 PM, cpblpublic wrote: >>>> I am looking for some reaally basic statistical tools. I have some >>>> sample data, some sample weights for those measurements, and I want to >>>> calculate a mean and a standard error of the mean. >>> >>> How about using a bootstrap? >>> >>> Array and weights: >>> >>>>> a = np.arange(100) >>>>> w = np.random.rand(100) >>>>> w = w / w.sum() >>> >>> Initialize: >>> >>>>> n = 1000 >>>>> ma = np.zeros(n) >>> >>> Save mean of each bootstrap sample: >>> >>>>> for i in range(n): >>> ? ....: ? ? idx = np.random.randint(0, 100, 100) >>> ? ....: ? ? ma[i] = np.dot(a[idx], w[idx]) >>> ? ....: >>> ? ....: >>> >>> Error in mean: >>> >>>>> ma.std() >>> ? 3.854023384833674 >>> >>> Sanity check: >>> >>>>> np.dot(w, a) >>> ? 49.231127299096954 >>>>> ma.mean() >>> ? 49.111478821225127 >>> >>> Hmm...should w[idx] be renormalized to sum to one in each bootstrap sample? >> >> Or perhaps there is no uncertainty about the weights, in which case: >> >>>> for i in range(n): >> ? ....: ? ? idx = np.random.randint(0, 100, 100) >> ? ....: ? ? ma[i] = np.dot(a[idx], w) >> ? ....: >> ? ....: >>>> ma.std() >> ? 3.2548815339711115 > > or maybe `w` reflects an underlying sampling scheme and you should > sample in the bootstrap according to w ? Yes.... > if weighted average is a sum of linear functions of (normal) > distributed random variables, it still depends on whether the > individual observations have the same or different variances, e.g. > http://en.wikipedia.org/wiki/Weighted_mean#Statistical_properties ...lots of possibilities. As you have shown the problem is not yet well defined. Not much specification needed for the weighted mean, lots needed for the standard error of the weighted mean. > What I can't figure out is whether if you assume simga_i = sigma for > all observation i, do we use the weighted or the unweighted variance > to get an estimate of sigma. And I'm not able to replicate with simple > calculations what statsmodels.WLS gives me. My guess: if all you want is sigma of the individual i and you know sigma is the same for all i, then I suppose you don't care about the weight. > > ??? > > Josef > > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From seb.haase at gmail.com Fri Sep 10 03:40:33 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Fri, 10 Sep 2010 09:40:33 +0200 Subject: [Numpy-discussion] ANNOUNCE: mahotas 0.5 In-Reply-To: <20100909225011.4b0ba037@oakeshott> References: <20100909225011.4b0ba037@oakeshott> Message-ID: Hi Luis, thanks for the announcement. How would you compare mahotas to scipy's ndimage ? Are you using ndimage in mahotas at all ? Thanks, Sebastian Haase On Fri, Sep 10, 2010 at 4:50 AM, Luis Pedro Coelho wrote: > Hello everyone, > > My numpy based image processing toolbox has just had a new release: 0.5 > > New features are: > ? Distance transform > ? bwperim() > ? freeimage interface [borrowed and improved from scikits.image] > ? zernike moment computation > > There were some fixes to the namespace (in particular, functions that > were previously available as mahotas.bbox.bbox are now just > mahotas.bbox), which is good, but might break code. Sorry, but the new > system is just so much better. > > Many small fixes and performance improvements all around. > > You can get it from pypi: http://pypi.python.org/pypi/mahotas > (including through easy_install mahotas or pip install mahotas). > > Bleeding edge is at: http://github.com/luispedro/mahotas > > API Documentation at http://packages.python.org/mahotas/ > (very complete, but rather ugly; will fix uglyness). > > Mailing-List (for Python-based computer vision related subjects in > general including this package): > http://groups.google.com/group/pythonvision > -- > Luis Pedro Coelho | Carnegie Mellon University | http://luispedro.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From radek.machulka at gmail.com Fri Sep 10 05:46:47 2010 From: radek.machulka at gmail.com (Radek Machulka) Date: Fri, 10 Sep 2010 11:46:47 +0200 Subject: [Numpy-discussion] Non-Zero Sub-Matrix Message-ID: Hi Folks, I have array (numpy.ndarray object) with non-zero elements cumulated 'somewhere' (like a array([[0,0,0,0],[0,1,1,0],[0,0,1,0],[0,0,0,0]])) and I need sub-array with just non-zero elements (array([[1,1],[0,1]])). I can do this with iterating throught an array, but I also found some magic tricks with boolen operators in indexing and nonzero function, but it is still not clear to me, how to use it. I am sure that this will be pretty easy for those who are familiar with this kind of array indexing and I hope I will be one of them soon :-). Thanks a lot Radek From pav at iki.fi Fri Sep 10 06:18:18 2010 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 10 Sep 2010 10:18:18 +0000 (UTC) Subject: [Numpy-discussion] Non-Zero Sub-Matrix References: Message-ID: Fri, 10 Sep 2010 11:46:47 +0200, Radek Machulka wrote: > I have array (numpy.ndarray object) with non-zero elements cumulated > 'somewhere' (like a array([[0,0,0,0],[0,1,1,0],[0,0,1,0],[0,0,0,0]])) > and I need sub-array with just non-zero elements (array([[1,1],[0,1]])). > I can do this with iterating throught an array, but I also found some > magic tricks with boolen operators in indexing and nonzero function, but > it is still not clear to me, how to use it. >>> x = np.array([[0,0,0,0],[0,1,1,0],[0,0,1,0],[0,0,0,0]]) >>> i, j = x.any(0).nonzero()[0], x.any(1).nonzero()[0] >>> x[i[:,None], j[None,:]] array([[1, 1], [0, 1]]) From tpk at kraussfamily.org Fri Sep 10 08:05:01 2010 From: tpk at kraussfamily.org (Tom K.) Date: Fri, 10 Sep 2010 05:05:01 -0700 (PDT) Subject: [Numpy-discussion] interrupting large matrix operations Message-ID: <29676118.post@talk.nabble.com> OK, I know it's my problem if I try to form a 15000x15000 array and take the cosine of each element, but the result is that my python session completely hangs - that is, the operation is not interruptible. t=np.arange(15360)/15.36e6 t.shape=(-1,1) X=np.cos(2*np.pi*750*(t-t.T)) I'd like to hit "control-c" and get out of this hung state. What would it take to support this? (I'm running ancient numpy and python at work, so if this is already supported in later versions, my apologies) - Tom K. -- View this message in context: http://old.nabble.com/interrupting-large-matrix-operations-tp29676118p29676118.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From cournape at gmail.com Fri Sep 10 08:22:03 2010 From: cournape at gmail.com (David Cournapeau) Date: Fri, 10 Sep 2010 21:22:03 +0900 Subject: [Numpy-discussion] interrupting large matrix operations In-Reply-To: <29676118.post@talk.nabble.com> References: <29676118.post@talk.nabble.com> Message-ID: On Fri, Sep 10, 2010 at 9:05 PM, Tom K. wrote: > > OK, I know it's my problem if I try to form a 15000x15000 array and take the > cosine of each element, but the result is that my python session completely > hangs - that is, the operation is not interruptible. > > t=np.arange(15360)/15.36e6 > t.shape=(-1,1) > X=np.cos(2*np.pi*750*(t-t.T)) > > > I'd like to hit "control-c" and get out of this hung state. > > What would it take to support this? It is difficult to support this in every case. The basic way to handle ctr+c is to regularly check whether the corresponding signal has been sent during computation. The problem is when to check this - too often, and it can significantly slow down the processing. For ufuncs, I am a bit surprised it is not done until the end of the processing, though. What happens exactly when you do Ctrl+C ? It may take a long time, but it should raise a keyboard interrupt at the end (or after the intermediate computation t-t.T which may take quite some time too). > (I'm running ancient numpy and python at work, so if this is already > supported in later versions, my apologies) What does ancient mean ? Could you give us the version (numpy.__version__) cheers, David From cournape at gmail.com Fri Sep 10 08:29:12 2010 From: cournape at gmail.com (David Cournapeau) Date: Fri, 10 Sep 2010 21:29:12 +0900 Subject: [Numpy-discussion] interrupting large matrix operations In-Reply-To: References: <29676118.post@talk.nabble.com> Message-ID: On Fri, Sep 10, 2010 at 9:22 PM, David Cournapeau wrote: > On Fri, Sep 10, 2010 at 9:05 PM, Tom K. wrote: >> >> OK, I know it's my problem if I try to form a 15000x15000 array and take the >> cosine of each element, but the result is that my python session completely >> hangs - that is, the operation is not interruptible. >> >> t=np.arange(15360)/15.36e6 >> t.shape=(-1,1) >> X=np.cos(2*np.pi*750*(t-t.T)) >> >> >> I'd like to hit "control-c" and get out of this hung state. >> >> What would it take to support this? One more question: are you hitting the swap significantly when doing the uninterruptible operation ? It may be blocked during allocation if the disk is used, in which case I am not sure there is anything we can do (if it is blocked inside malloc for example) - but my knowledge of signals is rather limited. cheers, David From tpk at kraussfamily.org Fri Sep 10 08:34:58 2010 From: tpk at kraussfamily.org (Tom K.) Date: Fri, 10 Sep 2010 05:34:58 -0700 (PDT) Subject: [Numpy-discussion] interrupting large matrix operations In-Reply-To: References: <29676118.post@talk.nabble.com> Message-ID: <29676348.post@talk.nabble.com> David Cournapeau wrote: > >> (I'm running ancient numpy and python at work, so if this is already >> supported in later versions, my apologies) > > What does ancient mean ? Could you give us the version (numpy.__version__) > > 1.0.4 Python 2.4.2 IPython 0.9.1 -- View this message in context: http://old.nabble.com/interrupting-large-matrix-operations-tp29676118p29676348.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From radek.machulka at gmail.com Fri Sep 10 08:35:46 2010 From: radek.machulka at gmail.com (Radek Machulka) Date: Fri, 10 Sep 2010 14:35:46 +0200 Subject: [Numpy-discussion] Non-Zero Sub-Matrix In-Reply-To: References: Message-ID: Thanks, but... >>> x = array([[0,0,0,0],[0,1,0,0],[0,0,1,1],[0,0,0,0]]) >>> x array([[0, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 1], [0, 0, 0, 0]]) >>> i, j = x.any(0).nonzero()[0], x.any(1).nonzero()[0] >>> x[i[:,None], j[None,:]] array([[1, 0], [0, 1], [0, 0]]) R. 2010/9/10, Pauli Virtanen : > Fri, 10 Sep 2010 11:46:47 +0200, Radek Machulka wrote: >> I have array (numpy.ndarray object) with non-zero elements cumulated >> 'somewhere' (like a array([[0,0,0,0],[0,1,1,0],[0,0,1,0],[0,0,0,0]])) >> and I need sub-array with just non-zero elements (array([[1,1],[0,1]])). >> I can do this with iterating throught an array, but I also found some >> magic tricks with boolen operators in indexing and nonzero function, but >> it is still not clear to me, how to use it. > >>>> x = np.array([[0,0,0,0],[0,1,1,0],[0,0,1,0],[0,0,0,0]]) >>>> i, j = x.any(0).nonzero()[0], x.any(1).nonzero()[0] >>>> x[i[:,None], j[None,:]] > array([[1, 1], > [0, 1]]) > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From pav at iki.fi Fri Sep 10 08:38:12 2010 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 10 Sep 2010 12:38:12 +0000 (UTC) Subject: [Numpy-discussion] Non-Zero Sub-Matrix References: Message-ID: Fri, 10 Sep 2010 14:35:46 +0200, Radek Machulka wrote: > Thanks, but... > >>>> x = array([[0,0,0,0],[0,1,0,0],[0,0,1,1],[0,0,0,0]]) x > array([[0, 0, 0, 0], > [0, 1, 0, 0], > [0, 0, 1, 1], > [0, 0, 0, 0]]) >>>> i, j = x.any(0).nonzero()[0], x.any(1).nonzero()[0] Should be >>> j, i = x.any(0).nonzero()[0], x.any(1).nonzero()[0] From david.froger at gmail.com Fri Sep 10 10:30:33 2010 From: david.froger at gmail.com (David Froger) Date: Fri, 10 Sep 2010 16:30:33 +0200 Subject: [Numpy-discussion] f2py: generic,public/private procedure Message-ID: Hy all, As a test case before writing something bigger, I'm trying to write a little Fortran module to compute the average of a array in these 4 cases: avg2d_float, avg2d_double avg3d_float, avg3d_double I want this module to be callable from both Fortran and Python, using f2py. 4 Fortran functions have to be written, and a generic Fortran function 'avg' overloads its. The Fortran module 'stat.f90' (containing the functions), a Fortran program 'stat_example.f90' and a Python script 'stat_example.py' are at the end of the email. (everything works fine: the compilation, f2py, the execution in Python and in Fortran.) My questions are: - Assumed shape array are not supported by f2py. However, it is much more handy to write in Fortran 'avg(array)' than 'avg(array,n1,n2,n3)'. Would it be possible to supported it by modifying the signature files? - Is there a way to avoid to write by hand the 'avg' Python function (in example_stat.py) ? - Isn't there no way to declare private the functions: avg2d_float, avg2d_double, avg3d_float,avg3d_double? It's embarassing its are visible from stat_example.f90. Uncommenting the lines in the top of the file stat.f90: !private !public:: avg and running: f2py -m f90stat -c stat.f90 give the error: use stat, only : avg2d_float Error: Symbol 'avg2d_float' referenced at (1) not found in module 'stat' Thanks for any suggestion, David * to create the Fortran executable: gfortran -c stat.f90 gfortran -c stat_example.f90 gfortran -o stat_example.x stat.o stat_example.o * to create the Python module: f2py -m f90stat -c stat.f90 * The Fortran module is: """ !file stat.f90 module stat implicit none !private !public:: avg interface avg module procedure avg2d_float, avg2d_double, avg3d_float, avg3d_double end interface contains function avg2d_float(array,n1,n2) result(average) implicit none real(kind=4):: average integer:: n1,n2 real(kind=4),dimension(n1,n2),intent(in):: array average = sum(array) average = average / n1 average = average / n2 end function avg2d_float function avg2d_double(array,n1,n2) result(average) implicit none real(kind=8):: average integer:: n1,n2 real(kind=8),dimension(n1,n2),intent(in):: array average = sum(array) average = average / n1 average = average / n2 end function avg2d_double function avg3d_float(array,n1,n2,n3) result(average) implicit none real(kind=4):: average integer:: n1,n2,n3 real(kind=4),dimension(n1,n2,n3),intent(in):: array average = sum(array) average = average / n1 average = average / n2 average = average / n3 end function avg3d_float function avg3d_double(array,n1,n2,n3) result(average) implicit none real(kind=8):: average integer:: n1,n2,n3 real(kind=8),dimension(n1,n2,n3),intent(in):: array average = sum(array) average = average / n1 average = average / n2 average = average / n3 end function avg3d_double end module stat """ * The Fortran program is: """ !file: stat_example.f90 program stat_example use stat implicit none integer,parameter:: n1=10,n2=10,n3=10 real(kind=4),dimension(n1,n2):: array2d_float real(kind=8),dimension(n1,n2):: array2d_double real(kind=4),dimension(n1,n2,n3):: array3d_float real(kind=8),dimension(n1,n2,n3):: array3d_double array2d_float = 4. array2d_double = 4. array3d_float = 4. array3d_double = 4. write(*,*) avg(array2d_float,n1,n2) write(*,*) avg(array2d_double,n1,n2) write(*,*) avg(array3d_float,n1,n2,n3) write(*,*) avg(array3d_double,n1,n2,n3) write(*,*) avg2d_float(array2d_float,n1,n2) end program stat_example """ * The Python script is: """ #!/usr/bin/env python #file stat_example.py import numpy as np import f90stat def avg(a): if a.ndim == 2: if a.dtype == 'float32': return f90stat.stat.avg2d_float(a) elif a.dtype == 'float64' : return f90stat.stat.avg2d_double(a) else: raise ValueError, 'dtype = %r unsupported.' % (a.dtype) elif a.ndim == 3: if a.dtype == 'float32': return f90stat.stat.avg3d_float(a) elif a.dtype == 'float64' : return f90stat.stat.avg3d_double(a) else: raise ValueError, 'dtype = %r unsupported.' % (a.dtype) else: raise ValueError, 'ndim = %r unsupported.' % (a.ndim) if __name__ == '__main__': a = np.arange(6.) a = a.reshape(2,3) print avg(a) """ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hagen at zhuliguan.net Fri Sep 10 13:36:11 2010 From: hagen at zhuliguan.net (=?UTF-8?B?SGFnZW4gRsO8cnN0ZW5hdQ==?=) Date: Fri, 10 Sep 2010 19:36:11 +0200 Subject: [Numpy-discussion] dot() performance depends on data? Message-ID: <4C8A6C8B.3030007@zhuliguan.net> Hi, I'm multiplying two 1000x1000 arrays with numpy.dot() and seeing significant performance differences depending on the data. It seems to take much longer on matrices with many zeros than on random ones. I don't know much about optimized MM implementations, but is this normal behavior for some reason? Cheers, Hagen -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: OpenPGP digital signature URL: From jsalvati at u.washington.edu Fri Sep 10 13:46:20 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Fri, 10 Sep 2010 10:46:20 -0700 Subject: [Numpy-discussion] dot() performance depends on data? In-Reply-To: <4C8A6C8B.3030007@zhuliguan.net> References: <4C8A6C8B.3030007@zhuliguan.net> Message-ID: Perhaps the ndarrays have different ordering (C vs Fortran order)? On Fri, Sep 10, 2010 at 10:36 AM, Hagen F?rstenau wrote: > Hi, > > I'm multiplying two 1000x1000 arrays with numpy.dot() and seeing > significant performance differences depending on the data. It seems to > take much longer on matrices with many zeros than on random ones. I > don't know much about optimized MM implementations, but is this normal > behavior for some reason? > > Cheers, > Hagen > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Fri Sep 10 13:46:40 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 10 Sep 2010 10:46:40 -0700 Subject: [Numpy-discussion] dot() performance depends on data? In-Reply-To: <4C8A6C8B.3030007@zhuliguan.net> References: <4C8A6C8B.3030007@zhuliguan.net> Message-ID: On Fri, Sep 10, 2010 at 10:36 AM, Hagen F?rstenau wrote: > I'm multiplying two 1000x1000 arrays with numpy.dot() and seeing > significant performance differences depending on the data. It seems to > take much longer on matrices with many zeros than on random ones. I > don't know much about optimized MM implementations, but is this normal > behavior for some reason? I don't see a difference on my computer. Could you post an example? From charlesr.harris at gmail.com Fri Sep 10 13:57:05 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 10 Sep 2010 11:57:05 -0600 Subject: [Numpy-discussion] dot() performance depends on data? In-Reply-To: <4C8A6C8B.3030007@zhuliguan.net> References: <4C8A6C8B.3030007@zhuliguan.net> Message-ID: On Fri, Sep 10, 2010 at 11:36 AM, Hagen F?rstenau wrote: > Hi, > > I'm multiplying two 1000x1000 arrays with numpy.dot() and seeing > significant performance differences depending on the data. It seems to > take much longer on matrices with many zeros than on random ones. I > don't know much about optimized MM implementations, but is this normal > behavior for some reason? > > Multiplication by zero used to be faster than multiplication by random numbers. However, modern hardware and compilers may have changed that to pretty much a wash. More likely you are seeing cache issues due to data localization or even variations in the time given the thread running the multiplication. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cpblpublic+numpy at gmail.com Fri Sep 10 13:58:08 2010 From: cpblpublic+numpy at gmail.com (Christopher Barrington-Leigh) Date: Fri, 10 Sep 2010 10:58:08 -0700 Subject: [Numpy-discussion] weighted mean; weighted standard error of the mean (sem) In-Reply-To: <14170_1284092042_1284092042_AANLkTi=PLFWR7FPrr_bHvNxt_jdDLMCrrw2WpYHFOgbw@mail.gmail.com> References: <14170_1284092042_1284092042_AANLkTi=PLFWR7FPrr_bHvNxt_jdDLMCrrw2WpYHFOgbw@mail.gmail.com> Message-ID: Interesting. Thanks Erin, Josef and Keith. There is a nice article on this at http://www.stata.com/support/faqs/stat/supweight.html. In my case, the model I've in mind is to assume that the expected value (mean) is the same for each sample, and that the weights are/should be normalised, whence a consistent estimator for sem is straightforward (if second moments can be assumed to be well behaved?). I suspect that this (survey-like) case is also one of the two most standard/most common expression that people want when they ask for an s.e. of the mean for a weighted dataset. The other would be when the weights are not to be normalised, but represent standard errors on the individual measurements. Surely what one wants, in the end, is a single function (or whatever) called mean or sem which calculates different values for different specified choices of model (assumptions)? And where possible that it has a default model in mind for when none is specified? thanks, Chris On Thu, Sep 9, 2010 at 9:13 PM, Keith Goodman wrote: > >>>> ma.std() > >> 3.2548815339711115 > > > > or maybe `w` reflects an underlying sampling scheme and you should > > sample in the bootstrap according to w ? > > Yes.... > > > if weighted average is a sum of linear functions of (normal) > > distributed random variables, it still depends on whether the > > individual observations have the same or different variances, e.g. > > http://en.wikipedia.org/wiki/Weighted_mean#Statistical_properties > > ...lots of possibilities. As you have shown the problem is not yet > well defined. Not much specification needed for the weighted mean, > lots needed for the standard error of the weighted mean. > > > What I can't figure out is whether if you assume simga_i = sigma for > > all observation i, do we use the weighted or the unweighted variance > > to get an estimate of sigma. And I'm not able to replicate with simple > > calculations what statsmodels.WLS gives me. > > My guess: if all you want is sigma of the individual i and you know > sigma is the same for all i, then I suppose you don't care about the > weight. > > > > > ??? > > > > Josef From adam at superbitbucket.com Fri Sep 10 14:40:54 2010 From: adam at superbitbucket.com (Adam) Date: Fri, 10 Sep 2010 14:40:54 -0400 Subject: [Numpy-discussion] multiple 2d vs n-dimensional arrays Message-ID: <4C8A7BB6.7050101@superbitbucket.com> I'm keeping a large number of data points in multiple 2d arrays, for example: class c(object): def __init__(self): self.a = np.zeros((24, 60)) self.b = np.zeros((24, 60)) ... After processing the data, I'm serializing these to disk for future reference/post-processing. It's a largish amount of data and is only going to get larger. Would it be more efficient (in terms of memory/disk storage to use a single n-dimensional array: self.a = np.zeros((24, 60, 5)) What other advantages (if any) would I gain from storing the data in a single array rather than multiple? The deeper into this project I get, the more I am probably going to need to correlate data points from one or more of the arrays to one or more of the other arrays. I think I just answered my own question... -- THE SuperBitBucket.com You know you're a geek when you can hold an entire conversation with friends in linux CLI talk, i.e. 'cat /proc/lookattheassonthat!' or 'cd /pub/beer' From josef.pktd at gmail.com Fri Sep 10 15:01:56 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 10 Sep 2010 15:01:56 -0400 Subject: [Numpy-discussion] weighted mean; weighted standard error of the mean (sem) In-Reply-To: References: <14170_1284092042_1284092042_AANLkTi=PLFWR7FPrr_bHvNxt_jdDLMCrrw2WpYHFOgbw@mail.gmail.com> Message-ID: On Fri, Sep 10, 2010 at 1:58 PM, Christopher Barrington-Leigh wrote: > Interesting. Thanks Erin, Josef and Keith. thanks to the stata page at least I figured out that WLS is aweights with asumption mu_i = mu import numpy as np from scikits.statsmodels import WLS w0 = np.arange(20) % 4 w = 1.*w0/w0.sum() y = 2 + np.random.randn(20) >>> res = WLS(y, np.ones(20), weights=w).fit() >>> print res.params, res.bse [ 2.29083069] [ 0.17562867] >>> m = np.dot(w, y) >>> m 2.2908306865128401 >>> s2u = 1/(nobs-1.) * np.dot(w, (y - m)**2) >>> s2u 0.030845429945278956 >>> np.sqrt(s2u) 0.17562867062435722 > > There is a nice article on this at > http://www.stata.com/support/faqs/stat/supweight.html. In my case, the > model I've in mind is to assume that the expected value (mean) is the same > for each sample, and that the weights are/should be normalised, whence a > consistent estimator for sem is straightforward (if second moments can > be assumed to be > well behaved?). I suspect that this (survey-like) case is also one of > the two most standard/most common > expression that people want when they ask for an s.e. of the mean for > a weighted dataset. The other would be when the weights are not to be > normalised, but represent standard errors on the individual > measurements. > > Surely what one wants, in the end, is a single function (or whatever) > called mean or sem which calculates different values for different > specified choices of model (assumptions)? And where possible that it has a > default model in mind for when none is specified? I find aweights and pweights still confusing, plus necessary auxillary assumptions. I don't find Stata docs very helpful, I almost never find a clear description of the formulas (and I don't have any Stata books). If you have or write some examples that show or apply in the different cases, then this would be very helpful to get a structure into this area, weighting and survey sampling, and population versus clustered or stratified sample statistics. I'm still pretty lost with the literature on surveys. Josef > > thanks, > Chris > > On Thu, Sep 9, 2010 at 9:13 PM, Keith Goodman wrote: >> >>>> ma.std() >> >> ? 3.2548815339711115 >> > >> > or maybe `w` reflects an underlying sampling scheme and you should >> > sample in the bootstrap according to w ? >> >> Yes.... >> >> > if weighted average is a sum of linear functions of (normal) >> > distributed random variables, it still depends on whether the >> > individual observations have the same or different variances, e.g. >> > http://en.wikipedia.org/wiki/Weighted_mean#Statistical_properties >> >> ...lots of possibilities. As you have shown the problem is not yet >> well defined. Not much specification needed for the weighted mean, >> lots needed for the standard error of the weighted mean. >> >> > What I can't figure out is whether if you assume simga_i = sigma for >> > all observation i, do we use the weighted or the unweighted variance >> > to get an estimate of sigma. And I'm not able to replicate with simple >> > calculations what statsmodels.WLS gives me. >> >> My guess: if all you want is sigma of the individual i and you know >> sigma is the same for all i, then I suppose you don't care about the >> weight. >> >> > >> > ??? >> > >> > Josef > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From borreguero at gmail.com Fri Sep 10 16:39:14 2010 From: borreguero at gmail.com (Jose Borreguero) Date: Fri, 10 Sep 2010 16:39:14 -0400 Subject: [Numpy-discussion] adding two pseudoinverse matrixes Message-ID: Dear Numpy users, I have to solve for Z in the following equation Z^(-1) = A^(-1) - B^(-1), where A and B are covariance matrices with zero determinant. I have never used pseudoinverse matrixes, could anybody please point to me any cautions I have to take when solving this equation for Z? The brute force approach linalg.pinv( linalg.pinv(A) - lingal.pinv(B) ) gives me a matrix with all entries equal to 'infinity'. -Jose Borreguero -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Sep 10 17:02:15 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 10 Sep 2010 15:02:15 -0600 Subject: [Numpy-discussion] adding two pseudoinverse matrixes In-Reply-To: References: Message-ID: On Fri, Sep 10, 2010 at 2:39 PM, Jose Borreguero wrote: > Dear Numpy users, > > I have to solve for Z in the following equation Z^(-1) = A^(-1) - B^(-1), > where A and B are covariance matrices with zero determinant. > > I have never used pseudoinverse matrixes, could anybody please point to me > any cautions I have to take when solving this equation for Z? The brute > force approach linalg.pinv( linalg.pinv(A) - lingal.pinv(B) ) gives me a > matrix with all entries equal to 'infinity'. > > Similar sorts of equations turn up in Kalman filters. You can also try tricks like Z = B * (B - A)^-1 * A . Where does this problem come from? There might be a better formulation. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From borreguero at gmail.com Fri Sep 10 17:11:51 2010 From: borreguero at gmail.com (Jose Borreguero) Date: Fri, 10 Sep 2010 17:11:51 -0400 Subject: [Numpy-discussion] adding two pseudoinverse matrixes In-Reply-To: References: Message-ID: Thanks for your suggestion, Chuck. The equation arises in the substraction of two harmonic potentials V and V': V' = 1/2 x^t * A^(-1) * x V= 1/2 x^t * B^(-1) * x V'-V = 1/2 x^t * ( A^(-1) - B^(-1) ) * x = 1/2 x^t * Z^(-1) * x A is the covariance matrix of the coordinates x in a molecular dynamics simulation, A = . Same goes for B. -Jose On Fri, Sep 10, 2010 at 5:02 PM, Charles R Harris wrote: > > > On Fri, Sep 10, 2010 at 2:39 PM, Jose Borreguero wrote: > >> Dear Numpy users, >> >> I have to solve for Z in the following equation Z^(-1) = A^(-1) - B^(-1), >> where A and B are covariance matrices with zero determinant. >> >> I have never used pseudoinverse matrixes, could anybody please point to me >> any cautions I have to take when solving this equation for Z? The brute >> force approach linalg.pinv( linalg.pinv(A) - lingal.pinv(B) ) gives me a >> matrix with all entries equal to 'infinity'. >> >> > Similar sorts of equations turn up in Kalman filters. You can also try > tricks like Z = B * (B - A)^-1 * A . Where does this problem come from? > There might be a better formulation. > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hagen at zhuliguan.net Fri Sep 10 18:57:55 2010 From: hagen at zhuliguan.net (=?ISO-8859-1?Q?Hagen_F=FCrstenau?=) Date: Sat, 11 Sep 2010 00:57:55 +0200 Subject: [Numpy-discussion] dot() performance depends on data? In-Reply-To: References: <4C8A6C8B.3030007@zhuliguan.net> Message-ID: <4C8AB7F3.4090105@zhuliguan.net> > I don't see a difference on my computer. Could you post an example? I'm attaching a small benchmark script. It multiplies two example arrays loaded from the file "data" (which you can download from http://zhuliguan.net/data (157K)), and compares this with multiplying two random arrays. On two different machines I get an output like the following: $ python mm.py 0.0121581554413 0.000909090042114 So the example arrays take more than 10 times as long! Hope this helps in understanding what's going on... Cheers, Hagen -------------- next part -------------- A non-text attachment was scrubbed... Name: mm.py Type: text/x-python Size: 303 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: OpenPGP digital signature URL: From josef.pktd at gmail.com Fri Sep 10 19:46:05 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 10 Sep 2010 19:46:05 -0400 Subject: [Numpy-discussion] weighted random integers Message-ID: I saw some questions on the web how to create non-uniform random integers in python. I don't know what the best way is but here is another way that looks reasonably fast >>> rvs = np.dot(np.random.multinomial(1, [0.1, 0.2, 0.5, 0.2], size=1000000),np.arange(4)) >>> np.bincount(rvs)/1000000. array([ 0.099741, 0.199943, 0.499317, 0.200999]) Josef From charlesr.harris at gmail.com Fri Sep 10 20:15:16 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 10 Sep 2010 18:15:16 -0600 Subject: [Numpy-discussion] weighted random integers In-Reply-To: References: Message-ID: On Fri, Sep 10, 2010 at 5:46 PM, wrote: > I saw some questions on the web how to create non-uniform random > integers in python. > > I don't know what the best way is but here is another way that looks > reasonably fast > > >>> rvs = np.dot(np.random.multinomial(1, [0.1, 0.2, 0.5, 0.2], > size=1000000),np.arange(4)) > > >>> np.bincount(rvs)/1000000. > array([ 0.099741, 0.199943, 0.499317, 0.200999]) > > This looks like a good case for the inverse cdf approach, at least for smallish ranges of integers. Searchsorted on an array of appropriate values should do the trick. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Sep 10 20:28:45 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 10 Sep 2010 18:28:45 -0600 Subject: [Numpy-discussion] weighted random integers In-Reply-To: References: Message-ID: On Fri, Sep 10, 2010 at 6:15 PM, Charles R Harris wrote: > > > On Fri, Sep 10, 2010 at 5:46 PM, wrote: > >> I saw some questions on the web how to create non-uniform random >> integers in python. >> >> I don't know what the best way is but here is another way that looks >> reasonably fast >> >> >>> rvs = np.dot(np.random.multinomial(1, [0.1, 0.2, 0.5, 0.2], >> size=1000000),np.arange(4)) >> >> >>> np.bincount(rvs)/1000000. >> array([ 0.099741, 0.199943, 0.499317, 0.200999]) >> >> > This looks like a good case for the inverse cdf approach, at least for > smallish ranges of integers. Searchsorted on an array of appropriate values > should do the trick. > > For instance, weight 0..3 by 1..4, then In [14]: w = arange(1,5) In [15]: p = cumsum(w)/float(w.sum()) In [16]: bincount(p.searchsorted(random(1000000)))/1e6 Out[16]: array([ 0.100336, 0.200382, 0.299132, 0.40015 ]) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Fri Sep 10 20:32:13 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 10 Sep 2010 20:32:13 -0400 Subject: [Numpy-discussion] weighted random integers In-Reply-To: References: Message-ID: On Fri, Sep 10, 2010 at 8:28 PM, Charles R Harris wrote: > > > On Fri, Sep 10, 2010 at 6:15 PM, Charles R Harris > wrote: >> >> >> On Fri, Sep 10, 2010 at 5:46 PM, wrote: >>> >>> I saw some questions on the web how to create non-uniform random >>> integers in python. >>> >>> I don't know what the best way is but here is another way that looks >>> reasonably fast >>> >>> >>> rvs = np.dot(np.random.multinomial(1, [0.1, 0.2, 0.5, 0.2], >>> >>> size=1000000),np.arange(4)) >>> >>> >>> np.bincount(rvs)/1000000. >>> array([ 0.099741, ?0.199943, ?0.499317, ?0.200999]) >>> >> >> This looks like a good case for the inverse cdf approach, at least for >> smallish ranges of integers. Searchsorted on an array of appropriate values >> should do the trick. >> > > For instance, weight 0..3 by 1..4, then > > In [14]: w = arange(1,5) > > In [15]: p = cumsum(w)/float(w.sum()) > > In [16]: bincount(p.searchsorted(random(1000000)))/1e6 > Out[16]: array([ 0.100336,? 0.200382,? 0.299132,? 0.40015 ]) Looks good, it feels faster and takes less memory, I guess. Josef > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From ben.root at ou.edu Fri Sep 10 20:33:01 2010 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 10 Sep 2010 19:33:01 -0500 Subject: [Numpy-discussion] multiple 2d vs n-dimensional arrays In-Reply-To: <4C8A7BB6.7050101@superbitbucket.com> References: <4C8A7BB6.7050101@superbitbucket.com> Message-ID: On Fri, Sep 10, 2010 at 1:40 PM, Adam wrote: > I'm keeping a large number of data points in multiple 2d arrays, for > example: > > class c(object): > def __init__(self): > self.a = np.zeros((24, 60)) > self.b = np.zeros((24, 60)) > ... > > After processing the data, I'm serializing these to disk for future > reference/post-processing. It's a largish amount of data and is only > going to get larger. > > Would it be more efficient (in terms of memory/disk storage to use a > single n-dimensional array: > > self.a = np.zeros((24, 60, 5)) > > What other advantages (if any) would I gain from storing the data in a > single array rather than multiple? The deeper into this project I get, > the more I am probably going to need to correlate data points from one > or more of the arrays to one or more of the other arrays. > > I think I just answered my own question... > > Adam, One argument *against* merging all of the data into a single array is that the array needs a contiguous portion of memory. If the quantity of data reaches a certain point, the OS may have to do more work in allocating the space for your array. Smaller chunks may fit better. Of course, this is entirely dependent upon usage patterns, available RAM, OS, the day of the week, and the color of your socks. YMMV. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Fri Sep 10 20:41:55 2010 From: cournape at gmail.com (David Cournapeau) Date: Sat, 11 Sep 2010 09:41:55 +0900 Subject: [Numpy-discussion] dot() performance depends on data? In-Reply-To: References: <4C8A6C8B.3030007@zhuliguan.net> Message-ID: On Sat, Sep 11, 2010 at 2:57 AM, Charles R Harris wrote: > > > On Fri, Sep 10, 2010 at 11:36 AM, Hagen F?rstenau > wrote: >> >> Hi, >> >> I'm multiplying two 1000x1000 arrays with numpy.dot() and seeing >> significant performance differences depending on the data. It seems to >> take much longer on matrices with many zeros than on random ones. I >> don't know much about optimized MM implementations, but is this normal >> behavior for some reason? >> > > Multiplication by zero used to be faster than multiplication by random > numbers. However, modern hardware and compilers may have changed that to > pretty much a wash. More likely you are seeing cache issues due to data > localization or even variations in the time given the thread running the > multiplication. That's actually most likely a denormal issue. The a and b matrix (from mm.py) have many very small numbers, which could cause numbers to be denormal. Maybe a has more denormals than b. Denormal cause significant performance issues on Intel hardware at least. Unfortunately, we don't have a way in numpy to check for denormal that I know of. cheers, David From charlesr.harris at gmail.com Fri Sep 10 20:44:35 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 10 Sep 2010 18:44:35 -0600 Subject: [Numpy-discussion] weighted random integers In-Reply-To: References: Message-ID: On Fri, Sep 10, 2010 at 6:32 PM, wrote: > On Fri, Sep 10, 2010 at 8:28 PM, Charles R Harris > wrote: > > > > > > On Fri, Sep 10, 2010 at 6:15 PM, Charles R Harris > > wrote: > >> > >> > >> On Fri, Sep 10, 2010 at 5:46 PM, wrote: > >>> > >>> I saw some questions on the web how to create non-uniform random > >>> integers in python. > >>> > >>> I don't know what the best way is but here is another way that looks > >>> reasonably fast > >>> > >>> >>> rvs = np.dot(np.random.multinomial(1, [0.1, 0.2, 0.5, 0.2], > >>> >>> size=1000000),np.arange(4)) > >>> > >>> >>> np.bincount(rvs)/1000000. > >>> array([ 0.099741, 0.199943, 0.499317, 0.200999]) > >>> > >> > >> This looks like a good case for the inverse cdf approach, at least for > >> smallish ranges of integers. Searchsorted on an array of appropriate > values > >> should do the trick. > >> > > > > For instance, weight 0..3 by 1..4, then > > > > In [14]: w = arange(1,5) > > > > In [15]: p = cumsum(w)/float(w.sum()) > > > > In [16]: bincount(p.searchsorted(random(1000000)))/1e6 > > Out[16]: array([ 0.100336, 0.200382, 0.299132, 0.40015 ]) > > Looks good, it feels faster and takes less memory, I guess. > > It's about 10x faster. In [30]: timeit p.searchsorted(random(1000000)) 10 loops, best of 3: 37.6 ms per loop In [31]: timeit np.dot(np.random.multinomial(1, [0.1, 0.2, 0.5, 0.2], size=1000000),np.arange(4)) 1 loops, best of 3: 363 ms per loop Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Sep 10 20:47:48 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 10 Sep 2010 18:47:48 -0600 Subject: [Numpy-discussion] dot() performance depends on data? In-Reply-To: References: <4C8A6C8B.3030007@zhuliguan.net> Message-ID: On Fri, Sep 10, 2010 at 6:41 PM, David Cournapeau wrote: > On Sat, Sep 11, 2010 at 2:57 AM, Charles R Harris > wrote: > > > > > > On Fri, Sep 10, 2010 at 11:36 AM, Hagen F?rstenau > > wrote: > >> > >> Hi, > >> > >> I'm multiplying two 1000x1000 arrays with numpy.dot() and seeing > >> significant performance differences depending on the data. It seems to > >> take much longer on matrices with many zeros than on random ones. I > >> don't know much about optimized MM implementations, but is this normal > >> behavior for some reason? > >> > > > > Multiplication by zero used to be faster than multiplication by random > > numbers. However, modern hardware and compilers may have changed that to > > pretty much a wash. More likely you are seeing cache issues due to data > > localization or even variations in the time given the thread running the > > multiplication. > > That's actually most likely a denormal issue. The a and b matrix (from > mm.py) have many very small numbers, which could cause numbers to be > denormal. Maybe a has more denormals than b. Denormal cause > significant performance issues on Intel hardware at least. > > Unfortunately, we don't have a way in numpy to check for denormal that > I know of. > > The matrices could be scaled up to check that. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Fri Sep 10 21:04:56 2010 From: cournape at gmail.com (David Cournapeau) Date: Sat, 11 Sep 2010 10:04:56 +0900 Subject: [Numpy-discussion] dot() performance depends on data? In-Reply-To: References: <4C8A6C8B.3030007@zhuliguan.net> Message-ID: On Sat, Sep 11, 2010 at 9:47 AM, Charles R Harris wrote: > > > On Fri, Sep 10, 2010 at 6:41 PM, David Cournapeau > wrote: >> >> On Sat, Sep 11, 2010 at 2:57 AM, Charles R Harris >> wrote: >> > >> > >> > On Fri, Sep 10, 2010 at 11:36 AM, Hagen F?rstenau >> > wrote: >> >> >> >> Hi, >> >> >> >> I'm multiplying two 1000x1000 arrays with numpy.dot() and seeing >> >> significant performance differences depending on the data. It seems to >> >> take much longer on matrices with many zeros than on random ones. I >> >> don't know much about optimized MM implementations, but is this normal >> >> behavior for some reason? >> >> >> > >> > Multiplication by zero used to be faster than multiplication by random >> > numbers. However, modern hardware and compilers may have changed that to >> > pretty much a wash. More likely you are seeing cache issues due to data >> > localization or even variations in the time given the thread running the >> > multiplication. >> >> That's actually most likely a denormal issue. The a and b matrix (from >> mm.py) have many very small numbers, which could cause numbers to be >> denormal. Maybe a has more denormals than b. Denormal cause >> significant performance issues on Intel hardware at least. >> >> Unfortunately, we don't have a way in numpy to check for denormal that >> I know of. >> > > The matrices could be scaled up to check that. Indeed - and I misread the script anyway, I should not investigate this kind of things after waking up :) Anyway, seems it is indeed a denormal issue, as adding a small (1e-10) constant gives same speed for both timings. cheers, David From charlesr.harris at gmail.com Fri Sep 10 21:10:43 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 10 Sep 2010 19:10:43 -0600 Subject: [Numpy-discussion] dot() performance depends on data? In-Reply-To: References: <4C8A6C8B.3030007@zhuliguan.net> Message-ID: On Fri, Sep 10, 2010 at 7:04 PM, David Cournapeau wrote: > On Sat, Sep 11, 2010 at 9:47 AM, Charles R Harris > wrote: > > > > > > On Fri, Sep 10, 2010 at 6:41 PM, David Cournapeau > > wrote: > >> > >> On Sat, Sep 11, 2010 at 2:57 AM, Charles R Harris > >> wrote: > >> > > >> > > >> > On Fri, Sep 10, 2010 at 11:36 AM, Hagen F?rstenau < > hagen at zhuliguan.net> > >> > wrote: > >> >> > >> >> Hi, > >> >> > >> >> I'm multiplying two 1000x1000 arrays with numpy.dot() and seeing > >> >> significant performance differences depending on the data. It seems > to > >> >> take much longer on matrices with many zeros than on random ones. I > >> >> don't know much about optimized MM implementations, but is this > normal > >> >> behavior for some reason? > >> >> > >> > > >> > Multiplication by zero used to be faster than multiplication by random > >> > numbers. However, modern hardware and compilers may have changed that > to > >> > pretty much a wash. More likely you are seeing cache issues due to > data > >> > localization or even variations in the time given the thread running > the > >> > multiplication. > >> > >> That's actually most likely a denormal issue. The a and b matrix (from > >> mm.py) have many very small numbers, which could cause numbers to be > >> denormal. Maybe a has more denormals than b. Denormal cause > >> significant performance issues on Intel hardware at least. > >> > >> Unfortunately, we don't have a way in numpy to check for denormal that > >> I know of. > >> > > > > The matrices could be scaled up to check that. > > Indeed - and I misread the script anyway, I should not investigate > this kind of things after waking up :) > > Anyway, seems it is indeed a denormal issue, as adding a small (1e-10) > constant gives same speed for both timings. > Ah, that's probably one of the reasons the lapack lstsqr's routine scales the matrices to a nice range. I was wondering. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From hagen at zhuliguan.net Sat Sep 11 05:58:56 2010 From: hagen at zhuliguan.net (=?UTF-8?B?SGFnZW4gRsO8cnN0ZW5hdQ==?=) Date: Sat, 11 Sep 2010 11:58:56 +0200 Subject: [Numpy-discussion] dot() performance depends on data? In-Reply-To: References: <4C8A6C8B.3030007@zhuliguan.net> Message-ID: <4C8B52E0.9060600@zhuliguan.net> > Anyway, seems it is indeed a denormal issue, as adding a small (1e-10) > constant gives same speed for both timings. With adding 1e-10 or clipping to 0 at 1e-150, I still get a slowdown of about 30% compared with the random arrays. Any explanation for that? Cheers, Hagen -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: OpenPGP digital signature URL: From matthieu.brucher at gmail.com Sat Sep 11 06:10:55 2010 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Sat, 11 Sep 2010 12:10:55 +0200 Subject: [Numpy-discussion] dot() performance depends on data? In-Reply-To: <4C8B52E0.9060600@zhuliguan.net> References: <4C8A6C8B.3030007@zhuliguan.net> <4C8B52E0.9060600@zhuliguan.net> Message-ID: Denormal numbers are a tricky beast. You may have to change the clip or the shift depending on the processor you have. It's no wonder that processors and thus compilers have options to round denormals to zero. Matthieu 2010/9/11 Hagen F?rstenau : >> Anyway, seems it is indeed a denormal issue, as adding a small (1e-10) >> constant gives same speed for both timings. > > With adding 1e-10 or clipping to 0 at 1e-150, I still get a slowdown of > about 30% compared with the random arrays. Any explanation for that? > > Cheers, > Hagen > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher From massimodisasha at gmail.com Sat Sep 11 08:45:38 2010 From: massimodisasha at gmail.com (Massimo Di Stefano) Date: Sat, 11 Sep 2010 14:45:38 +0200 Subject: [Numpy-discussion] scan array to extract min-max values (with if condition) Message-ID: Hello All, i need to extract data from an array, that are inside a rectangle area defined as : N, S, E, W = 234560.94503118, 234482.56929822, 921336.53116178, 921185.3779625 the data are in a csv (comma delimited text file, with 3 columns X,Y,Z) #X,Y,Z 3020081.5500,769999.3100,0.0300 3020086.2000,769991.6500,0.4600 3020099.6600,769996.2700,0.9000 ... ... i read it using " numpy.loadtxt " data : http://www.geofemengineering.it/data/csv.txt 5,3 mb (158735 rows) to extract data that are inside the boundy-box area (N, S, E, W) i'm using a loop inside a function like : import numpy as np def getMinMaxBB(data, N, S, E, W): mydata = data * 0.3048006096012 for i in range(len(mydata)): if mydata[i,0] < E or mydata[i,0] > W or mydata[i,1] < N or mydata[i,1] > S : if i == 0: newdata = np.array((mydata[i,0],mydata[i,1],mydata[i,2]), float) else : newdata = np.vstack((newdata,(mydata[i,0], mydata[i,1], mydata[i,2]))) results = {} results['Max_Z'] = newdata.max(0)[2] results['Min_Z'] = newdata.min(0)[2] results['Num_P'] = len(newdata) return results N, S, E, W = 234560.94503118, 234482.56929822, 921336.53116178, 921185.3779625 data = '/Users/sasha/csv.txt' mydata = np.loadtxt(data, comments='#', delimiter=',') out = getMinMaxBB(mydata, N, S, E, W) print out This method works, but maybe is not soo fast, have you any hints on how to improve code with better performance ? thanks!!! Massimo. From brett.olsen at gmail.com Sat Sep 11 10:19:49 2010 From: brett.olsen at gmail.com (Brett Olsen) Date: Sat, 11 Sep 2010 09:19:49 -0500 Subject: [Numpy-discussion] scan array to extract min-max values (with if condition) In-Reply-To: References: Message-ID: On Sat, Sep 11, 2010 at 7:45 AM, Massimo Di Stefano wrote: > Hello All, > > i need to extract data from an array, that are inside a > rectangle area defined as : > > N, S, E, W = 234560.94503118, 234482.56929822, 921336.53116178, 921185.3779625 > > the data are in a csv (comma delimited text file, with 3 columns X,Y,Z) > > #X,Y,Z > 3020081.5500,769999.3100,0.0300 > 3020086.2000,769991.6500,0.4600 > 3020099.6600,769996.2700,0.9000 > ... > ... > > i read it using " numpy.loadtxt " > > data : > > http://www.geofemengineering.it/data/csv.txt ? ? 5,3 mb (158735 rows) > > to extract data that are inside the boundy-box area (N, S, E, W) i'm using a loop > inside a function like : > > import numpy as np > > def getMinMaxBB(data, N, S, E, W): > ? ? ? ?mydata = data * 0.3048006096012 > ? ? ? ?for i in range(len(mydata)): > ? ? ? ? ? ? ? ?if mydata[i,0] < E or mydata[i,0] > W or mydata[i,1] < N or mydata[i,1] > S : > ? ? ? ? ? ? ? ? ? ? ? ?if i == 0: > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?newdata = np.array((mydata[i,0],mydata[i,1],mydata[i,2]), float) > ? ? ? ? ? ? ? ? ? ? ? ?else : > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?newdata = np.vstack((newdata,(mydata[i,0], mydata[i,1], mydata[i,2]))) > ? ? ? ?results = {} > ? ? ? ?results['Max_Z'] = newdata.max(0)[2] > ? ? ? ?results['Min_Z'] = newdata.min(0)[2] > ? ? ? ?results['Num_P'] = len(newdata) > ? ? ? ?return results > > > N, S, E, W = 234560.94503118, 234482.56929822, 921336.53116178, 921185.3779625 > data = '/Users/sasha/csv.txt' > mydata = np.loadtxt(data, comments='#', delimiter=',') > out = getMinMaxBB(mydata, N, S, E, W) > > print out Use boolean arrays to index the parts of your array that you want to look at: def newGetMinMax(data, N, S, E, W): mydata = data * 0.3048006096012 mask = np.zeros(mydata.shape[0], dtype=bool) mask |= mydata[:,0] < E mask |= mydata[:,0] > W mask |= mydata[:,1] < N mask |= mydata[:,1] > S results = {} results['Max_Z'] = mydata[mask,2].max() results['Min_Z'] = mydata[mask,2].min() results['Num_P'] = mask.sum() return results This runs about 5000 times faster on my machine. Brett From massimodisasha at gmail.com Sat Sep 11 12:51:56 2010 From: massimodisasha at gmail.com (Massimo Di Stefano) Date: Sat, 11 Sep 2010 18:51:56 +0200 Subject: [Numpy-discussion] scan array to extract min-max values (with if condition) In-Reply-To: References: Message-ID: That's awesome! masked array are defintley what i need! thanks to point my attention on it! best regards, Massimo. Il giorno 11/set/2010, alle ore 16.19, Brett Olsen ha scritto: > On Sat, Sep 11, 2010 at 7:45 AM, Massimo Di Stefano > wrote: >> Hello All, >> >> i need to extract data from an array, that are inside a >> rectangle area defined as : >> >> N, S, E, W = 234560.94503118, 234482.56929822, 921336.53116178, 921185.3779625 >> >> the data are in a csv (comma delimited text file, with 3 columns X,Y,Z) >> >> #X,Y,Z >> 3020081.5500,769999.3100,0.0300 >> 3020086.2000,769991.6500,0.4600 >> 3020099.6600,769996.2700,0.9000 >> ... >> ... >> >> i read it using " numpy.loadtxt " >> >> data : >> >> http://www.geofemengineering.it/data/csv.txt 5,3 mb (158735 rows) >> >> to extract data that are inside the boundy-box area (N, S, E, W) i'm using a loop >> inside a function like : >> >> import numpy as np >> >> def getMinMaxBB(data, N, S, E, W): >> mydata = data * 0.3048006096012 >> for i in range(len(mydata)): >> if mydata[i,0] < E or mydata[i,0] > W or mydata[i,1] < N or mydata[i,1] > S : >> if i == 0: >> newdata = np.array((mydata[i,0],mydata[i,1],mydata[i,2]), float) >> else : >> newdata = np.vstack((newdata,(mydata[i,0], mydata[i,1], mydata[i,2]))) >> results = {} >> results['Max_Z'] = newdata.max(0)[2] >> results['Min_Z'] = newdata.min(0)[2] >> results['Num_P'] = len(newdata) >> return results >> >> >> N, S, E, W = 234560.94503118, 234482.56929822, 921336.53116178, 921185.3779625 >> data = '/Users/sasha/csv.txt' >> mydata = np.loadtxt(data, comments='#', delimiter=',') >> out = getMinMaxBB(mydata, N, S, E, W) >> >> print out > > Use boolean arrays to index the parts of your array that you want to look at: > > def newGetMinMax(data, N, S, E, W): > mydata = data * 0.3048006096012 > mask = np.zeros(mydata.shape[0], dtype=bool) > mask |= mydata[:,0] < E > mask |= mydata[:,0] > W > mask |= mydata[:,1] < N > mask |= mydata[:,1] > S > results = {} > results['Max_Z'] = mydata[mask,2].max() > results['Min_Z'] = mydata[mask,2].min() > results['Num_P'] = mask.sum() > return results > > This runs about 5000 times faster on my machine. > > Brett > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From lpc at cmu.edu Sat Sep 11 13:56:11 2010 From: lpc at cmu.edu (Luis Pedro Coelho) Date: Sat, 11 Sep 2010 13:56:11 -0400 Subject: [Numpy-discussion] ANNOUNCE: mahotas 0.5 In-Reply-To: References: <20100909225011.4b0ba037@oakeshott> Message-ID: <201009111356.16053.lpc@cmu.edu> On Friday, September 10, 2010 03:40:33 am Sebastian Haase wrote: > Hi Luis, > > thanks for the announcement. How would you compare mahotas to scipy's > ndimage ? Are you using ndimage in mahotas at all ? Hi Sebastian, In general there is little overlap (there are 1 or 2 functions which are replicated). I wrote mahotas mostly to get functions that I wasn't finding elsewhere (or, occasionally, to make them faster). Ndimage, in some ways, contains more basic functions, mahotas has a bit more advanced functions (at the cost of having a somewhat idiosyncratically chosen set of functions). I do use ndimage in a couple of places (mostly to do convolution). So, they complement themselves rather than compete. Here are a couple of differences in philosophy: - ndimage is *always* n-D. Mahotas is mostly n-D but some functions are specialised to 2-D (patches always welcome if you want to extend them). - mahotas is written in templated C++, ndimage is C with macros. - ndimage tries to be very generic in its interfaces (which makes some of them hard to use at first), mahotas tries to have natural interfaces. HTH, Luis -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: From massimodisasha at gmail.com Sat Sep 11 15:53:27 2010 From: massimodisasha at gmail.com (Massimo Di Stefano) Date: Sat, 11 Sep 2010 21:53:27 +0200 Subject: [Numpy-discussion] scan array to extract min-max values (with if condition) In-Reply-To: References: Message-ID: <24AD26B0-C1F7-49B7-9EA5-81DCBF058197@gmail.com> Brett, i tried a different way to solve the problem, using : ############# import os fpath = '/Users/sasha/py/' input_fp = open( os.path.join(fpath, 'BE3730072600WC20050817.txt'), 'r' ) input_file = input_fp.readlines() N = 234560.94503118 S = 234482.56929822 E = 921336.53116178 W = 921185.3779625 xL = [] yL = [] zL = [] for index, line in enumerate( input_file ): if index == 0: print 'skipping header line...' else: x, y, z = line.split(',') xL.append( float(x) * 0.3048006096012 ) yL.append( float(y) * 0.3048006096012 ) zL.append( float(z) * 0.3048006096012 ) xLr = [] yLr = [] zLr = [] for coords in zip(xL, yL, zL): if W < coords[0] < E and S < coords[1] < N: xLr.append( coords[0] ) yLr.append( coords[1] ) zLr.append( coords[2] ) elements = len(xLr) minZ = min(zLr) maxZ = max(zLr) ############ using the same input file i posted early, it give me 966 elements instead of 158734 elements gived by your "MASK" example the input file contains 158734 elements, this means the mask code : > mask |= mydata[:,0] < E > mask |= mydata[:,0] > W > mask |= mydata[:,1] < N > mask |= mydata[:,1] > S is not working as aspected have you hints on how to get working the "MASK" code ? as it is now it pick all the points in the "mydata" array. thanks! Massimo. Il giorno 11/set/2010, alle ore 16.19, Brett Olsen ha scritto: > On Sat, Sep 11, 2010 at 7:45 AM, Massimo Di Stefano > wrote: >> Hello All, >> >> i need to extract data from an array, that are inside a >> rectangle area defined as : >> >> N, S, E, W = 234560.94503118, 234482.56929822, 921336.53116178, 921185.3779625 >> >> the data are in a csv (comma delimited text file, with 3 columns X,Y,Z) >> >> #X,Y,Z >> 3020081.5500,769999.3100,0.0300 >> 3020086.2000,769991.6500,0.4600 >> 3020099.6600,769996.2700,0.9000 >> ... >> ... >> >> i read it using " numpy.loadtxt " >> >> data : >> >> http://www.geofemengineering.it/data/csv.txt 5,3 mb (158735 rows) >> >> to extract data that are inside the boundy-box area (N, S, E, W) i'm using a loop >> inside a function like : >> >> import numpy as np >> >> def getMinMaxBB(data, N, S, E, W): >> mydata = data * 0.3048006096012 >> for i in range(len(mydata)): >> if mydata[i,0] < E or mydata[i,0] > W or mydata[i,1] < N or mydata[i,1] > S : >> if i == 0: >> newdata = np.array((mydata[i,0],mydata[i,1],mydata[i,2]), float) >> else : >> newdata = np.vstack((newdata,(mydata[i,0], mydata[i,1], mydata[i,2]))) >> results = {} >> results['Max_Z'] = newdata.max(0)[2] >> results['Min_Z'] = newdata.min(0)[2] >> results['Num_P'] = len(newdata) >> return results >> >> >> N, S, E, W = 234560.94503118, 234482.56929822, 921336.53116178, 921185.3779625 >> data = '/Users/sasha/csv.txt' >> mydata = np.loadtxt(data, comments='#', delimiter=',') >> out = getMinMaxBB(mydata, N, S, E, W) >> >> print out > > Use boolean arrays to index the parts of your array that you want to look at: > > def newGetMinMax(data, N, S, E, W): > mydata = data * 0.3048006096012 > mask = np.zeros(mydata.shape[0], dtype=bool) > mask |= mydata[:,0] < E > mask |= mydata[:,0] > W > mask |= mydata[:,1] < N > mask |= mydata[:,1] > S > results = {} > results['Max_Z'] = mydata[mask,2].max() > results['Min_Z'] = mydata[mask,2].min() > results['Num_P'] = mask.sum() > return results > > This runs about 5000 times faster on my machine. > > Brett > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Sat Sep 11 17:00:06 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Sat, 11 Sep 2010 23:00:06 +0200 Subject: [Numpy-discussion] scan array to extract min-max values (with if condition) In-Reply-To: <24AD26B0-C1F7-49B7-9EA5-81DCBF058197@gmail.com> References: <24AD26B0-C1F7-49B7-9EA5-81DCBF058197@gmail.com> Message-ID: <0CD4FE79-4EA7-4181-A119-55C3F96BEA0C@gmail.com> On Sep 11, 2010, at 9:53 PM, Massimo Di Stefano wrote: > have you hints on how to get working the "MASK" code ? > as it is now it pick all the points in the "mydata" array. Brett's code for the mask matched the loop of your post. However, taking a second look at it, I don't see why it would work. Mind trying something like # Selection on the NS axis yselect = (mydata[:,1] <= N) & (mydata[:,1] >= S) # selection on the EW axis xselect = (mydata[:,1] <= E) & (mydata[:,1] >= W) # Global selection selected_data = mydata[xselect & yselect] Now you can check in selected data that the coordinates are indeed in the rectangle that you want, and take the min/max of your data as needed. let me know how it goes P. From massimodisasha at gmail.com Sat Sep 11 17:46:49 2010 From: massimodisasha at gmail.com (Massimo Di Stefano) Date: Sat, 11 Sep 2010 23:46:49 +0200 Subject: [Numpy-discussion] scan array to extract min-max values (with if condition) In-Reply-To: <0CD4FE79-4EA7-4181-A119-55C3F96BEA0C@gmail.com> References: <24AD26B0-C1F7-49B7-9EA5-81DCBF058197@gmail.com> <0CD4FE79-4EA7-4181-A119-55C3F96BEA0C@gmail.com> Message-ID: Thanks Pierre, i tried it and all works fine and fast. my apologize :-( i used a wrong "if" statment to represent my needs if mydata[i,0] < E or mydata[i,0] > W or mydata[i,1] < N or mydata[i,1] > S : ^^ totally wrong for my needs^^ this "if" instead : if W < mydata[i,0] < E and S < mydata[i,1] < N: should reflect your example : yselect = (data[:,1] <= N) & (data[:,1] >= S) xselect = (data[:,0] <= E) & (data[:,0] >= W) selected_data = data[xselect & yselect] a question, how to code a masked array, as in the Brett's code, to reflect the new (right) if statment ? i'm asking this to try to learn how to use masked array. thanks a lot for your support!!! Massimo. Il giorno 11/set/2010, alle ore 23.00, Pierre GM ha scritto: > > On Sep 11, 2010, at 9:53 PM, Massimo Di Stefano wrote: > >> have you hints on how to get working the "MASK" code ? >> as it is now it pick all the points in the "mydata" array. > > Brett's code for the mask matched the loop of your post. However, taking a second look at it, I don't see why it would work. Mind trying something like > > # Selection on the NS axis > yselect = (mydata[:,1] <= N) & (mydata[:,1] >= S) > # selection on the EW axis > xselect = (mydata[:,1] <= E) & (mydata[:,1] >= W) > # Global selection > selected_data = mydata[xselect & yselect] > > Now you can check in selected data that the coordinates are indeed in the rectangle that you want, and take the min/max of your data as needed. > let me know how it goes > > P. > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From dyamins at gmail.com Sat Sep 11 19:56:44 2010 From: dyamins at gmail.com (dyamins at gmail.com) Date: Sat, 11 Sep 2010 23:56:44 +0000 Subject: [Numpy-discussion] least() of datetime objects doesn't work In-Reply-To: References: <20100819224003.GA8722@doriath.local> Message-ID: <671793467-1284249403-cardhu_decombobulator_blackberry.rim.net-30193775-@bda635.bisx.prod.on.blackberry> P Sent via BlackBerry from T-Mobile -----Original Message----- From: Charles R Harris Sender: numpy-discussion-bounces at scipy.org Date: Thu, 19 Aug 2010 18:03:29 To: Discussion of Numerical Python Reply-To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] lexsort() of datetime objects doesn't work _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion From brett.olsen at gmail.com Sun Sep 12 01:26:34 2010 From: brett.olsen at gmail.com (Brett Olsen) Date: Sun, 12 Sep 2010 00:26:34 -0500 Subject: [Numpy-discussion] scan array to extract min-max values (with if condition) In-Reply-To: References: <24AD26B0-C1F7-49B7-9EA5-81DCBF058197@gmail.com> <0CD4FE79-4EA7-4181-A119-55C3F96BEA0C@gmail.com> Message-ID: On Sat, Sep 11, 2010 at 4:46 PM, Massimo Di Stefano wrote: > Thanks Pierre, > > i tried it and all works fine and fast. > > my apologize :-( > > i used a wrong "if" statment to represent my needs > > if mydata[i,0] < E or mydata[i,0] > W or mydata[i,1] < N or mydata[i,1] > S : > > ^^ totally wrong for my needs^^ > > > this "if" ?instead : > > if W < mydata[i,0] < E and S < mydata[i,1] < N: > > should reflect your example : > > yselect = (data[:,1] <= N) & (data[:,1] >= S) > xselect = (data[:,0] <= E) & (data[:,0] >= W) > selected_data = data[xselect & yselect] > > > a question, how to code a masked array, > as in the Brett's code, to reflect the new (right) if statment ? Just replace the lines mask |= mydata[:,0] < E mask |= mydata[:,0] > W mask |= mydata[:,1] < N mask |= mydata[:,1] > S with mask &= mydata[:,0] < E mask &= mydata[:,0] > W mask &= mydata[:,1] < N mask &= mydata[:,1] > S Sorry, I wasn't paying attention to what you were actually trying to do and just duplicated the function of the code you supplied. There's a good primer on how to index with boolean arrays at http://www.scipy.org/Tentative_NumPy_Tutorial#head-d55e594d46b4f347c20efe1b4c65c92779f06268 that will explain why this works. Brett From felix.stoedter.ga at web.de Sun Sep 12 12:45:33 2010 From: felix.stoedter.ga at web.de (=?ISO-8859-15?Q?Felix_St=F6dter?=) Date: Sun, 12 Sep 2010 18:45:33 +0200 (CEST) Subject: [Numpy-discussion] Can't install numpy Message-ID: <419218887.9168336.1284309934003.JavaMail.fmail@mwmweb063> Hi numpy-team, unfortunately I can't install numpy. When I try to excecute the exe-file I get a message like "Pyhton 2.6 required". Pyhton is installed within another programme (Marc/Mentat). Can't he find it because of that? I know that it is possible to install numpy with this Python in Marc/Mentat. But how? Do you know what I can do about it? Thank you in advance. Kind regards, Felix Stoedter ___________________________________________________________ WEB.DE DSL SOMMER-SPECIAL: Surf & Phone Flat 16.000 f?r nur 19,99 €/mtl.!* http://produkte.web.de/go/DSL-Doppel-Flatrate/2 From pav at iki.fi Sun Sep 12 13:13:29 2010 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 12 Sep 2010 17:13:29 +0000 (UTC) Subject: [Numpy-discussion] Can't install numpy ... in Marc&Mentat (embedded Python) References: <419218887.9168336.1284309934003.JavaMail.fmail@mwmweb063> Message-ID: Sun, 12 Sep 2010 18:45:33 +0200, Felix St?dter wrote: > unfortunately I can't install numpy. When I try to excecute the exe-file > I get a message like "Pyhton 2.6 required". Pyhton is installed within > another programme (Marc/Mentat). Can't he find it because of that? I > know that it is possible to install numpy with this Python in > Marc/Mentat. But how? Do you know what I can do about it? I believe you should direct your question to MSC Software (whose product Marc&Mentat is), or their user support group. The point is that although installing Python modules is usually the same (i.e., if you have a pre-compiled binary package you want to install manually, copy the "numpy" folder to the correct folder, usually named "site-packages"), for Python bundled with some software the details of the installation may vary, and the best way to get good advice is to ask the authors of the particular product. -- Pauli Virtanen From david at silveregg.co.jp Sun Sep 12 21:09:32 2010 From: david at silveregg.co.jp (David) Date: Mon, 13 Sep 2010 10:09:32 +0900 Subject: [Numpy-discussion] Can't install numpy In-Reply-To: <419218887.9168336.1284309934003.JavaMail.fmail@mwmweb063> References: <419218887.9168336.1284309934003.JavaMail.fmail@mwmweb063> Message-ID: <4C8D79CC.3000005@silveregg.co.jp> On 09/13/2010 01:45 AM, Felix St?dter wrote: > Hi numpy-team, > > unfortunately I can't install numpy. When I try to excecute the exe-file I get a message like "Pyhton 2.6 required". Pyhton is installed within another programme (Marc/Mentat). Can't he find it because of that? I know that it is possible to install numpy with this Python in Marc/Mentat. But how? Do you know what I can do about it? What are you trying to do ? If Marc Mentat is using numpy internally (I don't know anything about that product), and you want to update numpy there, I strongly advise you *not* to install anything there unless they explicitely support it. If you just want to install numpy to use straight numpy, you should install it on top on python installed outside marc mentat, cheers, David From lorenz at MPA-Garching.MPG.DE Mon Sep 13 07:51:51 2010 From: lorenz at MPA-Garching.MPG.DE (Lorenz =?iso-8859-1?q?H=FCdepohl?=) Date: Mon, 13 Sep 2010 13:51:51 +0200 Subject: [Numpy-discussion] Difference in astype() for scalars and arrays? Message-ID: <201009131351.51541.lorenz@mpa-garching.mpg.de> Hi All, I'm fiddling with some endianness / binary representation issue and stumbled upon this puzzling behavior of astype(): It does not seem to work as expected - for me at least ;) - when given a scalar: >>> import numpy as np >>> np.array(42).astype(">i4").dtype dtype('>i4') >>> np.array(42, dtype=">i4").astype('>i4').dtype dtype('int32') >>> >>> np.array([42]).astype(">i4").dtype dtype('>i4') >>> np.array([42], dtype=">i4").astype('>i4').dtype dtype('>i4') Shouldn't I always get dtype('>i4')? Regards, Lorenz PS: >>> np.version.version '1.4.0.dev7417' From pav at iki.fi Mon Sep 13 08:16:49 2010 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 13 Sep 2010 12:16:49 +0000 (UTC) Subject: [Numpy-discussion] Difference in astype() for scalars and arrays? References: <201009131351.51541.lorenz@mpa-garching.mpg.de> Message-ID: Mon, 13 Sep 2010 13:51:51 +0200, Lorenz H?depohl wrote: > I'm fiddling with some endianness / binary representation issue and > stumbled upon this puzzling behavior of astype(): It does not seem to > work as expected - for me at least ;) - when given a scalar: > >>>> import numpy as np >>>> np.array(42).astype(">i4").dtype > dtype('>i4') >>>> np.array(42, dtype=">i4").astype('>i4').dtype > dtype('int32') Doesn't seem correct -- please file a bug ticket. -- Pauli Virtanen From oliphant at enthought.com Mon Sep 13 16:33:11 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Mon, 13 Sep 2010 15:33:11 -0500 Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? Message-ID: <321A3330-319E-43AA-A675-C993E1502E99@enthought.com> Are we ready to do this yet? I know there were some outstanding questions. Are there major concerns remaining? Or, is it just picking a time to freeze subversion when someone like David or Pauli who know how to move the repository to github correctly a time to do it. Thanks, -Travis From charlesr.harris at gmail.com Mon Sep 13 16:46:48 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 13 Sep 2010 14:46:48 -0600 Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? In-Reply-To: <321A3330-319E-43AA-A675-C993E1502E99@enthought.com> References: <321A3330-319E-43AA-A675-C993E1502E99@enthought.com> Message-ID: On Mon, Sep 13, 2010 at 2:33 PM, Travis Oliphant wrote: > > Are we ready to do this yet? > > I know there were some outstanding questions. Are there major concerns > remaining? Or, is it just picking a time to freeze subversion when someone > like David or Pauli who know how to move the repository to github correctly > a time to do it. > > I didn't see any problems, but I didn't use the test repository that much. Maybe we should just make the jump and iron out any little wrinkles as they show up. Losing small bits of history doesn't bother me much. I think you guys have a big chore ahead of you to merge the refactored stuff while keeping all the fixes that have gone in since the fork. Good Luck! Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Mon Sep 13 17:41:18 2010 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 13 Sep 2010 21:41:18 +0000 (UTC) Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? References: <321A3330-319E-43AA-A675-C993E1502E99@enthought.com> Message-ID: Mon, 13 Sep 2010 15:33:11 -0500, Travis Oliphant wrote: > Are we ready to do this yet? > > I know there were some outstanding questions. Are there major concerns > remaining? As far as the conversion is concerned, things should be OK. The bugs in svn-all-fast-export have been fixed in the meantime, and we have scripts for comparing checksums for each revision. It's hardly possible to lose anything. > Or, is it just picking a time to freeze subversion when > someone like David or Pauli who know how to move the repository to > github correctly a time to do it. I can upload a "final" repository today/tomorrow. If it seems OK, we can freeze SVN trunk a few days after that. Or we can freeze the trunk sooner than that after the "final" repo is up, and patch up things manually, if something turns up (unlikely). Pauli From pav at iki.fi Mon Sep 13 19:20:22 2010 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 13 Sep 2010 23:20:22 +0000 (UTC) Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? References: <321A3330-319E-43AA-A675-C993E1502E99@enthought.com> Message-ID: Mon, 13 Sep 2010 21:41:18 +0000, Pauli Virtanen wrote: [clip] > I can upload a "final" repository today/tomorrow. If it seems OK, we can > freeze SVN trunk a few days after that. > > Or we can freeze the trunk sooner than that after the "final" repo is > up, and patch up things manually, if something turns up (unlikely). Here we go: http://github.com/numpy/numpy http://github.com/numpy/vendor From charlesr.harris at gmail.com Mon Sep 13 20:08:39 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 13 Sep 2010 18:08:39 -0600 Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? In-Reply-To: References: <321A3330-319E-43AA-A675-C993E1502E99@enthought.com> Message-ID: Hi Pauli, On Mon, Sep 13, 2010 at 5:20 PM, Pauli Virtanen wrote: > Mon, 13 Sep 2010 21:41:18 +0000, Pauli Virtanen wrote: > [clip] > > I can upload a "final" repository today/tomorrow. If it seems OK, we can > > freeze SVN trunk a few days after that. > > > > Or we can freeze the trunk sooner than that after the "final" repo is > > up, and patch up things manually, if something turns up (unlikely). > > Here we go: > > http://github.com/numpy/numpy > > http://github.com/numpy/vendor > > What is the suggested work flow for the new repositories? Is the best way to use a github fork and push and pull from that? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Sep 13 20:15:01 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 13 Sep 2010 18:15:01 -0600 Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? In-Reply-To: References: <321A3330-319E-43AA-A675-C993E1502E99@enthought.com> Message-ID: On Mon, Sep 13, 2010 at 3:41 PM, Pauli Virtanen wrote: > Mon, 13 Sep 2010 15:33:11 -0500, Travis Oliphant wrote: > > Are we ready to do this yet? > > > > I know there were some outstanding questions. Are there major concerns > > remaining? > > As far as the conversion is concerned, things should be OK. > > The bugs in svn-all-fast-export have been fixed in the meantime, and we > have scripts for comparing checksums for each revision. It's hardly > possible to lose anything. > > > Or, is it just picking a time to freeze subversion when > > someone like David or Pauli who know how to move the repository to > > github correctly a time to do it. > > I can upload a "final" repository today/tomorrow. If it seems OK, we can > freeze SVN trunk a few days after that. > > Or we can freeze the trunk sooner than that after the "final" repo is up, > and patch up things manually, if something turns up (unlikely). > > I think we should freeze the svn repo as soon as possible. Pierre is still making commits there and unless there is an easy way to update the git repo from svn those sort of commits might be a small hassle. So, I suggest a short post to the list announcing a svn freeze. How does trac integrate with the new repo? Will the mailing lists still work? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Mon Sep 13 20:24:39 2010 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 14 Sep 2010 00:24:39 +0000 (UTC) Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? References: <321A3330-319E-43AA-A675-C993E1502E99@enthought.com> Message-ID: Mon, 13 Sep 2010 18:08:39 -0600, Charles R Harris wrote: [clip] > What is the suggested work flow for the new repositories? Is the best > way to use a github fork and push and pull from that? Yes, I'd personally work like that. Easier to keep private stuff separate. Pauli From fperez.net at gmail.com Mon Sep 13 20:38:09 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 13 Sep 2010 17:38:09 -0700 Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? In-Reply-To: References: <321A3330-319E-43AA-A675-C993E1502E99@enthought.com> Message-ID: Howdy, On Mon, Sep 13, 2010 at 5:24 PM, Pauli Virtanen wrote: > >> What is the suggested work flow for the new repositories? Is the best >> way to use a github fork and push and pull from that? > > Yes, I'd personally work like that. Easier to keep private stuff separate. > in case you find it useful, Matthew Brett wrote up a summary of the 'typical' git workflow in a way that should be easy to include in a project's doc tree and rebuild it with URLs specific to that project, for easy copy/paste. It's his gitwash git repo: http://github.com/matthew-brett/gitwash here's an example of gitwash 'rendered' for the IPython urls: http://ipython.scipy.org/doc/nightly/html/development/gitwash/index.html It's a good start of a fairly common workflow pattern, and if various projects find it useful and submit their fixes upstream, this pattern will become more familiar and easier to follow reliably by end users/developers. Cheers, f From pav at iki.fi Mon Sep 13 20:43:02 2010 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 14 Sep 2010 00:43:02 +0000 (UTC) Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? References: <321A3330-319E-43AA-A675-C993E1502E99@enthought.com> Message-ID: Mon, 13 Sep 2010 18:15:01 -0600, Charles R Harris wrote: [clip] > I think we should freeze the svn repo as soon as possible. Pierre is > still making commits there and unless there is an easy way to update the > git repo from svn those sort of commits might be a small hassle. It needs re-generation of the whole repo which is a bit slow, but it's reproducible and ends up with the same hashes, so in principle we can update later on. > So, I suggest a short post to the list announcing a svn freeze. > How does trac integrate with the new repo? It doesn't at the moment. There are some plugins, though: http://github.com/davglass/github-trac/ http://trac-hacks.org/wiki/GitPlugin/ However, I'm not sure if we really need anything else than a way to link to commits in bug reports (which can be done by pasting URLs anyway). I'd prefer to keep maintenance overhead low, and just use Github's source browser and timeline utilities. > Will the mailing lists still work? Yes, Github can AFAIK be configured to send emails to a given address. However, I guess the Numpy list must be configured to accept mails from Github. Pauli From bsouthey at gmail.com Mon Sep 13 21:10:57 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 13 Sep 2010 20:10:57 -0500 Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? In-Reply-To: References: <321A3330-319E-43AA-A675-C993E1502E99@enthought.com> Message-ID: On Mon, Sep 13, 2010 at 7:15 PM, Charles R Harris wrote: > > > On Mon, Sep 13, 2010 at 3:41 PM, Pauli Virtanen wrote: >> >> Mon, 13 Sep 2010 15:33:11 -0500, Travis Oliphant wrote: >> > Are we ready to do this yet? >> > >> > I know there were some outstanding questions. Are there major concerns >> > remaining? >> >> As far as the conversion is concerned, things should be OK. >> >> The bugs in svn-all-fast-export have been fixed in the meantime, and we >> have scripts for comparing checksums for each revision. It's hardly >> possible to lose anything. >> >> > Or, is it just picking a time to freeze subversion when >> > someone like David or Pauli who know how to move the repository to >> > github correctly a time to do it. >> >> I can upload a "final" repository today/tomorrow. If it seems OK, we can >> freeze SVN trunk a few days after that. >> >> Or we can freeze the trunk sooner than that after the "final" repo is up, >> and patch up things manually, if something turns up (unlikely). >> > > I think we should freeze the svn repo as soon as possible. Pierre is still > making commits there and unless there is an easy way to update the git repo > from svn those sort of commits might be a small hassle. > > So, I suggest a short post to the list announcing a svn freeze. How does > trac integrate with the new repo? Will the mailing lists still work? > > Chuck > > Can you also update the web site (front page and Developer Zone http://www.scipy.org/Developer_Zone ) with the new information? At least highlight that the git is now being used as presumably the svn version will no longer be maintained. It would also be nice for this information to included as a main point in the forthcoming release. Bruce From ben.root at ou.edu Mon Sep 13 21:34:39 2010 From: ben.root at ou.edu (Benjamin Root) Date: Mon, 13 Sep 2010 20:34:39 -0500 Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? In-Reply-To: References: <321A3330-319E-43AA-A675-C993E1502E99@enthought.com> Message-ID: On Mon, Sep 13, 2010 at 8:10 PM, Bruce Southey wrote: > On Mon, Sep 13, 2010 at 7:15 PM, Charles R Harris > wrote: > > > > > > On Mon, Sep 13, 2010 at 3:41 PM, Pauli Virtanen wrote: > >> > >> Mon, 13 Sep 2010 15:33:11 -0500, Travis Oliphant wrote: > >> > Are we ready to do this yet? > >> > > >> > I know there were some outstanding questions. Are there major concerns > >> > remaining? > >> > >> As far as the conversion is concerned, things should be OK. > >> > >> The bugs in svn-all-fast-export have been fixed in the meantime, and we > >> have scripts for comparing checksums for each revision. It's hardly > >> possible to lose anything. > >> > >> > Or, is it just picking a time to freeze subversion when > >> > someone like David or Pauli who know how to move the repository to > >> > github correctly a time to do it. > >> > >> I can upload a "final" repository today/tomorrow. If it seems OK, we can > >> freeze SVN trunk a few days after that. > >> > >> Or we can freeze the trunk sooner than that after the "final" repo is > up, > >> and patch up things manually, if something turns up (unlikely). > >> > > > > I think we should freeze the svn repo as soon as possible. Pierre is > still > > making commits there and unless there is an easy way to update the git > repo > > from svn those sort of commits might be a small hassle. > > > > So, I suggest a short post to the list announcing a svn freeze. How does > > trac integrate with the new repo? Will the mailing lists still work? > > > > Chuck > > > > > > Can you also update the web site (front page and Developer Zone > http://www.scipy.org/Developer_Zone ) with the new information? At > least highlight that the git is now being used as presumably the svn > version will no longer be maintained. > > It would also be nice for this information to included as a main point > in the forthcoming release. > > Bruce > Might also want to fix the README.txt file... The most current development version is always available from our subversion repository: http://svn.scipy.org/svn/numpy/trunk Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.l.goldsmith at gmail.com Mon Sep 13 22:24:55 2010 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Mon, 13 Sep 2010 19:24:55 -0700 Subject: [Numpy-discussion] [OT: MATLAB] Any way to globally make Matlab struct attributes Python-property-like Message-ID: I.e., I'd, at minimum, like to globally replace get(Handel, 'Property') with object.Property and set(Handel, 'Property', value) with object.Property = value to an arbitrary level of composition. (It's really getting cumbersome having to compound gets and sets all over the place while debugging.) (Sorry for the OT post; I thought I'd get a more sympathetic response here than on the MATLAB lists.) ;) Thanks! DG -- In science it often happens that scientists say, 'You know that's a really good argument; my position is mistaken,' and then they would actually change their minds and you never hear that old view from them again. They really do it. It doesn't happen as often as it should, because scientists are human and change is sometimes painful. But it happens every day. I cannot recall the last time something like that happened in politics or religion. - Carl Sagan, 1987 CSICOP address From robert.kern at gmail.com Mon Sep 13 23:13:49 2010 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 13 Sep 2010 22:13:49 -0500 Subject: [Numpy-discussion] [OT: MATLAB] Any way to globally make Matlab struct attributes Python-property-like In-Reply-To: References: Message-ID: On Mon, Sep 13, 2010 at 21:24, David Goldsmith wrote: > (Sorry for the OT post; I thought I'd get a more sympathetic response > here than on the MATLAB lists.) ;) I'm sorry, but that's *really* off-topic. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From josh.holbrook at gmail.com Mon Sep 13 23:18:29 2010 From: josh.holbrook at gmail.com (Joshua Holbrook) Date: Mon, 13 Sep 2010 19:18:29 -0800 Subject: [Numpy-discussion] [OT: MATLAB] Any way to globally make Matlab struct attributes Python-property-like In-Reply-To: References: Message-ID: you have my sympathy, but no hints. :( The only thing I can think of is something like a preprocessor. --Josh On Mon, Sep 13, 2010 at 7:13 PM, Robert Kern wrote: > On Mon, Sep 13, 2010 at 21:24, David Goldsmith wrote: >> (Sorry for the OT post; I thought I'd get a more sympathetic response >> here than on the MATLAB lists.) ;) > > I'm sorry, but that's *really* off-topic. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ? -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From josef.pktd at gmail.com Mon Sep 13 23:33:35 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 13 Sep 2010 23:33:35 -0400 Subject: [Numpy-discussion] [OT: MATLAB] Any way to globally make Matlab struct attributes Python-property-like In-Reply-To: References: Message-ID: On Mon, Sep 13, 2010 at 11:18 PM, Joshua Holbrook wrote: > you have my sympathy, but no hints. :( The only thing I can think of > is something like a preprocessor. > > --Josh > > On Mon, Sep 13, 2010 at 7:13 PM, Robert Kern wrote: >> On Mon, Sep 13, 2010 at 21:24, David Goldsmith wrote: >>> (Sorry for the OT post; I thought I'd get a more sympathetic response >>> here than on the MATLAB lists.) ;) >> >> I'm sorry, but that's *really* off-topic. I think it's a question for the python list about what to do after import re It shouldn't take more than a few hours to come up with the right regular expression and sub arguments. Josef >> >> -- >> Robert Kern >> >> "I have come to believe that the whole world is an enigma, a harmless >> enigma that is made terrible by our own mad attempt to interpret it as >> though it had an underlying truth." >> ? -- Umberto Eco >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From josh.holbrook at gmail.com Mon Sep 13 23:53:56 2010 From: josh.holbrook at gmail.com (Joshua Holbrook) Date: Mon, 13 Sep 2010 19:53:56 -0800 Subject: [Numpy-discussion] [OT: MATLAB] Any way to globally make Matlab struct attributes Python-property-like In-Reply-To: References: Message-ID: On Mon, Sep 13, 2010 at 7:33 PM, wrote: > On Mon, Sep 13, 2010 at 11:18 PM, Joshua Holbrook > wrote: >> you have my sympathy, but no hints. :( The only thing I can think of >> is something like a preprocessor. >> >> --Josh >> >> On Mon, Sep 13, 2010 at 7:13 PM, Robert Kern wrote: >>> On Mon, Sep 13, 2010 at 21:24, David Goldsmith wrote: >>>> (Sorry for the OT post; I thought I'd get a more sympathetic response >>>> here than on the MATLAB lists.) ;) >>> >>> I'm sorry, but that's *really* off-topic. > > I think it's a question for the python list about what to do after > import re > > It shouldn't take more than a few hours to come up with the right > regular expression and sub arguments. > > Josef > I'd love to see any solution(s) to fixing matlab's awkward OO syntax open-sourced. :D --Josh >>> >>> -- >>> Robert Kern >>> >>> "I have come to believe that the whole world is an enigma, a harmless >>> enigma that is made terrible by our own mad attempt to interpret it as >>> though it had an underlying truth." >>> ? -- Umberto Eco >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From robertwb at math.washington.edu Tue Sep 14 01:19:44 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 13 Sep 2010 22:19:44 -0700 Subject: [Numpy-discussion] Cython distutils Message-ID: I've pushed an implementation of http://wiki.cython.org/enhancements/distutils_preprocessing . This allows one do write, e.g. module_list = cythonize("*.pyx") in your setup.py, and it handle the .pyx dependencies (including transitive dependence of libraries) and does all the .pyx -> .c translation. (In particular, this does away with the need for a customized build_ext extension, which is especially nice for projects their own. There are also mechanisms for specifying more build information in the source file itself--see http://hg.cython.org/cython-devel/file/tip/tests/build/inline_distutils.srctree for an example. This will also be useful for non-distutils setups (e.g. pyximport and inline code). Try it out and let me know what you think in terms of features and API. - Robert From josh.holbrook at gmail.com Tue Sep 14 01:27:07 2010 From: josh.holbrook at gmail.com (Joshua Holbrook) Date: Mon, 13 Sep 2010 21:27:07 -0800 Subject: [Numpy-discussion] Hey datarray guys, did you see tablib in The Changelog? Message-ID: Seeing this made me think it might be of interest to some of us who want to work with tabular data: http://thechangelog.com/post/1112527716/tablib-pythonic-tabular-data-library It's definitely working in 2 dimensions instead of datarray's n dimensions, but it looks nice! Plus, import/export. Nice. Thoughts? Reactions? Discuss. --Josh From ndbecker2 at gmail.com Tue Sep 14 06:43:11 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 14 Sep 2010 06:43:11 -0400 Subject: [Numpy-discussion] test for long double support Message-ID: What is the recommended way to test for long double and clongdouble? Is it: #if NPY_SIZEOF_LONGDOUBLE == NPY_SIZEOF_DOUBLE From lorenz at MPA-Garching.MPG.DE Tue Sep 14 08:34:29 2010 From: lorenz at MPA-Garching.MPG.DE (Lorenz =?utf-8?q?H=C3=BCdepohl?=) Date: Tue, 14 Sep 2010 14:34:29 +0200 Subject: [Numpy-discussion] Difference in astype() for scalars and arrays? In-Reply-To: References: <201009131351.51541.lorenz@mpa-garching.mpg.de> Message-ID: <201009141434.29562.lorenz@mpa-garching.mpg.de> On Monday 13 September 2010 Pauli Virtanen wrote: > Doesn't seem correct -- please file a bug ticket. I did that: http://projects.scipy.org/numpy/ticket/1611 Regards, Lorenz From dalcinl at gmail.com Tue Sep 14 09:57:48 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 14 Sep 2010 10:57:48 -0300 Subject: [Numpy-discussion] Cython distutils In-Reply-To: References: Message-ID: On 14 September 2010 02:19, Robert Bradshaw wrote: > I've pushed an implementation of > http://wiki.cython.org/enhancements/distutils_preprocessing . This > allows one do write, e.g. > > ? ?module_list = cythonize("*.pyx") > What happens if you run 'python setup.py --help', for example? Will that trigger cythonization? > in your setup.py, and it handle the .pyx dependencies (including > transitive dependence of libraries) and does all the .pyx -> .c > translation. (In particular, this does away with the need for a > customized build_ext extension, which is especially nice for projects > their own. ?There are also mechanisms for specifying more build > information in the source file itself--see > http://hg.cython.org/cython-devel/file/tip/tests/build/inline_distutils.srctree > for an example. This will also be useful for non-distutils setups > (e.g. pyximport and inline code). > > Try it out and let me know what you think in terms of features and API. > > - Robert > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From ndbecker2 at gmail.com Tue Sep 14 11:47:24 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 14 Sep 2010 11:47:24 -0400 Subject: [Numpy-discussion] test for long double support References: Message-ID: Neal Becker wrote: > What is the recommended way to test for long double and clongdouble? > > Is it: > #if NPY_SIZEOF_LONGDOUBLE == NPY_SIZEOF_DOUBLE I guess I found it? #if HAVE_LONG_DOUBLE // defined in pyconfig.h From keekychen.shared at gmail.com Tue Sep 14 12:25:15 2010 From: keekychen.shared at gmail.com (kee chen) Date: Wed, 15 Sep 2010 00:25:15 +0800 Subject: [Numpy-discussion] how to store variable length string in array and get single char by it's position? Message-ID: Dear All, Suppose I have a list group some kind like DNA sequence: 1 ATGCATGCAATTGGCC 2 ATGCATGCAATTGGCCATCD 3 CATGCAATTGGCCCCCCCCC ...... 100000 CATGCAAATTGGCCCCCCCCC the string length of each item is not sure and may get change/update later, then how can I store above in a numpy array (include the ID) and easy to get the single value? for example 1. ATGCATGCAATTGGCC I want get the first T then I use something like array[1][1], means A[T]G...... and if I want to update the 3rd postion I can use array[1][2] = T to set the AT[G]C... to AT[T]C...? Thanks for any hints. Rgs, KC -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Tue Sep 14 12:33:20 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 14 Sep 2010 09:33:20 -0700 Subject: [Numpy-discussion] how to store variable length string in array and get single char by it's position? In-Reply-To: References: Message-ID: On Tue, Sep 14, 2010 at 9:25 AM, kee chen wrote: > Dear All, > > Suppose I have a list group some kind like DNA sequence: > > 1? ATGCATGCAATTGGCC > 2? ATGCATGCAATTGGCCATCD > 3? CATGCAATTGGCCCCCCCCC > ...... > 100000 CATGCAAATTGGCCCCCCCCC > > the string length of each item is not sure and may get change/update later, > then how can I store above in a numpy array (include the ID) and easy to get > the single value? > > for example > 1.? ATGCATGCAATTGGCC > I want get the first T then I use something like array[1][1], means > A[T]G......?? and if I want to update the 3rd postion I can use array[1][2] > = T to set the AT[G]C... to AT[T]C...? How about using a python list: >> array = ['ATGC', 'CATGA', 'A'] >> array[0][1] 'T' From bsouthey at gmail.com Tue Sep 14 14:02:10 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Tue, 14 Sep 2010 13:02:10 -0500 Subject: [Numpy-discussion] how to store variable length string in array and get single char by it's position? In-Reply-To: References: Message-ID: <4C8FB8A2.5040005@gmail.com> On 09/14/2010 11:33 AM, Keith Goodman wrote: > On Tue, Sep 14, 2010 at 9:25 AM, kee chen wrote: >> Dear All, >> >> Suppose I have a list group some kind like DNA sequence: >> >> 1 ATGCATGCAATTGGCC >> 2 ATGCATGCAATTGGCCATCD >> 3 CATGCAATTGGCCCCCCCCC >> ...... >> 100000 CATGCAAATTGGCCCCCCCCC >> >> the string length of each item is not sure and may get change/update later, >> then how can I store above in a numpy array (include the ID) and easy to get >> the single value? >> >> for example >> 1. ATGCATGCAATTGGCC >> I want get the first T then I use something like array[1][1], means >> A[T]G...... and if I want to update the 3rd postion I can use array[1][2] >> = T to set the AT[G]C... to AT[T]C...? > How about using a python list: > >>> array = ['ATGC', 'CATGA', 'A'] >>> array[0][1] > 'T' > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion Variable length is incompatible with numpy: 'NumPy arrays have a fixed size at creation, unlike Python lists (which can grow dynamically).' http://docs.scipy.org/doc/numpy/user/whatisnumpy.html So you have to allocate the space for the largest sequence - although you can try using Scipy's sparse matrix (where nucleotides/amino acids are coded in numbers starting at 1 such that zeros code the empty areas. If lists (or dictionaries) don't work for you, you might want to explore bioinformatics packages like 'pygr' (http://code.google.com/p/pygr/) and Biopython (http://biopython.org/wiki/Main_Page) or try more general approaches such hdf5 and pytables. Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Tue Sep 14 19:01:22 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Tue, 14 Sep 2010 18:01:22 -0500 Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? In-Reply-To: References: <321A3330-319E-43AA-A675-C993E1502E99@enthought.com> Message-ID: On Sep 13, 2010, at 3:46 PM, Charles R Harris wrote: > > > On Mon, Sep 13, 2010 at 2:33 PM, Travis Oliphant wrote: > > Are we ready to do this yet? > > I know there were some outstanding questions. Are there major concerns remaining? Or, is it just picking a time to freeze subversion when someone like David or Pauli who know how to move the repository to github correctly a time to do it. > > > I didn't see any problems, but I didn't use the test repository that much. Maybe we should just make the jump and iron out any little wrinkles as they show up. Losing small bits of history doesn't bother me much. > > I think you guys have a big chore ahead of you to merge the refactored stuff while keeping all the fixes that have gone in since the fork. Good Luck! I know. That's why I want to do it with git as much as possible... ;-) -Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Tue Sep 14 19:05:23 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Tue, 14 Sep 2010 18:05:23 -0500 Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? In-Reply-To: References: <321A3330-319E-43AA-A675-C993E1502E99@enthought.com> Message-ID: On Sep 13, 2010, at 6:20 PM, Pauli Virtanen wrote: > Mon, 13 Sep 2010 21:41:18 +0000, Pauli Virtanen wrote: > [clip] >> I can upload a "final" repository today/tomorrow. If it seems OK, we can >> freeze SVN trunk a few days after that. >> >> Or we can freeze the trunk sooner than that after the "final" repo is >> up, and patch up things manually, if something turns up (unlikely). > > Here we go: > > http://github.com/numpy/numpy > > http://github.com/numpy/vendor Thanks! This is really great work. So much thanks is due to Pauli and David for making this happen. I will get Aaron River to freeze the SVN trunk as well as work on trac integration. There may be some learning for using the git workflow, and but it sounds like there are some good tutorials for doing it. I've heard several people say that once they used git, they can't imagine going back to SVN. -Travis From dalcinl at gmail.com Tue Sep 14 19:34:37 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 14 Sep 2010 20:34:37 -0300 Subject: [Numpy-discussion] test for long double support In-Reply-To: References: Message-ID: On 14 September 2010 12:47, Neal Becker wrote: > Neal Becker wrote: > >> What is the recommended way to test for long double and clongdouble? >> >> Is it: >> #if NPY_SIZEOF_LONGDOUBLE == NPY_SIZEOF_DOUBLE > > I guess I found it? > > #if HAVE_LONG_DOUBLE ? ? ? ? ? ?// defined in pyconfig.h > I think these two are different things. For example, MSVC do support 'long double', it is a different type than 'double', but sizeof(long double) == sizeof(double). Also look at CPython's objimpl.h, union _gc_head, you will see an unprotected usage of 'long double', so it seems that CPython requires that the C compiler to support 'long double'. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From fperez.net at gmail.com Tue Sep 14 19:51:39 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 14 Sep 2010 16:51:39 -0700 Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? In-Reply-To: References: <321A3330-319E-43AA-A675-C993E1502E99@enthought.com> Message-ID: On Tue, Sep 14, 2010 at 4:05 PM, Travis Oliphant wrote: >> Here we go: >> >> ? ? ? http://github.com/numpy/numpy >> >> ? ? ? http://github.com/numpy/vendor > > > Thanks! ? This is really great work. ? So much thanks is due to Pauli and David for making this happen. ? ? I will get Aaron River to freeze the SVN trunk as well as work on trac integration. Ditto thanks! > There may be some learning for using the git workflow, and but it sounds like there are some good tutorials for doing it. I keep a list of resources here in case someone finds them useful: http://fperez.org/py4science/git.html > I've heard several people say that once they used git, they can't imagine going back to SVN. As you were writing this, Min RK and I were discussing on IRC: there are so many people who provide patches on github, since the process is so easy I couldn't think of going back from git to svn now. :) Cheers, f From kwatford+scipy at gmail.com Tue Sep 14 20:55:03 2010 From: kwatford+scipy at gmail.com (Ken Watford) Date: Tue, 14 Sep 2010 20:55:03 -0400 Subject: [Numpy-discussion] [OT: MATLAB] Any way to globally make Matlab struct attributes Python-property-like In-Reply-To: References: Message-ID: On Mon, Sep 13, 2010 at 10:24 PM, David Goldsmith wrote: > I.e., I'd, at minimum, like to globally replace > > get(Handel, 'Property') > > with > > object.Property > > and > > set(Handel, 'Property', value) > > with > > object.Property = value > > to an arbitrary level of composition. Both structures and objects of both handle and value types work exactly like you're asking for. You're probably talking about "handle graphics" objects (which are just opaque numeric values representing things related to the UI), which do still require the get/set functions. They're just doubles, after all. It should be feasible to write a class that wraps them for ease of use, but that wouldn't be a topic for here. If that intrigues you, try asking about it on StackOverflow. From millman at berkeley.edu Tue Sep 14 22:09:08 2010 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 14 Sep 2010 19:09:08 -0700 Subject: [Numpy-discussion] sf.net export controls for numpy/scipy Message-ID: Hello, I plan to update the export controls settings for both numpy and scipy to: "This project does NOT incorporate, access, call upon, or otherwise use encryption of any kind, including, but not limited to, open source algorithms and/or calls to encryption in the operating system or underlying platform." Unless I hear from someone that I overlooked something, I will make the changes tomorrow. Best, Jarrod From charlesr.harris at gmail.com Tue Sep 14 23:35:26 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 14 Sep 2010 21:35:26 -0600 Subject: [Numpy-discussion] Buildbot errors and failures Message-ID: Just thought I'd throw these out there, they've been showing for a while. Windows XP: =========== ERROR: test_fromfile_tofile_seeks (test_regression.TestRegression) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\buildbot\numpy\b11\numpy-install25\Lib\site-packages\numpy\core\tests\test_regression.py", line 1384, in test_fromfile_tofile_seeks ret = np.fromfile(f, count=4, dtype='u1') IOError: first argument must be an open file BSD : ===== gmake test /usr/local/bin/python2.6 -c 'import numpy,sys;sys.exit(not numpy.test("full", verbose=10).wasSuccessful())' Traceback (most recent call last): File "", line 1, in File "/home/doconnor/numpy-buildbot/b12/numpy-install/lib/python2.6/site-packages/numpy/__init__.py", line 136, in import add_newdocs File "/home/doconnor/numpy-buildbot/b12/numpy-install/lib/python2.6/site-packages/numpy/add_newdocs.py", line 9, in from numpy.lib import add_newdoc File "/home/doconnor/numpy-buildbot/b12/numpy-install/lib/python2.6/site-packages/numpy/lib/__init__.py", line 13, in from polynomial import * File "/home/doconnor/numpy-buildbot/b12/numpy-install/lib/python2.6/site-packages/numpy/lib/polynomial.py", line 17, in from numpy.linalg import eigvals, lstsq File "/home/doconnor/numpy-buildbot/b12/numpy-install/lib/python2.6/site-packages/numpy/linalg/__init__.py", line 48, in from linalg import * File "/home/doconnor/numpy-buildbot/b12/numpy-install/lib/python2.6/site-packages/numpy/linalg/linalg.py", line 23, in from numpy.linalg import lapack_lite ImportError: /usr/local/lib/liblapack.so.4: Undefined symbol "_gfortran_st_write" Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Wed Sep 15 03:21:20 2010 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 15 Sep 2010 09:21:20 +0200 Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? In-Reply-To: References: <321A3330-319E-43AA-A675-C993E1502E99@enthought.com> Message-ID: <20100915072120.GD13315@phare.normalesup.org> On Tue, Sep 14, 2010 at 04:51:39PM -0700, Fernando Perez wrote: > > I've heard several people say that once they used git, they can't imagine going back to SVN. > As you were writing this, Min RK and I were discussing on IRC: > there are so many people who provide patches on github, since > the process is so easy > I couldn't think of going back from git to svn now. While I find that git has its limitations (I have never made more user mistakes with data-loss consequences using git than with another VCS), github is just really awesome and way ahead of everything else, I believe. In terms of workflow in an open source project with many contributors, core and non-core, its a great gain. Gael From david at silveregg.co.jp Wed Sep 15 04:10:53 2010 From: david at silveregg.co.jp (David) Date: Wed, 15 Sep 2010 17:10:53 +0900 Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? In-Reply-To: <20100915072120.GD13315@phare.normalesup.org> References: <321A3330-319E-43AA-A675-C993E1502E99@enthought.com> <20100915072120.GD13315@phare.normalesup.org> Message-ID: <4C907F8D.40402@silveregg.co.jp> On 09/15/2010 04:21 PM, Gael Varoquaux wrote: > On Tue, Sep 14, 2010 at 04:51:39PM -0700, Fernando Perez wrote: >>> I've heard several people say that once they used git, they can't imagine going back to SVN. > >> As you were writing this, Min RK and I were discussing on IRC: > >> there are so many people who provide patches on github, since >> the process is so easy >> I couldn't think of going back from git to svn now. > > While I find that git has its limitations (I have never made more user > mistakes with data-loss consequences using git than with another VCS) It is very difficult to actually lose data with git thanks to the reflog: http://www.gitready.com/intermediate/2009/02/09/reflog-your-safety-net.html cheers, David From pav at iki.fi Wed Sep 15 04:38:17 2010 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 15 Sep 2010 10:38:17 +0200 Subject: [Numpy-discussion] test for long double support In-Reply-To: References: Message-ID: <1284539897.8587.2.camel@Nokia-N900-42-11> > Also look at CPython's objimpl.h, union _gc_head, you will see an > unprotected usage of 'long double', so it seems that CPython requires > that the C compiler to support 'long double'. Long double is IIRC in C89, so compiler support is probably not a problem nowadays. -- Pauli Virtanen From gael.varoquaux at normalesup.org Wed Sep 15 04:58:57 2010 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 15 Sep 2010 10:58:57 +0200 Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? In-Reply-To: <4C907F8D.40402@silveregg.co.jp> References: <321A3330-319E-43AA-A675-C993E1502E99@enthought.com> <20100915072120.GD13315@phare.normalesup.org> <4C907F8D.40402@silveregg.co.jp> Message-ID: <20100915085857.GE13315@phare.normalesup.org> On Wed, Sep 15, 2010 at 05:10:53PM +0900, David wrote: > It is very difficult to actually lose data with git thanks to the > reflog: > http://www.gitready.com/intermediate/2009/02/09/reflog-your-safety-net.html Unless you use the 'force' switches. I am trying very hard not to use them, as I have been advised by several good git users. But I keep getting in situations where people tell me that I need to use one of these switches. It's always a bit hard to explain how I get there, but I'll try and do so, so that knowledgeable people can advice me on the right solutions. Here is an example (in chronological order): 1) Branch out a feature branch 'feature' from master. 2) Develop in feature branch (cycle of code/commit...) 3) Hot bug on master, checkout master, attempt to fix bug. Bug fix induces other bugs, cycle of code/commit to fix them. 4) Decide that bug fix is not mature enough to push, but feature branch got reviewed and is. 5) Discover that I can't push from feature to origin/master. Conclude that I must merge back in master locally. Now I have a problem: at step 1 I should have created a branch. I did not. I need to go back and create a branch. This was happening at a sprint, and people that know git better than me helped me out. But the only way we found to sort this out was to create a branch at step 1, merge the branch with master, and 'reset -a' master at step 1. I thought it over quite a few times, and did not loose any data. However, I was very uncomfortable with the process (the 'reset -a'). What was the right solution (apart from 'create branch at step 1')? This really illustrates what git feels like me: linux in root mode, powerful, unsafe if you don't understand it well. Ga?l From markbak at gmail.com Wed Sep 15 05:01:21 2010 From: markbak at gmail.com (Mark Bakker) Date: Wed, 15 Sep 2010 11:01:21 +0200 Subject: [Numpy-discussion] array view with different shape Message-ID: Hello List, Can I make a view of an entire array but with a different shape? For example: a = zeros((2,3,4)) Now I want array b to be identical to a, but with shape (2,12). b = a; b.shape = (2,12) This doesn't work, of course, as also the shape of a changes. I know I can simply make a copy of a and change the shape, but can I make b to be a view of a (so that when I change a, then b changes as well)? Thanks, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From tmp50 at ukr.net Wed Sep 15 05:04:57 2010 From: tmp50 at ukr.net (Dmitrey) Date: Wed, 15 Sep 2010 12:04:57 +0300 Subject: [Numpy-discussion] ANN: OpenOpt 0.31, FuncDesigner 0.21, DerApproximator 0.21 Message-ID: Hi all, I'm glad to inform you about new releases: OpenOpt 0.31, FuncDesigner 0.21, DerApproximator 0.21 For details see http://forum.openopt.org/viewtopic.php?id=299 or visit our homepage http://openopt.org Regards, Dmitrey -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Wed Sep 15 05:06:13 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 15 Sep 2010 11:06:13 +0200 Subject: [Numpy-discussion] array view with different shape In-Reply-To: References: Message-ID: On Sep 15, 2010, at 11:01 AM, Mark Bakker wrote: > Hello List, > > Can I make a view of an entire array but with a different shape? > > For example: > > a = zeros((2,3,4)) Mark, Try that. b = a.reshape((2,12)) Now, >>> b.flat=1 >>> a array([[[ 1., 1., 1., 1.], [ 1., 1., 1., 1.], [ 1., 1., 1., 1.]], [[ 1., 1., 1., 1.], [ 1., 1., 1., 1.], [ 1., 1., 1., 1.]]]) From david at silveregg.co.jp Wed Sep 15 05:55:46 2010 From: david at silveregg.co.jp (David) Date: Wed, 15 Sep 2010 18:55:46 +0900 Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? In-Reply-To: <20100915085857.GE13315@phare.normalesup.org> References: <321A3330-319E-43AA-A675-C993E1502E99@enthought.com> <20100915072120.GD13315@phare.normalesup.org> <4C907F8D.40402@silveregg.co.jp> <20100915085857.GE13315@phare.normalesup.org> Message-ID: <4C909822.4010805@silveregg.co.jp> On 09/15/2010 05:58 PM, Gael Varoquaux wrote: > On Wed, Sep 15, 2010 at 05:10:53PM +0900, David wrote: >> It is very difficult to actually lose data with git thanks to the >> reflog: >> http://www.gitready.com/intermediate/2009/02/09/reflog-your-safety-net.html > > Unless you use the 'force' switches. I am trying very hard not to > use them, as I have been advised by several good git users. Well, yes and no. For example, you can not loose commits with any command that I know of thanks to the reflog (and this is true even if you use really "hard" commands like git reset --hard somecommit). > > But I keep getting in situations where people tell me that I need to use > one of these switches. It's always a bit hard to explain how I get there, > but I'll try and do so, so that knowledgeable people can advice me on the > right solutions. > > Here is an example (in chronological order): > > 1) Branch out a feature branch 'feature' from master. > > 2) Develop in feature branch (cycle of code/commit...) > > 3) Hot bug on master, checkout master, attempt to fix bug. Bug fix > induces other bugs, cycle of code/commit to fix them. > > 4) Decide that bug fix is not mature enough to push, but feature branch > got reviewed and is. > > 5) Discover that I can't push from feature to origin/master. Conclude > that I must merge back in master locally. > > Now I have a problem: at step 1 I should have created a branch. I did > not. I need to go back and create a branch. This was happening at a > sprint, and people that know git better than me helped me out. But the > only way we found to sort this out was to create a branch at step 1, > merge the branch with master, and 'reset -a' master at step 1. I thought > it over quite a few times, and did not loose any data. However, I was > very uncomfortable with the process (the 'reset -a'). I am not sure I understand your issue exactly. Do you mean you put some commits in the wrong branch ? I don't see how reset is related to that - I mean, I have used git for two years, and I don't even know what reset -a does, much less used it :) cheers, David From pav at iki.fi Wed Sep 15 06:08:21 2010 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 15 Sep 2010 10:08:21 +0000 (UTC) Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? References: <321A3330-319E-43AA-A675-C993E1502E99@enthought.com> <20100915072120.GD13315@phare.normalesup.org> <4C907F8D.40402@silveregg.co.jp> <20100915085857.GE13315@phare.normalesup.org> Message-ID: Wed, 15 Sep 2010 10:58:57 +0200, Gael Varoquaux wrote: [clip] > Now I have a problem: at step 1 I should have created a branch. I did > not. I need to go back and create a branch. This was happening at a > sprint, and people that know git better than me helped me out. But the > only way we found to sort this out was to create a branch at step 1, > merge the branch with master, and 'reset -a' master at step 1. I thought > it over quite a few times, and did not loose any data. However, I was > very uncomfortable with the process (the 'reset -a'). I would maybe have moved the stuff in master to a new branch via: $?git branch work master # move stuff in master to new branch $ git branch -f master origin/master # roll master back to origin/master Ok, here we used a --force switch. > What was the right solution (apart from 'create branch at step 1')? Pushing to origin/master from feature should be possible: $ git checkout feature $ git fetch origin $ git merge origin/master # if needed $ git push origin feature:master > This really illustrates what git feels like me: linux in root mode, > powerful, unsafe if you don't understand it well. Avoiding the --force switches and reset helps a bit. Also, personally I always tend to do history rewriting in a new branch, so I can easily re-do it, if necessary. $?git branch tmp original-branch ... do whatever history altering necessary ... ... check for mess-up ... $?git branch -f original-branch tmp # overwrite old branch $ git branch -d tmp Pauli From gael.varoquaux at normalesup.org Wed Sep 15 07:48:19 2010 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 15 Sep 2010 13:48:19 +0200 Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? In-Reply-To: <4C909822.4010805@silveregg.co.jp> References: <321A3330-319E-43AA-A675-C993E1502E99@enthought.com> <20100915072120.GD13315@phare.normalesup.org> <4C907F8D.40402@silveregg.co.jp> <20100915085857.GE13315@phare.normalesup.org> <4C909822.4010805@silveregg.co.jp> Message-ID: <20100915114819.GA29792@phare.normalesup.org> On Wed, Sep 15, 2010 at 06:55:46PM +0900, David wrote: > > Unless you use the 'force' switches. I am trying very hard not to > > use them, as I have been advised by several good git users. > Well, yes and no. For example, you can not loose commits with any > command that I know of thanks to the reflog (and this is true even if > you use really "hard" commands like git reset --hard somecommit). Cool, very handy. I wasn't aware of this. Is there a GUI for reflog (in any one of the many GUIs, I don't really care)? I don't use reflog much because I don't understand it, and don't really know what operations I can do on it (still learning). Ga?l From gael.varoquaux at normalesup.org Wed Sep 15 07:48:55 2010 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 15 Sep 2010 13:48:55 +0200 Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? In-Reply-To: References: <321A3330-319E-43AA-A675-C993E1502E99@enthought.com> <20100915072120.GD13315@phare.normalesup.org> <4C907F8D.40402@silveregg.co.jp> <20100915085857.GE13315@phare.normalesup.org> Message-ID: <20100915114855.GB29792@phare.normalesup.org> On Wed, Sep 15, 2010 at 10:08:21AM +0000, Pauli Virtanen wrote: > Pushing to origin/master from feature should be possible: > $ git checkout feature > $ git fetch origin > $ git merge origin/master # if needed > $ git push origin feature:master That's what I was missing! Thanks, Ga?l From cournape at gmail.com Wed Sep 15 08:54:24 2010 From: cournape at gmail.com (David Cournapeau) Date: Wed, 15 Sep 2010 21:54:24 +0900 Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? In-Reply-To: <20100915114819.GA29792@phare.normalesup.org> References: <321A3330-319E-43AA-A675-C993E1502E99@enthought.com> <20100915072120.GD13315@phare.normalesup.org> <4C907F8D.40402@silveregg.co.jp> <20100915085857.GE13315@phare.normalesup.org> <4C909822.4010805@silveregg.co.jp> <20100915114819.GA29792@phare.normalesup.org> Message-ID: On Wed, Sep 15, 2010 at 8:48 PM, Gael Varoquaux wrote: > On Wed, Sep 15, 2010 at 06:55:46PM +0900, David wrote: >> > Unless you use the 'force' switches. I am trying very hard not to >> > use them, as I have been advised by several good git users. > >> Well, yes and no. For example, you can not loose commits with any >> command that I know of thanks to the reflog (and this is true even if >> you use really "hard" commands like git reset --hard somecommit). > > Cool, very handy. I wasn't aware of this. Is there a GUI for reflog (in > any one of the many GUIs, I don't really care)? I don't use reflog much > because I don't understand it, and don't really know what operations I > can do on it (still learning). The basic way to use reflog is trivial: it just keeps a (time ordered) list of your commits and actions, independently of the branch. So for example, say you are in master, and by accident, you do: git reset --hard origin/master # origin/master is behind master git reflog will look like: b59aa8c HEAD@{0}: origin/master: updating HEAD ad5c0c0 HEAD@{1}: pull github master: Fast forward d11b913 HEAD@{2}: commit: ENH: add builder override support for compiled libraries. 2423a03 HEAD@{3}: commit: FEAT: add get_compiled_libraries to get all clib from all subpackages. ae4c57b HEAD@{4}: merge tmp: Fast forward f4c4b53 HEAD@{5}: github/master: updating HEAD ae4c57b HEAD@{6}: checkout: moving from tmp to master ae4c57b HEAD@{7}: checkout: moving from ae4c57bab5583759aab831eafce0c9b83a5f9c45 to tmp ... So you know exactly how to go back any point in time - that is until the reflog is "purged" (which is something like 30 days by default I think). The key point is that any operation you do in git w.r.t. commits is moving in the graph, so resetting, checking out, etc... do not lose anything. In particular, reset --hard only get you back to some node of that graph, without removing the old one. That's also useful when you need to revert some merge, etc... Although, as mentioned by pv, it does not preclude doing potentially "harming" operations in separate branches (if only because it is simpler). cheers, David From gael.varoquaux at normalesup.org Wed Sep 15 09:09:30 2010 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 15 Sep 2010 15:09:30 +0200 Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? In-Reply-To: References: <20100915072120.GD13315@phare.normalesup.org> <4C907F8D.40402@silveregg.co.jp> <20100915085857.GE13315@phare.normalesup.org> <4C909822.4010805@silveregg.co.jp> <20100915114819.GA29792@phare.normalesup.org> Message-ID: <20100915130930.GA31348@phare.normalesup.org> On Wed, Sep 15, 2010 at 09:54:24PM +0900, David Cournapeau wrote: > The basic way to use reflog is trivial: it just keeps a (time ordered) > list of your commits and actions, independently of the branch. So for > example, say you are in master, and by accident, you do: > git reset --hard origin/master # origin/master is behind master > git reflog > will look like: > b59aa8c HEAD@{0}: origin/master: updating HEAD > ad5c0c0 HEAD@{1}: pull github master: Fast forward > d11b913 HEAD@{2}: commit: ENH: add builder override support for > compiled libraries. > 2423a03 HEAD@{3}: commit: FEAT: add get_compiled_libraries to get all > clib from all subpackages. > ae4c57b HEAD@{4}: merge tmp: Fast forward > f4c4b53 HEAD@{5}: github/master: updating HEAD > ae4c57b HEAD@{6}: checkout: moving from tmp to master > ae4c57b HEAD@{7}: checkout: moving from > ae4c57bab5583759aab831eafce0c9b83a5f9c45 to tmp > ... I don't understand which line corresponds to the 'reset --hard' in the above reflog (sorry, I am slow). Is it the 'updating HEAD' one? Reading the above, it seemed to me that I could always revert my previous action (unless it impacted non-tracked files) using: git checkout HEAD@{0} I just tried it on a copy of a repo I had, and I get "You are in a 'detached HEAD'". From what I understand from the message (and Google) I simply need to do a "git checkout -b new_branch" to be in a state where I can work. However, it seem to me that when I switch back the the master branch, and merge my new_branch, I haven't "undone" the action I was trying to undo. Sorry for asking these questions, but I'd rather be dense and ask them while I am playing with a toy repo than suffer from my git inability later on a real situation, which usually happens at 2AM. Thanks for you git-foo Ga?l From ralf.gommers at googlemail.com Wed Sep 15 09:28:37 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 15 Sep 2010 21:28:37 +0800 Subject: [Numpy-discussion] Silent DeprecationWarnings under Python 2.7 onwards In-Reply-To: References: Message-ID: On Fri, Sep 10, 2010 at 12:05 AM, Peter < numpy-discussion at maubp.freeserve.co.uk> wrote: > Hi all, > > NumPy currently makes extensive use of the DeprecationWarning > class to alert users when some feature is going to be withdrawn. > However, as of Python 2.7, the DeprecationWarning is silent by > default, see: > > > http://docs.python.org/library/warnings.html#updating-code-for-new-versions-of-python > > This makes sense to me for deprecation warnings from Python > itself or the standard library - after all, Python 2.7 is the last of the > Python 2.x series. > The reason for the change is explained in the paragraph you link to, 2.7 being the final minor release in the 2.x series isn't it. There are many other packages/programs built on numpy, the user/developer distinction can be made in the same way as for Python itself. I fail to see a reason not to follow the lead of the Python core developers here. Cheers, Ralf > However, I believe that a silent DeprecationWarning is bad news > for third party libraries like NumPy where many end users are > coders and we want them to see these warnings by default. > Is anyone else concerned about this? A typical NumPy user > (on Python 2.7+ or Python 3.2+) may never see the warnings > (because they have to deliberately turn them on), the first they'll > know about it is when they upgrade to a new release and their > code suddenly stops working. > > Potentially NumPy would have to introduce its own NumPy > specific DeprecationWarning warning class, and use that. > > Peter > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Wed Sep 15 10:10:08 2010 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 15 Sep 2010 09:10:08 -0500 Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? In-Reply-To: <20100915130930.GA31348@phare.normalesup.org> References: <20100915072120.GD13315@phare.normalesup.org> <4C907F8D.40402@silveregg.co.jp> <20100915085857.GE13315@phare.normalesup.org> <4C909822.4010805@silveregg.co.jp> <20100915114819.GA29792@phare.normalesup.org> <20100915130930.GA31348@phare.normalesup.org> Message-ID: On Wed, Sep 15, 2010 at 8:09 AM, Gael Varoquaux < gael.varoquaux at normalesup.org> wrote: > On Wed, Sep 15, 2010 at 09:54:24PM +0900, David Cournapeau wrote: > > The basic way to use reflog is trivial: it just keeps a (time ordered) > > list of your commits and actions, independently of the branch. So for > > example, say you are in master, and by accident, you do: > > > git reset --hard origin/master # origin/master is behind master > > git reflog > > > will look like: > > > b59aa8c HEAD@{0}: origin/master: updating HEAD > > ad5c0c0 HEAD@{1}: pull github master: Fast forward > > d11b913 HEAD@{2}: commit: ENH: add builder override support for > > compiled libraries. > > 2423a03 HEAD@{3}: commit: FEAT: add get_compiled_libraries to get all > > clib from all subpackages. > > ae4c57b HEAD@{4}: merge tmp: Fast forward > > f4c4b53 HEAD@{5}: github/master: updating HEAD > > ae4c57b HEAD@{6}: checkout: moving from tmp to master > > ae4c57b HEAD@{7}: checkout: moving from > > ae4c57bab5583759aab831eafce0c9b83a5f9c45 to tmp > > ... > > I don't understand which line corresponds to the 'reset --hard' in the > above reflog (sorry, I am slow). Is it the 'updating HEAD' one? > > Reading the above, it seemed to me that I could always revert my previous > action (unless it impacted non-tracked files) using: > > git checkout HEAD@{0} > > I just tried it on a copy of a repo I had, and I get "You are in a > 'detached HEAD'". From what I understand from the message (and Google) I > simply need to do a "git checkout -b new_branch" to be in a state where I > can work. However, it seem to me that when I switch back the the master > branch, and merge my new_branch, I haven't "undone" the action I was > trying to undo. > > Sorry for asking these questions, but I'd rather be dense and ask them > while I am playing with a toy repo than suffer from my git inability > later on a real situation, which usually happens at 2AM. > > Thanks for you git-foo > > Ga?l > I have to ditto the earlier sentiment about how easy it is to mess up in git (as compared to svn). I read the git parable and (I thought) I understood the elegance of how git works, only to later discover that there was still plenty of details that I just "didn't get". It is probably a good thing that we are cutting our teeth on toy repositories first. Anyway, I would just like to voice two of the most important things I have learned about git. First, 'git revert' != 'svn revert' (seriously, Linus... why?). Second, 'git stash' is your friend. If you haven't come across stash yet, here is a quick use-case. Imagine you are working on adding a particular feature and have made some changes. Along the way, you discover an unrelated bug. You can 'stash' the changes you made so far, reverting back to the state at the last commit, and then fix that bug. You then commit *that* bugfix (to the proper branch, of course), then you can 'pop' the stashed changes and be back to where you were when you did the stash (with the bugfix). Lovely feature... :) Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Wed Sep 15 10:34:24 2010 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 15 Sep 2010 09:34:24 -0500 Subject: [Numpy-discussion] Silent DeprecationWarnings under Python 2.7 onwards In-Reply-To: References: Message-ID: On Wed, Sep 15, 2010 at 8:28 AM, Ralf Gommers wrote: > > > On Fri, Sep 10, 2010 at 12:05 AM, Peter < > numpy-discussion at maubp.freeserve.co.uk> wrote: > >> Hi all, >> >> NumPy currently makes extensive use of the DeprecationWarning >> class to alert users when some feature is going to be withdrawn. >> However, as of Python 2.7, the DeprecationWarning is silent by >> default, see: >> >> >> http://docs.python.org/library/warnings.html#updating-code-for-new-versions-of-python >> >> This makes sense to me for deprecation warnings from Python >> itself or the standard library - after all, Python 2.7 is the last of the >> Python 2.x series. >> > > The reason for the change is explained in the paragraph you link to, 2.7 > being the final minor release in the 2.x series isn't it. > > There are many other packages/programs built on numpy, the user/developer > distinction can be made in the same way as for Python itself. I fail to see > a reason not to follow the lead of the Python core developers here. > > Cheers, > Ralf > > > I have to agree a bit with Peter. I do understand the rationale of Python's position, however, I have to wonder what is the point of DeprecationWarning if it doesn't get displayed? The warning is supposed to give a heads-up to the developer to modify their code. Now, the argument can be made that a python developer should know to run python with those warnings unmuted. And I would agree for "true", career programmers. However, numpy/scipy/matplotlib have become environments unto themselves, catered to converts from Matlab, R, S+ and other such languages. I would argue that many of the "developers" are not typical programmers with proper development habits/skills. Many (myself included) are graduate students in scientific fields unrelated to computer science. I have to wonder how many of them would even be aware of the differences between python versions (or even which version they are using!). Anyway, my point is that the deprecation warnings are very valuable to display and that we need to keep in mind the audience that SciPy has. Maybe we don't necessarily turn them on by default (somehow), but maybe the documentation should highly recommend that they get turned on, thereby raising awareness on the part of the user. There is a side-benefit to mentioning the muted warnings issue in the documentation. If a developer later complains that a feature was removed without any deprecation notices, we could simply point to the documentation and say that we recommended turning the warnings on. My 2 cents, Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Wed Sep 15 11:07:52 2010 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 15 Sep 2010 15:07:52 +0000 (UTC) Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? References: <20100915072120.GD13315@phare.normalesup.org> <4C907F8D.40402@silveregg.co.jp> <20100915085857.GE13315@phare.normalesup.org> <4C909822.4010805@silveregg.co.jp> <20100915114819.GA29792@phare.normalesup.org> <20100915130930.GA31348@phare.normalesup.org> Message-ID: Wed, 15 Sep 2010 15:09:30 +0200, Gael Varoquaux wrote: [clip] > Reading the above, it seemed to me that I could always revert my > previous action (unless it impacted non-tracked files) using: > > git checkout HEAD@{0} > > I just tried it on a copy of a repo I had, and I get "You are in a > 'detached HEAD'". From what I understand from the message (and Google) I > simply need to do a "git checkout -b new_branch" to be in a state where > I can work. However, it seem to me that when I switch back the the > master branch, and merge my new_branch, I haven't "undone" the action I > was trying to undo. Suppose you just messed up `somebranch`. First, check its reflog: $ git reflog show somebranch and pick a state that seems good, usually somebranch@{1} if you didn't do new commits etc. on the branch yet: $ git checkout somebranch@{1} ... check now that this is the state you'd like somebranch to have... $ git branch -f somebranch somebranch@{1} which moves `somebranch` to its previous state. Note that $ git reflog does *NOT* show the reflog of the current branch -- it shows the reflog of HEAD, which can in some cases be confusing. The point is that HEAD is a meta-branch that corresponds to the current checkout, and so changes every time you use "git checkout". -- Pauli Virtanen From bsouthey at gmail.com Wed Sep 15 11:10:29 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Wed, 15 Sep 2010 10:10:29 -0500 Subject: [Numpy-discussion] Python2.4 and date-time - tickets 1475 and 1613 Message-ID: <4C90E1E5.9020908@gmail.com> Hi, Datetime support in Numpy requires ctypes - RuntimeError is raised because ctypes has only been included in Python starting with Python2.5. This Python2.4 dependency must be included in the documentation so I created ticket 1613 (http://projects.scipy.org/numpy/ticket/1613). A standalone version of ctypes is available for Python2.3 and Python2.4: http://starship.python.net/crew/theller/ctypes/ The numpy tests pass after installing ctypes under Python2.4. I think ticket 1475 is fixed by skipping that test in a similar way to the skipped tests in 'numpy/tests/test_ctypeslib.py'. Bruce PS A big thanks for fixing the other Python2.4 errors I mentioned with regards to scipy! From gael.varoquaux at normalesup.org Wed Sep 15 11:12:20 2010 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 15 Sep 2010 17:12:20 +0200 Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? In-Reply-To: References: <20100915072120.GD13315@phare.normalesup.org> <4C907F8D.40402@silveregg.co.jp> <20100915085857.GE13315@phare.normalesup.org> <4C909822.4010805@silveregg.co.jp> <20100915114819.GA29792@phare.normalesup.org> <20100915130930.GA31348@phare.normalesup.org> Message-ID: <20100915151220.GC31348@phare.normalesup.org> On Wed, Sep 15, 2010 at 03:07:52PM +0000, Pauli Virtanen wrote: > and pick a state that seems good, usually somebranch@{1} if you didn't do > new commits etc. on the branch yet: > $ git checkout somebranch@{1} > ... check now that this is the state you'd like somebranch to have... > $ git branch -f somebranch somebranch@{1} OK, the above was what I was missing. Thanks a lot, it's very useful. > Note that > $ git reflog > does *NOT* show the reflog of the current branch -- it shows the reflog > of HEAD, which can in some cases be confusing. The point is that HEAD is > a meta-branch that corresponds to the current checkout, and so changes > every time you use "git checkout". Thank you, I hadn't realized! Ga?l From charlesr.harris at gmail.com Wed Sep 15 12:10:48 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 15 Sep 2010 10:10:48 -0600 Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? In-Reply-To: <20100915151220.GC31348@phare.normalesup.org> References: <20100915072120.GD13315@phare.normalesup.org> <4C907F8D.40402@silveregg.co.jp> <20100915085857.GE13315@phare.normalesup.org> <4C909822.4010805@silveregg.co.jp> <20100915114819.GA29792@phare.normalesup.org> <20100915130930.GA31348@phare.normalesup.org> <20100915151220.GC31348@phare.normalesup.org> Message-ID: Now that we have the repository, let's revisit the .gitignore file that I removed from svn. IIRC, Stefan complained and Pauli more or less agreed with the removal. However, after browsing about the net, the suggested policy is that files that should be excluded in all the clones of the repository should go in .gitignore and local exclusions should go in .git/info/exclude. So it seems we should commit a .gitignore file just to help avoid accidental additions. As a start, I suggest the following contents: *.o *.a *.pyc *.swp *~ build Thoughts? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Sep 15 12:16:35 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 15 Sep 2010 11:16:35 -0500 Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? In-Reply-To: References: <20100915072120.GD13315@phare.normalesup.org> <4C907F8D.40402@silveregg.co.jp> <20100915085857.GE13315@phare.normalesup.org> <4C909822.4010805@silveregg.co.jp> <20100915114819.GA29792@phare.normalesup.org> <20100915130930.GA31348@phare.normalesup.org> <20100915151220.GC31348@phare.normalesup.org> Message-ID: On Wed, Sep 15, 2010 at 11:10, Charles R Harris wrote: > > > Now that we have the repository, let's revisit the .gitignore file that I > removed from svn. IIRC, Stefan complained and Pauli more or less agreed with > the removal. However, after browsing about the net, the suggested policy is > that files that should be excluded in all the clones of the repository > should go in .gitignore and local exclusions should go in .git/info/exclude. > So it seems we should commit a .gitignore file just to help avoid accidental > additions. As a start, I suggest the following contents: > > *.o > *.a > *.pyc > *.swp > *~ > build *.pyo *.so *.pyd .gdb_history dist -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From pav at iki.fi Wed Sep 15 12:27:20 2010 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 15 Sep 2010 16:27:20 +0000 (UTC) Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? References: <20100915072120.GD13315@phare.normalesup.org> <4C907F8D.40402@silveregg.co.jp> <20100915085857.GE13315@phare.normalesup.org> <4C909822.4010805@silveregg.co.jp> <20100915114819.GA29792@phare.normalesup.org> <20100915130930.GA31348@phare.normalesup.org> <20100915151220.GC31348@phare.normalesup.org> Message-ID: Wed, 15 Sep 2010 11:16:35 -0500, Robert Kern wrote: [clip: .gitignore] >> *.o >> *.a >> *.pyc >> *.swp >> *~ >> build > > *.pyo > *.so > *.pyd > .gdb_history > dist /build /dist /doc/build /doc/cdoc/build .\#* *.log *.egg-info /numpy/core/__svn_version__.py /doc/numpy.scipy.org/_build The .\#* is for emacs. But let's do this directly in Git, so that I don't have to bother regenerating the Numpy repo. -- Pauli Virtanen From charlesr.harris at gmail.com Wed Sep 15 12:50:44 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 15 Sep 2010 10:50:44 -0600 Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? In-Reply-To: References: <20100915072120.GD13315@phare.normalesup.org> <4C907F8D.40402@silveregg.co.jp> <20100915085857.GE13315@phare.normalesup.org> <4C909822.4010805@silveregg.co.jp> <20100915114819.GA29792@phare.normalesup.org> <20100915130930.GA31348@phare.normalesup.org> <20100915151220.GC31348@phare.normalesup.org> Message-ID: On Wed, Sep 15, 2010 at 10:27 AM, Pauli Virtanen wrote: > Wed, 15 Sep 2010 11:16:35 -0500, Robert Kern wrote: > [clip: .gitignore] > >> *.o > >> *.a > >> *.pyc > >> *.swp > >> *~ > >> build > > > > *.pyo > > *.so > > *.pyd > > .gdb_history > > dist > > /build > /dist > /doc/build > /doc/cdoc/build > .\#* > *.log > *.egg-info > /numpy/core/__svn_version__.py > /doc/numpy.scipy.org/_build > > The .\#* is for emacs. But let's do this directly in Git, so that I don't > have to bother regenerating the Numpy repo. > > OK, there is a version at http://github.com/charris/numpy/tree/misc Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From markbak at gmail.com Wed Sep 15 14:43:41 2010 From: markbak at gmail.com (Mark Bakker) Date: Wed, 15 Sep 2010 20:43:41 +0200 Subject: [Numpy-discussion] ANN: OpenOpt 0.31, FuncDesigner 0.21, DerApproximator 0.21 Message-ID: Dmitrey, Great to hear OpenOpt development is moving forward. I tried to easy_install on both a Mac and on Windows, but both installations crashed. With similar (maybe identical) errors on "atexit" commands. Any solution? Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Wed Sep 15 15:34:42 2010 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 15 Sep 2010 14:34:42 -0500 Subject: [Numpy-discussion] common significant diigits Message-ID: Hello, I am trying to solve a problem in matplotlib where I would have an array of floating point numbers and I want to quickly determine what is the closest common offset to a power of 10. In other words, if given: [12373.43, 12375.89, 12370.18], I would want returned something like either 12370.0, or the lowest common order of magnitude (in this case, 10). Is there some sort of neat math/numpy trick to figure this out? I already have a brute-force method with a while loop, but I am looking for something a little bit more elegant. Thanks, Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokhansever at gmail.com Wed Sep 15 15:41:41 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Wed, 15 Sep 2010 14:41:41 -0500 Subject: [Numpy-discussion] common significant diigits In-Reply-To: References: Message-ID: On Wed, Sep 15, 2010 at 2:34 PM, Benjamin Root wrote: > Hello, > > I am trying to solve a problem in matplotlib where I would have an array of > floating point numbers and I want to quickly determine what is the closest > common offset to a power of 10. In other words, if given: > > [12373.43, 12375.89, 12370.18], > > I would want returned something like either 12370.0, or the lowest common > order of magnitude (in this case, 10). > > Is there some sort of neat math/numpy trick to figure this out? I already > have a brute-force method with a while loop, but I am looking for something > a little bit more elegant. > > Thanks, > Ben Root > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > Probably not being very fail-safe here is my first guess: a = np.array([12373.43, 12375.89, 12370.18]) np.floor(a).min() -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From markbak at gmail.com Wed Sep 15 15:46:13 2010 From: markbak at gmail.com (Mark Bakker) Date: Wed, 15 Sep 2010 21:46:13 +0200 Subject: [Numpy-discussion] common significant diigits Message-ID: 10*np.floor(a/10).min() -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Sep 15 15:54:47 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 15 Sep 2010 13:54:47 -0600 Subject: [Numpy-discussion] common significant diigits In-Reply-To: References: Message-ID: On Wed, Sep 15, 2010 at 1:34 PM, Benjamin Root wrote: > Hello, > > I am trying to solve a problem in matplotlib where I would have an array of > floating point numbers and I want to quickly determine what is the closest > common offset to a power of 10. In other words, if given: > > [12373.43, 12375.89, 12370.18], > > I would want returned something like either 12370.0, or the lowest common > order of magnitude (in this case, 10). > > Is there some sort of neat math/numpy trick to figure this out? I already > have a brute-force method with a while loop, but I am looking for something > a little bit more elegant. > > Something along the lines of In [14]: ceil(log10(a.max() - a.min())) Out[14]: 1.0 ? I think this approach can be fixed up for whatever it is. I wasn't clear on that. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Wed Sep 15 16:26:06 2010 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 15 Sep 2010 13:26:06 -0700 Subject: [Numpy-discussion] array view with different shape In-Reply-To: References: Message-ID: <4C912BDE.2020507@noaa.gov> Mark Bakker wrote: > Can I make a view of an entire array but with a different shape? yup. > For example: > > a = zeros((2,3,4)) > > Now I want array b to be identical to a, but with shape (2,12). > > b = a; b.shape = (2,12) > > This doesn't work, of course, as also the shape of a changes. as ab and b are bound to the same single object. someone already gave you reshape(), but for the more general case, .view() is handy: In [11]: a = np.zeros((2,3,4)) In [12]: b = a.view() In [13]: b.shape = (2, 12) In [14]: a.shape Out[14]: (2, 3, 4) # so they are different objects. In [15]: a[0,0,0] = 1 In [16]: b Out[16]: array([[ 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], [ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]) # that do share data. note that .view() can do all sorts of nifty things like change the data type or make an np.matrix, or... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From ben.root at ou.edu Wed Sep 15 17:28:07 2010 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 15 Sep 2010 16:28:07 -0500 Subject: [Numpy-discussion] common significant diigits In-Reply-To: References: Message-ID: On Wed, Sep 15, 2010 at 2:54 PM, Charles R Harris wrote: > > > On Wed, Sep 15, 2010 at 1:34 PM, Benjamin Root wrote: > >> Hello, >> >> I am trying to solve a problem in matplotlib where I would have an array >> of floating point numbers and I want to quickly determine what is the >> closest common offset to a power of 10. In other words, if given: >> >> [12373.43, 12375.89, 12370.18], >> >> I would want returned something like either 12370.0, or the lowest common >> order of magnitude (in this case, 10). >> >> Is there some sort of neat math/numpy trick to figure this out? I already >> have a brute-force method with a while loop, but I am looking for something >> a little bit more elegant. >> >> > Something along the lines of > > In [14]: ceil(log10(a.max() - a.min())) > Out[14]: 1.0 > > ? I think this approach can be fixed up for whatever it is. I wasn't clear > on that. > > Chuck > Chuck, That does seem to work for most cases (and it even did better than my brute force approach when it comes to negative numbers). But there are still some edge cases. In particular a case like [0.99, 1.01] will be interpreted as having a common significant digit. I suspect another edge case would be if there are no common sig digs, but I could check for that by seeing if the order of magnitude of the range is less than the order of magnitude of the maximum value and if the order of magnitude of the max equals the order of magnitude of the min. I will look into it further, but if anyone is interested, I have attached a test script. Thanks, Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: commonsigs.py Type: application/octet-stream Size: 1242 bytes Desc: not available URL: From mfenner at gmail.com Wed Sep 15 17:38:49 2010 From: mfenner at gmail.com (Mark Fenner) Date: Wed, 15 Sep 2010 17:38:49 -0400 Subject: [Numpy-discussion] Two questions on indexing In-Reply-To: References: Message-ID: One method of using indices seems to be as follows: In [19]: a = N.array(range(6)).reshape(3,2) In [20]: i = N.indices([3,2]) In [21]: r,c = i In [22]: a[r,c] Out[22]: array([[0, 1], ? ? ? [2, 3], ? ? ? [4, 5]]) In [23]: a[tuple(i)] Out[23]: array([[0, 1], ? ? ? [2, 3], ? ? ? [4, 5]]) *****Question 1:***** For using the results of argmax, how would one "pad" the indices returned by argmax to be usable as follows (note, I resorted to method I found on numpy-discussion ... google "take_axis" ... that revolves around rolling the desired axis to the first position and then using the indices ... I'm curious how to make something like what I have below work). ?Incidentally, the number of dimensions can be arbitrary (for my needs): x = N.array([[2, 3], ? ? ? ? ? ? ? ? [1, 5]]) for a in [0, 1]: ? ? indices = N.argmax(x, axis=a) ? ? # perhaps something similar to the following? ? ? # grid = N.indices(x.shape) ? ? # grid[a] = indices # <--- this is the idea, but it fails to get the necessary result ? ? # usable = tuple(grid) ? ? assert x[usableIndices] == x.max(axis=a) # or similar *****Question 2:***** A separate question. Suppose I have a slice for indexing that looks like: [:, :, 2, :, 5] How can I get an indexing slice for all OTHER dimension values besides those specified. Conceptually, something like: [:, :, all but 2, :, all but 5] Incidentally, the goal is to construct a new array with all those "other" spots filled in with zero and the specified spots with their original values. ?Would it be easier to construct a 0-1 indicator array with 1s in the [:,:,2,:,5] positions and multiply it out? ?Humm, I may have just answered my own question. For argument sake, how would you do it with indexing/slicing? I suppose one develops some intuition as one gains experience with numpy with regards to when to (1) use clever matrix ops and when to (2) use clever slicing and when to (3) use a combination of both. Thanks, Mark From brett.olsen at gmail.com Wed Sep 15 17:49:57 2010 From: brett.olsen at gmail.com (Brett Olsen) Date: Wed, 15 Sep 2010 16:49:57 -0500 Subject: [Numpy-discussion] Two questions on indexing In-Reply-To: References: Message-ID: On Wed, Sep 15, 2010 at 4:38 PM, Mark Fenner wrote: > A separate question. ?Suppose I have a slice for indexing that looks like: > > [:, :, 2, :, 5] > > How can I get an indexing slice for all OTHER dimension values besides > those specified. ?Conceptually, something like: > > [:, :, all but 2, :, all but 5] > > Incidentally, the goal is to construct a new array with all those > "other" spots filled in with zero and the specified spots with their > original values. ?Would it be easier to construct a 0-1 indicator > array with 1s in the [:,:,2,:,5] positions and multiply it out? ?Humm, > I may have just answered my own question. > > For argument sake, how would you do it with indexing/slicing? ?I > suppose one develops some intuition as one gains experience with numpy > with regards to when to (1) use clever matrix ops and when to (2) use > clever slicing and when to (3) use a combination of both. This works, although I'm not sure how efficient it is compared to other methods: In [19]: a = N.arange(16).reshape(4,4) In [20]: a Out[20]: array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11], [12, 13, 14, 15]]) In [24]: a[:,N.arange(4) != 2] Out[24]: array([[ 0, 1, 3], [ 4, 5, 7], [ 8, 9, 11], [12, 13, 15]]) Brett From alan at ajackson.org Wed Sep 15 21:14:05 2010 From: alan at ajackson.org (alan at ajackson.org) Date: Wed, 15 Sep 2010 20:14:05 -0500 Subject: [Numpy-discussion] weighted mean; weighted standard error of the mean (sem) In-Reply-To: References: Message-ID: <20100915201405.449e868f@ajackson.org> >I am looking for some reaally basic statistical tools. I have some >sample data, some sample weights for those measurements, and I want to >calculate a mean and a standard error of the mean. > I have been working through an R training class, converting the examples and exercises to numpy and scipy. Some of what you want may be there. http://www.oplnk.net/~ajackson/software/ -- ----------------------------------------------------------------------- | Alan K. Jackson | To see a World in a Grain of Sand | | alan at ajackson.org | And a Heaven in a Wild Flower, | | www.ajackson.org | Hold Infinity in the palm of your hand | | Houston, Texas | And Eternity in an hour. - Blake | ----------------------------------------------------------------------- From stefan at sun.ac.za Thu Sep 16 01:50:57 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 16 Sep 2010 07:50:57 +0200 Subject: [Numpy-discussion] Can we freeze the subversion repository and move to github this week? In-Reply-To: References: <321A3330-319E-43AA-A675-C993E1502E99@enthought.com> Message-ID: On Wed, Sep 15, 2010 at 1:05 AM, Travis Oliphant wrote: > Thanks! ? This is really great work. ? So much thanks is due to Pauli and David for making this happen. ? ? I will get Aaron River to freeze the SVN trunk as well as work on trac integration. With the last re-install of Trac we integrated a Git browser (optimistic anticipation!): http://projects.scipy.org/numpy/browse_git But I think we can simply replace this with a link to GitHub now? Regards St?fan From friedrichromstedt at gmail.com Thu Sep 16 04:31:44 2010 From: friedrichromstedt at gmail.com (Friedrich Romstedt) Date: Thu, 16 Sep 2010 10:31:44 +0200 Subject: [Numpy-discussion] Two questions on indexing In-Reply-To: References: Message-ID: 2010/9/15 Mark Fenner : > One method of using indices seems to be as follows: > > In [19]: a = N.array(range(6)).reshape(3,2) > In [20]: i = N.indices([3,2]) > In [21]: r,c = i > In [22]: a[r,c] > Out[22]: > array([[0, 1], > ? ? ? [2, 3], > ? ? ? [4, 5]]) > > In [23]: a[tuple(i)] > Out[23]: > array([[0, 1], > ? ? ? [2, 3], > ? ? ? [4, 5]]) > > *****Question 1:***** > > For using the results of argmax, how would one "pad" the indices > returned by argmax to be usable as follows (note, I resorted to method > I found on numpy-discussion ... google "take_axis" ... that revolves > around rolling the desired axis to the first position and then using > the indices ... I'm curious how to make something like what I have > below work). ?Incidentally, the number of ?dimensions can be arbitrary > (for my needs): > > x = N.array([[2, 3], > ? ? ? ? ? ? ? ? [1, 5]]) > > for a in [0, 1]: > ? ? indices = N.argmax(x, axis=a) > > ? ? # perhaps something similar to the following? > ? ? # grid = N.indices(x.shape) > ? ? # grid[a] = indices # <--- this is the idea, but it fails to get > the necessary result > ? ? # usable = tuple(grid) > > ? ? assert x[usableIndices] == x.max(axis=a) # or similar I first will try to understand what you want. I go to the simplest case, 1-dimensional. There we have only one axis. x is sth like [1, 42, 3, 4]. numpy.argmax(x, axis=0) delivers 1 in this case (scalar). This is because there are no dimensions left besides the zeroth. So grid would be [[0, 1, 2, 3]]. Assigning 1 to this grid in the zeroth component of this grid would work in this case, and would set all the indexing elements to 1. Thus, when indexing with this grid, you would spread out the 42 on the whole axis (axis=0 in this case). Is that what you want? I for now assume that I got your point. Would be a good idea to post the desired output too. When going to higher dimensions, say x.shape = (10, 11, 12), then grid.shape = (3, 10, 11, 12). Meaning all elements of grid have again shape (10, 11, 12). Your argmax indices array, say in axis=0, has shape (11, 12). Indexed with some tuple (j, k) it gives the coordinate of the corresponing maximum element as (indices[j, k], j, k). Further grid[i, j, k] gives the index where to take from in the zeroth axis for element (i, j, k) of the result array. You want to spread on the zeroth axis in that case. Meaning you want to say grid[0][i, j, k] = indices[j, k] for all i. For the zeroth axis, this will work via broadcasting directly: grid[0] = indices. For the say 1st axis, axis=1, indices.shape = (10, 12). You want to say in this case grid[1][i, j, k] = indices[i, k]. For this to get it working, you would probably have to reshape with the a shape containing the 1 in the appropriate place, in this case grid[0] = indices.reshape((10, 1, 12)). This shouldn't be super-hard. Just take the original shape, and substitute the axis'th value by a one, and put this into .reshape(). With this reshaped indices, broadcasting will work as expected. I guess you want to do some stuff with the "corrected" indices, otherwise x.max(axis=a) would do the job? x = numpy.random.random((10, 11, 12)) for axis in xrange(0, 3): maximum_indices = x.argmax(axis=axis) full_shape = list(x.shape) full_shape[axis] = 1 grid = numpy.indices(x.shape) grid[axis] = maximum_indices.reshape(full_shape) There is some room for optimisation because we create the same grid all the time. I did not do this now. > *****Question 2:***** > > A separate question. ?Suppose I have a slice for indexing that looks like: > > [:, :, 2, :, 5] > > How can I get an indexing slice for all OTHER dimension values besides > those specified. ?Conceptually, something like: > > [:, :, all but 2, :, all but 5] > > Incidentally, the goal is to construct a new array with all those > "other" spots filled in with zero and the specified spots with their > original values. ?Would it be easier to construct a 0-1 indicator > array with 1s in the [:,:,2,:,5] positions and multiply it out? ?Humm, > I may have just answered my own question. > > For argument sake, how would you do it with indexing/slicing? ?I > suppose one develops some intuition as one gains experience with numpy > with regards to when to (1) use clever matrix ops and when to (2) use > clever slicing and when to (3) use a combination of both. I really think the multiplication approach is a good one, and you can speed it up using broadcastring. Assuming x.shape = (10, 11, 12, 13, 14), for your example, you would say then: factor = numpy.zeros(1, 1, 12, 1, 14, dtype=numpy.bool) factor[:, :, 2] = True factor[:, :, :, :, 5] = True x_masked = x * factor Notice that as I understand you, you want to have 1s in those cells, which have either 2 or 5 or both in their indices. This means, that a cell is zero iff. it has indices not containing the 2 and not containing the 5. On the contrary, when you want to keep only those which have 2 and 5 in their indices, you may use two separate factors as above. Friedrich From pav at iki.fi Thu Sep 16 04:58:46 2010 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 16 Sep 2010 08:58:46 +0000 (UTC) Subject: [Numpy-discussion] Numpy SVN frozen; move to Git Message-ID: Dear all, Numpy SVN repository is now frozen, and does not accept new commits. Future development should end up in the Git repository: http://github.com/numpy/numpy The next things on the TODO list: - Update any links that point to http://svn.scipy.org/svn/numpy or talk about SVN. E.g. numpy.org needs updating. - Put up documentation on how to contribute to Numpy via Git. Gitwash-generated stuff could be added to the Numpy docs. - Decide if the `numpy-svn` email list is still needed. Github has RSS feeds for the repositories (but it can also send email to the list, if we want to keep the list alive). - Core devs: create accounts on github.com and ask for push permissions to the numpy repository. Or, just push your changes to your personal forks, and send pull requests -- I'm sure we have enough people to handle it also this way. - Trac integration -- our bug DB will still stay at projects.scipy.org but other Trac functionality can maybe be integrated. Pauli From stefan at sun.ac.za Thu Sep 16 07:26:51 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 16 Sep 2010 13:26:51 +0200 Subject: [Numpy-discussion] Numpy SVN frozen; move to Git In-Reply-To: References: Message-ID: On Thu, Sep 16, 2010 at 10:58 AM, Pauli Virtanen wrote: > Numpy SVN repository is now frozen, and does not accept new commits. > Future development should end up in the Git repository: > > ? ?http://github.com/numpy/numpy Beautiful! It now takes mere seconds to clone NumPy (compared to several minutes from SVN). Thanks so much to yourself and David Cournapeau for making this happen. Regards St?fan From friedrichromstedt at gmail.com Thu Sep 16 08:02:37 2010 From: friedrichromstedt at gmail.com (Friedrich Romstedt) Date: Thu, 16 Sep 2010 14:02:37 +0200 Subject: [Numpy-discussion] Asking for opinions: Priops Message-ID: I just ran across the problem of priorities with ndarrays again and it keeps biting me. I did once ago a workaround to get my ``undarray`` class's methods be called when being the second operand of e.g. + . But since I wrote it, always Python crashes on exit with the message: Python-32(68665) malloc: *** error for object 0x239680: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug (Of course only if I imported the module. Occasionally I also observed Bus errors, and even segfaults.) I overloaded the numpy ops via numpy.set_numeric_ops() with self-written classes, which are *not* derived from numpy.ufunc, and do not resemble numpy ufuncs completely. So I want to do it properly this time. I therefore started with writing a Letter of Intent, and put it online on http://github.com/friedrichromstedt/priops . Opinions? Friedrich P.S.: I will start coding anyway, but it would be nice. P.P.S.: The package this originates from is also online, under http://github.com/friedrichromstedt/upy, or http://upy.sourceforge.net. I will probably create a small example script demonstrating the crash. From ralf.gommers at googlemail.com Thu Sep 16 09:21:05 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 16 Sep 2010 21:21:05 +0800 Subject: [Numpy-discussion] Silent DeprecationWarnings under Python 2.7 onwards In-Reply-To: References: Message-ID: On Wed, Sep 15, 2010 at 10:34 PM, Benjamin Root wrote: > On Wed, Sep 15, 2010 at 8:28 AM, Ralf Gommers > wrote: > >> >> >> On Fri, Sep 10, 2010 at 12:05 AM, Peter < >> numpy-discussion at maubp.freeserve.co.uk> wrote: >> >>> Hi all, >>> >>> NumPy currently makes extensive use of the DeprecationWarning >>> class to alert users when some feature is going to be withdrawn. >>> However, as of Python 2.7, the DeprecationWarning is silent by >>> default, see: >>> >>> >>> http://docs.python.org/library/warnings.html#updating-code-for-new-versions-of-python >>> >>> This makes sense to me for deprecation warnings from Python >>> itself or the standard library - after all, Python 2.7 is the last of the >>> Python 2.x series. >>> >> >> The reason for the change is explained in the paragraph you link to, 2.7 >> being the final minor release in the 2.x series isn't it. >> >> There are many other packages/programs built on numpy, the user/developer >> distinction can be made in the same way as for Python itself. I fail to see >> a reason not to follow the lead of the Python core developers here. >> >> Cheers, >> Ralf >> >> >> > I have to agree a bit with Peter. I do understand the rationale of > Python's position, however, I have to wonder what is the point of > DeprecationWarning if it doesn't get displayed? The warning is supposed to > give a heads-up to the developer to modify their code. > > Now, the argument can be made that a python developer should know to run > python with those warnings unmuted. And I would agree for "true", career > programmers. However, numpy/scipy/matplotlib have become environments unto > themselves, catered to converts from Matlab, R, S+ and other such > languages. I would argue that many of the "developers" are not typical > programmers with proper development habits/skills. Many (myself included) > are graduate students in scientific fields unrelated to computer science. I > have to wonder how many of them would even be aware of the differences > between python versions (or even which version they are using!). > > Anyway, my point is that the deprecation warnings are very valuable to > display and that we need to keep in mind the audience that SciPy has. Maybe > we don't necessarily turn them on by default (somehow), but maybe the > documentation should highly recommend that they get turned on, thereby > raising awareness on the part of the user. > Sure, documenting this well and recommending to turn on the warnings for example once after installation and once before upgrading makes a lot of sense. Places to document it: http://www.scipy.org/FAQ http://projects.scipy.org/numpy/wiki/TestingGuidelines Any others? Ralf > There is a side-benefit to mentioning the muted warnings issue in the > documentation. If a developer later complains that a feature was removed > without any deprecation notices, we could simply point to the documentation > and say that we recommended turning the warnings on. > > My 2 cents, > Ben Root > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Thu Sep 16 09:30:28 2010 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 16 Sep 2010 13:30:28 +0000 (UTC) Subject: [Numpy-discussion] Numpy SVN frozen; move to Git References: Message-ID: Thu, 16 Sep 2010 08:58:46 +0000, Pauli Virtanen wrote: > The next things on the TODO list: > > - Update any links that point to http://svn.scipy.org/svn/numpy > or talk about SVN. > > E.g. numpy.org needs updating. > > - Put up documentation on how to contribute to Numpy via Git. > Gitwash-generated stuff could be added to the Numpy docs. > > - Decide if the `numpy-svn` email list is still needed. > Github has RSS feeds for the repositories (but it can also send > email to the list, if we want to keep the list alive). > > - Core devs: create accounts on github.com and ask for push > permissions > to the numpy repository. > > Or, just push your changes to your personal forks, and send pull > requests -- I'm sure we have enough people to handle it also this > way. > > - Trac integration -- our bug DB will still stay at projects.scipy.org > but other Trac functionality can maybe be integrated. And a few more: - Buildbot and buildslaves. Maybe easiest by using Github's SVN interface? http://github.com/blog/626-announcing-svn-support - Doc editor etc. -- Pauli Virtanen From friedrichromstedt at gmail.com Thu Sep 16 11:47:07 2010 From: friedrichromstedt at gmail.com (Friedrich Romstedt) Date: Thu, 16 Sep 2010 17:47:07 +0200 Subject: [Numpy-discussion] Re-pinging two old issues Message-ID: I'd really love to see these two things fixed: bugreport1.py When comparing objects using the numpy mechanism for == or != (only those apparently), everything is cast to an ndarray, even if this results in an dtype=numpy.object array. E.g. for addition, this doesn't happen. Installing loggers ... Creating array a: ... [1 2] Calling a + 1 ... args = (array([1, 2]), 1) Calling a == 1 ... args = (array([1, 2]), array(1)) Calling a == X() ... args = (array([1, 2]), array(<__main__.X instance at 0x716300>, dtype=object)) bugreport2.py This one is really crucial to me. I have to overload the operators with callables. But when doing so, at least my Python crashes at exit with different outcome. Sometimes it even does not crash. I'd like to get some feedback from other users whether they can confirm the crashiness or if it is specific to my numpy. I'm using numpy 1.4.1, but I strongly believe this things are widely apart from the regions where fixes took place. My system is Mac OS X 10.6 Intel-based, 64bit, with a 32bit Python, and self-compiled numpy, and 8 GB RAM, ca. 1/3 free, and rebooting is rare for me. Friedrich -------------- next part -------------- A non-text attachment was scrubbed... Name: bugreport1.py Type: application/octet-stream Size: 623 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: bugreport2.py Type: application/octet-stream Size: 2767 bytes Desc: not available URL: From david at silveregg.co.jp Thu Sep 16 21:21:56 2010 From: david at silveregg.co.jp (David) Date: Fri, 17 Sep 2010 10:21:56 +0900 Subject: [Numpy-discussion] Numpy SVN frozen; move to Git In-Reply-To: References: Message-ID: <4C92C2B4.8020707@silveregg.co.jp> On 09/16/2010 10:30 PM, Pauli Virtanen wrote: > Thu, 16 Sep 2010 08:58:46 +0000, Pauli Virtanen wrote: >> The next things on the TODO list: >> >> - Update any links that point to http://svn.scipy.org/svn/numpy >> or talk about SVN. >> >> E.g. numpy.org needs updating. >> >> - Put up documentation on how to contribute to Numpy via Git. >> Gitwash-generated stuff could be added to the Numpy docs. >> >> - Decide if the `numpy-svn` email list is still needed. >> Github has RSS feeds for the repositories (but it can also send >> email to the list, if we want to keep the list alive). >> >> - Core devs: create accounts on github.com and ask for push >> permissions >> to the numpy repository. >> >> Or, just push your changes to your personal forks, and send pull >> requests -- I'm sure we have enough people to handle it also this >> way. >> >> - Trac integration -- our bug DB will still stay at projects.scipy.org >> but other Trac functionality can maybe be integrated. > > And a few more: > > - Buildbot and buildslaves. Maybe easiest by using Github's SVN > interface? > > http://github.com/blog/626-announcing-svn-support That's what I was thinking too - but it seems that the svn support is somewhat flaky. I keep getting svn: REPORT of '/cournape/numpy.git/!svn/vcc/default': 200 OK (http://svn.github.com). May be due to some network configuration, I don't know. Would be good if someone else could check, cheers, David From charlesr.harris at gmail.com Thu Sep 16 21:49:08 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 16 Sep 2010 19:49:08 -0600 Subject: [Numpy-discussion] Numpy SVN frozen; move to Git In-Reply-To: <4C92C2B4.8020707@silveregg.co.jp> References: <4C92C2B4.8020707@silveregg.co.jp> Message-ID: On Thu, Sep 16, 2010 at 7:21 PM, David wrote: > On 09/16/2010 10:30 PM, Pauli Virtanen wrote: > > Thu, 16 Sep 2010 08:58:46 +0000, Pauli Virtanen wrote: > >> The next things on the TODO list: > >> > >> - Update any links that point to http://svn.scipy.org/svn/numpy > >> or talk about SVN. > >> > >> E.g. numpy.org needs updating. > >> > >> - Put up documentation on how to contribute to Numpy via Git. > >> Gitwash-generated stuff could be added to the Numpy docs. > >> > >> - Decide if the `numpy-svn` email list is still needed. > >> Github has RSS feeds for the repositories (but it can also send > >> email to the list, if we want to keep the list alive). > >> > >> - Core devs: create accounts on github.com and ask for push > >> permissions > >> to the numpy repository. > >> > >> Or, just push your changes to your personal forks, and send pull > >> requests -- I'm sure we have enough people to handle it also this > >> way. > >> > >> - Trac integration -- our bug DB will still stay at > projects.scipy.org > >> but other Trac functionality can maybe be integrated. > > > > And a few more: > > > > - Buildbot and buildslaves. Maybe easiest by using Github's SVN > > interface? > > > > http://github.com/blog/626-announcing-svn-support > > That's what I was thinking too - but it seems that the svn support is > somewhat flaky. I keep getting svn: REPORT of > '/cournape/numpy.git/!svn/vcc/default': 200 OK (http://svn.github.com). > May be due to some network configuration, I don't know. Would be good if > someone else could check, > > IIRC, Stefan said that the newer version of the buildbot software works with GIT. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From mayankpj at gmail.com Fri Sep 17 01:16:51 2010 From: mayankpj at gmail.com (Mayank P Jain) Date: Fri, 17 Sep 2010 10:46:51 +0530 Subject: [Numpy-discussion] Viewer for 2D Numpy arrays (GUI) Message-ID: Currently I am exporting them to csv files, but I wonder if there is a viewer that can be used with native numpy array files to view and preferably modify the 2D arrays. Any help would be appreciated. Regards Mayank P Jain V R TechNiche Transportation Modeler/Planner Phone: +91 901 356 0583 -------------- next part -------------- An HTML attachment was scrubbed... URL: From nadavh at visionsense.com Fri Sep 17 02:38:29 2010 From: nadavh at visionsense.com (Nadav Horesh) Date: Fri, 17 Sep 2010 08:38:29 +0200 Subject: [Numpy-discussion] Viewer for 2D Numpy arrays (GUI) References: Message-ID: View 2D arrays: Most convenient: Matplotlib (imshow) As surface plot: Mayavi (Matplotlib has surface plots, but it is slow for large arrays) Modify files: I think the IDE spyder could help (and you can use mayavi/matplotlib within) Nadav -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? Mayank P Jain ????: ? 17-??????-10 08:16 ??: numpy-discussion ????: [Numpy-discussion] Viewer for 2D Numpy arrays (GUI) Currently I am exporting them to csv files, but I wonder if there is a viewer that can be used with native numpy array files to view and preferably modify the 2D arrays. Any help would be appreciated. Regards Mayank P Jain V R TechNiche Transportation Modeler/Planner Phone: +91 901 356 0583 begin_of_the_skype_highlighting??????????????+91 901 356 0583??????end_of_the_skype_highlighting -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3124 bytes Desc: not available URL: From mayankpj at gmail.com Fri Sep 17 02:53:26 2010 From: mayankpj at gmail.com (Mayank P Jain) Date: Fri, 17 Sep 2010 12:23:26 +0530 Subject: [Numpy-discussion] Viewer for 2D Numpy arrays (GUI) In-Reply-To: References: Message-ID: I thought about these options but what I need is excel like interface that displays the values for each cell and one can modify and save the files. This would be convenient way of saving large files in less space and at the same time, see them and would remove those additional steps of writing out and reading them in each time my program runs. Regards Mayank P Jain V R TechNiche Transportation Modeler/Planner Phone: +91 901 356 0583 2010/9/17 Nadav Horesh > View 2D arrays: > Most convenient: Matplotlib (imshow) > As surface plot: Mayavi (Matplotlib has surface plots, but it is slow for > large arrays) > > Modify files: > I think the IDE spyder could help (and you can use mayavi/matplotlib > within) > > Nadav > > > -----????? ??????----- > ???: numpy-discussion-bounces at scipy.org ??? Mayank P Jain > ????: ? 17-??????-10 08:16 > ??: numpy-discussion > ????: [Numpy-discussion] Viewer for 2D Numpy arrays (GUI) > > Currently I am exporting them to csv files, but I wonder if there is a > viewer that can be used with native numpy array files to view and > preferably > modify the 2D arrays. > Any help would be appreciated. > > > Regards > Mayank P Jain > > V R TechNiche > Transportation Modeler/Planner > Phone: +91 901 356 0583 begin_of_the_skype_highlighting +91 > 901 356 0583 end_of_the_skype_highlighting > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From massimodisasha at gmail.com Fri Sep 17 03:03:00 2010 From: massimodisasha at gmail.com (Massimo Di Stefano) Date: Fri, 17 Sep 2010 09:03:00 +0200 Subject: [Numpy-discussion] Viewer for 2D Numpy arrays (GUI) In-Reply-To: References: Message-ID: <6497605B-1F33-4976-9859-0D8B053D90A6@gmail.com> Hi, have yo already tryied Spyderlib : http://code.google.com/p/spyderlib/ a matlab-like environment based on pyqt you can store object like array in a QtTable and modify/save it. Ciao, Massimo. Il giorno 17/set/2010, alle ore 08.53, Mayank P Jain ha scritto: > I thought about these options but what I need is excel like interface that displays the values for each cell and one can modify and save the files. > > This would be convenient way of saving large files in less space and at the same time, see them and would remove those additional steps of writing out and reading them in each time my program runs. > > Regards > Mayank P Jain > > V R TechNiche > Transportation Modeler/Planner > Phone: +91 901 356 0583 > > > > 2010/9/17 Nadav Horesh > View 2D arrays: > Most convenient: Matplotlib (imshow) > As surface plot: Mayavi (Matplotlib has surface plots, but it is slow for large arrays) > > Modify files: > I think the IDE spyder could help (and you can use mayavi/matplotlib within) > > Nadav > > > -----????? ??????----- > ???: numpy-discussion-bounces at scipy.org ??? Mayank P Jain > ????: ? 17-??????-10 08:16 > ??: numpy-discussion > ????: [Numpy-discussion] Viewer for 2D Numpy arrays (GUI) > > Currently I am exporting them to csv files, but I wonder if there is a > viewer that can be used with native numpy array files to view and preferably > modify the 2D arrays. > Any help would be appreciated. > > > Regards > Mayank P Jain > > V R TechNiche > Transportation Modeler/Planner > Phone: +91 901 356 0583 begin_of_the_skype_highlighting +91 901 356 0583 end_of_the_skype_highlighting > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nicolas.Rougier at loria.fr Fri Sep 17 03:32:07 2010 From: Nicolas.Rougier at loria.fr (Nicolas Rougier) Date: Fri, 17 Sep 2010 09:32:07 +0200 Subject: [Numpy-discussion] Viewer for 2D Numpy arrays (GUI) In-Reply-To: <6497605B-1F33-4976-9859-0D8B053D90A6@gmail.com> References: <6497605B-1F33-4976-9859-0D8B053D90A6@gmail.com> Message-ID: <1284708727.20702.3.camel@sulfur> Maybe glumpy may be of some help: http://code.google.com/p/glumpy/ Nicolas On Fri, 2010-09-17 at 09:03 +0200, Massimo Di Stefano wrote: > Hi, > > > have yo already tryied Spyderlib : > > > http://code.google.com/p/spyderlib/ > > > a matlab-like environment based on pyqt > you can store object like array in a QtTable > and modify/save it. > > > Ciao, > > > Massimo. > > Il giorno 17/set/2010, alle ore 08.53, Mayank P Jain ha scritto: > > > I thought about these options but what I need is excel like > > interface that displays the values for each cell and one can modify > > and save the files. > > > > This would be convenient way of saving large files in less space and > > at the same time, see them and would remove those additional steps > > of writing out and reading them in each time my program runs. > > > > Regards > > Mayank P Jain > > > > V R TechNiche > > Transportation Modeler/Planner > > Phone: +91 901 356 0583 > > > > > > > > 2010/9/17 Nadav Horesh > > View 2D arrays: > > Most convenient: Matplotlib (imshow) > > As surface plot: Mayavi (Matplotlib has surface plots, but > > it is slow for large arrays) > > > > Modify files: > > I think the IDE spyder could help (and you can use > > mayavi/matplotlib within) > > > > Nadav > > > > < < -----????? ??????----- < < ???: numpy-discussion-bounces at scipy.org ??? Mayank P Jain < < ????: ? 17-??????-10 08:16 < < ??: numpy-discussion < < ????: [Numpy-discussion] Viewer for 2D Numpy arrays (GUI) > > > > > > Currently I am exporting them to csv files, but I wonder > > if there is a > > viewer that can be used with native numpy array files to > > view and preferably > > modify the 2D arrays. > > Any help would be appreciated. > > > > > > Regards > > Mayank P Jain > > > > V R TechNiche > > Transportation Modeler/Planner > > > > Phone: +91 901 356 0583 > > begin_of_the_skype_highlighting +91 901 356 > > 0583 end_of_the_skype_highlighting > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From slaunger at gmail.com Fri Sep 17 05:14:43 2010 From: slaunger at gmail.com (Kim Hansen) Date: Fri, 17 Sep 2010 11:14:43 +0200 Subject: [Numpy-discussion] Viewer for 2D Numpy arrays (GUI) In-Reply-To: References: Message-ID: 2010/9/17 Mayank P Jain > I thought about these options but what I need is excel like interface that > displays the values for each cell and one can modify and save the files. > > This would be convenient way of saving large files in less space and at the > same time, see them and would remove those additional steps of writing out > and reading them in each time my program runs. > > > > Personally I would store the numpy arrays in HDF5 files using the very convenient numpy-aware pytables module and then use the very fast and efficient HDF5 file viewer, VITables for the Excel like interface. If you have large amounts of data, this GUI is way faster to operate as compared to Excel and it has some convient utilities for filtering the data you want to see or edit. Best regards Kim -------------- next part -------------- An HTML attachment was scrubbed... URL: From numpy-discussion at maubp.freeserve.co.uk Fri Sep 17 07:01:10 2010 From: numpy-discussion at maubp.freeserve.co.uk (Peter) Date: Fri, 17 Sep 2010 12:01:10 +0100 Subject: [Numpy-discussion] Silent DeprecationWarnings under Python 2.7 onwards In-Reply-To: References: Message-ID: On Wed, Sep 15, 2010 at 3:34 PM, Benjamin Root wrote: > On Wed, Sep 15, 2010 at 8:28 AM, Ralf Gommers > wrote: >> >> >> On Fri, Sep 10, 2010 at 12:05 AM, Peter >>> http://docs.python.org/library/warnings.html#updating-code-for-new-versions-of-python >>> >>> This makes sense to me for deprecation warnings from Python >>> itself or the standard library - after all, Python 2.7 is the last of the >>> Python 2.x series. >> >> The reason for the change is explained in the paragraph you link to, 2.7 >> being the final minor release in the 2.x series isn't it. I didn't intend to suggest it was the official reason, rather this is my opinion: Given Python 2.7 is the last Python 2.x release, deprecation warnings from Python itself and the stdlib are only relevant to porting to Python 3, so having them silent makes sense for end users. >> There are many other packages/programs built on numpy, the user/developer >> distinction can be made in the same way as for Python itself. I fail to see >> a reason not to follow the lead of the Python core developers here. >> >> Cheers, >> Ralf > > I have to agree a bit with Peter.? I do understand the rationale of Python's > position, however, I have to wonder what is the point of DeprecationWarning > if it doesn't get displayed?? The warning is supposed to give a heads-up to > the developer to modify their code. > > Now, the argument can be made that a python developer should know to run > python with those warnings unmuted.? And I would agree for "true", career > programmers.? However, numpy/scipy/matplotlib have become environments unto > themselves, catered to converts from Matlab, R, S+ and other such > languages.? I would argue that many of the "developers" are not typical > programmers with proper development habits/skills.? Many (myself included) > are graduate students in scientific fields unrelated to computer science.? I > have to wonder how many of them would even be aware of the differences > between python versions? (or even which version they are using!). > > Anyway, my point is that the deprecation warnings are very valuable to > display and that we need to keep in mind the audience that SciPy has.? Maybe > we don't necessarily turn them on by default (somehow), but maybe the > documentation should highly recommend that they get turned on, thereby > raising awareness on the part of the user. We can make the warnings noisy by introducing a NumPyDeprecationWarning (which must not subclass DreprecationWarning as that would make it silent). This isn't very elegant though, and would mean existing scripts already using the warnings module to silence DreprecationWarning may need updating. At very least, I agree the NumPy/SciPy documentation should mention this issue of silent DeprecationWarnings on Python 2.7 (and whichever of the Python 3 series do this). > There is a side-benefit to mentioning the muted warnings issue in the > documentation.? If a developer later complains that a feature was removed > without any deprecation notices, we could simply point to the documentation > and say that we recommended turning the warnings on. It is an excuse, but at least documentation would have warned people. Peter From hodgson.neil at yahoo.co.uk Fri Sep 17 08:17:12 2010 From: hodgson.neil at yahoo.co.uk (Neil Hodgson) Date: Fri, 17 Sep 2010 12:17:12 +0000 (GMT) Subject: [Numpy-discussion] loadtxt stop Message-ID: <597388.34549.qm@web27501.mail.ukl.yahoo.com> Hi, I been looking around and could spot anything on this. Quite often I want to read a homogeneous block of data from within a file. The skiprows option is great for missing out the section before the data starts, but if there is anything below then loadtxt will choke. I wondered if there was a possibility to put an endmarker= ? For example, if I want to load text from a large! file that looks like this header line header line 1 2.0 3.0 2 4.5 5.7 ... 500 4.3 5.4 END more headers more headers 1 2.0 3.0 3.14 1.1414 2 4.5 5.7 1.14 3.1459 ... 500 4.3 5.4 0.000 0.001 END Then I can use skiprows=2, but loadtxt will choke when it gets to 'END'. To read t -------------- next part -------------- An HTML attachment was scrubbed... URL: From hodgson.neil at yahoo.co.uk Fri Sep 17 08:40:53 2010 From: hodgson.neil at yahoo.co.uk (Neil Hodgson) Date: Fri, 17 Sep 2010 12:40:53 +0000 (GMT) Subject: [Numpy-discussion] loadtxt stop Message-ID: <853737.68306.qm@web27507.mail.ukl.yahoo.com> oops, I meant to save my post but I sent it instead - doh! In the end, the question was; is worth adding start= and stop= markers into loadtxt to allow grabbing sections of a file between two known headers? I imagine it's something that people come up against regularly. Thanks, Neil ________________________________ From: Neil Hodgson To: numpy-discussion at scipy.org Sent: Fri, 17 September, 2010 14:17:12 Subject: loadtxt stop Hi, I been looking around and could spot anything on this. Quite often I want to read a homogeneous block of data from within a file. The skiprows option is great for missing out the section before the data starts, but if there is anything below then loadtxt will choke. I wondered if there was a possibility to put an endmarker= ? For example, if I want to load text from a large! file that looks like this header line header line 1 2.0 3.0 2 4.5 5.7 ... 500 4.3 5.4 END more headers more headers 1 2.0 3.0 3.14 1.1414 2 4.5 5.7 1.14 3.1459 ... 500 4.3 5.4 0.000 0.001 END Then I can use skiprows=2, but loadtxt will choke when it gets to 'END'. To read t -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Fri Sep 17 09:03:10 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 17 Sep 2010 15:03:10 +0200 Subject: [Numpy-discussion] loadtxt stop In-Reply-To: <853737.68306.qm@web27507.mail.ukl.yahoo.com> References: <853737.68306.qm@web27507.mail.ukl.yahoo.com> Message-ID: <6F03EC5F-F3F6-4149-864B-7748C7240B35@gmail.com> On Sep 17, 2010, at 2:40 PM, Neil Hodgson wrote: > oops, I meant to save my post but I sent it instead - doh! > > In the end, the question was; is worth adding start= and stop= markers into loadtxt to allow grabbing sections of a file between two known headers? I imagine it's something that people come up against regularly. genfromtxt comes with skip_header and skip_footer that do what you want. Earlier this week, I corrected a bug w/ skip_footer on the SVN (now git) version of the sources. Please check it out. Try to be as specific as possible with your input options, that'll make genfromtxt more efficient. From jh at physics.ucf.edu Fri Sep 17 09:41:43 2010 From: jh at physics.ucf.edu (Joe Harrington) Date: Fri, 17 Sep 2010 09:41:43 -0400 Subject: [Numpy-discussion] Viewer for 2D Numpy arrays (GUI) In-Reply-To: (numpy-discussion-request@scipy.org) References: Message-ID: There is a stand-alone program in wide use by astronomers that does 2D floating-point image display called ds9. It has a 2-way numpy interface and a rich set of image viewing/annotating features, including interactive colormapping, zoom/pan of very large images, n-up displays, movies, live row and column plots, etc. See http://hea-www.harvard.edu/RD/ds9/ There is also the lighter, native-Python ntv routine, which uses matplotlib: http://planets.ucf.edu/resources/open-source-software/ http://code.google.com/p/ntv/ This is a new package that can display 2D floating-point arrays as images and do some astro-specific stuff like point-source centering and extraction (aka photometry) from images. The author, Nate Lust, is eager for contributions, such as help in adding features and improving the web pages for it. It would probably be straightforward to add a table-viewing/editing feature to this, such that you would click in an image and get a spreadsheet-like interface centered where you clicked. --jh-- Date: Fri, 17 Sep 2010 10:46:51 +0530 From: Mayank P Jain Subject: [Numpy-discussion] Viewer for 2D Numpy arrays (GUI) To: numpy-discussion Message-ID: Content-Type: text/plain; charset="iso-8859-1" Currently I am exporting them to csv files, but I wonder if there is a viewer that can be used with native numpy array files to view and preferably modify the 2D arrays. Any help would be appreciated. Regards Mayank P Jain V R TechNiche Transportation Modeler/Planner Phone: +91 901 356 0583 From stefan at sun.ac.za Fri Sep 17 11:45:27 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 17 Sep 2010 17:45:27 +0200 Subject: [Numpy-discussion] Numpy SVN frozen; move to Git In-Reply-To: References: <4C92C2B4.8020707@silveregg.co.jp> Message-ID: On Fri, Sep 17, 2010 at 3:49 AM, Charles R Harris wrote: > IIRC, Stefan said that the newer version of the buildbot software works with > GIT. Looks like the only missing component is a Git Poller, and because this machine is behind such a scary firewall I'll have to ask the administrators to help. I'll send the request right away, so hopefully we'll be up and running after the weekend. Regards St?fan From gokhansever at gmail.com Fri Sep 17 11:53:34 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Fri, 17 Sep 2010 10:53:34 -0500 Subject: [Numpy-discussion] Viewer for 2D Numpy arrays (GUI) In-Reply-To: References: Message-ID: On Fri, Sep 17, 2010 at 12:16 AM, Mayank P Jain wrote: > Currently I am exporting them to csv files, but I wonder if there is a > viewer that can be used with native numpy array files to view and preferably > modify the 2D arrays. > Any help would be appreciated. > I would suggest using IPython + Matplotlib, and as a text editor Vim. I have just learnt how to select the same column in Vim [ http://stackoverflow.com/questions/3736678/select-all-column]. This trio is as simple as and yet as powerful as it gets for my array view + visualization tasks. -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Fri Sep 17 13:19:18 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 17 Sep 2010 13:19:18 -0400 Subject: [Numpy-discussion] weighted mean; weighted standard error of the mean (sem) In-Reply-To: References: <14170_1284092042_1284092042_AANLkTi=PLFWR7FPrr_bHvNxt_jdDLMCrrw2WpYHFOgbw@mail.gmail.com> Message-ID: On Fri, Sep 10, 2010 at 3:01 PM, wrote: > On Fri, Sep 10, 2010 at 1:58 PM, Christopher Barrington-Leigh > wrote: >> Interesting. Thanks Erin, Josef and Keith. > > thanks to the stata page at least I figured out that WLS is aweights > with asumption mu_i = mu > > import numpy as np > from scikits.statsmodels import WLS > w0 = np.arange(20) % 4 > w = 1.*w0/w0.sum() > y = 2 + np.random.randn(20) > >>>> res = WLS(y, np.ones(20), weights=w).fit() >>>> print res.params, res.bse > [ 2.29083069] [ 0.17562867] >>>> m = np.dot(w, y) >>>> m > 2.2908306865128401 >>>> s2u = 1/(nobs-1.) * np.dot(w, (y - m)**2) >>>> s2u > 0.030845429945278956 >>>> np.sqrt(s2u) > 0.17562867062435722 > > >> >> There is a nice article on this at >> http://www.stata.com/support/faqs/stat/supweight.html. In my case, the >> model I've in mind is to assume that the expected value (mean) is the same >> for each sample, and that the weights are/should be normalised, whence a >> consistent estimator for sem is straightforward (if second moments can >> be assumed to be >> well behaved?). I suspect that this (survey-like) case is also one of >> the two most standard/most common >> expression that people want when they ask for an s.e. of the mean for >> a weighted dataset. The other would be when the weights are not to be >> normalised, but represent standard errors on the individual >> measurements. >> >> Surely what one wants, in the end, is a single function (or whatever) >> called mean or sem which calculates different values for different >> specified choices of model (assumptions)? And where possible that it has a >> default model in mind for when none is specified? > > I find aweights and pweights still confusing, plus necessary auxillary > assumptions. > > I don't find Stata docs very helpful, I almost never find a clear > description of the formulas (and I don't have any Stata books). > > If you have or write some examples that show or apply in the different > cases, then this would be very helpful to get a structure into this > area, weighting and survey sampling, and population versus clustered > or stratified sample statistics. > > I'm still pretty lost with the literature on surveys. I found the formula collection for SPSS http://support.spss.com/productsext/statistics/documentation/19/clientindex.html#Manuals pdf file for algorithms Not much explanation, and sometimes it's not really clear what a variable stands for exactly, but a useful summary of formulas. Also the formulas might not always be for a general case, e.g. formulas for non-parametric tests seem to be missing tie-handling (from a quick look). More compressed than the details descriptions in SAS, but much more explicit than Stata and R without buying the books. The chapter on T Test Algorithm carries population frequencies throughout, this should work for weighted statistics, but maybe not for different complex sampling schemes (a-weights, p-weights,...). Josef > > Josef > > >> >> thanks, >> Chris >> >> On Thu, Sep 9, 2010 at 9:13 PM, Keith Goodman wrote: >>> >>>> ma.std() >>> >> ? 3.2548815339711115 >>> > >>> > or maybe `w` reflects an underlying sampling scheme and you should >>> > sample in the bootstrap according to w ? >>> >>> Yes.... >>> >>> > if weighted average is a sum of linear functions of (normal) >>> > distributed random variables, it still depends on whether the >>> > individual observations have the same or different variances, e.g. >>> > http://en.wikipedia.org/wiki/Weighted_mean#Statistical_properties >>> >>> ...lots of possibilities. As you have shown the problem is not yet >>> well defined. Not much specification needed for the weighted mean, >>> lots needed for the standard error of the weighted mean. >>> >>> > What I can't figure out is whether if you assume simga_i = sigma for >>> > all observation i, do we use the weighted or the unweighted variance >>> > to get an estimate of sigma. And I'm not able to replicate with simple >>> > calculations what statsmodels.WLS gives me. >>> >>> My guess: if all you want is sigma of the individual i and you know >>> sigma is the same for all i, then I suppose you don't care about the >>> weight. >>> >>> > >>> > ??? >>> > >>> > Josef >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > From ndbecker2 at gmail.com Fri Sep 17 13:42:41 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 17 Sep 2010 13:42:41 -0400 Subject: [Numpy-discussion] buffer arg to ndarray Message-ID: np.ndarray description says: buffer : object exposing buffer interface It's nice that it works with mmap: b = mmap.mmap (...) u = np.ndarray (buffer=b ...) but you wouldn't know it from the above description. It doesn't look to me that the object returned by mmap exposes the buffer interface, does it? In [26]: dir(b) Out[26]: ['__add__', '__class__', '__delattr__', '__delitem__', '__delslice__', '__doc__', '__format__', '__getattribute__', '__getitem__', '__getslice__', '__hash__', '__init__', '__len__', '__mul__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmul__', '__setattr__', '__setitem__', '__setslice__', '__sizeof__', '__str__', '__subclasshook__', 'close', 'find', 'flush', 'move', 'read', 'read_byte', 'readline', 'resize', 'rfind', 'seek', 'size', 'tell', 'write', 'write_byte'] From ndbecker2 at gmail.com Fri Sep 17 13:47:47 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 17 Sep 2010 13:47:47 -0400 Subject: [Numpy-discussion] restrictions on fancy indexing Message-ID: It's nice I can do: f = np.linspace (0, 1, 100) u[f<.1] = 0 cool, this seems to work also: u[np.abs(f)<.1] = 0 cool! But exactly what kind of expressions are possible here? Certainly not arbitrary code. From jsalvati at u.washington.edu Fri Sep 17 13:56:37 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Fri, 17 Sep 2010 10:56:37 -0700 Subject: [Numpy-discussion] restrictions on fancy indexing In-Reply-To: References: Message-ID: There's a tutorial here: http://www.scipy.org/Cookbook/Indexing Look down for the section on Fancy Indexing. On Fri, Sep 17, 2010 at 10:47 AM, Neal Becker wrote: > It's nice I can do: > > f = np.linspace (0, 1, 100) > u[f<.1] = 0 > > cool, this seems to work also: > > u[np.abs(f)<.1] = 0 > > cool! But exactly what kind of expressions are possible here? Certainly > not arbitrary code. > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Sep 17 14:00:02 2010 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 17 Sep 2010 13:00:02 -0500 Subject: [Numpy-discussion] buffer arg to ndarray In-Reply-To: References: Message-ID: On Fri, Sep 17, 2010 at 12:42, Neal Becker wrote: > np.ndarray description says: > ?buffer : object exposing buffer interface > > It's nice that it works with mmap: > > b = mmap.mmap (...) > u = np.ndarray (buffer=b ...) > > but you wouldn't know it from the above description. > > It doesn't look to me that the object returned by mmap exposes the buffer > interface, does it? The buffer interface is a C thing. You don't see Python-level methods. The cheapest way to test if an object can be made into a buffer is to try it: In [22]: m = mmap.mmap(f.fileno(), 0) In [23]: buffer(m) Out[23]: -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From ben.root at ou.edu Fri Sep 17 14:01:30 2010 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 17 Sep 2010 13:01:30 -0500 Subject: [Numpy-discussion] Viewer for 2D Numpy arrays (GUI) In-Reply-To: References: Message-ID: On Fri, Sep 17, 2010 at 10:53 AM, G?khan Sever wrote: > On Fri, Sep 17, 2010 at 12:16 AM, Mayank P Jain wrote: > >> Currently I am exporting them to csv files, but I wonder if there is >> a viewer that can be used with native numpy array files to view and >> preferably modify the 2D arrays. >> Any help would be appreciated. >> > > I would suggest using IPython + Matplotlib, and as a text editor Vim. I > have just learnt how to select the same column in Vim [ > http://stackoverflow.com/questions/3736678/select-all-column]. This trio > is as simple as and yet as powerful as it gets for my array view > + visualization tasks. > > -- > G?khan > > Neat! Is there anything vi can't do? Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From aarchiba at physics.mcgill.ca Fri Sep 17 14:04:56 2010 From: aarchiba at physics.mcgill.ca (Anne Archibald) Date: Fri, 17 Sep 2010 14:04:56 -0400 Subject: [Numpy-discussion] restrictions on fancy indexing In-Reply-To: References: Message-ID: On 17 September 2010 13:47, Neal Becker wrote: > It's nice I can do: > > f = np.linspace (0, 1, 100) > u[f<.1] = 0 > > cool, this seems to work also: > > u[np.abs(f)<.1] = 0 > > cool! ?But exactly what kind of expressions are possible here? ?Certainly > not arbitrary code. The short answer is, anything that yields a boolean or integer array. There's no syntactical magic here. It might be clearer to write it as: c = np.abs(f)<.1 u[c] = 0 As for what generates boolean arrays, well, they're just numpy arrays, you can mangle them any way you want. But in particular, < > == != are operators that take two arrays and yield a boolean array. Also useful are ~ | and &, which are the logical operators on boolean arrays. Anne From Chris.Barker at noaa.gov Fri Sep 17 14:51:13 2010 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 17 Sep 2010 11:51:13 -0700 Subject: [Numpy-discussion] loadtxt stop In-Reply-To: <853737.68306.qm@web27507.mail.ukl.yahoo.com> References: <853737.68306.qm@web27507.mail.ukl.yahoo.com> Message-ID: <4C93B8A1.30209@noaa.gov> Neil Hodgson wrote: > In the end, the question was; is worth adding start= and stop= markers > into loadtxt to allow grabbing sections of a file between two known > headers? I imagine it's something that people come up against regularly. maybe not so regular. However, a common use would be to be able load only n rows, which also does not appear to be supported. That would be nice. -Chris > Thanks, > Neil > > ------------------------------------------------------------------------ > *From:* Neil Hodgson > *To:* numpy-discussion at scipy.org > *Sent:* Fri, 17 September, 2010 14:17:12 > *Subject:* loadtxt stop > > Hi, > > I been looking around and could spot anything on this. Quite often I > want to read a homogeneous block of data from within a file. The > skiprows option is great for missing out the section before the data > starts, but if there is anything below then loadtxt will choke. I > wondered if there was a possibility to put an endmarker= ? > > For example, if I want to load text from a large! file that looks like this > > header line > header line > 1 2.0 3.0 > 2 4.5 5.7 > ... > 500 4.3 5.4 > END > more headers > more headers > 1 2.0 3.0 3.14 1.1414 > 2 4.5 5.7 1.14 3.1459 > ... > 500 4.3 5.4 0.000 0.001 > END > > Then I can use skiprows=2, but loadtxt will choke when it gets to > 'END'. To read t > > > > ------------------------------------------------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From efiring at hawaii.edu Fri Sep 17 15:09:14 2010 From: efiring at hawaii.edu (Eric Firing) Date: Fri, 17 Sep 2010 09:09:14 -1000 Subject: [Numpy-discussion] restrictions on fancy indexing In-Reply-To: References: Message-ID: <4C93BCDA.9000507@hawaii.edu> On 09/17/2010 08:04 AM, Anne Archibald wrote: > On 17 September 2010 13:47, Neal Becker wrote: >> It's nice I can do: >> >> f = np.linspace (0, 1, 100) >> u[f<.1] = 0 >> >> cool, this seems to work also: >> >> u[np.abs(f)<.1] = 0 >> >> cool! But exactly what kind of expressions are possible here? Certainly >> not arbitrary code. > > The short answer is, anything that yields a boolean or integer array. > There's no syntactical magic here. It might be clearer to write it as: > > c = np.abs(f)<.1 > u[c] = 0 > > As for what generates boolean arrays, well, they're just numpy arrays, > you can mangle them any way you want. But in particular,< > == != are > operators that take two arrays and yield a boolean array. Also useful > are ~ | and&, which are the logical operators on boolean arrays. It can be important to bear in mind that they are not actually logical operators, they are bitwise operators pressed into service. Functionally, they substitute for logical operators on boolean arrays, but one must watch out for their high precedence. This typically requires using parentheses where they would not be needed for the true python logical operators: With python scalars: a > b and c < d With numpy arrays: (a > b) & (c < d) Eric > > Anne > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From zachary.pincus at yale.edu Fri Sep 17 15:34:44 2010 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Fri, 17 Sep 2010 15:34:44 -0400 Subject: [Numpy-discussion] loadtxt stop In-Reply-To: <4C93B8A1.30209@noaa.gov> References: <853737.68306.qm@web27507.mail.ukl.yahoo.com> <4C93B8A1.30209@noaa.gov> Message-ID: >> In the end, the question was; is worth adding start= and stop= >> markers >> into loadtxt to allow grabbing sections of a file between two known >> headers? I imagine it's something that people come up against >> regularly. Simple enough to wrap your file in a new file-like object that stops coughing up lines when the delimiter is found, no? class TruncatingFile(object): def __init__(self, fh, delimiter='END'): self.fh = fh self.delimiter=delimiter self.done = False def readline(self): if self.done: return '' line = self.fh.readline() if line.strip() == self.delimiter: self.done = True return '' return line def __iter__(self): return self def next(self): line = self.fh.next() if line.strip() == self.delimiter: self.done = True raise StopIteration() return line from StringIO import StringIO c = StringIO("0 1\n2 3\nEND") numpy.loadtxt(TruncatingFile(c)) Though, really, it's annoying that numpy.loadtxt needs both the readline function *and* the iterator protocol. If it just used iterators, you could do: def truncator(fh, delimiter='END'): for line in fh: if line.strip() == delimiter: break yield line numpy.loadtxt(truncator(c)) Maybe I'll try to work up a patch for this. Zach On Sep 17, 2010, at 2:51 PM, Christopher Barker wrote: > Neil Hodgson wrote: >> In the end, the question was; is worth adding start= and stop= >> markers >> into loadtxt to allow grabbing sections of a file between two known >> headers? I imagine it's something that people come up against >> regularly. > > maybe not so regular. However, a common use would be to be able load > only n rows, which also does not appear to be supported. That would > be nice. > > -Chris > > > >> Thanks, >> Neil >> >> ------------------------------------------------------------------------ >> *From:* Neil Hodgson >> *To:* numpy-discussion at scipy.org >> *Sent:* Fri, 17 September, 2010 14:17:12 >> *Subject:* loadtxt stop >> >> Hi, >> >> I been looking around and could spot anything on this. Quite often I >> want to read a homogeneous block of data from within a file. The >> skiprows option is great for missing out the section before the data >> starts, but if there is anything below then loadtxt will choke. I >> wondered if there was a possibility to put an endmarker= ? >> >> For example, if I want to load text from a large! file that looks >> like this >> >> header line >> header line >> 1 2.0 3.0 >> 2 4.5 5.7 >> ... >> 500 4.3 5.4 >> END >> more headers >> more headers >> 1 2.0 3.0 3.14 1.1414 >> 2 4.5 5.7 1.14 3.1459 >> ... >> 500 4.3 5.4 0.000 0.001 >> END >> >> Then I can use skiprows=2, but loadtxt will choke when it gets to >> 'END'. To read t >> >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From zachary.pincus at yale.edu Fri Sep 17 15:50:27 2010 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Fri, 17 Sep 2010 15:50:27 -0400 Subject: [Numpy-discussion] loadtxt stop In-Reply-To: References: <853737.68306.qm@web27507.mail.ukl.yahoo.com> <4C93B8A1.30209@noaa.gov> Message-ID: <1A3A2705-4B90-4547-A58C-31B0D4EE3720@yale.edu> > Though, really, it's annoying that numpy.loadtxt needs both the > readline function *and* the iterator protocol. If it just used > iterators, you could do: > > def truncator(fh, delimiter='END'): > for line in fh: > if line.strip() == delimiter: > break > yield line > > numpy.loadtxt(truncator(c)) > > Maybe I'll try to work up a patch for this. That seemed easy... worth applying? Won't break compatibility, because the previous loadtxt required both fname.readline and fname.__iter__, while this requires only the latter. Index: numpy/lib/npyio.py =================================================================== --- numpy/lib/npyio.py (revision 8716) +++ numpy/lib/npyio.py (working copy) @@ -597,10 +597,11 @@ fh = bz2.BZ2File(fname) else: fh = open(fname, 'U') - elif hasattr(fname, 'readline'): - fh = fname else: - raise ValueError('fname must be a string or file handle') + try: + fh = iter(fname) + except: + raise ValueError('fname must be a string or file handle') X = [] def flatten_dtype(dt): @@ -633,14 +634,18 @@ # Skip the first `skiprows` lines for i in xrange(skiprows): - fh.readline() + try: + fh.next() + except StopIteration: + raise IOError('End-of-file reached before encountering data.') # Read until we find a line with some values, and use # it to estimate the number of columns, N. first_vals = None while not first_vals: - first_line = fh.readline() - if not first_line: # EOF reached + try: + first_line = fh.next() + except StopIteration: raise IOError('End-of-file reached before encountering data.') first_vals = split_line(first_line) N = len(usecols or first_vals) From ben.root at ou.edu Fri Sep 17 15:59:11 2010 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 17 Sep 2010 14:59:11 -0500 Subject: [Numpy-discussion] loadtxt stop In-Reply-To: <1A3A2705-4B90-4547-A58C-31B0D4EE3720@yale.edu> References: <853737.68306.qm@web27507.mail.ukl.yahoo.com> <4C93B8A1.30209@noaa.gov> <1A3A2705-4B90-4547-A58C-31B0D4EE3720@yale.edu> Message-ID: On Fri, Sep 17, 2010 at 2:50 PM, Zachary Pincus wrote: > > Though, really, it's annoying that numpy.loadtxt needs both the > > readline function *and* the iterator protocol. If it just used > > iterators, you could do: > > > > def truncator(fh, delimiter='END'): > > for line in fh: > > if line.strip() == delimiter: > > break > > yield line > > > > numpy.loadtxt(truncator(c)) > > > > Maybe I'll try to work up a patch for this. > > > That seemed easy... worth applying? Won't break compatibility, because > the previous loadtxt required both fname.readline and fname.__iter__, > while this requires only the latter. > > > Index: numpy/lib/npyio.py > =================================================================== > --- numpy/lib/npyio.py (revision 8716) > +++ numpy/lib/npyio.py (working copy) > @@ -597,10 +597,11 @@ > fh = bz2.BZ2File(fname) > else: > fh = open(fname, 'U') > - elif hasattr(fname, 'readline'): > - fh = fname > else: > - raise ValueError('fname must be a string or file handle') > + try: > + fh = iter(fname) > + except: > + raise ValueError('fname must be a string or file handle') > X = [] > > def flatten_dtype(dt): > @@ -633,14 +634,18 @@ > > # Skip the first `skiprows` lines > for i in xrange(skiprows): > - fh.readline() > + try: > + fh.next() > + except StopIteration: > + raise IOError('End-of-file reached before > encountering data.') > > # Read until we find a line with some values, and use > # it to estimate the number of columns, N. > first_vals = None > while not first_vals: > - first_line = fh.readline() > - if not first_line: # EOF reached > + try: > + first_line = fh.next() > + except StopIteration: > raise IOError('End-of-file reached before > encountering data.') > first_vals = split_line(first_line) > N = len(usecols or first_vals) > > So, this code will still raise an error for an empty file. Personally, I consider that a bug because I would expect to receive an empty array. I could understand raising an error for a non-empty file that does not contain anything useful. For comparison, Matlab returns an empty matrix for loading an emtpy text file. This has been a long-standing annoyance for me, along with the behavior with a single-line data file. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.pincus at yale.edu Fri Sep 17 16:04:25 2010 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Fri, 17 Sep 2010 16:04:25 -0400 Subject: [Numpy-discussion] loadtxt stop In-Reply-To: References: <853737.68306.qm@web27507.mail.ukl.yahoo.com> <4C93B8A1.30209@noaa.gov> <1A3A2705-4B90-4547-A58C-31B0D4EE3720@yale.edu> Message-ID: <39E21281-4723-4F1A-B9A9-C2B0D48B9183@yale.edu> On Sep 17, 2010, at 3:59 PM, Benjamin Root wrote: > So, this code will still raise an error for an empty file. > Personally, I consider that a bug because I would expect to receive > an empty array. I could understand raising an error for a non-empty > file that does not contain anything useful. For comparison, Matlab > returns an empty matrix for loading an emtpy text file. > > This has been a long-standing annoyance for me, along with the > behavior with a single-line data file. Agreed... I just wanted to make the patch as identical in behavior to the old version as possible. Though again, simple shims around loadtxt (as in my previous examples) can yield the desired behavior easily enough. From ben.root at ou.edu Fri Sep 17 16:13:09 2010 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 17 Sep 2010 15:13:09 -0500 Subject: [Numpy-discussion] loadtxt stop In-Reply-To: <39E21281-4723-4F1A-B9A9-C2B0D48B9183@yale.edu> References: <853737.68306.qm@web27507.mail.ukl.yahoo.com> <4C93B8A1.30209@noaa.gov> <1A3A2705-4B90-4547-A58C-31B0D4EE3720@yale.edu> <39E21281-4723-4F1A-B9A9-C2B0D48B9183@yale.edu> Message-ID: On Fri, Sep 17, 2010 at 3:04 PM, Zachary Pincus wrote: > On Sep 17, 2010, at 3:59 PM, Benjamin Root wrote: > > > So, this code will still raise an error for an empty file. > > Personally, I consider that a bug because I would expect to receive > > an empty array. I could understand raising an error for a non-empty > > file that does not contain anything useful. For comparison, Matlab > > returns an empty matrix for loading an emtpy text file. > > > > This has been a long-standing annoyance for me, along with the > > behavior with a single-line data file. > > Agreed... I just wanted to make the patch as identical in behavior to > the old version as possible. Though again, simple shims around loadtxt > (as in my previous examples) can yield the desired behavior easily > enough. > > Fair enough. No need to mix a bugfix with a feature request. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From mayankpj at gmail.com Fri Sep 17 22:27:15 2010 From: mayankpj at gmail.com (Mayank P Jain) Date: Sat, 18 Sep 2010 07:57:15 +0530 Subject: [Numpy-discussion] Viewer for 2D Numpy arrays (GUI) In-Reply-To: References: Message-ID: thanks everyone for wonderful suggestions. I am currently trying out spyder as it also gives me free interpreter and looks simple.......... but all the suggestions are really nice............ and will definitely be of use later for me............ Regards Mayank P Jain V R TechNiche Transportation Modeler/Planner Phone: +91 901 356 0583 On Fri, Sep 17, 2010 at 11:31 PM, Benjamin Root wrote: > On Fri, Sep 17, 2010 at 10:53 AM, G?khan Sever wrote: > >> On Fri, Sep 17, 2010 at 12:16 AM, Mayank P Jain wrote: >> >>> Currently I am exporting them to csv files, but I wonder if there is >>> a viewer that can be used with native numpy array files to view and >>> preferably modify the 2D arrays. >>> Any help would be appreciated. >>> >> >> I would suggest using IPython + Matplotlib, and as a text editor Vim. I >> have just learnt how to select the same column in Vim [ >> http://stackoverflow.com/questions/3736678/select-all-column]. This trio >> is as simple as and yet as powerful as it gets for my array view >> + visualization tasks. >> >> -- >> G?khan >> >> > Neat! Is there anything vi can't do? > > Ben Root > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Sat Sep 18 06:02:05 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 18 Sep 2010 03:02:05 -0700 Subject: [Numpy-discussion] Numpy SVN frozen; move to Git In-Reply-To: References: Message-ID: On Thu, Sep 16, 2010 at 1:58 AM, Pauli Virtanen wrote: > > > Numpy SVN repository is now frozen, and does not accept new commits. > Future development should end up in the Git repository: > > ? ?http://github.com/numpy/numpy Joining my voice to the collective thank you, I figured I'd pass along something I stumbled upon just now as I was reading on github's new (massively improved) pull requests; a nice description of branch management using git for a real-world scenario with release/development branches, etc: http://nvie.com/posts/a-successful-git-branching-model/ None of it is a new idea, but it's very nicely and concisely explained. Regards, f From vs at it.uu.se Sat Sep 18 08:09:24 2010 From: vs at it.uu.se (Virgil Stokes) Date: Sat, 18 Sep 2010 14:09:24 +0200 Subject: [Numpy-discussion] Financial TS models Message-ID: <4C94ABF4.5090806@it.uu.se> I am considering the development of an all Python package (with numpy and matplotlib) for the modeling and analysis of financial time series. This is a rather ambitious and could take sometime before I can have something that is useful. Thus, any suggestions, pointers, etc. to related work would be appreciated. Thank you, --V From josef.pktd at gmail.com Sat Sep 18 10:13:34 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 18 Sep 2010 10:13:34 -0400 Subject: [Numpy-discussion] Financial TS models In-Reply-To: <4C94ABF4.5090806@it.uu.se> References: <4C94ABF4.5090806@it.uu.se> Message-ID: On Sat, Sep 18, 2010 at 8:09 AM, Virgil Stokes wrote: > ?I am considering the development of an all Python package (with numpy and > matplotlib) for the modeling and analysis of financial time series. > > This is a rather ambitious and could take sometime before I can have something > that is useful. ?Thus, any suggestions, pointers, etc. to related work would be > appreciated. Depends on what you want to do, but I would join or build on top of an existing package. I just got distracted with an extended internet search after finding http://github.com/quantmind/dynts (They use Redis as an in-memory and persistent storage. After reading up a bit, I think this might be useful if you have a web front end http://github.com/lsbardel/jflow in mind, but maybe not as good as hdf5 for desktop work. Just guessing since I used neither, and I always worry about installation problems on Windows.) They just started public development but all packages are in BSD from what I have seen. Otherwise, I would build on top of pandas, scikits.timeseries or larry or tabular if you want to handle your own time variable. For specific financial time series, e.g. stocks, exchange rates, options, I have seen only bits and pieces, or only partially implemented code (with a BSD compatible license), outside of quantlib and it's python bindings. Maybe someone knows more about what's available. For the econometrics/statistical analysis I haven't seen much outside of pandas and statsmodels in this area (besides isolated examples and recipes). I started to write on this in the statsmodels sandbox (including simulators). "modeling and analysis of financial time series" is a big project, and to get any results within a reasonable amount of time (unless you are part of a large team) is to specialize on some pieces. This is just my impression, since I thought of doing the same thing, but didn't see any way to get very far. (I just spend some weekends just to get the data from the Federal Reserve and wrap the API for the economics data base (Fred) of the Federal Reserve Saint Louis, the boring storage backend is zipped csv-files) Josef > > Thank you, > --V > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From josef.pktd at gmail.com Sat Sep 18 10:33:44 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 18 Sep 2010 10:33:44 -0400 Subject: [Numpy-discussion] Financial TS models In-Reply-To: References: <4C94ABF4.5090806@it.uu.se> Message-ID: On Sat, Sep 18, 2010 at 10:13 AM, wrote: > On Sat, Sep 18, 2010 at 8:09 AM, Virgil Stokes wrote: >> ?I am considering the development of an all Python package (with numpy and >> matplotlib) for the modeling and analysis of financial time series. >> >> This is a rather ambitious and could take sometime before I can have something >> that is useful. ?Thus, any suggestions, pointers, etc. to related work would be >> appreciated. I should have just asked you: What do you have in mind? instead of writing my notes from the readings I just did into the email. I think any open source contributions in this area will be very useful with the increased popularity of python in Finance. Josef > > Depends on what you want to do, but I would join or build on top of an > existing package. > > I just got distracted with an extended internet search after finding > > http://github.com/quantmind/dynts > > (They use Redis as an in-memory and persistent storage. After reading > up a bit, I think this might be useful if you have a web front end > http://github.com/lsbardel/jflow in mind, but maybe not as good as > hdf5 for desktop work. Just guessing since I used neither, and I > always worry about installation problems on Windows.) > They just started public development but all packages are in BSD from > what I have seen. > > Otherwise, I would build on top of pandas, scikits.timeseries or larry > or tabular if you want to handle your own time variable. > > For specific financial time series, e.g. stocks, exchange rates, > options, I have seen only bits and pieces, or only partially > implemented code (with a BSD compatible license), outside of quantlib > and it's python bindings. > > Maybe someone knows more about what's available. > > For the econometrics/statistical analysis I haven't seen much outside > of pandas and statsmodels in this area (besides isolated examples and > recipes). I started to write on this in the statsmodels sandbox > (including simulators). > > ?"modeling and analysis of financial time series" is a big project, > and to get any results within a reasonable amount of time (unless you > are part of a large team) is to specialize on some pieces. > > This is just my impression, since I thought of doing the same thing, > but didn't see any way to get very far. > > (I just spend some weekends just to get the data from the Federal > Reserve and wrap the API for the economics data base (Fred) of the > Federal Reserve Saint Louis, the boring storage backend is zipped > csv-files) > > Josef > >> >> Thank you, >> --V >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > From charlesr.harris at gmail.com Sat Sep 18 11:12:18 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 18 Sep 2010 09:12:18 -0600 Subject: [Numpy-discussion] Numpy SVN frozen; move to Git In-Reply-To: References: Message-ID: On Sat, Sep 18, 2010 at 4:02 AM, Fernando Perez wrote: > On Thu, Sep 16, 2010 at 1:58 AM, Pauli Virtanen wrote: > > > > > > Numpy SVN repository is now frozen, and does not accept new commits. > > Future development should end up in the Git repository: > > > > http://github.com/numpy/numpy > > Joining my voice to the collective thank you, I figured I'd pass along > something I stumbled upon just now as I was reading on github's new > (massively improved) pull requests; a nice description of branch > management using git for a real-world scenario with > release/development branches, etc: > > http://nvie.com/posts/a-successful-git-branching-model/ > > None of it is a new idea, but it's very nicely and concisely explained. > > But the vim post was the best thing at the site ;) I've now got pathogen and snipmate in my .vim folder. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wesmckinn at gmail.com Sat Sep 18 17:45:35 2010 From: wesmckinn at gmail.com (Wes McKinney) Date: Sat, 18 Sep 2010 17:45:35 -0400 Subject: [Numpy-discussion] Financial TS models In-Reply-To: References: <4C94ABF4.5090806@it.uu.se> Message-ID: On Sat, Sep 18, 2010 at 10:33 AM, wrote: > On Sat, Sep 18, 2010 at 10:13 AM, ? wrote: >> On Sat, Sep 18, 2010 at 8:09 AM, Virgil Stokes wrote: >>> ?I am considering the development of an all Python package (with numpy and >>> matplotlib) for the modeling and analysis of financial time series. >>> >>> This is a rather ambitious and could take sometime before I can have something >>> that is useful. ?Thus, any suggestions, pointers, etc. to related work would be >>> appreciated. > > I should have just asked you: ? What do you have in mind? > instead of writing my notes from the readings I just did into the email. > > I think any open source contributions in this area will be very useful > with the increased popularity of python in Finance. > > Josef > > >> >> Depends on what you want to do, but I would join or build on top of an >> existing package. >> >> I just got distracted with an extended internet search after finding >> >> http://github.com/quantmind/dynts >> >> (They use Redis as an in-memory and persistent storage. After reading >> up a bit, I think this might be useful if you have a web front end >> http://github.com/lsbardel/jflow in mind, but maybe not as good as >> hdf5 for desktop work. Just guessing since I used neither, and I >> always worry about installation problems on Windows.) >> They just started public development but all packages are in BSD from >> what I have seen. >> >> Otherwise, I would build on top of pandas, scikits.timeseries or larry >> or tabular if you want to handle your own time variable. >> >> For specific financial time series, e.g. stocks, exchange rates, >> options, I have seen only bits and pieces, or only partially >> implemented code (with a BSD compatible license), outside of quantlib >> and it's python bindings. >> >> Maybe someone knows more about what's available. >> >> For the econometrics/statistical analysis I haven't seen much outside >> of pandas and statsmodels in this area (besides isolated examples and >> recipes). I started to write on this in the statsmodels sandbox >> (including simulators). >> >> ?"modeling and analysis of financial time series" is a big project, >> and to get any results within a reasonable amount of time (unless you >> are part of a large team) is to specialize on some pieces. >> >> This is just my impression, since I thought of doing the same thing, >> but didn't see any way to get very far. >> >> (I just spend some weekends just to get the data from the Federal >> Reserve and wrap the API for the economics data base (Fred) of the >> Federal Reserve Saint Louis, the boring storage backend is zipped >> csv-files) >> >> Josef >> >>> >>> Thank you, >>> --V >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > I'm also interested as well what you might have in mind. For econometric models, the place to start (and perhaps contribute) would definitely be statsmodels. For other models it depends. Over the next year or so I plan to be working as often as I can on implementing time series models either inside statsmodels (with a pandas interface so you can carry around metadata) or inside pandas itself (which is not necessarily intended for financial applications, but that's what I've used it for). I'm interested in both standard econometric models (see e.g. L?tkepohl's 2006 time series book) and Bayesian time series models. - Wes From d.l.goldsmith at gmail.com Sun Sep 19 02:56:28 2010 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Sat, 18 Sep 2010 23:56:28 -0700 Subject: [Numpy-discussion] [OT: MATLAB] Any way to globally make Matlab struct attributes Python-property-like Message-ID: > Date: Tue, 14 Sep 2010 20:55:03 -0400 > From: Ken Watford > Subject: Re: [Numpy-discussion] [OT: MATLAB] Any way to globally make > Matlab struct attributes Python-property-like > To: Discussion of Numerical Python > On Mon, Sep 13, 2010 at 10:24 PM, David Goldsmith > wrote: >> I.e., I'd, at minimum, like to globally replace >> >> get(Handel, 'Property') >> >> with >> >> object.Property >> >> and >> >> set(Handel, 'Property', value) >> >> with >> >> object.Property = value >> >> to an arbitrary level of composition. > > Both structures and objects of both handle and value types work > exactly like you're asking for. > > You're probably talking about "handle graphics" objects (which are > just opaque numeric values representing things related to the UI), > which do still require the get/set functions. Yes, my mistake, sorry (I figured out the distinction a day or two after I posted). > They're just doubles, > after all. It should be feasible to write a class that wraps them for > ease of use, but that wouldn't be a topic for here. If that intrigues > you, try asking about it on StackOverflow. Of course, I'll go there if/when need be, thanks! (For now, I've managed to weasel my way into using Python for my next assignment--yippee!) DG -- In science it often happens that scientists say, 'You know that's a really good argument; my position is mistaken,' and then they would actually change their minds and you never hear that old view from them again. They really do it. It doesn't happen as often as it should, because scientists are human and change is sometimes painful. But it happens every day. I cannot recall the last time something like that happened in politics or religion. - Carl Sagan, 1987 CSICOP address From josef.pktd at gmail.com Sun Sep 19 10:28:41 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 19 Sep 2010 10:28:41 -0400 Subject: [Numpy-discussion] weighted mean; weighted standard error of the mean (sem) In-Reply-To: References: <14170_1284092042_1284092042_AANLkTi=PLFWR7FPrr_bHvNxt_jdDLMCrrw2WpYHFOgbw@mail.gmail.com> Message-ID: On Fri, Sep 17, 2010 at 1:19 PM, wrote: > On Fri, Sep 10, 2010 at 3:01 PM, ? wrote: >> On Fri, Sep 10, 2010 at 1:58 PM, Christopher Barrington-Leigh >> wrote: >>> Interesting. Thanks Erin, Josef and Keith. >> >> thanks to the stata page at least I figured out that WLS is aweights >> with asumption mu_i = mu >> >> import numpy as np >> from scikits.statsmodels import WLS >> w0 = np.arange(20) % 4 >> w = 1.*w0/w0.sum() >> y = 2 + np.random.randn(20) >> >>>>> res = WLS(y, np.ones(20), weights=w).fit() >>>>> print res.params, res.bse >> [ 2.29083069] [ 0.17562867] >>>>> m = np.dot(w, y) >>>>> m >> 2.2908306865128401 >>>>> s2u = 1/(nobs-1.) * np.dot(w, (y - m)**2) >>>>> s2u >> 0.030845429945278956 >>>>> np.sqrt(s2u) >> 0.17562867062435722 >> >> >>> >>> There is a nice article on this at >>> http://www.stata.com/support/faqs/stat/supweight.html. In my case, the >>> model I've in mind is to assume that the expected value (mean) is the same >>> for each sample, and that the weights are/should be normalised, whence a >>> consistent estimator for sem is straightforward (if second moments can >>> be assumed to be >>> well behaved?). I suspect that this (survey-like) case is also one of >>> the two most standard/most common >>> expression that people want when they ask for an s.e. of the mean for >>> a weighted dataset. The other would be when the weights are not to be >>> normalised, but represent standard errors on the individual >>> measurements. >>> >>> Surely what one wants, in the end, is a single function (or whatever) >>> called mean or sem which calculates different values for different >>> specified choices of model (assumptions)? And where possible that it has a >>> default model in mind for when none is specified? >> >> I find aweights and pweights still confusing, plus necessary auxillary >> assumptions. >> >> I don't find Stata docs very helpful, I almost never find a clear >> description of the formulas (and I don't have any Stata books). >> >> If you have or write some examples that show or apply in the different >> cases, then this would be very helpful to get a structure into this >> area, weighting and survey sampling, and population versus clustered >> or stratified sample statistics. >> >> I'm still pretty lost with the literature on surveys. > > I found the formula collection for SPSS > http://support.spss.com/productsext/statistics/documentation/19/clientindex.html#Manuals > pdf file for algorithms > > Not much explanation, and sometimes it's not really clear what a > variable stands for exactly, but a useful summary of formulas. Also > the formulas might not always be for a general case, e.g. formulas for > non-parametric tests seem to be missing tie-handling (from a quick > look). > > More compressed than the details descriptions in SAS, but much more > explicit than Stata and R without buying the books. > > The chapter on T Test Algorithm carries population frequencies > throughout, this should work for weighted statistics, but maybe not > for different complex sampling schemes (a-weights, p-weights,...). here is the corresponding SAS documentation http://support.sas.com/documentation/cdl/en/statug/63347/HTML/default/viewer.htm#ttest_toc.htm and here is my first draft of it in statsmodels, http://bazaar.launchpad.net/~josef-pktd/statsmodels/statsmodels-josef-experimental-gsoc/annotate/head%3A/scikits/statsmodels/sandbox/stats/weightstats.py A few hours of translating the SPSS formula collection into sci-python, still buggy, insufficiently tested and insufficiently documented, and it doesn't have the extras yet that SAS has. It uses my currently preferred pattern of lazily evaluated classes (although I might switch decorators). Josef > > Josef > > >> >> Josef >> >> >>> >>> thanks, >>> Chris >>> >>> On Thu, Sep 9, 2010 at 9:13 PM, Keith Goodman wrote: >>>> >>>> ma.std() >>>> >> ? 3.2548815339711115 >>>> > >>>> > or maybe `w` reflects an underlying sampling scheme and you should >>>> > sample in the bootstrap according to w ? >>>> >>>> Yes.... >>>> >>>> > if weighted average is a sum of linear functions of (normal) >>>> > distributed random variables, it still depends on whether the >>>> > individual observations have the same or different variances, e.g. >>>> > http://en.wikipedia.org/wiki/Weighted_mean#Statistical_properties >>>> >>>> ...lots of possibilities. As you have shown the problem is not yet >>>> well defined. Not much specification needed for the weighted mean, >>>> lots needed for the standard error of the weighted mean. >>>> >>>> > What I can't figure out is whether if you assume simga_i = sigma for >>>> > all observation i, do we use the weighted or the unweighted variance >>>> > to get an estimate of sigma. And I'm not able to replicate with simple >>>> > calculations what statsmodels.WLS gives me. >>>> >>>> My guess: if all you want is sigma of the individual i and you know >>>> sigma is the same for all i, then I suppose you don't care about the >>>> weight. >>>> >>>> > >>>> > ??? >>>> > >>>> > Josef >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> > From zachary.pincus at yale.edu Sun Sep 19 10:47:09 2010 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Sun, 19 Sep 2010 10:47:09 -0400 Subject: [Numpy-discussion] loadtxt stop In-Reply-To: <1A3A2705-4B90-4547-A58C-31B0D4EE3720@yale.edu> References: <853737.68306.qm@web27507.mail.ukl.yahoo.com> <4C93B8A1.30209@noaa.gov> <1A3A2705-4B90-4547-A58C-31B0D4EE3720@yale.edu> Message-ID: >> Though, really, it's annoying that numpy.loadtxt needs both the >> readline function *and* the iterator protocol. If it just used >> iterators, you could do: >> >> def truncator(fh, delimiter='END'): >> for line in fh: >> if line.strip() == delimiter: >> break >> yield line >> >> numpy.loadtxt(truncator(c)) >> >> Maybe I'll try to work up a patch for this. http://projects.scipy.org/numpy/ticket/1616 Zach > > > That seemed easy... worth applying? Won't break compatibility, because > the previous loadtxt required both fname.readline and fname.__iter__, > while this requires only the latter. > > > Index: numpy/lib/npyio.py > =================================================================== > --- numpy/lib/npyio.py (revision 8716) > +++ numpy/lib/npyio.py (working copy) > @@ -597,10 +597,11 @@ > fh = bz2.BZ2File(fname) > else: > fh = open(fname, 'U') > - elif hasattr(fname, 'readline'): > - fh = fname > else: > - raise ValueError('fname must be a string or file handle') > + try: > + fh = iter(fname) > + except: > + raise ValueError('fname must be a string or file handle') > X = [] > > def flatten_dtype(dt): > @@ -633,14 +634,18 @@ > > # Skip the first `skiprows` lines > for i in xrange(skiprows): > - fh.readline() > + try: > + fh.next() > + except StopIteration: > + raise IOError('End-of-file reached before > encountering data.') > > # Read until we find a line with some values, and use > # it to estimate the number of columns, N. > first_vals = None > while not first_vals: > - first_line = fh.readline() > - if not first_line: # EOF reached > + try: > + first_line = fh.next() > + except StopIteration: > raise IOError('End-of-file reached before > encountering data.') > first_vals = split_line(first_line) > N = len(usecols or first_vals) > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From gokhansever at gmail.com Sun Sep 19 13:19:48 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Sun, 19 Sep 2010 12:19:48 -0500 Subject: [Numpy-discussion] Question about masked arrays Message-ID: Hello, Consider these two sets of container arrays --one defined as usual np array the others as ma arrays: all_measured = np.ma.zeros((16, 18)) all_predicted = np.ma.zeros((16, 18)) all_measured2 = np.zeros((16, 18)) all_predicted2 = np.zeros((16, 18)) I do a computation within a for loop to assign 16 set of measurements into a length of 18 arrays (thus constructing a 2D array to perform overall statistics and plotting.) For the simplicity I only show a portion of all_measured and all_measured2 as: all_measured masked_array(data = [[512.632175 527.33373 565.36541 567.53967 593.86833 570.31319 574.40965 582.72649 588.21336 618.48789 593.09007 620.33474 591.10203 611.06443 655.60614 638.13193 626.71769 625.63584] [626.6435 -- -- 1183.67671 1206.82453 1183.13248 1162.5514 1180.70062 1086.53246 1078.78711 997.1642 856.57159 645.35167 696.86947 778.40914 816.03059 862.88297 901.7237] ... all_measured2 array([[ 512.632175 , 527.33373 , 565.36541 , 567.53967 , 593.86833 , 570.31319 , 574.40965 , 582.72649 , 588.21336 , 618.48789 , 593.09007 , 620.33474 , 591.10203 , 611.06443 , 655.60614 , 638.13193 , 626.71769 , 625.63584 ], [ 626.6435 , 0. , 0. , 1183.67671 , 1206.82453 , 1183.13248 , 1162.5514 , 1180.70062 , 1086.53246 , 1078.78711 , 997.1642 , 856.57159 , 645.35167 , 696.86947 , 778.40914 , 816.03059 , 862.88297 , 901.7237 ],... The issue is why masked arrays casted to regular numpy arrays as in all_measured2 case? whereas a simple numpy function np.mean and ma equivalent np.ma.mean yields same results on all_measured? Because the former requires a priori knowledge about the type of arrays, however the latter doesn't necessitate such restriction. Hope this is clear. Thanks. -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From joschu at caltech.edu Mon Sep 20 01:17:22 2010 From: joschu at caltech.edu (John Schulman) Date: Sun, 19 Sep 2010 22:17:22 -0700 Subject: [Numpy-discussion] Inverting argsort(a, axis=0) to obtain column-wise ranks In-Reply-To: References: Message-ID: Argsort twice and you get the rank. a1.argsort(axis=0).argsort(axis=0) That's because argsort is it's own inverse when applied to the ranks. On Tue, Sep 7, 2010 at 1:01 PM, Alexander Michael wrote: > Calculating ranks by inverting the results of an argsort is > straightforward and fast for 1D arrays: > > indices = argsort(a1) > ranks = zeros_like(indices) > ranks[indices] = arange(len(indices)) > > I was wondering if there was an equally pithy way to do this for > multiple data samples stored column-wise in a 2D array. That is, is > there a trick to invert the results of argsort(a2, axis=0) without > iterating (in python) over the columns? > > Thanks, > Alex > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From joschu at caltech.edu Mon Sep 20 01:19:13 2010 From: joschu at caltech.edu (John Schulman) Date: Sun, 19 Sep 2010 22:19:13 -0700 Subject: [Numpy-discussion] Inverting argsort(a, axis=0) to obtain column-wise ranks In-Reply-To: References: Message-ID: D'oh, Zachary already gave that answer. On Sun, Sep 19, 2010 at 10:17 PM, John Schulman wrote: > Argsort twice and you get the rank. > > a1.argsort(axis=0).argsort(axis=0) > > That's because argsort is it's own inverse when applied to the ranks. > > On Tue, Sep 7, 2010 at 1:01 PM, Alexander Michael wrote: >> Calculating ranks by inverting the results of an argsort is >> straightforward and fast for 1D arrays: >> >> indices = argsort(a1) >> ranks = zeros_like(indices) >> ranks[indices] = arange(len(indices)) >> >> I was wondering if there was an equally pithy way to do this for >> multiple data samples stored column-wise in a 2D array. That is, is >> there a trick to invert the results of argsort(a2, axis=0) without >> iterating (in python) over the columns? >> >> Thanks, >> Alex >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > From dpinte at enthought.com Mon Sep 20 07:46:54 2010 From: dpinte at enthought.com (Didrik Pinte) Date: Mon, 20 Sep 2010 13:46:54 +0200 Subject: [Numpy-discussion] Viewer for 2D Numpy arrays (GUI) In-Reply-To: References: Message-ID: On Fri, Sep 17, 2010 at 8:53 AM, Mayank P Jain wrote: > I thought about these options but what I need is excel like interface that > displays the values for each cell and one can modify and save the files. > > This would be convenient way of saving large files in less space and at the > same time, see them and would remove those additional steps of writing out > and reading them in each time my program runs. > > Regards > Mayank P Jain You can potentially use an Array traits using Traits and view it using the ArrayEditor or a TabularEditor. See Traits, TraitsUI and Chaco documentation for more details about that. -- Didrik From robert.kern at gmail.com Mon Sep 20 14:05:26 2010 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 20 Sep 2010 13:05:26 -0500 Subject: [Numpy-discussion] Question about masked arrays In-Reply-To: References: Message-ID: On Sun, Sep 19, 2010 at 12:19, G?khan Sever wrote: > Hello, > Consider these two sets of container arrays --one defined as usual np array > the others as ma arrays: > ?? ?all_measured = np.ma.zeros((16, 18)) > ?? ?all_predicted = np.ma.zeros((16, 18)) > ?? ?all_measured2 = np.zeros((16, 18)) > ?? ?all_predicted2 = np.zeros((16, 18)) > I do a computation within a for loop to assign 16 set of measurements into a > length of 18 arrays (thus constructing a 2D array to perform overall > statistics and plotting.) For the simplicity I only show a portion of > all_measured and all_measured2 as: > all_measured > masked_array(data = > ?[[512.632175 527.33373 565.36541 567.53967 593.86833 570.31319 574.40965 > ??582.72649 588.21336 618.48789 593.09007 620.33474 591.10203 611.06443 > ??655.60614 638.13193 626.71769 625.63584] > ?[626.6435 -- -- 1183.67671 1206.82453 1183.13248 1162.5514 1180.70062 > ??1086.53246 1078.78711 997.1642 856.57159 645.35167 696.86947 778.40914 > ??816.03059 862.88297 901.7237] ... > all_measured2 > array([[ ?512.632175 ?, ? 527.33373 ? , ? 565.36541 ? , ? 567.53967 ? , > ?? ? ? ? ?593.86833 ? , ? 570.31319 ? , ? 574.40965 ? , ? 582.72649 ? , > ?? ? ? ? ?588.21336 ? , ? 618.48789 ? , ? 593.09007 ? , ? 620.33474 ? , > ?? ? ? ? ?591.10203 ? , ? 611.06443 ? , ? 655.60614 ? , ? 638.13193 ? , > ?? ? ? ? ?626.71769 ? , ? 625.63584 ? ], > ?? ? ? [ ?626.6435 ? ?, ? ? 0. ? ? ? ?, ? ? 0. ? ? ? ?, ?1183.67671 ? , > ?? ? ? ? 1206.82453 ? , ?1183.13248 ? , ?1162.5514 ? ?, ?1180.70062 ? , > ?? ? ? ? 1086.53246 ? , ?1078.78711 ? , ? 997.1642 ? ?, ? 856.57159 ? , > ?? ? ? ? ?645.35167 ? , ? 696.86947 ? , ? 778.40914 ? , ? 816.03059 ? , > ?? ? ? ? ?862.88297 ? , ? 901.7237 ? ?],... > The issue is why masked arrays casted to regular numpy arrays as in > all_measured2 case? whereas a simple numpy function np.mean and ma > equivalent np.ma.mean yields same results on all_measured? Because the > former requires?a priori?knowledge about the type of arrays, however the > latter doesn't?necessitate?such restriction. > Hope this is clear. Thanks. Are you asking about when masked arrays are casted to ndarrays (and thus losing the mask information)? Most times when a function uses asarray() or array() to explicitly cast the inputs to an ndarray. The reason that np.mean() gives the same result as np.ma.mean() is that it simply defers to the .mean() method on the object, so it works as expected on a masked array. Many other functions will not. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From gokhansever at gmail.com Mon Sep 20 16:23:34 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Mon, 20 Sep 2010 15:23:34 -0500 Subject: [Numpy-discussion] Question about masked arrays In-Reply-To: References: Message-ID: On Mon, Sep 20, 2010 at 1:05 PM, Robert Kern wrote: > Are you asking about when masked arrays are casted to ndarrays (and > thus losing the mask information)? Most times when a function uses > asarray() or array() to explicitly cast the inputs to an ndarray. The > reason that np.mean() gives the same result as np.ma.mean() is that it > simply defers to the .mean() method on the object, so it works as > expected on a masked array. Many other functions will not. > > -- > Robert Kern > Right guess. It is important for me to able to preserve masked array properties of an array. Otherwise losing the mask information yields unexpected results in some of my calculations. I could see from np.mean?? that mean function is indeed the object method. Also in /numpy/ma there is a conversion for np.zeros(). I guess in any case it is the user's responsibility to make sure that the operations are performed on a desired array type. -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmay31 at gmail.com Mon Sep 20 16:34:23 2010 From: rmay31 at gmail.com (Ryan May) Date: Mon, 20 Sep 2010 15:34:23 -0500 Subject: [Numpy-discussion] Question about masked arrays In-Reply-To: References: Message-ID: On Mon, Sep 20, 2010 at 3:23 PM, G?khan Sever wrote: > On Mon, Sep 20, 2010 at 1:05 PM, Robert Kern wrote: >> >> Are you asking about when masked arrays are casted to ndarrays (and >> thus losing the mask information)? Most times when a function uses >> asarray() or array() to explicitly cast the inputs to an ndarray. The >> reason that np.mean() gives the same result as np.ma.mean() is that it >> simply defers to the .mean() method on the object, so it works as >> expected on a masked array. Many other functions will not. >> >> -- >> Robert Kern > > Right guess. It is important for me to able to preserve masked array > properties of an array. Otherwise losing the mask information yields > unexpected results in some of my calculations. I could see from np.mean?? > that mean function is indeed the object method. Also in /numpy/ma there is a > conversion?for np.zeros(). I guess in any case it is the user's > responsibility to make sure that the operations are performed on a desired > array type. True, but in some cases the functions just blindly call asarray() or array() without thinking about using asanyarray(). If you encounter a basic numpy function that calls asarray() but would work fine with masked arrays (or other subclasses), feel free to file/post as a bug. It's good to get those cases fixed where possible. (I've done this in the past.) Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma From ben.root at ou.edu Mon Sep 20 16:34:48 2010 From: ben.root at ou.edu (Benjamin Root) Date: Mon, 20 Sep 2010 15:34:48 -0500 Subject: [Numpy-discussion] Question about masked arrays In-Reply-To: References: Message-ID: On Mon, Sep 20, 2010 at 3:23 PM, G?khan Sever wrote: > On Mon, Sep 20, 2010 at 1:05 PM, Robert Kern wrote: > >> Are you asking about when masked arrays are casted to ndarrays (and >> thus losing the mask information)? Most times when a function uses >> asarray() or array() to explicitly cast the inputs to an ndarray. The >> reason that np.mean() gives the same result as np.ma.mean() is that it >> simply defers to the .mean() method on the object, so it works as >> expected on a masked array. Many other functions will not. >> >> -- >> Robert Kern >> > > Right guess. It is important for me to able to preserve masked array > properties of an array. Otherwise losing the mask information yields > unexpected results in some of my calculations. I could see from np.mean?? > that mean function is indeed the object method. Also in /numpy/ma there is a > conversion for np.zeros(). I guess in any case it is the user's > responsibility to make sure that the operations are performed on a desired > array type. > > > Gokhan, I have been using masked arrays quite extensively. My take on them is that if a masked array makes sense in that operation, then they should still work with the regular functions. However, there have been many cases where a developer used np.asarray() instead of np.asanyarray() for their code, which causes the masked array object to lose the mask. If you encounter such situations, it is usually a bug and should be reported. Actually, that reminds me... watch out for np.polyfit() with masked arrays. It doesn't behave quite nicely with masked arrays and the results are deceptive. It may appear to be right, but it is not. Use np.ma.polyfit(). Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokhansever at gmail.com Mon Sep 20 17:03:10 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Mon, 20 Sep 2010 16:03:10 -0500 Subject: [Numpy-discussion] Question about masked arrays In-Reply-To: References: Message-ID: On Mon, Sep 20, 2010 at 3:34 PM, Benjamin Root wrote: > I have been using masked arrays quite extensively. My take on them is that > if a masked array makes sense in that operation, then they should still work > with the regular functions. However, there have been many cases where a > developer used np.asarray() instead of np.asanyarray() for their code, which > causes the masked array object to lose the mask. If you encounter such > situations, it is usually a bug and should be reported. > Do you think then in my case assigning the masked array to np.zeros and getting a non-masked return is a bug or I should make sure that I am providing a masked array equivalent zeros function before I start doing computation? And also if the inner execution could be clarified by asanyarray assertion why there is ma equivalent array operation functions? > > Actually, that reminds me... watch out for np.polyfit() with masked > arrays. It doesn't behave quite nicely with masked arrays and the results > are deceptive. It may appear to be right, but it is not. Use > np.ma.polyfit(). > Hah, you should see my scipy.optimize.leastsq in action. I estimate geometric mean and standard deviation parameters for log-normal aerosol population. It runs, with the values masked and unmasked (from above np.zeros and np.ma.zeros difference) but resulting all kind interesting estimates. > > Ben Root > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Mon Sep 20 17:09:52 2010 From: ben.root at ou.edu (Benjamin Root) Date: Mon, 20 Sep 2010 16:09:52 -0500 Subject: [Numpy-discussion] Question about masked arrays In-Reply-To: References: Message-ID: On Mon, Sep 20, 2010 at 4:03 PM, G?khan Sever wrote: > On Mon, Sep 20, 2010 at 3:34 PM, Benjamin Root wrote: > >> I have been using masked arrays quite extensively. My take on them is >> that if a masked array makes sense in that operation, then they should still >> work with the regular functions. However, there have been many cases where >> a developer used np.asarray() instead of np.asanyarray() for their code, >> which causes the masked array object to lose the mask. If you encounter >> such situations, it is usually a bug and should be reported. >> > > Do you think then in my case assigning the masked array to np.zeros and > getting a non-masked return is a bug or I should make sure that I am > providing a masked array equivalent zeros function before I start > doing computation? > > If you have any variable and you assign np.zeros() to it, then whatever was there previously is replaced. If you want to create a masked array, then use np.ma.zeros(), or take an array and put it through np.ma.masked_array() with the mask keyword to create the mask at the same time. Note that this approach supposedly uses the original array, so you aren't paying much of a penalty here. > And also if the inner execution could be clarified by asanyarray assertion > why there is ma equivalent array operation functions? > > That is a design question for the numpy gods... > >> Actually, that reminds me... watch out for np.polyfit() with masked >> arrays. It doesn't behave quite nicely with masked arrays and the results >> are deceptive. It may appear to be right, but it is not. Use >> np.ma.polyfit(). >> > > Hah, you should see my scipy.optimize.leastsq in action. I estimate > geometric mean and standard deviation parameters for log-normal aerosol > population. It runs, with the values masked and unmasked (from above > np.zeros and np.ma.zeros difference) but resulting all kind interesting > estimates. > > I wonder if the functions were originally intended to work with masked arrays, but at some point, some inner part of the algorithm became 'ma-unfriendly'... Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Mon Sep 20 17:17:57 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 20 Sep 2010 23:17:57 +0200 Subject: [Numpy-discussion] Question about masked arrays In-Reply-To: References: Message-ID: <14110F0D-B473-45B5-AF78-1296AEF1EE7A@gmail.com> On Sep 20, 2010, at 11:09 PM, Benjamin Root wrote: > > And also if the inner execution could be clarified by asanyarray assertion why there is ma equivalent array operation functions? > > That is a design question for the numpy gods... Well, asanyarray is not always a panacea, and can lead to problems with other subclasses of ndarray. I'd think twice before bluntly adapting a function to use np.asanyarray... Better explicit than implicit. A relatively simple way to take care of the issue is to improve the documentation, for example by adding a note pointing ti the corresponding np.ma function. > I wonder if the functions were originally intended to work with masked arrays, but at some point, some inner part of the algorithm became 'ma-unfriendly'... I don't think so. Anyway, as usual, please open a ticket when you run into a problem with masked arrays and I'll try to take care of it ASAP. From hagen at zhuliguan.net Mon Sep 20 17:34:58 2010 From: hagen at zhuliguan.net (=?UTF-8?B?SGFnZW4gRsO8cnN0ZW5hdQ==?=) Date: Mon, 20 Sep 2010 23:34:58 +0200 Subject: [Numpy-discussion] sum of outer products Message-ID: Hi, I don't know if I'm overlooking something obvious, but is there a compact way of computing the 3-array X_{ijk} = \sum_{l} A_{il}*B_{jl}*C_{kl} out of the 2-arrays A, B, and C? - Hagen From pav at iki.fi Mon Sep 20 17:49:23 2010 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 20 Sep 2010 21:49:23 +0000 (UTC) Subject: [Numpy-discussion] sum of outer products References: Message-ID: Mon, 20 Sep 2010 23:34:58 +0200, Hagen F?rstenau wrote: > I don't know if I'm overlooking something obvious, but is there a > compact way of computing the 3-array > > X_{ijk} = \sum_{l} A_{il}*B_{jl}*C_{kl} > > out of the 2-arrays A, B, and C? (A[:,newaxis,newaxis]*B[newaxis,:,newaxis]*C[newaxis,newaxis,:]).sum(axis=-1) From hagen at zhuliguan.net Mon Sep 20 18:13:46 2010 From: hagen at zhuliguan.net (=?UTF-8?B?SGFnZW4gRsO8cnN0ZW5hdQ==?=) Date: Tue, 21 Sep 2010 00:13:46 +0200 Subject: [Numpy-discussion] sum of outer products In-Reply-To: References: Message-ID: >> X_{ijk} = \sum_{l} A_{il}*B_{jl}*C_{kl} > > (A[:,newaxis,newaxis]*B[newaxis,:,newaxis]*C[newaxis,newaxis,:]).sum(axis=-1) Thanks for the quick solution and practical exercise in broadcasting! :-) However, this creates a temporary 4-array, right? Is there a way of avoiding this memory requirement? - Hagen From mpan at cesga.es Tue Sep 21 02:33:35 2010 From: mpan at cesga.es (Miguel Pan Fidalgo) Date: Tue, 21 Sep 2010 08:33:35 +0200 Subject: [Numpy-discussion] compile process of numpy-1.5.0 fails in Itanium cluster with icc Message-ID: <201009210833.35419.mpan@cesga.es> hi, I'm trying to install numpy-1.5.0 in our Itanium (IA-64) supercomputer [http://www.cesga.es/content/view/917/115/lang,en/ ]. This is my site.cfg (I have to install umfpack but I think this is not the problem): $ cat site.cfg [DEFAULT] library_dirs = /opt/cesga/fftw-3.2.2/lib:/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64 include_dirs = /opt/cesga/fftw-3.2.2/include:/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include #[umfpack] #umfpack_libs = umfpack [fftw] libraries = fftw3 [mkl] library_dirs = /opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64 lapack_libs = mkl_lapack mkl_libs = mkl_core The only hacks that I made: $ diff numpy/distutils/system_info.py ../numpy- scipy/numpy-1.5.0/numpy/distutils/system_info.py 818c818 < _lib_mkl = ['mkl','mkl_lapack','guide'] --- > _lib_mkl = ['mkl','vml','guide'] $ diff numpy/distutils/intelccompiler.py ../numpy- scipy/numpy-1.5.0/numpy/distutils/intelccompiler.py 11c11 < cc_exe = 'icc -O3 -openmp -lpthread -fPIC' --- > cc_exe = 'icc' And I have used this script to launch the compilation process: $ cat compilar.sh #!/bin/bash module load nose/0.10.4 fftw/3.2.2 ifort/11.1.056 icc/11.1.056 mkl/10.2.2 python setup.py config --compiler=intele --fcompiler=intele build install -- prefix=/home/cesga/mpan/numpy-1.5.0 exit 0 blas and lapack are detected properly: $ head -n 40 compilar.log Running from numpy source directory.F2PY Version 1 blas_opt_info: blas_mkl_info: FOUND: libraries = ['mkl_core', 'pthread'] library_dirs = ['/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/cesga/fftw-3.2.2/include', '/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include'] FOUND: libraries = ['mkl_core', 'pthread'] library_dirs = ['/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/cesga/fftw-3.2.2/include', '/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include'] lapack_opt_info: lapack_mkl_info: mkl_info: FOUND: libraries = ['mkl_core', 'pthread'] library_dirs = ['/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/cesga/fftw-3.2.2/include', '/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include'] FOUND: libraries = ['mkl_lapack', 'mkl_core', 'pthread'] library_dirs = ['/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/cesga/fftw-3.2.2/include', '/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include'] FOUND: libraries = ['mkl_lapack', 'mkl_core', 'pthread'] library_dirs = ['/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/cesga/fftw-3.2.2/include', '/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include'] running config running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options but, unfortunately, the compilation process ends with an unexpected realocation problem: $ tail -n 11 compilar.log /opt/cesga/intel/intel11.0/Compiler/11.1/056/bin/ia64/icc -shared build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o -Lbuild/temp.linux-ia64-2.4 -o build/lib.linux-ia64-2.4/numpy/core/umath_tests.so ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol inner1d_signature ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol innerwt_signature ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol matrix_multiply_signature ld: final link failed: Nonrepresentable section on output ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol inner1d_signature ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol innerwt_signature ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol matrix_multiply_signature ld: final link failed: Nonrepresentable section on output error: Command "/opt/cesga/intel/intel11.0/Compiler/11.1/056/bin/ia64/icc - shared build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o -Lbuild/temp.linux-ia64-2.4 -o build/lib.linux-ia64-2.4/numpy/core/umath_tests.so" failed with exit status 1 I have tried to link this files manually but I haven't any success: $ /opt/cesga/intel/intel11.0/Compiler/11.1/056/bin/ia64/icc -shared build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o -Lbuild/temp.linux-ia64-2.4 -o build/lib.linux-ia64-2.4/numpy/core/umath_tests.so ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol inner1d_signature ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol innerwt_signature ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol matrix_multiply_signature ld: final link failed: Nonrepresentable section on output $ /opt/cesga/intel/intel11.0/Compiler/11.1/056/bin/ia64/icc -fPIC -shared build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o -Lbuild/temp.linux-ia64-2.4 -o build/lib.linux-ia64-2.4/numpy/core/umath_tests.so ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol inner1d_signature ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol innerwt_signature ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol matrix_multiply_signature ld: final link failed: Nonrepresentable section on output $ /opt/cesga/intel/intel11.0/Compiler/11.1/056/bin/ia64/icc -i-dynamic -shared build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o -Lbuild/temp.linux-ia64-2.4 -o build/lib.linux-ia64-2.4/numpy/core/umath_tests.so ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol inner1d_signature ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol innerwt_signature ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol matrix_multiply_signature ld: final link failed: Nonrepresentable section on output with differents combinations (MKL 9.1, no optimatazions flags, no hacks.....) I have the same problem. any idea? -- Atentamente, Miguel Pan Fidalgo (Applications Technician) mail: mpan at cesga.es web: http://www.cesga.es Avda. de Vigo s/n 15705, Santiago de Compostela Telf.: +34 981 569810 - Fax: 981 594616 From pschmidtke at mmb.pcb.ub.es Tue Sep 21 02:55:35 2010 From: pschmidtke at mmb.pcb.ub.es (Peter Schmidtke) Date: Tue, 21 Sep 2010 08:55:35 +0200 Subject: [Numpy-discussion] unique 2d arrays Message-ID: <4C9856E7.6050206@mmb.pcb.ub.es> Dear all, I'd like to know if there is a pythonic / numpy way of retrieving unique lines of a 2d numpy array. In a way I have this : [[409 152] [409 152] [409 152] [409 152] [409 152] [409 152] [409 152] [409 152] [409 152] [409 152] [409 152] [426 193] [431 129]] And I'd like to get this : [[409 152] [426 193] [431 129]] How can I do this without workarounds like string concatenation or such things? Numpy.unique flattens the whole array so it's not really of use here. Cheers. -- Peter Schmidtke PhD Student Dept. Physical Chemistry Faculty of Pharmacy University of Barcelona From millman at berkeley.edu Tue Sep 21 03:13:45 2010 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 21 Sep 2010 00:13:45 -0700 Subject: [Numpy-discussion] [ANN] SciPy India 2010 Call for Presentations Message-ID: ========================== SciPy 2010 Call for Papers ========================== The second `SciPy India Conference `_ will be held from December 13th to 18th, 2010 at `IIIT-Hyderabad `_. At this conference, novel applications and breakthroughs made in the pursuit of science using Python are presented. Attended by leading figures from both academia and industry, it is an excellent opportunity to experience the cutting edge of scientific software development. The conference is followed by two days of tutorials and a code sprint, during which community experts provide training on several scientific Python packages. We invite you to take part by submitting a talk abstract on the conference website at: http://scipy.in Talk/Paper Submission ========================== We solicit talks and accompanying papers (either formal academic or magazine-style articles) that discuss topics regarding scientific computing using Python, including applications, teaching, development and research. Papers are included in the peer-reviewed conference proceedings, published online. Please note that submissions primarily aimed at the promotion of a commercial product or service will not be considered. Important Dates ========================== Monday, Oct. 11: Abstracts Due Saturday, Oct. 30: Schedule announced Tuesday, Nov. 30: Proceedings paper submission due Monday-Tuesday, Dec. 13-14: Conference Wednesday-Friday, Dec. 15-17: Tutorials/Sprints Saturday, Dec. 18: Sprints Organizers ========================== * Jarrod Millman, Neuroscience Institute, UC Berkeley, USA (Conference Co-Chair) * Prabhu Ramachandran, Department of Aerospace Engineering, IIT Bombay, India (Conference Co-Chair) * FOSSEE Team From a.h.jaffe at gmail.com Tue Sep 21 08:25:19 2010 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Tue, 21 Sep 2010 13:25:19 +0100 Subject: [Numpy-discussion] read ascii file from complex fortran format() -- genfromtxt Message-ID: <4C98A42F.7020201@gmail.com> Hi all, I've got an ascii file with a relatively complicated structure, originally written by fortran with the format: 135 format(a12,1x,2(f10.5,1x),i3,1x,4(f9.3,1x),4(i2,1x),3x, 1 16(f7.2,1x),i3,3x,f13.5,1x,f10.5,1x,f10.6,1x,i3,1x, 2 4(f10.6,1x), 2 i2,1x,f5.2,1x,f10.3,1x,i3,1x,f7.2,1x,f7.2,3x,4(f7.4,1x), 3 4(f7.2,1x),3x,f7.2,1x,i4,3x,f10.3,1x,14(f6.2,1x),i3,1x, 1 3x,2f10.5,8f11.2,2f10.5,f12.3,3x, 4 2(a6,1x),a23,1x,a22,1x,a22) Note, in particular, that many of the strings contain white space. Is there a relatively straightforward way to translate this into dtype (and delimiter?) arguments for use with genfromtxt or do I just have to do it by hand? Andrew From josef.pktd at gmail.com Tue Sep 21 10:57:08 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 21 Sep 2010 10:57:08 -0400 Subject: [Numpy-discussion] unique 2d arrays In-Reply-To: <4C9856E7.6050206@mmb.pcb.ub.es> References: <4C9856E7.6050206@mmb.pcb.ub.es> Message-ID: On Tue, Sep 21, 2010 at 2:55 AM, Peter Schmidtke wrote: > Dear all, > > I'd like to know if there is a pythonic / numpy way of retrieving unique > lines of a 2d numpy array. > > In a way I have this : > > [[409 152] > ?[409 152] > ?[409 152] > ?[409 152] > ?[409 152] > ?[409 152] > ?[409 152] > ?[409 152] > ?[409 152] > ?[409 152] > ?[409 152] > ?[426 193] > ?[431 129]] > > And I'd like to get this : > > [[409 152] > ?[426 193] > ?[431 129]] > > > How can I do this without workarounds like string concatenation or such > things? Numpy.unique flattens the whole array so it's not really of use > here. One possibility see thread at http://mail.scipy.org/pipermail/numpy-discussion/2009-August/044664.html Josef > > Cheers. > > -- Peter Schmidtke > > PhD Student > Dept. Physical Chemistry > Faculty of Pharmacy > University of Barcelona > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ralf.gommers at googlemail.com Tue Sep 21 10:57:38 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 21 Sep 2010 22:57:38 +0800 Subject: [Numpy-discussion] help from OS X 10.5 users wanted Message-ID: Hi, There have been many reports from OS X 10.5(.8) users that the provided binaries on Sourceforge for numpy versions >= 1.4.0 do not work, see http://projects.scipy.org/numpy/ticket/1322. If you're an OS X 10.5 user I would like your help to test if the same issue occurs for the following binary: http://sourceforge.net/projects/numpy/files/Temporary/numpy-2.0.0.dev-py2.6-osx10.5-python.org.dmg/download Please comment on the above ticket. Thanks, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From pschmidtke at mmb.pcb.ub.es Tue Sep 21 11:23:41 2010 From: pschmidtke at mmb.pcb.ub.es (Peter Schmidtke) Date: Tue, 21 Sep 2010 17:23:41 +0200 Subject: [Numpy-discussion] unique 2d arrays In-Reply-To: References: <4C9856E7.6050206@mmb.pcb.ub.es> Message-ID: <4C98CDFD.3090502@mmb.pcb.ub.es> Hey Josef, I didn't stumble upon these posts. Thanks for the hint...it doesn't look very pythonic or matlab like still. This would be a nice thing to have a unique function that is able to take an axis argument. Cheers. Peter josef.pktd at gmail.com wrote: > On Tue, Sep 21, 2010 at 2:55 AM, Peter Schmidtke > wrote: > >> Dear all, >> >> I'd like to know if there is a pythonic / numpy way of retrieving unique >> lines of a 2d numpy array. >> >> In a way I have this : >> >> [[409 152] >> [409 152] >> [409 152] >> [409 152] >> [409 152] >> [409 152] >> [409 152] >> [409 152] >> [409 152] >> [409 152] >> [409 152] >> [426 193] >> [431 129]] >> >> And I'd like to get this : >> >> [[409 152] >> [426 193] >> [431 129]] >> >> >> How can I do this without workarounds like string concatenation or such >> things? Numpy.unique flattens the whole array so it's not really of use >> here. >> > > One possibility see thread at > http://mail.scipy.org/pipermail/numpy-discussion/2009-August/044664.html > > Josef > > > > >> Cheers. >> >> -- Peter Schmidtke >> >> PhD Student >> Dept. Physical Chemistry >> Faculty of Pharmacy >> University of Barcelona >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Peter Schmidtke PhD Student Dept. Physical Chemistry Faculty of Pharmacy University of Barcelona From gokhansever at gmail.com Tue Sep 21 13:29:03 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Tue, 21 Sep 2010 12:29:03 -0500 Subject: [Numpy-discussion] unique 2d arrays In-Reply-To: <4C9856E7.6050206@mmb.pcb.ub.es> References: <4C9856E7.6050206@mmb.pcb.ub.es> Message-ID: On Tue, Sep 21, 2010 at 1:55 AM, Peter Schmidtke wrote: > Dear all, > > I'd like to know if there is a pythonic / numpy way of retrieving unique > lines of a 2d numpy array. > > In a way I have this : > > [[409 152] > [409 152] > [409 152] > [409 152] > [409 152] > [409 152] > [409 152] > [409 152] > [409 152] > [409 152] > [409 152] > [426 193] > [431 129]] > > And I'd like to get this : > > [[409 152] > [426 193] > [431 129]] > > > How can I do this without workarounds like string concatenation or such > things? Numpy.unique flattens the whole array so it's not really of use > here. > Here is one alternative: I[15]: a = np.array([[409, 152], [409, 152], [426, 193], [431, 129]]) I[16]: np.array(list(set(tuple(i) for i in a.tolist()))) O[16]: array([[409, 152], [426, 193], [431, 129]]) I[6]: %timeit np.unique(a.view([('',a.dtype)]*a.shape[1])).view(a.dtype).reshape(-1,a.shape[1]) 10000 loops, best of 3: 51 us per loop I[8]: %timeit np.array(list(set(tuple(i) for i in a.tolist()))) 10000 loops, best of 3: 31.4 us per loop # Try with a bigger array I[9]: k = np.array((a.tolist()*50000)) I[10]: %timeit np.array(list(set(tuple(i) for i in k.tolist()))) 1 loops, best of 3: 324 ms per loop I[11]: %timeit np.unique(k.view([('',k.dtype)]*k.shape[1])).view(k.dtype).reshape(-1,k.shape[1]) 1 loops, best of 3: 790 ms per loop Seems like faster on these tests comparing to the unique method. Also it is more readable. Still not uber Pythonic. Haskell has "nub" to remove duplicate list elements. http://www.haskell.org/ghc/docs/6.12.2/html/libraries/base-4.2.0.1/Data-List.html#v%3Anub -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.huard at gmail.com Tue Sep 21 13:29:09 2010 From: david.huard at gmail.com (David Huard) Date: Tue, 21 Sep 2010 13:29:09 -0400 Subject: [Numpy-discussion] read ascii file from complex fortran format() -- genfromtxt In-Reply-To: <4C98A42F.7020201@gmail.com> References: <4C98A42F.7020201@gmail.com> Message-ID: Have you tried http://code.google.com/p/python-fortranformat/ It's not officially released yet but it's probably worth a try. David H. On Tue, Sep 21, 2010 at 8:25 AM, Andrew Jaffe wrote: > Hi all, > > I've got an ascii file with a relatively complicated structure, > originally written by fortran with the format: > > 135 format(a12,1x,2(f10.5,1x),i3,1x,4(f9.3,1x),4(i2,1x),3x, > 1 16(f7.2,1x),i3,3x,f13.5,1x,f10.5,1x,f10.6,1x,i3,1x, > 2 4(f10.6,1x), > 2 i2,1x,f5.2,1x,f10.3,1x,i3,1x,f7.2,1x,f7.2,3x,4(f7.4,1x), > 3 4(f7.2,1x),3x,f7.2,1x,i4,3x,f10.3,1x,14(f6.2,1x),i3,1x, > 1 3x,2f10.5,8f11.2,2f10.5,f12.3,3x, > 4 2(a6,1x),a23,1x,a22,1x,a22) > > Note, in particular, that many of the strings contain white space. > > Is there a relatively straightforward way to translate this into dtype > (and delimiter?) arguments for use with genfromtxt or do I just have to > do it by hand? > > Andrew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Sep 21 13:43:53 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 21 Sep 2010 13:43:53 -0400 Subject: [Numpy-discussion] unique 2d arrays In-Reply-To: References: <4C9856E7.6050206@mmb.pcb.ub.es> Message-ID: On Tue, Sep 21, 2010 at 1:29 PM, G?khan Sever wrote: > > > On Tue, Sep 21, 2010 at 1:55 AM, Peter Schmidtke > wrote: >> >> Dear all, >> >> I'd like to know if there is a pythonic / numpy way of retrieving unique >> lines of a 2d numpy array. >> >> In a way I have this : >> >> [[409 152] >> ?[409 152] >> ?[409 152] >> ?[409 152] >> ?[409 152] >> ?[409 152] >> ?[409 152] >> ?[409 152] >> ?[409 152] >> ?[409 152] >> ?[409 152] >> ?[426 193] >> ?[431 129]] >> >> And I'd like to get this : >> >> [[409 152] >> ?[426 193] >> ?[431 129]] >> >> >> How can I do this without workarounds like string concatenation or such >> things? Numpy.unique flattens the whole array so it's not really of use >> here. > > Here is one alternative: > I[15]: a = np.array([[409, 152], [409, 152], [426, 193], [431, 129]]) > I[16]: np.array(list(set(tuple(i) for i in a.tolist()))) > O[16]: > array([[409, 152], > ?? ? ? [426, 193], > ?? ? ? [431, 129]]) > I[6]: %timeit > np.unique(a.view([('',a.dtype)]*a.shape[1])).view(a.dtype).reshape(-1,a.shape[1]) > 10000 loops, best of 3: 51 us per loop > I[8]: %timeit np.array(list(set(tuple(i) for i in a.tolist()))) > 10000 loops, best of 3: 31.4 us per loop > # Try with a bigger array > I[9]: k = np.array((a.tolist()*50000)) > I[10]: %timeit np.array(list(set(tuple(i) for i in k.tolist()))) > 1 loops, best of 3: 324 ms per loop > I[11]: %timeit > np.unique(k.view([('',k.dtype)]*k.shape[1])).view(k.dtype).reshape(-1,k.shape[1]) > 1 loops, best of 3: 790 ms per loop I'm a bit surprised, I think np.unique does some extra work to maintain the order. The tolist() might not be necessary if you iterate over rows. Josef > > Seems like faster on these tests comparing to the unique method. Also it is > more readable. Still not uber Pythonic. Haskell has "nub" to remove > duplicate list > elements.?http://www.haskell.org/ghc/docs/6.12.2/html/libraries/base-4.2.0.1/Data-List.html#v%3Anub > -- > G?khan > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From gokhansever at gmail.com Tue Sep 21 13:55:12 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Tue, 21 Sep 2010 12:55:12 -0500 Subject: [Numpy-discussion] unique 2d arrays In-Reply-To: References: <4C9856E7.6050206@mmb.pcb.ub.es> Message-ID: On Tue, Sep 21, 2010 at 12:43 PM, wrote: > I'm a bit surprised, I think np.unique does some extra work to > maintain the order. > The tolist() might not be necessary if you iterate over rows. > Testing again with a smaller k array and more repeats I[25]: k = np.array((a.tolist()*5000)) I[27]: %timeit -r 100 np.array(list(set(tuple(i) for i in k.tolist()))) 10 loops, best of 100: 31.3 ms per loop I[28]: %timeit -r 100 np.array(list(set(tuple(i) for i in k))) 10 loops, best of 100: 55.4 ms per loop I[30]: %timeit -r 100 np.unique(k.view([('',k.dtype)]*k.shape[1])).view(k.dtype).reshape(-1,k.shape[1]) 10 loops, best of 100: 60.5 ms per loop .tolist version is faster. Can you also verify this? -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From eadrogue at gmx.net Tue Sep 21 14:35:00 2010 From: eadrogue at gmx.net (Ernest =?iso-8859-1?Q?Adrogu=E9?=) Date: Tue, 21 Sep 2010 20:35:00 +0200 Subject: [Numpy-discussion] unique 2d arrays In-Reply-To: References: <4C9856E7.6050206@mmb.pcb.ub.es> Message-ID: <20100921183500.GA1964@doriath.local> 21/09/10 @ 12:55 (-0500), thus spake G?khan Sever: > On Tue, Sep 21, 2010 at 12:43 PM, wrote: > > > I'm a bit surprised, I think np.unique does some extra work to > > maintain the order. > > The tolist() might not be necessary if you iterate over rows. > > > > Testing again with a smaller k array and more repeats > > I[25]: k = np.array((a.tolist()*5000)) > > I[27]: %timeit -r 100 np.array(list(set(tuple(i) for i in k.tolist()))) > 10 loops, best of 100: 31.3 ms per loop > > I[28]: %timeit -r 100 np.array(list(set(tuple(i) for i in k))) > 10 loops, best of 100: 55.4 ms per loop > > I[30]: %timeit -r 100 > np.unique(k.view([('',k.dtype)]*k.shape[1])).view(k.dtype).reshape(-1,k.shape[1]) > 10 loops, best of 100: 60.5 ms per loop > > .tolist version is faster. Can you also verify this? I get the same results: In [14]: x=np.random.poisson(1.3, size=100000).reshape(-1,2) In [19]: %timeit np.array(tuple(set(map(tuple, x.tolist())))) 10 loops, best of 3: 86.5 ms per loop In [20]: %timeit np.array(tuple(set(map(tuple, x)))) 10 loops, best of 3: 125 ms per loop Bye. Ernest From kbasye1 at jhu.edu Tue Sep 21 15:48:30 2010 From: kbasye1 at jhu.edu (Ken Basye) Date: Tue, 21 Sep 2010 15:48:30 -0400 Subject: [Numpy-discussion] A proposed change to rollaxis() behavior for negative 'start' values Message-ID: <4C990C0E.5090904@jhu.edu> Hi Numpy Folks, A while back, I filed this ticket: http://projects.scipy.org/numpy/ticket/1441 suggesting a change to rollaxis() and some fixes to the doc and error reporting. Ralf Gommers suggested I float the behavior change here, so that's what I'm doing. The motivation for the change comes because it seems like there should be a simpler way to get some axis into the last position than to do this: >>> a = np.ones((3,4,5,6)) >>> b = np.rollaxis(a, axis=0, start=len(a.shape)) >>> b.shape (4, 5, 6, 3) But currently it seems there isn't because when you specify -1 as the 'start' argument, the axis is moved into the second-to-last position. My proposed change, which you can see on the ticket, would change that so that using -1 referred to the end position. Note that the use of negative 'start' arguments isn't currently documented and, in its current form, doesn't seem very useful. My proposal wouldn't change the behavior for positive 'start' values at all, and the interpretation of 'axis' arguments is also unaffected. If that's going to break too much code, here's a pathway that might be acceptable: Add a new function moveaxis() which works the way rollaxis() does for positive arguments but in the new way for negative arguments. Eventually, rollaxis could be deprecated to keep things tidy. This has the added advantage of using a name that seems to fit what the function does better - 'rollaxis' suggests a behavior like the roll() function which affects other axes, which isn't what happens. Thanks for listening; I'm a big fan of Numpy. Best, Ken Basye From aarchiba at physics.mcgill.ca Tue Sep 21 16:14:34 2010 From: aarchiba at physics.mcgill.ca (Anne Archibald) Date: Tue, 21 Sep 2010 16:14:34 -0400 Subject: [Numpy-discussion] A proposed change to rollaxis() behavior for negative 'start' values In-Reply-To: <4C990C0E.5090904@jhu.edu> References: <4C990C0E.5090904@jhu.edu> Message-ID: Hi Ken, This is a tricky one. The current behaviour of rollaxis is to remove the requested axis from the list of axes and then insert it before the axis specified. This is exactly how python's list insertion works: In [1]: a = range(10) In [3]: a.insert(-1,'a') In [4]: a Out[4]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 'a', 9] And indeed, there's no clean way to add something to the end of a list using insert (apart from the obvious a.insert(len(a),'b') ). For this you have .append(). Unfortunately numpy's rollaxis, while it agrees with insert in its behaviour, doesn't have a move_axis_to_end. The situation is also somewhat muddied by the fact that rollaxis also removes the axis from the original list of axes, so that the interpretation of index numbers is a little more subtle. But I think your suggested behaviour would be confusing because of the conflict with python's insert. How about allowing the string "end" as an argument to rollaxis to specify that the axis should go at the end? Anne On 21 September 2010 15:48, Ken Basye wrote: > Hi Numpy Folks, > ?A while back, I filed this ticket: > http://projects.scipy.org/numpy/ticket/1441 ?suggesting a change to > rollaxis() and some fixes to the doc and error reporting. ?Ralf Gommers > suggested I float the behavior change here, so that's what I'm doing. > > The motivation for the change comes because it seems like there should > be a simpler way to get some axis into the last position than to do this: > > ?>>> a = np.ones((3,4,5,6)) > ?>>> b = np.rollaxis(a, axis=0, start=len(a.shape)) > ?>>> b.shape > (4, 5, 6, 3) > > But currently it seems there isn't because when you specify -1 as the > 'start' argument, the axis is moved into the second-to-last position. > My proposed change, which you can see on the ticket, would change that > so that using -1 referred to the end position. ?Note that the use of > negative 'start' arguments isn't currently documented and, in its > current form, doesn't seem very useful. ?My proposal wouldn't change the > behavior for positive 'start' values at all, and the interpretation of > 'axis' arguments is also unaffected. > > If that's going to break too much code, here's a pathway that might be > acceptable: ?Add a new function moveaxis() which works the way > rollaxis() does for positive arguments but in the new way for negative > arguments. ?Eventually, rollaxis could be deprecated to keep things > tidy. ?This has the added advantage of using a name that seems to fit > what the function does better - 'rollaxis' suggests a behavior like the > roll() function which affects other axes, which isn't what happens. > > Thanks for listening; I'm a big fan of Numpy. > > Best, > ? Ken Basye > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From michael.s.gilbert at gmail.com Tue Sep 21 17:31:55 2010 From: michael.s.gilbert at gmail.com (Michael Gilbert) Date: Tue, 21 Sep 2010 17:31:55 -0400 Subject: [Numpy-discussion] real and imag functions should produce errors for object arrays Message-ID: <20100921173155.5216cdf6.michael.s.gilbert@gmail.com> Hi, The following example demonstrates a rather unexpected result: >>> import numpy >>> x = numpy.array( complex( 1.0 , 1.0 ) , numpy.object ) >>> print x.real (1+1j) >>> print x.imag 0 Shouldn't real and imag return an error in such a situation? Thanks, Mike From ben.root at ou.edu Tue Sep 21 17:44:21 2010 From: ben.root at ou.edu (Benjamin Root) Date: Tue, 21 Sep 2010 16:44:21 -0500 Subject: [Numpy-discussion] real and imag functions should produce errors for object arrays In-Reply-To: <20100921173155.5216cdf6.michael.s.gilbert@gmail.com> References: <20100921173155.5216cdf6.michael.s.gilbert@gmail.com> Message-ID: On Tue, Sep 21, 2010 at 4:31 PM, Michael Gilbert < michael.s.gilbert at gmail.com> wrote: > Hi, > > The following example demonstrates a rather unexpected result: > > >>> import numpy > >>> x = numpy.array( complex( 1.0 , 1.0 ) , numpy.object ) > >>> print x.real > (1+1j) > >>> print x.imag > 0 > > Shouldn't real and imag return an error in such a situation? > > Thanks, > Mike > Don't use 'numpy.object'. Because complex is a numerical type, numpy can handle it just fine. By setting dtype to numpy.object, numpy then treats it like an object rather than a numerical. x = numpy.array( complex(1.0, 1.0) ) should work just fine. I hope that helps! Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Tue Sep 21 17:50:54 2010 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 21 Sep 2010 14:50:54 -0700 Subject: [Numpy-discussion] real and imag functions should produce errors for object arrays In-Reply-To: References: <20100921173155.5216cdf6.michael.s.gilbert@gmail.com> Message-ID: Hi, On Tue, Sep 21, 2010 at 2:44 PM, Benjamin Root wrote: > On Tue, Sep 21, 2010 at 4:31 PM, Michael Gilbert > wrote: >> >> Hi, >> >> The following example demonstrates a rather unexpected result: >> >> >>> import numpy >> >>> x = numpy.array( complex( 1.0 , 1.0 ) , numpy.object ) >> >>> print x.real >> (1+1j) >> >>> print x.imag >> 0 >> >> Shouldn't real and imag return an error in such a situation? It looks like there was a decision to let 'real' and 'imag' pass quietly for non-numerical types: In [2]: a = np.array('hello', dtype=object) In [3]: a.real Out[3]: array('hello', dtype=object) In [4]: a.imag Out[4]: array(0, dtype=object) and In [6]: a = np.array('hello', dtype='S5') In [7]: a.real Out[7]: array('hello', dtype='|S5') In [8]: a.imag Out[8]: array('', dtype='|S5') I can see that that could be confusing. I suppose the alternative would be to raise an error for real and imag for non-numerical types at least. Best, Matthew From pav at iki.fi Tue Sep 21 17:50:08 2010 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 21 Sep 2010 21:50:08 +0000 (UTC) Subject: [Numpy-discussion] real and imag functions should produce errors for object arrays References: <20100921173155.5216cdf6.michael.s.gilbert@gmail.com> Message-ID: Tue, 21 Sep 2010 17:31:55 -0400, Michael Gilbert wrote: > The following example demonstrates a rather unexpected result: > >>>> import numpy >>>> x = numpy.array( complex( 1.0 , 1.0 ) , numpy.object ) > print x.real > (1+1j) >>>> print x.imag > 0 > > Shouldn't real and imag return an error in such a situation? It probably shouldn't do *that* at the least. http://projects.scipy.org/numpy/ticket/1618 -- Pauli Virtanen From pav at iki.fi Tue Sep 21 18:17:37 2010 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 21 Sep 2010 22:17:37 +0000 (UTC) Subject: [Numpy-discussion] real and imag functions should produce errors for object arrays References: <20100921173155.5216cdf6.michael.s.gilbert@gmail.com> Message-ID: Tue, 21 Sep 2010 21:50:08 +0000, Pauli Virtanen wrote: > Tue, 21 Sep 2010 17:31:55 -0400, Michael Gilbert wrote: >> The following example demonstrates a rather unexpected result: >> >>>>> import numpy >>>>> x = numpy.array( complex( 1.0 , 1.0 ) , numpy.object ) >> print x.real >> (1+1j) >>>>> print x.imag >> 0 >> >> Shouldn't real and imag return an error in such a situation? > > It probably shouldn't do *that* at the least. *that* == return a complex number from .real From ben.root at ou.edu Tue Sep 21 18:23:08 2010 From: ben.root at ou.edu (Benjamin Root) Date: Tue, 21 Sep 2010 17:23:08 -0500 Subject: [Numpy-discussion] real and imag functions should produce errors for object arrays In-Reply-To: References: <20100921173155.5216cdf6.michael.s.gilbert@gmail.com> Message-ID: On Tue, Sep 21, 2010 at 4:44 PM, Benjamin Root wrote: > On Tue, Sep 21, 2010 at 4:31 PM, Michael Gilbert < > michael.s.gilbert at gmail.com> wrote: > >> Hi, >> >> The following example demonstrates a rather unexpected result: >> >> >>> import numpy >> >>> x = numpy.array( complex( 1.0 , 1.0 ) , numpy.object ) >> >>> print x.real >> (1+1j) >> >>> print x.imag >> 0 >> >> Shouldn't real and imag return an error in such a situation? >> >> Thanks, >> Mike >> > > Don't use 'numpy.object'. Because complex is a numerical type, numpy can > handle it just fine. By setting dtype to numpy.object, numpy then treats it > like an object rather than a numerical. > > x = numpy.array( complex(1.0, 1.0) ) > > should work just fine. > > I hope that helps! > Ben Root > > I see that I have interpreted this thread as "Doctor, it hurts when I do this... Well, don't do that!" Sorry for the noise. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Sep 21 18:28:08 2010 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 21 Sep 2010 17:28:08 -0500 Subject: [Numpy-discussion] real and imag functions should produce errors for object arrays In-Reply-To: References: <20100921173155.5216cdf6.michael.s.gilbert@gmail.com> Message-ID: On Tue, Sep 21, 2010 at 17:17, Pauli Virtanen wrote: > Tue, 21 Sep 2010 21:50:08 +0000, Pauli Virtanen wrote: > >> Tue, 21 Sep 2010 17:31:55 -0400, Michael Gilbert wrote: >>> The following example demonstrates a rather unexpected result: >>> >>>>>> import numpy >>>>>> x = numpy.array( complex( 1.0 , 1.0 ) , numpy.object ) >>> print x.real >>> (1+1j) >>>>>> print x.imag >>> 0 >>> >>> Shouldn't real and imag return an error in such a situation? >> >> It probably shouldn't do *that* at the least. > > *that* == return a complex number from .real What is the alternative? I'm personally happy with saying that many of the operations we define on numpy arrays can be done because we know the types and that object arrays subvert this. numpy can't, without excessive amounts of magic, always know a sensible thing to do with object arrays, so we implement the fast thing to do. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From matthew.brett at gmail.com Tue Sep 21 18:35:06 2010 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 21 Sep 2010 15:35:06 -0700 Subject: [Numpy-discussion] real and imag functions should produce errors for object arrays In-Reply-To: References: <20100921173155.5216cdf6.michael.s.gilbert@gmail.com> Message-ID: Hi, On Tue, Sep 21, 2010 at 3:28 PM, Robert Kern wrote: > On Tue, Sep 21, 2010 at 17:17, Pauli Virtanen wrote: >> Tue, 21 Sep 2010 21:50:08 +0000, Pauli Virtanen wrote: >> >>> Tue, 21 Sep 2010 17:31:55 -0400, Michael Gilbert wrote: >>>> The following example demonstrates a rather unexpected result: >>>> >>>>>>> import numpy >>>>>>> x = numpy.array( complex( 1.0 , 1.0 ) , numpy.object ) >>>> print x.real >>>> (1+1j) >>>>>>> print x.imag >>>> 0 >>>> >>>> Shouldn't real and imag return an error in such a situation? >>> >>> It probably shouldn't do *that* at the least. >> >> *that* == return a complex number from .real > > What is the alternative? I'm personally happy with saying that many of > the operations we define on numpy arrays can be done because we know > the types and that object arrays subvert this. numpy can't, without > excessive amounts of magic, always know a sensible thing to do with > object arrays, so we implement the fast thing to do. I agree that special-casing object array .real to detect complex contents seems a bit messy, but it does make sense I think to raise an error from .real and .imag for non-numerical types. Best, Matthew From pav at iki.fi Tue Sep 21 18:40:00 2010 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 21 Sep 2010 22:40:00 +0000 (UTC) Subject: [Numpy-discussion] real and imag functions should produce errors for object arrays References: <20100921173155.5216cdf6.michael.s.gilbert@gmail.com> Message-ID: Tue, 21 Sep 2010 17:28:08 -0500, Robert Kern wrote: [clip] >> *that* == return a complex number from .real > > What is the alternative? I'm personally happy with saying that many of > the operations we define on numpy arrays can be done because we know the > types and that object arrays subvert this. numpy can't, without > excessive amounts of magic, always know a sensible thing to do with > object arrays, so we implement the fast thing to do. As I see it, the alternatives are 1) Not to define .real and .imag for object arrays 2) Define them as elementwise .real and .imag I don't clearly see the reason for >>> x.real is x True >>> x.imag array([0], dtype=object) But it is a minor corner case, and there may be backward compatibility issues in changing it. -- Pauli Virtanen From robert.kern at gmail.com Tue Sep 21 19:00:06 2010 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 21 Sep 2010 18:00:06 -0500 Subject: [Numpy-discussion] real and imag functions should produce errors for object arrays In-Reply-To: References: <20100921173155.5216cdf6.michael.s.gilbert@gmail.com> Message-ID: On Tue, Sep 21, 2010 at 17:40, Pauli Virtanen wrote: > Tue, 21 Sep 2010 17:28:08 -0500, Robert Kern wrote: > [clip] >>> *that* == return a complex number from .real >> >> What is the alternative? I'm personally happy with saying that many of >> the operations we define on numpy arrays can be done because we know the >> types and that object arrays subvert this. numpy can't, without >> excessive amounts of magic, always know a sensible thing to do with >> object arrays, so we implement the fast thing to do. > > As I see it, the alternatives are > > 1) Not to define .real and .imag for object arrays > > 2) Define them as elementwise .real and .imag > > I don't clearly see the reason for > >>>> x.real is x > True >>>> x.imag > array([0], dtype=object) The reason it is that way is probably because object and str dtypes were an afterthought when we extended .real and .imag to all dtypes. We used to only have those attributes on complex arrays. We added the behavior described above to all of the other dtypes to allow one to use the .real and .imag attributes regardless of whether the array was complex or real. I don't think we gave much thought to what the right thing to do with object arrays or str arrays. Perhaps we did think about it and decided that consistency with the other non-complex dtypes was more important than trying to expensively inspect the individual objects for .real and .imag attributes. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From sierra_mtnview at sbcglobal.net Tue Sep 21 19:01:47 2010 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Tue, 21 Sep 2010 16:01:47 -0700 Subject: [Numpy-discussion] Removing numpy numpy-1.1.0 in Python 2.5 Message-ID: <4C99395B.2010608@sbcglobal.net> I may have bounced a similar problem around here a few months ago, but this one is a bit more important to get an answer for. I'm about to distribute some report programs to three people who work together and should be using the same version of Python and libraries that I use, and a common application to us. However, to create two new report programs they do not have, I need to make sure we do have the same versions, and in particular need to use the numpy 1.2.0 I used for the report programs. To replace the old numpy and make sure we are all on on the same wave length, I wrote the description below. I want to know if what I say about removing 1.1.0 and installing 1.2.0 is correct. I think deleting the folder as below is the right way to go. numpy is not in Control Panel Add/Remove. Comments? ================================= Required Libraries for Sentinel Report Programs for Win XP ... You will need to replace your numpy-1.1.0 library. This is accomplished by first removing numpy-1.1.0 from the Python library located at C:\Python2.5\Lib\site-packages. Delete the numpy folder. Next execute the numpy-1.2.0-win32-superpack-python2.5.exe to install numpy 1.2.0. Try sentuser.py to make sure it runs properly. If not, contact me. ================================== -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet Want to lose weight? Move to the equator. You'll weigh about 0.1% less than at the poles. Centrifugal force is directed opposite gravity there--zero at the poles. -- WTW Web Page: From hilton at meteo.psu.edu Tue Sep 21 19:20:51 2010 From: hilton at meteo.psu.edu (Timothy W. Hilton) Date: Tue, 21 Sep 2010 16:20:51 -0700 Subject: [Numpy-discussion] slicing / indexing question Message-ID: <20100921232051.GA451@Tim.local> Hello, I have an indexing problem which I suspect has a simple solution, but I've not been able to piece together various threads I've read on this list to solve. I have an 80x1200x1200 nd.array of floats this_par. I have a 1200x1200 boolean array idx, and an 80-element float array pars. For each element of idx that is True, I wish to replace the corresponding 80x1x1 slice of this_par with the elements of pars. I've tried lots of variations on the theme of >>>this_par[idx[np.newaxis, ...]] = pars[:, np.newaxis, np.newaxis] but so far, no dice. Any help greatly appreciated! Thanks, Tim -- Timothy W. Hilton PhD Candidate, Department of Meteorology The Pennsylvania State University 503 Walker Building, University Park, PA 16802 hilton at meteo.psu.edu From josef.pktd at gmail.com Tue Sep 21 19:38:55 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 21 Sep 2010 19:38:55 -0400 Subject: [Numpy-discussion] Removing numpy numpy-1.1.0 in Python 2.5 In-Reply-To: <4C99395B.2010608@sbcglobal.net> References: <4C99395B.2010608@sbcglobal.net> Message-ID: On Tue, Sep 21, 2010 at 7:01 PM, Wayne Watson wrote: > ?I may have bounced a similar problem around here a few months ago, but > this one is a bit more important to get an answer for. > > I'm about to distribute some report programs to three people who work > together and should be using the same version of Python and libraries > that I use, and a common application to us. However, to create two new > report programs they do not have, I need to make sure we do have the > same versions, and in particular need to use the numpy 1.2.0 I used for > the report programs. > > To replace the old numpy and make sure we are all on on the same wave > length, I wrote the description below. ?I want to know if what I say > about removing 1.1.0 and installing 1.2.0 is correct. ?I think deleting > the folder as below is the right way to go. ?numpy is not in Control > Panel Add/Remove. I have some numpy versions in there I would recommend to first check Add/Remove, and delete from there and then delete the numpy folder in site-packages if it is still there. Why don't you upgrade to the latest release. If you are already upgrading, jumping to the latest might be better. Josef Comments? > > ================================= > ? ? ? ? ? Required Libraries for Sentinel Report Programs for Win XP > ... > You will need to replace your numpy-1.1.0 library. This is accomplished by > first removing numpy-1.1.0 from the Python library located at > > ? C:\Python2.5\Lib\site-packages. > > Delete the numpy folder. > > Next execute the numpy-1.2.0-win32-superpack-python2.5.exe to > install numpy 1.2.0. > > Try sentuser.py to make sure it runs properly. If not, contact me. > ================================== > > -- > ? ? ? ? ? ?Wayne Watson (Watson Adventures, Prop., Nevada City, CA) > > ? ? ? ? ? ? ?(121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) > ? ? ? ? ? ? ? Obz Site: ?39? 15' 7" N, 121? 2' 32" W, 2700 feet > > ? ? ? ? ? ? Want to lose weight? Move to the equator. You'll weigh > ? ? ? ? ? ? about 0.1% less than at the poles. Centrifugal force > ? ? ? ? ? ? is directed opposite gravity there--zero at the poles. > ? ? ? ? ? ? ? ? ? ? ? ? ? ? -- WTW > > > ? ? ? ? ? ? ? ? ? ? Web Page: > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From brett.olsen at gmail.com Tue Sep 21 19:54:03 2010 From: brett.olsen at gmail.com (Brett Olsen) Date: Tue, 21 Sep 2010 18:54:03 -0500 Subject: [Numpy-discussion] slicing / indexing question In-Reply-To: <20100921232051.GA451@Tim.local> References: <20100921232051.GA451@Tim.local> Message-ID: On Tue, Sep 21, 2010 at 6:20 PM, Timothy W. Hilton wrote: > Hello, > > I have an indexing problem which I suspect has a simple solution, but > I've not been able to piece together various threads I've read on this > list to solve. > > I have an 80x1200x1200 nd.array of floats this_par. ?I have a > 1200x1200 boolean array idx, and an 80-element float array pars. ?For > each element of idx that is True, I wish to replace the corresponding > 80x1x1 slice of this_par with the elements of pars. > > I've tried lots of variations on the theme of >>>>this_par[idx[np.newaxis, ...]] = pars[:, np.newaxis, np.newaxis] > but so far, no dice. > > Any help greatly appreciated! > > Thanks, > Tim This works, although I imagine it could be streamlined. In [1]: this_par = N.ones((2,4,4)) In [2]: idx = N.random.random((4,4)) > 0.5 In [3]: pars = N.arange(2) - 10 In [4]: this_par[:,idx] = N.tile(pars, (idx.sum(), 1)).transpose() In [5]: idx Out[5] array([[ True, False, True, False], [False, False, True, True], [False, False, False, False], [False, False, False, False]], dtype=bool) In [6]: this_par Out[6]: array([[[-10., 1., -10., 1.], [ 1., 1., -10., -10.], [ 1., 1., 1., 1.], [ 1., 1., 1., 1.]], [[ -9., 1., -9., 1.], [ 1., 1., -9., -9.], [ 1., 1., 1., 1.], [ 1., 1., 1., 1.]]]) Brett From josef.pktd at gmail.com Tue Sep 21 20:04:26 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 21 Sep 2010 20:04:26 -0400 Subject: [Numpy-discussion] slicing / indexing question In-Reply-To: References: <20100921232051.GA451@Tim.local> Message-ID: On Tue, Sep 21, 2010 at 7:54 PM, Brett Olsen wrote: > On Tue, Sep 21, 2010 at 6:20 PM, Timothy W. Hilton wrote: >> Hello, >> >> I have an indexing problem which I suspect has a simple solution, but >> I've not been able to piece together various threads I've read on this >> list to solve. >> >> I have an 80x1200x1200 nd.array of floats this_par. ?I have a >> 1200x1200 boolean array idx, and an 80-element float array pars. ?For >> each element of idx that is True, I wish to replace the corresponding >> 80x1x1 slice of this_par with the elements of pars. >> >> I've tried lots of variations on the theme of >>>>>this_par[idx[np.newaxis, ...]] = pars[:, np.newaxis, np.newaxis] >> but so far, no dice. >> >> Any help greatly appreciated! >> >> Thanks, >> Tim > > This works, although I imagine it could be streamlined. > > In [1]: this_par = N.ones((2,4,4)) > In [2]: idx = N.random.random((4,4)) > 0.5 > In [3]: pars = N.arange(2) - 10 > In [4]: this_par[:,idx] = N.tile(pars, (idx.sum(), 1)).transpose() > In [5]: idx > Out[5] > array([[ True, False, True, False], > ? ? ? ? [False, False, True, True], > ? ? ? ? [False, False, False, False], > ? ? ? ? [False, False, False, False]], dtype=bool) > In [6]: this_par > Out[6]: > array([[[-10., ? 1., -10., ? 1.], > ? ? ? ? ?[ ? 1., ? 1., -10., -10.], > ? ? ? ? ?[ ? 1., ? 1., ? ?1., ? 1.], > ? ? ? ? ?[ ? 1., ? 1., ? ?1., ? 1.]], > ? ? ? ? [[ -9., ? 1., ?-9., ? 1.], > ? ? ? ? ?[ ? 1., ? 1., ?-9., ?-9.], > ? ? ? ? ?[ ? 1., ? 1., ? ?1., ? 1.], > ? ? ? ? ?[ ? 1., ? 1., ? ?1., ? 1.]]]) introspection works easier with an example indexing with slice and boolean seems to flatten the boolean part, then we need only one newaxis ? seems to work >>> this_par[:,idx].shape # = N.tile(pars, (idx.sum(), 1)).transpose() (2, 8) >>> idx array([[ True, False, False, False], [ True, True, True, False], [ True, True, False, False], [ True, False, True, False]], dtype=bool) >>> this_par[:,idx] = pars[:,N.newaxis] >>> this_par array([[[-10., 1., 1., 1.], [-10., -10., -10., 1.], [-10., -10., 1., 1.], [-10., 1., -10., 1.]], [[ -9., 1., 1., 1.], [ -9., -9., -9., 1.], [ -9., -9., 1., 1.], [ -9., 1., -9., 1.]]]) Josef > Brett > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From matthew.brett at gmail.com Tue Sep 21 21:15:08 2010 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 21 Sep 2010 18:15:08 -0700 Subject: [Numpy-discussion] real and imag functions should produce errors for object arrays In-Reply-To: References: <20100921173155.5216cdf6.michael.s.gilbert@gmail.com> Message-ID: Hi, > I see that I have interpreted this thread as "Doctor, it hurts when I do > this...? Well, don't do that!"? Sorry for the noise. It's all good - a reply is almost always more friendly and helpful than no reply ;) See you, Matthew From sierra_mtnview at sbcglobal.net Tue Sep 21 22:32:33 2010 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Tue, 21 Sep 2010 19:32:33 -0700 Subject: [Numpy-discussion] Removing numpy numpy-1.1.0 in Python 2.5 In-Reply-To: References: <4C99395B.2010608@sbcglobal.net> Message-ID: <4C996AC1.2060902@sbcglobal.net> I don't see it in Add/Remove. Whoops. It is on this Win7 machine. I need to check my XP machine. I'll be back when I figure out if that's right. My not always recollection was that it was not on Add/Remove under XP. I'm dealing with people who barely understand software, so keeping this simple is the best thing to do. This version of the application is the last, and we might as well treat it as frozen. I really don't want to try to correct people's install problems or attempts to move ahead with the various libraries we use over the internet, so simple is it. On 9/21/2010 4:38 PM, josef.pktd at gmail.com wrote: > On Tue, Sep 21, 2010 at 7:01 PM, Wayne Watson > wrote: >> I may have bounced a similar problem around here a few months ago, but >> this one is a bit more important to get an answer for. >> >> I'm about to distribute some report programs to three people who work >> together and should be using the same version of Python and libraries >> that I use, and a common application to us. However, to create two new >> report programs they do not have, I need to make sure we do have the >> same versions, and in particular need to use the numpy 1.2.0 I used for >> the report programs. >> >> To replace the old numpy and make sure we are all on on the same wave >> length, I wrote the description below. I want to know if what I say >> about removing 1.1.0 and installing 1.2.0 is correct. I think deleting >> the folder as below is the right way to go. numpy is not in Control >> Panel Add/Remove. > I have some numpy versions in there > > I would recommend to first check Add/Remove, and delete from there and > then delete the numpy folder in site-packages if it is still there. > > Why don't you upgrade to the latest release. If you are already > upgrading, jumping to the latest might be better. > > Josef > > > Comments? >> ================================= >> Required Libraries for Sentinel Report Programs for Win XP >> ... >> You will need to replace your numpy-1.1.0 library. This is accomplished by >> first removing numpy-1.1.0 from the Python library located at >> >> C:\Python2.5\Lib\site-packages. >> >> Delete the numpy folder. >> >> Next execute the numpy-1.2.0-win32-superpack-python2.5.exe to >> install numpy 1.2.0. >> >> Try sentuser.py to make sure it runs properly. If not, contact me. >> ================================== >> >> -- >> Wayne Watson (Watson Adventures, Prop., Nevada City, CA) >> >> (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) >> Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet >> >> Want to lose weight? Move to the equator. You'll weigh >> about 0.1% less than at the poles. Centrifugal force >> is directed opposite gravity there--zero at the poles. >> -- WTW >> >> >> Web Page: >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet Want to lose weight? Move to the equator. You'll weigh about 0.1% less than at the poles. Centrifugal force is directed opposite gravity there--zero at the poles. -- WTW Web Page: From josef.pktd at gmail.com Tue Sep 21 22:49:17 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 21 Sep 2010 22:49:17 -0400 Subject: [Numpy-discussion] Removing numpy numpy-1.1.0 in Python 2.5 In-Reply-To: <4C996AC1.2060902@sbcglobal.net> References: <4C99395B.2010608@sbcglobal.net> <4C996AC1.2060902@sbcglobal.net> Message-ID: On Tue, Sep 21, 2010 at 10:32 PM, Wayne Watson wrote: > ?I don't see ?it in ?Add/Remove. ?Whoops. ?It is on this Win7 machine. > I need to check my XP machine. ?I'll be back when I figure out if that's > right. My not always recollection was that it was not on Add/Remove > under XP. > > I'm dealing with people who barely understand software, so keeping this > simple is the best thing to do. This version of the application is the > last, and we might as well treat it as frozen. ?I really don't want to > try to correct people's install problems or attempts to move ahead with > the various libraries we use over the internet, so simple is it. If you want to have a frozen application, the best might be to use py2exe, but maybe not worth the effort, if they don't mess with the installed python packages. Josef > > On 9/21/2010 4:38 PM, josef.pktd at gmail.com wrote: >> On Tue, Sep 21, 2010 at 7:01 PM, Wayne Watson >> ?wrote: >>> ? I may have bounced a similar problem around here a few months ago, but >>> this one is a bit more important to get an answer for. >>> >>> I'm about to distribute some report programs to three people who work >>> together and should be using the same version of Python and libraries >>> that I use, and a common application to us. However, to create two new >>> report programs they do not have, I need to make sure we do have the >>> same versions, and in particular need to use the numpy 1.2.0 I used for >>> the report programs. >>> >>> To replace the old numpy and make sure we are all on on the same wave >>> length, I wrote the description below. ?I want to know if what I say >>> about removing 1.1.0 and installing 1.2.0 is correct. ?I think deleting >>> the folder as below is the right way to go. ?numpy is not in Control >>> Panel Add/Remove. >> I have some numpy versions in there >> >> I would recommend to first check Add/Remove, and delete from there and >> then delete the numpy folder in site-packages if it is still there. >> >> Why don't you upgrade to the latest release. If you are already >> upgrading, jumping to the latest might be better. >> >> Josef >> >> >> Comments? >>> ================================= >>> ? ? ? ? ? ?Required Libraries for Sentinel Report Programs for Win XP >>> ... >>> You will need to replace your numpy-1.1.0 library. This is accomplished by >>> first removing numpy-1.1.0 from the Python library located at >>> >>> ? ?C:\Python2.5\Lib\site-packages. >>> >>> Delete the numpy folder. >>> >>> Next execute the numpy-1.2.0-win32-superpack-python2.5.exe to >>> install numpy 1.2.0. >>> >>> Try sentuser.py to make sure it runs properly. If not, contact me. >>> ================================== >>> >>> -- >>> ? ? ? ? ? ? Wayne Watson (Watson Adventures, Prop., Nevada City, CA) >>> >>> ? ? ? ? ? ? ? (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) >>> ? ? ? ? ? ? ? ?Obz Site: ?39? 15' 7" N, 121? 2' 32" W, 2700 feet >>> >>> ? ? ? ? ? ? ?Want to lose weight? Move to the equator. You'll weigh >>> ? ? ? ? ? ? ?about 0.1% less than at the poles. Centrifugal force >>> ? ? ? ? ? ? ?is directed opposite gravity there--zero at the poles. >>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?-- WTW >>> >>> >>> ? ? ? ? ? ? ? ? ? ? ?Web Page: >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > -- > ? ? ? ? ? ?Wayne Watson (Watson Adventures, Prop., Nevada City, CA) > > ? ? ? ? ? ? ?(121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) > ? ? ? ? ? ? ? Obz Site: ?39? 15' 7" N, 121? 2' 32" W, 2700 feet > > ? ? ? ? ? ? Want to lose weight? Move to the equator. You'll weigh > ? ? ? ? ? ? about 0.1% less than at the poles. Centrifugal force > ? ? ? ? ? ? is directed opposite gravity there--zero at the poles. > ? ? ? ? ? ? ? ? ? ? ? ? ? ? -- WTW > > > ? ? ? ? ? ? ? ? ? ? Web Page: > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From keekychen.shared at gmail.com Wed Sep 22 02:04:03 2010 From: keekychen.shared at gmail.com (keekychen.shared) Date: Wed, 22 Sep 2010 14:04:03 +0800 Subject: [Numpy-discussion] dtype from |S10 to object in array? Message-ID: <4C999C53.9060502@gmail.com> Dear All, See below code pls, import sicpy import numpy as np x = np.zeros((2,),dtype=('i4,f4,a10')) x[:] = [(1,2.,'Hello'),(2,3.,"World")] y = x['f2'] #array(['Hello', 'World'], dtype='|S10') x['f2'] = y x #array([(1, 2.0, 'Hello'), (2, 3.0, 'World')], dtype=[('f0', ' References: <4C98A42F.7020201@gmail.com> Message-ID: <4C99E096.9040200@gmail.com> On 21/09/2010 18:29, David Huard wrote: > Have you tried > > http://code.google.com/p/python-fortranformat/ > > It's not officially released yet but it's probably worth a try. Well, I noticed it, bugt the website does say "This is a work in progress, a working version is not yet available"! Andrew > > David H. > > On Tue, Sep 21, 2010 at 8:25 AM, Andrew Jaffe > wrote: > > Hi all, > > I've got an ascii file with a relatively complicated structure, > originally written by fortran with the format: > > 135 format(a12,1x,2(f10.5,1x),i3,1x,4(f9.3,1x),4(i2,1x),3x, > 1 16(f7.2,1x),i3,3x,f13.5,1x,f10.5,1x,f10.6,1x,i3,1x, > 2 4(f10.6,1x), > 2 i2,1x,f5.2,1x,f10.3,1x,i3,1x,f7.2,1x,f7.2,3x,4(f7.4,1x), > 3 4(f7.2,1x),3x,f7.2,1x,i4,3x,f10.3,1x,14(f6.2,1x),i3,1x, > 1 3x,2f10.5,8f11.2,2f10.5,f12.3,3x, > 4 2(a6,1x),a23,1x,a22,1x,a22) > > Note, in particular, that many of the strings contain white space. > > Is there a relatively straightforward way to translate this into dtype > (and delimiter?) arguments for use with genfromtxt or do I just have to > do it by hand? > > Andrew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From pgmdevlist at gmail.com Wed Sep 22 07:51:15 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 22 Sep 2010 13:51:15 +0200 Subject: [Numpy-discussion] read ascii file from complex fortran format() -- genfromtxt In-Reply-To: <4C98A42F.7020201@gmail.com> References: <4C98A42F.7020201@gmail.com> Message-ID: On Sep 21, 2010, at 2:25 PM, Andrew Jaffe wrote: > Hi all, > > I've got an ascii file with a relatively complicated structure, > originally written by fortran with the format: > > 135 format(a12,1x,2(f10.5,1x),i3,1x,4(f9.3,1x),4(i2,1x),3x, > 1 16(f7.2,1x),i3,3x,f13.5,1x,f10.5,1x,f10.6,1x,i3,1x, > 2 4(f10.6,1x), > 2 i2,1x,f5.2,1x,f10.3,1x,i3,1x,f7.2,1x,f7.2,3x,4(f7.4,1x), > 3 4(f7.2,1x),3x,f7.2,1x,i4,3x,f10.3,1x,14(f6.2,1x),i3,1x, > 1 3x,2f10.5,8f11.2,2f10.5,f12.3,3x, > 4 2(a6,1x),a23,1x,a22,1x,a22) > > Note, in particular, that many of the strings contain white space. > > Is there a relatively straightforward way to translate this into dtype > (and delimiter?) arguments for use with genfromtxt or do I just have to > do it by hand? By hand might be the easiest. If possible, try to put the empty spaces at the beginning of the string (eg, your 3x,f13.5 would be 16 characters). Use `autostrip=True`. Use a sequence as delimiter (like in your case (12,11,11,3,10,10,10,10,3,3,10,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,4,16...)), that should be more efficient. From josef.pktd at gmail.com Wed Sep 22 08:14:34 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 22 Sep 2010 08:14:34 -0400 Subject: [Numpy-discussion] dtype from |S10 to object in array? In-Reply-To: <4C999C53.9060502@gmail.com> References: <4C999C53.9060502@gmail.com> Message-ID: On Wed, Sep 22, 2010 at 2:04 AM, keekychen.shared wrote: > ?Dear All, > > See below code pls, > > import sicpy > import numpy as np > > x = np.zeros((2,),dtype=('i4,f4,a10')) > x[:] = [(1,2.,'Hello'),(2,3.,"World")] > > y = x['f2'] > #array(['Hello', 'World'], > ? ? ?dtype='|S10') > > x['f2'] = y > x > #array([(1, 2.0, 'Hello'), (2, 3.0, 'World')], > ? ? ?dtype=[('f0', ' > y = y.astype('object') > y > array([Hello, World], dtype=object) > > > x['f2'] = y > array([(1, 2.0, 'HellWorld'), (2, 3.0, '\x00\x00\x00\x00\x00\x00\x18')], > ? ? ?dtype=[('f0', ' > ##here comes the problem: the f2 col type has not been changed and the > data is not I wanted... > ---------------------------------------------------------------------------- > > here is why I need using this: > suppose I have a datasource, csv, sql based db or what ever look like this: > > 1, 2.0, 'Hello' > 2, 3.0, 'World' > 3, 2.0, 'other string' > > I want to read them to a numpy array and process it's columns, it has no > problem for processing the float or int type but string. > After reading the manual and found the object dtype may store variable > string then I want to exact the string col into an new array, try to > process it then store back to the numpy "matrix" then store it back to > the data source. > > May I know how I can do that? I do not care performance now. I *guess* that you can do this by changing only the type of the string column to object dt = np.dtype([('f0', '>> x = np.zeros((2,),dtype=('i4,f4,O')) >>> x[:] = [(1,2.,'Hello'),(2,3.,"World")] >>> x array([(1, 2.0, 'Hello'), (2, 3.0, 'World')], dtype=[('f0', '>> x['f2'] = 'hellooooooooooooooooo' >>> x array([(1, 2.0, 'hellooooooooooooooooo'), (2, 3.0, 'hellooooooooooooooooo')], dtype=[('f0', ' > > Thanks for any hints > > Rgs, > > KC > > > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ndbecker2 at gmail.com Wed Sep 22 10:10:06 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 22 Sep 2010 10:10:06 -0400 Subject: [Numpy-discussion] searching binary data Message-ID: A colleague of mine posed the following problem. He wants to search large files of binary data for sequences. I thought of using mmap (to avoid reading all data into memory at once) and then turning this into a numpy array (using buffer=). But, how to then efficiently find a sequence? Note that I'm not committed to using numpy or mmap as part of the solution. From cournape at gmail.com Wed Sep 22 10:23:24 2010 From: cournape at gmail.com (David Cournapeau) Date: Wed, 22 Sep 2010 23:23:24 +0900 Subject: [Numpy-discussion] searching binary data In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 11:10 PM, Neal Becker wrote: > A colleague of mine posed the following problem. ?He wants to search large > files of binary data for sequences. > Is there a reason why you cannot use one of the classic string search algorithms applied to the bytestream ? David From ndbecker2 at gmail.com Wed Sep 22 10:25:41 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 22 Sep 2010 10:25:41 -0400 Subject: [Numpy-discussion] searching binary data References: Message-ID: David Cournapeau wrote: > On Wed, Sep 22, 2010 at 11:10 PM, Neal Becker wrote: >> A colleague of mine posed the following problem. He wants to search >> large files of binary data for sequences. >> > > Is there a reason why you cannot use one of the classic string search > algorithms applied to the bytestream ? > What would you suggest? Keep in mind the file is to big to fit into memory all at once. From josef.pktd at gmail.com Wed Sep 22 10:27:09 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 22 Sep 2010 10:27:09 -0400 Subject: [Numpy-discussion] printable random seed ? Message-ID: I would like to generate random numbers based on a random seed, for example what numpy.random does if the seed is not specified. But I would also like to print out the initial state, so I can replicate the random numbers. Can I get a human readable or printable version of the initial state? Alternatively, what's a good way to randomly generate an initial state? I could draw an integer with randint and use it as seed. Is this the best way? Josef From ndbecker2 at gmail.com Wed Sep 22 10:32:22 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 22 Sep 2010 10:32:22 -0400 Subject: [Numpy-discussion] printable random seed ? References: Message-ID: josef.pktd at gmail.com wrote: > I would like to generate random numbers based on a random seed, for > example what numpy.random does if the seed is not specified. But I > would also like to print out the initial state, so I can replicate the > random numbers. > > Can I get a human readable or printable version of the initial state? > Alternatively, what's a good way to randomly generate an initial > state? > > I could draw an integer with randint and use it as seed. Is this the best > way? > > Josef import struct import os seed = struct.unpack ('I', os.urandom (4))[0] print seed From robert.kern at gmail.com Wed Sep 22 10:34:50 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 22 Sep 2010 09:34:50 -0500 Subject: [Numpy-discussion] printable random seed ? In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 09:27, wrote: > I would like to generate random numbers based on a random seed, for > example what numpy.random does if the seed is not specified. But I > would also like to print out the initial state, so I can replicate the > random numbers. > > Can I get a human readable or printable version of the initial state? [~] |13> prng = np.random.RandomState() [~] |14> prng2 = np.random.RandomState() [~] |15> prng.randint(100, size=10) array([74, 62, 56, 94, 86, 59, 69, 94, 42, 18]) [~] |16> prng2.randint(100, size=10) array([21, 58, 34, 55, 9, 81, 45, 3, 93, 62]) [~] |17> prng2.set_state(prng.get_state()) [~] |18> prng.randint(100, size=10) array([37, 57, 2, 0, 68, 9, 75, 88, 11, 7]) [~] |19> prng2.randint(100, size=10) array([37, 57, 2, 0, 68, 9, 75, 88, 11, 7]) [~] |20> prng.get_state() ('MT19937', array([1368120112, 957462593, 2623310617, 4207155283, 446940397, 3506388262, 4104366519, 371500243, 4029407620, 899392379, .... 1843090101, 2484333397, 4085469971, 306955884, 23307203, 1640066622, 48186677, 637144011, 854838500], dtype=uint32), 26, 0, 2.5933437794758841e-288) > Alternatively, what's a good way to ?randomly generate an initial > state? np.random.RandomState() will do the best thing on each platform. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From robert.kern at gmail.com Wed Sep 22 10:29:20 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 22 Sep 2010 09:29:20 -0500 Subject: [Numpy-discussion] searching binary data In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 09:10, Neal Becker wrote: > A colleague of mine posed the following problem. ?He wants to search large > files of binary data for sequences. > > I thought of using mmap (to avoid reading all data into memory at once) and > then turning this into a numpy array (using buffer=). > > But, how to then efficiently find a sequence? mmap objects have most of the usual string methods: [~] |2> f = open('./scratch/foo.py', 'r+b') [~] |4> m = mmap.mmap(f.fileno(), 0) [~] |6> m.find('import') 11 [~] |7> m[11:17] 'import' -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From dalcinl at gmail.com Wed Sep 22 10:35:59 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 22 Sep 2010 11:35:59 -0300 Subject: [Numpy-discussion] [PATCH] F2Py on Python 3 Message-ID: It seems that lib2to3 does not process the main f2py bootstrap script that gets autogenerated by f2py's setup.py. The trivial patch below replaces the print statements with sys.stderr.write() calls. After that change, f2py works just fine in Python 3.2 Index: numpy/f2py/setup.py =================================================================== --- numpy/f2py/setup.py (revision 8716) +++ numpy/f2py/setup.py (working copy) @@ -62,7 +62,7 @@ except ValueError: pass os.environ["NO_SCIPY_IMPORT"]="f2py" if mode=="g3-numpy": - print >> sys.stderr, "G3 f2py support is not implemented, yet." + sys.stderr.write("G3 f2py support is not implemented, yet.\n") sys.exit(1) elif mode=="2e-numeric": from f2py2e import main @@ -72,7 +72,7 @@ elif mode=="2e-numpy": from numpy.f2py import main else: - print >> sys.stderr, "Unknown mode:",`mode` + sys.stderr.write("Unknown mode: '%s'\n" % mode) sys.exit(1) main() '''%(os.path.basename(sys.executable))) -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From cournape at gmail.com Wed Sep 22 10:40:24 2010 From: cournape at gmail.com (David Cournapeau) Date: Wed, 22 Sep 2010 23:40:24 +0900 Subject: [Numpy-discussion] searching binary data In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 11:25 PM, Neal Becker wrote: > David Cournapeau wrote: > >> On Wed, Sep 22, 2010 at 11:10 PM, Neal Becker wrote: >>> A colleague of mine posed the following problem. ?He wants to search >>> large files of binary data for sequences. >>> >> >> Is there a reason why you cannot use one of the classic string search >> algorithms applied to the bytestream ? >> > > What would you suggest? ?Keep in mind the file is to big to fit into memory > all at once. Do you care about speed ? String search and even regular expression are supposed to work on mmap data, but I have never used them on large datasets, so I don't know how they would perform. Otherwise, depending on the data and whether you can afford pre-computing, algorithms like Knuth Morris Pratt can speed things up. But I would assume you would have to do it in C to hope any speed gain compared to python string search . cheers, David From charlesr.harris at gmail.com Wed Sep 22 11:21:48 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 22 Sep 2010 09:21:48 -0600 Subject: [Numpy-discussion] [PATCH] F2Py on Python 3 In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 8:35 AM, Lisandro Dalcin wrote: > It seems that lib2to3 does not process the main f2py bootstrap script > that gets autogenerated by f2py's setup.py. The trivial patch below > replaces the print statements with sys.stderr.write() calls. After > that change, f2py works just fine in Python 3.2 > > > Index: numpy/f2py/setup.py > =================================================================== > --- numpy/f2py/setup.py (revision 8716) > +++ numpy/f2py/setup.py (working copy) > @@ -62,7 +62,7 @@ > except ValueError: pass > os.environ["NO_SCIPY_IMPORT"]="f2py" > if mode=="g3-numpy": > - print >> sys.stderr, "G3 f2py support is not implemented, yet." > + sys.stderr.write("G3 f2py support is not implemented, yet.\n") > sys.exit(1) > elif mode=="2e-numeric": > from f2py2e import main > @@ -72,7 +72,7 @@ > elif mode=="2e-numpy": > from numpy.f2py import main > else: > - print >> sys.stderr, "Unknown mode:",`mode` > + sys.stderr.write("Unknown mode: '%s'\n" % mode) > sys.exit(1) > main() > '''%(os.path.basename(sys.executable))) > > > Looks reasonable. Want to try running this through the git process just for practice? Of course, it is only two lines so if git seems a bit much I'll make the change for you. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokhansever at gmail.com Wed Sep 22 11:46:19 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Wed, 22 Sep 2010 10:46:19 -0500 Subject: [Numpy-discussion] slicing / indexing question In-Reply-To: <20100921232051.GA451@Tim.local> References: <20100921232051.GA451@Tim.local> Message-ID: On Tue, Sep 21, 2010 at 6:20 PM, Timothy W. Hilton wrote: > Hello, > > I have an indexing problem which I suspect has a simple solution, but > I've not been able to piece together various threads I've read on this > list to solve. > > I have an 80x1200x1200 nd.array of floats this_par. I have a > 1200x1200 boolean array idx, and an 80-element float array pars. For > each element of idx that is True, I wish to replace the corresponding > 80x1x1 slice of this_par with the elements of pars. > > I've tried lots of variations on the theme of > >>>this_par[idx[np.newaxis, ...]] = pars[:, np.newaxis, np.newaxis] > but so far, no dice. > > Any help greatly appreciated! > > Thanks, > Tim > This is a tough indexing question for me. I can visualize the arrays and the required selections in my mind, but hard to put them into application as in this case. Beyond 2D array operations indexing and broadcasting operations get somewhat complex to interpret at the first look and likewise later when I read and decipher my own code, not to mention if someone else wants to understand that same code. What do you do with such big arrays? It is good to see new atmospheric sciences people in the lists. Our department is still swarming with IDL and Matlab users. -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Sep 22 11:46:40 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 22 Sep 2010 11:46:40 -0400 Subject: [Numpy-discussion] printable random seed ? In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 10:32 AM, Neal Becker wrote: > josef.pktd at gmail.com wrote: > >> I would like to generate random numbers based on a random seed, for >> example what numpy.random does if the seed is not specified. But I >> would also like to print out the initial state, so I can replicate the >> random numbers. >> >> Can I get a human readable or printable version of the initial state? >> Alternatively, what's a good way to ?randomly generate an initial >> state? >> >> I could draw an integer with randint and use it as seed. Is this the best >> way? >> >> Josef > > ? ? ? ?import struct > ? ? ? ?import os > ? ? ? ?seed = struct.unpack ('I', os.urandom (4))[0] > ? ? ? ?print seed >>> os.urandom(4) '\x02\xcf\xd5`' >>> np.array(os.urandom(4)).view(int) array(-452038899) >>> import struct >>> seed = struct.unpack ('I', os.urandom (4))[0] >>> seed 3650333822L >>> np.random.seed(seed) Traceback (most recent call last): File "", line 1, in np.random.seed(seed) File "mtrand.pyx", line 593, in mtrand.RandomState.seed (numpy\random\mtrand\mtrand.c:4786) OverflowError: long int too large to convert to int Josef > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From josef.pktd at gmail.com Wed Sep 22 11:48:54 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 22 Sep 2010 11:48:54 -0400 Subject: [Numpy-discussion] printable random seed ? In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 10:34 AM, Robert Kern wrote: > On Wed, Sep 22, 2010 at 09:27, ? wrote: >> I would like to generate random numbers based on a random seed, for >> example what numpy.random does if the seed is not specified. But I >> would also like to print out the initial state, so I can replicate the >> random numbers. >> >> Can I get a human readable or printable version of the initial state? > > [~] > |13> prng = np.random.RandomState() > > [~] > |14> prng2 = np.random.RandomState() > > [~] > |15> prng.randint(100, size=10) > array([74, 62, 56, 94, 86, 59, 69, 94, 42, 18]) > > [~] > |16> prng2.randint(100, size=10) > array([21, 58, 34, 55, ?9, 81, 45, ?3, 93, 62]) > > [~] > |17> prng2.set_state(prng.get_state()) > > [~] > |18> prng.randint(100, size=10) > array([37, 57, ?2, ?0, 68, ?9, 75, 88, 11, ?7]) > > [~] > |19> prng2.randint(100, size=10) > array([37, 57, ?2, ?0, 68, ?9, 75, 88, 11, ?7]) > > [~] > |20> prng.get_state() > > ('MT19937', > ?array([1368120112, ?957462593, 2623310617, 4207155283, ?446940397, > ? ? ? 3506388262, 4104366519, ?371500243, 4029407620, ?899392379, > .... > ? ? ? 1843090101, 2484333397, 4085469971, ?306955884, ? 23307203, > ? ? ? 1640066622, ? 48186677, ?637144011, ?854838500], dtype=uint32), > ?26, > ?0, > ?2.5933437794758841e-288) > >> Alternatively, what's a good way to ?randomly generate an initial >> state? > > np.random.RandomState() will do the best thing on each platform. Thanks, I need to think about whether I want to save the full get_state() tuple (or cheat) Josef > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ? -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Wed Sep 22 11:53:22 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 22 Sep 2010 10:53:22 -0500 Subject: [Numpy-discussion] printable random seed ? In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 10:48, wrote: > On Wed, Sep 22, 2010 at 10:34 AM, Robert Kern wrote: >> On Wed, Sep 22, 2010 at 09:27, ? wrote: >>> I would like to generate random numbers based on a random seed, for >>> example what numpy.random does if the seed is not specified. But I >>> would also like to print out the initial state, so I can replicate the >>> random numbers. >>> >>> Can I get a human readable or printable version of the initial state? >> >> [~] >> |13> prng = np.random.RandomState() >> >> [~] >> |14> prng2 = np.random.RandomState() >> >> [~] >> |15> prng.randint(100, size=10) >> array([74, 62, 56, 94, 86, 59, 69, 94, 42, 18]) >> >> [~] >> |16> prng2.randint(100, size=10) >> array([21, 58, 34, 55, ?9, 81, 45, ?3, 93, 62]) >> >> [~] >> |17> prng2.set_state(prng.get_state()) >> >> [~] >> |18> prng.randint(100, size=10) >> array([37, 57, ?2, ?0, 68, ?9, 75, 88, 11, ?7]) >> >> [~] >> |19> prng2.randint(100, size=10) >> array([37, 57, ?2, ?0, 68, ?9, 75, 88, 11, ?7]) >> >> [~] >> |20> prng.get_state() >> >> ('MT19937', >> ?array([1368120112, ?957462593, 2623310617, 4207155283, ?446940397, >> ? ? ? 3506388262, 4104366519, ?371500243, 4029407620, ?899392379, >> .... >> ? ? ? 1843090101, 2484333397, 4085469971, ?306955884, ? 23307203, >> ? ? ? 1640066622, ? 48186677, ?637144011, ?854838500], dtype=uint32), >> ?26, >> ?0, >> ?2.5933437794758841e-288) >> >>> Alternatively, what's a good way to ?randomly generate an initial >>> state? >> >> np.random.RandomState() will do the best thing on each platform. > > Thanks, I need to think about whether I want to save the full > get_state() tuple (or cheat) Save the whole thing. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From sierra_mtnview at sbcglobal.net Wed Sep 22 12:16:36 2010 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Wed, 22 Sep 2010 09:16:36 -0700 Subject: [Numpy-discussion] Removing numpy numpy-1.1.0 in Python 2.5 In-Reply-To: References: <4C99395B.2010608@sbcglobal.net> <4C996AC1.2060902@sbcglobal.net> Message-ID: <4C9A2BE4.4020009@sbcglobal.net> I've considered it, but it's way too time consuming to work out the details. I spent a week some time ago dealing with a simple test program, maybe 10 lines of code, trying to get it to work, which I think I did. I just need to work what I've got. On 9/21/2010 7:49 PM, josef.pktd at gmail.com wrote: > On Tue, Sep 21, 2010 at 10:32 PM, Wayne Watson > wrote: >> I don't see it in Add/Remove. Whoops. It is on this Win7 machine. >> I need to check my XP machine. I'll be back when I figure out if that's >> right. My not always recollection was that it was not on Add/Remove >> under XP. >> >> I'm dealing with people who barely understand software, so keeping this >> simple is the best thing to do. This version of the application is the >> last, and we might as well treat it as frozen. I really don't want to >> try to correct people's install problems or attempts to move ahead with >> the various libraries we use over the internet, so simple is it. > If you want to have a frozen application, the best might be to use > py2exe, but maybe not worth the effort, if they don't mess with the > installed python packages. > > Josef > > >> On 9/21/2010 4:38 PM, josef.pktd at gmail.com wrote: >>> On Tue, Sep 21, 2010 at 7:01 PM, Wayne Watson >>> wrote: >>>> I may have bounced a similar problem around here a few months ago, but >>>> this one is a bit more important to get an answer for. >>>> >>>> I'm about to distribute some report programs to three people who work >>>> together and should be using the same version of Python and libraries >>>> that I use, and a common application to us. However, to create two new >>>> report programs they do not have, I need to make sure we do have the >>>> same versions, and in particular need to use the numpy 1.2.0 I used for >>>> the report programs. >>>> >>>> To replace the old numpy and make sure we are all on on the same wave >>>> length, I wrote the description below. I want to know if what I say >>>> about removing 1.1.0 and installing 1.2.0 is correct. I think deleting >>>> the folder as below is the right way to go. numpy is not in Control >>>> Panel Add/Remove. >>> I have some numpy versions in there >>> >>> I would recommend to first check Add/Remove, and delete from there and >>> then delete the numpy folder in site-packages if it is still there. >>> >>> Why don't you upgrade to the latest release. If you are already >>> upgrading, jumping to the latest might be better. >>> >>> Josef >>> >>> >>> Comments? >>>> ================================= >>>> Required Libraries for Sentinel Report Programs for Win XP >>>> ... >>>> You will need to replace your numpy-1.1.0 library. This is accomplished by >>>> first removing numpy-1.1.0 from the Python library located at >>>> >>>> C:\Python2.5\Lib\site-packages. >>>> >>>> Delete the numpy folder. >>>> >>>> Next execute the numpy-1.2.0-win32-superpack-python2.5.exe to >>>> install numpy 1.2.0. >>>> >>>> Try sentuser.py to make sure it runs properly. If not, contact me. >>>> ================================== >>>> >>>> -- >>>> Wayne Watson (Watson Adventures, Prop., Nevada City, CA) >>>> >>>> (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) >>>> Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet >>>> >>>> Want to lose weight? Move to the equator. You'll weigh >>>> about 0.1% less than at the poles. Centrifugal force >>>> is directed opposite gravity there--zero at the poles. >>>> -- WTW >>>> >>>> >>>> Web Page: >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> -- >> Wayne Watson (Watson Adventures, Prop., Nevada City, CA) >> >> (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) >> Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet >> >> Want to lose weight? Move to the equator. You'll weigh >> about 0.1% less than at the poles. Centrifugal force >> is directed opposite gravity there--zero at the poles. >> -- WTW >> >> >> Web Page: >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet Want to lose weight? Move to the equator. You'll weigh about 0.1% less than at the poles. Centrifugal force is directed opposite gravity there--zero at the poles. -- WTW Web Page: From charlesr.harris at gmail.com Wed Sep 22 12:48:33 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 22 Sep 2010 10:48:33 -0600 Subject: [Numpy-discussion] [PATCH] F2Py on Python 3 In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 8:35 AM, Lisandro Dalcin wrote: > It seems that lib2to3 does not process the main f2py bootstrap script > that gets autogenerated by f2py's setup.py. The trivial patch below > replaces the print statements with sys.stderr.write() calls. After > that change, f2py works just fine in Python 3.2 > > > Index: numpy/f2py/setup.py > =================================================================== > --- numpy/f2py/setup.py (revision 8716) > +++ numpy/f2py/setup.py (working copy) > @@ -62,7 +62,7 @@ > except ValueError: pass > os.environ["NO_SCIPY_IMPORT"]="f2py" > if mode=="g3-numpy": > - print >> sys.stderr, "G3 f2py support is not implemented, yet." > + sys.stderr.write("G3 f2py support is not implemented, yet.\n") > sys.exit(1) > elif mode=="2e-numeric": > from f2py2e import main > @@ -72,7 +72,7 @@ > elif mode=="2e-numpy": > from numpy.f2py import main > else: > - print >> sys.stderr, "Unknown mode:",`mode` > + sys.stderr.write("Unknown mode: '%s'\n" % mode) > sys.exit(1) > main() > '''%(os.path.basename(sys.executable))) > > > I'm also wondering if we shouldn't raise an error instead of writing to stderr. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Wed Sep 22 13:28:08 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Wed, 22 Sep 2010 12:28:08 -0500 Subject: [Numpy-discussion] Removing numpy numpy-1.1.0 in Python 2.5 In-Reply-To: <4C9A2BE4.4020009@sbcglobal.net> References: <4C99395B.2010608@sbcglobal.net> <4C996AC1.2060902@sbcglobal.net> <4C9A2BE4.4020009@sbcglobal.net> Message-ID: <4C9A3CA8.5060406@gmail.com> On 09/22/2010 11:16 AM, Wayne Watson wrote: > I've considered it, but it's way too time consuming to work out the > details. I spent a week some time ago dealing with a simple test > program, maybe 10 lines of code, trying to get it to work, which I think > I did. I just need to work what I've got. > > On 9/21/2010 7:49 PM, josef.pktd at gmail.com wrote: >> On Tue, Sep 21, 2010 at 10:32 PM, Wayne Watson >> wrote: >>> I don't see it in Add/Remove. Whoops. It is on this Win7 machine. >>> I need to check my XP machine. I'll be back when I figure out if that's >>> right. My not always recollection was that it was not on Add/Remove >>> under XP. >>> >>> I'm dealing with people who barely understand software, so keeping this >>> simple is the best thing to do. This version of the application is the >>> last, and we might as well treat it as frozen. I really don't want to >>> try to correct people's install problems or attempts to move ahead with >>> the various libraries we use over the internet, so simple is it. >> If you want to have a frozen application, the best might be to use >> py2exe, but maybe not worth the effort, if they don't mess with the >> installed python packages. >> >> Josef >> >> >>> On 9/21/2010 4:38 PM, josef.pktd at gmail.com wrote: >>>> On Tue, Sep 21, 2010 at 7:01 PM, Wayne Watson >>>> wrote: >>>>> I may have bounced a similar problem around here a few months ago, but >>>>> this one is a bit more important to get an answer for. >>>>> >>>>> I'm about to distribute some report programs to three people who work >>>>> together and should be using the same version of Python and libraries >>>>> that I use, and a common application to us. However, to create two new >>>>> report programs they do not have, I need to make sure we do have the >>>>> same versions, and in particular need to use the numpy 1.2.0 I used for >>>>> the report programs. >>>>> >>>>> To replace the old numpy and make sure we are all on on the same wave >>>>> length, I wrote the description below. I want to know if what I say >>>>> about removing 1.1.0 and installing 1.2.0 is correct. I think deleting >>>>> the folder as below is the right way to go. numpy is not in Control >>>>> Panel Add/Remove. >>>> I have some numpy versions in there >>>> >>>> I would recommend to first check Add/Remove, and delete from there and >>>> then delete the numpy folder in site-packages if it is still there. >>>> >>>> Why don't you upgrade to the latest release. If you are already >>>> upgrading, jumping to the latest might be better. >>>> >>>> Josef >>>> >>>> >>>> Comments? >>>>> ================================= >>>>> Required Libraries for Sentinel Report Programs for Win XP >>>>> ... >>>>> You will need to replace your numpy-1.1.0 library. This is accomplished by >>>>> first removing numpy-1.1.0 from the Python library located at >>>>> >>>>> C:\Python2.5\Lib\site-packages. >>>>> >>>>> Delete the numpy folder. >>>>> >>>>> Next execute the numpy-1.2.0-win32-superpack-python2.5.exe to >>>>> install numpy 1.2.0. >>>>> >>>>> Try sentuser.py to make sure it runs properly. If not, contact me. >>>>> ================================== >>>>> >>>>> -- >>>>> Wayne Watson (Watson Adventures, Prop., Nevada City, CA) >>>>> >>>>> (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) >>>>> Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet >>>>> >>>>> Want to lose weight? Move to the equator. You'll weigh >>>>> about 0.1% less than at the poles. Centrifugal force >>>>> is directed opposite gravity there--zero at the poles. >>>>> -- WTW >>>>> >>>>> >>>>> Web Page: >>>>> >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> >>> -- >>> Wayne Watson (Watson Adventures, Prop., Nevada City, CA) >>> >>> (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) >>> Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet >>> >>> Want to lose weight? Move to the equator. You'll weigh >>> about 0.1% less than at the poles. Centrifugal force >>> is directed opposite gravity there--zero at the poles. >>> -- WTW >>> >>> >>> Web Page: >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> On my old Windows XP (service pack 2!) running Python 2.5.2, running 'numpy-1.1.0-win32-superpack-python2.5.exe' registers numpy (Python 2.5 numpy-1.1.0). So you really have to remove numpy from the registry - usually via the 'Add/Remove programs' and then install the new version. (There is probably some script that can do that like these that I have zero clue if these will work that I got from a google search http://mcpmag.com/articles/2005/05/09/automated-uninstall.aspx, http://www.ehow.com/how_5185226_uninstall-software-vb-script.html) Failing that it gets rather dangerous : Just install 'numpy-1.2.0-win32-superpack-python2.5.exe' without removing the previous version. But that WILL leave a minefield of 'orphan' files. So if you delete the numpy egg file and directory in 'C:\Python2.5\Lib\site-packages\', numpy entry is still in the registry (appears in 'Add/Remove programs'). Then you can then install 'numpy-1.2.0-win32-superpack-python2.5.exe' and get a new numpy entry in the registry. There may be still traces of numpy in the registry but probably not sufficient to create major problems. Bruce From dalcinl at gmail.com Wed Sep 22 14:08:21 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 22 Sep 2010 15:08:21 -0300 Subject: [Numpy-discussion] [PATCH] F2Py on Python 3 In-Reply-To: References: Message-ID: On 22 September 2010 13:48, Charles R Harris wrote: > > > On Wed, Sep 22, 2010 at 8:35 AM, Lisandro Dalcin wrote: >> >> It seems that lib2to3 does not process the main f2py bootstrap script >> that gets autogenerated by f2py's setup.py. The trivial patch below >> replaces the print statements with sys.stderr.write() calls. After >> that change, f2py works just fine in Python 3.2 >> >> >> Index: numpy/f2py/setup.py >> =================================================================== >> --- numpy/f2py/setup.py (revision 8716) >> +++ numpy/f2py/setup.py (working copy) >> @@ -62,7 +62,7 @@ >> ? ? except ValueError: pass >> ?os.environ["NO_SCIPY_IMPORT"]="f2py" >> ?if mode=="g3-numpy": >> - ? ?print >> sys.stderr, "G3 f2py support is not implemented, yet." >> + ? ?sys.stderr.write("G3 f2py support is not implemented, yet.\n") >> ? ? sys.exit(1) >> ?elif mode=="2e-numeric": >> ? ? from f2py2e import main >> @@ -72,7 +72,7 @@ >> ?elif mode=="2e-numpy": >> ? ? from numpy.f2py import main >> ?else: >> - ? ?print >> sys.stderr, "Unknown mode:",`mode` >> + ? ?sys.stderr.write("Unknown mode: '%s'\n" % mode) >> ? ? sys.exit(1) >> ?main() >> ?'''%(os.path.basename(sys.executable))) >> >> > > I'm also wondering if we shouldn't raise an error instead of writing to > stderr. > sys.exit(1) below the sys.stderr.write() is supposed to exit Python with an error status to the invoker of f2py. So I think nothing more have to be done. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From ndbecker2 at gmail.com Wed Sep 22 14:22:38 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 22 Sep 2010 14:22:38 -0400 Subject: [Numpy-discussion] printable random seed ? References: Message-ID: josef.pktd at gmail.com wrote: > On Wed, Sep 22, 2010 at 10:32 AM, Neal Becker wrote: >> josef.pktd at gmail.com wrote: >> >>> I would like to generate random numbers based on a random seed, for >>> example what numpy.random does if the seed is not specified. But I >>> would also like to print out the initial state, so I can replicate the >>> random numbers. >>> >>> Can I get a human readable or printable version of the initial state? >>> Alternatively, what's a good way to randomly generate an initial >>> state? >>> >>> I could draw an integer with randint and use it as seed. Is this the >>> best way? >>> >>> Josef >> >> import struct >> import os >> seed = struct.unpack ('I', os.urandom (4))[0] >> print seed > > >>>> os.urandom(4) > '\x02\xcf\xd5`' >>>> np.array(os.urandom(4)).view(int) > array(-452038899) >>>> import struct >>>> seed = struct.unpack ('I', os.urandom (4))[0] >>>> seed > 3650333822L >>>> np.random.seed(seed) > Traceback (most recent call last): > File "", line 1, in > np.random.seed(seed) > File "mtrand.pyx", line 593, in mtrand.RandomState.seed > (numpy\random\mtrand\mtrand.c:4786) > OverflowError: long int too large to convert to int > File to fit :) That code was used with my own random number lib based on boost From sierra_mtnview at sbcglobal.net Wed Sep 22 14:23:02 2010 From: sierra_mtnview at sbcglobal.net (Wayne Watson) Date: Wed, 22 Sep 2010 11:23:02 -0700 Subject: [Numpy-discussion] Removing numpy numpy-1.1.0 in Python 2.5 In-Reply-To: <4C9A3CA8.5060406@gmail.com> References: <4C99395B.2010608@sbcglobal.net> <4C996AC1.2060902@sbcglobal.net> <4C9A2BE4.4020009@sbcglobal.net> <4C9A3CA8.5060406@gmail.com> Message-ID: <4C9A4986.3010503@sbcglobal.net> On 9/22/2010 10:28 AM, Bruce Southey wrote: ... > > On my old Windows XP (service pack 2!) running Python 2.5.2, running > 'numpy-1.1.0-win32-superpack-python2.5.exe' registers numpy (Python 2.5 > numpy-1.1.0). > > So you really have to remove numpy from the registry - usually via the > 'Add/Remove programs' and then install the new version. (There is > probably some script that can do that like these that I have zero clue > if these will work that I got from a google search > http://mcpmag.com/articles/2005/05/09/automated-uninstall.aspx, > http://www.ehow.com/how_5185226_uninstall-software-vb-script.html) > > Failing that it gets rather dangerous : > > Just install 'numpy-1.2.0-win32-superpack-python2.5.exe' without > removing the previous version. But that WILL leave a minefield of > 'orphan' files. So if you delete the numpy egg file and directory in > 'C:\Python2.5\Lib\site-packages\', numpy entry is still in the registry > (appears in 'Add/Remove programs'). Then you can then install > 'numpy-1.2.0-win32-superpack-python2.5.exe' and get a new numpy entry in > the registry. There may be still traces of numpy in the registry but > probably not sufficient to create major problems. > > Bruce > Interesting. As it turns out, on XP numpy is in Add/Remove, so it can be removed there. Should be a problem. -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39? 15' 7" N, 121? 2' 32" W, 2700 feet Want to lose weight? Move to the equator. You'll weigh about 0.1% less than at the poles. Centrifugal force is directed opposite gravity there--zero at the poles. -- WTW Web Page: From aarchiba at physics.mcgill.ca Wed Sep 22 14:22:50 2010 From: aarchiba at physics.mcgill.ca (Anne Archibald) Date: Wed, 22 Sep 2010 14:22:50 -0400 Subject: [Numpy-discussion] slicing / indexing question In-Reply-To: <20100921232051.GA451@Tim.local> References: <20100921232051.GA451@Tim.local> Message-ID: On 21 September 2010 19:20, Timothy W. Hilton wrote: > I have an 80x1200x1200 nd.array of floats this_par. ?I have a > 1200x1200 boolean array idx, and an 80-element float array pars. ?For > each element of idx that is True, I wish to replace the corresponding > 80x1x1 slice of this_par with the elements of pars. > > I've tried lots of variations on the theme of >>>>this_par[idx[np.newaxis, ...]] = pars[:, np.newaxis, np.newaxis] > but so far, no dice. How about this? In [1]: A = np.zeros((2,3,5)) In [2]: B = np.array([1,2]) In [3]: C = np.zeros((3,5), dtype=np.bool) In [4]: C[1,1] = True In [5]: C[2,3] = True In [6]: C Out[6]: array([[False, False, False, False, False], [False, True, False, False, False], [False, False, False, True, False]], dtype=bool) In [7]: A[:,C] = B[:,np.newaxis] In [8]: A Out[8]: array([[[ 0., 0., 0., 0., 0.], [ 0., 1., 0., 0., 0.], [ 0., 0., 0., 1., 0.]], [[ 0., 0., 0., 0., 0.], [ 0., 2., 0., 0., 0.], [ 0., 0., 0., 2., 0.]]]) The key is that indexing with C replaces the two axes C is indexing with only one; boolean indexing necessarily flattens the relevant axes. You can check this with (e.g.) A[:,C].shape. Be careful with these "mixed" indexing modes (partly fancy indexing, partly slicing) as they can sometimes seem to reorder your axes for you: In [16]: np.zeros((2,3,7))[:,np.ones(5,dtype=int),np.ones(5,dtype=int)].shape Out[16]: (2, 5) In [17]: np.zeros((2,3,7))[np.ones(5,dtype=int),:,np.ones(5,dtype=int)].shape Out[17]: (5, 3) In [18]: np.zeros((2,3,7))[np.ones(5,dtype=int),np.ones(5,dtype=int),:].shape Out[18]: (5, 7) Anne From Chris.Barker at noaa.gov Wed Sep 22 15:23:05 2010 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 22 Sep 2010 12:23:05 -0700 Subject: [Numpy-discussion] Removing numpy numpy-1.1.0 in Python 2.5 In-Reply-To: <4C9A2BE4.4020009@sbcglobal.net> References: <4C99395B.2010608@sbcglobal.net> <4C996AC1.2060902@sbcglobal.net> <4C9A2BE4.4020009@sbcglobal.net> Message-ID: <4C9A5799.9070805@noaa.gov> Wayne Watson wrote: > I've considered it, but it's way too time consuming to work out the > details. I spent a week some time ago dealing with a simple test > program, maybe 10 lines of code, trying to get it to work, which I think > I did. I just need to work what I've got. There is a lot to be said for something that works already. However, I think it is generally a bad idea to distribute software based on python that relies on the standard python install with particular packages -- this will only work if you assume that that machine will not use python for anything else -- if two pieces of software both do this, but need different versions of packages, you are guaranteed breakage. RedHat did it with python way back when (I have no idea if they've changed that -- but it was pain). ESRI (GIS system) delivers the standard Windows Python version something with its GIS tools, and things break if you upgrade parts of it. Apple uses its system Python for some of its admin tools, and again, relies on who knows which versions of which packages. All this points to delivering python to your users separately from the standard install. If it's one app, use py2exe, py2app, pyInstaller, bbfreeze, whatever. If it's a collection of apps, you may want to built it on a custom python install, and/or maybe virtualenv. Anyway, if you know your users, and know they will, for sure, not use Python for anything else, then it's not a problem, but I think it's hard to know that. py2exe can be a pain, but it's usually short term pain -- once you have it working for your project, it keeps working (untill you add a new package anyway...) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From sebastian.walter at gmail.com Wed Sep 22 15:31:43 2010 From: sebastian.walter at gmail.com (Sebastian Walter) Date: Wed, 22 Sep 2010 21:31:43 +0200 Subject: [Numpy-discussion] Asking for opinions: Priops In-Reply-To: References: Message-ID: Hello Friedrich, I have read your proposal. You describe issues that I have also encountered several times. I believe that your priops approach would be an improvement over the current overloading of binary operators. That being said, I think the issue is not so much numpy but rather the way Python implements operator overloading using methods like __add__ and __radd__. Hence, your suggestion seems to be a Python Enhancement Proposal and should be discussed without any references to numpy or bugs related to numpy.set_numeric_ops. Maybe you could also have a look at Go's interfaces (Googles programming language) which seems to be somewhat related to your approach. Also, have you checked the Python mail archive? Issues like that tend to be discussed from time to time. On a more practical note: Why exactly do you use set_numeric_ops? You could also 1) use numpy.ndarrays with dtype=object 2) or create new numpy.ndarray -like class and set __array_priority__ > 2 both approaches work well for me. just my 2 cents, Sebastian On Thu, Sep 16, 2010 at 2:02 PM, Friedrich Romstedt wrote: > I just ran across the problem of priorities with ndarrays again and it > keeps biting me. ?I did once ago a workaround to get my ``undarray`` > class's methods be called when being the second operand of e.g. > + . ?But since I wrote it, always Python crashes > on exit with the message: > > Python-32(68665) malloc: *** error for object 0x239680: incorrect > checksum for freed object - object was probably modified after being > freed. > *** set a breakpoint in malloc_error_break to debug > > (Of course only if I imported the module. ?Occasionally I also > observed Bus errors, and even segfaults.) > I overloaded the numpy ops via numpy.set_numeric_ops() with > self-written classes, which are *not* derived from numpy.ufunc, and do > not resemble numpy ufuncs completely. > > So I want to do it properly this time. > > I therefore started with writing a Letter of Intent, and put it online > on http://github.com/friedrichromstedt/priops . > > Opinions? > > Friedrich > > P.S.: I will start coding anyway, but it would be nice. > > P.P.S.: The package this originates from is also online, under > http://github.com/friedrichromstedt/upy, or > http://upy.sourceforge.net. ?I will probably create a small example > script demonstrating the crash. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From rosswilliamson.spt at gmail.com Wed Sep 22 16:38:09 2010 From: rosswilliamson.spt at gmail.com (Ross Williamson) Date: Wed, 22 Sep 2010 15:38:09 -0500 Subject: [Numpy-discussion] Indexing and lookup issues Message-ID: Hi everyone I suspect this is easy but I'm stuck say I have a 1D array: t = [10,11,12] and a 2D array: id = [[0,1,0] [0,2,0] [2,0,2]] In could in IDL do y = t[id] which would produce: y = [[10,11,10] [10,12,10] [12,10,12]] i.e. use the indexes in id on the lookup array t. Is there an easy way to do this in numpy? Cheers Ross -- Ross Williamson University of Chicago Department of Astronomy & Astrophysics 773-834-9785 (office) 312-504-3051 (Cell) From aarchiba at physics.mcgill.ca Wed Sep 22 16:42:26 2010 From: aarchiba at physics.mcgill.ca (Anne Archibald) Date: Wed, 22 Sep 2010 16:42:26 -0400 Subject: [Numpy-discussion] Indexing and lookup issues In-Reply-To: References: Message-ID: On 22 September 2010 16:38, Ross Williamson wrote: > Hi everyone > > I suspect this is easy but I'm stuck > > say I have a 1D array: > > t = [10,11,12] > > and a 2D array: > > id = [[0,1,0] > [0,2,0] > [2,0,2]] > > In could in IDL do y = t[id] which would produce: > > y = [[10,11,10] > [10,12,10] > [12,10,12]] > > i.e. use the indexes in id on the lookup array t. > > Is there an easy way to do this in numpy? In [1]: t = np.array([10,11,12]) In [2]: id = np.array([[0,1,0], [0,2,0], [2,0,2]]) In [3]: t[id] Out[3]: array([[10, 11, 10], [10, 12, 10], [12, 10, 12]]) Anne From rosswilliamson.spt at gmail.com Wed Sep 22 16:46:52 2010 From: rosswilliamson.spt at gmail.com (Ross Williamson) Date: Wed, 22 Sep 2010 15:46:52 -0500 Subject: [Numpy-discussion] Indexing and lookup issues In-Reply-To: References: Message-ID: Thank you Anne I was doing something totally stupid at my end. Ross On Wed, Sep 22, 2010 at 3:42 PM, Anne Archibald wrote: > On 22 September 2010 16:38, Ross Williamson > wrote: >> Hi everyone >> >> I suspect this is easy but I'm stuck >> >> say I have a 1D array: >> >> t = [10,11,12] >> >> and a 2D array: >> >> id = [[0,1,0] >> [0,2,0] >> [2,0,2]] >> >> In could in IDL do y = t[id] which would produce: >> >> y = [[10,11,10] >> [10,12,10] >> [12,10,12]] >> >> i.e. use the indexes in id on the lookup array t. >> >> Is there an easy way to do this in numpy? > > In [1]: t = np.array([10,11,12]) > > In [2]: id = np.array([[0,1,0], [0,2,0], [2,0,2]]) > > In [3]: t[id] > Out[3]: > array([[10, 11, 10], > ? ? ? [10, 12, 10], > ? ? ? [12, 10, 12]]) > > Anne > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Ross Williamson University of Chicago Department of Astronomy & Astrophysics 773-834-9785 (office) 312-504-3051 (Cell) From charlesr.harris at gmail.com Wed Sep 22 21:11:22 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 22 Sep 2010 19:11:22 -0600 Subject: [Numpy-discussion] Asking for opinions: Priops In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 1:31 PM, Sebastian Walter < sebastian.walter at gmail.com> wrote: > Hello Friedrich, > > I have read your proposal. You describe issues that I have also > encountered several times. > I believe that your priops approach would be an improvement over the > current overloading of binary operators. > That being said, I think the issue is not so much numpy but rather the > way Python implements operator overloading using methods like __add__ > and __radd__. Hence, your suggestion seems to be a Python Enhancement > Proposal and should be discussed without any references to numpy or > bugs related to numpy.set_numeric_ops. > Maybe you could also have a look at Go's interfaces (Googles > programming language) which seems to be somewhat related to your > approach. Also, have you checked the Python mail archive? Issues like > that tend to be discussed from time to time. > > On a more practical note: Why exactly do you use set_numeric_ops? You > could also > 1) use numpy.ndarrays with dtype=object > 2) or create new numpy.ndarray -like class and set __array_priority__ > 2 > both approaches work well for me. > > just my 2 cents, > Sebastian > > > > On Thu, Sep 16, 2010 at 2:02 PM, Friedrich Romstedt > wrote: > > I just ran across the problem of priorities with ndarrays again and it > > keeps biting me. I did once ago a workaround to get my ``undarray`` > > class's methods be called when being the second operand of e.g. > > + . But since I wrote it, always Python crashes > > on exit with the message: > > > > Python-32(68665) malloc: *** error for object 0x239680: incorrect > > checksum for freed object - object was probably modified after being > > freed. > > *** set a breakpoint in malloc_error_break to debug > > > > (Of course only if I imported the module. Occasionally I also > > observed Bus errors, and even segfaults.) > > I overloaded the numpy ops via numpy.set_numeric_ops() with > > self-written classes, which are *not* derived from numpy.ufunc, and do > > not resemble numpy ufuncs completely. > > > > So I want to do it properly this time. > > > > I therefore started with writing a Letter of Intent, and put it online > > on http://github.com/friedrichromstedt/priops . > > > > Opinions? > > > > Friedrich > > > > P.S.: I will start coding anyway, but it would be nice. > > > > P.P.S.: The package this originates from is also online, under > > http://github.com/friedrichromstedt/upy, or > > http://upy.sourceforge.net. I will probably create a small example > > script demonstrating the crash. > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Sep 22 22:16:48 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 22 Sep 2010 20:16:48 -0600 Subject: [Numpy-discussion] [PATCH] F2Py on Python 3 In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 12:08 PM, Lisandro Dalcin wrote: > On 22 September 2010 13:48, Charles R Harris > wrote: > > > > > > On Wed, Sep 22, 2010 at 8:35 AM, Lisandro Dalcin > wrote: > >> > >> It seems that lib2to3 does not process the main f2py bootstrap script > >> that gets autogenerated by f2py's setup.py. The trivial patch below > >> replaces the print statements with sys.stderr.write() calls. After > >> that change, f2py works just fine in Python 3.2 > >> > >> > >> Index: numpy/f2py/setup.py > >> =================================================================== > >> --- numpy/f2py/setup.py (revision 8716) > >> +++ numpy/f2py/setup.py (working copy) > >> @@ -62,7 +62,7 @@ > >> except ValueError: pass > >> os.environ["NO_SCIPY_IMPORT"]="f2py" > >> if mode=="g3-numpy": > >> - print >> sys.stderr, "G3 f2py support is not implemented, yet." > >> + sys.stderr.write("G3 f2py support is not implemented, yet.\n") > >> sys.exit(1) > >> elif mode=="2e-numeric": > >> from f2py2e import main > >> @@ -72,7 +72,7 @@ > >> elif mode=="2e-numpy": > >> from numpy.f2py import main > >> else: > >> - print >> sys.stderr, "Unknown mode:",`mode` > >> + sys.stderr.write("Unknown mode: '%s'\n" % mode) > >> sys.exit(1) > >> main() > >> '''%(os.path.basename(sys.executable))) > >> > >> > > > > I'm also wondering if we shouldn't raise an error instead of writing to > > stderr. > > > > sys.exit(1) below the sys.stderr.write() is supposed to exit Python > with an error status to the invoker of f2py. So I think nothing more > have to be done. > > > Done in 29cccb6. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Wed Sep 22 23:14:13 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 22 Sep 2010 20:14:13 -0700 Subject: [Numpy-discussion] [PATCH] F2Py on Python 3 In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 7:16 PM, Charles R Harris wrote: > > Done in? 29cccb6. > Mmh, I think it broke something: File "/home/fperez/tmp/src/scipy/numpy/numpy/distutils/command/build.py", line 37, in run old_build.run(self) File "/usr/lib/python2.6/distutils/command/build.py", line 135, in run self.run_command(cmd_name) File "/usr/lib/python2.6/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command cmd_obj.run() File "/home/fperez/tmp/src/scipy/numpy/numpy/distutils/command/build_scripts.py", line 39, in run self.scripts = self.generate_scripts(self.scripts) File "/home/fperez/tmp/src/scipy/numpy/numpy/distutils/command/build_scripts.py", line 24, in generate_scripts script = func(build_dir) File "numpy/f2py/setup.py", line 78, in generate_f2py_py '''%(os.path.basename(sys.executable))) TypeError: not enough arguments for format string If I revert to the previous commit, it installs fine again. Cheers, f From charlesr.harris at gmail.com Thu Sep 23 00:00:00 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 22 Sep 2010 22:00:00 -0600 Subject: [Numpy-discussion] [PATCH] F2Py on Python 3 In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 9:14 PM, Fernando Perez wrote: > On Wed, Sep 22, 2010 at 7:16 PM, Charles R Harris > wrote: > > > > Done in 29cccb6. > > > > Mmh, I think it broke something: > > File "/home/fperez/tmp/src/scipy/numpy/numpy/distutils/command/build.py", > line 37, in run > old_build.run(self) > File "/usr/lib/python2.6/distutils/command/build.py", line 135, in run > self.run_command(cmd_name) > File "/usr/lib/python2.6/distutils/cmd.py", line 333, in run_command > self.distribution.run_command(command) > File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command > cmd_obj.run() > File > "/home/fperez/tmp/src/scipy/numpy/numpy/distutils/command/build_scripts.py", > line 39, in run > self.scripts = self.generate_scripts(self.scripts) > File > "/home/fperez/tmp/src/scipy/numpy/numpy/distutils/command/build_scripts.py", > line 24, in generate_scripts > script = func(build_dir) > File "numpy/f2py/setup.py", line 78, in generate_f2py_py > '''%(os.path.basename(sys.executable))) > TypeError: not enough arguments for format string > > If I revert to the previous commit, it installs fine again. > > I wondered about that, it is probably the old `mode` vs plain old mode. I had just hoped it was tested. Can you check that out? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Sep 23 00:26:43 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 22 Sep 2010 22:26:43 -0600 Subject: [Numpy-discussion] [PATCH] F2Py on Python 3 In-Reply-To: References: Message-ID: On Wed, Sep 22, 2010 at 10:00 PM, Charles R Harris < charlesr.harris at gmail.com> wrote: > > > On Wed, Sep 22, 2010 at 9:14 PM, Fernando Perez wrote: > >> On Wed, Sep 22, 2010 at 7:16 PM, Charles R Harris >> wrote: >> > >> > Done in 29cccb6. >> > >> >> Mmh, I think it broke something: >> >> File "/home/fperez/tmp/src/scipy/numpy/numpy/distutils/command/build.py", >> line 37, in run >> old_build.run(self) >> File "/usr/lib/python2.6/distutils/command/build.py", line 135, in run >> self.run_command(cmd_name) >> File "/usr/lib/python2.6/distutils/cmd.py", line 333, in run_command >> self.distribution.run_command(command) >> File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command >> cmd_obj.run() >> File >> "/home/fperez/tmp/src/scipy/numpy/numpy/distutils/command/build_scripts.py", >> line 39, in run >> self.scripts = self.generate_scripts(self.scripts) >> File >> "/home/fperez/tmp/src/scipy/numpy/numpy/distutils/command/build_scripts.py", >> line 24, in generate_scripts >> script = func(build_dir) >> File "numpy/f2py/setup.py", line 78, in generate_f2py_py >> '''%(os.path.basename(sys.executable))) >> TypeError: not enough arguments for format string >> >> If I revert to the previous commit, it installs fine again. >> >> > I wondered about that, it is probably the old `mode` vs plain old mode. I > had just hoped it was tested. Can you check that out? > > Should be fixed in 8f6114b. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Thu Sep 23 02:20:20 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 23 Sep 2010 14:20:20 +0800 Subject: [Numpy-discussion] A proposed change to rollaxis() behavior for negative 'start' values In-Reply-To: References: <4C990C0E.5090904@jhu.edu> Message-ID: On Wed, Sep 22, 2010 at 4:14 AM, Anne Archibald wrote: > Hi Ken, > > This is a tricky one. The current behaviour of rollaxis is to remove > the requested axis from the list of axes and then insert it before the > axis specified. This is exactly how python's list insertion works: > > In [1]: a = range(10) > > In [3]: a.insert(-1,'a') > > In [4]: a > Out[4]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 'a', 9] > > And indeed, there's no clean way to add something to the end of a list > using insert (apart from the obvious a.insert(len(a),'b') ). For this > you have .append(). Unfortunately numpy's rollaxis, while it agrees > with insert in its behaviour, doesn't have a move_axis_to_end. The > situation is also somewhat muddied by the fact that rollaxis also > removes the axis from the original list of axes, so that the > interpretation of index numbers is a little more subtle. But I think > your suggested behaviour would be confusing because of the conflict > with python's insert. How about allowing the string "end" as an > argument to rollaxis to specify that the axis should go at the end? > Allowing "end" is an easy solution, but note that moving an axis to the end is already possible: >>> a = np.ones((3,4,5,6)) >>> np.rollaxis(a, 2, len(a)+1).shape # roll axis to to last position (3, 4, 6, 5) Not consistent with insert though, there you would use len(a) instead of len(a)+1. It's a little ugly, but perhaps just documenting this is no worse than allowing a string or adding yet another function. Ralf > Anne > > On 21 September 2010 15:48, Ken Basye wrote: > > Hi Numpy Folks, > > A while back, I filed this ticket: > > http://projects.scipy.org/numpy/ticket/1441 suggesting a change to > > rollaxis() and some fixes to the doc and error reporting. Ralf Gommers > > suggested I float the behavior change here, so that's what I'm doing. > > > > The motivation for the change comes because it seems like there should > > be a simpler way to get some axis into the last position than to do this: > > > > >>> a = np.ones((3,4,5,6)) > > >>> b = np.rollaxis(a, axis=0, start=len(a.shape)) > > >>> b.shape > > (4, 5, 6, 3) > > > > But currently it seems there isn't because when you specify -1 as the > > 'start' argument, the axis is moved into the second-to-last position. > > My proposed change, which you can see on the ticket, would change that > > so that using -1 referred to the end position. Note that the use of > > negative 'start' arguments isn't currently documented and, in its > > current form, doesn't seem very useful. My proposal wouldn't change the > > behavior for positive 'start' values at all, and the interpretation of > > 'axis' arguments is also unaffected. > > > > If that's going to break too much code, here's a pathway that might be > > acceptable: Add a new function moveaxis() which works the way > > rollaxis() does for positive arguments but in the new way for negative > > arguments. Eventually, rollaxis could be deprecated to keep things > > tidy. This has the added advantage of using a name that seems to fit > > what the function does better - 'rollaxis' suggests a behavior like the > > roll() function which affects other axes, which isn't what happens. > > > > Thanks for listening; I'm a big fan of Numpy. > > > > Best, > > Ken Basye > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Sep 23 06:32:27 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 23 Sep 2010 04:32:27 -0600 Subject: [Numpy-discussion] A proposed change to rollaxis() behavior for negative 'start' values In-Reply-To: References: <4C990C0E.5090904@jhu.edu> Message-ID: On Thu, Sep 23, 2010 at 12:20 AM, Ralf Gommers wrote: > > > On Wed, Sep 22, 2010 at 4:14 AM, Anne Archibald < > aarchiba at physics.mcgill.ca> wrote: > >> Hi Ken, >> >> This is a tricky one. The current behaviour of rollaxis is to remove >> the requested axis from the list of axes and then insert it before the >> axis specified. This is exactly how python's list insertion works: >> >> In [1]: a = range(10) >> >> In [3]: a.insert(-1,'a') >> >> In [4]: a >> Out[4]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 'a', 9] >> >> And indeed, there's no clean way to add something to the end of a list >> using insert (apart from the obvious a.insert(len(a),'b') ). For this >> you have .append(). Unfortunately numpy's rollaxis, while it agrees >> with insert in its behaviour, doesn't have a move_axis_to_end. The >> situation is also somewhat muddied by the fact that rollaxis also >> removes the axis from the original list of axes, so that the >> interpretation of index numbers is a little more subtle. But I think >> your suggested behaviour would be confusing because of the conflict >> with python's insert. How about allowing the string "end" as an >> argument to rollaxis to specify that the axis should go at the end? >> > > Allowing "end" is an easy solution, but note that moving an axis to the end > is already possible: > > > >>> a = np.ones((3,4,5,6)) > >>> np.rollaxis(a, 2, len(a)+1).shape # roll axis to to last position > (3, 4, 6, 5) > > Not consistent with insert though, there you would use len(a) instead of > len(a)+1. It's a little ugly, but perhaps just documenting this is no worse > than allowing a string or adding yet another function. > > It is a common enough operation that it would be nice to have a less cumbersome way to specify it. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From friedrichromstedt at gmail.com Thu Sep 23 06:54:16 2010 From: friedrichromstedt at gmail.com (Friedrich Romstedt) Date: Thu, 23 Sep 2010 12:54:16 +0200 Subject: [Numpy-discussion] Asking for opinions: Priops In-Reply-To: References: Message-ID: Hi Sebastian, Thanks for your reply! 2010/9/22 Sebastian Walter : > [...] I think the issue is not so much numpy but rather the > way Python implements operator overloading using methods like __add__ > and __radd__. ?Hence, your suggestion seems to be a Python Enhancement > Proposal and should be discussed without any references to numpy or > bugs related to numpy.set_numeric_ops. Yeah, I agree, but i refrained from spamming the list with two separate mails, although I see now it would have been better, and would have brought more eyes on it. For the PEP, I will look into this and will check the mailing lists. For the technical things, I think the implementation of the operations must lie in the classes, and thus __add__ etc. are in principle okay. But as described in the new README, there is need to organise this functions. priop (note the rename, "priops" turned out to be always spelled "priop" in the source code) could define this new layer. > Maybe you could also have a look at Go's interfaces (Googles > programming language) which seems to be somewhat related to your > approach. I will try. Can you provide an URL? > On a more practical note: Why exactly do you use set_numeric_ops? You > could also > 1) use numpy.ndarrays with dtype=object This is too slow. And it's eating up memory because of the Python objects stored with all their bells and whistles. > 2) or create new numpy.ndarray -like class and set ?__array_priority__ > 2 > both approaches work well for me. I wanted to avoid exactly this because I think priop is a better approach, via set_numeric_ops(). The new URL is: http://github.com/friedrichromstedt/priop (just for the rename, which may be discussable). It contains now also an implementation, which was much less hard than expected ... Friedrich From kbasye1 at jhu.edu Thu Sep 23 10:42:58 2010 From: kbasye1 at jhu.edu (Ken Basye) Date: Thu, 23 Sep 2010 10:42:58 -0400 Subject: [Numpy-discussion] A proposed change to rollaxis(), behavior for negative 'start' values In-Reply-To: References: Message-ID: <4C9B6772.2080709@jhu.edu> Anne says: This is a tricky one. The current behaviour of rollaxis is to remove the requested axis from the list of axes and then insert it before the axis specified. This is exactly how python's list insertion works: In [1]: a = range(10) In [3]: a.insert(-1,'a') In [4]: a Out[4]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 'a', 9] And indeed, there's no clean way to add something to the end of a list using insert (apart from the obvious a.insert(len(a),'b') ). For this you have .append(). Unfortunately numpy's rollaxis, while it agrees with insert in its behaviour, doesn't have a move_axis_to_end. The situation is also somewhat muddied by the fact that rollaxis also removes the axis from the original list of axes, so that the interpretation of index numbers is a little more subtle. But I think your suggested behaviour would be confusing because of the conflict with python's insert. How about allowing the string "end" as an argument to rollaxis to specify that the axis should go at the end? Anne Ralf says: Allowing "end" is an easy solution, but note that moving an axis to the end > is already possible: > > >>> > >>> a = np.ones((3,4,5,6)) >>> > >>> np.rollaxis(a, 2, len(a)+1).shape # roll axis to to last position >>> > (3, 4, 6, 5) > > Not consistent with insert though, there you would use len(a) instead of > len(a)+1. It's a little ugly, but perhaps just documenting this is no worse > than allowing a string or adding yet another function. > > Chuck says: It is a common enough operation that it would be nice to have a less cumbersome way to specify it. Ralf, that's not going to work generally, because len(a) is a.shape[0], right? It works in the example above but if I had done >>> a = np.ones((1,2,3,4)) it wouldn't work. In my original posting, I suggested >>> np.rollaxis(a, 2, len(a.shape)) which I think will work fine, but it still seems cumbersome, as Chuck points out. I'm going to take one more tilt at the windmill. To me, the connection with Python's insert() is tenuous: insert() is a destructive operation whereas rollaxis() returns a new array with the requested change, also rollaxis is doing the delete and insert in one step. I know the implementation uses insert() and delete(), but I think the natural way to think about rollaxis() is in terms of moving an axis from one position to another and there is no Python analog to that. Anne's comment that "the situation is somehat muddied..." is precisely why I think the change is worth making - if the meaning of the 'start' argument were changed from "The axis is rolled until it lies before this position." to "The axis is rolled until it lies *at* this position." then it seems to me there's no confusion. The current description "The axis is rolled until it lies before this position." is ambiguous because "this position" seems like it means "position in the shape of the array" but then what's the position before 0? So how about "The axis is rolled until it lies before the axis currently at this position." That's much closer, but now it's hard to make sense of using len(a.shape) as an argument, since there is no axis currently at len(a.shape). To really capture the muddiness, we need to say something like "The axis is rolled until it lies before the axis currently at this position, unless the value is len(a.shape), in which case the axis is rolled to the end" - yuck. Again, note that the proposed change applies only to positions specified with negative indices; positive indices will all work as before. Best, Ken -------------- next part -------------- An HTML attachment was scrubbed... URL: From dagss at student.matnat.uio.no Thu Sep 23 11:11:41 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 23 Sep 2010 17:11:41 +0200 Subject: [Numpy-discussion] Asking for opinions: Priops In-Reply-To: References: Message-ID: <4C9B6E2D.5050706@student.matnat.uio.no> Friedrich Romstedt wrote: > I just ran across the problem of priorities with ndarrays again and it > keeps biting me. I did once ago a workaround to get my ``undarray`` > class's methods be called when being the second operand of e.g. > + . But since I wrote it, always Python crashes > on exit with the message: > > Python-32(68665) malloc: *** error for object 0x239680: incorrect > checksum for freed object - object was probably modified after being > freed. > *** set a breakpoint in malloc_error_break to debug > > (Of course only if I imported the module. Occasionally I also > observed Bus errors, and even segfaults.) > I overloaded the numpy ops via numpy.set_numeric_ops() with > self-written classes, which are *not* derived from numpy.ufunc, and do > not resemble numpy ufuncs completely. > > So I want to do it properly this time. > > I therefore started with writing a Letter of Intent, and put it online > on http://github.com/friedrichromstedt/priops . > > Opinions? > I haven't had time to go into the details, but I love the fact that somebody is about to deal with this problem, it's been bothering me as well. Something that is slightly related that one might as well test is the Sage coercion model. If you haven't, you may want to see if you get anything useful out of http://wiki.sagemath.org/coercion. Essentially, perhaps what you have sketched up + an ability to extend the graph with object conversion routes would be perfect for my own uses. So you can define a function with overloads (A, B) and (A, C), but also that objects of type D can be converted to C (and how). For instance, consider: np.array([1,2,3]) + [1,2,3] Here, list-> array could be handled through a defined coercion to array, rather than having to add an overload for list for every method taking an array. Dag Sverre From dagss at student.matnat.uio.no Thu Sep 23 11:14:06 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 23 Sep 2010 17:14:06 +0200 Subject: [Numpy-discussion] Asking for opinions: Priops In-Reply-To: <4C9B6E2D.5050706@student.matnat.uio.no> References: <4C9B6E2D.5050706@student.matnat.uio.no> Message-ID: <4C9B6EBE.6090304@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Friedrich Romstedt wrote: > >> I just ran across the problem of priorities with ndarrays again and it >> keeps biting me. I did once ago a workaround to get my ``undarray`` >> class's methods be called when being the second operand of e.g. >> + . But since I wrote it, always Python crashes >> on exit with the message: >> >> Python-32(68665) malloc: *** error for object 0x239680: incorrect >> checksum for freed object - object was probably modified after being >> freed. >> *** set a breakpoint in malloc_error_break to debug >> >> (Of course only if I imported the module. Occasionally I also >> observed Bus errors, and even segfaults.) >> I overloaded the numpy ops via numpy.set_numeric_ops() with >> self-written classes, which are *not* derived from numpy.ufunc, and do >> not resemble numpy ufuncs completely. >> >> So I want to do it properly this time. >> >> I therefore started with writing a Letter of Intent, and put it online >> on http://github.com/friedrichromstedt/priops . >> >> Opinions? >> >> > I haven't had time to go into the details, but I love the fact that > somebody is about to deal with this problem, it's been bothering me as well. > > Something that is slightly related that one might as well test is the > Sage coercion model. If you haven't, you may want to see if you get > anything useful out of http://wiki.sagemath.org/coercion. > > Essentially, perhaps what you have sketched up + an ability to extend > the graph with object conversion routes would be perfect for my own > uses. So you can define a function with overloads (A, B) and (A, C), but > also that objects of type D can be converted to C (and how). For > instance, consider: > > np.array([1,2,3]) + [1,2,3] > > Here, list-> array could be handled through a defined coercion to array, > rather than having to add an overload for list for every method taking > an array. > > Btw, I was just using numPy as an example, not suggesting that NumPy adopt priops (we can "eliminate" NumPy through __array_priority__ , as long as everyone else use priops?). MyObject() + [1,2,3], with MyObject only knowing about np.ndarray, would have been better... Dag Sverre From aarchiba at physics.mcgill.ca Thu Sep 23 11:32:09 2010 From: aarchiba at physics.mcgill.ca (Anne Archibald) Date: Thu, 23 Sep 2010 11:32:09 -0400 Subject: [Numpy-discussion] A proposed change to rollaxis() behavior for negative 'start' values In-Reply-To: References: <4C990C0E.5090904@jhu.edu> Message-ID: On 23 September 2010 02:20, Ralf Gommers wrote: > > > On Wed, Sep 22, 2010 at 4:14 AM, Anne Archibald > wrote: >> >> Hi Ken, >> >> This is a tricky one. The current behaviour of rollaxis is to remove >> the requested axis from the list of axes and then insert it before the >> axis specified. This is exactly how python's list insertion works: >> >> In [1]: a = range(10) >> >> In [3]: a.insert(-1,'a') >> >> In [4]: a >> Out[4]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 'a', 9] >> >> And indeed, there's no clean way to add something to the end of a list >> using insert (apart from the obvious a.insert(len(a),'b') ). For this >> you have .append(). Unfortunately numpy's rollaxis, while it agrees >> with insert in its behaviour, doesn't have a move_axis_to_end. The >> situation is also somewhat muddied by the fact that rollaxis also >> removes the axis from the original list of axes, so that the >> interpretation of index numbers is a little more subtle. But I think >> your suggested behaviour would be confusing because of the conflict >> with python's insert. How about allowing the string "end" as an >> argument to rollaxis to specify that the axis should go at the end? > > Allowing "end" is an easy solution, but note that moving an axis to the end > is already possible: > >>>> a = np.ones((3,4,5,6)) >>>> np.rollaxis(a, 2, len(a)+1).shape? # roll axis to to last position > (3, 4, 6, 5) > > Not consistent with insert though, there you would use len(a) instead of > len(a)+1. It's a little ugly, but perhaps just documenting this is no worse > than allowing a string or adding yet another function. Just a quick correction: len(a) gives a.shape[0], while what you want is actually len(a.shape). So: In [1]: a = np.zeros((2,3,4,5,6)) In [2]: len(a) Out[2]: 2 In [8]: np.rollaxis(a,0,len(a.shape)).shape Out[8]: (3, 4, 5, 6, 2) So it behaves just like insert. But "len(a.shape)" is rather cumbersome, especially if you haven't given a a name yet: d = (a+b*c).rollaxis(2,'end') Anne > Ralf > > >> >> Anne >> >> On 21 September 2010 15:48, Ken Basye wrote: >> > Hi Numpy Folks, >> > ?A while back, I filed this ticket: >> > http://projects.scipy.org/numpy/ticket/1441 ?suggesting a change to >> > rollaxis() and some fixes to the doc and error reporting. ?Ralf Gommers >> > suggested I float the behavior change here, so that's what I'm doing. >> > >> > The motivation for the change comes because it seems like there should >> > be a simpler way to get some axis into the last position than to do >> > this: >> > >> > ?>>> a = np.ones((3,4,5,6)) >> > ?>>> b = np.rollaxis(a, axis=0, start=len(a.shape)) >> > ?>>> b.shape >> > (4, 5, 6, 3) >> > >> > But currently it seems there isn't because when you specify -1 as the >> > 'start' argument, the axis is moved into the second-to-last position. >> > My proposed change, which you can see on the ticket, would change that >> > so that using -1 referred to the end position. ?Note that the use of >> > negative 'start' arguments isn't currently documented and, in its >> > current form, doesn't seem very useful. ?My proposal wouldn't change the >> > behavior for positive 'start' values at all, and the interpretation of >> > 'axis' arguments is also unaffected. >> > >> > If that's going to break too much code, here's a pathway that might be >> > acceptable: ?Add a new function moveaxis() which works the way >> > rollaxis() does for positive arguments but in the new way for negative >> > arguments. ?Eventually, rollaxis could be deprecated to keep things >> > tidy. ?This has the added advantage of using a name that seems to fit >> > what the function does better - 'rollaxis' suggests a behavior like the >> > roll() function which affects other axes, which isn't what happens. >> > >> > Thanks for listening; I'm a big fan of Numpy. >> > >> > Best, >> > ? Ken Basye >> > >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From rmay31 at gmail.com Thu Sep 23 11:37:50 2010 From: rmay31 at gmail.com (Ryan May) Date: Thu, 23 Sep 2010 10:37:50 -0500 Subject: [Numpy-discussion] A proposed change to rollaxis() behavior for negative 'start' values In-Reply-To: References: <4C990C0E.5090904@jhu.edu> Message-ID: On Thu, Sep 23, 2010 at 10:32 AM, Anne Archibald wrote: > Just a quick correction: len(a) gives a.shape[0], while what you want > is actually len(a.shape). So: > > In [1]: a = np.zeros((2,3,4,5,6)) > > In [2]: len(a) > Out[2]: 2 > > In [8]: np.rollaxis(a,0,len(a.shape)).shape > Out[8]: (3, 4, 5, 6, 2) > > So it behaves just like insert. But "len(a.shape)" is rather > cumbersome, especially if you haven't given a a name yet: It's available as a.ndim Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma From njs at pobox.com Thu Sep 23 12:33:05 2010 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 23 Sep 2010 09:33:05 -0700 Subject: [Numpy-discussion] A proposed change to rollaxis() behavior for negative 'start' values In-Reply-To: <4C990C0E.5090904@jhu.edu> References: <4C990C0E.5090904@jhu.edu> Message-ID: On Tue, Sep 21, 2010 at 12:48 PM, Ken Basye wrote: > If that's going to break too much code, here's a pathway that might be > acceptable: ?Add a new function moveaxis() which works the way > rollaxis() does for positive arguments but in the new way for negative > arguments. ?Eventually, rollaxis could be deprecated to keep things > tidy. ?This has the added advantage of using a name that seems to fit > what the function does better - 'rollaxis' suggests a behavior like the > roll() function which affects other axes, which isn't what happens. My 2 cents: +1 on a new function, but I'd change the behavior for positive arguments too. Currently, the API is (AFAICT): You give the index of the axis you want to move, and you give the index of the axis that you want the first axis to be moved in front of. This is super confusing! I propose that a much better API would be: You give the index of the axis you want to move, and you give the index you *want* that axis to have. So we'd have the invariant: b = np.moveaxis(a, i, j) assert a.shape[i] == b.shape[j] This is way easier to think about, at least for me. And it solves the problem with negative indices too. BTW, note that that the documentation for rollaxis is actually self-contradictory at the moment: http://docs.scipy.org/doc/numpy/reference/generated/numpy.rollaxis.html At the top it seems to document the behavior that I propose ("Roll the specified axis backwards, until it lies *in a given* position."), and then in the details it describes the actual behavior("The axis is rolled until it lies *before* this position"). I take this as further evidence that the current behavior is unnatural and confusing :-). -- Nathaniel From hilton at meteo.psu.edu Thu Sep 23 12:53:54 2010 From: hilton at meteo.psu.edu (Timothy W. Hilton) Date: Thu, 23 Sep 2010 09:53:54 -0700 Subject: [Numpy-discussion] slicing / indexing question In-Reply-To: References: <20100921232051.GA451@Tim.local> Message-ID: <20100923165354.GA4498@Tim.local> Hi Brett, Josef, G?khan, and Anne, Many thanks for the suggestions! This indexing problem was not as straight-forward as I had anticipated. I have it working now, thanks to your help. G?khan, I'm driving an ecosystem--atmosphere carbon dioxide flux model with MODIS measurements, so this_par and pars are parameter values that I need to combine with the measurements. The arrays represent time series of model parameters and MODIS reflectances and land surface products (enhanced vegetation index, land cover class, vegetation dynamics) at 1 km resolution for 1200 km by 1200 km "tiles". There are lots of IDL and Matlab folks in my department too. I've been using R and, more recently, Scipy... Being able to test out code offline on my own machine without worrying about licenses (not only cost but also talking to license servers, availability of site licenses, etc.) is a big big help! Along those lines, many thanks also to all who have put so much time and energy into developing python and Scipy. It's a great platform! -Tim On Wed, Sep 2010, 22 at 02:22:50PM -0400, Anne Archibald wrote: > On 21 September 2010 19:20, Timothy W. Hilton wrote: > > > I have an 80x1200x1200 nd.array of floats this_par. ?I have a > > 1200x1200 boolean array idx, and an 80-element float array pars. ?For > > each element of idx that is True, I wish to replace the corresponding > > 80x1x1 slice of this_par with the elements of pars. > > > > I've tried lots of variations on the theme of > >>>>this_par[idx[np.newaxis, ...]] = pars[:, np.newaxis, np.newaxis] > > but so far, no dice. > > How about this? > > > In [1]: A = np.zeros((2,3,5)) > > In [2]: B = np.array([1,2]) > > In [3]: C = np.zeros((3,5), dtype=np.bool) > > In [4]: C[1,1] = True > > In [5]: C[2,3] = True > > In [6]: C > Out[6]: > array([[False, False, False, False, False], > [False, True, False, False, False], > [False, False, False, True, False]], dtype=bool) > > In [7]: A[:,C] = B[:,np.newaxis] > > In [8]: A > Out[8]: > array([[[ 0., 0., 0., 0., 0.], > [ 0., 1., 0., 0., 0.], > [ 0., 0., 0., 1., 0.]], > > [[ 0., 0., 0., 0., 0.], > [ 0., 2., 0., 0., 0.], > [ 0., 0., 0., 2., 0.]]]) > > The key is that indexing with C replaces the two axes C is indexing > with only one; boolean indexing necessarily flattens the relevant > axes. You can check this with (e.g.) A[:,C].shape. > > Be careful with these "mixed" indexing modes (partly fancy indexing, > partly slicing) as they can sometimes seem to reorder your axes for > you: > > In [16]: np.zeros((2,3,7))[:,np.ones(5,dtype=int),np.ones(5,dtype=int)].shape > Out[16]: (2, 5) > > In [17]: np.zeros((2,3,7))[np.ones(5,dtype=int),:,np.ones(5,dtype=int)].shape > Out[17]: (5, 3) > > In [18]: np.zeros((2,3,7))[np.ones(5,dtype=int),np.ones(5,dtype=int),:].shape > Out[18]: (5, 7) > > Anne > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From stefan at sun.ac.za Thu Sep 23 13:47:25 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 23 Sep 2010 19:47:25 +0200 Subject: [Numpy-discussion] Commit mailing list Message-ID: Hi all, The commit mailing list is running again. Regards St?fan From dalcinl at gmail.com Thu Sep 23 21:22:33 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 23 Sep 2010 22:22:33 -0300 Subject: [Numpy-discussion] [PATCH] F2Py on Python 3 In-Reply-To: References: Message-ID: On 23 September 2010 01:26, Charles R Harris wrote: > > > On Wed, Sep 22, 2010 at 10:00 PM, Charles R Harris > wrote: >> >> >> On Wed, Sep 22, 2010 at 9:14 PM, Fernando Perez >> wrote: >>> >>> On Wed, Sep 22, 2010 at 7:16 PM, Charles R Harris >>> wrote: >>> > >>> > Done in? 29cccb6. >>> > >>> >>> Mmh, I think it broke something: >>> >>> ?File >>> "/home/fperez/tmp/src/scipy/numpy/numpy/distutils/command/build.py", >>> line 37, in run >>> ? ?old_build.run(self) >>> ?File "/usr/lib/python2.6/distutils/command/build.py", line 135, in run >>> ? ?self.run_command(cmd_name) >>> ?File "/usr/lib/python2.6/distutils/cmd.py", line 333, in run_command >>> ? ?self.distribution.run_command(command) >>> ?File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command >>> ? ?cmd_obj.run() >>> ?File >>> "/home/fperez/tmp/src/scipy/numpy/numpy/distutils/command/build_scripts.py", >>> line 39, in run >>> ? ?self.scripts = self.generate_scripts(self.scripts) >>> ?File >>> "/home/fperez/tmp/src/scipy/numpy/numpy/distutils/command/build_scripts.py", >>> line 24, in generate_scripts >>> ? ?script = func(build_dir) >>> ?File "numpy/f2py/setup.py", line 78, in generate_f2py_py >>> ? ?'''%(os.path.basename(sys.executable))) >>> TypeError: not enough arguments for format string >>> >>> If I revert to the previous commit, it installs fine again. >>> >> >> I wondered about that, it is probably the old `mode` vs? plain old mode. I >> had just hoped it was tested. Can you check that out? >> > > Should be fixed in 8f6114b. > I had just hoped it was tested... [sorry, I could not resist! ;-) ] diff --git a/numpy/f2py/setup.py b/numpy/f2py/setup.py index aac3d33..37aab19 100644 --- a/numpy/f2py/setup.py +++ b/numpy/f2py/setup.py @@ -62,7 +62,7 @@ for mode in ["g3-numpy", "2e-numeric", "2e-numarray", "2e-numpy except ValueError: pass os.environ["NO_SCIPY_IMPORT"]="f2py" if mode=="g3-numpy": - sys.stderr.write("G3 f2py support is not implemented, yet.\n") + sys.stderr.write("G3 f2py support is not implemented, yet.\\n") sys.exit(1) elif mode=="2e-numeric": from f2py2e import main @@ -72,7 +72,7 @@ elif mode=="2e-numarray": elif mode=="2e-numpy": from numpy.f2py import main else: - sys.stderr.write("Unknown mode: " + repr(mode) + "\n") + sys.stderr.write("Unknown mode: " + repr(mode) + "\\n") sys.exit(1) main() '''%(os.path.basename(sys.executable))) -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From charlesr.harris at gmail.com Thu Sep 23 22:34:44 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 23 Sep 2010 20:34:44 -0600 Subject: [Numpy-discussion] [PATCH] F2Py on Python 3 In-Reply-To: References: Message-ID: On Thu, Sep 23, 2010 at 7:22 PM, Lisandro Dalcin wrote: > On 23 September 2010 01:26, Charles R Harris > wrote: > > > > > > On Wed, Sep 22, 2010 at 10:00 PM, Charles R Harris > > wrote: > >> > >> > >> On Wed, Sep 22, 2010 at 9:14 PM, Fernando Perez > >> wrote: > >>> > >>> On Wed, Sep 22, 2010 at 7:16 PM, Charles R Harris > >>> wrote: > >>> > > >>> > Done in 29cccb6. > >>> > > >>> > >>> Mmh, I think it broke something: > >>> > >>> File > >>> "/home/fperez/tmp/src/scipy/numpy/numpy/distutils/command/build.py", > >>> line 37, in run > >>> old_build.run(self) > >>> File "/usr/lib/python2.6/distutils/command/build.py", line 135, in run > >>> self.run_command(cmd_name) > >>> File "/usr/lib/python2.6/distutils/cmd.py", line 333, in run_command > >>> self.distribution.run_command(command) > >>> File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command > >>> cmd_obj.run() > >>> File > >>> > "/home/fperez/tmp/src/scipy/numpy/numpy/distutils/command/build_scripts.py", > >>> line 39, in run > >>> self.scripts = self.generate_scripts(self.scripts) > >>> File > >>> > "/home/fperez/tmp/src/scipy/numpy/numpy/distutils/command/build_scripts.py", > >>> line 24, in generate_scripts > >>> script = func(build_dir) > >>> File "numpy/f2py/setup.py", line 78, in generate_f2py_py > >>> '''%(os.path.basename(sys.executable))) > >>> TypeError: not enough arguments for format string > >>> > >>> If I revert to the previous commit, it installs fine again. > >>> > >> > >> I wondered about that, it is probably the old `mode` vs plain old mode. > I > >> had just hoped it was tested. Can you check that out? > >> > > > > Should be fixed in 8f6114b. > > > > I had just hoped it was tested... [sorry, I could not resist! ;-) ] > > diff --git a/numpy/f2py/setup.py b/numpy/f2py/setup.py > index aac3d33..37aab19 100644 > --- a/numpy/f2py/setup.py > +++ b/numpy/f2py/setup.py > @@ -62,7 +62,7 @@ for mode in ["g3-numpy", "2e-numeric", > "2e-numarray", "2e-numpy > except ValueError: pass > os.environ["NO_SCIPY_IMPORT"]="f2py" > if mode=="g3-numpy": > - sys.stderr.write("G3 f2py support is not implemented, yet.\n") > + sys.stderr.write("G3 f2py support is not implemented, yet.\\n") > sys.exit(1) > elif mode=="2e-numeric": > from f2py2e import main > @@ -72,7 +72,7 @@ elif mode=="2e-numarray": > elif mode=="2e-numpy": > from numpy.f2py import main > else: > - sys.stderr.write("Unknown mode: " + repr(mode) + "\n") > + sys.stderr.write("Unknown mode: " + repr(mode) + "\\n") > sys.exit(1) > main() > '''%(os.path.basename(sys.executable))) > > Arrggghhhh, I forgot the double backslashes :-( Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Fri Sep 24 01:14:05 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Fri, 24 Sep 2010 01:14:05 -0400 Subject: [Numpy-discussion] dtype from |S10 to object in array? In-Reply-To: <4C999C53.9060502@gmail.com> References: <4C999C53.9060502@gmail.com> Message-ID: Assignment will never change the data-type of an already allocated array. So, x['f2'] = y will try to convert the array to 'S10'. On my system I get: array([(1, 2.0, 'Hello'), (2, 3.0, 'World')], dtype=[('f0', ' Dear All, > > See below code pls, > > import sicpy > import numpy as np > > x = np.zeros((2,),dtype=('i4,f4,a10')) > x[:] = [(1,2.,'Hello'),(2,3.,"World")] > > y = x['f2'] > #array(['Hello', 'World'], > dtype='|S10') > > x['f2'] = y > x > #array([(1, 2.0, 'Hello'), (2, 3.0, 'World')], > dtype=[('f0', ' > y = y.astype('object') > y > array([Hello, World], dtype=object) > > > x['f2'] = y > array([(1, 2.0, 'HellWorld'), (2, 3.0, '\x00\x00\x00\x00\x00\x00\x18')], > dtype=[('f0', ' > ##here comes the problem: the f2 col type has not been changed and the > data is not I wanted... > ---------------------------------------------------------------------------- > > here is why I need using this: > suppose I have a datasource, csv, sql based db or what ever look like this: > > 1, 2.0, 'Hello' > 2, 3.0, 'World' > 3, 2.0, 'other string' > > I want to read them to a numpy array and process it's columns, it has no > problem for processing the float or int type but string. > After reading the manual and found the object dtype may store variable > string then I want to exact the string col into an new array, try to > process it then store back to the numpy "matrix" then store it back to > the data source. > > May I know how I can do that? I do not care performance now. > > > Thanks for any hints > > Rgs, > > KC > > > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion --- Travis Oliphant Enthought, Inc. oliphant at enthought.com 1-512-536-1057 http://www.enthought.com From asmund.hjulstad at gmail.com Fri Sep 24 08:15:08 2010 From: asmund.hjulstad at gmail.com (=?ISO-8859-1?Q?=C5smund_Hjulstad?=) Date: Fri, 24 Sep 2010 15:15:08 +0300 Subject: [Numpy-discussion] F2py and Absoft Fortran compiler v11.0 (unable to locate U77.lib) [patch] Message-ID: Hi all, I have a small patch to absoft.py: Compiling my fortran extension using f2py and distutils from Numpy 1.5, with Python 2.7 on Windows, using Absoft v11.0 and MSVC 2008 Pro, first, it complained about not finding the library U77.lib. Removing this from the library list (in absoft.py) gave me this error: error LNK2001: unresolved external symbol __absoft_s_cat This symbol is now exported from amisc.lib, and after adding this to the library list everything works fine. The end result is the following patch: (or just replacing U77 with amisc) --- C:/Python27/Lib/site-packages/numpy/distutils/fcompiler/absoft_patched.py Fri Sep 24 15:10:57 2010 +++ C:/Python27/Lib/site-packages/numpy/distutils/fcompiler/absoft.py Fri Sep 24 15:09:52 2010 @@ -91,7 +91,9 @@ def get_libraries(self): opt = FCompiler.get_libraries(self) - if self.get_version() >= '10.0': + if self.get_version() >= '11.0': + opt.extend(['af90math', 'afio', 'af77math', 'amisc']) + elif self.get_version() >= '10.0': opt.extend(['af90math', 'afio', 'af77math', 'U77']) elif self.get_version() >= '8.0': opt.extend(['f90math','fio','f77math','U77']) BTW, the performance increase compared to gfortran v4.5.0 is really noticeable, perhaps more than 50%. Best regards, ?smund Hjulstad -------------- next part -------------- An HTML attachment was scrubbed... URL: From mat.yeates at gmail.com Fri Sep 24 12:25:48 2010 From: mat.yeates at gmail.com (Mathew Yeates) Date: Fri, 24 Sep 2010 09:25:48 -0700 Subject: [Numpy-discussion] constructing an array from memory Message-ID: I'm trying to do something ... unusual. gdb support scripting with Python. From within my python script, I can get the address of a contiguous area of memory that stores a fortran array. I want to creat a NumPy array using "frombuffer". I see that the CPython API supports the creation of a buffer, but, is there an easier, more direct, way? -Mathew From zachary.pincus at yale.edu Fri Sep 24 13:21:01 2010 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Fri, 24 Sep 2010 13:21:01 -0400 Subject: [Numpy-discussion] constructing an array from memory In-Reply-To: References: Message-ID: <4C10A4F2-B73C-4E33-9E31-911BEE4BCE86@yale.edu> > I'm trying to do something ... unusual. > > gdb support scripting with Python. From within my python script, I can > get the address of a contiguous area of memory that stores a fortran > array. I want to creat a NumPy array using "frombuffer". I see that > the CPython API supports the creation of a buffer, but, is there an > easier, more direct, way? Here's how I do a similar task: numpy.ndarray(shape, dtype=dtype, buffer=(ctypes.c_char*size_in_bytes).from_address(address)) You may need the strides or order parameters, as well. Perhaps there's an easier way to create a buffer from an integer memory address, but this seems pretty straightforward. Zach From mat.yeates at gmail.com Fri Sep 24 13:25:25 2010 From: mat.yeates at gmail.com (Mathew Yeates) Date: Fri, 24 Sep 2010 10:25:25 -0700 Subject: [Numpy-discussion] constructing an array from memory In-Reply-To: <4C10A4F2-B73C-4E33-9E31-911BEE4BCE86@yale.edu> References: <4C10A4F2-B73C-4E33-9E31-911BEE4BCE86@yale.edu> Message-ID: Thank a lot. I was wading through the Python C API. This is much simpler. -Mathew On Fri, Sep 24, 2010 at 10:21 AM, Zachary Pincus wrote: >> I'm trying to do something ... unusual. >> >> gdb support scripting with Python. From within my python script, I can >> get the address of a contiguous area of memory that stores a ?fortran >> array. I want to creat a NumPy array using "frombuffer". I see that >> the CPython API supports the creation of a buffer, but, is there an >> easier, more direct, way? > > Here's how I do a similar task: > numpy.ndarray(shape, dtype=dtype, > buffer=(ctypes.c_char*size_in_bytes).from_address(address)) > > You may need the strides or order parameters, as well. > > Perhaps there's an easier way to create a buffer from an integer > memory address, but this seems pretty straightforward. > > Zach > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Fri Sep 24 13:30:29 2010 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 24 Sep 2010 12:30:29 -0500 Subject: [Numpy-discussion] constructing an array from memory In-Reply-To: References: Message-ID: On Fri, Sep 24, 2010 at 11:25, Mathew Yeates wrote: > I'm trying to do something ... unusual. > > gdb support scripting with Python. From within my python script, I can > get the address of a contiguous area of memory that stores a ?fortran > array. I want to creat a NumPy array using "frombuffer". I see that > the CPython API supports the creation of a buffer, but, is there an > easier, more direct, way? You can also make an object with an .__array_interface__ attribute describing the data. [~] |5> x = np.arange(10) [~] |6> x.__array_interface__ {'data': (68583072, False), 'descr': [('', ' np.lib.stride_tricks.DummyArray?? Type: type Base Class: String Form: Namespace: Interactive File: /Library/Frameworks/Python.framework/Versions/6.1/lib/python2.6/site-packages/numpy/lib/stride_tricks.py Source: class DummyArray(object): """ Dummy object that just exists to hang __array_interface__ dictionaries and possibly keep alive a reference to a base array. """ def __init__(self, interface, base=None): self.__array_interface__ = interface self.base = base Constructor information: Definition: np.lib.stride_tricks.DummyArray(self, interface, base=None) Then np.asarray() will consume that object to make an ndarray that references the given memory. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From dalcinl at gmail.com Fri Sep 24 17:31:00 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 24 Sep 2010 18:31:00 -0300 Subject: [Numpy-discussion] [PATCH] F2Py on Python 3 In-Reply-To: References: Message-ID: On 23 September 2010 23:34, Charles R Harris wrote: > > > On Thu, Sep 23, 2010 at 7:22 PM, Lisandro Dalcin wrote: >> >> On 23 September 2010 01:26, Charles R Harris >> wrote: >> > >> > >> > On Wed, Sep 22, 2010 at 10:00 PM, Charles R Harris >> > wrote: >> >> >> >> >> >> On Wed, Sep 22, 2010 at 9:14 PM, Fernando Perez >> >> wrote: >> >>> >> >>> On Wed, Sep 22, 2010 at 7:16 PM, Charles R Harris >> >>> wrote: >> >>> > >> >>> > Done in? 29cccb6. >> >>> > >> >>> >> >>> Mmh, I think it broke something: >> >>> >> >>> ?File >> >>> "/home/fperez/tmp/src/scipy/numpy/numpy/distutils/command/build.py", >> >>> line 37, in run >> >>> ? ?old_build.run(self) >> >>> ?File "/usr/lib/python2.6/distutils/command/build.py", line 135, in >> >>> run >> >>> ? ?self.run_command(cmd_name) >> >>> ?File "/usr/lib/python2.6/distutils/cmd.py", line 333, in run_command >> >>> ? ?self.distribution.run_command(command) >> >>> ?File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command >> >>> ? ?cmd_obj.run() >> >>> ?File >> >>> >> >>> "/home/fperez/tmp/src/scipy/numpy/numpy/distutils/command/build_scripts.py", >> >>> line 39, in run >> >>> ? ?self.scripts = self.generate_scripts(self.scripts) >> >>> ?File >> >>> >> >>> "/home/fperez/tmp/src/scipy/numpy/numpy/distutils/command/build_scripts.py", >> >>> line 24, in generate_scripts >> >>> ? ?script = func(build_dir) >> >>> ?File "numpy/f2py/setup.py", line 78, in generate_f2py_py >> >>> ? ?'''%(os.path.basename(sys.executable))) >> >>> TypeError: not enough arguments for format string >> >>> >> >>> If I revert to the previous commit, it installs fine again. >> >>> >> >> >> >> I wondered about that, it is probably the old `mode` vs? plain old >> >> mode. I >> >> had just hoped it was tested. Can you check that out? >> >> >> > >> > Should be fixed in 8f6114b. >> > >> >> I had just hoped it was tested... [sorry, I could not resist! ;-) ] >> >> diff --git a/numpy/f2py/setup.py b/numpy/f2py/setup.py >> index aac3d33..37aab19 100644 >> --- a/numpy/f2py/setup.py >> +++ b/numpy/f2py/setup.py >> @@ -62,7 +62,7 @@ for mode in ["g3-numpy", "2e-numeric", >> "2e-numarray", "2e-numpy >> ? ? except ValueError: pass >> ?os.environ["NO_SCIPY_IMPORT"]="f2py" >> ?if mode=="g3-numpy": >> - ? ?sys.stderr.write("G3 f2py support is not implemented, yet.\n") >> + ? ?sys.stderr.write("G3 f2py support is not implemented, yet.\\n") >> ? ? sys.exit(1) >> ?elif mode=="2e-numeric": >> ? ? from f2py2e import main >> @@ -72,7 +72,7 @@ elif mode=="2e-numarray": >> ?elif mode=="2e-numpy": >> ? ? from numpy.f2py import main >> ?else: >> - ? ?sys.stderr.write("Unknown mode: " + repr(mode) + "\n") >> + ? ?sys.stderr.write("Unknown mode: " + repr(mode) + "\\n") >> ? ? sys.exit(1) >> ?main() >> ?'''%(os.path.basename(sys.executable))) >> > > Arrggghhhh, I forgot the double backslashes :-( > Cython does a lot of stuff like this (though it generates C code), and I prefer use raw string prefixes when hacking on it. These backslash scapes are nightmare for my brain to parse, plus it is very easy for yourself or others to make a mistake later. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From gburdell1 at gmail.com Fri Sep 24 21:56:25 2010 From: gburdell1 at gmail.com (George) Date: Sat, 25 Sep 2010 01:56:25 +0000 (UTC) Subject: [Numpy-discussion] Slicing, sum, etc. reduces rank of array? Message-ID: I couldn't find an answer to my newbie question, so I'm posting it here. I have: a=numpy.array([[1,2],[3,4]]) b=numpy.array([[5,6],[7,8]]) Via broadcasting, I know that a*[[5],[7]]=numpy.array([[5,10],[21,28]]) Being a recent convert from MATLAB, I expected the same result from a*b[:,0], assuming b[:,0] would be the column vector [[5],[7]]. Unfortunately, I was wrong. b[:,0] is apparently a 1-rank array of shape (2,). This causes a*b[:,0] to evaluate as a*numpy.array([[5,7]])=numpy.array([[5,14],[15,28]]) instead of a*numpy.array([[5],[7]]) To get the result I desire, the only way I've been able to come up with is a*b[:,0].reshape(2,1) to "coerce" b[:,0] into a column vector. Is there an easier way to do this, without having to do the reshape? I find similar things happen when I use other operations (e.g. "sum") that also seem to reduce the array rank. For example, I would expect numpy.sum(b,1) to also be a "column vector," but it also evaluates to a 1-rank array [11, 15] with shape (2,) Any thoughts, suggestions? From ben.root at ou.edu Fri Sep 24 22:23:26 2010 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 24 Sep 2010 21:23:26 -0500 Subject: [Numpy-discussion] Slicing, sum, etc. reduces rank of array? In-Reply-To: References: Message-ID: On Fri, Sep 24, 2010 at 8:56 PM, George wrote: > I couldn't find an answer to my newbie question, so I'm posting it here. > > I have: > > a=numpy.array([[1,2],[3,4]]) > b=numpy.array([[5,6],[7,8]]) > > Via broadcasting, I know that > a*[[5],[7]]=numpy.array([[5,10],[21,28]]) > > Being a recent convert from MATLAB, I expected the same result from > a*b[:,0], > assuming b[:,0] would be the column vector [[5],[7]]. > > Unfortunately, I was wrong. b[:,0] is apparently a 1-rank array of shape > (2,). > This causes a*b[:,0] to evaluate as > a*numpy.array([[5,7]])=numpy.array([[5,14],[15,28]]) instead of > a*numpy.array([[5],[7]]) > > To get the result I desire, the only way I've been able to come up with is > > a*b[:,0].reshape(2,1) > > to "coerce" b[:,0] into a column vector. Is there an easier way to do this, > without having to do the reshape? > > I find similar things happen when I use other operations (e.g. "sum") that > also > seem to reduce the array rank. > > For example, I would expect numpy.sum(b,1) to also be a "column vector," > but it > also evaluates to a 1-rank array [11, 15] with shape (2,) > > Any thoughts, suggestions? > > This has bitten me several times in the past. While there are some neat tricks around this issue, the one sure-fire, blunt-object solution to the problem is the np.atleast_2d() function. There is also a 1d and 3d variant (although the 3d variant messes around a bit with the order of the axes...). I will leave the more elegant solutions to others to give. Ben Root > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Fri Sep 24 22:31:48 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 24 Sep 2010 22:31:48 -0400 Subject: [Numpy-discussion] Slicing, sum, etc. reduces rank of array? In-Reply-To: References: Message-ID: On Fri, Sep 24, 2010 at 10:23 PM, Benjamin Root wrote: > On Fri, Sep 24, 2010 at 8:56 PM, George wrote: >> >> I couldn't find an answer to my newbie question, so I'm posting it here. >> >> I have: >> >> a=numpy.array([[1,2],[3,4]]) >> b=numpy.array([[5,6],[7,8]]) >> >> Via broadcasting, I know that >> a*[[5],[7]]=numpy.array([[5,10],[21,28]]) >> >> Being a recent convert from MATLAB, I expected the same result from >> a*b[:,0], >> assuming b[:,0] would be the column vector [[5],[7]]. >> >> Unfortunately, I was wrong. b[:,0] is apparently a 1-rank array of shape >> (2,). >> This causes a*b[:,0] to evaluate as >> a*numpy.array([[5,7]])=numpy.array([[5,14],[15,28]]) instead of >> a*numpy.array([[5],[7]]) >> >> To get the result I desire, the only way I've been able to come up with is >> >> a*b[:,0].reshape(2,1) >> >> to "coerce" b[:,0] into a column vector. Is there an easier way to do >> this, >> without having to do the reshape? >> >> I find similar things happen when I use other operations (e.g. "sum") that >> also >> seem to reduce the array rank. >> >> For example, I would expect numpy.sum(b,1) to also be a "column vector," >> but it >> also evaluates to a 1-rank array [11, 15] with shape (2,) >> >> Any thoughts, suggestions? >> > > This has bitten me several times in the past.? While there are some neat > tricks around this issue, the one sure-fire, blunt-object solution to the > problem is the np.atleast_2d() function.? There is also a 1d and 3d variant > (although the 3d variant messes around a bit with the order of the axes...). np.atleast_2d() leaves you with a row vector my preferred generic version since I found it,, has been np.expand_dims(a.sum(axis), axis) special solution for given axis b[:,0:1] slice instead of index b[:,0][:,None] add axis back in I managed to get used to it, finally, but None is almost the most frequent index in numpy (maybe 3rd place) Josef > > I will leave the more elegant solutions to others to give. > > Ben Root > >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From charlesr.harris at gmail.com Sat Sep 25 09:52:45 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 25 Sep 2010 07:52:45 -0600 Subject: [Numpy-discussion] F2py and Absoft Fortran compiler v11.0 (unable to locate U77.lib) [patch] In-Reply-To: References: Message-ID: On Fri, Sep 24, 2010 at 6:15 AM, ?smund Hjulstad wrote: > > Hi all, > > I have a small patch to absoft.py: > > Compiling my fortran extension using f2py and distutils from Numpy 1.5, > with Python 2.7 on Windows, using Absoft v11.0 and MSVC 2008 Pro, first, it > complained about not finding the library U77.lib. Removing this from the > library list (in absoft.py) gave me this error: > > error LNK2001: unresolved external symbol __absoft_s_cat > > This symbol is now exported from amisc.lib, and after adding this to the > library list everything works fine. > > The end result is the following patch: (or just replacing U77 with amisc) > --- > C:/Python27/Lib/site-packages/numpy/distutils/fcompiler/absoft_patched.py > Fri Sep 24 15:10:57 2010 > +++ C:/Python27/Lib/site-packages/numpy/distutils/fcompiler/absoft.py > Fri Sep 24 15:09:52 2010 > @@ -91,7 +91,9 @@ > > def get_libraries(self): > opt = FCompiler.get_libraries(self) > - if self.get_version() >= '10.0': > + if self.get_version() >= '11.0': > + opt.extend(['af90math', 'afio', 'af77math', 'amisc']) > + elif self.get_version() >= '10.0': > opt.extend(['af90math', 'afio', 'af77math', 'U77']) > elif self.get_version() >= '8.0': > opt.extend(['f90math','fio','f77math','U77']) > > > BTW, the performance increase compared to gfortran v4.5.0 is really > noticeable, perhaps more than 50%. > > Done in 75cebc1. It is better to open a ticket for these things as otherwise they are likely to slip unnoticed into history. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at pythonlibrary.org Thu Sep 16 10:08:23 2010 From: mike at pythonlibrary.org (Mike Driscoll) Date: Thu, 16 Sep 2010 09:08:23 -0500 Subject: [Numpy-discussion] GSoC Article for PSF and my blog Message-ID: Hi, I am working on an article for the Python Software Foundation's blog, http://pyfound.blogspot.com/, about the various Python projects that were worked on during this year's Google Summer of Code program. They want me to write up something about what projects were worked on and what the results were. I found your project information here: http://wiki.python.org/moin/SummerOfCode/2010 Anyway, since the PSF blog article will be brief, I thought I would also write up a longer article about your projects on my personal blog as well. The information I found on the Python wiki page was pretty brief, so I would appreciate it if you could tell me the following: 1) What was worked on during the GSoC project 2) How many students helped with a guess at how many hours were put in 3) Your experiences being a mentor If possible, I would like the student's perspective too. Feel free to forward my information to them. Also feel free to opt out and I won't write anything more than the already public info I can find. Thanks a lot for your help! -- ----------------- Mike Driscoll Blog: http://blog.pythonlibrary.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From longstaff2009 at gmail.com Sun Sep 26 21:57:11 2010 From: longstaff2009 at gmail.com (Ben Longstaff) Date: Mon, 27 Sep 2010 11:27:11 +0930 Subject: [Numpy-discussion] Matlab IO Warning in mio5.py Message-ID: Is the warning message produced by the test script below an issue I need to address if I am to use the numpy and scipy libraries? The script below gives me the warning below on both windows 7 and vista Thanks for your time Ben Python Script #!/path/to/python import re, string, sys from numpy import * import scipy.weave print "Hello, World!" Warning Message C:\Python26\lib\site-packages\scipy\io\matlab\mio5.py:90: RuntimeWarning: __builtin__.file size changed, may indicate binary incompatibility from mio5_utils import VarReader5 Hello, World! -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sun Sep 26 22:56:57 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 26 Sep 2010 22:56:57 -0400 Subject: [Numpy-discussion] Matlab IO Warning in mio5.py In-Reply-To: References: Message-ID: On Sun, Sep 26, 2010 at 9:57 PM, Ben Longstaff wrote: > Is the warning message produced by the test script below an issue I need to > address if I am to use the numpy and scipy libraries? The script below gives > me the warning below on both windows 7 and vista > Thanks for your time > Ben > > Python Script > > #!/path/to/python > import re, string, sys > from numpy import * > import scipy.weave > print "Hello, World!" > > Warning Message > C:\Python26\lib\site-packages\scipy\io\matlab\mio5.py:90: RuntimeWarning: > __builtin__.file size changed, may indicate binary incompatibility > ? from mio5_utils import VarReader5 > > Hello, World! Which numpy and scipy versions are you using? e.g. scipy.__version__ np.__version__ This looks like a warning from cython, but the versions are still binary compatible, and there shouldn't be any problems. (with all numpy except 1.4.0 and 2.x in repository) Josef > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From longstaff2009 at gmail.com Sun Sep 26 23:03:06 2010 From: longstaff2009 at gmail.com (Ben Longstaff) Date: Mon, 27 Sep 2010 12:33:06 +0930 Subject: [Numpy-discussion] Matlab IO Warning in mio5.py In-Reply-To: References: Message-ID: numpy 1.5.0 (installed from numpy-1.5.0-win32-superpack-python2.6.exe from sourceforge.net) scipy 0.8.0 (installed from scipy-0.8.0-win32-superpack-python2.6.exe from sourceforge.net) python 2.6.6 (installed with ActivePython 2.6.6.15) On Mon, Sep 27, 2010 at 12:26 PM, wrote: > On Sun, Sep 26, 2010 at 9:57 PM, Ben Longstaff > wrote: > > Is the warning message produced by the test script below an issue I need > to > > address if I am to use the numpy and scipy libraries? The script below > gives > > me the warning below on both windows 7 and vista > > Thanks for your time > > Ben > > > > Python Script > > > > #!/path/to/python > > import re, string, sys > > from numpy import * > > import scipy.weave > > print "Hello, World!" > > > > Warning Message > > C:\Python26\lib\site-packages\scipy\io\matlab\mio5.py:90: RuntimeWarning: > > __builtin__.file size changed, may indicate binary incompatibility > > from mio5_utils import VarReader5 > > > > Hello, World! > > Which numpy and scipy versions are you using? e.g. scipy.__version__ > np.__version__ > > This looks like a warning from cython, but the versions are still > binary compatible, and there shouldn't be any problems. > (with all numpy except 1.4.0 and 2.x in repository) > > Josef > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sun Sep 26 23:42:37 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 26 Sep 2010 23:42:37 -0400 Subject: [Numpy-discussion] Matlab IO Warning in mio5.py In-Reply-To: References: Message-ID: On Sun, Sep 26, 2010 at 11:03 PM, Ben Longstaff wrote: > numpy 1.5.0 (installed from numpy-1.5.0-win32-superpack-python2.6.exe from > sourceforge.net) > scipy 0.8.0 (installed from scipy-0.8.0-win32-superpack-python2.6.exe from > sourceforge.net) > python 2.6.6 (installed with ActivePython 2.6.6.15) It's a safe combination, and the cython warning has no effect. (Maybe a disclaimer: There's always a chance that new things show up, but I haven't heard of any problems.) Josef > > On Mon, Sep 27, 2010 at 12:26 PM, wrote: >> >> On Sun, Sep 26, 2010 at 9:57 PM, Ben Longstaff >> wrote: >> > Is the warning message produced by the test script below an issue I need >> > to >> > address if I am to use the numpy and scipy libraries? The script below >> > gives >> > me the warning below on both windows 7 and vista >> > Thanks for your time >> > Ben >> > >> > Python Script >> > >> > #!/path/to/python >> > import re, string, sys >> > from numpy import * >> > import scipy.weave >> > print "Hello, World!" >> > >> > Warning Message >> > C:\Python26\lib\site-packages\scipy\io\matlab\mio5.py:90: >> > RuntimeWarning: >> > __builtin__.file size changed, may indicate binary incompatibility >> > ? from mio5_utils import VarReader5 >> > >> > Hello, World! >> >> Which numpy and scipy versions are you using? e.g. scipy.__version__ >> np.__version__ >> >> This looks like a warning from cython, but the versions are still >> binary compatible, and there shouldn't be any problems. >> (with all numpy except 1.4.0 and 2.x in repository) >> >> Josef >> >> >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From josef.pktd at gmail.com Sun Sep 26 23:56:42 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 26 Sep 2010 23:56:42 -0400 Subject: [Numpy-discussion] Matlab IO Warning in mio5.py In-Reply-To: References: Message-ID: On Sun, Sep 26, 2010 at 11:42 PM, wrote: > On Sun, Sep 26, 2010 at 11:03 PM, Ben Longstaff wrote: >> numpy 1.5.0 (installed from numpy-1.5.0-win32-superpack-python2.6.exe from >> sourceforge.net) >> scipy 0.8.0 (installed from scipy-0.8.0-win32-superpack-python2.6.exe from >> sourceforge.net) >> python 2.6.6 (installed with ActivePython 2.6.6.15) > > It's a safe combination, and the cython warning has no effect. > > (Maybe a disclaimer: There's always a chance that new things show up, > but I haven't heard of any problems.) Just a side comment: ActiveState is distributing (since June) "New packages for data centric financial modeling and scientific computing applications (available in ActivePython Business, Enterprise, and OEM Editions): * NumPy: The fundamental library needed for financial and scientific computing with Python * SciPy: A suite of scientific tools for Python, depends on the NumPy library * matplotlib: A Python numerical plotting library " because of changes in US financial regulation. So maybe we get some additional testing out of it. Josef > > Josef > >> >> On Mon, Sep 27, 2010 at 12:26 PM, wrote: >>> >>> On Sun, Sep 26, 2010 at 9:57 PM, Ben Longstaff >>> wrote: >>> > Is the warning message produced by the test script below an issue I need >>> > to >>> > address if I am to use the numpy and scipy libraries? The script below >>> > gives >>> > me the warning below on both windows 7 and vista >>> > Thanks for your time >>> > Ben >>> > >>> > Python Script >>> > >>> > #!/path/to/python >>> > import re, string, sys >>> > from numpy import * >>> > import scipy.weave >>> > print "Hello, World!" >>> > >>> > Warning Message >>> > C:\Python26\lib\site-packages\scipy\io\matlab\mio5.py:90: >>> > RuntimeWarning: >>> > __builtin__.file size changed, may indicate binary incompatibility >>> > ? from mio5_utils import VarReader5 >>> > >>> > Hello, World! >>> >>> Which numpy and scipy versions are you using? e.g. scipy.__version__ >>> np.__version__ >>> >>> This looks like a warning from cython, but the versions are still >>> binary compatible, and there shouldn't be any problems. >>> (with all numpy except 1.4.0 and 2.x in repository) >>> >>> Josef >>> >>> >>> > >>> > _______________________________________________ >>> > NumPy-Discussion mailing list >>> > NumPy-Discussion at scipy.org >>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> > >>> > >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > From charlesr.harris at gmail.com Mon Sep 27 00:07:19 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 26 Sep 2010 22:07:19 -0600 Subject: [Numpy-discussion] Matlab IO Warning in mio5.py In-Reply-To: References: Message-ID: On Sun, Sep 26, 2010 at 9:56 PM, wrote: > On Sun, Sep 26, 2010 at 11:42 PM, wrote: > > On Sun, Sep 26, 2010 at 11:03 PM, Ben Longstaff > wrote: > >> numpy 1.5.0 (installed from numpy-1.5.0-win32-superpack-python2.6.exe > from > >> sourceforge.net) > >> scipy 0.8.0 (installed from scipy-0.8.0-win32-superpack-python2.6.exe > from > >> sourceforge.net) > >> python 2.6.6 (installed with ActivePython 2.6.6.15) > > > > It's a safe combination, and the cython warning has no effect. > > > > (Maybe a disclaimer: There's always a chance that new things show up, > > but I haven't heard of any problems.) > > Just a side comment: ActiveState is distributing (since June) > > "New packages for data centric financial modeling and scientific > computing applications (available in ActivePython Business, > Enterprise, and OEM Editions): > * NumPy: The fundamental library needed for financial and > scientific computing with Python > * SciPy: A suite of scientific tools for Python, depends on the > NumPy library > * matplotlib: A Python numerical plotting library > " > > because of changes in US financial regulation. So maybe we get some > additional testing out of it. > > Josef > > > > > Josef > > > >> > >> On Mon, Sep 27, 2010 at 12:26 PM, wrote: > >>> > >>> On Sun, Sep 26, 2010 at 9:57 PM, Ben Longstaff < > longstaff2009 at gmail.com> > >>> wrote: > >>> > Is the warning message produced by the test script below an issue I > need > >>> > to > >>> > address if I am to use the numpy and scipy libraries? The script > below > >>> > gives > >>> > me the warning below on both windows 7 and vista > >>> > Thanks for your time > >>> > Ben > >>> > > >>> > Python Script > >>> > > >>> > #!/path/to/python > >>> > import re, string, sys > >>> > from numpy import * > >>> > import scipy.weave > >>> > print "Hello, World!" > >>> > > >>> > Warning Message > >>> > C:\Python26\lib\site-packages\scipy\io\matlab\mio5.py:90: > >>> > RuntimeWarning: > >>> > __builtin__.file size changed, may indicate binary incompatibility > >>> > from mio5_utils import VarReader5 > >>> > > >>> > Hello, World! > >>> > >>> Which numpy and scipy versions are you using? e.g. scipy.__version__ > >>> np.__version__ > >>> > >>> This looks like a warning from cython, but the versions are still > >>> binary compatible, and there shouldn't be any problems. > >>> (with all numpy except 1.4.0 and 2.x in repository) > >>> > Didn't Ralph catch the warnings on import for the released binaries? Maybe he missed one. Also, I don't recall seeing seeing that one before, not that it is necessarily a problem. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Mon Sep 27 00:26:05 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 27 Sep 2010 00:26:05 -0400 Subject: [Numpy-discussion] Matlab IO Warning in mio5.py In-Reply-To: References: Message-ID: On Mon, Sep 27, 2010 at 12:07 AM, Charles R Harris wrote: > > > On Sun, Sep 26, 2010 at 9:56 PM, wrote: >> >> On Sun, Sep 26, 2010 at 11:42 PM, ? wrote: >> > On Sun, Sep 26, 2010 at 11:03 PM, Ben Longstaff >> > wrote: >> >> numpy 1.5.0 (installed from numpy-1.5.0-win32-superpack-python2.6.exe >> >> from >> >> sourceforge.net) >> >> scipy 0.8.0 (installed from scipy-0.8.0-win32-superpack-python2.6.exe >> >> from >> >> sourceforge.net) >> >> python 2.6.6 (installed with ActivePython 2.6.6.15) >> > >> > It's a safe combination, and the cython warning has no effect. >> > >> > (Maybe a disclaimer: There's always a chance that new things show up, >> > but I haven't heard of any problems.) >> >> Just a side comment: ActiveState is distributing (since June) >> >> "New packages for data centric financial modeling and scientific >> computing applications (available in ActivePython Business, >> Enterprise, and OEM Editions): >> ? ?* NumPy: The fundamental library needed for financial and >> scientific computing with Python >> ? ?* SciPy: A suite of scientific tools for Python, depends on the >> NumPy library >> ? ?* matplotlib: A Python numerical plotting library >> " >> >> because of changes in US financial regulation. So maybe we get some >> additional testing out of it. >> >> Josef >> >> > >> > Josef >> > >> >> >> >> On Mon, Sep 27, 2010 at 12:26 PM, wrote: >> >>> >> >>> On Sun, Sep 26, 2010 at 9:57 PM, Ben Longstaff >> >>> >> >>> wrote: >> >>> > Is the warning message produced by the test script below an issue I >> >>> > need >> >>> > to >> >>> > address if I am to use the numpy and scipy libraries? The script >> >>> > below >> >>> > gives >> >>> > me the warning below on both windows 7 and vista >> >>> > Thanks for your time >> >>> > Ben >> >>> > >> >>> > Python Script >> >>> > >> >>> > #!/path/to/python >> >>> > import re, string, sys >> >>> > from numpy import * >> >>> > import scipy.weave >> >>> > print "Hello, World!" >> >>> > >> >>> > Warning Message >> >>> > C:\Python26\lib\site-packages\scipy\io\matlab\mio5.py:90: >> >>> > RuntimeWarning: >> >>> > __builtin__.file size changed, may indicate binary incompatibility >> >>> > ? from mio5_utils import VarReader5 >> >>> > >> >>> > Hello, World! >> >>> >> >>> Which numpy and scipy versions are you using? e.g. scipy.__version__ >> >>> np.__version__ >> >>> >> >>> This looks like a warning from cython, but the versions are still >> >>> binary compatible, and there shouldn't be any problems. >> >>> (with all numpy except 1.4.0 and 2.x in repository) >> >>> > > Didn't Ralph catch the warnings on import for the released binaries? Maybe > he missed one. Also, I don't recall seeing seeing that one before, not that > it is necessarily a problem. matlab\mio5.py:90: RuntimeWarning was reported during scipy 0.8 rc2 testing I don't know which warnings were silenced Josef > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From faltet at pytables.org Mon Sep 27 08:28:21 2010 From: faltet at pytables.org (Francesc Alted) Date: Mon, 27 Sep 2010 14:28:21 +0200 Subject: [Numpy-discussion] str/bytes object from arr.data? Message-ID: <201009271428.21808.faltet@pytables.org> Hi, Anybody knows a way to get a str object (bytes for Python >= 2.6) out of a buffer object (i.e. the .data attribute of ndarrays) without copying data? I need this for avoid creating a new function that deals with buffer objects (instead of reusing the one for str/byte objects that I already have). So, a matter of laziness :-) Thanks, -- Francesc Alted From zachary.pincus at yale.edu Mon Sep 27 09:04:14 2010 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Mon, 27 Sep 2010 09:04:14 -0400 Subject: [Numpy-discussion] str/bytes object from arr.data? In-Reply-To: <201009271428.21808.faltet@pytables.org> References: <201009271428.21808.faltet@pytables.org> Message-ID: As str objects are supposed to be immutable, I think anything "official" that makes a string from a numpy array is supposed to copy the data. But I think you can use ctypes to wrap a pointer and a length as a python string. Zach On Sep 27, 2010, at 8:28 AM, Francesc Alted wrote: > Hi, > > Anybody knows a way to get a str object (bytes for Python >= 2.6) > out of > a buffer object (i.e. the .data attribute of ndarrays) without copying > data? > > I need this for avoid creating a new function that deals with buffer > objects (instead of reusing the one for str/byte objects that I > already > have). So, a matter of laziness :-) > > Thanks, > > -- > Francesc Alted > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From ralf.gommers at googlemail.com Mon Sep 27 09:46:42 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 27 Sep 2010 21:46:42 +0800 Subject: [Numpy-discussion] Matlab IO Warning in mio5.py In-Reply-To: References: Message-ID: On Mon, Sep 27, 2010 at 12:07 PM, Charles R Harris < charlesr.harris at gmail.com> wrote: > > On Sun, Sep 26, 2010 at 11:03 PM, Ben Longstaff >> wrote: >> >> >>> > Warning Message >> >>> > C:\Python26\lib\site-packages\scipy\io\matlab\mio5.py:90: >> >>> > RuntimeWarning: >> >>> > __builtin__.file size changed, may indicate binary incompatibility >> >>> > from mio5_utils import VarReader5 >> >>> > >> >>> > Hello, World! >> >>> >> >>> Which numpy and scipy versions are you using? e.g. scipy.__version__ >> >>> np.__version__ >> >>> >> >>> This looks like a warning from cython, but the versions are still >> >>> binary compatible, and there shouldn't be any problems. >> >>> (with all numpy except 1.4.0 and 2.x in repository) >> >>> >> > > Didn't Ralph catch the warnings on import for the released binaries? Maybe > he missed one. Also, I don't recall seeing seeing that one before, not that > it is necessarily a problem. > Hmm, missed this one, also can't reproduce it on either XP or OS X with the same numpy/scipy versions. If these warnings are platform-dependent shouldn't that be considered a Cython bug? Related commits are r6331 and r6512 in scipy. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From friedrichromstedt at gmail.com Mon Sep 27 10:40:37 2010 From: friedrichromstedt at gmail.com (Friedrich Romstedt) Date: Mon, 27 Sep 2010 16:40:37 +0200 Subject: [Numpy-discussion] Asking for opinions: Priops In-Reply-To: <4C9B6E2D.5050706@student.matnat.uio.no> References: <4C9B6E2D.5050706@student.matnat.uio.no> Message-ID: 2010/9/23 Dag Sverre Seljebotn : > Essentially, perhaps what you have sketched up + an ability to extend > the graph with object conversion routes would be perfect for my own > uses. So you can define a function with overloads (A, B) and (A, C), but > also that objects of type D can be converted to C (and how). For > instance, consider: > > np.array([1,2,3]) + [1,2,3] > > Here, list-> array could be handled through a defined coercion to array, > rather than having to add an overload for list for every method taking > an array. This seems to be a good thing. Let's reason about this for some moment. Atm, the relation is *not* transitive. (I.e., (A, B) and (B, C) does imply nothing for (A, C). A, B, C classes.) But such kind of transitivity is what you mean, if (A, B) is defined and there is a graph egde in the "conversion graph" (B, C), then (A, C) can choose the (A, B) route via C -> B conversion; here the notation (B, C) in the conversion graph means "conversion from C to B". I don't see a clear solution at the end satisfying me. It seems that one really has to conduct a search in the additional conversion graph. Since this is expensive, I believe it would probably be good to derive a "ConversionPriop" from priop.Priop. What are your thoughts now? I feel it useful to add this conversion graph, since it creates many egdes in the resulting effective coercion graph, which do not have all to be specified explicitly, but which exist. Before, this conversion graph exists already in the sense of subclassing, i.e., if your second object is a subclass of `list`, in your example, it will be sufficient to define the edge with `list`, and it will find the edge. Would this suffice for your needs? It could be that this is even safer than magic conversion pathes. Maybe it's even better, more straightforward. I'm really unsure about this. Please give me your feedback. > np.array([1,2,3]) + [1,2,3] For this, a) numpy.ndarray could be subclassing list, if it doesn't do already, but then it would look like list concatenation. b) It would be sufficient to add the edge with `list` as outlined before. So, now I tend to *not* adding the functionality ... :-/ Maybe the strongest argument in favour of adding it nevertheless is, that, in our example, ndarray can be seen at a "view" of a list. So operations with objects that can be seen as something else should be supported, especially if the class it is seen as is more deep in the class hierarchy. This means, edges (ndarray, ndarray) in fact should match on (ndarray, list), although `ndarray` is more deep than `list`. (Opposed to the case one defines (ndarray, list), and this matches (ndarrary, list_subclass), seeing only the `list` functionality in `list_subclass`.) Would it, having this in mind, suffice to add a transitivity not only for subclasses, but also for superclasses? If (A, B) shall be coerced and C is a subclass of B, i.e. B a superclass of C, then (A, B) should translate to (A, C), using C(b) for the operand? This means, C *extends* the functionality of B only. I'm feeling quite good with this, because it's less arbitrary than allowing for *all* conversion routes. Actually it's nothing more than giving the class C as the conversion function, since classes are callable. And since not all subclasses maybe need more than a superclass instance for construction ..., the graph still has to created manually. Looking further into this, it seems to suffice classes which can be constructed from their superclass instances only. E.g. class C just needs to be registered as an "extending class", and if in (A, B) it holds that ``isinstance(b, C)`` while (A, B) is not defined but (A, C), then the (A, C) edge will be called via the C(b) constructor. Conversion would just happen using a *take* argument to the registration, optionally being a string defining the kwarg used in the C constructor for construction from base class instances. E.g.: priop.extender(numpy.ndarray, constructor=numpy.asarray, extends=list) priop.extender(my_class, take='universal') # *universal* may be some late kwarg in the constructor. # Automatically applies to all classes `my_class` derives from. Unfortunately, ndarray isn't a subclass of list. Maybe this would be worth a NEP, if it is technically possible. Otherwise, the first form would suffice it. ? It got a bit longish, Friedrich P.S.: I like the fact that I understood that "conversion a->b" means "B extends A". As opposed to the fact "B functionality is a subset of A functionality". From dagss at student.matnat.uio.no Mon Sep 27 13:18:58 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 27 Sep 2010 19:18:58 +0200 Subject: [Numpy-discussion] Asking for opinions: Priops In-Reply-To: References: <4C9B6E2D.5050706@student.matnat.uio.no> Message-ID: <4CA0D202.9040201@student.matnat.uio.no> Friedrich Romstedt wrote: > 2010/9/23 Dag Sverre Seljebotn : > >> Essentially, perhaps what you have sketched up + an ability to extend >> the graph with object conversion routes would be perfect for my own >> uses. So you can define a function with overloads (A, B) and (A, C), but >> also that objects of type D can be converted to C (and how). For >> instance, consider: >> >> np.array([1,2,3]) + [1,2,3] >> >> Here, list-> array could be handled through a defined coercion to array, >> rather than having to add an overload for list for every method taking >> an array. >> > > This seems to be a good thing. Let's reason about this for some > moment. Atm, the relation is *not* transitive. (I.e., (A, B) and (B, > C) does imply nothing for (A, C). A, B, C classes.) But such kind of > transitivity is what you mean, if (A, B) is defined and there is a > graph egde in the "conversion graph" (B, C), then (A, C) can choose > the (A, B) route via C -> B conversion; here the notation (B, C) in > the conversion graph means "conversion from C to B". > > I don't see a clear solution at the end satisfying me. It seems that > one really has to conduct a search in the additional conversion graph. > Since this is expensive, I believe it would probably be good to > derive a "ConversionPriop" from priop.Priop. > > What are your thoughts now? > You can just cache all lookups in the conversion graph, so that after some initialization all lookups are O(1). There's a limited number of types in a Python runtime, and the actual distinct lookups performed are likely to not be many. I don't see that as a problem at all. I'll try to remember to get back to the rest in a week or so...I'm handing in my MSc on Thursday :-) Dag Sverre From faltet at pytables.org Mon Sep 27 13:24:03 2010 From: faltet at pytables.org (Francesc Alted) Date: Mon, 27 Sep 2010 19:24:03 +0200 Subject: [Numpy-discussion] str/bytes object from arr.data? In-Reply-To: References: <201009271428.21808.faltet@pytables.org> Message-ID: <201009271924.03440.faltet@pytables.org> A Monday 27 September 2010 15:04:14 Zachary Pincus escrigu?: > As str objects are supposed to be immutable, I think anything > "official" that makes a string from a numpy array is supposed to copy > the data. But I think you can use ctypes to wrap a pointer and a > length as a python string. Yeah, ctypes is very powerful indeed. Thanks! -- Francesc Alted From butterw at gmail.com Mon Sep 27 18:51:59 2010 From: butterw at gmail.com (Peter Butterworth) Date: Tue, 28 Sep 2010 00:51:59 +0200 Subject: [Numpy-discussion] mean of empty sequence gives nan In-Reply-To: References: Message-ID: In numpy 1.5.0, I got the following for mean of an empty sequence (or array): In [21]: mean([]) Warning: invalid value encountered in double_scalars Out[21]: nan Is this behaviour expected ? Also, would it be possible to have more explicit warning messages about the problem being related numpy ? It seems these message cause quite a bit of confusion to users. The following would be more helpful : Warning: invalid value encountered in numpy double_scalars From robert.kern at gmail.com Mon Sep 27 18:55:57 2010 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 27 Sep 2010 17:55:57 -0500 Subject: [Numpy-discussion] mean of empty sequence gives nan In-Reply-To: References: Message-ID: On Mon, Sep 27, 2010 at 17:51, Peter Butterworth wrote: > In numpy 1.5.0, I got the following for mean of an empty sequence (or array): > > In [21]: mean([]) > Warning: invalid value encountered in double_scalars > Out[21]: nan > > Is this behaviour expected ? np.sum([]) / len([]) -> 0.0 / 0 -> nan, so yes. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From markbak at gmail.com Tue Sep 28 04:26:28 2010 From: markbak at gmail.com (Mark Bakker) Date: Tue, 28 Sep 2010 10:26:28 +0200 Subject: [Numpy-discussion] can savetxt write arrays with ndim > 2 ? Message-ID: Hello list, I tried to savetxt an array with ndim = 3, but I get an error: In [252]: savetxt('test.dat',a) /Library/Frameworks/Python.framework/Versions/6.2/lib/python2.6/site-packages/numpy/lib/io.py in savetxt(fname, X, fmt, delimiter) 784 785 for row in X: --> 786 fh.write(format % tuple(row) + '\n') 787 788 import re TypeError: float argument required, not numpy.ndarray I don't see this anywhere in the documentation of savetxt. And if that is a restriction it should probably be caught with an assertion. Numpy version 1.4.0. I see why ndim>2 may cause problems, as there is now way on 'loadtxt' to figure out the shape of the array, but I think it should throw an assertion and the documentation should be updated. Thanks, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsalvati at u.washington.edu Tue Sep 28 14:19:54 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Tue, 28 Sep 2010 11:19:54 -0700 Subject: [Numpy-discussion] Trying out datarray Message-ID: I am not sure if this is the correct place for such questions, but here goes: I am curious about datarray, but I haven't been able to get it to work. The module datarray does not appear to have a class DataArray (or DatArray). So I am confused how I am supposed to use it. Can anyone advise? My other question is whether datarray will be able to handle multiple data types in the same object; i.e. does it have the functionality of recarrays and R data.frames? John -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsseabold at gmail.com Tue Sep 28 14:24:11 2010 From: jsseabold at gmail.com (Skipper Seabold) Date: Tue, 28 Sep 2010 14:24:11 -0400 Subject: [Numpy-discussion] Trying out datarray In-Reply-To: References: Message-ID: On Tue, Sep 28, 2010 at 2:19 PM, John Salvatier wrote: > I am not sure if this is the correct place for such questions, but here > goes: > > I am curious about datarray, but I haven't been able to get it to work. The > module datarray does not appear to have a class DataArray (or DatArray). So > I am confused how I am supposed to use it. Can anyone advise? > I believe there's an "extra" layer... from datarray.datarray import DataArray http://fperez.github.com/datarray-doc/0.0.3/index.html Skipper From fperez.net at gmail.com Tue Sep 28 20:43:42 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 28 Sep 2010 17:43:42 -0700 Subject: [Numpy-discussion] Trying out datarray In-Reply-To: References: Message-ID: On Tue, Sep 28, 2010 at 11:19 AM, John Salvatier wrote: > My other question is whether datarray will be able to handle multiple data > types in the same object; i.e. does it have the functionality of recarrays > and R data.frames? > The actual array container is a normal numpy array, so its dtype can indeed be a structured one. The only changes datarray makes are 'on the outside', i.e. to axis management, not to the internal data contained in the individual cells of the array. Cheers, f From jsalvati at u.washington.edu Tue Sep 28 20:46:32 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Tue, 28 Sep 2010 17:46:32 -0700 Subject: [Numpy-discussion] Trying out datarray In-Reply-To: References: Message-ID: OK, that makes sense. Thanks guys! On Tue, Sep 28, 2010 at 5:43 PM, Fernando Perez wrote: > On Tue, Sep 28, 2010 at 11:19 AM, John Salvatier > wrote: > > My other question is whether datarray will be able to handle multiple > data > > types in the same object; i.e. does it have the functionality of > recarrays > > and R data.frames? > > > > The actual array container is a normal numpy array, so its dtype can > indeed be a structured one. The only changes datarray makes are 'on > the outside', i.e. to axis management, not to the internal data > contained in the individual cells of the array. > > Cheers, > > f > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmorton at bitfurnace.com Wed Sep 29 02:01:06 2010 From: dmorton at bitfurnace.com (Damien Morton) Date: Wed, 29 Sep 2010 16:01:06 +1000 Subject: [Numpy-discussion] indexed arrays ignoring duplicates Message-ID: lets say i have arrays: a = array((1,2,3,4,5)) indices = array((1,1,1,1)) and i perform operation: a[indices] += 1 the result is array([1, 3, 3, 4, 5]) in other words, the duplicates in?indices?are ignored if I wanted the duplicates not to be ignored, resulting in: array([1, 6, 3, 4, 5]) how would I go about this? the example above is somewhat trivial, what follows is exactly what I am trying to do: def inflate(self,pressure): ? ? faceforces = pressure * cross(self.verts[self.faces[:,1]]-self.verts[self.faces[:,0]], self.verts[self.faces[:,2]]-self.verts[self.faces[:,0]]) ? ? self.verts[self.faces[:,0]] += faceforces ? ? self.verts[self.faces[:,1]] += faceforces ? ? self.verts[self.faces[:,2]] += faceforces def constrain_lengths(self): ? ? vectors = self.verts[self.constraints[:,1]] - self.verts[self.constraints[:,0]] ? ? lengths = sqrt(sum(square(vectors), axis=1)) ? ? correction = 0.5 * (vectors.T * (1 - (self.restlengths / lengths))).T ? ? self.verts[self.constraints[:,0]] += correction ? ? self.verts[self.constraints[:,1]] -= correction def compute_normals(self): ? ? facenormals = cross(self.verts[self.faces[:,1]]-self.verts[self.faces[:,0]], self.verts[self.faces[:,2]]-self.verts[self.faces[:,0]]) ? ? self.normals.fill(0) ? ? self.normals[self.faces[:,0]] += facenormals ? ? self.normals[self.faces[:,1]] += facenormals ? ? self.normals[self.faces[:,2]] += facenormals ? ? lengths = sqrt(sum(square(self.normals), axis=1)) ? ? self.normals = (self.normals.T / lengths).T Ive been getting some very buggy results as a result of duplicates being ignored in my indexed inplace add/sub operations. From faltet at pytables.org Wed Sep 29 04:17:46 2010 From: faltet at pytables.org (Francesc Alted) Date: Wed, 29 Sep 2010 10:17:46 +0200 Subject: [Numpy-discussion] NPZ format Message-ID: <201009291017.46955.faltet@pytables.org> Hi, I'm going to give a seminar about serialization, and I'd like to describe the .npy format. I noticed that there is a variant of it called .npz that can pack several arrays in one single file. However, .npz does not use compression at all and I'm wondering what's the reason. I suppose that this is because you don't want to loose the possibility to memmap saved arrays, but can someone confirm this? Thanks, -- Francesc Alted From gordon at tolomea.com Wed Sep 29 08:25:04 2010 From: gordon at tolomea.com (Gordon Wrigley) Date: Wed, 29 Sep 2010 22:25:04 +1000 Subject: [Numpy-discussion] looking for code advice Message-ID: Hi First the disclaimer: This is my first numpy experience, so I have next to no idea what I'm doing. I've muddled through and managed to put together some code for my current problem and now that I have it going I'd like to hear any comments people may have on both my solution and other ways of approaching the problem. I have two goals here, I'd like to make the process run faster and I'd like to broaden my understanding of numpy as I can see from my brief use of it that it is a remarkably powerful tool. Now to the problem at hand. I find this difficult to explain but will try as best I can. The best word I have for the process is decimation. The input and output are both 3 dimensional arrays of uint8's. The output is half the size of the input along each dimension. Each cell [x,y,z] in the output corresponds to the 2x2x2 block [2*x:2*x+2, 2*y:2*y+2, 2*z:2*z+2] in the input. The tricky bit is in how the correspondence works. If all the cells in the input block have the same value then the cell in the output block will also have that value. Otherwise the output cell will have the value MIXED. Here is my current solution, from my limited testing it seems to produce the result I'm after. def decimate(data_in): in_x, in_y, in_z = data_in.shape out_x = in_x / 2 out_y = in_y / 2 out_z = in_z / 2 out_shape = out_x, out_y, out_z out_size = product(out_shape) # figure out which chunks are homogeneous reshaped_array = data_in.reshape(out_x, 2, out_y, 2, out_z, 2).transpose(0,2,4,1,3,5).reshape(out_x, out_y, out_z, 8) min_array = numpy.amin(reshaped_array, axis=3) max_array = numpy.amax(reshaped_array, axis=3) equal_array = numpy.equal(min_array, max_array) # select the actual value for the homogeneous chunks and MIXED for the heterogeneous decimated_array = data_in[::2,::2,::2] mixed_array = numpy.tile(MIXED, out_size).reshape(out_shape) data_out = numpy.where(equal_array, decimated_array, mixed_array) return data_out For the curious this is will be used to build a voxel octtree for a 3d graphics application. The final setup will be more complicated, this is the minimum that will let me get up and running. Regards Gordon P.S. congrats on numpy, it is a very impressive tool, I've only scraped the surface and it's already impressed me several times over. -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Sep 29 09:19:23 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 29 Sep 2010 09:19:23 -0400 Subject: [Numpy-discussion] looking for code advice In-Reply-To: References: Message-ID: On Wed, Sep 29, 2010 at 8:25 AM, Gordon Wrigley wrote: > Hi > First the disclaimer: This is my first numpy experience, so I have next to > no idea what I'm doing. > I've muddled through and managed to put together some code for my current > problem and now that I have it going I'd like to hear any comments people > may have on both my solution and other ways of approaching the problem. > I have two goals here, I'd like to make the process run faster?and I'd like > to broaden my understanding of numpy as I can see from my brief use of it > that it is a remarkably powerful tool. > Now to the problem at hand. I find this difficult to explain but will try as > best I can. > The best word I have for the process is decimation. The input and output are > both 3 dimensional arrays of uint8's. The output is half the size of the > input along each dimension. Each cell [x,y,z] in the output corresponds to > the 2x2x2 block [2*x:2*x+2,?2*y:2*y+2,?2*z:2*z+2] in the input. The tricky > bit is in how the correspondence works. If all the cells in the input block > have the same value then the cell in the output block will also have that > value. Otherwise the output cell will have the value MIXED. > Here is my current solution, from my limited testing it seems to produce the > result I'm after. > def decimate(data_in): > ?? ?in_x, in_y, in_z = data_in.shape > ?? ?out_x = in_x / 2 > ?? ?out_y = in_y / 2 > ?? ?out_z = in_z / 2 > ?? ?out_shape = out_x, out_y, out_z > ?? ?out_size = product(out_shape) > ?? ?# figure out which chunks are homogeneous > ?? ?reshaped_array = data_in.reshape(out_x, 2, out_y, 2, out_z, > 2).transpose(0,2,4,1,3,5).reshape(out_x, out_y, out_z, 8) > ?? ?min_array = numpy.amin(reshaped_array, axis=3) > ?? ?max_array = numpy.amax(reshaped_array, axis=3) > ?? ?equal_array = numpy.equal(min_array, max_array) > ?? ?# select the actual value for the homogeneous chunks and MIXED for the > heterogeneous > ?? ?decimated_array = data_in[::2,::2,::2] > ?? ?mixed_array = numpy.tile(MIXED, out_size).reshape(out_shape) > ?? ?data_out = numpy.where(equal_array, decimated_array, mixed_array) data_out = numpy.where(equal_array, decimated_array, MIXED) should work I don't see anything else, unless there is something in scipy.ndimage. I have to remember your reshape trick for 3d. (I don't know how many temporary arrays this creates.) Josef > ?? ?return data_out > For the curious this is will be used to build a voxel octtree for a 3d > graphics application. The final setup will be more complicated, this is the > minimum that will let me get up and running. > Regards > Gordon > P.S. congrats on numpy, it is a very impressive tool, I've only scraped the > surface and it's already impressed me several times over. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From tkg at lanl.gov Wed Sep 29 10:09:30 2010 From: tkg at lanl.gov (Thomas Kirk Gamble) Date: Wed, 29 Sep 2010 08:09:30 -0600 Subject: [Numpy-discussion] convert FORTRAN exponential format text to float Message-ID: <1285769370.23296.134.camel@koolasuchus.lanl.gov> I need to convert numbers read from a text file to floating point. The numbers are in the format 1.538D-06 (written by a FORTRAN application) and have varying amounts of whitespace between them from line to line. The function fromstring() deals with the whitespace just fine but 'dtype=float' doesn't correctly convert the data. It sees every thing up to the 'D' and ignores the rest(I assume expecting an 'e' to indicate the exponential). I was able to get around this using re.sub() to change the 'D' to 'e' in the string before using fromstring(), but I was wondering if python has any way to directly read this data as float? Google didn't find an answer. From david.froger at gmail.com Wed Sep 29 10:36:26 2010 From: david.froger at gmail.com (David Froger) Date: Wed, 29 Sep 2010 16:36:26 +0200 Subject: [Numpy-discussion] convert FORTRAN exponential format text to float In-Reply-To: <1285769370.23296.134.camel@koolasuchus.lanl.gov> References: <1285769370.23296.134.camel@koolasuchus.lanl.gov> Message-ID: Did you try loadtxt? I try to output something in the format 1.538D-06 with Fortran in order to test reading it with loadtxt, but I always get 1.538E-06. Where does the 'D' come from? -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.froger at gmail.com Wed Sep 29 10:45:57 2010 From: david.froger at gmail.com (David Froger) Date: Wed, 29 Sep 2010 16:45:57 +0200 Subject: [Numpy-discussion] convert FORTRAN exponential format text to float In-Reply-To: References: <1285769370.23296.134.camel@koolasuchus.lanl.gov> Message-ID: program write_txt real(kind=8):: x open(10,file='data.txt') do i = 1,10 x = i*2. write(10,fmt='(2(D12.3))') x,x**2 enddo close(10) end program write_txt In [1]: x,y = loadtxt('data.txt',unpack=True) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) /home/dfroger/tmp/ in () /usr/lib/python2.5/site-packages/numpy/lib/io.pyc in loadtxt(fname, dtype, comments, delimiter, converters, skiprows, usecols, unpack) 350 351 # Convert each value according to its column and store --> 352 X.append(tuple([conv(val) for (conv, val) in zip(converters, vals)])) 353 354 if len(dtype_types) > 1: ValueError: invalid literal for float(): 0.200D+01 Maybe the the arguments 'converters' of loadtxt can help? -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.froger at gmail.com Wed Sep 29 10:54:30 2010 From: david.froger at gmail.com (David Froger) Date: Wed, 29 Sep 2010 16:54:30 +0200 Subject: [Numpy-discussion] convert FORTRAN exponential format text to float In-Reply-To: References: <1285769370.23296.134.camel@koolasuchus.lanl.gov> Message-ID: program write_txt real(kind=8):: x open(10,file='data.txt') do i = 1,10 x = i*2. write(10,fmt='(2(D12.3))') x,x**2 enddo close(10) end program write_txt In [1]: def expDtofloat(s): ...: return float(s.replace('D','E')) ...: In [2]: x,y = loadtxt('data.txt',unpack=True,converters={0:expDtofloat,1:expDtofloat}) Not really better than re.sub() ... -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Sep 29 12:15:08 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 29 Sep 2010 11:15:08 -0500 Subject: [Numpy-discussion] indexed arrays ignoring duplicates In-Reply-To: References: Message-ID: On Wed, Sep 29, 2010 at 01:01, Damien Morton wrote: > lets say i have arrays: > > a = array((1,2,3,4,5)) > indices = array((1,1,1,1)) > > and i perform operation: > > a[indices] += 1 > > the result is > > array([1, 3, 3, 4, 5]) > > in other words, the duplicates in?indices?are ignored > > if I wanted the duplicates not to be ignored, resulting in: > > array([1, 6, 3, 4, 5]) > > how would I go about this? Use numpy.bincount() instead. The reason for the current behavior is that Python compiles the "x[i] += y" construct into three separate orthogonal operations. tmp = x.__getitem__(i) val = tmp.__iadd__(y) x.__setitem__(i, val) Each of these operations has well-defined semantics in numpy arrays primarily designed for other use cases. There is no way for each of them to know that they are in the "x[i] += y" idiom in order to do something different to achieve the semantics you want. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From tkg at lanl.gov Wed Sep 29 12:32:48 2010 From: tkg at lanl.gov (Thomas Kirk Gamble) Date: Wed, 29 Sep 2010 10:32:48 -0600 Subject: [Numpy-discussion] convert FORTRAN exponential format text to float In-Reply-To: References: <1285769370.23296.134.camel@koolasuchus.lanl.gov> Message-ID: <1285777968.26439.27.camel@koolasuchus.lanl.gov> On Wed, 2010-09-29 at 16:36 +0200, David Froger wrote: > Did you try loadtxt? I try to output something in the format 1.538D-06 > with Fortran in order to test reading it with loadtxt, but I always > get 1.538E-06. Where does the 'D' come from? No, I didn't, but I will. Preserving the 'D' isn't important, only proper conversion of the string to float. The data was created by an atmosphere modeling program called AMOEBA that was written in FORTRAN. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From josef.pktd at gmail.com Wed Sep 29 12:36:10 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 29 Sep 2010 12:36:10 -0400 Subject: [Numpy-discussion] looking for code advice In-Reply-To: References: Message-ID: On Wed, Sep 29, 2010 at 9:19 AM, wrote: > On Wed, Sep 29, 2010 at 8:25 AM, Gordon Wrigley wrote: >> Hi >> First the disclaimer: This is my first numpy experience, so I have next to >> no idea what I'm doing. >> I've muddled through and managed to put together some code for my current >> problem and now that I have it going I'd like to hear any comments people >> may have on both my solution and other ways of approaching the problem. >> I have two goals here, I'd like to make the process run faster?and I'd like >> to broaden my understanding of numpy as I can see from my brief use of it >> that it is a remarkably powerful tool. >> Now to the problem at hand. I find this difficult to explain but will try as >> best I can. >> The best word I have for the process is decimation. The input and output are >> both 3 dimensional arrays of uint8's. The output is half the size of the >> input along each dimension. Each cell [x,y,z] in the output corresponds to >> the 2x2x2 block [2*x:2*x+2,?2*y:2*y+2,?2*z:2*z+2] in the input. The tricky >> bit is in how the correspondence works. If all the cells in the input block >> have the same value then the cell in the output block will also have that >> value. Otherwise the output cell will have the value MIXED. >> Here is my current solution, from my limited testing it seems to produce the >> result I'm after. >> def decimate(data_in): >> ?? ?in_x, in_y, in_z = data_in.shape >> ?? ?out_x = in_x / 2 >> ?? ?out_y = in_y / 2 >> ?? ?out_z = in_z / 2 >> ?? ?out_shape = out_x, out_y, out_z >> ?? ?out_size = product(out_shape) >> ?? ?# figure out which chunks are homogeneous >> ?? ?reshaped_array = data_in.reshape(out_x, 2, out_y, 2, out_z, >> 2).transpose(0,2,4,1,3,5).reshape(out_x, out_y, out_z, 8) >> ?? ?min_array = numpy.amin(reshaped_array, axis=3) >> ?? ?max_array = numpy.amax(reshaped_array, axis=3) >> ?? ?equal_array = numpy.equal(min_array, max_array) maybe ptp==0 is faster numpy.ptp(a, axis=None, out=None) Range of values (maximum - minimum) along an axis. Josef >> ?? ?# select the actual value for the homogeneous chunks and MIXED for the >> heterogeneous >> ?? ?decimated_array = data_in[::2,::2,::2] >> ?? ?mixed_array = numpy.tile(MIXED, out_size).reshape(out_shape) >> ?? ?data_out = numpy.where(equal_array, decimated_array, mixed_array) > > data_out = numpy.where(equal_array, decimated_array, MIXED) > should work > > I don't see anything else, unless there is something in scipy.ndimage. > > I have to remember your reshape trick for 3d. (I don't know how many > temporary arrays this creates.) > > Josef > >> ?? ?return data_out >> For the curious this is will be used to build a voxel octtree for a 3d >> graphics application. The final setup will be more complicated, this is the >> minimum that will let me get up and running. >> Regards >> Gordon >> P.S. congrats on numpy, it is a very impressive tool, I've only scraped the >> surface and it's already impressed me several times over. >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > From pav at iki.fi Wed Sep 29 13:00:56 2010 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 29 Sep 2010 17:00:56 +0000 (UTC) Subject: [Numpy-discussion] indexed arrays ignoring duplicates References: Message-ID: Wed, 29 Sep 2010 11:15:08 -0500, Robert Kern wrote: [clip: inplace addition with duplicates] > Use numpy.bincount() instead. It might be worthwhile to add a separate helper function for this purpose. Bincount makes a copy that could be avoided, and it is difficult to find if you don't know about this trick. -- Pauli Virtanen From robert.kern at gmail.com Wed Sep 29 13:28:10 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 29 Sep 2010 12:28:10 -0500 Subject: [Numpy-discussion] indexed arrays ignoring duplicates In-Reply-To: References: Message-ID: On Wed, Sep 29, 2010 at 12:00, Pauli Virtanen wrote: > Wed, 29 Sep 2010 11:15:08 -0500, Robert Kern wrote: > [clip: inplace addition with duplicates] >> Use numpy.bincount() instead. > > It might be worthwhile to add a separate helper function for this > purpose. Bincount makes a copy that could be avoided, and it is difficult > to find if you don't know about this trick. I'm fairly certain that most of the arrays used are fairly small, as such things are reckoned. I'm not sure that in-place modification would win us much. And I'm not sure what other name for the function would make it easier to find. AFAICT, using bincount() this way is not really a "trick"; it's just the right way to do exactly this job. If anything, "x.fill(0);x[i] += 1" is the "trick". -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From ariver at enthought.com Wed Sep 29 13:48:33 2010 From: ariver at enthought.com (Aaron River) Date: Wed, 29 Sep 2010 12:48:33 -0500 Subject: [Numpy-discussion] Improvements to SciPy.org Server In-Reply-To: References: Message-ID: Greetings Gentle Folk, I'm happy to announce that the server hosting SciPy.org, as well as other OSS projects, has been migrated into a cluster assigned to our community support efforts. The base instance itself remains relatively unchanged, but is now running atop a more capable set of servers. (Improvements to the services themselves are next in line.) Yesterday afternoon, I allowed the system to run with the configurations carried over from the old hardware, while watching the services for warning signs of trouble. Then, that evening, I loosened things up and adjusted configurations to better utilize the available resources. Though I always expect there to be "gotchas", I've thus far observed much improved response times and throughput. If you do experience any problems, please do not hesitate to let us know. We hope you enjoy the improvements to SciPy.org. Thanks, -- Aaron River Sr IT Administrator Enthought, Inc. -- From d.p.reichert at sms.ed.ac.uk Wed Sep 29 15:03:12 2010 From: d.p.reichert at sms.ed.ac.uk (David Reichert) Date: Wed, 29 Sep 2010 15:03:12 -0400 Subject: [Numpy-discussion] Multiplying list of matrices with list of vectors Message-ID: Hi, I have a list of matrices W_k I'd like to multiply with a list of vectors v_k, or another way of looking at it, writing all W_k into a 3d array and all v_k into a 2d matrix/array, I'd like to compute matrix R as R_ik = sum_j W_ijk h_jk. Is there a fast way of doing that in numpy? Regards, David -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: not available URL: From mdroe at stsci.edu Wed Sep 29 15:59:50 2010 From: mdroe at stsci.edu (Michael Droettboom) Date: Wed, 29 Sep 2010 15:59:50 -0400 Subject: [Numpy-discussion] numpydoc: "Other Parameters" section Message-ID: <4CA39AB6.6080809@stsci.edu> We're having trouble using numpydoc formatted docstrings for one of our own project. It seems that the "Other parameters" section is not getting included in the output. A grep across Numpy's own source code (git master) reveals that this kind of section is used in only one place -- the docstring for "recarray". Curiously, the "Other Parameters" section is displayed in the documentation editor here: http://docs.scipy.org/numpy/docs/numpy.core.records.recarray/ But not in the generated Numpy docs online here: http://docs.scipy.org/doc/numpy/reference/generated/numpy.recarray.html Is this a bug? Mike -- Michael Droettboom Science Software Branch Space Telescope Science Institute Baltimore, Maryland, USA From charlesr.harris at gmail.com Wed Sep 29 16:19:40 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 29 Sep 2010 14:19:40 -0600 Subject: [Numpy-discussion] numpydoc: "Other Parameters" section In-Reply-To: <4CA39AB6.6080809@stsci.edu> References: <4CA39AB6.6080809@stsci.edu> Message-ID: On Wed, Sep 29, 2010 at 1:59 PM, Michael Droettboom wrote: > We're having trouble using numpydoc formatted docstrings for one of our > own project. It seems that the "Other parameters" section is not > getting included in the output. > > A grep across Numpy's own source code (git master) reveals that this > kind of section is used in only one place -- the docstring for > "recarray". Curiously, the "Other Parameters" section is displayed in > the documentation editor here: > > http://docs.scipy.org/numpy/docs/numpy.core.records.recarray/ > > But not in the generated Numpy docs online here: > > http://docs.scipy.org/doc/numpy/reference/generated/numpy.recarray.html > > Is this a bug? > > Probably. The Other Parameters section is one of those ideas that sounded good at the time but didn't end up used in practice. I expect that nobody noticed its absence before. Chuck. -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Sep 29 16:39:35 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 29 Sep 2010 16:39:35 -0400 Subject: [Numpy-discussion] numpydoc: "Other Parameters" section In-Reply-To: References: <4CA39AB6.6080809@stsci.edu> Message-ID: On Wed, Sep 29, 2010 at 4:19 PM, Charles R Harris wrote: > > > On Wed, Sep 29, 2010 at 1:59 PM, Michael Droettboom wrote: >> >> We're having trouble using numpydoc formatted docstrings for one of our >> own project. ?It seems that the "Other parameters" section is not >> getting included in the output. >> >> A grep across Numpy's own source code (git master) reveals that this >> kind of section is used in only one place -- the docstring for >> "recarray". ?Curiously, the "Other Parameters" section is displayed in >> the documentation editor here: >> >> ? http://docs.scipy.org/numpy/docs/numpy.core.records.recarray/ >> >> But not in the generated Numpy docs online here: >> >> ? http://docs.scipy.org/doc/numpy/reference/generated/numpy.recarray.html >> >> Is this a bug? >> > > Probably. The Other Parameters section is one of those ideas that sounded > good at the time but didn't end up used in practice. I expect that nobody > noticed its absence before. http://code.google.com/p/pydocweb/issues/detail?id=45 and scipy-dev mailinglist around March 18 Josef > Chuck. > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From josef.pktd at gmail.com Wed Sep 29 16:56:35 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 29 Sep 2010 16:56:35 -0400 Subject: [Numpy-discussion] numpydoc: "Other Parameters" section In-Reply-To: References: <4CA39AB6.6080809@stsci.edu> Message-ID: On Wed, Sep 29, 2010 at 4:39 PM, wrote: > On Wed, Sep 29, 2010 at 4:19 PM, Charles R Harris > wrote: >> >> >> On Wed, Sep 29, 2010 at 1:59 PM, Michael Droettboom wrote: >>> >>> We're having trouble using numpydoc formatted docstrings for one of our >>> own project. ?It seems that the "Other parameters" section is not >>> getting included in the output. >>> >>> A grep across Numpy's own source code (git master) reveals that this >>> kind of section is used in only one place -- the docstring for >>> "recarray". ?Curiously, the "Other Parameters" section is displayed in >>> the documentation editor here: >>> >>> ? http://docs.scipy.org/numpy/docs/numpy.core.records.recarray/ >>> >>> But not in the generated Numpy docs online here: >>> >>> ? http://docs.scipy.org/doc/numpy/reference/generated/numpy.recarray.html >>> >>> Is this a bug? >>> >> >> Probably. The Other Parameters section is one of those ideas that sounded >> good at the time but didn't end up used in practice. I expect that nobody >> noticed its absence before. > > http://code.google.com/p/pydocweb/issues/detail?id=45 > > and scipy-dev mailinglist around March 18 We discussed the same problem with Other Parameters, which is also a (independent ?) problem in the doc editor. But browsing the thread and ticket, I don't see any solution. Josef > > Josef > >> Chuck. >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > From gokhansever at gmail.com Wed Sep 29 17:01:57 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Wed, 29 Sep 2010 16:01:57 -0500 Subject: [Numpy-discussion] Appending/combining masked arrays Message-ID: Hello, Consider these two simple masked arrays: I[188]: a = np.ma.masked_equal([1,2,3], value=2) I[189]: b = np.ma.masked_equal([4,3,2], value=2) An operation like this voids the mask: I[190]: np.append(a,b) O[190]: masked_array(data = [1 2 3 4 3 2], mask = False, fill_value = 999999) In my real use case I have two lists (shown simplified versions): I[193]: all_measured[5::14][1:] O[193]: [masked_array(data = [425.82268 441.8043 432.69865 433.75158 420.42552 469.73359 483.80741 427.66887 466.7487 452.64255 438.14488 428.38871 416.38598 432.92884 440.74705 415.00694 430.1807 446.02079 428.1408 428.21708 461.37897 453.43518 433.90081 426.88591 451.15683 426.07399 410.7971 455.19179 389.01905 485.69204 505.35355 523.30598 502.00168 491.85421 485.75839 473.37061 459.24917 438.47531 424.09222 411.82773 409.27676 366.24813 362.0136 385.61986 350.38855 357.10589 390.84878 390.53565 332.60864 368.45913], mask = [False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False], fill_value = 1e+20) , masked_array(data = [762.00897 756.79155 773.59503 757.97867 746.20204 752.0657], mask = [False False False False False False], fill_value = 1e+20) ] I[194]: all_predicted[5::14][1:] O[194]: [masked_array(data = [601.587925396 615.382975948 637.565961135 662.845855035 630.180285797 910.714363555 886.048093691 912.616380221 1087.38406572 789.0075947 777.900831884 733.319025182 750.579326854 752.627618389 696.521605131 633.362074267 722.437789869 730.89750503 692.179530427 703.786215707 808.592649936 1006.89797524 818.839286207 767.260255009 787.622382926 831.332970348 949.016807581 783.981396594 643.768619685 654.417348215 681.516301642 753.577103851 654.092465489 628.484105951 691.461588689 800.901347497 630.894132084 610.977386345 512.926749811 653.74866061 587.915074604 531.106658494 562.265237436 606.32672755 563.281067561 546.715211886 604.210379352 475.66452212 454.426293217 656.039874394], mask = [False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False False], fill_value = 1e+20) , masked_array(data = [891.251806903 833.185882945 840.250000752 831.649215796 883.534378034 841.970022166], mask = [False False False False False False], fill_value = 1e+20) ] These lists have 42 varying size masked arrays in each. I want to be able to combine each list in one array --preferably in a masked array for not losing the mask information so that I can perform some overall statistics. What is the way to solve this issue? Thanks -- G?khan From pgmdevlist at gmail.com Wed Sep 29 17:37:39 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 29 Sep 2010 23:37:39 +0200 Subject: [Numpy-discussion] Appending/combining masked arrays In-Reply-To: References: Message-ID: <63312AF6-0C68-47FC-A9BD-253DB6684813@gmail.com> On Sep 29, 2010, at 11:01 PM, G?khan Sever wrote: > Hello, > > Consider these two simple masked arrays: > > I[188]: a = np.ma.masked_equal([1,2,3], value=2) > > I[189]: b = np.ma.masked_equal([4,3,2], value=2) > > An operation like this voids the mask: > > I[190]: np.append(a,b) > O[190]: > masked_array(data = [1 2 3 4 3 2], > mask = False, > fill_value = 999999) You're using a standard numpy function on a masked array. It's hardly surprising that you run into some issues. You should use the np.ma equivalent. Except of course that the equivalent doesn't exist yet... Please open a ticket. In the mean time, do it manually: * combining the .data of each item (masked array) of your list is straightforward * repeat for the masks. I advise you to use the getmaskarray function to get the mask of each item, otherwise you may get a `nomask` in the middle which will complicate things * combine the the two results in a new masked array. From josef.pktd at gmail.com Wed Sep 29 17:46:03 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 29 Sep 2010 17:46:03 -0400 Subject: [Numpy-discussion] Appending/combining masked arrays In-Reply-To: <63312AF6-0C68-47FC-A9BD-253DB6684813@gmail.com> References: <63312AF6-0C68-47FC-A9BD-253DB6684813@gmail.com> Message-ID: On Wed, Sep 29, 2010 at 5:37 PM, Pierre GM wrote: > > On Sep 29, 2010, at 11:01 PM, G?khan Sever wrote: > >> Hello, >> >> Consider these two simple masked arrays: >> >> I[188]: a = np.ma.masked_equal([1,2,3], value=2) >> >> I[189]: b = np.ma.masked_equal([4,3,2], value=2) >> >> An operation like this voids the mask: >> >> I[190]: np.append(a,b) >> O[190]: >> masked_array(data = [1 2 3 4 3 2], >> ? ? ? ? ? ? mask = False, >> ? ? ? fill_value = 999999) > > You're using a standard numpy function on a masked array. It's hardly surprising that you run into some issues. You should use the np.ma equivalent. Except of course that the equivalent doesn't exist yet... Please open a ticket. > > In the mean time, do it manually: > * combining the .data of each item (masked array) of your list is straightforward > * repeat for the masks. I advise you to use the getmaskarray function to get the mask of each item, otherwise you may get a `nomask` in the middle which will complicate things > * combine the the two results in a new masked array. any of the ma stack array function might also work, or not? Josef > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From gokhansever at gmail.com Wed Sep 29 18:01:49 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Wed, 29 Sep 2010 17:01:49 -0500 Subject: [Numpy-discussion] Appending/combining masked arrays In-Reply-To: <63312AF6-0C68-47FC-A9BD-253DB6684813@gmail.com> References: <63312AF6-0C68-47FC-A9BD-253DB6684813@gmail.com> Message-ID: > You're using a standard numpy function on a masked array. It's hardly surprising that you run into some issues. You should use the np.ma equivalent. Except of course that the equivalent doesn't exist yet... Please open a ticket. Here it comes -> http://projects.scipy.org/numpy/ticket/1623 > In the mean time, do it manually: > * combining the .data of each item (masked array) of your list is straightforward > * repeat for the masks. I advise you to use the getmaskarray function to get the mask of each item, otherwise you may get a `nomask` in the middle which will complicate things > * combine the the two results in a new masked array. Does this look good? I[256]: a = np.ma.masked_equal([1,2,3], value=2) I[257]: b = np.ma.masked_equal([4,3,2, 5], value=2) I[258]: c = np.append(a.data, b.data) I[251]: cmask = np.append(np.ma.getmaskarray (a), np.ma.getmaskarray (b)) I[252]: c O[252]: array([1, 2, 3, 4, 3, 2, 5]) I[253]: cmask O[253]: array([False, True, False, False, False, True, False], dtype=bool) I[254]: d = np.ma.masked_array(c, cmask) I[255]: d O[255]: masked_array(data = [1 -- 3 4 3 -- 5], mask = [False True False False False True False], fill_value = 999999) Probably I will need a loop to construct a similar ma array for 7 different masked arrays in a list. -- G?khan From pgmdevlist at gmail.com Wed Sep 29 18:09:05 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 30 Sep 2010 00:09:05 +0200 Subject: [Numpy-discussion] Appending/combining masked arrays In-Reply-To: References: <63312AF6-0C68-47FC-A9BD-253DB6684813@gmail.com> Message-ID: <396B8A95-5635-4779-87D2-456F906AA46C@gmail.com> On Sep 29, 2010, at 11:46 PM, josef.pktd at gmail.com wrote: > > any of the ma stack array function might also work, or not? In np.ma.extras ? Most likely, yes From gokhansever at gmail.com Wed Sep 29 18:15:15 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Wed, 29 Sep 2010 17:15:15 -0500 Subject: [Numpy-discussion] Appending/combining masked arrays In-Reply-To: <396B8A95-5635-4779-87D2-456F906AA46C@gmail.com> References: <63312AF6-0C68-47FC-A9BD-253DB6684813@gmail.com> <396B8A95-5635-4779-87D2-456F906AA46C@gmail.com> Message-ID: On Wed, Sep 29, 2010 at 5:09 PM, Pierre GM wrote: > > On Sep 29, 2010, at 11:46 PM, josef.pktd at gmail.com wrote: >> >> any of the ma stack array function might also work, or not? > > In np.ma.extras ? Most likely, yes This seems to do what I want: I[262]: np.ma.hstack(all_measured) O[262]: masked_array(data = [382.4828 375.57736 387.56132 ..., 371.29348 364.26819 374.77477], mask = [False False False ..., False False False], fill_value = 1e+20) I don't know why there are replicas under np.ma.extras. -- G?khan From pgmdevlist at gmail.com Wed Sep 29 18:19:55 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 30 Sep 2010 00:19:55 +0200 Subject: [Numpy-discussion] Appending/combining masked arrays In-Reply-To: References: <63312AF6-0C68-47FC-A9BD-253DB6684813@gmail.com> Message-ID: <849BD287-2F67-4B40-B841-D14D0213EC93@gmail.com> On Sep 30, 2010, at 12:01 AM, G?khan Sever wrote: >> You're using a standard numpy function on a masked array. It's hardly surprising that you run into some issues. You should use the np.ma equivalent. Except of course that the equivalent doesn't exist yet... Please open a ticket. > > Here it comes -> http://projects.scipy.org/numpy/ticket/1623 Thank you > Probably I will need a loop to construct a similar ma array for 7 > different masked arrays in a list. I think so. Just do what you would do if you had only ndarrays, but do it twice, once on the .data part and once on the .mask part, keeping in mind that you may have a nomask in the way, as I mentioned before (so use a getmaskarray to get the .mask) From pgmdevlist at gmail.com Wed Sep 29 18:22:07 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 30 Sep 2010 00:22:07 +0200 Subject: [Numpy-discussion] Appending/combining masked arrays In-Reply-To: References: <63312AF6-0C68-47FC-A9BD-253DB6684813@gmail.com> <396B8A95-5635-4779-87D2-456F906AA46C@gmail.com> Message-ID: <111119AD-E2B0-4398-99EB-58FE35C16662@gmail.com> On Sep 30, 2010, at 12:15 AM, G?khan Sever wrote: > On Wed, Sep 29, 2010 at 5:09 PM, Pierre GM wrote: >> >> On Sep 29, 2010, at 11:46 PM, josef.pktd at gmail.com wrote: >>> >>> any of the ma stack array function might also work, or not? >> >> In np.ma.extras ? Most likely, yes > > This seems to do what I want: > > I[262]: np.ma.hstack(all_measured) > > > I don't know why there are replicas under np.ma.extras. Well, hstack is implemented in numpy/ma/extras.py but this file is automatically imported w/ np.ma From jh at physics.ucf.edu Wed Sep 29 18:52:21 2010 From: jh at physics.ucf.edu (Joe Harrington) Date: Wed, 29 Sep 2010 18:52:21 -0400 Subject: [Numpy-discussion] numpydoc: "Other Parameters" section In-Reply-To: (numpy-discussion-request@scipy.org) References: Message-ID: > It seems that the "Other parameters" section is not > getting included in the output. The "Other parameters" is not often used in numpy because there are not generally dozens of "tweaking" parameters in its basic numerical routines. It will probably be more common in certain scipy pages, and if other packages adopted the format, it could get used a lot. Plotting packages, for example, often have a ton of "tweaking" parameters (line width, line color, line pattern, major axis tick interval, minor axis tick interval, x axis label, y axis label, title, axis font, title font, tick strings, etc.) and could use this section to good effect to sort those out from the parameters of primary interest. This would keep the output section and so forth near the top of the docstring for easy reading. So, it would be better to fix the problem than deprecate the section. --jh-- From gordon at tolomea.com Wed Sep 29 18:57:32 2010 From: gordon at tolomea.com (Gordon Wrigley) Date: Thu, 30 Sep 2010 08:57:32 +1000 Subject: [Numpy-discussion] looking for code advice In-Reply-To: References: Message-ID: Hi Josef Thanks for your comments. > numpy.ptp(a, axis=None, out=None) > Range of values (maximum - minimum) along an axis. I feel silly for not having noticed that function, it's right next to amin and amax in the docs. Both your suggestions improved the performance, but only a little, regardless they make the code simpler which is always a good thing. This is what I have currently: def decimate(data_in): in_x, in_y, in_z = data_in.shape out_x = in_x / 2 out_y = in_y / 2 out_z = in_z / 2 out_shape = out_x, out_y, out_z out_size = product(out_shape) # figure out which chunks are homogeneous reshaped_array = data_in.reshape(out_x, 2, out_y, 2, out_z, 2).transpose(0,2,4,1,3,5).reshape(out_x, out_y, out_z, 8) not_equal_array = numpy.ptp(reshaped_array, axis=3) # select the actual value for the homogeneous chunks and MIXED for the heterogeneous decimated_array = reshaped_array[:,:,:,0] data_out = numpy.where(not_equal_array, MIXED, decimated_array) return data_out Regards Gordon On Thu, Sep 30, 2010 at 2:36 AM, wrote: > On Wed, Sep 29, 2010 at 9:19 AM, wrote: > > On Wed, Sep 29, 2010 at 8:25 AM, Gordon Wrigley > wrote: > >> Hi > >> First the disclaimer: This is my first numpy experience, so I have next > to > >> no idea what I'm doing. > >> I've muddled through and managed to put together some code for my > current > >> problem and now that I have it going I'd like to hear any comments > people > >> may have on both my solution and other ways of approaching the problem. > >> I have two goals here, I'd like to make the process run faster and I'd > like > >> to broaden my understanding of numpy as I can see from my brief use of > it > >> that it is a remarkably powerful tool. > >> Now to the problem at hand. I find this difficult to explain but will > try as > >> best I can. > >> The best word I have for the process is decimation. The input and output > are > >> both 3 dimensional arrays of uint8's. The output is half the size of the > >> input along each dimension. Each cell [x,y,z] in the output corresponds > to > >> the 2x2x2 block [2*x:2*x+2, 2*y:2*y+2, 2*z:2*z+2] in the input. The > tricky > >> bit is in how the correspondence works. If all the cells in the input > block > >> have the same value then the cell in the output block will also have > that > >> value. Otherwise the output cell will have the value MIXED. > >> Here is my current solution, from my limited testing it seems to produce > the > >> result I'm after. > >> def decimate(data_in): > >> in_x, in_y, in_z = data_in.shape > >> out_x = in_x / 2 > >> out_y = in_y / 2 > >> out_z = in_z / 2 > >> out_shape = out_x, out_y, out_z > >> out_size = product(out_shape) > >> # figure out which chunks are homogeneous > >> reshaped_array = data_in.reshape(out_x, 2, out_y, 2, out_z, > >> 2).transpose(0,2,4,1,3,5).reshape(out_x, out_y, out_z, 8) > >> min_array = numpy.amin(reshaped_array, axis=3) > >> max_array = numpy.amax(reshaped_array, axis=3) > >> equal_array = numpy.equal(min_array, max_array) > > maybe ptp==0 is faster > > numpy.ptp(a, axis=None, out=None) > Range of values (maximum - minimum) along an axis. > > Josef > > >> # select the actual value for the homogeneous chunks and MIXED for > the > >> heterogeneous > >> decimated_array = data_in[::2,::2,::2] > >> mixed_array = numpy.tile(MIXED, out_size).reshape(out_shape) > >> data_out = numpy.where(equal_array, decimated_array, mixed_array) > > > > data_out = numpy.where(equal_array, decimated_array, MIXED) > > should work > > > > I don't see anything else, unless there is something in scipy.ndimage. > > > > I have to remember your reshape trick for 3d. (I don't know how many > > temporary arrays this creates.) > > > > Josef > > > >> return data_out > >> For the curious this is will be used to build a voxel octtree for a 3d > >> graphics application. The final setup will be more complicated, this is > the > >> minimum that will let me get up and running. > >> Regards > >> Gordon > >> P.S. congrats on numpy, it is a very impressive tool, I've only scraped > the > >> surface and it's already impressed me several times over. > >> > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at scipy.org > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmorton at bitfurnace.com Wed Sep 29 21:03:40 2010 From: dmorton at bitfurnace.com (Damien Morton) Date: Thu, 30 Sep 2010 11:03:40 +1000 Subject: [Numpy-discussion] indexed arrays ignoring duplicates In-Reply-To: References: Message-ID: On Thu, Sep 30, 2010 at 3:28 AM, Robert Kern wrote: > On Wed, Sep 29, 2010 at 12:00, Pauli Virtanen wrote: >> Wed, 29 Sep 2010 11:15:08 -0500, Robert Kern wrote: >> [clip: inplace addition with duplicates] >>> Use numpy.bincount() instead. >> >> It might be worthwhile to add a separate helper function for this >> purpose. Bincount makes a copy that could be avoided, and it is difficult >> to find if you don't know about this trick. > > I'm fairly certain that most of the arrays used are fairly small, as > such things are reckoned. I'm not sure that in-place modification > would win us much. And I'm not sure what other name for the function > would make it easier to find. AFAICT, using bincount() this way is not > really a "trick"; it's just the right way to do exactly this job. If > anything, "x.fill(0);x[i] += 1" is the "trick". bincount only works for gathering/accumulating scalars. Even the 'weights' parameter is limited to scalars. I propose the name 'gather()' for the helper function that does this. From josef.pktd at gmail.com Wed Sep 29 21:11:20 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 29 Sep 2010 21:11:20 -0400 Subject: [Numpy-discussion] indexed arrays ignoring duplicates In-Reply-To: References: Message-ID: On Wed, Sep 29, 2010 at 9:03 PM, Damien Morton wrote: > On Thu, Sep 30, 2010 at 3:28 AM, Robert Kern wrote: >> On Wed, Sep 29, 2010 at 12:00, Pauli Virtanen wrote: >>> Wed, 29 Sep 2010 11:15:08 -0500, Robert Kern wrote: >>> [clip: inplace addition with duplicates] >>>> Use numpy.bincount() instead. >>> >>> It might be worthwhile to add a separate helper function for this >>> purpose. Bincount makes a copy that could be avoided, and it is difficult >>> to find if you don't know about this trick. >> >> I'm fairly certain that most of the arrays used are fairly small, as >> such things are reckoned. I'm not sure that in-place modification >> would win us much. And I'm not sure what other name for the function >> would make it easier to find. AFAICT, using bincount() this way is not >> really a "trick"; it's just the right way to do exactly this job. If >> anything, "x.fill(0);x[i] += 1" is the "trick". > > bincount only works for gathering/accumulating scalars. Even the > 'weights' parameter is limited to scalars. Do you mean that bincount only works with 1d arrays? I also think that this is a major limitation of it. > > I propose the name 'gather()' for the helper function that does this. I don't think "gather" is an obvious name to search for. Josef > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From fperez.net at gmail.com Wed Sep 29 21:22:38 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 29 Sep 2010 18:22:38 -0700 Subject: [Numpy-discussion] Improvements to SciPy.org Server In-Reply-To: References: Message-ID: On Wed, Sep 29, 2010 at 10:48 AM, Aaron River wrote: > Greetings Gentle Folk, > > I'm happy to announce that the server hosting SciPy.org, > as well as other OSS projects, has been migrated into a > cluster assigned to our community support efforts. As someone who has been grateful, yet frustrated over the years because the flakiness of the server has impacted ipython on occasion, I'd like to thank you for this! I can tell right away that things are *much* better, because this page had become very hit-or-miss on whether it would even render at all (and when it did, it usually took forever to come back): http://www.scipy.org/Topical_Software and now it's instant. I hadn't seen that page render this quickly since the day I created it as a tiny list of bookmarks some 6 years ago! So many, many thanks :) f From charlesr.harris at gmail.com Wed Sep 29 21:34:36 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 29 Sep 2010 19:34:36 -0600 Subject: [Numpy-discussion] Multiplying list of matrices with list of vectors In-Reply-To: References: Message-ID: On Wed, Sep 29, 2010 at 1:03 PM, David Reichert wrote: > Hi, > > I have a list of matrices W_k I'd like to multiply with a list of vectors > v_k, > or another way of looking at it, writing all W_k into a 3d array and all > v_k into a 2d matrix/array, I'd like to compute matrix R as > > R_ik = sum_j W_ijk h_jk. > > > Is there a fast way of doing that in numpy? > > There is a way to do it: > Some operations on stacks of small matrices are easy to get, for instance, > +,-,*,/, and matrix multiply. The last is the interesting one. If A and B > are stacks of matrices with the same number of dimensions with the matrices > stored in the last two indices, then > > sum(A[...,:,:,newaxis]*B[...,newaxis,:,:], axis=-2) > > is the matrix-wise multiplication of the two stacks. If B is replaced by a > stack of 1D vectors, x, it is even simpler: > > sum(A[...,:,:]*x[...,newaxis,:], axis=-1) > > This doesn't go through BLAS, but for large stacks of small matrices it > might be even faster than BLAS because BLAS is kinda slow for small > matrices. It is probably a bit faster but uses more memory than using, say, list comprehensions. There is infrastructure in Numpy to build ufuncs for this sort of thing but it hasn't been used yet. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Sep 29 23:17:18 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 29 Sep 2010 22:17:18 -0500 Subject: [Numpy-discussion] indexed arrays ignoring duplicates In-Reply-To: References: Message-ID: On Wed, Sep 29, 2010 at 20:11, wrote: > On Wed, Sep 29, 2010 at 9:03 PM, Damien Morton wrote: >> On Thu, Sep 30, 2010 at 3:28 AM, Robert Kern wrote: >>> On Wed, Sep 29, 2010 at 12:00, Pauli Virtanen wrote: >>>> Wed, 29 Sep 2010 11:15:08 -0500, Robert Kern wrote: >>>> [clip: inplace addition with duplicates] >>>>> Use numpy.bincount() instead. >>>> >>>> It might be worthwhile to add a separate helper function for this >>>> purpose. Bincount makes a copy that could be avoided, and it is difficult >>>> to find if you don't know about this trick. >>> >>> I'm fairly certain that most of the arrays used are fairly small, as >>> such things are reckoned. I'm not sure that in-place modification >>> would win us much. And I'm not sure what other name for the function >>> would make it easier to find. AFAICT, using bincount() this way is not >>> really a "trick"; it's just the right way to do exactly this job. If >>> anything, "x.fill(0);x[i] += 1" is the "trick". >> >> bincount only works for gathering/accumulating scalars. Even the >> 'weights' parameter is limited to scalars. > > Do you mean that bincount only works with 1d arrays? I also think that > this is a major limitation of it. Feel free to change it. I think that extending the weights array to allow greater dimensions is an obvious extension of the current semantics. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From dmorton at bitfurnace.com Wed Sep 29 23:24:55 2010 From: dmorton at bitfurnace.com (Damien Morton) Date: Thu, 30 Sep 2010 13:24:55 +1000 Subject: [Numpy-discussion] indexed arrays ignoring duplicates In-Reply-To: References: Message-ID: On Thu, Sep 30, 2010 at 11:11 AM, wrote: >> bincount only works for gathering/accumulating scalars. Even the >> 'weights' parameter is limited to scalars. > > Do you mean that bincount only works with 1d arrays? I also think that > this is a major limitation of it. >>> from numpy import * >>> a = array((1,2,2,3,3)) >>> w = array(((1,2),(3,4),(5,6),(7,8),(9,10))) >>> bincount(a,weights=w) Traceback (most recent call last): File "", line 1, in ValueError: object too deep for desired array >>> w0 = array((1,2,3,4,5)) >>> bincount(a,weights=w0) array([ 0., 1., 5., 9.]) >> I propose the name 'gather()' for the helper function that does this. > > I don't think "gather" is an obvious name to search for. "gather" is the name that the GPGPU community uses to describe this kind of operation. Not just for summation but for any kind of indexed reducing operation. From dmorton at bitfurnace.com Wed Sep 29 23:41:27 2010 From: dmorton at bitfurnace.com (Damien Morton) Date: Thu, 30 Sep 2010 13:41:27 +1000 Subject: [Numpy-discussion] indexed arrays ignoring duplicates In-Reply-To: References: Message-ID: >>> I propose the name 'gather()' for the helper function that does this. >> >> I don't think "gather" is an obvious name to search for. > > "gather" is the name that the GPGPU community uses to describe this > kind of operation. Not just for summation but for any kind of indexed > reducing operation. gather(a,to_indices, b, from_indices, reduce_op) From josef.pktd at gmail.com Thu Sep 30 00:07:28 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 30 Sep 2010 00:07:28 -0400 Subject: [Numpy-discussion] indexed arrays ignoring duplicates In-Reply-To: References: Message-ID: On Wed, Sep 29, 2010 at 11:24 PM, Damien Morton wrote: > On Thu, Sep 30, 2010 at 11:11 AM, ? wrote: >>> bincount only works for gathering/accumulating scalars. Even the >>> 'weights' parameter is limited to scalars. >> >> Do you mean that bincount only works with 1d arrays? I also think that >> this is a major limitation of it. > >>>> from numpy import * >>>> a = array((1,2,2,3,3)) >>>> w = array(((1,2),(3,4),(5,6),(7,8),(9,10))) >>>> bincount(a,weights=w) > Traceback (most recent call last): > ?File "", line 1, in > ValueError: object too deep for desired array >>>> w0 = array((1,2,3,4,5)) >>>> bincount(a,weights=w0) > array([ 0., ?1., ?5., ?9.]) Since I'm not a C person to change bincount, how about >>> a = np.array((1,2,2,3,3)) >>> w = np.array(((1,2),(3,4),(5,6),(7,8),(9,10))) >>> a2 = np.array((1,2,2,3,3))[:,None]-1 + np.array([0, a.max()]) >>> a array([1, 2, 2, 3, 3]) >>> w array([[ 1, 2], [ 3, 4], [ 5, 6], [ 7, 8], [ 9, 10]]) >>> np.bincount(a2.ravel(),weights=w.ravel()).reshape(2,-1).T array([[ 1., 2.], [ 8., 10.], [ 16., 18.]]) I never thought of doing this before and I have been using bincount for some time. > >>> I propose the name 'gather()' for the helper function that does this. >> >> I don't think "gather" is an obvious name to search for. > > "gather" is the name that the GPGPU community uses to describe this > kind of operation. Not just for summation but for any kind of indexed > reducing operation. Some group functions that Travis is planning, might go in this direction. Josef > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From josef.pktd at gmail.com Thu Sep 30 00:45:14 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 30 Sep 2010 00:45:14 -0400 Subject: [Numpy-discussion] indexed arrays ignoring duplicates In-Reply-To: References: Message-ID: On Thu, Sep 30, 2010 at 12:07 AM, wrote: > On Wed, Sep 29, 2010 at 11:24 PM, Damien Morton wrote: >> On Thu, Sep 30, 2010 at 11:11 AM, ? wrote: >>>> bincount only works for gathering/accumulating scalars. Even the >>>> 'weights' parameter is limited to scalars. >>> >>> Do you mean that bincount only works with 1d arrays? I also think that >>> this is a major limitation of it. >> >>>>> from numpy import * >>>>> a = array((1,2,2,3,3)) >>>>> w = array(((1,2),(3,4),(5,6),(7,8),(9,10))) >>>>> bincount(a,weights=w) >> Traceback (most recent call last): >> ?File "", line 1, in >> ValueError: object too deep for desired array >>>>> w0 = array((1,2,3,4,5)) >>>>> bincount(a,weights=w0) >> array([ 0., ?1., ?5., ?9.]) > > Since I'm not a C person to change bincount, how about > >>>> a = np.array((1,2,2,3,3)) >>>> w = np.array(((1,2),(3,4),(5,6),(7,8),(9,10))) >>>> a2 = np.array((1,2,2,3,3))[:,None]-1 + np.array([0, a.max()]) >>>> a > array([1, 2, 2, 3, 3]) >>>> w > array([[ 1, ?2], > ? ? ? [ 3, ?4], > ? ? ? [ 5, ?6], > ? ? ? [ 7, ?8], > ? ? ? [ 9, 10]]) >>>> np.bincount(a2.ravel(),weights=w.ravel()).reshape(2,-1).T > array([[ ?1., ? 2.], > ? ? ? [ ?8., ?10.], > ? ? ? [ 16., ?18.]]) > > I never thought of doing this before and I have been using bincount > for some time. for future search, this seems to work >>> w = np.arange(5*4).reshape(5,4) >>> a = np.random.randint(5,8, size=5) >>> a array([6, 5, 6, 5, 7]) >>> w array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11], [12, 13, 14, 15], [16, 17, 18, 19]]) >>> a2 = a[:,None]-a.min() + (a.ptp()+1) * np.arange(w.shape[1]) >>> np.bincount(a2.ravel(),weights=w.ravel()).reshape(w.shape[1],-1).T array([[ 16., 18., 20., 22.], [ 8., 10., 12., 14.], [ 16., 17., 18., 19.]]) will include row of zeros for indices between a.min and a.max that have zero count Josef > > >> >>>> I propose the name 'gather()' for the helper function that does this. >>> >>> I don't think "gather" is an obvious name to search for. >> >> "gather" is the name that the GPGPU community uses to describe this >> kind of operation. Not just for summation but for any kind of indexed >> reducing operation. > > Some group functions that Travis is planning, might go in this direction. > > Josef > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > From seb.haase at gmail.com Thu Sep 30 03:15:17 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Thu, 30 Sep 2010 09:15:17 +0200 Subject: [Numpy-discussion] Improvements to SciPy.org Server In-Reply-To: References: Message-ID: On Thu, Sep 30, 2010 at 3:22 AM, Fernando Perez wrote: > On Wed, Sep 29, 2010 at 10:48 AM, Aaron River wrote: >> Greetings Gentle Folk, >> >> I'm happy to announce that the server hosting SciPy.org, >> as well as other OSS projects, has been migrated into a >> cluster assigned to our community support efforts. > > As someone who has been grateful, yet frustrated over the years > because the flakiness of the server has impacted ipython on occasion, > I'd like to thank you for this! > > I can tell right away that things are *much* better, because this page > had become very hit-or-miss on whether it would even render at all > (and when it did, it usually took forever to come back): > > http://www.scipy.org/Topical_Software > > and now it's instant. ?I hadn't seen that page render this quickly > since the day I created it as a tiny list of bookmarks some 6 years > ago! > > So many, many thanks :) > On this note, I looked at this page and the first two links I looked at (pycode & graceplot) where broken ... Is there a tool to highlight broken links in some way .... ? Obviously this is a general problem of wiki sites that exceed a certain life time ;-) (maybe there is a moinmoin plugin ...) Thanks for scipy, Sebastian Haase From scott.sinclair.za at gmail.com Thu Sep 30 04:15:20 2010 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Thu, 30 Sep 2010 10:15:20 +0200 Subject: [Numpy-discussion] Trouble cloning numpy Github repo over HTTP Message-ID: Hi, I'm behind a firewall that doesn't allow me to use the git protocol so I can't use the git:// URL. I see the following problem: $ git clone http://github.com/numpy/numpy.git numpy Initialized empty Git repository in /home/scott/external_repos/numpy/.git/ error: RPC failed; result=22, HTTP code = 417 I never have trouble cloning other repos off of Github over HTTP. See http://support.github.com/discussions/repos/4323-error-rpc-failed-result22-http-code-411 Does anyone else see this problem? Thanks, Scott From ben.root at ou.edu Thu Sep 30 10:15:16 2010 From: ben.root at ou.edu (Benjamin Root) Date: Thu, 30 Sep 2010 09:15:16 -0500 Subject: [Numpy-discussion] Trouble cloning numpy Github repo over HTTP In-Reply-To: References: Message-ID: On Thu, Sep 30, 2010 at 3:15 AM, Scott Sinclair wrote: > Hi, > > I'm behind a firewall that doesn't allow me to use the git protocol so > I can't use the git:// URL. > > I see the following problem: > > $ git clone http://github.com/numpy/numpy.git numpy > Initialized empty Git repository in /home/scott/external_repos/numpy/.git/ > error: RPC failed; result=22, HTTP code = 417 > > I never have trouble cloning other repos off of Github over HTTP. > > See > http://support.github.com/discussions/repos/4323-error-rpc-failed-result22-http-code-411 > > Does anyone else see this problem? > > Thanks, > Scott > HTTP code 417 is: *417 - Expectation Failed* The expectation given in the Expect request-header could not be fulfilled by the server. Because of this, I suspect it has something to do with the http proxy, but given that you aren't having issues with other repos, I am not sure exactly how that can be the case. An incorrectly configured proxy should have the same issue with other http-based repos... Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From ariver at enthought.com Thu Sep 30 11:15:56 2010 From: ariver at enthought.com (Aaron River) Date: Thu, 30 Sep 2010 10:15:56 -0500 Subject: [Numpy-discussion] Trouble cloning numpy Github repo over HTTP In-Reply-To: References: Message-ID: Hello Scott, If you're allowed access to arbitrary https urls, try: git clone https://github.com/numpy/numpy.git numpy -- Aaron On Thu, Sep 30, 2010 at 03:15, Scott Sinclair wrote: > > Hi, > > I'm behind a firewall that doesn't allow me to use the git protocol so > I can't use the git:// URL. > > I see the following problem: > > $ git clone http://github.com/numpy/numpy.git numpy > Initialized empty Git repository in /home/scott/external_repos/numpy/.git/ > error: RPC failed; result=22, HTTP code = 417 > > I never have trouble cloning other repos off of Github over HTTP. > > See http://support.github.com/discussions/repos/4323-error-rpc-failed-result22-http-code-411 > > Does anyone else see this problem? > > Thanks, > Scott > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From rspeer at MIT.EDU Thu Sep 30 11:39:26 2010 From: rspeer at MIT.EDU (Rob Speer) Date: Thu, 30 Sep 2010 11:39:26 -0400 Subject: [Numpy-discussion] datarray repositories have diverged Message-ID: There's some DataArray code that I've had for a while, but I just finished it up and tested it today. Most notably, it includes a new __str__ for DataArrays that does some nice layout and includes ticks: In [9]: print d_arr country year --------- ------------------------------------------------- 1994 1998 2002 2006 2010 Netherlan -0.505758 0.096597 1.083148 -0.450156 0.172754 Uruguay 1.772182 -0.113394 -0.781307 1.002416 -0.64925 Germany -2.013874 0.283947 1.170848 -0.504823 0.448497 Spain -0.725844 0.909713 -1.191371 -0.465167 -1.518764 (The layout functions in datarray.print_grid actually work with any ndarray, so you can use it as an alternative to the __str__ in NumPy.) However, I notice that all the new development on datarray is happening on Fernando Perez's branch, which mine diverged from long ago. I forked from Lluis (jesusabdullah)'s branch, which was the most active at the time, and I got all but the most recent changes merged back in. But that branch in turn was never merged back into fperez's. The divergence point is even before I added the syntax for named axes and ticks by name (such as arr.named['Spain', 1994:2010] or arr.year.named[2010]). Does it make sense to merge my branch into the main one? http://github.com/rspeer/datarray -- Rob From scott.sinclair.za at gmail.com Thu Sep 30 12:02:05 2010 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Thu, 30 Sep 2010 18:02:05 +0200 Subject: [Numpy-discussion] Trouble cloning numpy Github repo over HTTP In-Reply-To: References: Message-ID: On 30 September 2010 17:15, Aaron River wrote: > If you're allowed access to arbitrary https urls, try: > > git clone https://github.com/numpy/numpy.git numpy Thanks Aaron. I'm pretty sure I tried that and failed, but I'm at home now (no proxy) but will try again tomorrow. Looks like our proxy might be at fault. Cheers, Scott From charlesr.harris at gmail.com Thu Sep 30 12:08:41 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 30 Sep 2010 10:08:41 -0600 Subject: [Numpy-discussion] FYI: Nose Message-ID: Looks like official nose support for Python 3.x might be in the pipeline . Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Sep 30 12:20:16 2010 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 30 Sep 2010 11:20:16 -0500 Subject: [Numpy-discussion] NPZ format In-Reply-To: <201009291017.46955.faltet@pytables.org> References: <201009291017.46955.faltet@pytables.org> Message-ID: On Wed, Sep 29, 2010 at 03:17, Francesc Alted wrote: > Hi, > > I'm going to give a seminar about serialization, and I'd like to > describe the .npy format. ?I noticed that there is a variant of it > called .npz that can pack several arrays in one single file. > > However, .npz does not use compression at all and I'm wondering what's > the reason. ?I suppose that this is because you don't want to loose the > possibility to memmap saved arrays, but can someone confirm this? While I suspect it's possible, I'm certain we don't have any code that actually does it. Most likely the author assumed that it would be faster (or tested it to be faster with their CPU/hard disk configuration) to not compress. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From d.p.reichert at sms.ed.ac.uk Thu Sep 30 12:21:05 2010 From: d.p.reichert at sms.ed.ac.uk (David Reichert) Date: Thu, 30 Sep 2010 12:21:05 -0400 Subject: [Numpy-discussion] Multiplying list of matrices with list of vectors In-Reply-To: References: Message-ID: Hi, Thanks for the tip. For small matrices this method seems to be faster, however, for matrices of the size I'm using it actually seems to be slower than simply looping... Regards, David On Wed, Sep 29, 2010 at 9:34 PM, Charles R Harris wrote: > > > On Wed, Sep 29, 2010 at 1:03 PM, David Reichert > wrote: > >> Hi, >> >> I have a list of matrices W_k I'd like to multiply with a list of vectors >> v_k, >> or another way of looking at it, writing all W_k into a 3d array and all >> v_k into a 2d matrix/array, I'd like to compute matrix R as >> >> R_ik = sum_j W_ijk h_jk. >> >> >> Is there a fast way of doing that in numpy? >> >> > There is a way to do it: > > > Some operations on stacks of small matrices are easy to get, for > instance, > > +,-,*,/, and matrix multiply. The last is the interesting one. If A and B > > are stacks of matrices with the same number of dimensions with the > matrices > > stored in the last two indices, then > > > > sum(A[...,:,:,newaxis]*B[...,newaxis,:,:], axis=-2) > > > > is the matrix-wise multiplication of the two stacks. If B is replaced by > a > > stack of 1D vectors, x, it is even simpler: > > > > sum(A[...,:,:]*x[...,newaxis,:], axis=-1) > > > > This doesn't go through BLAS, but for large stacks of small matrices it > > might be even faster than BLAS because BLAS is kinda slow for small > > matrices. > > It is probably a bit faster but uses more memory than using, say, list > comprehensions. There is infrastructure in Numpy to build ufuncs for this > sort of thing but it hasn't been used yet. > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: not available URL: From xscript at gmx.net Thu Sep 30 12:22:23 2010 From: xscript at gmx.net (=?utf-8?Q?Llu=C3=ADs?=) Date: Thu, 30 Sep 2010 18:22:23 +0200 Subject: [Numpy-discussion] datarray repositories have diverged In-Reply-To: (Rob Speer's message of "Thu, 30 Sep 2010 11:39:26 -0400") References: Message-ID: <87sk0r6w1s.fsf@ginnungagap.bsc.es> Rob Speer writes: > However, I notice that all the new development on datarray is > happening on Fernando Perez's branch, which mine diverged from long > ago. I forked from Lluis (jesusabdullah)'s branch, which was the most > active at the time, and I got all but the most recent changes merged > back in. But that branch in turn was never merged back into fperez's. Ups! I thought my master branch was obsolete after the first sprint, so I deleted it and re-branched from fperez's. Thus, I suppose that comparing against my current master won't be useful to you. BTW, my fix branches are incomplete (no tests and doc have been updated), but in the future, how should they be merged (if they should be)? I mean, should datarray fork from the new github numpy into a new repository owned by a "datarray" user? I don't know much about how these kind of things are managed on github, but I remember some comments about that. apa! -- "And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer." -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth From rspeer at MIT.EDU Thu Sep 30 12:41:42 2010 From: rspeer at MIT.EDU (Rob Speer) Date: Thu, 30 Sep 2010 12:41:42 -0400 Subject: [Numpy-discussion] datarray repositories have diverged In-Reply-To: <87sk0r6w1s.fsf@ginnungagap.bsc.es> References: <87sk0r6w1s.fsf@ginnungagap.bsc.es> Message-ID: The fact that I wasn't around for the sprint probably has a lot to do with how much the code had diverged. But it's not too bad -- I merged Fernando's branch into mine and only had to change a couple of things to make the tests pass. There seem to be two general patterns for decentralized projects on GitHub: either you have one de facto leader who owns what everyone considers the main branch (this is what datarray is doing now, with Fernando as the leader), or you create a GitHub "organization" that owns the main branch and make a bunch of key people members of the organization (which is what numpy is doing). The way you'd usually get something merged in this kind of project is to send a pull request to the leader using the "Pull Request" button. But in this case, I'm basically making my pull request on the mailing list, because it's not straightforward enough for a simple pull request. -- Rob On Thu, Sep 30, 2010 at 12:22 PM, Llu?s wrote: > Rob Speer writes: > >> However, I notice that all the new development on datarray is >> happening on Fernando Perez's branch, which mine diverged from long >> ago. I forked from Lluis (jesusabdullah)'s branch, which was the most >> active at the time, and I got all but the most recent changes merged >> back in. But that branch in turn was never merged back into fperez's. > > Ups! I thought my master branch was obsolete after the first sprint, so > I deleted it and re-branched from fperez's. Thus, I suppose that > comparing against my current master won't be useful to you. > > BTW, my fix branches are incomplete (no tests and doc have been > updated), but in the future, how should they be merged (if they should > be)? ?I mean, should datarray fork from the new github numpy into a new > repository owned by a "datarray" user? I don't know much about how these > kind of things are managed on github, but I remember some comments about > that. > > apa! > > -- > ?"And it's much the same thing with knowledge, for whenever you learn > ?something new, the whole world becomes that much richer." > ?-- The Princess of Pure Reason, as told by Norton Juster in The Phantom > ?Tollbooth > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From mdroe at stsci.edu Thu Sep 30 15:13:04 2010 From: mdroe at stsci.edu (Michael Droettboom) Date: Thu, 30 Sep 2010 15:13:04 -0400 Subject: [Numpy-discussion] numpydoc: "Other Parameters" section In-Reply-To: References: Message-ID: <4CA4E140.5090400@stsci.edu> Is the solution as simple as the attached diff? It works for me, but I don't understand all the implications. Mike On 09/29/2010 06:52 PM, Joe Harrington wrote: >> It seems that the "Other parameters" section is not >> getting included in the output. >> > The "Other parameters" is not often used in numpy because there are > not generally dozens of "tweaking" parameters in its basic numerical > routines. It will probably be more common in certain scipy pages, and > if other packages adopted the format, it could get used a lot. > Plotting packages, for example, often have a ton of "tweaking" > parameters (line width, line color, line pattern, major axis tick > interval, minor axis tick interval, x axis label, y axis label, title, > axis font, title font, tick strings, etc.) and could use this section > to good effect to sort those out from the parameters of primary > interest. This would keep the output section and so forth near the > top of the docstring for easy reading. > > So, it would be better to fix the problem than deprecate the section. > > --jh-- > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Michael Droettboom Science Software Branch Space Telescope Science Institute Baltimore, Maryland, USA -------------- next part -------------- A non-text attachment was scrubbed... Name: other_parameters.diff Type: text/x-patch Size: 1989 bytes Desc: not available URL: From josef.pktd at gmail.com Thu Sep 30 15:41:08 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 30 Sep 2010 15:41:08 -0400 Subject: [Numpy-discussion] numpydoc: "Other Parameters" section In-Reply-To: <4CA4E140.5090400@stsci.edu> References: <4CA4E140.5090400@stsci.edu> Message-ID: On Thu, Sep 30, 2010 at 3:13 PM, Michael Droettboom wrote: > Is the solution as simple as the attached diff? ?It works for me, but I > don't understand all the implications. just minor point using "title" might be better now >>> 'other parameters'.capitalize() 'Other parameters' >>> 'other parameters'.title() 'Other Parameters' (I would say looks ok to me, except I never looked much at the code.) Thanks, Josef > > Mike > > On 09/29/2010 06:52 PM, Joe Harrington wrote: >>> >>> It seems that the "Other parameters" section is not >>> getting included in the output. >>> >> >> The "Other parameters" is not often used in numpy because there are >> not generally dozens of "tweaking" parameters in its basic numerical >> routines. ?It will probably be more common in certain scipy pages, and >> if other packages adopted the format, it could get used a lot. >> Plotting packages, for example, often have a ton of "tweaking" >> parameters (line width, line color, line pattern, major axis tick >> interval, minor axis tick interval, x axis label, y axis label, title, >> axis font, title font, tick strings, etc.) and could use this section >> to good effect to sort those out from the parameters of primary >> interest. ?This would keep the output section and so forth near the >> top of the docstring for easy reading. >> >> So, it would be better to fix the problem than deprecate the section. >> >> --jh-- >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > -- > Michael Droettboom > Science Software Branch > Space Telescope Science Institute > Baltimore, Maryland, USA > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From irving at naml.us Thu Sep 30 16:30:48 2010 From: irving at naml.us (Geoffrey Irving) Date: Fri, 1 Oct 2010 09:30:48 +1300 Subject: [Numpy-discussion] constant shaded triangle mesh in mayavi2 Message-ID: Hello, I'm not sure where the correct place to ask questions about Mayavi, so feel free to redirect me elsewhere. I have a triangle mesh with a bunch of data on each face. The only color-relevant argument to triangular_mesh I know about is scalars, which is one value per vertex. Is there a way to set color per triangle, ideally with control over rgb separately so I can visualize three different fields at once? Thanks, Geoffrey From robert.kern at gmail.com Thu Sep 30 16:38:21 2010 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 30 Sep 2010 15:38:21 -0500 Subject: [Numpy-discussion] constant shaded triangle mesh in mayavi2 In-Reply-To: References: Message-ID: On Thu, Sep 30, 2010 at 15:30, Geoffrey Irving wrote: > Hello, > > I'm not sure where the correct place to ask questions about Mayavi, so > feel free to redirect me elsewhere. https://mail.enthought.com/mailman/listinfo/enthought-dev > I have a triangle mesh with a bunch of data on each face. ?The only > color-relevant argument to triangular_mesh I know about is scalars, > which is one value per vertex. ?Is there a way to set color per > triangle, ideally with control over rgb separately so I can visualize > three different fields at once? Pretty difficult, unfortunately. The only way I have found is to assign UV texture coordinates to the vertices and slap on a texture. Assigning the UV coordinates is usually not easy. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From irving at naml.us Thu Sep 30 17:00:32 2010 From: irving at naml.us (Geoffrey Irving) Date: Fri, 1 Oct 2010 10:00:32 +1300 Subject: [Numpy-discussion] constant shaded triangle mesh in mayavi2 In-Reply-To: References: Message-ID: On Fri, Oct 1, 2010 at 9:38 AM, Robert Kern wrote: > On Thu, Sep 30, 2010 at 15:30, Geoffrey Irving wrote: >> Hello, >> >> I'm not sure where the correct place to ask questions about Mayavi, so >> feel free to redirect me elsewhere. > > https://mail.enthought.com/mailman/listinfo/enthought-dev > >> I have a triangle mesh with a bunch of data on each face. ?The only >> color-relevant argument to triangular_mesh I know about is scalars, >> which is one value per vertex. ?Is there a way to set color per >> triangle, ideally with control over rgb separately so I can visualize >> three different fields at once? > > Pretty difficult, unfortunately. The only way I have found is to > assign UV texture coordinates to the vertices and slap on a texture. > Assigning the UV coordinates is usually not easy. Is this a limitation in VTK, TVTK, or the Mayavi python code? The VTK documentation seems to imply it supports cell data on polygon meshes (though I don't know how one would set them): http://www.vtk.org/doc/release/5.6/html/a01445.html "Point and cell attribute values (e.g., scalars, vectors, etc.) also are represented" I'm happy to hack TVTK or Mayavi (in order of happiness) to support cell data if VTK supports it. Geoffrey From robert.kern at gmail.com Thu Sep 30 17:18:34 2010 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 30 Sep 2010 16:18:34 -0500 Subject: [Numpy-discussion] constant shaded triangle mesh in mayavi2 In-Reply-To: References: Message-ID: On Thu, Sep 30, 2010 at 16:00, Geoffrey Irving wrote: > On Fri, Oct 1, 2010 at 9:38 AM, Robert Kern wrote: >> On Thu, Sep 30, 2010 at 15:30, Geoffrey Irving wrote: >>> Hello, >>> >>> I'm not sure where the correct place to ask questions about Mayavi, so >>> feel free to redirect me elsewhere. >> >> https://mail.enthought.com/mailman/listinfo/enthought-dev >> >>> I have a triangle mesh with a bunch of data on each face. ?The only >>> color-relevant argument to triangular_mesh I know about is scalars, >>> which is one value per vertex. ?Is there a way to set color per >>> triangle, ideally with control over rgb separately so I can visualize >>> three different fields at once? >> >> Pretty difficult, unfortunately. The only way I have found is to >> assign UV texture coordinates to the vertices and slap on a texture. >> Assigning the UV coordinates is usually not easy. > > Is this a limitation in VTK, TVTK, or the Mayavi python code? ?The VTK > documentation seems to imply it supports cell data on polygon meshes > (though I don't know how one would set them): > > ? ?http://www.vtk.org/doc/release/5.6/html/a01445.html > ? ?"Point and cell attribute values (e.g., scalars, vectors, etc.) > also are represented" > > I'm happy to hack TVTK or Mayavi (in order of happiness) to support > cell data if VTK supports it. It's a VTK restriction. You can store whatever data you like on the on the vertices or cells. There is no mapper (that I am aware of) that takes that data and interprets them as RGB values on a per-vertex or per-cell basis. There are mappers that turn scalar vertex and cell attribute values into colors through a colormap table, but that's not what you are asking for. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From irving at naml.us Thu Sep 30 17:26:05 2010 From: irving at naml.us (Geoffrey Irving) Date: Fri, 1 Oct 2010 10:26:05 +1300 Subject: [Numpy-discussion] constant shaded triangle mesh in mayavi2 In-Reply-To: References: Message-ID: On Fri, Oct 1, 2010 at 10:18 AM, Robert Kern wrote: > On Thu, Sep 30, 2010 at 16:00, Geoffrey Irving wrote: >> On Fri, Oct 1, 2010 at 9:38 AM, Robert Kern wrote: >>> On Thu, Sep 30, 2010 at 15:30, Geoffrey Irving wrote: >>>> Hello, >>>> >>>> I'm not sure where the correct place to ask questions about Mayavi, so >>>> feel free to redirect me elsewhere. >>> >>> https://mail.enthought.com/mailman/listinfo/enthought-dev >>> >>>> I have a triangle mesh with a bunch of data on each face. ?The only >>>> color-relevant argument to triangular_mesh I know about is scalars, >>>> which is one value per vertex. ?Is there a way to set color per >>>> triangle, ideally with control over rgb separately so I can visualize >>>> three different fields at once? >>> >>> Pretty difficult, unfortunately. The only way I have found is to >>> assign UV texture coordinates to the vertices and slap on a texture. >>> Assigning the UV coordinates is usually not easy. >> >> Is this a limitation in VTK, TVTK, or the Mayavi python code? ?The VTK >> documentation seems to imply it supports cell data on polygon meshes >> (though I don't know how one would set them): >> >> ? ?http://www.vtk.org/doc/release/5.6/html/a01445.html >> ? ?"Point and cell attribute values (e.g., scalars, vectors, etc.) >> also are represented" >> >> I'm happy to hack TVTK or Mayavi (in order of happiness) to support >> cell data if VTK supports it. > > It's a VTK restriction. You can store whatever data you like on the on > the vertices or cells. There is no mapper (that I am aware of) that > takes that data and interprets them as RGB values on a per-vertex or > per-cell basis. There are mappers that turn scalar vertex and cell > attribute values into colors through a colormap table, but that's not > what you are asking for. Visualizing a signle cell scalar field is still quite useful (the data doesn't correspond to color). In other words, the per-cell part of the question is more important than than the RGB part. Is there a current way to set a cell attribute value on a Mayavi triangular_mesh? Thanks, Geoffrey From robert.kern at gmail.com Thu Sep 30 18:39:49 2010 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 30 Sep 2010 17:39:49 -0500 Subject: [Numpy-discussion] constant shaded triangle mesh in mayavi2 In-Reply-To: References: Message-ID: On Thu, Sep 30, 2010 at 16:26, Geoffrey Irving wrote: > On Fri, Oct 1, 2010 at 10:18 AM, Robert Kern wrote: >> On Thu, Sep 30, 2010 at 16:00, Geoffrey Irving wrote: >>> On Fri, Oct 1, 2010 at 9:38 AM, Robert Kern wrote: >>>> On Thu, Sep 30, 2010 at 15:30, Geoffrey Irving wrote: >>>>> Hello, >>>>> >>>>> I'm not sure where the correct place to ask questions about Mayavi, so >>>>> feel free to redirect me elsewhere. >>>> >>>> https://mail.enthought.com/mailman/listinfo/enthought-dev >>>> >>>>> I have a triangle mesh with a bunch of data on each face. ?The only >>>>> color-relevant argument to triangular_mesh I know about is scalars, >>>>> which is one value per vertex. ?Is there a way to set color per >>>>> triangle, ideally with control over rgb separately so I can visualize >>>>> three different fields at once? >>>> >>>> Pretty difficult, unfortunately. The only way I have found is to >>>> assign UV texture coordinates to the vertices and slap on a texture. >>>> Assigning the UV coordinates is usually not easy. >>> >>> Is this a limitation in VTK, TVTK, or the Mayavi python code? ?The VTK >>> documentation seems to imply it supports cell data on polygon meshes >>> (though I don't know how one would set them): >>> >>> ? ?http://www.vtk.org/doc/release/5.6/html/a01445.html >>> ? ?"Point and cell attribute values (e.g., scalars, vectors, etc.) >>> also are represented" >>> >>> I'm happy to hack TVTK or Mayavi (in order of happiness) to support >>> cell data if VTK supports it. >> >> It's a VTK restriction. You can store whatever data you like on the on >> the vertices or cells. There is no mapper (that I am aware of) that >> takes that data and interprets them as RGB values on a per-vertex or >> per-cell basis. There are mappers that turn scalar vertex and cell >> attribute values into colors through a colormap table, but that's not >> what you are asking for. > > Visualizing a signle cell scalar field is still quite useful (the data > doesn't correspond to color). ?In other words, the per-cell part of > the question is more important than than the RGB part. ?Is there a > current way to set a cell attribute value on a Mayavi triangular_mesh? You're better off not using triangular_mesh(). Construct the mesh separately from the surface. s = mlab.pipeline.triangular_mesh_source(x,y,z,triangles) s.data.cell_data.scalars = .... # Your data here. surf = mlab.pipeline.surface(s) surf.contours.filled_contours = True -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From charlesr.harris at gmail.com Thu Sep 30 23:04:23 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 30 Sep 2010 21:04:23 -0600 Subject: [Numpy-discussion] numpydoc: "Other Parameters" section In-Reply-To: <4CA4E140.5090400@stsci.edu> References: <4CA4E140.5090400@stsci.edu> Message-ID: On Thu, Sep 30, 2010 at 1:13 PM, Michael Droettboom wrote: > Is the solution as simple as the attached diff? It works for me, but I > don't understand all the implications. > > Pauli should probably sign off on it. Why not open a ticket and mark it for review? If no comments are forthcoming, then just commit it. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: