From robert.kern at gmail.com Sat Jul 1 00:53:14 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 30 Jun 2006 23:53:14 -0500 Subject: [Numpy-discussion] Mailing lists for Trac tickets and SVN checkins Message-ID: We now have mailing lists set up to receive notifications of changes to Trac tickets and SVN checkins for both NumPy and SciPy. We do not have Gmane gateways for them, yet. http://www.scipy.org/Mailing_Lists -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From schofield at ftw.at Sat Jul 1 06:52:46 2006 From: schofield at ftw.at (Ed Schofield) Date: Sat, 1 Jul 2006 12:52:46 +0200 Subject: [Numpy-discussion] convertcode.py and float defaults In-Reply-To: <44A58569.9080504@ee.byu.edu> References: <44A47854.1050106@ieee.org> <44A4F004.60809@ieee.org> <44A55986.8040905@esrf.fr> <44A569BF.30501@ee.byu.edu> <463e11f90606301142v5351b76r39b1d730fde7faa8@mail.gmail.com> <44A5739F.7020701@ee.byu.edu> <44A57AA3.1040405@ee.byu.edu> <20060630194438.GA6065@ssh.cv.nrao.edu> <44A58569.9080504@ee.byu.edu> Message-ID: On 30/06/2006, at 10:11 PM, Travis Oliphant wrote: > I should have worded this as: > > "People who want arange to return floats *as a default* should be > directed to linspace" > > So, basically, arange is not going to change. > > Because of this, shifting over was a cinch. I still need to write > the > convert-script code that inserts dtype=int > in routines that use old defaults: *plea* anybody want to write > that?? Hmm ... couldn't we make the transition easier and more robust by writing compatibility interfaces for zeros, ones, empty, called e.g. intzeros, intones, intempty? These functions could of course live in oldnumeric.py. Then we can get convertcode.py to do a simple search and replace -- and, more importantly, it's easy for users to do the same manually should they choose not to use convertcode.py. I could work on this this weekend. I'm pleased we're changing the defaults to float. The combination of the int defaults and silent downcasting with in-place operators trips me up once every few months when I forget to specify dtype=float explicitly. Another wart gone from NumPy! I'm surprised and impressed that the community's willing to make this change after 10+ years with int defaults. I think it's a small but important improvement in usability. -- Ed From aisaac at american.edu Sat Jul 1 12:09:03 2006 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 1 Jul 2006 12:09:03 -0400 Subject: [Numpy-discussion] convertcode.py and float defaults In-Reply-To: References: <44A47854.1050106@ieee.org> <44A4F004.60809@ieee.org><44A55986.8040905@esrf.fr> <44A569BF.30501@ee.byu.edu> <463e11f90606301142v5351b76r39b1d730fde7faa8@mail.gmail.com> <44A5739F.7020701@ee.byu.edu> <44A57AA3.1040405@ee.byu.edu><20060630194438.GA6065@ssh.cv.nrao.edu><44A58569.9080504@ee.byu.edu> Message-ID: On Sat, 1 Jul 2006, Ed Schofield apparently wrote: > couldn't we make the transition easier and more robust by > writing compatibility interfaces for zeros, ones, empty, > called e.g. intzeros, intones, intempty I think Robert or Tim suggested int.zeros() etc. fwiw, Alan Isaac From charlesr.harris at gmail.com Sat Jul 1 13:48:11 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 1 Jul 2006 11:48:11 -0600 Subject: [Numpy-discussion] Time for beta1 of NumPy 1.0 In-Reply-To: References: <44A47854.1050106@ieee.org> <44A4F004.60809@ieee.org> <44A55986.8040905@esrf.fr> <44A569BF.30501@ee.byu.edu> <463e11f90606301142v5351b76r39b1d730fde7faa8@mail.gmail.com> <44A5739F.7020701@ee.byu.edu> Message-ID: On 6/30/06, Robert Kern wrote: > > Travis Oliphant wrote: > > > Comments? > > Whatever else you do, leave arange() alone. It should never have accepted > floats > in the first place. Hear, hear. Using floats in arange is a lousy temptation that must be avoided. Apart from that I think that making float64 the default for most things is the right way to go. Numpy is primarily for numeric computation, and numeric computation is primarily in float64. Specialist areas like imaging can be dealt with as special cases. BTW, can someone suggest the best way to put new code into Numpy at this point? Is there a test branch of some sort? I have some free time coming up in a few weeks and would like to do the following: 1) add a left/right option to searchsorted, 2) add faster normals to random, 3) add the MWC8222 generator to random, 3) add the kind keyword to the functional forms of sort (sort, argsort) as in numarray. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Jul 1 13:56:34 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 1 Jul 2006 11:56:34 -0600 Subject: [Numpy-discussion] Time for beta1 of NumPy 1.0 In-Reply-To: References: <44A47854.1050106@ieee.org> <44A4F004.60809@ieee.org> <44A55986.8040905@esrf.fr> <44A569BF.30501@ee.byu.edu> <463e11f90606301142v5351b76r39b1d730fde7faa8@mail.gmail.com> <44A5739F.7020701@ee.byu.edu> Message-ID: All, This is bit off topic, but a while ago there were some complaints about the usefulness of distutils. I note that KDE has gone over to using cmake after trying scon. I am not familiar with cmake, but perhaps someone here knows more and can comment on its suitability. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Sat Jul 1 14:30:13 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 01 Jul 2006 12:30:13 -0600 Subject: [Numpy-discussion] Time for beta1 of NumPy 1.0 In-Reply-To: References: <44A47854.1050106@ieee.org> <44A4F004.60809@ieee.org> <44A55986.8040905@esrf.fr> <44A569BF.30501@ee.byu.edu> <463e11f90606301142v5351b76r39b1d730fde7faa8@mail.gmail.com> <44A5739F.7020701@ee.byu.edu> Message-ID: <44A6BF35.5080400@ieee.org> Charles R Harris wrote: > > > On 6/30/06, *Robert Kern* > wrote: > > Travis Oliphant wrote: > > > Comments? > > Whatever else you do, leave arange() alone. It should never have > accepted floats > in the first place. > > > Hear, hear. Using floats in arange is a lousy temptation that must be > avoided. Apart from that I think that making float64 the default for > most things is the right way to go. Numpy is primarily for numeric > computation, and numeric computation is primarily in float64. > Specialist areas like imaging can be dealt with as special cases. > > BTW, can someone suggest the best way to put new code into Numpy at > this point? Is there a test branch of some sort? My favorite is to make changes in piece-meal steps and just commit them to the turnk as they get created. I think your projects 2 and 4 could be done that way. If a change requires a more elaborate re-write, then I usually construct a branch, switch over to the branch and make changes there. When I'm happy with the result, the branch is merged back into the trunk. Be careful with branches though. It is easy to get too far away from main-line trunk development (although at this point the trunk should be stabilizing toward release 1.0). 1) To construct a branch (just a copy of the trunk): (Make note of the revision number when you create the branch-- you can get it later but it's easier to just record it at copy). svn cp http://svn.scipy.org/svn/numpy/trunk http://svn.scipy.org/svn/numpy/branches/ 2) To switch to using the branch: svn switch http://svn.scipy.org/svn/numpy/branches/ You can also just have another local directory where you work on the branch so that you still have a local directory with the main trunk. Just check out the branch: svn co http://svn.scipy.org/svn/numpy/branches/ mybranch 3) To merge back: a) Get back to the trunk repository: svn switch http://svn.scipy.org/svn/numpy/trunk or go to your local copy of the trunk and do an svn update b) Merge the changes from the branch back in to your local copy of the trunk: svn merge -r :HEAD http://svn.scipy.org/svn/numpy/branches/ This assumes that is the revision number when the branch is created c) You have to now commit your local copy of the trunk (after you've dealt with and resolved any potential conflicts). If your branch is continuing a while, you may need to update your branch with changes that have happened in the main-line trunk. This will make it easier to merge back when you are done. To update your branch with changes from the main trunk do: svn merge -r : http://svn.scipy.org/svn/numpy/trunk where is the last revision number you used to update your branch (or the revision number at which you made your branch) and is the ending revision number for changes in the trunk you'd like to merge. Here is a good link explaining the process more. http://svnbook.red-bean.com/en/1.1/ch04s03.html -Travis -Travis From charlesr.harris at gmail.com Sat Jul 1 14:50:09 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 1 Jul 2006 12:50:09 -0600 Subject: [Numpy-discussion] Time for beta1 of NumPy 1.0 In-Reply-To: <44A6BF35.5080400@ieee.org> References: <44A47854.1050106@ieee.org> <44A4F004.60809@ieee.org> <44A55986.8040905@esrf.fr> <44A569BF.30501@ee.byu.edu> <463e11f90606301142v5351b76r39b1d730fde7faa8@mail.gmail.com> <44A5739F.7020701@ee.byu.edu> <44A6BF35.5080400@ieee.org> Message-ID: Thanks Travis, Your directions are very helpful and much appreciated. Chuck On 7/1/06, Travis Oliphant wrote: > > Charles R Harris wrote: > > > > > > On 6/30/06, *Robert Kern* > > wrote: > > > > Travis Oliphant wrote: > > > > > Comments? > > > > Whatever else you do, leave arange() alone. It should never have > > accepted floats > > in the first place. > > > > > > Hear, hear. Using floats in arange is a lousy temptation that must be > > avoided. Apart from that I think that making float64 the default for > > most things is the right way to go. Numpy is primarily for numeric > > computation, and numeric computation is primarily in float64. > > Specialist areas like imaging can be dealt with as special cases. > > > > BTW, can someone suggest the best way to put new code into Numpy at > > this point? Is there a test branch of some sort? > My favorite is to make changes in piece-meal steps and just commit them > to the turnk as they get created. I think your projects 2 and 4 could > be done that way. > > If a change requires a more elaborate re-write, then I usually construct > a branch, switch over to the branch and make changes there. When I'm > happy with the result, the branch is merged back into the trunk. > > Be careful with branches though. It is easy to get too far away from > main-line trunk development (although at this point the trunk should be > stabilizing toward release 1.0). > > 1) To construct a branch (just a copy of the trunk): > > (Make note of the revision number when you create the branch-- you can > get it later but it's easier to just record it at copy). > > svn cp http://svn.scipy.org/svn/numpy/trunk > http://svn.scipy.org/svn/numpy/branches/ > > 2) To switch to using the branch: > > svn switch http://svn.scipy.org/svn/numpy/branches/ > > You can also just have another local directory where you work on the > branch so that you still have a local directory with the main trunk. > Just check out the branch: > > svn co http://svn.scipy.org/svn/numpy/branches/ mybranch > > 3) To merge back: > > a) Get back to the trunk repository: > > svn switch http://svn.scipy.org/svn/numpy/trunk > > or go to your local copy of the trunk and do an svn update > > b) Merge the changes from the branch back in to your local copy of the > trunk: > > svn merge -r :HEAD > http://svn.scipy.org/svn/numpy/branches/ > > This assumes that is the revision number when the branch > is created > > c) You have to now commit your local copy of the trunk (after you've > dealt with and resolved any potential conflicts). > > If your branch is continuing a while, you may need to update your branch > with changes that have happened in the main-line trunk. This will make > it easier to merge back when you are done. > > To update your branch with changes from the main trunk do: > > svn merge -r : http://svn.scipy.org/svn/numpy/trunk > > where is the last revision number you used to update your > branch (or the revision number at which you made your branch) and > is the ending revision number for changes in the trunk you'd like to > merge. > > Here is a good link explaining the process more. > > http://svnbook.red-bean.com/en/1.1/ch04s03.html > > > > -Travis > > > > -Travis > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Sat Jul 1 14:53:43 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 01 Jul 2006 12:53:43 -0600 Subject: [Numpy-discussion] Time for beta1 of NumPy 1.0 In-Reply-To: References: <44A47854.1050106@ieee.org> <44A4F004.60809@ieee.org> <44A55986.8040905@esrf.fr> <44A569BF.30501@ee.byu.edu> <463e11f90606301142v5351b76r39b1d730fde7faa8@mail.gmail.com> <44A5739F.7020701@ee.byu.edu> <44A6BF35.5080400@ieee.org> Message-ID: <44A6C4B7.7080606@ieee.org> Charles R Harris wrote: > Thanks Travis, > > Your directions are very helpful and much appreciated. I placed these instructions at http://projects.scipy.org/scipy/numpy/wiki/MakingBranches Please make any changes needed to that wiki page. -Travis From robert.kern at gmail.com Sat Jul 1 15:18:59 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 01 Jul 2006 14:18:59 -0500 Subject: [Numpy-discussion] Time for beta1 of NumPy 1.0 In-Reply-To: References: <44A47854.1050106@ieee.org> <44A4F004.60809@ieee.org> <44A55986.8040905@esrf.fr> <44A569BF.30501@ee.byu.edu> <463e11f90606301142v5351b76r39b1d730fde7faa8@mail.gmail.com> <44A5739F.7020701@ee.byu.edu> Message-ID: Charles R Harris wrote: > All, > > This is bit off topic, but a while ago there were some complaints about > the usefulness of distutils. I note that KDE has gone over to using > cmake after trying scon. I am not familiar with cmake, but perhaps > someone here knows more and can comment on its suitability. None at all. I have nightmares about it every time I need to rebuild VTK. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Sat Jul 1 15:36:20 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 01 Jul 2006 14:36:20 -0500 Subject: [Numpy-discussion] Time for beta1 of NumPy 1.0 In-Reply-To: <44A6C4B7.7080606@ieee.org> References: <44A47854.1050106@ieee.org> <44A4F004.60809@ieee.org> <44A55986.8040905@esrf.fr> <44A569BF.30501@ee.byu.edu> <463e11f90606301142v5351b76r39b1d730fde7faa8@mail.gmail.com> <44A5739F.7020701@ee.byu.edu> <44A6BF35.5080400@ieee.org> <44A6C4B7.7080606@ieee.org> Message-ID: Travis Oliphant wrote: > Charles R Harris wrote: >> Thanks Travis, >> >> Your directions are very helpful and much appreciated. > I placed these instructions at > > http://projects.scipy.org/scipy/numpy/wiki/MakingBranches > > Please make any changes needed to that wiki page. I will add (here as well as the wiki) that using the svnmerge tool to be enormously helpful in maintaining branches. http://www.dellroad.org/svnmerge/index Among other things, it makes merge commit messages with the contents of the individual commit messages, so history isn't lost when changes are merged back into the trunk. Here is how I tend to set things up for bidirectional merging: (untested with this specific example, though) $ cd ~/svn/scipy $ svn cp http://svn.scipy.org/svn/scipy/trunk http://svn.scipy.org/svn/scipy/branches/mine $ svnmerge init http://svn.scipy.org/svn/scipy/branches/mine $ svn commit -F svnmerge-commit-message.txt $ svn switch http://svn.scipy.org/svn/scipy/branches/mine $ svnmerge init http://svn.scipy.org/svn/scipy/trunk $ svn commit -F svnmerge-commit-message.txt Then, when you need to pull in changes from the trunk, view them with $ svnmerge avail and pull them in with $ svnmerge merge $ svn ci -F svnmerge-commit-message.txt When you're finally done with the branch, the same procedure on the trunk pulls in all of the (real, not merged in from the trunk) changes you've made to the branch. Also, if you're only going to be making changes in one directory, I've found that it's much easier to simply branch that directory and svn switch just that directory over. That way, you don't have to worry about pulling in everyone else's changes to the rest of the package into the branch. You can just svn up. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From fullung at gmail.com Sat Jul 1 15:37:34 2006 From: fullung at gmail.com (Albert Strasheim) Date: Sat, 1 Jul 2006 21:37:34 +0200 Subject: [Numpy-discussion] Time for beta1 of NumPy 1.0 In-Reply-To: Message-ID: <017b01c69d45$cce1c450$0a84a8c0@dsp.sun.ac.za> Hey Chuck > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of Charles R Harris > Sent: 01 July 2006 19:57 > To: Robert Kern > Cc: numpy-discussion at lists.sourceforge.net > Subject: Re: [Numpy-discussion] Time for beta1 of NumPy 1.0 > > All, > > This is bit off topic, but a while ago there were some complaints about > the usefulness of distutils. I note that KDE has gone over to using cmake > after trying scon. I am not familiar with cmake, but perhaps someone here > knows more and can comment on its suitability. CMake definately warrants investigation, but I think SCons might be a better way to go. I think it would make it easier to reuse large parts of the existing build code (e.g. conv_template.py could be converted into a SCons builder without too much effort). Reusing parts of distutils and setuptools would also be easier if the new tool is somehow Python-aware. I think the main problem with distutils in the NumPy context is that it was never designed to build C/Fortran code over so many platforms with to many possible build configurations. python setup.py install works pretty well, but any kind of non-default configuration can become a bit hairy, despite the excellent work on NumPy extensions to distutils. I'd like to take a stab at doing something with SCons in a few weeks' time. Does anybody want to brainstorm on some ideas for what is needed from a better build system for NumPy? Maybe a wiki page? Regards, Albert From robert.kern at gmail.com Sat Jul 1 15:44:57 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 01 Jul 2006 14:44:57 -0500 Subject: [Numpy-discussion] Time for beta1 of NumPy 1.0 In-Reply-To: <017b01c69d45$cce1c450$0a84a8c0@dsp.sun.ac.za> References: <017b01c69d45$cce1c450$0a84a8c0@dsp.sun.ac.za> Message-ID: Albert Strasheim wrote: > I'd like to take a stab at doing something with SCons in a few weeks' time. > Does anybody want to brainstorm on some ideas for what is needed from a > better build system for NumPy? Maybe a wiki page? I strongly believe that we need to be using whatever build system is the standard for Python packages. I'm happy to see distutils go away in favor of something better, but that "something better" needs to be actively promoted as *the* replacement for distutils for *all* Python packages, not just numpy. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ndarray at mac.com Sat Jul 1 16:07:36 2006 From: ndarray at mac.com (Sasha) Date: Sat, 1 Jul 2006 16:07:36 -0400 Subject: [Numpy-discussion] Time for beta1 of NumPy 1.0 In-Reply-To: References: <017b01c69d45$cce1c450$0a84a8c0@dsp.sun.ac.za> Message-ID: On 7/1/06, Robert Kern wrote: > ... > I strongly believe that we need to be using whatever build system is the > standard for Python packages. I'm happy to see distutils go away in favor of > something better, but that "something better" needs to be actively promoted as > *the* replacement for distutils for *all* Python packages, not just numpy. > I strongly agree. Distutils is in such a bad shape partially because extension packages either not use distutils or extend distutils in non-standard ways. Python-dev is not an easy crowd to deal with, but in the long run investing effort in improving core distutils will pay off. From ndarray at mac.com Sat Jul 1 16:28:11 2006 From: ndarray at mac.com (Sasha) Date: Sat, 1 Jul 2006 16:28:11 -0400 Subject: [Numpy-discussion] convertcode.py and float defaults In-Reply-To: References: <44A47854.1050106@ieee.org> <44A569BF.30501@ee.byu.edu> <463e11f90606301142v5351b76r39b1d730fde7faa8@mail.gmail.com> <44A5739F.7020701@ee.byu.edu> <44A57AA3.1040405@ee.byu.edu> <20060630194438.GA6065@ssh.cv.nrao.edu> <44A58569.9080504@ee.byu.edu> Message-ID: I don't see how that will simplify the transition. Convertcode will still need to detect use of the dtype argument (keyword or positional). Simple s/zeros/int.zeros/ will not work. I read Ed's suggestion as retaining current default in intzeros so that intzeros(n, float) is valid. On the other hand Tim's int.zeros would not take dtype argument because dtype is already bound as self. The bottom line: int.zeros will not work and intzeros(n, float) is ugly. I would not mind oldnumeric.zeros, but context aware convertcode is still worth the effort. Let's see how far I will get with that ... On 7/1/06, Alan G Isaac wrote: > On Sat, 1 Jul 2006, Ed Schofield apparently wrote: > > couldn't we make the transition easier and more robust by > > writing compatibility interfaces for zeros, ones, empty, > > called e.g. intzeros, intones, intempty > > > I think Robert or Tim suggested int.zeros() etc. > > fwiw, > Alan Isaac > > > > > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From tim.hochberg at cox.net Sat Jul 1 19:34:18 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Sat, 01 Jul 2006 16:34:18 -0700 Subject: [Numpy-discussion] convertcode.py and float defaults In-Reply-To: References: <44A47854.1050106@ieee.org> <44A569BF.30501@ee.byu.edu> <463e11f90606301142v5351b76r39b1d730fde7faa8@mail.gmail.com> <44A5739F.7020701@ee.byu.edu> <44A57AA3.1040405@ee.byu.edu> <20060630194438.GA6065@ssh.cv.nrao.edu> <44A58569.9080504@ee.byu.edu> Message-ID: <44A7067A.4010004@cox.net> Sasha wrote: > I don't see how that will simplify the transition. Convertcode will > still need to detect use of the dtype argument (keyword or > positional). Simple s/zeros/int.zeros/ will not work. I read Ed's > suggestion as retaining current default in intzeros so that > intzeros(n, float) is valid. On the other hand Tim's int.zeros would > not take dtype argument because dtype is already bound as self. > It's just like a game of telephone! That was Robert's suggestion not mine. What I said was: Personally, given no other constraints, I would probably just get rid of the defaults all together and make the user choose. Since I've been dragged back into this again let me make a quick comment. If we are choosing a floating point default, there are at least two other choices that make as much sense as using float64. The first possibility is to use the same thing that python uses, that is 'float'. On my box and probably most current boxes that turns out to be float64 anyway, but choosing 'float' as the default rather than 'float64' will change the way numpy is expected to behave as hardware and / or Python evolves. The second choice is to use the longest floating point type available on a given platform, that is, 'longfloat'. Again, on my box that is the same as using float64, but on other boxes I suspect it gives somewhat different results. The advantage of using 'float64' as the default is that we can expect programs to run consistently across platforms. The advantage of choosing 'float' is that interactions with Python proper may be less suprising when python's float is not 'float64. The advantage of using 'longfloat' is that it is the safest type to use when interacting with other unknown types. I don't care much which gets chosen, but I think we should know which of these we intend and why. Since there often the same thing at present I have a suspicion that these three cases may be conflated in some people heads. -tim > The bottom line: int.zeros will not work and intzeros(n, float) is > ugly. I would not mind oldnumeric.zeros, but context aware convertcode > is still worth the effort. Let's see how far I will get with that ... > > > > On 7/1/06, Alan G Isaac wrote: > >> On Sat, 1 Jul 2006, Ed Schofield apparently wrote: >> >>> couldn't we make the transition easier and more robust by >>> writing compatibility interfaces for zeros, ones, empty, >>> called e.g. intzeros, intones, intempty >>> >> I think Robert or Tim suggested int.zeros() etc. >> >> fwiw, >> Alan Isaac >> >> >> >> >> >> Using Tomcat but need to do more? Need to support web services, security? >> Get stuff done quickly with pre-integrated technology to make your job easier >> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> >> > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From oliphant.travis at ieee.org Sat Jul 1 23:14:26 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 01 Jul 2006 21:14:26 -0600 Subject: [Numpy-discussion] Scipy SVN server slow or down? Message-ID: <44A73A12.1090103@ieee.org> Is anybody else having trouble connecting to the SciPy SVN server? It just started failing with could not connect to server errors in the last hour. -Travis From oliphant.travis at ieee.org Sat Jul 1 23:19:57 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 01 Jul 2006 21:19:57 -0600 Subject: [Numpy-discussion] Ctypes support in NumPy Message-ID: <44A73B5D.60002@ieee.org> I've been playing a bit with ctypes and realized that with a little help, it could be made much easier to interface with NumPy arrays. Thus, I added a ctypes attribute to the NumPy array. If ctypes is installed, this attribute returns a "conversion" object otherwise an AttributeError is raised. The ctypes-conversion object has attributes which return c_types aware objects so that the information can be passed directly to c-code (as an integer, the number of dimensions can already be passed using c-types). The information available and it's corresponding c_type is data - c_void_p shape, strides - c_int * nd or c_long * nd or c_longlong * nd depending on platform -Travis From ter18 at 163.com Wed Jul 5 23:44:43 2006 From: ter18 at 163.com (=?GB2312?B?N9TCMTUtMTbI1S/Jz7qj?=) Date: Thu, 6 Jul 2006 11:44:43 +0800 Subject: [Numpy-discussion] =?GB2312?B?1MvTw0VYQ0VMuMS9+LLGzvG53MDtus3K0LOh06o8YWQ+?= Message-ID: An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sun Jul 2 00:42:50 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 01 Jul 2006 23:42:50 -0500 Subject: [Numpy-discussion] Scipy SVN server slow or down? In-Reply-To: <44A73A12.1090103@ieee.org> References: <44A73A12.1090103@ieee.org> Message-ID: Travis Oliphant wrote: > Is anybody else having trouble connecting to the SciPy SVN server? It > just started failing with could not connect to server errors in the last > hour. It works intermittently for me. The scipy.org server has apparently been using up a lot of bandwidth over the past few days (I have this information second-hand so I don't know why exactly). It was affecting our network connectivity at Enthought. Its upload bandwidth was capped at 1 Mbps yesterday. I'll try to escalate this issue, but both Monday and Tuesday are holidays for us. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Norbert.Nemec.list at gmx.de Sun Jul 2 05:47:11 2006 From: Norbert.Nemec.list at gmx.de (Norbert Nemec) Date: Sun, 02 Jul 2006 11:47:11 +0200 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> Message-ID: <44A7961F.9030907@gmx.de> I agree. Currently the order of the output of unique is undefined. Defining it in such a way that it produces a sorted array will not break any compatibility. My idea would be something like def unique(arr,sort=True): if hasattr(arr,'flatten'): tmp = arr.flatten() tmp.sort() idx = concatenate([True],tmp[1:]!=tmp[:-1]) return tmp[idx] else: # for compatibility: set = {} for item in inseq: set[item] = None if sort: return asarray(sorted(set.keys())) else: return asarray(set.keys()) Does anybody know about the internals of the python "set"? How is .keys() implemented? I somehow have really doubts about the efficiency of this method. David Huard wrote: > Hi, > > Numpy's unique(x) returns an array x with repetitions removed. > However, since it returns asarray(dict.keys()), the resulting array is > not sorted, worse, the original order may not be conserved. I think > that unique() should return a sorted array, like its matlab homonym. > > Regards, > > David Huard > ------------------------------------------------------------------------ > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From robert.kern at gmail.com Sun Jul 2 06:37:29 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 02 Jul 2006 05:37:29 -0500 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <44A7961F.9030907@gmx.de> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> Message-ID: Norbert Nemec wrote: > I agree. > > Currently the order of the output of unique is undefined. Defining it in > such a way that it produces a sorted array will not break any compatibility. > > My idea would be something like > > def unique(arr,sort=True): > if hasattr(arr,'flatten'): > tmp = arr.flatten() > tmp.sort() > idx = concatenate([True],tmp[1:]!=tmp[:-1]) > return tmp[idx] > else: # for compatibility: > set = {} > for item in inseq: > set[item] = None > if sort: > return asarray(sorted(set.keys())) > else: > return asarray(set.keys()) > > Does anybody know about the internals of the python "set"? How is > .keys() implemented? I somehow have really doubts about the efficiency > of this method. Well, that's a dictionary, not a set, but they both use the same algorithm. They are both hash tables. If you need more specific details about how the hash tables are implemented, the source (Object/dictobject.c)is the best place for them. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From tim.leslie at gmail.com Sun Jul 2 06:53:58 2006 From: tim.leslie at gmail.com (Tim Leslie) Date: Sun, 2 Jul 2006 20:53:58 +1000 Subject: [Numpy-discussion] nan_to_num() SystemException Message-ID: Hi All, The following script: import numpy as N print N.__version__ a = N.array([1,2,3,4], dtype=N.float64) a.dtype = a.dtype.newbyteorder('big') print N.nan_to_num(a) gives the following exception: 0.9.9.2707 Traceback (most recent call last): File "memmap_nan.py", line 5, in ? print N.nan_to_num(a) File "/usr/lib/python2.4/site-packages/numpy/lib/type_check.py", line 127, in nan_to_num y[are_inf] = maxf SystemError: error return without exception set Could someone shed some light on this? I'm at a bit of a loss as where to go. I had a poke around inside type_check.py but didn't get very far. I confirmed that are_inf in type_check.py is [False, False, False, False], as expected. I changed the value of maxf to different values (0, maxf+1, maxf-1, maxf+0.00000001, minf, 37), and with any of these different values the SystemError goes away. Beyond that, I'm not sure how to chase this one down. Cheers, Tim Leslie From mclella at brhlaw.com Sun Jul 2 09:33:44 2006 From: mclella at brhlaw.com (Jahan Mclelland) Date: Sun, 2 Jul 2006 06:33:44 -0700 Subject: [Numpy-discussion] with qexau Message-ID: <000001c69ddc$239cea90$6338a8c0@ucq22> Hi, V / / A G R A from 3 , 33 $ and man y other at http://uadesaxecoin.com - - - - - - And he told her what had happened in Madam Puddifoots teashop. : so then, he finished several minutes later, as the final bit of crumble disappeared, she jumps up, right, and says, Ill see you around, Harry, and runs out of the place! He put down his spoon and looked at Hermione. I mean, what was all that about? What was going on? Hermione glanced over at the back of Chos head and sighed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fullung at gmail.com Sun Jul 2 10:24:36 2006 From: fullung at gmail.com (Albert Strasheim) Date: Sun, 2 Jul 2006 16:24:36 +0200 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: <44A73B5D.60002@ieee.org> Message-ID: <001301c69de3$3f0d4890$0a84a8c0@dsp.sun.ac.za> Hello all Travis Oliphant wrote: > I've been playing a bit with ctypes and realized that with a little > help, it could be made much easier to interface with NumPy arrays. > Thus, I added a ctypes attribute to the NumPy array. If ctypes is > installed, this attribute returns a "conversion" object otherwise an > AttributeError is raised. > > The ctypes-conversion object has attributes which return c_types aware > objects so that the information can be passed directly to c-code (as an > integer, the number of dimensions can already be passed using c-types). > > The information available and it's corresponding c_type is > > data - c_void_p > shape, strides - c_int * nd or c_long * nd or c_longlong * nd > depending on platform I did a few tests and this seems to work nicely: In [133]: printf = ctypes.cdll.msvcrt.printf In [134]: printf.argtypes = [ctypes.c_char_p, ctypes.c_void_p] In [135]: x = N.array([1,2,3]) In [136]: printf('%p\n', x.ctypes.data) 01CC8AC0 Out[136]: 9 In [137]: hex(x.__array_interface__['data'][0]) Out[137]: '0x1cc8ac0' It would be nice if we could the _as_parameter_ magic to work as well. See this thread: http://aspn.activestate.com/ASPN/Mail/Message/ctypes-users/3122558 If I understood Thomas correctly, in the presence of argtypes an an instance, say x, with _as_parameter_, the following is done to convert the instance to something that the function accepts as its nth argument: func.argtypes[n].from_param(x._as_parameter_) However, if I try passing x directly to printf, I get this: In [147]: printf('%p\n', x) ... ArgumentError: argument 2: exceptions.TypeError: wrong type However, this much works: In [148]: ctypes.c_void_p.from_param(x._as_parameter_) Out[148]: So I don't understand why the conversion isn't happening automatically. Another quirk I noticed is that non-void pointers' from_param can't seem to be used with ints. For example: In [167]: ctypes.POINTER(ctypes.c_double).from_param(x._as_parameter_) ... TypeError: expected LP_c_double instance instead of int But this works: In [168]: ctypes.POINTER(ctypes.c_double).from_address(x._as_parameter_) Out[168]: I don't think this is too much of an issue though -- you could wrap all your functions to take c_void_ps. If you happen to pass an int32 NumPy array to a function expecting a double*, you might run into problems though. Maybe there should be a way to get a pointer to the NumPy array data as a POINTER(c_double) if it is known that the array's dtype is float64. Ditto for c_int/int32 and the others. Regards, Albert From ndarray at mac.com Sun Jul 2 12:17:58 2006 From: ndarray at mac.com (Sasha) Date: Sun, 2 Jul 2006 12:17:58 -0400 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <44A7961F.9030907@gmx.de> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> Message-ID: On 7/2/06, Norbert Nemec wrote: > ... > Does anybody know about the internals of the python "set"? How is > .keys() implemented? I somehow have really doubts about the efficiency > of this method. > Set implementation (Objects/setobject.c) is a copy and paste job from dictobject with values removed. As a result it is heavily optimized for the case of string valued keys - a case that is almost irrelevant for numpy. I think something like the following (untested, 1d only) will probably be much faster and sorted: def unique(x): s = sort(x) r = empty_like(s) r[:-1] = s[1:] r[-1] = s[0] return s[r != s] From haase at msg.ucsf.edu Sun Jul 2 13:20:51 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sun, 02 Jul 2006 10:20:51 -0700 Subject: [Numpy-discussion] problem building Numeric on python 2.5 In-Reply-To: <449B1F15.1020501@cox.net> References: <449B1F15.1020501@cox.net> Message-ID: <44A80073.2070805@msg.ucsf.edu> Hi, It seems that there is a current discussion on python-dev http://www.gossamer-threads.com/lists/engine?do=post_view_flat;post=497487;page=1;mh=-1;list=python;sb=post_latest_reply;so=ASC and the warning should go away by 2.5b2 So: don't change any code quite yet .... Cheers, -Sebastian Haase Tim Hochberg wrote: > Russell E. Owen wrote: >> I just installed python 2.5b1 on my Mac (10.4 ppc) and can't seem to get >> Numeric 24.2 installed. It seems to build fine (no obvious error >> messages), but when I try to import it I get: >> Python 2.5b1 (r25b1:47038M, Jun 20 2006, 16:17:55) >> [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin >> Type "help", "copyright", "credits" or "license" for more information. >> >>>>> import Numeric >>>>> >> __main__:1: ImportWarning: Not importing directory >> '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-pac >> kages/Numeric': missing __init__.py >> >> >> Any ideas? Is it somehow incompatible with python 2.5b1? >> > Import warning is a new 'feature' of 2.5. It warns if there are > directories on sys.path that are *not* packages. I'll refer you to the > py-dev archives if you want figure out the motivation for that. So, if > everything seems to work, there's a good chance that nothing's wrong, > but that your just seeing a complaint due to this new behaviour. If you > check recent messages on Python-dev someone just posted a recipe for > suppressing this warning. > > -tim > > >> For what it's worth, numarray builds and installs fine. I've not tried >> numpy or any other packages yet. >> >> -- Russell From bsouthey at gmail.com Sun Jul 2 16:23:47 2006 From: bsouthey at gmail.com (Bruce Southey) Date: Sun, 2 Jul 2006 15:23:47 -0500 Subject: [Numpy-discussion] Time for beta1 of NumPy 1.0 In-Reply-To: <017b01c69d45$cce1c450$0a84a8c0@dsp.sun.ac.za> References: <017b01c69d45$cce1c450$0a84a8c0@dsp.sun.ac.za> Message-ID: Hi, Linux Weekly News (http://lwn.net) had an very interesting article on KDE's switch on June 19, 2006 by Alexander Neundorf: http://lwn.net/Articles/187923/ The full article is at: http://lwn.net/Articles/188693/ This should be freely available to all. Also, the current US Linux Magazine (June or July 2006 ) has a small feature on cmake as well. Regards Bruce On 7/1/06, Albert Strasheim wrote: > Hey Chuck > > > -----Original Message----- > > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > > discussion-bounces at lists.sourceforge.net] On Behalf Of Charles R Harris > > Sent: 01 July 2006 19:57 > > To: Robert Kern > > Cc: numpy-discussion at lists.sourceforge.net > > Subject: Re: [Numpy-discussion] Time for beta1 of NumPy 1.0 > > > > All, > > > > This is bit off topic, but a while ago there were some complaints about > > the usefulness of distutils. I note that KDE has gone over to using cmake > > after trying scon. I am not familiar with cmake, but perhaps someone here > > knows more and can comment on its suitability. > > CMake definately warrants investigation, but I think SCons might be a better > way to go. I think it would make it easier to reuse large parts of the > existing build code (e.g. conv_template.py could be converted into a SCons > builder without too much effort). Reusing parts of distutils and setuptools > would also be easier if the new tool is somehow Python-aware. > > I think the main problem with distutils in the NumPy context is that it was > never designed to build C/Fortran code over so many platforms with to many > possible build configurations. python setup.py install works pretty well, > but any kind of non-default configuration can become a bit hairy, despite > the excellent work on NumPy extensions to distutils. > > I'd like to take a stab at doing something with SCons in a few weeks' time. > Does anybody want to brainstorm on some ideas for what is needed from a > better build system for NumPy? Maybe a wiki page? > > Regards, > > Albert > > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From jonathan.taylor at stanford.edu Sun Jul 2 16:38:28 2006 From: jonathan.taylor at stanford.edu (Jonathan Taylor) Date: Sun, 02 Jul 2006 13:38:28 -0700 Subject: [Numpy-discussion] byteorder question In-Reply-To: <44A5AD36.7070906@ee.byu.edu> References: <44A5A9AC.5070707@stanford.edu> <44A5AD36.7070906@ee.byu.edu> Message-ID: <44A82EC4.4040804@stanford.edu> Thanks. The byteswap works for me now, but I have another problem... -------------------------------------------------------------------- >>> import numpy as N >>> d = N.dtype(N.float64) >>> d.newbyteorder('big') dtype('>f8') >>> D = d.newbyteorder('big') >>> >>> x = N.zeros((10,10), D) >>> N.nan_to_num(x) Traceback (most recent call last): File "", line 1, in ? File "/home/jtaylo/python/lib/python2.4/site-packages/numpy/lib/type_check.py", line 127, in nan_to_num y[are_inf] = maxf SystemError: error return without exception set >>> >>> ------------------------------------------------------------ Here is what maxf, minf are: ---------------------------------------------------------------- SystemError: error return without exception set /home/jtaylo/python/lib/python2.4/site-packages/numpy/lib/type_check.py in nan_to_num() 126 y[are_nan] = 0 --> 127 y[are_inf] = maxf 128 y[are_neg_inf] = minf ipdb> maxf array(1.7976931348623157e+308) ipdb> minf -1.7976931348623157e+308 ----------------------------------------------------------------------- ----------------------------------------------------------- -- Jonathan -- ------------------------------------------------------------------------ Jonathan Taylor Tel: 650.723.9230 Dept. of Statistics Fax: 650.725.8977 Sequoia Hall, 137 www-stat.stanford.edu/~jtaylo 390 Serra Mall Stanford, CA 94305 From jonathan.taylor at stanford.edu Sun Jul 2 16:44:42 2006 From: jonathan.taylor at stanford.edu (Jonathan Taylor) Date: Sun, 02 Jul 2006 13:44:42 -0700 Subject: [Numpy-discussion] nan_to_num() SystemException In-Reply-To: References: Message-ID: <44A8303A.5010005@stanford.edu> Ooops -- I posted this same question. Sorry. One more bit of info: ipdb> maxf array(1.7976931348623157e+308) ipdb> minf -1.7976931348623157e+308 ipdb> type(maxf) ipdb> type(minf) ipdb> y[are_inf] = float(maxf) ipdb> ipdb> y[are_inf] array([], dtype=float64) --Jonathan Tim Leslie wrote: > Hi All, > > The following script: > > import numpy as N > print N.__version__ > a = N.array([1,2,3,4], dtype=N.float64) > a.dtype = a.dtype.newbyteorder('big') > print N.nan_to_num(a) > > gives the following exception: > > 0.9.9.2707 > Traceback (most recent call last): > File "memmap_nan.py", line 5, in ? > print N.nan_to_num(a) > File "/usr/lib/python2.4/site-packages/numpy/lib/type_check.py", > line 127, in nan_to_num > y[are_inf] = maxf > SystemError: error return without exception set > > Could someone shed some light on this? I'm at a bit of a loss as where > to go. I had a poke around inside type_check.py but didn't get very > far. I confirmed that are_inf in type_check.py is [False, False, > False, False], as expected. I changed the value of maxf to different > values (0, maxf+1, maxf-1, maxf+0.00000001, minf, 37), and with any of > these different values the SystemError goes away. Beyond that, I'm not > sure how to chase this one down. > > Cheers, > > Tim Leslie > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- ------------------------------------------------------------------------ Jonathan Taylor Tel: 650.723.9230 Dept. of Statistics Fax: 650.725.8977 Sequoia Hall, 137 www-stat.stanford.edu/~jtaylo 390 Serra Mall Stanford, CA 94305 From elventear at gmail.com Sun Jul 2 17:32:26 2006 From: elventear at gmail.com (Pepe Barbe) Date: Sun, 2 Jul 2006 16:32:26 -0500 Subject: [Numpy-discussion] Problems with list 2 array Message-ID: <3e73596b0607021432r79589c05w1e90fb48fa6e949b@mail.gmail.com> Hi, I have some data I've generated and stored in a PyTables database. Then I retrieve the data in a script and do some processing. I have two different datasets that have been generated in the same way, and one works perfectly as intended while the other gives me an error. I'll explain in the following lines what happens. The information in question is stored in a list that contains 10 one dimensional arrays. The data is stored in a variable called prob. If I execute array(prob) on the data coming of one of the sources it is converted in a 10xn numpy array. If I do the same thing on the other I get the following: In [56]: array(prob) --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /Users/elventear/Projects/workspace/libSVM Scripts/src/ TypeError: a float is required I have no idea why it is complaining about this. I've compared the data coming from the two sources and they seem the same. In both cases, they are like this: In [57]: type(prob) Out[57]: In [58]: type(prob[0]) Out[58]: In [59]: type(prob[0][0]) Out[59]: I have no idea where else to look to solve this problem. As I said the data is generated identically starting from two different data sources and then processed identically with the same script. From one data source it just works while the other complains. Thanks, Pepe From pgmdevlist at mailcan.com Sun Jul 2 18:48:37 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Sun, 2 Jul 2006 18:48:37 -0400 Subject: [Numpy-discussion] Problems with list 2 array In-Reply-To: <3e73596b0607021432r79589c05w1e90fb48fa6e949b@mail.gmail.com> References: <3e73596b0607021432r79589c05w1e90fb48fa6e949b@mail.gmail.com> Message-ID: <200607021848.37228.pgmdevlist@mailcan.com> Pepe, > In [56]: array(prob) > --------------------------------------------------------------------------- > exceptions.TypeError Traceback (most > recent call last) > > /Users/elventear/Projects/workspace/libSVM Scripts/src/ > > TypeError: a float is required Make sure that all the elements of your `prob` list have the same length. If not, then numpy can't create an array from your list and raises the exception you see. From webb.sprague at gmail.com Sun Jul 2 19:36:14 2006 From: webb.sprague at gmail.com (Webb Sprague) Date: Sun, 2 Jul 2006 16:36:14 -0700 Subject: [Numpy-discussion] Does zero based indexing drive anyone else crazy? Message-ID: Hi Numpeans, I have been working on a web-based scientific application for about a year, most of which had been written in either Matlab or SPLUS/R. My task has been to make it "driveable" through an online interface (if anyone cares about mortality forecasting, drop me an email and we can chat about it offline). I chose Python/Numpy for the language because Python and Numpy are both so full featured and easy to work with (except for one little thing...), and neither Matlab nor R could gracefully deal with CGI programming (misguided propaganda notwithstanding). However.... I have spent a huge amount of my time fixing and bending my head around off-by-one errors caused by trying to index matrices using 0 to n-1. The problem is two-fold (threefold if you count my limited IQ...): one, all the formulas in the literature use 1 to n indexing except for some small exceptions. Second and more important, it is far more natural to program if the indices are aligned with the counts of the elements (I think there is a way to express that idea in modern algebra but I can't recall it). This lets you say "how many are there? Three--ok, grab the third one and do whatever to it" etc. Or "how many? zero--ok don't do anything". With zero-based indexing, you are always translating between counts and indices, but such translation is never a problem in one-based indexing. Given the long history of python and its ancestry in C (for which zero based indexing made lots of sense since it dovetailed with thinking in memory offsets in systems programming), there is probably nothing to be done now. I guess I just want to vent, but also to ask if anyone has found any way to deal with this issue in their own scientific programming. Or maybe I am the only with this problem, and if I were a real programmer would translate into zero indexing without even noticing.... Anyway, thanks for listening... From wbaxter at gmail.com Sun Jul 2 21:58:47 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 3 Jul 2006 10:58:47 +0900 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) Message-ID: Here's another possible now or never change: fix rand(), eye(), ones(), zeros(), and empty() to ALL take either a tuple argument or plain list. I know this has been discussed before, but I really don't see why these methods can't be overloaded to accept either one. Here are some wrappers I cooked up that basically do the trick, with a few exceptions which could easily be ironed out with a little input from someone who knows python better than me. I've got versions of rand(), eye() and ones() below. empty() and zeros() should basically be the same as ones(). Actually I guess it's not a "now or never change", because this should be completely backwards compatible. But still I find myself very frequently typing things like ones(2,2), and I'm getting tired of it not working. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- import types import numpy def xrand(*varg): """xrand(d0, d1, ..., dn) -> random values Return an array of the given dimensions which is initialized to random numbers from a uniform distribution in the range [0,1). xrand(d0, d1, ..., dn) -> random values or xrand((d0, d1, ..., dn)) -> random values """ if len(varg) == 0 or type(varg[0]) != types.TupleType: return rand(*varg) else: if len(varg) != 1: raise TypeError('Argument should either be a tuple or an argument list') else: return rand(*varg[0]) def xeye(N, *varg,**kwarg): """xeye(N, M=None, k=0, dtype=float64) eye returns a N-by-M 2-d array where the k-th diagonal is all ones, and everything else is zeros. """ if hasattr(N,'__getitem__') and hasattr(N,'__len__'): if len(N) == 0 or len(N) > 2: raise TypeError('First tuple to eye should be length 1 or 2') if len(N) == 1: return numpy.eye(N[0], None, *varg, **kwarg) else: return numpy.eye(N[0], N[1], *varg, **kwarg) return numpy.eye(N,*varg,**kwarg) def xones(shape, *varg, **kwarg): """xones(shape, dtype=, order='C') xones(shape, dtype=int_) returns an array of the given dimensions which is initialized to all ones. """ if hasattr(shape,'__getitem__'): return numpy.ones(shape,*varg,**kwarg) i = 0 for x in varg: if type(x)==types.IntType: i+=1 else: break tshape = (shape,) if i>0: tshape += tuple( varg[0:i] ) args = varg[i:] return numpy.ones(tshape, *args, **kwarg) def test(): xrand() xrand(2) xrand(2,2) xrand((2,2)) xeye(1) xeye(2,2) xeye(2,2) xeye(2,2,0,numpy.float64) xeye(2,2,k=0,dtype=numpy.float64) xeye((2,2),0,numpy.float64) xeye((2,2),k=0,dtype=numpy.float64) xones(1) xones([1]) xones([1]) xones(2,2) xones([2,2]) xones((2,2)) xones(numpy.array([2,2])) xones(1,numpy.float64) xones([1],numpy.float64) xones([1],numpy.float64) xones(2,2,numpy.float64) xones([2,2],numpy.float64) xones((2,2),numpy.float64) xones(numpy.array([2,2]),numpy.float64) xones(1,dtype=numpy.float64) xones([1],dtype=numpy.float64) xones([1],dtype=numpy.float64) xones(2,2,dtype=numpy.float64) xones([2,2],dtype=numpy.float64) xones((2,2),dtype=numpy.float64) xones(numpy.array([2,2]),dtype=numpy.float64) From wbaxter at gmail.com Sun Jul 2 22:16:26 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 3 Jul 2006 11:16:26 +0900 Subject: [Numpy-discussion] Combining arrays together Message-ID: What's the best way to combine say several 2-d arrays together into a grid? Here's the best I can see: >>> a = eye(2,2) >>> b = 2*a >>> c = 3*a >>> d = 4*a >>> r_[c_[a,b],c_[c,d]] array([[1, 0, 2, 0], [0, 1, 0, 2], [3, 0, 4, 0], [0, 3, 0, 4]]) In matlab you'd get the same effect by saying: [ a, b; c, d ] Compared to that, r_[c_[a,b],c_[c,d]] looks quite a mess. Would be nice if there were some operator like c_ that took a special argument that introduced a new row. Like maybe: c_[a,b, newrow, c,d] or maybe you could abuse the syntax and make something like this work: c_[a,b : c,d] or perhaps an empty argument could work? c_[a,b ,, c,d] Or empty tuple: c_[a,b, (), c,d] Hmm... I see there's already something in the code for handling 'special directives': >>> c_[a,b,'r',c,d] matrix([[1, 0, 2, 0, 3, 0, 4, 0], [0, 1, 0, 2, 0, 3, 0, 4]]) 'r' seems to turn the results into a matrix. Maybe that could be used to recognize a newline or something: >>> c_[a,b,'\n',c,d] Traceback (most recent call last): File "", line 1, in ? File "C:\Python24\Lib\site-packages\numpy\lib\index_tricks.py", line 239, in __getitem__ raise ValueError, "Unknown special directive." ValueError: Unknown special directive. --Bill -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at cox.net Sun Jul 2 23:22:25 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Sun, 02 Jul 2006 20:22:25 -0700 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: References: Message-ID: <44A88D71.6000904@cox.net> Bill Baxter wrote: > Here's another possible now or never change: > fix rand(), eye(), ones(), zeros(), and empty() to ALL take either a > tuple argument or plain list. Since a tuple seems to work fine as an argument I imagine you means something else. > > I know this has been discussed before, but I really don't see why > these methods can't be overloaded to accept either one. > Here are some wrappers I cooked up that basically do the trick, with a > few exceptions which could easily be ironed out with a little input > from someone who knows python better than me. I've got versions of > rand(), eye() and ones() below. empty() and zeros() should basically > be the same as ones(). > > Actually I guess it's not a "now or never change", because this should > be completely backwards compatible. But still I find myself very > frequently typing things like ones(2,2), and I'm getting tired of it > not working. Ah! You mean accepts either multiple args or a sequence. I don't like this kind of "helpful" gimmickry. I suspect that the errors that it masks in some inscrutable way more than make up for the minor time that it saves one on occasion. So mark me down as -1. -tim > > > ------------------------------------------------------------------------ > > > import types > import numpy > > def xrand(*varg): > """xrand(d0, d1, ..., dn) -> random values > > Return an array of the given dimensions which is initialized to > random numbers from a uniform distribution in the range [0,1). > > xrand(d0, d1, ..., dn) -> random values > or > xrand((d0, d1, ..., dn)) -> random values > """ > if len(varg) == 0 or type(varg[0]) != types.TupleType: > return rand(*varg) > else: > if len(varg) != 1: > raise TypeError('Argument should either be a tuple or an argument list') > else: > return rand(*varg[0]) > > > def xeye(N, *varg,**kwarg): > """xeye(N, M=None, k=0, dtype=float64) > > eye returns a N-by-M 2-d array where the k-th diagonal is all ones, > and everything else is zeros. > """ > > if hasattr(N,'__getitem__') and hasattr(N,'__len__'): > if len(N) == 0 or len(N) > 2: > raise TypeError('First tuple to eye should be length 1 or 2') > if len(N) == 1: > return numpy.eye(N[0], None, *varg, **kwarg) > else: > return numpy.eye(N[0], N[1], *varg, **kwarg) > > return numpy.eye(N,*varg,**kwarg) > > > def xones(shape, *varg, **kwarg): > """xones(shape, dtype=, order='C') > > xones(shape, dtype=int_) returns an array of the given > dimensions which is initialized to all ones. > """ > > if hasattr(shape,'__getitem__'): > return numpy.ones(shape,*varg,**kwarg) > > i = 0 > for x in varg: > if type(x)==types.IntType: > i+=1 > else: > break > > tshape = (shape,) > if i>0: > tshape += tuple( varg[0:i] ) > args = varg[i:] > return numpy.ones(tshape, *args, **kwarg) > > > def test(): > xrand() > xrand(2) > xrand(2,2) > xrand((2,2)) > > xeye(1) > xeye(2,2) > xeye(2,2) > xeye(2,2,0,numpy.float64) > xeye(2,2,k=0,dtype=numpy.float64) > xeye((2,2),0,numpy.float64) > xeye((2,2),k=0,dtype=numpy.float64) > > xones(1) > xones([1]) > xones([1]) > xones(2,2) > xones([2,2]) > xones((2,2)) > xones(numpy.array([2,2])) > > xones(1,numpy.float64) > xones([1],numpy.float64) > xones([1],numpy.float64) > xones(2,2,numpy.float64) > xones([2,2],numpy.float64) > xones((2,2),numpy.float64) > xones(numpy.array([2,2]),numpy.float64) > > xones(1,dtype=numpy.float64) > xones([1],dtype=numpy.float64) > xones([1],dtype=numpy.float64) > xones(2,2,dtype=numpy.float64) > xones([2,2],dtype=numpy.float64) > xones((2,2),dtype=numpy.float64) > xones(numpy.array([2,2]),dtype=numpy.float64) > > > > ------------------------------------------------------------------------ > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From aisaac at american.edu Mon Jul 3 00:04:48 2006 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 3 Jul 2006 00:04:48 -0400 Subject: [Numpy-discussion] Combining arrays together In-Reply-To: References: Message-ID: On Mon, 3 Jul 2006, Bill Baxter apparently wrote: > What's the best way to combine say several 2-d arrays > together into a grid? >>> help(N.bmat) Help on function bmat in module numpy.core.defmatrix: bmat(obj, ldict=None, gdict=None) Build a matrix object from string, nested sequence, or array. Ex: F = bmat('A, B; C, D') F = bmat([[A,B],[C,D]]) F = bmat(r_[c_[A,B],c_[C,D]]) all produce the same Matrix Object [ A B ] [ C D ] if A, B, C, and D are appropriately shaped 2-d arrays. hth, Alan Isaac From aisaac at american.edu Mon Jul 3 00:24:47 2006 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 3 Jul 2006 00:24:47 -0400 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: References: Message-ID: On Mon, 3 Jul 2006, Bill Baxter apparently wrote: > Here's another possible now or never change: > fix rand(), eye(), ones(), zeros(), and empty() to ALL take either a tuple > argument or plain list. > I know this has been discussed before, but I really don't > see why these methods can't be overloaded to accept either > one. I think the discussion has been slightly different than this. The "numpy way" for array creation is generally to specify dimension as tuples. A small number of functions violate this, which is an unhappy inconsistency. Specifically, rand() and randn() violate this. (Perhaps one could also say that eye() violates this; I do not yet have an opinion.) I argue that rand and randn should accept a tuple as the first argument. Whether the old behavior is also allowed, I have no opinion. But the numpy-consistent behavior should definitely be allowed. I perhaps wrongly understood Robert to argue that the current behavior of rand and randn is not a wart since i. alternative tuple-accepting functions are available and ii. the suprising behavior is documented. This seems quite wrong to me, and I am farily confident that such an argument would not be offered except in defence of legacy code. In fact, I would argue that if rand and randn are not "fixed" to accept a tuple, then they should be moved into a compatability module and not be considered part of numpy. Cheers, Alan Isaac From aisaac at american.edu Mon Jul 3 00:27:59 2006 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 3 Jul 2006 00:27:59 -0400 Subject: [Numpy-discussion] Does zero based indexing drive anyone else crazy? In-Reply-To: References: Message-ID: On Sun, 2 Jul 2006, Webb Sprague apparently wrote: > I have spent a huge amount of my time fixing and bending > my head around off-by-one errors caused by trying to index > matrices using 0 to n-1. I come from GAUSS so I am symphathetic, but in the end zero-based indexing is usually great. Anyway, ideally you will rely on vector/matrix operations rather than constantly tracking indices. fwiw, Alan From robert.kern at gmail.com Mon Jul 3 00:37:49 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 02 Jul 2006 23:37:49 -0500 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: References: Message-ID: Alan G Isaac wrote: > I argue that rand and randn should accept a tuple as the > first argument. Whether the old behavior is also allowed, > I have no opinion. But the numpy-consistent behavior should > definitely be allowed. I perhaps wrongly understood Robert > to argue that the current behavior of rand and randn is not > a wart since i. alternative tuple-accepting functions are > available and ii. the suprising behavior is documented. > This seems quite wrong to me, and I am farily confident that > such an argument would not be offered except in defence of > legacy code. i. Yes, you're still misunderstanding my arguments. ii. I'm bloody sick of rehashing it, so I won't be responding further. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Mon Jul 3 00:57:41 2006 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 03 Jul 2006 13:57:41 +0900 Subject: [Numpy-discussion] Time for beta1 of NumPy 1.0 In-Reply-To: <017b01c69d45$cce1c450$0a84a8c0@dsp.sun.ac.za> References: <017b01c69d45$cce1c450$0a84a8c0@dsp.sun.ac.za> Message-ID: <44A8A3C5.4040600@ar.media.kyoto-u.ac.jp> Albert Strasheim wrote: > Hey Chuck > > >> -----Original Message----- >> From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- >> discussion-bounces at lists.sourceforge.net] On Behalf Of Charles R Harris >> Sent: 01 July 2006 19:57 >> To: Robert Kern >> Cc: numpy-discussion at lists.sourceforge.net >> Subject: Re: [Numpy-discussion] Time for beta1 of NumPy 1.0 >> >> All, >> >> This is bit off topic, but a while ago there were some complaints about >> the usefulness of distutils. I note that KDE has gone over to using cmake >> after trying scon. I am not familiar with cmake, but perhaps someone here >> knows more and can comment on its suitability. >> > > CMake definately warrants investigation, but I think SCons might be a better > way to go. I think it would make it easier to reuse large parts of the > existing build code (e.g. conv_template.py could be converted into a SCons > builder without too much effort). Reusing parts of distutils and setuptools > would also be easier if the new tool is somehow Python-aware. > > I think the main problem with distutils in the NumPy context is that it was > never designed to build C/Fortran code over so many platforms with to many > possible build configurations. python setup.py install works pretty well, > but any kind of non-default configuration can become a bit hairy, despite > the excellent work on NumPy extensions to distutils. > > I'd like to take a stab at doing something with SCons in a few weeks' time. > Does anybody want to brainstorm on some ideas for what is needed from a > better build system for NumPy? Maybe a wiki page? > I have a small experience with scons, as a replacement of the auto* tools for small packages on my own (requirements for cross-building, library and header dependency, build of libraries, etc...). So I am willing to share my somewhat limited experience with scons (the code I am building scons with is using cblas/clapack, and has libraries + some unit testing, so we would not start from scratch). Also, I have access to x86 and ppc linux + mac os x + windows easily, which makes it easy to test on some common platforms, David P.S: Some comments on scons : I don't know distutils, so I can only compare to autotools: from *my* experience, you should think about scons as a Makefile replacement, and as a build framework to build onto. The main pro of scons: - having a real language for build rules programming is a real plus. It makes it much easier to extend that autoconf, for example (debugging m4 macro is not something I can enjoy much, and I am sure I am not alone). - the dependency checking works great - parallel build is explicitly handled - scons knows how to build library (static and shared) on the plateforms it supports - can be included in the project so scons does not need to be installed if needed (I have never used this feature myself). The main cons: - configuration part: there are some tools to test library/header a la autoconf, but this is far from great in the present situation, mainly because of the next point - option handling from the command line: there is some support, but nothing is automatic. On the long run, this is painful. - No support for library versioning; I am not sure about rpath support, which is useful for non-standard path installation. I don't know how difficult it would be to implement for all platforms (I have some - can be slow for big projects ? I have seen quite big projects (eg ardour: several hundred of .c and .h files) using scons, and it was not really slow, and I don't think it would be a problem for something like numpy which size is nothing compared to kde. To sum it up: as a make replacement, from a developer POV, it works great. As a tool for *distribution*, I am less convinced. For people familiar with autotools, scons is a great automake replacement. Everything else has to be implemented: autoconf, libtool, etc... My understanding is that those two tools (autoconf and libtool) are the ones most needed for numpy, so there is a lot of work to do if we want to use scons. From kwgoodman at gmail.com Mon Jul 3 01:27:52 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Sun, 2 Jul 2006 22:27:52 -0700 Subject: [Numpy-discussion] None as missing value Message-ID: I have a list x >> x [[1, None], [2, 3]] that I generate outside of numpy (with plain python). What is the best way to convert x into an array? This doesn't work >> asarray(x) array([[1, None], [2, 3]], dtype=object) <-- I'm hoping for something like dtype=float64 Is there something better than None to represent missing values so that when I convert to numpy arrays (actually matrices) I'll be all set? (I could use -99, but that would be even more embarrassing than my python skills.) If there is nothing better than None, what's a fast way to take care of the None's if x is faily large? From tim.leslie at gmail.com Mon Jul 3 01:46:12 2006 From: tim.leslie at gmail.com (Tim Leslie) Date: Mon, 3 Jul 2006 15:46:12 +1000 Subject: [Numpy-discussion] None as missing value In-Reply-To: References: Message-ID: On 7/3/06, Keith Goodman wrote: > I have a list x > > >> x > [[1, None], [2, 3]] > > that I generate outside of numpy (with plain python). What is the best > way to convert x into an array? This doesn't work > > >> asarray(x) > > array([[1, None], > [2, 3]], dtype=object) <-- I'm hoping for something like dtype=float64 > > Is there something better than None to represent missing values so > that when I convert to numpy arrays (actually matrices) I'll be all > set? (I could use -99, but that would be even more embarrassing than > my python skills.) > > If there is nothing better than None, what's a fast way to take care > of the None's if x is faily large? You might want to have a look at the masked array module in numpy (numpy.ma). The following example might help to get you started. >>> import numpy as N >>> x = [[1, None], [2, 3]] >>> m = N.ma.array(x, mask=N.equal(x, None)) >>> print m [[1 --] [2 3]] Cheers, Tim > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From aisaac at american.edu Mon Jul 3 01:54:17 2006 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 3 Jul 2006 01:54:17 -0400 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: References: Message-ID: > Alan G Isaac wrote: >> I argue that rand and randn should accept a tuple as the >> first argument. Whether the old behavior is also allowed, >> I have no opinion. But the numpy-consistent behavior should >> definitely be allowed. I perhaps wrongly understood Robert >> to argue that the current behavior of rand and randn is not >> a wart since i. alternative tuple-accepting functions are >> available and ii. the suprising behavior is documented. >> This seems quite wrong to me, and I am farily confident that >> such an argument would not be offered except in defence of >> legacy code. On Sun, 02 Jul 2006, Robert Kern apparently wrote: > i. Yes, you're still misunderstanding my arguments. > ii. I'm bloody sick of rehashing it, so I won't be responding further. Sorry, I should not have said: "not a wart". I perhaps should have instead said: "an acceptable wart", due to issues of backward compatability. At least that's what you implied here: http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3150643 And note that you emphasized the availability of the alternative functions here: http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3150702 I made the documentation comment based on your action in response to this conversation: adding documentation. You make a claim not an argument when you say: http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3150643 *Changing* the API of rand() and randn() doesn't solve any problem. *Removing* them might. Your primary argument against changing the API, as far as I can see, is that allowing *both* the extant behavior and the numpy consistent behavior will result in confusing code. http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3150643 Is this a knock-down argument? I think not. But in any case, I did not argue (above) for the combined behaviors: only for the numpy-consistent behavior. (Or for removing rand and randn, an action which I view as inferior but acceptable, and which you seem---at the link above---willing to consider.) To repeat a point I made before: http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3150728 numpy should take a step so that this question goes away, rather than maintain the status quo and see it crop up continually. (I.e., its recurrence should be understood to signal a problem.) Apologies in advance for any misrepresentations, Alan Isaac From oliphant.travis at ieee.org Mon Jul 3 01:50:56 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun, 02 Jul 2006 23:50:56 -0600 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: <001301c69de3$3f0d4890$0a84a8c0@dsp.sun.ac.za> References: <001301c69de3$3f0d4890$0a84a8c0@dsp.sun.ac.za> Message-ID: <44A8B040.7070007@ieee.org> Albert Strasheim wrote: > I did a few tests and this seems to work nicely: > Hey Albert, I read the post you linked to on the ctypes mailing list. I hope I didn't step on any toes with what I did in NumPy. I was just working on a ctypes interface and realized that a lot of the cruft to convert to what ctypes was expecting could and should be handled in a default way. The conversion of the shapes and strides information to the "right-kind" of ctypes integer plus the inclusion of ctypes in Python 2.5 was enough to convince me to put some kind of hook into the array object. I decided to make the ctypes attribute return an object so that the object could grow in the future additional attributes and/or methods to make it easier to interface with ctypes. I looked a bit at the source code and was disappointed to see that the _as_parameter_ approach is pretty limited. While there is talk of supporting a tuple return of _as_parameter_ in the source code comments, there is no evidence in the source itself of supporting it. There is also the changed way of handling additional arguments when argtypes is set on the function which uses the from_param method. Unfortunately, as Thomas responds to your post, the from_param method must be on one of the ctypes to work. You have to add support specifically for one of the c-data types. I think the _as_parameter_ approach returning a tuple that could be interpreted as the right ctype was better because it let other objects play the ctypes game. Basically, what you need is a type-map just like swig uses. But, now that ctypes is in Python, it will be slower to change. That's a bit unfortunate. But, ultimately, it works fine now. I don't know what is really gained by applying an argtypes to a function call anyway --- some kind of "type-checking". Is that supposed to be safer. For NumPy extension modules, type checking is only a small part of the memory-violation danger. In-correct array bounds and/or striding is far more common - not-to mention unaligned memory areas and/or unwriteable ones (like a read-only memory-mapped file). Thus, you're going to have to write a small "error-checking" code in Python anyway that calls out to the C-library with the right arguments. So, basically you write an extension module that calls c-code just as you did before, but now the entire "extension" module can all be in Python because the call to an arbitrary C-library is made using ctypes. For arrays, you will typically need to pass one or more of the data, the dimension information, the stride information, and the number of dimensions. The data-type will be known about because function calls usually handle only a specific data-type. Thus, I started with a ctypes object that produces this needed data in the format that ctypes needs, so it can be very easy to use an array with the ctypes module. Frankly, I'm quite impressed with the ease of accessing C-code available using c-types. It quite rivals f2py in enjoyment using it. One thing I like about c-types over Pyrex, for example is that it lets you separate the C-code from the Python code instead of "mixing it all together" I wouldn't be surprised if c-types doesn't become the dominant way to interface C/C++ and possibly even Fortran code (but it has tougher competition in f2py) once it grows up a little with additional ease-of-use. -Travis From oliphant.travis at ieee.org Mon Jul 3 01:55:13 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun, 02 Jul 2006 23:55:13 -0600 Subject: [Numpy-discussion] None as missing value In-Reply-To: References: Message-ID: <44A8B141.70506@ieee.org> Keith Goodman wrote: > I have a list x > > >>> x >>> > [[1, None], [2, 3]] > > that I generate outside of numpy (with plain python). What is the best > way to convert x into an array? This doesn't work > > >>> asarray(x) >>> > > array([[1, None], > [2, 3]], dtype=object) <-- I'm hoping for something like dtype=float64 > > Is there something better than None to represent missing values so > that when I convert to numpy arrays (actually matrices) I'll be all > set? (I could use -99, but that would be even more embarrassing than > my python skills.) > You can use a masked array specifically, or use nan's for missing values and just tell Python you want a floating-point array (because it finds the None object it's guessing incorrectly you want an "object" array. asarray(x, dtype=float) array([[ 1. , nan], [ 2. , 3. ]]) -Travis From pgmdevlist at mailcan.com Mon Jul 3 02:01:30 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Mon, 3 Jul 2006 02:01:30 -0400 Subject: [Numpy-discussion] None as missing value In-Reply-To: References: Message-ID: <200607030201.30697.pgmdevlist@mailcan.com> Keith, > > Is there something better than None to represent missing values so > > that when I convert to numpy arrays (actually matrices) I'll be all > > set? (I could use -99, but that would be even more embarrassing than > > my python skills.) As Tim suggested, have a look at the masked array module. However, the result will NOT be exportable to matrices, unless you fill the missing value first (for example, with -99 ;)). I use MaskedArrays a lot, they're quite flexible. An alternative would be to use nan instead of None: >>> import numpy as N >>> x = [[1, nan], [2, 3]] >>> print N.matrix(x) [[ 1. nan] [ 2. 3. ]] Of course, the solution will depend on what you need... From oliphant.travis at ieee.org Mon Jul 3 02:14:05 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 03 Jul 2006 00:14:05 -0600 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: <001301c69de3$3f0d4890$0a84a8c0@dsp.sun.ac.za> References: <001301c69de3$3f0d4890$0a84a8c0@dsp.sun.ac.za> Message-ID: <44A8B5AD.5020003@ieee.org> Albert Strasheim wrote: > I did a few tests and this seems to work nicely: > > In [133]: printf = ctypes.cdll.msvcrt.printf > > In [134]: printf.argtypes = [ctypes.c_char_p, ctypes.c_void_p] > > In [135]: x = N.array([1,2,3]) > > In [136]: printf('%p\n', x.ctypes.data) > 01CC8AC0 > Out[136]: 9 > > In [137]: hex(x.__array_interface__['data'][0]) > Out[137]: '0x1cc8ac0' > > It would be nice if we could the _as_parameter_ magic to work as well. See > this thread: > > http://aspn.activestate.com/ASPN/Mail/Message/ctypes-users/3122558 > > If I understood Thomas correctly, in the presence of argtypes an an > instance, say x, with _as_parameter_, the following is done to convert the > instance to something that the function accepts as its nth argument: > > func.argtypes[n].from_param(x._as_parameter_) > Unfortunately, from the source code this is not true. It would be an improvement, but the source code shows that the from_param of each type does something special and only works with particular kinds of data-types --- basic Python types or ctypes types. I did not see evidence that the _as_parameter_ method was called within any of the from_param methods of _ctypes.c > However, if I try passing x directly to printf, I get this: > > In [147]: printf('%p\n', x) > ... > ArgumentError: argument 2: exceptions.TypeError: wrong type > > However, this much works: > > In [148]: ctypes.c_void_p.from_param(x._as_parameter_) > Out[148]: > > So I don't understand why the conversion isn't happening automatically. > Despite any advertisement, the code is just not there in ctypes to do it when argtypes are present. Dealing with non-ctypes data is apparently not handled when argtypes are present. Get-rid of the argtypes setting and it will work (because then the _as_parameter_ method is called.... > Another quirk I noticed is that non-void pointers' from_param can't seem to > be used with ints. For example: > Yeah from the code it looks like each from_param method has it's own implementation that expects it's own set of "acceptable" things. There does not seem to be any way for an object to inform it appropriately. > I don't think this is too much of an issue though -- you could wrap all your > functions to take c_void_ps. If you happen to pass an int32 NumPy array to a > function expecting a double*, you might run into problems though. > Yeah, but you were going to run into trouble anyway. I don't really see a lot of "value-added" in the current type-checking c-types provides and would just ignore it at this point. Build a Python function that calls out to the c-function. > Maybe there should be a way to get a pointer to the NumPy array data as a > POINTER(c_double) if it is known that the array's dtype is float64. Ditto > for c_int/int32 and the others. > I could see value in arr.ctypes.data_as() arr.ctypes.strides_as() arr.ctypes.shape_as() methods which allow returning the data as different kinds of c-types things instead of the defaults --- Perhaps we just make data, strides, and shapes methods with an optional argument. -Travis From wbaxter at gmail.com Mon Jul 3 02:59:51 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 3 Jul 2006 15:59:51 +0900 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: References: Message-ID: On 7/3/06, Alan G Isaac wrote: > > > Your primary argument against changing the API, as far as > I can see, is that allowing *both* the extant behavior and > the numpy consistent behavior will result in confusing code. > http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3150643 > Is this a knock-down argument? I think not. In particular the argument was that it would make for code that's confusing for users to read. I.e. in some places the users see 'rand(2,2,3)' and in other places 'rand((2,2,3))'. I really don't see anything confusing about that. There's only one interpretation of either of those that makes sense. If there's any confusion issue I think it's more likely to come from looking at the source code of rand() itself. But even that is pretty minor given a few minutes to write a some comments about what's going on. Personally I think allowing the separate argument variations on these few methods would be a good thing. It makes ones() and zeros() more like Matlab's for one. But also it just looks cleaner to say ones(2,3,5) than it does to say ones((2,3,5)). I understand the general objections to it. -- It's kind of hacky with the *args,*kwargs; -- it leads to "more that one way to do it"; -- makes the implementation code a little harder to write and read (but I say it makes user code EASIER to write/read); -- can make IDEs confused (PyCrust at least tries suggests *args,**kwags as arguments); and -- isn't always possible to have both tuple and separate shape values if the arg after the shape arguments is also a number, like func(shape, num=0) But in this case, since these are functions that are both really simple and get a lot of use, I think it's worth it to make them easier to use, even if it uglifies the implementation. At this point since I've already got an implementation of this which works great for everything I want it to do, I'm not really going to be affected by whatever numpy decides to go with. I'll just wrap numpy with my functions. Making my own wrapper layer for my favorite numpy customizations was something I'd been meaning to do anyway. But I do think this is a change that would make numpy more user friendly. And as Alan points out, it seems to be a repeat discussion. I suspect that's because they are all functions newbies will encounter early on when they're trying to understand the logic behind numpy. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Mon Jul 3 03:26:21 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 3 Jul 2006 16:26:21 +0900 Subject: [Numpy-discussion] Combining arrays together In-Reply-To: References: Message-ID: Neat, didn't know about that. But, grr, always returns matrix regardless of argument types. --bb On 7/3/06, Alan G Isaac wrote: > > On Mon, 3 Jul 2006, Bill Baxter apparently wrote: > > What's the best way to combine say several 2-d arrays > > together into a grid? > > > >>> help(N.bmat) > Help on function bmat in module numpy.core.defmatrix: > > bmat(obj, ldict=None, gdict=None) > Build a matrix object from string, nested sequence, or array. > > Ex: F = bmat('A, B; C, D') > F = bmat([[A,B],[C,D]]) > F = bmat(r_[c_[A,B],c_[C,D]]) > > all produce the same Matrix Object [ A B ] > [ C D ] > > if A, B, C, and D are appropriately shaped 2-d arrays. > > hth, > Alan Isaac > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fullung at gmail.com Mon Jul 3 03:48:54 2006 From: fullung at gmail.com (Albert Strasheim) Date: Mon, 3 Jul 2006 09:48:54 +0200 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: <44A8B040.7070007@ieee.org> Message-ID: <009101c69e75$21b39fa0$0a84a8c0@dsp.sun.ac.za> Hello all, Travis Oliphant wrote: > Hey Albert, I read the post you linked to on the ctypes mailing list. > I hope I didn't step on any toes with what I did in NumPy. I was just Certainly not. This is great stuff! > working on a ctypes interface and realized that a lot of the cruft to > convert to what ctypes was expecting could and should be handled in a > default way. The conversion of the shapes and strides information to > the "right-kind" of ctypes integer plus the inclusion of ctypes in > Python 2.5 was enough to convince me to put some kind of hook into the > array object. I decided to make the ctypes attribute return an object > so that the object could grow in the future additional attributes and/or > methods to make it easier to interface with ctypes. Ah! So arr.ctypes.* is a collection of things that one typically needs to pass to C functions to get them to do their work, i.e. a pointer to data and some description of the data buffer (shape, strides, etc.). Very nice. > > Basically, what you need is a type-map just like swig uses. But, now > that ctypes is in Python, it will be slower to change. That's a bit > unfortunate. If we find the ctypes in Python 2.5 to be missing some features, maybe Thomas Heller could release "ctypes2" to tide us over until Python 2.6. But I think ctypes as it will appear in Python 2.5 is already excellent. > But, ultimately, it works fine now. I don't know what is really gained > by applying an argtypes to a function call anyway --- some kind of > "type-checking". Is that supposed to be safer. Yes, type-checking mostly. Some interesting things might happen when you're passing structs by value. But hopefully it just works. > For NumPy extension modules, type checking is only a small part of the > memory-violation danger. In-correct array bounds and/or striding is far > more common - not-to mention unaligned memory areas and/or unwriteable > ones (like a read-only memory-mapped file). Agreed. > Thus, you're going to have to write a small "error-checking" code in > Python anyway that calls out to the C-library with the right > arguments. So, basically you write an extension module that calls > c-code just as you did before, but now the entire "extension" module can > all be in Python because the call to an arbitrary C-library is made > using ctypes. Exactly. And once you have your DLL/shared library, there is no need to compile anything again. Another useful benefit on Windows is that you can build your extension in debug mode without having to have a debug build of Python. This is very useful. > > Frankly, I'm quite impressed with the ease of accessing C-code available > using c-types. It quite rivals f2py in enjoyment using it. Indeed. Viva ctypes! > Regards, Albert From oliphant.travis at ieee.org Mon Jul 3 03:51:06 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 03 Jul 2006 01:51:06 -0600 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: References: Message-ID: <44A8CC6A.40901@ieee.org> Hmmm..... One thing that bothers me is that it seems that those arguing *against* this behavior are relatively long-time users of Python while those arguing *for* it are from what I can tell somewhat new to Python/NumPy. I'm not sure what this means. Is the current behavior a *wart* you get used to or a clear *feature* providing some gain in programming efficiency. If new users complain about something, my tendency is to listen openly to the discussion and then discourage the implementation only if there is a clear reason to. With this one, I'm not so sure of the clear reason. I can see that "program parsers" would have a harder time with a "flexible" calling convention. But, in my calculus, user-interface benefit outweighs programmer benefit (all things being equal). It's easy as a programmer to get caught up in a very rigid system when many users want flexibility. I must confess that I don't like looking at ones((5,5)) either. I much prefer ones(5,5) or even ones([5,5]). But perhaps what this shows is something I've heard Robert discuss before that ihas not received enough attention. NumPy really has at least two "users" 1) application developers and 2) prototype developers (the MATLAB crowd for lack of a better term). These two groups usually want different functionality (and in reality most of us probably fall in both groups on different occasions). The first clamors for more rigidity and conformity even at the expense of user interfaces. These people usually want 1) long_but_explanatory_function names 2) rigid calling syntax 3) strict name-space control The second group which wants to get something prototyped and working quickly wants 1) short easy-to-remember names 2) flexible calling syntax 3) all-in-one name-space control My hypothesis is that when you first start with NumPy (especially with a MATLAB or IDL history) you seem to start out in group 2 and stay there for quick projects. Then, as code-size increases and applications get larger, you become more like group 1. I think both groups have valid points to make and both styles can be useful and one point or another. Perhaps, the solution is the one I have barely begun to recognize, that others of you have probably already seen. A) Make numpy a good "library" for group 1. B) Make a shallow "name-space" (like pylab) with the properties of group 2. Perhaps a good solution is to formalize the discussion and actually place in NumPy a name-space package much like Bill has done privately. -Travis From fullung at gmail.com Mon Jul 3 04:11:21 2006 From: fullung at gmail.com (Albert Strasheim) Date: Mon, 3 Jul 2006 10:11:21 +0200 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: <44A8B5AD.5020003@ieee.org> Message-ID: <009201c69e78$44fa0d20$0a84a8c0@dsp.sun.ac.za> Hello all Travis Oliphant wrote: > > Unfortunately, from the source code this is not true. It would be an > improvement, but the source code shows that the from_param of each type > does something special and only works with particular kinds of > data-types --- basic Python types or ctypes types. I did not see > evidence that the _as_parameter_ method was called within any of the > from_param methods of _ctypes.c To summarise, I think we've come to the conclusion that one should avoid argtypes when mixing NumPy with ctypes? (at least for now) The extensions to .ctypes you propose below should make it possible to use NumPy arrays with argtypes set. "Raw" C functions will probably be wrapped by a Python function 99.9% of the time for error checking, etc. This hides the need to call the .ctypes stuff from the user. > > > Maybe there should be a way to get a pointer to the NumPy array data as > a > > POINTER(c_double) if it is known that the array's dtype is float64. > Ditto > > for c_int/int32 and the others. > > > > I could see value in > > arr.ctypes.data_as() > arr.ctypes.strides_as() > arr.ctypes.shape_as() > > methods which allow returning the data as different kinds of c-types > things instead of the defaults --- Perhaps we just make data, strides, > and shapes methods with an optional argument. Agreed. If you really like argtypes, arr.ctypes.data_as() would be perfect for doing the necessary work to make sure ctypes accepts the array. arr.ctypes.data_as(c_type) could be implemented as ctypes.cast(x.ctypes.data, ctypes.POINTER(c_type)) c_void_p, c_char_p and c_wchar_p are special cases that aren't going to work here, so maybe it should be ctypes.cast(x.ctypes.data, c_type) in which case one mostly call it as arr.ctypes.data_as(POINTER(c_type)). Regards, Albert From svetosch at gmx.net Mon Jul 3 04:55:26 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Mon, 03 Jul 2006 10:55:26 +0200 Subject: [Numpy-discussion] Combining arrays together In-Reply-To: References: Message-ID: <44A8DB7E.60205@gmx.net> Bill Baxter schrieb: > Neat, didn't know about that. But, grr, always returns matrix > regardless of argument types. > --bb > Well Bill mayb you should have stayed with matrices ;-) But I also see no reason why it shouldn't be expected work for 2d-arrays in general. (Or maybe even for more dimensions as well?) Perhaps you should file this as a new ticket. cheers, sven From cimrman3 at ntc.zcu.cz Mon Jul 3 05:35:15 2006 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Mon, 03 Jul 2006 11:35:15 +0200 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> Message-ID: <44A8E4D3.5000603@ntc.zcu.cz> Sasha wrote: > On 7/2/06, Norbert Nemec wrote: >> ... >> Does anybody know about the internals of the python "set"? How is >> .keys() implemented? I somehow have really doubts about the efficiency >> of this method. >> > Set implementation (Objects/setobject.c) is a copy and paste job from > dictobject with values removed. As a result it is heavily optimized > for the case of string valued keys - a case that is almost irrelevant > for numpy. > > I think something like the following (untested, 1d only) will probably > be much faster and sorted: > > def unique(x): > s = sort(x) > r = empty_like(s) > r[:-1] = s[1:] > r[-1] = s[0] > return s[r != s] There are 1d array set operations like this already in numpy (numpy/lib/arraysetops.py - unique1d, ...) r. From simon at arrowtheory.com Mon Jul 3 05:51:38 2006 From: simon at arrowtheory.com (Simon Burton) Date: Mon, 3 Jul 2006 11:51:38 +0200 Subject: [Numpy-discussion] Does zero based indexing drive anyone else crazy? In-Reply-To: References: Message-ID: <20060703115138.0315d198.simon@arrowtheory.com> On Sun, 2 Jul 2006 16:36:14 -0700 "Webb Sprague" wrote: > > Given the long history of python and its ancestry in C (for which zero > based indexing made lots of sense since it dovetailed with thinking in > memory offsets in systems programming), there is probably nothing to > be done now. I guess I just want to vent, but also to ask if anyone > has found any way to deal with this issue in their own scientific > programming. the mathemaician John Conway, in his book, "the book of numbers" has a brilliant discussion of just this issue. It does indeed make sense mathematically. Simon. From svetosch at gmx.net Mon Jul 3 05:53:43 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Mon, 03 Jul 2006 11:53:43 +0200 Subject: [Numpy-discussion] None as missing value In-Reply-To: <44A8B141.70506@ieee.org> References: <44A8B141.70506@ieee.org> Message-ID: <44A8E927.9000405@gmx.net> Travis Oliphant schrieb: > > You can use a masked array specifically, or use nan's for missing values > and just tell Python you want a floating-point array (because it finds > the None object it's guessing incorrectly you want an "object" array. > > asarray(x, dtype=float) > > array([[ 1. , nan], > [ 2. , 3. ]]) > Is there anything else besides None which is recognized/converted to numpy.nan? Put differently, where can I find documentation about basic nan definition and handling in numpy? (I have the numpy book which covers isnan etc., when you have the nans already set up.) I was also a bit surprised at the following behavior: >>> a = numpy.asarray([1,1]) >>> a array([1, 1]) >>> a[0]=numpy.nan >>> a array([0, 1]) Is this a bug or intended? Thanks, Sven From svetosch at gmx.net Mon Jul 3 06:44:10 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Mon, 03 Jul 2006 12:44:10 +0200 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: <44A8CC6A.40901@ieee.org> References: <44A8CC6A.40901@ieee.org> Message-ID: <44A8F4FA.2060200@gmx.net> I agree with everything in your post, so I'm really happy you're a central figure of numpy! As for rand(), ones() etc.: I don't mind (too much) the double pair of parentheses in ones((5,5)), but I find Robert's proposal in an earlier thread ("If you want a function that takes tuples, use numpy.random.random().") a little impractical, because I'm mostly one of the prototypers as you defined them. I've come to realize that for me (and from what I read, for Bill as well) rand() is not adding convenience because we cannot settle on any habit (neither single nor double pairs of parentheses) and therefore we continue to get one or the other wrong when we're not paying full attention. Anything that allows me to develop a consistent habit is fine with me! Thanks much, Sven Travis Oliphant schrieb: > Hmmm..... One thing that bothers me is that it seems that those > arguing *against* this behavior are relatively long-time users of Python > while those arguing *for* it are from what I can tell somewhat new to > Python/NumPy. I'm not sure what this means. > > Is the current behavior a *wart* you get used to or a clear *feature* > providing some gain in programming efficiency. > > If new users complain about something, my tendency is to listen openly > to the discussion and then discourage the implementation only if there > is a clear reason to. > > With this one, I'm not so sure of the clear reason. I can see that > "program parsers" would have a harder time with a "flexible" calling > convention. But, in my calculus, user-interface benefit outweighs > programmer benefit (all things being equal). > > It's easy as a programmer to get caught up in a very rigid system when > many users want flexibility. > > I must confess that I don't like looking at ones((5,5)) either. I much > prefer ones(5,5) or even ones([5,5]). > > But perhaps what this shows is something I've heard Robert discuss > before that ihas not received enough attention. NumPy really has at > least two "users" 1) application developers and 2) prototype developers > (the MATLAB crowd for lack of a better term). > > These two groups usually want different functionality (and in reality > most of us probably fall in both groups on different occasions). The > first clamors for more rigidity and conformity even at the expense of > user interfaces. These people usually want > > 1) long_but_explanatory_function names > 2) rigid calling syntax > 3) strict name-space control > > The second group which wants to get something prototyped and working > quickly wants > > 1) short easy-to-remember names > 2) flexible calling syntax > 3) all-in-one name-space control > > My hypothesis is that when you first start with NumPy (especially with a > MATLAB or IDL history) you seem to start out in group 2 and stay there > for quick projects. Then, as code-size increases and applications get > larger, you become more like group 1. > > I think both groups have valid points to make and both styles can be > useful and one point or another. Perhaps, the solution is the one I have > barely begun to recognize, that others of you have probably already seen. > > A) Make numpy a good "library" for group 1. > B) Make a shallow "name-space" (like pylab) with the properties of group 2. > > Perhaps a good solution is to formalize the discussion and actually > place in NumPy a name-space package much like Bill has done privately. > > -Travis > > > > > > > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From stefan at sun.ac.za Mon Jul 3 07:48:04 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Mon, 3 Jul 2006 13:48:04 +0200 Subject: [Numpy-discussion] Combining arrays together In-Reply-To: References: Message-ID: <20060703114803.GA7427@mentat.za.net> On Mon, Jul 03, 2006 at 11:16:26AM +0900, Bill Baxter wrote: > What's the best way to combine say several 2-d arrays together into a grid? > Here's the best I can see: > > >>> a = eye(2,2) > >>> b = 2*a > >>> c = 3*a > >>> d = 4*a > >>> r_[c_[a,b],c_[c,d]] > array([[1, 0, 2, 0], > [0, 1, 0, 2], > [3, 0, 4, 0], > [0, 3, 0, 4]]) > > In matlab you'd get the same effect by saying: [ a, b; c, d ] > > Compared to that, r_[c_[a,b],c_[c,d]] looks quite a mess. You could always explicitly write out what you are doing, i.e. In [47]: N.vstack((N.hstack((a,b)), N.hstack((c,d)))) Out[47]: array([[ 1., 0., 2., 0.], [ 0., 1., 0., 2.], [ 3., 0., 4., 0.], [ 0., 3., 0., 4.]]) St?fan From acorriga at gmu.edu Mon Jul 3 08:36:53 2006 From: acorriga at gmu.edu (Andrew Corrigan) Date: Mon, 03 Jul 2006 08:36:53 -0400 Subject: [Numpy-discussion] arbitrary number of newaxis Message-ID: <44A90F65.2010805@gmu.edu> In a function I'm writing, I multiply two arrays together: A and B. where A.ndim == 2 and I don't know B.ndim in advance If I knew B.ndim == 3, then I would write A[:,:,newaxis,newaxis,newaxis]*B[newaxis,newaxis,...] or if I knew that B.ndim == 1 then I would write A[:,:,newaxis*B[newaxis,newaxis,...] but I don't know B.ndim. Essentially I want to say something like: A[:,:,repeat(newaxis, B.ndim)]*B[newaxis,newaxis,...] How can I express what I mean, such that it actually works? From ndarray at mac.com Mon Jul 3 08:54:43 2006 From: ndarray at mac.com (Sasha) Date: Mon, 3 Jul 2006 08:54:43 -0400 Subject: [Numpy-discussion] arbitrary number of newaxis In-Reply-To: <44A90F65.2010805@gmu.edu> References: <44A90F65.2010805@gmu.edu> Message-ID: On 7/3/06, Andrew Corrigan wrote: > ... Essentially I want to say something like: > A[:,:,repeat(newaxis, B.ndim)]*B[newaxis,newaxis,...] > > How can I express what I mean, such that it actually works? >>> A[(slice(None),)*2 + (newaxis,)*B.ndim] From fullung at gmail.com Mon Jul 3 08:59:11 2006 From: fullung at gmail.com (Albert Strasheim) Date: Mon, 3 Jul 2006 14:59:11 +0200 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: <44A73B5D.60002@ieee.org> Message-ID: <00b201c69ea0$7ab7a710$0a84a8c0@dsp.sun.ac.za> Hello all Travis Oliphant wrote: > The ctypes-conversion object has attributes which return c_types aware > objects so that the information can be passed directly to c-code (as an > integer, the number of dimensions can already be passed using c-types). > > The information available and it's corresponding c_type is > > data - c_void_p > shape, strides - c_int * nd or c_long * nd or c_longlong * nd > depending on platform Stefan and I did some more experiments and it seems like .ctypes.strides isn't doing the right thing for subarrays. For example: In [52]: x = N.rand(3,4) In [57]: [x.ctypes.strides[i] for i in range(x.ndim)] Out[57]: [32, 8] This looks fine. But for this subarray: In [56]: [x[1:3,1:4].ctypes.strides[i] for i in range(x.ndim)] Out[56]: [32, 8] In this case, I think one wants strides[0] (the row stride) to return 40. .ctypes.data already seems to do the right thing: In [60]: x.ctypes.data Out[60]: c_void_p(31685288) In [61]: x[1:3,1:4].ctypes.data Out[61]: c_void_p(31685328) In [62]: 31685288-31685328 Out[62]: 40 What would be a good way of dealing with discontiguous arrays? It seems like one might want to disable their .ctypes attribute. Regards, Albert From david.huard at gmail.com Mon Jul 3 09:19:07 2006 From: david.huard at gmail.com (David Huard) Date: Mon, 3 Jul 2006 09:19:07 -0400 Subject: [Numpy-discussion] iterate along a ray: linear algebra? In-Reply-To: <51f97e530606300713w1c167cf3j10c36d24f87326cf@mail.gmail.com> References: <51f97e530606300713w1c167cf3j10c36d24f87326cf@mail.gmail.com> Message-ID: <91cf711d0607030619v3155ea22o6529189f8a4f0fc8@mail.gmail.com> Hi Stephen, I don't know much about image analysis, but in the scipy tutorial (7.2Filtering), there is an example of an image filter that highlights the edges of an image. If I guess correctly, it finds the smoothing spline of the image and then computes the derivative of the spline. A high derivative means that the image intensity shifts rapidly, so maybe that could help you. David 2006/6/30, stephen emslie : > > I am in the process of implementing an image processing algorithm that > requires following rays extending outwards from a starting point and > calculating the intensity derivative at each point. The idea is to find the > point where the difference in intensity goes beyond a particular threshold. > > Specifically I'm examining an image of an eye to find the pupil, and the > edge of the pupil is a sharp change in intensity. > > How does one iterate along a line in a 2d matrix, and is there a better > way to do this? Is this a problem that linear algebra can help with? > > Thanks > Stephen Emslie > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at cox.net Mon Jul 3 10:19:29 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon, 03 Jul 2006 07:19:29 -0700 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: <44A8CC6A.40901@ieee.org> References: <44A8CC6A.40901@ieee.org> Message-ID: <44A92771.2040904@cox.net> Travis Oliphant wrote: > Hmmm..... One thing that bothers me is that it seems that those > arguing *against* this behavior are relatively long-time users of Python > while those arguing *for* it are from what I can tell somewhat new to > Python/NumPy. I'm not sure what this means. > > Is the current behavior a *wart* you get used to or a clear *feature* > providing some gain in programming efficiency. > > If new users complain about something, my tendency is to listen openly > to the discussion and then discourage the implementation only if there > is a clear reason to. > > With this one, I'm not so sure of the clear reason. I can see that > "program parsers" would have a harder time with a "flexible" calling > convention. But, in my calculus, user-interface benefit outweighs > programmer benefit (all things being equal). > > It's easy as a programmer to get caught up in a very rigid system when > many users want flexibility. > > I must confess that I don't like looking at ones((5,5)) either. I much > prefer ones(5,5) or even ones([5,5]). > I'm not sure why more people don't write the second version. It's significantly easier to read since the mixed brackets stand out better. If one buys into Guido's description of what tuples and lists are for, then it's also more appropriate since a shape is homogeneous, variable length sequence (as opposed to fixed length heterogeneous collection which would be more appropriate to represent using a tuple). Not everyone buys into that, using tuples as essentially immutable lists, but it's easier to read in any event. I also find: ones([5, 5], dt) clearer than: ones(5, 5, dt) or, more dramatic, consider: ones([dx, dy], dt) versus ones(dx, dy, dt). Brrrr! A side note: one reason I'm big on specifying the dtype is that according to Kahan (many papers available at http://www.cs.berkeley.edu/~wkahan/) the single best thing you can do to check that an implementation is numerically sane is to examine the results at different precisions (say float32 and float64 since they're commonly available) and verify that the results don't go off the rails. Kahan is oft quoted by Tim Peters who seems to have one of the better grasps of the fiddly aspects of floating point in the Python community, so I give his views a lot of weight. Since I try to program with this in mind, at least for touchy numerical code, I end up parameterizing things based on dtype anyway. Then it's relatively easy to check that things are behaving simply by changing the dtype that is passed in. > But perhaps what this shows is something I've heard Robert discuss > before that ihas not received enough attention. NumPy really has at > least two "users" 1) application developers and 2) prototype developers > (the MATLAB crowd for lack of a better term). > > These two groups usually want different functionality (and in reality > most of us probably fall in both groups on different occasions). The > first clamors for more rigidity and conformity even at the expense of > user interfaces. These people usually want > > 1) long_but_explanatory_function names > 2) rigid calling syntax > One might also call it consistent calling syntax, but yes I'd put myself in this group and having consistent calling syntax catches many errors that would otherwise pass silently. It's also easier to figure out what's going one when coming back to functions written in the distant pass if their is only one calling syntax. > 3) strict name-space control > This combined with more explanatory names mentioned in 1 help make it easier to decipher code written in yesteryear. > The second group which wants to get something prototyped and working > quickly wants > > 1) short easy-to-remember names > 2) flexible calling syntax > 3) all-in-one name-space control > > My hypothesis is that when you first start with NumPy (especially with a > MATLAB or IDL history) you seem to start out in group 2 and stay there > for quick projects. Then, as code-size increases and applications get > larger, you become more like group 1. > > I think both groups have valid points to make and both styles can be > useful and one point or another. Perhaps, the solution is the one I have > barely begun to recognize, that others of you have probably already seen. > > A) Make numpy a good "library" for group 1. > B) Make a shallow "name-space" (like pylab) with the properties of group 2. > > Perhaps a good solution is to formalize the discussion and actually > place in NumPy a name-space package much like Bill has done privately. > +1 -tim From aisaac at american.edu Mon Jul 3 10:58:02 2006 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 3 Jul 2006 10:58:02 -0400 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: <44A8F4FA.2060200@gmx.net> References: <44A8CC6A.40901@ieee.org> <44A8F4FA.2060200@gmx.net> Message-ID: On Mon, 03 Jul 2006, Sven Schreiber apparently wrote: > Anything that allows me to develop a consistent habit is fine with me! Consistency! That is exactly the issue, especially for those who wish to teach with numpy. I do not want to tell my students to write ones([5,5]) but rand(5,5) and although relatively new to Python I actually like the practice of providing dimensions in a list or tuple. But I care less about the choice of convention than about adherence to the convention. Cheers, Alan Isaac From jnc at ecs.soton.ac.uk Mon Jul 3 11:16:53 2006 From: jnc at ecs.soton.ac.uk (John Carter) Date: Mon, 03 Jul 2006 16:16:53 +0100 Subject: [Numpy-discussion] Error building numpy from svn Message-ID: <7.0.1.0.0.20060703153308.01aa0340@ecs.soton.ac.uk> Hi, I'm getting errors when I try and build numpy from the svn. I've followed the instructions on Installing Sci/Py/Windows at http://www.scipy.org/Installing_SciPy/Windows?highlight=%28%28----%28-%2A%29%28%5Cr%29%3F%5Cn%29%28.%2A%29CategoryInstallation%5Cb%29 I've downloaded, built and tested Atlas, Lapack, etc My computer is set up as follows: MinGW 3.4.2 Cygwin 3.4.4 (used for Atlas and Lapack) Win XP SP2 I've tried building using Cygwin instead MinGW with similar results. (not using cygwin python) I also have VC++ 6 and VC++ Express installed, but neither is on the path when I'm attempting to build numpy. I normally have no problems building my own python extensions using numarray, numpy or PIL. I use pyrex or home rolled code. These all work with 2.3 and 2.4. I'd be grateful of any pointers as to what might be wrong, Thanks in advance John =================================================================================== Using Python 2.3 D:.\numpy> setup.py config --compiler=mingwg2 build --compiler=mingw32 bdist_wininst ...... compile options: '-DNO_ATLAS_INFO=2 -Id:\work\Programming\numerical\numpy\numpy\core\include -Ibuild\src.win32-2.3\nump \core -Id:\work\Programming\numerical\numpy\numpy\core\src -Id:\work\Programming\numerical\numpy\numpy\core\include -IC \PYTHON23\include -IC:\PYTHON23\PC -c' C:\MINGW\BIN\g77.exe -shared build\temp.win32-2.3\Release\work\programming\numerical\numpy\numpy\linalg\lapack_litemodu e.o -Ld:\work\Programming\numerical\libs -LC:/MINGW/BIN/../lib/gcc/mingw32/3.4.2 -LC:\PYTHON23\libs -LC:\PYTHON23\PCBui d -llapack -llapack -lf77blas -lcblas -latlas -lpython23 -lgcc -lg2c -o build\lib.win32-2.3\numpy\linalg\lapack_lite.py C:/MINGW/BIN/../lib/gcc/mingw32/3.4.2/libgcc.a(__main.o)(.text+0x4f): undefined reference to `__EH_FRAME_BEGIN__' C:/MINGW/BIN/../lib/gcc/mingw32/3.4.2/libgcc.a(__main.o)(.text+0x73): undefined reference to `__EH_FRAME_BEGIN__' collect2: ld returned 1 exit status ========================================================================================= Using Python 2.4 D:.\numpy> setup.py config --compiler=mingwg2 build --compiler=mingw32 bdist_wininst Running from numpy source directory. No module named __svn_version__ F2PY Version 2_2727 blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not find in C:\PYTHON24\lib libraries mkl,vml,guide not find in C:\ libraries mkl,vml,guide not find in C:\PYTHON24\libs NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS FOUND: libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] library_dirs = ['d:\\work\\Programming\\numerical\\libs'] language = c No module named msvccompiler in numpy.distutils, trying from distutils.. Traceback (most recent call last): File "D:\work\Programming\numerical\numpy\setup.py", line 84, in ? setup_package() File "D:\work\Programming\numerical\numpy\setup.py", line 77, in setup_package configuration=configuration ) File "D:\work\Programming\numerical\numpy\numpy\distutils\core.py", line 144, in setup config = configuration() File "D:\work\Programming\numerical\numpy\setup.py", line 43, in configuration config.add_subpackage('numpy') File "D:\work\Programming\numerical\numpy\numpy\distutils\misc_util.py", line 740, in add_subpackage caller_level = 2) File "D:\work\Programming\numerical\numpy\numpy\distutils\misc_util.py", line 723, in get_subpackage caller_level = caller_level + 1) File "D:\work\Programming\numerical\numpy\numpy\distutils\misc_util.py", line 670, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File ".\numpy\setup.py", line 9, in configuration config.add_subpackage('core') File "D:\work\Programming\numerical\numpy\numpy\distutils\misc_util.py", line 740, in add_subpackage caller_level = 2) File "D:\work\Programming\numerical\numpy\numpy\distutils\misc_util.py", line 723, in get_subpackage caller_level = caller_level + 1) File "D:\work\Programming\numerical\numpy\numpy\distutils\misc_util.py", line 670, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File "d:\work\Programming\numerical\numpy\numpy\core\setup.py", line 207, in configuration blas_info = get_info('blas_opt',0) File "D:\work\Programming\numerical\numpy\numpy\distutils\system_info.py", line 256, in get_info return cl().get_info(notfound_action) File "D:\work\Programming\numerical\numpy\numpy\distutils\system_info.py", line 397, in get_info self.calc_info() File "D:\work\Programming\numerical\numpy\numpy\distutils\system_info.py", line 1244, in calc_info atlas_version = get_atlas_version(**version_info) File "D:\work\Programming\numerical\numpy\numpy\distutils\system_info.py", line 1085, in get_atlas_version library_dirs=config.get('library_dirs', []), File "D:\work\Programming\numerical\numpy\numpy\distutils\command\config.py", line 101, in get_output self._check_compiler() File "D:\work\Programming\numerical\numpy\numpy\distutils\command\config.py", line 34, in _check_compiler old_config._check_compiler(self) File "C:\PYTHON24\lib\distutils\command\config.py", line 107, in _check_compiler dry_run=self.dry_run, force=1) File "D:\work\Programming\numerical\numpy\numpy\distutils\ccompiler.py", line 333, in new_compiler compiler = klass(None, dry_run, force) File "C:\PYTHON24\lib\distutils\msvccompiler.py", line 211, in __init__ self.__macros = MacroExpander(self.__version) File "C:\PYTHON24\lib\distutils\msvccompiler.py", line 112, in __init__ self.load_macros(version) File "C:\PYTHON24\lib\distutils\msvccompiler.py", line 133, in load_macros raise DistutilsPlatformError, \ distutils.errors.DistutilsPlatformError: The .NET Framework SDK needs to be installed before building extensions for Pyt hon. D:.\numpy> Dr. John N. Carter jnc at ecs.soton.ac.uk ISIS http://www.ecs.soton.ac.uk/~jnc/ From pgmdevlist at mailcan.com Mon Jul 3 12:40:04 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Mon, 3 Jul 2006 12:40:04 -0400 Subject: [Numpy-discussion] None as missing value In-Reply-To: <44A8E927.9000405@gmx.net> References: <44A8B141.70506@ieee.org> <44A8E927.9000405@gmx.net> Message-ID: <200607031240.05066.pgmdevlist@mailcan.com> > I was also a bit surprised at the following behavior: >>> a = numpy.asarray([1,1]) >>> a array([1, 1]) >>> a[0]=numpy.nan >>> a array([0, 1]) Seems to affect only the int_ arrays: >>> a = numpy.asarray([1,1], dtype=float_) >>> a array([1., 1.]) >>> a[0]=numpy.nan >>> a array([ nan, 1. ]) From david.huard at gmail.com Mon Jul 3 13:00:35 2006 From: david.huard at gmail.com (David Huard) Date: Mon, 3 Jul 2006 13:00:35 -0400 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <44A8E4D3.5000603@ntc.zcu.cz> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> <44A8E4D3.5000603@ntc.zcu.cz> Message-ID: <91cf711d0607031000l286da52fn30d98798057f12c5@mail.gmail.com> Here is a quick benchmark between numpy's unique, unique1d and sasha's unique: x = rand(100000)*100 x = x.astype('i') %timeit unique(x) 10 loops, best of 3: 525 ms per loop %timeit unique_sasha(x) 100 loops, best of 3: 10.7 ms per loop timeit unique1d(x) 100 loops, best of 3: 12.6 ms per loop So I wonder what is the added value of unique? Could unique1d simply become unique ? Cheers, David P.S. I modified sasha's version to account for the case where all elements are identical, which returned an empty array. def unique_sasha(x): s = sort(x) r = empty(s.shape, float) r[:-1] = s[1:] r[-1] = NaN return s[r != s] 2006/7/3, Robert Cimrman : > > Sasha wrote: > > On 7/2/06, Norbert Nemec wrote: > >> ... > >> Does anybody know about the internals of the python "set"? How is > >> .keys() implemented? I somehow have really doubts about the efficiency > >> of this method. > >> > > Set implementation (Objects/setobject.c) is a copy and paste job from > > dictobject with values removed. As a result it is heavily optimized > > for the case of string valued keys - a case that is almost irrelevant > > for numpy. > > > > I think something like the following (untested, 1d only) will probably > > be much faster and sorted: > > > > def unique(x): > > s = sort(x) > > r = empty_like(s) > > r[:-1] = s[1:] > > r[-1] = s[0] > > return s[r != s] > > There are 1d array set operations like this already in numpy > (numpy/lib/arraysetops.py - unique1d, ...) > > r. > > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndarray at mac.com Mon Jul 3 14:07:00 2006 From: ndarray at mac.com (Sasha) Date: Mon, 3 Jul 2006 14:07:00 -0400 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: References: <44A8CC6A.40901@ieee.org> <44A8F4FA.2060200@gmx.net> Message-ID: On 7/3/06, Alan G Isaac wrote: > .... > Consistency! That is exactly the issue, > especially for those who wish to teach with numpy. > > I do not want to tell my students to write > ones([5,5]) > but > rand(5,5) > and although relatively new to Python > I actually like the practice of providing > dimensions in a list or tuple. > Consistency is already lost because 1d case allows both ones(5) and ones([5]) (and even ones((5,)) if anyone can tolerate that abomination). I don't think those who argue for sequence only are willing to require ones([5]). Remember, "A Foolish Consistency is the Hobgoblin of Little Minds" (Ralph Waldo Emerson (1803?1882), adopted without attribution as a section heading in PEP 8 ). I think the current situation strikes the right balance between convenience and consistency. From Chris.Barker at noaa.gov Mon Jul 3 14:30:20 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 03 Jul 2006 11:30:20 -0700 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: <44A92771.2040904@cox.net> References: <44A8CC6A.40901@ieee.org> <44A92771.2040904@cox.net> Message-ID: <44A9623C.6040708@noaa.gov> Tim Hochberg wrote: > I also find: > > ones([5, 5], dt) > > clearer than: > > ones(5, 5, dt) > > or, more dramatic, consider: ones([dx, dy], dt) versus ones(dx, dy, dt). > Brrrr! And I imagine this is EXACTLY why Numeric was originally designed that way: when there are more arguments to pass, then a shape is essentially a single entity. Also consider this: MyShape = a.shape b = ones(MyShape, N.float_) much better than: b = ones(*MyShape, N.float_) At least I think so. > Kahan is oft quoted by Tim Peters who seems to have one of the better > grasps of the fiddly aspects of floating point in the Python community, > so I give his views a lot of weight. I took at class with Kahan at Berkeley (that I barely passed), he is definitely someone to listen to. > NumPy really has at >> least two "users" 1) application developers and 2) prototype developers FWIW: I came from a heavy MATLAB background, still do far more prototyping that really application writing (though I certainly do the later), and I want: >> 1) longish_explanatory names >> 2) consistent calling syntax (note that I've edited those a bit...) And above all: >> 3) strict name-space control In my experience that that is a difference between "prototyping" and "using Python like Matlab". The sooner that you learn to use Python like Python, rather than trying to use it like Matlab, the better off you'll be. Python is an excellent language for prototyping, but even for that, you're better off working with, rather than against, its features. As for the current issue: 1) Make everything take a sequence for the size argument. 2) dump the "convenience functions" into a compatibility module. Is it so hard to write: import numpy.random.uniform as rand (though I notice that numpy.random.rand() is the inconsistent "convenience function" -- darn) Sasha wrote: > Consistency is already lost because 1d case allows both ones(5) and > ones([5]) (and even ones((5,)) if anyone can tolerate that > abomination). I don't think those who argue for sequence only are > willing to require ones([5]). But I'd be happy to ban ones(5) (by the way, doesn't Matlab's ones(5) produce a 5X5 matrix?). We certainly don't' want to require a list, that would totally go against Python's concept of duck typing. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Mon Jul 3 14:42:14 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 03 Jul 2006 11:42:14 -0700 Subject: [Numpy-discussion] Does zero based indexing drive anyone else crazy? In-Reply-To: References: Message-ID: <44A96506.6010904@noaa.gov> Webb Sprague wrote: > it is far more natural to program if the indices are aligned with the > counts of the elements I suggest that it's only more natural if that's what you're used to -- i.e. you come form other languages that do it that way. I fairly rarely get bitten by indexing 1, rather than zero, but I save a lot of errors that I used to get in MATLAB by the way python does slices: len(a[i:j]) == j - i and: l[:j] + l[j:] == l or: r_[a[:i],a[i:]] == a for numpy arrays. I suppose you could have one-indexing and the python slicing, but I think that would be even more error prone. > zero > based indexing made lots of sense since it dovetailed with thinking in > memory offsets in systems programming it also dovetails nicely into using an array to represent a grid of values: i = (X - MinX) / deltaX rather than i = (X - MinX) / deltaX + 1 X = i*deltaX rather than X = (i-1)*deltaX In Fortran, you can choose where you want your array indexing to start, and I found myself starting with zero more often than 1, and I was never a C programmer. > I guess I just want to vent, but also to ask if anyone > has found any way to deal with this issue in their own scientific > programming. You'll get used to it. There are disadvantages either way, but after switching from primarily Matlab to primarily Python, I like zero-based indexing better. Perhaps you will too. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From nikhil.padmanabhan at gmail.com Mon Jul 3 14:47:55 2006 From: nikhil.padmanabhan at gmail.com (Nikhil Padmanabhan) Date: Mon, 3 Jul 2006 14:47:55 -0400 Subject: [Numpy-discussion] dtype conversion errors in cumsum Message-ID: <5CEABBC2-4D17-450D-8088-D50EA6504F88@gmail.com> Hi, (I saw some discussion of this on the mailing list, but was not sure if there was a solution reached...) If I try to use cumsum on a numpy array with the dtype keyword, I get the wrong answer if dtype does not match that of the array. More specifically, Python 2.4.3 (#1, Apr 3 2006, 18:07:18) [GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin Type "help", "copyright", "credits" or "license" for more information. ->> import numpy ->> numpy.__version__ '0.9.9.2727' ->> a = numpy.arange(10002) ->> a.cumsum(dtype='f4')[-1] 0.0 ->> a.cumsum(dtype='f8')[-1] 0.0 ->> a.cumsum()[-1] 50015001 ->> a.sum() 50015001 ->> a.sum(dtype='f8') 50015001.0 Note that this only seems to affect cumsum(); sum() works just fine. And cumsum works fine for small arrays.... Am I misunderstanding what the dtype keyword in cumsum is, or doing something silly? I should not be overflowing any type bounds (I think...) This is with the SVN numpy, on an iBook G4 running OS X 10.4.6 Thanks in advance, -- Nikhil ------------------------------------ Nikhil Padmanabhan nikhil.padmanabhan at gmail.com From svetosch at gmx.net Mon Jul 3 15:01:24 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Mon, 03 Jul 2006 21:01:24 +0200 Subject: [Numpy-discussion] None as missing value In-Reply-To: <200607031240.05066.pgmdevlist@mailcan.com> References: <44A8B141.70506@ieee.org> <44A8E927.9000405@gmx.net> <200607031240.05066.pgmdevlist@mailcan.com> Message-ID: <44A96984.2020303@gmx.net> Pierre GM schrieb: >> I was also a bit surprised at the following behavior: >>>> a = numpy.asarray([1,1]) >>>> a > array([1, 1]) >>>> a[0]=numpy.nan >>>> a > array([0, 1]) > > Seems to affect only the int_ arrays: > >>>> a = numpy.asarray([1,1], dtype=float_) >>>> a > array([1., 1.]) >>>> a[0]=numpy.nan >>>> a > array([ nan, 1. ]) > Sure it works with floats. The question is, should there maybe be an error if the type of the assigned value doesn't match the dtype of the array, instead of silently doing something unexpected? Consider the following (with still the same *integer* array a from my post above): >>> a[0]=0.1 >>> a array([0, 1]) >>> a[0]='0' Traceback (most recent call last): File "", line 1, in ? TypeError: an integer is required This TypeError makes a lot of sense, but why does it only react to strings, not to floats? Cheers, Sven From oliphant.travis at ieee.org Mon Jul 3 15:38:17 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 03 Jul 2006 13:38:17 -0600 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: <00b201c69ea0$7ab7a710$0a84a8c0@dsp.sun.ac.za> References: <00b201c69ea0$7ab7a710$0a84a8c0@dsp.sun.ac.za> Message-ID: <44A97229.1020605@ieee.org> Albert Strasheim wrote: > Stefan and I did some more experiments and it seems like .ctypes.strides > isn't doing the right thing for subarrays. > > For example: > > In [52]: x = N.rand(3,4) > In [57]: [x.ctypes.strides[i] for i in range(x.ndim)] > Out[57]: [32, 8] > > This looks fine. But for this subarray: > > In [56]: [x[1:3,1:4].ctypes.strides[i] for i in range(x.ndim)] > Out[56]: [32, 8] > > In this case, I think one wants strides[0] (the row stride) to return 40. > Why do you think that? All sliced arrays keep the same strides information as their "parents". This is the essence of a "view". The striding is exactly the same as before (the data hasn't moved anywhere), only the starting point and the bounds have changed. > .ctypes.data already seems to do the right thing: > > In [60]: x.ctypes.data > Out[60]: c_void_p(31685288) > > In [61]: x[1:3,1:4].ctypes.data > Out[61]: c_void_p(31685328) > > In [62]: 31685288-31685328 > Out[62]: 40 > > What would be a good way of dealing with discontiguous arrays? It seems like > one might want to disable their .ctypes attribute. > > No, not at all. Discontiguous arrays are easily handled simply by using the strides information to step through the array in each dimension instead of "assuming" contiguousness. Perhaps there is some confusion about what the strides actually represent. It's quite easy to write C-code that takes stride information as well which will then work with discontiguous arrays. The benefit of this approach is that you avoid copying data when you don't really have to. There should be very little performance penalty in most algorithms as well as the strides calculation is not much more than adding 1 to the current pointer. -Travis From svetosch at gmx.net Mon Jul 3 15:56:04 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Mon, 03 Jul 2006 21:56:04 +0200 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: References: <44A8CC6A.40901@ieee.org> <44A8F4FA.2060200@gmx.net> Message-ID: <44A97654.1030800@gmx.net> Sasha schrieb: > > Consistency is already lost because 1d case allows both ones(5) and > ones([5]) (and even ones((5,)) if anyone can tolerate that > abomination). I don't think those who argue for sequence only are > willing to require ones([5]). Imho consistency is not lost there precisely because one can use ones((5)) if one is so inclined. So the double-parentheses habit (or call it tuple-habit if you like) goes through. In contrast, rand((5,5)) fails, and that is what breaks consistency. > > Remember, "A Foolish Consistency is the Hobgoblin of Little Minds" > (Ralph Waldo Emerson (1803?1882), adopted without attribution as a > section heading in PEP 8 ). A "little mind" is an accurate description of myself in the numpy field. But I believe that in order to become a success, numpy must take into account the needs of little minds as well. > > I think the current situation strikes the right balance between > convenience and consistency. I was arguing that it's inconvenient exactly because it's inconsistent, so I don't see the tradeoff here. (Given that I'm fairly indifferent between one or two pairs of parentheses.) In any case I promise to shut up about this when 1.0(beta) is out, but I think Alan is right that under the status quo there will be a constant stream of the same newbie question that I asked. cheers, Sven From ndarray at mac.com Mon Jul 3 16:23:37 2006 From: ndarray at mac.com (Sasha) Date: Mon, 3 Jul 2006 16:23:37 -0400 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: <44A97654.1030800@gmx.net> References: <44A8CC6A.40901@ieee.org> <44A8F4FA.2060200@gmx.net> <44A97654.1030800@gmx.net> Message-ID: On 7/3/06, Sven Schreiber wrote: > ... > I was arguing that it's inconvenient exactly because it's inconsistent, > so I don't see the tradeoff here. (Given that I'm fairly indifferent > between one or two pairs of parentheses.) > It is convenient to be able to write zeros(5) and rand(5,5) instead of consistent zeros([5]) and rand([5,5]). Note that inconsistency will not be fully cured by allowing rand([5,5]) because rand([5,5], float32) will still not work. Actually, I can see some value in a random numbers generator that conforms to the ones/zeros interface because that may promote use of random data in testing instead of ones/zeros. How about rands(shape, dtype=float, low=default_low(dtype), high=default_high(dtype)) ? Where default_{low,high}(floatN) should probably be {0,1} but for integer types some other default may make more sense. From npadmana at Princeton.EDU Mon Jul 3 16:35:40 2006 From: npadmana at Princeton.EDU (Nikhil Padmanabhan) Date: Mon, 3 Jul 2006 16:35:40 -0400 Subject: [Numpy-discussion] dtype error in cumsum Message-ID: <5B56AACF-C28E-406B-8F90-FD86F1253213@princeton.edu> (I sent the following from my gmail account, but that does not seem to have been accepted by sourceforge -- apologies in advance if it shows up twice) Hi, (I saw some discussion of this on the mailing list, but was not sure if there was a solution reached, so I thought I'd ask again....) If I try to use cumsum on a numpy array with the dtype keyword, I get the wrong answer if dtype does not match that of the array. More specifically, Python 2.4.3 (#1, Apr 3 2006, 18:07:18) [GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin Type "help", "copyright", "credits" or "license" for more information. ->> import numpy ->> numpy.__version__ '0.9.9.2727' ->> a = numpy.arange(10002) ->> a.cumsum(dtype='f4')[-1] 0.0 ->> a.cumsum(dtype='f8')[-1] 0.0 ->> a.cumsum()[-1] 50015001 ->> a.sum() 50015001 ->> a.sum(dtype='f8') 50015001.0 Note that this only seems to affect cumsum(); sum() works just fine. And cumsum works fine for small arrays.... Am I misunderstanding what the dtype keyword in cumsum is, or doing something silly? I should not be overflowing any type bounds (I think...) This is with the SVN numpy, on an iBook G4 running OS X 10.4.6 Thanks in advance, -- Nikhil ------------------------------------ Nikhil Padmanabhan npadmana at princeton.edu http://wwwphy.princeton.edu/~npadmana nikhil@(609) 258-4355 From ndarray at mac.com Mon Jul 3 16:40:53 2006 From: ndarray at mac.com (Sasha) Date: Mon, 3 Jul 2006 16:40:53 -0400 Subject: [Numpy-discussion] dtype error in cumsum In-Reply-To: <5B56AACF-C28E-406B-8F90-FD86F1253213@princeton.edu> References: <5B56AACF-C28E-406B-8F90-FD86F1253213@princeton.edu> Message-ID: It did get through all right. SF is a mess. In any case what you observe is clearly a bug. Looks like uninitialized memory somewhere. Please file a bug report at http://projects.scipy.org/scipy/numpy/newticket . On 7/3/06, Nikhil Padmanabhan wrote: > (I sent the following from my gmail account, but that does not seem > to have been accepted by sourceforge -- apologies in advance if it > shows up twice) > > Hi, > > (I saw some discussion of this on the mailing list, but was not sure > if there was a solution reached, so I thought I'd ask again....) If I > try to use cumsum on a numpy array with the dtype keyword, I get the > wrong answer if dtype does not match that of the array. More > specifically, > > Python 2.4.3 (#1, Apr 3 2006, 18:07:18) > [GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > ->> import numpy > ->> numpy.__version__ > '0.9.9.2727' > ->> a = numpy.arange(10002) > ->> a.cumsum(dtype='f4')[-1] > 0.0 > ->> a.cumsum(dtype='f8')[-1] > 0.0 > ->> a.cumsum()[-1] > 50015001 > ->> a.sum() > 50015001 > ->> a.sum(dtype='f8') > 50015001.0 > > Note that this only seems to affect cumsum(); sum() works just fine. > And cumsum works fine for small arrays.... > > Am I misunderstanding what the dtype keyword in cumsum is, or doing > something silly? I should not be overflowing any type bounds (I > think...) > > This is with the SVN numpy, on an iBook G4 running OS X 10.4.6 > > Thanks in advance, > -- Nikhil > > ------------------------------------ > Nikhil Padmanabhan > npadmana at princeton.edu > http://wwwphy.princeton.edu/~npadmana > nikhil@(609) 258-4355 > > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From torgil.svensson at gmail.com Mon Jul 3 17:39:04 2006 From: torgil.svensson at gmail.com (Torgil Svensson) Date: Mon, 3 Jul 2006 23:39:04 +0200 Subject: [Numpy-discussion] iterate along a ray: linear algebra? In-Reply-To: <51f97e530606300713w1c167cf3j10c36d24f87326cf@mail.gmail.com> References: <51f97e530606300713w1c167cf3j10c36d24f87326cf@mail.gmail.com> Message-ID: I've done something similar a few years ago (numarray,numeric). I started roughly at the middle and did 64 points from a reference point (xc,yc). This point together with a point at the edge of the image (xp,yp) also defined a reference angle (a0). (ysize,xsize) is the shape of the intensity image. I used the following code to calculate points of interest: na=64 xr,yr=xsize-xc,ysize-yc a0=arctan2(yp-yr,xp-xc) if a0<0: a0+=2*pi ac=arctan2([yr,yr,-yc,-yc],[xr,-xc,-xc,xr]) if numarray: ac[ac<0]+=2*pi else: ac=choose(ac<0,(ac,ac+2*pi)) a1,a2,a3,a4=ac rmaxfn={ 0: lambda a: a<=a1 and xr/cos(a-0.0*pi) or yr/cos(0.5*pi-a), 1: lambda a: a<=a2 and yr/cos(a-0.5*pi) or xc/cos(1.0*pi-a), 2: lambda a: a<=a3 and xc/cos(a-1.0*pi) or yc/cos(1.5*pi-a), 3: lambda a: a<=a4 and yc/cos(a-1.5*pi) or xr/cos(2.0*pi-a) } angles=arange(a0,a0+2*pi,2*pi/na) if numarray: angles[angles>=2*pi]-=2*pi else: angles=choose(angles>=2*pi,(angles,angles-2*pi)) nr=int(ceil(sqrt(max(yc,yr)**2+max(xc,xr)**2))) crmax=array([int(floor(rmaxfn[floor(a*2/pi)](a))) for a in angles]) cr=outerproduct(ones(na),arange(float(nr))) ca=outerproduct(angles,ones(nr)) x=cr*cos(ca)+xc y=cr*sin(ca)+yc After this I did cubic spline interpolation in the image with these points and did something useful. I don't know how relevant this is to you and it doesn't use the linear algebra package but it might give you some hint. If you find out a nifty way to do your rays please post on this thread. Sidenote -- Watch my explicit float argument to arange and even putting in pi there in one case. There's a discussion on this list that floats in arange are troublesome On 6/30/06, stephen emslie wrote: > I am in the process of implementing an image processing algorithm that > requires following rays extending outwards from a starting point and > calculating the intensity derivative at each point. The idea is to find the > point where the difference in intensity goes beyond a particular threshold. > > Specifically I'm examining an image of an eye to find the pupil, and the > edge of the pupil is a sharp change in intensity. > > How does one iterate along a line in a 2d matrix, and is there a better way > to do this? Is this a problem that linear algebra can help with? > > Thanks > Stephen Emslie > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From Fernando.Perez at colorado.edu Mon Jul 3 18:41:11 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon, 03 Jul 2006 16:41:11 -0600 Subject: [Numpy-discussion] A C++ library and the new array interface: the best approach? Message-ID: <44A99D07.5030706@colorado.edu> Hi all, I'd like to ask for a bit of guidance on the best path to follow regarding the integration of an existing C++ library with numpy, given the recent developments on these topics: 1. the new array interface specification 2. the ctypes discussions 3. pyrex Our situation is the following: an existing in-house C++ library defines a templated class (Tensor), in many ways similar to a numpy array, but with its own extra functionality for other things. There are many functions in this code which take or return Tensors. I now need to interface this library with a Python code which is fully numpy-based, and hopefully be able to go back and forth between the C++ and python sides without unnecessary copies. Since the C++ codebase is somewhat large and already exists, and will continue to evolve as a standalone C++ system, something tells me I should just bite the typemap bullet and SWIG the darn thing. But I've never written anything beyond the most trivial typemaps, and I'm not 100% sure on exactly how to take advantage of the new array interface with SWIG. I read all the docs under - http://svn.scipy.org/svn/PEP - http://www.scipy.org/BaseArray - http://numeric.scipy.org/array_interface.html - the pyrex/array interface wiki pages (the site is not responding right now, so I can't give a URL) but my lack of familiarity with all the details of new type creation got me a bit lost. I'm sure the information I need is all there, but right now I don't really see the forest with all the leaves in my face. I've also read the various recent threads on ctypes, as well was the one initiated by Joris: http://article.gmane.org/gmane.comp.python.numeric.general/6296 So I'd like to know if SWIG is really the best way out in this particular case (and any advice on taking advantage of the array interface via SWIG would be appreciated), or if ctypes or pyrex could be used here. I'm quite happy using pyrex in other contexts, but I know it doesn't directly support C++. However, since I have access to all the code, perhaps a pure C layer could be used to bridge the C++/pyrex gap. Or given the recent praise for ctypes, perhaps that can be an option? Any advice will be greatly appreciated. Best, f From charlesr.harris at gmail.com Mon Jul 3 18:51:17 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 3 Jul 2006 16:51:17 -0600 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: References: <44A8CC6A.40901@ieee.org> <44A8F4FA.2060200@gmx.net> <44A97654.1030800@gmx.net> Message-ID: +1 for tuples, Because dimensions *are* tuples. I think using tuples is a good habit to get into and agree with Robert Kern's comments about using the new random number interface instead of the old compatibility functions rand and randn. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Mon Jul 3 19:06:42 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 03 Jul 2006 17:06:42 -0600 Subject: [Numpy-discussion] A C++ library and the new array interface: the best approach? In-Reply-To: <44A99D07.5030706@colorado.edu> References: <44A99D07.5030706@colorado.edu> Message-ID: <44A9A302.90901@ieee.org> Fernando Perez wrote: > Hi all, > > but my lack of familiarity with all the details of new type creation got me a > bit lost. I'm sure the information I need is all there, but right now I don't > really see the forest with all the leaves in my face. I've also read the > various recent threads on ctypes, as well was the one initiated by Joris: > > http://article.gmane.org/gmane.comp.python.numeric.general/6296 > > > So I'd like to know if SWIG is really the best way out in this particular case > (and any advice on taking advantage of the array interface via SWIG would be > appreciated), or if ctypes or pyrex could be used here. I'm quite happy using > pyrex in other contexts, but I know it doesn't directly support C++. However, > since I have access to all the code, perhaps a pure C layer could be used to > bridge the C++/pyrex gap. Or given the recent praise for ctypes, perhaps that > can be an option? > I don't have a lot of time to respond right now. But here are some thoughts. I'm not sure I'd use SWIG at this point. I'd probably either a) use ctypes to call the C++ library (if it is compiled as a *shared* library) although to be honest I'm not sure if calling a C++ causes any grief. b) byte-the bullet and use Boost python (which other's who have integrated C++ code with Python seem to really like). I've never wrapped my mind around Boost python, though, so I'm not much help there. c) write a custom Tensor class that either inherits from the ndarray or is a mix-in class with the ndarray as a component. If it were me, I'd probably be doing a combination of a and c. I've just finished the chapter in my book that covers and compares f2py, weave, pyrex and ctypes integration. I've been waiting on finishing this chapter before sending updates. You should get the update by the end of the week. Perhaps more later. -Travis > Any advice will be greatly appreciated. > > Best, > > f > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From ltaylor at llnl.gov Mon Jul 3 19:13:39 2006 From: ltaylor at llnl.gov (Lee Taylor) Date: Mon, 3 Jul 2006 16:13:39 -0700 (PDT) Subject: [Numpy-discussion] Namespaces in header files [Was: Time for beta1 of NumPy 1.0] In-Reply-To: <44A47854.1050106@ieee.org> References: <44A47854.1050106@ieee.org> Message-ID: On Thu, 29 Jun 2006, Travis Oliphant wrote: > I think it's time for the first beta-release of NumPy 1.0 > > I'd like to put it out within 2 weeks. Please make any comments or > voice major concerns so that the 1.0 release series can be as stable as > possible. One issue I ran across that I have not seen addressed is the namespace of arrayobject.h. I'm not refering to C++ namespaces but prefixing symbols to avoid clashes with user's code. The externals start with PyArray. But I had symbol redefinition errors for byte, MAX_DIMS, and ERR. That is, I already had defines for MAX_DIMS and ERR and a typedef for byte in my code. When adding a numpy interface to my library I had to undef these symbols before including arrayobject.h. Is there a way to move implemention defines, like ERR, into a separate header. Or if they're part of the API, prefix the symbols? Lee Taylor From strawman at astraw.com Mon Jul 3 20:22:32 2006 From: strawman at astraw.com (Andrew Straw) Date: Mon, 03 Jul 2006 17:22:32 -0700 Subject: [Numpy-discussion] A C++ library and the new array interface: the best approach? In-Reply-To: <44A99D07.5030706@colorado.edu> References: <44A99D07.5030706@colorado.edu> Message-ID: <44A9B4C8.10907@astraw.com> Dear Fernando, A couple of quick thoughts: A) Despite my love of Pyrex, it probably isn't the tool for the job -- it doesn't play particularly well with C++, and it would require you to write custom code for every function wrapped. B) It sounds like you want something that will more-or-less automatically generate the python interface from the header files. To my mind, this suggests SWIG or boost. Boost uses some seriously fancy (and mind-bending) C++, which can either be a good thing or a bad thing, but it certainly boosts compile time! There are doubtless other ways. For example, perhaps you could try using gccxml to auto-generate something like a Python/ctypes wrapper. C) The core of whatever you do will probably involve writing code to view a Tensor as an object that has the __array_struct__ interface. That way numpy/scipy/whatever can use the data with no copying. So that's a good place to start and whatever you learn there will probably be useful regardless of whatever tools you end up with. You'll probably also want the converse function so that you can view objects exposing __array_struct__ (numpy arrays) as a Tensor. Anyhow, that's not much, but perhaps it'll help. Cheers! Andrew Fernando Perez wrote: >Hi all, > >I'd like to ask for a bit of guidance on the best path to follow regarding the >integration of an existing C++ library with numpy, given the recent >developments on these topics: > >1. the new array interface specification >2. the ctypes discussions >3. pyrex > >Our situation is the following: an existing in-house C++ library defines a >templated class (Tensor), in many ways similar to a numpy array, but with its >own extra functionality for other things. There are many functions in this >code which take or return Tensors. I now need to interface this library with >a Python code which is fully numpy-based, and hopefully be able to go back and >forth between the C++ and python sides without unnecessary copies. > >Since the C++ codebase is somewhat large and already exists, and will continue >to evolve as a standalone C++ system, something tells me I should just bite >the typemap bullet and SWIG the darn thing. But I've never written anything >beyond the most trivial typemaps, and I'm not 100% sure on exactly how to take >advantage of the new array interface with SWIG. I read all the docs under > > - http://svn.scipy.org/svn/PEP > > - http://www.scipy.org/BaseArray > > - http://numeric.scipy.org/array_interface.html > > - the pyrex/array interface wiki pages (the site is not responding right >now, so I can't give a URL) > > >but my lack of familiarity with all the details of new type creation got me a >bit lost. I'm sure the information I need is all there, but right now I don't >really see the forest with all the leaves in my face. I've also read the >various recent threads on ctypes, as well was the one initiated by Joris: > >http://article.gmane.org/gmane.comp.python.numeric.general/6296 > > >So I'd like to know if SWIG is really the best way out in this particular case >(and any advice on taking advantage of the array interface via SWIG would be >appreciated), or if ctypes or pyrex could be used here. I'm quite happy using >pyrex in other contexts, but I know it doesn't directly support C++. However, >since I have access to all the code, perhaps a pure C layer could be used to >bridge the C++/pyrex gap. Or given the recent praise for ctypes, perhaps that >can be an option? > >Any advice will be greatly appreciated. > >Best, > >f > >Using Tomcat but need to do more? Need to support web services, security? >Get stuff done quickly with pre-integrated technology to make your job easier >Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo >http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From kwgoodman at gmail.com Mon Jul 3 21:53:33 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 3 Jul 2006 18:53:33 -0700 Subject: [Numpy-discussion] Does zero based indexing drive anyone else crazy? In-Reply-To: References: Message-ID: On 7/2/06, Webb Sprague wrote: > Given the long history of python and its ancestry in C (for which zero > based indexing made lots of sense since it dovetailed with thinking in > memory offsets in systems programming), there is probably nothing to > be done now. I guess I just want to vent, but also to ask if anyone > has found any way to deal with this issue in their own scientific > programming. Aha! Guido himself prefers starting the index at one. Here's a code snippet from a fun article he wrote about optimizing python code: import time def timing(f, n, a): print f.__name__, r = range(n) t1 = time.clock() for i in r: f(a); f(a); f(a); f(a); f(a); f(a); f(a); f(a); f(a); f(a) t2 = time.clock() print round(t2-t1, 3) http://www.python.org/doc/essays/list2str/ Notice he chose t1 and t2 instead of t0 and t1. QED From egilber2 at cs.uiuc.edu Mon Jul 3 21:58:27 2006 From: egilber2 at cs.uiuc.edu (Eric Gilbert) Date: Mon, 03 Jul 2006 20:58:27 -0500 Subject: [Numpy-discussion] Research study on CVS - you can help and get paid! Message-ID: I am a Computer Science graduate student at the University of Illinois at Urbana-Champaign. I need your help in a study I want to conduct. I have created a project called LifeSource that visualizes CVS repositories. You can find screenshots and info here: http://social.cs.uiuc.edu/people/gilbert (info on me) http://social.cs.uiuc.edu/people/gilbert/research.html (info on LifeSource, the CVS project) LifeSource appeared as a Work-in-Progress paper at ACM's CHI 2006 in Montreal (http://www.chi2006.org). I applied LifeSource to the Azureus and Gaim CVS repositories. I would like to apply the same visualization to your project. Next, I want to see if you as the visualized community find it accurate, helpful, disturbing, etc. In order to participate in the study, you will need to view the LifeSource visualization once a week (on the web) for four weeks and respond to two short online surveys. You will receive a $20 gift certificate for your participation. I can fully explain the process in the near future - just email me back. I will provide all of the information of the methodology of the study online. (We can set up a blog for study members to leave further comments.) Please let me know what you think. I plan on using the study in a long paper to CHI 2007. Cheers, Eric P.S. I apologize for the spam and will send no more. From wbaxter at gmail.com Mon Jul 3 23:39:28 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 4 Jul 2006 12:39:28 +0900 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: References: <44A8CC6A.40901@ieee.org> <44A8F4FA.2060200@gmx.net> <44A97654.1030800@gmx.net> Message-ID: I just noticed that array.transpose() takes either a tuple or separate arguments. I'm not trying to imply anything, just I was thinking such a thing had no precedent in numpy, but it turns out it does. And as an aside, the docstring for .transpose() needs fixing. It says: >>> a = array([[1,2],[3,4]]) >>> a array([[1, 2], [3, 4]]) >>> a.transpose() array([[1, 3], [3, 4]]) Whoa -- transpose destroys any 2's in your input and turns them into 3's! How does one use this Trac thing again? I couldn't find a "submit a bug" link anywhere on the web scipy.org web page. I found it once before, but now I can't seem to locate it. --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Jul 3 23:45:17 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 03 Jul 2006 22:45:17 -0500 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: References: <44A8CC6A.40901@ieee.org> <44A8F4FA.2060200@gmx.net> <44A97654.1030800@gmx.net> Message-ID: Bill Baxter wrote: > How does one use this Trac thing again? I couldn't find a "submit a > bug" link anywhere on the web scipy.org web page. I > found it once before, but now I can't seem to locate it. http://projects.scipy.org/scipy/scipy You need to register first. Click the "Register" link in the upper right. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Mon Jul 3 23:54:19 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 03 Jul 2006 22:54:19 -0500 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: References: <44A8CC6A.40901@ieee.org> <44A8F4FA.2060200@gmx.net> <44A97654.1030800@gmx.net> Message-ID: Robert Kern wrote: > Bill Baxter wrote: >> How does one use this Trac thing again? I couldn't find a "submit a >> bug" link anywhere on the web scipy.org web page. I >> found it once before, but now I can't seem to locate it. > > http://projects.scipy.org/scipy/scipy > > You need to register first. Click the "Register" link in the upper right. Or rather, http://projects.scipy.org/scipy/numpy -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From michael.sorich at gmail.com Tue Jul 4 00:44:56 2006 From: michael.sorich at gmail.com (Michael Sorich) Date: Tue, 4 Jul 2006 14:14:56 +0930 Subject: [Numpy-discussion] MA bug or feature? In-Reply-To: References: <16761e100606210146q7683c94bu5bd2699caa6b95cf@mail.gmail.com> <200606210612.09374.pgmdevlist@mailcan.com> Message-ID: <16761e100607032144w65013936jbbf9772f429f4be9@mail.gmail.com> On 6/23/06, Paul Dubois wrote: > In the original MA in Numeric, I decided that to constantly check for masks > that didn't actually mask anything was not a good idea. It punishes normal > use with a very expensive check that is rarely going to be true. > > If you are in a setting where you do not want this behavior, but instead > want masks removed whenever possible, you may wish to wrap or replace things > like masked_array so that they call make_mask with flag = 1: > > y = masked_array(data, make_mask(maskdata, flag=1)) > > y will have no mask if maskdata is all false. > Hi Paul. If this is purely for optimisation, is there perhaps a better way to do so in which the optimisation is hidden? For example if the optimisation is in regard to the case in which none of the elements is masked, an alternative approach may be to make a subclass of ndarray and cache the result of the any method. e.g. import numpy as N def asmask(data): if isinstance(data, Mask): return data else: return Mask(data) class Mask(N.ndarray): __array_priority__ = 10.0 def __new__(subtype, data): ret = N.array(data, N.bool_) return ret.view(Mask) def __init__(self, data): self._any = None def any(self): if self._any is None: self._any = N.ndarray.any(self) return self._any def __setitem__(self, index, value): self._any = None N.ndarray.__setitem__(self, index, value) The biggest problem I have with the current setup is the inconsistency between the behaviour of the array when the mask is nomask vs a boolarray with all False. Another example of this is when one changes the mask on an element. This is not possible when the mask is nomask print N.version.version ma1 = N.ma.array([1,2,3], mask=[False, False, False]) print ma1.mask ma1.mask[2] = True ma2 = N.ma.array([1,2,3], mask=False) print ma2.mask ma2.mask[2] = True ----- output 0.9.8 [False False False] [False False True] False Traceback (most recent call last): File "D:\eclipse\Mask\src\mask\__init__.py", line 111, in ? ma2.mask[2] = True TypeError: object does not support item assignment From pgmdevlist at mailcan.com Tue Jul 4 02:39:59 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Tue, 4 Jul 2006 02:39:59 -0400 Subject: [Numpy-discussion] MA bug or feature? In-Reply-To: <16761e100607032144w65013936jbbf9772f429f4be9@mail.gmail.com> References: <16761e100606210146q7683c94bu5bd2699caa6b95cf@mail.gmail.com> <16761e100607032144w65013936jbbf9772f429f4be9@mail.gmail.com> Message-ID: <200607040240.00075.pgmdevlist@mailcan.com> Michael, I wonder whether the Mask class you suggest is not a bit overkill. There should be enough tools in the existing MA module to do what we want. And I don't wanna think about compatibility the number of changes in the MA code that'd be required (but I'm lazy)... For the sake of consistency and optimization, I still think it could be easier (and cleaner) to make `nomask` the default for a MaskedArray without masked values. That could for example be implemented by forcing `nomask` at the creation of the MaskedArray with an extra `if mask and not mask.any(): mask=nomask`, or by using Paul's make_mask( flag=1) trick. Masking some specific values could still be done when mask is nomask with an intermediary MA.getmaskarray() step. On a side note, modifying an existing mask is a delicate matter. Everything's OK if you use masks as a way to hide existing data, it's more complex when initially you have some holes in your dataset... From fullung at gmail.com Tue Jul 4 03:57:10 2006 From: fullung at gmail.com (Albert Strasheim) Date: Tue, 4 Jul 2006 09:57:10 +0200 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: <44A97229.1020605@ieee.org> Message-ID: <00f301c69f3f$740616f0$0a84a8c0@dsp.sun.ac.za> Hello all > > In this case, I think one wants strides[0] (the row stride) to return > 40. > > > > Why do you think that? > > All sliced arrays keep the same strides information as their > "parents". This is the essence of a "view". The striding is exactly > the same as before (the data hasn't moved anywhere), only the starting > point and the bounds have changed. Sorry, I was suffering some temporary strides brain damage. :-) Regards, Albert From fullung at gmail.com Tue Jul 4 04:44:56 2006 From: fullung at gmail.com (Albert Strasheim) Date: Tue, 4 Jul 2006 10:44:56 +0200 Subject: [Numpy-discussion] A C++ library and the new array interface: the best approach? In-Reply-To: <44A9A302.90901@ieee.org> References: <44A99D07.5030706@colorado.edu> <44A9A302.90901@ieee.org> Message-ID: <20060704084456.GB4754@dogbert.sdsl.sun.ac.za> Hello all On Mon, 03 Jul 2006, Travis Oliphant wrote: > Fernando Perez wrote: > > Hi all, > > > > but my lack of familiarity with all the details of new type creation got me a > > bit lost. I'm sure the information I need is all there, but right now I don't > > really see the forest with all the leaves in my face. I've also read the > > various recent threads on ctypes, as well was the one initiated by Joris: > > > > http://article.gmane.org/gmane.comp.python.numeric.general/6296 > > > > So I'd like to know if SWIG is really the best way out in this particular case > > (and any advice on taking advantage of the array interface via SWIG would be > > appreciated), or if ctypes or pyrex could be used here. I'm quite happy using > > pyrex in other contexts, but I know it doesn't directly support C++. However, > > since I have access to all the code, perhaps a pure C layer could be used to > > bridge the C++/pyrex gap. Or given the recent praise for ctypes, perhaps that > > can be an option? > > I don't have a lot of time to respond right now. But here are some > thoughts. > > I'm not sure I'd use SWIG at this point. At least for C code, I would definately agree. If you already know SWIG, you might want to try it, but I started without any knowledge of SWIG and ctypes when I wanted to wrap a pretty basic library, and after a week of puzzling over SWIG, I learned ctypes and wrapped the library in a matter of hours. > I'd probably either > > a) use ctypes to call the C++ library (if it is compiled as a *shared* > library) although to be honest I'm not sure if calling a C++ causes any > grief. If I understand correctly, because each C++ compiler potentially mangles C++ identifier names in a different way, there isn't a way for ctypes to know what to look for in a shared library. > b) byte-the bullet and use Boost python (which other's who have > integrated C++ code with Python seem to really like). I've never > wrapped my mind around Boost python, though, so I'm not much help there. If there is any way for you to "hide" you C++ code behind an extern "C" interface, i.e. write some C-style wrappers on top of your C++ code, then you can use ctypes. As luck would have it, this is exactly what the libsvm authors did already, which made wrapping libsvm with ctypes relatively straightforward. If you want to achieve things like Python classes deriving from C++ classes and vice versa, Boost-Python is what you want to look at. But as others have noted, prepare to wait for the compiler when wrapping anything of even moderate size. In Boost-Python world there's Pyste that allows you to generate code. I think Pyste uses GCC-XML underneath. I played with Pyste for a day or two long ago, but at that stage the code it made differed greatly from what I wrote when doing Boost-Python wrapping by hand, so it was a bit strange. In general, I got the feeling that Pyste wasn't flexible enough to wrap any kind of C++ library. But maybe you're lucky and your C++ code is just right to wrap easily with Pyste. > c) write a custom Tensor class that either inherits from the ndarray or > is a mix-in class with the ndarray as a component. Don't know much about this option. In summary (keeping in mind that I'm a bit of a ctypes zealot at present), I'd try the "C interface on top of C++ with ctypes" approach, and build an ndarray subclass on top these C functions. Regards, Albert From pau.gargallo at gmail.com Tue Jul 4 05:25:32 2006 From: pau.gargallo at gmail.com (Pau Gargallo) Date: Tue, 4 Jul 2006 11:25:32 +0200 Subject: [Numpy-discussion] tentative numpy tutorial Message-ID: <6ef8f3380607040225t30948964o49b0f570b8768edf@mail.gmail.com> hi all, motivated by the lack of free documentation for NumPy, with some friends, we started writing a tutorial, that we would like to see in scipy.org. After some time, the project have started to loose its initial impetus. Now, we put the current unfinished version in http://www.scipy.org/Tentative_NumPy_Tutorial with the hope that, with the help of other users, it will be possible to build a useful document for beginners. We are not Numpy experts nor native English speakers (as you probably realized ;-), so the current version needs to be reviewed. Moreover, there are huge holes marked with (TODO) tags waiting for motivated people to fill them. In my opinion, the lack of free documentation is preventing many users to get started with NumPy: - Many NumPy features are only documented in the Guide to NumPy. - (In my opinion) New users are not going to buy the Guide before knowing about this features. So there is a kind of dead lock here. It would be nice, if NumPy 1.0 could be released with a complete free tutorial, don't you think so? then click the edit button! thanks, pau From jnc at ecs.soton.ac.uk Tue Jul 4 05:18:53 2006 From: jnc at ecs.soton.ac.uk (John Carter) Date: Tue, 04 Jul 2006 10:18:53 +0100 Subject: [Numpy-discussion] Error building numpy from svn, 2nd attempt Message-ID: <7.0.1.0.0.20060704100002.01f8fc38@ecs.soton.ac.uk> Hi, As is the way posting to a news group stirs the brain cell into activity and the problem is solved. or rather shifted. I've downloaded the candidate version of mingw32 and using that to build numpy/scipy works, or rather it builds the extensions for Python 2.3 I believe that there are still problems as both numpy and scipy fail their tests. I'm also dubious as to whether they are finding Altas and Lapack. Configuration: Windows XP SP2 Cygwin = gcc version 3.4.4 (cygming special) (gdc 0.12, using dmd 0.125) MinGW = gcc version 3.4.5 (mingw special) Python = Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32 scipy, numpy, Atlas, Lapack all downloaded within the last 2 days I've cut and pasted the error messages from the tests below. Any help gratefully received. John ===== numpy ===== D:.\numerical> python Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.test() Found 5 tests for numpy.distutils.misc_util Found 3 tests for numpy.lib.getlimits Found 30 tests for numpy.core.numerictypes Found 13 tests for numpy.core.umath Found 3 tests for numpy.core.scalarmath Found 8 tests for numpy.lib.arraysetops Found 42 tests for numpy.lib.type_check Found 101 tests for numpy.core.multiarray Found 36 tests for numpy.core.ma Found 10 tests for numpy.lib.twodim_base Found 10 tests for numpy.core.defmatrix Found 1 tests for numpy.lib.ufunclike Found 38 tests for numpy.lib.function_base Found 3 tests for numpy.dft.helper Found 1 tests for numpy.lib.polynomial Found 7 tests for numpy.core.records Found 26 tests for numpy.core.numeric Found 4 tests for numpy.lib.index_tricks Found 46 tests for numpy.lib.shape_base Found 0 tests for __main__ ........................................................................................................ ........................................................................................................ .........E.............................................................................................. ........................... ====================================================================== ERROR: check_simple (numpy.lib.tests.test_twodim_base.test_histogram2d) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python23\Lib\site-packages\numpy\lib\tests\test_twodim_base.py", line 137, in check_simple np.random.seed(1) File "mtrand.pyx", line 311, in mtrand.RandomState.seed SystemError: C:\sf\python\dist\src-maint23\Objects\longobject.c:240: bad argument to internal function ---------------------------------------------------------------------- Ran 387 tests in 1.391s FAILED (errors=1) >>> ===== scipy ===== D:.\numerical> python Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import scipy >>> scipy.test() Overwriting lib= from C:\PYTHON23\lib\si te-packages\scipy\lib\__init__.pyc (was from C:\PYTHON23\lib\site-packages\numpy\lib\__init__.pyc) Overwriting fft= from scipy.fftpack.basic (was from numpy.dft.f ftpack) Overwriting ifft= from scipy.fftpack.basic (was from numpy.df t.fftpack) Found 4 tests for scipy.io.array_import Found 128 tests for scipy.linalg.fblas Found 397 tests for scipy.ndimage Found 10 tests for scipy.integrate.quadpack Found 97 tests for scipy.stats.stats Found 47 tests for scipy.linalg.decomp Found 2 tests for scipy.integrate.quadrature Found 95 tests for scipy.sparse.sparse Found 20 tests for scipy.fftpack.pseudo_diffs Found 6 tests for scipy.optimize.optimize Found 5 tests for scipy.interpolate.fitpack Found 1 tests for scipy.interpolate Found 12 tests for scipy.io.mmio Found 10 tests for scipy.stats.morestats Found 4 tests for scipy.linalg.lapack Found 18 tests for scipy.fftpack.basic Found 4 tests for scipy.linsolve.umfpack Found 4 tests for scipy.optimize.zeros Found 41 tests for scipy.linalg.basic Found 2 tests for scipy.maxentropy.maxentropy Found 358 tests for scipy.special.basic Found 128 tests for scipy.lib.blas.fblas Found 7 tests for scipy.linalg.matfuncs **************************************************************** WARNING: clapack module is empty ----------- See scipy/INSTALL.txt for troubleshooting. Notes: * If atlas library is not found by numpy/distutils/system_info.py, then scipy uses flapack instead of clapack. **************************************************************** Found 42 tests for scipy.lib.lapack Found 1 tests for scipy.optimize.cobyla Found 16 tests for scipy.lib.blas Found 1 tests for scipy.integrate Found 14 tests for scipy.linalg.blas Found 70 tests for scipy.stats.distributions Found 4 tests for scipy.fftpack.helper Found 4 tests for scipy.signal.signaltools Found 0 tests for __main__ Don't worry about a warning regarding the number of bytes read. Warning: 1000000 bytes requested, 20 bytes read. .......caxpy:n=4 ..caxpy:n=3 ....ccopy:n=4 ..ccopy:n=3 .............cscal:n=4 ....cswap:n=4 ..cswap:n=3 .....daxpy:n=4 ..daxpy:n=3 ....dcopy:n=4 ..dcopy:n=3 .............dscal:n=4 ....dswap:n=4 ..dswap:n=3 .....saxpy:n=4 ..saxpy:n=3 ....scopy:n=4 ..scopy:n=3 .............sscal:n=4 ....sswap:n=4 ..sswap:n=3 .....zaxpy:n=4 ..zaxpy:n=3 ....zcopy:n=4 ..zcopy:n=3 .............zscal:n=4 ....zswap:n=4 ..zswap:n=3 ........................................................................................................................ ........................................................................................................................ ........................................................................................................................ ........................................................................................................................ .........................................................................Took 13 points. ..........Resizing... 16 17 24 Resizing... 20 7 35 Resizing... 23 7 47 Resizing... 24 25 58 Resizing... 28 7 68 Resizing... 28 27 73 .....Use minimum degree ordering on A'+A. ........................Use minimum degree ordering on A'+A. ...................Resizing... 16 17 24 Resizing... 20 7 35 Resizing... 23 7 47 Resizing... 24 25 58 Resizing... 28 7 68 Resizing... 28 27 73 .....Use minimum degree ordering on A'+A. .................Resizing... 16 17 24 Resizing... 20 7 35 Resizing... 23 7 47 Resizing... 24 25 58 Resizing... 28 7 68 Resizing... 28 27 73 .....Use minimum degree ordering on A'+A. ......................................C:\PYTHON23\lib\site-packages\scipy\interpolate\fitpack2.py:410: UserWarning: The coefficients of the spline returned have been computed as the minimal norm least-squares solution of a (numerically) rank deficient system (deficiency=7). If deficiency is large, the results may be inaccurate. Deficiency may strongly depend on the value of eps. warnings.warn(message) ....................Ties preclude use of exact statistic. ..Ties preclude use of exact statistic. ........ **************************************************************** WARNING: clapack module is empty ----------- See scipy/INSTALL.txt for troubleshooting. Notes: * If atlas library is not found by numpy/distutils/system_info.py, then scipy uses flapack instead of clapack. **************************************************************** ....................data-ftype: z compared to data D Calling _superlu.zgssv Use minimum degree ordering on A'+A. .data-ftype: c compared to data F Calling _superlu.cgssv Use minimum degree ordering on A'+A. .data-ftype: d compared to data d Calling _superlu.dgssv Use minimum degree ordering on A'+A. .data-ftype: s compared to data f Calling _superlu.sgssv Use minimum degree ordering on A'+A. ........................................................................................................................ ........................................................................................................................ ....................................Gegenbauer, a = 2.7337413228 ........................................................................................................................ .............caxpy:n=4 ..caxpy:n=3 ....ccopy:n=4 ..ccopy:n=3 .............cscal:n=4 ....cswap:n=4 ..cswap:n=3 .....daxpy:n=4 ..daxpy:n=3 ....dcopy:n=4 ..dcopy:n=3 .............dscal:n=4 ....dswap:n=4 ..dswap:n=3 .....saxpy:n=4 ..saxpy:n=3 ....scopy:n=4 ..scopy:n=3 .............sscal:n=4 ....sswap:n=4 ..sswap:n=3 .....zaxpy:n=4 ..zaxpy:n=3 ....zcopy:n=4 ..zcopy:n=3 .............zscal:n=4 ....zswap:n=4 ..zswap:n=3 ...Result may be inaccurate, approximate err = 4.1928136851e-009 ...Result may be inaccurate, approximate err = 7.27595761418e-012 .............................................F................Residual: 1.05006950319e-007 . **************************************************************** WARNING: cblas module is empty ----------- See scipy/INSTALL.txt for troubleshooting. Notes: * If atlas library is not found by numpy/distutils/system_info.py, then scipy uses fblas instead of cblas. **************************************************************** ............................................................................................ ====================================================================== FAIL: check_simple (scipy.optimize.tests.test_cobyla.test_cobyla) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python23\Lib\site-packages\scipy\optimize\tests\test_cobyla.py", line 20, in check_simple assert_almost_equal(x, [x0, x1], decimal=5) File "C:\Python23\Lib\site-packages\numpy\testing\utils.py", line 152, in assert_almost_equal return assert_array_almost_equal(actual, desired, decimal, err_msg) File "C:\Python23\Lib\site-packages\numpy\testing\utils.py", line 222, in assert_array_almost_equal header='Arrays are not almost equal') File "C:\Python23\Lib\site-packages\numpy\testing\utils.py", line 207, in assert_array_compare assert cond, msg AssertionError: Arrays are not almost equal (mismatch 100.0%) x: array([ 4.957975 , 0.64690335]) y: array([ 4.95535625, 0.66666667]) ---------------------------------------------------------------------- Ran 1552 tests in 5.828s FAILED (failures=1) >>> Dr. John N. Carter. E-Mail : jnc at ecs.soton.ac.uk Building 1, Room 2005 http://www.ecs.soton.ac.uk/~jnc/ Information: Signals, Images, Systems Phone : +44 (0) 23 8059 2405 School of Electronics & Computer Science, Fax : +44 (0) 23 8059 4498 Southampton University, Hants, UK SO17 1BJ. From cimrman3 at ntc.zcu.cz Tue Jul 4 05:51:20 2006 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Tue, 04 Jul 2006 11:51:20 +0200 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <91cf711d0607031000l286da52fn30d98798057f12c5@mail.gmail.com> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> <44A8E4D3.5000603@ntc.zcu.cz> <91cf711d0607031000l286da52fn30d98798057f12c5@mail.gmail.com> Message-ID: <44AA3A18.9060300@ntc.zcu.cz> David Huard wrote: > Here is a quick benchmark between numpy's unique, unique1d and sasha's > unique: > > x = rand(100000)*100 > x = x.astype('i') > > %timeit unique(x) > 10 loops, best of 3: 525 ms per loop > > %timeit unique_sasha(x) > 100 loops, best of 3: 10.7 ms per loop > > timeit unique1d(x) > 100 loops, best of 3: 12.6 ms per loop > > So I wonder what is the added value of unique? > Could unique1d simply become unique ? It looks like unique1d and friends could use same facelifting with new numpy features like boolean indexing :) r. From theller at python.net Tue Jul 4 05:56:45 2006 From: theller at python.net (Thomas Heller) Date: Tue, 04 Jul 2006 11:56:45 +0200 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: <44A73B5D.60002@ieee.org> References: <44A73B5D.60002@ieee.org> Message-ID: Travis Oliphant schrieb: > I've been playing a bit with ctypes and realized that with a little > help, it could be made much easier to interface with NumPy arrays. > Thus, I added a ctypes attribute to the NumPy array. If ctypes is > installed, this attribute returns a "conversion" object otherwise an > AttributeError is raised. > > The ctypes-conversion object has attributes which return c_types aware > objects so that the information can be passed directly to c-code (as an > integer, the number of dimensions can already be passed using c-types). > > The information available and it's corresponding c_type is > > data - c_void_p > shape, strides - c_int * nd or c_long * nd or c_longlong * nd > depending on platform I've also played a little, and I think one important limitation in ctypes is that items in the argtypes list have to be ctypes types. If that limitation is removed (see the attached trivial patch) one can write a class that implements 'from_param' and accepts ctypes arrays as well as numpy arrays as argument in function calls (Maybe the _as_parameter_ stuff needs cleanup as well). The attached shape.py script implements this class, and has two examples. The 'from_param' method checks the correct shape and itemtype of the arrays that are passed as parameter. Thomas -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: shape.py URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: from_param.patch URL: From simon.anders at uibk.ac.at Tue Jul 4 06:03:07 2006 From: simon.anders at uibk.ac.at (Simon Anders) Date: Tue, 04 Jul 2006 12:03:07 +0200 Subject: [Numpy-discussion] A C++ library and the new array interface: the best approach? In-Reply-To: <44A99D07.5030706@colorado.edu> References: <44A99D07.5030706@colorado.edu> Message-ID: <44AA3CDB.8060502@uibk.ac.at> Hi Fernando, Fernando Perez schrieb: [...] > So I'd like to know if SWIG is really the best way out in this particular case > (and any advice on taking advantage of the array interface via SWIG would be > appreciated), or if ctypes or pyrex could be used here. I'm quite happy using > pyrex in other contexts, but I know it doesn't directly support C++. However, > since I have access to all the code, perhaps a pure C layer could be used to > bridge the C++/pyrex gap. Or given the recent praise for ctypes, perhaps that > can be an option? I'm not so sure either, whether SWIG is the way to go. If, however, you decide to give it a try, you could try a little tool I am currently working on. It is basically a SWIG typemap definition which allows for easy use of numpy arrays from C++. In brief, I have defined a C++ template class around numpy's own PyArrayObject structure which allows for convenient access of the array data using C++-like techniques and takes care of type checking and the like. As it adds only non-virtual methods and no fields, the objects are memory-wise identical to numpy's own array structure and can hence be passed between C++ and Python without any copying of data or metadata. The original idea behind was a bit the opposite of your problem. I had program for a numerical computation, written in Python, and wanted to rewrite the most performance-critical parts in C++. With my SWIG typemap, I can now write the C++ part using my template class and wrap it such that these objects appear to Python as numpy array objects. For your problem, it might be a way to write a kind of casting functions that takes one of your tensor objects and transforms it into my numpy objects. This should be possible with only copying metadata (i.e. fields containing dimensions and the like) but with leaving the actual data in place. My stuff is still a bit work in progress, and so I don't want to post it here yet, but it may help you to not have to start from scratch, as the typemap code would help you to get started and might be easy to adjust to your needs. So, if you are interested, send me a mail, and I would appreciate any comments from you on how to make my tool into seomthing really reusable. Regards, Simon -- +--- | Simon Anders, Dipl. Phys. | Institut fuer Theoretische Physik, Universitaet Innsbruck, Austria | Tel. +43-512-507-6207, Fax -2919 | preferred (permanent) e-mail: sanders at fs.tum.de From theller at python.net Tue Jul 4 06:00:42 2006 From: theller at python.net (Thomas Heller) Date: Tue, 04 Jul 2006 12:00:42 +0200 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: References: <44A73B5D.60002@ieee.org> Message-ID: Thomas Heller schrieb: > I've also played a little, and I think one important limitation in ctypes > is that items in the argtypes list have to be ctypes types. Thi swas misleading: I mean that this limitation should probably be removed, because it prevents a lot of things one could do. Thomas From jan_braun at gmx.net Tue Jul 4 06:10:18 2006 From: jan_braun at gmx.net (Jan-Matthias Braun) Date: Tue, 4 Jul 2006 12:10:18 +0200 Subject: [Numpy-discussion] KeyError with float96 in linalg.py: polyfit Message-ID: <200607041210.20480.jan_braun@gmx.net> Hi all, I'm testing some computations with float96 at the moment and right now I have problems with polyfit raising a KeyError for the keycode 'g', which is floatxx with xx>64. I am getting a KeyError using polyfit on some float96 values. The used Routines seem to know nothing about this type. My main question is: have I missed something? Shouldn't this type be used? Below is a more detailed descripton. Thanks in advance, Jan ---------------------------------------------------------------------------- In file numpy/linalg/linalg.py, the following definitions at lines 26ff seem to be the offending ones: # Helper routines _array_kind = {'i':0, 'l': 0, 'q': 0, 'f': 0, 'd': 0, 'F': 1, 'D': 1} _array_precision = {'i': 1, 'l': 1, 'q': 1, 'f': 0, 'd': 1, 'F': 0, 'D': 1} _array_type = [['f', 'd'], ['F', 'D']] Here the new typecodes are missing. I tried # Helper routines _array_kind = {'i':0, 'l': 0, 'q': 0, 'f': 0, 'd': 0, 'g': '0', 'F': 1, 'D':1, 'G':1} _array_precision = {'i': 1, 'l': 1, 'q': 1, 'f': 0, 'd': 1, 'g': 1, 'F': 0, 'D': 1, 'G': 1} _array_type = [['f', 'd', 'g'], ['F', 'D', 'G']] which gets me a step further to a TypeError: File "lib/python2.3/site-packages/numpy/linalg/linalg.py", line 454, in lstsq bstar[:b.shape[0],:n_rhs] = b.copy() TypeError: array cannot be safely cast to required type (Question: Why only one typecode for a type which varies in bitlength on different platforms? On Opteron CPU's I've seen float128 with 'g'?) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From fullung at gmail.com Tue Jul 4 06:11:53 2006 From: fullung at gmail.com (Albert Strasheim) Date: Tue, 4 Jul 2006 12:11:53 +0200 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: Message-ID: <00fe01c69f52$4614c9e0$0a84a8c0@dsp.sun.ac.za> Hey Thomas Thomas Heller wrote: > Thomas Heller schrieb: > > I've also played a little, and I think one important limitation in > ctypes > > is that items in the argtypes list have to be ctypes types. > > Thi swas misleading: I mean that this limitation should probably be > removed, because it prevents a lot of things one could do. What's your thinking on getting these changes made to ctypes and on ctypes' future development in general? Presumably you can't change it too much with the Python 2.5 release coming up, but it would be a shame if we had to wait until Python 2.6 to get the changes you suggested (and other goodies, like the array interface). Regards, Albert From simon at arrowtheory.com Tue Jul 4 07:47:58 2006 From: simon at arrowtheory.com (Simon Burton) Date: Tue, 4 Jul 2006 13:47:58 +0200 Subject: [Numpy-discussion] A C++ library and the new array interface: the best approach? In-Reply-To: <44A99D07.5030706@colorado.edu> References: <44A99D07.5030706@colorado.edu> Message-ID: <20060704134758.0dae19d8.simon@arrowtheory.com> On Mon, 03 Jul 2006 16:41:11 -0600 Fernando Perez wrote: > > So I'd like to know if SWIG is really the best way out in this particular case > (and any advice on taking advantage of the array interface via SWIG would be > appreciated), or if ctypes or pyrex could be used here. I'm quite happy using > pyrex in other contexts, but I know it doesn't directly support C++. However, > since I have access to all the code, perhaps a pure C layer could be used to > bridge the C++/pyrex gap. Or given the recent praise for ctypes, perhaps that > can be an option? Pyrex can handle some C++. Eg. make objects, and call methods. You will need to search the pyrex email archives for all the tricks to get this to work. Simon. From theller at python.net Tue Jul 4 07:49:42 2006 From: theller at python.net (Thomas Heller) Date: Tue, 04 Jul 2006 13:49:42 +0200 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: <00fe01c69f52$4614c9e0$0a84a8c0@dsp.sun.ac.za> References: <00fe01c69f52$4614c9e0$0a84a8c0@dsp.sun.ac.za> Message-ID: Albert Strasheim schrieb: > Hey Thomas > > Thomas Heller wrote: >> Thomas Heller schrieb: >> > I've also played a little, and I think one important limitation in >> ctypes >> > is that items in the argtypes list have to be ctypes types. >> >> Thi swas misleading: I mean that this limitation should probably be >> removed, because it prevents a lot of things one could do. > > What's your thinking on getting these changes made to ctypes and on ctypes' > future development in general? > > Presumably you can't change it too much with the Python 2.5 release coming > up, but it would be a shame if we had to wait until Python 2.6 to get the > changes you suggested (and other goodies, like the array interface). I have asked on python-dev, let's wait for the answer. I hope that at least the limitation that I mentioned can be removed in Python 2.5. The goal of my post was to show that (without this restriction) a lot can already be done in Python, of course it would be better if this could be implemented in C and integrated in ctypes. For the numpy/ctypes inegration I'm not absolutely sure what would be needed most: Is there a need to convert between ctypes and numpy arrays? If numpy arrays can be passed to ctypes foreign functions maybe there is no need at all for the conversion. We could probably even live with helper code like that I posted outside of ctypes... Thomas From stephenemslie at gmail.com Tue Jul 4 08:00:29 2006 From: stephenemslie at gmail.com (stephen emslie) Date: Tue, 4 Jul 2006 13:00:29 +0100 Subject: [Numpy-discussion] iterate along a ray: linear algebra? In-Reply-To: References: <51f97e530606300713w1c167cf3j10c36d24f87326cf@mail.gmail.com> Message-ID: <51f97e530607040500y5e4becja4b8d86c019129cc@mail.gmail.com> I've found some matlab code that seems to do the same sort of thing. Interestingly enough it just uses trigonomotry to do find the x,y positions in the matrix that correspond to the ray at a particular angle. I had origionally discarded this idea because I thought there must be a more efficient way to do it, but perhaps not. I still have a lot to learn about numpy :) Stephen (sorry for the double post to you Torgil) On 7/3/06, Torgil Svensson wrote: > > I've done something similar a few years ago (numarray,numeric). I > started roughly at the middle and did 64 points from a reference point > (xc,yc). This point together with a point at the edge of the image > (xp,yp) also defined a reference angle (a0). (ysize,xsize) is the > shape of the intensity image. > > I used the following code to calculate points of interest: > > na=64 > xr,yr=xsize-xc,ysize-yc > a0=arctan2(yp-yr,xp-xc) > if a0<0: a0+=2*pi > ac=arctan2([yr,yr,-yc,-yc],[xr,-xc,-xc,xr]) > if numarray: > ac[ac<0]+=2*pi > else: > ac=choose(ac<0,(ac,ac+2*pi)) > a1,a2,a3,a4=ac > rmaxfn={ > 0: lambda a: a<=a1 and xr/cos(a-0.0*pi) or yr/cos(0.5*pi-a), > 1: lambda a: a<=a2 and yr/cos(a-0.5*pi) or xc/cos(1.0*pi-a), > 2: lambda a: a<=a3 and xc/cos(a-1.0*pi) or yc/cos(1.5*pi-a), > 3: lambda a: a<=a4 and yc/cos(a-1.5*pi) or xr/cos(2.0*pi-a) > } > angles=arange(a0,a0+2*pi,2*pi/na) > if numarray: > angles[angles>=2*pi]-=2*pi > else: > angles=choose(angles>=2*pi,(angles,angles-2*pi)) > nr=int(ceil(sqrt(max(yc,yr)**2+max(xc,xr)**2))) > crmax=array([int(floor(rmaxfn[floor(a*2/pi)](a))) for a in angles]) > cr=outerproduct(ones(na),arange(float(nr))) > ca=outerproduct(angles,ones(nr)) > x=cr*cos(ca)+xc > y=cr*sin(ca)+yc > > After this I did cubic spline interpolation in the image with these > points and did something useful. I don't know how relevant this is to > you and it doesn't use the linear algebra package but it might give > you some hint. > > If you find out a nifty way to do your rays please post on this thread. > > Sidenote -- Watch my explicit float argument to arange and even > putting in pi there in one case. There's a discussion on this list > that floats in arange are troublesome > > > On 6/30/06, stephen emslie wrote: > > I am in the process of implementing an image processing algorithm that > > requires following rays extending outwards from a starting point and > > calculating the intensity derivative at each point. The idea is to find > the > > point where the difference in intensity goes beyond a particular > threshold. > > > > Specifically I'm examining an image of an eye to find the pupil, and the > > edge of the pupil is a sharp change in intensity. > > > > How does one iterate along a line in a 2d matrix, and is there a better > way > > to do this? Is this a problem that linear algebra can help with? > > > > Thanks > > Stephen Emslie > > > > Using Tomcat but need to do more? Need to support web services, > security? > > Get stuff done quickly with pre-integrated technology to make your job > > easier > > Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fullung at gmail.com Tue Jul 4 08:58:05 2006 From: fullung at gmail.com (Albert Strasheim) Date: Tue, 4 Jul 2006 14:58:05 +0200 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: References: <00fe01c69f52$4614c9e0$0a84a8c0@dsp.sun.ac.za> Message-ID: <20060704125805.GA7962@dogbert.sdsl.sun.ac.za> Hello all On Tue, 04 Jul 2006, Thomas Heller wrote: > Albert Strasheim schrieb: > > Hey Thomas > > > > Thomas Heller wrote: > >> Thomas Heller schrieb: > >> > I've also played a little, and I think one important limitation in > >> ctypes > >> > is that items in the argtypes list have to be ctypes types. > >> > >> Thi swas misleading: I mean that this limitation should probably be > >> removed, because it prevents a lot of things one could do. > > > > What's your thinking on getting these changes made to ctypes and on ctypes' > > future development in general? > > > > Presumably you can't change it too much with the Python 2.5 release coming > > up, but it would be a shame if we had to wait until Python 2.6 to get the > > changes you suggested (and other goodies, like the array interface). > > I have asked on python-dev, let's wait for the answer. > I hope that at least the limitation that I mentioned can be removed in Python 2.5. Sounds great. > The goal of my post was to show that (without this restriction) a lot can > already be done in Python, of course it would be better if this could be > implemented in C and integrated in ctypes. > > For the numpy/ctypes inegration I'm not absolutely sure what would be needed most: > > Is there a need to convert between ctypes and numpy arrays? If numpy arrays can > be passed to ctypes foreign functions maybe there is no need at all for the conversion. > We could probably even live with helper code like that I posted outside of ctypes... I think there are basically two ways for a C library to work with regards to memory allocation: 1. let the user allocate the array/struct/whatever and pass a pointer to the library to manipulate 2. let the library allocate the array/struct/whatever, manipulate it and return the pointer to the user I think the first case is pretty much covered. Where in the past you would create the array or struct on the stack or allocate it on the heap with malloc, you now create a ctypes Structure or a NumPy array and pass that to the C function. In the second case, one would want to wrap a NumPy array around the ctype so that you can manipulate the data returned by the library. I don't know if this second scenario is very common -- hopefully not. If not, then having ctypes implement the array interface isn't too critical, since you wouldn't typically need to make a NumPy array from existing data. What do you think? Regards, Albert From steffen.loeck at gmx.de Tue Jul 4 09:19:24 2006 From: steffen.loeck at gmx.de (Steffen Loeck) Date: Tue, 4 Jul 2006 15:19:24 +0200 Subject: [Numpy-discussion] Speed degression? Message-ID: <200607041519.24636.steffen.loeck@gmx.de> Hi all, i made some speed tests using the sin-function and the %-operation to compare Numeric, numpy 0.9.8 and numpy 0.9.9.2732. As result the latest numpy version seems to be very slow in comparison to the two other candidates. Results (in usec per loop): sin-array mod-array Numeric 134 18 numpy 0.9.8 97 55 numpy 0.9.9.2732 204 316 numpy 0.9.8 + math 38 numpy 0.9.9.2732 + math 161 Numeric + math 23 The used scripts can be found at the end. Can anyone verify my results and explain the observed speed degression? Thanks, Steffen sin-scripts: /usr/lib/python2.3/timeit.py -s "from Numeric import sin,zeros,arange; x=zeros(10, 'd'); x[0]=0.1" "for i in arange(9): x[i+1]=sin(x[i])" /usr/lib/python2.3/timeit.py -s "from numpy import sin,zeros,arange; x=zeros(10, 'd'); x[0]=0.1" "for i in arange(9): x[i+1]=sin(x[i])" /usr/lib/python2.3/timeit.py -s "from math import sin; from numpy import zeros,arange; x=zeros(10, 'd');x[0]=0.1" "for i in arange(9): x[i+1]=sin(x[i])" /usr/lib/python2.3/timeit.py -s "from math import sin; from Numeric import zeros,arange; x=zeros(10, 'd'); x[0]=0.1" "for i in arange(9): x[i+1]=sin(x[i] )" %-scripts /usr/lib/python2.3/timeit.py -s "from Numeric import zeros,arange; x=zeros(10, 'd'); x[0]=0.1" "for i in arange(9): x[i+1]=(x[i]+1.1)%(1.0)" /usr/lib/python2.3/timeit.py -s "from numpy import zeros,arange; x=zeros(10, 'd'); x[0]=0.1" "for i in arange(9): x[i+1]=(x[i]+1.1)%(1.0)" From david.huard at gmail.com Tue Jul 4 09:28:02 2006 From: david.huard at gmail.com (David Huard) Date: Tue, 4 Jul 2006 09:28:02 -0400 Subject: [Numpy-discussion] Error building numpy from svn, 2nd attempt In-Reply-To: <7.0.1.0.0.20060704100002.01f8fc38@ecs.soton.ac.uk> References: <7.0.1.0.0.20060704100002.01f8fc38@ecs.soton.ac.uk> Message-ID: <91cf711d0607040628m6914bd9cj60957595d1da0ee2@mail.gmail.com> Hi John, Here is a patch to fix the first error in test_twodim_base.py. I'm sorry I can't help you with the rest. David 2006/7/4, John Carter : > > Hi, > > As is the way posting to a news group stirs the brain cell into > activity and the problem is solved. or rather shifted. > > I've downloaded the candidate version of mingw32 and using that to > build numpy/scipy works, or rather it builds the extensions for Python 2.3 > > I believe that there are still problems as both numpy and scipy fail > their tests. I'm also dubious as to whether they are finding Altas and > Lapack. > > Configuration: > > Windows XP SP2 > Cygwin = gcc version 3.4.4 (cygming special) (gdc 0.12, using dmd 0.125) > MinGW = gcc version 3.4.5 (mingw special) > Python = Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit > (Intel)] on win32 > > scipy, numpy, Atlas, Lapack all downloaded within the last 2 days > > > I've cut and pasted the error messages from the tests below. > > Any help gratefully received. > > John > > > ===== > numpy > ===== > D:.\numerical> python > Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on > win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > >>> numpy.test() > Found 5 tests for numpy.distutils.misc_util > Found 3 tests for numpy.lib.getlimits > Found 30 tests for numpy.core.numerictypes > Found 13 tests for numpy.core.umath > Found 3 tests for numpy.core.scalarmath > Found 8 tests for numpy.lib.arraysetops > Found 42 tests for numpy.lib.type_check > Found 101 tests for numpy.core.multiarray > Found 36 tests for numpy.core.ma > Found 10 tests for numpy.lib.twodim_base > Found 10 tests for numpy.core.defmatrix > Found 1 tests for numpy.lib.ufunclike > Found 38 tests for numpy.lib.function_base > Found 3 tests for numpy.dft.helper > Found 1 tests for numpy.lib.polynomial > Found 7 tests for numpy.core.records > Found 26 tests for numpy.core.numeric > Found 4 tests for numpy.lib.index_tricks > Found 46 tests for numpy.lib.shape_base > Found 0 tests for __main__ > > ........................................................................................................ > > ........................................................................................................ > > .........E.............................................................................................. > ........................... > ====================================================================== > ERROR: check_simple (numpy.lib.tests.test_twodim_base.test_histogram2d) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "C:\Python23\Lib\site-packages\numpy\lib\tests\test_twodim_base.py", > line 137, in check_simple > np.random.seed(1) > File "mtrand.pyx", line 311, in mtrand.RandomState.seed > SystemError: C:\sf\python\dist\src-maint23\Objects\longobject.c:240: > bad argument to internal function > > ---------------------------------------------------------------------- > Ran 387 tests in 1.391s > > FAILED (errors=1) > > >>> > > ===== > scipy > ===== > D:.\numerical> python > Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on > win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> import scipy > >>> scipy.test() > Overwriting lib= 'C:\PYTHON23\lib\site-packages\scipy\lib\__init__.pyc'> from > C:\PYTHON23\lib\si > te-packages\scipy\lib\__init__.pyc (was 'C:\PYTHON23\lib\site-packages\numpy\lib\__init__.pyc'> > from C:\PYTHON23\lib\site-packages\numpy\lib\__init__.pyc) > Overwriting fft= from scipy.fftpack.basic > (was from numpy.dft.f > ftpack) > Overwriting ifft= from > scipy.fftpack.basic (was from numpy.df > t.fftpack) > Found 4 tests for scipy.io.array_import > Found 128 tests for scipy.linalg.fblas > Found 397 tests for scipy.ndimage > Found 10 tests for scipy.integrate.quadpack > Found 97 tests for scipy.stats.stats > Found 47 tests for scipy.linalg.decomp > Found 2 tests for scipy.integrate.quadrature > Found 95 tests for scipy.sparse.sparse > Found 20 tests for scipy.fftpack.pseudo_diffs > Found 6 tests for scipy.optimize.optimize > Found 5 tests for scipy.interpolate.fitpack > Found 1 tests for scipy.interpolate > Found 12 tests for scipy.io.mmio > Found 10 tests for scipy.stats.morestats > Found 4 tests for scipy.linalg.lapack > Found 18 tests for scipy.fftpack.basic > Found 4 tests for scipy.linsolve.umfpack > Found 4 tests for scipy.optimize.zeros > Found 41 tests for scipy.linalg.basic > Found 2 tests for scipy.maxentropy.maxentropy > Found 358 tests for scipy.special.basic > Found 128 tests for scipy.lib.blas.fblas > Found 7 tests for scipy.linalg.matfuncs > > **************************************************************** > WARNING: clapack module is empty > ----------- > See scipy/INSTALL.txt for troubleshooting. > Notes: > * If atlas library is not found by numpy/distutils/system_info.py, > then scipy uses flapack instead of clapack. > **************************************************************** > > Found 42 tests for scipy.lib.lapack > Found 1 tests for scipy.optimize.cobyla > Found 16 tests for scipy.lib.blas > Found 1 tests for scipy.integrate > Found 14 tests for scipy.linalg.blas > Found 70 tests for scipy.stats.distributions > Found 4 tests for scipy.fftpack.helper > Found 4 tests for scipy.signal.signaltools > Found 0 tests for __main__ > > Don't worry about a warning regarding the number of bytes read. > Warning: 1000000 bytes requested, 20 bytes read. > .......caxpy:n=4 > ..caxpy:n=3 > ....ccopy:n=4 > ..ccopy:n=3 > .............cscal:n=4 > ....cswap:n=4 > ..cswap:n=3 > .....daxpy:n=4 > ..daxpy:n=3 > ....dcopy:n=4 > ..dcopy:n=3 > .............dscal:n=4 > ....dswap:n=4 > ..dswap:n=3 > .....saxpy:n=4 > ..saxpy:n=3 > ....scopy:n=4 > ..scopy:n=3 > .............sscal:n=4 > ....sswap:n=4 > ..sswap:n=3 > .....zaxpy:n=4 > ..zaxpy:n=3 > ....zcopy:n=4 > ..zcopy:n=3 > .............zscal:n=4 > ....zswap:n=4 > ..zswap:n=3 > > ........................................................................................................................ > > ........................................................................................................................ > > ........................................................................................................................ > > ........................................................................................................................ > > .........................................................................Took > 13 points. > ..........Resizing... 16 17 24 > Resizing... 20 7 35 > Resizing... 23 7 47 > Resizing... 24 25 58 > Resizing... 28 7 68 > Resizing... 28 27 73 > .....Use minimum degree ordering on A'+A. > ........................Use minimum degree ordering on A'+A. > ...................Resizing... 16 17 24 > Resizing... 20 7 35 > Resizing... 23 7 47 > Resizing... 24 25 58 > Resizing... 28 7 68 > Resizing... 28 27 73 > .....Use minimum degree ordering on A'+A. > .................Resizing... 16 17 24 > Resizing... 20 7 35 > Resizing... 23 7 47 > Resizing... 24 25 58 > Resizing... 28 7 68 > Resizing... 28 27 73 > .....Use minimum degree ordering on A'+A. > > ......................................C:\PYTHON23\lib\site-packages\scipy\interpolate\fitpack2.py:410: > UserWarning: > The coefficients of the spline returned have been computed as the > minimal norm least-squares solution of a (numerically) rank deficient > system (deficiency=7). If deficiency is large, the results may be > inaccurate. Deficiency may strongly depend on the value of eps. > warnings.warn(message) > ....................Ties preclude use of exact statistic. > ..Ties preclude use of exact statistic. > ........ > **************************************************************** > WARNING: clapack module is empty > ----------- > See scipy/INSTALL.txt for troubleshooting. > Notes: > * If atlas library is not found by numpy/distutils/system_info.py, > then scipy uses flapack instead of clapack. > **************************************************************** > > ....................data-ftype: z compared to data D > Calling _superlu.zgssv > Use minimum degree ordering on A'+A. > .data-ftype: c compared to data F > Calling _superlu.cgssv > Use minimum degree ordering on A'+A. > .data-ftype: d compared to data d > Calling _superlu.dgssv > Use minimum degree ordering on A'+A. > .data-ftype: s compared to data f > Calling _superlu.sgssv > Use minimum degree ordering on A'+A. > > ........................................................................................................................ > > ........................................................................................................................ > ....................................Gegenbauer, a = 2.7337413228 > > ........................................................................................................................ > .............caxpy:n=4 > ..caxpy:n=3 > ....ccopy:n=4 > ..ccopy:n=3 > .............cscal:n=4 > ....cswap:n=4 > ..cswap:n=3 > .....daxpy:n=4 > ..daxpy:n=3 > ....dcopy:n=4 > ..dcopy:n=3 > .............dscal:n=4 > ....dswap:n=4 > ..dswap:n=3 > .....saxpy:n=4 > ..saxpy:n=3 > ....scopy:n=4 > ..scopy:n=3 > .............sscal:n=4 > ....sswap:n=4 > ..sswap:n=3 > .....zaxpy:n=4 > ..zaxpy:n=3 > ....zcopy:n=4 > ..zcopy:n=3 > .............zscal:n=4 > ....zswap:n=4 > ..zswap:n=3 > ...Result may be inaccurate, approximate err = 4.1928136851e-009 > ...Result may be inaccurate, approximate err = 7.27595761418e-012 > .............................................F................Residual: > 1.05006950319e-007 > . > **************************************************************** > WARNING: cblas module is empty > ----------- > See scipy/INSTALL.txt for troubleshooting. > Notes: > * If atlas library is not found by numpy/distutils/system_info.py, > then scipy uses fblas instead of cblas. > **************************************************************** > > > ............................................................................................ > ====================================================================== > FAIL: check_simple (scipy.optimize.tests.test_cobyla.test_cobyla) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "C:\Python23\Lib\site-packages\scipy\optimize\tests\test_cobyla.py", > line 20, in check_simple > assert_almost_equal(x, [x0, x1], decimal=5) > File "C:\Python23\Lib\site-packages\numpy\testing\utils.py", line > 152, in assert_almost_equal > return assert_array_almost_equal(actual, desired, decimal, err_msg) > File "C:\Python23\Lib\site-packages\numpy\testing\utils.py", line > 222, in assert_array_almost_equal > header='Arrays are not almost equal') > File "C:\Python23\Lib\site-packages\numpy\testing\utils.py", line > 207, in assert_array_compare > assert cond, msg > AssertionError: > Arrays are not almost equal > > (mismatch 100.0%) > x: array([ 4.957975 , 0.64690335]) > y: array([ 4.95535625, 0.66666667]) > > ---------------------------------------------------------------------- > Ran 1552 tests in 5.828s > > FAILED (failures=1) > > >>> > > > > > > Dr. John N. Carter. E-Mail : > jnc at ecs.soton.ac.uk > Building 1, Room > 2005 http://www.ecs.soton.ac.uk/~jnc/ > Information: Signals, Images, Systems Phone : +44 (0) 23 8059 > 2405 > School of Electronics & Computer Science, Fax : +44 (0) 23 8059 > 4498 > Southampton University, Hants, UK SO17 1BJ. > > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: histogram2d_test.patch Type: text/x-patch Size: 675 bytes Desc: not available URL: From jensj at fysik.dtu.dk Tue Jul 4 10:25:42 2006 From: jensj at fysik.dtu.dk (=?UTF-8?B?SmVucyBKw7hyZ2VuIE1vcnRlbnNlbg==?=) Date: Tue, 04 Jul 2006 16:25:42 +0200 Subject: [Numpy-discussion] Why is my array not contiguous? Message-ID: <44AA7A66.9070809@servfys.fysik.dtu.dk> Hi! With numpy-0.9.9.2726, I do this: >>> x = arange(4) >>> y = x[newaxis, :] I would expect both arrays to be contiguous: >>> x.flags.contiguous, y.flags.contiguous (True, False) Shouldn't y be contiguous? Maybe it's because of the strange strides: >>> y.strides (0, 4) >>> y.strides = (16, 4) >>> y.flags.contiguous True Jens J?rgen From bart.vandereycken at cs.kuleuven.be Tue Jul 4 11:16:16 2006 From: bart.vandereycken at cs.kuleuven.be (Bart Vandereycken) Date: Tue, 04 Jul 2006 17:16:16 +0200 Subject: [Numpy-discussion] Dot as a method Message-ID: Hi all, reading the thread "Ransom proposals" I was wondering why there isn't a ndarray.dot() method? There is already a scipy.sparse.dot() so this would fit nicely in the whole idea of polymorphism. Bart From cookedm at physics.mcmaster.ca Tue Jul 4 16:09:38 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Tue, 4 Jul 2006 16:09:38 -0400 Subject: [Numpy-discussion] KeyError with float96 in linalg.py: polyfit In-Reply-To: <200607041210.20480.jan_braun@gmx.net> References: <200607041210.20480.jan_braun@gmx.net> Message-ID: <20060704160938.37a2f70a@arbutus.physics.mcmaster.ca> On Tue, 4 Jul 2006 12:10:18 +0200 Jan-Matthias Braun wrote: > Hi all, > > I'm testing some computations with float96 at the moment and right now I > have problems with polyfit raising a KeyError for the keycode 'g', which is > floatxx with xx>64. Use longdouble instead of float96; it'll make your code portable. > I am getting a KeyError using polyfit on some float96 values. The used > Routines seem to know nothing about this type. > > My main question is: have I missed something? Shouldn't this type be used? > Below is a more detailed descripton. polyfit uses LAPACK to do least-squares for the fit, and LAPACK doesn't handle any real types besides single- and double-precision. And Numpy (as a design decision) doesn't like casting to a lower precision. > In file numpy/linalg/linalg.py, the following definitions at lines 26ff > seem to be the offending ones: > # Helper routines > _array_kind = {'i':0, 'l': 0, 'q': 0, 'f': 0, 'd': 0, 'F': 1, 'D': > 1} _array_precision = {'i': 1, 'l': 1, 'q': 1, 'f': 0, 'd': 1, 'F': 0, 'D': > 1} _array_type = [['f', 'd'], ['F', 'D']] > > Here the new typecodes are missing. I tried > # Helper routines > _array_kind = {'i':0, 'l': 0, 'q': 0, 'f': 0, 'd': 0, 'g': '0', > 'F': 1, 'D':1, 'G':1} > _array_precision = {'i': 1, 'l': 1, 'q': 1, 'f': 0, 'd': 1, 'g': 1, > 'F': 0, 'D': 1, 'G': 1} > _array_type = [['f', 'd', 'g'], ['F', 'D', 'G']] > > which gets me a step further to a TypeError: > > File "lib/python2.3/site-packages/numpy/linalg/linalg.py", line 454, in > lstsq > bstar[:b.shape[0],:n_rhs] = b.copy() > TypeError: array cannot be safely cast to required type That would be a better error. I'm going to leave it like this for now, though. Instead, the linalg module will be converted to use dtypes. > (Question: Why only one typecode for a type which varies in bitlength on > different platforms? On Opteron CPU's I've seen float128 with 'g'?) Typecodes are deprecated (they date back to Numeric), so we're not bothering to add new ones. The equivalent to 'g' is the dtype longdouble. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From josh8912 at yahoo.com Tue Jul 4 17:23:20 2006 From: josh8912 at yahoo.com (JJ) Date: Tue, 4 Jul 2006 14:23:20 -0700 (PDT) Subject: [Numpy-discussion] arguments for stats.chi2 Message-ID: <20060704212320.30425.qmail@web51712.mail.yahoo.com> Hello. I have a very simple question. I would like to generate a number of random variables from the chi2 distribution. If I wanted these for the normal distribution, the code could be stats.norm.rvs(size=5,loc=100,scale=1). But stats.chi2.rvs(size=5,df=10,loc=0,scale=1) or stats.chi2.rvs(df=10,loc=0,scale=1) or stats.chi2.rvs(df=10)does not work. Can anyone tell me what the proper syntax would be for this? Thanks JJ __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From pfdubois at gmail.com Tue Jul 4 17:54:32 2006 From: pfdubois at gmail.com (Paul Dubois) Date: Tue, 4 Jul 2006 14:54:32 -0700 Subject: [Numpy-discussion] MA bug or feature? In-Reply-To: <200607040240.00075.pgmdevlist@mailcan.com> References: <16761e100606210146q7683c94bu5bd2699caa6b95cf@mail.gmail.com> <16761e100607032144w65013936jbbf9772f429f4be9@mail.gmail.com> <200607040240.00075.pgmdevlist@mailcan.com> Message-ID: Some things to note: The mask is copy-on-write. Don't mess with that. You can't just poke values into an existing mask, it may be shared with other arrays. I do not agree that there is any 'inconsistency'. It may be someone's concept of the class that if there is a mask then at least one value is on, but that was not my design. I believe if you try your ideas you'll find it slows other people down, if not you. Perhaps with all of Travis' new machinery, subclassing works. It didn't used to, and I haven't kept up. On 7/3/06, Pierre GM wrote: > > Michael, > I wonder whether the Mask class you suggest is not a bit overkill. There > should be enough tools in the existing MA module to do what we want. And I > don't wanna think about compatibility the number of changes in the MA code > that'd be required (but I'm lazy)... > > For the sake of consistency and optimization, I still think it could be > easier > (and cleaner) to make `nomask` the default for a MaskedArray without > masked > values. That could for example be implemented by forcing `nomask` at the > creation of the MaskedArray with an extra > `if mask and not mask.any(): mask=nomask`, or by using Paul's > make_mask( flag=1) trick. > > Masking some specific values could still be done when mask is nomask with > an > intermediary MA.getmaskarray() step. > > On a side note, modifying an existing mask is a delicate matter. > Everything's > OK if you use masks as a way to hide existing data, it's more complex when > initially you have some holes in your dataset... > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Wed Jul 5 00:03:34 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 5 Jul 2006 13:03:34 +0900 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again Message-ID: Just wanted to make one last effort get a .T attribute for arrays, so that you can flip axes with a simple "a.T" instead of "a.transpose()", as with numpy matrix objects. If I recall, the main objection raised before was that there are lots of ways to transpose n-dimensional data. Fine, but the fact is that 2D arrays are pretty darn common, and so are a special case worth optimizing for. Furthermore transpose() won't go away if you do need to do some specific kind of axes swapping other than the default, so noone is really going to be harmed by adding it. I propose to make .T a synonym for .swapaxes(-2,-1) {*}, i.e. the last two axes are interchanged. This should also make it useful in many N-d array cases (whereas the default of .transpose() -- to completely reverse the order of all the axes -- is seldom what you want). Part of the thinking is that when you print an N-d array it's the last two dimensions that get printed like 2-d matrices separated by blank likes. You can think of it as some number of stacks of 2-d matrices. So this .T would just transpose those 2-d matrices in the printout. Those are the parts that are generally most contiguous in memory also, so it makes sense for 2-d matrix bits to be stored in those last two dimensions. Then, if there is a .T, it makes sense to also have .H which would basically be equivalent to .T.conjugate(). Finally, the matrix class has .A to get the underlying array -- it would also be nice to have a .M on array as a shortcut for asmatrix(). This one would be very handy for matrix users, I think, but I could go either way on that, having abandoned matrix myself. Ex: ones([4,4]).M Other possibilities: - Make .T a function, so that you can pass it the same info as .transpose(). Then the shortcut becomes a.T(), which isn't as nice, and isn't consistent with matrix's .T any more. - Just make .T raise an error for ndim>2. But I don't really see any benefit in making it an error as opposed to defining a reasonable default behavior. - Make .T on a 1-dim array return a 2-dim Nx1 array. (My default suggestion is to just leave it alone if ndim < 2, an exception would be another possiblility). Would make an easy way to create column vectors from arrays, but I can think of nothing else in Numpy that acts that way. This is not a 1.0 must have, as it introduces no backward compatibility issues. But it would be trivial to add if the will is there. {*} except that negative axes for swapaxes doesn't seem work currently, so instead it would need to be something like: a.transpose( a.shape[:-2] + (a.shape[-1],a.shape[-2]) ) with a check for "if ndim > 1", of course. --Bill -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Wed Jul 5 00:58:45 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 5 Jul 2006 13:58:45 +0900 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: Message-ID: Slight correction. {*} except that negative axes for swapaxes doesn't seem work currently, so > instead it would need to be something like: > a.transpose( a.shape[:-2] + (a.shape[-1],a.shape[-2]) ) > with a check for "if ndim > 1", of course. > Apparently a.swapaxes(-2,-1) does work, and it does exactly what I am suggesting, including leaving zero-d and 1-d arrays alone. Not sure why I thought it wasn't working. So in short my proposal is to: -- make a.T a property of array that returns a.swapaxes(-2,-1), -- make a.H a property of array that returns a.conjugate().swapaxes(-2,-1) and maybe -- make a.M a property of array that returns numpy.asmatrix(a) --Bill -------------- next part -------------- An HTML attachment was scrubbed... URL: From 67753110 at 163.com Wed Jul 5 01:42:19 2006 From: 67753110 at 163.com (=?GB2312?B?1tC5+r3wyvS5pNK1zfg=?=) Date: Wed, 5 Jul 2006 13:42:19 +0800 Subject: [Numpy-discussion] =?GB2312?B?xOPSqsHLveK1xKOsvs3U2rTL09C96cnco6E=?= Message-ID: numpy-discussion:??! ? ????????www.cmpi.cn,???? ? ????????? ?! ?????????????? ??????? ?????????????? 67753110 at 163.com ?????????????? 2006-07-05 From wbaxter at gmail.com Wed Jul 5 02:41:19 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 5 Jul 2006 15:41:19 +0900 Subject: [Numpy-discussion] Simple 3D drawing options? Message-ID: I tried to get MayaVi and VTK working under Win32/MSVC.Net a while back failed miserably. Is there some simple, out-of-the-box, precompiled, no-brains-required solution for creating 3D plots? Preferably one that doesn't require compiling anything. --Bill -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Wed Jul 5 03:17:16 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 5 Jul 2006 16:17:16 +0900 Subject: [Numpy-discussion] Simple 3D drawing options? In-Reply-To: <63eb7fa90607042357j4053b114yda0227c91f38b99c@mail.gmail.com> References: <63eb7fa90607042357j4053b114yda0227c91f38b99c@mail.gmail.com> Message-ID: I like the idea of enthon, but I hadn't given it much thought because it seems to lag so far behind in versions. Python 2.3, Numpy 0.9.6, etc. But maybe that's not as big a deal as I've been thinking? Two questions: * Can it happily co-exist with separate installs of more up-to-date versions of things? Like can I have Python 2.4 installed at the same time? Or a different Python 2.3? * Or is it user-upgradeable in a piecemeal manner? E.g. could I install Numpy 0.9.8 over top of enthon's older version of Numpy? --bb On 7/5/06, Edin Salkovi? wrote: > > Have you tried: > http://code.enthought.com/enthon/ > > Cheers, > Edin > > On 7/5/06, Bill Baxter wrote: > > I tried to get MayaVi and VTK working under Win32/MSVC.Net a while back > > failed miserably. > > > > Is there some simple, out-of-the-box, precompiled, no-brains-required > > solution for creating 3D plots? Preferably one that doesn't require > > compiling anything. > > > > --Bill > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ferenc.Pintye at eu.decoma.com Wed Jul 5 08:52:43 2006 From: Ferenc.Pintye at eu.decoma.com (Ferenc.Pintye at eu.decoma.com) Date: Wed, 5 Jul 2006 14:52:43 +0200 Subject: [Numpy-discussion] (no subject) Message-ID: Hi, how could i get the name of an array in a string ? (info command ?!) thanks f. From david.huard at gmail.com Wed Jul 5 09:23:12 2006 From: david.huard at gmail.com (David Huard) Date: Wed, 5 Jul 2006 09:23:12 -0400 Subject: [Numpy-discussion] arguments for stats.chi2 In-Reply-To: <20060704212320.30425.qmail@web51712.mail.yahoo.com> References: <20060704212320.30425.qmail@web51712.mail.yahoo.com> Message-ID: <91cf711d0607050623u5cf327bbx187749c407dc044d@mail.gmail.com> HI JJ, try stats.chi2.rvs(10) stats.chi2.rvs(10, loc=0, scale =100, size = 5) df is not a keyword argument, so writing df=10 explicitely was causing the error. David 2006/7/4, JJ : > > Hello. I have a very simple question. I would like > to generate a number of random variables from the chi2 > distribution. If I wanted these for the normal > distribution, the code could be > stats.norm.rvs(size=5,loc=100,scale=1). But > stats.chi2.rvs(size=5,df=10,loc=0,scale=1) or > stats.chi2.rvs(df=10,loc=0,scale=1) or > stats.chi2.rvs(df=10)does not work. Can anyone tell > me what the proper syntax would be for this? > Thanks JJ > > __________________________________________________ > Do You Yahoo!? > Tired of spam? Yahoo! Mail has the best spam protection around > http://mail.yahoo.com > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ferenc.pintye at steelsky.de Wed Jul 5 08:51:36 2006 From: ferenc.pintye at steelsky.de (ferenc pintye) Date: Wed, 5 Jul 2006 14:51:36 +0200 Subject: [Numpy-discussion] (no subject) Message-ID: <000c01c6a045$383f06c0$b90614ac@eu.decoma.com> Hi, how could i get the name of an array in a string ? thanks f. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexandre.fayolle at logilab.fr Wed Jul 5 11:49:58 2006 From: alexandre.fayolle at logilab.fr (Alexandre Fayolle) Date: Wed, 5 Jul 2006 17:49:58 +0200 Subject: [Numpy-discussion] (no subject) In-Reply-To: <000c01c6a045$383f06c0$b90614ac@eu.decoma.com> References: <000c01c6a045$383f06c0$b90614ac@eu.decoma.com> Message-ID: <20060705154958.GF26050@crater.logilab.fr> On Wed, Jul 05, 2006 at 02:51:36PM +0200, ferenc pintye wrote: > Hi, > > how could i get the name of an array in a string ? if by this you mean "the name of an identifier refering to [whatever]", this is a FAQ, and the answer is 1. you don't want to do that 2. your problem is not well behaved (there can be many answers to the questions) 3. you can use something along the lines of for name, item in locals().iteritems(): if item is my_array: break else: name = "anonymous object" -- Alexandre Fayolle LOGILAB, Paris (France) Formations Python, Zope, Plone, Debian: http://www.logilab.fr/formations D?veloppement logiciel sur mesure: http://www.logilab.fr/services Informatique scientifique: http://www.logilab.fr/science -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 481 bytes Desc: Digital signature URL: From robert.kern at gmail.com Wed Jul 5 11:51:25 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 05 Jul 2006 10:51:25 -0500 Subject: [Numpy-discussion] (no subject) In-Reply-To: References: Message-ID: Ferenc.Pintye at eu.decoma.com wrote: > > Hi, > > how could i get the name of an array in a string ? > (info command ?!) You can't. http://mail.python.org/pipermail/python-list/2005-June/286574.html -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From tim.hochberg at cox.net Wed Jul 5 18:14:44 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Wed, 05 Jul 2006 15:14:44 -0700 Subject: [Numpy-discussion] Dot as a method In-Reply-To: References: Message-ID: <44AC39D4.3030104@cox.net> Bart Vandereycken wrote: > Hi all, > > reading the thread "Ransom proposals" I was wondering why there isn't a > ndarray.dot() method? There is already a scipy.sparse.dot() so this > would fit nicely in the whole idea of polymorphism. > Are you sure about that? The problem with a dot method (aside from a over proliferation of methods in general) is that to be done correctly you want to choose a particular implementation of dot based *both* of its arguments. A method does a good job dispatching on a single argument, but falls down when dispatching on two types. Let's look at this specific case. Imagine that in addition ndarray.dot and sparse.dot, we also stick a dot method on ma.masked, etc. Now in order to fully exploit polymorphism and get maximum efficiency, we want asparsearray.dot(amaskedarray) to correctly treat the masked values (ma.dot treats them as zeros) and to not instantiate a dense version of the sparsearray. But in order to do that all three classes need to know about the other two. That's possible, if messy since all three of these are known in advance, but this approach becomes untenable if you classes outside core numpy or scipy to participate as full citizens. What does appear fit well here is generic functions / multimethods / protocols as discussed in some detail on pydev-3000 a couple of months ago. This would allow classes defined outside of core numpy and scipy to work correctly and efficiently with dot as long as they register appropriate versions of dot. If I wasn't swamped right at the moment I'd prototype this up and see how it works in practice. -tim From bhendrix at enthought.com Wed Jul 5 19:58:31 2006 From: bhendrix at enthought.com (Bryce Hendrix) Date: Wed, 05 Jul 2006 18:58:31 -0500 Subject: [Numpy-discussion] ANN: Python Enthought Edition Version 1.0.0.beta3 Released Message-ID: <44AC5227.9030902@enthought.com> Enthought is pleased to announce the release of Python Enthought Edition Version 1.0.0.beta3 (http://code.enthought.com/enthon/) -- a python distribution for Windows. 1.0.0.beta3 Release Notes: -------------------- Version 1.0.0.beta3 of Python Enthought Edition is the first version based on Python 2.4.3 and includes updates to nearly every package. This is the third and (hopefully) last beta release. This release includes version 1.0.8 of the Enthought Tool Suite (ETS) Package and bug fixes-- you can look at the release notes for this ETS version here: http://svn.enthought.com/downloads/enthought/changelog-release.1.0.8.html About Python Enthought Edition: ------------------------------- Python 2.4.3, Enthought Edition is a kitchen-sink-included Python distribution for Windows including the following packages out of the box: Numeric SciPy IPython Enthought Tool Suite wxPython PIL mingw f2py MayaVi Scientific Python VTK and many more... More information is available about all Open Source code written and released by Enthought, Inc. at http://code.enthought.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Wed Jul 5 20:13:29 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Wed, 5 Jul 2006 17:13:29 -0700 Subject: [Numpy-discussion] tentative numpy tutorial In-Reply-To: <6ef8f3380607040225t30948964o49b0f570b8768edf@mail.gmail.com> References: <6ef8f3380607040225t30948964o49b0f570b8768edf@mail.gmail.com> Message-ID: On 7/4/06, Pau Gargallo wrote: > motivated by the lack of free documentation for NumPy, with some > friends, we started writing a tutorial, that we would like to see in > scipy.org. After some time, the project have started to loose its > initial impetus. Now, we put the current unfinished version in > > http://www.scipy.org/Tentative_NumPy_Tutorial > > with the hope that, with the help of other users, it will be possible > to build a useful document for beginners. I think it looks great. From myeates at jpl.nasa.gov Wed Jul 5 20:37:32 2006 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Wed, 05 Jul 2006 17:37:32 -0700 Subject: [Numpy-discussion] simple question Message-ID: <44AC5B4C.10804@jpl.nasa.gov> What is the typical way of doing the following starting with a 0 matrix, set all values to 1 when a certain condition is met, set to -1 when another condition is met, left alone if neither condition is met. Mathew From kwgoodman at gmail.com Wed Jul 5 20:45:10 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Wed, 5 Jul 2006 17:45:10 -0700 Subject: [Numpy-discussion] simple question In-Reply-To: <44AC5B4C.10804@jpl.nasa.gov> References: <44AC5B4C.10804@jpl.nasa.gov> Message-ID: On 7/5/06, Mathew Yeates wrote: > What is the typical way of doing the following > starting with a 0 matrix, set all values to 1 when a certain condition > is met, set to -1 when another condition is met, left alone if neither > condition is met. This works on recent versions of numpy: >> x = asmatrix(zeros((2,2))) >> x matrix([[0, 0], [0, 0]]) >> y = asmatrix(rand(2,2)) >> y matrix([[ 0.85219404, 0.48311427], [ 0.41026966, 0.2184193 ]]) >> x[y > 0.5] = 1 >> x[y < 0.5] = -1 >> x matrix([[ 1, -1], [-1, -1]]) From lists.steve at arachnedesign.net Wed Jul 5 20:45:14 2006 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Wed, 5 Jul 2006 20:45:14 -0400 Subject: [Numpy-discussion] tentative numpy tutorial In-Reply-To: References: <6ef8f3380607040225t30948964o49b0f570b8768edf@mail.gmail.com> Message-ID: <29506E7D-02A3-4C71-AEE6-FEA30D701042@arachnedesign.net> >> motivated by the lack of free documentation for NumPy, with some >> friends, we started writing a tutorial, that we would like to see in >> scipy.org. After some time, the project have started to loose its >> initial impetus. Now, we put the current unfinished version in >> >> http://www.scipy.org/Tentative_NumPy_Tutorial >> >> with the hope that, with the help of other users, it will be possible >> to build a useful document for beginners. > > I think it looks great. Agreed .. thanks for whipping that together! -steve From myeates at jpl.nasa.gov Wed Jul 5 20:47:02 2006 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Wed, 05 Jul 2006 17:47:02 -0700 Subject: [Numpy-discussion] simple question In-Reply-To: References: <44AC5B4C.10804@jpl.nasa.gov> Message-ID: <44AC5D86.9050201@jpl.nasa.gov> ohhhh. I was looking at using "where" Keith Goodman wrote: > On 7/5/06, Mathew Yeates wrote: >> What is the typical way of doing the following >> starting with a 0 matrix, set all values to 1 when a certain condition >> is met, set to -1 when another condition is met, left alone if neither >> condition is met. > > This works on recent versions of numpy: > >>> x = asmatrix(zeros((2,2))) > >>> x > > matrix([[0, 0], > [0, 0]]) > >>> y = asmatrix(rand(2,2)) > >>> y > > matrix([[ 0.85219404, 0.48311427], > [ 0.41026966, 0.2184193 ]]) > >>> x[y > 0.5] = 1 > >>> x[y < 0.5] = -1 > >>> x > > matrix([[ 1, -1], > [-1, -1]]) > From paustin at eos.ubc.ca Wed Jul 5 20:51:04 2006 From: paustin at eos.ubc.ca (Philip Austin) Date: Wed, 5 Jul 2006 17:51:04 -0700 Subject: [Numpy-discussion] A C++ library and the new array interface: the best approach? In-Reply-To: <44A9A302.90901@ieee.org> References: <44A99D07.5030706@colorado.edu> <44A9A302.90901@ieee.org> Message-ID: <17580.24184.539788.121229@eos.ubc.ca> Travis Oliphant writes: > b) byte-the bullet and use Boost python (which other's who have > integrated C++ code with Python seem to really like). I've never > wrapped my mind around Boost python, though, so I'm not much help there. I've updated my set of boost helper functions to work with numpy -- there's a Readme at: http://www.eos.ubc.ca/research/clouds/software/pythonlibs/num_util/num_util_release2 (http://tinyurl.com/eo6fe) A simple example which shows some of the boost python features (reference counting, C++ versions of python types, docstrings, exceptions) is appended below. When called with python script http://tinyurl.com/zy57a it produces the output http://tinyurl.com/fr7xo Regards, Phil Austin -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: simpletest.cpp URL: From tom.denniston at alum.dartmouth.org Wed Jul 5 20:58:37 2006 From: tom.denniston at alum.dartmouth.org (Tom Denniston) Date: Wed, 5 Jul 2006 19:58:37 -0500 Subject: [Numpy-discussion] PyObject arrays, pickling, tofile Message-ID: It seems from my cursory look that numpy doesn't support the pickling or the tofile of PyObject arrays. I was wondering whether such functionlity is in the pipeline or if there is little interest. Personally, I would really like to see the feature in numpy but don't know if I am up to implementing it myself. --Tom From wbaxter at gmail.com Wed Jul 5 21:03:19 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu, 6 Jul 2006 10:03:19 +0900 Subject: [Numpy-discussion] simple question In-Reply-To: <44AC5D86.9050201@jpl.nasa.gov> References: <44AC5B4C.10804@jpl.nasa.gov> <44AC5D86.9050201@jpl.nasa.gov> Message-ID: That's the kind of thing that should maybe be on the NumpyExample list: http://www.scipy.org/Numpy_Example_List But currently little syntactical tricks like that aren't listed there. I'm not sure how you'd list that even. There is an example like that ( a[a<0.5]=0 ) on the Numpy for Matlab Users page, though. http://www.scipy.org/NumPy_for_Matlab_Users It's supposed to be for matlab users, but if you just ignore the Matlab column, the main table there is still a pretty good list of examples of numpy syntax and basic functions. --bb On 7/6/06, Mathew Yeates wrote: > > ohhhh. I was looking at using "where" > > > > Keith Goodman wrote: > > On 7/5/06, Mathew Yeates wrote: > >> What is the typical way of doing the following > >> starting with a 0 matrix, set all values to 1 when a certain condition > >> is met, set to -1 when another condition is met, left alone if neither > >> condition is met. > > > > This works on recent versions of numpy: > > > >>> x = asmatrix(zeros((2,2))) > > > >>> x > > > > matrix([[0, 0], > > [0, 0]]) > > > >>> y = asmatrix(rand(2,2)) > > > >>> y > > > > matrix([[ 0.85219404, 0.48311427], > > [ 0.41026966, 0.2184193 ]]) > > > >>> x[y > 0.5] = 1 > > > >>> x[y < 0.5] = -1 > > > >>> x > > > > matrix([[ 1, -1], > > [-1, -1]]) > > > > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- William V. Baxter III OLM Digital Kono Dens Building Rm 302 1-8-8 Wakabayashi Setagaya-ku Tokyo, Japan 154-0023 +81 (3) 3422-3380 -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Wed Jul 5 22:39:19 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu, 6 Jul 2006 11:39:19 +0900 Subject: [Numpy-discussion] Determining if two arrays share data Message-ID: Often when I'm doing interactive prototyping I find myself wanting to check whether two arrays are sharing a copy of the same data. It seems like there ought to be a concise way to do that, but right now seems like the only way is with a little function like this: def same_base(a,b): ab = a.base if ab is None: ab = a bb = b.base if bb is None: bb = b return ab is bb is there some easier built-in way? Does the above function even cover all the bases? (so to speak...) It would be easier if a non-copy array pointed to itself as base rather than 'None'. Then you could just check a.base == b.base. --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From satgroup at postbox.auckland.ac.nz Wed Jul 5 22:54:30 2006 From: satgroup at postbox.auckland.ac.nz (Satellite Data Research Group) Date: Thu, 6 Jul 2006 14:54:30 +1200 Subject: [Numpy-discussion] Problems installing numpy0.9.8 on MacOSX (10.4) Message-ID: <20060706145430.t48yabubgdsscowo@webmail.auckland.ac.nz> Dear Sir/Madam I am trying to install numpy0.9.8 onto Mac version 10.4 and Python2.4 and am having trouble doing so. I had successfully installed inumpy0.9.8 on Python2.3, so am very confused as to why it is not working on Python2.4 After typing 'python setup.py build' and then 'python setup.py install' from the terminal I get a host of errors, whereas with Python2.3 it worked perfectly. Then in python typing 'from numpy import *' then trying something like 'array((range(10))' I get an error saying array cannot be found, whereas on Python2.3 this worked. I was wondering if anyone else had trouble installing numpy on Python2.4 using Mac 10.4, and if anyone could help me rectify the problem. Help would be greatly appreciated. Sincerely Stephen Johnston ----------------------------------------------------------------------- This mail sent through University of Auckland http://www.auckland.ac.nz From tim.leslie at gmail.com Wed Jul 5 23:00:08 2006 From: tim.leslie at gmail.com (Tim Leslie) Date: Thu, 6 Jul 2006 13:00:08 +1000 Subject: [Numpy-discussion] Problems installing numpy0.9.8 on MacOSX (10.4) In-Reply-To: <20060706145430.t48yabubgdsscowo@webmail.auckland.ac.nz> References: <20060706145430.t48yabubgdsscowo@webmail.auckland.ac.nz> Message-ID: On 7/6/06, Satellite Data Research Group wrote: > Dear Sir/Madam > > I am trying to install numpy0.9.8 onto Mac version 10.4 and Python2.4 > and am having trouble doing so. I had successfully installed > inumpy0.9.8 on Python2.3, so am very confused as to why it is not > working on Python2.4 > > After typing 'python setup.py build' and then 'python setup.py install' > from the terminal I get a host of errors, whereas with Python2.3 it > worked perfectly. Then in python typing 'from numpy import *' then > trying something like 'array((range(10))' I get an error saying array > cannot be found, whereas on Python2.3 this worked. > Hi Stephen, Could you please post the exact error messages you get. Without these it's not possible to diagnose your problem. Cheers, Tim. > I was wondering if anyone else had trouble installing numpy on > Python2.4 using Mac 10.4, and if anyone could help me rectify the > problem. Help would be greatly appreciated. > > Sincerely > > Stephen Johnston > > ----------------------------------------------------------------------- > This mail sent through University of Auckland http://www.auckland.ac.nz > > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From oliphant.travis at ieee.org Wed Jul 5 23:17:00 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 05 Jul 2006 21:17:00 -0600 Subject: [Numpy-discussion] PyObject arrays, pickling, tofile In-Reply-To: References: Message-ID: <44AC80AC.1030708@ieee.org> Tom Denniston wrote: > It seems from my cursory look that numpy doesn't support the pickling > or the tofile of PyObject arrays. Yes, you can pickle PyObject arrays, that is supposed to work. What is not working? You can't use tofile on PyObject arrays, because I didn't have clear idea what to do. Use Pickle for PyObject arrays. -Travis From satgroup at postbox.auckland.ac.nz Wed Jul 5 23:23:50 2006 From: satgroup at postbox.auckland.ac.nz (Satellite Data Research Group) Date: Thu, 6 Jul 2006 15:23:50 +1200 Subject: [Numpy-discussion] Numpy not working (revised) Message-ID: <20060706152350.fgppt3uyop6o848k@webmail.auckland.ac.nz> Dear all I am trying to get numpy working with MacOSX 10.4 on an intel processor. I had it working for Python2.3.5, but after installing Python 2.4 it didn't work on either python version. After inputing: sudo python setup.py build and then sudo python setup.py install and also trying to load up a basic array, I get the following errors. The errors that I get repeatedly during the installation process seem to mainly be (without showing you the entire massive output, unless you need to see it): building extension "numpy.core.umath" sources adding 'build/src.darwin-8.7.1-i386-2.3/numpy/core/config.h' to sources. adding 'build/src.darwin-8.7.1-i386-2.3/numpy/core/__ufunc_api.h' to sources. adding 'build/src.darwin-8.7.1-i386-2.3/numpy/core/src' to include_dirs. numpy.core - nothing done with h_files= ['build/src.darwin-8.7.1-i386-2.3/numpy/core/config.h', 'build/src.darwin-8.7.1-i386-2.3/numpy/core/__multiarray_api.h', 'build/src.darwin-8.7.1-i386-2.3/numpy/core/__ufunc_api.h'] building extension "numpy.core._dotblas" sources adding 'numpy/core/blasdot/_dotblas.c' to sources. building extension "numpy.lib._compiled_base" sources building extension "numpy.dft.fftpack_lite" sources building extension "numpy.linalg.lapack_lite" sources adding 'numpy/linalg/lapack_litemodule.c' to sources. building extension "numpy.random.mtrand" sources customize NAGFCompiler customize AbsoftFCompiler customize IbmFCompiler Could not locate executable g77 Could not locate executable f77 Could not locate executable gfortran Could not locate executable f95 customize GnuFCompiler customize Gnu95FCompiler customize G95FCompiler customize GnuFCompiler customize Gnu95FCompiler customize NAGFCompiler customize NAGFCompiler using config gcc options: '-fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 -Wall -Wstrict-prototypes' compile options: '-Inumpy/core/src -Inumpy/core/include -I/System/Library/Frameworks/Python.framework/Versions/2.3/include/python2.3 -c' gcc: _configtest.c _configtest.c:7:2: error: #error No _WIN32 _configtest.c:7:2: error: #error No _WIN32 The above lines of output are repeated a few times. If it makes any difference, which I am sure that it does, I am running on an intel chip. In Python, when typing >>> from numpy import * I get: Running from numpy source directory. Then trying to make an array, I get: >>> vector = array((2,3,4,5,6)) Traceback (most recent call last): File "", line 1, in ? NameError: name 'array' is not defined This is in Python2.3.5. However, my collegue had numpy working on Python2.3.5, with the exact same machine just now. Just for background information, we are needing numpy to use pyhdf, to load and analyse hdf files. Thank you so much for your help. Sincerely Stephen Johnston p.s my apologies for the multiple requests, this one just a revised version of my previous attempt at asking for help ----------------------------------------------------------------------- This mail sent through University of Auckland http://www.auckland.ac.nz From oliphant.travis at ieee.org Wed Jul 5 23:48:18 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 05 Jul 2006 21:48:18 -0600 Subject: [Numpy-discussion] Numpy not working (revised) In-Reply-To: <20060706152350.fgppt3uyop6o848k@webmail.auckland.ac.nz> References: <20060706152350.fgppt3uyop6o848k@webmail.auckland.ac.nz> Message-ID: <44AC8802.2010105@ieee.org> Satellite Data Research Group wrote: > Dear all > > I am trying to get numpy working with MacOSX 10.4 on an intel > processor. I had it working for Python2.3.5, but after installing > Python 2.4 it didn't work on either python version. > > After inputing: > sudo python setup.py build > and then > sudo python setup.py install > > and also trying to load up a basic array, I get the following errors. > The errors that I get repeatedly during the installation process seem > to mainly be (without showing you the entire massive output, unless you > need to see it): > > > building extension "numpy.core.umath" sources > adding 'build/src.darwin-8.7.1-i386-2.3/numpy/core/config.h' to sources. > adding 'build/src.darwin-8.7.1-i386-2.3/numpy/core/__ufunc_api.h' to sources. > adding 'build/src.darwin-8.7.1-i386-2.3/numpy/core/src' to include_dirs. > numpy.core - nothing done with h_files= > ['build/src.darwin-8.7.1-i386-2.3/numpy/core/config.h', > 'build/src.darwin-8.7.1-i386-2.3/numpy/core/__multiarray_api.h', > 'build/src.darwin-8.7.1-i386-2.3/numpy/core/__ufunc_api.h'] > > building extension "numpy.core._dotblas" sources > adding 'numpy/core/blasdot/_dotblas.c' to sources. > building extension "numpy.lib._compiled_base" sources > building extension "numpy.dft.fftpack_lite" sources > building extension "numpy.linalg.lapack_lite" sources > adding 'numpy/linalg/lapack_litemodule.c' to sources. > building extension "numpy.random.mtrand" sources > customize NAGFCompiler > customize AbsoftFCompiler > customize IbmFCompiler > Could not locate executable g77 > Could not locate executable f77 > Could not locate executable gfortran > Could not locate executable f95 > > customize GnuFCompiler > customize Gnu95FCompiler > customize G95FCompiler > customize GnuFCompiler > customize Gnu95FCompiler > customize NAGFCompiler > customize NAGFCompiler using config > gcc options: '-fno-strict-aliasing -Wno-long-double -no-cpp-precomp > -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 -Wall > -Wstrict-prototypes' > compile options: '-Inumpy/core/src -Inumpy/core/include > -I/System/Library/Frameworks/Python.framework/Versions/2.3/include/python2.3 > -c' > gcc: _configtest.c > > _configtest.c:7:2: error: #error No _WIN32 > _configtest.c:7:2: error: #error No _WIN32 > > The above lines of output are repeated a few times. If it makes any > difference, which I am sure that it does, I am running on an intel chip. > > This is not a problem. These are "configuration" "error" that the system uses to incorporate platform-specific code. > In Python, when typing > >>>> from numpy import * >>>> > I get: > Running from numpy source directory. > > This is the problem. The error is telling you that you can't import numpy from the directory where numpy was built (the import mechanism gets confused between the system-installed numpy and the directory structure underneath). Just exit Python and cd to another directory. Then numpy should work fine. -Travis From lists.steve at arachnedesign.net Wed Jul 5 23:56:14 2006 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Wed, 5 Jul 2006 23:56:14 -0400 Subject: [Numpy-discussion] Numpy not working (revised) In-Reply-To: <20060706152350.fgppt3uyop6o848k@webmail.auckland.ac.nz> References: <20060706152350.fgppt3uyop6o848k@webmail.auckland.ac.nz> Message-ID: <768C2AFA-2F66-4C27-A924-5B19746C796D@arachnedesign.net> > I am trying to get numpy working with MacOSX 10.4 on an intel > processor. I had it working for Python2.3.5, but after installing > Python 2.4 it didn't work on either python version. Hi Stephen, I've been able to get numpy/scipy installed on my MacBook Pro (although I do get three unit test failures when I run nump.test(1) -- which I'll post next) One thing I'm noticing in what you printed out is that for some reason your doing something (linking?) to the version of Python that comes with your mac (2.3). Look at this output that you posted: > -I/System/Library/Frameworks/Python.framework/Versions/2.3/include/ > python2.3 Maybe you should do a: "python setups.py clean" and also remove your numpy build directory and start over. I guess you also have to make sure that all of your python 2.4 binaries come first (in terms of environment var/PATH) before the 2.3 binaries. Also .. one more thing, I also noticed that you didn't have any fortran compilers in your path, you can get gfortran like so: $curl -O http://easynews.dl.sourceforge.net/sourceforge/hpc/gfortran- intel-bin.tar.gz $ sudo tar -C / -xzf gfortran-intel-bin.tar.gz and it'll put gfortran in your /usr/local/bin (I got that info from http://bmistudents.blogspot.com/2006/04/on-installation-of-proper- python.html) -steve From lists.steve at arachnedesign.net Thu Jul 6 00:02:16 2006 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Thu, 6 Jul 2006 00:02:16 -0400 Subject: [Numpy-discussion] Mac OS 10.4 test errors Message-ID: <47338856-A9EA-4A08-A14B-6581CE1CD429@arachnedesign.net> Hi folks, I'm able to build and install numpy on my intel mac OS 10.4.7 but I'm getting 3 unit test failures. I'm using gcc 4.0.1 and GNU Fortran 95 (GCC) 4.2.0 20060401 (if that's helpful to know). The relevant output from my ipython session is below. Doing some of the stuff I've found in random tutorials doesn't seem to be a problem .. though I do get seg faults when trying to do some scipy tutorial plotting stuff .. which I may post to the scipy list a bit later ... I just want to make sure I'm installing everything correctly first. Thanks for any insight, -steve In [3]: numpy.__version__ Out[3]: '0.9.9.2735' In [4]: numpy.test(1) .... ... cut some stuff out until first error .. ... ====================================================================== ERROR: check_basic (numpy.core.tests.test_defmatrix.test_algebra) ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/local/lib/python2.4/site-packages/numpy/core/tests/ test_defmatrix.py", line 125, in check_basic Ainv = linalg.inv(A) File "/opt/local/lib/python2.4/site-packages/numpy/linalg/ linalg.py", line 122, in inv return wrap(solve(a, identity(a.shape[0]))) File "/opt/local/lib/python2.4/site-packages/numpy/linalg/ linalg.py", line 109, in solve results = lapack_routine(n_eq, n_rhs, a, n_eq, pivots, b, n_eq, 0) LapackError: Parameter ipiv is not of type PyArray_INT in lapack_lite.dgesv ====================================================================== ERROR: check_instance_methods (numpy.core.tests.test_defmatrix.test_matrix_return) ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/local/lib/python2.4/site-packages/numpy/core/tests/ test_defmatrix.py", line 157, in check_instance_methods f = eval('a.%s' % attrib) File "", line 0, in ? File "/opt/local/lib/python2.4/site-packages/numpy/core/ defmatrix.py", line 293, in getI return matrix(inv(self)) File "/opt/local/lib/python2.4/site-packages/numpy/linalg/ linalg.py", line 122, in inv return wrap(solve(a, identity(a.shape[0]))) File "/opt/local/lib/python2.4/site-packages/numpy/linalg/ linalg.py", line 109, in solve results = lapack_routine(n_eq, n_rhs, a, n_eq, pivots, b, n_eq, 0) LapackError: Parameter ipiv is not of type PyArray_INT in lapack_lite.dgesv ====================================================================== ERROR: check_basic (numpy.core.tests.test_defmatrix.test_properties) ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/local/lib/python2.4/site-packages/numpy/core/tests/ test_defmatrix.py", line 48, in check_basic assert allclose(linalg.inv(A), mA.I) File "/opt/local/lib/python2.4/site-packages/numpy/linalg/ linalg.py", line 122, in inv return wrap(solve(a, identity(a.shape[0]))) File "/opt/local/lib/python2.4/site-packages/numpy/linalg/ linalg.py", line 109, in solve results = lapack_routine(n_eq, n_rhs, a, n_eq, pivots, b, n_eq, 0) LapackError: Parameter ipiv is not of type PyArray_INT in lapack_lite.dgesv ---------------------------------------------------------------------- From satgroup at postbox.auckland.ac.nz Thu Jul 6 01:02:41 2006 From: satgroup at postbox.auckland.ac.nz (Satellite Data Research Group) Date: Thu, 6 Jul 2006 17:02:41 +1200 Subject: [Numpy-discussion] Numpy import problem Message-ID: <20060706170241.hq0fq9gssdys8kg4@webmail.auckland.ac.nz> Hi, I have a problem when importing numpy into python 2.4.1. I did a successful (I think) build and install of Numpy 0.9.8 on a mac osx (10.4) intel, but when i try to import numpy into python i get this error: >>> from numpy import* import core -> failed: Inappropriate file type for dynamic loading import lib -> failed: Inappropriate file type for dynamic loading import linalg -> failed: Inappropriate file type for dynamic loading import dft -> failed: Inappropriate file type for dynamic loading import random -> failed: Inappropriate file type for dynamic loading Traceback (most recent call last): File "", line 1, in -toplevel- from numpy import* File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/numpy/__init__.py", line 49, in -toplevel- import add_newdocs File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/numpy/add_newdocs.py", line 2, in -toplevel- from lib import add_newdoc File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/numpy/lib/__init__.py", line 5, in -toplevel- from type_check import * File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/numpy/lib/type_check.py", line 8, in -toplevel- import numpy.core.numeric as _nx File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/numpy/core/__init__.py", line 5, in -toplevel- import multiarray ImportError: Inappropriate file type for dynamic loading Any help anyone could offer as to the cause of this would be most welcome, Cheers, Joe Corbett ps. installing numpy through python 2.3 worked and I was able to import numpy into 2.3 fine. ----------------------------------------------------------------------- This mail sent through University of Auckland http://www.auckland.ac.nz From stefan at sun.ac.za Thu Jul 6 03:21:18 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu, 6 Jul 2006 09:21:18 +0200 Subject: [Numpy-discussion] Determining if two arrays share data In-Reply-To: References: Message-ID: <20060706072118.GB1070@mentat.za.net> On Thu, Jul 06, 2006 at 11:39:19AM +0900, Bill Baxter wrote: > Often when I'm doing interactive prototyping I find myself wanting to check > whether two arrays are sharing a copy of the same data. > > It seems like there ought to be a concise way to do that, but right now seems > like the only way is with a little function like this: > > def same_base(a,b): > ab = a.base > if ab is None: ab = a > bb = b.base > if bb is None: bb = b > return ab is bb > > is there some easier built-in way? Does the above function even cover all the > bases? (so to speak...) Say you have x = N.array([1,2,3,4]) and y = x.reshape((2,2)) then x and y share the same data. You can see this when you do x.__array_interface__['data'][0] == y.__array_interface__['data'][0] Still, this only holds for full data views. If you had z = y[1:,1:] then the data memory position would differ. Cheers St?fan From fullung at gmail.com Thu Jul 6 04:08:36 2006 From: fullung at gmail.com (Albert Strasheim) Date: Thu, 6 Jul 2006 10:08:36 +0200 Subject: [Numpy-discussion] Comparing arrays with dtypes with field names Message-ID: <00ef01c6a0d3$61b34500$0a84a8c0@dsp.sun.ac.za> Hello all I noticed that arrays with dtypes with field names don't compare correctly when using the equality operator. Example: In [17]: dt = N.dtype({'names' : ['f1', 'f2'], 'formats' : [N.intc, N.intc]}) In [18]: x = N.array([(0,0)], dtype=dt) In [19]: y = N.array([(0,0)], dtype=dt) In [20]: x['f1']==y['f1'] Out[20]: array([True], dtype=bool) In [21]: x['f2']==y['f2'] Out[21]: array([True], dtype=bool) In [22]: x==y Out[22]: False The question arises as to what should be returned when comparing these arrays. One option would be an array with dtype=bool which would indicate that each item in the array is equal in f1 and f2. Another option would be an array with dtype N.dtype({'names' : ['f1', 'f2'], 'formats' : [bool,bool]}) I think this second option could be very useful for some applications, where you might want to know something like: is the (i,j)th element of this array the same in f1 and f2 but different in f3? Any thoughts? Regards, Albert From oliphant.travis at ieee.org Thu Jul 6 04:20:46 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 06 Jul 2006 02:20:46 -0600 Subject: [Numpy-discussion] Comparing arrays with dtypes with field names In-Reply-To: <00ef01c6a0d3$61b34500$0a84a8c0@dsp.sun.ac.za> References: <00ef01c6a0d3$61b34500$0a84a8c0@dsp.sun.ac.za> Message-ID: <44ACC7DE.8080708@ieee.org> Albert Strasheim wrote: > Hello all > > I noticed that arrays with dtypes with field names don't compare correctly > when using the equality operator. > > Example: > > In [17]: dt = N.dtype({'names' : ['f1', 'f2'], > 'formats' : [N.intc, N.intc]}) > > In [18]: x = N.array([(0,0)], dtype=dt) > > In [19]: y = N.array([(0,0)], dtype=dt) > > In [20]: x['f1']==y['f1'] > Out[20]: array([True], dtype=bool) > > In [21]: x['f2']==y['f2'] > Out[21]: array([True], dtype=bool) > > In [22]: x==y > Out[22]: False > > The question arises as to what should be returned when comparing these > arrays. One option would be an array with dtype=bool which would indicate > that each item in the array is equal in f1 and f2. Currently, in SVN, record-arrays now can be compared (== or != only). The comparison returns a single bool array: For equality the comparison is basically: res = True for name in x.dtype.fields[-1]: res = logical_and(res, x[name] == y[name]) while for not-equal comparisons it is: res = False for name in x.dtype.fields[-1]: res = logical_or(res, x[name] != y[name]) > Another option would be > an array with dtype > > N.dtype({'names' : ['f1', 'f2'], 'formats' : [bool,bool]}) > > I think this second option could be very useful for some applications, where > you might want to know something like: is the (i,j)th element of this array > the same in f1 and f2 but different in f3? > This is a bit too sophisticated and does not work very well for nested arrays. The implemented comparison works for nested arrays just fine. You can just test for what you want directly (x['f1'] == y['f2']) & (x['f1'] != y['f3']) to handle the fewer particular cases where the other approach might be useful. -Travis From pau.gargallo at gmail.com Thu Jul 6 04:50:44 2006 From: pau.gargallo at gmail.com (Pau Gargallo) Date: Thu, 6 Jul 2006 10:50:44 +0200 Subject: [Numpy-discussion] simple question In-Reply-To: References: <44AC5B4C.10804@jpl.nasa.gov> <44AC5D86.9050201@jpl.nasa.gov> Message-ID: <6ef8f3380607060150ucaab1d8v8df523a05d9b6d2@mail.gmail.com> On 7/6/06, Bill Baxter wrote: > That's the kind of thing that should maybe be on the NumpyExample list: > http://www.scipy.org/Numpy_Example_List > But currently little syntactical tricks like that aren't listed there. I'm > not sure how you'd list that even. > There is a (tentative) introduction to 'fancy' indexing in the tentative numpy tutorial. http://www.scipy.org/Tentative_NumPy_Tutorial#head-0dffc419afa7d77d51062d40d2d84143db8216c2 i'm going to add the '[ ]' operator to the Numpy example list. pau From rob at hooft.net Thu Jul 6 05:03:15 2006 From: rob at hooft.net (Rob W.W. Hooft) Date: Thu, 06 Jul 2006 11:03:15 +0200 Subject: [Numpy-discussion] Determining if two arrays share data In-Reply-To: <20060706072118.GB1070@mentat.za.net> References: <20060706072118.GB1070@mentat.za.net> Message-ID: <44ACD1D3.9060608@hooft.net> Stefan van der Walt wrote: > On Thu, Jul 06, 2006 at 11:39:19AM +0900, Bill Baxter wrote: > >>Often when I'm doing interactive prototyping I find myself wanting to check >>whether two arrays are sharing a copy of the same data. > > > Say you have > > x = N.array([1,2,3,4]) > > and > > y = x.reshape((2,2)) > > then x and y share the same data. You can see this when you do > > x.__array_interface__['data'][0] == y.__array_interface__['data'][0] > > Still, this only holds for full data views. If you had > > z = y[1:,1:] > > then the data memory position would differ. I am still using Numeric, but even with that background I think that the problem is ill-defined. Consider the following interactive session: Python 2.3b1+ (#2, Jun 10 2003, 20:53:51) [GCC 3.0.2 20010905 (Red Hat Linux 7.1 3.0.1-3)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import Numeric >>> a=Numeric.arange(20) >>> b=a[::2] >>> c=a[1::2] >>> a array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]) >>> b array([ 0, 2, 4, 6, 8, 10, 12, 14, 16, 18]) >>> c array([ 1, 3, 5, 7, 9, 11, 13, 15, 17, 19]) >>> b[0]=99 >>> c[0]=99 >>> a array([99, 99, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]) Do "b" and "c" share a copy of the same data? None of the values in either array is visible in the other, but both share data with "a"! Regards, Rob -- Rob W.W. Hooft || rob at hooft.net || http://www.hooft.net/people/rob/ From fullung at gmail.com Thu Jul 6 05:45:02 2006 From: fullung at gmail.com (Albert Strasheim) Date: Thu, 6 Jul 2006 11:45:02 +0200 Subject: [Numpy-discussion] Determining if two arrays share data In-Reply-To: <44ACD1D3.9060608@hooft.net> Message-ID: <00fd01c6a0e0$da387790$0a84a8c0@dsp.sun.ac.za> Hello all > > Python 2.3b1+ (#2, Jun 10 2003, 20:53:51) > [GCC 3.0.2 20010905 (Red Hat Linux 7.1 3.0.1-3)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import Numeric > >>> a=Numeric.arange(20) > >>> b=a[::2] > >>> c=a[1::2] > >>> a > array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, > 16, 17, 18, > 19]) > >>> b > array([ 0, 2, 4, 6, 8, 10, 12, 14, 16, 18]) > >>> c > array([ 1, 3, 5, 7, 9, 11, 13, 15, 17, 19]) > >>> b[0]=99 > >>> c[0]=99 > >>> a > array([99, 99, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, > 16, 17, 18, > 19]) > > > Do "b" and "c" share a copy of the same data? None of the values in > either array is visible in the other, but both share data with "a"! I think we might be talking about two related but different concepts. One is sharing of data between arrays, the other is whether the data overlaps. Let's assume we can get at the starting address of the array in memory via the array interface or whatever, and the length of the array in bytes. To determine whether two arrays overlap, find the smallest data address of the two arrays. If the data address of the other array is smaller than the sum of the smallest data address and its corresponding length, you have overlap. I'm not quite sure how to check whether two arrays share the same underlying data buffer. Flags don't tell the whole story here. Regards, Albert From zhang.le.misc at gmail.com Thu Jul 6 06:31:17 2006 From: zhang.le.misc at gmail.com (Zhang Le) Date: Thu, 6 Jul 2006 11:31:17 +0100 Subject: [Numpy-discussion] bug in svn/numpy/numpy/doc/swig/numpy.i Message-ID: <4e7ed7700607060331i45969753v4b609237467578d2@mail.gmail.com> Hi, I found a bug in svn/numpy/numpy/doc/swig/numpy.i: a ')' is missed at line 72 which causes a compilation error: if (is_array(input) && (typecode == PyArray_NOTYPE || PyArray_EquivTypenums(array_type(input), typecode))) { ^ missing ary = (PyArrayObject*) input; Cheers, Zhang Le From rob at hooft.net Thu Jul 6 07:09:55 2006 From: rob at hooft.net (Rob W.W. Hooft) Date: Thu, 06 Jul 2006 13:09:55 +0200 Subject: [Numpy-discussion] Determining if two arrays share data In-Reply-To: <00fd01c6a0e0$da387790$0a84a8c0@dsp.sun.ac.za> References: <00fd01c6a0e0$da387790$0a84a8c0@dsp.sun.ac.za> Message-ID: <44ACEF83.4060503@hooft.net> Albert Strasheim wrote: > I think we might be talking about two related but different concepts. One is > sharing of data between arrays, the other is whether the data overlaps. When is it useful to know whether data buffers overlap? As long as they are disjoint, it should be irrelevant at the higher level of the program. > Let's assume we can get at the starting address of the array in memory via > the array interface or whatever, and the length of the array in bytes. > > To determine whether two arrays overlap, find the smallest data address of > the two arrays. If the data address of the other array is smaller than the > sum of the smallest data address and its corresponding length, you have > overlap. Don't forget the strides. Rob -- Rob W.W. Hooft || rob at hooft.net || http://www.hooft.net/people/rob/ From oliphant.travis at ieee.org Thu Jul 6 07:17:49 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 06 Jul 2006 05:17:49 -0600 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: Message-ID: <44ACF15D.6030607@ieee.org> Bill Baxter wrote: > So in short my proposal is to: > -- make a.T a property of array that returns a.swapaxes(-2,-1), > -- make a.H a property of array that returns > a.conjugate().swapaxes(-2,-1) > and maybe > -- make a.M a property of array that returns numpy.asmatrix(a) I've tentatively implemented all of these suggestions as well as adding the .A attribute to the ndarray as well (so that all sub-classes and array scalars can get back a view as an ndarray). I did this to make it easier to do matrix-like calculations with or with-out matrices. Matrix-calculation flexibility is still a sore-spot for many and I think these syntatical-sugar attributes will help long term. If there are no strong objections, then the recent MATH attribute checkin will stay. If there are major objections, then we can back them out without too much trouble as well. -Travis From oliphant.travis at ieee.org Thu Jul 6 07:30:52 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 06 Jul 2006 05:30:52 -0600 Subject: [Numpy-discussion] Speed degression? In-Reply-To: <200607041519.24636.steffen.loeck@gmx.de> References: <200607041519.24636.steffen.loeck@gmx.de> Message-ID: <44ACF46C.8050507@ieee.org> Steffen Loeck wrote: > Hi all, > > i made some speed tests using the sin-function and the %-operation to compare > Numeric, numpy 0.9.8 and numpy 0.9.9.2732. > As result the latest numpy version seems to be very slow in comparison to the > two other candidates. > > Results (in usec per loop): > sin-array mod-array > > Numeric 134 18 > > numpy 0.9.8 97 55 > > numpy 0.9.9.2732 204 316 > > numpy 0.9.8 + math 38 > > numpy 0.9.9.2732 + math 161 > > Numeric + math 23 > > > The used scripts can be found at the end. > > Can anyone verify my results and explain the observed speed degression? > I can't verify the results, but it's possible that the indexing code for a single integer is a bit slower. However, you forgot to test the "use Python lists" approach which is the fastest of all on this code. python -m timeit -s "x = [0.1]*10;" "for i in xrange(9): x[i+1]=(x[i]+1.1)%(1.0)" Using Numeric/NumPy is always slower than lists if you are doing element-by-element processing. NumPy is slower than Numeric for such processing. But, Numeric was always slower than pure Python too for such processing. So, the moral of the story is: "don't compute like that with NumPy (or Numeric)" With that said. I can add back the 1-d integer check in the indexing code to see if that speeds things up... -Travis From svetosch at gmx.net Thu Jul 6 07:45:54 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Thu, 06 Jul 2006 13:45:54 +0200 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44ACF15D.6030607@ieee.org> References: <44ACF15D.6030607@ieee.org> Message-ID: <44ACF7F2.1050904@gmx.net> Travis Oliphant schrieb: > Bill Baxter wrote: >> So in short my proposal is to: >> -- make a.T a property of array that returns a.swapaxes(-2,-1), >> -- make a.H a property of array that returns >> a.conjugate().swapaxes(-2,-1) >> and maybe >> -- make a.M a property of array that returns numpy.asmatrix(a) > > I've tentatively implemented all of these suggestions as well as adding > the .A attribute to the ndarray as well (so that all sub-classes and > array scalars can get back a view as an ndarray). > > I did this to make it easier to do matrix-like calculations with or > with-out matrices. Matrix-calculation flexibility is still a sore-spot > for many and I think these syntatical-sugar attributes will help long term. > I think this is great, thanks to Bill for suggesting it and to Travis for implementing it! So the only convenience of matrices over pure arrays that remains (afaics): -) .I for inverse; actually, why not add that to arrays as well as "syntactic sugar"? -) * being the matrix product instead of element-wise; Now, I could live with using dot and I don't want to push anything, but maybe this is the right time to consider another operator symbol as a shortcut for the dot function to be used with arrays? (Unfortunately right now I can't think of any sensible character(s) for that...) -) ** analogously for powers. For me this is less important. -) Being able to distinguish between row and column vectors; I guess this is just not possible with arrays... If --apart from the previous changes-- numpy had the .I for arrays I guess this would get me to say goodbye to matrices. The rest of the list would be a welcome luxury. I believe this whole thing has the potential to unify the use of numpy by eventually making the matrix subclass redundant. IMO that would be more transparent for new users and would increase the popularity of numpy! Cheers, Sven From bart.vandereycken at cs.kuleuven.be Thu Jul 6 07:49:57 2006 From: bart.vandereycken at cs.kuleuven.be (Bart Vandereycken) Date: Thu, 06 Jul 2006 13:49:57 +0200 Subject: [Numpy-discussion] Dot as a method In-Reply-To: <44AC39D4.3030104@cox.net> References: <44AC39D4.3030104@cox.net> Message-ID: Tim Hochberg wrote: > Bart Vandereycken wrote: >> Hi all, >> >> reading the thread "Ransom proposals" I was wondering why there isn't a >> ndarray.dot() method? There is already a scipy.sparse.dot() so this >> would fit nicely in the whole idea of polymorphism. >> > Are you sure about that? > > The problem with a dot method (aside from a over proliferation of > methods in general) is that to be done correctly you want to choose a > particular implementation of dot based *both* of its arguments. A method > does a good job dispatching on a single argument, but falls down when > dispatching on two types. Let's look at this specific case. Imagine that > in addition ndarray.dot and sparse.dot, we also stick a dot method on > ma.masked, etc. Now in order to fully exploit polymorphism and get > maximum efficiency, we want asparsearray.dot(amaskedarray) to correctly > treat the masked values (ma.dot treats them as zeros) and to not > instantiate a dense version of the sparsearray. But in order to do that > all three classes need to know about the other two. That's possible, if > messy since all three of these are known in advance, but this approach > becomes untenable if you classes outside core numpy or scipy to > participate as full citizens. You're right that you need to put the class specific dot code somewhere. However, it would be better if this can be done in a clean way. By the way I don't fixate on methods or functions, I just want something useful ;-) Suppose I have written my own matrix class (custmatrix) with it's own fast matrix-vector product. How can I extend numpy code so that is uses this fast multiplication when I want a_custmatrix*a_ndarray or a_ndarray*a_custmatrix? > What does appear fit well here is generic functions / multimethods / > protocols as discussed in some detail on pydev-3000 a couple of months > ago. This would allow classes defined outside of core numpy and scipy to > work correctly and efficiently with dot as long as they register > appropriate versions of dot. If I wasn't swamped right at the moment I'd > prototype this up and see how it works in practice. Right, I was thinking about multimethods too after reading http://www.artima.com/weblogs/viewpost.jsp?thread=101605 The other solution that simulates double dispatching is the Visitor pattern, but it has a lot of disadvantages (difficult to add custom matrices, slow?, all the matrices have to be children of an abstract class). Bart From arnd.baecker at web.de Thu Jul 6 08:06:25 2006 From: arnd.baecker at web.de (Arnd Baecker) Date: Thu, 6 Jul 2006 14:06:25 +0200 (CEST) Subject: [Numpy-discussion] Speed degression? In-Reply-To: <44ACF46C.8050507@ieee.org> References: <200607041519.24636.steffen.loeck@gmx.de> <44ACF46C.8050507@ieee.org> Message-ID: Hi Travis, On Thu, 6 Jul 2006, Travis Oliphant wrote: > Steffen Loeck wrote: > > Hi all, > > > > i made some speed tests using the sin-function and the %-operation to compare > > Numeric, numpy 0.9.8 and numpy 0.9.9.2732. > > As result the latest numpy version seems to be very slow in comparison to the > > two other candidates. > > > > Results (in usec per loop): > > sin-array mod-array > > > > Numeric 134 18 > > numpy 0.9.8 97 55 > > numpy 0.9.9.2732 204 316 > > numpy 0.9.8 + math 38 > > numpy 0.9.9.2732 + math 161 > > Numeric + math 23 > > > > Can anyone verify my results and explain the observed speed degression? > > > > I can't verify the results, but it's possible that the indexing code for > a single integer is a bit slower. On the 64Bit machine I get Results (in usec per loop): sin-array mod-array Numeric 34 7 numpy 0.9.9.2746 56 92 numpy 0.9.9.2746 + math 42 Numeric + math 8 Though this does not do the comparison with 0.9.8, it can roughly reproduce the ratios above. In particular the mod seems not just a "bit slower", but more than a factor 5. Also the " + math" variants for the sin suffered by a factor of 4... > However, you forgot to test the "use Python lists" approach which is the > fastest of all on this code. > > python -m timeit -s "x = [0.1]*10;" "for i in xrange(9): > x[i+1]=(x[i]+1.1)%(1.0)" > > > Using Numeric/NumPy is always slower than lists if you are doing > element-by-element processing. NumPy is slower than Numeric for such > processing. But, Numeric was always slower than pure Python too for > such processing. So, the moral of the story is: > > "don't compute like that with NumPy (or Numeric)" > Well, the example is a benchmark and not the real code ... To me it seems that it clearly points out some significant slow-down. > With that said. I can add back the 1-d integer check in the indexing > code to see if that speeds things up... That would be great - I will re-run the tests on the Opteron then. Many thanks, Arnd From wbaxter at gmail.com Thu Jul 6 08:58:01 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu, 6 Jul 2006 21:58:01 +0900 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44ACF7F2.1050904@gmx.net> References: <44ACF15D.6030607@ieee.org> <44ACF7F2.1050904@gmx.net> Message-ID: On 7/6/06, Sven Schreiber wrote: > > Travis Oliphant schrieb: > > Bill Baxter wrote: > >> So in short my proposal is to: > >> -- make a.T a property of array that returns a.swapaxes(-2,-1), > >> -- make a.H a property of array that returns > >> a.conjugate().swapaxes(-2,-1) > >> and maybe > >> -- make a.M a property of array that returns numpy.asmatrix(a) > > > > I've tentatively implemented all of these suggestions as well as adding > > the .A attribute to the ndarray as well (so that all sub-classes and > > array scalars can get back a view as an ndarray). That's great. > > > I did this to make it easier to do matrix-like calculations with or > > with-out matrices. Matrix-calculation flexibility is still a sore-spot > > for many and I think these syntatical-sugar attributes will help long > term. I think it will help too. The .M will make it easer to go from array to matrix types, and with the .A it was already easy to go from matrix to array. And the .T and .H will make it easier to do linear algebra numerics without switching over to matrix. One thing just occurred to me though, will the .M be available on matrices too? If not then one will have to be careful when using a mix of matrix and array to make sure you only use the .M on the latter and not the former. I think this is great, thanks to Bill for suggesting it and to Travis > for implementing it! Yes indeed, thanks! So the only convenience of matrices over pure arrays that remains (afaics): > > -) .I for inverse; actually, why not add that to arrays as well as > "syntactic sugar"? I don't even think it .I belongs in matrix. Hideously expensive calculations should not be masquerading as simple attribute accesses! Furthermore, maybe I'm just not doing the right kind of math, but inverses just don't come up that often for me. And if they do, it's usually in the context of solving a linear system Ax = b, where taking the inverse of A is generally a bad approach. So I'm negative on the inverse attribute in general, and double negative on it for arrays. Transpose is an operation that makes sense for a lot of kinds of data even non-matrix, but inverse is really only the multiplicative matrix inverse. Besides, given the new .M attribute, all you need to do to get the matrix inverse is .M.I. or .M.I.A if you want to get back to array. But sheesh, inv() is just 3 letters + parens. Going to .I only saves you 3 keystrokes. Going from .transpose() to .T saves 10. -) * being the matrix product instead of element-wise; Now, I could live > with using dot and I don't want to push anything, but maybe this is the > right time to consider another operator symbol as a shortcut for the dot > function to be used with arrays? (Unfortunately right now I can't think > of any sensible character(s) for that...) I don't think Python offers much flexibility in choosing characters to use as operators. That's one place where I think Haskell and ML are pretty neat. You can define your own in-fix operator symbols like (*) or -*- or <*> or .* like matlab. Whatever you want as long as it's a string of operator characters. Python can't do that, though. -) ** analogously for powers. For me this is less important. I don't have much need for that one. -) Being able to distinguish between row and column vectors; I guess > this is just not possible with arrays... Yeh that doesn't seem to be in the cards with array. But with the .M attribute you'll be able to get a column from a 1d array with a.M.T. Maybe a little improvement over a[None].transpose()? I guess I'd probably use a[None].T, though. If --apart from the previous changes-- numpy had the .I for arrays I > guess this would get me to say goodbye to matrices. The rest of the list > would be a welcome luxury. I believe this whole thing has the potential > to unify the use of numpy by eventually making the matrix subclass > redundant. IMO that would be more transparent for new users and would > increase the popularity of numpy! That was pretty much my conclusion. Ideally if you start off with matrices you would never encounter an array anywhere, and it would be just like using Matlab. :-) If it were like that I might still be using it. But it just doesn't seem to work out that way. One thing that would really help would be one of these mini-decorator modules Travis was talking about. In this case it would change all functions to return matrices. So if you do something like "import numpy.matrix" then after that ones() and zeros() and rand() etc will all be functions that return matrices. The .M attrib will make it easier. Instead of asmatrix(ones(3,3)) everywhere you'll have ones(3,3).M everywhere. But it's still not as nice as it would be if you could just import a package and have ones() etc return a matrix by default. I personally think matrix should be allowed to have more than 2 dimensions. Less than 2, no, but more than 2 should be ok. --bill -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Thu Jul 6 08:59:04 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 06 Jul 2006 06:59:04 -0600 Subject: [Numpy-discussion] Speed degression? In-Reply-To: References: <200607041519.24636.steffen.loeck@gmx.de> <44ACF46C.8050507@ieee.org> Message-ID: <44AD0918.9010702@ieee.org> Arnd Baecker wrote: > Hi >>> i made some speed tests using the sin-function and the %-operation to compare >>> Numeric, numpy 0.9.8 and numpy 0.9.9.2732. >>> As result the latest numpy version seems to be very slow in comparison to the >>> two other candidates. >>> Arrgggh. It looks like a pre-mature return statement was causing scalarmath not to load and therefore none of it's speed ups are present until this SVN. Thanks for the speed test... -Travis From arnd.baecker at web.de Thu Jul 6 09:36:11 2006 From: arnd.baecker at web.de (Arnd Baecker) Date: Thu, 6 Jul 2006 15:36:11 +0200 (CEST) Subject: [Numpy-discussion] Speed degression? In-Reply-To: <44AD0918.9010702@ieee.org> References: <200607041519.24636.steffen.loeck@gmx.de> <44ACF46C.8050507@ieee.org> <44AD0918.9010702@ieee.org> Message-ID: On Thu, 6 Jul 2006, Travis Oliphant wrote: > Arnd Baecker wrote: > > Hi > >>> i made some speed tests using the sin-function and the %-operation to compare > >>> Numeric, numpy 0.9.8 and numpy 0.9.9.2732. > >>> As result the latest numpy version seems to be very slow in comparison to the > >>> two other candidates. > >>> > > Arrgggh. > > It looks like a pre-mature return statement was causing scalarmath not > to load and therefore none of it's speed ups are present until this SVN. > > Thanks for the speed test... Well, the thanks for this go to Steffen (who wrote the text above and the test, not me ;-). Concerning the speed the thanks go to you: Results (in usec per loop): sin-array mod-array Numeric 34 7 numpy 0.9.9.2732 56 92 numpy 0.9.9.2732 + math 42 Numeric + math 8 *and now* Results (in usec per loop): sin-array mod-array Numeric 33 7 numpy 0.9.9.2749 30 17.6 numpy 0.9.9.2749 + math 13 Numeric + math 8 Bingo! Only numpy 0.9.9.2749 + math is slower than Numeric+math by a factor 1.6 and mod-array is 2.5 times slower than numpy. It would be nice if you could have a look at the module operation, if possible ... Many thanks, Arnd From tim.hochberg at cox.net Thu Jul 6 10:15:40 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 06 Jul 2006 07:15:40 -0700 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44ACF7F2.1050904@gmx.net> References: <44ACF15D.6030607@ieee.org> <44ACF7F2.1050904@gmx.net> Message-ID: <44AD1B0C.1070909@cox.net> Sven Schreiber wrote: > Travis Oliphant schrieb: > >> Bill Baxter wrote: >> >>> So in short my proposal is to: >>> -- make a.T a property of array that returns a.swapaxes(-2,-1), >>> -- make a.H a property of array that returns >>> a.conjugate().swapaxes(-2,-1) >>> and maybe >>> -- make a.M a property of array that returns numpy.asmatrix(a) >>> >> I've tentatively implemented all of these suggestions as well as adding >> the .A attribute to the ndarray as well (so that all sub-classes and >> array scalars can get back a view as an ndarray). >> >> I did this to make it easier to do matrix-like calculations with or >> with-out matrices. Matrix-calculation flexibility is still a sore-spot >> for many and I think these syntatical-sugar attributes will help long term. >> >> > > I think this is great, thanks to Bill for suggesting it and to Travis > for implementing it! > > So the only convenience of matrices over pure arrays that remains (afaics): > > -) .I for inverse; actually, why not add that to arrays as well as > "syntactic sugar"? > Because it encourages people to do the wrong thing numerically speaking? My understanding is that one almost never wants to compute the inverse directly, at least not if you're subsequently going to multiply it with something, instead you want to use linalg.solve or some other similar approach. > -) * being the matrix product instead of element-wise; Now, I could live > with using dot and I don't want to push anything, but maybe this is the > right time to consider another operator symbol as a shortcut for the dot > function to be used with arrays? (Unfortunately right now I can't think > of any sensible character(s) for that...) > See my earlier note on why using a method (which '*' is in essence) for dot is not really a good idea. We can do better. Oh, and this would completely hose all the non matrix heads out there as well. > -) ** analogously for powers. For me this is less important. > > -) Being able to distinguish between row and column vectors; I guess > this is just not possible with arrays... > Why can't you distinguish between them the same way that the matrix class does? Shape [1, N] is a row array, shape [N,1] is column array. > If --apart from the previous changes-- numpy had the .I for arrays I > guess this would get me to say goodbye to matrices. The rest of the list > would be a welcome luxury. I believe this whole thing has the potential > to unify the use of numpy by eventually making the matrix subclass > redundant. IMO that would be more transparent for new users and would > increase the popularity of numpy! >>> len(dir(numpy.ndarray)) - len(dir(int)) 99 I'm not particularly excited about the matrix class growing ever more cruft, but c'est la vie. I really need to finish up arraykit..... -tim From wbaxter at gmail.com Thu Jul 6 10:48:59 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu, 6 Jul 2006 23:48:59 +0900 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44AD1B0C.1070909@cox.net> References: <44ACF15D.6030607@ieee.org> <44ACF7F2.1050904@gmx.net> <44AD1B0C.1070909@cox.net> Message-ID: On 7/6/06, Tim Hochberg wrote: > > > -) Being able to distinguish between row and column vectors; I guess > > this is just not possible with arrays... > > > Why can't you distinguish between them the same way that the matrix > class does? Shape [1, N] is a row array, shape [N,1] is column array. Yep, that works. But there are still various annoyances. - You have to remeber to specify extra brackets all the time. Like array([[1,2,3]]) or array([[1],[2],[3]]). - And a slice of a vector out of a matrix has to be pumped back up to 2-D. If x has ndim==2, then to get a column out of it you have to do x[:,i,None] instead of just x[:,i]. To get a row you need x[j,None] instead of just x[j] Not horrible, but it feels a little klunky if you're used to something like Matlab. So matrix gets rid of a few annoyances like that ... and replaces them with a few of its own. :-) --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at cox.net Thu Jul 6 11:04:15 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 06 Jul 2006 08:04:15 -0700 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44AD1B0C.1070909@cox.net> References: <44ACF15D.6030607@ieee.org> <44ACF7F2.1050904@gmx.net> <44AD1B0C.1070909@cox.net> Message-ID: <44AD266F.8070707@cox.net> Tim Hochberg wrote: > Sven Schreiber wrote: > >> Travis Oliphant schrieb: >> >> >>> Bill Baxter wrote: >>> >>> >>>> So in short my proposal is to: >>>> -- make a.T a property of array that returns a.swapaxes(-2,-1), >>>> -- make a.H a property of array that returns >>>> a.conjugate().swapaxes(-2,-1) >>>> and maybe >>>> -- make a.M a property of array that returns numpy.asmatrix(a) >>>> >>>> >>> I've tentatively implemented all of these suggestions as well as adding >>> the .A attribute to the ndarray as well (so that all sub-classes and >>> array scalars can get back a view as an ndarray). >>> >>> I did this to make it easier to do matrix-like calculations with or >>> with-out matrices. Matrix-calculation flexibility is still a sore-spot >>> for many and I think these syntatical-sugar attributes will help long term. >>> >>> >>> >> I think this is great, thanks to Bill for suggesting it and to Travis >> for implementing it! >> >> So the only convenience of matrices over pure arrays that remains (afaics): >> >> -) .I for inverse; actually, why not add that to arrays as well as >> "syntactic sugar"? >> >> > Because it encourages people to do the wrong thing numerically speaking? > My understanding is that one almost never wants to compute the inverse > directly, at least not if you're subsequently going to multiply it with > something, instead you want to use linalg.solve or some other similar > approach. > Also, having an attribute access do an expensive operation behind the scenes seems antisocial. > >> -) * being the matrix product instead of element-wise; Now, I could live >> with using dot and I don't want to push anything, but maybe this is the >> right time to consider another operator symbol as a shortcut for the dot >> function to be used with arrays? (Unfortunately right now I can't think >> of any sensible character(s) for that...) >> >> > See my earlier note on why using a method (which '*' is in essence) for > dot is not really a good idea. Actually, thinking about this some more, this isn't a very compelling reason since multiply suffers from the same effect. Better reasons: 1. Consistency. Unless '/' is overloaded as well, then some of the operators will be elementwise and some will be matrix (+/- go both ways) making the interface inconsistent and thus harder to learn. 2. Backwards compatibility. 3. I was going to say narrowness of interface, since I had this fantasy that dot allowed axis to be specified, and thus we'd still need a separate dot function in addition to '*'. However, that must be something I have set up locally. (I realize that there's a separate multiply ufunc, but the ufuncs are all the same so I don't really count them against the overall complexity of numpy). Curmudgeonly yours, -tim > We can do better. Oh, and this would > completely hose all the non matrix heads out there as well. > > >> -) ** analogously for powers. For me this is less important. >> >> -) Being able to distinguish between row and column vectors; I guess >> this is just not possible with arrays... >> >> > Why can't you distinguish between them the same way that the matrix > class does? Shape [1, N] is a row array, shape [N,1] is column array. > >> If --apart from the previous changes-- numpy had the .I for arrays I >> guess this would get me to say goodbye to matrices. The rest of the list >> would be a welcome luxury. I believe this whole thing has the potential >> to unify the use of numpy by eventually making the matrix subclass >> redundant. IMO that would be more transparent for new users and would >> increase the popularity of numpy! >> > > >>> len(dir(numpy.ndarray)) - len(dir(int)) > 99 > > I'm not particularly excited about the matrix class growing ever more > cruft, but c'est la vie. I really need to finish up arraykit..... > > -tim > > > > > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From tim.hochberg at cox.net Thu Jul 6 11:45:13 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 06 Jul 2006 08:45:13 -0700 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44ACF7F2.1050904@gmx.net> <44AD1B0C.1070909@cox.net> Message-ID: <44AD3009.3030607@cox.net> Bill Baxter wrote: > On 7/6/06, *Tim Hochberg* > wrote: > > > -) Being able to distinguish between row and column vectors; I guess > > this is just not possible with arrays... > > > Why can't you distinguish between them the same way that the matrix > class does? Shape [1, N] is a row array, shape [N,1] is column array. > > > Yep, that works. But there are still various annoyances. > - You have to remeber to specify extra brackets all the time. Like > array([[1,2,3]]) or array([[1],[2],[3]]). This one I can't get excited about. If you are actually creating that many constant arrays, just define rowarray and colarray functions that add the appropriate dimensions for you. > - And a slice of a vector out of a matrix has to be pumped back up to > 2-D. If x has ndim==2, then to get a column out of it you have to do > x[:,i,None] instead of just x[:,i]. To get a row you need x[j,None] > instead of just x[j] Alternatively x[:,i:i+1], although that's not much better. > > Not horrible, but it feels a little klunky if you're used to something > like Matlab. Well Matlab is geared to matrices. The ndarray object has always been more or less a tensor. I can't help feeling that loading it up with matrix like methds is just going to lead to confusion and trouble. I would rather work things out so that we can have a pure matrix class and a pure ndarray class coexist in some sensible way. Figuring out how to do that well would have fringe benefits for other stuff (masked arrays, sparse arrays, user defined arrays of various types). > So matrix gets rid of a few annoyances like that ... and replaces them > with a few of its own. :-) In theory matrix should not be annoying to Matlab users since it's whole purpose is to keep matlab users happy. I think the big problem with matrix is that none of the developers use it as far as I know, so no one is motivated to clean up the rough edges. -tim From ndarray at mac.com Thu Jul 6 12:21:43 2006 From: ndarray at mac.com (Sasha) Date: Thu, 6 Jul 2006 12:21:43 -0400 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44ACF15D.6030607@ieee.org> References: <44ACF15D.6030607@ieee.org> Message-ID: I would like to raise a few objections going from mild to strong: 1. .T : I am mildly against it. As an inexpensive operation that returns a view (so that a.T[...] = makes sense) it is a reasonable candidate for an attribute. Unfortunately reversing the order of axes at least as reasonable as swapaxes(-2,-1) and swapaxes(-2,-1) is invalid for rank < 2. My main objection is that a.T is fairly cryptic - is there any other language that uses attribute for transpose? Adding .T to arrays will lead to less readable code because in expressions like "a * b.T" it will not be clear whether * is a matrix or elemenwise multiplication. 2. .H : This is an O(n^2) complexity operation returning a copy so it is not appropriate for an attribute. It does not make much sense for any type other than complex, so it's use is limited. 3. .M : I am strongly against this. It will create a circular dependency between ndarray and matrix. I would expect that asmatrix is mostly used to convert function arguments and for this purpose @matrix_args decorator would be a better solution to reduce code clutter. 4. .A : I have no clue what this one does, so I won't comment. On 7/6/06, Travis Oliphant wrote: > Bill Baxter wrote: > > So in short my proposal is to: > > -- make a.T a property of array that returns a.swapaxes(-2,-1), > > -- make a.H a property of array that returns > > a.conjugate().swapaxes(-2,-1) > > and maybe > > -- make a.M a property of array that returns numpy.asmatrix(a) > > I've tentatively implemented all of these suggestions as well as adding > the .A attribute to the ndarray as well (so that all sub-classes and > array scalars can get back a view as an ndarray). > > I did this to make it easier to do matrix-like calculations with or > with-out matrices. Matrix-calculation flexibility is still a sore-spot > for many and I think these syntatical-sugar attributes will help long term. > > If there are no strong objections, then the recent MATH attribute > checkin will stay. If there are major objections, then we can back them > out without too much trouble as well. > > -Travis > > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From Chris.Barker at noaa.gov Thu Jul 6 12:35:43 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 06 Jul 2006 09:35:43 -0700 Subject: [Numpy-discussion] simple question In-Reply-To: <44AC5D86.9050201@jpl.nasa.gov> References: <44AC5B4C.10804@jpl.nasa.gov> <44AC5D86.9050201@jpl.nasa.gov> Message-ID: <44AD3BDF.8020107@noaa.gov> Mathew Yeates wrote: > ohhhh. I was looking at using "where" There's nothing wrong with where: >>> y = N.asmatrix(N.rand(3,3)) >>> y matrix([[ 0.29741635, 0.78756994, 0.641378 ], [ 0.0198837 , 0.71677631, 0.76068183], [ 0.84904382, 0.80169706, 0.23877389]]) >>> x = N.asmatrix(N.where(y > 0.5, 1, 0)) >>> x matrix([[0, 1, 1], [0, 1, 1], [1, 1, 0]]) And did you really mean matrix? or would 2-d arrays be fine? -CHB -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From svetosch at gmx.net Thu Jul 6 12:40:46 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Thu, 06 Jul 2006 18:40:46 +0200 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44AD266F.8070707@cox.net> References: <44ACF15D.6030607@ieee.org> <44ACF7F2.1050904@gmx.net> <44AD1B0C.1070909@cox.net> <44AD266F.8070707@cox.net> Message-ID: <44AD3D0E.1060706@gmx.net> Tim Hochberg schrieb: >>> -) .I for inverse; actually, why not add that to arrays as well as >>> "syntactic sugar"? >>> >>> >> Because it encourages people to do the wrong thing numerically speaking? >> My understanding is that one almost never wants to compute the inverse >> directly, at least not if you're subsequently going to multiply it with >> something, instead you want to use linalg.solve or some other similar >> approach. >> > Also, having an attribute access do an expensive operation behind the > scenes seems antisocial. Ok, you (and Bill) may have a point there. (Although I'm not sure it's the best way to educate naive users like me about numerical efficiency by making inverse harder to access. Otherwise you should rename linalg.inv --way too easy!-- to linalg.okay_im_going_to_compute_the_inverse_for_you_but_its_probably_a_stupid_thing_to_do() ;-) However, then why is it possible for matrices?? It's just seems incoherent to me. Maybe you guys as the developers of numpy should really make up your mind about the future of matrices in numpy. Either it's supported, then eventually I would expect matrix versions of ones, zeros, eye, for example. (Although eye().M would come close enough I guess.) Or you decide that you don't really like all the problems that it implies and officially declare it unsupported. That would be valuable information for users like me who have had (and still sometimes have) a hard time figuring out whether I should be using arrays or matrices. > >> >>> -) * being the matrix product instead of element-wise; Now, I could live >>> with using dot and I don't want to push anything, but maybe this is the >>> right time to consider another operator symbol as a shortcut for the dot >>> function to be used with arrays? (Unfortunately right now I can't think >>> of any sensible character(s) for that...) >>> > 2. Backwards compatibility. I meant a *new* operator symbol. > Curmudgeonly yours, > Well I don't know what that means, so here's my hopefully equally incomprehensible reaction: Mit freundlichen Gr??en, Sven From Chris.Barker at noaa.gov Thu Jul 6 12:42:27 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 06 Jul 2006 09:42:27 -0700 Subject: [Numpy-discussion] Numpy import problem In-Reply-To: <20060706170241.hq0fq9gssdys8kg4@webmail.auckland.ac.nz> References: <20060706170241.hq0fq9gssdys8kg4@webmail.auckland.ac.nz> Message-ID: <44AD3D73.8080602@noaa.gov> Satellite Data Research Group wrote: > Hi, I have a problem when importing numpy into python 2.4.1. I did a > successful (I think) build and install of Numpy 0.9.8 on a mac osx > (10.4) intel, Which Python 2.4.1 are you using? It would be great if you would give the Python2.4.3 version found here a try: http://www.pythonmac.org/packages/py24-fat/index.html We're trying to establish it as the "recommended" Python for OS-X 10.4, on both Intel and PPC machines. In any case, we need to know what build of Python you're using. -CHB -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From kwgoodman at gmail.com Thu Jul 6 12:59:29 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 6 Jul 2006 09:59:29 -0700 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44AD3D0E.1060706@gmx.net> References: <44ACF15D.6030607@ieee.org> <44ACF7F2.1050904@gmx.net> <44AD1B0C.1070909@cox.net> <44AD266F.8070707@cox.net> <44AD3D0E.1060706@gmx.net> Message-ID: On 7/6/06, Sven Schreiber wrote: > Maybe you guys as the developers of numpy should really make up your > mind about the future of matrices in numpy. Either it's supported, then > eventually I would expect matrix versions of ones, zeros, eye, for > example. (Although eye().M would come close enough I guess.) Or you > decide that you don't really like all the problems that it implies and > officially declare it unsupported. > That would be valuable information for users like me who have had (and > still sometimes have) a hard time figuring out whether I should be using > arrays or matrices. If we use it (matrices) they will build it. All of the big problems I've run into while using matrices have been fixed within hours of posting the problem to the mailing list. So for anyone who wants better matrix support, all you have to do is start using matrices. The rest follows naturally: send any problems you have to the list and answer any questions you can from new matrix users. From wbaxter at gmail.com Thu Jul 6 13:21:44 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 7 Jul 2006 02:21:44 +0900 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> Message-ID: On 7/7/06, Sasha wrote: > > I would like to raise a few objections going from mild to strong: > > 1. .T : I am mildly against it. As an inexpensive operation that > returns a view (so that a.T[...] = makes sense) it is a reasonable > candidate for an attribute. Unfortunately reversing the order of axes at least as reasonable as > swapaxes(-2,-1) I suppose reversing the order changes you from C ordering to Fortran ordering? Other than that I can't think of any good examples of why you'd want to completely reverse the order of all your axes. I think it's much more common to want to swap just two axes, and the last two seem a logical choice since a) in the default C-ordering they're the closest together in memory and b) they're the axes that are printed contiguously when you say "print A". and swapaxes(-2,-1) is > invalid for rank < 2. > At least in numpy 0.9.8, it's not invalid, it just doesn't do anything. My main objection is that a.T is fairly cryptic > - is there any other language that uses attribute for transpose? Does it matter what other languages do? It's not _that_ cryptic. The standard way to write transpose is with a little T superscript in the upper right. We can't do that with ASCII so the T just appears after the dot. Makes perfect sense to me. I'd vote for an operator if it were possible in python. Something like A^T would be neat, maybe, or matlab's single-quote operator. Adding .T to arrays will lead to less readable code because in > expressions like "a * b.T" it will not be clear whether * is a matrix > or elemenwise multiplication. That seems a pretty weak argument, since there are already lots of expressions you can write that don't make it clear whether some operation is a matrix operation or array operation. You could write a * b.transpose(1,0) right now and still not know whether it was matrix or element-wise multiplication. Or doing A[:,1] when you know A is 2-D -- does it give you a 1D thing back or a 2D thing back? That just comes down to it being difficult to determine the class of objects in Python by looking at code. > 2. .H : This is an O(n^2) complexity operation returning a copy so > it is not appropriate for an attribute. Not sure how you get O(n^2). It just requires flipping the sign on the imaginary part of each element in the array. So in my book that's O(n). But that does make it more expensive than O(1) transpose, yes. It does not make much sense > for any type other than complex, so it's use is limited. I personally don't think I have ever used a hermitian transpose in my life. So I can't really can't say how useful it is. But the makers of Matlab decided to make single quote ( e.g. A' ) be the hermitian transpose operator, and dot-single quote (e.g. A.') be the regular transpose. So I'm assuming it's common enough that the folks behind Matlab thought it wise to make it the 'default' style of transpose and give it a one-character operator. That's about the only evidence I have that it's a useful operation, though. In general, though, I do know that when you take good-ole algorithms for reals and extend them to complex numbers, things that were transpose for the reals become hermitian transposes for the complex version. 3. .M : I am strongly against this. It will create a circular > dependency between ndarray and matrix. I would expect that asmatrix > is mostly used to convert function arguments and for this purpose > @matrix_args decorator would be a better solution to reduce code > clutter. I'm kindof ambivalent about this one too. Assuming matrix is going to stay around and we actually want to encourage people to use it, I think having a .M on arrays is an improvement over the current situation. Arguments to functions expecting matrices are as you say one place where conversions are needed, but another place is on funtions like zeros and ones and empty. With the .M you can just say ones(2,2).M. But probably a better solution would be to have matrix versions of these in the library as an optional module to import so people could, say, import them as M and use M.ones(2,2). It does seem to me that in some sense matrix is supposed to be 'just another customized array subclass', like sparse or masked, so to have array aware of it this one particular subclass makes me a little uneasy. But if matrix really should be considered to be on par with array, then it makes sense. It's just like a mutually recursive data structure. Or you can think of matrix's inheritance from array being an implementation detail. 4. .A : I have no clue what this one does, so I won't comment. It returns the array. I think the idea was that you would always be able to say .A with array or anything derived from it. Currently you have to know you have a matrix before you can use the .A attribute. If you were wrong and it was actually an array, then you'll get an exception. It would be nicer to have X.A just return X if X is already an array. In short, I'm most enthusiastic about the .T attribute. Then, given a .T, it makes sense to have a .H, both to be consistent with matrix, but also since it seems to be a big deal in other math packages like matlab. Then given the current situation, I like the .M but I can imagine other ways to make .M less necessary. --bb On 7/6/06, Travis Oliphant wrote: > > Bill Baxter wrote: > > > So in short my proposal is to: > > > -- make a.T a property of array that returns a.swapaxes(-2,-1), > > > -- make a.H a property of array that returns > > > a.conjugate().swapaxes(-2,-1) > > > and maybe > > > -- make a.M a property of array that returns numpy.asmatrix(a) > > > > I've tentatively implemented all of these suggestions as well as adding > > the .A attribute to the ndarray as well (so that all sub-classes and > > array scalars can get back a view as an ndarray). > > > > I did this to make it easier to do matrix-like calculations with or > > with-out matrices. Matrix-calculation flexibility is still a sore-spot > > for many and I think these syntatical-sugar attributes will help long > term. > > > > If there are no strong objections, then the recent MATH attribute > > checkin will stay. If there are major objections, then we can back them > > out without too much trouble as well. > > > > -Travis > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at cox.net Thu Jul 6 13:51:14 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 06 Jul 2006 10:51:14 -0700 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44AD3D0E.1060706@gmx.net> References: <44ACF15D.6030607@ieee.org> <44ACF7F2.1050904@gmx.net> <44AD1B0C.1070909@cox.net> <44AD266F.8070707@cox.net> <44AD3D0E.1060706@gmx.net> Message-ID: <44AD4D92.7020403@cox.net> Sven Schreiber wrote: > Tim Hochberg schrieb: > > >>>> -) .I for inverse; actually, why not add that to arrays as well as >>>> "syntactic sugar"? >>>> >>>> >>>> >>> Because it encourages people to do the wrong thing numerically speaking? >>> My understanding is that one almost never wants to compute the inverse >>> directly, at least not if you're subsequently going to multiply it with >>> something, instead you want to use linalg.solve or some other similar >>> approach. >>> >>> >> Also, having an attribute access do an expensive operation behind the >> scenes seems antisocial. >> > > Ok, you (and Bill) may have a point there. (Although I'm not sure it's > the best way to educate naive users like me about numerical efficiency > by making inverse harder to access. Otherwise you should rename > linalg.inv --way too easy!-- to > linalg.okay_im_going_to_compute_the_inverse_for_you_but_its_probably_a_stupid_thing_to_do() > ;-) > > However, then why is it possible for matrices?? It's just seems > incoherent to me. > Because matrix users asked for it I imagine. Since I don't use the matrix, I certainly wouldn't have objected. Also, the matrix class is supposed to look a lot like matlab and I assume matlab has some trivial way to produce an inverse -- even if it is a bad idea. > Maybe you guys as the developers of numpy should really make up your > mind about the future of matrices in numpy. Either it's supported, then > eventually I would expect matrix versions of ones, zeros, eye, for > example. (Although eye().M would come close enough I guess.) Or you > decide that you don't really like all the problems that it implies and > officially declare it unsupported. > That would be valuable information for users like me who have had (and > still sometimes have) a hard time figuring out whether I should be using > arrays or matrices. > Well, the matrix class could use a champion I think. Preferably someone who does a lot of matrix work and is capable of doing some development on it, at least on the python end. Without that I imagine it will stay kind of incoherent. > > >>> >>> >>>> -) * being the matrix product instead of element-wise; Now, I could live >>>> with using dot and I don't want to push anything, but maybe this is the >>>> right time to consider another operator symbol as a shortcut for the dot >>>> function to be used with arrays? (Unfortunately right now I can't think >>>> of any sensible character(s) for that...) >>>> >>>> >> 2. Backwards compatibility. >> > > I meant a *new* operator symbol. > Oh. Well, we're kind of short on those at the moment. The only thing I know of that would work is to abuse the call operator, thus dot(A, B) <=> (A)(B) That's kind of weird though and it's never gotten much support. > >> Curmudgeonly yours, >> A curmudgeon is an old, cranky, stubborn fixed in his ways kind of person. >> > Well I don't know what that means, so here's my hopefully equally > incomprehensible reaction: > Mit freundlichen Gr??en, > With great (big? much?) friendship (friendliness?)? -tim > Sven > > > From ndarray at mac.com Thu Jul 6 14:25:39 2006 From: ndarray at mac.com (Sasha) Date: Thu, 6 Jul 2006 14:25:39 -0400 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> Message-ID: On 7/6/06, Bill Baxter wrote: > > > On 7/7/06, Sasha wrote: > ... I think it's much > more common to want to swap just two axes, and the last two seem a logical > choice since a) in the default C-ordering they're the closest together in > memory and b) they're the axes that are printed contiguously when you say > "print A". It all depends on how you want to interpret a rank-K tensor. You seem to advocate a view that it is a (K-2)-rank array of matrices and .T is an element-wise transpose operation. Alternatively I can expect that it is a matrix of (K-2)-rank arrays and then .T should be swapaxes(0,1). Do you have real-life applications of swapaxes(-2,-1) for rank > 2? > > > and swapaxes(-2,-1) is > > invalid for rank < 2. > > > At least in numpy 0.9.8, it's not invalid, it just doesn't do anything. > That's bad. What sense does it make to swap non-existing axes? Many people would expect transpose of a vector to be a matrix. This is the case in S+ and R. > > My main objection is that a.T is fairly cryptic > > - is there any other language that uses attribute for transpose? > > > Does it matter what other languages do? It's not _that_ cryptic. If something is clear and natural, chances are it was done before. For me prior art is always a useful guide when making a design choice. For example, in R, the transpose operation is t(a) and works on rank <= 2 only always returning rank-2. K (an APL-like language) overloads unary '+' to do swapaxes(0,1) for rank>=2 and nothing for lower rank. Both R and K solutions are implementable in Python with R using 3 characters and K using 1(!) compared to your two-character ".T" notation. I would suggest that when inventing something new, you should consider prior art and explain how you invention is better. That's why what other languages do matter. (After all, isn't 'T' chosen because "transpose" starts with "t" in the English language?) > The standard way to write transpose is with a little T superscript in the upper > right. We can't do that with ASCII so the T just appears after the dot. > Makes perfect sense to me. I'd vote for an operator if it were possible in > python. Something like A^T would be neat, maybe, or matlab's single-quote > operator. > Well, you could overload __rpow__ for a singleton T and spell it A**T ... (I hope no one will take that proposal seriosely). Visually, A.T looks more like a subscript rather than superscript. > > Adding .T to arrays will lead to less readable code because in > > expressions like "a * b.T" it will not be clear whether * is a matrix > > or elemenwise multiplication. > > > That seems a pretty weak argument, since there are already lots of > expressions you can write that don't make it clear whether some operation is > a matrix operation or array operation. This may be a weak argument for someone used to matrix notation, but for me seeing a.T means: beware - tricky stuff here. > You could write a * b.transpose(1,0) > right now and still not know whether it was matrix or element-wise > multiplication. Why would anyone do that if b was a matrix? > > 2. .H : This is an O(n^2) complexity operation returning a copy so > > it is not appropriate for an attribute. > > Not sure how you get O(n^2). It just requires flipping the sign on the > imaginary part of each element in the array. So in my book that's O(n). > But that does make it more expensive than O(1) transpose, yes. > In my book n is the size of the matrix as in "n x n matrix", but the argument stays with O(n) as well. > But probably a better solution > would be to have matrix versions of these in the library as an optional > module to import so people could, say, import them as M and use M.ones(2,2). > This is the solution used by ma, which is another argument for it. > In short, I'm most enthusiastic about the .T attribute. Then, given a .T, > it makes sense to have a .H, both to be consistent with matrix, but also > since it seems to be a big deal in other math packages like matlab. Then > given the current situation, I like the .M but I can imagine other ways to > make .M less necessary. > I only raised a mild objection against .T, but the slippery slope argument makes me dislike it much more. At the very least I would like to see a discussion of why a.T is better than t(a) or +a. From tim.hochberg at cox.net Thu Jul 6 14:50:40 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 06 Jul 2006 11:50:40 -0700 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> Message-ID: <44AD5B80.4010905@cox.net> Sasha wrote: > On 7/6/06, Bill Baxter wrote: > >> On 7/7/06, Sasha wrote: >> ... I think it's much >> more common to want to swap just two axes, and the last two seem a logical >> choice since a) in the default C-ordering they're the closest together in >> memory and b) they're the axes that are printed contiguously when you say >> "print A". >> > > It all depends on how you want to interpret a rank-K tensor. You seem > to advocate a view that it is a (K-2)-rank array of matrices and .T is > an element-wise transpose operation. Alternatively I can expect that > it is a matrix of (K-2)-rank arrays and then .T should be > swapaxes(0,1). Do you have real-life applications of swapaxes(-2,-1) > for rank > 2? > I do for what it's worth. At various times I use arrays of shape (n, n, d) (matrices of rank-1 arrays at you suggest above) and arrays of shape (d, n, n) (vectors of matrices as Bill proposes). Using swapaxes(-2, -1) would be right only half the time, but it would the current defaults for transpose are essentially never right for rank > 2. Then again they are easy to explain. > >>> and swapaxes(-2,-1) is >>> invalid for rank < 2. >>> >>> >> At least in numpy 0.9.8, it's not invalid, it just doesn't do anything. >> >> > > That's bad. What sense does it make to swap non-existing axes? Many > people would expect transpose of a vector to be a matrix. This is the > case in S+ and R. > So this is essentially turning a row vector into a column vector? Is that right? >>> My main objection is that a.T is fairly cryptic >>> - is there any other language that uses attribute for transpose? >>> >> Does it matter what other languages do? It's not _that_ cryptic. >> > > If something is clear and natural, chances are it was done before. > For me prior art is always a useful guide when making a design choice. > For example, in R, the transpose operation is t(a) and works on rank > <= 2 only always returning rank-2. K (an APL-like language) overloads > unary '+' to do swapaxes(0,1) for rank>=2 and nothing for lower rank. > Both R and K solutions are implementable in Python with R using 3 > characters and K using 1(!) compared to your two-character ".T" > notation. Overloading '+' sure seems perverse, but maybe that's just me. > I would suggest that when inventing something new, you > should consider prior art and explain how you invention is better. > That's why what other languages do matter. (After all, isn't 'T' > chosen because "transpose" starts with "t" in the English language?) > > >> The standard way to write transpose is with a little T superscript in the upper >> right. We can't do that with ASCII so the T just appears after the dot. >> Makes perfect sense to me. I'd vote for an operator if it were possible in >> python. Something like A^T would be neat, maybe, or matlab's single-quote >> operator. >> >> > > Well, you could overload __rpow__ for a singleton T and spell it A**T > ... (I hope no one will take that proposal seriosely). Visually, A.T > looks more like a subscript rather than superscript. > No, no no. Overload __rxor__, then you can spell it A^t, A^h, etc. Much better ;-). [Sadly, I almost like that....] > >>> Adding .T to arrays will lead to less readable code because in >>> expressions like "a * b.T" it will not be clear whether * is a matrix >>> or elemenwise multiplication. >>> >> That seems a pretty weak argument, since there are already lots of >> expressions you can write that don't make it clear whether some operation is >> a matrix operation or array operation. >> > > This may be a weak argument for someone used to matrix notation, but > for me seeing a.T means: beware - tricky stuff here. > > >> You could write a * b.transpose(1,0) >> right now and still not know whether it was matrix or element-wise >> multiplication. >> > > Why would anyone do that if b was a matrix? > > > >>> 2. .H : This is an O(n^2) complexity operation returning a copy so >>> it is not appropriate for an attribute. >>> >> Not sure how you get O(n^2). It just requires flipping the sign on the >> imaginary part of each element in the array. So in my book that's O(n). >> But that does make it more expensive than O(1) transpose, yes. >> >> > In my book n is the size of the matrix as in "n x n matrix", but the > argument stays with O(n) as well. > > > > >> But probably a better solution >> would be to have matrix versions of these in the library as an optional >> module to import so people could, say, import them as M and use M.ones(2,2). >> >> > > This is the solution used by ma, which is another argument for it. > > > >> In short, I'm most enthusiastic about the .T attribute. Then, given a .T, >> it makes sense to have a .H, both to be consistent with matrix, but also >> since it seems to be a big deal in other math packages like matlab. Then >> given the current situation, I like the .M but I can imagine other ways to >> make .M less necessary. >> >> > > I only raised a mild objection against .T, but the slippery slope > argument makes me dislike it much more. At the very least I would > like to see a discussion of why a.T is better than t(a) or +a. > Here's a half baked thought: if the objection to t(A) is that it doesn't mirror the formulae where t appears as a subscript after A. Conceivably, __call__ could be defined so that A(x) returns x(A). That's kind of perverse, but it means that A(t), A(h), etc. could all work appropriately for suitably defined singletons. These singletons could either be assembeled in some abbreviations namespace or brought in by the programmer using "import transpose as t", etc. The latter works for doing t(a) as well of course. -tim From robert.kern at gmail.com Thu Jul 6 14:54:29 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 06 Jul 2006 13:54:29 -0500 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44ACF15D.6030607@ieee.org> References: <44ACF15D.6030607@ieee.org> Message-ID: Travis Oliphant wrote: > Bill Baxter wrote: >> So in short my proposal is to: >> -- make a.T a property of array that returns a.swapaxes(-2,-1), >> -- make a.H a property of array that returns >> a.conjugate().swapaxes(-2,-1) >> and maybe >> -- make a.M a property of array that returns numpy.asmatrix(a) > > I've tentatively implemented all of these suggestions as well as adding > the .A attribute to the ndarray as well (so that all sub-classes and > array scalars can get back a view as an ndarray). > > I did this to make it easier to do matrix-like calculations with or > with-out matrices. Matrix-calculation flexibility is still a sore-spot > for many and I think these syntatical-sugar attributes will help long term. > > If there are no strong objections, then the recent MATH attribute > checkin will stay. If there are major objections, then we can back them > out without too much trouble as well. Like Sasha, I'm mildly opposed to .T (as a synonym for .transpose()) and much more opposed to the rest (including .T being a synonym for .swapaxes(-2, -1)). It's not often that a proposal carries with it its own slippery-slope argument against itself. I don't think that just because arrays are often used for linear algebra that linear algebra assumptions should be built in to the core array type. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From alexander.belopolsky at gmail.com Thu Jul 6 15:18:43 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 6 Jul 2006 15:18:43 -0400 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44AD5B80.4010905@cox.net> References: <44ACF15D.6030607@ieee.org> <44AD5B80.4010905@cox.net> Message-ID: On 7/6/06, Tim Hochberg wrote: > ... > So this is essentially turning a row vector into a column vector? Is > that right? > Being a definition, this is neither right nor wrong. It all depends on what you are using it for. If you want to distinguish row and column vectors, you have to use rank-2, but it is convenient to to think of rank-1 as an equivalent of either row or column and rank-0 as a 1x1 array. If you define transpose as a rank-2 only operation, but allow promotion of rank < 2 to rank 2, you end up with the rules of S+. From alexander.belopolsky at gmail.com Thu Jul 6 15:37:27 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 6 Jul 2006 15:37:27 -0400 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44AD5B80.4010905@cox.net> References: <44ACF15D.6030607@ieee.org> <44AD5B80.4010905@cox.net> Message-ID: On 7/6/06, Tim Hochberg wrote: > ... > Overloading '+' sure seems perverse, but maybe that's just me. > The first time I saw it, it seemed perverse to me as well, but it actually make a lot of sense: 1. It is visually appealing as in '+' makes '|' from '-' and '-' from '|' and looks close enough to 't'. 2. It puts an otherwise useless operator to work. 3. Prefix spelling suggests that it should be swapaxes(0,1) rather than swapaxes(-2,-1), which is the choice made by K. 4. You can't get any shorter than that (at least using a fixed width font :-). 5. It already does the right thing for rank<2. From ndarray at mac.com Thu Jul 6 15:38:00 2006 From: ndarray at mac.com (Sasha) Date: Thu, 6 Jul 2006 15:38:00 -0400 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44AD5B80.4010905@cox.net> Message-ID: On 7/6/06, Tim Hochberg wrote: > ... > Overloading '+' sure seems perverse, but maybe that's just me. > The first time I saw it, it seemed perverse to me as well, but it actually make a lot of sense: 1. It is visually appealing as in '+' makes '|' from '-' and '-' from '|' and looks close enough to 't'. 2. It puts an otherwise useless operator to work. 3. Prefix spelling suggests that it should be swapaxes(0,1) rather than swapaxes(-2,-1), which is the choice made by K. 4. You can't get any shorter than that (at least using a fixed width font :-). 5. It already does the right thing for rank<2. From tim.hochberg at cox.net Thu Jul 6 16:23:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 06 Jul 2006 13:23:03 -0700 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44AD5B80.4010905@cox.net> Message-ID: <44AD7127.2090307@cox.net> Alexander Belopolsky wrote: > On 7/6/06, Tim Hochberg wrote: >> ... >> Overloading '+' sure seems perverse, but maybe that's just me. >> > The first time I saw it, it seemed perverse to me as well, but it > actually make a lot of sense: > > 1. It is visually appealing as in '+' makes '|' from '-' and '-' from > '|' and looks close enough to 't'. It looks even closer to ? (dagger if that doesn't make it through) which is the symbol used for the hermitian adjoint. > 2. It puts an otherwise useless operator to work. > 3. Prefix spelling suggests that it should be swapaxes(0,1) rather > than swapaxes(-2,-1), which is the choice made by K. > 4. You can't get any shorter than that (at least using a fixed width > font :-). > 5. It already does the right thing for rank<2. Perhaps it's not as perverse as it first appears. Although I still don't have to like it ;-) -tim > > From myeates at jpl.nasa.gov Thu Jul 6 16:23:49 2006 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Thu, 06 Jul 2006 13:23:49 -0700 Subject: [Numpy-discussion] simple question In-Reply-To: References: <44AC5B4C.10804@jpl.nasa.gov> Message-ID: <44AD7155.6070901@jpl.nasa.gov> Not working. A[row,all_dates == 10] = -1 where all_dates is a matrix with column length of 14 [[960111,..,.. and A is a matrix with same column length I get IndexError: arrays used as indices must be of integer type when I print out all_dates == 10 I get [True True True True True True True True True False False False True True]] I experimented with "<" instead of "==" but I still get boolean values as indices. Any help? Mathew Keith Goodman wrote: > On 7/5/06, Mathew Yeates wrote: >> What is the typical way of doing the following >> starting with a 0 matrix, set all values to 1 when a certain condition >> is met, set to -1 when another condition is met, left alone if neither >> condition is met. > > This works on recent versions of numpy: > >>> x = asmatrix(zeros((2,2))) > >>> x > > matrix([[0, 0], > [0, 0]]) > >>> y = asmatrix(rand(2,2)) > >>> y > > matrix([[ 0.85219404, 0.48311427], > [ 0.41026966, 0.2184193 ]]) > >>> x[y > 0.5] = 1 > >>> x[y < 0.5] = -1 > >>> x > > matrix([[ 1, -1], > [-1, -1]]) > From ndarray at mac.com Thu Jul 6 16:30:41 2006 From: ndarray at mac.com (Sasha) Date: Thu, 6 Jul 2006 16:30:41 -0400 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> Message-ID: On 7/6/06, Robert Kern wrote: > ... > I don't think that just because arrays are often used for linear algebra that > linear algebra assumptions should be built in to the core array type. > In addition, transpose is a (rank-2) array or matrix operation and not a linear algebra operation. Transpose corresponds to the "adjoint" linear algebra operation if you represent vectors as single column matrices and co-vectors as single-row matrices. This is a convenient representation followed by much of the relevant literature, but it does not alow generalization beyond rank-2. Another useful feature is that inner product can be calculated as the matrix product as long as you accept a 1x1 matrix for a scalar. This feature does not work beyond rank-2 either because in order to do tensor inner product you have to be explicit about the axes being collapsed (for example using Einstein notation). Since ndarray does not distinguish between upper an lower indices, it is not possible distinguish between vectors and co-vectors in any way other than using matrix convention. This makes ndarrays a poor model for linear algebra tensors. From oliphant at ee.byu.edu Thu Jul 6 16:36:48 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 06 Jul 2006 14:36:48 -0600 Subject: [Numpy-discussion] simple question In-Reply-To: <44AD7155.6070901@jpl.nasa.gov> References: <44AC5B4C.10804@jpl.nasa.gov> <44AD7155.6070901@jpl.nasa.gov> Message-ID: <44AD7460.7050109@ee.byu.edu> Mathew Yeates wrote: >Not working. >A[row,all_dates == 10] = -1 where all_dates is a matrix with column >length of 14 [[960111,..,.. >and A is a matrix with same column length > >I get >IndexError: arrays used as indices must be of integer type > >when I print out all_dates == 10 >I get >[True True True True True True True True True False False False True True]] > >I experimented with "<" instead of "==" but I still get boolean values >as indices. > >Any help? > > What version are you using? Can you give an example that shows the error. It's hard to guess the type of all the variables. The following works for me. import numpy print numpy.__version__ A = numpy.matrix(rand(3,14)) all_dates = array([10,10,1,10,1,10,0,10,0,10,0,1,10,1]) row = 2 A[row, all_dates == 10] -Trvis From ndarray at mac.com Thu Jul 6 16:41:44 2006 From: ndarray at mac.com (Sasha) Date: Thu, 6 Jul 2006 16:41:44 -0400 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44AD7127.2090307@cox.net> References: <44ACF15D.6030607@ieee.org> <44AD5B80.4010905@cox.net> <44AD7127.2090307@cox.net> Message-ID: On 7/6/06, Tim Hochberg wrote: > ... > It looks even closer to ? (dagger if that doesn't make it through) which > is the symbol used for the hermitian adjoint. If it pleases the matlab crowd, '+' can be defined to do the hermitian adjoint. on the complex type. > ... > Perhaps it's not as perverse as it first appears. Although I still don't > have to like it ;-) I don't like it either, but I don't like .T even more. These days I hate functionality I cannot google for. Call me selfish, but I already know what unary '+' can do to a higher rank array, but with .T I will always have to look up which axes it swaps ... From myeates at jpl.nasa.gov Thu Jul 6 16:42:08 2006 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Thu, 06 Jul 2006 13:42:08 -0700 Subject: [Numpy-discussion] simple question In-Reply-To: <44AD7460.7050109@ee.byu.edu> References: <44AC5B4C.10804@jpl.nasa.gov> <44AD7155.6070901@jpl.nasa.gov> <44AD7460.7050109@ee.byu.edu> Message-ID: <44AD75A0.8060809@jpl.nasa.gov> The very example you give produces IndexError: arrays used as indices must be of integer type this is with 0.9.8 Also .....while your example says "rand" I had to say numpy.rand This is on WindowsXP Mathew Travis Oliphant wrote: > Mathew Yeates wrote: > >> Not working. >> A[row,all_dates == 10] = -1 where all_dates is a matrix with column >> length of 14 [[960111,..,.. >> and A is a matrix with same column length >> >> I get >> IndexError: arrays used as indices must be of integer type >> >> when I print out all_dates == 10 >> I get >> [True True True True True True True True True False False False True >> True]] >> >> I experimented with "<" instead of "==" but I still get boolean >> values as indices. >> >> Any help? >> >> > What version are you using? Can you give an example that shows the > error. It's hard to guess the type of all the variables. The > following works for me. > > import numpy > print numpy.__version__ > A = numpy.matrix(rand(3,14)) > all_dates = array([10,10,1,10,1,10,0,10,0,10,0,1,10,1]) > row = 2 > A[row, all_dates == 10] > > > > > > -Trvis > > From kwgoodman at gmail.com Thu Jul 6 16:43:00 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 6 Jul 2006 13:43:00 -0700 Subject: [Numpy-discussion] simple question In-Reply-To: <44AD7460.7050109@ee.byu.edu> References: <44AC5B4C.10804@jpl.nasa.gov> <44AD7155.6070901@jpl.nasa.gov> <44AD7460.7050109@ee.byu.edu> Message-ID: On 7/6/06, Travis Oliphant wrote: > Mathew Yeates wrote: > > >Not working. > >A[row,all_dates == 10] = -1 where all_dates is a matrix with column > >length of 14 [[960111,..,.. > >and A is a matrix with same column length > > > >I get > >IndexError: arrays used as indices must be of integer type > > > >when I print out all_dates == 10 > >I get > >[True True True True True True True True True False False False True True]] > > > >I experimented with "<" instead of "==" but I still get boolean values > >as indices. > > > >Any help? > > > > > What version are you using? Can you give an example that shows the > error. It's hard to guess the type of all the variables. The following > works for me. > > import numpy > print numpy.__version__ > A = numpy.matrix(rand(3,14)) > all_dates = array([10,10,1,10,1,10,0,10,0,10,0,1,10,1]) > row = 2 > A[row, all_dates == 10] This is what NASA is doing (and what I would like to do): >> A[row, asmatrix(all_dates == 10)] --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /home/kwg/ /usr/local/lib/python2.4/site-packages/numpy/core/defmatrix.py in __getitem__(self, index) 122 123 def __getitem__(self, index): --> 124 out = N.ndarray.__getitem__(self, index) 125 # Need to swap if slice is on first index 126 # or there is an integer on the second ValueError: too many indices for array From myeates at jpl.nasa.gov Thu Jul 6 17:20:51 2006 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Thu, 06 Jul 2006 14:20:51 -0700 Subject: [Numpy-discussion] More zaniness Re: simple question In-Reply-To: <44AD75A0.8060809@jpl.nasa.gov> References: <44AC5B4C.10804@jpl.nasa.gov> <44AD7155.6070901@jpl.nasa.gov> <44AD7460.7050109@ee.byu.edu> <44AD75A0.8060809@jpl.nasa.gov> Message-ID: <44AD7EB3.1060207@jpl.nasa.gov> okay, I went back to the binary windows distrib. Based on Keths code I wrote >> print numpy.asmatrix(all_dates == start_dates[row],dtype=int) [[0 0 0 0 0 0 0 0 0 0 0 1 0 0]] >> [row,numpy.asmatrix(all_dates == start_dates[row],dtype=int)] = -1 >> print A[row,:] [[-1. -1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]] huh? It set the first 2 elements and not the 12'th!! Mathew From kwgoodman at gmail.com Thu Jul 6 17:43:34 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 6 Jul 2006 14:43:34 -0700 Subject: [Numpy-discussion] More zaniness Re: simple question In-Reply-To: <44AD7EB3.1060207@jpl.nasa.gov> References: <44AC5B4C.10804@jpl.nasa.gov> <44AD7155.6070901@jpl.nasa.gov> <44AD7460.7050109@ee.byu.edu> <44AD75A0.8060809@jpl.nasa.gov> <44AD7EB3.1060207@jpl.nasa.gov> Message-ID: On 7/6/06, Mathew Yeates wrote: > okay, I went back to the binary windows distrib. Based on Keths code I wrote > > >> print numpy.asmatrix(all_dates == start_dates[row],dtype=int) > [[0 0 0 0 0 0 0 0 0 0 0 1 0 0]] > >> [row,numpy.asmatrix(all_dates == start_dates[row],dtype=int)] = -1 > >> print A[row,:] > [[-1. -1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]] > > huh? It set the first 2 elements and not the 12'th!! You are assigning -1 to column 0 thirteen times and assinging -1 to column 1 once. For now, until boolean indexing works with matrices, I would just use brute force: A[row,where(all_dates.A == 10)[0]] Or you can do all rows at once with A[all_dates== 10] where all_dates is the same size as A. From oliphant at ee.byu.edu Thu Jul 6 17:43:59 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 06 Jul 2006 15:43:59 -0600 Subject: [Numpy-discussion] More zaniness Re: simple question In-Reply-To: <44AD7EB3.1060207@jpl.nasa.gov> References: <44AC5B4C.10804@jpl.nasa.gov> <44AD7155.6070901@jpl.nasa.gov> <44AD7460.7050109@ee.byu.edu> <44AD75A0.8060809@jpl.nasa.gov> <44AD7EB3.1060207@jpl.nasa.gov> Message-ID: <44AD841F.2020401@ee.byu.edu> Mathew Yeates wrote: >okay, I went back to the binary windows distrib. Based on Keths code I wrote > > >> print numpy.asmatrix(all_dates == start_dates[row],dtype=int) >[[0 0 0 0 0 0 0 0 0 0 0 1 0 0]] > >> [row,numpy.asmatrix(all_dates == start_dates[row],dtype=int)] = -1 > >> print A[row,:] >[[-1. -1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]] > >huh? It set the first 2 elements and not the 12'th!! > > Indexing has changed in SVN python, but in general, index matrices are not what you want because the dimensionality of the index arrays means something and matrices have two dimensions. So use arrays for indexing. -Travis From oliphant at ee.byu.edu Thu Jul 6 17:49:53 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 06 Jul 2006 15:49:53 -0600 Subject: [Numpy-discussion] Speed degression? In-Reply-To: References: <200607041519.24636.steffen.loeck@gmx.de> <44ACF46C.8050507@ieee.org> <44AD0918.9010702@ieee.org> Message-ID: <44AD8581.5030904@ee.byu.edu> Arnd Baecker wrote: >Bingo! Only numpy 0.9.9.2749 + math is slower than >Numeric+math by a factor 1.6 and mod-array is 2.5 times slower >than numpy. >It would be nice if you could have a look at the module operation, >if possible ... > > O.K. I looked at how Python computed modulo and implemented the same thing for both umath (the remainder function) and scalarmath. The result is a significant speed up for modulo...yeah... I also placed in hooks so you can replace the scalarmath (for int, float, and complex) with the Python version of math (this works because the int, float, and complex scalars are sub-classes of the corresponding Python object). import numpy.core.scalarmath as ncs # Replace "scalarmath" with standard Python arithmetic. ncs.use_pythonmath() # Replace "scalarmath" with actual scalarmath code. ncs.use_scalarmath() This can test the speed of the implementations. -Travis From tim.hochberg at cox.net Thu Jul 6 18:11:08 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 06 Jul 2006 15:11:08 -0700 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> Message-ID: <44AD8A7C.5030408@cox.net> Sasha wrote: > On 7/6/06, Robert Kern wrote: > >> ... >> I don't think that just because arrays are often used for linear algebra that >> linear algebra assumptions should be built in to the core array type. >> >> > > In addition, transpose is a (rank-2) array or matrix operation and not > a linear algebra operation. Transpose corresponds to the "adjoint" > linear algebra operation if you represent vectors as single column > matrices and co-vectors as single-row matrices. This is a convenient > representation followed by much of the relevant literature, but it > does not alow generalization beyond rank-2. Another useful feature is > that inner product can be calculated as the matrix product as long as > you accept a 1x1 matrix for a scalar. This feature does not work > beyond rank-2 either because in order to do tensor inner product you > have to be explicit about the axes being collapsed (for example using > Einstein notation). > At various times, I've thought about how one might do Einstein notation within Python. About the best I could come up with was: A.ijk * B.klm or A("ijk") * B("klm") Neither is spectacular, the first is a cleaner notation, but conceptually messy since it abuses getattr. Both require some intermediate pseudo object that wraps the array as well as info about the indexing. > Since ndarray does not distinguish between upper an lower indices, it > is not possible distinguish between vectors and co-vectors in any way > other than using matrix convention. This makes ndarrays a poor model > for linear algebra tensors. > My tensor math is rusty, but isn't it possible to represent all ones tensors as either covariant and contravariant and just embed the information about the metric into the product operator? It would seem that the inability to specify lower and upper indices is not truly limiting, but the inability to specify what axis to contract over is a fundamental limitation of sorts. I'm sure I'm partly influenced by my feeling that in practice upper and lower indices (aka contra- and covariant- and mixed-tensors) would be a pain in the neck, but a more capable inner product operator might well be useful if we could come up with correct syntax. -tim From wbaxter at gmail.com Thu Jul 6 21:56:43 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 7 Jul 2006 10:56:43 +0900 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44AD8A7C.5030408@cox.net> References: <44ACF15D.6030607@ieee.org> <44AD8A7C.5030408@cox.net> Message-ID: Tim Wrote: > That second argument is particularly uncompelling, but I think I agree > that in a vacuum swapaxes(-2,-1) would be a better choice for .T than > reversing the axes. However, we're not in a vacuum and there are several > reasons not to do this. > 1. A.T and A.transpose() should really have the same behavior. There may be a certain economy to that but I don't see why it should necessarily be so. Especially if it's agreed that the behavior .transpose() is not very useful. The use case for .T is primarily to make linear algebra stuff easier. If you're doing n-dim stuff and need something specific, you'll use the more general .transpose(). 2. Changing A.transpose would be one more backwards compatibility issue. Maybe it's a change worth making though, if we are right in saying that the current .transpose() for ndim>2 is hardly ever what you want. 3. Since, as far as I can tell there's not concise way of spelling > A.swapaxes(-2,-1) in terms of A.transpose it would make documenting and > explaining the default case harder. > Huh? A.swapaxes (-2,-1) is pretty concise. Why should it have to have an explanation in terms of A.transpose? Here's the explanation for the documentation: "A.T returns A with the last two axes transposed. It is equivalent to A.swapaxes (-2,-1). For a 2-d array, this is the usual matrix transpose." This just is a non-issue. Sasha wrote: > > more common to want to swap just two axes, and the last two seem a > logical > > choice since a) in the default C-ordering they're the closest together > in > > memory and b) they're the axes that are printed contiguously when you > say > > "print A". > > It all depends on how you want to interpret a rank-K tensor. You seem > to advocate a view that it is a (K-2)-rank array of matrices and .T is > an element-wise transpose operation. Alternatively I can expect that > it is a matrix of (K-2)-rank arrays and then .T should be > swapaxes(0,1). Do you have real-life applications of swapaxes(-2,-1) > for rank > 2? > Yep, like Tim said. The usage is say a N sets of basis vectors. Each set of basis vectors is a matrix. And say I have a different basis associated with each of N points in space. Usually I'll want to print it out organized by basis vector set. I.e. look at the matrix associated with each of the points. So it makes sense to organize it as shape=(N,a,b) so that if I print it I get something that's easy to interpret. If I set it up as shape=(a,b,N) then what's easiest to see in the print output is all N first basis vectors, all N second basis vectors, etc. Also again in a C memory layout, the last two axes are closest in memory, so it's more cache friendly to have the bits that will usually be used together in computations be on the trailing end. In matlab (which is fortran order), I do things the other way, with the N at the end of the shape. (And note that Matlab prints out the first two axes contiguously.) Either way swapaxes(-2,-1) is likely more likely to be what you want than .transpose(). > > and swapaxes(-2,-1) is > > > invalid for rank < 2. > > > > > At least in numpy 0.9.8, it's not invalid, it just doesn't do anything. > > > > > That's bad. What sense does it make to swap non-existing axes? Many > people would expect transpose of a vector to be a matrix. This is the > case in S+ and R. > Well, I would be really happy for .T to return an (N,1) column vector if handed an (N,) 1-d array. But I'm pretty sure that would raise more furuor among the readers of the list than leaving it 1-d. > > My main objection is that a.T is fairly cryptic > > > - is there any other language that uses attribute for transpose? > > > > > > Does it matter what other languages do? It's not _that_ cryptic. > If something is clear and natural, chances are it was done before. The thing is most other numerical computing languages were designed for doing numerical computing. They weren't designed originally for writing general purpose software, like Python was. So in matlab, for instance, transpose is a simple single-quote. But that doesn't help us decide what it should be in numpy. For me prior art is always a useful guide when making a design choice. > For example, in R, the transpose operation is t(a) and works on rank > <= 2 only always returning rank-2. I have serious reservations about a function called t(). x,y,z, and t are probably all in the top 10 variable names in scientific computing. K (an APL-like language) overloads > unary '+' to do swapaxes(0,1) for rank>=2 and nothing for lower rank. Hmm. That's kind of interesting, it seems like an abuse of notation to me. And precedence might be an issue too. The precedence of unary + isn't as high as attribute access. Anyway, as far as the meaning of + in K, I'm guessing K's arrays are in Fortran order, so (0,1) axes vary the fastest. I couldn't find any documentation for the K language from a quick search, though. Both R and K solutions are implementable in Python with R using 3 > characters and K using 1(!) compared to your two-character ".T" > notation. I would suggest that when inventing something new, you > should consider prior art and explain how you invention is better. > That's why what other languages do matter. (After all, isn't 'T' > chosen because "transpose" starts with "t" in the English language?) Yes you're right. My main thought was just what I said above, that there probably aren't too many other examples that can really apply in this case, both because most numerical computing languages are custom-designed for numerical computing, and also because Python's attributes are also kind of uncommon among programming languages. So it's worth looking at other examples, but in the end it has to be something that makes sense for a numerical computing package written in Python, and there aren't too many examples of that around. > You could write a * b.transpose(1,0) > > right now and still not know whether it was matrix or element-wise > > multiplication. > > Why would anyone do that if b was a matrix? > Maybe because, like you, they think "that a.T is fairly cryptic". > But probably a better solution > > would be to have matrix versions of these in the library as an optional > > module to import so people could, say, import them as M and use M.ones > (2,2). > > > This is the solution used by ma, which is another argument for it. > Yeh, I'm starting to think that's better than slapping an M attribute on arrays, too. Is it hard to write a module like that? I only raised a mild objection against .T, but the slippery slope > argument makes me dislike it much more. At the very least I would > like to see a discussion of why a.T is better than t(a) or +a. > * A.T puts the T on the proper side of A, so in that sense it looks more like the standard math notation. * A.T has precedence that roughly matches the standard math notation * t(A) uses an impossibly short function name that's likely to conflict with local variable names. To avoid the conflict people will just end up using it as numpy.t(A), at which point it's value as a shortcut for transpose is nullified. Or they'll have to do a mini-import within specific functions ("from numpy import t") to localize the namespace pollution. But at that point they might as well just say " t= numpy.transpose". * t(A) puts the transpose operator on the wrong side of A * +A puts the transpose operator on the wrong side of A also. * +A implies addition. The general rule with operator overloading is that the overload should have the same general meaning as the original operator. So overloading * for matrix multiplication makes sense. Overloading & for would be a bad idea. New users looking at something like A + +B are pretty certain to be confused because they think they know what + means, but they're wrong. If you see A + B.T, you either know what it means or you know immediately that you don't know what it means and you go look it up. * +A has different precedence than the usual transpose operator. (But I can't think of a case where that would make a difference now.) Tim Hochberg wrote: > > Well, you could overload __rpow__ for a singleton T and spell it A**T > > ... (I hope no one will take that proposal seriosely). Visually, A.T > > looks more like a subscript rather than superscript. > > > No, no no. Overload __rxor__, then you can spell it A^t, A^h, etc. Much > better ;-). [Sadly, I almost like that....] > Ouch! No way! It's got even worse precedence problems than the +A proposal. How about A+B^t ? And you still have to introduce 'h' and 't' into the global namespace for it to work. Here's a half baked thought: if the objection to t(A) is that it doesn't > mirror the formulae where t appears as a subscript after A. Conceivably, > __call__ could be defined so that A(x) returns x(A). That's kind of > perverse, but it means that A(t), A(h), etc. could all work > appropriately for suitably defined singletons. These singletons could > either be assembeled in some abbreviations namespace or brought in by > the programmer using "import transpose as t", etc. The latter works for > doing t(a) as well of course. Same problem with the need for global t. And it is kind of perverse, besides. Robert Kern wrote: > Like Sasha, I'm mildly opposed to .T (as a synonym for .transpose()) and > much > more opposed to the rest (including .T being a synonym for .swapaxes(-2, > -1)). > It's not often that a proposal carries with it its own slippery-slope > argument > against itself. > The slippery slope argument only applies to the .M, not the .T or .H. And I think if there's a matrixutils module with redefinitions of ones and zeros etc, and if other functions are all truly fixed to preserve matrix when matrix is passed in, then I agree, there's not so much need for .M. I don't think that just because arrays are often used for linear algebra > that > linear algebra assumptions should be built in to the core array type. It's not just that "arrays can be used for linear algebra". It's that linear algebra is the single most popular kind of numerical computing in the world! It's the foundation for a countless many fields. What you're saying is like "grocery stores shouldn't devote so much shelf space to food, because food is just one of the products people buy", or "this mailing list shouldn't be conducted in English, because English is just one of the languages people can speak here", or "I don't think my keyboard should devote so much space to the A-Z keys, because there are so many characters in the Unicode character set that could be there instead", or to quote from a particular comedy troop: "Ah, how about Cheddar?" "Well, we don't get much call for it around here, sir." "Not much ca- It's the single most popular cheese in the world!" "Not round here, sir." Linear algebra is pretty much the 'cheddar' of the numerical computing world. But it's more than that. It's like the yeast of the beer world. Pretty much everything starts with it as a base. It makes sense to make it as convenient as possible to do with numpy, even if it is a "special case". I wish I could think of some sort of statistics or google search I could cite to back this claim up, but as far as my academic background from high school though Ph.D. goes, linear algebra is a mighty big deal, not merely an "also ran" in the world of math or numerical computing. Sasha Wrote: > In addition, transpose is a (rank-2) array or matrix operation and not > a linear algebra operation. Transpose corresponds to the "adjoint" > linear algebra operation if you represent vectors as single column > matrices and co-vectors as single-row matrices. This is a convenient > representation followed by much of the relevant literature, but it > does not alow generalization beyond rank-2. > I would be willing to accept a .T that just threw an exception if ndim were > 2. That's what Matlab does with its transpose operator. I don't like that behavior myself -- it seems wasteful when it could just have some well defined behavior that would let it be useful at least some of the time on N-d arrays. I don't like it either, but I don't like .T even more. These days I > hate functionality I cannot google for. Call me selfish, but I > already know what unary '+' can do to a higher rank array, but with .T > I will always have to look up which axes it swaps ... I think '.T' is more likely to be searchable than '+'. And when you say you already know what unary + can do, you mean because you've used K? That's not much use to the typical user, who also thinks they know what a unary + does, but they'd be wrong in this case. So, in summary, I vote for: - Keep the .T and the .H on array - Get rid of .M - Instead implement a matrix helper module that could be imported as M, allowing M.ones(...) etc. And also: - Be diligent about fixing any errors from matrix users along the lines of " numpy.foo returns an array when given a matrix" (Travis has been good about this -- but we need to keep it up.) Part of the motivation for .M attribute was just as a band-aid on the problem of matrices getting turned into arrays. Having .M means you can just slap a .M on the end of any result you aren't sure about. It's better (but harder) to fix the upstream problem of functions not preserving subtypes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Thu Jul 6 22:20:58 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 7 Jul 2006 11:20:58 +0900 Subject: [Numpy-discussion] Enhancing dot() Message-ID: On 7/7/06, Tim Hochberg wrote: > I'd caution here though that the H is another thing that's going to > encourage people to write code that's less accurate and slower than it > needs to be. Consider the simple equation: > > [Side note: Here's something I periodically think about -- it would be > nice if dot could take multiple arguments and chain them into a series > of matrix multiplies so that dot(dot(a, b), a.H) could be written dot(a, > b, a.H). I'll use that here for clarity]. > > x = dot(a, b, a.H) > > versus: > > x = dot(a.real, b, a.real.transpose()) + dot(a.imag, b, a.imag.transpose > ()) > > The latter should be about twice as fast and won't have any pesky > imaginary residuals showing up from the finite precision of the various > operations. The funny thing is that having a dot(a,b,c,...) would lead to the exact same kind of hidden performance problems you're arguing against. The cost of a series of matrix multiplications can vary dramatically with the order in which you perform them. E.g. A*B*C*v where A,B,C are 2-dim and v is a column vector. Then you should do (A*(B*(C*v))), and definitely not ((A*B)*C)*v) But that said, dot could maybe be made smart enough to figure out the best order. That would be a nice feature. dot(A, dot(B, dot(C, v))) is pretty darn ugly. I seem to remember finding the best order being an example of a dynamic programming problem, and probably O(N^2) in the number of terms. But presumably N is never going to be very big, and the N==2 (no choice) and N=3 (only one choice) cases could be optimized. --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Jul 6 22:31:07 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 06 Jul 2006 21:31:07 -0500 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44AD8A7C.5030408@cox.net> Message-ID: Bill Baxter wrote: > Robert Kern wrote: > > Like Sasha, I'm mildly opposed to .T (as a synonym for .transpose()) > and much > more opposed to the rest (including .T being a synonym for > .swapaxes(-2, -1)). > It's not often that a proposal carries with it its own > slippery-slope argument > against itself. > > The slippery slope argument only applies to the .M, not the .T or .H. No, it was the "Let's have a .T attribute. And if we're going to do that, then we should also do this. And this. And this." > I don't think that just because arrays are often used for linear > algebra that > > linear algebra assumptions should be built in to the core array type. > > It's not just that "arrays can be used for linear algebra". It's that > linear algebra is the single most popular kind of numerical computing in > the world! It's the foundation for a countless many fields. What > you're saying is like "grocery stores shouldn't devote so much shelf > space to food, because food is just one of the products people buy", or [etc.] I'm sorry, but the argument-by-inappropriate-analogy is not convincing. Just because linear algebra is "the base" for a lot of numerical computing does not mean that everyone is using numpy arrays for linear algebra all the time. Much less does it mean that all of those conventions you've devised should be shoved into the core array type. I hold a higher standard for the design of the core array type than I do for the stuff around it. "It's convenient for what I do," just doesn't rise to that level. There has to be more of an argument for it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From wbaxter at gmail.com Thu Jul 6 22:56:11 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 7 Jul 2006 11:56:11 +0900 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44AD8A7C.5030408@cox.net> Message-ID: On 7/7/06, Robert Kern wrote: > > Bill Baxter wrote: > > Robert Kern wrote: > > > > > > The slippery slope argument only applies to the .M, not the .T or .H. > > No, it was the "Let's have a .T attribute. And if we're going to do that, > then > we should also do this. And this. And this." There's no slippery slope there. It's just "Let's have a .T attribute, and if we have that then we should have .H also." Period. The slope stops there. The .M and .A are a separate issue. > I don't think that just because arrays are often used for linear > > algebra that > > > > linear algebra assumptions should be built in to the core array > type. > > > > It's not just that "arrays can be used for linear algebra". It's that > > linear algebra is the single most popular kind of numerical computing in > > the world! It's the foundation for a countless many fields. What > > you're saying is like "grocery stores shouldn't devote so much shelf > > space to food, because food is just one of the products people buy", or > [etc.] > > I'm sorry, but the argument-by-inappropriate-analogy is not convincing. > Just > because linear algebra is "the base" for a lot of numerical computing does > not > mean that everyone is using numpy arrays for linear algebra all the time. > Much > less does it mean that all of those conventions you've devised should be > shoved > into the core array type. I hold a higher standard for the design of the > core > array type than I do for the stuff around it. "It's convenient for what I > do," > just doesn't rise to that level. There has to be more of an argument for > it. My argument is not that "it's convenient for what I do", it's that "it's convenient for what 90% of users want to do". But unfortunately I can't think of a good way to back up that claim with any sort of numbers. But here's one I just found: http://www.netlib.org/master_counts2.html download statistics for various numerical libraries on netlib.org. The top 4 are all linear algebra related: /lapack 37,373,505 /lapack/lug 19,908,865 /scalapack 14,418,172 /linalg 11,091,511 The next three are more like general computing issues: parallelization lib, performance monitoring, benchmarks: /pvm3 10,360,012 /performance 7,999,140 /benchmark 7,775,600 Then the next one is more linear algebra. And that seems to hold pretty far down the list. It looks like mostly stuff that's either linear algebra related or parallelization/benchmarking related. And as another example, there's the success of higher level numerical environments like Matlab (and maybe R and S? and Mathematica, and Maple?) that have strong support for linear algebra right in the core, not requiring users to go into some syntax/library ghetto to use that functionality. I am also curious, given the number of times I've heard this nebulous argument of "there are lots kinds of numerical computing that don't invlolve linear algebra", that no one ever seems to name any of these "lots of kinds". Statistics, maybe? But you can find lots of linear algebra in statistics. --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Jul 7 00:23:06 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 06 Jul 2006 23:23:06 -0500 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44AD8A7C.5030408@cox.net> Message-ID: Bill Baxter wrote: > On 7/7/06, *Robert Kern* > wrote: > > Bill Baxter wrote: > > Robert Kern wrote: [snip] > > I don't think that just because arrays are often used for linear > > algebra that > > > > linear algebra assumptions should be built in to the core > array type. > > > > It's not just that "arrays can be used for linear algebra". It's > that > > linear algebra is the single most popular kind of numerical > computing in > > the world! It's the foundation for a countless many fields. What > > you're saying is like "grocery stores shouldn't devote so much shelf > > space to food, because food is just one of the products people > buy", or > [etc.] > > I'm sorry, but the argument-by-inappropriate-analogy is not > convincing. Just > because linear algebra is "the base" for a lot of numerical > computing does not > mean that everyone is using numpy arrays for linear algebra all the > time. Much > less does it mean that all of those conventions you've devised > should be shoved > into the core array type. I hold a higher standard for the design of > the core > array type than I do for the stuff around it. "It's convenient for > what I do," > just doesn't rise to that level. There has to be more of an argument > for it. > > My argument is not that "it's convenient for what I do", it's that "it's > convenient for what 90% of users want to do". But unfortunately I can't > think of a good way to back up that claim with any sort of numbers. [snip] > I am also curious, given the number of times I've heard this nebulous > argument of "there are lots kinds of numerical computing that don't > invlolve linear algebra", that no one ever seems to name any of these > "lots of kinds". Statistics, maybe? But you can find lots of linear > algebra in statistics. That's because I'm not waving my hands at general fields of application. I'm talking about how people actually use array objects on a line-by-line basis. If I represent a dataset as an array and fit a nonlinear function to that dataset, am I using linear algebra at some level? Sure! Does having a .T attribute on that array help me at all? No. Arguing about how fundamental linear algebra is to numerical endeavors is entirely besides the point. I'm not saying that people who do use arrays for linear algebra are rare or unimportant. It's that syntactical convenience for one set of conventional ways to use an array object, by itself, is not a good enough reason to add stuff to the core array object. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant.travis at ieee.org Fri Jul 7 00:26:12 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 06 Jul 2006 22:26:12 -0600 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes Message-ID: <44ADE264.2030903@ieee.org> This is a call for a vote on each of the math attributes. Please post your vote as +1 : support +0 : don't care so go ahead -0 : don't care so why do it -1 : against Vote on the following issues separately: 1) .T Have some kind of .T attribute If >0 on this then: a) .T == .swapaxes(-2,-1) b) .T == .transpose() c) .T raises error for ndim > 2 d) .T returns (N,1) array for length (N,) array e) .T returns self for ndim < 2 2) .H returns .T.conj() 3) .M returns matrix version of array 4) .A returns basearray (useful for sub-classes). -Travis From oliphant.travis at ieee.org Fri Jul 7 00:28:19 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 06 Jul 2006 22:28:19 -0600 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE264.2030903@ieee.org> References: <44ADE264.2030903@ieee.org> Message-ID: <44ADE2E3.60102@ieee.org> Travis Oliphant wrote: > This is a call for a vote on each of the math attributes. Please post > your vote as > > +1 : support > +0 : don't care so go ahead > -0 : don't care so why do it > -1 : against > > Vote on the following issues separately: > > > > 1) .T Have some kind of .T attribute > > +1 > If >0 on this then: > > a) .T == .swapaxes(-2,-1) > +0 > b) .T == .transpose() > -0 > c) .T raises error for ndim > 2 > -1 > d) .T returns (N,1) array for length (N,) array > > +1 > e) .T returns self for ndim < 2 > > -1 > 2) .H returns .T.conj() > > > +0 > 3) .M returns matrix version of array > > +0 > 4) .A returns basearray (useful for sub-classes). > +0 -Travis From ndarray at mac.com Fri Jul 7 00:34:49 2006 From: ndarray at mac.com (Sasha) Date: Fri, 7 Jul 2006 00:34:49 -0400 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44AD8A7C.5030408@cox.net> Message-ID: On 7/6/06, Bill Baxter wrote: > ... > Yep, like Tim said. The usage is say a N sets of basis vectors. Each set > of basis vectors is a matrix. This brings up a feature that I really miss from numpy: an ability to do array([f(x) for x in a]) without python overhead. APL-like languages have a notion of "adverb" - a higher level operator that maps a function to a function. Numpy has some adverbs implemented as attributes to ufuncs: for example add.reduce is the same as +/ in K and add.accumulate is the same as +\ ('/' and '\' are 'over' and 'scan' adverbs in K). However, there is no way to do f/ or f\ where f is an arbitrary dyadic function. The equivalent of array([f(x) for x in a]) is spelled f'(a) in K (' is an adverb 'each'). The transpose operator (+) is swaps the first two axes, so in order to apply to the array of matrices, one would have to do +:'a (: in +: disambiguates + as a unary operator). I don't know of a good way to introduce adverbs in numpy, nor can I think of a good way to do list comprehensions, but array friendly versions of map, filter and reduce may be a good addition. These higher order functions may take an optional axes argument to deal with the higher rank arrays and may be optimized to recognize ufuncs so that map(f, a) could call f(a) and reduce(f, a) could do f.reduce(a) when f is a ufunc. [snip] > Either way swapaxes(-2,-1) is likely more likely to be what you want than > .transpose(). > Agree, but swapaxes(0, 1) is a close runner-up which is also known as zip in python. > Well, I would be really happy for .T to return an (N,1) column vector if > handed an (N,) 1-d array. But I'm pretty sure that would raise more furuor > among the readers of the list than leaving it 1-d. > Would you be even happier if .T would return a matrix? I hope not because my .M objection will apply. Maybe we can compromize by implementing a.T so that it raises ValueError unless rank(a) == 2 or at least unless rank(a) <= 2? > I have serious reservations about a function called t(). x,y,z, and t are > probably all in the top 10 variable names in scientific computing. > What about T()? > > > K (an APL-like language) overloads > > unary '+' to do swapaxes(0,1) for rank>=2 and nothing for lower rank. > > > Hmm. That's kind of interesting, it seems like an abuse of notation to me. > And precedence might be an issue too. The precedence of unary + isn't as > high as attribute access. It is high enough AFAICT - higher than any binary operator. > Anyway, as far as the meaning of + in K, I'm > guessing K's arrays are in Fortran order, so (0,1) axes vary the fastest. No, K has 1d arrays only, but they can be nested. Matrices are arrays of arrays and tensors are arrays of arrays of arrays ..., but you are right (0,1) swap is faster than (-2,-1) swap and this motivated the choice for the primitive. > I couldn't find any documentation for the K language from a quick search, > though. Kx Systems, the company behind K has replaced K with Q and pulled old manuals from the web. Q is close enough to K: see http://kx.com/q/d/k.txt for a terse summary. [snip] > > Why would anyone do that if b was a matrix? > Maybe because, like you, they think "that a.T is fairly cryptic". > If they are like me, they will not use numpy.matrix to begin with :-). > > > > But probably a better solution > > > would be to have matrix versions of these in the library as an optional > > > module to import so people could, say, import them as M and use > M.ones(2,2). > > > > > > > This is the solution used by ma, which is another argument for it. > > Yeh, I'm starting to think that's better than slapping an M attribute on > arrays, too. Is it hard to write a module like that? > Writing matrixutils with def zeros(shape, dtype=float): return asmatrix(zeros(shape, dtype)) is trivial, but matrixutils.zeros will have two python function calls overhead. This may be a case for making zeros a class method of ndarray that can be written in a way that will make inherited matrix.zeros do the right thing with no overhead. [snip] > * +A implies addition. No, it does not. Unary '+' is a noop. Does * imply multiplication or ** imply pow in f(*args, **kwds) to you? > The general rule with operator overloading is that > the overload should have the same general meaning as the original operator. Unary '+' has no preset meaning in plain python. It can be interpreted as transpose if you think of scalars as 1x1 matrices. > So overloading * for matrix multiplication makes sense. It depends on what you consider part of "general meaning". If the commutativity property is part of it then overloading * for matrix multiplication doesn't make sense. If the "general meaning" of unary + includes x = +x invariant, then you are right, but I am willing to relax that to x = ++x invariant when x is a non-symmetric matrix. > ... New users looking at something like A + +B are pretty > certain to be confused because they think they know what + means, but > they're wrong. In my experience new users don't realize that unary + is defined for arrays. Use of unary + with non-literal numbers is exotic enough that new users seeing "something like A + +B" will not assume that they know what it means. [snip] > * +A has different precedence than the usual transpose operator. (But I > can't think of a case where that would make a difference now.) > Maybe you can't because it doesn't? :-) > I would be willing to accept a .T that just threw an exception if ndim were > > 2. Aha! Let's start with an error unless ndim != 2. It is always easier to add good features than to remove bad ones. From robert.kern at gmail.com Fri Jul 7 00:47:59 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 06 Jul 2006 23:47:59 -0500 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE264.2030903@ieee.org> References: <44ADE264.2030903@ieee.org> Message-ID: Travis Oliphant wrote: > This is a call for a vote on each of the math attributes. Please post > your vote as > > +1 : support > +0 : don't care so go ahead > -0 : don't care so why do it > -1 : against > > Vote on the following issues separately: > > 1) .T Have some kind of .T attribute > > If >0 on this then: > > a) .T == .swapaxes(-2,-1) > b) .T == .transpose() > c) .T raises error for ndim > 2 > d) .T returns (N,1) array for length (N,) array > e) .T returns self for ndim < 2 The fact that there are a, b, c, d, and e makes me fully -1 on this. In the face of ambiguity, refuse the temptation to guess. > 2) .H returns .T.conj() -1 > 3) .M returns matrix version of array -1 > 4) .A returns basearray (useful for sub-classes). -1 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ndarray at mac.com Fri Jul 7 00:53:31 2006 From: ndarray at mac.com (Sasha) Date: Fri, 7 Jul 2006 00:53:31 -0400 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE264.2030903@ieee.org> References: <44ADE264.2030903@ieee.org> Message-ID: On 7/7/06, Travis Oliphant wrote: > 1) .T Have some kind of .T attribute > -1 (but -0 if raises an error if ndim != 2) > If >0 on this then: > > a) .T == .swapaxes(-2,-1) > > b) .T == .transpose() > > c) .T raises error for ndim > 2 > > d) .T returns (N,1) array for length (N,) array > > e) .T returns self for ndim < 2 > Write-in: f) .T raises error for ndim != 2 (still <0) > > 2) .H returns .T.conj() > -1 > > 3) .M returns matrix version of array > -1 > > 4) .A returns basearray (useful for sub-classes). > -1 From tim.hochberg at cox.net Fri Jul 7 02:03:48 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 06 Jul 2006 23:03:48 -0700 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44AD8A7C.5030408@cox.net> Message-ID: <44ADF944.5020003@cox.net> Sasha wrote: > On 7/6/06, Bill Baxter wrote: > >> ... >> Yep, like Tim said. The usage is say a N sets of basis vectors. Each set >> of basis vectors is a matrix. >> > > This brings up a feature that I really miss from numpy: an ability to do > > array([f(x) for x in a]) > Please note that there is now a fromiter function so that much of the overhead of the above function can be removed by using: numpy.fromiter((f(x) for x in a), float) This won't generate an intermediate list or use significantly extra storage. I doubt it's a full replacement for adverbs as you've described below though. -tim > without python overhead. APL-like languages have a notion of "adverb" > - a higher level operator that maps a function to a function. Numpy > has some adverbs implemented as attributes to ufuncs: for example > add.reduce is the same as +/ in K and add.accumulate is the same as +\ > ('/' and '\' are 'over' and 'scan' adverbs in K). However, there is no > way to do f/ or f\ where f is an arbitrary dyadic function. > > The equivalent of array([f(x) for x in a]) is spelled f'(a) in K (' is > an adverb 'each'). The transpose operator (+) is swaps the first two > axes, so in order to apply to the array of matrices, one would have to > do +:'a (: in +: disambiguates + as a unary operator). > > I don't know of a good way to introduce adverbs in numpy, nor can I > think of a good way to do list comprehensions, but array friendly > versions of map, filter and reduce may be a good addition. These > higher order functions may take an optional axes argument to deal with > the higher rank arrays and may be optimized to recognize ufuncs so > that map(f, a) could call f(a) and reduce(f, a) could do f.reduce(a) > when f is a ufunc. > > [snip] > >> Either way swapaxes(-2,-1) is likely more likely to be what you want than >> .transpose(). >> >> > > Agree, but swapaxes(0, 1) is a close runner-up which is also known as > zip in python. > > >> Well, I would be really happy for .T to return an (N,1) column vector if >> handed an (N,) 1-d array. But I'm pretty sure that would raise more furuor >> among the readers of the list than leaving it 1-d. >> >> > > Would you be even happier if .T would return a matrix? I hope not > because my .M objection will apply. Maybe we can compromize by > implementing a.T so that it raises ValueError unless rank(a) == 2 or > at least unless rank(a) <= 2? > > >> I have serious reservations about a function called t(). x,y,z, and t are >> probably all in the top 10 variable names in scientific computing. >> >> > > What about T()? > > >>> K (an APL-like language) overloads >>> unary '+' to do swapaxes(0,1) for rank>=2 and nothing for lower rank. >>> >> Hmm. That's kind of interesting, it seems like an abuse of notation to me. >> And precedence might be an issue too. The precedence of unary + isn't as >> high as attribute access. >> > > It is high enough AFAICT - higher than any binary operator. > > >> Anyway, as far as the meaning of + in K, I'm >> guessing K's arrays are in Fortran order, so (0,1) axes vary the fastest. >> > > No, K has 1d arrays only, but they can be nested. Matrices are arrays > of arrays and tensors are arrays of arrays of arrays ..., but you are > right (0,1) swap is faster than (-2,-1) swap and this motivated the > choice for the primitive. > > >> I couldn't find any documentation for the K language from a quick search, >> though. >> > > Kx Systems, the company behind K has replaced K with Q and pulled old > manuals from the web. Q is close enough to K: see > http://kx.com/q/d/k.txt for a terse summary. > > [snip] > >>> Why would anyone do that if b was a matrix? >>> >> Maybe because, like you, they think "that a.T is fairly cryptic". >> >> > If they are like me, they will not use numpy.matrix to begin with :-). > > >>>> But probably a better solution >>>> would be to have matrix versions of these in the library as an optional >>>> module to import so people could, say, import them as M and use >>>> >> M.ones(2,2). >> >>> This is the solution used by ma, which is another argument for it. >>> >> Yeh, I'm starting to think that's better than slapping an M attribute on >> arrays, too. Is it hard to write a module like that? >> >> > > Writing matrixutils with > > def zeros(shape, dtype=float): > return asmatrix(zeros(shape, dtype)) > > is trivial, but matrixutils.zeros will have two python function calls > overhead. This may be a case for making zeros a class method of > ndarray that can be written in a way that will make inherited > matrix.zeros do the right thing with no overhead. > > [snip] > >> * +A implies addition. >> > No, it does not. Unary '+' is a noop. Does * imply multiplication or > ** imply pow in f(*args, **kwds) to you? > > >> The general rule with operator overloading is that >> the overload should have the same general meaning as the original operator. >> > Unary '+' has no preset meaning in plain python. It can be interpreted > as transpose if you think of scalars as 1x1 matrices. > > >> So overloading * for matrix multiplication makes sense. >> > > It depends on what you consider part of "general meaning". If the > commutativity property is part of it then overloading * for matrix > multiplication doesn't make sense. If the "general meaning" of unary + > includes x = +x invariant, then you are right, but I am willing to > relax that to x = ++x invariant when x is a non-symmetric matrix. > > >> ... New users looking at something like A + +B are pretty >> certain to be confused because they think they know what + means, but >> they're wrong. >> > > In my experience new users don't realize that unary + is defined for > arrays. Use of unary + with non-literal numbers is exotic enough that > new users seeing "something like A + +B" will not assume that they > know what it means. > > [snip] > >> * +A has different precedence than the usual transpose operator. (But I >> can't think of a case where that would make a difference now.) >> >> > Maybe you can't because it doesn't? :-) > > >> I would be willing to accept a .T that just threw an exception if ndim were >> >>> 2. >>> > > Aha! Let's start with an error unless ndim != 2. It is always easier > to add good features than to remove bad ones. > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From wbaxter at gmail.com Fri Jul 7 02:17:36 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 7 Jul 2006 15:17:36 +0900 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44AD8A7C.5030408@cox.net> Message-ID: On 7/7/06, Robert Kern wrote: > > Bill Baxter wrote: > > I am also curious, given the number of times I've heard this nebulous > > argument of "there are lots kinds of numerical computing that don't > > invlolve linear algebra", that no one ever seems to name any of these > > "lots of kinds". Statistics, maybe? But you can find lots of linear > > algebra in statistics. > > That's because I'm not waving my hands at general fields of application. > I'm > talking about how people actually use array objects on a line-by-line > basis. If > I represent a dataset as an array and fit a nonlinear function to that > dataset, > am I using linear algebra at some level? Sure! Does having a .T attribute > on > that array help me at all? No. Arguing about how fundamental linear > algebra is > to numerical endeavors is entirely besides the point. Ok. If line-by-line usage is what everyone really means, then I'll get off the linear algebra soap box, but that's not what it sounded like to me. So, if you want to talk line-by-line, I really can't talk about much beside my own code. But I just grepped through it and out of 2445 non-empty lines of code: 927 lines contain '=' 390 lines contain a '[' 75 lines contain matrix,asmatrix, or mat ==> 47 lines contain a '.T' or '.transpose' of some sort. <== 33 lines contain array, or asarray, or asanyarray 24 lines contain 'rand(' --- I use it for generating bogus test data a lot 17 lines contain 'newaxis' or 'NewAxis' 16 lines contain 'zeros(' 13 lines contain 'dot(' 12 lines contain 'empty(' 8 lines contain 'ones(' 7 lines contain 'inv(' I'm pretty new to numpy, so that's all the code I got right now. I'm sure I've written many more lines of emails about numpy than I have lines of actual numpy code. :-/ But from that, I can say that -- at least in my code -- transpose is pretty common. If someone can point me to some larger codebases written in numpy or numeric, I'd be happy to do a similar analysis of those. I'm not saying that people who do use arrays for linear algebra are rare or > unimportant. It's that syntactical convenience for one set of conventional > ways > to use an array object, by itself, is not a good enough reason to add > stuff to > the core array object. I wish I had a way to magically find out the distribution of array dimensions used by all numpy and numeric code out there. My guess is it would be something like 1-d: 50%, 2-d: 30%, 3-d: 10%, everything else: 10%. I can't think of a good way to even get an estimate on that. But in any event, I'm positive ndims==2 is a significant percentage of all usages. It seems like the opponents to this idea are suggesting the distribution is more flat than that. But whatever the distribution is, it has to have a fairly light tail since memory usage is exponential in ndim. If ndim == 20, then it takes 8 megabytes just to store the smallest possible non-degenerate array of float64s (i.e. a 2x2x2x2x...) It seems crazy to even be arguing this. Transposing is not some specialized esoteric operation. It's important enough that R and S give it a one letter function, and Matlab, Scilab, K all give it a single-character operator. [*] Whoever designed the numpy.matrix class also thought it was worthy of a shortcut, and I think came up with a pretty good syntax for it. And the people who invented math itself decided it was worth assigning a 1-character exponent to it. So I think there's a clear argument for having a .T attribute. But ok, let's say you're right, and a lot of people won't use it. Fine. IT WILL DO THEM ABSOLUTELY NO HARM. They don't have to use it if they don't like it! Just ignore it. Unlike a t() function, .T doesn't pollute any namespace users can define symbols in, so you really can just ignore it if you're not interested in using it. It won't get in your way. For the argument that ndarray should be pure like the driven snow, just a raw container for n-dimensional data, I think that's what the basearray thing that goes into Python itself should be. ndarray is part of numpy and numpy is for numerical computing. Regards, --Bill [*] Full disclosure: I did find two counter-examples -- Maple and Mathematica. Maple has only a transpose() function and Mathematica has only Transpose[] (but you can use [esc]tr[esc] as a shortcut) However, both of those packages are primarily known for their _symbolic_ math capabilities, not their number crunching, so they less are similar to numpy than R,S,K,Matlab and Scilab in that regard. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at cox.net Fri Jul 7 02:26:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 06 Jul 2006 23:26:04 -0700 Subject: [Numpy-discussion] Enhancing dot() In-Reply-To: References: Message-ID: <44ADFE7C.5040506@cox.net> Bill Baxter wrote: > On 7/7/06, *Tim Hochberg* > wrote: > > I'd caution here though that the H is another thing that's going to > encourage people to write code that's less accurate and slower than it > needs to be. Consider the simple equation: > > [Side note: Here's something I periodically think about -- it > would be > nice if dot could take multiple arguments and chain them into a series > of matrix multiplies so that dot(dot(a, b), a.H) could be written > dot(a, > b, a.H). I'll use that here for clarity]. > > x = dot(a, b, a.H) > > versus: > > x = dot(a.real, b, a.real.transpose()) + dot(a.imag, b, > a.imag.transpose()) > > The latter should be about twice as fast and won't have any pesky > imaginary residuals showing up from the finite precision of the > various > operations. > > > The funny thing is that having a dot(a,b,c,...) would lead to the > exact same kind of hidden performance problems you're arguing against. Not exactly arguing -- this isn't why I don't like H and friends -- just noting that this is one of the traps that people are likely to fall into when transferring equations to code. > The cost of a series of matrix multiplications can vary dramatically > with the order in which you perform them. E.g. A*B*C*v where A,B,C > are 2-dim and v is a column vector. Then you should do (A*(B*(C*v))), > and definitely not ((A*B)*C)*v) That's a good point. > But that said, dot could maybe be made smart enough to figure out the > best order. That would be a nice feature. > dot(A, dot(B, dot(C, v))) is pretty darn ugly. I seem to remember > finding the best order being an example of a dynamic programming > problem, and probably O(N^2) in the number of terms. But presumably N > is never going to be very big, and the N==2 (no choice) and N=3 (only > one choice) cases could be optimized. You could probably just brute force it with out too much trouble. As you say O(n^2) is not that daunting when n is usually 2 or 3 and rarely over 5. Perhaps cache the last few configurations tested since the same matrix size combinations will likely occur over and over. Another interesting thought is one just wrapped an array with a wrapper that says "use the conjugate of what I'm wrapping", then dot(a, b H(a)) could do the write thing and only compute the real part of the equation, assuming H(a) created the wrapper. That's starting to get complex though. -tim From wbaxter at gmail.com Fri Jul 7 02:29:06 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 7 Jul 2006 15:29:06 +0900 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE264.2030903@ieee.org> References: <44ADE264.2030903@ieee.org> Message-ID: On 7/7/06, Travis Oliphant wrote: > > > 1) .T Have some kind of .T attribute +1 If >0 on this then: > > a) .T == .swapaxes(-2,-1) +1 b) .T == .transpose() +0 c) .T raises error for ndim > 2 +0 d) .T returns (N,1) array for length (N,) array +1 e) .T returns self for ndim < 2 +0 2) .H returns .T.conj() +1 3) .M returns matrix version of array -1 4) .A returns basearray (useful for sub-classes). +0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Fri Jul 7 02:44:14 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 7 Jul 2006 15:44:14 +0900 Subject: [Numpy-discussion] Enhancing dot() In-Reply-To: <44ADFE7C.5040506@cox.net> References: <44ADFE7C.5040506@cox.net> Message-ID: On 7/7/06, Tim Hochberg wrote: > > > The funny thing is that having a dot(a,b,c,...) would lead to the > > exact same kind of hidden performance problems you're arguing against. > Not exactly arguing -- this isn't why I don't like H and friends -- just > noting that this is one of the traps that people are likely to fall into > when transferring equations to code. There's a strong argument to be made that the whole design of most array math packages is flawed and leads to inefficient code. The classic example is something like: A = B + C - 2*D where all the matrices are 2million x 2million. I think for numpy that would basically do: tmp = B+C tmp2 = 2*D tmp3 = tmp - tmp2 A = tmp3 Allocating three huge tmp variables and probably doing an extra copy or two in there, when the best thing to do would be more like: A = D A *= -2 A += C A += B Or something like that. The point is that you give up any notion of having optimal code the minute you start using something like numpy. And you do so happily for the ability to get stuff done faster and have nicer looking, more readable code in the end. When everything works, that's when you hit the "go fast button" if you even really need to. --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at cox.net Fri Jul 7 02:53:29 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 06 Jul 2006 23:53:29 -0700 Subject: [Numpy-discussion] Enhancing dot() In-Reply-To: References: <44ADFE7C.5040506@cox.net> Message-ID: <44AE04E9.9040105@cox.net> Bill Baxter wrote: > On 7/7/06, *Tim Hochberg* > wrote: > > > The funny thing is that having a dot(a,b,c,...) would lead to the > > exact same kind of hidden performance problems you're arguing > against. > Not exactly arguing -- this isn't why I don't like H and friends > -- just > noting that this is one of the traps that people are likely to > fall into > when transferring equations to code. > > > There's a strong argument to be made that the whole design of most > array math packages is flawed and leads to inefficient code. The > classic example is something like: > A = B + C - 2*D > where all the matrices are 2million x 2million. I think for numpy > that would basically do: > tmp = B+C > tmp2 = 2*D > tmp3 = tmp - tmp2 > A = tmp3 > > Allocating three huge tmp variables and probably doing an extra copy > or two in there, when the best thing to do would be more like: > A = D > A *= -2 > A += C > A += B > > Or something like that. Yeah -- that's not quite right since you've clobbered B. But that is essentially the dark road that people go down in performance critical sections on occasion. > The point is that you give up any notion of having optimal code the > minute you start using something like numpy. And you do so happily > for the ability to get stuff done faster and have nicer looking, more > readable code in the end. When everything works, that's when you hit > the "go fast button" if you even really need to. That's what numexpr is for. A = numexpr.evaluate("B + C - 2*D") Evaluates this considerably faster than numpy and doesn't chew up all that extra memory. Actually that's why it's faster -- it's much more memory friendly. It's also a lot less flexible, although that's been improving. Still you want numpy as a base, but numexpr is a viable alternative for those critical sections where once you would resort to x= or worse, three argument ufuncs. -tim > > --bb From tim.hochberg at cox.net Fri Jul 7 03:24:46 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri, 07 Jul 2006 00:24:46 -0700 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44AD8A7C.5030408@cox.net> Message-ID: <44AE0C3E.5090508@cox.net> Bill Baxter wrote: > On 7/7/06, *Robert Kern* > wrote: > > Bill Baxter wrote: > > I am also curious, given the number of times I've heard this > nebulous > > argument of "there are lots kinds of numerical computing that don't > > invlolve linear algebra", that no one ever seems to name any of > these > > "lots of kinds". Statistics, maybe? But you can find lots of > linear > > algebra in statistics. > > That's because I'm not waving my hands at general fields of > application. I'm > talking about how people actually use array objects on a > line-by-line basis. If > I represent a dataset as an array and fit a nonlinear function to > that dataset, > am I using linear algebra at some level? Sure! Does having a .T > attribute on > that array help me at all? No. Arguing about how fundamental > linear algebra is > to numerical endeavors is entirely besides the point. > > > Ok. If line-by-line usage is what everyone really means, then I'll > get off the linear algebra soap box, but that's not what it sounded > like to me. > > So, if you want to talk line-by-line, I really can't talk about much > beside my own code. But I just grepped through it and out of 2445 > non-empty lines of code: > > 927 lines contain '=' > 390 lines contain a '[' > 75 lines contain matrix,asmatrix, or mat > ==> 47 lines contain a '.T' or '.transpose' of some sort. <== > 33 lines contain array, or asarray, or asanyarray > 24 lines contain 'rand(' --- I use it for generating bogus test data > a lot > 17 lines contain 'newaxis' or 'NewAxis' > 16 lines contain 'zeros(' > 13 lines contain 'dot(' > 12 lines contain 'empty(' > 8 lines contain 'ones(' > 7 lines contain 'inv(' In my main project theres about 26 KLOC (including blank lines), 700 or so of which use numeric (I prefix everything with np. so it's easy to count. Of those lines 29 use transpose, and of those 29 lines at most 9 could use a T attribute. It's probably far less than that since I didn't check the dimensionality of the arrays involved. Somewhere between 0 and 5 seems likely. > > I'm pretty new to numpy, so that's all the code I got right now. I'm > sure I've written many more lines of emails about numpy than I have > lines of actual numpy code. :-/ > > But from that, I can say that -- at least in my code -- transpose is > pretty common. If someone can point me to some larger codebases > written in numpy or numeric, I'd be happy to do a similar analysis of > those. > > > I'm not saying that people who do use arrays for linear algebra > are rare or > unimportant. It's that syntactical convenience for one set of > conventional ways > to use an array object, by itself, is not a good enough reason to > add stuff to > the core array object. > > > I wish I had a way to magically find out the distribution of array > dimensions used by all numpy and numeric code out there. My guess is > it would be something like 1-d: 50%, 2-d: 30%, 3-d: 10%, everything > else: 10%. I can't think of a good way to even get an estimate on > that. But in any event, I'm positive ndims==2 is a significant > percentage of all usages. It seems like the opponents to this idea > are suggesting the distribution is more flat than that. But whatever > the distribution is, it has to have a fairly light tail since memory > usage is exponential in ndim. If ndim == 20, then it takes 8 > megabytes just to store the smallest possible non-degenerate array of > float64s ( i.e. a 2x2x2x2x...) I would guess that it falls off fast after n=3, but that's just a guess. Personally, the majority of my code deals in 3D arrays (2x2xN and 4x4xN for the most part). These are arrays of vectors holding scattering data at N different frequency or time points. The 2D arrays that I do use are for rendering imaging (the actual rendering is done in C since Python wasn't fast enough and numpy wasn't really suitable for it). So, you see that for me at least, a T attribute is complete cruft. Useless for the 3D arrays, not needed for the 2D arrays, and again useless for the 1D arrays. I suspect that in general, the image processing types, who use a lot of 2D arrays, are probably not heavy users of transpose, but I'm not certain of that. > > It seems crazy to even be arguing this. Transposing is not some > specialized esoteric operation. It's important enough that R and S > give it a one letter function, and Matlab, Scilab, K all give it a > single-character operator. [*] Whoever designed the numpy.matrix > class also thought it was worthy of a shortcut, and I think came up > with a pretty good syntax for it. And the people who invented math > itself ?! > decided it was worth assigning a 1-character exponent to it. > > So I think there's a clear argument for having a .T attribute. But > ok, let's say you're right, and a lot of people won't use it. Fine. > IT WILL DO THEM ABSOLUTELY NO HARM. They don't have to use it if they > don't like it! Just ignore it. Unlike a t() function, .T doesn't > pollute any namespace users can define symbols in, so you really can > just ignore it if you're not interested in using it. It won't get in > your way. This is a completely bogus argument. All features cost -- good and ill alike. There's implementation cost and maintenance cost, both likely small in this case, but not zero. There's cognitive costs associated with trying to hold all of the various numpy methods, attributes and functions in ones head at once. There's pedagogical costs trying to explain how things fit together. There's community costs in that people who are allegedly coding with core numpy end up using mutually incomprehensible dialects. TANSTAFL. The ndarray object has far too many methods and attributes already IMO, and you have not made a case that I find convincing that this is important enough to further cruftify it. > > For the argument that ndarray should be pure like the driven snow, > just a raw container for n-dimensional data, Did anyone make that argument. No? I didn't think so. > I think that's what the basearray thing that goes into Python itself > should be. ndarray is part of numpy and numpy is for numerical > computing. And? Regards, -tim > > Regards, > --Bill > > [*] Full disclosure: I did find two counter-examples -- Maple and > Mathematica. Maple has only a transpose() function and Mathematica > has only Transpose[] (but you can use [esc]tr[esc] as a shortcut) > However, both of those packages are primarily known for their > _symbolic_ math capabilities, not their number crunching, so they less > are similar to numpy than R,S,K,Matlab and Scilab in that regard. > > ------------------------------------------------------------------------ > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From steffen.loeck at gmx.de Fri Jul 7 03:39:15 2006 From: steffen.loeck at gmx.de (Steffen Loeck) Date: Fri, 7 Jul 2006 09:39:15 +0200 Subject: [Numpy-discussion] Speed degression? In-Reply-To: <44AD8581.5030904@ee.byu.edu> References: <200607041519.24636.steffen.loeck@gmx.de> <44AD8581.5030904@ee.byu.edu> Message-ID: <200607070939.15600.steffen.loeck@gmx.de> On Thursday 06 July 2006, Travis Oliphant wrote: > This can test the speed of the implementations. With the latest modifications i get the following Results (in usec per loop): ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?sin-array ? ? ? ? ? ?mod-array ? use_scalarmath/use_pythonmath? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Numeric ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?134 ? ? ? ? ? ? ? ? ? 18 ? ? ? ? ? ?numpy 0.9.8 ? ? ? ? ? ? ? ? ? ? ? ? ? ? 97 ? ? ? ? ? ? ? ? ? 55 ?numpy 0.9.9.2764 ? ? ? ? ? ? ? ? ? ? ?114/78 ? ? ? ? ? ? ? ?55/21 ? ? ?numpy 0.9.8 + math ? ? ? ? ? ? ? ? ? ? ?38 ?numpy 0.9.9.2764 + math ? ? ? ? ? ? ? 38/23 ? ?Numeric + math ? ? ? ? ? ? ? ? ? ? ? ? ?23 The newest version seems to be faster then Numeric in all sin-array tests. Also the speed of mod-array has improved a lot. Many thanks, Steffen From jensj at fysik.dtu.dk Fri Jul 7 03:51:56 2006 From: jensj at fysik.dtu.dk (=?UTF-8?B?SmVucyBKw7hyZ2VuIE1vcnRlbnNlbg==?=) Date: Fri, 07 Jul 2006 09:51:56 +0200 Subject: [Numpy-discussion] Why is my array not contiguous? Message-ID: <44AE129C.4060909@servfys.fysik.dtu.dk> Hi! With numpy-0.9.9.2726, I do this: >>> x = arange(4) >>> y = x[newaxis, :] I would expect both arrays to be contiguous: >>> x.flags.contiguous, y.flags.contiguous (True, False) Shouldn't y be contiguous? Maybe it's because of the strange strides: >>> y.strides (0, 4) >>> y.strides = (16, 4) >>> y.flags.contiguous True Jens J?rgen From svetosch at gmx.net Fri Jul 7 04:40:36 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Fri, 07 Jul 2006 10:40:36 +0200 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE264.2030903@ieee.org> References: <44ADE264.2030903@ieee.org> Message-ID: <44AE1E04.6080409@gmx.net> Travis Oliphant schrieb: > > 1) .T Have some kind of .T attribute > +0 (the discussion in the .T thread convinced me it's better to keep the matrix playground as a separate subclass, and so it's not important for me what happens with pure arrays) > > 2) .H returns .T.conj() +0 > > > 3) .M returns matrix version of array +1 !! (Either that, or I will soon get on your nerves demanding matrix versions of ones, zeros, rand, eye etc.!) > > > 4) .A returns basearray (useful for sub-classes). +0 ? (Already exists for matrix, doesn't it?) -sven From schofield at ftw.at Fri Jul 7 05:06:00 2006 From: schofield at ftw.at (Ed Schofield) Date: Fri, 7 Jul 2006 11:06:00 +0200 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE264.2030903@ieee.org> References: <44ADE264.2030903@ieee.org> Message-ID: On 07/07/2006, at 6:26 AM, Travis Oliphant wrote: > 1) .T Have some kind of .T attribute -1 The meaning of .T is unclear for ndim != 2, as the plethora of sub- vote options shows. And we already have a matrix object designed explicitly for the case ndim == 2. Let's make matrix objects more useful, rather than give up on matrices and hard-wire 2d concepts into arrays. I'm willing to help with this, and I'll post some ideas to a different thread. > If >0 on this then: > > a) .T == .swapaxes(-2,-1) > > b) .T == .transpose() > > c) .T raises error for ndim > 2 > > d) .T returns (N,1) array for length (N,) array > > e) .T returns self for ndim < 2 (Just for the record, I think option (d) is positively evil.) > 2) .H returns .T.conj() -1 I'm against this for the same reasons I'm against .T, but more strongly, due to the extra memory allocation. > 3) .M returns matrix version of array -0 This would have the advantage of making .T and .H redundant for 2-d arrays, since they'd be accessible as matrix attributes anyway. But I don't see a strong enough case for this as an extra attribute, since asmatrix() and mat() are available anyway. I would like to deprecate mat() as an alias for matrix (by moving mat to the oldnumeric namespace); this would break the expectation that mat() perform a copy, and allow users to write >>> from numpy import asmatrix as mat instead if they need a short, simple syntax that avoids the copy overhead. It's also trivial for any serious user of matrices to write wrapper functions for ones, zeros, etc., such as def matones(*args): return asmatrix(ones(*args)) There might be an argument for including these in e.g. the numpy.matrix namespace. This would make a .M attribute even less compelling. > 4) .A returns basearray (useful for sub-classes). -0 This could be a good idea, but I'm not clear enough on its implications to vote either way. -- Ed From wbaxter at gmail.com Fri Jul 7 08:31:06 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 7 Jul 2006 21:31:06 +0900 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44AE0C3E.5090508@cox.net> References: <44AD8A7C.5030408@cox.net> <44AE0C3E.5090508@cox.net> Message-ID: I think the thread to this point can be pretty much summarized by: while True: Bill: "2D transpose is common so it should have a nice syntax" Tim, Robert, Sasha, and Ed: "No it's not." Very well. I think it may be a self fulfilling prophecy, though. I.e. if matrix operations are cumbersome to use, then -- surprise surprise -- the large user base for matrix-like operations never materializes. Potential converts just give numpy the pass, and go to Octave or Scilab, or stick with Matlab, R or S instead. Why all the fuss about the .T? Because any changes to functions (like making ones() return a matrix) can easily be worked around on the user side, as has been pointed out. But as far as I know -- do correct me if I'm wrong -- there's no good way for a user to add an attribute to an existing class. After switching from matrices back to arrays, .T was the only thing I really missed from numpy.matrix. I would be all for a matrix class that was on equal footing with array and as easy to use as matrices in Matlab. But my experience using numpy.matrixwas far from that, and, given the lack of enthusiasm for matrices around here, that seems unlikely to change. However, I'm anxious to see what Ed has up his sleeves in the other thread. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Fri Jul 7 09:03:21 2006 From: bsouthey at gmail.com (Bruce Southey) Date: Fri, 7 Jul 2006 08:03:21 -0500 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE264.2030903@ieee.org> References: <44ADE264.2030903@ieee.org> Message-ID: On 7/6/06, Travis Oliphant wrote: > > This is a call for a vote on each of the math attributes. Please post > your vote as > > +1 : support > +0 : don't care so go ahead > -0 : don't care so why do it > -1 : against > > Vote on the following issues separately: > > > > 1) .T Have some kind of .T attribute -1 because I prefer what is already present although I would prefer the option of inplace or a taking a copy. Also I don't find .T particularly intuitive (for example, R and SAS IML have a t() function - SAS IML also uses a backtick but that has not been very portable). > > If >0 on this then: > > a) .T == .swapaxes(-2,-1) > > b) .T == .transpose() > > c) .T raises error for ndim > 2 > > d) .T returns (N,1) array for length (N,) array > > e) .T returns self for ndim < 2 > > > 2) .H returns .T.conj() -/+0 I don't care either way. > > > 3) .M returns matrix version of array -1 as I think asmatrix etc is clearer. > > > 4) .A returns basearray (useful for sub-classes). -1 as I think something like asarray etc is clearer and require explicit calling. > > > > -Travis > > > > > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From pjssilva at ime.usp.br Fri Jul 7 09:13:59 2006 From: pjssilva at ime.usp.br (Paulo Jose da Silva e Silva) Date: Fri, 07 Jul 2006 10:13:59 -0300 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE264.2030903@ieee.org> References: <44ADE264.2030903@ieee.org> Message-ID: <1152278040.5478.11.camel@localhost.localdomain> Em Qui, 2006-07-06 ?s 22:26 -0600, Travis Oliphant escreveu: > This is a call for a vote on each of the math attributes. Please post > your vote as > > +1 : support > +0 : don't care so go ahead > -0 : don't care so why do it > -1 : against > > Vote on the following issues separately: > > > > 1) .T Have some kind of .T attribute +1 > > If >0 on this then: > > a) .T == .swapaxes(-2,-1) > +0 > b) .T == .transpose() > -0 > c) .T raises error for ndim > 2 > +0 > d) .T returns (N,1) array for length (N,) array > +0 > e) .T returns self for ndim < 2 -1 > 2) .H returns .T.conj() > +0 > 3) .M returns matrix version of array > -1 > 4) .A returns basearray (useful for sub-classes). > +0 Paulo -- Paulo Jos? da Silva e Silva Professor Assistente do Dep. de Ci?ncia da Computa??o (Assistant Professor of the Computer Science Dept.) Universidade de S?o Paulo - Brazil e-mail: pjssilva at ime.usp.br Web: http://www.ime.usp.br/~pjssilva Teoria ? o que n?o entendemos o (Theory is something we don't) suficiente para chamar de pr?tica. (understand well enough to call) (practice) From schofield at ftw.at Fri Jul 7 09:49:41 2006 From: schofield at ftw.at (Ed Schofield) Date: Fri, 07 Jul 2006 15:49:41 +0200 Subject: [Numpy-discussion] What's wrong with matrices? Message-ID: <44AE6675.108@ftw.at> Bill Baxter wrote: > I think the thread to this point can be pretty much summarized by: > > while True: > Bill: "2D transpose is common so it should have a nice syntax" > Tim, Robert, Sasha, and Ed: "No it's not." > > Very well. I think it may be a self fulfilling prophecy, though. > I.e. if matrix operations are cumbersome to use, then -- surprise > surprise -- the large user base for matrix-like operations never > materializes. Potential converts just give numpy the pass, and go to > Octave or Scilab, or stick with Matlab, R or S instead. > > Why all the fuss about the .T? Because any changes to functions (like > making ones() return a matrix) can easily be worked around on the user > side, as has been pointed out. But as far as I know -- do correct me > if I'm wrong -- there's no good way for a user to add an attribute to > an existing class. After switching from matrices back to arrays, .T > was the only thing I really missed from numpy.matrix. > > I would be all for a matrix class that was on equal footing with array > and as easy to use as matrices in Matlab. But my experience using > numpy.matrix was far from that, and, given the lack of enthusiasm for > matrices around here, that seems unlikely to change. However, I'm > anxious to see what Ed has up his sleeves in the other thread. Okay ... ... let's make this the thread ;) I'd like to know why you, Sven, and anyone else on the list have gone back to using arrays after trying matrices. What was inconvenient about them? I'd like a nice juicy list. The whole purpose of the matrix class is to simplify 2d linear algebra. Where is it failing? I also went back to arrays after trying out matrices for some 2d linear algebra tasks, since I found that using matrices increased my code's complexity. I can describe the problems I had with them later, but first I'd like to hear of others' experiences. I'd like to help to make matrices more usable. Tell me what you want, and I'll work on some patches. -- Ed From rhetland at mac.com Fri Jul 7 09:57:12 2006 From: rhetland at mac.com (Robert Hetland) Date: Fri, 7 Jul 2006 09:57:12 -0400 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> Message-ID: <9BC1D67B-C7BB-41F9-B136-5ACD4604DDD5@mac.com> On Jul 6, 2006, at 2:54 PM, Robert Kern wrote: > I don't think that just because arrays are often used for linear > algebra that > linear algebra assumptions should be built in to the core array type. True. This argues against the MAH attributes. However, I use transpose often when not dealing with linear algebra, in particular with reading in data, and putting various columns into variables. Also, occasional in plotting (which expects things in 'backward' order relative to x-y space), and communicating between fortran programs (which typically use 'forward' order (x, y, z)) and numpy (backward -- (z, x, y)). I am very much in favor of .T, but it should be a full .transpose(), not just swap the last two axes. I don't care so much for the others. +1 for .T == .transpose() -Rob -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhetland at mac.com Fri Jul 7 10:06:32 2006 From: rhetland at mac.com (Robert Hetland) Date: Fri, 7 Jul 2006 10:06:32 -0400 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE264.2030903@ieee.org> References: <44ADE264.2030903@ieee.org> Message-ID: <37D34CB6-6BFE-4A6D-A8DD-1003C4440A96@mac.com> +1 1b) .T == .transpose() -1 all others On Jul 7, 2006, at 12:26 AM, Travis Oliphant wrote: > > This is a call for a vote on each of the math attributes. Please post > your vote as > > +1 : support > +0 : don't care so go ahead > -0 : don't care so why do it > -1 : against > > Vote on the following issues separately: > > > > 1) .T Have some kind of .T attribute > > If >0 on this then: > > a) .T == .swapaxes(-2,-1) > > b) .T == .transpose() > > c) .T raises error for ndim > 2 > > d) .T returns (N,1) array for length (N,) array > > e) .T returns self for ndim < 2 > > > 2) .H returns .T.conj() > > > 3) .M returns matrix version of array > > > 4) .A returns basearray (useful for sub-classes). > > > > -Travis > > > > > > Using Tomcat but need to do more? Need to support web services, > security? > Get stuff done quickly with pre-integrated technology to make your > job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel? > cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From stefan at sun.ac.za Fri Jul 7 10:06:57 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 7 Jul 2006 16:06:57 +0200 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE264.2030903@ieee.org> References: <44ADE264.2030903@ieee.org> Message-ID: <20060707140657.GE30177@mentat.za.net> On Thu, Jul 06, 2006 at 10:26:12PM -0600, Travis Oliphant wrote: > 1) .T Have some kind of .T attribute -1, since the expected behaviour of .T is different depending on problem context. > a) .T == .swapaxes(-2,-1) The fact that this was proposed just demonstrates the fact. If you have a (10,5) array, .T gives the expected (5,10) array. But if you have a (10,10,3) RGB image as an ndarray, .T gives a (10,3,10) array... which is certainly not what you'd expect in an image processing context. St?fan From pebarrett at gmail.com Fri Jul 7 10:22:47 2006 From: pebarrett at gmail.com (Paul Barrett) Date: Fri, 7 Jul 2006 10:22:47 -0400 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE264.2030903@ieee.org> References: <44ADE264.2030903@ieee.org> Message-ID: <40e64fa20607070722v4c00b769sec4c33b1b7a5f278@mail.gmail.com> On 7/7/06, Travis Oliphant wrote: > > This is a call for a vote on each of the math attributes. Please post > your vote as > > +1 : support > +0 : don't care so go ahead > -0 : don't care so why do it > -1 : against > > Vote on the following issues separately: > > > > 1) .T Have some kind of .T attribute -1 > If >0 on this then: > > a) .T == .swapaxes(-2,-1) > > b) .T == .transpose() > > c) .T raises error for ndim > 2 > > d) .T returns (N,1) array for length (N,) array > > e) .T returns self for ndim < 2 > > > 2) .H returns .T.conj() -1 > 3) .M returns matrix version of array -1 > 4) .A returns basearray (useful for sub-classes). -1 From gnurser at googlemail.com Fri Jul 7 10:26:41 2006 From: gnurser at googlemail.com (George Nurser) Date: Fri, 7 Jul 2006 15:26:41 +0100 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <9BC1D67B-C7BB-41F9-B136-5ACD4604DDD5@mac.com> References: <44ACF15D.6030607@ieee.org> <9BC1D67B-C7BB-41F9-B136-5ACD4604DDD5@mac.com> Message-ID: <1d1e6ea70607070726u6e6d8e33v26765b5a6f6c9cd3@mail.gmail.com> On 07/07/06, Robert Hetland wrote: [snip] > However, I use transpose often when not dealing with linear algebra, in > particular with reading in data, and putting various columns into > variables. Also, occasional in plotting (which expects things in 'backward' > order relative to x-y space), and communicating between fortran programs > (which typically use 'forward' order (x, y, z)) and numpy (backward -- (z, > x, y)). > This is my usage as well. Also my primitive knowledge of numpy requires use of the transpose when iterating over indexes from where. Moreover I think the notation .T is perfectly reasonable. So I agree with: > I am very much in favor of .T, but it should be a full .transpose(), not > just swap the last two axes. I don't care so much for the others. +1 for .T == .transpose() George Nurser. From gnurser at googlemail.com Fri Jul 7 10:31:02 2006 From: gnurser at googlemail.com (George Nurser) Date: Fri, 7 Jul 2006 15:31:02 +0100 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE264.2030903@ieee.org> References: <44ADE264.2030903@ieee.org> Message-ID: <1d1e6ea70607070731n7f83b303n4ef67482e6dfbbaf@mail.gmail.com> On 07/07/06, Travis Oliphant wrote: > > This is a call for a vote on each of the math attributes. Please post > your vote as > > +1 : support > +0 : don't care so go ahead > -0 : don't care so why do it > -1 : against > > Vote on the following issues separately: > > > > 1) .T Have some kind of .T attribute +1 > > If >0 on this then: > > a) .T == .swapaxes(-2,-1) -1 > > b) .T == .transpose() > > c) .T raises error for ndim > 2 -1 > > d) .T returns (N,1) array for length (N,) array -1 > > e) .T returns self for ndim < 2 +1 > > > 2) .H returns .T.conj() -1 > > > 3) .M returns matrix version of array -1 > > > 4) .A returns basearray (useful for sub-classes). -0 George. From gruben at bigpond.net.au Fri Jul 7 10:45:26 2006 From: gruben at bigpond.net.au (Gary Ruben) Date: Sat, 08 Jul 2006 00:45:26 +1000 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44AD8A7C.5030408@cox.net> Message-ID: <44AE7386.9000007@bigpond.net.au> Sasha wrote: > On 7/6/06, Bill Baxter wrote: >> ... >> Yep, like Tim said. The usage is say a N sets of basis vectors. Each set >> of basis vectors is a matrix. > > This brings up a feature that I really miss from numpy: an ability to do > > array([f(x) for x in a]) > > without python overhead. I'd find this really useful too. I'm doing lots of this in my recent code: array([f(x,y) for x in a for y in b]).reshape(xxx) Gary R. From wbaxter at gmail.com Fri Jul 7 11:07:58 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 8 Jul 2006 00:07:58 +0900 Subject: [Numpy-discussion] What's wrong with matrices? In-Reply-To: <44AE6675.108@ftw.at> References: <44AE6675.108@ftw.at> Message-ID: On 7/7/06, Ed Schofield wrote: > > Bill Baxter wrote: > > I would be all for a matrix class that was on equal footing with array > > and as easy to use as matrices in Matlab. But my experience using > > numpy.matrix was far from that, and, given the lack of enthusiasm for > > matrices around here, that seems unlikely to change. However, I'm > > anxious to see what Ed has up his sleeves in the other thread. > > Okay ... ... let's make this the thread ;) > I'd like to know why you, Sven, and anyone else on the list have gone > back to using arrays after trying matrices. What was inconvenient about > them? I'd like a nice juicy list. The whole purpose of the matrix > class is to simplify 2d linear algebra. Where is it failing? Okay, here are a few that come to mind. 1) Functions that take a matrix but return an array. Maybe these are all fixed now. But they better be fixed not just in numpy but in scipy too. To me this implies there needs to be some standard idiom for how to write a generic array-protocol-using function so that you don't have to think about it. 2) At the time I was using matrix, scalar * matrix was broken. Fixed now, but that kind of thing just shouldn't happen. There should be a tests for basic operations like that if there aren't already. 3) mat() doesn't make sense as a shortcut for matrix construction. It only saves 3 letters over typing matrix(), and asmatrix is generally more useful. So mat() should be a synonym for asmatrix() 4) eye,empty,rand,ones,zeros,arange and anything else that builds an array from scratch or from a python list should have a matrix equivalent. 5) I've got squeezes like crazy all over my matrix-using code. Maybe this was a bug in 0.9.5 or so that's been fixed? I do seem to recall some problem with indexing or c_ or something that was causing matrices to grow extra levels of length 1 axes. Again, like the scalar*matrix bug, things like that shouldn't happen. 6) No good way to do elementwise operations? Sometimes you just want to do an elementwise mult or divide or exponentiation. I guess you're supposed to do Z = asmatrix(X.A * Y.A). Yah, right. 7) Finally, once all that is fixed, I find the slavish adherance to ndim=2 to be too restrictive. a) Sometimes it's useful to have a stack of matrices. Pretty often, in fact, for me. I guess I could use a python list of matrices or an object array of matrix or something, but I also think there are times when it's useful to treat different pairs of axes as the 'matrix' part. So I'd like matrices to be able to have ndim>2. b) On the other end, I think ndim<2 is useful sometimes too. Take a function like mean(), for example. With no arguments the return value is a 1x1 matrix (as opposed to a scalar). That just looks silly. And it doesn't work everywhere a scalar would, e.g. with a python built-in like round(). Or take indexing. It seems odd to me that where() reurns a tuple of shape==(1,N) objects instead of just (N,) . Maybe I can get over that though, as long as it works for indexing (which it seems it does). But I think the scalar return case is a real issue. Here's another: sum(). For an array you can do sum(sum(a)) and get a scalar if a is 2-d, but for matrix sum(sum(m)) is the same as sum(m). And along these lines, m[newaxis] just silently doesn't do anything. That doesn't seem right. That's all I can think of for now. Beyond that, looming in the back of my mind there's the question of what happens when we decide we want to have sparse matrices? Or masked sparse matrices, for that matter. These specialty array objects all seem to me like they should be some sort of mix-ins. The features they provide are (conceptually, anyway) independent. Concrete sibling classes doesn't really seem like the right inheritance relationship. > I'd like to help to make matrices more usable. Tell me what you want, > and I'll work on some patches. Thanks for giving it a shot. I'm currently trying to compile numpy from svn on Win32 for the first time myself. If it goes well I can probably help run tests or maybe even on some coding. But I still have yet to get past ATLAS. --Bill -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.taylor at stanford.edu Fri Jul 7 11:18:25 2006 From: jonathan.taylor at stanford.edu (Jonathan Taylor) Date: Fri, 07 Jul 2006 08:18:25 -0700 Subject: [Numpy-discussion] numpy book Message-ID: <44AE7B41.3020703@stanford.edu> I bought a copy of the numpy book sometime in January which I have since lost in a careless mistake. Is there a way to get another (updated) copy? I couldn't see how this was done on the website. Thanks, Jonathan -- ------------------------------------------------------------------------ Jonathan Taylor Tel: 650.723.9230 Dept. of Statistics Fax: 650.725.8977 Sequoia Hall, 137 www-stat.stanford.edu/~jtaylo 390 Serra Mall Stanford, CA 94305 From charlesr.harris at gmail.com Fri Jul 7 11:20:42 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 7 Jul 2006 09:20:42 -0600 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <1d1e6ea70607070731n7f83b303n4ef67482e6dfbbaf@mail.gmail.com> References: <44ADE264.2030903@ieee.org> <1d1e6ea70607070731n7f83b303n4ef67482e6dfbbaf@mail.gmail.com> Message-ID: Does anyone use transpose for anything besides two dimensional arrays? For arrays that aren't matrices at heart the meaning of transpose is pretty arbitrary, so the only virtue I see in having an attribute would be less typeing. So put it in the documentation. As an aside, being able to specify axis in dot might obviate many cases of *.T and *.H and make inner a special case. > > > 1) .T Have some kind of .T attribute +1 > > > If >0 on this then: > > > > a) .T == .swapaxes(-2,-1) -1 > b) .T == .transpose() +1 > c) .T raises error for ndim > 2 -1 > > > d) .T returns (N,1) array for length (N,) array -1 > e) .T returns self for ndim < 2 +1 > > > 2) .H returns .T.conj() +1 > > > 3) .M returns matrix version of array no opinion. > > > 4) .A returns basearray (useful for sub-classes). no opinion Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin.wiechert at gmx.de Fri Jul 7 11:33:54 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Fri, 7 Jul 2006 17:33:54 +0200 Subject: [Numpy-discussion] ix_ and boolean indices Message-ID: <200607071733.54278.martin.wiechert@gmx.de> Hi all, for me M [ix_(I, J)] does not work if I, J are boolean arrays. Is this intended or a bug/missing feature? And is there a way (other than I = where (I) [0] etc.) to make it work? Thanks, Martin From david.huard at gmail.com Fri Jul 7 11:39:26 2006 From: david.huard at gmail.com (David Huard) Date: Fri, 7 Jul 2006 11:39:26 -0400 Subject: [Numpy-discussion] Missing docstrings Message-ID: <91cf711d0607070839t8f57aedoa4aa002aeddaaa70@mail.gmail.com> Hi, For the beta release, it would be nice if every function had a docstring, even a small one. See Ticket #174 Missing docstrings: In [116]: ['issubsctype', 'unicode_', 'string', 'float96', 'pkgload', 'void', 'unicode0', 'void0', 'object0', 'memmap', 'nan_to_num', 'PackageLoader', 'object_', 'dtype', 'unsignedinteger', 'uintc', 'uint0', 'uint8', 'chararray', 'uint64', 'finfo', 'add_newdoc', 'array_repr', 'array_str', 'longlong', 'int16', 'mat', 'uint', 'correlate', 'int64', 'choose', 'complexfloating', 'recarray', 'mean', 'str_', 'ulonglong', 'matrix', 'uint32', 'byte', 'ctypes_load_library', 'signedinteger', 'ndim', 'number', 'bool8', 'msort', 'bool_', 'inexact', 'broadcast', 'short', 'ubyte', 'std', 'double', 'require', 'take', 'issubclass_', 'longfloat', 'deprecate', 'bincount', 'array2string', 'float64', 'ushort', 'float_', 'geterrobj', 'iterable', 'intp', 'flexible', 'sctype2char', 'longdouble', 'flatiter', 'generic', 'show_config', 'i0', 'uintp', 'character', 'uint16', 'float32', 'int32', 'integer', 'get_printoptions', 'seterrobj', 'add_docstring', 'intc', 'var', 'int_', 'histogram', 'issubdtype', 'int0', 'int8', 'record', 'obj2sctype', 'single', 'floating', 'test', 'string0'] In the process of adding docstrings, I noted some slight bugs: bincount returns an error if the input array mixes float and ints, while it accepts lists with mixed floats and ints. histogram does not deal gracefully with the range attribute. All values greater than range[1] are put in the last bin, which is probably not the expected behavior. Cheers, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Fri Jul 7 11:45:17 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 7 Jul 2006 08:45:17 -0700 Subject: [Numpy-discussion] What's wrong with matrices? In-Reply-To: <44AE6675.108@ftw.at> References: <44AE6675.108@ftw.at> Message-ID: On 7/7/06, Ed Schofield wrote: > I'd like to help to make matrices more usable. Tell me what you want, > and I'll work on some patches. I can't pass up an offer like that. DIAG diag(M) returns an array. It would be nice if diag(M) returned asmatrix(diag(M)).T. It would also be nice if you can construct a diagonal matrix directly from what is returned from diag(M). But right now you can't: >> x matrix([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) >> diag(x) array([0, 4, 8]) >> d = asmatrix(diag(x)).T >> d matrix([[0], [4], [8]]) >> diag(d) array([0]) <-- this should be a 3x3 matrix MATRIX CONSTRUCTION Making it easier to construct matrices would be a big help. Could the following function be made to return matrices? ones, zeros, rand, randn, eye, linspace, empty I guess the big decision is how to tell numpy to use matrices. How about from numpy.matrix import ones, zeros ? I would prefer something even more global. Something that acts like the global variable 'usematrix = True'. Once that is declared, the default changes from array to matrix. INDEXING >> x matrix([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) >> y matrix([[0], [1], [2]]) >> x[y>1,:] matrix([[6]]) This is a big one for me. If x[y>1,:] returned >> x[2,:] matrix([[6, 7, 8]]) then there would no longer be a need for array :) From charlesr.harris at gmail.com Fri Jul 7 11:48:15 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 7 Jul 2006 09:48:15 -0600 Subject: [Numpy-discussion] Enhancing dot() In-Reply-To: References: <44ADFE7C.5040506@cox.net> Message-ID: On 7/7/06, Bill Baxter wrote: > > On 7/7/06, Tim Hochberg wrote: > > > > The funny thing is that having a dot(a,b,c,...) would lead to the > > > exact same kind of hidden performance problems you're arguing against. > > Not exactly arguing -- this isn't why I don't like H and friends -- just > > > > noting that this is one of the traps that people are likely to fall into > > when transferring equations to code. > > A = D > A *= -2 > A += C > A += B > I would like to write something like: A = D.copy().times(-2).plus(C).plus(B) i.e. copy produces a "register", the rest is reverse Polish, and = "stores" the result. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pau.gargallo at gmail.com Fri Jul 7 11:51:04 2006 From: pau.gargallo at gmail.com (Pau Gargallo) Date: Fri, 7 Jul 2006 17:51:04 +0200 Subject: [Numpy-discussion] ix_ and boolean indices In-Reply-To: <200607071733.54278.martin.wiechert@gmx.de> References: <200607071733.54278.martin.wiechert@gmx.de> Message-ID: <6ef8f3380607070851j2bbe48f6meacb67e8e9195dd3@mail.gmail.com> On 7/7/06, Martin Wiechert wrote: > Hi all, > > for me > > M [ix_(I, J)] > > does not work if I, J are boolean arrays. Is this intended or a bug/missing > feature? > > And is there a way (other than I = where (I) [0] etc.) to make it work? > > Thanks, > Martin it is a recent feature. It works for me on version '0.9.9.2660' pau >>> import numpy >>> numpy.__version__ '0.9.9.2660' >>> a = numpy.rand(3,4) >>> a array([[ 0.24347161, 0.25636386, 0.64373189, 0.82730095], [ 0.02062571, 0.12244009, 0.60053928, 0.10624435], [ 0.75472591, 0.00614411, 0.75388955, 0.40481918]]) >>> I = a[:,0]<0.5 >>> J = a[0,:]<0.5 >>> a[numpy.ix_(I,J)] array([[ 0.24347161, 0.25636386], [ 0.02062571, 0.12244009]]) From Chris.Barker at noaa.gov Fri Jul 7 14:02:21 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 07 Jul 2006 11:02:21 -0700 Subject: [Numpy-discussion] Numpy import problem In-Reply-To: <20060707130305.2svvwr0wuntgkosk@webmail.auckland.ac.nz> References: <20060706170241.hq0fq9gssdys8kg4@webmail.auckland.ac.nz> <44AD3D73.8080602@noaa.gov> <20060707130305.2svvwr0wuntgkosk@webmail.auckland.ac.nz> Message-ID: <44AEA1AD.6060605@noaa.gov> Satellite Data Research Group wrote: > Quoting Christopher Barker : >> Which Python 2.4.1 are you using? It would be great if you would give >> the Python2.4.3 version found here a try: >> >> http://www.pythonmac.org/packages/py24-fat/index.html > Thanks for that, python 2.4.3 works perfectly. > Cheers, Joe Corbett There is now a Numpy0.9.8 package on that page too. Please let me know (or write to the macpython list) if you have a problem with it. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant.travis at ieee.org Fri Jul 7 13:42:35 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 07 Jul 2006 11:42:35 -0600 Subject: [Numpy-discussion] ix_ and boolean indices In-Reply-To: <200607071733.54278.martin.wiechert@gmx.de> References: <200607071733.54278.martin.wiechert@gmx.de> Message-ID: <44AE9D0B.3060909@ieee.org> Martin Wiechert wrote: > Hi all, > > for me > > M [ix_(I, J)] > > does not work if I, J are boolean arrays. Is this intended or a bug/missing > feature? > Which version? Using boolean arrays as separate indices was a recent feature. You have to get SVN to use it. -Travis From Chris.Barker at noaa.gov Fri Jul 7 14:11:22 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 07 Jul 2006 11:11:22 -0700 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44AD8A7C.5030408@cox.net> Message-ID: <44AEA3CA.7060207@noaa.gov> Robert Kern wrote: > Just > because linear algebra is "the base" for a lot of numerical computing does not > mean that everyone is using numpy arrays for linear algebra all the time. Much > less does it mean that all of those conventions you've devised should be shoved > into the core array type. I totally agree here. What bugged me most about MATLAB was that it was so darn Matrix/Linear Algebra centric. Yes, much of the code I wrote used linear algebra, but mostly it was a tiny (though critical) part of the actual code: Lots of code to set up a matrix equation, then solve it. The solve it was one line of code. For the rest, I prefer an array approach. A Matrix/Linear Algebra centric approach is good for some things, but I think it should be all or nothing. If you want it, then there should be a Matrix package, that includes the Matrix object, AND a matrix version of all the utility functions, like ones, zeros, etc. So all you would have to do is do: from numpy.matrix import * instead of from numpy import * and you'd get all the same stuff. Most of what would need to be added to the matrix package would be pretty easy, boiler plate code. Then we'd need a bunch more testing to root out all the operations that returned arrays where they should return matrices. If there is no one that wants to do all that work, then we have our answer. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant.travis at ieee.org Fri Jul 7 14:13:14 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 07 Jul 2006 12:13:14 -0600 Subject: [Numpy-discussion] What's wrong with matrices? In-Reply-To: References: <44AE6675.108@ftw.at> Message-ID: <44AEA43A.9020904@ieee.org> Bill Baxter wrote: > On 7/7/06, *Ed Schofield* > > wrote: > > Okay ... ... let's make this the thread ;) > I'd like to know why you, Sven, and anyone else on the list have gone > back to using arrays after trying matrices. What was inconvenient > about > them? I'd like a nice juicy list. The whole purpose of the matrix > class is to simplify 2d linear algebra. Where is it failing? > > > Okay, here are a few that come to mind. > 1) Functions that take a matrix but return an array. Maybe these are > all fixed now. But they better be fixed not just in numpy but in > scipy too. To me this implies there needs to be some standard idiom > for how to write a generic array-protocol-using function so that you > don't have to think about it. A lot of these are fixed. The mechanism for handling this is in-place: either using asanyarray in the function or (more generally) using a decorator that wraps the arguments with asarray and returns the output with __array_wrap__. But, we need people to help with fleshing it out. > > 2) At the time I was using matrix, scalar * matrix was broken. Fixed > now, but that kind of thing just shouldn't happen. There should be a > tests for basic operations like that if there aren't already. We need people to write and implement the tests. It's one way everybody can contribute. I do use matrices occasionally (not never as has been implied). But, I do more coding than linear algebra (particularly with images), therefore my need for matrix math is reduced. Nonetheless, I've been very willing to make well-defined changes that are needed. Most problems cannot be found out without people who use and test the code. > > 3) mat() doesn't make sense as a shortcut for matrix construction. It > only saves 3 letters over typing matrix(), and asmatrix is generally > more useful. So mat() should be a synonym for asmatrix() I'd be willing to make that change, but it will break some people's SciPy code. > > 4) eye,empty,rand,ones,zeros,arange and anything else that builds an > array from scratch or from a python list should have a matrix equivalent Would from numpy.defmatrix import ones, zeros, ... work? > > 5) I've got squeezes like crazy all over my matrix-using code. Maybe > this was a bug in 0.9.5 or so that's been fixed? I do seem to recall > some problem with indexing or c_ or something that was causing > matrices to grow extra levels of length 1 axes. Again, like the > scalar*matrix bug, things like that shouldn't happen. Sure, but it's going to happen in a beta release... That's why we need testers. As I recall, most bugs with matrices have been fixed fairly quickly as soon as they are reported. > > 6) No good way to do elementwise operations? Sometimes you just want > to do an elementwise mult or divide or exponentiation. I guess you're > supposed to do Z = asmatrix(X.A * Y.A). Yah, right. This is a problem with a dearth of infix operators. In fact, if we had a good way to write matrix multiplication as an infix operator, perhaps there wouldn't be any need for matrices. I'm really not sure how to fix the problem (the .M attribute of arrays was an attempt to make it easier): (X.A * Y.A).M But, there is always multiply(X,Y) > > 7) Finally, once all that is fixed, I find the slavish adherance to > ndim=2 to be too restrictive. > a) Sometimes it's useful to have a stack of matrices. Pretty > often, in fact, for me. I guess I could use a python list of matrices > or an object array of matrix or something, but I also think there are > times when it's useful to treat different pairs of axes as the > 'matrix' part. So I'd like matrices to be able to have ndim>2. I suppose this restriction could be lifted. > b) On the other end, I think ndim<2 is useful sometimes too. Take > a function like mean(), for example. With no arguments the return > value is a 1x1 matrix (as opposed to a scalar). Have you checked lately. It's a scalar now... This has been fixed. > Or take indexing. It seems odd to me that where() reurns a tuple of > shape==(1,N) objects instead of just (N,) . The way to fix some of these is to return arrays for indexing instead of allowing matrices. But, matrices that are less than 2-d just don't make sense. > Maybe I can get over that though, as long as it works for indexing > (which it seems it does). But I think the scalar return case is a > real issue. Here's another: sum(). For an array you can do > sum(sum(a)) and get a scalar if a is 2-d, but for matrix sum(sum(m)) > is the same as sum(m). And along these lines, m[newaxis] just > silently doesn't do anything. That doesn't seem right. These are just semantic questions. It's no surprise that sum(sum(m)) returns the same as sum(m) for a matrix because summing over the same axis won't change the result. You have to sum over both axes in a matrix. Thanks for the feedback. -Travis From mark at mitre.org Fri Jul 7 14:36:18 2006 From: mark at mitre.org (Mark Heslep) Date: Fri, 07 Jul 2006 14:36:18 -0400 Subject: [Numpy-discussion] Numpy subversion compatibility with Scipy release? Message-ID: <44AEA9A2.7060308@mitre.org> Is there any general sync point with development Numpy from subversion and the SciPy releases? Ive got Numpy 0.9.9.2761 and Scipy 0.4.9 installed with (I believe several) breakages, in particular: > In [8]: from scipy import special ... > /usr/lib/python2.4/site-packages/scipy/linalg/basic.py > 20 > conjugate,ravel,r_,mgrid,take,ones,dot,transpose,sqrt,add,real > 21 import numpy > ---> 22 from numpy import asarray_chkfinite, outerproduct, > concatenate, reshape, single > 23 from numpy import matrix as Matrix > 24 import calc_lwork > > ImportError: cannot import name outerproduct I suppose Scipy is not picking up the deprecation of outerproduct. No surprise that bleeding edge Numpy subversion doesn't play with a SciPy release; I was just wondering if there generally used/known way to make it happen. Do I need to fall back to a Numpy release? Or move forward on SciPy? Mark From robert.kern at gmail.com Fri Jul 7 14:46:04 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 07 Jul 2006 13:46:04 -0500 Subject: [Numpy-discussion] Numpy subversion compatibility with Scipy release? In-Reply-To: <44AEA9A2.7060308@mitre.org> References: <44AEA9A2.7060308@mitre.org> Message-ID: Mark Heslep wrote: > Is there any general sync point with development Numpy from subversion > and the SciPy releases? Ive got Numpy 0.9.9.2761 and Scipy 0.4.9 > installed with (I believe several) breakages, in particular: > >> In [8]: from scipy import special > ... >> /usr/lib/python2.4/site-packages/scipy/linalg/basic.py >> 20 >> conjugate,ravel,r_,mgrid,take,ones,dot,transpose,sqrt,add,real >> 21 import numpy >> ---> 22 from numpy import asarray_chkfinite, outerproduct, >> concatenate, reshape, single >> 23 from numpy import matrix as Matrix >> 24 import calc_lwork >> >> ImportError: cannot import name outerproduct > I suppose Scipy is not picking up the deprecation of outerproduct. No > surprise that bleeding edge Numpy subversion doesn't play with a SciPy > release; I was just wondering if there generally used/known way to make > it happen. Do I need to fall back to a Numpy release? Or move forward > on SciPy? The releases are synched such that the numpy version is "twice" (by an idiosyncratic form of arithmetic) that of the scipy version. numpy 0.9.8 <-> scipy 0.4.9 . The latest SVN revisions should match (it's a bug, otherwise). numpy 2761 is recent enough that an SVN checkout of scipy will probably be fine. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david.huard at gmail.com Fri Jul 7 14:46:53 2006 From: david.huard at gmail.com (David Huard) Date: Fri, 07 Jul 2006 14:46:53 -0400 Subject: [Numpy-discussion] Missing docstrings Message-ID: <1152298014.5062.21.camel@localhost> Hi, For the first release, it would be nice if every function had a docstring, even a small one. There are 279 callable items in the numpy namespace, and 94 of those lack a docstring, albeit most of those probably don't get much usage. To help the process, I filed Ticket #174 and attached a couple of docstrings, feel free to add your own. Missing docstrings: ['issubsctype', 'unicode_', 'string', 'float96', 'pkgload', 'void', 'unicode0', 'void0', 'object0', 'memmap', 'nan_to_num', 'PackageLoader', 'object_', 'dtype', 'unsignedinteger', 'uintc', 'uint0', 'uint8', 'chararray', 'uint64', 'finfo', 'add_newdoc', 'array_repr', 'array_str', 'longlong', 'int16', 'mat', 'uint', 'correlate', 'int64', 'choose', 'complexfloating', 'recarray', 'mean', 'str_', 'ulonglong', 'matrix', 'uint32', 'byte', 'ctypes_load_library', 'signedinteger', 'ndim', 'number', 'bool8', 'msort', 'bool_', 'inexact', 'broadcast', 'short', 'ubyte', 'std', 'double', 'require', 'take', 'issubclass_', 'longfloat', 'deprecate', 'bincount', 'array2string', 'float64', 'ushort', 'float_', 'geterrobj', 'iterable', 'intp', 'flexible', 'sctype2char', 'longdouble', 'flatiter', 'generic', 'show_config', 'i0', 'uintp', 'character', 'uint16', 'float32', 'int32', 'integer', 'get_printoptions', 'seterrobj', 'add_docstring', 'intc', 'var', 'int_', 'histogram', 'issubdtype', 'int0', 'int8', 'record', 'obj2sctype', 'single', 'floating', 'test', 'string0'] Cheers, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark at mitre.org Fri Jul 7 15:42:01 2006 From: mark at mitre.org (Mark Heslep) Date: Fri, 07 Jul 2006 15:42:01 -0400 Subject: [Numpy-discussion] OptimizationDemo [was: Numpy subversion compatibility with Scipy release? ] In-Reply-To: References: <44AEA9A2.7060308@mitre.org> Message-ID: <44AEB909.6040206@mitre.org> Robert Kern wrote: > > The latest SVN revisions should match (it's a bug, otherwise). numpy 2761 is > recent enough that an SVN checkout of scipy will probably be fine. > Yes SVN scipy/numpy fix the problem. Sorry, I missed Travis's post on scipy-user here: http://projects.scipy.org/pipermail/scipy-user/2006-June/008438.html to that effect. I should just get specific: I was simply trying to run the basic optimization demo on the front page of the wiki as check on SVN (http://www.scipy.org/Cookbook/OptimizationDemo1) and it appears SVN scipy/numpy breaks matplotlib -0.87.2-1 (fc5) in the demo. Do I need SVN matplotlib as well? Mark From the Opt. Demo: > In [1]: from scipy import arange, special, optimize > In [2]: x = arange(0,10,0.01) > In [4]: for k in arange(0.5,5.5): > ...: y = special.jv(k,x) > ...: plot(x,y) > ...: > --------------------------------------------------------------------------- > exceptions.ZeroDivisionError Traceback (most recent call ... ... > /usr/lib/python2.4/site-packages/matplotlib/ticker.py in > bin_boundaries(self, vmin, vmax) > 766 def bin_boundaries(self, vmin, vmax): > 767 nbins = self._nbins > --> 768 scale, offset = scale_range(vmin, vmax, nbins) > 769 vmin -= offset > 770 vmax -= offset > > /usr/lib/python2.4/site-packages/matplotlib/ticker.py in > scale_range(vmin, vmax, n, threshold) > 731 dv = abs(vmax - vmin) > 732 meanv = 0.5*(vmax+vmin) > --> 733 var = dv/max(abs(vmin), abs(vmax)) > 734 if var < 1e-12: > 735 return 1.0, 0.0 > > ZeroDivisionError: float division From robert.kern at gmail.com Fri Jul 7 15:56:36 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 07 Jul 2006 14:56:36 -0500 Subject: [Numpy-discussion] OptimizationDemo [was: Numpy subversion compatibility with Scipy release? ] In-Reply-To: <44AEB909.6040206@mitre.org> References: <44AEA9A2.7060308@mitre.org> <44AEB909.6040206@mitre.org> Message-ID: Mark Heslep wrote: > Robert Kern wrote: >> The latest SVN revisions should match (it's a bug, otherwise). numpy 2761 is >> recent enough that an SVN checkout of scipy will probably be fine. >> > Yes SVN scipy/numpy fix the problem. Sorry, I missed Travis's post on > scipy-user here: > http://projects.scipy.org/pipermail/scipy-user/2006-June/008438.html > to that effect. > > I should just get specific: I was simply trying to run the basic > optimization demo on the front page of the wiki as check on SVN > (http://www.scipy.org/Cookbook/OptimizationDemo1) and it appears SVN > scipy/numpy breaks matplotlib -0.87.2-1 (fc5) in the demo. Do I need SVN > matplotlib as well? Yes, you do, thanks to the removal of the Numeric typecodes (Int32, etc.). However, the error you get looks to be unrelated. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at ee.byu.edu Fri Jul 7 16:21:44 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 07 Jul 2006 14:21:44 -0600 Subject: [Numpy-discussion] .M .A .T .H attribute result Message-ID: <44AEC258.9040800@ee.byu.edu> I didn't compile the results, but the discussion on the idea of adding new attributes to the array object led to the following result. Added: .T attribute to mean self.transpose() .T This was rather controversial with many possibilities emerging. In the end, I think the common case of going back and forth between C-order and Fortran-order codes in a wide variety of settings convinced me to make .T a short-hand for .transpose() and add it as an attribute. This is now the behavior in SVN. Right now, for self.ndim < 2, this just returns a new reference to self (perhaps it should return a new view instead). .M While some were in favor, too many people opposed this (although the circular reference argument was not convincing). Instead a numpy.matlib module was started to store matrix versions of the standard array-creation functions and mat was re-labeled to "asmatrix" so that a copy is not made by default. .A A few were in favor, but as this is just syntactic sugar for .__array__() or asarray(obj) or .view(ndarray) it was thrown out because it is not used enough to add an additional attribute .H A few were in favor, but this can now be written .T.conj() which is not bad so does not get a new attribute. -Travis From cookedm at physics.mcmaster.ca Fri Jul 7 16:22:20 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri, 7 Jul 2006 16:22:20 -0400 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <1d1e6ea70607070726u6e6d8e33v26765b5a6f6c9cd3@mail.gmail.com> References: <44ACF15D.6030607@ieee.org> <9BC1D67B-C7BB-41F9-B136-5ACD4604DDD5@mac.com> <1d1e6ea70607070726u6e6d8e33v26765b5a6f6c9cd3@mail.gmail.com> Message-ID: <20060707162220.66d3d122@arbutus.physics.mcmaster.ca> On Fri, 7 Jul 2006 15:26:41 +0100 "George Nurser" wrote: > On 07/07/06, Robert Hetland wrote: > [snip] > > However, I use transpose often when not dealing with linear algebra, in > > particular with reading in data, and putting various columns into > > variables. Also, occasional in plotting (which expects things in > > 'backward' order relative to x-y space), and communicating between > > fortran programs (which typically use 'forward' order (x, y, z)) and > > numpy (backward -- (z, x, y)). > > > This is my usage as well. Also my primitive knowledge of numpy > requires use of the transpose when iterating over indexes from where. > Moreover I think the notation .T is perfectly reasonable. So I agree > with: same. > > > I am very much in favor of .T, but it should be a full .transpose(), not > > just swap the last two axes. I don't care so much for the others. > > +1 for .T == .transpose() Another +1 from me. If transpose was a shorter word I wouldn't care :-) -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From oliphant at ee.byu.edu Fri Jul 7 16:39:06 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 07 Jul 2006 14:39:06 -0600 Subject: [Numpy-discussion] [SciPy-dev] Weave, numpy, external libraries and conflicting typedefs In-Reply-To: References: <17582.7994.142818.120995@prpc.aero.iitb.ac.in> <44AE9E12.5000703@ieee.org> <17582.46882.286683.846598@prpc.aero.iitb.ac.in> <44AEBE1F.8010304@ee.byu.edu> Message-ID: <44AEC66A.2070507@ee.byu.edu> Fernando Perez wrote: >On 7/7/06, Travis Oliphant wrote: > > > >>Also, (in latest SVN) the MAXMIN macros can be avoided using >> >>#define PYA_NOMAXMIN >> >>before including arrayobject.h >> >> > >Mmh, this looks crufty to me: special cases like these look bad in a >library, and break the 'just works' ideal we all strive for, IMHO. > > But, it fixes the problem he's having without breaking anybody else's code that uses the MAX / MIN macros, already. Besides, the PY_ARRAY_TYPES_PREFIX business is a lot more crufty. I'm not opposed to putting a *short* prefix in front of everything (the Int32, Float64, stuff came from numarray which now has it's own back-ward compatible header where it could be placed now anyway). Perhaps npy_ would be a suitable prefix. That way we could get rid of the cruft entirely. I suppose we could also provide the noprefix.h header that defines the old un-prefixed cases for "backwards NumPy compatibility". -Travis From svetosch at gmx.net Fri Jul 7 16:44:42 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Fri, 07 Jul 2006 22:44:42 +0200 Subject: [Numpy-discussion] What's wrong with matrices? In-Reply-To: <44AE6675.108@ftw.at> References: <44AE6675.108@ftw.at> Message-ID: <44AEC7BA.6000205@gmx.net> Ed Schofield schrieb: > > Okay ... ... let's make this the thread ;) > I'd like to know why you, Sven, and anyone else on the list have gone > back to using arrays after trying matrices. What was inconvenient about > them? I'd like a nice juicy list. The whole purpose of the matrix > class is to simplify 2d linear algebra. Where is it failing? No, no, I must have given the wrong impression, I'm still in the matrix camp. My main complaint would have been absence of equivalents for ones, zeros, etc., but it seems Travis has introduced numpy.matlib exactly for that, which is great. Before that, I sometimes felt like a second-class citizen because many people on the list argued that users should invent their own workarounds (which btw I have done in the meantime, but that's not the purpose of a ready-to-use matrix package, is it?). > > I'd like to help to make matrices more usable. Tell me what you want, > and I'll work on some patches. > Well if numpy.matlib does what I think it does there are no pressing issues for me right now. Element-wise multiplication isn't that important for me. Of course I'll let you know as soon as something occurs to me ;-) Thanks for offering, Sven From fperez.net at gmail.com Fri Jul 7 16:53:45 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 7 Jul 2006 14:53:45 -0600 Subject: [Numpy-discussion] [SciPy-dev] Weave, numpy, external libraries and conflicting typedefs In-Reply-To: <44AEC66A.2070507@ee.byu.edu> References: <17582.7994.142818.120995@prpc.aero.iitb.ac.in> <44AE9E12.5000703@ieee.org> <17582.46882.286683.846598@prpc.aero.iitb.ac.in> <44AEBE1F.8010304@ee.byu.edu> <44AEC66A.2070507@ee.byu.edu> Message-ID: On 7/7/06, Travis Oliphant wrote: > I'm not opposed to putting a *short* prefix in front of everything (the > Int32, Float64, stuff came from numarray which now has it's own > back-ward compatible header where it could be placed now anyway). > Perhaps npy_ would be a suitable prefix. > > That way we could get rid of the cruft entirely. Well, now is your chance to clean up all the APIs, /including/ the C ones :) npy_ or NPy, I'm not too sure what conventions you are following at the C naming level. I'm all for cruft removal and making things easy to use out of the box, even at the cost of making the transition a bit more work. Remember, that's a one time cost. And numpy is so good that people /will/ transition from Numeric eventually, so might as well make the end result as nice and appealing as possible. As other tools (like matplotlib) move eventually to numpy-only support, the incentive to making the switch will really go up for just about anyone using python for numerical work. At the risk of sounding a bit harsh, I think you can then say, 'take the pain for the switch if you really want all the new goodies'. Those who positively, absolutely can't update from Numeric can then just keep a frozen codebase. It's not like you're breaking Numeric 24.2 or deleting it from the internet :) Cheers, f From tim.hochberg at cox.net Fri Jul 7 17:21:02 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri, 07 Jul 2006 14:21:02 -0700 Subject: [Numpy-discussion] extended dot In-Reply-To: <44AEA43A.9020904@ieee.org> References: <44AE6675.108@ftw.at> <44AEA43A.9020904@ieee.org> Message-ID: <44AED03E.5040808@cox.net> So, I put together of a prototype dot function dot( From tim.hochberg at cox.net Fri Jul 7 17:21:43 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri, 07 Jul 2006 14:21:43 -0700 Subject: [Numpy-discussion] extended dot In-Reply-To: <44AED03E.5040808@cox.net> References: <44AE6675.108@ftw.at> <44AEA43A.9020904@ieee.org> <44AED03E.5040808@cox.net> Message-ID: <44AED067.4060501@cox.net> Tim Hochberg wrote: > > > So, I put together of a prototype dot function > > dot( > Ooops! This wasn't supposed to go out yet, sorry. More later. -tim From mark at mitre.org Fri Jul 7 17:27:49 2006 From: mark at mitre.org (Mark Heslep) Date: Fri, 07 Jul 2006 17:27:49 -0400 Subject: [Numpy-discussion] OptimizationDemo [was: Numpy subversion compatibility with Scipy release? ] RESOLVED In-Reply-To: References: <44AEA9A2.7060308@mitre.org> <44AEB909.6040206@mitre.org> Message-ID: <44AED1D5.1050609@mitre.org> The problem was matplotlib-0.87.2-1; it is broken wrt to the Demo. The lastest matplotlib .3 works. In summary, to run the demo, the following versions were required: Released Packages, all built from tar balls: ------------------------------------------------------ matplotlib-0.87.3 , ie, the current package .2 in Fedora extras fails, meaning I needed to chase down the matplotlib dependencies myself (eg tk.h in tk-devel) Numpy 0.9.8 (latest Fedora package is 0.9.5 which is too old) Scipy 0.4.9 And, at least as of now, svn check outs from the numpy, scipy, and matplotlib subversion repos also work together for Demo1. Mark Robert Kern wrote: > Mark Heslep wrote: > >> Robert Kern wrote: >> >>> The latest SVN revisions should match (it's a bug, otherwise). numpy 2761 is >>> recent enough that an SVN checkout of scipy will probably be fine. >>> >>> >> Yes SVN scipy/numpy fix the problem. Sorry, I missed Travis's post on >> scipy-user here: >> http://projects.scipy.org/pipermail/scipy-user/2006-June/008438.html >> to that effect. >> >> I should just get specific: I was simply trying to run the basic >> optimization demo on the front page of the wiki as check on SVN >> (http://www.scipy.org/Cookbook/OptimizationDemo1) and it appears SVN >> scipy/numpy breaks matplotlib -0.87.2-1 (fc5) in the demo. Do I need SVN >> matplotlib as well? >> > > Yes, you do, thanks to the removal of the Numeric typecodes (Int32, etc.). > However, the error you get looks to be unrelated. > > From tim.hochberg at cox.net Fri Jul 7 18:01:39 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri, 07 Jul 2006 15:01:39 -0700 Subject: [Numpy-discussion] extended dot Message-ID: <44AED9C3.8040409@cox.net> So I put together a prototype for an extended dot function that takes multiple arguments. This allows multiple dots to be computed in a single call thusly: dot(dot(dot(a, b), c), d) => dotn(a, b, c, d) On Bill Baxter's suggestion, dotn attempts to do the dots in an order that minimizes operations. That appears to work fine, although I wasn't very careful and I wouldn't be at all surprised if some problems crop up with that part. The interesting thing is that, since dot can perform both matrix and dot products, it's not associative. That is, dot(a, dot(b, c) != dot(dot(a, b), c) in general. The simplest examples is three vectors. >>> dot(dot([1,2,3], [3,2,1]), [1,1,1]) array([10, 10, 10]) >>> dot([1,2,3], dot([3,2,1], [1,1,1])) array([ 6, 12, 18]) That's mind numbingly obvious in retrospect, but means that my simple minded dot product optimizer is all wrong because it can change the order of evaluation in such a way that the result changes. That means two things: 1. I need to pick an effective order of evaluation. So, dotn(a, b, c ...) will be evaluated as if the products were evaluated in left to right order. 2. The optimizer needs to respect that order when rearranging the order in which the products are performed. Dot products must remain dot products and matrix products must remain matrix products under any transformations that take place. Anyway, that's that for now. I'll report back when I get it fixed up. -tim From ndarray at mac.com Fri Jul 7 18:41:27 2006 From: ndarray at mac.com (Sasha) Date: Fri, 7 Jul 2006 18:41:27 -0400 Subject: [Numpy-discussion] Broken tMIN/tMAX macros Message-ID: Travis' recent change highlighted the definitions of tMIN/tMAX macros. Typed min/max were a subject for some heated discussion between Linux kernel developers many years ago that resulted in the following definitions in the curent kernel: """ /* * min()/max() macros that also do * strict type-checking.. See the * "unnecessary" pointer comparison. */ #define min(x,y) ({ \ typeof(x) _x = (x); \ typeof(y) _y = (y); \ (void) (&_x == &_y); \ _x < _y ? _x : _y; }) #define max(x,y) ({ \ typeof(x) _x = (x); \ typeof(y) _y = (y); \ (void) (&_x == &_y); \ _x > _y ? _x : _y; }) /* * ..and if you can't take the strict * types, you can specify one yourself. * * Or not use min/max at all, of course. */ #define min_t(type,x,y) \ ({ type __x = (x); type __y = (y); __x < __y ? __x: __y; }) #define max_t(type,x,y) \ ({ type __x = (x); type __y = (y); __x > __y ? __x: __y; }) """ The idea is to force people to use _t versions unless the types of x and y are exactly the same. The numpy's tMIN and tMAX are clearly addressing the same problem, but current definitions #define tMAX(a,b,typ) {typ _x_=(a); typ _y_=(b); _x_>_y_ ? _x_ : _y_} #define tMIN(a,b,typ) {typ _x_=(a); typ _y_=(b); _x_<_y_ ? _x_ : _y_} are unlikely to work with any compiler. Linux kernel uses gcc trick of wrapping a block in parenthesis to get an expression but I don't think this is acceptable in numpy code. Not surprizingly, these macros are not used anywhere. I propose to remove them. From kwgoodman at gmail.com Fri Jul 7 19:01:47 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 7 Jul 2006 16:01:47 -0700 Subject: [Numpy-discussion] What's wrong with matrices? In-Reply-To: <44AEA43A.9020904@ieee.org> References: <44AE6675.108@ftw.at> <44AEA43A.9020904@ieee.org> Message-ID: On 7/7/06, Travis Oliphant wrote: > Bill Baxter wrote: > > 4) eye,empty,rand,ones,zeros,arange and anything else that builds an > > array from scratch or from a python list should have a matrix equivalent > Would > > from numpy.defmatrix import ones, zeros, ... > > work? Can defmatrix be shortened to matrix? So from numpy.matrix import ones, zeros, ... From oliphant at ee.byu.edu Fri Jul 7 21:02:53 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 07 Jul 2006 19:02:53 -0600 Subject: [Numpy-discussion] NumPy C-API now has prefixes Message-ID: <44AF043D.8010805@ee.byu.edu> I just committed a big change to the NumPy SVN (r2773-r2777) which adds the prefix npy_ or NPY_ to all names not otherwise pre-fixed. There is also a noprefix.h header that allows you to use the names without the prefixes defined, as before Plus: 1) The special FLAG names with _FLAGS now have the _FLAGS removed 2) The PY_ARRAY_TYPES_PREFIX is ignored. 3) The tMIN/tMAX macros are removed 4) MAX_DIMS --> NPY_MAXDIMS 5) OWN_DATA --> NPY_OWNDATA There is the header oldnumeric.h that can be used for compatibility with the Numeric C-API (including the names CONTIGUOUS and OWN_DATA). Please, try out the new C-API and let's get the bugs wrinkled out. Hopefully this will give us a more solid foundation for the future... I've already committed changes to matplotlib SVN that allow it to work with old and new NumPy. -Travis From fperez.net at gmail.com Fri Jul 7 21:06:58 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 7 Jul 2006 19:06:58 -0600 Subject: [Numpy-discussion] NumPy C-API now has prefixes In-Reply-To: <44AF043D.8010805@ee.byu.edu> References: <44AF043D.8010805@ee.byu.edu> Message-ID: On 7/7/06, Travis Oliphant wrote: > I just committed a big change to the NumPy SVN (r2773-r2777) which adds > the prefix npy_ or NPY_ to all names not otherwise pre-fixed. > > There is also a noprefix.h header that allows you to use the names > without the prefixes defined, as before > > Plus: > > 1) The special FLAG names with _FLAGS now have the _FLAGS removed > 2) The PY_ARRAY_TYPES_PREFIX is ignored. > 3) The tMIN/tMAX macros are removed > 4) MAX_DIMS --> NPY_MAXDIMS > 5) OWN_DATA --> NPY_OWNDATA Make sure scipy builds after these, I think I just saw it not build with 'OWN_DATA' errors. Maybe I just caught you in-between commits... f From fperez.net at gmail.com Fri Jul 7 21:15:08 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 7 Jul 2006 19:15:08 -0600 Subject: [Numpy-discussion] I think something broke badly today in numpy/scipy... Message-ID: Hi all, I updated earlier today (about 4 hours ago) numpy/scipy SVN, and all of a sudden my codes broke left and right. Backing off to In [3]: numpy.__version__ Out[3]: '0.9.9.2737' In [4]: scipy.__version__ Out[4]: '0.5.0.2044' things are OK again. I am really sorry not to be able to provide a narrow test case, but I am utterly swamped right now, and need to finish a number of things. Given how bracketing this particular problem takes about 1/2 hour in recompilations alone for each revision, I really can't do it right now. There is no exception, no traceback, it's just that various numerical algorithms fail to converge. This code is complex and it uses numpy, scipy and weave, so the problem could be anywhere. Again, my apologies for the super-vague report. But perhaps knowing that the change is fairly recent, someone may have an eureka moment :) If nobody finds anything obvious, I might be able to spend some serious debugging time next week on this, but for now I'll just retreat to a working version and try to finish my things. Cheers, f From fullung at gmail.com Fri Jul 7 21:22:36 2006 From: fullung at gmail.com (Albert Strasheim) Date: Sat, 8 Jul 2006 03:22:36 +0200 Subject: [Numpy-discussion] NumPy C-API now has prefixes In-Reply-To: <44AF043D.8010805@ee.byu.edu> Message-ID: <00f001c6a22c$fec10080$01eaa8c0@dsp.sun.ac.za> Hello all > Please, > > try out the new C-API and let's get the bugs wrinkled out. > > Hopefully this will give us a more solid foundation for the future... > I've already committed changes to matplotlib SVN that allow it to work > with old and new NumPy. What implications, if any, do these changes have for C extension skeleton on the wiki? http://www.scipy.org/Cookbook/C_Extensions Regards, Albert From stefan at sun.ac.za Fri Jul 7 21:22:50 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Sat, 8 Jul 2006 03:22:50 +0200 Subject: [Numpy-discussion] NumPy C-API now has prefixes In-Reply-To: References: <44AF043D.8010805@ee.byu.edu> Message-ID: <20060708012249.GA21318@mentat.za.net> On Fri, Jul 07, 2006 at 07:06:58PM -0600, Fernando Perez wrote: > On 7/7/06, Travis Oliphant wrote: > > I just committed a big change to the NumPy SVN (r2773-r2777) which adds > > the prefix npy_ or NPY_ to all names not otherwise pre-fixed. > > > > There is also a noprefix.h header that allows you to use the names > > without the prefixes defined, as before > > > > Plus: > > > > 1) The special FLAG names with _FLAGS now have the _FLAGS removed > > 2) The PY_ARRAY_TYPES_PREFIX is ignored. > > 3) The tMIN/tMAX macros are removed > > 4) MAX_DIMS --> NPY_MAXDIMS > > 5) OWN_DATA --> NPY_OWNDATA > > Make sure scipy builds after these, I think I just saw it not build > with 'OWN_DATA' errors. Maybe I just caught you in-between commits... I see the following before the compilation breaks: Lib/cluster/src/vq_wrap.cpp: In function ?PyObject* ARGOUT_argout(char*, char*, int*, int)?: Lib/cluster/src/vq_wrap.cpp:734: error: ?OWN_DATA? was not declared in this scope Lib/cluster/src/vq_wrap.cpp: In function ?int char_to_size(char)?: Lib/cluster/src/vq_wrap.cpp:582: warning: control reaches end of non-void function Lib/cluster/src/vq_wrap.cpp: In function ?int char_to_numtype(char)?: Lib/cluster/src/vq_wrap.cpp:590: warning: control reaches end of non-void function Lib/cluster/src/vq_wrap.cpp: At global scope: Lib/cluster/src/vq_wrap.cpp:147: warning: ?void* SWIG_TypeQuery(const char*)? defined but not used Lib/cluster/src/vq_wrap.cpp:301: warning: ?void SWIG_addvarlink(PyObject*, char*, PyObject* (*)(), int (*)(PyObject*))? defined but not used Lib/cluster/src/vq_wrap.cpp:315: warning: ?int SWIG_ConvertPtr(PyObject*, void**, swig_type_info*, int)? defined but not used Lib/cluster/src/vq_wrap.cpp:516: warning: ?PyObject* l_output_helper(PyObject*, PyObject*)? defined but not used In file included from /usr/include/python2.4/Python.h:8, from Lib/cluster/src/vq_wrap.cpp:176: /usr/include/python2.4/pyconfig.h:835:1: warning: "_POSIX_C_SOURCE" redefined In file included from /usr/include/string.h:26, from Lib/cluster/src/vq_wrap.cpp:27: /usr/include/features.h:150:1: warning: this is the location of the previous definition In file included from Lib/cluster/src/vq_wrap.cpp:499: Lib/cluster/src/vq.h:57:7: warning: no newline at end of file Lib/cluster/src/vq_wrap.cpp: In function ?PyObject* ARGOUT_argout(char*, char*, int*, int)?: Lib/cluster/src/vq_wrap.cpp:734: error: ?OWN_DATA? was not declared in this scope Lib/cluster/src/vq_wrap.cpp: In function ?int char_to_size(char)?: Lib/cluster/src/vq_wrap.cpp:582: warning: control reaches end of non-void function Lib/cluster/src/vq_wrap.cpp: In function ?int char_to_numtype(char)?: Lib/cluster/src/vq_wrap.cpp:590: warning: control reaches end of non-void function Lib/cluster/src/vq_wrap.cpp: At global scope: Lib/cluster/src/vq_wrap.cpp:147: warning: ?void* SWIG_TypeQuery(const char*)? defined but not used Lib/cluster/src/vq_wrap.cpp:301: warning: ?void SWIG_addvarlink(PyObject*, char*, PyObject* (*)(), int (*)(PyObject*))? defined but not used Lib/cluster/src/vq_wrap.cpp:315: warning: ?int SWIG_ConvertPtr(PyObject*, void**, swig_type_info*, int)? defined but not used Lib/cluster/src/vq_wrap.cpp:516: warning: ?PyObject* l_output_helper(PyObject*, PyObject*)? defined but not used error: Command "g++ -pthread -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -fPIC -I/home/stefan/lib/python2.4/site-packages/numpy/core/include -I/usr/include/python2.4 -c Lib/cluster/src/vq_wrap.cpp -o build/temp.linux-i686-2.4/Lib/cluster/src/vq_wrap.o" failed with exit status 1 Error building package. St?fan From haase at msg.ucsf.edu Fri Jul 7 21:37:32 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 07 Jul 2006 18:37:32 -0700 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE2E3.60102@ieee.org> References: <44ADE264.2030903@ieee.org> <44ADE2E3.60102@ieee.org> Message-ID: <44AF0C5C.9040104@msg.ucsf.edu> Travis Oliphant wrote: > Travis Oliphant wrote: >> This is a call for a vote on each of the math attributes. Please post >> your vote as >> >> +1 : support >> +0 : don't care so go ahead >> -0 : don't care so why do it >> -1 : against >> >> Vote on the following issues separately: >> >> >> >> 1) .T Have some kind of .T attribute >> >> +1 >> If >0 on this then: >> >> a) .T == .swapaxes(-2,-1) >> +1 >> b) .T == .transpose() >> -0 >> c) .T raises error for ndim > 2 >> -1 >> d) .T returns (N,1) array for length (N,) array >> >> +1 >> e) .T returns self for ndim < 2 >> >> -1 >> 2) .H returns .T.conj() >> >> >> +0 > >> 3) .M returns matrix version of array >> >> +0 >> 4) .A returns basearray (useful for sub-classes). >> +0 Sebastian Haase From haase at msg.ucsf.edu Fri Jul 7 21:37:26 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 07 Jul 2006 18:37:26 -0700 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44AD8A7C.5030408@cox.net> Message-ID: <44AF0C56.7020906@msg.ucsf.edu> Sasha wrote: > On 7/6/06, Bill Baxter wrote: >> ... >> Yep, like Tim said. The usage is say a N sets of basis vectors. Each set >> of basis vectors is a matrix. > > This brings up a feature that I really miss from numpy: an ability to do > > array([f(x) for x in a]) > > without python overhead. APL-like languages have a notion of "adverb" > - a higher level operator that maps a function to a function. Numpy > has some adverbs implemented as .... Hi, I was just reading through this thread and noticed that the above might be possibly done best with(a little extended version of) the numexpr module. Am I right !? Just wanted to post this comment about a package I'm really looking forward to using once I convert from numarray. Thanks for numpy !! Sebastian Haase UCSF From oliphant at ee.byu.edu Fri Jul 7 21:58:51 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 07 Jul 2006 19:58:51 -0600 Subject: [Numpy-discussion] Easy way to get NumPy code to compile Message-ID: <44AF115B.1040105@ee.byu.edu> If you have been a user of NumPy. The easy way to update your code so that it compiles even with the latest changes to the naming-scheme is to replace #include "numpy/arrayobject.h" with #include "numpy/noprefix.h" This will generate the prefix-less names (and the other _FLAGS and OWN_DATA names that your code may depend on). It also includes arrayobject.h and so there is no need to do that twice. This should help people who already have code that works with NumPy. -Travis From oliphant at ee.byu.edu Fri Jul 7 22:24:50 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 07 Jul 2006 20:24:50 -0600 Subject: [Numpy-discussion] SciPy should now compile and install Message-ID: <44AF1772.5010804@ee.byu.edu> All changes needed for scipy to compile and install with the new NumPy are now done. As a side benefit, the numarray C-API compatibility module also received a test as it is now used to compile scipy/ndimage So, SVN versions of NumPy / SciPy / and Matplotlib should all work together now. -Travis From jk985 at tom.com Mon Jul 10 23:22:55 2006 From: jk985 at tom.com (=?GB2312?B?N9TCMTUtMTbI1S/Jz7qj?=) Date: Tue, 11 Jul 2006 11:22:55 +0800 Subject: [Numpy-discussion] =?GB2312?B?1MvTw0VYQ0VMuMS9+LLGzvG53MDtus3K0LOh06rP+jxhZD4=?= Message-ID: An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Sat Jul 8 03:17:14 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 08 Jul 2006 01:17:14 -0600 Subject: [Numpy-discussion] NumPy beta release plan Message-ID: <44AF5BFA.70509@ieee.org> Some of you may have noticed that things have been changing rapidly in the NumPy world (if you were out of the office in June then all the activity may seem overwhelming). All of this activity is based on the fact that the upcoming beta release will mean a feature freeze for NumPy. As a review: The biggest changes over the past 3 months have been 1) Capitalized-type-names (Float32, Complex64, Int8, etc) are now only available in the numpy.oldnumeric namespace (this is the namespace that convertcode.py replaces for Numeric). We are trying to wean you off character codes as much as possible. They are still there, of course but should only be used in special cases not as a general rule. 2) Un-specified data-types now default to floating point. To help with code you have that relies on integer data-types you can either use functions from numpy.oldnumeric where the functions are still defined with the integer-default data-types or use functions in numpy/lib/convdtype to replace type-less versions of ones, zeros, empty with empty(..., dtype=int). 3) C-API names have prefix PyArray_ (like always), NPY_ or npy_. The NPY_ and npy_ prefixes are new and were done to remove the likelihood of name collisions when NumPy is used with another library. The old (and un-prefixed) names are accessible by importing numpy/noprefix.h instead of numpy/arrayobject.h It is fine to use noprefix.h in-place of arrayobject.h if you expect to have no naming conflicts. This is what NumPy itself does. 4) The flag combinations with _FLAGS in the name have the _FLAGS removed (but are prefixed with NPY_). Again the old names are still available in numpy/noprefix.h 5) The Numarray C-API is now available as numpy/libnumarray.h as long as you use the directory returned from numpy.get_numarray_include() as an argument to -I in the compile command. More minor changes: 1) ctypes attribute added for ease of working with ctypes data 2) T attribute added as a convenience for .transpose() Personally, I don't want to make any more changes so that we can make a 1.0 beta release that will freeze the API. Because of my guarantee to keep SVN versions of NumPy/SciPy/matplotlib working, changes probably cause more headache for me than anyone else. Unless there is a serious issue pointed out, the only thing that should be changed at this point are bug-fixes, documentation strings, and added tests. Once 1.0b1 is released nothing but those things can be added. I'd like July to be a much more calm month. We should get the 1.0b1 out in the next couple of weeks. That way perhaps 1.0b2 can be out by the SciPy conference. I can see the beta release period taking several months with only bug-fixes/docstring/testing improvements happening over that time. So, take this message as a warning for the up-coming feature freeze on NumPy and an invitation to contribute docstrings and unit-tests. I hope the rapid pace of June development did not scare too many people. Please voice your concerns if you have them. Best regards, -Travis O. From st at sigmasquared.net Sat Jul 8 04:04:20 2006 From: st at sigmasquared.net (Stephan Tolksdorf) Date: Sat, 08 Jul 2006 10:04:20 +0200 Subject: [Numpy-discussion] NumPy beta release plan In-Reply-To: <44AF5BFA.70509@ieee.org> References: <44AF5BFA.70509@ieee.org> Message-ID: <44AF6704.4080407@sigmasquared.net> Hi Travis, thanks for putting all the effort into NumPy. > Unless there is a serious issue pointed out, the only thing that should > be changed at this point are bug-fixes, documentation strings, and added > tests. Once 1.0b1 is released nothing but those things can be added. The beginning of the beta phase might also be the best time to address the most severe NumPy.distutils issues, so that all changes can be extensively tested during the beta: - warning messages are only shown if compilation fails (NumPy should build warning free) - there are some problems on Windows (ticket #114) - 64bit/Python 2.5 issues? Ciao, Stephan From oliphant.travis at ieee.org Sat Jul 8 05:33:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 08 Jul 2006 03:33:01 -0600 Subject: [Numpy-discussion] I think something broke badly today in numpy/scipy... In-Reply-To: References: Message-ID: <44AF7BCD.4080302@ieee.org> Fernando Perez wrote: > Hi all, > > I updated earlier today (about 4 hours ago) numpy/scipy SVN, and all > of a sudden my codes broke left and right. Backing off to > > Hey Fernando. I think I found the problem. It was the same problem causing the BFGS test to fail in SciPy. It can be shown by looking at sk = [0,-1.0,1.0] print 250.0*sk[:,newaxis] This should give something useful (not zeros). The problem was a seemingly harmless change in _IsContiguous that allowed for 0-strided arrays to be called contiguous. This caused havoc with the multiplication function later which relied on an accurate _IsContiguous function. This change was in r2765 (I suspect you checked out a change with that in it). The change could cause problems where-ever you use newaxis with scalar multiplication. The problem should be fixed in SVN. Very sorry... -Travis From wbaxter at gmail.com Sat Jul 8 06:10:04 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 8 Jul 2006 19:10:04 +0900 Subject: [Numpy-discussion] building ATLAS on Win32 - endless compile loop Message-ID: Trying to compile Atlas from source on Win32 with Cygwin. Anyone seen 'make install' for ATLAS go into and endless loop before? More info: Following the instructions here: Installing SciPy/Windows - SciPy.org At this step: "5) As prompted by the config script, execute make install arch=YOUR_ARCHITECTURE and wait for approx > 15min." It was taking much longer than 15 minutes, but I figured my laptop is just slow or something. So I left it on when I went to bed. It was still cranking away 6 hours later when I got up. Just thought I'd check if anyone had seen that before. My laptop is a Pentium M, which isn't one of the options on the architecture list. So I picked "unknown". --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From schofield at ftw.at Sat Jul 8 07:07:11 2006 From: schofield at ftw.at (Ed Schofield) Date: Sat, 8 Jul 2006 13:07:11 +0200 Subject: [Numpy-discussion] Args for rand and randn: call for a vote Message-ID: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> Last week's discussion on rand() and randn() seemed to indicate a sentiment that they ought to take tuples for consistency with ones, zeros, eye, identity, and empty -- that, although they are supposed to be convenience functions, they are inconvenient precisely because of their inconsistency with these other functions. This issue has been raised many times over the past several months. Travis made a change in r2572 to allow tuples as arguments, then took it out again a few hours later, apparently unsure about whether this was a good idea. I'd like to call for a vote on what people would prefer, and then ask Travis to make a final pronouncement before the feature freeze. * Should numpy.rand and numpy.randn accept sequences of dimensions as arguments, like rand((3,3)), as an alternative to rand(3,3)? OR * Should rand((3,3)) and randn((3,3)) continue to raise a TypeError? From schofield at ftw.at Sat Jul 8 07:16:02 2006 From: schofield at ftw.at (Ed Schofield) Date: Sat, 8 Jul 2006 13:16:02 +0200 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> Message-ID: <2014D1A2-6D90-45B6-879D-A82D9EDE8F76@ftw.at> On 08/07/2006, at 1:07 PM, Ed Schofield wrote: > Last week's discussion on rand() and randn() seemed to indicate a > sentiment that they ought to take tuples for consistency with ones, > zeros, eye, identity, and empty -- that, although they are supposed > to be convenience functions, they are inconvenient precisely because > of their inconsistency with these other functions. This issue has > been raised many times over the past several months. > > Travis made a change in r2572 to allow tuples as arguments, then took > it out again a few hours later, apparently unsure about whether this > was a good idea. > > I'd like to call for a vote on what people would prefer, and then ask > Travis to make a final pronouncement before the feature freeze. > > > * Should numpy.rand and numpy.randn accept sequences of dimensions as > arguments, like rand((3,3)), as an alternative to rand(3,3)? +1 From wbaxter at gmail.com Sat Jul 8 08:21:06 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 8 Jul 2006 21:21:06 +0900 Subject: [Numpy-discussion] What's wrong with matrices? In-Reply-To: References: <44AE6675.108@ftw.at> <44AEA43A.9020904@ieee.org> Message-ID: My preferred way to import numpy currently is: import numpy as num It would be nice if I could do: import numpy.matrix as num And basically have access to all the same stuff that's in the base numpy but have everything set up in a matrix-like way, so num.ones returns a matrix instead of an array, etc. (I don't care so much about the particular name... matlib, matrixlib, matrixdefault, or whatever -- I just care about the functionality) The way I understand this matlib module is that I would do something along the lines of one of these: 1) import numpy as num from numpy.matlib import ones, zeros,... or 2) import numpy as num import numpy.matlib as M Either way, the matrix functions aren't in the same namespace as general numpy functions, which makes it feel like something of a bolt-on rather than a second equally valid way to use numpy. So what I'd like is 3) import numpy.matlib as num I think the way to get 3) may be to just do a "from numpy import *" at the beginning of the numpy.matlib module, before redefining particular functions with matrix versions. Maybe that's a bad idea, though? Or maybe this is the way Travis and others planned for it to work from the beginning? At any rate, versions 1) and 2) should also be supported for the times you don't want to use matlib as the default, but still want access to it. --bill -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter at sonderport.dk Sat Jul 8 10:58:00 2006 From: peter at sonderport.dk (Peter =?ISO-8859-1?Q?S=F8ndergaard?=) Date: Sat, 08 Jul 2006 16:58:00 +0200 Subject: [Numpy-discussion] building ATLAS on Win32 - endless compile loop In-Reply-To: References: Message-ID: <1152370680.2347.1.camel@katholt.sonderport.dk> > My laptop is a Pentium M, which isn't one of the options on the > architecture list. So I picked "unknown". > In that case ATLAS propably does a full search instead of the much smaller architectural default one. This takes much longer, but not infinitely. Cheers, Peter From svetosch at gmx.net Sat Jul 8 09:23:15 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Sat, 08 Jul 2006 15:23:15 +0200 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> Message-ID: <44AFB1C3.6030804@gmx.net> Ed Schofield schrieb: > > I'd like to call for a vote on what people would prefer, and then ask > Travis to make a final pronouncement before the feature freeze. > > > > > * Should numpy.rand and numpy.randn accept sequences of dimensions as > arguments, like rand((3,3)), as an alternative to rand(3,3)? > +1 (Thanks Ed for trying to reach a conclusion on this.) -sven From pebarrett at gmail.com Sat Jul 8 09:31:20 2006 From: pebarrett at gmail.com (Paul Barrett) Date: Sat, 8 Jul 2006 09:31:20 -0400 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> Message-ID: <40e64fa20607080631w2a9d72ebh2a186b4cac70f383@mail.gmail.com> On 7/8/06, Ed Schofield wrote: > > Last week's discussion on rand() and randn() seemed to indicate a > sentiment that they ought to take tuples for consistency with ones, > zeros, eye, identity, and empty -- that, although they are supposed > to be convenience functions, they are inconvenient precisely because > of their inconsistency with these other functions. This issue has > been raised many times over the past several months. > > Travis made a change in r2572 to allow tuples as arguments, then took > it out again a few hours later, apparently unsure about whether this > was a good idea. > > I'd like to call for a vote on what people would prefer, and then ask > Travis to make a final pronouncement before the feature freeze. > > > > > * Should numpy.rand and numpy.randn accept sequences of dimensions as > arguments, like rand((3,3)), as an alternative to rand(3,3)? +1 --- I'm all for consistency! > OR > > > * Should rand((3,3)) and randn((3,3)) continue to raise a TypeError? From wbaxter at gmail.com Sat Jul 8 09:33:16 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 8 Jul 2006 22:33:16 +0900 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <44AFB1C3.6030804@gmx.net> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> <44AFB1C3.6030804@gmx.net> Message-ID: > > * Should numpy.rand and numpy.randn accept sequences of dimensions as > > arguments, like rand((3,3)), as an alternative to rand(3,3)? > > +1 -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Sat Jul 8 10:24:58 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 8 Jul 2006 23:24:58 +0900 Subject: [Numpy-discussion] What's wrong with matrices? In-Reply-To: <44AEA43A.9020904@ieee.org> References: <44AE6675.108@ftw.at> <44AEA43A.9020904@ieee.org> Message-ID: On 7/8/06, Travis Oliphant wrote: > > > Okay, here are a few that come to mind. > > 1) Functions that take a matrix but return an array. Maybe these are > > all fixed now. But they better be fixed not just in numpy but in > > scipy too. To me this implies there needs to be some standard idiom > > for how to write a generic array-protocol-using function so that you > > don't have to think about it. > > A lot of these are fixed. The mechanism for handling this is in-place: > either using asanyarray in the function or (more generally) using a > decorator that wraps the arguments with asarray and returns the output > with __array_wrap__. That sounds good. Is there a description of what that does and how to use it anywhere you could point me to? > 5) I've got squeezes like crazy all over my matrix-using code. Maybe > > this was a bug in 0.9.5 or so that's been fixed? I do seem to recall > > some problem with indexing or c_ or something that was causing > > matrices to grow extra levels of length 1 axes. Again, like the > > scalar*matrix bug, things like that shouldn't happen. > Sure, but it's going to happen in a beta release... That's why we need > testers. As I recall, most bugs with matrices have been fixed fairly > quickly as soon as they are reported. What do you mean by "beta release"? Are the odds considered betas and evens releases? Or do you mean just everything prior to 1.0 is actually beta? I haven't seen anything actually labeled as "beta" yet. And yes, you've done a fantastic job fixing bugs quickly, and getting releases out in a pretty timely manner too. Many thanks for that. > > > 6) No good way to do elementwise operations? Sometimes you just want > > to do an elementwise mult or divide or exponentiation. I guess you're > > supposed to do Z = asmatrix(X.A * Y.A). Yah, right. > This is a problem with a dearth of infix operators. In fact, if we had > a good way to write matrix multiplication as an infix operator, perhaps > there wouldn't be any need for matrices. Actually I was more after what you mentioned later -- the multiply(a,b) function. I see they're all there -- multiply, divide, power. That's all I wanted, because I know the Python operator overload situation doesn't really allow more than that. > ...So I'd like matrices to be able to have ndim>2. > I suppose this restriction could be lifted. I think that would be an improvement for people who want to use matrix as their only ndarray data type. > b) On the other end, I think ndim<2 is useful sometimes too. Take > > a function like mean(), for example. With no arguments the return > > value is a 1x1 matrix (as opposed to a scalar). > Have you checked lately. It's a scalar now... This has been fixed. Nope, haven't checked more recently than the latest release. I'm in the process of trying to build numpy, though. > Or take indexing. It seems odd to me that where() reurns a tuple of > > shape==(1,N) objects instead of just (N,) . The way to fix some of these is to return arrays for indexing instead of > allowing matrices. But, matrices that are less than 2-d just don't > make sense. I guess not. Looking at what matlab does more closely I see that it does report size (1,1) for all scalars, and either 1,N or N,1 for all vectors. It just hides it better. Namely when printing you don't get as many superfluous brackets (in fact matlab doesn't print any brackets). > Maybe I can get over that though, as long as it works for indexing > > (which it seems it does). But I think the scalar return case is a > > real issue. Here's another: sum(). For an array you can do > > sum(sum(a)) and get a scalar if a is 2-d, but for matrix sum(sum(m)) > > is the same as sum(m). And along these lines, m[newaxis] just > > silently doesn't do anything. That doesn't seem right. > > These are just semantic questions. It's no surprise that sum(sum(m)) > returns the same as sum(m) for a matrix because summing over the same > axis won't change the result. You have to sum over both axes in a > matrix. Right, it is perfectly logical given how sum is implemented. I guess I'm just used to Matlab's way which acts more like sum on arrays. But maybe numpy's way is ok too. --bill -------------- next part -------------- An HTML attachment was scrubbed... URL: From dd55 at cornell.edu Sat Jul 8 10:30:59 2006 From: dd55 at cornell.edu (Darren Dale) Date: Sat, 8 Jul 2006 10:30:59 -0400 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> Message-ID: <200607081030.59826.dd55@cornell.edu> On Saturday 08 July 2006 7:07 am, Ed Schofield wrote: > * Should numpy.rand and numpy.randn accept sequences of dimensions as > arguments, like rand((3,3)), as an alternative to rand(3,3)? +1 -- Darren S. Dale, Ph.D. dd55 at cornell.edu From sransom at nrao.edu Sat Jul 8 10:40:19 2006 From: sransom at nrao.edu (Scott Ransom) Date: Sat, 8 Jul 2006 10:40:19 -0400 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> Message-ID: <20060708144019.GB12588@ssh.cv.nrao.edu> > * Should numpy.rand and numpy.randn accept sequences of dimensions as > arguments, like rand((3,3)), as an alternative to rand(3,3)? +1 -- Scott M. Ransom Address: NRAO Phone: (434) 296-0320 520 Edgemont Rd. email: sransom at nrao.edu Charlottesville, VA 22903 USA GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From dd55 at cornell.edu Sat Jul 8 10:45:53 2006 From: dd55 at cornell.edu (Darren Dale) Date: Sat, 8 Jul 2006 10:45:53 -0400 Subject: [Numpy-discussion] NumPy beta release plan In-Reply-To: <44AF6704.4080407@sigmasquared.net> References: <44AF5BFA.70509@ieee.org> <44AF6704.4080407@sigmasquared.net> Message-ID: <200607081045.53198.dd55@cornell.edu> On Saturday 08 July 2006 4:04 am, Stephan Tolksdorf wrote: > Hi Travis, > > thanks for putting all the effort into NumPy. > > > Unless there is a serious issue pointed out, the only thing that should > > be changed at this point are bug-fixes, documentation strings, and added > > tests. Once 1.0b1 is released nothing but those things can be added. > > The beginning of the beta phase might also be the best time to address > the most severe NumPy.distutils issues, so that all changes can be > extensively tested during the beta: The number of people in my lab using numpy/scipy/matplotlib is starting to increase. Last week I was trying to get a couple of graduate students set up. They were using MacOS X 10.4 with gcc3.3, and we were never successful in getting numpy to build. I think the problem was the same as reported at http://mail.python.org/pipermail/pythonmac-sig/2006-March/016703.html. We ended up getting things installed by installing scipy with fink. > - warning messages are only shown if compilation fails (NumPy should > build warning free) One of the guys I was working with tried installing every compiler out there to get rid of all the compiler not found messages that were reported during the attempted numpy builds. I guess he thought these messages were critical errors, since they are rendered in red lettering. Darren From listservs at mac.com Sat Jul 8 11:11:02 2006 From: listservs at mac.com (listservs at mac.com) Date: Sat, 8 Jul 2006 11:11:02 -0400 Subject: [Numpy-discussion] bdist_mpkg Message-ID: As of recent svn updates, I can no longer build numpy binaries for OS X using bdist_mpkg. How do I restore this capability, or is there a better way of building Mac binaries? Thanks, C. -- Christopher Fonnesbeck + Atlanta, GA + fonnesbeck at mac.com + Contact me on AOL IM using email address From fonnesbeck at mac.com Sat Jul 8 11:16:21 2006 From: fonnesbeck at mac.com (Christopher Fonnesbeck) Date: Sat, 8 Jul 2006 11:16:21 -0400 Subject: [Numpy-discussion] Fwd: bdist_mpkg References: Message-ID: <6C4B2D05-5E9C-4C37-AF53-5750983094B3@mac.com> Never mind, I see that it is in setupegg.py. C. Begin forwarded message: > From: listservs at mac.com > Date: July 8, 2006 11:11:02 AM EDT > To: numpy-discussion at lists.sourceforge.net > Subject: bdist_mpkg > > As of recent svn updates, I can no longer build numpy binaries for > OS X using bdist_mpkg. How do I restore this capability, or is > there a better way of building Mac binaries? > > Thanks, > C. > > -- > Christopher Fonnesbeck > + Atlanta, GA > + fonnesbeck at mac.com > + Contact me on AOL IM using email address > > -- Christopher Fonnesbeck + Atlanta, GA + fonnesbeck at mac.com + Contact me on AOL IM using email address -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2365 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 186 bytes Desc: This is a digitally signed message part URL: From listservs at mac.com Sat Jul 8 11:16:43 2006 From: listservs at mac.com (listservs at mac.com) Date: Sat, 8 Jul 2006 11:16:43 -0400 Subject: [Numpy-discussion] Fwd: bdist_mpkg References: Message-ID: <35C4DE0B-141F-447D-9A8F-2B30895AD704@mac.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Never mind, I see that it is in setupegg.py. C. Begin forwarded message: > From: listservs at mac.com > Date: July 8, 2006 11:11:02 AM EDT > To: numpy-discussion at lists.sourceforge.net > Subject: bdist_mpkg > > As of recent svn updates, I can no longer build numpy binaries for > OS X using bdist_mpkg. How do I restore this capability, or is > there a better way of building Mac binaries? > > Thanks, > C. > > -- > Christopher Fonnesbeck > + Atlanta, GA > + fonnesbeck at mac.com > + Contact me on AOL IM using email address > > - -- Christopher Fonnesbeck + Atlanta, GA + fonnesbeck at mac.com + Contact me on AOL IM using email address - -- Christopher Fonnesbeck + Atlanta, GA + fonnesbeck at mac.com + Contact me on AOL IM using email address -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (Darwin) iD8DBQFEr8xbkeka2iCbE4wRAmxWAJ9jx/jFRJW0N4vCrh5H3SmN9IffvgCfWrRH pT90YecS/SM47DEzfeOSWKU= =D8D1 -----END PGP SIGNATURE----- From erin.sheldon at gmail.com Sat Jul 8 11:23:15 2006 From: erin.sheldon at gmail.com (Erin Sheldon) Date: Sat, 8 Jul 2006 11:23:15 -0400 Subject: [Numpy-discussion] recarray field names In-Reply-To: <4418B223.5020201@ee.byu.edu> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> <76fbbdec3883fc386929a698021fbf8b@stsci.edu> <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> <4418AAB9.3080200@ee.byu.edu> <4418AC54.1060202@colorado.edu> <4418B223.5020201@ee.byu.edu> Message-ID: <331116dc0607080823h7f3c7e4ar578ea660fa6c547e@mail.gmail.com> It was suggested that I put off this discussion until we were closer to the 1.0 release. Perhaps now is a good time to bring it up once again? The quick summary: accessing field names has some oddness that needs cleaning up. On 3/15/06, Travis Oliphant wrote: > Fernando Perez wrote: > > > > > Mmh, just curious: I wonder about the wisdom of that overloading of a > > 'magic' key (-1). It will make thinks like > > > > for name in t.dtype.fields: > > > No real wisdom. More organic growth. Intially I didn't have an ordered > list of fields, but as more complicated data-type descriptors were > supported, this became an important thing to have. I should have > probably added a n additional element to the PyArray_Descr structure. > Remember, it was growing out of the old PyArray_Descr already and I was > a little concerned about changing it too much (there are ramifications > of changing this structure in several places). > > So, instead of adding a "ordered_names" tuple to the dtype object, I > just stuck it in the fields dictionary. I agree it's kind of odd > sitting there. > > It probably deserves a re-factoring and pulling that out into a new > attribute of the dtype object. > > This would mean that the PyArray_Descr structure would need a new object > (names perhaps), and it would need to be tracked. > > Not a huge change and probably worth it before the next release. > > > -Travis From listservs at mac.com Sat Jul 8 11:39:25 2006 From: listservs at mac.com (listservs at mac.com) Date: Sat, 8 Jul 2006 11:39:25 -0400 Subject: [Numpy-discussion] more bdist_mpkg issues (setuptools) Message-ID: <7949CE47-7151-42F6-8DE6-AC017CB0EDC3@mac.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Now that I figured out how to get at bdist_mpkg, I am experiencing issues with building extensions. I am trying to set them up as before with distutils except with the following import statement: from setuptools import setup, Extension However, my f2py extension is not recognized: building 'PyMC.flib' extension error: unknown file type '.f' (from 'PyMC/flib.f') Any pointers to what I might be missing are most appreciated. Thanks, C. - -- Christopher Fonnesbeck + Atlanta, GA + fonnesbeck at mac.com + Contact me on AOL IM using email address -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (Darwin) iD8DBQFEr9Gvkeka2iCbE4wRAg+YAJwPIDDMCQ7GB0CEriKZAFBO8RM3dwCfX1PY Uz0/44rWk1n5i6U9o6Q7pVw= =j9vl -----END PGP SIGNATURE----- From aisaac at american.edu Sat Jul 8 12:15:05 2006 From: aisaac at american.edu (Alan Isaac) Date: Sat, 08 Jul 2006 09:15:05 -0700 Subject: [Numpy-discussion] Args for rand and randn: call for a vote Message-ID: <44AFDA09.5060209@american.edu> Ed Schofield wrote: > * Should numpy.rand and numpy.randn accept sequences of dimensions as > arguments, like rand((3,3)), as an alternative to rand(3,3)? > For allowing tuples: +1 For *only* allowing tuples: +1 Alan Isaac From fperez.net at gmail.com Sat Jul 8 15:03:17 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 8 Jul 2006 13:03:17 -0600 Subject: [Numpy-discussion] I think something broke badly today in numpy/scipy... In-Reply-To: <44AF7BCD.4080302@ieee.org> References: <44AF7BCD.4080302@ieee.org> Message-ID: On 7/8/06, Travis Oliphant wrote: > Hey Fernando. I think I found the problem. It was the same problem > causing the BFGS test to fail in SciPy. [...] > The problem should be fixed in SVN. Confirmed. With: numpy version: 0.9.9.2788 scipy version: 0.5.0.2057 I checked on two machines (home/office) and all is good now. I only had to make a tiny change to my hand-written extensions for the new C API names, and all checks OK now. > Very sorry... Considering how much you've given us, I don't really think you even have the /right/ to apologize to anyone ;) Many thanks for such a remarkable fix, given my impossibly vague bug report. You rock. f From charlesr.harris at gmail.com Sat Jul 8 15:36:23 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 8 Jul 2006 13:36:23 -0600 Subject: [Numpy-discussion] NumPy beta release plan In-Reply-To: <200607081045.53198.dd55@cornell.edu> References: <44AF5BFA.70509@ieee.org> <44AF6704.4080407@sigmasquared.net> <200607081045.53198.dd55@cornell.edu> Message-ID: Hi Travis, Are you going to make a development branch so other things can continue going into Numpy until the 1.0 release? Chuck On 7/8/06, Darren Dale wrote: > > On Saturday 08 July 2006 4:04 am, Stephan Tolksdorf wrote: > > Hi Travis, > > > > thanks for putting all the effort into NumPy. > > > > > Unless there is a serious issue pointed out, the only thing that > should > > > be changed at this point are bug-fixes, documentation strings, and > added > > > tests. Once 1.0b1 is released nothing but those things can be added. > > > > The beginning of the beta phase might also be the best time to address > > the most severe NumPy.distutils issues, so that all changes can be > > extensively tested during the beta: > > The number of people in my lab using numpy/scipy/matplotlib is starting to > increase. Last week I was trying to get a couple of graduate students set > up. > They were using MacOS X 10.4 with gcc3.3, and we were never successful in > getting numpy to build. I think the problem was the same as reported at > http://mail.python.org/pipermail/pythonmac-sig/2006-March/016703.html. We > ended up getting things installed by installing scipy with fink. > > > - warning messages are only shown if compilation fails (NumPy should > > build warning free) > > One of the guys I was working with tried installing every compiler out > there > to get rid of all the compiler not found messages that were reported > during > the attempted numpy builds. I guess he thought these messages were > critical > errors, since they are rendered in red lettering. > > Darren > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Jul 8 15:38:17 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 08 Jul 2006 14:38:17 -0500 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> Message-ID: Ed Schofield wrote: > Last week's discussion on rand() and randn() seemed to indicate a > sentiment that they ought to take tuples for consistency with ones, > zeros, eye, identity, and empty -- that, although they are supposed > to be convenience functions, they are inconvenient precisely because > of their inconsistency with these other functions. This issue has > been raised many times over the past several months. > > Travis made a change in r2572 to allow tuples as arguments, then took > it out again a few hours later, apparently unsure about whether this > was a good idea. He was experimenting to see if it was technically easy or not. > I'd like to call for a vote on what people would prefer, and then ask > Travis to make a final pronouncement before the feature freeze. > > * Should numpy.rand and numpy.randn accept sequences of dimensions as > arguments, like rand((3,3)), as an alternative to rand(3,3)? -1 This is worse than the problem it's trying to solve. You'll still see the inconsistent rand(3,3) in people's code. > * Should rand((3,3)) and randn((3,3)) continue to raise a TypeError? This is a false dichotomy. There are more choices here. * Remove rand and randn (at least from the toplevel namespace) and promote the use of random_sample and standard_normal which already follow the tuple convention. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From fperez.net at gmail.com Sat Jul 8 15:50:17 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 8 Jul 2006 13:50:17 -0600 Subject: [Numpy-discussion] .M .A .T .H attribute result In-Reply-To: <44AEC258.9040800@ee.byu.edu> References: <44AEC258.9040800@ee.byu.edu> Message-ID: Hi all, On 7/7/06, Travis Oliphant wrote: > > I didn't compile the results, but the discussion on the idea of adding > new attributes to the array object led to the following result. > > Added: .T attribute to mean self.transpose() [...] > .H > > A few were in favor, but this can now be written .T.conj() which is not > bad so does not get a new attribute. I didn't have strong feelings one way or another on this, so I didn't vote, but at this point I'd like to make a comment before the freeze. Given that .T went in, I'd argue that .H should be in as well. Basically, I now think these two should have been considered as a bundle and not as separate options. The reason isn't (just) my OCD surfacing again, but the fact that the hermitian conjugate plays /exactly/ the role that transposition plays, in regards to defining norms and (positive definite) inner products, when complex arrays are in play. The fact that numpy has excellent complex support is a major source of joy for many. I think that having .T but not .H would be a big wart in this regard. If you are trying to write code for inner products with complex arrays, the natural language change from real ones is: dot(A.T,B) -> dot(A.H,B) For people who play with quantum mechanics this is an everyday need (self-adjoint operators require expressions like this all the time), but I suspect its use will be common in any field requiring normed spaces with complex arrays. Just my 1e-2j Cheers, f From charlesr.harris at gmail.com Sat Jul 8 15:57:41 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 8 Jul 2006 13:57:41 -0600 Subject: [Numpy-discussion] .M .A .T .H attribute result In-Reply-To: References: <44AEC258.9040800@ee.byu.edu> Message-ID: Hi all, On 7/8/06, Fernando Perez wrote: > > Hi all, > > On 7/7/06, Travis Oliphant wrote: > > > > I didn't compile the results, but the discussion on the idea of adding > > new attributes to the array object led to the following result. > > > > Added: .T attribute to mean self.transpose() > > [...] > > > .H > > > > A few were in favor, but this can now be written .T.conj() which is not > > bad so does not get a new attribute. > > I didn't have strong feelings one way or another on this, so I didn't > vote, but at this point I'd like to make a comment before the freeze. > Given that .T went in, I'd argue that .H should be in as well. > Basically, I now think these two should have been considered as a > bundle and not as separate options. +1. H is just the complex counterpart of T. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Jul 8 18:04:46 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 08 Jul 2006 17:04:46 -0500 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> Message-ID: Ed Schofield wrote: > Last week's discussion on rand() and randn() seemed to indicate a > sentiment that they ought to take tuples for consistency with ones, > zeros, eye, identity, and empty -- that, although they are supposed > to be convenience functions, they are inconvenient precisely because > of their inconsistency with these other functions. This issue has > been raised many times over the past several months. > > Travis made a change in r2572 to allow tuples as arguments, then took > it out again a few hours later, apparently unsure about whether this > was a good idea. > > I'd like to call for a vote on what people would prefer, and then ask > Travis to make a final pronouncement before the feature freeze. I would like to ask about the purpose of calling for a vote, here. What authority do you intend the result to have? If you are just asking for a straw poll of opinions from the list to inform Travis' decision, do you think that he hasn't read the previous discussions? Are previous non-participants being drawn out of the woodwork? Or do you intend the majority winner of the vote to force Travis' decision? In which case, who comprises the voting membership, and why? Voting is a notoriously bad way to make decisions in software development. Open source did not change that. It should be a very rare occurrence and happen in very specific circumstances. Two CFVs in as many days is entirely unreasonable. And when a vote *is* taken, it should be done with much more care than I think you are showing here (specifically, a CFV should not be making the case for only one of the choices). Karl Fogel's book _Producing Open Source Software_ thoroughly discusses these issues: http://producingoss.com/html-chunk/consensus-democracy.html#voting -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pau.gargallo at gmail.com Sat Jul 8 18:59:38 2006 From: pau.gargallo at gmail.com (Pau Gargallo) Date: Sun, 9 Jul 2006 00:59:38 +0200 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> Message-ID: <6ef8f3380607081559v62daf4f7t9c3224536e070c0@mail.gmail.com> On 7/8/06, Robert Kern wrote: > Ed Schofield wrote: > > * Should numpy.rand and numpy.randn accept sequences of dimensions as > > arguments, like rand((3,3)), as an alternative to rand(3,3)? > > > * Should rand((3,3)) and randn((3,3)) continue to raise a TypeError? > > This is a false dichotomy. There are more choices here. > > * Remove rand and randn (at least from the toplevel namespace) and promote the > use of random_sample and standard_normal which already follow the tuple convention. > i just wanted to point out another possible choice: * enable numpy.rand((3,3)) and make numpy.rand(3,3) raise an error as zeros and ones do. I suppose that you all discussed a lot about this choice also, but it still seems very reasonable to me :-( pau From robert.kern at gmail.com Sat Jul 8 19:15:25 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 08 Jul 2006 18:15:25 -0500 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <6ef8f3380607081559v62daf4f7t9c3224536e070c0@mail.gmail.com> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> <6ef8f3380607081559v62daf4f7t9c3224536e070c0@mail.gmail.com> Message-ID: Pau Gargallo wrote: > On 7/8/06, Robert Kern wrote: >> Ed Schofield wrote: >>> * Should numpy.rand and numpy.randn accept sequences of dimensions as >>> arguments, like rand((3,3)), as an alternative to rand(3,3)? >>> * Should rand((3,3)) and randn((3,3)) continue to raise a TypeError? >> This is a false dichotomy. There are more choices here. >> >> * Remove rand and randn (at least from the toplevel namespace) and promote the >> use of random_sample and standard_normal which already follow the tuple convention. > > i just wanted to point out another possible choice: > > * enable numpy.rand((3,3)) and make numpy.rand(3,3) raise an error > as zeros and ones do. > > I suppose that you all discussed a lot about this choice also, but it > still seems very reasonable to me :-( Extensively! The problem I have with it is that there are already functions which do exactly that. Why have two functions that do exactly the same thing with exactly the same interface but different names? I say, pick one and get rid of the other if you must remove rand(3,3). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From josh8912 at yahoo.com Sat Jul 8 16:22:03 2006 From: josh8912 at yahoo.com (JJ) Date: Sat, 8 Jul 2006 20:22:03 +0000 (UTC) Subject: [Numpy-discussion] What's wrong with matrices? References: <44AE6675.108@ftw.at> Message-ID: Hello. My 2 cents may be late on this, but I do have some thoughts as a matlab user switching over to numpy/scipy. I will just list my points, and I have a general question at the end. 1) I vote for a simple way to allow matrices to be the default (vs arrays), with all matrix operations/functions/range selections returning matrices. 2) Allowing more than 2D matrices would be nice, but I dont need them often. 3) In some operations, especially range selection operations, a N,1 matrix is turned into a 1,N matrix. This is confusing relative to matlab and problematic, in my view. 3) From my experience so far, range selection is by far the most problematic part of using numpy. In matlab, range selection is very easy and intuitive. For example, if X is a 10 x 5 matrix, a = X[X[:,1]<3,1:3] works just fine. In numpy it is much more confusing and the code to do this is relatively lengthy. It would be nice if range selection for a matrix could be done using any combination of boolean and/or integer matrices (not lists or arrays, necessarily). For example, boolean selection for rows and integer selection for columns. 4) It would be very nice if range selection could be done with real numbers, such as 1.0, 2.0, etc. Matlab can do this and it makes the code much simpler. 5) If X is a 1,n matrix, then X.T should return a n,1 matrix, I think. I guess thats my main points. And now I have a question. What do people think about the role of numpy/scipy vs. R (other than one is python and one is not)? What are the benefits of numpy/scipy over R? In the future, do numpy users want stat libraries like R has? Or put another way, when numpy/scipy is mature, when would you use numpy and when would you use R? Thanks. JJ (John) From sransom at nrao.edu Sat Jul 8 21:23:29 2006 From: sransom at nrao.edu (Scott Ransom) Date: Sat, 8 Jul 2006 21:23:29 -0400 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> <6ef8f3380607081559v62daf4f7t9c3224536e070c0@mail.gmail.com> Message-ID: <20060709012329.GA32684@ssh.cv.nrao.edu> On Sat, Jul 08, 2006 at 06:15:25PM -0500, Robert Kern wrote: > Pau Gargallo wrote: > > On 7/8/06, Robert Kern wrote: > >> Ed Schofield wrote: > >>> * Should numpy.rand and numpy.randn accept sequences of dimensions as > >>> arguments, like rand((3,3)), as an alternative to rand(3,3)? > >>> * Should rand((3,3)) and randn((3,3)) continue to raise a TypeError? > >> This is a false dichotomy. There are more choices here. > >> > >> * Remove rand and randn (at least from the toplevel namespace) and promote the > >> use of random_sample and standard_normal which already follow the tuple convention. > > > > i just wanted to point out another possible choice: > > > > * enable numpy.rand((3,3)) and make numpy.rand(3,3) raise an error > > as zeros and ones do. > > > > I suppose that you all discussed a lot about this choice also, but it > > still seems very reasonable to me :-( > > Extensively! The problem I have with it is that there are already functions > which do exactly that. Why have two functions that do exactly the same thing > with exactly the same interface but different names? I say, pick one and get rid > of the other if you must remove rand(3,3). Robert, what you say is a perfectly valid solution for the spirit of the first selection. My +1 vote was really for _some_ functions in the top level numpy namespace that take tuple "shape" parameters as input and generate uniform and gaussian normal random deviates. If those functions are random_sample (although that is a bad name, IMO, "random" is better -- which is already in there) and standard_normal that is fine by me. The problem was (is?) that rand and randn are not consistent with everything else. _That_ is why people are having issues. Scott -- Scott M. Ransom Address: NRAO Phone: (434) 296-0320 520 Edgemont Rd. email: sransom at nrao.edu Charlottesville, VA 22903 USA GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From haase at msg.ucsf.edu Sun Jul 9 01:16:55 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sat, 08 Jul 2006 22:16:55 -0700 Subject: [Numpy-discussion] build problem: numpy with cygwin mingw Message-ID: <44B09147.1030107@msg.ucsf.edu> Hi, I just tried my first build from a svn checkout on windows using cygwin. I want to use the non cygwin python - i.e. the non-cygwin (mingw) compiler more. I get a presumably very basic error: Even though I follow the simple instructions on http://www.scipy.org/Installing_SciPy/Windows calling python.exe setup.py config --compiler=mingw32 build --compiler=mingw32 bdist_wininst I noticed that gcc gets called without the "-mno-cygwin". From this follows that _WIN32 is not defined and than I get: Could not locate executable gfortran Could not locate executable f95 customize GnuFCompiler customize GnuFCompiler customize GnuFCompiler using config C compiler: gcc -O2 -Wall -Wstrict-prototypes compile options: '-Ic:\python24\include -Inumpy\core\src -Inumpy\core\include -I c:\python24\include -Ic:\python24\PC -c' gcc -O2 -Wall -Wstrict-prototypes -Ic:\python24\include -Inumpy\core\src -Inumpy\core\include -Ic:\python24\include -Ic:\python24\PC -c _configtest.c -o _config test.o In file included from c:/python24/include/Python.h:82, from _configtest.c:2: c:/python24/include/intobject.h:41: error: parse error before "PyInt_AsUnsignedLongLongMask" The error in the last line I traced back to this line: #define PY_LONG_LONG __int64 in pyconfig.h which is used because #if defined(__GNUC__) && defined(_WIN32) is not true if gcc is called without "-mno-cygwin" I don't have cygwin's python installed. I neither have cygwin's non-mingw gcc installed.... According to http://www.python.org/doc/current/inst/tweak-flags.html the --compiler=mingw32 should tell distutils to use the "Cygwin in no-cygwin mode" !?!? What's wrong here ? Thanks Sebastian Haase PS: I don't even care about ATLAS... at this point PPS: Is there maybe a precompiled version later than 0.9.8 avaible !? From oliphant.travis at ieee.org Sun Jul 9 01:20:47 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 08 Jul 2006 23:20:47 -0600 Subject: [Numpy-discussion] NumPy beta release plan In-Reply-To: References: <44AF5BFA.70509@ieee.org> <44AF6704.4080407@sigmasquared.net> <200607081045.53198.dd55@cornell.edu> Message-ID: <44B0922F.6030205@ieee.org> Charles R Harris wrote: > Hi Travis, > > Are you going to make a development branch so other things can > continue going into Numpy until the 1.0 release? Something like that is a good idea. I'm open to suggestions... -Travis From schofield at ftw.at Sun Jul 9 01:58:02 2006 From: schofield at ftw.at (Ed Schofield) Date: Sun, 9 Jul 2006 07:58:02 +0200 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> Message-ID: <2CFD8EBF-9685-4A7A-9905-125A21BE769C@ftw.at> On 09/07/2006, at 12:04 AM, Robert Kern wrote: > Ed Schofield wrote: >> Last week's discussion on rand() and randn() seemed to indicate a >> sentiment that they ought to take tuples for consistency with ones, >> zeros, eye, identity, and empty -- that, although they are supposed >> to be convenience functions, they are inconvenient precisely because >> of their inconsistency with these other functions. This issue has >> been raised many times over the past several months. >> >> Travis made a change in r2572 to allow tuples as arguments, then took >> it out again a few hours later, apparently unsure about whether this >> was a good idea. >> >> I'd like to call for a vote on what people would prefer, and then ask >> Travis to make a final pronouncement before the feature freeze. > > I would like to ask about the purpose of calling for a vote, here. > What > authority do you intend the result to have? If you are just asking > for a straw > poll of opinions from the list to inform Travis' decision, do you > think that he > hasn't read the previous discussions? Are previous non-participants > being drawn > out of the woodwork? ... No authority at all -- it's just a straw poll. My intention is merely to ascertain whether there's indeed a groundswell of public opinion for this change among NumPy's users, as is my suspicion. Previous non-participants are indeed coming out of the woodwork. The reason I've listed only one alternative to the status quo is that this is a simple, concrete proposal that has bubbled up several times from the discussion, which may have broad enough support to tip Travis's decision. I know that you, Robert, disagree with the proposal and have put forward an alternative. Fair enough, but let's now hear what others have to say. -- Ed From schofield at ftw.at Sun Jul 9 02:35:29 2006 From: schofield at ftw.at (Ed Schofield) Date: Sun, 9 Jul 2006 08:35:29 +0200 Subject: [Numpy-discussion] What's wrong with matrices? In-Reply-To: References: <44AE6675.108@ftw.at> Message-ID: <3FE6FE60-D0B9-422B-A73C-1B832FDADB1B@ftw.at> On 08/07/2006, at 10:22 PM, JJ wrote: > 3) In some operations, especially range selection operations, a N, > 1 matrix is > turned into a 1,N matrix. This is confusing relative to matlab and > problematic, > in my view. This sounds like a bug. Can you give any examples of this happening with the latest release or SVN version? > > > 5) If X is a 1,n matrix, then X.T should return a n,1 matrix, I > think. This should be the case. Could you post a code snippet that violates it? Thanks for your feedback, JJ! -- Ed From robert.kern at gmail.com Sun Jul 9 02:44:07 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 09 Jul 2006 01:44:07 -0500 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <2CFD8EBF-9685-4A7A-9905-125A21BE769C@ftw.at> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> <2CFD8EBF-9685-4A7A-9905-125A21BE769C@ftw.at> Message-ID: Ed Schofield wrote: > On 09/07/2006, at 12:04 AM, Robert Kern wrote: > >> I would like to ask about the purpose of calling for a vote, here. >> What >> authority do you intend the result to have? If you are just asking >> for a straw >> poll of opinions from the list to inform Travis' decision, do you >> think that he >> hasn't read the previous discussions? Are previous non-participants >> being drawn >> out of the woodwork? ... > > No authority at all -- it's just a straw poll. My intention is > merely to ascertain whether there's indeed a groundswell of public > opinion for this change among NumPy's users, as is my suspicion. > > Previous non-participants are indeed coming out of the woodwork. > > The reason I've listed only one alternative to the status quo is that > this is a simple, concrete proposal that has bubbled up several times > from the discussion, which may have broad enough support to tip > Travis's decision. > > I know that you, Robert, disagree with the proposal and have put > forward an alternative. Fair enough, but let's now hear what others > have to say. The problem I have is that the way this happened is ineffective at doing that honestly. Fogel's book has a good discussion about how these should be conducted. You can't just pick the options that are convenient for you. You can't argue for your favorite in the CFV. You should provide pointers to the previous discussions so that newcomers can understand what the arguments are. You should encourage people to come up with alternatives that might achieve consensus rather than just picking the ones on the ballot. Otherwise, the poll just serves as rhetorical ploy, not a way to ascertain people's opinions. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From st at sigmasquared.net Sun Jul 9 02:56:44 2006 From: st at sigmasquared.net (Stephan Tolksdorf) Date: Sun, 09 Jul 2006 08:56:44 +0200 Subject: [Numpy-discussion] build problem: numpy with cygwin mingw In-Reply-To: <44B09147.1030107@msg.ucsf.edu> References: <44B09147.1030107@msg.ucsf.edu> Message-ID: <44B0A8AC.2030008@sigmasquared.net> Hi Sebastian, > I just tried my first build from a svn checkout on windows using > cygwin. I want to use the non cygwin python - i.e. the non-cygwin > (mingw) compiler more. > I get a presumably very basic error: > Even though I follow the simple instructions on > http://www.scipy.org/Installing_SciPy/Windows did you apply the patch from ticket #114, as mentioned on top of that page? The Cygwin build is known not to work otherwise (due to incorrect compiler flags, as you noticed)... Stephan From josh8912 at yahoo.com Sun Jul 9 04:10:27 2006 From: josh8912 at yahoo.com (JJ) Date: Sun, 9 Jul 2006 08:10:27 +0000 (UTC) Subject: [Numpy-discussion] What's wrong with matrices? References: <44AE6675.108@ftw.at> <3FE6FE60-D0B9-422B-A73C-1B832FDADB1B@ftw.at> Message-ID: Ed Schofield ftw.at> writes: > > > On 08/07/2006, at 10:22 PM, JJ wrote: > > > 3) In some operations, especially range selection operations, a N, > > 1 matrix is > > turned into a 1,N matrix. This is confusing relative to matlab and > > problematic, > > in my view. > > This sounds like a bug. Can you give any examples of this happening > with the latest release or SVN version? > > > > > > > 5) If X is a 1,n matrix, then X.T should return a n,1 matrix, I > > think. > > This should be the case. Could you post a code snippet that violates > it? > > Thanks for your feedback, JJ! > ----------------------------------------- Hello Ed: Here are a couple of examples off the top of my head: a = mat(arange(10)) a.shape = (5,2) b = a.copy() c = hstack((a,b)) # should return a matrix type(c) a[where(a[:,0]<6)[0],0] #should return a column matrix([[0, 2, 4]]) My version is about a month old, so maybe these are fixed. Since we are on the topic, I dont understand why where() returns a tupple. To me, it just means adding an extra '[0]' in many places in my code. I would vote for where() retuning a matrix (if matrix is the default in the session). My comment on transpose of a 1,n matrix producing a n,1 matrix was in response to a vote question on the list. Also, I would vote for X.A*Y.A to return a matrix, if matrix is the default for a session. Lastly, this might not be the right place for this comment, but it would be nice if pylab allowed numpy matrices. Then code like: plot(d.A.ravel().tolist()) could be changed to plot(d) Hope this is of some help. JJ From josh8912 at yahoo.com Sun Jul 9 04:23:54 2006 From: josh8912 at yahoo.com (JJ) Date: Sun, 9 Jul 2006 08:23:54 +0000 (UTC) Subject: [Numpy-discussion] What's wrong with matrices? References: <44AE6675.108@ftw.at> <3FE6FE60-D0B9-422B-A73C-1B832FDADB1B@ftw.at> Message-ID: Ed Schofield ftw.at> writes: > > > On 08/07/2006, at 10:22 PM, JJ wrote: > > > 3) In some operations, especially range selection operations, a N, > > 1 matrix is > > turned into a 1,N matrix. This is confusing relative to matlab and > > problematic, > > in my view. > > This sounds like a bug. Can you give any examples of this happening > with the latest release or SVN version? > > > > > > > 5) If X is a 1,n matrix, then X.T should return a n,1 matrix, I > > think. > > This should be the case. Could you post a code snippet that violates > it? > > Thanks for your feedback, JJ! >-------------------------------------------------- Hi Ed: Here are a couple more items: c = cov(a) # a is a matrix type(c) # should be a matrix Also, if a is 5x2, then cov(a) is 5,5. Is that just a convention? I would expect the cov to be 2,2, as in matlab. It means I have to use more transpose methods. D is a 1,n matrix (D<2).nonzero()[0] As per my earlier comment on which(), I wish that .nonzero() did not return a tupple. From gruben at bigpond.net.au Sun Jul 9 04:54:39 2006 From: gruben at bigpond.net.au (Gary Ruben) Date: Sun, 09 Jul 2006 18:54:39 +1000 Subject: [Numpy-discussion] NumPy beta release plan In-Reply-To: <44B0922F.6030205@ieee.org> References: <44AF5BFA.70509@ieee.org> <44AF6704.4080407@sigmasquared.net> <200607081045.53198.dd55@cornell.edu> <44B0922F.6030205@ieee.org> Message-ID: <44B0C44F.7020601@bigpond.net.au> Hi Travis, My suggestion: Because you said you want to do a feature freeze, this should be pretty easy to handle. Wait for a feature freeze point then create a 1.0 branch. From then, new stuff can continue going into the trunk. Bug fixes go into the trunk and the 1.0 branch. Gary R. Travis Oliphant wrote: > Charles R Harris wrote: >> Hi Travis, >> >> Are you going to make a development branch so other things can >> continue going into Numpy until the 1.0 release? > Something like that is a good idea. I'm open to suggestions... > > -Travis From MAILER-DAEMON at dodoma.magic.fr Sun Jul 9 06:07:08 2006 From: MAILER-DAEMON at dodoma.magic.fr (Mail Delivery Subsystem) Date: Sun, 9 Jul 2006 12:07:08 +0200 Subject: [Numpy-discussion] Returned mail: see transcript for details Message-ID: <200607091007.k69A78aK031065@dodoma.magic.fr> The original message was received at Sun, 9 Jul 2006 12:07:07 +0200 from asuncion.magic.fr [195.154.233.103] ----- The following addresses had permanent fatal errors ----- (reason: 550 5.1.1 ... User unknown) ----- Transcript of session follows ----- ... while talking to [62.210.158.43]: >>> RCPT To: <<< 550 5.1.1 ... User unknown 550 5.1.1 ... User unknown From numpy-discussion at lists.sourceforge.net Sun Jul 9 06:06:28 2006 From: numpy-discussion at lists.sourceforge.net (numpy-discussion at lists.sourceforge.net) Date: Sun, 9 Jul 2006 12:06:28 +0200 Subject: Hello Message-ID: <200607091006.k69A6W5U030335@manille.magic.fr> A non-text attachment was scrubbed... Name: not available Type: multipart/mixed Size: 49 bytes Desc: not available URL: From svetosch at gmx.net Sun Jul 9 10:55:55 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Sun, 09 Jul 2006 16:55:55 +0200 Subject: [Numpy-discussion] What's wrong with matrices? In-Reply-To: References: <44AE6675.108@ftw.at> <3FE6FE60-D0B9-422B-A73C-1B832FDADB1B@ftw.at> Message-ID: <44B118FB.7040004@gmx.net> JJ schrieb: > ----------------------------------------- > Hello Ed: > Here are a couple of examples off the top of my head: > > a = mat(arange(10)) > a.shape = (5,2) > b = a.copy() > c = hstack((a,b)) # should return a matrix > type(c) > This hstack bug has been fixed recently. > > a[where(a[:,0]<6)[0],0] #should return a column > matrix([[0, 2, 4]]) Don't know anything about this stuff. > > My version is about a month old, so maybe these are fixed. Since we are on the > topic, I dont understand why where() returns a tupple. To me, it just means > adding an extra '[0]' in many places in my code. I would vote for where() > retuning a matrix (if matrix is the default in the session). My comment on > transpose of a 1,n matrix producing a n,1 matrix was in response to a vote > question on the list. That discussion was about pure arrays, not the matrix subclass. > matrix is the default for a session. Lastly, this might not be the right place > for this comment, but it would be nice if pylab allowed numpy matrices. I agree, that would be nice! Maybe we can team up and invade the matplotlib mailing list some time ;-) -sven From esheldon at kicp.uchicago.edu Sun Jul 9 10:55:36 2006 From: esheldon at kicp.uchicago.edu (Erin Sheldon) Date: Sun, 9 Jul 2006 09:55:36 -0500 Subject: [Numpy-discussion] recarray field names Message-ID: <20060709095536.A5542@cfcp.uchicago.edu> Hi everyone - (sigh) These gmail issues are really annoying...so I apologize if the gmail version of this message shows up in a few days. I brought up the issue a while back of having a simple way to access the field names of an array. The quick summary: accessing field names has some oddness that needs cleaning up. The argument that this is important isn't too hard to make: most astronomers work with lists of objects with a slew of properties and these are most naturally stored in a recarray. We use generic file formats like FITS that don't require the user to know what is in the file ahead of time, but once the data is read this information should be easily accessible, both interactively and in code. I'm sure similar arguments can be made for other users. Thanks, Erin On 3/15/06, Travis Oliphant wrote: - Hide quoted text - > Fernando Perez wrote: > > > > > Mmh, just curious: I wonder about the wisdom of that overloading of a > > 'magic' key (-1). It will make thinks like > > > > for name in t.dtype.fields: > > > No real wisdom. More organic growth. Intially I didn't have an ordered > list of fields, but as more complicated data-type descriptors were > supported, this became an important thing to have. I should have > probably added a n additional element to the PyArray_Descr structure. > Remember, it was growing out of the old PyArray_Descr already and I was > a little concerned about changing it too much (there are ramifications > of changing this structure in several places). > > So, instead of adding a "ordered_names" tuple to the dtype object, I > just stuck it in the fields dictionary. I agree it's kind of odd > sitting there. > > It probably deserves a re-factoring and pulling that out into a new > attribute of the dtype object. > > This would mean that the PyArray_Descr structure would need a new object > (names perhaps), and it would need to be tracked. > > Not a huge change and probably worth it before the next release. > > > -Travis From haase at msg.ucsf.edu Sun Jul 9 15:54:52 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sun, 09 Jul 2006 12:54:52 -0700 Subject: [Numpy-discussion] build problem: numpy with cygwin mingw In-Reply-To: <44B0A8AC.2030008@sigmasquared.net> References: <44B09147.1030107@msg.ucsf.edu> <44B0A8AC.2030008@sigmasquared.net> Message-ID: <44B15F0C.7030300@msg.ucsf.edu> Hi Stephan, thanks a lot ! Of course I did NOT read the small print at the top ;-) Now I just have to learn how to apply a patch ;-) (Funny, that after more than ten years of Linux I still don't know this ... ) Thanks again - I'm sure it will work now. (I hope) - Sebastian Stephan Tolksdorf wrote: > Hi Sebastian, > > > I just tried my first build from a svn checkout on windows using > > cygwin. I want to use the non cygwin python - i.e. the non-cygwin > > (mingw) compiler more. > > I get a presumably very basic error: > > Even though I follow the simple instructions on > > http://www.scipy.org/Installing_SciPy/Windows > > did you apply the patch from ticket #114, as mentioned on top of that > page? The Cygwin build is known not to work otherwise (due to incorrect > compiler flags, as you noticed)... > > Stephan From haase at msg.ucsf.edu Sun Jul 9 16:09:31 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sun, 09 Jul 2006 13:09:31 -0700 Subject: [Numpy-discussion] build problem: numpy with cygwin mingw In-Reply-To: <44B15F0C.7030300@msg.ucsf.edu> References: <44B09147.1030107@msg.ucsf.edu> <44B0A8AC.2030008@sigmasquared.net> <44B15F0C.7030300@msg.ucsf.edu> Message-ID: <44B1627B.8060708@msg.ucsf.edu> Hi all, I was to quick with my last reply. The patch in ticket 114 (http://projects.scipy.org/scipy/numpy/ticket/114) refers (among others) to a file distutils/system_info.py but my windows python (version 2.4.3) does not contain any file like that in C:\Python24\Lib\distutils What am I missing this time !? Thanks, Sebastian Haase Sebastian Haase wrote: > Hi Stephan, > thanks a lot ! Of course I did NOT read the small print at the top ;-) > > Now I just have to learn how to apply a patch ;-) > (Funny, that after more than ten years of Linux I still don't know this > ... ) > > Thanks again - I'm sure it will work now. > (I hope) > - Sebastian > > > Stephan Tolksdorf wrote: >> Hi Sebastian, >> >> > I just tried my first build from a svn checkout on windows using >> > cygwin. I want to use the non cygwin python - i.e. the non-cygwin >> > (mingw) compiler more. >> > I get a presumably very basic error: >> > Even though I follow the simple instructions on >> > http://www.scipy.org/Installing_SciPy/Windows >> >> did you apply the patch from ticket #114, as mentioned on top of that >> page? The Cygwin build is known not to work otherwise (due to incorrect >> compiler flags, as you noticed)... >> >> Stephan > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From lists.steve at arachnedesign.net Sun Jul 9 17:06:45 2006 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Sun, 9 Jul 2006 17:06:45 -0400 Subject: [Numpy-discussion] build problem: numpy with cygwin mingw In-Reply-To: <44B1627B.8060708@msg.ucsf.edu> References: <44B09147.1030107@msg.ucsf.edu> <44B0A8AC.2030008@sigmasquared.net> <44B15F0C.7030300@msg.ucsf.edu> <44B1627B.8060708@msg.ucsf.edu> Message-ID: On Jul 9, 2006, at 4:09 PM, Sebastian Haase wrote: > Hi all, > I was to quick with my last reply. > The patch in ticket 114 (http://projects.scipy.org/scipy/numpy/ > ticket/114) > refers (among others) to a file > distutils/system_info.py > but my windows python (version 2.4.3) does not contain any file > like that in > C:\Python24\Lib\distutils > > What am I missing this time !? Perhaps it's referring to the distutils folder in the numpy package .. should be in: /path/to/numpy/checkout/numpy/distutils/system_info.py -steve From haase at msg.ucsf.edu Sun Jul 9 17:54:19 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sun, 09 Jul 2006 14:54:19 -0700 Subject: [Numpy-discussion] build problem: numpy with cygwin mingw In-Reply-To: References: <44B09147.1030107@msg.ucsf.edu> <44B0A8AC.2030008@sigmasquared.net> <44B15F0C.7030300@msg.ucsf.edu> <44B1627B.8060708@msg.ucsf.edu> Message-ID: <44B17B0B.7080609@msg.ucsf.edu> Steve Lianoglou wrote: > On Jul 9, 2006, at 4:09 PM, Sebastian Haase wrote: > >> Hi all, >> I was to quick with my last reply. >> The patch in ticket 114 >> (http://projects.scipy.org/scipy/numpy/ticket/114) >> refers (among others) to a file >> distutils/system_info.py >> but my windows python (version 2.4.3) does not contain any file like >> that in >> C:\Python24\Lib\distutils >> >> What am I missing this time !? > > Perhaps it's referring to the distutils folder in the numpy package .. > should be in: > /path/to/numpy/checkout/numpy/distutils/system_info.py > Hi Steve, Thanks for the hint. Before I was looking for numpy/distutils (which did not exist) and now I found it as numpy/numpy/distutils. Done. Worked - thanks. Now, I was trying to build SciPy - without getting through the ATALS/Lapack precedure ! So I get the expected Warnings (... NOT AVAILABLE) but then (right after those) I get an error: File "C:\Python24\Lib\site-packages\numpy\distutils\system_info.py", line 403, in get_info raise self.notfounderror,self.notfounderror.__doc__ numpy.distutils.system_info.NotFoundError: Some third-party program or library is not found. Does SciPy not come anymore with a fall back non-Lapack version included !? Thanks, Sebastian Haase From haase at msg.ucsf.edu Sun Jul 9 17:59:20 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sun, 09 Jul 2006 14:59:20 -0700 Subject: [Numpy-discussion] numpy dtype str is ' Hi Travis, Is it possible to change dtype str() [and repr() ?] of "standard types" to a more user friendly string: Example: instead of '>> N.array([1,2]) [1 2] >>> _.dtype ' Message-ID: Travis Oliphant ieee.org> writes: > > > Some of you may have noticed that things have been changing rapidly in > the NumPy world (if you were out of the office in June then all the > activity may seem overwhelming). > > All of this activity is based on the fact that the upcoming beta release > will mean a feature freeze for NumPy. Recently numpy.distutils switched to install to /usr/local/lib instead of /usr/lib. Is that intended? I'd prefer to have it back at /usr/lib in 1.0beta. Anyway, I'm really looking forward to seeing a 1.0 release. Keep up the good work. Regards, Christian From wbaxter at gmail.com Sun Jul 9 21:08:44 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 10 Jul 2006 10:08:44 +0900 Subject: [Numpy-discussion] building ATLAS on Win32 - endless compile loop In-Reply-To: <1152370680.2347.1.camel@katholt.sonderport.dk> References: <1152370680.2347.1.camel@katholt.sonderport.dk> Message-ID: I did some searching on the web and found that Pentium M is more or less a Pentium III. After selecting "Pentium III" in the ATLAS setup, the build worked. Still, on my ~1GHz laptop took several hours, nothing like the 15 minutes advertised on the Wiki. (I updated the wiki accordingly) Just out of curiostiy what processor is the official Numpy win32 distribution built for? Or does it just not use optimized blas/lapack? --Bill On 7/8/06, Peter S?ndergaard wrote: > > > > My laptop is a Pentium M, which isn't one of the options on the > > architecture list. So I picked "unknown". > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sun Jul 9 21:13:43 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 09 Jul 2006 20:13:43 -0500 Subject: [Numpy-discussion] NumPy beta release plan In-Reply-To: References: <44AF5BFA.70509@ieee.org> Message-ID: Christian Kristukat wrote: > Travis Oliphant ieee.org> writes: > >> Some of you may have noticed that things have been changing rapidly in >> the NumPy world (if you were out of the office in June then all the >> activity may seem overwhelming). >> >> All of this activity is based on the fact that the upcoming beta release >> will mean a feature freeze for NumPy. > > Recently numpy.distutils switched to install to /usr/local/lib instead of > /usr/lib. Is that intended? I'd prefer to have it back at /usr/lib in > 1.0beta. It shouldn't have. It shouldn't be doing anything different in that respect than core distutils. Are other distutils packages installing to /usr/lib? Nothing in the code should be making that choice. It is up to the configuration of Python (whatever prefix that the interpreter was installed to) or overridden by system or user configuration files. I believe that Debian recently changed its configuration to default to /usr/local (for good reason!). Perhaps that is causing what you see. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ckkart at hoc.net Sun Jul 9 21:59:06 2006 From: ckkart at hoc.net (Christian Kristukat) Date: Mon, 10 Jul 2006 01:59:06 +0000 (UTC) Subject: [Numpy-discussion] NumPy beta release plan References: <44AF5BFA.70509@ieee.org> Message-ID: Robert Kern gmail.com> writes: > I believe that Debian recently changed its configuration to default to > /usr/local (for good reason!). Perhaps that is causing what you see. > I just found out that this is feature/bug of opensuse 10.1. I'm beginning to like it but can you tell what you have to do to force it install to /usr/lib? Btw. (OFF TOPIC) have you noticed that the scipy mailing list is just swallowing mails but not spreading them? Regards, Christian From robert.kern at gmail.com Sun Jul 9 22:08:47 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 09 Jul 2006 21:08:47 -0500 Subject: [Numpy-discussion] NumPy beta release plan In-Reply-To: References: <44AF5BFA.70509@ieee.org> Message-ID: Christian Kristukat wrote: > Robert Kern gmail.com> writes: > >> I believe that Debian recently changed its configuration to default to >> /usr/local (for good reason!). Perhaps that is causing what you see. > > I just found out that this is feature/bug of opensuse 10.1. I'm beginning to > like it but can you tell what you have to do to force it install to /usr/lib? It's probably a good idea to leave it as-is. OS distributions like to control all files in /usr (except /usr/local) and leave /usr/local for sysadmins and users. But if you need more control: http://docs.python.org/inst/alt-install-windows.html (ignore the inappropriate filename; it talks about alternate installation locations for all platforms) http://docs.python.org/inst/config-syntax.html > Btw. (OFF TOPIC) have you noticed that the scipy mailing list is just > swallowing mails but not spreading them? No. That could be a problem on our end. Please let Jeff Strunk know about it. If you can provide details like a local copy of the email that got sent and swallowed, that would be very good. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From wbaxter at gmail.com Sun Jul 9 23:30:24 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 10 Jul 2006 12:30:24 +0900 Subject: [Numpy-discussion] Testing a self-compiled numpy Message-ID: I got numpy compiled according the the instruction on the Wiki, but is there some way to try it out without wiping out my stable install of 0.9.8? I tried modifying my PYTHONPATH to put the new numpy build directory first, but 'import numpy' still picks up the installed 0.9.8 version. I also tried running python from the build directory. Then 'import numpy' prints out "Running from numpy source directory", but the numpy thus imported seems not to have any attributes. Suggestions? --bill -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Sun Jul 9 23:55:35 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Sun, 9 Jul 2006 20:55:35 -0700 Subject: [Numpy-discussion] Testing a self-compiled numpy In-Reply-To: References: Message-ID: On 7/9/06, Bill Baxter wrote: > I got numpy compiled according the the instruction on the Wiki, but is there > some way to try it out without wiping out my stable install of 0.9.8? > > I tried modifying my PYTHONPATH to put the new numpy build directory first, > but 'import numpy' still picks up the installed 0.9.8 version. One quick hack is to install the new numpy somewhere else and then rename the directory containing 0.9.8 to numpySTOP. Then you don't have to worry about the ordering of the path. I install in /usr/local/scipy/numpy (then it is easy to remove numpy without worrying about leaving behind any files) and then I create the file /usr/lib/python2.4/site-packages/addlocalpath.pth that contains the lines: /usr/local/scipy/scipy/lib/python2.4/site-packages/ /usr/local/scipy/numpy/lib/python2.4/site-packages/ Then there is no need to play with the path. From wbaxter at gmail.com Mon Jul 10 00:42:03 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 10 Jul 2006 13:42:03 +0900 Subject: [Numpy-discussion] Testing a self-compiled numpy In-Reply-To: References: Message-ID: Thanks for the reply Keith. On 7/10/06, Keith Goodman wrote: > One quick hack is to install the new numpy somewhere else and then > rename the directory containing 0.9.8 to numpySTOP. Then you don't > have to worry about the ordering of the path. The only thing is that the numpy installer built by the windows instructions on the Wiki doesn't give you an option of where to install numpy. It installs straight to {PYTHON_DIR}\lib\site-packages\numpy. (The command suggested by the wiki is: c:\path\to\python.exe setup.py config --compiler=mingw32 build --compiler=mingw32 bdist_wininst ) But yeh, in a similar vein I can just rename the current numpy directory to numpy_98 or something like that before installing, and that seems to work. I just thought there would be a more sophisticated way to test various verisons of modules in Python than renaming directories. I install in /usr/local/scipy/numpy (then it is easy to remove numpy > without worrying about leaving behind any files) and then I create the > file /usr/lib/python2.4/site-packages/addlocalpath.pth that contains > the lines: > > /usr/local/scipy/scipy/lib/python2.4/site-packages/ > /usr/local/scipy/numpy/lib/python2.4/site-packages/ > > Then there is no need to play with the path. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Jul 10 01:21:30 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 9 Jul 2006 23:21:30 -0600 Subject: [Numpy-discussion] NumPy beta release plan In-Reply-To: References: <44AF5BFA.70509@ieee.org> Message-ID: Hi, On 7/9/06, Robert Kern wrote: > > Christian Kristukat wrote: > > Robert Kern gmail.com> writes: > > > >> I believe that Debian recently changed its configuration to default to > >> /usr/local (for good reason!). Perhaps that is causing what you see. > > > > I just found out that this is feature/bug of opensuse 10.1. I'm > beginning to > > like it but can you tell what you have to do to force it install to > /usr/lib? Suse does that, but so does python by default if you compile it from scratch; Suse is just following the path of least resistence. I don't like it much, I think distributions should leave /usr/local empty, but when you compile your own stuff it should arguably end up in /usr/local by default. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Mon Jul 10 01:41:38 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Sun, 9 Jul 2006 22:41:38 -0700 Subject: [Numpy-discussion] Testing a self-compiled numpy In-Reply-To: References: Message-ID: On 7/9/06, Bill Baxter wrote: > Thanks for the reply Keith. > > > On 7/10/06, Keith Goodman wrote: > > One quick hack is to install the new numpy somewhere else and then > > rename the directory containing 0.9.8 to numpySTOP. Then you don't > > have to worry about the ordering of the path. > > > The only thing is that the numpy installer built by the windows > instructions on the Wiki doesn't give you an option of where to install > numpy. It installs straight to > {PYTHON_DIR}\lib\site-packages\numpy. (The command > suggested by the wiki is: > c:\path\to\python.exe setup.py config --compiler=mingw32 build > --compiler=mingw32 bdist_wininst > ) Doesn't --prefix=/install/numpy/here/ work on windows? From arnd.baecker at web.de Mon Jul 10 01:59:14 2006 From: arnd.baecker at web.de (Arnd Baecker) Date: Mon, 10 Jul 2006 07:59:14 +0200 (CEST) Subject: [Numpy-discussion] logspace behaviour/documentation In-Reply-To: References: Message-ID: Hi Alan, sorry, for answering so late - your message slipped through ... On Fri, 30 Jun 2006, Alan Isaac wrote: > On Fri, 30 Jun 2006, T) Arnd Baecker wrote: > > I am wondering a bit about the the behaviour of logspace: > > http://www.mathworks.com/access/helpdesk/help/techdoc/ref/logspace.html > > fwiw, Alright, if one wants to keep compatibility with matlab, the behaviour of logspace should be unchanged. I'd suggest that something along def logspace(start,stop,num=50,endpoint=True,base=10.0): """Evenly spaced numbers on a logarithmic scale. Return 'int(num)' evenly spaced samples on a logarithmic scale from 'base'**'start' to 'base'**'stop'. If 'endpoint' is True, the last sample is 'base'**'stop'.""" is clearer than the present one Docstring: Evenly spaced numbers on a logarithmic scale. Computes int(num) evenly spaced exponents from start to stop. If endpoint=True, then last exponent is stop. Returns base**exponents. (Note that I am not a native speaker - surprise - ; so maybe the present one is already clear enough?) Best, Arnd From ckkart at hoc.net Mon Jul 10 05:18:53 2006 From: ckkart at hoc.net (Christian Kristukat) Date: Mon, 10 Jul 2006 09:18:53 +0000 (UTC) Subject: [Numpy-discussion] NumPy beta release plan References: <44AF5BFA.70509@ieee.org> Message-ID: Charles R Harris gmail.com> writes: > > > Hi, > > On 7/9/06, Robert Kern gmail.com> wrote: > Christian Kristukat wrote:> Robert Kern gmail.com> writes:>>> I believe that Debian recently changed its configuration to default to > >> /usr/local (for good reason!). Perhaps that is causing what you see.>> I just found out that this is feature/bug of opensuse 10.1. I'm beginning to> like it but can you tell what you have to do to force it install to /usr/lib? > > > Suse does that, but so does python by default if you compile it from scratch; I think every autoconf managed source package out there will install itself to /usr/local if you don't specify the --prefix. So this is not a python thing. > Suse is just following the path of least resistence. I don't like it much, I > think distributions should leave /usr/local empty, but when you compile your > own stuff it should arguably end up in /usr/local by default. The python modules shipped with suse are installed where python resides and that is in /usr. But beginning with suse 10.1 user installed packages go to /usr/local. What I don't like about that method so much is, that when you build a rpm with distutils it gets installed in /usr/local/, too, even though the system does handle third party rpms in the systems rpm database. Christian From nwagner at iam.uni-stuttgart.de Mon Jul 10 05:26:57 2006 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Mon, 10 Jul 2006 11:26:57 +0200 Subject: [Numpy-discussion] Converting a list Message-ID: <44B21D61.70304@iam.uni-stuttgart.de> Hi all, I have a list consisting of arrays of different size data = [array([-1. +0.j, -1.33333333+0.j, -1.66666667+0.j]), array([-2.+0.j , -2.-0.66666667j, -2.-1.33333333j]), array([-2. -2.j, -1.33333333-2.j, -0.66666667-2.j]), array([ 0.-2.j , 0.-1.66666667j, 0.-1.33333333j]), array([ 6.12323400e-17-1.j , -2.58819045e-01-0.96592583j, -5.00000000e-01-0.8660254j , -7.07106781e-01-0.70710678j, -8.66025404e-01-0.5j , -9.65925826e-01-0.25881905j])] type(data) = shape(data) results in shape(data) = Traceback (most recent call last): File "sinai.py", line 107, in ? p = polygon(P) File "sinai.py", line 67, in polygon print 'shape(data) = ',shape(data) File "/usr/lib64/python2.4/site-packages/numpy/core/fromnumeric.py", line 258, in shape result = asarray(a).shape File "/usr/lib64/python2.4/site-packages/numpy/core/numeric.py", line 119, in asarray return array(a, dtype, copy=False, order=order) TypeError: a float is required Is this a bug ? Nils From robert.kern at gmail.com Mon Jul 10 05:57:51 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 10 Jul 2006 04:57:51 -0500 Subject: [Numpy-discussion] Converting a list In-Reply-To: <44B21D61.70304@iam.uni-stuttgart.de> References: <44B21D61.70304@iam.uni-stuttgart.de> Message-ID: Nils Wagner wrote: > Hi all, > > I have a list consisting of arrays of different size > > data = [array([-1. +0.j, -1.33333333+0.j, -1.66666667+0.j]), > array([-2.+0.j , -2.-0.66666667j, -2.-1.33333333j]), > array([-2. -2.j, -1.33333333-2.j, -0.66666667-2.j]), array([ > 0.-2.j , 0.-1.66666667j, 0.-1.33333333j]), array([ > 6.12323400e-17-1.j , -2.58819045e-01-0.96592583j, > -5.00000000e-01-0.8660254j , -7.07106781e-01-0.70710678j, > -8.66025404e-01-0.5j , -9.65925826e-01-0.25881905j])] > > type(data) = > > shape(data) results in > > shape(data) = > Traceback (most recent call last): > File "sinai.py", line 107, in ? > p = polygon(P) > File "sinai.py", line 67, in polygon > print 'shape(data) = ',shape(data) > File "/usr/lib64/python2.4/site-packages/numpy/core/fromnumeric.py", > line 258, in shape > result = asarray(a).shape > File "/usr/lib64/python2.4/site-packages/numpy/core/numeric.py", line > 119, in asarray > return array(a, dtype, copy=False, order=order) > TypeError: a float is required > > Is this a bug ? The error message is unhelpful, certainly, but the *fact* that an exception is raised is not a bug. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From nwagner at iam.uni-stuttgart.de Mon Jul 10 08:33:52 2006 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Mon, 10 Jul 2006 14:33:52 +0200 Subject: [Numpy-discussion] Number of digits Message-ID: <44B24930.1080603@iam.uni-stuttgart.de> Hi all, how can I increase the number of digits in the output of str(.) ? >>> lam**2 493.48022005446808 >>> str(lam**2) '493.480220054' '493.4802200544680' would be fine. Nils From david.douard at logilab.fr Mon Jul 10 09:09:07 2006 From: david.douard at logilab.fr (David Douard) Date: Mon, 10 Jul 2006 15:09:07 +0200 Subject: [Numpy-discussion] Number of digits In-Reply-To: <44B24930.1080603@iam.uni-stuttgart.de> References: <44B24930.1080603@iam.uni-stuttgart.de> Message-ID: <20060710130906.GA16160@crater.logilab.fr> On Mon, Jul 10, 2006 at 02:33:52PM +0200, Nils Wagner wrote: > Hi all, > > how can I increase the number of digits in the output of str(.) ? > > >>> lam**2 > 493.48022005446808 > >>> str(lam**2) > '493.480220054' > > '493.4802200544680' would be fine. Maybe something like >>> '%.13f'%(lam**2) David -- David Douard LOGILAB, Paris (France) Formations Python, Zope, Plone, Debian : http://www.logilab.fr/formations D?veloppement logiciel sur mesure : http://www.logilab.fr/services Informatique scientifique : http://www.logilab.fr/science -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From tim.hochberg at cox.net Mon Jul 10 09:30:06 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon, 10 Jul 2006 06:30:06 -0700 Subject: [Numpy-discussion] Number of digits In-Reply-To: <44B24930.1080603@iam.uni-stuttgart.de> References: <44B24930.1080603@iam.uni-stuttgart.de> Message-ID: <44B2565E.8010208@cox.net> Nils Wagner wrote: > Hi all, > > how can I increase the number of digits in the output of str(.) ? > You can't as far as I know. For floats, you can use "%.nf". For example: "%.13f" % 493.4802200544680 -tim > >>>> lam**2 >>>> > 493.48022005446808 > >>>> str(lam**2) >>>> > '493.480220054' > > '493.4802200544680' would be fine. > > Nils > > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From robert.kern at gmail.com Mon Jul 10 09:46:33 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 10 Jul 2006 08:46:33 -0500 Subject: [Numpy-discussion] Number of digits In-Reply-To: <44B2565E.8010208@cox.net> References: <44B24930.1080603@iam.uni-stuttgart.de> <44B2565E.8010208@cox.net> Message-ID: Tim Hochberg wrote: > Nils Wagner wrote: >> Hi all, >> >> how can I increase the number of digits in the output of str(.) ? >> > You can't as far as I know. For floats, you can use "%.nf". For example: > > "%.13f" % 493.4802200544680 The problem is is that he doesn't have a float. He has one of our float64scalar objects. The str() of a real Python float will give as many digits as are necessary to recreate number and no more (or maybe one or two more). A str() of a float64scalar will round according to some rule that I haven't figured out, yet. It doesn't seem to be configurable with numpy.set_printoptions(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david.douard at logilab.fr Mon Jul 10 10:08:23 2006 From: david.douard at logilab.fr (David Douard) Date: Mon, 10 Jul 2006 16:08:23 +0200 Subject: [Numpy-discussion] Number of digits In-Reply-To: References: <44B24930.1080603@iam.uni-stuttgart.de> <44B2565E.8010208@cox.net> Message-ID: <20060710140823.GB16160@crater.logilab.fr> On Mon, Jul 10, 2006 at 08:46:33AM -0500, Robert Kern wrote: > Tim Hochberg wrote: > > Nils Wagner wrote: > >> Hi all, > >> > >> how can I increase the number of digits in the output of str(.) ? > >> > > You can't as far as I know. For floats, you can use "%.nf". For example: > > > > "%.13f" % 493.4802200544680 > > The problem is is that he doesn't have a float. He has one of our float64scalar > objects. The str() of a real Python float will give as many digits as are > necessary to recreate number and no more (or maybe one or two more). A str() of > a float64scalar will round according to some rule that I haven't figured out, > yet. It doesn't seem to be configurable with numpy.set_printoptions(). This is a different ptoblem from the one exposed by Nils. I mean, tha fact that str() on numpy.float64 objects is somewhat obscure is a problem that should obviously be addressed some day. However, as far as I understand Nils' message, the "%.13f" trick is enough for what he need. But I may be wrong... He just want to "increase the number of digits", not have the "optimal" number of digits (as long as this is meaningfull). But I may have missed something. David -- David Douard LOGILAB, Paris (France) Formations Python, Zope, Plone, Debian : http://www.logilab.fr/formations D?veloppement logiciel sur mesure : http://www.logilab.fr/services Informatique scientifique : http://www.logilab.fr/science -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From robert.kern at gmail.com Mon Jul 10 10:06:47 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 10 Jul 2006 09:06:47 -0500 Subject: [Numpy-discussion] Number of digits In-Reply-To: References: <44B24930.1080603@iam.uni-stuttgart.de> <44B2565E.8010208@cox.net> Message-ID: Robert Kern wrote: > Tim Hochberg wrote: >> Nils Wagner wrote: >>> Hi all, >>> >>> how can I increase the number of digits in the output of str(.) ? >>> >> You can't as far as I know. For floats, you can use "%.nf". For example: >> >> "%.13f" % 493.4802200544680 > > The problem is is that he doesn't have a float. He has one of our float64scalar > objects. The str() of a real Python float will give as many digits as are > necessary to recreate number and no more (or maybe one or two more). A str() of > a float64scalar will round according to some rule that I haven't figured out, > yet. It doesn't seem to be configurable with numpy.set_printoptions(). No, that's entirely wrong. Please ignore the ravings of a madman. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From bhendrix at enthought.com Mon Jul 10 10:16:46 2006 From: bhendrix at enthought.com (Bryce Hendrix) Date: Mon, 10 Jul 2006 09:16:46 -0500 Subject: [Numpy-discussion] Testing a self-compiled numpy In-Reply-To: References: Message-ID: <44B2614E.7060101@enthought.com> Keith Goodman wrote: >> The only thing is that the numpy installer built by the windows >> instructions on the Wiki doesn't give you an option of where to install >> numpy. It installs straight to >> {PYTHON_DIR}\lib\site-packages\numpy. (The command >> suggested by the wiki is: >> c:\path\to\python.exe setup.py config --compiler=mingw32 build >> --compiler=mingw32 bdist_wininst >> ) >> > > Doesn't --prefix=/install/numpy/here/ work on windows? > For our Windows Enthon project, the build command is setup.py config --compiler=mingw32 build --compiler=mingw32 install --prefix=path_to_where_numpy_goes. Bryce -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Jul 10 10:15:38 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 10 Jul 2006 09:15:38 -0500 Subject: [Numpy-discussion] Number of digits In-Reply-To: <20060710140823.GB16160@crater.logilab.fr> References: <44B24930.1080603@iam.uni-stuttgart.de> <44B2565E.8010208@cox.net> <20060710140823.GB16160@crater.logilab.fr> Message-ID: David Douard wrote: > On Mon, Jul 10, 2006 at 08:46:33AM -0500, Robert Kern wrote: >> Tim Hochberg wrote: >>> Nils Wagner wrote: >>>> Hi all, >>>> >>>> how can I increase the number of digits in the output of str(.) ? >>>> >>> You can't as far as I know. For floats, you can use "%.nf". For example: >>> >>> "%.13f" % 493.4802200544680 >> The problem is is that he doesn't have a float. He has one of our float64scalar >> objects. The str() of a real Python float will give as many digits as are >> necessary to recreate number and no more (or maybe one or two more). A str() of >> a float64scalar will round according to some rule that I haven't figured out, >> yet. It doesn't seem to be configurable with numpy.set_printoptions(). > > This is a different ptoblem from the one exposed by Nils. I mean, tha > fact that str() on numpy.float64 objects is somewhat obscure is a > problem that should obviously be addressed some day. However, as far as > I understand Nils' message, the "%.13f" trick is enough for what he > need. But I may be wrong... He just want to "increase the number of > digits", not have the "optimal" number of digits (as long as this is > meaningfull). > > But I may have missed something. No, you didn't miss anything. I just haven't gotten enough sleep. I thought that Python floats had a __str__ that computed just as many places as necessary, but it looks like it just rounds at 12 places. This is not an adjustable parameter (barring modifying the C code of the interpreter). I also thought that the float scalars were being printed differently. However, the code for the float64scalar __str__ just gets the double value, creates a Python float object from it, and returns the __str__ result from that object. Everything works as intended. Nothing to see here. Move along. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From tom.denniston at alum.dartmouth.org Mon Jul 10 11:42:50 2006 From: tom.denniston at alum.dartmouth.org (Tom Denniston) Date: Mon, 10 Jul 2006 10:42:50 -0500 Subject: [Numpy-discussion] PyObject arrays, pickling, tofile In-Reply-To: <44AC80AC.1030708@ieee.org> References: <44AC80AC.1030708@ieee.org> Message-ID: Travis, Thanks for your help. Sorry for taking so long to get back to you. I wasn't on email for a few days. I seem to be able to pickle and restore PyObject arrays but then concatenates and takes give me the following error: TypeError: fields with object members not yet supported. What is even stranger is my stack trace stops at the call to numpy in my code and doesn't show where it is failing in the numpy code. The same operation on the numpy array that hasn't been pickled and loaded back, works fine. --Tom From parejkoj at drexel.edu Mon Jul 10 12:46:45 2006 From: parejkoj at drexel.edu (John Parejko) Date: Mon, 10 Jul 2006 12:46:45 -0400 Subject: [Numpy-discussion] recarray field names Message-ID: <44B28475.9010602@drexel.edu> Howdy! I just wanted to voice my agreement with this statment by Erin Sheldon: > I brought up the issue a while back of having a simple way to > access the field names of an array. The quick summary: accessing > field names has some oddness that needs cleaning up. Sometimes a['name'], sometimes a.field('name'). I vastly prefer the first version; it has become my favored way of dealing with my FITS and ASCII data. But using the bracket notation sometimes gives an error, and I haven't quite figured out what the circumstances are. Also, "for name in a.dtype.fields:" or "for name in a.fields:" would be very, very handy. Could this sort of thing get put in before the beta (or is it already in, and I missed the boat, again?). Thanks John -- ************************* John Parejko Department of Physics and Astronomy 215 895-2786 Drexel University Philadelphia, PA ************************** From pgmdevlist at mailcan.com Mon Jul 10 13:46:07 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Mon, 10 Jul 2006 13:46:07 -0400 Subject: [Numpy-discussion] Converting a list In-Reply-To: References: <44B21D61.70304@iam.uni-stuttgart.de> Message-ID: <200607101346.07555.pgmdevlist@mailcan.com> On Monday 10 July 2006 05:57, Robert Kern wrote: > Nils Wagner wrote: > > Hi all, > > > > I have a list consisting of arrays of different size ... > > The error message is unhelpful, certainly, but the *fact* that an exception > is raised is not a bug. Attempting to build an array from a list of sequences with different sizes seem to be a rather common mistake (judging from the recurrence of the topic on that list). As Robert pointed, the current error message is not helpful. Could it be possible to change it ? A "(inconsistent sizes ?)" could at least be added... From rowen at cesmail.net Mon Jul 10 14:17:02 2006 From: rowen at cesmail.net (Russell E. Owen) Date: Mon, 10 Jul 2006 11:17:02 -0700 Subject: [Numpy-discussion] NumPy beta release plan References: <44AF5BFA.70509@ieee.org> Message-ID: In article <44AF5BFA.70509 at ieee.org>, Travis Oliphant wrote: > 3) C-API names have prefix PyArray_ (like always), NPY_ or npy_. > > The NPY_ and npy_ prefixes are new and were done to remove the > likelihood of name collisions when NumPy is used with another > library. The old (and un-prefixed) names are accessible by > importing numpy/noprefix.h instead of numpy/arrayobject.h This may be too late or too picky, but... This seems like too many choices, leading to possible confusion when reading other people's extensions or the numpy code itself. Could you standardize on one (I like npy_) or two (if you have to keep PyArray_ for some reason and can't put it in a backwards-compatibility header somewhere). Overall it's great. I'm really glad to hear you're so close to freezing the features. -- Russell From kwgoodman at gmail.com Mon Jul 10 14:30:09 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 10 Jul 2006 11:30:09 -0700 Subject: [Numpy-discussion] .M .A .T .H attribute result In-Reply-To: <44AEC258.9040800@ee.byu.edu> References: <44AEC258.9040800@ee.byu.edu> Message-ID: On 7/7/06, Travis Oliphant wrote: > a numpy.matlib module was started to store matrix versions of the > standard array-creation functions and mat was re-labeled to "asmatrix" > so that a copy is not made by default. Holy crap! It works. This is great. Thank you. >> import numpy.matlib >> numpy.__version__ '0.9.9.2788' >> from numpy.matlib import * >> rand(2,2) matrix([[ 0.23834437, 0.60329722], <--- matrix by default [ 0.03907811, 0.55134035]]) >> ones((2,2)) matrix([[ 1., 1.], <--- matrix by default [ 1., 1.]]) >> numpy.matlib. numpy.matlib.N numpy.matlib.__name__ numpy.matlib.eye numpy.matlib.__all__ numpy.matlib.__new__ numpy.matlib.identity numpy.matlib.__class__ numpy.matlib.__reduce__ numpy.matlib.matrix numpy.matlib.__delattr__ numpy.matlib.__reduce_ex__ numpy.matlib.ndarray numpy.matlib.__dict__ numpy.matlib.__repr__ numpy.matlib.ones numpy.matlib.__doc__ numpy.matlib.__setattr__ numpy.matlib.rand numpy.matlib.__file__ numpy.matlib.__str__ numpy.matlib.randn numpy.matlib.__getattribute__ numpy.matlib.array numpy.matlib.zeros numpy.matlib.__hash__ numpy.matlib.asmatrix numpy.matlib.__init__ numpy.matlib.empty From Chris.Barker at noaa.gov Mon Jul 10 14:30:59 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 10 Jul 2006 11:30:59 -0700 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> Message-ID: <44B29CE3.9010504@noaa.gov> Ed Schofield wrote: > * Should numpy.rand and numpy.randn accept sequences of dimensions as > arguments, like rand((3,3)), as an alternative to rand(3,3)? +1 +1 for ONLY allowing tuples. Or just get rid of it. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From robert.kern at gmail.com Mon Jul 10 14:41:04 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 10 Jul 2006 13:41:04 -0500 Subject: [Numpy-discussion] Converting a list In-Reply-To: <200607101346.07555.pgmdevlist@mailcan.com> References: <44B21D61.70304@iam.uni-stuttgart.de> <200607101346.07555.pgmdevlist@mailcan.com> Message-ID: Pierre GM wrote: > On Monday 10 July 2006 05:57, Robert Kern wrote: >> Nils Wagner wrote: >>> Hi all, >>> >>> I have a list consisting of arrays of different size > ... >> The error message is unhelpful, certainly, but the *fact* that an exception >> is raised is not a bug. > > Attempting to build an array from a list of sequences with different sizes > seem to be a rather common mistake (judging from the recurrence of the topic > on that list). > As Robert pointed, the current error message is not helpful. > Could it be possible to change it ? A "(inconsistent sizes ?)" could at least > be added... People have looked at this before. IIRC, the conclusion was that at the specific place where the exception needs to be raised, the information that it came from a "ragged" input is lost, and that exception can be raised in other circumstances as well (so it can't just be reworded). The number of cases that array() tries to handle automatically is quite large, and that makes for hairy code. OTOH, if you can pinpoint a place where we can raise a more specific exception without changing the semantics of the function, I'd be happy to apply your patch. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at ee.byu.edu Mon Jul 10 14:54:43 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 10 Jul 2006 12:54:43 -0600 Subject: [Numpy-discussion] NumPy beta release plan In-Reply-To: References: <44AF5BFA.70509@ieee.org> Message-ID: <44B2A273.4050502@ee.byu.edu> Christian Kristukat wrote: >Travis Oliphant ieee.org> writes: > > > >>Some of you may have noticed that things have been changing rapidly in >>the NumPy world (if you were out of the office in June then all the >>activity may seem overwhelming). >> >>All of this activity is based on the fact that the upcoming beta release >>will mean a feature freeze for NumPy. >> >> > >Recently numpy.distutils switched to install to /usr/local/lib instead of >/usr/lib. Is that intended? I'd prefer to have it back at /usr/lib in >1.0beta. > > What gets installed in /usr/lib ? I'm pretty sure the install directory is determined by the Python installation. Exceptions to this are probably bugs. -Travis From oliphant at ee.byu.edu Mon Jul 10 15:04:07 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 10 Jul 2006 13:04:07 -0600 Subject: [Numpy-discussion] recarray field names In-Reply-To: <44B28475.9010602@drexel.edu> References: <44B28475.9010602@drexel.edu> Message-ID: <44B2A4A7.6050405@ee.byu.edu> John Parejko wrote: >Howdy! I just wanted to voice my agreement with this statment by Erin Sheldon: > > > I brought up the issue a while back of having a simple way to > > access the field names of an array. The quick summary: accessing > > field names has some oddness that needs cleaning up. > >Sometimes a['name'], sometimes a.field('name'). I vastly prefer the first >version; it has become my favored way of dealing with my FITS and ASCII >data. But using the bracket notation sometimes gives an error, and I >haven't quite figured out what the circumstances are. > >Also, "for name in a.dtype.fields:" or "for name in a.fields:" would be >very, very handy. > >Could this sort of thing get put in before the beta (or is it already in, >and I missed the boat, again?). > > > It's actually there already. The issue at hand is that the fields dictionary contains an additional member keyed by a -1. This odd-ball entry in the dictionary is to obtain an "ordered" list of the fields. The order is determined by the offset. This is a warty situation but the functionality is there. Probably a better solution is to add a names attribute to the dtype object that returns the ordered list. In C, the PyArray_Descr structure would grow an additional names member that contains the ordered list of names instead of sticking it as a -1 key entry in the fields dictionary which was and is a hack. Let's schedule this for pre 1.0 beta -Travis From oliphant at ee.byu.edu Mon Jul 10 15:06:10 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 10 Jul 2006 13:06:10 -0600 Subject: [Numpy-discussion] NumPy beta release plan In-Reply-To: References: <44AF5BFA.70509@ieee.org> Message-ID: <44B2A522.4000501@ee.byu.edu> Russell E. Owen wrote: >In article <44AF5BFA.70509 at ieee.org>, > Travis Oliphant wrote: > > > >>3) C-API names have prefix PyArray_ (like always), NPY_ or npy_. >> >> The NPY_ and npy_ prefixes are new and were done to remove the >> likelihood of name collisions when NumPy is used with another >> library. The old (and un-prefixed) names are accessible by >> importing numpy/noprefix.h instead of numpy/arrayobject.h >> >> > >This may be too late or too picky, but... > >This seems like too many choices, leading to possible confusion when >reading other people's extensions or the numpy code itself. Could you >standardize on one (I like npy_) or two (if you have to keep PyArray_ >for some reason and can't put it in a backwards-compatibility header >somewhere). > > > There is a logic behind it though. The C-API function calls all have PyArray_ prefixes (as always in Numeric) The macro captialized constants have NPY_ prefixes Newly defined types with lower-case names have npy_ prefixes -Travis From ndarray at mac.com Mon Jul 10 16:10:56 2006 From: ndarray at mac.com (Sasha) Date: Mon, 10 Jul 2006 16:10:56 -0400 Subject: [Numpy-discussion] NumPy beta release plan In-Reply-To: References: <44AF5BFA.70509@ieee.org> Message-ID: I would like to add my voice to a call for single prefix (I like PyArray_ for everything, but can live with npy_). Presumably npy_ vs. NPY_ variation is supposed to distinguish between macros and C language elements, but as long as the stem is properly capitalized, I don't see a problem of using the same prefix for everything. In any case, if more than one prefix is used, it would be helpful to get some guidance on what to use when. On 7/10/06, Russell E. Owen wrote: > In article <44AF5BFA.70509 at ieee.org>, > Travis Oliphant wrote: > > > 3) C-API names have prefix PyArray_ (like always), NPY_ or npy_. > > > > The NPY_ and npy_ prefixes are new and were done to remove the > > likelihood of name collisions when NumPy is used with another > > library. The old (and un-prefixed) names are accessible by > > importing numpy/noprefix.h instead of numpy/arrayobject.h > > This may be too late or too picky, but... > > This seems like too many choices, leading to possible confusion when > reading other people's extensions or the numpy code itself. Could you > standardize on one (I like npy_) or two (if you have to keep PyArray_ > for some reason and can't put it in a backwards-compatibility header > somewhere). > > Overall it's great. I'm really glad to hear you're so close to freezing > the features. > > -- Russell > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From oliphant at ee.byu.edu Mon Jul 10 19:09:48 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 10 Jul 2006 17:09:48 -0600 Subject: [Numpy-discussion] recarray field names In-Reply-To: <44B28475.9010602@drexel.edu> References: <44B28475.9010602@drexel.edu> Message-ID: <44B2DE3C.7060801@ee.byu.edu> John Parejko wrote: >Howdy! I just wanted to voice my agreement with this statment by Erin Sheldon: > > > I brought up the issue a while back of having a simple way to > > access the field names of an array. The quick summary: accessing > > field names has some oddness that needs cleaning up. > >Sometimes a['name'], sometimes a.field('name'). I vastly prefer the first >version; it has become my favored way of dealing with my FITS and ASCII >data. But using the bracket notation sometimes gives an error, and I >haven't quite figured out what the circumstances are. > > Bracketed version should never give an error. If you see it, it's a bug. >Also, "for name in a.dtype.fields:" or "for name in a.fields:" would be >very, very handy. > > You can do this, now but each key in the fields dictionary might not be unique because titles will also be keys in the dictionary. A unique list of ordered names can now (in NumPy SVN) be obtained using for name in a.dtype.names -Travis From oliphant at ee.byu.edu Mon Jul 10 19:32:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 10 Jul 2006 17:32:04 -0600 Subject: [Numpy-discussion] NumPy beta release plan In-Reply-To: References: <44AF5BFA.70509@ieee.org> Message-ID: <44B2E374.60607@ee.byu.edu> Sasha wrote: >I would like to add my voice to a call for single prefix (I like >PyArray_ for everything, but can live with npy_). Presumably npy_ >vs. NPY_ variation is supposed to distinguish between macros and C >language elements, but as long as the stem is properly capitalized, I >don't see a problem of using the same prefix for everything. > >In any case, if more than one prefix is used, it would be helpful to >get some guidance on what to use when. > > PyArray_ --- Functions and/or macro-functions NPY_ --- Constants or enums npy_ --- typedef'd types (npy_int16, npy_float32, npy_intp) -Travis From tom.denniston at alum.dartmouth.org Mon Jul 10 20:59:25 2006 From: tom.denniston at alum.dartmouth.org (Tom Denniston) Date: Mon, 10 Jul 2006 19:59:25 -0500 Subject: [Numpy-discussion] Strange error taking log of PyObject array Message-ID: The following works on a float array but not an object array. It gives a very strange error message. (Pdb) numpy.log(numpy.array([19155613843.7], dtype=object)) *** AttributeError: 'float' object has no attribute 'log' (Pdb) numpy.log([19155613843.7]) array([ 23.67586166]) From oliphant.travis at ieee.org Mon Jul 10 21:46:21 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 10 Jul 2006 19:46:21 -0600 Subject: [Numpy-discussion] Strange error taking log of PyObject array In-Reply-To: References: Message-ID: <44B302ED.1080104@ieee.org> Tom Denniston wrote: > The following works on a float array but not an object array. It > gives a very strange error message. > > (Pdb) numpy.log(numpy.array([19155613843.7], dtype=object)) > *** AttributeError: 'float' object has no attribute 'log' > This is expected behavior. For object arrays the ufuncs look to the objects themselves for implementation. In this case. The ufunc tries to call the log method of each entry in the array. The first entry is a float (which as the error shows) does not have a log attribute and so you get a failure. -Travis From swwkofbr at ec.hokudai.ac.jp Tue Jul 11 03:32:53 2006 From: swwkofbr at ec.hokudai.ac.jp (lighting displays) Date: Tue, 11 Jul 2006 05:32:53 -0200 Subject: [Numpy-discussion] rotten foul. spotted teensy Message-ID: <800C2E05027B674.1345572@ec.hokudai.ac.jp> G0ldmark Industries, Inc ( G D K I ) THIS ST()CK IS E> Just tested the lastest SVN and it works as advertised. Thanks Travis. An unrelated question: why does this work for arrays but not recarrays? In [24]: mydescriptor = [('age',float64),('Nchildren',int8),('weight',float32)] In [25]: a = array([(64,2,75.0),(25,0,60.0)], dtype=mydescriptor) In [26]: a = recarray([(64,2,75.0),(25,0,60.0)], dtype=mydescriptor) --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /Users/esheldon/ TypeError: __new__() got an unexpected keyword argument 'dtype' I understand that I could use the formats and names keywords, but this seems a little inconsistent. Erin On 7/10/06, Travis Oliphant wrote: > John Parejko wrote: > > >Howdy! I just wanted to voice my agreement with this statment by Erin > >Sheldon: > > > > > I brought up the issue a while back of having a simple way to > > > access the field names of an array. The quick summary: accessing > > > field names has some oddness that needs cleaning up. > > > >Sometimes a['name'], sometimes a.field('name'). I vastly prefer the first > >version; it has become my favored way of dealing with my FITS and ASCII > >data. But using the bracket notation sometimes gives an error, and I > >haven't quite figured out what the circumstances are. > > > > > > Bracketed version should never give an error. If you see it, it's a bug. > > >Also, "for name in a.dtype.fields:" or "for name in a.fields:" would be > >very, very handy. > > > > > You can do this, now but each key in the fields dictionary might not be > unique because titles will also be keys in the dictionary. > > A unique list of ordered names can now (in NumPy SVN) be obtained using > > for name in a.dtype.names > > > -Travis > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From oliphant.travis at ieee.org Tue Jul 11 00:19:44 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 10 Jul 2006 22:19:44 -0600 Subject: [Numpy-discussion] recarray field names In-Reply-To: <20060710225459.A24557@cfcp.uchicago.edu> References: <20060710225459.A24557@cfcp.uchicago.edu> Message-ID: <44B326E0.6020406@ieee.org> Erin Sheldon wrote: > Just tested the lastest SVN and it works as advertised. Thanks > Travis. > > An unrelated question: why does this work for arrays but not recarrays? > > > In [24]: mydescriptor = [('age',float64),('Nchildren',int8),('weight',float32)] > > In [25]: a = array([(64,2,75.0),(25,0,60.0)], dtype=mydescriptor) > > In [26]: a = recarray([(64,2,75.0),(25,0,60.0)], dtype=mydescriptor) > --------------------------------------------------------------------------- > exceptions.TypeError Traceback (most recent > call last) > > /Users/esheldon/ > > TypeError: __new__() got an unexpected keyword argument 'dtype' > > I understand that I could use the formats and names keywords, but > this seems a little inconsistent. > > Well there are a couple of reasons this doesn't work. 1) the recarray constructor is similar to the ndarray constructor. Neither of these take list arguments as inputs. So, while I've added dtype as the keyword for the formats input of the recarray constructor, the second example will still fail. 2) Even were you to use the rec.array function (which is analagous to the numpy.array function) to produce the error, it does not take a dtype keyword. This is inconsistent. But, the interface for the record array class came from numarray. I tried to stay compatible with those functions so the numarray people would have an easier time adapting. Perhaps we should add a dtype keyword argument to all the constructors in numpy/core/records.py which over-rides the formats, names, and titles keywords so that you could do a = rec.array([...], dtype=mine) analgously to a = array([...], dtype=mine) -Travis From nadavh at visionsense.com Tue Jul 11 01:33:41 2006 From: nadavh at visionsense.com (Nadav Horesh) Date: Tue, 11 Jul 2006 07:33:41 +0200 Subject: [Numpy-discussion] Converting a list Message-ID: <07C6A61102C94148B8104D42DE95F7E8C8EFF4@exchange2k.envision.co.il> Do you mean: >> map(shape, data) -----Original Message----- From: numpy-discussion-bounces at lists.sourceforge.net on behalf of Nils Wagner Sent: Mon 10-Jul-06 12:26 To: numpy-discussion at lists.sourceforge.net Cc: Subject: [Numpy-discussion] Converting a list Hi all, I have a list consisting of arrays of different size data = [array([-1. +0.j, -1.33333333+0.j, -1.66666667+0.j]), array([-2.+0.j , -2.-0.66666667j, -2.-1.33333333j]), array([-2. -2.j, -1.33333333-2.j, -0.66666667-2.j]), array([ 0.-2.j , 0.-1.66666667j, 0.-1.33333333j]), array([ 6.12323400e-17-1.j , -2.58819045e-01-0.96592583j, -5.00000000e-01-0.8660254j , -7.07106781e-01-0.70710678j, -8.66025404e-01-0.5j , -9.65925826e-01-0.25881905j])] type(data) = shape(data) results in shape(data) = Traceback (most recent call last): File "sinai.py", line 107, in ? p = polygon(P) File "sinai.py", line 67, in polygon print 'shape(data) = ',shape(data) File "/usr/lib64/python2.4/site-packages/numpy/core/fromnumeric.py", line 258, in shape result = asarray(a).shape File "/usr/lib64/python2.4/site-packages/numpy/core/numeric.py", line 119, in asarray return array(a, dtype, copy=False, order=order) TypeError: a float is required Is this a bug ? Nils ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From tim.hochberg at cox.net Tue Jul 11 00:35:35 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon, 10 Jul 2006 21:35:35 -0700 Subject: [Numpy-discussion] Strange error taking log of PyObject array In-Reply-To: <44B302ED.1080104@ieee.org> References: <44B302ED.1080104@ieee.org> Message-ID: <44B32A97.4070003@cox.net> Travis Oliphant wrote: > Tom Denniston wrote: > >> The following works on a float array but not an object array. It >> gives a very strange error message. >> >> (Pdb) numpy.log(numpy.array([19155613843.7], dtype=object)) >> *** AttributeError: 'float' object has no attribute 'log' >> >> > This is expected behavior. For object arrays the ufuncs look to the > objects themselves for implementation. In this case. The ufunc tries > to call the log method of each entry in the array. The first entry is a > float (which as the error shows) does not have a log attribute and so > you get a failure. > It seems like we could do better than this. Couldn't the ufunc call *itself* on on objects in an object array? That is, ufunc(object_array) would be more or less equivalent to map(ufunc, object_array) [mostly less, but hopefully you get the idea]. I haven't looked into the gory details, and I'm not likely to soon since I'm tied up with other stuff, but I figured I'd toss the idea into the ring. -tim From esheldon at kicp.uchicago.edu Tue Jul 11 01:01:57 2006 From: esheldon at kicp.uchicago.edu (Erin Sheldon) Date: Tue, 11 Jul 2006 00:01:57 -0500 Subject: [Numpy-discussion] recarray field names Message-ID: <20060711000157.A24853@cfcp.uchicago.edu> On 7/11/06, Travis Oliphant wrote: > Erin Sheldon wrote: > > Just tested the lastest SVN and it works as advertised. Thanks > > Travis. > > > > An unrelated question: why does this work for arrays but not recarrays? > > > > > > In [24]: mydescriptor = > > [('age',float64),('Nchildren',int8),('weight',float32)] > > > > In [25]: a = array([(64,2,75.0),(25,0,60.0)], dtype=mydescriptor) > > > > In [26]: a = recarray([(64,2,75.0),(25,0,60.0)], dtype=mydescriptor) > > --------------------------------------------------------------------------- > > exceptions.TypeError Traceback (most recent > > call last) > > > > /Users/esheldon/ > > > > TypeError: __new__() got an unexpected keyword argument 'dtype' > > > > I understand that I could use the formats and names keywords, but > > this seems a little inconsistent. > > > > > Well there are a couple of reasons this doesn't work. > > 1) the recarray constructor is similar to the ndarray constructor. > > Neither of these take list arguments as inputs. So, while I've added > dtype as the > keyword for the formats input of the recarray constructor, the second > example will still fail. > > 2) Even were you to use the rec.array function (which is analagous to > the numpy.array function) to produce the error, it does not take a dtype > keyword. > > This is inconsistent. But, the interface for the record array class > came from numarray. I tried to stay compatible with those functions so > the numarray people would have an easier time adapting. I see. > > Perhaps we should add a dtype keyword argument to all the > constructors in numpy/core/records.py which over-rides the formats, > names, and titles keywords so that you could do > > a = rec.array([...], dtype=mine) > > analgously to > > a = array([...], dtype=mine) > The advantage of this is the ability to initialize the memory. One can get this functionality with core.records.fromrecords, but that is buried pretty far down. I realize now that all the functionality is in the array object. Other than the .field() method, one gets (I think) all the functionality from using the above array declaration and then a repeat() call to get a record array. I suppose then that recarray is really only for compatibility except for .field() Is there a compelling reason not to have a .field() method for arrays so that people in the know can just skip recarray altogether? >>> mydescriptor = [('age',int16),('Nchildren',int16),('weight',float32)] >>> a = array([(0,0,0.0)], dtype=mydescriptor) >>> a = repeat(a, num) From oliphant.travis at ieee.org Tue Jul 11 01:45:49 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 10 Jul 2006 23:45:49 -0600 Subject: [Numpy-discussion] Keyword argument order changed in records.py Message-ID: <44B33B0D.5000908@ieee.org> In an effort to preserve consistency, I made a change to the keyword argument order in the functions in records.py. In addition, all of the functions now take a dtype= keyword argument. This keyword argument is placed first along with other needed keyword arguments. When the dtype keyword is given then it's argument is converted to a data-type object and used in the record array. Otherwise (and only when dtype=None) the formats, names, titles, aligned, and byteorder arguments are used to construct the data-type object. These arguments are always last in the keyword argument order. Bascially, when these are given the rec.format_parser class is run to construct a data-type object from them: Thus using these keywords in place of data-type is equivalent to dtype = format_parser(formats, names, titles, aligned, byteorder)._descr Please let me know if this causes problems. As long as you used keyword arguments, the changes should have no effect on code written for NumPy. Best, -Travis From olivetti at itc.it Tue Jul 11 05:32:48 2006 From: olivetti at itc.it (Emanuele Olivetti) Date: Tue, 11 Jul 2006 11:32:48 +0200 Subject: [Numpy-discussion] how to use argsort result? Message-ID: <44B37040.2040905@itc.it> Hi, I don't understand how to use argsort results. I have a 2D matrix and I want to sort values in each row and obtain the index array of that sorting. Argsort(1) is what I need, but the problem is how to use its result in order to obtain a sorted matrix. Here is the simple example: A = array([[2,3,1],[5,4,6]]) indexes = a.argsort(1) now indexes is: array([[2, 0, 1], [1, 0, 2]]) I'd like to apply indexes to A and obtain: array([[1, 2, 3], [4, 5, 6]]) or better, I'm interested both in a subset of indexes, i.e. indexes[:,1:], and the related values of A matrix. How can I do this? If I simpy say: A[indexes] I get an IndexError. Thanks in advance, Emanuele P.S. numpy.__version__ is '0.9.8'. From pau.gargallo at gmail.com Tue Jul 11 05:40:44 2006 From: pau.gargallo at gmail.com (Pau Gargallo) Date: Tue, 11 Jul 2006 11:40:44 +0200 Subject: [Numpy-discussion] how to use argsort result? In-Reply-To: <44B37040.2040905@itc.it> References: <44B37040.2040905@itc.it> Message-ID: <6ef8f3380607110240i24f6f219w2fa112a113f04291@mail.gmail.com> here goes a first try: A[arange(2)[:,newaxis],indexes] pau On 7/11/06, Emanuele Olivetti wrote: > Hi, > I don't understand how to use argsort results. I have a 2D matrix and > I want to sort values in each row and obtain the index array of that > sorting. Argsort(1) is what I need, but the problem is how to use its > result in order to obtain a sorted matrix. Here is the simple example: > > A = array([[2,3,1],[5,4,6]]) > indexes = a.argsort(1) > > now indexes is: > array([[2, 0, 1], > [1, 0, 2]]) > > I'd like to apply indexes to A and obtain: > array([[1, 2, 3], > [4, 5, 6]]) > > or better, I'm interested both in a subset of indexes, i.e. indexes[:,1:], and > the related values of A matrix. > > How can I do this? If I simpy say: A[indexes] I get an IndexError. > > Thanks in advance, > > Emanuele > > P.S. numpy.__version__ is '0.9.8'. > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From olivetti at itc.it Tue Jul 11 05:57:28 2006 From: olivetti at itc.it (Emanuele Olivetti) Date: Tue, 11 Jul 2006 11:57:28 +0200 Subject: [Numpy-discussion] how to use argsort result? In-Reply-To: <6ef8f3380607110240i24f6f219w2fa112a113f04291@mail.gmail.com> References: <44B37040.2040905@itc.it> <6ef8f3380607110240i24f6f219w2fa112a113f04291@mail.gmail.com> Message-ID: <44B37608.2000003@itc.it> Wow. I have to study much more indexing. It works pretty well. Just to help indexing newbie like on using your advice: A[arange(A.shape[0])[:,newaxis],indexes] Thanks a lot! Emanuele Pau Gargallo wrote: > here goes a first try: > > A[arange(2)[:,newaxis],indexes] From stefan at sun.ac.za Tue Jul 11 06:20:03 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Tue, 11 Jul 2006 12:20:03 +0200 Subject: [Numpy-discussion] how to use argsort result? In-Reply-To: <44B37040.2040905@itc.it> References: <44B37040.2040905@itc.it> Message-ID: <20060711102003.GB22798@mentat.za.net> On Tue, Jul 11, 2006 at 11:32:48AM +0200, Emanuele Olivetti wrote: > Hi, > I don't understand how to use argsort results. I have a 2D matrix and > I want to sort values in each row and obtain the index array of that > sorting. Argsort(1) is what I need, but the problem is how to use its > result in order to obtain a sorted matrix. Here is the simple example: > > A = array([[2,3,1],[5,4,6]]) > indexes = a.argsort(1) > > now indexes is: > array([[2, 0, 1], > [1, 0, 2]]) > > I'd like to apply indexes to A and obtain: > array([[1, 2, 3], > [4, 5, 6]]) > > or better, I'm interested both in a subset of indexes, i.e. indexes[:,1:], and > the related values of A matrix. > > How can I do this? If I simpy say: A[indexes] I get an IndexError. Something's not quite right here. The argsort docstring states that: argsort(a,axis=-1) return the indices into a of the sorted array along the given axis, so that take(a,result,axis) is the sorted array. But N.take(A,A.argsort()) breaks. Either this is a bug, or the docstring needs to be updated. Cheers St?fan From pau.gargallo at gmail.com Tue Jul 11 06:37:23 2006 From: pau.gargallo at gmail.com (Pau Gargallo) Date: Tue, 11 Jul 2006 12:37:23 +0200 Subject: [Numpy-discussion] how to use argsort result? In-Reply-To: <20060711102003.GB22798@mentat.za.net> References: <44B37040.2040905@itc.it> <20060711102003.GB22798@mentat.za.net> Message-ID: <6ef8f3380607110337y7f07790dj42e0812864d1e08d@mail.gmail.com> On 7/11/06, Stefan van der Walt wrote: > On Tue, Jul 11, 2006 at 11:32:48AM +0200, Emanuele Olivetti wrote: > > Hi, > > I don't understand how to use argsort results. I have a 2D matrix and > > I want to sort values in each row and obtain the index array of that > > sorting. Argsort(1) is what I need, but the problem is how to use its > > result in order to obtain a sorted matrix. Here is the simple example: > > > > A = array([[2,3,1],[5,4,6]]) > > indexes = a.argsort(1) > > > > now indexes is: > > array([[2, 0, 1], > > [1, 0, 2]]) > > > > I'd like to apply indexes to A and obtain: > > array([[1, 2, 3], > > [4, 5, 6]]) > > > > or better, I'm interested both in a subset of indexes, i.e. indexes[:,1:], and > > the related values of A matrix. > > > > How can I do this? If I simpy say: A[indexes] I get an IndexError. > > Something's not quite right here. The argsort docstring states that: > > argsort(a,axis=-1) return the indices into a of the sorted array > along the given axis, so that take(a,result,axis) is the sorted array. > > But > > N.take(A,A.argsort()) breaks. Either this is a bug, or the docstring > needs to be updated. > > Cheers > St?fan > I think the docstring is wrong, because take doesn't do that. if you N.take(A,A.argsort(1), 1), it doesn't break but it doesn't sort A neither. Take seems to peek entire columns, but take's docstring is missing. For the argsort docstring, it may be usefull to indicate that if one do >>> ind = indices(A.shape) >>> ind[ax] = A.argsort(axis=ax) then A[ind] is the sorted array. pau From schofield at ftw.at Tue Jul 11 06:37:25 2006 From: schofield at ftw.at (Ed Schofield) Date: Tue, 11 Jul 2006 12:37:25 +0200 Subject: [Numpy-discussion] optimize.leastsq crashs my interpreter In-Reply-To: References: <003301c6a1b8$80ca16e0$1001a8c0@JELLE> <44AE505D.8050905@ftw.at> <44AE775A.30004@ftw.at> Message-ID: <44B37F65.1090508@ftw.at> John Hassler wrote: > Ed Schofield ftw.at> writes: > > >> Hmmm ... it could be an ATLAS problem. What's your processor? I built >> the SciPy 0.4.9 binaries against Pearu's ATLAS binaries for Pentium 2, >> thinking that this would give maximum compatibility ... >> >> Or perhaps it's something else. Could someone with this problem please >> post a backtrace? >> > > > This computer is an AMD Athlon 1600+ running Windows XP. > > > > All of the versions of scipy using numpy crash with XP whenever I access > any of the functions in "optimize" or "integrate" which (I assume) call the > Fortran libraries. > > In the current version, running scipy.test() gives an "unhandled > exception." Debug shows a pointer to: > 020CA9C3 xorps xmm6,xmm6 > > > > Some other information: > >>>> scipy.__version__ '0.4.9' >>>> > > >>>> scipy.__numpy_version__ '0.9.8' >>>> > > >>>> scipy.show_numpy_config() >>>> > > atlas_threads_info: NOT AVAILABLE > > blas_opt_info: libraries = ['f77blas', 'cblas', 'atlas'] library_dirs = > ['C:\\Libraries\\ATLAS_3.6.0_WIN_P4'] define_macros = [('ATLAS_INFO', > '"\\"3.6.0\\""')] language = c include_dirs = > ['C:\\Libraries\\ATLAS_3.6.0_WIN_P4'] > > plus similar stuff. Probably the important thing is the "ATLAS ... P4" > line. > Thanks, John -- this is helpful. (Hans too, thanks for your testing). This looks like a problem with the NumPy build. Travis, could you have compiled the Win32 binaries accidentally against the P4/SSE2 ATLAS library? -- Ed From pau.gargallo at gmail.com Tue Jul 11 06:54:22 2006 From: pau.gargallo at gmail.com (Pau Gargallo) Date: Tue, 11 Jul 2006 12:54:22 +0200 Subject: [Numpy-discussion] int 'up'casted to uint Message-ID: <6ef8f3380607110354u6db272b0t125a9f13a92d1180@mail.gmail.com> hi, looking at the upcasting table at http://www.scipy.org/Tentative_NumPy_Tutorial#head-4c1d53fe504adc97baf27b65513b4b97586a4fc5 I saw that int's are sometimes casted to uint's. In [3]: a = array([3],int16) In [5]: b = array([4],uint32) In [7]: a+b Out[7]: array([7], dtype=uint32) is that intended? pau From stefan at sun.ac.za Tue Jul 11 07:10:53 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Tue, 11 Jul 2006 13:10:53 +0200 Subject: [Numpy-discussion] how to use argsort result? In-Reply-To: <6ef8f3380607110337y7f07790dj42e0812864d1e08d@mail.gmail.com> References: <44B37040.2040905@itc.it> <20060711102003.GB22798@mentat.za.net> <6ef8f3380607110337y7f07790dj42e0812864d1e08d@mail.gmail.com> Message-ID: <20060711111053.GC24667@mentat.za.net> On Tue, Jul 11, 2006 at 12:37:23PM +0200, Pau Gargallo wrote: > > Something's not quite right here. The argsort docstring states that: > > > > argsort(a,axis=-1) return the indices into a of the sorted array > > along the given axis, so that take(a,result,axis) is the sorted array. > > > > But > > > > N.take(A,A.argsort()) breaks. Either this is a bug, or the docstring > > needs to be updated. > > > > Cheers > > St?fan > > > > I think the docstring is wrong, because take doesn't do that. > if you N.take(A,A.argsort(1), 1), it doesn't break but it doesn't sort > A neither. > > Take seems to peek entire columns, but take's docstring is missing. > > For the argsort docstring, it may be usefull to indicate that if one do > >>> ind = indices(A.shape) > >>> ind[ax] = A.argsort(axis=ax) > then A[ind] is the sorted array. We can always adapt the documentation at http://numeric.scipy.org/numpydoc/numpy-9.html#pgfId-36425 into a docstring. I'll file a ticket. Cheers St?fan From Norbert.Nemec.list at gmx.de Tue Jul 11 11:00:50 2006 From: Norbert.Nemec.list at gmx.de (Norbert Nemec) Date: Tue, 11 Jul 2006 17:00:50 +0200 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <91cf711d0607031000l286da52fn30d98798057f12c5@mail.gmail.com> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> <44A8E4D3.5000603@ntc.zcu.cz> <91cf711d0607031000l286da52fn30d98798057f12c5@mail.gmail.com> Message-ID: <44B3BD22.70003@gmx.de> unique1d is based on ediff1d, so it really calculates many differences and compares those to 0.0 This is inefficient, even though this is hidden by the general inefficiency of Python (It might be the reason for the two milliseconds, though) What is more: subtraction works only for numbers, while the various proposed versions use only comparison which works for any data type (as long as it can be sorted) My own version tried to capture all possible cases that the current unique captures. Sasha's version only works for numpy arrays and has a problem for arrays with all identical entries. David's version only works for numpy arrays of types that can be converted to float. I would once more propose to use my own version as given before: def unique(arr,sort=True): if hasattr(arr,'flatten'): tmp = arr.flatten() tmp.sort() idx = concatenate([True],tmp[1:]!=tmp[:-1]) return tmp[idx] else: # for compatibility: set = {} for item in inseq: set[item] = None if sort: return asarray(sorted(set.keys())) else: return asarray(set.keys()) Greetings, Norbert From aisaac at american.edu Tue Jul 11 11:47:27 2006 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 11 Jul 2006 11:47:27 -0400 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <44B3BD22.70003@gmx.de> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> <44A8E4D3.5000603@ntc.zcu.cz><91cf711d0607031000l286da52fn30d98798057f12c5@mail.gmail.com> <44B3BD22.70003@gmx.de> Message-ID: On Tue, 11 Jul 2006, Norbert Nemec apparently wrote: > else: # for compatibility: > set = {} > for item in inseq: > set[item] = None > if sort: > return asarray(sorted(set.keys())) > else: > return asarray(set.keys()) I'm currently in major caffeine deficit, but aside from backward compatability, how is this better than: else: #for compatability items = list(set(inseq)) if sort: items.sort() return asarray(items) Alan Isaac PS Actually, making a list of a set may already sort? No time to check now ... PPS For to 2.3, need set = sets.Set From tim.hochberg at cox.net Tue Jul 11 12:02:21 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue, 11 Jul 2006 09:02:21 -0700 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <44B3BD22.70003@gmx.de> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> <44A8E4D3.5000603@ntc.zcu.cz> <91cf711d0607031000l286da52fn30d98798057f12c5@mail.gmail.com> <44B3BD22.70003@gmx.de> Message-ID: <44B3CB8D.9010308@cox.net> Norbert Nemec wrote: > unique1d is based on ediff1d, so it really calculates many differences > and compares those to 0.0 > > This is inefficient, even though this is hidden by the general > inefficiency of Python (It might be the reason for the two milliseconds, > though) > > What is more: subtraction works only for numbers, while the various > proposed versions use only comparison which works for any data type (as > long as it can be sorted) > My first question is: why? What's the attraction in returning a sorted answer here? Returning an unsorted array is potentially faster, depending on the algorithm chosen, and sorting after the fact is trivial. If one was going to spend extra complexity on something, I'd think it would be better spent on preserving the input order. Second, some objects can be compared for equality and hashed, but not sorted (Python's complex number's come to mind). If one is going to worry about subtraction so as to keep things general, it makes sense to also avoid sorting as well Sasha's slick algorithm not withstanding. Third, I propose that whatever the outcome of the sorting issue, I would propose that unique have the same interface as the other structural array operations. That is: unique(anarray, axis=0): ... The default axis=0 is for compatibility with the other, somewhat similar functions. Axis=None would return the flattened, uniquified data, axis=# would uniquify the result along that axis. Regards, -tim > My own version tried to capture all possible cases that the current > unique captures. > > Sasha's version only works for numpy arrays and has a problem for arrays > with all identical entries. > > David's version only works for numpy arrays of types that can be > converted to float. > > I would once more propose to use my own version as given before: > > def unique(arr,sort=True): > if hasattr(arr,'flatten'): > tmp = arr.flatten() > tmp.sort() > idx = concatenate([True],tmp[1:]!=tmp[:-1]) > return tmp[idx] > else: # for compatibility: > set = {} > for item in inseq: > set[item] = None > if sort: > return asarray(sorted(set.keys())) > else: > return asarray(set.keys()) > > > Greetings, > Norbert > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From oliphant.travis at ieee.org Tue Jul 11 12:03:38 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 11 Jul 2006 10:03:38 -0600 Subject: [Numpy-discussion] int 'up'casted to uint In-Reply-To: <6ef8f3380607110354u6db272b0t125a9f13a92d1180@mail.gmail.com> References: <6ef8f3380607110354u6db272b0t125a9f13a92d1180@mail.gmail.com> Message-ID: <44B3CBDA.8080605@ieee.org> Pau Gargallo wrote: > hi, > > looking at the upcasting table at > http://www.scipy.org/Tentative_NumPy_Tutorial#head-4c1d53fe504adc97baf27b65513b4b97586a4fc5 > I saw that int's are sometimes casted to uint's. > > In [3]: a = array([3],int16) > In [5]: b = array([4],uint32) > In [7]: a+b > Out[7]: array([7], dtype=uint32) > > is that intended? > It's a bug. The result should be int64. I've fixed it in SVN. -Travis From tim.hochberg at cox.net Tue Jul 11 12:06:23 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue, 11 Jul 2006 09:06:23 -0700 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <44B3CB8D.9010308@cox.net> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> <44A8E4D3.5000603@ntc.zcu.cz> <91cf711d0607031000l286da52fn30d98798057f12c5@mail.gmail.com> <44B3BD22.70003@gmx.de> <44B3CB8D.9010308@cox.net> Message-ID: <44B3CC7F.2070000@cox.net> Tim Hochberg wrote: > Norbert Nemec wrote: > >> unique1d is based on ediff1d, so it really calculates many differences >> and compares those to 0.0 >> >> This is inefficient, even though this is hidden by the general >> inefficiency of Python (It might be the reason for the two milliseconds, >> though) >> >> What is more: subtraction works only for numbers, while the various >> proposed versions use only comparison which works for any data type (as >> long as it can be sorted) >> >> > My first question is: why? What's the attraction in returning a sorted > answer here? Returning an unsorted array is potentially faster, > depending on the algorithm chosen, and sorting after the fact is > trivial. If one was going to spend extra complexity on something, I'd > think it would be better spent on preserving the input order. > > Second, some objects can be compared for equality and hashed, but not > sorted (Python's complex number's come to mind). If one is going to > worry about subtraction so as to keep things general, it makes sense to > also avoid sorting as well Sasha's slick algorithm not withstanding. > > Third, I propose that whatever the outcome of the sorting issue, I would > propose that unique have the same interface as the other structural > array operations. That is: > > unique(anarray, axis=0): > ... > > The default axis=0 is for compatibility with the other, somewhat similar > functions. Axis=None would return the flattened, uniquified data, > axis=# would uniquify the result along that axis. > Hmmm. Of course that precludes it returning an actual array for axis!=None. That might be considered suboptimal... -tim > Regards, > > -tim > > >> My own version tried to capture all possible cases that the current >> unique captures. >> >> Sasha's version only works for numpy arrays and has a problem for arrays >> with all identical entries. >> >> David's version only works for numpy arrays of types that can be >> converted to float. >> >> I would once more propose to use my own version as given before: >> >> def unique(arr,sort=True): >> if hasattr(arr,'flatten'): >> tmp = arr.flatten() >> tmp.sort() >> idx = concatenate([True],tmp[1:]!=tmp[:-1]) >> return tmp[idx] >> else: # for compatibility: >> set = {} >> for item in inseq: >> set[item] = None >> if sort: >> return asarray(sorted(set.keys())) >> else: >> return asarray(set.keys()) >> >> >> Greetings, >> Norbert >> >> >> >> ------------------------------------------------------------------------- >> Using Tomcat but need to do more? Need to support web services, security? >> Get stuff done quickly with pre-integrated technology to make your job easier >> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> >> >> >> > > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From robert.kern at gmail.com Tue Jul 11 12:23:42 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 11 Jul 2006 11:23:42 -0500 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <44B3CB8D.9010308@cox.net> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> <44A8E4D3.5000603@ntc.zcu.cz> <91cf711d0607031000l286da52fn30d98798057f12c5@mail.gmail.com> <44B3BD22.70003@gmx.de> <44B3CB8D.9010308@cox.net> Message-ID: Tim Hochberg wrote: > Norbert Nemec wrote: >> unique1d is based on ediff1d, so it really calculates many differences >> and compares those to 0.0 >> >> This is inefficient, even though this is hidden by the general >> inefficiency of Python (It might be the reason for the two milliseconds, >> though) >> >> What is more: subtraction works only for numbers, while the various >> proposed versions use only comparison which works for any data type (as >> long as it can be sorted) >> > My first question is: why? What's the attraction in returning a sorted > answer here? Returning an unsorted array is potentially faster, > depending on the algorithm chosen, and sorting after the fact is > trivial. If one was going to spend extra complexity on something, I'd > think it would be better spent on preserving the input order. One issue is that uniquifying numpy arrays using Python dicts, while expected O(n) in terms of complexity, is really slow compared to sorting because of the overhead in getting the elements out of the numpy arrays and into Python objects. For the cases where sorting works (your caveat below is quite correct), it's really quite good for numpy arrays. OTOH, if one were to implement a hash table in C, that might potentially be faster and more robust, but that is spending a *large* amount of extra complexity. > Second, some objects can be compared for equality and hashed, but not > sorted (Python's complex number's come to mind). If one is going to > worry about subtraction so as to keep things general, it makes sense to > also avoid sorting as well Sasha's slick algorithm not withstanding. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pau.gargallo at gmail.com Tue Jul 11 12:24:34 2006 From: pau.gargallo at gmail.com (Pau Gargallo) Date: Tue, 11 Jul 2006 18:24:34 +0200 Subject: [Numpy-discussion] int 'up'casted to uint In-Reply-To: <44B3CBDA.8080605@ieee.org> References: <6ef8f3380607110354u6db272b0t125a9f13a92d1180@mail.gmail.com> <44B3CBDA.8080605@ieee.org> Message-ID: <6ef8f3380607110924p66b8de0ds55971dfb8c9ff892@mail.gmail.com> On 7/11/06, Travis Oliphant wrote: > Pau Gargallo wrote: > > hi, > > > > looking at the upcasting table at > > http://www.scipy.org/Tentative_NumPy_Tutorial#head-4c1d53fe504adc97baf27b65513b4b97586a4fc5 > > I saw that int's are sometimes casted to uint's. > > > > In [3]: a = array([3],int16) > > In [5]: b = array([4],uint32) > > In [7]: a+b > > Out[7]: array([7], dtype=uint32) > > > > is that intended? > > > It's a bug. The result should be int64. I've fixed it in SVN. > Thanks!! From cimrman3 at ntc.zcu.cz Tue Jul 11 12:49:41 2006 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Tue, 11 Jul 2006 18:49:41 +0200 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <44B3BD22.70003@gmx.de> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> <44A8E4D3.5000603@ntc.zcu.cz> <91cf711d0607031000l286da52fn30d98798057f12c5@mail.gmail.com> <44B3BD22.70003@gmx.de> Message-ID: <44B3D6A5.6060404@ntc.zcu.cz> Norbert Nemec wrote: > unique1d is based on ediff1d, so it really calculates many differences > and compares those to 0.0 > > This is inefficient, even though this is hidden by the general > inefficiency of Python (It might be the reason for the two milliseconds, > though) > > What is more: subtraction works only for numbers, while the various > proposed versions use only comparison which works for any data type (as > long as it can be sorted) I agree that unique1d works only for numbers, but that is what it was meant for... well for integers only, in fact - everyone here surely knows, that comparing floats with != does not work well. Note also that it was written before logical indexing and other neat stuff were not possible in numpy - every improvement is welcome! (btw. I cannot recall why I used subtraction and testing for zero instead of just comparisons - maybe remnants from my old matlab days and its logical arrays - ediff1d should disappear from the other functions in arraysetops) > My own version tried to capture all possible cases that the current > unique captures. > > Sasha's version only works for numpy arrays and has a problem for arrays > with all identical entries. > > David's version only works for numpy arrays of types that can be > converted to float. comparing floats... > I would once more propose to use my own version as given before: > > def unique(arr,sort=True): > if hasattr(arr,'flatten'): > tmp = arr.flatten() > tmp.sort() > idx = concatenate([True],tmp[1:]!=tmp[:-1]) > return tmp[idx] > else: # for compatibility: > set = {} > for item in inseq: > set[item] = None > if sort: > return asarray(sorted(set.keys())) > else: > return asarray(set.keys()) Have you considered using set instead of dict? Just curious :-) r. From Chris.Barker at noaa.gov Tue Jul 11 13:47:46 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 11 Jul 2006 10:47:46 -0700 Subject: [Numpy-discussion] array interface, PIL and PyGame Message-ID: <44B3E442.7080606@noaa.gov> Hi all, Over on the PIL mailing list, someone asked about some possible additions to PIL to facilitate copy-free moving of data between PyGame and PIL. I sent a note suggesting that the array interface might be just the ticket. These were the responses: Pete Shinners wrote: > Yeah, this would be an ideal solution. I hope more of the base array stuff can > get into Python 2.6. > > We did look at the array proposal, but haven't been able to jump on it yet > because of adoption. I guess it needs another package or two to get the ball > rolling. So that's an advocacy issue, and an illustration of why getting it into the standard lib is critical. Fredrik Lundh wrote: > unfortunately, the "array interface" model isn't even close to be able > to describe a PIL image memory (the "Imaging" structure)... Darn. It was my understanding that we thought that it was close to being able to describe an image... so what have we missed. I'm out of my technical depth, but I've encouraged Fredrik to bring the discussion here. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant at ee.byu.edu Tue Jul 11 14:04:37 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 11 Jul 2006 12:04:37 -0600 Subject: [Numpy-discussion] array interface, PIL and PyGame In-Reply-To: <44B3E442.7080606@noaa.gov> References: <44B3E442.7080606@noaa.gov> Message-ID: <44B3E835.1040207@ee.byu.edu> Christopher Barker wrote: >Hi all, > >Over on the PIL mailing list, someone asked about some possible >additions to PIL to facilitate copy-free moving of data between PyGame >and PIL. I sent a note suggesting that the array interface might be just >the ticket. These were the responses: > >Pete Shinners wrote: > > >>Yeah, this would be an ideal solution. I hope more of the base array stuff can >>get into Python 2.6. >> >>We did look at the array proposal, but haven't been able to jump on it yet >>because of adoption. I guess it needs another package or two to get the ball >>rolling. >> >> > >So that's an advocacy issue, and an illustration of why getting it into >the standard lib is critical. > >Fredrik Lundh wrote: > > >>unfortunately, the "array interface" model isn't even close to be able >>to describe a PIL image memory (the "Imaging" structure)... >> >> > > > We will all welcome Fredrik's comments. But, I think he is exaggerating the situation a bit. It is true that the Imaging structure of PIL is a more general memory model for 2-d arrays. My understanding is that it basically allows for an array of pointers to memory where each pointer points to a different line (or chunk) in the image (kind of like a C-array). So, basically, the problem is that you may have an image that cannot be described as a single block of memory or a singly-strided array. I believe, thought, that sometimes you do have a PIL image that is basically a single chunk of memory. So, while a general-purpose "memory-sharing" situation might be difficult. I think some sharing (of each chunk for example) could be done. Even still, the array interface (the Python side) does technically handle the PIL case because the default is to use the sequence interface to access elements of the array. It would be nice if Fredrik were more willing to help establish a standard, though. Calling it "not close" but giving no alternative is not helpful. >Darn. It was my understanding that we thought that it was close to being >able to describe an image... so what have we missed. I'm out of my >technical depth, but I've encouraged Fredrik to bring the discussion here. > > -Travis From david.huard at gmail.com Tue Jul 11 14:27:37 2006 From: david.huard at gmail.com (David Huard) Date: Tue, 11 Jul 2006 14:27:37 -0400 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <44B3D6A5.6060404@ntc.zcu.cz> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> <44A8E4D3.5000603@ntc.zcu.cz> <91cf711d0607031000l286da52fn30d98798057f12c5@mail.gmail.com> <44B3BD22.70003@gmx.de> <44B3D6A5.6060404@ntc.zcu.cz> Message-ID: <91cf711d0607111127w21bb4b3cj30a3c31b46e136f0@mail.gmail.com> Tim Hochberg wrote: > My first question is: why? What's the attraction in returning a sorted > answer here? Returning an unsorted array is potentially faster, > depending on the algorithm chosen, and sorting after the fact is > trivial. If one was going to spend extra complexity on something, I'd > think it would be better spent on preserving the input order. There is a unique function in matlab that returns a sorted vector. I think a lot of people will expect a numpy and matlab functions with identical names to behave similarly. If we want to preserve the input order, we'd have to choose a convention about whose value's order is retained: do we keep the order of the first value found or the last one ? Here is the benchmark. Sorry Norbert for not including your code the first time, it turns out that with Alain's suggestion, its the fastest one both for lists and arrays. x = rand(100000)*100 x = x.astype('i') l = list(x) For array x: In [166]: timeit unique_alan(x) # with set instead of dict 100 loops, best of 3: 8.8 ms per loop In [167]: timeit unique_norbert(x) 100 loops, best of 3: 8.8 ms per loop In [168]: timeit unique_sasha(x) 100 loops, best of 3: 10.8 ms per loop In [169]: timeit unique(x) 10 loops, best of 3: 50.4 ms per loop In [170]: timeit unique1d(x) 10 loops, best of 3: 13.2 ms per loop For list l: In [196]: timeit unique_norbert(l) 10 loops, best of 3: 29 ms per loop In [197]: timeit unique_alan(l) # with set instead of dict 10 loops, best of 3: 14.5 ms per loop In [193]: timeit unique(l) 10 loops, best of 3: 29.6 ms per loop Note: In Norbert function, setting sort=False for flattenable objects returns a sorted array anyway. So I'd suggest to remove the sort keyword, sort if the datatype is sortable, and don't sort if its not. David -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: unique_test.py Type: text/x-python-script Size: 994 bytes Desc: not available URL: From perry at stsci.edu Tue Jul 11 14:28:31 2006 From: perry at stsci.edu (Perry Greenfield) Date: Tue, 11 Jul 2006 14:28:31 -0400 Subject: [Numpy-discussion] array interface, PIL and PyGame In-Reply-To: <44B3E835.1040207@ee.byu.edu> References: <44B3E442.7080606@noaa.gov> <44B3E835.1040207@ee.byu.edu> Message-ID: On Jul 11, 2006, at 2:04 PM, Travis Oliphant wrote: >> > We will all welcome Fredrik's comments. But, I think he is > exaggerating > the situation a bit. > > It is true that the Imaging structure of PIL is a more general memory > model for 2-d arrays. My understanding is that it basically allows > for > an array of pointers to memory where each pointer points to a > different > line (or chunk) in the image (kind of like a C-array). > > So, basically, the problem is that you may have an image that > cannot be > described as a single block of memory or a singly-strided array. I > believe, thought, that sometimes you do have a PIL image that is > basically a single chunk of memory. > > So, while a general-purpose "memory-sharing" situation might be > difficult. I think some sharing (of each chunk for example) could be > done. > > Even still, the array interface (the Python side) does technically > handle the PIL case because the default is to use the sequence > interface > to access elements of the array. > > It would be nice if Fredrik were more willing to help establish a > standard, though. Calling it "not close" but giving no alternative is > not helpful. To expand on this a bit, I think Fredrik is misreading the intent of the array interface somewhat. It's not that we are promising that the array data structure will suit his every need. Far from it. It's that it allows him (or us) to provide convenience functions that allow accessing data in PIL without jumping through lots of annoying hoops. It should be possible to convert PIL image data to arrays easily. Does that mean that all associated information is propagated as part of the array object? Of course not. But this information is obtainable by other means anyway. Even in the case where different chunks of an image are in different memory locations and there is no simple way of avoiding copying the data, so what? I still much prefer the data be copied to an array so that numpy functionality can be applied to the array and have to avoid calling a sequence of operations to convert the data to a string and then to an array. At least one copy is avoided, but avoiding copying isn't the entire justification. Think of it this way, if PIL strength is that it can convert images between many different formats (no doubt, copying data in the process) then arrays should be one of the supported formats, no? Perry From oliphant.travis at ieee.org Tue Jul 11 15:37:15 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 11 Jul 2006 13:37:15 -0600 Subject: [Numpy-discussion] Quicker image transfer, tobuffer? In-Reply-To: References: <44B2EC45.8020402@noaa.gov> Message-ID: <44B3FDEB.1000909@ieee.org> Here is a simple approach to allowing the PIL to export the array interface. This allows NumPy to create a suitable array from a PIL image very easily: At the top of Image.py add the following if sys.byteorder == 'little': _ENDIAN = '<' else: _ENDIAN = '>' _MODE_CONV = { # official modes "1": ('|b1', None), "L": ('|u1', None), "I": ('%si4' % _ENDIAN, None), "F": ('%sf4' % _ENDIAN, None), "P": ('|u1', None), "RGB": ('|u1', 3), "RGBX": ('|u1', 4), "RGBA": ('|u1', 4), "CMYK": ('|u1', 4), "YCbCr": ('|u1', 4), # Experimental modes include I;16, I;16B, RGBa, BGR;15, # and BGR;24. Use these modes only if you know exactly # what you're doing... } def _conv_type_shape(im): shape = im.size typ, extra = _MODE_CONV[im.mode] if extra is None: return shape, typ shape += (extra,) return shape, typ In the Image class structure add def __get_array_interface__(self): new = {} shape, typestr = _conv_type_shape(self) new['shape'] = shape new['typestr'] = typestr new['data'] = self.tostring() return new __array_interface__ = property(__get_array_interface__, None, doc="array interface") With this addition you can then do import Image, numpy im = Image.open('lena.jpg') a = numpy.asarray(im) and you will get a suitable read-only array pointing to the string produced by tostring. This would be a nice thing to add to the PIL. -Travis Oliphant From jonathan.taylor at utoronto.ca Tue Jul 11 16:07:40 2006 From: jonathan.taylor at utoronto.ca (Jonathan Taylor) Date: Tue, 11 Jul 2006 16:07:40 -0400 Subject: [Numpy-discussion] What's wrong with matrices? In-Reply-To: <44AE6675.108@ftw.at> References: <44AE6675.108@ftw.at> Message-ID: <463e11f90607111307s8bb9344w42d30b844cdaf0b4@mail.gmail.com> I agree the real problem with matrices is they seem awkward to work with compared to arrays because numpy seems so array centric. The only advantage I see is getting .T to do transposes and * to do matrix multiplication. I hope numpy reaches a point where it is as natural to use matrices as arrays. I'd also vote for the inclusion of the following two functions col and row. Inspired by R equivelents they let you do some indexing very easily such as getting the values of the upper triangle of the matrix. E.g. vals = m[row(m) > col(m)] Cheers, Jon. def col(m): """col(m) returns a matrix of the same size of m where each element contains an integer denoting which column it is in. For example, >>> m = eye(3) >>> m array([[1, 0, 0], [0, 1, 0], [0, 0, 1]]) >>> col(m) array([[0, 1, 2], [0, 1, 2], [0, 1, 2]]) """ assert len(m.shape) == 2, "should be a matrix" return N.indices(m.shape)[1] def row(m): """row(m) returns a matrix of the same size of m where each element contains an integer denoting which row it is in. For example, >>> m = eye(3) >>> m array([[1, 0, 0], [0, 1, 0], [0, 0, 1]]) >>> row(m) array([[0, 0, 0], [1, 1, 1], [2, 2, 2]]) """ assert len(m.shape) == 2, "should be a matrix" return N.indices(m.shape)[0] On 7/7/06, Ed Schofield wrote: > Bill Baxter wrote: > > I think the thread to this point can be pretty much summarized by: > > > > while True: > > Bill: "2D transpose is common so it should have a nice syntax" > > Tim, Robert, Sasha, and Ed: "No it's not." > > > > Very well. I think it may be a self fulfilling prophecy, though. > > I.e. if matrix operations are cumbersome to use, then -- surprise > > surprise -- the large user base for matrix-like operations never > > materializes. Potential converts just give numpy the pass, and go to > > Octave or Scilab, or stick with Matlab, R or S instead. > > > > Why all the fuss about the .T? Because any changes to functions (like > > making ones() return a matrix) can easily be worked around on the user > > side, as has been pointed out. But as far as I know -- do correct me > > if I'm wrong -- there's no good way for a user to add an attribute to > > an existing class. After switching from matrices back to arrays, .T > > was the only thing I really missed from numpy.matrix. > > > > I would be all for a matrix class that was on equal footing with array > > and as easy to use as matrices in Matlab. But my experience using > > numpy.matrix was far from that, and, given the lack of enthusiasm for > > matrices around here, that seems unlikely to change. However, I'm > > anxious to see what Ed has up his sleeves in the other thread. > > Okay ... ... let's make this the thread ;) > I'd like to know why you, Sven, and anyone else on the list have gone > back to using arrays after trying matrices. What was inconvenient about > them? I'd like a nice juicy list. The whole purpose of the matrix > class is to simplify 2d linear algebra. Where is it failing? > > I also went back to arrays after trying out matrices for some 2d linear > algebra tasks, since I found that using matrices increased my code's > complexity. I can describe the problems I had with them later, but > first I'd like to hear of others' experiences. > > I'd like to help to make matrices more usable. Tell me what you want, > and I'll work on some patches. > > -- Ed > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From kevin at cazabon.com Tue Jul 11 16:10:29 2006 From: kevin at cazabon.com (kevin at cazabon.com) Date: Tue, 11 Jul 2006 22:10:29 +0200 Subject: [Numpy-discussion] [Image-SIG] Quicker image transfer, tobuffer? References: <44B2EC45.8020402@noaa.gov> <44B3FDEB.1000909@ieee.org> Message-ID: <013401c6a526$0eb5e5e0$650aa8c0@duallie> Although I'm not really up to speed on the array interface, accessing the pixel data in a PIL image isn't really that difficult in C/C++... the only challenge I would see (besides tracking the channels/padding correctly... trivial) would be getting the pointer into Python to pass it to NumPy. I've written a few modules in C that directly modify the PIL buffer data, with simple code such as attached (lines 186-214 show it clearly). (This is a module that does unsharp-masking and gaussian blur on PIL images... Fredrik is welcome to include this directly into the PIL library if he sees fit, for which I'll gladly remove ANY licensing restrictions) Kevin. ----- Original Message ----- From: "Travis Oliphant" To: Cc: "numpy-discussion" Sent: Tuesday, July 11, 2006 9:37 PM Subject: Re: [Image-SIG] Quicker image transfer, tobuffer? > > Here is a simple approach to allowing the PIL to export the array > interface. > > This allows NumPy to create a suitable array from a PIL image very easily: > > > At the top of Image.py add the following > > if sys.byteorder == 'little': > _ENDIAN = '<' > else: > _ENDIAN = '>' > > _MODE_CONV = { > > # official modes > "1": ('|b1', None), > "L": ('|u1', None), > "I": ('%si4' % _ENDIAN, None), > "F": ('%sf4' % _ENDIAN, None), > "P": ('|u1', None), > "RGB": ('|u1', 3), > "RGBX": ('|u1', 4), > "RGBA": ('|u1', 4), > "CMYK": ('|u1', 4), > "YCbCr": ('|u1', 4), > > # Experimental modes include I;16, I;16B, RGBa, BGR;15, > # and BGR;24. Use these modes only if you know exactly > # what you're doing... > > } > > def _conv_type_shape(im): > shape = im.size > typ, extra = _MODE_CONV[im.mode] > if extra is None: > return shape, typ > shape += (extra,) > return shape, typ > > > > In the Image class structure add > > def __get_array_interface__(self): > new = {} > shape, typestr = _conv_type_shape(self) > new['shape'] = shape > new['typestr'] = typestr > new['data'] = self.tostring() > return new > > __array_interface__ = property(__get_array_interface__, None, > doc="array interface") > > > > With this addition you can then do > > import Image, numpy > > im = Image.open('lena.jpg') > a = numpy.asarray(im) > > and you will get a suitable read-only array pointing to the string > produced by tostring. > > > This would be a nice thing to add to the PIL. > > -Travis Oliphant > > > _______________________________________________ > Image-SIG maillist - Image-SIG at python.org > http://mail.python.org/mailman/listinfo/image-sig > -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: PILusm.cpp URL: From myeates at jpl.nasa.gov Tue Jul 11 16:49:15 2006 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Tue, 11 Jul 2006 13:49:15 -0700 Subject: [Numpy-discussion] looking for the Numpy to do it Message-ID: <44B40ECB.8060408@jpl.nasa.gov> I can handle the following problem by iterating through some indices but I'm looking for a more elegant solution. If I have a 1d array, I want to find a contiguous nonzero region about a given index. For example, if a=[1,2,3,0,40,50,60,0,7,8,9] and we start with the index of 5, then I want the indices 4,5,6 Any gurus out there? Mathew From ndarray at mac.com Tue Jul 11 17:03:49 2006 From: ndarray at mac.com (Sasha) Date: Tue, 11 Jul 2006 17:03:49 -0400 Subject: [Numpy-discussion] looking for the Numpy to do it In-Reply-To: <44B40ECB.8060408@jpl.nasa.gov> References: <44B40ECB.8060408@jpl.nasa.gov> Message-ID: Here is the solution of a half of the problem: >>> a=array([1,2,3,0,40,50,60,0,7,8,9]) >>> 5+where(logical_and.accumulate(a[5:]!=0)) array([5, 6]) the rest is left as an exercise to the reader :-) Hint a[::-1] will reverse a. On 7/11/06, Mathew Yeates wrote: > I can handle the following problem by iterating through some indices but > I'm looking for a more elegant solution. > > If I have a 1d array, I want to find a contiguous nonzero region about a > given index. For example, if a=[1,2,3,0,40,50,60,0,7,8,9] and we start > with the index of 5, then I want the indices 4,5,6 > > Any gurus out there? > > Mathew > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From oliphant at ee.byu.edu Tue Jul 11 17:15:06 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 11 Jul 2006 15:15:06 -0600 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> Message-ID: <44B414DA.1050300@ee.byu.edu> Ed Schofield wrote: >Last week's discussion on rand() and randn() seemed to indicate a >sentiment that they ought to take tuples for consistency with ones, >zeros, eye, identity, and empty -- that, although they are supposed >to be convenience functions, they are inconvenient precisely because >of their inconsistency with these other functions. This issue has >been raised many times over the past several months. > >Travis made a change in r2572 to allow tuples as arguments, then took >it out again a few hours later, apparently unsure about whether this >was a good idea. > >I'd like to call for a vote on what people would prefer, and then ask >Travis to make a final pronouncement before the feature freeze. > > > > This is my thinking about the rand and randn situation: I really like the rand and randn functions. I use them all the time and want quick and easy access to them. In retrospect, the rand and randn functions probably should have been placed in a separate "user" namespace like pylab or numlab or something like that to distinguish it from "numpy the library." But, we don't have something like that in place at this point, so what to do now? I'm opposed to any suggestion to get rid of the rand(3,3) calling syntax. One reason is that I like the syntax for this function and I've used it a lot. It comes from MLab in Numeric and so it is needed to support compatibility with Numeric as well. So, we can't just get rid of it entirely. Another big reason is that there is already a function that uses ONLY the tuple syntax to do exactly the same thing. Look at the docstrings for rand and randn to get the names. If you are opposed to the rand and randn syntax then just ignore those functions and use numpy.random.random_sample(tuple) numpy.random.standard_normal(tuple) So, I'm opposed to getting rid of the *args based syntax. My feelings are weaker regarding adding the capability to rand and randn to accept a tuple. I did test it out and it does seem feasible to add this feature at the cost of an additional comparison. I know Robert is opposed to it but I'm not sure I understand completely why. Please correct me if I'm wrong, but I think it has something to do with making the random_sample and standard_normal functions irrelevant and unnecessary combined with my hypothesis that Robert doesn't like the *args-style syntax. Therefore, adding support to rand and randn for tuples, would make them the default random-number generators and there would be proliferating code that was "harder to read" because of the different usages. Personally, I'm not that opposed to "different" calling conventions, but I respect the opinions of the wider NumPy community. In sum: rand and randn have to live somewhere and accept their current calling convention. I would not be opposed at this point to taking them out of top-level numpy and putting them instead into a "numlab" name-space (for lack of a better name). However, I'm not so sure that making a "numlab" name-space really solves any of the issues that are being debated in this case. So, I'm not going to spend any time doing it... -Travis From myeates at jpl.nasa.gov Tue Jul 11 17:17:18 2006 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Tue, 11 Jul 2006 14:17:18 -0700 Subject: [Numpy-discussion] looking for the Numpy to do it In-Reply-To: References: <44B40ECB.8060408@jpl.nasa.gov> Message-ID: <44B4155E.7080400@jpl.nasa.gov> I bestow upon Sasha the mantle of guru. Sasha wrote: > Here is the solution of a half of the problem: > >>>> a=array([1,2,3,0,40,50,60,0,7,8,9]) >>>> 5+where(logical_and.accumulate(a[5:]!=0)) > array([5, 6]) > > the rest is left as an exercise to the reader :-) > > Hint a[::-1] will reverse a. > > > On 7/11/06, Mathew Yeates wrote: >> I can handle the following problem by iterating through some indices but >> I'm looking for a more elegant solution. >> >> If I have a 1d array, I want to find a contiguous nonzero region about a >> given index. For example, if a=[1,2,3,0,40,50,60,0,7,8,9] and we start >> with the index of 5, then I want the indices 4,5,6 >> >> Any gurus out there? >> >> Mathew >> >> >> >> ------------------------------------------------------------------------- >> >> Using Tomcat but need to do more? Need to support web services, >> security? >> Get stuff done quickly with pre-integrated technology to make your >> job easier >> Download IBM WebSphere Application Server v.1.0.1 based on Apache >> Geronimo >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> > From filip at ftv.pl Tue Jul 11 17:31:56 2006 From: filip at ftv.pl (Filip Wasilewski) Date: Tue, 11 Jul 2006 23:31:56 +0200 Subject: [Numpy-discussion] Quicker image transfer, tobuffer? In-Reply-To: <44B3FDEB.1000909@ieee.org> References: <44B2EC45.8020402@noaa.gov> <44B3FDEB.1000909@ieee.org> Message-ID: <1785255519.20060711233156@gmail.com> Hi Travis, this is a great example of the __array_interface__ usage. I have spotted some problems after patching the Image.py module and trying to display an array created from Image in matplotlib. First issue is a minor one. There is a difference in axis order between ndarray and PIL: def _conv_type_shape(im): shape = im.size[::-1] typ, extra = _MODE_CONV[im.mode] if extra is None: return shape, typ shape += (extra,) return shape, typ The second seems to be more complex and may be a more general. The memory of string created by self.tostring() seems to be deallocated before array is created (v 0.9.9.2788, win). Everything works fine after storing the reference to data, but this probably should be done somewhere else: def __get_array_interface__(self): new = {} shape, typestr = _conv_type_shape(self) new['shape'] = shape new['typestr'] = typestr new['data'] = self.tostring() self._str_data = new['data'] # a dirty hack return new best, fw From oliphant at ee.byu.edu Tue Jul 11 18:01:53 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 11 Jul 2006 16:01:53 -0600 Subject: [Numpy-discussion] Quicker image transfer, tobuffer? In-Reply-To: <1785255519.20060711233156@gmail.com> References: <44B2EC45.8020402@noaa.gov> <44B3FDEB.1000909@ieee.org> <1785255519.20060711233156@gmail.com> Message-ID: <44B41FD1.6090506@ee.byu.edu> Filip Wasilewski wrote: >Hi Travis, > >this is a great example of the __array_interface__ usage. > > > Just to complete the example: With the Image.py patch that adds the __array_interface__ you can do import Image, pylab im = Image.open('somefile.jpg') pylab.imshow(im, origin='lower') and get a nice picture in the matplotlib window (at least if you are running NumPy). -Travis From oliphant at ee.byu.edu Tue Jul 11 18:02:14 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 11 Jul 2006 16:02:14 -0600 Subject: [Numpy-discussion] Quicker image transfer, tobuffer? In-Reply-To: <1785255519.20060711233156@gmail.com> References: <44B2EC45.8020402@noaa.gov> <44B3FDEB.1000909@ieee.org> <1785255519.20060711233156@gmail.com> Message-ID: <44B41FE6.9010908@ee.byu.edu> Filip Wasilewski wrote: >Hi Travis, > >this is a great example of the __array_interface__ usage. > > > >The second seems to be more complex and may be a more general. The >memory of string created by self.tostring() seems to be deallocated >before array is created (v 0.9.9.2788, win). >Everything works fine after storing the reference to data, but this >probably should be done somewhere else: > > def __get_array_interface__(self): > new = {} > shape, typestr = _conv_type_shape(self) > new['shape'] = shape > new['typestr'] = typestr > new['data'] = self.tostring() > self._str_data = new['data'] # a dirty hack > return new > > > This is now fixed in NumPy. The problem was that when the "buffer" interface was used a reference to the object was kept (but not the buffer). In this case it's the reference to the buffer that is needed. -Travis From strawman at astraw.com Tue Jul 11 18:25:36 2006 From: strawman at astraw.com (Andrew Straw) Date: Tue, 11 Jul 2006 15:25:36 -0700 Subject: [Numpy-discussion] Quicker image transfer, tobuffer? In-Reply-To: <44B41FD1.6090506@ee.byu.edu> References: <44B2EC45.8020402@noaa.gov> <44B3FDEB.1000909@ieee.org> <1785255519.20060711233156@gmail.com> <44B41FD1.6090506@ee.byu.edu> Message-ID: <44B42560.4030000@astraw.com> Travis Oliphant wrote: >Filip Wasilewski wrote: > > > >>Hi Travis, >> >>this is a great example of the __array_interface__ usage. >> >> >> >> >> > >Just to complete the example: > >With the Image.py patch that adds the __array_interface__ > >you can do > >import Image, pylab > >im = Image.open('somefile.jpg') >pylab.imshow(im, origin='lower') > >and get a nice picture in the matplotlib window (at least if you are >running NumPy). > > Actually, this has been in MPL for a while. For example, see the image_demo3.py example. You don't need the __array_interface__ for this bit of functionality. From efiring at hawaii.edu Tue Jul 11 18:38:15 2006 From: efiring at hawaii.edu (Eric Firing) Date: Tue, 11 Jul 2006 12:38:15 -1000 Subject: [Numpy-discussion] Quicker image transfer, tobuffer? In-Reply-To: <44B42560.4030000@astraw.com> References: <44B2EC45.8020402@noaa.gov> <44B3FDEB.1000909@ieee.org> <1785255519.20060711233156@gmail.com> <44B41FD1.6090506@ee.byu.edu> <44B42560.4030000@astraw.com> Message-ID: <44B42857.1020907@hawaii.edu> Andrew Straw wrote: > Actually, this has been in MPL for a while. For example, see the > image_demo3.py example. You don't need the __array_interface__ for this > bit of functionality. It's broken. The first problem is that the kw "aspect = 'preserve'" is no longer needed or supported. Removing that (as I will do in svn shortly), I get a somewhat scrambled image. Eric From efiring at hawaii.edu Tue Jul 11 18:55:54 2006 From: efiring at hawaii.edu (Eric Firing) Date: Tue, 11 Jul 2006 12:55:54 -1000 Subject: [Numpy-discussion] Quicker image transfer, tobuffer? In-Reply-To: <44B42857.1020907@hawaii.edu> References: <44B2EC45.8020402@noaa.gov> <44B3FDEB.1000909@ieee.org> <1785255519.20060711233156@gmail.com> <44B41FD1.6090506@ee.byu.edu> <44B42560.4030000@astraw.com> <44B42857.1020907@hawaii.edu> Message-ID: <44B42C7A.7070701@hawaii.edu> Eric Firing wrote: > Andrew Straw wrote: > > >>Actually, this has been in MPL for a while. For example, see the >>image_demo3.py example. You don't need the __array_interface__ for this >>bit of functionality. > > > It's broken. > > The first problem is that the kw "aspect = 'preserve'" is no longer > needed or supported. Removing that (as I will do in svn shortly), I get > a somewhat scrambled image. Correction: I did fix the first problem, and the second problem is not at all what I thought. Instead, the examples/data/lena.jpg file in my svn mpl directory is corrupted. I have no idea why. Looking directly at the version on svn via the svn browser, I see that it is corrupted also. Eric From jdhunter at ace.bsd.uchicago.edu Tue Jul 11 18:55:53 2006 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Tue, 11 Jul 2006 17:55:53 -0500 Subject: [Numpy-discussion] [matplotlib-devel] Quicker image transfer, tobuffer? In-Reply-To: <44B42C7A.7070701@hawaii.edu> (Eric Firing's message of "Tue, 11 Jul 2006 12:55:54 -1000") References: <44B2EC45.8020402@noaa.gov> <44B3FDEB.1000909@ieee.org> <1785255519.20060711233156@gmail.com> <44B41FD1.6090506@ee.byu.edu> <44B42560.4030000@astraw.com> <44B42857.1020907@hawaii.edu> <44B42C7A.7070701@hawaii.edu> Message-ID: <87r70r7nuu.fsf@peds-pc311.bsd.uchicago.edu> >>>>> "Eric" == Eric Firing writes: Eric> Correction: I did fix the first problem, and the second Eric> problem is not at all what I thought. Instead, the Eric> examples/data/lena.jpg file in my svn mpl directory is Eric> corrupted. I have no idea why. Looking directly at the This usually happens whenever Andrew commits -- don't know why (platform dependent new line problem, perhaps?) peds-pc311:~/mpl> svn log | grep astraw|head r2480 | astraw | 2006-06-15 06:33:07 -0500 (Thu, 15 Jun 2006) | 1 line r2430 | astraw | 2006-06-06 15:12:33 -0500 (Tue, 06 Jun 2006) | 1 line r2279 | astraw | 2006-04-10 10:35:31 -0500 (Mon, 10 Apr 2006) | 3 lines r2180 | astraw | 2006-03-20 15:38:12 -0600 (Mon, 20 Mar 2006) | 1 line JDH From josh8912 at yahoo.com Tue Jul 11 19:16:45 2006 From: josh8912 at yahoo.com (JJ) Date: Tue, 11 Jul 2006 23:16:45 +0000 (UTC) Subject: [Numpy-discussion] suggestions for Matrix-related changes Message-ID: Hello. For what its worth, as a newly ex-matlab user I would like to make a few suggestions on use of matrices in numpy. As per earlier discussions, I like the idea of being able to choose matrices as the default (vs arrays). But if possible, it would be nice if all functions etc that took matrices also returned matrices. I know effort has been made on this. Here are my suggestions: 1) is it possible to get the function unique() to work with matrices, perhaps with a unique_rows() function to work with matrices of more than one column? 2) It would be very convienient to have some simple way to delete selected columns of a matrix. For example, in matlab the command is X[:,[3,5,7]]=[] to delete the three selected columns. It would be nice if such a command would also work with selections, as in X[:,A[0,:]<4] = [], where X and A are matrices. For me, the single most frustrating and time-consuming aspect of switching to and using numpy is determing how to select columns/rows of arrays/matrices in different situations. In addition to being time consuming to figure out (relative to Matlab), often the code is quite verbose. I have made a few suggestions on this topic in an earlier post (under the title "Whats wrong with matrices?"). 3) It would be nice if matrices could be used for iterations. For example, if M was a 1 x n matrix, it would be nice to be able to use: for i in M: and iterate over the individual items in M. 4) It would be nice if the linear algebra package and other packages returned matrices if given matrices. For example, if M is a matrix, svd(M) now returns arrays. Just some suggestions. I wish I knew more so I could help implement them. Maybe one day. JJ From fperez.net at gmail.com Tue Jul 11 19:19:18 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 11 Jul 2006 17:19:18 -0600 Subject: [Numpy-discussion] [matplotlib-devel] Quicker image transfer, tobuffer? In-Reply-To: <87r70r7nuu.fsf@peds-pc311.bsd.uchicago.edu> References: <44B3FDEB.1000909@ieee.org> <1785255519.20060711233156@gmail.com> <44B41FD1.6090506@ee.byu.edu> <44B42560.4030000@astraw.com> <44B42857.1020907@hawaii.edu> <44B42C7A.7070701@hawaii.edu> <87r70r7nuu.fsf@peds-pc311.bsd.uchicago.edu> Message-ID: On 7/11/06, John Hunter wrote: > >>>>> "Eric" == Eric Firing writes: > > Eric> Correction: I did fix the first problem, and the second > Eric> problem is not at all what I thought. Instead, the > Eric> examples/data/lena.jpg file in my svn mpl directory is > Eric> corrupted. I have no idea why. Looking directly at the > > This usually happens whenever Andrew commits -- don't know why > (platform dependent new line problem, perhaps?) Is that file tagged as binary in the repo? If it is, it should be impervious to OS-dependent EOL conventions... Cheers, f From filip at ftv.pl Tue Jul 11 19:36:11 2006 From: filip at ftv.pl (Filip Wasilewski) Date: Wed, 12 Jul 2006 01:36:11 +0200 Subject: [Numpy-discussion] Array interface and builtin array.array example Message-ID: <146309645.20060712013611@gmail.com> Hi, the way of accessing data with __array_interface__, as shown by Travis in [1], also works nicely when used with builtin array.array (if someone here is still using it;). Time to convert array.array to ndarray is O(N) but can be made O(1) just by simple subclassing. [1] http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3191164 cheers, fw ----------------------------------------------------------------------- #!/usr/bin/env python # -*- coding: utf-8 -*- import array as _array import sys if sys.byteorder == 'little': _ENDIAN = '<' else: _ENDIAN = '>' _TYPES_CONV ={ 'c': '|u%%d', #character 1 'b': '|i%%d', #signed integer 1 'B': '|u%%d', #unsigned integer 1 'u': '%su%%d' % _ENDIAN, #Unicode character 2 'h': '%si%%d' % _ENDIAN, #signed integer 2 'H': '%su%%d' % _ENDIAN, #unsigned integer 2 'i': '%si%%d' % _ENDIAN, #signed integer 2 (4?) 'I': '%su%%d' % _ENDIAN, #unsigned integer 2 (4?) 'l': '%si%%d' % _ENDIAN, #signed integer 4 'L': '%su%%d' % _ENDIAN, #unsigned integer 4 'f': '%sf%%d' % _ENDIAN, #floating point 4 'd': '%sf%%d' % _ENDIAN, #floating point 8 } class array(_array.array): def __get_array_interface__(self): new = {} shape, typestr = (self.__len__(),), (_TYPES_CONV[self.typecode] % self.itemsize) new['shape'] = shape new['typestr'] = typestr new['data'] = (self.buffer_info()[0], False) # writable return new __array_interface__ = property(__get_array_interface__, None, doc="array interface") if __name__ == '__main__': size = 1000000 typecode = 'f' new = array(typecode, xrange(size)) old = _array.array(typecode, xrange(size)) import numpy from time import clock as time t1 = time() nd = numpy.asarray(new) t1 = time() - t1 #print nd t2 = time() nd = numpy.asarray(old) t2 = time() - t2 #print nd print "new:", t1 print "old:", t2 #EOF From strawman at astraw.com Tue Jul 11 19:38:34 2006 From: strawman at astraw.com (Andrew Straw) Date: Tue, 11 Jul 2006 16:38:34 -0700 Subject: [Numpy-discussion] [matplotlib-devel] Quicker image transfer, tobuffer? In-Reply-To: <87r70r7nuu.fsf@peds-pc311.bsd.uchicago.edu> References: <44B2EC45.8020402@noaa.gov> <44B3FDEB.1000909@ieee.org> <1785255519.20060711233156@gmail.com> <44B41FD1.6090506@ee.byu.edu> <44B42560.4030000@astraw.com> <44B42857.1020907@hawaii.edu> <44B42C7A.7070701@hawaii.edu> <87r70r7nuu.fsf@peds-pc311.bsd.uchicago.edu> Message-ID: <44B4367A.3010708@astraw.com> John Hunter wrote: >>>>>>"Eric" == Eric Firing writes: >>>>>> >>>>>> > > Eric> Correction: I did fix the first problem, and the second > Eric> problem is not at all what I thought. Instead, the > Eric> examples/data/lena.jpg file in my svn mpl directory is > Eric> corrupted. I have no idea why. Looking directly at the > >This usually happens whenever Andrew commits -- don't know why >(platform dependent new line problem, perhaps?) > >peds-pc311:~/mpl> svn log | grep astraw|head >r2480 | astraw | 2006-06-15 06:33:07 -0500 (Thu, 15 Jun 2006) | 1 line >r2430 | astraw | 2006-06-06 15:12:33 -0500 (Tue, 06 Jun 2006) | 1 line >r2279 | astraw | 2006-04-10 10:35:31 -0500 (Mon, 10 Apr 2006) | 3 >lines >r2180 | astraw | 2006-03-20 15:38:12 -0600 (Mon, 20 Mar 2006) | 1 line > > > Hmm -- "usually happens"? I never noticed that. And I'm mystified as to whether the output of svn log shows that. Let me know if I play any more evil-line-ending tricks. Anyhow, I think I fixed the corrupted file issue. I changed the deleted the svn:eol-style property and added the set svn:mime-type property to image/jpg and re-uploaded lena.jpg. I suspect this may have been a victim of the cvs2svn switch, or perhaps I never checked it into cvs properly. Cheers! Andrew From oliphant at ee.byu.edu Tue Jul 11 19:41:55 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 11 Jul 2006 17:41:55 -0600 Subject: [Numpy-discussion] suggestions for Matrix-related changes In-Reply-To: References: Message-ID: <44B43743.5090403@ee.byu.edu> JJ wrote: >Hello. For what its worth, as a newly ex-matlab user I would like to make a few >suggestions on use of matrices in numpy. As per earlier discussions, I like the >idea of being able to choose matrices as the default (vs arrays). But if >possible, it would be nice if all functions etc that took matrices also returned >matrices. I know effort has been made on this. Here are my suggestions: > >1) is it possible to get the function unique() to work with matrices, perhaps >with a unique_rows() function to work with matrices of more than one column? > >2) It would be very convienient to have some simple way to delete selected >columns of a matrix. > This is a good idea. It would be nice to address it at some point. There is a Python syntax for it, but we are not using it yet: del X[...] Of course one of the problems with this syntax (as opposed to a function that returns a new array) is that because X can share it's data with other arrays, you can't just re-size it's memory or other arrays depending on that chunk of memory will be in deep trouble. So, we are probably not going to be able to have a "syntax-style" delete. But, some kind of function that returns an array with specific entries deleted would be nice. >3) It would be nice if matrices could be used for iterations. For example, if M >was a 1 x n matrix, it would be nice to be able to use: for i in M: and >iterate over the individual items in M. > > They can be used as iterators. The problem here is simply convention (rows are iterated over first). We could over-ride the iterator behavior of matrices, though to handle 1xn and nx1 matrices identically if that is desirable. >4) It would be nice if the linear algebra package and other packages returned >matrices if given matrices. For example, if M is a matrix, svd(M) now returns > > Svd returns matrices now. Except for the list of singular values which is still an array. Do you want a 1xn matrix instead of an array? -Travis From oliphant at ee.byu.edu Tue Jul 11 19:43:09 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 11 Jul 2006 17:43:09 -0600 Subject: [Numpy-discussion] Array interface and builtin array.array example In-Reply-To: <146309645.20060712013611@gmail.com> References: <146309645.20060712013611@gmail.com> Message-ID: <44B4378D.7010701@ee.byu.edu> Filip Wasilewski wrote: >Hi, > >the way of accessing data with __array_interface__, as shown by Travis >in [1], also works nicely when used with builtin array.array (if someone >here is still using it;). > >Time to convert array.array to ndarray is O(N) but can be made O(1) just >by simple subclassing. > >[1] http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3191164 > > > This is exactly the kind of thing I'd like to see get into Python. Thanks for picking up the ball.. -Travis From josh8912 at yahoo.com Tue Jul 11 20:00:47 2006 From: josh8912 at yahoo.com (JJ) Date: Wed, 12 Jul 2006 00:00:47 +0000 (UTC) Subject: [Numpy-discussion] bug in stats.binom? Or is it just me. Message-ID: Am I using the wrong syntax for the binom.ppf command, or is there a bug? >>> stats.binom.ppf(.975,100,.5) Traceback (most recent call last): File "", line 1, in ? File "/usr/lib64/python2.4/site-packages/scipy/stats/distributions.py", line 3590, in ppf insert(output,cond,self._ppf(*goodargs) + loc) File "/usr/lib64/python2.4/site-packages/numpy/lib/function_base.py", line 501, in insert return _insert(arr, mask, vals) TypeError: array cannot be safely cast to required type >>> ----------------- The info pages for binom.ppf state that: binom.ppf(q,n,pr,loc=0) - percent point function (inverse of cdf --- percentiles) So I would expect binom.ppf to take three variables. I expected the function to return a number, such as is done in matlab: N = 100 alpha = 0.05 p1 = 0.30 cutoff = binoinv(1-alpha, N, p1) cutoff = 38 Any suggestions? JJ From michael.sorich at gmail.com Tue Jul 11 20:07:56 2006 From: michael.sorich at gmail.com (Michael Sorich) Date: Wed, 12 Jul 2006 09:37:56 +0930 Subject: [Numpy-discussion] suggestions for Matrix-related changes In-Reply-To: References: Message-ID: <16761e100607111707i4d336f9cke1a0b92c0d474f9e@mail.gmail.com> On 7/12/06, JJ wrote: > 2) It would be very convienient to have some simple way to delete selected > columns of a matrix. For example, in matlab the command is X[:,[3,5,7]]=[] to > delete the three selected columns. It would be nice if such a command would > also work with selections, as in X[:,A[0,:]<4] = [], where X and A are matrices. +1. In R negative integers are used for this purpose and a copy of the array is returned. e.g. > x = 1:10 > x [1] 1 2 3 4 5 6 7 8 9 10 > x[-1] [1] 2 3 4 5 6 7 8 9 10 > x[c(-1,-2)] [1] 3 4 5 6 7 8 9 10 > x[-c(1,2)] [1] 3 4 5 6 7 8 9 10 I like the current use of negative indices in numpy, but I do find myself missing the ability to easily make a copy of the array without certain indices. Mike From kwgoodman at gmail.com Tue Jul 11 20:11:12 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 11 Jul 2006 17:11:12 -0700 Subject: [Numpy-discussion] suggestions for Matrix-related changes In-Reply-To: <44B43743.5090403@ee.byu.edu> References: <44B43743.5090403@ee.byu.edu> Message-ID: On 7/11/06, Travis Oliphant wrote: > JJ wrote: > >4) It would be nice if the linear algebra package and other packages returned > >matrices if given matrices. For example, if M is a matrix, svd(M) now returns > > > > > Svd returns matrices now. Except for the list of singular values which > is still an array. Do you want a 1xn matrix instead of an array? That sounds good to me. The same goes for eig and eigh: >> eigval,eigvec = linalg.eig(rand(2,2)) >> eigval array([-0.06035002, 0.14320639]) >> eigvec matrix([[ 0.54799954, -0.83647863], [-0.83647863, -0.54799954]]) From josh8912 at yahoo.com Tue Jul 11 20:23:28 2006 From: josh8912 at yahoo.com (JJ) Date: Wed, 12 Jul 2006 00:23:28 +0000 (UTC) Subject: [Numpy-discussion] suggestions for Matrix-related changes References: <44B43743.5090403@ee.byu.edu> Message-ID: Travis Oliphant ee.byu.edu> writes: > But, some kind of function that returns an array with specific > entries deleted would be nice. I agree. This would be just fine. > We could over-ride the iterator > behavior of matrices, though to handle 1xn and nx1 matrices > identically if that is desirable. I had tried this iteration on a month-old version of numpy and it did not work. I guess this now has been changed. I just updated my copy but have not yet tried it. An over-ride might be nice. But just off the topic, could you get a matrix of real numbers such as A= [[1.0 2.0,3.0]] to be used to select rows/colums as in B[:,A]? I guess this would require a hidden conversion to integers, as well as code to handle selection using a matrix. > Svd returns matrices now. Except for the list of singular values > which is still an array. Do you want a 1xn matrix instead of an > array? I had just tried this with my new version of numpy, but I had used svd as follows: import scipy.linalg as la res = la.svd(M) That returned arrays, but I see that using: res = linalg.svd(M) returns matrices. Apparently, both numpy and scipy have linalg packages, which differ. I did not know that. Whoops. From ckkart at hoc.net Tue Jul 11 21:11:04 2006 From: ckkart at hoc.net (Christian Kristukat) Date: Wed, 12 Jul 2006 01:11:04 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?bdist=5Frpm_failure?= Message-ID: Hi, currently the bdist_rpm build method seems to be quite unstable. It works for some recent svn revisions, for the current 2804, however, not. The error messages begin with: building 'numpy.core.multiarray' extension compiling C sources C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -march=i586 -mtune=i686 -fmessage-length=0 -Wall - D_FORTIFY_SOURCE=2 -g -O2 -g -m32 -march=i586 -mtune=i686 -fmessage-length=0 -D_FORTIFY_SOURCE=2 -fPIC creating build/temp.linux-i686-2.4 creating build/temp.linux-i686-2.4/numpy creating build/temp.linux-i686-2.4/numpy/core creating build/temp.linux-i686-2.4/numpy/core/src compile options: '-Ibuild/src.linux-i686-2.4/numpy/core/src -Inumpy/core/include -Ibuild/src.linux-i686-2.4/n umpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' gcc: numpy/core/src/multiarraymodule.c numpy/core/src/multiarraymodule.c:24:28: error: numpy/noprefix.h: No such file or directory numpy/core/src/multiarraymodule.c:33: error: expected ?=?, ?,?, ?;?, ?asm? or ?__attribute__? before ?*? toke n Regards, Christian From oliphant.travis at ieee.org Tue Jul 11 21:24:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 11 Jul 2006 19:24:01 -0600 Subject: [Numpy-discussion] bdist_rpm failure In-Reply-To: References: Message-ID: <44B44F31.2020508@ieee.org> Christian Kristukat wrote: > Hi, > currently the bdist_rpm build method seems to be quite unstable. It works for > some recent svn revisions, for the current 2804, however, not. > The error messages begin with: > > building 'numpy.core.multiarray' extension > compiling C sources > C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -march=i586 > -mtune=i686 -fmessage-length=0 -Wall - > D_FORTIFY_SOURCE=2 -g -O2 -g -m32 -march=i586 -mtune=i686 -fmessage-length=0 > -D_FORTIFY_SOURCE=2 -fPIC > > creating build/temp.linux-i686-2.4 > creating build/temp.linux-i686-2.4/numpy > creating build/temp.linux-i686-2.4/numpy/core > creating build/temp.linux-i686-2.4/numpy/core/src > compile options: '-Ibuild/src.linux-i686-2.4/numpy/core/src -Inumpy/core/include > -Ibuild/src.linux-i686-2.4/n > umpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' > gcc: numpy/core/src/multiarraymodule.c > numpy/core/src/multiarraymodule.c:24:28: error: numpy/noprefix.h: No such file > or directory > numpy/core/src/multiarraymodule.c:33: error: expected ?=?, ?,?, ?;?, ?asm? or > ?__attribute__? before ?*? toke > n > > Make sure you get rid of the MANIFEST file in the source directory before trying to run sdist or bdist_rpm. The MANIFEST file is not being deleted when it is dated... -Travis From david.huard at gmail.com Tue Jul 11 22:36:09 2006 From: david.huard at gmail.com (David Huard) Date: Tue, 11 Jul 2006 22:36:09 -0400 Subject: [Numpy-discussion] bug in stats.binom? Or is it just me. In-Reply-To: References: Message-ID: <91cf711d0607111936l3e93a50aja3b7235df15d616@mail.gmail.com> There probably is a bug because stats.binom.ppf(.975, 100, .5) crashes the ipython shell. win2k, P3, latest binary release. David 2006/7/11, JJ : > > Am I using the wrong syntax for the binom.ppf command, or is there a bug? > > >>> stats.binom.ppf(.975,100,.5) > Traceback (most recent call last): > File "", line 1, in ? > File "/usr/lib64/python2.4/site-packages/scipy/stats/distributions.py", > line > 3590, in ppf > insert(output,cond,self._ppf(*goodargs) + loc) > File "/usr/lib64/python2.4/site-packages/numpy/lib/function_base.py", > line > 501, in insert > return _insert(arr, mask, vals) > TypeError: array cannot be safely cast to required type > >>> > ----------------- > The info pages for binom.ppf state that: > binom.ppf(q,n,pr,loc=0) > - percent point function (inverse of cdf --- percentiles) > So I would expect binom.ppf to take three variables. > > I expected the function to return a number, such as is done in matlab: > N = 100 > alpha = 0.05 > p1 = 0.30 > cutoff = binoinv(1-alpha, N, p1) > > cutoff = > > 38 > Any suggestions? > > > JJ > > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndarray at mac.com Tue Jul 11 22:38:31 2006 From: ndarray at mac.com (Sasha) Date: Tue, 11 Jul 2006 22:38:31 -0400 Subject: [Numpy-discussion] Array interface and builtin array.array example In-Reply-To: <44B4378D.7010701@ee.byu.edu> References: <146309645.20060712013611@gmail.com> <44B4378D.7010701@ee.byu.edu> Message-ID: I had similar hopes when I submited the array interface patch and announced it on python-dev . I am still waiting for anyone to comment on it :-( On 7/11/06, Travis Oliphant wrote: > Filip Wasilewski wrote: > > >Hi, > > > >the way of accessing data with __array_interface__, as shown by Travis > >in [1], also works nicely when used with builtin array.array (if someone > >here is still using it;). > > > >Time to convert array.array to ndarray is O(N) but can be made O(1) just > >by simple subclassing. > > > >[1] http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3191164 > > > > > > > This is exactly the kind of thing I'd like to see get into Python. > Thanks for picking up the ball.. > > > -Travis > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From david.huard at gmail.com Tue Jul 11 22:42:23 2006 From: david.huard at gmail.com (David Huard) Date: Tue, 11 Jul 2006 22:42:23 -0400 Subject: [Numpy-discussion] suggestions for Matrix-related changes In-Reply-To: References: Message-ID: <91cf711d0607111942n76a5671fy87690ca67ad0bc03@mail.gmail.com> 2006/7/11, JJ : > > 1) is it possible to get the function unique() to work with matrices, > perhaps > with a unique_rows() function to work with matrices of more than one > column? The problem is that applying unique to different rows will return vectors with different lengths. So you could not return an array, much less a matrix. You'd have to return a list of arrays or 1D matrices. David -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Tue Jul 11 22:50:46 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 11 Jul 2006 19:50:46 -0700 Subject: [Numpy-discussion] suggestions for Matrix-related changes In-Reply-To: <91cf711d0607111942n76a5671fy87690ca67ad0bc03@mail.gmail.com> References: <91cf711d0607111942n76a5671fy87690ca67ad0bc03@mail.gmail.com> Message-ID: On 7/11/06, David Huard wrote: > > > 2006/7/11, JJ : > > 1) is it possible to get the function unique() to work with matrices, > perhaps > > with a unique_rows() function to work with matrices of more than one > column? > > > The problem is that applying unique to different rows will return vectors > with different lengths. So you could not return an array, much less a > matrix. ...unless it returned the unique rows instead of the unique elements in each row. So if the matrix is 1 2 2 3 4 5 1 2 2 then the unique rows would be 1 2 2 3 4 5 From robert.kern at gmail.com Wed Jul 12 00:05:35 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 11 Jul 2006 23:05:35 -0500 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <44B414DA.1050300@ee.byu.edu> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> <44B414DA.1050300@ee.byu.edu> Message-ID: Travis Oliphant wrote: > So, I'm opposed to getting rid of the *args based syntax. My feelings > are weaker regarding adding the capability to rand and randn to accept a > tuple. I did test it out and it does seem feasible to add this feature > at the cost of an additional comparison. I know Robert is opposed to it > but I'm not sure I understand completely why. > > Please correct me if I'm wrong, but I think it has something to do with > making the random_sample and standard_normal functions irrelevant and > unnecessary combined with my hypothesis that Robert doesn't like the > *args-style syntax. Therefore, adding support to rand and randn for > tuples, would make them the default random-number generators and there > would be proliferating code that was "harder to read" because of the > different usages. My opposition has much to do with my natural orneriness, I suppose. However, for most of this argument I've never seen a reason why rand() ought to be considered a peer of ones() and zeros() and thus have any need to be "consistent" in this one respect with them. I've always considered ones() and zeros() to be fundamental constructors of basic arrays of an arbitrary dtype that (most likely) you'll be mangling up immediately. I've seen rand() and all of the other RandomState methods as simply functions that return arrays. In that respect, they have more in common with arange() than anything else. However, I've come to realize that because rand() is exposed in the top-level namespace, other people probably *are* seeing it as another fundamental constructor of arrays (albeit of a single dtype). When I was writing the package, I never even considered that some of its functions might be imported into the top-level namespace. I'm somewhat more sympathetic to the confusion caused by the rand()-as-constructor viewpoint, now. That's why my currently preferred compromise position is to remove rand() from numpy.* and leave it in numpy.random*. I also have found that I just don't use those functions convenient in "real" code. They usually come into play when I need one or lots of arbitrary, but non-degenerate-with-high-probability arrays to as a test or demo input to another piece of code. Otherwise, I probably need something more sophisticated. Manually shifting and scaling standard variates makes the code more verbose without making it more comprehensible. I do indeed dislike functions that try to be clever in determining what the user wanted by the number and types of arguments provided. Even if they are easy to implement robustly (as in this case), I think that it makes explaining the function more complicated. It also makes the same function be called in two very different ways; I find this break in consistency rather more egregious than two different functions, used for different purposes in different circumstances, being called in two different ways. In my experience, it's much more likely to cause confusion. I'm currently trudging my way through a mountain of someone else's C++ code at the moment. There are good uses of C++'s function overloading, for example to do type polymorphism on the same number of arguments when templates don't quite cut it. However, if it is abused to create fundamentally different ways to call the same function, I've found that the readability drops rapidly. I have not been a happy camper. I have yet to see a good exposition of the actual *problems* the current situation is causing. Most of the argument in favor of changing anything has been a call for consistency. However, consistency can never be an end of itself. It must always be considered a means to achieve a comprehensible, flexible, usable API. But it's only one of several means to that goal, and even that goal must be weighed against other goals, too. When real problems were discussed, I didn't find the solution to fit those problems. Alan said that it was an annoyance (at the very least) to have to teach students that rand() is called differently from ones(). The answer here is to not teach rand(); use the functions that follow the one convention that you want to teach. Also teach the students to read docstrings so if they come across rand() in their copious spare time, they know what's what. Another almost-compelling-to-me real problem was someone saying that they always had to pause when writing rand() and think about what the calling convention was. My answer is similar: don't use rand(). This problem, as described, is a "write-only" problem; if you want consistency in the calling convention between ones() et al. and your source of random numbers, it's there. You might run into uses of rand() in other people's code, but it'll never confuse you as to what the calling convention is. However, I do think that the presence of rand() in the numpy.* namespace is probably throwing people off. They default to rand() and chafe against its API even though they'd be much happier with the tuple-based API. But mostly my opposition follows from this observation: making rand() cleverly handle its argument tuple *does not solve the problem*. A polymorphic rand() will *still* be inconsistent with ones() and zeros() because ones() and zeros() won't be polymorphic, too. And they can't be really; the fact that they take other arguments besides the shape tuple makes the implementation and use idioms rather harder than for rand(). And mark my words, if we make rand() polymorphic, we will get just as many newbies coming to the list asking why ones(3, 4) doesn't work. I've already described how I feel that this would just trade one inconsistency (between different functions) for another (between different uses of the same function). I find the latter much worse. To summarize: while I'd rather just leave things as they are, I realize that feeling is more from spite than anything else, and I'm above that. Mostly. I *do* think that the brouhaha will rapidly diminish if rand() and rand() are simply removed from numpy.* and left in numpy.random.* where they belong. While I'm sure that some who have been pushing for consistency the hardest will still grumble a bit, I strongly suspect that no one would have thought to complain if this had been the configuration from the very beginning. Okay, now I think I've officially spent more time on this email than I ever did using or implementing rand(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From josh8912 at yahoo.com Wed Jul 12 01:01:14 2006 From: josh8912 at yahoo.com (JJ) Date: Tue, 11 Jul 2006 22:01:14 -0700 (PDT) Subject: [Numpy-discussion] suggestions for Matrix-related changes In-Reply-To: Message-ID: <20060712050114.90890.qmail@web51704.mail.yahoo.com> > ...unless it returned the unique rows instead of the > unique elements > in each row. So if the matrix is > > 1 2 2 > 3 4 5 > 1 2 2 > > then the unique rows would be > > 1 2 2 > 3 4 5 > Hello Keith. Yes, that is what I mean also. JJ __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From fperez.net at gmail.com Wed Jul 12 02:02:26 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 12 Jul 2006 00:02:26 -0600 Subject: [Numpy-discussion] bdist_rpm failure In-Reply-To: <44B44F31.2020508@ieee.org> References: <44B44F31.2020508@ieee.org> Message-ID: On 7/11/06, Travis Oliphant wrote: > Make sure you get rid of the MANIFEST file in the source directory > before trying to run sdist or bdist_rpm. The MANIFEST file is not being > deleted when it is dated... Since both numpy/scipy have a MANIFEST.in, this bit of code (from ipython's setup.py) is probably a good idea to put in: # BEFORE importing distutils, remove MANIFEST. distutils doesn't properly # update it when the contents of directories change. if os.path.exists('MANIFEST'): os.remove('MANIFEST') I committed this as I don't see any problem with it (ipython has used it for years). Feel free to revert if you object. I also added #!/usr/bin/env python to both of them. They were set as executable scripts but without a proper shebang line, causing the interesting mis-behavior of being treated like shell scripts. Which means the first import os line would simply give you a weird-looking cross-hairs cursor, and a gigantic file called 'os' sitting on your directory once you clicked. Why? That's the ImageMagick 'import' command, which takes a screenshot in X bitmap format :) This one bit me a few times, so I decided to also fix it, but feel free to revert if you see a good reason for the existing behavior. Cheers, f From wbaxter at gmail.com Wed Jul 12 02:09:55 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 12 Jul 2006 15:09:55 +0900 Subject: [Numpy-discussion] Args for rand and randn, and workarounds Message-ID: > > And mark my words, if we make rand() polymorphic, > we will get just as many newbies coming to the list asking why ones(3, 4) > doesn't work. > I think you're probably right there, at least for Matlab converts. Matlab allows either way of calling for all of rand, ones, zeros, eye, and in general tries to make *every* function accept *any* possible combination of arguments that might make sense. So having rand() accept both, but not the others, will undoubtedly be unexpected to Matlab users. And the argument against funky overloading will be weaker since they'll be able to say "well rand() does it? why can't ones()?" Fortunately it's pretty easy to define one's own custom versions of these little helper functions to make them just the way you like them. Or for rand(), just "from numpy.random import uniform_sample as rand" if that's the version you like. This isn't a panacea, but you can then collect all these into a little customization module, say "mynum.py" which is somewhere on your PYTHONPATH, and then set up all your customizations with a "from mynum import *" You can even go further if you wish and define the PYTHONSTARTUP environment variable to point to a file that does "from mynum import *" so that every time you start up the interactive interpreter you have your definitions there already. I'm sure this is all well known to you long-time Python'ers out there, but I thought I'd mention it since there are probably other numpy users out there who, like me, are pretty new to Python. --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Wed Jul 12 02:52:57 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 12 Jul 2006 00:52:57 -0600 Subject: [Numpy-discussion] Args for rand and randn, and workarounds In-Reply-To: References: Message-ID: <44B49C49.1040906@ieee.org> Bill Baxter wrote: > > And mark my words, if we make rand() polymorphic, > > we will get just as many newbies coming to the list asking why > ones(3, 4) > > doesn't work. > > > I think you're probably right there, at least for Matlab converts. > Matlab allows either way of calling for all of rand, ones, zeros, eye, > and in general tries to make *every* function accept *any* possible > combination of arguments that might make sense. So having rand() > accept both, but not the others, will undoubtedly be unexpected to > Matlab users. And the argument against funky overloading will be > weaker since they'll be able to say "well rand() does it? why can't > ones()?" > > Fortunately it's pretty easy to define one's own custom versions of > these little helper functions to make them just the way you like > them. Or for rand(), just "from numpy.random import uniform_sample > as rand" if that's the version you like. Because of this. I've removed the global_namespace functions (fft, ifft, rand, and randn) from numpy. They are *no longer* in the top-level name-space. If you want them, setup a startup-file appropriately. -Travis From a.u.r.e.l.i.a.n at gmx.net Wed Jul 12 03:10:40 2006 From: a.u.r.e.l.i.a.n at gmx.net (Johannes Loehnert) Date: Wed, 12 Jul 2006 09:10:40 +0200 Subject: [Numpy-discussion] suggestions for Matrix-related changes In-Reply-To: <16761e100607111707i4d336f9cke1a0b92c0d474f9e@mail.gmail.com> References: <16761e100607111707i4d336f9cke1a0b92c0d474f9e@mail.gmail.com> Message-ID: <200607120910.40686.a.u.r.e.l.i.a.n@gmx.net> On Wednesday 12 July 2006 02:07, Michael Sorich wrote: > On 7/12/06, JJ wrote: > > 2) It would be very convienient to have some simple way to delete > > selected columns of a matrix. For example, in matlab the command is > > X[:,[3,5,7]]=[] to delete the three selected columns. It would be nice > > if such a command would also work with selections, as in X[:,A[0,:]<4] = > > [], where X and A are matrices. > (...) > I like the current use of negative indices in numpy, but I do find > myself missing the ability to easily make a copy of the array without > certain indices. Maybe use complex numbers? draft: >>> a = arange(10) >>> print a[ 3j ] [ 0 1 2 4 5 6 7 8 9 ] Johannes From wbaxter at gmail.com Wed Jul 12 04:24:53 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 12 Jul 2006 17:24:53 +0900 Subject: [Numpy-discussion] Testing a self-compiled numpy In-Reply-To: <44B2614E.7060101@enthought.com> References: <44B2614E.7060101@enthought.com> Message-ID: On 7/10/06, Bryce Hendrix wrote: > > Keith Goodman wrote: > > Doesn't --prefix=/install/numpy/here/ work on windows? > > For our Windows Enthon project, the build command is > > setup.py config --compiler=mingw32 build --compiler=mingw32 install -- > prefix=path_to_where_numpy_goes. > Thanks for the info. I added it to the "building numpy/scipy on windows" wiki. Still looking for help on how to get SciPy to build on Windows if anyone knows anything about that. I posted my symptoms to the SciPy list, but got no response. --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Wed Jul 12 04:47:15 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 12 Jul 2006 17:47:15 +0900 Subject: [Numpy-discussion] Args for rand and randn, and workarounds In-Reply-To: <44B49C49.1040906@ieee.org> References: <44B49C49.1040906@ieee.org> Message-ID: On 7/12/06, Travis Oliphant wrote: > > > Because of this. I've removed the global_namespace functions (fft, > ifft, rand, and randn) from numpy. They are *no longer* in the > top-level name-space. If you want them, setup a startup-file > appropriately. > Any hints as to where we can find them now? (more generally -- how is one supposed to find stuff in big python packages to begin with? Like -- "show me anything with 'rand' in the name".) Also, numpy.matlib.rand() needs to be updated to point to the new location as well. --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Wed Jul 12 05:00:56 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Wed, 12 Jul 2006 11:00:56 +0200 Subject: [Numpy-discussion] Args for rand and randn, and workarounds In-Reply-To: References: <44B49C49.1040906@ieee.org> Message-ID: <20060712090056.GA21547@mentat.za.net> On Wed, Jul 12, 2006 at 05:47:15PM +0900, Bill Baxter wrote: > On 7/12/06, Travis Oliphant wrote: > > > Because of this. I've removed the global_namespace functions (fft, > ifft, rand, and randn) from numpy. They are *no longer* in the > top-level name-space. If you want them, setup a startup-file > appropriately. > > > > Any hints as to where we can find them now? > (more generally -- how is one supposed to find stuff in big python packages to > begin with? Like -- "show me anything with 'rand' in the name".) It is in numpy.random.rand. If you are using ipython, you can do import numpy numpy.*rand*? which will print out a list of all members that contain "rand". In this case numpy.random. To do a deeper search, use pydoc. You can start it with pydoc -g which should pop up a graphical interface, where you can type in a search term. Click on the results to open the appropriate documentation web page. > Also, numpy.matlib.rand() needs to be updated to point to the new location as > well. Already fixed in SVN. Cheers St?fan From svetosch at gmx.net Wed Jul 12 05:44:46 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Wed, 12 Jul 2006 11:44:46 +0200 Subject: [Numpy-discussion] suggestions for Matrix-related changes In-Reply-To: References: <44B43743.5090403@ee.byu.edu> Message-ID: <44B4C48E.5050602@gmx.net> JJ schrieb: > Travis Oliphant ee.byu.edu> writes: > >> Svd returns matrices now. Except for the list of singular values >> which is still an array. Do you want a 1xn matrix instead of an >> array? Although I'm a matrix supporter, I'm not sure here. Afaics the pro argument is to have *everything* a matrix when you're in that camp. Fair enough. But then it's already not clear if you want a row or a column, and you carry an extra dimension around, which is sometimes annoying e.g. for cumulation of the values, which I do a lot (for eigenvalues, that is). So for my personal use I came to the conclusion that the status quo of numpy (array for the value list, matrix for the decomp) is just fine. So maybe the people in favor of values-in-1xn-matrices can tell why they need to matrix-multiply the value array afterwards, because that's the only benefit I can see here. > > I had just tried this with my new version of numpy, but I had used svd > as follows: > import scipy.linalg as la > res = la.svd(M) > That returned arrays, but I see that using: > res = linalg.svd(M) > returns matrices. Apparently, both numpy and scipy have linalg > packages, which differ. I did not know that. Whoops. > I'm trying to get by with numpy (good that kron was brought over!), but eventually I will need scipy -- I was hoping that all the matrix discussion in the numpy list implicitly applied to scipy as well. Is that not true? Cheers, Sven From svetosch at gmx.net Wed Jul 12 06:21:05 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Wed, 12 Jul 2006 12:21:05 +0200 Subject: [Numpy-discussion] Args for rand and randn, and workarounds In-Reply-To: <44B49C49.1040906@ieee.org> References: <44B49C49.1040906@ieee.org> Message-ID: <44B4CD11.70105@gmx.net> Travis Oliphant schrieb: > > Because of this. I've removed the global_namespace functions (fft, > ifft, rand, and randn) from numpy. They are *no longer* in the > top-level name-space. If you want them, setup a startup-file > appropriately. > Ok I'm glad that's settled; however, do I understand correctly (after looking at the new numpy.matlib for the first time), that numpy.matlib.rand() still doesn't accept tuples, and that the tuple-accepting "full" version is not exposed in numpy.matlib? If so, it would be perfect if numpy.matlib could be augmented by a matrix analogue pointing to numpy.random.random() or whatever its full name is. Then the situation for matrix users is actually quite nice, just have to import everything from numpy.matlib. Thanks, Sven From T.Menneer at soton.ac.uk Wed Jul 12 06:57:48 2006 From: T.Menneer at soton.ac.uk (Tamaryn Menneer) Date: Wed, 12 Jul 2006 11:57:48 +0100 Subject: [Numpy-discussion] "ImportError: No module named numeric" Message-ID: <200607121057.k6CAvrCc028781@mta1.iss.soton.ac.uk> Hi I'm running Python 2.4 on Windows XP. I've installed NumPy, and run the simple test of "import numeric" but I get the error "ImportError: No module named numeric" in return. The module numeric is located in C:\Python24\Lib\site-packages\numpy\core, but even when I set PYTHONPATH to this it returns the same error. Imports of modules in the Lib directory work fine, but everything I've tried within site-packages results the no module error. I originally installed Python24 in Program Files, and thought this might be causing the problem with a hardwired pathname somewhere, but having uninstalled and reinstalled directly on the C drive, I still get the same problem. Any guidance would be very much appreciated. I'm virtually a complete newbie to all this, but I'll try my best to understand any responses! Thank you :-) Tam. From svetosch at gmx.net Wed Jul 12 07:02:13 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Wed, 12 Jul 2006 13:02:13 +0200 Subject: [Numpy-discussion] "ImportError: No module named numeric" In-Reply-To: <200607121057.k6CAvrCc028781@mta1.iss.soton.ac.uk> References: <200607121057.k6CAvrCc028781@mta1.iss.soton.ac.uk> Message-ID: <44B4D6B5.3090605@gmx.net> Tamaryn Menneer schrieb: > Hi > > I'm running Python 2.4 on Windows XP. I've installed NumPy, and run the > simple test of "import numeric" but I get the error "ImportError: No module > named numeric" in return. The module numeric is located in > C:\Python24\Lib\site-packages\numpy\core, but even when I set PYTHONPATH to > this it returns the same error. Imports of modules in the Lib directory work > fine, but everything I've tried within site-packages results the no module > error. > > numpy is not numeric, so just do "import numpy" to test the install. Just ignore all the dir structure under ...\numpy\, as a user you don't usually need to look at it. -sven From stefan at sun.ac.za Wed Jul 12 07:03:43 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Wed, 12 Jul 2006 13:03:43 +0200 Subject: [Numpy-discussion] "ImportError: No module named numeric" In-Reply-To: <200607121057.k6CAvrCc028781@mta1.iss.soton.ac.uk> References: <200607121057.k6CAvrCc028781@mta1.iss.soton.ac.uk> Message-ID: <20060712110343.GA30468@mentat.za.net> Hi Tamaryn On Wed, Jul 12, 2006 at 11:57:48AM +0100, Tamaryn Menneer wrote: > Hi > > I'm running Python 2.4 on Windows XP. I've installed NumPy, and run the > simple test of "import numeric" but I get the error "ImportError: No module > named numeric" in return. The module numeric is located in > C:\Python24\Lib\site-packages\numpy\core, but even when I set PYTHONPATH to > this it returns the same error. Imports of modules in the Lib directory work > fine, but everything I've tried within site-packages results the no module > error. Set your PYTHONPATH to c:\Python24\Lib\site-packages and then try from numpy import oldnumeric as numeric > Any guidance would be very much appreciated. I'm virtually a complete newbie > to all this, but I'll try my best to understand any responses! If you are just starting out with numpy, you shouldn't need to import the numeric module at all. Take a look at the documentation on http://www.scipy.org/Documentation Good luck! St?fan From theller at python.net Wed Jul 12 07:29:56 2006 From: theller at python.net (Thomas Heller) Date: Wed, 12 Jul 2006 13:29:56 +0200 Subject: [Numpy-discussion] Array interface and builtin array.array example In-Reply-To: References: <146309645.20060712013611@gmail.com> <44B4378D.7010701@ee.byu.edu> Message-ID: Sasha schrieb: > I had similar hopes when I submited the array interface patch > > and announced it on python-dev > . > > I am still waiting for anyone to comment on it :-( > The *relevant* comment is here: http://aspn.activestate.com/ASPN/Mail/Message/python-dev/3078372 Thomas From aisaac at american.edu Wed Jul 12 09:42:11 2006 From: aisaac at american.edu (Alan G Isaac) Date: Wed, 12 Jul 2006 09:42:11 -0400 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at><44B414DA.1050300@ee.byu.edu> Message-ID: Robert makes his case clearly and persuasively. Without pretending to challenge his argument in any way, I would just like to clarify what is at issue for some of the teaching crowd (or for me in any case). - Get up and running very quickly even with students who lack a programming background. This means having rand() and randn() in the top-level namespace is nice, since I use them early and often. - Avoid confusion and frustration. This is the basis for having a "consistent" calling convention for array constructors (pace Robert's arguments about consistency). On Tue, 11 Jul 2006, Robert Kern apparently wrote: > And mark my words, if we make rand() polymorphic, we will > get just as many newbies coming to the list asking why > ones(3, 4) doesn't work. That is plausible. If polymorphism is chosen for rand() and randn(), I suppose I would address this by documenting the current API as present for backwards compatability only. That allows a quick answer, but perhaps does not preclude the questions. Cheers, Alan Isaac From karol.langner at kn.pl Wed Jul 12 09:40:36 2006 From: karol.langner at kn.pl (Karol Langner) Date: Wed, 12 Jul 2006 15:40:36 +0200 Subject: [Numpy-discussion] Array interface and builtin array.array example In-Reply-To: References: <146309645.20060712013611@gmail.com> Message-ID: <200607121540.37465.karol.langner@kn.pl> On Wednesday 12 July 2006 13:29, Thomas Heller wrote: > Sasha schrieb: > > I had similar hopes when I submited the array interface patch > > >id=5470&atid=305470> and announced it on python-dev > > . > > > > I am still waiting for anyone to comment on it :-( > > The *relevant* comment is here: > > http://aspn.activestate.com/ASPN/Mail/Message/python-dev/3078372 > > Thomas A related PEP _is_ being prepared - it can be found at http://svn.scipy.org/svn/PEP/ (created by Travis). In fact, I am working on it for Google Summer of Code, as I wrote in a different thread. I think that if everyone interested in getting some kind of array interface into Python core focused on a common effort, it would be more effecient and probably make a stronger impression. I created a page about it at http://scipy.org/BaseArray, and will be adding more information very soon. Contribution and comments from annyone interested are very welcome. I really hope we can get this moving allong, and do it right. Karol -- written by Karol Langner ?ro lip 12 15:31:09 CEST 2006 From theller at python.net Wed Jul 12 10:00:59 2006 From: theller at python.net (Thomas Heller) Date: Wed, 12 Jul 2006 16:00:59 +0200 Subject: [Numpy-discussion] Array interface and builtin array.array example In-Reply-To: <200607121540.37465.karol.langner@kn.pl> References: <146309645.20060712013611@gmail.com> <200607121540.37465.karol.langner@kn.pl> Message-ID: Karol Langner schrieb: > On Wednesday 12 July 2006 13:29, Thomas Heller wrote: >> Sasha schrieb: >> > I had similar hopes when I submited the array interface patch >> > > >id=5470&atid=305470> and announced it on python-dev >> > . >> > >> > I am still waiting for anyone to comment on it :-( >> >> The *relevant* comment is here: >> >> http://aspn.activestate.com/ASPN/Mail/Message/python-dev/3078372 >> >> Thomas > > A related PEP _is_ being prepared - it can be found at > http://svn.scipy.org/svn/PEP/ (created by Travis). I know. But, hiding it somewhere on the internet doesn't lead anywhere. You (the authors) should follow the official PEP process, instead: submit an official PEP, post it for feedback, and so on. Thomas From karol.langner at kn.pl Wed Jul 12 10:15:23 2006 From: karol.langner at kn.pl (Karol Langner) Date: Wed, 12 Jul 2006 16:15:23 +0200 Subject: [Numpy-discussion] Array interface and builtin array.array example In-Reply-To: References: <146309645.20060712013611@gmail.com> <200607121540.37465.karol.langner@kn.pl> Message-ID: <200607121615.23441.karol.langner@kn.pl> On Wednesday 12 July 2006 16:00, Thomas Heller wrote: > Karol Langner schrieb: > > On Wednesday 12 July 2006 13:29, Thomas Heller wrote: > >> Sasha schrieb: > >> > I had similar hopes when I submited the array interface patch > >> > >> >up_ id=5470&atid=305470> and announced it on python-dev > >> > . > >> > > >> > I am still waiting for anyone to comment on it :-( > >> > >> The *relevant* comment is here: > >> > >> http://aspn.activestate.com/ASPN/Mail/Message/python-dev/3078372 > >> > >> Thomas > > > > A related PEP _is_ being prepared - it can be found at > > http://svn.scipy.org/svn/PEP/ (created by Travis). > > I know. But, hiding it somewhere on the internet doesn't lead anywhere. > You (the authors) should follow the official PEP process, instead: > submit an official PEP, post it for feedback, and so on. I haven't actually thought about this, but I guess that is will happen sooner or later. I started working on the code in the scipt svn repository, and I'm not to well oriented how the dev procedure works. Maybe Travis can tell us about long-term strategy for the PEP, if there already is some? There was a PEP back in 2001 concerning MD-arrays (http://www.python.org/dev/peps/pep-0209/), which is was even featured in the "PEP parade" (http://www.python.org/doc/essays/pepparade.html), but got deferred, abandoned, withdrawn, or something else. Karol -- written by Karol Langner ?ro lip 12 16:04:22 CEST 2006 From jnc at ecs.soton.ac.uk Wed Jul 12 12:37:41 2006 From: jnc at ecs.soton.ac.uk (John Carter) Date: Wed, 12 Jul 2006 17:37:41 +0100 Subject: [Numpy-discussion] Building SciPy under Windows attn. Bill Baxter Message-ID: <7.0.1.0.0.20060712172713.0b628218@ecs.soton.ac.uk> I had problems building SciPy and NumPy under Windows. They went away when I stopped using the stable version of gcc and used 3.4.5 I think the problem was related to differences in cygwin and mingw32. SciPy built and everything I've needed works but the self test fails. John Dr. John N. Carter jnc at ecs.soton.ac.uk ISIS http://www.ecs.soton.ac.uk/~jnc/ From mark at mitre.org Wed Jul 12 13:42:44 2006 From: mark at mitre.org (Mark Heslep) Date: Wed, 12 Jul 2006 13:42:44 -0400 Subject: [Numpy-discussion] Numpy array from ctypes pointer object? Message-ID: <44B53494.2000005@mitre.org> I don't see a clean way to create a numpy array from a ctypes pointer object. Is the __array_interface_ in ctypes the thing thats missing needed to make this happen? I've followed Albert's Scipy cookbook on ctypes here http://www.scipy.org/Cookbook/Ctypes <-- getting dated now for numpy SVN but the idea is clear This shows clean ways to 1) Create a ctype array object from a numpy array: npptr= nparray.__array_interface__['data'][0] ctarray = (c_double * nparray.size).from_address(npptr) 2) Create a numpy array from a ctypes array (not a pointer): ctarray = (c_double * 2)() nparray = N.array( ctarray ) But if I'm starting with say, a POINTER(c_int) and a separate size argument that's returned from a ctypes foreign function or provided by _fields_ element in a ctypes structure, then I'm unclear how to provide the pointer in Python to numpy. The numpy methods or creating an array in Python as I see it are basically two: N.array, N.asarray <-- require a sequence object N.fromstring, N.frombuffer <-- not available unless given c_char_p and even thats wrong if I don't want zero termination. The attributes in nparray.__array_interface_ are not writable, so no joy there. On the C side the PyArray_SimpleNewFromData( ..dimensions, ...data ptr) C API does the job nicely. Is there a ctypes paradigm for SimpleNew...? Mark From oliphant.travis at ieee.org Wed Jul 12 13:47:50 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 12 Jul 2006 11:47:50 -0600 Subject: [Numpy-discussion] Numpy array from ctypes pointer object? In-Reply-To: <44B53494.2000005@mitre.org> References: <44B53494.2000005@mitre.org> Message-ID: <44B535C6.1080307@ieee.org> Mark Heslep wrote: > I don't see a clean way to create a numpy array from a ctypes pointer object. Is the __array_interface_ in ctypes the thing thats missing needed to make this happen? I've followed Albert's Scipy cookbook on ctypes here > > On the C side the PyArray_SimpleNewFromData( ..dimensions, ...data ptr) C API does the job nicely. Is there a ctypes paradigm for SimpleNew...? > Can you somehow call this function using ctypes? -Travis From ndarray at mac.com Wed Jul 12 14:33:40 2006 From: ndarray at mac.com (Sasha) Date: Wed, 12 Jul 2006 14:33:40 -0400 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> <44B414DA.1050300@ee.byu.edu> Message-ID: Let me repeat my suggestion that was lost in this long thread: Add rands(shape, dtype=float, min=default_min(dtype), max=default_max(dtype)) to the top level. Suitable defaults can be discussed. A more flexible variation could be rands(shape, dtype=float, algorithm=default_algorithm(dtype)), but that would probably be an overkill. I think this will help teaching: rands is similar to zeros and ones, but with few bells and whistles to be covered in the graduate course. On 7/12/06, Alan G Isaac wrote: > Robert makes his case clearly and persuasively. > Without pretending to challenge his argument in any way, > I would just like to clarify what is at issue > for some of the teaching crowd (or for me in any case). > > - Get up and running very quickly even with students who > lack a programming background. This means having rand() > and randn() in the top-level namespace is nice, since > I use them early and often. > - Avoid confusion and frustration. This is the basis for > having a "consistent" calling convention for array > constructors (pace Robert's arguments about consistency). > > > On Tue, 11 Jul 2006, Robert Kern apparently wrote: > > And mark my words, if we make rand() polymorphic, we will > > get just as many newbies coming to the list asking why > > ones(3, 4) doesn't work. > > That is plausible. > If polymorphism is chosen for rand() and randn(), I suppose > I would address this by documenting the current API as > present for backwards compatability only. That allows > a quick answer, but perhaps does not preclude the questions. > > Cheers, > Alan Isaac > > > > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From mark at mitre.org Wed Jul 12 15:06:14 2006 From: mark at mitre.org (Mark Heslep) Date: Wed, 12 Jul 2006 15:06:14 -0400 Subject: [Numpy-discussion] Numpy array from ctypes pointer object? In-Reply-To: <44B535C6.1080307@ieee.org> References: <44B53494.2000005@mitre.org> <44B535C6.1080307@ieee.org> Message-ID: <44B54826.2010002@mitre.org> Travis Oliphant wrote: > Mark Heslep wrote: >> I don't see a clean way to create a numpy array from a ctypes pointer >> object. Is the __array_interface_ in ctypes the thing thats missing >> needed to make this happen? I've followed Albert's Scipy cookbook >> on ctypes here >> >> On the C side the PyArray_SimpleNewFromData( ..dimensions, ...data >> ptr) C API does the job nicely. Is there a ctypes paradigm for >> SimpleNew...? >> > Can you somehow call this function using ctypes? > > -Travis That might work, though indirectly. As I think I understand from ctypes docs: Ctypes uses functions exposed in a shared library, macros existing only a header are not available. If its PyArray... is a macro then I a) need to compile and make a little library directly from arrayobject.h or b) need to use the root function upon which the macro is based, PyArrayNew? ctypes has built-in access to the functions in the core python library, but Im not sure which library contains all the PyArray API calls, since its all typically hidden by distutils and setup.py Mark From theller at python.net Wed Jul 12 15:15:13 2006 From: theller at python.net (Thomas Heller) Date: Wed, 12 Jul 2006 21:15:13 +0200 Subject: [Numpy-discussion] Numpy array from ctypes pointer object? In-Reply-To: <44B54826.2010002@mitre.org> References: <44B53494.2000005@mitre.org> <44B535C6.1080307@ieee.org> <44B54826.2010002@mitre.org> Message-ID: Mark Heslep schrieb: > Travis Oliphant wrote: >> Mark Heslep wrote: >>> I don't see a clean way to create a numpy array from a ctypes pointer >>> object. Is the __array_interface_ in ctypes the thing thats missing >>> needed to make this happen? I've followed Albert's Scipy cookbook >>> on ctypes here >>> >>> On the C side the PyArray_SimpleNewFromData( ..dimensions, ...data >>> ptr) C API does the job nicely. Is there a ctypes paradigm for >>> SimpleNew...? >>> >> Can you somehow call this function using ctypes? >> >> -Travis > That might work, though indirectly. As I think I understand from ctypes > docs: Ctypes uses functions exposed in a shared library, macros > existing only a header are not available. If its PyArray... is a macro > then I a) need to compile and make a little library directly from > arrayobject.h or b) need to use the root function upon which the macro > is based, PyArrayNew? Sure, macros will not work. > ctypes has built-in access to the functions in the core python library, > but Im not sure which library contains all the PyArray API calls, since > its all typically hidden by distutils and setup.py On Linux it doesn't matter, since you don't need to know which library exports a function (If I understand shared libs in Linux correctly). At least dlsym can be called with a NULL handle. On windows, functions can only be retrieved from exactly the library that exposes them. Plus, the function must explicitly marked exported either by compiler directives in the source code of by listing them in a .def file. Thomas From oliphant.travis at ieee.org Wed Jul 12 15:23:16 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 12 Jul 2006 13:23:16 -0600 Subject: [Numpy-discussion] Numpy array from ctypes pointer object? In-Reply-To: <44B53494.2000005@mitre.org> References: <44B53494.2000005@mitre.org> Message-ID: <44B54C24.2070301@ieee.org> Mark Heslep wrote: > I don't see a clean way to create a numpy array from a ctypes pointer object. Is the __array_interface_ in ctypes the thing thats missing needed to make this happen? I've followed Albert's Scipy cookbook on ctypes here > > http://www.scipy.org/Cookbook/Ctypes <-- getting dated now for numpy SVN but the idea is clear > > This shows clean ways to > 1) Create a ctype array object from a numpy array: > npptr= nparray.__array_interface__['data'][0] > ctarray = (c_double * nparray.size).from_address(npptr) > > 2) Create a numpy array from a ctypes array (not a pointer): > ctarray = (c_double * 2)() > nparray = N.array( ctarray ) > > But if I'm starting with say, a POINTER(c_int) and a separate size argument that's returned from a ctypes foreign function or provided by _fields_ element in a ctypes structure, then I'm unclear how to provide the pointer in Python to numpy. The numpy methods or creating an array in Python as I see it are basically two: > > N.array, N.asarray <-- require a sequence object > N.fromstring, N.frombuffer <-- not available unless given c_char_p and even thats wrong if I don't want zero termination. > The problem here is that from Python NumPy has no way to create an ndarray from a pointer. Doing this creates a situtation where it is unclear who owns the memory. It is probably best to wrap the pointer into some kind of object exposing the buffer protocol and then pass that to frombuffer (or ndarray.__new__). When an ndarray is using memory that is not its own, it expects another object to be "in charge" of that memory and the ndarray will point its base attribute to it and increment its reference count. What should the object that is "in charge" of the memory be? Perhaps a suitable utility function could be created that can work with ctypes to create ndarrays from ctypes memory locations and either own or disown the data. This needs to be thought through a bit, however. -Travis > The attributes in nparray.__array_interface_ are not writable, so no joy there. > > On the C side the PyArray_SimpleNewFromData( ..dimensions, ...data ptr) C API does the job nicely. Is there a ctypes paradigm for SimpleNew...? > > Mark > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From oliphant.travis at ieee.org Wed Jul 12 15:25:59 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 12 Jul 2006 13:25:59 -0600 Subject: [Numpy-discussion] Numpy array from ctypes pointer object? In-Reply-To: <44B54826.2010002@mitre.org> References: <44B53494.2000005@mitre.org> <44B535C6.1080307@ieee.org> <44B54826.2010002@mitre.org> Message-ID: <44B54CC7.4040509@ieee.org> Mark Heslep wrote: > Travis Oliphant wrote: > >> Mark Heslep wrote: >> >>> I don't see a clean way to create a numpy array from a ctypes pointer >>> object. Is the __array_interface_ in ctypes the thing thats missing >>> needed to make this happen? I've followed Albert's Scipy cookbook >>> on ctypes here >>> >>> On the C side the PyArray_SimpleNewFromData( ..dimensions, ...data >>> ptr) C API does the job nicely. Is there a ctypes paradigm for >>> SimpleNew...? >>> >>> >> Can you somehow call this function using ctypes? >> >> -Travis >> > That might work, though indirectly. As I think I understand from ctypes > docs: Ctypes uses functions exposed in a shared library, macros > existing only a header are not available. If its PyArray... is a macro > then I a) need to compile and make a little library directly from > arrayobject.h or b) need to use the root function upon which the macro > is based, PyArrayNew? > > This is more complicated because all the C-API functions are actually just pointers stored in _ARRAY_API of multiarray. So, something would have to be built to interpret the C-pointers in that C-Object. I'm not sure that is possible. -Travis From mark at mitre.org Wed Jul 12 16:03:32 2006 From: mark at mitre.org (Mark Heslep) Date: Wed, 12 Jul 2006 16:03:32 -0400 Subject: [Numpy-discussion] Numpy array from ctypes pointer object? In-Reply-To: <44B54C24.2070301@ieee.org> References: <44B53494.2000005@mitre.org> <44B54C24.2070301@ieee.org> Message-ID: <44B55594.6090901@mitre.org> Travis Oliphant wrote: > The problem here is that from Python NumPy has no way to create an > ndarray from a pointer. Doing this creates a situtation where it is > unclear who owns the memory. It is probably best to wrap the pointer > into some kind of object exposing the buffer protocol and then pass > that to frombuffer (or ndarray.__new__). Yep thats where I just ended up: from ctypes import * import numpy as N ... func = pythonapi.PyBuffer_FromMemory func.restype = py_object buffer = func( im.imageData, size_of_the_data ) <----imageData = ctypes.LP_c_ubyte object return N.frombuffer( buffer, N.uint8 ) Works! Im curious though: the several projects recently using ctypes and numpy to wrap libraries (Pygame SDL, OpenGL, svm) must have come across the issue of using a creating a numpy array from a ctypes pointer. Ill have to look further. > When an ndarray is using memory that is not its own, it expects > another object to be "in charge" of that memory and the ndarray will > point its base attribute to it and increment its reference count. > What should the object that is "in charge" of the memory be? > Perhaps a suitable utility function could be created that can work > with ctypes to create ndarrays from ctypes memory locations and either > own or disown the data. > I suppose that is still the case w/ PyBuffer_From.. above. That is, the underlying im.imageData pointer can not be released before buffer. Mark > This needs to be thought through a bit, however. > > -Travis > > > >> The attributes in nparray.__array_interface_ are not writable, so no >> joy there. >> >> On the C side the PyArray_SimpleNewFromData( ..dimensions, ...data >> ptr) C API does the job nicely. Is there a ctypes paradigm for >> SimpleNew...? >> >> Mark >> >> >> >> ------------------------------------------------------------------------- >> >> Using Tomcat but need to do more? Need to support web services, >> security? >> Get stuff done quickly with pre-integrated technology to make your >> job easier >> Download IBM WebSphere Application Server v.1.0.1 based on Apache >> Geronimo >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> > From oliphant.travis at ieee.org Wed Jul 12 16:11:58 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 12 Jul 2006 14:11:58 -0600 Subject: [Numpy-discussion] Numpy array from ctypes pointer object? In-Reply-To: <44B55594.6090901@mitre.org> References: <44B53494.2000005@mitre.org> <44B54C24.2070301@ieee.org> <44B55594.6090901@mitre.org> Message-ID: <44B5578E.3010007@ieee.org> Mark Heslep wrote: > Travis Oliphant wrote: > >> The problem here is that from Python NumPy has no way to create an >> ndarray from a pointer. Doing this creates a situtation where it is >> unclear who owns the memory. It is probably best to wrap the pointer >> into some kind of object exposing the buffer protocol and then pass >> that to frombuffer (or ndarray.__new__). >> > Yep thats where I just ended up: > > from ctypes import * > import numpy as N > ... > func = pythonapi.PyBuffer_FromMemory > func.restype = py_object > buffer = func( im.imageData, size_of_the_data ) > <----imageData = ctypes.LP_c_ubyte object > return N.frombuffer( buffer, N.uint8 ) > > Works! Nice job! > Im curious though: the several projects recently using ctypes > and numpy to wrap libraries (Pygame SDL, OpenGL, svm) must have come > across the issue of using a creating a numpy array from a ctypes > pointer. Ill have to look further. > > It depends on whether or not the library creates memory or not. Not every library manages memory (some expect you to pass in memory already owned --- this is easy to do already with ctypes and numpy). >> When an ndarray is using memory that is not its own, it expects >> another object to be "in charge" of that memory and the ndarray will >> point its base attribute to it and increment its reference count. >> What should the object that is "in charge" of the memory be? >> Perhaps a suitable utility function could be created that can work >> with ctypes to create ndarrays from ctypes memory locations and either >> own or disown the data. >> >> > I suppose that is still the case w/ PyBuffer_From.. above. That is, the > underlying im.imageData pointer can not be released before buffer. > Yes, you are right. It is the memory that is most critical. Who owns the memory pointed to by im.imageData? When will it be freed? The ndarray object is holding a reference to the Python buffer object which is just *hoping* that the memory it was initiated with is not going to be freed before it gets deallocated (which won't happen until at least the ndarray object is deallocated). So, managing the memory can be a bit tricky. But, if you are sure that im.imageData memory will not be freed then you are O.K. -Travis From fullung at gmail.com Wed Jul 12 16:23:32 2006 From: fullung at gmail.com (Albert Strasheim) Date: Wed, 12 Jul 2006 22:23:32 +0200 Subject: [Numpy-discussion] Numpy array from ctypes pointer object? In-Reply-To: <44B5578E.3010007@ieee.org> Message-ID: <00e201c6a5f1$0b939300$0100000a@dsp.sun.ac.za> Hello all Various people wrote: > > Im curious though: the several projects recently using ctypes > > and numpy to wrap libraries (Pygame SDL, OpenGL, svm) must have come > > across the issue of using a creating a numpy array from a ctypes > > pointer. Ill have to look further. > > > It depends on whether or not the library creates memory or not. Not > every library manages memory (some expect you to pass in memory already > owned --- this is easy to do already with ctypes and numpy). I see two main models for sending back data from C land. One is to have the C function allocate memory and return the pointer to the caller. This raises the question of who owns the memory. It also leads to some *very* interesting crashes on Windows when you start freeing memory allocated in code linked against one runtime in code linked against another runtime. I think, if you have the option, avoid this kind of thing at all costs. The other model has functions that take their normal arguments, and pointers to buffers where they can store their results. Typically the caller would also specify the size of the buffer. If the buffer is large enough for the function to do its work, it uses it. If not, it returns an error code. This way, one entity is always in charge of the memory. If you're writing C code that you plan to use with ctypes and NumPy, I think the second model will lead to more robust code and less time spent debugging crashes and memory leaks. libsvm (which I'm wrapping with ctypes) mostly follows the second model, except when training new models, in which case it returns a pointer to a newly allocated structure. To deal with this, I keep a pointer to this allocated memory in a Python object that has the following function: def __del__(self): libsvm.svm_destroy_model(self.model) By providing this destroy function, libsvm avoids the problem of mixing allocation and deallocation across runtimes. Regards, Albert From prabhu at aero.iitb.ac.in Wed Jul 12 17:03:32 2006 From: prabhu at aero.iitb.ac.in (Prabhu Ramachandran) Date: Thu, 13 Jul 2006 02:33:32 +0530 Subject: [Numpy-discussion] [SciPy-dev] Weave, numpy, external libraries and conflicting typedefs In-Reply-To: <44AEC66A.2070507@ee.byu.edu> References: <17582.7994.142818.120995@prpc.aero.iitb.ac.in> <44AE9E12.5000703@ieee.org> <17582.46882.286683.846598@prpc.aero.iitb.ac.in> <44AEBE1F.8010304@ee.byu.edu> <44AEC66A.2070507@ee.byu.edu> Message-ID: <17589.25508.570735.848513@prpc.aero.iitb.ac.in> I sent this reply on 9th but the message seems to have never made it to scipy-dev and is still pending moderator approval on numpy-discussion. I had attached a patch for weave that made the email larger than 40 KB. I don't have checkin privileges to scipy/numpy. So if someone would be kind enough to apply the patch for me, let me know and I'll send you the patch off-list. >>>>> "Travis" == Travis Oliphant writes: [...] Travis> I'm not opposed to putting a *short* prefix in front of Travis> everything (the Int32, Float64, stuff came from numarray Travis> which now has it's own back-ward compatible header where Travis> it could be placed now anyway). Perhaps npy_ would be a Travis> suitable prefix. Travis> That way we could get rid of the cruft entirely. Travis> I suppose we could also provide the noprefix.h header that Travis> defines the old un-prefixed cases for "backwards NumPy Travis> compatibility". Travis, you rock! Thanks for fixing this in SVN. All the problems I was having earlier are gone. Here is a patch for weave's SWIG support that should be applied to scipy. The SWIG runtime layout changed in version 1.3.28 and this broke weave support. This should be fixed soon (hopefully!) in SWIG CVS and the following patch will ensure that weave works against it. Many thanks once again for the speedy fixes! cheers, prabhu From aisaac at american.edu Wed Jul 12 17:33:48 2006 From: aisaac at american.edu (Alan G Isaac) Date: Wed, 12 Jul 2006 17:33:48 -0400 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at><44B414DA.1050300@ee.byu.edu> Message-ID: On Wed, 12 Jul 2006, Sasha apparently wrote: > Let me repeat my suggestion that was lost in this long thread: > Add rands(shape, dtype=float, min=default_min(dtype), max=default_max(dtype)) > to the top level. Suitable defaults can be discussed. A more flexible > variation could > be rands(shape, dtype=float, algorithm=default_algorithm(dtype)), but > that would probably be an overkill. My only reason for not exploring this is that recent decisions seem to preclude it. Specifically, nothing from the random module remains in the numpy namespace, I think. Cheers, Alan Isaac From ndarray at mac.com Wed Jul 12 17:37:34 2006 From: ndarray at mac.com (Sasha) Date: Wed, 12 Jul 2006 17:37:34 -0400 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> <44B414DA.1050300@ee.byu.edu> Message-ID: On 7/12/06, Alan G Isaac wrote: > On Wed, 12 Jul 2006, Sasha apparently wrote: [snip] > > Add rands(shape, dtype=float, min=default_min(dtype), max=default_max(dtype)) > > to the top level. Suitable defaults can be discussed. A more flexible > > variation could > > be rands(shape, dtype=float, algorithm=default_algorithm(dtype)), but > > that would probably be an overkill. > > My only reason for not exploring this is that recent > decisions seem to preclude it. Specifically, nothing > from the random module remains in the numpy namespace, To the contrary, the recent changes cleared the way for a better random number generator in the numpy namespace. With my proposal, I would predict that rands(n) and rands((n,m)) will be used a lot in tests and examples while more sophisticated functionality will be easily discoverable via help(rands). I can also see some benefit in having rands(n, bool) that is not available at the moment. From laidler at stsci.edu Wed Jul 12 17:42:08 2006 From: laidler at stsci.edu (Victoria G. Laidler) Date: Wed, 12 Jul 2006 17:42:08 -0400 Subject: [Numpy-discussion] how to use argsort result? In-Reply-To: <44B37608.2000003@itc.it> References: <44B37040.2040905@itc.it> <6ef8f3380607110240i24f6f219w2fa112a113f04291@mail.gmail.com> <44B37608.2000003@itc.it> Message-ID: <44B56CB0.2080507@stsci.edu> Hi, Pardon me if I'm reprising an earlier discussion, as I'm new to the list. But is there a reason that this obscure syntax A[arange(2)[:,newaxis],indexes] A[arange(A.shape[0])[:,newaxis],indexes] is preferable to the intuitively reasonable thing that the Original Poster did? A[indexes] Doesn't it violate the pythonic "principle of least surprise" for the simpler syntax not to work? As a casual numpy user, I can't imagine being able to remember the obscure syntax in order to use the result of an argsort. curiously, Vicki Laidler (numarray user who came from IDL) Emanuele Olivetti wrote: >Wow. I have to study much more indexing. It works pretty well. > >Just to help indexing newbie like on using your advice: >A[arange(A.shape[0])[:,newaxis],indexes] > >Thanks a lot! > >Emanuele > >Pau Gargallo wrote: > > >>here goes a first try: >> >>A[arange(2)[:,newaxis],indexes] >> >> > > >------------------------------------------------------------------------- >Using Tomcat but need to do more? Need to support web services, security? >Get stuff done quickly with pre-integrated technology to make your job easier >Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo >http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From mark at mitre.org Wed Jul 12 17:46:39 2006 From: mark at mitre.org (Mark Heslep) Date: Wed, 12 Jul 2006 17:46:39 -0400 Subject: [Numpy-discussion] Numpy array from ctypes pointer object? In-Reply-To: <00e201c6a5f1$0b939300$0100000a@dsp.sun.ac.za> References: <00e201c6a5f1$0b939300$0100000a@dsp.sun.ac.za> Message-ID: <44B56DBF.9060906@mitre.org> Albert Strasheim wrote: > Hello all > > Various people wrote: > >>> Im curious though: the several projects recently using ctypes >>> and numpy to wrap libraries (Pygame SDL, OpenGL, svm) must have come >>> across the issue of using a creating a numpy array from a ctypes >>> pointer. Ill have to look further. >>> >>> >> It depends on whether or not the library creates memory or not. Not >> every library manages memory (some expect you to pass in memory already >> owned --- this is easy to do already with ctypes and numpy). >> > > I see two main models for sending back data from C land. > > One is to have the C function allocate memory and return the pointer to the > caller. This raises the question of who owns the memory. It also leads to > some *very* interesting crashes on Windows when you start freeing memory > allocated in code linked against one runtime in code linked against another > runtime. I think, if you have the option, avoid this kind of thing at all > costs. > > The other model has functions that take their normal arguments, and pointers > to buffers where they can store their results. Typically the caller would > also specify the size of the buffer. If the buffer is large enough for the > function to do its work, it uses it. If not, it returns an error code. This > way, one entity is always in charge of the memory. > > If you're writing C code that you plan to use with ctypes and NumPy, I think > the second model will lead to more robust code and less time spent debugging > crashes and memory leaks. > Yes, I agree, C libs written with your model #2 would make life much easier to create robust wrappers. Many of the ones that come to my mind, and deal with array like data and thus relevant, are model #1 types. I think the reason this is so goes something like this: Joe Foo developer & friends writes an entire library API in C. To make the API complete and easy to use Joe includes getting started 'make the data' C functions. Examples: SDL: SDL_surface* SDL_CreateRGBSurface(params) Opencv: IplImage* cvCreateImage(params), ... libdc1394: uint_t * capture buffer //created by the OS 1394 driver etc. If the Joe doesn't do this then the first thing Joe's 'users' must do to create the simplest application is call malloc, sizeof( figure out what element type...), blah, blah, i.e. do lots of memory management having zilch to do with the problem at hand. Further, Joe, being conscientious, includes boat loads of examples all using the 'make the data' calls. > libsvm (which I'm wrapping with ctypes) mostly follows the second model, > except when training new models, in which case it returns a pointer to a > newly allocated structure. To deal with this, I keep a pointer to this > allocated memory in a Python object that has the following function: > > def __del__(self): > libsvm.svm_destroy_model(self.model) > > Nice > By providing this destroy function, libsvm avoids the problem of mixing > allocation and deallocation across runtimes. > > Regards, > > Albert > > -Mark From oliphant.travis at ieee.org Wed Jul 12 18:42:55 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 12 Jul 2006 16:42:55 -0600 Subject: [Numpy-discussion] Patch against Image.py in the PIL Message-ID: <44B57AEF.3080300@ieee.org> Attached is a patch that makes PIL Image objects both export and consume the array interface. -Travis -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: pil_patch_against_358.txt URL: From robert.kern at gmail.com Wed Jul 12 19:16:47 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 12 Jul 2006 18:16:47 -0500 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> <44B414DA.1050300@ee.byu.edu> Message-ID: Sasha wrote: > Let me repeat my suggestion that was lost in this long thread: > > Add rands(shape, dtype=float, min=default_min(dtype), max=default_max(dtype)) > to the top level. Suitable defaults can be discussed. A more flexible > variation could > be rands(shape, dtype=float, algorithm=default_algorithm(dtype)), but > that would probably be an overkill. > > I think this will help teaching: rands is similar to zeros and ones, > but with few bells and whistles to be covered in the graduate course. Well, write it up, stick it as a method in RandomState, and when we can play with it, we can discuss whether it should go into numpy.*. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From wbaxter at gmail.com Wed Jul 12 22:29:46 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu, 13 Jul 2006 11:29:46 +0900 Subject: [Numpy-discussion] numpy distutils problem on Windows Message-ID: In numpy/distutils/command/config.py: 'cookedm' added a get_output() command on June 9. This get_output function uses os.WEXITSTATUS and various other os.W*functions. These do not exist in Python on Windows. Is there some other way to achieve the same thing without those? For now, just commenting out those error checking lines seems to do the trick. --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From cookedm at physics.mcmaster.ca Wed Jul 12 22:48:35 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed, 12 Jul 2006 22:48:35 -0400 Subject: [Numpy-discussion] numpy distutils problem on Windows In-Reply-To: References: Message-ID: <20060712224835.30c2ebda@arbutus.physics.mcmaster.ca> On Thu, 13 Jul 2006 11:29:46 +0900 "Bill Baxter" wrote: > In numpy/distutils/command/config.py: > 'cookedm' added a get_output() command on June 9. > This get_output function uses os.WEXITSTATUS and various other > os.W*functions. > > These do not exist in Python on Windows. > > Is there some other way to achieve the same thing without those? > > For now, just commenting out those error checking lines seems to do the > trick. Shoot, you're right. I didn't see that in the docs for the os module. I've fixed it in svn (I think -- I don't have a Windows box to test on). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From wbaxter at gmail.com Wed Jul 12 23:12:36 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu, 13 Jul 2006 12:12:36 +0900 Subject: [Numpy-discussion] Building SciPy under Windows attn. Bill Baxter In-Reply-To: <7.0.1.0.0.20060712172713.0b628218@ecs.soton.ac.uk> References: <7.0.1.0.0.20060712172713.0b628218@ecs.soton.ac.uk> Message-ID: Thanks, that seems to have done the trick! I've got a SVN Scipy now! I updated the building scipy wiki page with this piece of advice. --bb On 7/13/06, John Carter wrote: > > I had problems building SciPy and NumPy under Windows. They went away > when I stopped using the stable version of gcc and used 3.4.5 > > I think the problem was related to differences in cygwin and mingw32. > > SciPy built and everything I've needed works but the self test fails. > > John > > Dr. John N. Carter jnc at ecs.soton.ac.uk > ISIS http://www.ecs.soton.ac.uk/~jnc/ > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- William V. Baxter III OLM Digital Kono Dens Building Rm 302 1-8-8 Wakabayashi Setagaya-ku Tokyo, Japan 154-0023 +81 (3) 3422-3380 -------------- next part -------------- An HTML attachment was scrubbed... URL: From haase at msg.ucsf.edu Thu Jul 13 00:14:47 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 12 Jul 2006 21:14:47 -0700 Subject: [Numpy-discussion] prebuild windows - was: Re: Building SciPy under Windows attn. Bill Baxter In-Reply-To: References: <7.0.1.0.0.20060712172713.0b628218@ecs.soton.ac.uk> Message-ID: <44B5C8B7.70407@msg.ucsf.edu> Hi, The latest cygwin gcc seems to be version 3.4.4. Are (relatively) new SVN builds (both numpy and scipy) available somewhere for download ? (I would like to experience the new "arr.T" feature ;-) ) Thanks Sebastian Haase Bill Baxter wrote: > Thanks, that seems to have done the trick! > I've got a SVN Scipy now! > > I updated the building scipy wiki page with this piece of advice. > > --bb > > On 7/13/06, * John Carter* > wrote: > > I had problems building SciPy and NumPy under Windows. They went away > when I stopped using the stable version of gcc and used 3.4.5 > > I think the problem was related to differences in cygwin and mingw32. > > SciPy built and everything I've needed works but the self test fails. > > John > > Dr. John N. Carter jnc at ecs.soton.ac.uk > ISIS http://www.ecs.soton.ac.uk/~jnc/ > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, > security? > Get stuff done quickly with pre-integrated technology to make your > job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > > -- > William V. Baxter III > OLM Digital > Kono Dens Building Rm 302 > 1-8-8 Wakabayashi Setagaya-ku > Tokyo, Japan 154-0023 > +81 (3) 3422-3380 > > > ------------------------------------------------------------------------ > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From wbaxter at gmail.com Thu Jul 13 01:24:12 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu, 13 Jul 2006 14:24:12 +0900 Subject: [Numpy-discussion] determining if you have ATLAS or not Message-ID: How can you probe numpy for info about what sort of BLAS/LAPACK you have, or other build configuration info? Searching the ml archives I turned up this one hint from Travis, which can be embodied in code thusly: import numpy def have_blas(): return id(numpy.dot) != id(numpy.core.multiarray.dot) A) is that function correct? and B) Is that the most you can find out? --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Thu Jul 13 02:16:30 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu, 13 Jul 2006 15:16:30 +0900 Subject: [Numpy-discussion] determining if you have ATLAS or not In-Reply-To: References: Message-ID: Terrific. Nils sent me the answer: scipy.show_config() or > numpy.show_config() > will give you some useful information. > > And here it goes straight to the wiki http://www.scipy.org/Installing_SciPy/Windows --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebzur at pin.if.uz.zgora.pl Wed Jul 12 19:07:31 2006 From: sebzur at pin.if.uz.zgora.pl (=?ISO-8859-2?Q?Sebastian_=AFurek?=) Date: Thu, 13 Jul 2006 00:07:31 +0100 Subject: [Numpy-discussion] RandomArray module Message-ID: Hi All, Has anyone worked with the RandomArray module? I wonder, if it's OK to use its pseudo-random numbers generators, or maybe I shall find more trusted methods (ie. ran1 from Numerical Recipes)? Please, give some comments. Thanks. Sebastian From faltet at carabos.com Thu Jul 13 03:40:27 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu, 13 Jul 2006 09:40:27 +0200 Subject: [Numpy-discussion] RandomArray module In-Reply-To: References: Message-ID: <200607130940.28154.faltet@carabos.com> A Dijous 13 Juliol 2006 01:07, Sebastian ?urek va escriure: > Hi All, > > Has anyone worked with the RandomArray module? I wonder, > if it's OK to use its pseudo-random numbers generators, or > maybe I shall find more trusted methods (ie. ran1 from Numerical Recipes)? I'm not an expert, but my understanding is that the 'random' module that comes with NumPy is made with kind of state-of-the-art random generators, so it should be fine for most of purposes. However, the experts, and in particular Robert Kern (I think he is the implementor), may want to precise this point. > > Please, give some comments. Thanks. Done ;-) -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From pau.gargallo at gmail.com Thu Jul 13 04:35:33 2006 From: pau.gargallo at gmail.com (Pau Gargallo) Date: Thu, 13 Jul 2006 10:35:33 +0200 Subject: [Numpy-discussion] how to use argsort result? In-Reply-To: <44B56CB0.2080507@stsci.edu> References: <44B37040.2040905@itc.it> <6ef8f3380607110240i24f6f219w2fa112a113f04291@mail.gmail.com> <44B37608.2000003@itc.it> <44B56CB0.2080507@stsci.edu> Message-ID: <6ef8f3380607130135x1fe1bcau606a5fa46a49497d@mail.gmail.com> On 7/12/06, Victoria G. Laidler wrote: > Hi, > > Pardon me if I'm reprising an earlier discussion, as I'm new to the list. > > But is there a reason that this obscure syntax > > A[arange(2)[:,newaxis],indexes] > > A[arange(A.shape[0])[:,newaxis],indexes] > > is preferable to the intuitively reasonable thing that the Original > Poster did? > > A[indexes] > i don't think so. The obscure syntax is just a way you can solve the problem with the current state of NumPy. Of course, a more clearer syntax would be better, but for this, something in NumPy should be changed. This other syntax is longer but clearer: ind = indices(A.shape) ind[ax] = A.argsort(axis=ax) A[ind] Which brings me to the question: Would it be reasonable if argsort returned the complete tuple of indices, so that A[A.argsort(ax)] would work ? pau From oliphant.travis at ieee.org Thu Jul 13 05:07:25 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 13 Jul 2006 03:07:25 -0600 Subject: [Numpy-discussion] how to use argsort result? In-Reply-To: <6ef8f3380607130135x1fe1bcau606a5fa46a49497d@mail.gmail.com> References: <44B37040.2040905@itc.it> <6ef8f3380607110240i24f6f219w2fa112a113f04291@mail.gmail.com> <44B37608.2000003@itc.it> <44B56CB0.2080507@stsci.edu> <6ef8f3380607130135x1fe1bcau606a5fa46a49497d@mail.gmail.com> Message-ID: <44B60D4D.2080509@ieee.org> Pau Gargallo wrote: > On 7/12/06, Victoria G. Laidler wrote: > >> Hi, >> >> Pardon me if I'm reprising an earlier discussion, as I'm new to the list. >> >> But is there a reason that this obscure syntax >> >> A[arange(2)[:,newaxis],indexes] >> >> A[arange(A.shape[0])[:,newaxis],indexes] >> >> is preferable to the intuitively reasonable thing that the Original >> Poster did? >> >> A[indexes] >> >> > > i don't think so. > The obscure syntax is just a way you can solve the problem with the > current state of NumPy. Of course, a more clearer syntax would be > better, but for this, something in NumPy should be changed. > > This other syntax is longer but clearer: > ind = indices(A.shape) > ind[ax] = A.argsort(axis=ax) > A[ind] > > Which brings me to the question: > > Would it be reasonable if argsort returned the complete tuple of > indices, so that > A[A.argsort(ax)] would work ? > > I think this is reasonable. We would need a way for the argsort() function to work as it does now. I'm not sure if anybody actually uses the multidimensional behavior of argsort now, but it's been in Numeric for a long time. -Travis From pau.gargallo at gmail.com Thu Jul 13 05:52:31 2006 From: pau.gargallo at gmail.com (Pau Gargallo) Date: Thu, 13 Jul 2006 11:52:31 +0200 Subject: [Numpy-discussion] int 'up'casted to uint In-Reply-To: <6ef8f3380607110924p66b8de0ds55971dfb8c9ff892@mail.gmail.com> References: <6ef8f3380607110354u6db272b0t125a9f13a92d1180@mail.gmail.com> <44B3CBDA.8080605@ieee.org> <6ef8f3380607110924p66b8de0ds55971dfb8c9ff892@mail.gmail.com> Message-ID: <6ef8f3380607130252m5afcc214kcd68f1b9bb5ba7ea@mail.gmail.com> On 7/11/06, Pau Gargallo wrote: > On 7/11/06, Travis Oliphant wrote: > > Pau Gargallo wrote: > > > hi, > > > > > > looking at the upcasting table at > > > http://www.scipy.org/Tentative_NumPy_Tutorial#head-4c1d53fe504adc97baf27b65513b4b97586a4fc5 > > > I saw that int's are sometimes casted to uint's. > > > > > > In [3]: a = array([3],int16) > > > In [5]: b = array([4],uint32) > > > In [7]: a+b > > > Out[7]: array([7], dtype=uint32) > > > > > > is that intended? > > > > > It's a bug. The result should be int64. I've fixed it in SVN. > > > > Thanks!! > hi Travis, now uint64+int gives float64. Obtaining a float from ints surprise me. But anyway, I don't know if there is a better choice. pau From adam at epcc.ed.ac.uk Thu Jul 13 06:02:07 2006 From: adam at epcc.ed.ac.uk (Adam Carter) Date: Thu, 13 Jul 2006 11:02:07 +0100 Subject: [Numpy-discussion] LAPACK Problem with "import numpy" on AIX Message-ID: <200607131002.k6DA272v009473@e450.epcc.ed.ac.uk> Hi all, I'm new to this list so apologies if this is a solved problem, but I haven't been able to find anything in the archives. I've just installed Python 2.4.3 and Numpy-0.9.8 on AIX, and the configure/make/install of Python and the install of Numpy _appeared_ to go smoothly. However, now when I run python and type >>> import numpy I get import linalg -> failed: No module named lapack_lite I'm not sure if this is just a warning, but I suspect not, as when I later try import scipy, it results in a traceback: >>> import scipy Traceback (most recent call last): File "", line 1, in ? File "/hpcx/home/z001/z001/adam/packages/Python-2.4.3/lib/python2.4/site-packages /scipy/__init__.py", line 34, in ? del linalg NameError: name 'linalg' is not defined Can anytone tell me what I'm doing wrong? I've not explicitly installed any LAPACK library but I was hpoing to use an existing (vendor provided) LAPACK. The environment variable $LAPACK was set to point to this library at compile and run-time. Do I need lapack_lite aswell? Where can I get it? If I need this other LAPACK, how can I ensure that my code uses the optimised version of LAPACK already on this system I'm using? Any advice very welcome. Thanks in advance, Adam ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Dr Adam Carter (Applications Consultant) |epcc| e: adam at epcc.ed.ac.uk w: http://www.epcc.ed.ac.uk/~adam r: 3405 JCMB t: +44 131 650 6009 From wbaxter at gmail.com Thu Jul 13 06:08:19 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu, 13 Jul 2006 19:08:19 +0900 Subject: [Numpy-discussion] LAPACK Problem with "import numpy" on AIX In-Reply-To: <200607131002.k6DA272v009473@e450.epcc.ed.ac.uk> References: <200607131002.k6DA272v009473@e450.epcc.ed.ac.uk> Message-ID: On 7/13/06, Adam Carter wrote: > > Hi all, > > Do I need lapack_lite aswell? Where can I get it? If I need this other > LAPACK, how can I ensure that my code uses the optimised version of LAPACK > already on this system I'm using? Lapack_lite is in the numpy sources. I think it's the fallback if you don't have a LAPACK. You can check the config that got built with: numpy.show_config() But that doesn't help much if you can't import numpy. --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam at epcc.ed.ac.uk Thu Jul 13 06:19:16 2006 From: adam at epcc.ed.ac.uk (Adam Carter) Date: Thu, 13 Jul 2006 11:19:16 +0100 Subject: [Numpy-discussion] LAPACK Problem with "import numpy" on AIX In-Reply-To: Message-ID: <200607131019.k6DAJG4s010054@e450.epcc.ed.ac.uk> > Lapack_lite is in the numpy sources. I think it's the fallback if you don't have a LAPACK. Thanks Bill, that's what I thought, but I don't understand why it can't find it. > You can check the config that got built with: > numpy.show_config() OK, so it appears that NumPy is imported then, albeit with the warning. The result of the above line is included below. So maybe the problem lies with SciPy? Perhaps I should take my problem to scipy-users. Unless anyone has any further insight? Cheers, Adam >>> numpy.show_config() blas_info: libraries = ['blas'] library_dirs = ['/usr/lib'] language = f77 lapack_info: libraries = ['lapack'] library_dirs = ['/usr/local/lib'] language = f77 atlas_threads_info: NOT AVAILABLE blas_opt_info: libraries = ['blas'] library_dirs = ['/usr/lib'] language = f77 define_macros = [('NO_ATLAS_INFO', 1)] atlas_blas_threads_info: NOT AVAILABLE lapack_opt_info: libraries = ['lapack', 'blas'] library_dirs = ['/usr/local/lib', '/usr/lib'] language = f77 define_macros = [('NO_ATLAS_INFO', 1)] atlas_info: NOT AVAILABLE lapack_mkl_info: NOT AVAILABLE blas_mkl_info: NOT AVAILABLE atlas_blas_info: NOT AVAILABLE mkl_info: NOT AVAILABLE From pau.gargallo at gmail.com Thu Jul 13 08:09:07 2006 From: pau.gargallo at gmail.com (Pau Gargallo) Date: Thu, 13 Jul 2006 14:09:07 +0200 Subject: [Numpy-discussion] how to use argsort result? In-Reply-To: <44B60D4D.2080509@ieee.org> References: <44B37040.2040905@itc.it> <6ef8f3380607110240i24f6f219w2fa112a113f04291@mail.gmail.com> <44B37608.2000003@itc.it> <44B56CB0.2080507@stsci.edu> <6ef8f3380607130135x1fe1bcau606a5fa46a49497d@mail.gmail.com> <44B60D4D.2080509@ieee.org> Message-ID: <6ef8f3380607130509s4f787ce8na1f7ac58b07f1dd4@mail.gmail.com> On 7/13/06, Travis Oliphant wrote: > Pau Gargallo wrote: > > On 7/12/06, Victoria G. Laidler wrote: > > > >> Hi, > >> > >> Pardon me if I'm reprising an earlier discussion, as I'm new to the list. > >> > >> But is there a reason that this obscure syntax > >> > >> A[arange(2)[:,newaxis],indexes] > >> > >> A[arange(A.shape[0])[:,newaxis],indexes] > >> > >> is preferable to the intuitively reasonable thing that the Original > >> Poster did? > >> > >> A[indexes] > >> > >> > > > > i don't think so. > > The obscure syntax is just a way you can solve the problem with the > > current state of NumPy. Of course, a more clearer syntax would be > > better, but for this, something in NumPy should be changed. > > > > This other syntax is longer but clearer: > > ind = indices(A.shape) > > ind[ax] = A.argsort(axis=ax) > > A[ind] > > > > Which brings me to the question: > > > > Would it be reasonable if argsort returned the complete tuple of > > indices, so that > > A[A.argsort(ax)] would work ? > > > > > I think this is reasonable. We would need a way for the argsort() > function to work as it does now. I'm not sure if anybody actually uses > the multidimensional behavior of argsort now, but it's been in Numeric > for a long time. > actually I use the multidimensional behavior of argmin and argmax in its current state, and found it useful as it is, even if A[A.argmax(ax)] doesn't work. May be a keyword could be added, so that A.argxxx( axis=ax, indices=True ) returns the tuple of indices. (The keyword name and default should be discussed) I don't know if that's *the* way, but ... pau From charges at humortadela.com.br Thu Jul 13 09:21:53 2006 From: charges at humortadela.com.br (Humortadela) Date: Thu, 13 Jul 2006 08:21:53 -0500 Subject: [Numpy-discussion] Voce Recebeu uma Charge do Humortadela Message-ID: <6f7beb686bf7e1e84df7db96919ccb5a@humortadela.com.br> An HTML attachment was scrubbed... URL: From charges at humortadela.com.br Thu Jul 13 09:21:52 2006 From: charges at humortadela.com.br (Humortadela) Date: Thu, 13 Jul 2006 08:21:52 -0500 Subject: [Numpy-discussion] Voce Recebeu uma Charge do Humortadela Message-ID: <6f7beb686bf7e1e84df7db96919ccb5a@humortadela.com.br> An HTML attachment was scrubbed... URL: From Jonathan.Peirce at nottingham.ac.uk Thu Jul 13 08:55:05 2006 From: Jonathan.Peirce at nottingham.ac.uk (Jon Peirce) Date: Thu, 13 Jul 2006 13:55:05 +0100 Subject: [Numpy-discussion] generalized_inverse Message-ID: <44B642A9.3090805@psychology.nottingham.ac.uk> There used to be a function generalized_inverse in the numpy.linalg module (certainly in 0.9.2). In numpy0.9.8 it seems to have been moved to the numpy.linalg.old subpackage. Does that mean it's being dropped? Did it have to move? Now i have to add code to my package to try both locations because my users might have any version... :-( Jon -- Jon Peirce Nottingham University +44 (0)115 8467176 (tel) +44 (0)115 9515324 (fax) http://www.psychopy.org This message has been checked for viruses but the contents of an attachment may still contain software viruses, which could damage your computer system: you are advised to perform your own checks. Email communications with the University of Nottingham may be monitored as permitted by UK legislation. From berkes at gatsby.ucl.ac.uk Thu Jul 13 09:11:00 2006 From: berkes at gatsby.ucl.ac.uk (Pietro Berkes) Date: Thu, 13 Jul 2006 14:11:00 +0100 (BST) Subject: [Numpy-discussion] A few comments (MDP conversion) Message-ID: Dear numpys, a couple of weeks ago Tiziano and I completed the conversion of our data processing library MDP to numpy. We collected a few ideas and questions: - we found the convertcode.py module quite useful to perform a first, low-level conversion. We had some problem when 'typecode' was used as a keyword argument because the parser only converts 'typecode=' to 'dtype=' and skips the cases where there is an additional space character before the equal sign ('typecode ='). Other names that might be easily converted are 'ArrayType' and 'NewAxis'. - some functions changed the columns/rows conventions ('cov', for example). It would be really helpful to explicitly write this in the list of necessary changes in the documentation. It could be nice to have the file 'convertcode.py' to print a warning every time one of this functions is used in the code. - the linalg functions svd, eig, inv, pinv, diag, and possibly others perform an upcast, e.g. from 'f' to 'd'. This is apparently because the litelapack module only wraps double precision routines. Wouldn't it be more consistent to cast the results back to the numerical type of the input? Otherwise, the user always has to take care of the casting, which makes the use of single precision arrays quite cumbersome. - we found a bug in the 'eig' function in the case the solution is complex: in (linalg.py, line 269) 'Complex' should be 'w.dtype' We wondered whether it would be possible to obtain SVN write access in orderto be able to fix this kind of issues by ourselves in the future. We could also contribute docstrings for some of the functions. In general, we found the conversion to numpy quite straightforward and would like to thank you all for the great work! Cheers, Pietro Berkes and Tiziano Zito http://mdp-toolkit.sourceforge.net/ From bsouthey at gmail.com Thu Jul 13 11:07:55 2006 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 13 Jul 2006 10:07:55 -0500 Subject: [Numpy-discussion] RandomArray module In-Reply-To: References: Message-ID: Hi, As Francesc meintioned, Robert Kern did a great job of replacing ranlib: "numpy.random uses the Mersenne Twister. RANLIB is dead! Long live MT19937!" So throw away ran1! Regards Bruce On 7/12/06, Sebastian ?urek wrote: > Hi All, > > Has anyone worked with the RandomArray module? I wonder, > if it's OK to use its pseudo-random numbers generators, or > maybe I shall find more trusted methods (ie. ran1 from Numerical Recipes)? > > Please, give some comments. Thanks. > > Sebastian > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From nwagner at iam.uni-stuttgart.de Thu Jul 13 11:43:06 2006 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 13 Jul 2006 17:43:06 +0200 Subject: [Numpy-discussion] [Fwd: Re: A few comments (MDP conversion)] Message-ID: <44B66A0A.8030002@iam.uni-stuttgart.de> -------------- next part -------------- An embedded message was scrubbed... From: unknown sender Subject: no subject Date: no date Size: 38 URL: From nwagner at iam.uni-stuttgart.de Thu Jul 13 11:42:24 2006 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 13 Jul 2006 17:42:24 +0200 Subject: [Numpy-discussion] A few comments (MDP conversion) In-Reply-To: References: Message-ID: <44B669E0.7020804@iam.uni-stuttgart.de> Pietro Berkes wrote: > Dear numpys, > > a couple of weeks ago Tiziano and I completed the conversion of our data > processing library MDP to numpy. We collected a few ideas and > questions: > > - we found the convertcode.py module quite useful to perform a first, > low-level conversion. We had some problem when 'typecode' was > used as a keyword argument because the parser only converts > 'typecode=' to 'dtype=' and skips the cases where there is an > additional space character before the equal sign ('typecode ='). > Other names that might be easily converted are 'ArrayType' and > 'NewAxis'. > > - some functions changed the columns/rows conventions ('cov', for > example). It would be really helpful to explicitly write this in the > list of necessary changes in the documentation. It could be nice > to have the file 'convertcode.py' to print a warning every time one > of this functions is used in the code. > > - the linalg functions svd, eig, inv, pinv, diag, and possibly > others perform an upcast, e.g. from 'f' to 'd'. This is apparently > because the litelapack module only wraps double precision routines. > Wouldn't it be more consistent to cast the results back to the > numerical type of the input? Otherwise, the user always has to take > care of the casting, which makes the use of single precision > arrays quite cumbersome. > > - we found a bug in the 'eig' function in the case the solution > is complex: in (linalg.py, line 269) 'Complex' should be 'w.dtype' > > Did you file a ticket ? http://projects.scipy.org/scipy/numpy It seems to be line 281 instead of 269. I am using latest svn. BTW, in linalg.py in def pinv: there is another Complex with capital C. Nils > We wondered whether it would be possible to obtain SVN write access in > orderto be able to fix this kind of issues by ourselves in the future. > We could also contribute docstrings for some of the functions. > > In general, we found the conversion to numpy quite straightforward > and would like to thank you all for the great work! > > Cheers, > Pietro Berkes and Tiziano Zito > http://mdp-toolkit.sourceforge.net/ > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > --------------040602050709060203000700-- From berkes at gatsby.ucl.ac.uk Thu Jul 13 12:39:56 2006 From: berkes at gatsby.ucl.ac.uk (Pietro Berkes) Date: Thu, 13 Jul 2006 17:39:56 +0100 (BST) Subject: [Numpy-discussion] [Fwd: Re: A few comments (MDP conversion)] In-Reply-To: <44B66A0A.8030002@iam.uni-stuttgart.de> References: <44B66A0A.8030002@iam.uni-stuttgart.de> Message-ID: On Thu, 13 Jul 2006, Nils Wagner wrote: > It seems to be line 281 instead of 269. I am using latest svn. > BTW, in linalg.py in > > def pinv: > > there is another Complex with capital C. Well, the problem is not really the capital 'C', but rather the lack of quotation marks... From robert.kern at gmail.com Thu Jul 13 13:26:34 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 13 Jul 2006 12:26:34 -0500 Subject: [Numpy-discussion] RandomArray module In-Reply-To: References: Message-ID: Sebastian ?urek wrote: > Hi All, > > Has anyone worked with the RandomArray module? I wonder, > if it's OK to use its pseudo-random numbers generators, or > maybe I shall find more trusted methods (ie. ran1 from Numerical Recipes)? At this point in time, I don't recommend using RandomArray if you can avoid it. The RANLIB library that underlies it is quite well respected, but it is also quite old. The field has moved on since it was written. ran1 is no better. If you can make the leap to numpy instead of Numeric, the PRNG we use is the Mersenne Twister which beats the pants off RANLIB and anything from NR. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From efiring at hawaii.edu Thu Jul 13 13:51:31 2006 From: efiring at hawaii.edu (Eric Firing) Date: Thu, 13 Jul 2006 07:51:31 -1000 Subject: [Numpy-discussion] how to use argsort result? In-Reply-To: <6ef8f3380607130135x1fe1bcau606a5fa46a49497d@mail.gmail.com> References: <44B37040.2040905@itc.it> <6ef8f3380607110240i24f6f219w2fa112a113f04291@mail.gmail.com> <44B37608.2000003@itc.it> <44B56CB0.2080507@stsci.edu> <6ef8f3380607130135x1fe1bcau606a5fa46a49497d@mail.gmail.com> Message-ID: <44B68823.8020702@hawaii.edu> > > Would it be reasonable if argsort returned the complete tuple of > indices, so that > A[A.argsort(ax)] would work ? +1 This is the behavior one would naturally expect. Eric From webb.sprague at gmail.com Thu Jul 13 15:42:30 2006 From: webb.sprague at gmail.com (Webb Sprague) Date: Thu, 13 Jul 2006 12:42:30 -0700 Subject: [Numpy-discussion] Finding indices for all rows which contain nan (or anything else) Message-ID: Does anyone have some vectorized code that pulls out all the row indices for any row which has an nan (or a number less than 1 or whatever). I want to subsequently be able to perform an operation with all the good rows. See the imaginary code below. a = numpy.array([[1,2],[nan,1], [2,3]]) is_row_nan(a) == array([1]) ii = numpy.negative(is_row_nan(a)) a[ii,:] # these are the ones I want. Hopefully this is array([[1,2],[2,3]]) I can imagine doing this with a loop or with (maybe) some fancy set union stuff, but I am at a loss for vectorized versions. Thanks From robert.kern at gmail.com Thu Jul 13 15:56:11 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 13 Jul 2006 14:56:11 -0500 Subject: [Numpy-discussion] Finding indices for all rows which contain nan (or anything else) In-Reply-To: References: Message-ID: Webb Sprague wrote: > Does anyone have some vectorized code that pulls out all the row > indices for any row which has an nan (or a number less than 1 or > whatever). I want to subsequently be able to perform an operation > with all the good rows. See the imaginary code below. > > a = numpy.array([[1,2],[nan,1], [2,3]]) > is_row_nan(a) == array([1]) > ii = numpy.negative(is_row_nan(a)) > > a[ii,:] # these are the ones I want. Hopefully this is array([[1,2],[2,3]]) > > I can imagine doing this with a loop or with (maybe) some fancy set > union stuff, but I am at a loss for vectorized versions. (Untested) def is_row_nan(a): return numpy.isnan(a).any(axis=-1) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From webb.sprague at gmail.com Thu Jul 13 16:02:13 2006 From: webb.sprague at gmail.com (Webb Sprague) Date: Thu, 13 Jul 2006 13:02:13 -0700 Subject: [Numpy-discussion] Finding indices for all rows which contain nan (or anything else) In-Reply-To: References: Message-ID: On 7/13/06, Robert Kern wrote: > Webb Sprague wrote: > > Does anyone have some vectorized code that pulls out all the row > > def is_row_nan(a): > return numpy.isnan(a).any(axis=-1) I knew there was a way, but I didn't know to check any() and all(). Thanks to all (I love free software lists!) W From cookedm at physics.mcmaster.ca Thu Jul 13 17:11:49 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu, 13 Jul 2006 17:11:49 -0400 Subject: [Numpy-discussion] A few comments (MDP conversion) In-Reply-To: References: Message-ID: On Jul 13, 2006, at 09:11 , Pietro Berkes wrote: > Dear numpys, > > a couple of weeks ago Tiziano and I completed the conversion of our > data > processing library MDP to numpy. We collected a few ideas and > questions: > > - we found the convertcode.py module quite useful to perform a first, > low-level conversion. We had some problem when 'typecode' was > used as a keyword argument because the parser only converts > 'typecode=' to 'dtype=' and skips the cases where there is an > additional space character before the equal sign ('typecode ='). > Other names that might be easily converted are 'ArrayType' and > 'NewAxis'. fixed in svn. > - some functions changed the columns/rows conventions ('cov', for > example). It would be really helpful to explicitly write this in the > list of necessary changes in the documentation. It could be nice > to have the file 'convertcode.py' to print a warning every time one > of this functions is used in the code. Do you have a list of these? > - the linalg functions svd, eig, inv, pinv, diag, and possibly > others perform an upcast, e.g. from 'f' to 'd'. This is apparently > because the litelapack module only wraps double precision routines. > Wouldn't it be more consistent to cast the results back to the > numerical type of the input? Otherwise, the user always has to take > care of the casting, which makes the use of single precision > arrays quite cumbersome. That makes sense. I'll have a look at it. > - we found a bug in the 'eig' function in the case the solution > is complex: in (linalg.py, line 269) 'Complex' should be 'w.dtype' fixed in svn. > We wondered whether it would be possible to obtain SVN write access in > orderto be able to fix this kind of issues by ourselves in the future. > We could also contribute docstrings for some of the functions. I don't know about svn write access (that's not up to me), but we're perfectly willing to take patches made with 'svn diff', and uploaded to our bug tracker. > In general, we found the conversion to numpy quite straightforward > and would like to thank you all for the great work! > > Cheers, > Pietro Berkes and Tiziano Zito > http://mdp-toolkit.sourceforge.net/ -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From cookedm at physics.mcmaster.ca Thu Jul 13 17:13:13 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu, 13 Jul 2006 17:13:13 -0400 Subject: [Numpy-discussion] [Fwd: Re: A few comments (MDP conversion)] In-Reply-To: References: <44B66A0A.8030002@iam.uni-stuttgart.de> Message-ID: On Jul 13, 2006, at 12:39 , Pietro Berkes wrote: > On Thu, 13 Jul 2006, Nils Wagner wrote: > >> It seems to be line 281 instead of 269. I am using latest svn. >> BTW, in linalg.py in >> >> def pinv: >> >> there is another Complex with capital C. > > Well, the problem is not really the capital 'C', but rather the > lack of > quotation marks... Guess I've got more work to do to get rid of the typecodes there... -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From bhendrix at enthought.com Thu Jul 13 18:48:45 2006 From: bhendrix at enthought.com (Bryce Hendrix) Date: Thu, 13 Jul 2006 17:48:45 -0500 Subject: [Numpy-discussion] ANN: Python Enthought Edition Version 1.0.0.beta4 Released Message-ID: <44B6CDCD.7010206@enthought.com> Enthought is pleased to announce the release of Python Enthought Edition Version 1.0.0.beta4 (http://code.enthought.com/enthon/) -- a python distribution for Windows. 1.0.0.beta4 Release Notes: -------------------- There are two known issues: * No documentation is included due to problems with the chm. Instead, all documentation for this beta is available on the web at http://code.enthought.com/enthon/docs. The official 1.0.0 will include a chm containing all of our docs again. * IPython may cause problems when starting the first time if a previous version of IPython was ran. If you see "WARNING: could not import user config", either follow the directions which follow the warning. Unless something terrible is discovered between now and the next release, we intend on releasing 1.0.0 on July 25th. This release includes version 1.0.9 of the Enthought Tool Suite (ETS) Package and bug fixes-- you can look at the release notes for this ETS version here: http://svn.enthought.com/downloads/enthought/changelog-release.1.0.9.html About Python Enthought Edition: ------------------------------- Python 2.4.3, Enthought Edition is a kitchen-sink-included Python distribution for Windows including the following packages out of the box: Numpy SciPy IPython Enthought Tool Suite wxPython PIL mingw f2py MayaVi Scientific Python VTK and many more... More information is available about all Open Source code written and released by Enthought, Inc. at http://code.enthought.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From webb.sprague at gmail.com Thu Jul 13 19:55:33 2006 From: webb.sprague at gmail.com (Webb Sprague) Date: Thu, 13 Jul 2006 16:55:33 -0700 Subject: [Numpy-discussion] numpy.average(fooArray, axis=0): dropping nans from calculation Message-ID: Could someone recommend a way to average an array along the columns without propagating the nans and without turning them into some weird number which bias the result? I guess I can just keep using an indexing array for fooArray, but if there is somehting more graceful, I would love to know. Boy missing data is a pain in the neck... Thanks again! From nvf at MIT.EDU Thu Jul 13 20:08:19 2006 From: nvf at MIT.EDU (Nick Fotopoulos) Date: Thu, 13 Jul 2006 20:08:19 -0400 Subject: [Numpy-discussion] numpy.vectorize performance Message-ID: Dear all, I often make use of numpy.vectorize to make programs read more like the physics equations I write on paper. numpy.vectorize is basically a wrapper for numpy.frompyfunc. Reading Travis's Scipy Book (mine is dated Jan 6 2005) kind of suggests to me that it returns a full- fledged ufunc exactly like built-in ufuncs. First, is this true? Second, how is the performance? i.e., are my functions performing approximately as fast as they could be or would they still gain a great deal of speed by rewriting it in C or some other compiled python accelerator? As an aside, I've found the following function decorator to be helpful for readability, and perhaps others will enjoy it or improve upon it: def autovectorized(f): """Function decorator to do vectorization only as necessary. vectorized functions fail for scalar inputs.""" def wrapper(input): if type(input) == numpy.arraytype: return numpy.vectorize(f)(input) return f(input) return wrapper For those unfamiliar to the syntactic joys of Python 2.4, you can then use this as: @autovectorized def myOtherwiseScalarFunction(*args): ... and now the function will work with both numpy arrays and scalars. Take care, Nick From efiring at hawaii.edu Thu Jul 13 20:44:15 2006 From: efiring at hawaii.edu (Eric Firing) Date: Thu, 13 Jul 2006 14:44:15 -1000 Subject: [Numpy-discussion] numpy.average(fooArray, axis=0): dropping nans from calculation In-Reply-To: References: Message-ID: <44B6E8DF.3010707@hawaii.edu> Webb Sprague wrote: > Could someone recommend a way to average an array along the columns > without propagating the nans and without turning them into some weird > number which bias the result? I guess I can just keep using an > indexing array for fooArray, but if there is somehting more graceful, > I would love to know. Something like this: import numpy as N ym = N.ma.masked_where(N.isnan(y), y) yaverage = N.ma.average(ym) > > Boy missing data is a pain in the neck... It certainly is, but Masked Arrays ease the pain. Eric From robert.kern at gmail.com Fri Jul 14 02:35:19 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 14 Jul 2006 01:35:19 -0500 Subject: [Numpy-discussion] A few comments (MDP conversion) In-Reply-To: References: Message-ID: Pietro Berkes wrote: > We wondered whether it would be possible to obtain SVN write access in > orderto be able to fix this kind of issues by ourselves in the future. > We could also contribute docstrings for some of the functions. The best way to get SVN privileges in any open source project is by posting a constant stream of good patches to the bug tracker such that it becomes less work for us to give you access than manually applying the patches ourselves. ;-) http://projects.scipy.org/scipy/numpy Note that you will need to register a username/password in order to create new tickets. We have had to institute this policy due to egregious amounts of ticket spam. I have no doubt that you will provide such patches and be given SVN access in short order. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From nwagner at iam.uni-stuttgart.de Fri Jul 14 02:43:32 2006 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Fri, 14 Jul 2006 08:43:32 +0200 Subject: [Numpy-discussion] [Fwd: Re: A few comments (MDP conversion)] In-Reply-To: References: <44B66A0A.8030002@iam.uni-stuttgart.de> Message-ID: <44B73D14.3040804@iam.uni-stuttgart.de> Pietro Berkes wrote: > On Thu, 13 Jul 2006, Nils Wagner wrote: > > >> It seems to be line 281 instead of 269. I am using latest svn. >> BTW, in linalg.py in >> >> def pinv: >> >> there is another Complex with capital C. >> > > Well, the problem is not really the capital 'C', but rather the lack of > quotation marks... > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > >>> p = linalg.pinv(a) Traceback (most recent call last): File "", line 1, in ? File "/usr/lib64/python2.4/site-packages/numpy/linalg/linalg.py", line 426, in pinv if issubclass(a.dtype.dtype, complexfloating): AttributeError: 'numpy.dtype' object has no attribute 'dtype' Nils From svetosch at gmx.net Fri Jul 14 03:25:49 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Fri, 14 Jul 2006 09:25:49 +0200 Subject: [Numpy-discussion] numpy.average(fooArray, axis=0): dropping nans from calculation In-Reply-To: References: Message-ID: <44B746FD.9030408@gmx.net> Webb Sprague schrieb: > Could someone recommend a way to average an array along the columns > without propagating the nans and without turning them into some weird > number which bias the result? I guess I can just keep using an > indexing array for fooArray, but if there is somehting more graceful, > I would love to know. You could take advantage of the nan-related functions: >>> help(numpy.nansum) Help on function nansum in module numpy.lib.function_base: nansum(a, axis=-1) Sum the array over the given axis, treating NaNs as 0. > > Boy missing data is a pain in the neck... > Btw, do you know what is treated as NaN in numpy when getting the data from some external source (file, python list etc.), apart from None? I asked that on this list but it apparently went unnoticed. Cheers, Sven From svetosch at gmx.net Fri Jul 14 03:32:42 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Fri, 14 Jul 2006 09:32:42 +0200 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: <44B642A9.3090805@psychology.nottingham.ac.uk> References: <44B642A9.3090805@psychology.nottingham.ac.uk> Message-ID: <44B7489A.1050108@gmx.net> Jon Peirce schrieb: > There used to be a function generalized_inverse in the numpy.linalg > module (certainly in 0.9.2). > > In numpy0.9.8 it seems to have been moved to the numpy.linalg.old > subpackage. Does that mean it's being dropped? Did it have to move? Now > i have to add code to my package to try both locations because my users > might have any version... :-( > > Maybe I don't understand, but what's wrong with numpy.linalg.pinv? -sven From a.h.jaffe at gmail.com Fri Jul 14 04:22:43 2006 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Fri, 14 Jul 2006 09:22:43 +0100 Subject: [Numpy-discussion] debian floating point exceptions/numpy/f2py -- sse Message-ID: Hi All, I have just switched from RHEL to debian, and all of a sudden I started getting floating point exception errors in various contexts. Apparently, this has to do with some faulty error stuff in glibc, specifically related to the sse. I would prefer to fix the actual problem, but I can't seem to be able to get the recommended 'apt-get source glibc' incantation to work (I'm not root, but I can sudo.) I was able to fix some of these issues by simply downgrading ATLAS to not use sse instructions anymore. But f2py still links with sse and sse2 by default. I can't quite understand the configuration well enough to work out how to turn it off. Can someone give me any guidance? Thanks, Andrew From svetosch at gmx.net Fri Jul 14 04:51:44 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Fri, 14 Jul 2006 10:51:44 +0200 Subject: [Numpy-discussion] numpy.average(fooArray, axis=0): dropping nans from calculation In-Reply-To: References: <44B746FD.9030408@gmx.net> <44B75184.5010109@gmx.net> Message-ID: <44B75B20.8020609@gmx.net> Curzio Basso schrieb: >> Well try it out and see for yourself ;-) > > good point :-) > >> But for sums it doesn't make a difference, right... Note that it's >> called nan*sum* and not nanwhatever. > > sure, I was still thinking about the first post which was referring to > the average... > > qrz > Right, having to count the Nans then makes the masked-array solution more attractive, that's true. So maybe that's a feature request, complementing the nansum function by a nanaverage? From david at ar.media.kyoto-u.ac.jp Fri Jul 14 05:31:44 2006 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 14 Jul 2006 18:31:44 +0900 Subject: [Numpy-discussion] debian floating point exceptions/numpy/f2py -- sse In-Reply-To: References: Message-ID: <44B76480.80700@ar.media.kyoto-u.ac.jp> Andrew Jaffe wrote: > Hi All, > > I have just switched from RHEL to debian, and all of a sudden I started > getting floating point exception errors in various contexts. > > Apparently, this has to do with some faulty error stuff in glibc, > specifically related to the sse. I would prefer to fix the actual > problem, but I can't seem to be able to get the recommended 'apt-get > source glibc' incantation to work (I'm not root, but I can sudo.) > What does not work ? The apt-get source part ? The actual building ? Basically, if the sources are OK, you just need to do - fakeroot dpkg-source -x name_of_dsc_file - cd name_of_package - fakeroot dpkg-buildpackage And that's it > I was able to fix some of these issues by simply downgrading ATLAS to > not use sse instructions anymore. > > But f2py still links with sse and sse2 by default. I can't quite > understand the configuration well enough to work out how to turn it off. > Can someone give me any guidance? > The way it is supposed to work, at least on debian and ubuntu, is that you never link to the sse/sse2 versions, but to the non-optimized versions. After, the dynamic loader will get the right one (ie optimized if available) instead of the one linked (this of course only works for dynamic linking). You can check which library is picked with a ldconfig - p | grep lapack (for lapack functions, and so on...) David From aisaac at american.edu Fri Jul 14 09:26:19 2006 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 14 Jul 2006 09:26:19 -0400 Subject: [Numpy-discussion] numpy.average(fooArray, axis=0): dropping nans from calculation In-Reply-To: <44B75B20.8020609@gmx.net> References: <44B746FD.9030408@gmx.net> <44B75184.5010109@gmx.net><44B75B20.8020609@gmx.net> Message-ID: On Fri, 14 Jul 2006, Sven Schreiber apparently wrote: > So maybe that's a feature request, complementing the > nansum function by a nanaverage? This is not an objection; just an observation. It has always seemed to me that such descriptive statistics make more sense as class attributes. In this case, something like a NanDstat class. fwiw, Alan Isaac From laidler at stsci.edu Fri Jul 14 11:56:38 2006 From: laidler at stsci.edu (Victoria G. Laidler) Date: Fri, 14 Jul 2006 11:56:38 -0400 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: <44B7489A.1050108@gmx.net> References: <44B642A9.3090805@psychology.nottingham.ac.uk> <44B7489A.1050108@gmx.net> Message-ID: <44B7BEB6.8070608@stsci.edu> Sven Schreiber wrote: >Jon Peirce schrieb: > > >>There used to be a function generalized_inverse in the numpy.linalg >>module (certainly in 0.9.2). >> >>In numpy0.9.8 it seems to have been moved to the numpy.linalg.old >>subpackage. Does that mean it's being dropped? Did it have to move? Now >>i have to add code to my package to try both locations because my users >>might have any version... :-( >> >> >> >> > >Maybe I don't understand, but what's wrong with numpy.linalg.pinv? > > Er, what's a pinv? It doesn't sound anything like a generalized_inverse. Vicki Laidler From jonathan.taylor at stanford.edu Fri Jul 14 11:59:49 2006 From: jonathan.taylor at stanford.edu (Jonathan Taylor) Date: Fri, 14 Jul 2006 08:59:49 -0700 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: <44B7BEB6.8070608@stsci.edu> References: <44B642A9.3090805@psychology.nottingham.ac.uk> <44B7489A.1050108@gmx.net> <44B7BEB6.8070608@stsci.edu> Message-ID: <44B7BF75.2080708@stanford.edu> pseudoinverse it's the same name matlab uses: http://www.mathworks.com/access/helpdesk/help/techdoc/ref/pinv.html Victoria G. Laidler wrote: >Sven Schreiber wrote: > > > >>Jon Peirce schrieb: >> >> >> >> >>>There used to be a function generalized_inverse in the numpy.linalg >>>module (certainly in 0.9.2). >>> >>>In numpy0.9.8 it seems to have been moved to the numpy.linalg.old >>>subpackage. Does that mean it's being dropped? Did it have to move? Now >>>i have to add code to my package to try both locations because my users >>>might have any version... :-( >>> >>> >>> >>> >>> >>> >>Maybe I don't understand, but what's wrong with numpy.linalg.pinv? >> >> >> >> >Er, what's a pinv? It doesn't sound anything like a generalized_inverse. > >Vicki Laidler > > > >------------------------------------------------------------------------- >Using Tomcat but need to do more? Need to support web services, security? >Get stuff done quickly with pre-integrated technology to make your job easier >Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo >http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > -- ------------------------------------------------------------------------ I'm part of the Team in Training: please support our efforts for the Leukemia and Lymphoma Society! http://www.active.com/donate/tntsvmb/tntsvmbJTaylor GO TEAM !!! ------------------------------------------------------------------------ Jonathan Taylor Tel: 650.723.9230 Dept. of Statistics Fax: 650.725.8977 Sequoia Hall, 137 www-stat.stanford.edu/~jtaylo 390 Serra Mall Stanford, CA 94305 From laidler at stsci.edu Fri Jul 14 12:16:58 2006 From: laidler at stsci.edu (Victoria G. Laidler) Date: Fri, 14 Jul 2006 12:16:58 -0400 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: <44B7BF75.2080708@stanford.edu> References: <44B642A9.3090805@psychology.nottingham.ac.uk> <44B7489A.1050108@gmx.net> <44B7BEB6.8070608@stsci.edu> <44B7BF75.2080708@stanford.edu> Message-ID: <44B7C37A.7010801@stsci.edu> Jonathan Taylor wrote: > pseudoinverse > > it's the same name matlab uses: > > http://www.mathworks.com/access/helpdesk/help/techdoc/ref/pinv.html Thanks for the explanation. I'm puzzled by the naming choice, however. Standard best practice in writing software is to give understandable names, to improve readability and code maintenance. Obscure abbreviations like "pinv" pretty much went out with the FORTRAN 9-character limit for variable names. It's very unusual to see them in new software nowadays, and it always looks unprofessional to me. I understand that for interactive use, short names are more convenient; but shouldn't they be available aliases to the more general names? Since numpy is primarily a software library, I wouldn't expect it to sacrifice a standard best-practice in order to make things more convenient for interactive use. If the concern is for for matlab compatibility, maybe a synonym module numpy.as_matlab could define all the synonyms, that matlab users could then use? That would make more sense to me than inflicting obscure matlab names on the rest of the user community. Vicki Laidler > > Victoria G. Laidler wrote: > >> Sven Schreiber wrote: >> >> >> >>> Jon Peirce schrieb: >>> >>> >>> >>> >>>> There used to be a function generalized_inverse in the numpy.linalg >>>> module (certainly in 0.9.2). >>>> >>>> In numpy0.9.8 it seems to have been moved to the numpy.linalg.old >>>> subpackage. Does that mean it's being dropped? Did it have to move? >>>> Now i have to add code to my package to try both locations because >>>> my users might have any version... :-( >>>> >>>> >>>> >>>> >>> >>> Maybe I don't understand, but what's wrong with numpy.linalg.pinv? >>> >>> >>> >> >> Er, what's a pinv? It doesn't sound anything like a generalized_inverse. >> >> Vicki Laidler >> >> >> >> ------------------------------------------------------------------------- >> >> Using Tomcat but need to do more? Need to support web services, >> security? >> Get stuff done quickly with pre-integrated technology to make your >> job easier >> Download IBM WebSphere Application Server v.1.0.1 based on Apache >> Geronimo >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> >> > From strawman at astraw.com Fri Jul 14 12:28:57 2006 From: strawman at astraw.com (Andrew Straw) Date: Fri, 14 Jul 2006 09:28:57 -0700 Subject: [Numpy-discussion] debian floating point exceptions/numpy/f2py -- sse In-Reply-To: References: Message-ID: <44B7C649.8080204@astraw.com> GNU libc version 2.3.2 has a bug[1] "feclearexcept() error on CPUs with SSE" (fixed in 2.3.3) which has been submitted to Debian[2] but not fixed in sarge. See http://www.its.caltech.edu/~astraw/coding.html#id3 for more information and .debs which fix the problem. [1] http://sources.redhat.com/bugzilla/show_bug.cgi?id=10 [2] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=279294 If only somebody could show how this was a security issue, we could get Debian to finally release a fix in sarge for this (frequently asked) question. Andrew Jaffe wrote: > Hi All, > > I have just switched from RHEL to debian, and all of a sudden I started > getting floating point exception errors in various contexts. > > Apparently, this has to do with some faulty error stuff in glibc, > specifically related to the sse. I would prefer to fix the actual > problem, but I can't seem to be able to get the recommended 'apt-get > source glibc' incantation to work (I'm not root, but I can sudo.) > > I was able to fix some of these issues by simply downgrading ATLAS to > not use sse instructions anymore. > > But f2py still links with sse and sse2 by default. I can't quite > understand the configuration well enough to work out how to turn it off. > Can someone give me any guidance? > > Thanks, > > Andrew > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From efiring at hawaii.edu Fri Jul 14 12:28:53 2006 From: efiring at hawaii.edu (Eric Firing) Date: Fri, 14 Jul 2006 06:28:53 -1000 Subject: [Numpy-discussion] numpy.average(fooArray, axis=0): dropping nans from calculation In-Reply-To: References: <44B746FD.9030408@gmx.net> <44B75184.5010109@gmx.net> <44B75B20.8020609@gmx.net> Message-ID: <44B7C645.2090507@hawaii.edu> Alan G Isaac wrote: > On Fri, 14 Jul 2006, Sven Schreiber apparently wrote: > >>So maybe that's a feature request, complementing the >>nansum function by a nanaverage? > > > This is not an objection; just an observation. > It has always seemed to me that such descriptive > statistics make more sense as class attributes. > In this case, something like a NanDstat class. Attached is something like that, in case someone finds it useful. It is designed to replace something I wrote a long time ago for matlab. It is only very lightly tested, so use with care. Eric -------------- next part -------------- A non-text attachment was scrubbed... Name: mstdgap.py Type: text/x-python Size: 5659 bytes Desc: not available URL: From a.h.jaffe at gmail.com Fri Jul 14 12:35:36 2006 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Fri, 14 Jul 2006 17:35:36 +0100 Subject: [Numpy-discussion] debian floating point exceptions/numpy/f2py -- sse In-Reply-To: <44B76480.80700@ar.media.kyoto-u.ac.jp> References: <44B76480.80700@ar.media.kyoto-u.ac.jp> Message-ID: <44B7C7D8.3090800@gmail.com> David Cournapeau wrote: > Andrew Jaffe wrote: >> Hi All, >> >> I have just switched from RHEL to debian, and all of a sudden I started >> getting floating point exception errors in various contexts. >> >> Apparently, this has to do with some faulty error stuff in glibc, >> specifically related to the sse. I would prefer to fix the actual >> problem, but I can't seem to be able to get the recommended 'apt-get >> source glibc' incantation to work (I'm not root, but I can sudo.) >> > What does not work ? The apt-get source part ? The actual building ? > > Basically, if the sources are OK, you just need to do > - fakeroot dpkg-source -x name_of_dsc_file > - cd name_of_package > - fakeroot dpkg-buildpackage > > And that's it Aha -- I didn't know about fakeroot... Thanks! That was the problem with apt-get source. I'm compiling the patched version now... wow, is it slow! >> I was able to fix some of these issues by simply downgrading ATLAS to >> not use sse instructions anymore. >> >> But f2py still links with sse and sse2 by default. I can't quite >> understand the configuration well enough to work out how to turn it off. >> Can someone give me any guidance? >> > The way it is supposed to work, at least on debian and ubuntu, is that > you never link to the sse/sse2 versions, but to the non-optimized > versions. After, the dynamic loader will get the right one (ie optimized > if available) instead of the one linked (this of course only works for > dynamic linking). You can check which library is picked with a ldconfig > - p | grep lapack (for lapack functions, and so on...) The problem with f2py isn't the atlas/lapack linkage, which it does correctly, but the fact that it automatically appends -sse2 to the g77 compile options; I couldn't figure out how to turn that off! Although now I'm not so sure, since I can never get my self-compiled executable version of my fortran code to give the same error as when it runs within python. But with the patched glibc, I think I'm alright in any event! Thanks! A From a.h.jaffe at gmail.com Fri Jul 14 12:35:36 2006 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Fri, 14 Jul 2006 17:35:36 +0100 Subject: [Numpy-discussion] debian floating point exceptions/numpy/f2py -- sse In-Reply-To: <44B76480.80700@ar.media.kyoto-u.ac.jp> References: <44B76480.80700@ar.media.kyoto-u.ac.jp> Message-ID: <44B7C7D8.3090800@gmail.com> David Cournapeau wrote: > Andrew Jaffe wrote: >> Hi All, >> >> I have just switched from RHEL to debian, and all of a sudden I started >> getting floating point exception errors in various contexts. >> >> Apparently, this has to do with some faulty error stuff in glibc, >> specifically related to the sse. I would prefer to fix the actual >> problem, but I can't seem to be able to get the recommended 'apt-get >> source glibc' incantation to work (I'm not root, but I can sudo.) >> > What does not work ? The apt-get source part ? The actual building ? > > Basically, if the sources are OK, you just need to do > - fakeroot dpkg-source -x name_of_dsc_file > - cd name_of_package > - fakeroot dpkg-buildpackage > > And that's it Aha -- I didn't know about fakeroot... Thanks! That was the problem with apt-get source. I'm compiling the patched version now... wow, is it slow! >> I was able to fix some of these issues by simply downgrading ATLAS to >> not use sse instructions anymore. >> >> But f2py still links with sse and sse2 by default. I can't quite >> understand the configuration well enough to work out how to turn it off. >> Can someone give me any guidance? >> > The way it is supposed to work, at least on debian and ubuntu, is that > you never link to the sse/sse2 versions, but to the non-optimized > versions. After, the dynamic loader will get the right one (ie optimized > if available) instead of the one linked (this of course only works for > dynamic linking). You can check which library is picked with a ldconfig > - p | grep lapack (for lapack functions, and so on...) The problem with f2py isn't the atlas/lapack linkage, which it does correctly, but the fact that it automatically appends -sse2 to the g77 compile options; I couldn't figure out how to turn that off! Although now I'm not so sure, since I can never get my self-compiled executable version of my fortran code to give the same error as when it runs within python. But with the patched glibc, I think I'm alright in any event! Thanks! A From nvf at MIT.EDU Fri Jul 14 12:43:38 2006 From: nvf at MIT.EDU (Nick Fotopoulos) Date: Fri, 14 Jul 2006 12:43:38 -0400 Subject: [Numpy-discussion] numpy.vectorize performance In-Reply-To: <44B6FECD.8000004@cox.net> References: <44B6FECD.8000004@cox.net> Message-ID: <432FD99A-ECC4-423C-B089-C5077D230A17@mit.edu> On Jul 13, 2006, at 10:17 PM, Tim Hochberg wrote: > Nick Fotopoulos wrote: >> Dear all, >> >> I often make use of numpy.vectorize to make programs read more >> like the physics equations I write on paper. numpy.vectorize is >> basically a wrapper for numpy.frompyfunc. Reading Travis's Scipy >> Book (mine is dated Jan 6 2005) kind of suggests to me that it >> returns a full- fledged ufunc exactly like built-in ufuncs. >> >> First, is this true? > Well according to type(), the result of frompyfunc is indeed of > type ufunc, so I would say the answer to that is yes. >> Second, how is the performance? > A little timing indicates that it's not good (about 30 X slower for > computing x**2 than doing it using x*x on an array). . That's not > frompyfunc (or vectorizes) fault though. It's calling a python > function at each point, so the python function call overhead is > going to kill you. Not to mention instantiating an actual Python > object or objects at each point. That's unfortunate since I tend to nest functions quite deeply and then scipy.integrate.quad over them, which I'm sure results in a ridiculous number of function calls. Are anonymous lambdas any different than named functions in terms of performance? > >> i.e., are my functions performing approximately as fast as they >> could be or would they still gain a great deal of speed by >> rewriting it in C or some other compiled python accelerator? >> > Can you give examples of what these functions look like? You might > gain a great deal of speed by rewriting them in numpy in the > correct way. Or perhaps not, but it's probably worth showing some > examples so we can offer suggestions or at least admit that we are > stumped. This is by far the slowest bit of my code. I cache the results, so it's not too bad, but any upstream tweak can take a lot of CPU time to propagate. @autovectorized def dnsratezfunc(z): """Take coalescence time into account."" def integrand(zf): return Pz(z,zf)*NSbirthzfunc(zf) return quad(integrand,delayedZ(2e5*secperyear+1,z),5)[0] dnsratez = lambdap*dnsratezfunc(zs) where: # Neutron star formation rate is a delayed version of star formation rate NSbirthzfunc = autovectorized(lambda z: SFRz(delayedZ (1e8*secperyear,z))) def Pz(z_c,z_f): """Return the probability density per unit redshift of a DNS coalescence at z_c given a progenitor formation at z_f. """ return P(t(z_c,z_f))*dtdz(z_c) and there are many further nested levels of function calls. If the act of calling a function is more expensive than actually executing it and I value speed over readability/code reuse, I can inline Pz's function calls and inline the unvectorized NSbirthzfunc to reduce the calling stack a bit. Any other suggestions? Thanks, Tim. Take care, Nick From tim.hochberg at cox.net Fri Jul 14 12:56:32 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri, 14 Jul 2006 09:56:32 -0700 Subject: [Numpy-discussion] numpy.vectorize performance In-Reply-To: <432FD99A-ECC4-423C-B089-C5077D230A17@mit.edu> References: <44B6FECD.8000004@cox.net> <432FD99A-ECC4-423C-B089-C5077D230A17@mit.edu> Message-ID: <44B7CCC0.8050100@cox.net> Nick Fotopoulos wrote: > On Jul 13, 2006, at 10:17 PM, Tim Hochberg wrote: > > >> Nick Fotopoulos wrote: >> >>> Dear all, >>> >>> I often make use of numpy.vectorize to make programs read more >>> like the physics equations I write on paper. numpy.vectorize is >>> basically a wrapper for numpy.frompyfunc. Reading Travis's Scipy >>> Book (mine is dated Jan 6 2005) kind of suggests to me that it >>> returns a full- fledged ufunc exactly like built-in ufuncs. >>> >>> First, is this true? >>> >> Well according to type(), the result of frompyfunc is indeed of >> type ufunc, so I would say the answer to that is yes. >> >>> Second, how is the performance? >>> >> A little timing indicates that it's not good (about 30 X slower for >> computing x**2 than doing it using x*x on an array). . That's not >> frompyfunc (or vectorizes) fault though. It's calling a python >> function at each point, so the python function call overhead is >> going to kill you. Not to mention instantiating an actual Python >> object or objects at each point. >> > > That's unfortunate since I tend to nest functions quite deeply and > then scipy.integrate.quad over them, which I'm sure results in a > ridiculous number of function calls. Are anonymous lambdas any > different than named functions in terms of performance? > Sorry, no. Under the covers they're the same. >>> i.e., are my functions performing approximately as fast as they >>> could be or would they still gain a great deal of speed by >>> rewriting it in C or some other compiled python accelerator? >>> >>> >> Can you give examples of what these functions look like? You might >> gain a great deal of speed by rewriting them in numpy in the >> correct way. Or perhaps not, but it's probably worth showing some >> examples so we can offer suggestions or at least admit that we are >> stumped. >> > > This is by far the slowest bit of my code. I cache the results, so > it's not too bad, but any upstream tweak can take a lot of CPU time > to propagate. > > @autovectorized > def dnsratezfunc(z): > """Take coalescence time into account."" > def integrand(zf): > return Pz(z,zf)*NSbirthzfunc(zf) > return quad(integrand,delayedZ(2e5*secperyear+1,z),5)[0] > dnsratez = lambdap*dnsratezfunc(zs) > > where: > > # Neutron star formation rate is a delayed version of star formation > rate > NSbirthzfunc = autovectorized(lambda z: SFRz(delayedZ > (1e8*secperyear,z))) > > def Pz(z_c,z_f): > """Return the probability density per unit redshift of a DNS > coalescence at z_c given a progenitor formation at z_f. """ > return P(t(z_c,z_f))*dtdz(z_c) > > and there are many further nested levels of function calls. If the > act of calling a function is more expensive than actually executing > it and I value speed over readability/code reuse, I can inline Pz's > function calls and inline the unvectorized NSbirthzfunc to reduce the > calling stack a bit. Any other suggestions? > I think I'd try psyco (http://psyco.sourceforge.net/). That's pretty painless to try and may result in a significant improvement. -tim From travis at enthought.com Fri Jul 14 13:19:11 2006 From: travis at enthought.com (Travis N. Vaught) Date: Fri, 14 Jul 2006 12:19:11 -0500 Subject: [Numpy-discussion] ANN: SciPy 2006 Schedule/Early Registration Reminder Message-ID: <44B7D20F.8060701@enthought.com> Greetings, The SciPy 2006 Conference (http://www.scipy.org/SciPy2006) is August 17-18 this year. The deadline for early registration is *today*, July 14, 2006. The registration price will increase from $100 to $150 after today. You can register online at https://www.enthought.com/scipy06 . We invite everyone attending the conference to also attend the Coding Sprints on Monday-Tuesday , August 14-15 and also the Tutorials Wednesday, August 16. There is no additional charge for these sessions. A *tentative* schedule of talks has now been posted. http://www.scipy.org/SciPy2006/Schedule We look forward to seeing you at CalTech in August! Best, Travis From robert.kern at gmail.com Fri Jul 14 13:29:39 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 14 Jul 2006 12:29:39 -0500 Subject: [Numpy-discussion] [Fwd: Re: A few comments (MDP conversion)] In-Reply-To: <44B73D14.3040804@iam.uni-stuttgart.de> References: <44B66A0A.8030002@iam.uni-stuttgart.de> <44B73D14.3040804@iam.uni-stuttgart.de> Message-ID: Nils Wagner wrote: > >>> p = linalg.pinv(a) > Traceback (most recent call last): > File "", line 1, in ? > File "/usr/lib64/python2.4/site-packages/numpy/linalg/linalg.py", line > 426, in pinv > if issubclass(a.dtype.dtype, complexfloating): > AttributeError: 'numpy.dtype' object has no attribute 'dtype' Unless if you have something substantive to actually add to the discussion, the ticket you submitted is sufficient to report the bug. There are reasons we have a bug tracker, and one of them is to keep otherwise contentless bug reports off the mailing list. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From nvf at MIT.EDU Fri Jul 14 13:39:07 2006 From: nvf at MIT.EDU (Nick Fotopoulos) Date: Fri, 14 Jul 2006 13:39:07 -0400 Subject: [Numpy-discussion] numpy.vectorize performance In-Reply-To: <44B7CCC0.8050100@cox.net> References: <44B6FECD.8000004@cox.net> <432FD99A-ECC4-423C-B089-C5077D230A17@mit.edu> <44B7CCC0.8050100@cox.net> Message-ID: On Jul 14, 2006, at 12:56 PM, Tim Hochberg wrote: > I think I'd try psyco (http://psyco.sourceforge.net/). That's > pretty painless to try and may result in a significant improvement. I've been doing more and more development on my PPC Mac, where psyco is not an option. If the speed issue really gets to me, I can run things with psyco on a linux box. Thanks, Nick From wbaxter at gmail.com Fri Jul 14 15:01:29 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 15 Jul 2006 04:01:29 +0900 Subject: [Numpy-discussion] [Fwd: Re: A few comments (MDP conversion)] In-Reply-To: References: <44B66A0A.8030002@iam.uni-stuttgart.de> <44B73D14.3040804@iam.uni-stuttgart.de> Message-ID: On 7/15/06, Robert Kern wrote: > Nils Wagner wrote: > > > >>> p = linalg.pinv(a) > > Traceback (most recent call last): > > File "", line 1, in ? > > File "/usr/lib64/python2.4/site-packages/numpy/linalg/linalg.py", line > > 426, in pinv > > if issubclass(a.dtype.dtype, complexfloating): > > AttributeError: 'numpy.dtype' object has no attribute 'dtype' > > Unless if you have something substantive to actually add to the discussion, the > ticket you submitted is sufficient to report the bug. There are reasons we have > a bug tracker, and one of them is to keep otherwise contentless bug reports off > the mailing list. If that's really the policy, then this page should probably be changed: http://www.scipy.org/Developer_Zone Quote: "Bugs should be reported to one of the appropriate Mailing Lists. Do this first, and open a ticket on the corresponding developer's wiki if necessary." --Bill From robert.kern at gmail.com Fri Jul 14 15:11:31 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 14 Jul 2006 14:11:31 -0500 Subject: [Numpy-discussion] [Fwd: Re: A few comments (MDP conversion)] In-Reply-To: References: <44B66A0A.8030002@iam.uni-stuttgart.de> <44B73D14.3040804@iam.uni-stuttgart.de> Message-ID: Bill Baxter wrote: > On 7/15/06, Robert Kern wrote: >> Nils Wagner wrote: >> >>> >>> p = linalg.pinv(a) >>> Traceback (most recent call last): >>> File "", line 1, in ? >>> File "/usr/lib64/python2.4/site-packages/numpy/linalg/linalg.py", line >>> 426, in pinv >>> if issubclass(a.dtype.dtype, complexfloating): >>> AttributeError: 'numpy.dtype' object has no attribute 'dtype' >> Unless if you have something substantive to actually add to the discussion, the >> ticket you submitted is sufficient to report the bug. There are reasons we have >> a bug tracker, and one of them is to keep otherwise contentless bug reports off >> the mailing list. > > If that's really the policy, then this page should probably be changed: > http://www.scipy.org/Developer_Zone > > Quote: > "Bugs should be reported to one of the appropriate Mailing Lists. Do > this first, and open a ticket on the corresponding developer's wiki if > necessary." Yes, that *should* be fixed. In January it was a reasonable policy, but with the ticket mailing lists, it's not useful anymore. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cookedm at physics.mcmaster.ca Fri Jul 14 15:21:11 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri, 14 Jul 2006 15:21:11 -0400 Subject: [Numpy-discussion] [Fwd: Re: A few comments (MDP conversion)] In-Reply-To: References: <44B66A0A.8030002@iam.uni-stuttgart.de> <44B73D14.3040804@iam.uni-stuttgart.de> Message-ID: <20060714152111.29c9d4a6@arbutus.physics.mcmaster.ca> On Fri, 14 Jul 2006 12:29:39 -0500 Robert Kern wrote: > Nils Wagner wrote: > > > >>> p = linalg.pinv(a) > > Traceback (most recent call last): > > File "", line 1, in ? > > File "/usr/lib64/python2.4/site-packages/numpy/linalg/linalg.py", line > > 426, in pinv > > if issubclass(a.dtype.dtype, complexfloating): > > AttributeError: 'numpy.dtype' object has no attribute 'dtype' > > Unless if you have something substantive to actually add to the discussion, > the ticket you submitted is sufficient to report the bug. There are reasons > we have a bug tracker, and one of them is to keep otherwise contentless bug > reports off the mailing list. For an obvious typo like this, made on a recent commit (i.e., in the last day), it's probably just enough to email the committer (me, in this case). Or, at least, I don't mind. [this kind of error sneaking by is b/c we don't have enough test cases :-)] -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From svetosch at gmx.net Fri Jul 14 17:16:28 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Fri, 14 Jul 2006 23:16:28 +0200 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: <44B7C37A.7010801@stsci.edu> References: <44B642A9.3090805@psychology.nottingham.ac.uk> <44B7489A.1050108@gmx.net> <44B7BEB6.8070608@stsci.edu> <44B7BF75.2080708@stanford.edu> <44B7C37A.7010801@stsci.edu> Message-ID: <44B809AC.5080701@gmx.net> Victoria G. Laidler schrieb: > > I understand that for interactive use, short names are more convenient; > but shouldn't they be available aliases to the more general names? Since > numpy is primarily a software library, I wouldn't expect it to sacrifice > a standard best-practice in order to make things more convenient for > interactive use. I don't necessarily agree that numpy should aim to be primarily a library, but I'm with you concerning the alias idea. However, iirc there was some discussion recently on this list about the dual solution (long names as well as short ones in parallel), and some important numpy people had some reservations, although I don't remember exactly what those were -- probably some Python Zen issues ("there should be only one way to get to Rome", was that it? -- just kidding). > > If the concern is for for matlab compatibility, maybe a synonym module > numpy.as_matlab could define all the synonyms, that matlab users could > then use? That would make more sense to me than inflicting obscure > matlab names on the rest of the user community. >From superficially browsing through the numpy guide my subjective impression is that function names are mostly pretty short. So maybe the alias thing should work the other way around, making long names available in a module numpy.long_names_for_typing_addicts (again, a bad joke...) cheers, Sven From kwgoodman at gmail.com Fri Jul 14 17:54:09 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 14 Jul 2006 14:54:09 -0700 Subject: [Numpy-discussion] [Fwd: Re: A few comments (MDP conversion)] In-Reply-To: References: <44B66A0A.8030002@iam.uni-stuttgart.de> <44B73D14.3040804@iam.uni-stuttgart.de> Message-ID: On 7/14/06, Bill Baxter wrote: > On 7/15/06, Robert Kern wrote: > > Nils Wagner wrote: > > > > > >>> p = linalg.pinv(a) > > > Traceback (most recent call last): > > > File "", line 1, in ? > > > File "/usr/lib64/python2.4/site-packages/numpy/linalg/linalg.py", line > > > 426, in pinv > > > if issubclass(a.dtype.dtype, complexfloating): > > > AttributeError: 'numpy.dtype' object has no attribute 'dtype' > > > > Unless if you have something substantive to actually add to the discussion, the > > ticket you submitted is sufficient to report the bug. There are reasons we have > > a bug tracker, and one of them is to keep otherwise contentless bug reports off > > the mailing list. > > If that's really the policy, then this page should probably be changed: > http://www.scipy.org/Developer_Zone > > Quote: > "Bugs should be reported to one of the appropriate Mailing Lists. Do > this first, and open a ticket on the corresponding developer's wiki if > necessary." I like seeing the bug reports on the list. It is an easy way to get alerts of what to look out for. From robert.kern at gmail.com Fri Jul 14 18:00:36 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 14 Jul 2006 17:00:36 -0500 Subject: [Numpy-discussion] [Fwd: Re: A few comments (MDP conversion)] In-Reply-To: References: <44B66A0A.8030002@iam.uni-stuttgart.de> <44B73D14.3040804@iam.uni-stuttgart.de> Message-ID: Keith Goodman wrote: > I like seeing the bug reports on the list. It is an easy way to get > alerts of what to look out for. There are read-only mailing lists for new and updated tickets: http://www.scipy.org/Mailing_Lists -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From fperez.net at gmail.com Fri Jul 14 20:52:58 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 14 Jul 2006 18:52:58 -0600 Subject: [Numpy-discussion] Scalars, comparisons and indexing. Is this a bug? Message-ID: Hi all, I just got bit by this problem, and I'm not really sure if this is something that should be considered a numpy bug or not. It is a bit unpleasant, at the least, though it can be easily worked around. Here's a simple demonstration: In [32]: a=array([1,2]) In [33]: b=1 In [34]: (1,2)[a[0]==b] --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /home/fperez/research/code/mwmerge/mwadap-merge/mwadap/test/ TypeError: tuple indices must be integers Whereas this works: In [35]: c=2 In [36]: (1,2)[c==b] Out[36]: 1 Basically, it seems that array scalars, upon comparisons, return something that doesn't 'looke enough like an int' to python for it to let you use it as an index, it's a 'boolscalar': In [38]: a0==b0 Out[38]: True In [39]: type _ -------> type(_) Out[39]: Advice? Is this a numpy bug? Or should it just be left alone, and this kind of inconvenience will disappear when 2.5 is out, with __index__ support? Cheers, f From kwgoodman at gmail.com Fri Jul 14 21:22:18 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 14 Jul 2006 18:22:18 -0700 Subject: [Numpy-discussion] Atlas and dual core Message-ID: Is there much speed to be gained by compiling atlas for a dual core system? I'm running Ubuntu on a Macbook. It's the first time I've had a dual core system. My one line benchmark shows that the Macbook is slow compared to my (old) desktop. >> t1=time.time();x=randn(500,1000);x = x*x.T;a,b=linalg.eigh(x);t2=time.time();print t2-t1 1.31429600716 My desktop is less than half of that. From wbaxter at gmail.com Fri Jul 14 21:26:06 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 15 Jul 2006 10:26:06 +0900 Subject: [Numpy-discussion] Scalars, comparisons and indexing. Is this a bug? In-Reply-To: References: Message-ID: I believe that's the problem that the indexing PEP from Travis is supposed to address: http://www.python.org/dev/peps/pep-0357/ So I think there's not much anyone can do about it untill the PEP is accepted and implemented. Maybe you can cast to int? > In [34]: (1,2)[int(a[0]==b)] --bb On 7/15/06, Fernando Perez wrote: > Hi all, > > I just got bit by this problem, and I'm not really sure if this is > something that should be considered a numpy bug or not. It is a bit > unpleasant, at the least, though it can be easily worked around. > Here's a simple demonstration: > > In [32]: a=array([1,2]) > > In [33]: b=1 > > In [34]: (1,2)[a[0]==b] > --------------------------------------------------------------------------- > exceptions.TypeError Traceback (most > recent call last) > > /home/fperez/research/code/mwmerge/mwadap-merge/mwadap/test/ > > TypeError: tuple indices must be integers > > > Whereas this works: > > In [35]: c=2 > > In [36]: (1,2)[c==b] > Out[36]: 1 > > Basically, it seems that array scalars, upon comparisons, return > something that doesn't 'looke enough like an int' to python for it to > let you use it as an index, it's a 'boolscalar': > > In [38]: a0==b0 > Out[38]: True > > In [39]: type _ > -------> type(_) > Out[39]: > > > Advice? Is this a numpy bug? Or should it just be left alone, and > this kind of inconvenience will disappear when 2.5 is out, with > __index__ support? > > Cheers, > > f > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- William V. Baxter III OLM Digital Kono Dens Building Rm 302 1-8-8 Wakabayashi Setagaya-ku Tokyo, Japan 154-0023 +81 (3) 3422-3380 From fperez.net at gmail.com Fri Jul 14 21:36:09 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 14 Jul 2006 19:36:09 -0600 Subject: [Numpy-discussion] Scalars, comparisons and indexing. Is this a bug? In-Reply-To: References: Message-ID: On 7/14/06, Bill Baxter wrote: > I believe that's the problem that the indexing PEP from Travis is > supposed to address: > http://www.python.org/dev/peps/pep-0357/ > So I think there's not much anyone can do about it untill the PEP is > accepted and implemented. > > Maybe you can cast to int? > > In [34]: (1,2)[int(a[0]==b)] Yup, that's the workaround I'm using. I was just wondering if comparisons between array scalars shouldn't return /true/ booleans (which can be used as indices) instead of array scalar booleans. I realize that the __index__ support in 2.5 will make this point moot (the PEP you point to), but perhaps this particular one can actually be fixed today, for all users of python pre-2.5. However, I haven't really wrapped my head enough around all the subtleties of array scalars to know whether 'fixing' this particular problem will introduce other, worse ones. Cheers, f From lists.steve at arachnedesign.net Sat Jul 15 10:57:28 2006 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Sat, 15 Jul 2006 10:57:28 -0400 Subject: [Numpy-discussion] Atlas and dual core In-Reply-To: References: Message-ID: <49832F8C-3FE7-4575-912F-AC4AFF440F55@arachnedesign.net> Hi Keith, I don't have any answers for you but ... 1) Did you install ubuntu by way of Parallels, or are you running linux "full steam" 2) If you did install ubuntu alongside/over/whatever mac os x and didn't use parallels .. are there any links you can share w/ a few tutorials? Is everything (else: like hardware support) working well? (Maybe off list since it's OT -- but maybe others are interested) 3) If you still have your Mac OS X partition lying around, the full r- platform installer [1] for the mac comes with an installer for a threaded compile of ATLAS (specidifally for the intel duo's .. somehow) so maybe you'd like to try to use that and save yourself some time ... but perhaps that whole malloc vs dmalloc report [2] might apply to this as well ... which I guess would suggest to use linux for maximal performance (refer to q 2 :-) -steve [1] : http://cran.r-project.org/bin/macosx/ [2] : http://sekhon.berkeley.edu/macosx/intel.html On Jul 14, 2006, at 9:22 PM, Keith Goodman wrote: > Is there much speed to be gained by compiling atlas for a dual core > system? > > I'm running Ubuntu on a Macbook. It's the first time I've had a dual > core system. > > My one line benchmark shows that the Macbook is slow compared to my > (old) desktop. > >>> t1=time.time();x=randn(500,1000);x = > x*x.T;a,b=linalg.eigh(x);t2=time.time();print t2-t1 > 1.31429600716 > > My desktop is less than half of that. > > > ---------------------------------------------------------------------- > --- > Using Tomcat but need to do more? Need to support web services, > security? > Get stuff done quickly with pre-integrated technology to make your > job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel? > cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From kwgoodman at gmail.com Sat Jul 15 11:14:52 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Sat, 15 Jul 2006 08:14:52 -0700 Subject: [Numpy-discussion] Atlas and dual core In-Reply-To: <49832F8C-3FE7-4575-912F-AC4AFF440F55@arachnedesign.net> References: <49832F8C-3FE7-4575-912F-AC4AFF440F55@arachnedesign.net> Message-ID: On 7/15/06, Steve Lianoglou wrote: > 1) Did you install ubuntu by way of Parallels, or are you running > linux "full steam" Yes, it is a full steam dual boot system. My default boot is Ubuntu. > 2) If you did install ubuntu alongside/over/whatever mac os x and > didn't use parallels .. are there any links you can share w/ a few > tutorials? Is everything (else: like hardware support) working well? > (Maybe off list since it's OT -- but maybe others are interested) Here are the references I used: http://bin-false.org/?p=17 http://www.planetisaac.com/articles/ubuntuinstall.html (The second one, which was an annotated version of the first one, seems to have disappeared.) No, not everything is working. And the installation is not easy (for me). I think it will become easy in a few months. But numpy and scipy were easy to compile! If you run into any problems, just send me a note off list. From david.huard at gmail.com Sat Jul 15 21:38:17 2006 From: david.huard at gmail.com (David Huard) Date: Sat, 15 Jul 2006 21:38:17 -0400 Subject: [Numpy-discussion] numpy.vectorize performance In-Reply-To: <432FD99A-ECC4-423C-B089-C5077D230A17@mit.edu> References: <44B6FECD.8000004@cox.net> <432FD99A-ECC4-423C-B089-C5077D230A17@mit.edu> Message-ID: <91cf711d0607151838p69149c73q5d479dbecaae78a0@mail.gmail.com> 2006/7/14, Nick Fotopoulos : > > Any other suggestions? > Hi Nick, I had some success by coding the integrand in fortran and wrapping it with f2py. If your probability density function is standard, you may find it in the flib library of the PyMC module of Chris Fonnesbeck ( a library of likelihood functions coded in f77) and save the trouble. Hope this helps, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Sun Jul 16 00:01:36 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 15 Jul 2006 22:01:36 -0600 Subject: [Numpy-discussion] numpy.vectorize performance In-Reply-To: References: Message-ID: <44B9BA20.7000600@ieee.org> Nick Fotopoulos wrote: > Dear all, > > I often make use of numpy.vectorize to make programs read more like > the physics equations I write on paper. numpy.vectorize is basically > a wrapper for numpy.frompyfunc. Reading Travis's Scipy Book (mine is > dated Jan 6 2005) kind of suggests to me that it returns a full- > fledged ufunc exactly like built-in ufuncs. > > First, is this true? Yes, it is true. But, it is a ufunc on Python object data-types. It is calling the underlying Python function at every point in the loop. > Second, how is the performance? i.e., are my > functions performing approximately as fast as they could be or would > they still gain a great deal of speed by rewriting it in C or some > other compiled python accelerator? > Absolutely the functions could be made faster to avoid the call back into Python at each evaluation stage. I don't think it would be too hard to replace the function-call with something else that could be evaluated more quickly. But, this has not been done yet. > As an aside, I've found the following function decorator to be > helpful for readability, and perhaps others will enjoy it or improve > upon it: > Thanks for the decorator. This should be put on the www.scipy.org wiki. -Travis From oliphant.travis at ieee.org Sun Jul 16 00:03:33 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 15 Jul 2006 22:03:33 -0600 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: <44B642A9.3090805@psychology.nottingham.ac.uk> References: <44B642A9.3090805@psychology.nottingham.ac.uk> Message-ID: <44B9BA95.9070908@ieee.org> Jon Peirce wrote: > There used to be a function generalized_inverse in the numpy.linalg > module (certainly in 0.9.2). > > In numpy0.9.8 it seems to have been moved to the numpy.linalg.old > subpackage. Does that mean it's being dropped? No. We are just emphasizing the new names. The old names are just there for compatibility with Numeric. The new names have been there from the beginning of NumPy releases. So, just call it numpy.linalg.pinv and it will work in all versions. -Travis From oliphant.travis at ieee.org Sun Jul 16 00:06:07 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 15 Jul 2006 22:06:07 -0600 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: <44B7BEB6.8070608@stsci.edu> References: <44B642A9.3090805@psychology.nottingham.ac.uk> <44B7489A.1050108@gmx.net> <44B7BEB6.8070608@stsci.edu> Message-ID: <44B9BB2F.3030605@ieee.org> Victoria G. Laidler wrote: > Sven Schreiber wrote: > > >> Jon Peirce schrieb: >> >> >> >>> There used to be a function generalized_inverse in the numpy.linalg >>> module (certainly in 0.9.2). >>> >>> In numpy0.9.8 it seems to have been moved to the numpy.linalg.old >>> subpackage. Does that mean it's being dropped? Did it have to move? Now >>> i have to add code to my package to try both locations because my users >>> might have any version... :-( >>> >>> >>> >>> >>> >> Maybe I don't understand, but what's wrong with numpy.linalg.pinv? >> >> >> > Er, what's a pinv? It doesn't sound anything like a generalized_inverse. > > 'pseudo'-inverse. It's the name MATLAB uses for the thing. There are many choices for a "generalized_inverse" which is actually a mis-nomer for what is being done. The Moore-Penrose pseudo-inverse is a particular form of the generalized_inverse (and the one being computed). -Travis From oliphant.travis at ieee.org Sun Jul 16 00:21:26 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 15 Jul 2006 22:21:26 -0600 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: <44B7C37A.7010801@stsci.edu> References: <44B642A9.3090805@psychology.nottingham.ac.uk> <44B7489A.1050108@gmx.net> <44B7BEB6.8070608@stsci.edu> <44B7BF75.2080708@stanford.edu> <44B7C37A.7010801@stsci.edu> Message-ID: <44B9BEC6.4050507@ieee.org> Victoria G. Laidler wrote: > Jonathan Taylor wrote: > > >> pseudoinverse >> >> it's the same name matlab uses: >> >> http://www.mathworks.com/access/helpdesk/help/techdoc/ref/pinv.html >> > > Thanks for the explanation. > > I'm puzzled by the naming choice, however. Standard best practice in > writing software is to give understandable names, to improve readability > and code maintenance. Obscure abbreviations like "pinv" pretty much went > out with the FORTRAN 9-character limit for variable names. It's very > unusual to see them in new software nowadays, and it always looks > unprofessional to me. > I appreciate this feedback. It's a question that comes up occasionally, so I'll at least give my opinion on the matter which may shed some light on it. I disagree with the general "long-name" concept when it comes to "very-common" operations. It's easy to take an idea and over-generalize it for the sake of consistency. I've seen too many codes where very long names actually get in the way of code readability. Someone reading code will have to know what an operation actually is to understand it. A name like "generalized_inverse" doesn't convey any intrinsic meaning to the non-practitioner anyway. You always have to "know" what the function is "really" doing. All that's needed is a "unique" name. I've found that long names are harder to remember (there's more opportunity for confusion about how much of the full name was actually used and how any words were combined). A particularly ludicrous case, for example, was the fact that the very common SVD (whose acronym everybody doing linear algebra uses) was named in LinearAlgebra (an unecessarily long module name to begin with) with the horribly long and unsightly name of singular_value_decomposition. I suppose this was done just for the sake of "code readability." It's not that we're concerned with MATLAB compatibility. But, frankly I've never heard that the short names MATLAB uses for some very common operations are a liability. So, when a common operation has a short, easily-remembered name that is in common usage, why not use it? That's basically the underlying philosophy. NumPy has too many very basic operations to try and create very_long_names for them. I know there are differing opinions out there. I can understand that. That's why I suspect that many codes I will want to use will be written with easy_to_understand_but_very_long names and I'll grin and bear the extra horizontal space that it takes up in my code. -Travis From cookedm at physics.mcmaster.ca Sun Jul 16 04:26:49 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Sun, 16 Jul 2006 04:26:49 -0400 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: <44B9BEC6.4050507@ieee.org> References: <44B642A9.3090805@psychology.nottingham.ac.uk> <44B7489A.1050108@gmx.net> <44B7BEB6.8070608@stsci.edu> <44B7BF75.2080708@stanford.edu> <44B7C37A.7010801@stsci.edu> <44B9BEC6.4050507@ieee.org> Message-ID: <95E43A43-20D0-4F19-A1AA-DDAD52727F92@physics.mcmaster.ca> On Jul 16, 2006, at 00:21 , Travis Oliphant wrote: > Victoria G. Laidler wrote: >> Jonathan Taylor wrote: >> >>> pseudoinverse >>> >>> it's the same name matlab uses: >>> >>> http://www.mathworks.com/access/helpdesk/help/techdoc/ref/pinv.html >>> >> >> Thanks for the explanation. >> >> I'm puzzled by the naming choice, however. Standard best practice in >> writing software is to give understandable names, to improve >> readability >> and code maintenance. Obscure abbreviations like "pinv" pretty >> much went >> out with the FORTRAN 9-character limit for variable names. It's very >> unusual to see them in new software nowadays, and it always looks >> unprofessional to me. >> > I appreciate this feedback. It's a question that comes up > occasionally, > so I'll at least give my opinion on the matter which may shed some > light > on it. > > I disagree with the general "long-name" concept when it comes to > "very-common" operations. It's easy to take an idea and > over-generalize it for the sake of consistency. I've seen too many > codes where very long names actually get in the way of code > readability. How are pseudoinverse and inverse "very common"? (Especially given that one of the arguments for not having a .I attribute for inverse on matrices is that that's usually the wrong way to go about solving equations.) > Someone reading code will have to know what an operation actually > is to > understand it. A name like "generalized_inverse" doesn't convey any > intrinsic meaning to the non-practitioner anyway. You always have to > "know" what the function is "really" doing. All that's needed is a > "unique" name. I've found that long names are harder to remember > (there's more opportunity for confusion about how much of the full > name > was actually used and how any words were combined). As has been argued before, short names have their own problems with remembering what they are. I also find that when reading code with short names, I go slower, because I have to stop and think what that short name is (particularly bad are short names that drop vowels, like lstsq -- I can't pronounce that!). I'm not very good at creating hash tables in my head from short names to long ones. The currently exported names in numpy.linalg are solve, inv, cholesky, eigvals, eigvalsh, eig, eigh, svd, pinv, det, lstsq, and norm. Of these, 'lstsq' is the worst offender, IMHO (superfluous dropped vowels). 'inv' and 'pinv' are the next, then the 'eig*' names. 'least_squares' would be better than 'lstsq'. 'inverse' is not much longer than 'inv', and is more descriptive. I don't think 'pinv' is that common to need a short name; 'pseudoinverse' would be better (not all generalized inverses are pseudoinverses). Give me these three and I'll be happy :-) Personally, I'd prefer 'eigenvalues' and 'eigen' instead of 'eigvals' and 'eig', but I can live with the current names. 'det' is fine, as it's used in mathematical notation. 'cholesky' is also fine, as it's a word at least. I'd have to look at the docstring to find how to use it, but that would be the same for "cholesky_decomposition". [btw, I'm ok with numpy.dft now: the names there make sense, because they're constructed logically. Once you know the scheme, you can see right away that 'irfftn' is 'inverse real FFT, n-dimensional'.] > > A particularly ludicrous case, for example, was the fact that the very > common SVD (whose acronym everybody doing linear algebra uses) was > named > in LinearAlgebra (an unecessarily long module name to begin with) with > the horribly long and unsightly name of singular_value_decomposition. > I suppose this was done just for the sake of "code readability." I agree; that's stupid. > It's not that we're concerned with MATLAB compatibility. But, frankly > I've never heard that the short names MATLAB uses for some very common > operations are a liability. So, when a common operation has a short, > easily-remembered name that is in common usage, why not use it? > > That's basically the underlying philosophy. NumPy has too many very > basic operations to try and create very_long_names for them. > > I know there are differing opinions out there. I can understand > that. > That's why I suspect that many codes I will want to use will be > written > with easy_to_understand_but_very_long names and I'll grin and bear the > extra horizontal space that it takes up in my code. -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From charlesr.harris at gmail.com Sun Jul 16 11:23:36 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 16 Jul 2006 09:23:36 -0600 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: <44B9BEC6.4050507@ieee.org> References: <44B642A9.3090805@psychology.nottingham.ac.uk> <44B7489A.1050108@gmx.net> <44B7BEB6.8070608@stsci.edu> <44B7BF75.2080708@stanford.edu> <44B7C37A.7010801@stsci.edu> <44B9BEC6.4050507@ieee.org> Message-ID: On 7/15/06, Travis Oliphant wrote: > > Victoria G. Laidler wrote: > > Jonathan Taylor wrote: > > It's not that we're concerned with MATLAB compatibility. But, frankly > I've never heard that the short names MATLAB uses for some very common > operations are a liability. So, when a common operation has a short, > easily-remembered name that is in common usage, why not use it? > > That's basically the underlying philosophy. NumPy has too many very > basic operations to try and create very_long_names for them. > > I know there are differing opinions out there. I can understand that. > That's why I suspect that many codes I will want to use will be written > with easy_to_understand_but_very_long names and I'll grin and bear the > extra horizontal space that it takes up in my code. What is needed in the end is a good index with lots of crossreferences. Name choices are just choices, there is no iso standard for function names that I know of. There are short names have been used for so long that everyone knows them (sin, cos, ...), some names come in two standard forms (arcsin, asin) some are fortran conventions (arctan2), some are matlab conventions (pinv, chol). One always has to learn what the names for things are in any new language, so the best thing is to make it easy to find out. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Sun Jul 16 11:47:08 2006 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 16 Jul 2006 11:47:08 -0400 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: <95E43A43-20D0-4F19-A1AA-DDAD52727F92@physics.mcmaster.ca> References: <44B642A9.3090805@psychology.nottingham.ac.uk> <44B7489A.1050108@gmx.net> <44B7BEB6.8070608@stsci.edu><44B7BF75.2080708@stanford.edu> <44B7C37A.7010801@stsci.edu><44B9BEC6.4050507@ieee.org><95E43A43-20D0-4F19-A1AA-DDAD52727F92@physics.mcmaster.ca> Message-ID: On Sun, 16 Jul 2006, "David M. Cooke" apparently wrote: > 'inverse' is not much longer than 'inv', and is more descriptive But 'inv' is quite universal (can you name a matrix language that uses 'inverse' instead?) and I think unambiguous (what might it be confused with?). Cheers, Alan Isaac From aisaac at american.edu Sun Jul 16 11:47:07 2006 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 16 Jul 2006 11:47:07 -0400 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: References: <44B642A9.3090805@psychology.nottingham.ac.uk><44B7489A.1050108@gmx.net> <44B7BEB6.8070608@stsci.edu><44B7BF75.2080708@stanford.edu> <44B7C37A.7010801@stsci.edu><44B9BEC6.4050507@ieee.org> Message-ID: On Sun, 16 Jul 2006, Charles R Harris apparently wrote: > What is needed in the end is a good index with lots of > crossreferences. Name choices are just choices I mostly agree with this (although I think Matlab made some bad choices in naming). As a point of reference for a useful index see http://www.mathworks.com/access/helpdesk/help/techdoc/ref/refbookl.html Cheers, Alan Isaac From josh.p.marshall at gmail.com Sun Jul 16 20:22:07 2006 From: josh.p.marshall at gmail.com (Josh Marshall) Date: Mon, 17 Jul 2006 10:22:07 +1000 Subject: [Numpy-discussion] Packaging numpy with py2app Message-ID: <660C6421-8C71-4B4D-A2CC-33C3FF089DF9@gmail.com> Back in December last year, I was building a PyObjC application that embedded numpy (scipy_core at the time), scipy and matplotlib. I ran into a few issues doing so, some of which were resolved. One was the inability for scipy to run from a zipped site-packages. I worked around this by expanding the embedded site-packages.zip into a site- packages directory in the same location. For reference, the thread can be found at: http://www.scipy.net/pipermail/scipy-dev/2005-December/004551.html Come a few months later, I have needed to update to the latest version of numpy (and therefore scipy and matplotlib). I have not yet updated to the universal build of Python, still running 2.4.1, although I will do so if it is known to fix any issues. (I don't have too much time at the moment, and building the latest versions of numpy and matplotlib for a universal build scares me). I managed to get it working again, which required: 1) Setting packages=['matplotlib','numpy'] in setup.py's options for py2app. 2) Modifying the py2app/apptemplate/lib/site.py file to include 'sys.path.append(_parent + '/site-packages')' before the same line with .zip appended to the file name. 3) Adding setup_options['options']['py2app']['includes'].extend (['pytz.zoneinfo.UTC']) to the setup.py, this is required by matplotlib. I believe (2) is a bug in py2app (I am running 0.3.1). Packages included using 'packages=' are not added to site-packages.zip, but rather are in their own site-packages directory. I am not sure whether this is the intended behaviour or a bug, but it is good for me, since numpy and matplotlib won't run when compressed. Now, it seems I am doomed to continue to have to find work-arounds to get numpy and matplotlib working in a standalone .app. Is there a chance we can come up with a py2app recipe for numpy, matplotlib and scipy? What other alternatives are there? Regards, Josh From bob at redivi.com Sun Jul 16 20:46:06 2006 From: bob at redivi.com (Bob Ippolito) Date: Sun, 16 Jul 2006 17:46:06 -0700 Subject: [Numpy-discussion] [Pythonmac-SIG] Packaging numpy with py2app In-Reply-To: <660C6421-8C71-4B4D-A2CC-33C3FF089DF9@gmail.com> References: <660C6421-8C71-4B4D-A2CC-33C3FF089DF9@gmail.com> Message-ID: On Jul 16, 2006, at 5:22 PM, Josh Marshall wrote: > Back in December last year, I was building a PyObjC application that > embedded numpy (scipy_core at the time), scipy and matplotlib. I ran > into a few issues doing so, some of which were resolved. One was the > inability for scipy to run from a zipped site-packages. I worked > around this by expanding the embedded site-packages.zip into a site- > packages directory in the same location. For reference, the thread > can be found at: > http://www.scipy.net/pipermail/scipy-dev/2005-December/004551.html > > Come a few months later, I have needed to update to the latest > version of numpy (and therefore scipy and matplotlib). I have not yet > updated to the universal build of Python, still running 2.4.1, > although I will do so if it is known to fix any issues. (I don't have > too much time at the moment, and building the latest versions of > numpy and matplotlib for a universal build scares me). If you're still using 2.4.1 you're going to want to add LSPrefersPPC=True to the plist. Otherwise your app will not run on i386, because it will not know to use Rosetta. > I managed to get it working again, which required: > 1) Setting packages=['matplotlib','numpy'] in setup.py's options for > py2app. That could be handled with recipes, or with eggs when py2app grows support for that. Recipes or workarounds are the only way until then. > 2) Modifying the py2app/apptemplate/lib/site.py file to include > 'sys.path.append(_parent + '/site-packages')' before the same line > with .zip appended to the file name. That's a bug, but not the right fix. Trunk is fixed. > 3) Adding setup_options['options']['py2app']['includes'].extend > (['pytz.zoneinfo.UTC']) to the setup.py, this is required by > matplotlib. That should be part of a matplotlib recipe. > Now, it seems I am doomed to continue to have to find work-arounds to > get numpy and matplotlib working in a standalone .app. Is there a > chance we can come up with a py2app recipe for numpy, matplotlib and > scipy? What other alternatives are there? Sure, someone can write the recipes and send me a patch. I don't currently use matplotlib, numpy, or scipy nor do I have an example I can test with so I'm not going to do it. -bob From rlw at stsci.edu Sun Jul 16 21:25:24 2006 From: rlw at stsci.edu (Rick White) Date: Sun, 16 Jul 2006 21:25:24 -0400 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: References: <44B642A9.3090805@psychology.nottingham.ac.uk> <44B7489A.1050108@gmx.net> <44B7BEB6.8070608@stsci.edu><44B7BF75.2080708@stanford.edu> <44B7C37A.7010801@stsci.edu><44B9BEC6.4050507@ieee.org><95E43A43-20D0-4F19-A1AA-DDAD52727F92@physics.mcmaster.ca> Message-ID: <3A6EC819-DED9-4C23-80E5-3BBDE0859EBF@stsci.edu> On Jul 16, 2006, at 11:47 AM, Alan G Isaac wrote: > On Sun, 16 Jul 2006, "David M. Cooke" apparently wrote: >> 'inverse' is not much longer than 'inv', and is more descriptive > > But 'inv' is quite universal (can you name a matrix language > that uses 'inverse' instead?) and I think unambiguous (what > might it be confused with?). IDL uses invert, so inv is not exactly universal. I'm personally a fan of names that can be used in interactive sessions at the command line, which argues for shorter names. But it is nice to have names where you can type just a few characters and use tab-completion to fill in the rest of the name. Then the important thing is not the full length of the name but having the first 3 or 4 characters be memorable. So I'd rather have "pseudoinverse" because I can probably find it by just typing "ps". From kwgoodman at gmail.com Sun Jul 16 21:39:19 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Sun, 16 Jul 2006 18:39:19 -0700 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: <3A6EC819-DED9-4C23-80E5-3BBDE0859EBF@stsci.edu> References: <44B642A9.3090805@psychology.nottingham.ac.uk> <44B7489A.1050108@gmx.net> <44B7BEB6.8070608@stsci.edu> <44B7BF75.2080708@stanford.edu> <44B7C37A.7010801@stsci.edu> <44B9BEC6.4050507@ieee.org> <95E43A43-20D0-4F19-A1AA-DDAD52727F92@physics.mcmaster.ca> <3A6EC819-DED9-4C23-80E5-3BBDE0859EBF@stsci.edu> Message-ID: On 7/16/06, Rick White wrote: > On Jul 16, 2006, at 11:47 AM, Alan G Isaac wrote: > > > On Sun, 16 Jul 2006, "David M. Cooke" apparently wrote: > >> 'inverse' is not much longer than 'inv', and is more descriptive > > > > But 'inv' is quite universal (can you name a matrix language > > that uses 'inverse' instead?) and I think unambiguous (what > > might it be confused with?). > > IDL uses invert, so inv is not exactly universal. > > I'm personally a fan of names that can be used in interactive > sessions at the command line, which argues for shorter names. But it > is nice to have names where you can type just a few characters and > use tab-completion to fill in the rest of the name. Then the > important thing is not the full length of the name but having the > first 3 or 4 characters be memorable. So I'd rather have > "pseudoinverse" because I can probably find it by just typing "ps". I prfr shrtr nams lke inv eig and sin. From david.huard at gmail.com Mon Jul 17 10:10:55 2006 From: david.huard at gmail.com (David Huard) Date: Mon, 17 Jul 2006 10:10:55 -0400 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <91cf711d0607111127w21bb4b3cj30a3c31b46e136f0@mail.gmail.com> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> <44A8E4D3.5000603@ntc.zcu.cz> <91cf711d0607031000l286da52fn30d98798057f12c5@mail.gmail.com> <44B3BD22.70003@gmx.de> <44B3D6A5.6060404@ntc.zcu.cz> <91cf711d0607111127w21bb4b3cj30a3c31b46e136f0@mail.gmail.com> Message-ID: <91cf711d0607170710q63ef1114rc55240741f2dd1f1@mail.gmail.com> Hi, I attached a patch for unique (with a test case) based on Norbert's suggestion. I removed the sort keyword since iterable arrays would be sorted anyway. The function uses a python set, and I was wondering if it is ok to assume that everyone running numpy has a python version >= 2.3 ? David -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: unique.patch Type: text/x-patch Size: 2146 bytes Desc: not available URL: From fullung at gmail.com Mon Jul 17 10:59:36 2006 From: fullung at gmail.com (Albert Strasheim) Date: Mon, 17 Jul 2006 16:59:36 +0200 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <91cf711d0607170710q63ef1114rc55240741f2dd1f1@mail.gmail.com> Message-ID: <027701c6a9b1$9e9b6f10$0100000a@dsp.sun.ac.za> Hey David > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of David Huard > Sent: 17 July 2006 16:11 > To: numpy-discussion at lists.sourceforge.net > Subject: Re: [Numpy-discussion] unique() should return a sorted array > > Hi, > > I attached a patch for unique (with a test case) based on Norbert's > suggestion. I removed the sort keyword since iterable arrays would be > sorted anyway. The function uses a python set, and I was wondering if it > is ok to assume that everyone running numpy has a python version >= 2.3 ? I think it has been discussed on the list that Python >= 2.3 is assumed. However, according to the Python documentation, the built-in set type is new in Python 2.4. Regards, Albert From jg307 at cam.ac.uk Mon Jul 17 11:16:42 2006 From: jg307 at cam.ac.uk (James Graham) Date: Mon, 17 Jul 2006 16:16:42 +0100 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <027701c6a9b1$9e9b6f10$0100000a@dsp.sun.ac.za> References: <027701c6a9b1$9e9b6f10$0100000a@dsp.sun.ac.za> Message-ID: <44BBA9DA.9050008@cam.ac.uk> Albert Strasheim wrote: > I think it has been discussed on the list that Python >= 2.3 is assumed. > However, according to the Python documentation, the built-in set type is new > in Python 2.4. Sets are available in python 2.3 as part of the sets module so it is possible. However afaict the provided patch does not use the module and so will need to be adapted for use in 2.3. -- "You see stars that clear have been dead for years But the idea just lives on..." -- Bright Eyes From wright at esrf.fr Mon Jul 17 11:19:48 2006 From: wright at esrf.fr (Jon Wright) Date: Mon, 17 Jul 2006 17:19:48 +0200 Subject: [Numpy-discussion] generalized_inverse and documentation in general In-Reply-To: References: <44B642A9.3090805@psychology.nottingham.ac.uk> <44B7489A.1050108@gmx.net> <44B7BEB6.8070608@stsci.edu> <44B7BF75.2080708@stanford.edu> <44B7C37A.7010801@stsci.edu> <44B9BEC6.4050507@ieee.org> <95E43A43-20D0-4F19-A1AA-DDAD52727F92@physics.mcmaster.ca> <3A6EC819-DED9-4C23-80E5-3BBDE0859EBF@stsci.edu> Message-ID: <44BBAA94.8010009@esrf.fr> Keith Goodman wrote: >I prfr shrtr nams lke inv eig and sin. > > Dwn wth vwls!! Srsly thgh: >>> from numpy import linalg >>> help(linalg.inv) Help on function inv in module numpy.linalg.linalg: inv(a) >>> ??? While I prefer inverse to inv, I don't really care as long as the word "inverse" appears in the docstring and it faithfully promises that it is going to try to do an inverse in the matrix linear algebra sense and not in the one_over_x or power_minus_one sense or bit inversion (~ x, numpy.invert) sense. It would be handy to know which exception which will be raised for a singular matrix, what are valid arguments (eg, square shape, numbers), what is the type of the output (' testnumpywiki.txt ==testwiki.py:== import doctest doctest.testfile("testnumpywiki.txt") $ python testwiki.py > problems.txt problems.txt is 37kB in size (83 of 1028) throwing out the blanklines issues via: doctest.testfile("testnumpywiki.txt", optionflags=doctest.NORMALIZE_WHITESPACE) reduces this to 24kB (62 of 1028). ... most cases are not important, just needing to be fixed for formatting on the wiki or flagged as version dependent, but a few are worth checking out the intentions, eg: ********************************************************************** File "testnumpywiki.txt", line 69, in testnumpywiki.txt Failed example: a[:,b2] Exception raised: Traceback (most recent call last): File "c:\python24\lib\doctest.py", line 1243, in __run compileflags, 1) in test.globs File "", line 1, in ? a[:,b2] IndexError: arrays used as indices must be of integer type ********************************************************************** File "testnumpywiki.txt", line 893, in testnumpywiki.txt Failed example: ceil(a) # nearest integers greater-than or equal to a Expected: array([-1., -1., -0., 1., 2., 2.]) Got: array([-1., -1., 0., 1., 2., 2.]) ********************************************************************** File "testnumpywiki.txt", line 1162, in testnumpywiki.txt Failed example: cov(T,P) # covariance between temperature and pressure Expected: 3.9541666666666657 Got: array([[ 1.97583333, 3.95416667], [ 3.95416667, 8.22916667]]) ********************************************************************** File "testnumpywiki.txt", line 2235, in testnumpywiki.txt Failed example: type(a[0]) Expected: Got: From skip at pobox.com Mon Jul 17 11:32:34 2006 From: skip at pobox.com (skip at pobox.com) Date: Mon, 17 Jul 2006 10:32:34 -0500 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <44BBA9DA.9050008@cam.ac.uk> References: <027701c6a9b1$9e9b6f10$0100000a@dsp.sun.ac.za> <44BBA9DA.9050008@cam.ac.uk> Message-ID: <17595.44434.740132.140969@montanaro.dyndns.org> James> Sets are available in python 2.3 as part of the sets module so it James> is possible. However afaict the provided patch does not use the James> module and so will need to be adapted for use in 2.3. I got so tired of the ugly test for the set builtin during our 2.3-to-2.4 transition that I finally just added if sys.hexversion < 0x2040000: from sets import Set as set import __builtin__ __builtin__.set = set to sitecustomize.py. I'm not suggesting that scipy's installer should do the same, but the change worked for us. Skip From haase at msg.ucsf.edu Mon Jul 17 12:40:48 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 17 Jul 2006 09:40:48 -0700 Subject: [Numpy-discussion] numarray bug: dot product between 2x2 and 3x2x3 on Mac different from PC Message-ID: <200607170940.48872.haase@msg.ucsf.edu> Hi! This is what I got: >>> import numarray as na >>> na.__version__ '1.4.0' >>>bbb=na.zeros((2,3,3), naFloat32) >>>bbb[0,:,:]=1 >>>bbb[1,:,:]=2 >>>ccc=na.transpose(bbb,(1,0,2)) >>>d=na.array([[1,0],[0,1]]) >>>na.dot(d,ccc) [[[ 1. ?1. ?1.] ? [ 1. ?1. ?1.] ? [ 1. ?1. ?1.]] ?[[ 2. ?2. ?2.] ? [ 2. ?2. ?2.] ? [ 2. ?2. ?2.]]] This is on a PC (Windows or Linux): But if you do the same thing on Mac, the result will be: [[[ 1. 1. 1.] ? [ 2. 2. 2.] ? [ 1. 1. 1.]] ?[[ 2. 2. 2.] ? [ 1. 1. 1.] ? [ 2. 2. 2.]]] Can someone confirm this ? (the new numpy on windows also gives the first result) Thanks, Sebastian Haase From Chris.Barker at noaa.gov Mon Jul 17 14:24:34 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 17 Jul 2006 11:24:34 -0700 Subject: [Numpy-discussion] numarray bug: dot product between 2x2 and 3x2x3 on Mac different from PC In-Reply-To: <200607170940.48872.haase@msg.ucsf.edu> References: <200607170940.48872.haase@msg.ucsf.edu> Message-ID: <44BBD5E2.5080703@noaa.gov> Sebastian Haase wrote: > Hi! > This is what I got: > Can someone confirm this ? This is what I get on Linux (ix86, Fedora core4, python 2.4.3) >>> import numarray as na >>> na.__version__ '1.5.0' >>> bbb=na.zeros((2,3,3), na.Float32) >>> bbb[0,:,:]=1 >>> bbb[1,:,:]=2 >>> bbb array([[[ 1., 1., 1.], [ 1., 1., 1.], [ 1., 1., 1.]], [[ 2., 2., 2.], [ 2., 2., 2.], [ 2., 2., 2.]]], type=Float32) >>> ccc=na.transpose(bbb,(1,0,2)) >>> ccc array([[[ 1., 1., 1.], [ 2., 2., 2.]], [[ 1., 1., 1.], [ 2., 2., 2.]], [[ 1., 1., 1.], [ 2., 2., 2.]]], type=Float32) >>> d=na.array([[1,0],[0,1]]) >>> d array([[1, 0], [0, 1]]) >>> na.dot(d,ccc) array([[[ 1., 1., 1.], [ 2., 2., 2.], [ 1., 1., 1.]], [[ 2., 2., 2.], [ 1., 1., 1.], [ 2., 2., 2.]]], type=Float32) >>> -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Mon Jul 17 14:29:46 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 17 Jul 2006 11:29:46 -0700 Subject: [Numpy-discussion] numarray bug: dot product between 2x2 and 3x2x3 on Mac different from PC In-Reply-To: <200607170940.48872.haase@msg.ucsf.edu> References: <200607170940.48872.haase@msg.ucsf.edu> Message-ID: <44BBD71A.1050400@noaa.gov> And on my Mac: OS-X 10.4.*, PPC, universal python 2.4.3: >>> import numarray as na >>> na.__version__ '1.5.1' >>> bbb=na.zeros((2,3,3), na.Float32) >>> bbb[0,:,:]=1 >>> bbb[1,:,:]=2 >>> ccc=na.transpose(bbb,(1,0,2)) >>> d=na.array([[1,0],[0,1]]) >>> na.dot(d,ccc) array([[[ 1., 1., 1.], [ 2., 2., 2.], [ 1., 1., 1.]], [[ 2., 2., 2.], [ 1., 1., 1.], [ 2., 2., 2.]]], type=Float32) -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From fperez.net at gmail.com Mon Jul 17 14:31:50 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 17 Jul 2006 12:31:50 -0600 Subject: [Numpy-discussion] 64-bit build woes Message-ID: Hi all, I received this message today from a collaborator. I don't have direct access to this box, but he posted fairly detailed logs. Has anyone seen a similar issue with current code? If not, I'll try to track down further with him what the problem may actually be. Thanks for any help, f ============ Original message: We are having trouble with scipy (not)compiling on the 64-bit machine, and it seems to be related to the intp type. I put the log file and connected files at http://www.math.ohiou.edu/~mjm/pickup/scipy.log http://www.math.ohiou.edu/~mjm/pickup/fortranobject.c http://www.math.ohiou.edu/~mjm/pickup/fortranobject.h The relevant parts are at the end of scipy.log: creating build/temp.linux-x86_64-2.3/Lib/fftpack/src compile options: '-Ibuild/src.linux-x86_64-2.3 -I/usr/lib64/python2.3/site-packages/numpy/core/include -I/usr/include/python2.3 -c' gcc: Lib/fftpack/src/drfft.c gcc: Lib/fftpack/src/zfft.c gcc: build/src.linux-x86_64-2.3/fortranobject.c build/src.linux-x86_64-2.3/fortranobject.c: In function `PyFortranObject_New': build/src.linux-x86_64-2.3/fortranobject.c:55: error: syntax error before "intp" build/src.linux-x86_64-2.3/fortranobject.c:60: error: syntax error before "intp" build/src.linux-x86_64-2.3/fortranobject.c: In function `fortran_getattr': build/src.linux-x86_64-2.3/fortranobject.c:179: error: syntax error before "intp" build/src.linux-x86_64-2.3/fortranobject.c: In function `fortran_setattr': build/src.linux-x86_64-2.3/fortranobject.c:243: error: syntax error before ')' token build/src.linux-x86_64-2.3/fortranobject.c:245: error: syntax error before ')' token ... I don't see "intp" in fortranobject.c, so it must be included from elsewhere. From haase at msg.ucsf.edu Mon Jul 17 14:40:43 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 17 Jul 2006 11:40:43 -0700 Subject: [Numpy-discussion] numarray bug: dot product between 2x2 and 3x2x3 on Mac different from PC In-Reply-To: <44BBD71A.1050400@noaa.gov> References: <200607170940.48872.haase@msg.ucsf.edu> <44BBD71A.1050400@noaa.gov> Message-ID: <200607171140.43249.haase@msg.ucsf.edu> Thanks Chris, If this got fixed towards numarray version 1.5.1 it looks like it is now on both Linux and Mac different(!) from the numpy result I got on Window !! On Linux I get: >>> import numpy as N >>> N.__version__ '0.9.9.2823' >>> bbb=N.zeros((2,3,3), N.float32) >>> bbb[0,:,:]=1 >>> bbb[1,:,:]=2 >>> bbb [[[ 1. 1. 1.] [ 1. 1. 1.] [ 1. 1. 1.]] [[ 2. 2. 2.] [ 2. 2. 2.] [ 2. 2. 2.]]] >>> ccc=N.transpose(bbb,(1,0,2)) >>> d=N.array([[1,0],[0,1]]) >>> d [[1 0] [0 1]] >>> N.dot(d,ccc) Traceback (most recent call last): File "", line 1, in ? TypeError: array cannot be safely cast to required type >>> dd=d.astype(N.float32) >>> N.dot(dd,ccc) [[[ 1. 1. 1.] [ 1. 1. 1.] [ 1. 1. 1.]] [[ 2. 2. 2.] [ 2. 2. 2.] [ 2. 2. 2.]]] >>> The TypeError looks like a numpy bug ! Thanks, Sebastian Haase On Monday 17 July 2006 11:29, Christopher Barker wrote: > And on my Mac: > > OS-X 10.4.*, PPC, universal python 2.4.3: > >>> import numarray as na > >>> na.__version__ > > '1.5.1' > > >>> bbb=na.zeros((2,3,3), na.Float32) > >>> bbb[0,:,:]=1 > >>> bbb[1,:,:]=2 > >>> ccc=na.transpose(bbb,(1,0,2)) > >>> d=na.array([[1,0],[0,1]]) > >>> na.dot(d,ccc) > > array([[[ 1., 1., 1.], > [ 2., 2., 2.], > [ 1., 1., 1.]], > > [[ 2., 2., 2.], > [ 1., 1., 1.], > [ 2., 2., 2.]]], type=Float32) From oliphant.travis at ieee.org Mon Jul 17 15:10:42 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 17 Jul 2006 13:10:42 -0600 Subject: [Numpy-discussion] numarray bug: dot product between 2x2 and 3x2x3 on Mac different from PC In-Reply-To: <200607171140.43249.haase@msg.ucsf.edu> References: <200607170940.48872.haase@msg.ucsf.edu> <44BBD71A.1050400@noaa.gov> <200607171140.43249.haase@msg.ucsf.edu> Message-ID: <44BBE0B2.4090002@ieee.org> Sebastian Haase wrote: > Traceback (most recent call last): > File "", line 1, in ? > TypeError: array cannot be safely cast to required type > >>>> dd=d.astype(N.float32) >>>> N.dot(dd,ccc) >>>> > [[[ 1. 1. 1.] > [ 1. 1. 1.] > [ 1. 1. 1.]] > > [[ 2. 2. 2.] > [ 2. 2. 2.] > [ 2. 2. 2.]]] > > > > The TypeError looks like a numpy bug ! > I don't see why this is a bug. You are trying to coerce a 32-bit integer to a 32-bit float. That is going to lose precision and so you get the error indicated. -Travis From oliphant.travis at ieee.org Mon Jul 17 15:20:12 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 17 Jul 2006 13:20:12 -0600 Subject: [Numpy-discussion] 64-bit build woes In-Reply-To: References: Message-ID: <44BBE2EC.7000308@ieee.org> Fernando Perez wrote: > Hi all, > > I received this message today from a collaborator. I don't have > direct access to this box, but he posted fairly detailed logs. Has > anyone seen a similar issue with current code? If not, I'll try to > track down further with him what the problem may actually be. > This looks like a problem with left-over headers and/or C-API files being picked up. Make sure the old header files are deleted and he has a fresh install of NumPy from SVN (with the build directory deleted before re-building). Look in /usr/lib64/python2.3/site-packages/numpy/core/include/__multiarray_api.h to make sure there are now isolated intp references (particularly look at PyArray_New). If there are, then the NumPy build was not clean. -Travis From fperez.net at gmail.com Mon Jul 17 15:23:50 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 17 Jul 2006 13:23:50 -0600 Subject: [Numpy-discussion] 64-bit build woes In-Reply-To: <44BBE2EC.7000308@ieee.org> References: <44BBE2EC.7000308@ieee.org> Message-ID: On 7/17/06, Travis Oliphant wrote: > Fernando Perez wrote: > > Hi all, > > > > I received this message today from a collaborator. I don't have > > direct access to this box, but he posted fairly detailed logs. Has > > anyone seen a similar issue with current code? If not, I'll try to > > track down further with him what the problem may actually be. > > > > This looks like a problem with left-over headers and/or C-API files > being picked up. Make sure the old header files are deleted and he has > a fresh install of NumPy from SVN (with the build directory deleted > before re-building). > > Look in > > /usr/lib64/python2.3/site-packages/numpy/core/include/__multiarray_api.h > > to make sure there are now isolated intp references (particularly look at PyArray_New). > If there are, then the NumPy build was not clean. Thanks, Travis. I just wanted to make sure it wasn't a more widespread problem. I'll track it down with my colleague in private then. Cheers, f From haase at msg.ucsf.edu Mon Jul 17 15:26:33 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 17 Jul 2006 12:26:33 -0700 Subject: [Numpy-discussion] numarray bug: dot product between 2x2 and 3x2x3 on Mac different from PC In-Reply-To: <44BBE0B2.4090002@ieee.org> References: <200607170940.48872.haase@msg.ucsf.edu> <200607171140.43249.haase@msg.ucsf.edu> <44BBE0B2.4090002@ieee.org> Message-ID: <200607171226.33383.haase@msg.ucsf.edu> On Monday 17 July 2006 12:10, Travis Oliphant wrote: > Sebastian Haase wrote: > > Traceback (most recent call last): > > File "", line 1, in ? > > TypeError: array cannot be safely cast to required type > > > >>>> dd=d.astype(N.float32) > >>>> N.dot(dd,ccc) > > > > [[[ 1. 1. 1.] > > [ 1. 1. 1.] > > [ 1. 1. 1.]] > > > > [[ 2. 2. 2.] > > [ 2. 2. 2.] > > [ 2. 2. 2.]]] > > > > > > > > The TypeError looks like a numpy bug ! > > I don't see why this is a bug. You are trying to coerce a 32-bit > integer to a 32-bit float. That is going to lose precision and so you > get the error indicated. > > -Travis In numarray I do not get an error. Would the error go away if I had 64 bit float !? It seems though that having ones and twos in an int array should fit just fine into a float32 array !? - Sebastian Haase From oliphant.travis at ieee.org Mon Jul 17 15:38:29 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 17 Jul 2006 13:38:29 -0600 Subject: [Numpy-discussion] numarray bug: dot product between 2x2 and 3x2x3 on Mac different from PC In-Reply-To: <200607171226.33383.haase@msg.ucsf.edu> References: <200607170940.48872.haase@msg.ucsf.edu> <200607171140.43249.haase@msg.ucsf.edu> <44BBE0B2.4090002@ieee.org> <200607171226.33383.haase@msg.ucsf.edu> Message-ID: <44BBE735.5010109@ieee.org> Sebastian Haase wrote: > On Monday 17 July 2006 12:10, Travis Oliphant wrote: > >> Sebastian Haase wrote: >> >>> Traceback (most recent call last): >>> File "", line 1, in ? >>> TypeError: array cannot be safely cast to required type >>> >>> >>>>>> dd=d.astype(N.float32) >>>>>> N.dot(dd,ccc) >>>>>> >>> [[[ 1. 1. 1.] >>> [ 1. 1. 1.] >>> [ 1. 1. 1.]] >>> >>> [[ 2. 2. 2.] >>> [ 2. 2. 2.] >>> [ 2. 2. 2.]]] >>> >>> >>> >>> The TypeError looks like a numpy bug ! >>> >> I don't see why this is a bug. You are trying to coerce a 32-bit >> integer to a 32-bit float. That is going to lose precision and so you >> get the error indicated. >> >> -Travis >> > In numarray I do not get an error. Would the error go away if I had 64 bit > float !? It seems though that having ones and twos in an int array should > fit just fine into a float32 array !? > This could be considered a bug in numarray. It's force-casting the result. That isn't the normal behavior of mixed-type functions. Also, the policy on type-casting is not to search the array to see if it's possible to do the conversion on every element (that would be slow on large arrays). The policy is to base the decision only on the data-types themselves (i.e. whether it's *possible* to lose precision*). -Travis *There is one exception to this policy in NumPy: 64-bit integers are allowed to be cast to 64-bit doubles --- other-wise on you would get a lot of non-standard long-doubles showing up on 64-bit systems. This policy was decided after discussion last year. From haase at msg.ucsf.edu Mon Jul 17 16:48:05 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 17 Jul 2006 13:48:05 -0700 Subject: [Numpy-discussion] numarray bug: dot product between 2x2 and 3x2x3 on Mac different from PC In-Reply-To: <44BBE735.5010109@ieee.org> References: <200607170940.48872.haase@msg.ucsf.edu> <200607171226.33383.haase@msg.ucsf.edu> <44BBE735.5010109@ieee.org> Message-ID: <200607171348.05776.haase@msg.ucsf.edu> On Monday 17 July 2006 12:38, Travis Oliphant wrote: > Sebastian Haase wrote: > > On Monday 17 July 2006 12:10, Travis Oliphant wrote: > >> Sebastian Haase wrote: > >>> Traceback (most recent call last): > >>> File "", line 1, in ? > >>> TypeError: array cannot be safely cast to required type > >>> > >>>>>> dd=d.astype(N.float32) > >>>>>> N.dot(dd,ccc) > >>> > >>> [[[ 1. 1. 1.] > >>> [ 1. 1. 1.] > >>> [ 1. 1. 1.]] > >>> > >>> [[ 2. 2. 2.] > >>> [ 2. 2. 2.] > >>> [ 2. 2. 2.]]] > >>> > >>> > >>> > >>> The TypeError looks like a numpy bug ! > >> > >> I don't see why this is a bug. You are trying to coerce a 32-bit > >> integer to a 32-bit float. That is going to lose precision and so you > >> get the error indicated. > >> > >> -Travis > > > > In numarray I do not get an error. Would the error go away if I had 64 > > bit float !? It seems though that having ones and twos in an int array > > should fit just fine into a float32 array !? > > This could be considered a bug in numarray. It's force-casting the > result. That isn't the normal behavior of mixed-type functions. > > Also, the policy on type-casting is not to search the array to see if > it's possible to do the conversion on every element (that would be slow > on large arrays). The policy is to base the decision only on the > data-types themselves (i.e. whether it's *possible* to lose precision*). > > -Travis > > > > *There is one exception to this policy in NumPy: 64-bit integers are > allowed to be cast to 64-bit doubles --- other-wise on you would get a > lot of non-standard long-doubles showing up on 64-bit systems. This > policy was decided after discussion last year. OK - understood. Combining int32 with float64 proves to be less cumbersome ... Any idea on my main question ? What is the dot product of a 2x2 and 3x2x3 supposed to look like ? Why are numarray and numpy giving different answers ?? Thanks, Sebastian Haase From kwgoodman at gmail.com Mon Jul 17 18:50:57 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 17 Jul 2006 15:50:57 -0700 Subject: [Numpy-discussion] Displaying all the rows of large matrix Message-ID: How do you display all of the rows of a matrix? >> x = zeros((334,3)) >> x matrix([[ 0., 0., 0.], [ 0., 0., 0.], [ 0., 0., 0.], ..., [ 0., 0., 0.], [ 0., 0., 0.], [ 0., 0., 0.]]) From oliphant.travis at ieee.org Mon Jul 17 19:00:42 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 17 Jul 2006 17:00:42 -0600 Subject: [Numpy-discussion] Displaying all the rows of large matrix In-Reply-To: References: Message-ID: <44BC169A.3070807@ieee.org> Keith Goodman wrote: > How do you display all of the rows of a matrix? > help(numpy.set_prinoptions) Look at the threshold keyword numpy.set_printoptions(threshold = 2000) for your example -Travis From kwgoodman at gmail.com Mon Jul 17 19:37:12 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 17 Jul 2006 16:37:12 -0700 Subject: [Numpy-discussion] Displaying all the rows of large matrix In-Reply-To: <44BC169A.3070807@ieee.org> References: <44BC169A.3070807@ieee.org> Message-ID: On 7/17/06, Travis Oliphant wrote: > Keith Goodman wrote: > > How do you display all of the rows of a matrix? > > > > help(numpy.set_prinoptions) That is great! Now I can change the precision as well. Eight significant figures is too precise for me. Does anyone out there save the print defaults across sessions? How do you do it? Does numpy look for any startup files (~/.numpyrc)? From wbaxter at gmail.com Mon Jul 17 19:59:14 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 18 Jul 2006 08:59:14 +0900 Subject: [Numpy-discussion] Startup .numpyrc file Message-ID: On 7/18/06, Keith Goodman wrote: > On 7/17/06, Travis Oliphant wrote: > > Keith Goodman wrote: > Does anyone out there save the print defaults across sessions? How do you do it? > > Does numpy look for any startup files (~/.numpyrc)? If you set a PYTHONSTARTUP environment variable then Python will run the script it points to at startup of interactive sessions. But I wonder if maybe some numpy-specific startup script would be good. My current PYTHONSTARTUP is importing numpy, just so I can define some numpy shortcuts for my interactive sessions. That's fine if you always use Numpy in your interactive sessions, but if not, then it's just dead weight. --bb From oliphant.travis at ieee.org Mon Jul 17 20:10:36 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 17 Jul 2006 18:10:36 -0600 Subject: [Numpy-discussion] [SciPy-user] Arithmetic Errors In-Reply-To: <20060717235553.GA14362@redwoodscientific.com> References: <20060717205116.GA11043@redwoodscientific.com> <44BC0161.8070007@ieee.org> <20060717235553.GA14362@redwoodscientific.com> Message-ID: <44BC26FC.8050408@ieee.org> John Lawless wrote: > Travis, > > Thanks! > > 1). I haven't found any documentation on dtype='O'. (I purchased your > trelgol book but it hasn't arrived yet.) Does 'O' guarantee no > wrong answers? > The object data-type uses Python objects instead of low-level C-types for the calculations. So, it gives the same calculations that Python would do (but of course it's much slower). > 2). My actual code was more complex than the example I posted. It was > giving correct answers until I increased the dataset size. Then, > luckily, the result became obviously wrong. I can go through a > code and try to coerce everything to double but, when debugging a > large code, how can one ever be sure that all types are coerced > correctly if no errors are generated? > NumPy uses c data-types for calculations. It is therefore, *much* faster, but you have to take precautions about overflowing on integer operations. > 3). AFAIK, checking for overflows should take no CPU time whatsoever > unless an exception is actually generated. This is true for floating point operations, but you were doing integer multiplication. There is no support for hardware multiply overflow in NumPy (is there even such a thing?). Python checks for overflow on integer arithmetic by doing some additional calculations. It would be possible to add slower, integer over-flow checking ufuncs to NumPy if this was desired and you could replace the standard non-checking functions pretty easily. -Travis From oliphant.travis at ieee.org Mon Jul 17 20:11:31 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 17 Jul 2006 18:11:31 -0600 Subject: [Numpy-discussion] Startup .numpyrc file In-Reply-To: References: Message-ID: <44BC2733.60002@ieee.org> Bill Baxter wrote: > On 7/18/06, Keith Goodman wrote: > >> On 7/17/06, Travis Oliphant wrote: >> >>> Keith Goodman wrote: >>> >> Does anyone out there save the print defaults across sessions? How do you do it? >> >> Does numpy look for any startup files (~/.numpyrc)? >> > > If you set a PYTHONSTARTUP environment variable then Python will run > the script it points to at startup of interactive sessions. > > But I wonder if maybe some numpy-specific startup script would be > good. My current PYTHONSTARTUP is importing numpy, just so I can > define some numpy shortcuts for my interactive sessions. That's fine > if you always use Numpy in your interactive sessions, but if not, then > it's just dead weight. > > The standard answer is to use ipython which gives you a wealth of startup options. -Travis From oliphant.travis at ieee.org Mon Jul 17 23:01:07 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 17 Jul 2006 21:01:07 -0600 Subject: [Numpy-discussion] Beta release on Thursday Message-ID: <44BC4EF3.5020103@ieee.org> I'd like to make release 1.0beta on Thursday. Please submit bug-reports and fixes before then. -Travis From nvf at MIT.EDU Mon Jul 17 23:30:46 2006 From: nvf at MIT.EDU (Nick Fotopoulos) Date: Mon, 17 Jul 2006 23:30:46 -0400 Subject: [Numpy-discussion] numpy.vectorize performance In-Reply-To: <44B9BA20.7000600@ieee.org> References: <44B9BA20.7000600@ieee.org> Message-ID: On Jul 16, 2006, at 12:01 AM, Travis Oliphant wrote: > Thanks for the decorator. This should be put on the www.scipy.org > wiki. I've been looking over the wiki and am not sure where the best place would be for such a snippet. Would it go with the numpy examples under vectorize or perhaps in a cookbook somewhere? This seems more specialized than the basic numpy examples, but not worthy of its own cookbook. In general, what do you do with constructs that seem useful, but aren't useful enough to just include somewhere in NumPy/ SciPy? How would anyone think to look for a tip like this? Also, thanks for your helpful responses and additional thanks to Travis for the book update. Take care, Nick From steffen.loeck at gmx.de Tue Jul 18 03:24:25 2006 From: steffen.loeck at gmx.de (Steffen Loeck) Date: Tue, 18 Jul 2006 09:24:25 +0200 Subject: [Numpy-discussion] Speed degression? In-Reply-To: <44AD8581.5030904@ee.byu.edu> References: <200607041519.24636.steffen.loeck@gmx.de> <44AD8581.5030904@ee.byu.edu> Message-ID: <200607180924.25652.steffen.loeck@gmx.de> > I also placed in hooks so you can replace the scalarmath (for int, > float, and complex) with the Python version of math (this works because > the int, float, and complex scalars are sub-classes of the corresponding > Python object). Just for completeness some more tests using pythonmath/scalarmath for int, float or both (in usec per loop): sin - array mod - array xx (a) - (no import of numpy.core.scalarmath) numpy 0.9.9.2800 152 76.5 numpy 0.9.9.2800 + math 50.2 (b) - (use_pythonmath(xx)) numpy 0.9.9.2800 107 60.4 (int) numpy 0.9.9.2800 + math 32.7 numpy 0.9.9.2800 148 43 (float) numpy 0.9.9.2800 + math 50.7 numpy 0.9.9.2800 109 26.5 (int, float) numpy 0.9.9.2800 + math 32.4 (c) - (use_scalarmath(xx)) numpy 0.9.9.2800 149 77.1 (int) numpy 0.9.9.2800 + math 50.7 numpy 0.9.9.2800 147 74.3 (float) numpy 0.9.9.2800 + math 50.7 numpy 0.9.9.2800 148 73.5 (int, float) numpy 0.9.9.2800 + math 50.8 Maybe use_pythonmath(int, float, complex) should be set as default? Many thanks, Steffen From aisaac at american.edu Tue Jul 18 04:10:49 2006 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 18 Jul 2006 04:10:49 -0400 Subject: [Numpy-discussion] A Different Arithmetic Error: += In-Reply-To: <20060718053657.GA19656@redwoodscientific.com> References: <20060718053657.GA19656@redwoodscientific.com> Message-ID: On Mon, 17 Jul 2006, John Lawless apparently wrote: >>>> from scipy import * >>>> a = array((1.2)) >>>> a += 1.3j >>>> a > array(1.2) > Shouldn't this generate either an error or an up-cast, rather than > silently discarding the imaginary part? As I understand it: it cannot upcast, as the '+=' operation will use only the memory initially allocated for a. Cheers, Alan Isaac From theller at python.net Tue Jul 18 04:32:38 2006 From: theller at python.net (Thomas Heller) Date: Tue, 18 Jul 2006 10:32:38 +0200 Subject: [Numpy-discussion] Cannot build numpy svn on Windows Message-ID: Building numpy from svn on Windows with Python 2.4.3 fails: c:\svn\numpy\numpy\core\include\numpy\arrayobject.h(986) : fatal error C1017: invalid integer constant expression When I change this line in the generated config.h file: #define NPY_ALLOW_THREADS WITH_THREAD to this one: #define NPY_ALLOW_THREADS 1 then I can build. Thomas From oliphant.travis at ieee.org Tue Jul 18 04:46:30 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 18 Jul 2006 02:46:30 -0600 Subject: [Numpy-discussion] numarray bug: dot product between 2x2 and 3x2x3 on Mac different from PC In-Reply-To: <200607171348.05776.haase@msg.ucsf.edu> References: <200607170940.48872.haase@msg.ucsf.edu> <200607171226.33383.haase@msg.ucsf.edu> <44BBE735.5010109@ieee.org> <200607171348.05776.haase@msg.ucsf.edu> Message-ID: <44BC9FE6.70702@ieee.org> Sebastian Haase wrote: > On Monday 17 July 2006 12:38, Travis Oliphant wrote: > > Any idea on my main question ? > What is the dot product of a 2x2 and 3x2x3 supposed to look like ? > Why are numarray and numpy giving different answers ?? > I'm pretty sure the dot-product in Numeric (and I guess numarray too) was broken for larger than 2-dimensions. This was fixed several months ago in NumPy. NumPy dot-product gives the following result a.shape is (I,L) b.shape is (J,L,K) Then c=dot(a,b) will have c.shape = (I,J,K) with c[i,j,k] = sum(a[i,:]*b[j,:,k]) I'm not even sure what Numeric is computing in this case. -Travis From gruben at bigpond.net.au Tue Jul 18 06:30:37 2006 From: gruben at bigpond.net.au (Gary Ruben) Date: Tue, 18 Jul 2006 20:30:37 +1000 Subject: [Numpy-discussion] numpy.vectorize performance In-Reply-To: References: <44B9BA20.7000600@ieee.org> Message-ID: <44BCB84D.1080008@bigpond.net.au> Nick Fotopoulos wrote: > I've been looking over the wiki and am not sure where the best place > would be for such a snippet. Would it go with the numpy examples > under vectorize or perhaps in a cookbook somewhere? Yes. It seems to me like a cookbook example. In the utopian future, when there are as many cookbook examples as OReilly have, it'll be time for a reorganisation, but for now, make it a cookbook entry. Gary R From mail at stevesimmons.com Tue Jul 18 07:59:30 2006 From: mail at stevesimmons.com (Stephen Simmons) Date: Tue, 18 Jul 2006 21:59:30 +1000 Subject: [Numpy-discussion] Advice for grouping recarrays Message-ID: <44BCCD22.7060607@stevesimmons.com> Hi, Does anyone have any suggestions for summarising data in numpy? The quick description is that I want to do something like the SQL statement: SELECT sum(field1), sum(field2) FROM table GROUP BY field3; The more accurate description is that my data is stored in PyTables HDF format, with 24 monthly files, each with 4m records describing how customers performed that month. Each record looks something like this: ('200604', 651404500000L, '800', 'K', 12L, 162.0, 2000.0, 0.054581, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 2.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 8.80, 0.86, 7.80 17.46, 0.0, 70.0, 0.0, 70.0, -142.93, 0.0, 2000.0, 2063.93, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, -9.71, 7.75, 87.46, 77.75, -3.45, 0.22, -0.45, -0.57, 73.95) The first 5 fields are status fields (month_string, account_number, product_code, account_status, months_since_customer_joined). The remaining 48 fields represent different aspects of the customer's performance during that month. I read 100,000 of these records at a time and turn them into a numpy recarray with: dat = hdf_table.read(start=pos, stop=pos+block_size) dat = numpy.asarray(dat._flatArray, dtype=dat.array_descr) I'd like to reduce these 96m records x 53 fields down to monthly averages for each tuple (month_string, months_since_customer_joined) which in the case above is ('200604', 12L). This will let me compare the performance of newly acquired customers at the same point in their lifecycle as customers acquired 1 or 2 years ago. The end result should be a dataset something like res[month_index, months_since_customer_joined] = array([ num_records, sum_field_5, sum_field_6, sum_field_7, ... sum_field_52 ]) with a shape of (24, 24, 49). I've played around with lexsort(), take(), sum(), etc, but get very confused and end up feeling that I'm making things more complicated than they need to be. So any advice from numpy veterans on how best to proceed would be very welcome! Cheers Stephen From tom.denniston at alum.dartmouth.org Tue Jul 18 09:49:51 2006 From: tom.denniston at alum.dartmouth.org (Tom Denniston) Date: Tue, 18 Jul 2006 08:49:51 -0500 Subject: [Numpy-discussion] Advice for grouping recarrays In-Reply-To: <44BCCD22.7060607@stevesimmons.com> References: <44BCCD22.7060607@stevesimmons.com> Message-ID: I suggest lexsort itertools.groupby of the indices take I think it would be really great if numpy had the first two as a function or something like that. It is really useful to be able to take an array and bucket it and apply further numpy operations like accumulation functions. On 7/18/06, Stephen Simmons wrote: > Hi, > > Does anyone have any suggestions for summarising data in numpy? > > The quick description is that I want to do something like the SQL statement: > SELECT sum(field1), sum(field2) FROM table GROUP BY field3; > > The more accurate description is that my data is stored in PyTables HDF > format, with 24 monthly files, each with 4m records describing how > customers performed that month. Each record looks something like this: > ('200604', 651404500000L, '800', 'K', 12L, 162.0, 2000.0, 0.054581, 0.0, > 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 2.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, > 8.80, 0.86, 7.80 17.46, 0.0, 70.0, 0.0, 70.0, -142.93, 0.0, 2000.0, > 2063.93, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, -9.71, 7.75, > 87.46, 77.75, -3.45, 0.22, -0.45, -0.57, 73.95) > The first 5 fields are status fields (month_string, account_number, > product_code, account_status, months_since_customer_joined). The > remaining 48 fields represent different aspects of the customer's > performance during that month. I read 100,000 of these records at a time > and turn them into a numpy recarray with: > dat = hdf_table.read(start=pos, stop=pos+block_size) > dat = numpy.asarray(dat._flatArray, dtype=dat.array_descr) > > I'd like to reduce these 96m records x 53 fields down to monthly > averages for each tuple (month_string, months_since_customer_joined) > which in the case above is ('200604', 12L). This will let me compare the > performance of newly acquired customers at the same point in their > lifecycle as customers acquired 1 or 2 years ago. > > The end result should be a dataset something like > res[month_index, months_since_customer_joined] > = array([ num_records, sum_field_5, sum_field_6, sum_field_7, ... > sum_field_52 ]) > with a shape of (24, 24, 49). > > I've played around with lexsort(), take(), sum(), etc, but get very > confused and end up feeling that I'm making things more complicated than > they need to be. So any advice from numpy veterans on how best to > proceed would be very welcome! > > Cheers > > Stephen > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From kwgoodman at gmail.com Tue Jul 18 09:54:44 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 18 Jul 2006 06:54:44 -0700 Subject: [Numpy-discussion] Cannot build numpy svn on Windows In-Reply-To: References: Message-ID: On 7/18/06, Thomas Heller wrote: > When I change this line in the generated config.h file: > > #define NPY_ALLOW_THREADS WITH_THREAD > > to this one: > > #define NPY_ALLOW_THREADS 1 > > then I can build. What part of numpy is threaded? From emsellem at obs.univ-lyon1.fr Tue Jul 18 10:23:53 2006 From: emsellem at obs.univ-lyon1.fr (Eric Emsellem) Date: Tue, 18 Jul 2006 16:23:53 +0200 Subject: [Numpy-discussion] fast way of doing "cross-multiplications" ? Message-ID: <44BCEEF9.1030200@obs.univ-lyon1.fr> Hi, I have a specific quantity to derive from an array, and I am at the moment unable to do it for a too large array because it just takes too long! So I am looking for an advice on how to efficiently compute such a quantity: I have 3 arrays of N floats (x[...], y[..], z[..]) and I wish to do: result = 0. for i in range(N) : for j in range(i+1,N,1) : result += 1. / sqrt((x[j] - x[i])**2 + (y[j] - y[i])**2 + (z[j] - z[i])**2) Of course the procedure written above is very inefficient and I thought of doing: result = 0. for i in range(N) : result += 1. / sqrt((x[i+1:] - x[i])**2 + (y[i+1:] - y[i])**2 + (z[i+1:] - z[i])**2) Still, this is quite slow and not workable for very large arrays (> 10^6 floats per array). Any hint on how to speed things up here? Thanks in advance! Eric From wbaxter at gmail.com Tue Jul 18 10:45:34 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 18 Jul 2006 23:45:34 +0900 Subject: [Numpy-discussion] fast way of doing "cross-multiplications" ? In-Reply-To: <44BCEEF9.1030200@obs.univ-lyon1.fr> References: <44BCEEF9.1030200@obs.univ-lyon1.fr> Message-ID: Maybe this will help -- it computes the squared distances between a bunch of points: def dist2(x,c): """ Calculates squared distance between two sets of points. n2 = dist2(x, c) D = DIST2(X, C) takes two matrices of vectors and calculates the squared Euclidean distance between them. Both matrices must be of the same column dimension. If X has M rows and N columns, and C has L rows and N columns, then the result has M rows and L columns. The I, Jth entry is the squared distance from the Ith row of X to the Jth row of C. """ ndata, dimx = x.shape ncentres, dimc = c.shape if dimx != dimc: raise ValueError('Data dimension does not match dimension of centres') n2 = (x*x).sum(1)[:,numpy.newaxis] + (c*c).sum(1) - 2*dot(x,T(c)) # Rounding errors occasionally cause negative entries in n2 #if (n2<0).any(): # n2[n2<0] = 0 return n2 Take 1.0/numpy.sqrt(dist2(V,V)) and from there you can probably get the sum with sum() calls. It's obviously not as efficient as it could be since it'll be computing both halves of a symmetric matrix, but it might be faster than nested Python loops. (The above was adapted from a routine in Netlab). --bb On 7/18/06, Eric Emsellem wrote: > Hi, > > I have a specific quantity to derive from an array, and I am at the > moment unable to do it for a too large array because it just takes too > long! So I am looking for an advice on how to efficiently compute such a > quantity: > > I have 3 arrays of N floats (x[...], y[..], z[..]) and I wish to do: > > result = 0. > for i in range(N) : > for j in range(i+1,N,1) : > result += 1. / sqrt((x[j] - x[i])**2 + (y[j] - y[i])**2 + (z[j] - > z[i])**2) > > > Of course the procedure written above is very inefficient and I thought > of doing: > > result = 0. > for i in range(N) : > result += 1. / sqrt((x[i+1:] - x[i])**2 + (y[i+1:] - y[i])**2 + > (z[i+1:] - z[i])**2) > > Still, this is quite slow and not workable for very large arrays (> 10^6 > floats per array). > > Any hint on how to speed things up here? > > Thanks in advance! > > Eric From perry at stsci.edu Tue Jul 18 11:23:53 2006 From: perry at stsci.edu (Perry Greenfield) Date: Tue, 18 Jul 2006 11:23:53 -0400 Subject: [Numpy-discussion] fast way of doing "cross-multiplications" ? In-Reply-To: <44BCEEF9.1030200@obs.univ-lyon1.fr> References: <44BCEEF9.1030200@obs.univ-lyon1.fr> Message-ID: <502862AF-514C-41CA-804E-420CE76FEE37@stsci.edu> On Jul 18, 2006, at 10:23 AM, Eric Emsellem wrote: > Hi, > > I have a specific quantity to derive from an array, and I am at the > moment unable to do it for a too large array because it just takes too > long! So I am looking for an advice on how to efficiently compute > such a > quantity: > > I have 3 arrays of N floats (x[...], y[..], z[..]) and I wish to do: > > result = 0. > for i in range(N) : > for j in range(i+1,N,1) : > result += 1. / sqrt((x[j] - x[i])**2 + (y[j] - y[i])**2 + (z > [j] - > z[i])**2) > > > Of course the procedure written above is very inefficient and I > thought > of doing: > > result = 0. > for i in range(N) : > result += 1. / sqrt((x[i+1:] - x[i])**2 + (y[i+1:] - y[i])**2 + > (z[i+1:] - z[i])**2) > > Still, this is quite slow and not workable for very large arrays (> > 10^6 > floats per array). > > Any hint on how to speed things up here? > > Thanks in advance! > > Eric Perhaps I'm misunderstanding the last variant but don't you want something like: result = 0. for i in range(N) : result += add.reduce(1. / sqrt((x[i+1:] - x[i])**2 + (y[i+1:] - y [i])**2 + (z[i+1:] - z[i])**2)) instead since the expression yields an array with a decreasing size each iteration? But besides that, it seems you are asking to do roughly 10^12 of these computations for 10^6 points. I don't see any way to avoid that given what you are computing. The solution Bill Baxter gives is fine (I think, I haven't looked at it closely), but the usual problem of doing it without any looping is that it requires an enormous amount of memory (~10^12 element arrays) if I'm not mistaken. Since your second example is iterating over large arrays (most of the time, not near the end), I'd be surprised if you can do much better than that (the looping overhead should be negligible for such large arrays). Do you have examples written in other languages that run much faster? I guess I would be surprised to see it possible to do more than a few times faster in any language without some very clever optimizations. Perry From emsellem at obs.univ-lyon1.fr Tue Jul 18 11:36:04 2006 From: emsellem at obs.univ-lyon1.fr (Eric Emsellem) Date: Tue, 18 Jul 2006 17:36:04 +0200 Subject: [Numpy-discussion] fast way of doing "cross-multiplications" ? In-Reply-To: <502862AF-514C-41CA-804E-420CE76FEE37@stsci.edu> References: <44BCEEF9.1030200@obs.univ-lyon1.fr> <502862AF-514C-41CA-804E-420CE76FEE37@stsci.edu> Message-ID: <44BCFFE4.6070502@obs.univ-lyon1.fr> thanks for the tips. (indeed your "add.reduce" is correct: I just wrote this down too quickly, in the script I have a "sum" included). And yes you are right for the memory issue, so I may just keep the loop in and try to make it work on a fast PC...(or use parallel processes) (is "sum" different than "add.reduce"?) thanks again to both Bill Baxter and Perry Greenfield for their fast (and helpful!) answers. cheers Eric Perry Greenfield wrote: > > On Jul 18, 2006, at 10:23 AM, Eric Emsellem wrote: > >> Hi, >> >> I have a specific quantity to derive from an array, and I am at the >> moment unable to do it for a too large array because it just takes too >> long! So I am looking for an advice on how to efficiently compute such a >> quantity: >> >> I have 3 arrays of N floats (x[...], y[..], z[..]) and I wish to do: >> >> result = 0. >> for i in range(N) : >> for j in range(i+1,N,1) : >> result += 1. / sqrt((x[j] - x[i])**2 + (y[j] - y[i])**2 + (z[j] - >> z[i])**2) >> >> >> Of course the procedure written above is very inefficient and I thought >> of doing: >> >> result = 0. >> for i in range(N) : >> result += 1. / sqrt((x[i+1:] - x[i])**2 + (y[i+1:] - y[i])**2 + >> (z[i+1:] - z[i])**2) >> >> Still, this is quite slow and not workable for very large arrays (> 10^6 >> floats per array). >> >> Any hint on how to speed things up here? >> >> Thanks in advance! >> >> Eric > > Perhaps I'm misunderstanding the last variant but don't you want > something like: > > result = 0. > for i in range(N) : > result += add.reduce(1. / sqrt((x[i+1:] - x[i])**2 + (y[i+1:] - > y[i])**2 + > (z[i+1:] - z[i])**2)) > > instead since the expression yields an array with a decreasing size > each iteration? > > But besides that, it seems you are asking to do roughly 10^12 of these > computations for 10^6 points. I don't see any way to avoid that given > what you are computing. The solution Bill Baxter gives is fine (I > think, I haven't looked at it closely), but the usual problem of doing > it without any looping is that it requires an enormous amount of > memory (~10^12 element arrays) if I'm not mistaken. Since your second > example is iterating over large arrays (most of the time, not near the > end), I'd be surprised if you can do much better than that (the > looping overhead should be negligible for such large arrays). Do you > have examples written in other languages that run much faster? I guess > I would be surprised to see it possible to do more than a few times > faster in any language without some very clever optimizations. > > Perry From fullung at gmail.com Tue Jul 18 11:51:18 2006 From: fullung at gmail.com (Albert Strasheim) Date: Tue, 18 Jul 2006 17:51:18 +0200 Subject: [Numpy-discussion] Cannot build numpy svn on Windows In-Reply-To: Message-ID: <031001c6aa82$021af7b0$0100000a@dsp.sun.ac.za> Hello all > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of Keith Goodman > Sent: 18 July 2006 15:55 > To: Thomas Heller > Cc: numpy-discussion at lists.sourceforge.net > Subject: Re: [Numpy-discussion] Cannot build numpy svn on Windows > > On 7/18/06, Thomas Heller wrote: > > > When I change this line in the generated config.h file: > > > > #define NPY_ALLOW_THREADS WITH_THREAD > > > > to this one: > > > > #define NPY_ALLOW_THREADS 1 > > > > then I can build. This might be due to some changes Travis made. http://projects.scipy.org/scipy/numpy/changeset/2833 I was able to build r2834 on Windows without problems. Are you still seeing this error? > What part of numpy is threaded? This thread stuff is so that NumPy releases/doesn't release the GIL in certain places. Regards, Albert From tim.hochberg at cox.net Tue Jul 18 12:13:23 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue, 18 Jul 2006 09:13:23 -0700 Subject: [Numpy-discussion] fast way of doing "cross-multiplications" ? In-Reply-To: <44BCFFE4.6070502@obs.univ-lyon1.fr> References: <44BCEEF9.1030200@obs.univ-lyon1.fr> <502862AF-514C-41CA-804E-420CE76FEE37@stsci.edu> <44BCFFE4.6070502@obs.univ-lyon1.fr> Message-ID: <44BD08A3.1030600@cox.net> Eric Emsellem wrote: > thanks for the tips. (indeed your "add.reduce" is correct: I just wrote > this down too quickly, in the script I have a "sum" included). > > And yes you are right for the memory issue, so I may just keep the loop > in and try to make it work on a fast PC...(or use parallel processes) > > (is "sum" different than "add.reduce"?) > > thanks again to both Bill Baxter and Perry Greenfield for their fast > (and helpful!) answers. > I just wanted to add that there are faster, but considerably complicated ways to attack this class of problems. The one I've looked at in the past was the fast multipole method and I believe there are others. I'm not sure whether these can be implemented efficiently in numpy, but you may want to take a look into this kind of more sophisticated/complicated approach if brute forcing the calculation doesn't work. -tim > cheers > > Eric > > Perry Greenfield wrote: > >> On Jul 18, 2006, at 10:23 AM, Eric Emsellem wrote: >> >> >>> Hi, >>> >>> I have a specific quantity to derive from an array, and I am at the >>> moment unable to do it for a too large array because it just takes too >>> long! So I am looking for an advice on how to efficiently compute such a >>> quantity: >>> >>> I have 3 arrays of N floats (x[...], y[..], z[..]) and I wish to do: >>> >>> result = 0. >>> for i in range(N) : >>> for j in range(i+1,N,1) : >>> result += 1. / sqrt((x[j] - x[i])**2 + (y[j] - y[i])**2 + (z[j] - >>> z[i])**2) >>> >>> >>> Of course the procedure written above is very inefficient and I thought >>> of doing: >>> >>> result = 0. >>> for i in range(N) : >>> result += 1. / sqrt((x[i+1:] - x[i])**2 + (y[i+1:] - y[i])**2 + >>> (z[i+1:] - z[i])**2) >>> >>> Still, this is quite slow and not workable for very large arrays (> 10^6 >>> floats per array). >>> >>> Any hint on how to speed things up here? >>> >>> Thanks in advance! >>> >>> Eric >>> >> Perhaps I'm misunderstanding the last variant but don't you want >> something like: >> >> result = 0. >> for i in range(N) : >> result += add.reduce(1. / sqrt((x[i+1:] - x[i])**2 + (y[i+1:] - >> y[i])**2 + >> (z[i+1:] - z[i])**2)) >> >> instead since the expression yields an array with a decreasing size >> each iteration? >> >> But besides that, it seems you are asking to do roughly 10^12 of these >> computations for 10^6 points. I don't see any way to avoid that given >> what you are computing. The solution Bill Baxter gives is fine (I >> think, I haven't looked at it closely), but the usual problem of doing >> it without any looping is that it requires an enormous amount of >> memory (~10^12 element arrays) if I'm not mistaken. Since your second >> example is iterating over large arrays (most of the time, not near the >> end), I'd be surprised if you can do much better than that (the >> looping overhead should be negligible for such large arrays). Do you >> have examples written in other languages that run much faster? I guess >> I would be surprised to see it possible to do more than a few times >> faster in any language without some very clever optimizations. >> >> Perry >> > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From theller at python.net Tue Jul 18 12:23:18 2006 From: theller at python.net (Thomas Heller) Date: Tue, 18 Jul 2006 18:23:18 +0200 Subject: [Numpy-discussion] Cannot build numpy svn on Windows In-Reply-To: <031001c6aa82$021af7b0$0100000a@dsp.sun.ac.za> References: <031001c6aa82$021af7b0$0100000a@dsp.sun.ac.za> Message-ID: Albert Strasheim schrieb: > Hello all > >> -----Original Message----- >> From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- >> discussion-bounces at lists.sourceforge.net] On Behalf Of Keith Goodman >> Sent: 18 July 2006 15:55 >> To: Thomas Heller >> Cc: numpy-discussion at lists.sourceforge.net >> Subject: Re: [Numpy-discussion] Cannot build numpy svn on Windows >> >> On 7/18/06, Thomas Heller wrote: >> >> > When I change this line in the generated config.h file: >> > >> > #define NPY_ALLOW_THREADS WITH_THREAD >> > >> > to this one: >> > >> > #define NPY_ALLOW_THREADS 1 >> > >> > then I can build. > > This might be due to some changes Travis made. > > http://projects.scipy.org/scipy/numpy/changeset/2833 > > I was able to build r2834 on Windows without problems. Are you still seeing > this error? No, I can build now. Travis fixed it immediately, probably with the change that you refer to above. Thomas From ndarray at mac.com Tue Jul 18 12:34:24 2006 From: ndarray at mac.com (Sasha) Date: Tue, 18 Jul 2006 12:34:24 -0400 Subject: [Numpy-discussion] fast way of doing "cross-multiplications" ? In-Reply-To: <44BCFFE4.6070502@obs.univ-lyon1.fr> References: <44BCEEF9.1030200@obs.univ-lyon1.fr> <502862AF-514C-41CA-804E-420CE76FEE37@stsci.edu> <44BCFFE4.6070502@obs.univ-lyon1.fr> Message-ID: On 7/18/06, Eric Emsellem wrote: [...] > (is "sum" different than "add.reduce"?) "sum" is a wrapper around ndarray.sum method, while add.reduce is a ufunc method. At the C level they are the same, but "sum" may be a little slower for the small arrays. > python -m timeit -s "from numpy import add, zeros, sum; x = zeros(10)" "sum(x)" 100000 loops, best of 3: 5.53 usec per loop > python -m timeit -s "from numpy import add, zeros, sum; x = zeros(10)" "add.reduce(x)" 100000 loops, best of 3: 2.71 usec per loop In the new code, I would recommend using the ndarray.sum method instead of either of the two. BTW, is this an optimization opportunity: compare > python -m timeit -s "from numpy import add, zeros, sum; x = zeros(10); r=add.reduce" "r(x)" 100000 loops, best of 3: 2.55 usec per loop and > python -m timeit -s "from numpy import zeros; x = zeros(10)" "x.sum()" 100000 loops, best of 3: 3.88 usec per loop ? From williams at astro.ox.ac.uk Tue Jul 18 12:49:28 2006 From: williams at astro.ox.ac.uk (Michael Williams) Date: Tue, 18 Jul 2006 17:49:28 +0100 Subject: [Numpy-discussion] updated Ubuntu Dapper packages for numpy, matplotlib, and scipy online In-Reply-To: <4496D1AC.8030100@astraw.com> References: <4496D1AC.8030100@astraw.com> Message-ID: <20060718164928.GB21578@astro.ox.ac.uk> Hi Andrew (and others), On Mon, Jun 19, 2006 at 09:32:44AM -0700, Andrew Straw wrote: >I have updated the apt repository I maintain for Ubuntu's Dapper, which >now includes: > >numpy >matplotlib >scipy > >Each package is from a recent SVN checkout and should thus be regarded >as "bleeding edge". The repository has a new URL: >http://debs.astraw.com/dapper/ I intend to keep this repository online >for an extended duration. If you want to put this repository in your >sources list, you need to add the following lines to >/etc/apt/sources.list:: > deb http://debs.astraw.com/ dapper/ > deb-src http://debs.astraw.com/ dapper/ I am unable to access these repositories (which sound very useful, and for which I am grateful to Andrew!). apt-get update gives "Failed to fetch http://debs.astraw.com/dapper/Release.gpg Could not resolve ?debs.astraw.com?" I am also unable to access the repositories listed on the website: deb http://sefton.astraw.com/ubuntu/ dapper/ deb-src http://sefton.astraw.com/ubuntu/ dapper/ for the same reason. Does anyone know where they've gone and if they're coming back? Cheers, -- Michael Williams Rudolph Peierls Centre for Theoretical Physics University of Oxford From robert.kern at gmail.com Tue Jul 18 13:02:09 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 18 Jul 2006 12:02:09 -0500 Subject: [Numpy-discussion] fast way of doing "cross-multiplications" ? In-Reply-To: <44BD08A3.1030600@cox.net> References: <44BCEEF9.1030200@obs.univ-lyon1.fr> <502862AF-514C-41CA-804E-420CE76FEE37@stsci.edu> <44BCFFE4.6070502@obs.univ-lyon1.fr> <44BD08A3.1030600@cox.net> Message-ID: Tim Hochberg wrote: > I just wanted to add that there are faster, but considerably complicated > ways to attack this class of problems. The one I've looked at in the > past was the fast multipole method and I believe there are others. I'm > not sure whether these can be implemented efficiently in numpy, but you > may want to take a look into this kind of more sophisticated/complicated > approach if brute forcing the calculation doesn't work. Idesbald Van den Bosch will be giving a talk at SciPy'06 on implementing FMM and the multilevel fast multipole algorithm (MLFMA) using SciPy and weave. http://www.scipy.org/SciPy2006 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From fperez.net at gmail.com Tue Jul 18 13:25:30 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 18 Jul 2006 11:25:30 -0600 Subject: [Numpy-discussion] fast way of doing "cross-multiplications" ? In-Reply-To: <44BD08A3.1030600@cox.net> References: <44BCEEF9.1030200@obs.univ-lyon1.fr> <502862AF-514C-41CA-804E-420CE76FEE37@stsci.edu> <44BCFFE4.6070502@obs.univ-lyon1.fr> <44BD08A3.1030600@cox.net> Message-ID: On 7/18/06, Tim Hochberg wrote: > Eric Emsellem wrote: > > thanks for the tips. (indeed your "add.reduce" is correct: I just wrote > > this down too quickly, in the script I have a "sum" included). > > > > And yes you are right for the memory issue, so I may just keep the loop > > in and try to make it work on a fast PC...(or use parallel processes) > > > > (is "sum" different than "add.reduce"?) > > > > thanks again to both Bill Baxter and Perry Greenfield for their fast > > (and helpful!) answers. > > > I just wanted to add that there are faster, but considerably complicated > ways to attack this class of problems. The one I've looked at in the > past was the fast multipole method and I believe there are others. I'm > not sure whether these can be implemented efficiently in numpy, but you > may want to take a look into this kind of more sophisticated/complicated > approach if brute forcing the calculation doesn't work. Indeed, FMM is the best known method that can turn this O(n^2) problem into O(n*log(n)). As Tim indicates, there is no easy way out of this one. Incidentally, there is a talk about FMM on the scipy'06 schedule, in case you are going to attend. An alternative approach to FMM (conceptually similar in some ways) is described here: http://amath.colorado.edu/faculty/fperez/talks/0604_sanum_mwadap.pdf Unfortunately this isn't exactly a half-hour optimization effort, as the required machinery is pretty heavy duty. And yes, this has all been done in python and runs on current numpy/scipy (though it has Fortran, C and C++ sprinkled as needed). Cheers, f From ndarray at mac.com Tue Jul 18 13:54:41 2006 From: ndarray at mac.com (Sasha) Date: Tue, 18 Jul 2006 13:54:41 -0400 Subject: [Numpy-discussion] A Different Arithmetic Error: += In-Reply-To: References: <20060718053657.GA19656@redwoodscientific.com> Message-ID: On 7/18/06, Alan G Isaac wrote: > it cannot upcast, as the '+=' operation will use only the > memory initially allocated for a. Not true: >>> x = [2,3] >>> x += array(2) >>> type(x) This is just the design choice made by numpy. I don't see the need for an error. Augmented assignment is a sufficiently advanced feature that those who use it can be expected to know what it does. Loosing imaginary part maybe a more compelling reason for an error than loosing precision, but it would be clearly wrong to have different types behave differently. Silent downcasting from float to int is actually a useful feature. From strawman at astraw.com Tue Jul 18 14:45:32 2006 From: strawman at astraw.com (Andrew Straw) Date: Tue, 18 Jul 2006 11:45:32 -0700 Subject: [Numpy-discussion] updated Ubuntu Dapper packages for numpy, matplotlib, and scipy online In-Reply-To: <20060718164928.GB21578@astro.ox.ac.uk> References: <4496D1AC.8030100@astraw.com> <20060718164928.GB21578@astro.ox.ac.uk> Message-ID: <44BD2C4C.4020205@astraw.com> Michael Williams wrote: >>Hi Andrew (and others), >> >>On Mon, Jun 19, 2006 at 09:32:44AM -0700, Andrew Straw wrote: >> >> > > >>>>I have updated the apt repository I maintain for Ubuntu's Dapper, which >>>>now includes: >>>> >>>>numpy >>>>matplotlib >>>>scipy >>>> >>>>Each package is from a recent SVN checkout and should thus be regarded >>>>as "bleeding edge". The repository has a new URL: >>>>http://debs.astraw.com/dapper/ I intend to keep this repository online >>>>for an extended duration. If you want to put this repository in your >>>>sources list, you need to add the following lines to >>>>/etc/apt/sources.list:: >>>>deb http://debs.astraw.com/ dapper/ >>>>deb-src http://debs.astraw.com/ dapper/ >>>> >>>> >> >> >> >>I am unable to access these repositories (which sound very useful, and >>for which I am grateful to Andrew!). apt-get update gives >> >> "Failed to fetch http://debs.astraw.com/dapper/Release.gpg Could not >> resolve ?debs.astraw.com?" >> >> >> > > Hmm, that looks like DNS error. My repository is still up and online... debs.astraw.com is actually a "CNAME" record, which aliases another domain name. The canonical domain name uses a dynamic DNS to point to a dynamically assigned IP addresses. Your system (probably in your DNS name resolution) must be having some issue with some of that. Unfortunately, you can't just plug in the IP address, because I'm using Apache virtual hosting to serve a website at "debs.astraw.com" differently from other websites. Anyhow, here's what I get, hopefully it will help you fix your DNS issue. $ host debs.astraw.com debs.astraw.com is an alias for astraw-office.kicks-ass.net. astraw-office.kicks-ass.net has address 131.215.28.162 debs.astraw.com is an alias for astraw-office.kicks-ass.net. debs.astraw.com is an alias for astraw-office.kicks-ass.net. >>I am also unable to access the repositories listed on the website: >> >> deb http://sefton.astraw.com/ubuntu/ dapper/ >> deb-src http://sefton.astraw.com/ubuntu/ dapper/ >> >> > > Eek, I replaced that with the new one location and some more info. >>for the same reason. Does anyone know where they've gone and if they're >>coming back? >> >> >> > > I'm planning on keeping them around for a while -- at least until numpy, scipy, and matplotlib get integrated into my flavor-of-the-year Debian/Ubuntu release in a sufficiently up-to-date version. From oliphant.travis at ieee.org Tue Jul 18 14:50:57 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 18 Jul 2006 12:50:57 -0600 Subject: [Numpy-discussion] What to do about numarray extension modules? Message-ID: <44BD2D91.1020800@ieee.org> I'd like to get suggestions about what to do about the remaining numarray extension modules that are not addressed in NumPy (convolve, image, nd_image). There will be people coming from numarray that will be using these modules. Of course, my opinion is that they should not have been placed in Numarray to begin with as the fit better as modules in SciPy. But, given the situation. Should we bring them over to NumPy? Or, perhaps provide a small package that contain these modules separately? With the numarray c_api that is now available in NumPy, these can be easily ported. It would be nice to have decided this by 1.0beta release at the end of the week. Discussion please... -Travis From robert.kern at gmail.com Tue Jul 18 15:09:56 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 18 Jul 2006 14:09:56 -0500 Subject: [Numpy-discussion] What to do about numarray extension modules? In-Reply-To: <44BD2D91.1020800@ieee.org> References: <44BD2D91.1020800@ieee.org> Message-ID: Travis Oliphant wrote: > I'd like to get suggestions about what to do about the remaining > numarray extension modules that are not addressed in NumPy (convolve, > image, nd_image). There will be people coming from numarray that will > be using these modules. > > Of course, my opinion is that they should not have been placed in > Numarray to begin with as the fit better as modules in SciPy. > > But, given the situation. Should we bring them over to NumPy? Or, > perhaps provide a small package that contain these modules separately? I'm definitely -1 on putting them in numpy. numpy is large enough as it is. I'm -0 on making a standalone package. The time would be better spent on making sure that scipy subpackages can be downloaded, built and installed individually. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant.travis at ieee.org Tue Jul 18 15:22:14 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 18 Jul 2006 13:22:14 -0600 Subject: [Numpy-discussion] What to do about numarray extension modules? In-Reply-To: References: <44BD2D91.1020800@ieee.org> Message-ID: <44BD34E6.2020603@ieee.org> Robert Kern wrote: > Travis Oliphant wrote: > >> I'd like to get suggestions about what to do about the remaining >> numarray extension modules that are not addressed in NumPy (convolve, >> image, nd_image). There will be people coming from numarray that will >> be using these modules. >> >> Of course, my opinion is that they should not have been placed in >> Numarray to begin with as the fit better as modules in SciPy. >> >> But, given the situation. Should we bring them over to NumPy? Or, >> perhaps provide a small package that contain these modules separately? >> > > I'm definitely -1 on putting them in numpy. numpy is large enough as it is. I'm > -0 on making a standalone package. The time would be better spent on making sure > that scipy subpackages can be downloaded, built and installed individually. > > I think I like this idea as well. I don't really want to grow NumPy. But, I'm curious what the current numarray users think. -Travis From filipwasilewski at gmail.com Tue Jul 18 15:21:46 2006 From: filipwasilewski at gmail.com (Filip Wasilewski) Date: Tue, 18 Jul 2006 21:21:46 +0200 Subject: [Numpy-discussion] A bunch of tickets Message-ID: Hi, I have been recently working with arrays of objects and noticed some problems. Here are the tickets: Array improperly created from numpy.poly1d object: http://projects.scipy.org/scipy/numpy/ticket/185 Can't create matrix of dtype=object directly from list (problem of ndarray.__new__): http://projects.scipy.org/scipy/numpy/ticket/186 Subclassing does not always work for numpy.matrix: http://projects.scipy.org/scipy/numpy/ticket/187 best, Filip From haase at msg.ucsf.edu Tue Jul 18 15:25:23 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 18 Jul 2006 12:25:23 -0700 Subject: [Numpy-discussion] What to do about numarray extension modules? In-Reply-To: References: <44BD2D91.1020800@ieee.org> Message-ID: <200607181225.23671.haase@msg.ucsf.edu> I'm using nd_image extensively. Considering that SciPy seem to install much easier these days than I remember I would be just happy to put them in to SciPy. Some numarray people prefer a transitional "special place". I remember that there was a discussion maybe a year ago of making SciPy so modular that (some?) sub-packages would be installable completely independent anyway - that would be the best :-) -Sebastian Haase On Tuesday 18 July 2006 12:09, Robert Kern wrote: > Travis Oliphant wrote: > > I'd like to get suggestions about what to do about the remaining > > numarray extension modules that are not addressed in NumPy (convolve, > > image, nd_image). There will be people coming from numarray that will > > be using these modules. > > > > Of course, my opinion is that they should not have been placed in > > Numarray to begin with as the fit better as modules in SciPy. > > > > But, given the situation. Should we bring them over to NumPy? Or, > > perhaps provide a small package that contain these modules separately? > > I'm definitely -1 on putting them in numpy. numpy is large enough as it is. > I'm -0 on making a standalone package. The time would be better spent on > making sure that scipy subpackages can be downloaded, built and installed > individually. From nvf at MIT.EDU Tue Jul 18 15:28:40 2006 From: nvf at MIT.EDU (Nick Fotopoulos) Date: Tue, 18 Jul 2006 15:28:40 -0400 Subject: [Numpy-discussion] What to do about numarray extension modules? Message-ID: Travis wrote: > Of course, my opinion is that they should not have been placed in > Numarray to begin with as the fit better as modules in SciPy. Can we not put them into SciPy now? Nick From perry at stsci.edu Tue Jul 18 15:34:46 2006 From: perry at stsci.edu (Perry Greenfield) Date: Tue, 18 Jul 2006 15:34:46 -0400 Subject: [Numpy-discussion] What to do about numarray extension modules? In-Reply-To: <44BD34E6.2020603@ieee.org> References: <44BD2D91.1020800@ieee.org> <44BD34E6.2020603@ieee.org> Message-ID: On Jul 18, 2006, at 3:22 PM, Travis Oliphant wrote: > Robert Kern wrote: >> Travis Oliphant wrote: >> >>> I'd like to get suggestions about what to do about the remaining >>> numarray extension modules that are not addressed in NumPy >>> (convolve, >>> image, nd_image). There will be people coming from numarray >>> that will >>> be using these modules. >>> >>> Of course, my opinion is that they should not have been placed in >>> Numarray to begin with as the fit better as modules in SciPy. >>> >>> But, given the situation. Should we bring them over to NumPy? Or, >>> perhaps provide a small package that contain these modules >>> separately? >>> >> >> I'm definitely -1 on putting them in numpy. numpy is large enough >> as it is. I'm >> -0 on making a standalone package. The time would be better spent >> on making sure >> that scipy subpackages can be downloaded, built and installed >> individually. >> >> > I think I like this idea as well. I don't really want to grow > NumPy. > But, I'm curious what the current numarray users think. > Our view, briefly, is that until it is easy to install scipy, or that scipy supports modular installs, that we need one or the other to support our distributions. The first choice (bundling with numpy) would be more convenient for us, but is not necessary should they be available separately. So, in short, either is fine with us. Perry From svetosch at gmx.net Tue Jul 18 15:37:27 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Tue, 18 Jul 2006 21:37:27 +0200 Subject: [Numpy-discussion] Beta release on Thursday In-Reply-To: <44BC4EF3.5020103@ieee.org> References: <44BC4EF3.5020103@ieee.org> Message-ID: <44BD3877.5010002@gmx.net> Travis Oliphant schrieb: > I'd like to make release 1.0beta on Thursday. Please submit bug-reports > and fixes before then. > Just two things I noticed: >>> import numpy as n import dft -> failed: module compiled against version 90709 of C-API but this version of numpy is 9090d >>> n.__version__ '0.9.9.2840' And TODO.txt looks pretty outdated, maybe to avoid a wrong impression and confusion when a lot of new people turn to numpy because of 1.0 status this should be changed (or removed?). Cheers, Sven From filip at ftv.pl Tue Jul 18 15:37:39 2006 From: filip at ftv.pl (Filip Wasilewski) Date: Tue, 18 Jul 2006 21:37:39 +0200 Subject: [Numpy-discussion] A bunch of tickets Message-ID: <151688431.20060718213739@gmail.com> Hi, I have posted this mail from my google account a while ago, but it obviously was blocked by sourceforge. What about that proposal of moving numpy-discussion to scipy host? Anyway, I have been recently working with arrays of objects and noticed some problems. Here are the tickets: Array improperly created from numpy.poly1d objects: http://projects.scipy.org/scipy/numpy/ticket/185 Can't create matrix of dtype=object directly from list (problem of ndarray.__new__): http://projects.scipy.org/scipy/numpy/ticket/186 Subclassing does not always work for numpy.matrix: http://projects.scipy.org/scipy/numpy/ticket/187 best, Filip From perry at stsci.edu Tue Jul 18 15:45:59 2006 From: perry at stsci.edu (Perry Greenfield) Date: Tue, 18 Jul 2006 15:45:59 -0400 Subject: [Numpy-discussion] What to do about numarray extension modules? In-Reply-To: <44BD2D91.1020800@ieee.org> References: <44BD2D91.1020800@ieee.org> Message-ID: On Jul 18, 2006, at 2:50 PM, Travis Oliphant wrote: > > I'd like to get suggestions about what to do about the remaining > numarray extension modules that are not addressed in NumPy (convolve, > image, nd_image). There will be people coming from numarray that > will > be using these modules. > > Of course, my opinion is that they should not have been placed in > Numarray to begin with as the fit better as modules in SciPy. At that at the time (some years ago now), that wasn't an option for us for the modules that we contributed since we couldn't use Numeric. I'm glad that it's now possible to integrate these with numpy, but it doesn't do much good now to lament that it didn't happen that way in the past. Perry From Pierre.Barbier_de_Reuille at sophia.inria.fr Tue Jul 18 15:32:36 2006 From: Pierre.Barbier_de_Reuille at sophia.inria.fr (Pierre Barbier de Reuille) Date: Tue, 18 Jul 2006 20:32:36 +0100 Subject: [Numpy-discussion] What to do about numarray extension modules? In-Reply-To: <200607181225.23671.haase@msg.ucsf.edu> References: <44BD2D91.1020800@ieee.org> <200607181225.23671.haase@msg.ucsf.edu> Message-ID: <44BD3754.8020609@sophia.inria.fr> Sebastian Haase wrote: > I'm using nd_image extensively. Considering that SciPy seem to install much > easier these days than I remember I would be just happy to put them in to > SciPy. > Well, what is the content of nd_image and image, compared to scipy.ndimage ? > Some numarray people prefer a transitional "special place". I remember that > there was a discussion maybe a year ago of making SciPy so modular that > (some?) sub-packages would be installable completely independent anyway - > that would be the best :-) > > -Sebastian Haase > > On Tuesday 18 July 2006 12:09, Robert Kern wrote: > >> Travis Oliphant wrote: >> >>> I'd like to get suggestions about what to do about the remaining >>> numarray extension modules that are not addressed in NumPy (convolve, >>> image, nd_image). There will be people coming from numarray that will >>> be using these modules. >>> >>> Of course, my opinion is that they should not have been placed in >>> Numarray to begin with as the fit better as modules in SciPy. >>> >>> But, given the situation. Should we bring them over to NumPy? Or, >>> perhaps provide a small package that contain these modules separately? >>> >> I'm definitely -1 on putting them in numpy. numpy is large enough as it is. >> I'm -0 on making a standalone package. The time would be better spent on >> making sure that scipy subpackages can be downloaded, built and installed >> individually. >> > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From oliphant at ee.byu.edu Tue Jul 18 16:21:42 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 18 Jul 2006 14:21:42 -0600 Subject: [Numpy-discussion] Beta release on Thursday In-Reply-To: <44BD3877.5010002@gmx.net> References: <44BC4EF3.5020103@ieee.org> <44BD3877.5010002@gmx.net> Message-ID: <44BD42D6.2020806@ee.byu.edu> Sven Schreiber wrote: >Travis Oliphant schrieb: > > >>I'd like to make release 1.0beta on Thursday. Please submit bug-reports >>and fixes before then. >> >> >> > >Just two things I noticed: > > Thanks for commenting > > >>>>import numpy as n >>>> >>>> >import dft -> failed: module compiled against version 90709 of C-API but >this version of numpy is 9090d > > > Make sure to do a clean build: i.e. rm -fr build python setup.py install Unfortunately, the dependency tracking does not re-build the sub-packages of numpy by itself when there is a header-file change. >And TODO.txt looks pretty outdated, maybe to avoid a wrong impression >and confusion when a lot of new people turn to numpy because of 1.0 >status this should be changed (or removed?). > > Good idea. -Travis From laidler at stsci.edu Tue Jul 18 16:23:14 2006 From: laidler at stsci.edu (Victoria G. Laidler) Date: Tue, 18 Jul 2006 16:23:14 -0400 Subject: [Numpy-discussion] What to do about numarray extension modules? In-Reply-To: <44BD2D91.1020800@ieee.org> References: <44BD2D91.1020800@ieee.org> Message-ID: <44BD4332.9030305@stsci.edu> I'm a numarray user that relies on convolve. I would hope that one of the criteria by which to make such decisions is "make the transition for numarray and numeric users as painless as possible". Changing namespaces and introducing additional dependencies is not painless. Thus unless there is a *compelling* argument to move them out, I would leave them in. (From the numarray-user point of view, it would be "moving them out" even though they aren't there yet.) Personally I wouldn't think that "numpy is big enough already" or "it fits better in scipy" to be compelling reasons. So, +1 to include them in numpy. That said, personally I would argue that convolve is qualitatively different from image and ndimage. Convolution is a common mathematical operation on arrays; thus I think it belongs in numpy. If any or all of them are not included in numpy, I would strongly prefer a small separate package than to put them in scipy. I try to minimize dependencies in my code; I really don't want to suddenly depend on all of scipy just so I can do a convolution! Vicki Laidler Travis Oliphant wrote: >I'd like to get suggestions about what to do about the remaining >numarray extension modules that are not addressed in NumPy (convolve, >image, nd_image). There will be people coming from numarray that will >be using these modules. > >Of course, my opinion is that they should not have been placed in >Numarray to begin with as the fit better as modules in SciPy. > >But, given the situation. Should we bring them over to NumPy? Or, >perhaps provide a small package that contain these modules separately? > >With the numarray c_api that is now available in NumPy, these can be >easily ported. It would be nice to have decided this by 1.0beta release >at the end of the week. > >Discussion please... > >-Travis > > >------------------------------------------------------------------------- >Take Surveys. Earn Cash. Influence the Future of IT >Join SourceForge.net's Techsay panel and you'll get the chance to share your >opinions on IT & business topics through brief surveys -- and earn cash >http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From oliphant at ee.byu.edu Tue Jul 18 16:25:01 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 18 Jul 2006 14:25:01 -0600 Subject: [Numpy-discussion] What to do about numarray extension modules? In-Reply-To: References: <44BD2D91.1020800@ieee.org> Message-ID: <44BD439D.8060807@ee.byu.edu> Perry Greenfield wrote: > > On Jul 18, 2006, at 2:50 PM, Travis Oliphant wrote: > >> >> I'd like to get suggestions about what to do about the remaining >> numarray extension modules that are not addressed in NumPy (convolve, >> image, nd_image). There will be people coming from numarray that will >> be using these modules. >> >> Of course, my opinion is that they should not have been placed in >> Numarray to begin with as the fit better as modules in SciPy. > > > At that at the time (some years ago now), that wasn't an option for > us for the modules that we contributed since we couldn't use Numeric. > I'm glad that it's now possible to integrate these with numpy, but it > doesn't do much good now to lament that it didn't happen that way in > the past. Of course. I was just thinking aloud again. That's the beauty of "organic" growth :-) It does have it's own charm. -Travis From williams at astro.ox.ac.uk Tue Jul 18 16:36:12 2006 From: williams at astro.ox.ac.uk (Michael Williams) Date: Tue, 18 Jul 2006 21:36:12 +0100 Subject: [Numpy-discussion] [SciPy-user] updated Ubuntu Dapper packages for numpy, matplotlib, and scipy online In-Reply-To: <20060718164928.GB21578@astro.ox.ac.uk> References: <4496D1AC.8030100@astraw.com> <20060718164928.GB21578@astro.ox.ac.uk> Message-ID: <20060718203612.GA25724@astro.ox.ac.uk> On Tue, Jul 18, 2006 at 11:45:32AM -0700, Andrew Straw wrote: >>I am unable to access these repositories (which sound very useful, and >>for which I am grateful to Andrew!). > >Hmm, that looks like DNS error. My repository is still up and online... It is indeed a DNS error. I should have tested from a different machine before posting. Thanks for the quick reply. >>for the same reason. Does anyone know where they've gone and if >>they're coming back? >> >I'm planning on keeping them around for a while -- at least until >numpy, scipy, and matplotlib get integrated into my flavor-of-the-year >Debian/Ubuntu release in a sufficiently up-to-date version. Great. I've installed them now, and the seem to be working fine. Thanks very much! Cheers, -- Mike From oliphant at ee.byu.edu Tue Jul 18 16:43:21 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 18 Jul 2006 14:43:21 -0600 Subject: [Numpy-discussion] What to do about numarray extension modules? In-Reply-To: <44BD4332.9030305@stsci.edu> References: <44BD2D91.1020800@ieee.org> <44BD4332.9030305@stsci.edu> Message-ID: <44BD47E9.6010102@ee.byu.edu> Victoria G. Laidler wrote: > I'm a numarray user that relies on convolve. Thanks for voicing your opinion. We should be clear about what we are talking about. 1) There is already a convolve in NumPy (it's the same one that was in Numeric). It seems the one in numarray is compatible. I'm talking about the following numarray functions (please let me know if I'm missing anything). * all of nd_image --- currently this is in scipy (as ndimage) and can already be built and installed as a separate package --- we need to work out distribution, though. * all of image/ combine.py median average minimum threshold _image.py translate * from Convolve/ iraf_frame.py lineshape.py Convolve.py/ correlate2d convolve2d boxcar For now, I will port the image and Convolve packages and place them in the scipy/sandbox area. From there we can figure out what to do with them. -Travis From jmiller at stsci.edu Tue Jul 18 18:37:06 2006 From: jmiller at stsci.edu (Todd Miller) Date: Tue, 18 Jul 2006 18:37:06 -0400 Subject: [Numpy-discussion] numarray bug: dot product between 2x2 and 3x2x3 on Mac different from PC In-Reply-To: <200607171348.05776.haase@msg.ucsf.edu> References: <200607170940.48872.haase@msg.ucsf.edu> <200607171226.33383.haase@msg.ucsf.edu> <44BBE735.5010109@ieee.org> <200607171348.05776.haase@msg.ucsf.edu> Message-ID: <44BD6292.5050600@stsci.edu> Sebastian Haase wrote: > OK - understood. Combining int32 with float64 proves to be less > cumbersome ... > > Why are numarray and numpy giving different answers ? > I have a little insight as to what is going on here but no fix. numarray has two versions of dot(): 1. The original dot() is standalone, easy to install, and basically works. This dot() does however have the type coercion wart where Int32 dotted with Float32 silently loses precision. 2. A more optimized dot(), ported from Numeric, depends on an external BLAS. This makes it faster but also generally harder to install. This optimized dot() has the higher dimensional array bug you discovered. For arrays of rank > 2, it's totally broken. Since Mac OS-X has a standard BLAS, it uses the optimized dot() by default and exposes the bug. The optimized dot() is easy to turn off in cfg_packages.py by commenting out the lines: # if os.path.exists('/System/Library/Frameworks/vecLib.framework'): # USE_LAPACK = True Since numarray is being phased out and neither issue is a serious problem for STScI, we're not planning to fix them. If it's causing someone major hardship let us know and we'll reconsider. Todd From Chris.Barker at noaa.gov Tue Jul 18 18:57:42 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 18 Jul 2006 15:57:42 -0700 Subject: [Numpy-discussion] What to do about numarray extension modules? In-Reply-To: References: <44BD2D91.1020800@ieee.org> Message-ID: <44BD6766.2020507@noaa.gov> Robert Kern wrote: > I'm definitely -1 on putting them in numpy. numpy is large enough as it is. I'm > -0 on making a standalone package. The time would be better spent on making sure > that scipy subpackages can be downloaded, built and installed individually. +1 The whole SciPy installation thing is HUGE! I now I still don't use any of SciPy because I can't get it all built. Is it yet possible to build SciPy on Fedora core 4 without lots of painful machinations? When numpy was called SciPy.core, there was a lot of talk about how SciPy needed to be broken up into individual packages, particularly the Fortran and non-Fortran ones separated. It was a good idea then, and it still is. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From josh.p.marshall at gmail.com Tue Jul 18 20:32:06 2006 From: josh.p.marshall at gmail.com (Josh Marshall) Date: Wed, 19 Jul 2006 10:32:06 +1000 Subject: [Numpy-discussion] [Pythonmac-SIG] Packaging numpy with py2app In-Reply-To: <1CC7BE8A-F534-4256-A5A2-FEA2E1D2725A@redivi.com> References: <660C6421-8C71-4B4D-A2CC-33C3FF089DF9@gmail.com> <44BC1B92.4090009@noaa.gov> <44BC1E6E.9020502@noaa.gov> <51B08665-6077-4E6F-875D-D217990B1714@redivi.com> <60E2AEE1-9BF1-4B62-A9DC-7A497310A681@gmail.com> <213C1E2D-AE4D-4578-9E22-FB49C1ACAA75@gmail.com> <9148A14B-7462-47EC-9262-52802E5A9CB4@gmail.com> <1CC7BE8A-F534-4256-A5A2-FEA2E1D2725A@redivi.com> Message-ID: <0039ED8C-EA8A-47BE-9967-200AF66D173A@gmail.com> Thanks for the info on how the various recipes work, Bob. Very helpful. On 19/07/2006, at 9:28 AM, Bob Ippolito wrote: > The recipe mechanism doesn't allow for it because it doesn't > generally make sense. There are very few packages that can find > their resources in an alternative manner. I'm not totally opposed > to adding another feature to support that use case, but throwing it > directly in the resources dir seems like a bad idea to do > automatically from a recipe. The files should sit under some kind > of matplotlib hierarchy. It would be nicer to see matplotlib just > figure out how to make their app work from a zip... pygame does, > and it has fonts and icons. I think matplotlib can access files inside the zip, since it searches inside the following directories upon startup and is frozen-aware. MatplotlibEmbedSample.app/Contents/Resources/lib/python2.4/site- packages.zip/matplotlib/mpl-data MatplotlibEmbedSample.app/Contents/mpl-data MatplotlibEmbedSample.app/Contents/Resources/mpl-data So I have been attempting to put the data files in the zip. > loader_files puts data files *in the zip* Indeed it does. When I use the following recipe: (noting that the recipe's filename can't be "matplotlib.py", I've been using "matplotlib_recipe.py"): def check(cmd, mf): m = mf.findNode('matplotlib') if m is None or m.filename is None: return None import matplotlib, glob, os mpl_datapath = matplotlib.get_data_path() mpl_datafilelist = glob.glob(mpl_datapath + r'/*') # only include individual files from the mpl-data directory # this should exclude directories, which should only be Matplotlib.nib mpl_datafilelist = [mpl_file for mpl_file in mpl_datafilelist if os.path.isfile(mpl_file)] mf.import_hook('pytz.zoneinfo', m, ['UTC']) mf.import_hook('matplotlib.numerix', m, ['random_array']) return dict( loader_files = [ ('matplotlib', mpl_datafilelist), ], ) The data files then get put in the matplotlib directory in the zip. However, matplotlib searches in the mpl-data directory *inside* the matplotlib directory, as shown in the search paths above. I have not been able to find a way to include a directory within the matplotlib directory. If I use loader_files = [ ('matplotlib/mpl-data', mpl_datafilelist), ], py2app barfs when trying to copy the files since the directory doesn't exist. Any ideas Bob? Chris, can you please copy my email when replying to the thread? I'm getting the digest, so I don't get your messages for quite some time otherwise. Christopher Barker wrote: > I wonder what the plan should be for recipes for MPL and the like. > As Bob mentioned, perhaps it's best for the developer to find a way > to manually include what they want, as there is a LOT of stuff in > MPL (and certainly SciPy!) and much of will not be used by a given > application. If we can get numpy and scipy to work without a recipe that just includes the packages we can definitely do this. The problem here is not with py2app, but rather with numpy/scipy themselves and their crazy custom importer which doesn't work with a zipped site-packages. I am close to getting matplotlib working with a decent non-naive recipe. This means it'll only include what is being used by the application. > It seems the only way to make a recipe that is useful would be for > it to include EVERYTHING. That may not be a bad default, but there > should certainly be a way to turn it off if need be. No, a useful and "proper" recipe just adds the files that are not referenced directly through imports, ie., data files. This is what I am doing with matplotlib. > Also, I got a numpy app to work yesterday by using: > packages=['numpy'] The recipe I sent out in the zip yesterday did exactly that. Bob has added it to py2app trunk. There is a similar recipe for scipy and matplotlib as well. These will do until we get proper ones working. The packages option does include *everything*, by doing a full copy of the package. This means that source, documentation, example files, etc gets included along with the bytecode files. > That then includes the entire numpy package. That works, but it > puts in a lot of stuff I don't need (dft, linear_algebra, etc). > Maybe the recipe should include only the core stuff, and let the > user add the extra packages, if need be. AFAICT, this can't really be done. From looking at the numpy structure, there are subpackages 'testing','core','lib','linalg','dft','random','f2py', which are all loaded in the numpy __init__.py via the following: pkgload = PackageLoader() pkgload('testing','core','lib','linalg','dft','random','f2py', verbose=NUMPY_IMPORT_VERBOSE,postpone=False) The doc string for PackageLoader.__call__() says: > This function is intended to shorten the need to import many of > subpackages, say of scipy, constantly with statements such as > > import scipy.linalg, scipy.fftpack, scipy.etc... > > Instead, you can say: > > import scipy > scipy.pkgload('linalg','fftpack',...) > > or > > scipy.pkgload() so it seems like the entire purpose of PackageLoader is to make life difficult for me, just to save a few lines of typing. :) Seriously, can a numpy developer tell me why PackageLoader is necessary? > Or should numpy be fixed so that it doesn't do weird imports? The weird imports are okay, but they need to be able to work from a site-packages.zip. That's the next job. Hopefully I'll get time to look at it before numpy goes too far into beta. Scipy is a long way off. Cheers all, Josh From haase at msg.ucsf.edu Tue Jul 18 20:40:45 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 18 Jul 2006 17:40:45 -0700 Subject: [Numpy-discussion] numarray uses "rank" instead "ndim" Message-ID: <200607181740.45980.haase@msg.ucsf.edu> Hi, (Thanks to Travis and Todd for answering my last "dot"-question) I'm trying to adjust my numarray based code to also run with numpy. It seems that a (maybe the only left) problem is that numarray had arr.rank giving the value of len(arr.shape) while in numpy this is now called arr.ndim Is the "ndim" attribute already added in the CVS version (just now I could not connect to the CVS server) ? Or else, could it be added to numarray to ease the transition !? Another way might be to go back to len(arr.shape) - but that would be quite ugly ... Also I noted that a numpy array object does not allow "sticking" a new attribute into it: In numarray I could just say arr.xyz0='test' like with any (simple) Python object - in numpy I get an AttributeError: 'numpy.ndarray' object has no attribute 'xyz0' !? Thanks for everybody help. Sebastian Haase From oliphant at ee.byu.edu Tue Jul 18 20:45:18 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 18 Jul 2006 18:45:18 -0600 Subject: [Numpy-discussion] numarray uses "rank" instead "ndim" In-Reply-To: <200607181740.45980.haase@msg.ucsf.edu> References: <200607181740.45980.haase@msg.ucsf.edu> Message-ID: <44BD809E.8070106@ee.byu.edu> Sebastian Haase wrote: >Hi, >(Thanks to Travis and Todd for answering my last "dot"-question) > >I'm trying to adjust my numarray based code to also run with numpy. >It seems that a (maybe the only left) problem is that numarray had >arr.rank giving the value of len(arr.shape) >while in numpy this is now called >arr.ndim > >Is the "ndim" attribute already added in the CVS version (just now I could not >connect to the CVS server) ? Or else, could it be added to numarray to ease >the transition !? >Another way might be to go back to len(arr.shape) - but that would be quite >ugly ... > >Also I noted that a numpy array object does not allow "sticking" a new >attribute into it: In numarray I could just say arr.xyz0='test' like with >any (simple) Python object - in numpy I get an >AttributeError: 'numpy.ndarray' object has no attribute 'xyz0' >!? > > Why don't you try using a simple sub-class of numpy's ndarray class myarray(numpy.ndarray): pass That way your new array can look like a numarray array object and have the same methods and attributes. I've thought about even adding something like this to the numarray directory. -Travis From bob at redivi.com Tue Jul 18 20:57:29 2006 From: bob at redivi.com (Bob Ippolito) Date: Tue, 18 Jul 2006 17:57:29 -0700 Subject: [Numpy-discussion] [Pythonmac-SIG] Packaging numpy with py2app In-Reply-To: <0039ED8C-EA8A-47BE-9967-200AF66D173A@gmail.com> References: <660C6421-8C71-4B4D-A2CC-33C3FF089DF9@gmail.com> <44BC1B92.4090009@noaa.gov> <44BC1E6E.9020502@noaa.gov> <51B08665-6077-4E6F-875D-D217990B1714@redivi.com> <60E2AEE1-9BF1-4B62-A9DC-7A497310A681@gmail.com> <213C1E2D-AE4D-4578-9E22-FB49C1ACAA75@gmail.com> <9148A14B-7462-47EC-9262-52802E5A9CB4@gmail.com> <1CC7BE8A-F534-4256-A5A2-FEA2E1D2725A@redivi.com> <0039ED8C-EA8A-47BE-9967-200AF66D173A@gmail.com> Message-ID: On Jul 18, 2006, at 5:32 PM, Josh Marshall wrote: > Thanks for the info on how the various recipes work, Bob. Very > helpful. > > On 19/07/2006, at 9:28 AM, Bob Ippolito wrote: >> The recipe mechanism doesn't allow for it because it doesn't >> generally make sense. There are very few packages that can find >> their resources in an alternative manner. I'm not totally opposed >> to adding another feature to support that use case, but throwing it >> directly in the resources dir seems like a bad idea to do >> automatically from a recipe. The files should sit under some kind >> of matplotlib hierarchy. It would be nicer to see matplotlib just >> figure out how to make their app work from a zip... pygame does, >> and it has fonts and icons. > > I think matplotlib can access files inside the zip, since it searches > inside the following directories upon startup and is frozen-aware. > > MatplotlibEmbedSample.app/Contents/Resources/lib/python2.4/site- > packages.zip/matplotlib/mpl-data > MatplotlibEmbedSample.app/Contents/mpl-data > MatplotlibEmbedSample.app/Contents/Resources/mpl-data > > So I have been attempting to put the data files in the zip. > >> loader_files puts data files *in the zip* > > Indeed it does. When I use the following recipe: (noting that the > recipe's filename can't be "matplotlib.py", I've been using > "matplotlib_recipe.py"): > > def check(cmd, mf): > m = mf.findNode('matplotlib') > if m is None or m.filename is None: > return None > > import matplotlib, glob, os > mpl_datapath = matplotlib.get_data_path() > mpl_datafilelist = glob.glob(mpl_datapath + r'/*') > > # only include individual files from the mpl-data directory > # this should exclude directories, which should only be > Matplotlib.nib > mpl_datafilelist = [mpl_file for mpl_file in mpl_datafilelist > if os.path.isfile(mpl_file)] > > mf.import_hook('pytz.zoneinfo', m, ['UTC']) > mf.import_hook('matplotlib.numerix', m, ['random_array']) > return dict( > loader_files = [ > ('matplotlib', mpl_datafilelist), > ], > ) > > The data files then get put in the matplotlib directory in the zip. > However, matplotlib searches in the mpl-data directory *inside* the > matplotlib directory, as shown in the search paths above. > > I have not been able to find a way to include a directory within the > matplotlib directory. If I use > loader_files = [ > ('matplotlib/mpl-data', mpl_datafilelist), > ], > py2app barfs when trying to copy the files since the directory > doesn't exist. Any ideas Bob? This should work in svn (r13). You can easily run the development version straight from a svn checkout: svn co http://svn.pythonmac.org/py2app/py2app/trunk py2app cd py2app python setup.py develop That'll set it up such that the "installed" py2app is actually what's sitting in your svn checkout. You can make changes without re- installing or anything. -bob From josh.p.marshall at gmail.com Tue Jul 18 22:26:51 2006 From: josh.p.marshall at gmail.com (Josh Marshall) Date: Wed, 19 Jul 2006 12:26:51 +1000 Subject: [Numpy-discussion] [Pythonmac-SIG] Packaging numpy with py2app In-Reply-To: References: <660C6421-8C71-4B4D-A2CC-33C3FF089DF9@gmail.com> <44BC1B92.4090009@noaa.gov> <44BC1E6E.9020502@noaa.gov> <51B08665-6077-4E6F-875D-D217990B1714@redivi.com> <60E2AEE1-9BF1-4B62-A9DC-7A497310A681@gmail.com> <213C1E2D-AE4D-4578-9E22-FB49C1ACAA75@gmail.com> <9148A14B-7462-47EC-9262-52802E5A9CB4@gmail.com> <1CC7BE8A-F534-4256-A5A2-FEA2E1D2725A@redivi.com> <0039ED8C-EA8A-47BE-9967-200AF66D173A@gmail.com> Message-ID: <98FDFEDF-CB5D-4014-8214-653FC4CA1CCA@gmail.com> Thanks for the fix Bob. Unfortunately Matplotlib does not work with zipped data files, after all that. So, we'll leave the recipes as is, as they work for now. I suspect the way forward is to get numpy/Matplotlib/scipy working with setuptools and using pkg_resources to manage the data files. Any thoughts on this? From bob at redivi.com Tue Jul 18 23:15:29 2006 From: bob at redivi.com (Bob Ippolito) Date: Tue, 18 Jul 2006 20:15:29 -0700 Subject: [Numpy-discussion] [Pythonmac-SIG] Packaging numpy with py2app In-Reply-To: <98FDFEDF-CB5D-4014-8214-653FC4CA1CCA@gmail.com> References: <660C6421-8C71-4B4D-A2CC-33C3FF089DF9@gmail.com> <44BC1B92.4090009@noaa.gov> <44BC1E6E.9020502@noaa.gov> <51B08665-6077-4E6F-875D-D217990B1714@redivi.com> <60E2AEE1-9BF1-4B62-A9DC-7A497310A681@gmail.com> <213C1E2D-AE4D-4578-9E22-FB49C1ACAA75@gmail.com> <9148A14B-7462-47EC-9262-52802E5A9CB4@gmail.com> <1CC7BE8A-F534-4256-A5A2-FEA2E1D2725A@redivi.com> <0039ED8C-EA8A-47BE-9967-200AF66D173A@gmail.com> <98FDFEDF-CB5D-4014-8214-653FC4CA1CCA@gmail.com> Message-ID: On Jul 18, 2006, at 7:26 PM, Josh Marshall wrote: > Thanks for the fix Bob. > > Unfortunately Matplotlib does not work with zipped data files, > after all that. So, we'll leave the recipes as is, as they work for > now. > > I suspect the way forward is to get numpy/Matplotlib/scipy working > with setuptools and using pkg_resources to manage the data files. > Any thoughts on this? Yes, that is the way. -bob From haase at msg.ucsf.edu Wed Jul 19 00:04:47 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 18 Jul 2006 21:04:47 -0700 Subject: [Numpy-discussion] calling a f2py made module with numarray (mem shared with numpy ?) Message-ID: <44BDAF5F.40701@msg.ucsf.edu> Hi, Having to use Fortran again, I had a look at f2py .. I looks really great !! Everything seems to run out of the box !! Thanks a lot ! I think I tried intend(inout) and then fed in an array object that really was an numarray as opposed to a numpy array. I was surprised that the array did not get altered by the function ! If I assign numpyArray = numpy.asarray(numarrayArr) don't numpyArray and numarrayArr share identical memory for their data? Another observation I made was that a C-contiguous array of rank 2 was not accepted by 'intend(inout)' (the error message said that it had to be FORTRAN-contiguous - as opposed to the f2py documentation I read online http://cens.ioc.ee/projects/f2py2e/usersguide/index.html#attributes which says : fortran OR c contiguous ) Thanks for another great tool ! Sebastian Haase From robert.kern at gmail.com Wed Jul 19 00:16:50 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 18 Jul 2006 23:16:50 -0500 Subject: [Numpy-discussion] [Pythonmac-SIG] Packaging numpy with py2app In-Reply-To: <0039ED8C-EA8A-47BE-9967-200AF66D173A@gmail.com> References: <660C6421-8C71-4B4D-A2CC-33C3FF089DF9@gmail.com> <44BC1B92.4090009@noaa.gov> <44BC1E6E.9020502@noaa.gov> <51B08665-6077-4E6F-875D-D217990B1714@redivi.com> <60E2AEE1-9BF1-4B62-A9DC-7A497310A681@gmail.com> <213C1E2D-AE4D-4578-9E22-FB49C1ACAA75@gmail.com> <9148A14B-7462-47EC-9262-52802E5A9CB4@gmail.com> <1CC7BE8A-F534-4256-A5A2-FEA2E1D2725A@redivi.com> <0039ED8C-EA8A-47BE-9967-200AF66D173A@gmail.com> Message-ID: Josh Marshall wrote: > so it seems like the entire purpose of PackageLoader is to make life > difficult for me, just to save a few lines of typing. :) Seriously, > can a numpy developer tell me why PackageLoader is necessary? I can't think of a good reason why it's used in __init__.py the way it is (it used to have postpone=True). If Pearu, who wrote that bit of code, doesn't speak up by Thursday, I'll have it removed in favor of regular imports for the beta. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant.travis at ieee.org Wed Jul 19 00:48:30 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 18 Jul 2006 22:48:30 -0600 Subject: [Numpy-discussion] [Pythonmac-SIG] Packaging numpy with py2app In-Reply-To: References: <660C6421-8C71-4B4D-A2CC-33C3FF089DF9@gmail.com> <44BC1B92.4090009@noaa.gov> <44BC1E6E.9020502@noaa.gov> <51B08665-6077-4E6F-875D-D217990B1714@redivi.com> <60E2AEE1-9BF1-4B62-A9DC-7A497310A681@gmail.com> <213C1E2D-AE4D-4578-9E22-FB49C1ACAA75@gmail.com> <9148A14B-7462-47EC-9262-52802E5A9CB4@gmail.com> <1CC7BE8A-F534-4256-A5A2-FEA2E1D2725A@redivi.com> <0039ED8C-EA8A-47BE-9967-200AF66D173A@gmail.com> Message-ID: <44BDB99E.70406@ieee.org> Robert Kern wrote: > Josh Marshall wrote: > >> so it seems like the entire purpose of PackageLoader is to make life >> difficult for me, just to save a few lines of typing. :) Seriously, >> can a numpy developer tell me why PackageLoader is necessary? >> > > I can't think of a good reason why it's used in __init__.py the way it is (it > used to have postpone=True). If Pearu, who wrote that bit of code, doesn't speak > up by Thursday, I'll have it removed in favor of regular imports for the beta. > > This will probably help with ticket #145 as well. A user of ActiveState's Komodo on Windows gets a program crash which started when the PackageLoader started getting used in 0.9.4 AFAIK, The two main purposes of the package loader are docstring manipulation and postponing imports. Right now, it's probably only the docstring portion of the code that is really being used. I would appreciate hearing Pearu's comments, however. From robert.kern at gmail.com Wed Jul 19 00:52:52 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 18 Jul 2006 23:52:52 -0500 Subject: [Numpy-discussion] [Pythonmac-SIG] Packaging numpy with py2app In-Reply-To: <44BDB99E.70406@ieee.org> References: <660C6421-8C71-4B4D-A2CC-33C3FF089DF9@gmail.com> <44BC1B92.4090009@noaa.gov> <44BC1E6E.9020502@noaa.gov> <51B08665-6077-4E6F-875D-D217990B1714@redivi.com> <60E2AEE1-9BF1-4B62-A9DC-7A497310A681@gmail.com> <213C1E2D-AE4D-4578-9E22-FB49C1ACAA75@gmail.com> <9148A14B-7462-47EC-9262-52802E5A9CB4@gmail.com> <1CC7BE8A-F534-4256-A5A2-FEA2E1D2725A@redivi.com> <0039ED8C-EA8A-47BE-9967-200AF66D173A@gmail.com> <44BDB99E.70406@ieee.org> Message-ID: Travis Oliphant wrote: > Robert Kern wrote: >> Josh Marshall wrote: >> >>> so it seems like the entire purpose of PackageLoader is to make life >>> difficult for me, just to save a few lines of typing. :) Seriously, >>> can a numpy developer tell me why PackageLoader is necessary? >>> >> I can't think of a good reason why it's used in __init__.py the way it is (it >> used to have postpone=True). If Pearu, who wrote that bit of code, doesn't speak >> up by Thursday, I'll have it removed in favor of regular imports for the beta. >> > This will probably help with ticket #145 as well. A user of > ActiveState's Komodo on Windows gets a program crash which started when > the PackageLoader started getting used in 0.9.4 > > AFAIK, The two main purposes of the package loader are docstring > manipulation and postponing imports. Right now, it's probably only the > docstring portion of the code that is really being used. There's also the configurable verbosity and hiding of ImportErrors. Admittedly, I dislike both of those. Currently, I don't think any of these features are worth messing with something as fundamental as importing. Especially not in __init__.py . It always works out to introduce fragility in something that needs to be rock-solid. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From bob at redivi.com Wed Jul 19 01:27:04 2006 From: bob at redivi.com (Bob Ippolito) Date: Tue, 18 Jul 2006 22:27:04 -0700 Subject: [Numpy-discussion] [Pythonmac-SIG] Packaging numpy with py2app In-Reply-To: References: <660C6421-8C71-4B4D-A2CC-33C3FF089DF9@gmail.com> <44BC1B92.4090009@noaa.gov> <44BC1E6E.9020502@noaa.gov> <51B08665-6077-4E6F-875D-D217990B1714@redivi.com> <60E2AEE1-9BF1-4B62-A9DC-7A497310A681@gmail.com> <213C1E2D-AE4D-4578-9E22-FB49C1ACAA75@gmail.com> <9148A14B-7462-47EC-9262-52802E5A9CB4@gmail.com> <1CC7BE8A-F534-4256-A5A2-FEA2E1D2725A@redivi.com> <0039ED8C-EA8A-47BE-9967-200AF66D173A@gmail.com> Message-ID: <61EE48D5-1689-4C46-9CDA-C1BF76500B5E@redivi.com> On Jul 18, 2006, at 9:16 PM, Robert Kern wrote: > Josh Marshall wrote: >> so it seems like the entire purpose of PackageLoader is to make life >> difficult for me, just to save a few lines of typing. :) Seriously, >> can a numpy developer tell me why PackageLoader is necessary? > > I can't think of a good reason why it's used in __init__.py the way > it is (it > used to have postpone=True). If Pearu, who wrote that bit of code, > doesn't speak > up by Thursday, I'll have it removed in favor of regular imports > for the beta. Note that all you need is the bytecode, it doesn't have to actually execute. So you can still use whatever custom import junk if you really want to: def _modulegraph_hints(): import foo You should do it in a dead function instead of "if 0" because dead functions will never be stripped by the compiler but dead code blocks sometimes are. -bob From verveer at embl-heidelberg.de Wed Jul 19 03:10:51 2006 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Wed, 19 Jul 2006 09:10:51 +0200 Subject: [Numpy-discussion] What to do about numarray extension modules? In-Reply-To: <44BD2D91.1020800@ieee.org> References: <44BD2D91.1020800@ieee.org> Message-ID: <8B1708B4-FE99-4323-8CDA-20F23C16BDDA@embl-heidelberg.de> I am the author of nd_image. In my opinion the package should remain in SciPy, where it indeed belongs. The only reason it was in numarray is that there was no such thing as SciPy for numarray. Although it is relatively bug-free and has a reasonable amount of image processing functions, it could stand a lot of improvement. Including it in a base NumPy package would be a mistake, as it is not a fully optimized and finished package. nd_image should install easily as a separate package, hence it could be included in SciPy while maintaining the possibility for standalone usage. Peter > > I'd like to get suggestions about what to do about the remaining > numarray extension modules that are not addressed in NumPy (convolve, > image, nd_image). There will be people coming from numarray that > will > be using these modules. > > Of course, my opinion is that they should not have been placed in > Numarray to begin with as the fit better as modules in SciPy. > > But, given the situation. Should we bring them over to NumPy? Or, > perhaps provide a small package that contain these modules separately? > > With the numarray c_api that is now available in NumPy, these can be > easily ported. It would be nice to have decided this by 1.0beta > release > at the end of the week. > > Discussion please... > > -Travis > > > ---------------------------------------------------------------------- > --- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to > share your > opinions on IT & business topics through brief surveys -- and earn > cash > http://www.techsay.com/default.php? > page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From oliphant.travis at ieee.org Wed Jul 19 03:57:22 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 19 Jul 2006 01:57:22 -0600 Subject: [Numpy-discussion] Improved creation of object arrays Message-ID: <44BDE5E2.2060703@ieee.org> I made a small change to the array creation function so that if you explicitly specify dtype=object, then the logic for determining the shape of the array is changed. The new logic for constructing dtype=object arrays from Python sequences is that the shape is determined by nested lists (and only lists). The nesting must be consistent or else the elements of the last conforming lists are assumed to be objects. Here are some examples: array([[1,2],3,4],dtype=object) returns a 1-d array of shape (3,) array([[1,2],[3,4]],dtype=object) returns a 2-d array of shape (2,2) array([(1,2),(3,4)],dtype=object) returns a 1-d array of shape (2,) array([],dtype=object) returns a 0-d array of shape () array([[],[],[]],dtype=object) returns a 1-d array of shape (3,) array([[3,4],[5,6],None],dtype=object) returns a 1-d array of shape (3,) You have to actually specify dtype=object to get this, otherwise the old code-path will be taken. -Travis From fullung at gmail.com Wed Jul 19 08:22:15 2006 From: fullung at gmail.com (Albert Strasheim) Date: Wed, 19 Jul 2006 14:22:15 +0200 Subject: [Numpy-discussion] Allocating discontiguous arrays Message-ID: <035201c6ab2d$f88ce8a0$0100000a@dsp.sun.ac.za> Hello all In some situations, I have to work with very large matrices. My Windows machine has 3 GB RAM, so I would expect to be able to use most of my process's address space for my matrix. Unfortunately, with matrices much larger than 700 or 800 MB, one starts running into heap fragmentation problems: even though there's 2 GB available to your process, it isn't available in one contiguous block. To see this, you can try the following code which tries to allocate a ~1792 MB 2-d array or a list of 1-d arrays that add up to the same size: import numpy as N fdtype = N.dtype(' For 1-d inputs I think r_ should act like vstack, and c_ should act like column_stack. Currently r_ and c_ both act like hstack for 1-d inputs. Background: I keep getting bitten by the fact that this doesn't work: >>> a = array([1,2,3]) >>> b = array([[1,2,3],[2,3,4]]) >>> c = array([4,5,6]) >>> r_[b,c] **error** and that this doesn't return a 2x3 >>> d = r_[a,c] array([1, 2, 3, 4, 5, 6]) To get what I want I need something like : >>> r_[[a],[c]] array([[1, 2, 3], [4, 5, 6]]) And to get them columnwise I need the likes of: >>> c_[a[:,newaxis], c[:,newaxis]] array([[1, 4], [2, 5], [3, 6]]) It seems like the r_ (which I think of as a "row concatenator") should assume that 1-d arrays are rows (like vstack does), and the c_ ("column concatenator") should assume 1-d arrays are columns (like column_stack does). Then you'd have: >>> r_[a,c] array([[1, 2, 3], [4, 5, 6]]) >>> c_[a,c] array([[1, 4], [2, 5], [3, 6]]) At any rate, r_'s behavior is currently different from vstack wrt 1-d arrays, and identitcal to c_ and hstack. Vstack, hstack, and column_stack give the following: >>> vstack([a,c]) array([[1, 2, 3], [4, 5, 6]]) >>> hstack([a,c]) array([1, 2, 3, 4, 5, 6]) >>> column_stack([a,c]) array([[1, 4], [2, 5], [3, 6]]) Also isn't it odd that there's a column_stack, but no row_stack? I guess that's because row_stack would just be an alias for vstack, but still. --bb From sebzur at pin.if.uz.zgora.pl Wed Jul 19 09:33:08 2006 From: sebzur at pin.if.uz.zgora.pl (=?UTF-8?B?U2ViYXN0aWFuIMW7dXJlaw==?=) Date: Wed, 19 Jul 2006 14:33:08 +0100 Subject: [Numpy-discussion] RandomArray module In-Reply-To: References: Message-ID: Thanks a lot Francescm, Bruce and Robert! Your response was all the essentials I wanted to know. As soon as I'll be back from my vacations time I'll try numpys PRNG. Sebastian From svetosch at gmx.net Wed Jul 19 09:06:34 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Wed, 19 Jul 2006 15:06:34 +0200 Subject: [Numpy-discussion] r_, c_, hstack, and vstack with 1-d arrays In-Reply-To: References: Message-ID: <44BE2E5A.80404@gmx.net> Bill Baxter schrieb: > For 1-d inputs I think r_ should act like vstack, and c_ should act > like column_stack. > Currently r_ and c_ both act like hstack for 1-d inputs. Well Bill, as I said before, you should have stayed with matrices ;-) Seriously, I think all the quirks you mentioned recently are now gone. I mean what you're experiencing now ist just logical from the point of view that rows and columns are 2d concepts, so a 1d array just isn't enough. Having said that, however, here are some more constructive comments: First, I think using c_ is not recommended, it doesn't even appear in the book. But I have no idea why that is so, I'm using it also (with matrices), so I think it's valid to question that deprecation. Indeed it seems natural to make it act like column_stack for 1d arrays!? Second, changing r_ would obviously break some people's codes. Two workarounds: You switch to matrices (sorry, couldn't resist), or you use vstack (trivial, sorry again). > > Also isn't it odd that there's a column_stack, but no row_stack? I > guess that's because row_stack would just be an alias for vstack, but > still. > I agree that for the poor non-matrix-users it would be good to have a pair of names that obviously belong together, instead of the current vstack/column_stack situation; with matrices you have the choice of r_/c_ or vstack/hstack that do exactly what you want (as you know). Good luck, Sven From wbaxter at gmail.com Wed Jul 19 09:39:12 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 19 Jul 2006 22:39:12 +0900 Subject: [Numpy-discussion] r_, c_, hstack, and vstack with 1-d arrays In-Reply-To: <44BE2E5A.80404@gmx.net> References: <44BE2E5A.80404@gmx.net> Message-ID: On 7/19/06, Sven Schreiber wrote: > Bill Baxter schrieb: > > For 1-d inputs I think r_ should act like vstack, and c_ should act > > like column_stack. > > Currently r_ and c_ both act like hstack for 1-d inputs. > > Well Bill, as I said before, you should have stayed with matrices ;-) :-D > Seriously, I think all the quirks you mentioned recently are now gone. Yeh, probably so. The main one remaining for me is that matrix can't take more than 2-D data. So if I decide to go, say, from storing a single polygon (N points x 2 coords), to a set of deformed versions of the polygon (M deformations x N points x 2 coords), then suddenly I have to switch a bunch of matrix types back to array types, and find and fix all the small differences between the two, and deal with two different types in my code (array and matrix) instead of one type everywhere (array). Or maybe I have to use a Python list of matrix. Maybe it wouldn't be as bad as I imagine. I haven't actually gone through such a conversion with matrices. It just seems like it would probably be more pain than it is to add an extra dimension without changing data type. > I mean what you're experiencing now ist just logical from the point of > view that rows and columns are 2d concepts, so a 1d array just isn't enough. True, but other parts of Numpy handle that ambiguity more gracefully. Like dot(), or column_stack. > Having said that, however, here are some more constructive comments: > > First, I think using c_ is not recommended, it doesn't even appear in > the book. But I have no idea why that is so, I'm using it also (with > matrices), so I think it's valid to question that deprecation. Indeed it > seems natural to make it act like column_stack for 1d arrays!? Hmm, yeh, what is the status on r_ and c_? My understanding was that they were just new/experimental, and thus not documented in the book. I think they're quite handy, so I hope they're not going away. There was a discussion previously about changing the names to something a little more attractive, but I've gotten pretty used to the names now. > Second, changing r_ would obviously break some people's codes. Two > workarounds: You switch to matrices (sorry, couldn't resist), or you use > vstack (trivial, sorry again). Well, if they are new, probably not that much code. > > Also isn't it odd that there's a column_stack, but no row_stack? I > > guess that's because row_stack would just be an alias for vstack, but > > still. > I agree that for the poor non-matrix-users it would be good to have a > pair of names that obviously belong together, instead of the current > vstack/column_stack situation; with matrices you have the choice of > r_/c_ or vstack/hstack that do exactly what you want (as you know). --bill From yves.frederix at cs.kuleuven.be Wed Jul 19 10:45:08 2006 From: yves.frederix at cs.kuleuven.be (Yves Frederix) Date: Wed, 19 Jul 2006 16:45:08 +0200 Subject: [Numpy-discussion] astype() causes segmentation fault when converting from string Message-ID: <44BE4574.3040704@cs.kuleuven.be> Hi, The latest svn version of numpy seems to be causing a segmentation fault when converting an array containing strings to floats using .astype(). It only seems to happen when converting 'from' strings. Casting between numerical types and 'to' strings is no problem. :yves $ python -c "import numpy as N; print N.__version__; a = N.asarray('2'); a.astype(float);print 'OK';" 0.9.9.2844 Segmentation fault An older version of numpy (the one I used before this upgrade) works perfectly though: :yves $ python -c "import numpy as N; print N.__version__; a = N.asarray('2'); a.astype(float);print 'OK';" 0.9.9.2655 OK Cheers, YVES From bioinformed at gmail.com Wed Jul 19 11:24:40 2006 From: bioinformed at gmail.com (Kevin Jacobs ) Date: Wed, 19 Jul 2006 11:24:40 -0400 Subject: [Numpy-discussion] Performance of defmatrix.py:103(__array_finalize__) Message-ID: <2e1434c10607190824l322ae6d4p128dc9c35a637f16@mail.gmail.com> Hi all, Is it expected that the following script would spend over half of the total execution time in defmatrix.py:103(__array_finalize__)? from numpy import matrix m=matrix( (1,1) ) for i in range(1000000): m[0,0] = 0. Python version is 2.5b1, NumPy 0.9.8. More importantly, is there a way to avoid this overhead? Thanks, -Kevin Jacobs -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.u.r.e.l.i.a.n at gmx.net Wed Jul 19 11:22:23 2006 From: a.u.r.e.l.i.a.n at gmx.net (Johannes Loehnert) Date: Wed, 19 Jul 2006 17:22:23 +0200 Subject: [Numpy-discussion] confirm 98d7f7f15c27f5122181163442d621a0b506d01c In-Reply-To: References: Message-ID: <200607191722.24226.a.u.r.e.l.i.a.n@gmx.net> On Wednesday 19 July 2006 14:41, numpy-discussion-request at lists.sourceforge.net wrote: > Your membership in the mailing list Numpy-discussion has been disabled > due to excessive bounces The last bounce received from you was dated > 19-Jul-2006. You will not get any more messages from this list until > you re-enable your membership. You will receive 3 more reminders like > this before your membership in the list is deleted. > > To re-enable your membership, you can simply respond to this message > (leaving the Subject: line intact), or visit the confirmation page at > > > https://lists.sourceforge.net/lists/confirm/numpy-discussion/98d7f7f15c27f5 >122181163442d621a0b506d01c > > > You can also visit your membership page at > > > https://lists.sourceforge.net/lists/options/numpy-discussion/a.u.r.e.l.i.a. >n%40gmx.net > > > On your membership page, you can change various delivery options such > as your email address and whether you get digests or not. As a > reminder, your membership password is > > numpylist > > If you have any questions or problems, you can contact the list owner > at > > numpy-discussion-owner at lists.sourceforge.net From oliphant.travis at ieee.org Wed Jul 19 11:34:13 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 19 Jul 2006 09:34:13 -0600 Subject: [Numpy-discussion] astype() causes segmentation fault when converting from string In-Reply-To: <44BE4574.3040704@cs.kuleuven.be> References: <44BE4574.3040704@cs.kuleuven.be> Message-ID: <44BE50F5.9090609@ieee.org> Yves Frederix wrote: > Hi, > > The latest svn version of numpy seems to be causing a segmentation fault > when converting an array containing strings to floats using .astype(). > It only seems to happen when converting 'from' strings. Casting between > numerical types and 'to' strings is no problem. > > :yves $ python -c "import numpy as N; print N.__version__; a = > N.asarray('2'); a.astype(float);print 'OK';" > 0.9.9.2844 > Segmentation fault > > A bad attempt at allowing other threads to run is at fault. We can't release the GIL for casting to and/or from strings.... This should be fixed now. -Travis From bioinformed at gmail.com Wed Jul 19 11:36:41 2006 From: bioinformed at gmail.com (Kevin Jacobs ) Date: Wed, 19 Jul 2006 11:36:41 -0400 Subject: [Numpy-discussion] Performance of defmatrix.py:103(__array_finalize__) In-Reply-To: <2e1434c10607190824l322ae6d4p128dc9c35a637f16@mail.gmail.com> References: <2e1434c10607190824l322ae6d4p128dc9c35a637f16@mail.gmail.com> Message-ID: <2e1434c10607190836g282d7e70hcf34145b222ad462@mail.gmail.com> On 7/19/06, Kevin Jacobs wrote: > > Is it expected that the following script would spend over half of the > total execution time in defmatrix.py:103(__array_finalize__)? > Confirmed that a similar overhead exists for arrays, though not as extreme. My application requires the ability to assign random elements, so any speed ups here will result be quite significant. Thanks, -Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Wed Jul 19 11:46:20 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 19 Jul 2006 09:46:20 -0600 Subject: [Numpy-discussion] Performance of defmatrix.py:103(__array_finalize__) In-Reply-To: <2e1434c10607190824l322ae6d4p128dc9c35a637f16@mail.gmail.com> References: <2e1434c10607190824l322ae6d4p128dc9c35a637f16@mail.gmail.com> Message-ID: <44BE53CC.1020002@ieee.org> Kevin Jacobs wrote: > Hi all, > > Is it expected that the following script would spend over half of the > total execution time in defmatrix.py:103(__array_finalize__)? > > from numpy import matrix > > m=matrix( (1,1) ) > for i in range(1000000): > m[0,0] = 0. > > > Python version is 2.5b1, NumPy 0.9.8. > > More importantly, is there a way to avoid this overhead? This over-head should be removed in latest SVN NumPy. Basically, at each point in the loop the 0. is being converted to a matrix. You could probably avoid the over-head by doing the conversion first to a 0-d array. val = array(0.) m = matrix( (1,1) ) for i in range(10000): m[0,0] = val -Travis From oliphant.travis at ieee.org Wed Jul 19 12:08:52 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 19 Jul 2006 10:08:52 -0600 Subject: [Numpy-discussion] Improved creation of object arrays In-Reply-To: <44BDE5E2.2060703@ieee.org> References: <44BDE5E2.2060703@ieee.org> Message-ID: <44BE5914.3050200@ieee.org> Travis Oliphant wrote: > I made a small change to the array creation function so that if you > explicitly specify dtype=object, then the logic for determining the > shape of the array is changed. > > The new logic for constructing dtype=object arrays from Python sequences > is that the shape is determined by nested lists (and only lists). The > nesting must be consistent or else the elements of the last conforming > lists are assumed to be objects. > Update: Now lists and tuples are both allowed for determining nested lists (that was needed for record arrays). The shapes still have to match for a new dimension to be "detected" -Travis From oliphant at ee.byu.edu Wed Jul 19 17:25:41 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed, 19 Jul 2006 15:25:41 -0600 Subject: [Numpy-discussion] Performance of defmatrix.py:103(__array_finalize__) In-Reply-To: <2e1434c10607191416p7dc055e1ie2194133e4d9dd3d@mail.gmail.com> References: <2e1434c10607190824l322ae6d4p128dc9c35a637f16@mail.gmail.com> <2e1434c10607190836g282d7e70hcf34145b222ad462@mail.gmail.com> <44BE5663.1090405@ieee.org> <2e1434c10607191416p7dc055e1ie2194133e4d9dd3d@mail.gmail.com> Message-ID: <44BEA355.7030406@ee.byu.edu> Kevin Jacobs wrote: > On 7/19/06, *Travis Oliphant* > wrote: > > Kevin Jacobs > wrote: > > On 7/19/06, *Kevin Jacobs > > >>* > > >> > wrote: > > > > Is it expected that the following script would spend over > half of > > the total execution time in > defmatrix.py:103(__array_finalize__)? > > > > > > Confirmed that a similar overhead exists for arrays, though not as > > extreme. My application requires the ability to assign random > > elements, so any speed ups here will result be quite significant. > > > There is an optimization for this kind of assignment in current NumPy > SVN which "should" avoid the array creation over-head. > > > > There still seems to be a huge performance penalty with using a matrix > to set elements. Consider this code where X is initially a matrix: Are you now using SVN NumPy, or are we still looking at NumPy 0.9.8 numbers? -Travis From bioinformed at gmail.com Wed Jul 19 17:31:05 2006 From: bioinformed at gmail.com (Kevin Jacobs ) Date: Wed, 19 Jul 2006 17:31:05 -0400 Subject: [Numpy-discussion] Performance of defmatrix.py:103(__array_finalize__) In-Reply-To: <44BEA355.7030406@ee.byu.edu> References: <2e1434c10607190824l322ae6d4p128dc9c35a637f16@mail.gmail.com> <2e1434c10607190836g282d7e70hcf34145b222ad462@mail.gmail.com> <44BE5663.1090405@ieee.org> <2e1434c10607191416p7dc055e1ie2194133e4d9dd3d@mail.gmail.com> <44BEA355.7030406@ee.byu.edu> Message-ID: <2e1434c10607191431u1f01c851v6d21e7efe771db01@mail.gmail.com> On 7/19/06, Travis Oliphant wrote: > > Are you now using SVN NumPy, or are we still looking at NumPy 0.9.8numbers? Sorry -- SVN updated as of about two hours ago. Thanks, -Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: From webb.sprague at gmail.com Wed Jul 19 18:06:00 2006 From: webb.sprague at gmail.com (Webb Sprague) Date: Wed, 19 Jul 2006 15:06:00 -0700 Subject: [Numpy-discussion] Pretty printing an array: no ellipses? Message-ID: I am not sure where to look for this, sorry if it is RTFM or JPS ("just plain stupid"): Is there a way to set a default to print the entire array, rather than an ellipses version of it? If not, why doesn't pprint.pformat(numpy.random.normal(0,1,(100, 100)), width=1000) at least give me something off the screen? Thx! From xaz39 at 163.com Sun Jul 23 20:47:56 2006 From: xaz39 at 163.com (=?GB2312?B?IjfUwjI5LTMwyNUvyc+6oyI=?=) Date: Mon, 24 Jul 2006 08:47:56 +0800 Subject: [Numpy-discussion] =?GB2312?B?cmU61MvTw0lFvLzK9czhuN/Qp8LKvqvS5sn6svo=?= Message-ID: An HTML attachment was scrubbed... URL: From stephen.walton at csun.edu Thu Jul 20 00:44:08 2006 From: stephen.walton at csun.edu (Stephen Walton) Date: Wed, 19 Jul 2006 21:44:08 -0700 Subject: [Numpy-discussion] Atlas and dual core In-Reply-To: References: Message-ID: <44BF0A18.4090406@csun.edu> Keith Goodman wrote: >Is there much speed to be gained by compiling atlas for a dual core system? > > I imagine the answer is yes. However, Clint Whaley literally just today announced the first release of a beta version of ATLAS whose configuration utility actually supports Core Duo, although probably only under OSX. Check Clint's message at http://sourceforge.net/mailarchive/forum.php?thread_id=25989825&forum_id=426 By the way, everyone should also read the immediately previous message, posted a few weeks earlier by Clint. gcc 4.x gives significantly worse floating point performance than gcc 3.x on most x86 platforms. Something to be aware of if you're benchmarking. He's communicated about the problem to the gcc developers, but is, ah, less than confident they'll fix the problem. Steve From davidgrant at gmail.com Thu Jul 20 01:59:11 2006 From: davidgrant at gmail.com (David Grant) Date: Wed, 19 Jul 2006 22:59:11 -0700 Subject: [Numpy-discussion] Profiling line-by-line Message-ID: Is there any way to do line-by-line profiling in Python? The profiling results can tell me how much time is spent in all functions, but within a given function I can't get any idea of how much time was spent on each line. For example, in the example below, I can see that graphWidth.py is taking all the time, but there are many lines of code in graphWidth.py that aren't function calls, and I have no way of knowing which lines are the bottlenecks. I'm using hotspot currently, by the way. ncalls tottime percall cumtime percall filename:lineno(function) 1 0.215 0.215 0.221 0.221 graphWidth.py:6(graphWidth) 27 0.001 0.000 0.003 0.000 oldnumeric.py:472(all) 26 0.002 0.000 0.002 0.000 oldnumeric.py:410(sum) 26 0.001 0.000 0.002 0.000 oldnumeric.py:163(_wrapit) 26 0.001 0.000 0.001 0.000 oldnumeric.py:283(argmin) 26 0.000 0.000 0.000 0.000 numeric.py:111(asarray) 0 0.000 0.000 profile:0(profiler) Thanks, -- David Grant -------------- next part -------------- An HTML attachment was scrubbed... URL: From arnd.baecker at web.de Thu Jul 20 03:06:35 2006 From: arnd.baecker at web.de (Arnd Baecker) Date: Thu, 20 Jul 2006 09:06:35 +0200 (CEST) Subject: [Numpy-discussion] Profiling line-by-line In-Reply-To: References: Message-ID: On Wed, 19 Jul 2006, David Grant wrote: > Is there any way to do line-by-line profiling in Python? The profiling > results can tell me how much time is spent in all functions, but within a > given function I can't get any idea of how much time was spent on each line. > For example, in the example below, I can see that graphWidth.py is taking > all the time, but there are many lines of code in graphWidth.py that aren't > function calls, and I have no way of knowing which lines are the > bottlenecks. I'm using hotspot currently, by the way. > > ncalls tottime percall cumtime percall filename:lineno(function) > 1 0.215 0.215 0.221 0.221 graphWidth.py:6(graphWidth) > 27 0.001 0.000 0.003 0.000 oldnumeric.py:472(all) > 26 0.002 0.000 0.002 0.000 oldnumeric.py:410(sum) > 26 0.001 0.000 0.002 0.000 oldnumeric.py:163(_wrapit) > 26 0.001 0.000 0.001 0.000 oldnumeric.py:283(argmin) > 26 0.000 0.000 0.000 0.000 numeric.py:111(asarray) > 0 0.000 0.000 profile:0(profiler) > > Thanks, You might give hotshot2kcachegrind a try. See http://mail.enthought.com/pipermail/enthought-dev/2006-January/001075.html for more details. The screenshots http://www.physik.tu-dresden.de/~baecker/tmp/bench_traits/kcachegrind_screenshot.png http://kcachegrind.sourceforge.net/cgi-bin/show.cgi/KcacheGrindShot4 might give you an idea how things will look. More importantly note that profiling in connection with ufuncs seems problematic: See this thread (unfortunately split into several pieces, not sure if I got all of them): http://thread.gmane.org/gmane.comp.python.numeric.general/5309/focus=5309 http://thread.gmane.org/gmane.comp.python.numeric.general/5311/focus=5316 http://thread.gmane.org/gmane.comp.python.numeric.general/5337/focus=5337 (I always wanted to write this up for the wiki, but "real work" is interfering too strongly at the moment ;-). Good luck with profiling, Arnd From stefan at sun.ac.za Thu Jul 20 07:51:12 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu, 20 Jul 2006 13:51:12 +0200 Subject: [Numpy-discussion] Pretty printing an array: no ellipses? In-Reply-To: References: Message-ID: <20060720115112.GJ23629@mentat.za.net> On Wed, Jul 19, 2006 at 03:06:00PM -0700, Webb Sprague wrote: > I am not sure where to look for this, sorry if it is RTFM or JPS > ("just plain stupid"): > > Is there a way to set a default to print the entire array, rather than > an ellipses version of it? If not, why doesn't > pprint.pformat(numpy.random.normal(0,1,(100, 100)), width=1000) at > least give me something off the screen? Try N.set_printoptions(threshold=N.nan) St?fan From Ferenc.Pintye at eu.decoma.com Thu Jul 20 08:36:12 2006 From: Ferenc.Pintye at eu.decoma.com (Ferenc.Pintye at eu.decoma.com) Date: Thu, 20 Jul 2006 14:36:12 +0200 Subject: [Numpy-discussion] vector to tensor matrix speed up Message-ID: Hi users, i have some problem in Numpy with indexing speed for array to tensor matrix transport. With 200000 cycles it's 9sec ! (without sort(eigvals() - functions) Many thanks f. # stress values/matrix in array form for 200000 points out = zeros((200000,11),Float32) # #.....out = filling matrix ...etc... # # stress tensor matrix eig = zeros((3,3),Float32) # #output for eigvalues eigwert = array([0,0,0]) # for j in arange(0,200000): eig[0,0] = out[j,1] eig[1,1] = out[j,2] eig[2,2] = out[j,3] # eig[0,1] = out[j,4] eig[0,2] = out[j,6] eig[1,0] = out[j,4] eig[1,2] = out[j,5] eig[2,0] = out[j,6] eig[2,1] = out[j,5] # eigwert = sort(eigvals(eig)) out[j,7] = eigwert[2] out[j,8] = eigwert[1] out[j,9] = eigwert[0] out[j,10] = abs(eigwert[0]-eigwert[2]) # From tom.denniston at alum.dartmouth.org Thu Jul 20 09:46:05 2006 From: tom.denniston at alum.dartmouth.org (Tom Denniston) Date: Thu, 20 Jul 2006 08:46:05 -0500 Subject: [Numpy-discussion] numpy.var and numpy.std Message-ID: Are these functions supposed to compute variance and standard deviation? They don't have docstrings and I can't find anything about them in Travis' manual. From david.huard at gmail.com Thu Jul 20 09:58:12 2006 From: david.huard at gmail.com (David Huard) Date: Thu, 20 Jul 2006 09:58:12 -0400 Subject: [Numpy-discussion] numpy.var and numpy.std In-Reply-To: References: Message-ID: <91cf711d0607200658w6d4f7fd1s51173eeeca63a9b7@mail.gmail.com> Yes, In SVN, std has a docstring and I submitted one for var in ticket #174. David 2006/7/20, Tom Denniston : > > Are these functions supposed to compute variance and standard > deviation? They don't have docstrings and I can't find anything about > them in Travis' manual. > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share > your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tom.denniston at alum.dartmouth.org Thu Jul 20 12:35:29 2006 From: tom.denniston at alum.dartmouth.org (Tom Denniston) Date: Thu, 20 Jul 2006 11:35:29 -0500 Subject: [Numpy-discussion] numpy.var and numpy.std In-Reply-To: <91cf711d0607200658w6d4f7fd1s51173eeeca63a9b7@mail.gmail.com> References: <91cf711d0607200658w6d4f7fd1s51173eeeca63a9b7@mail.gmail.com> Message-ID: Sounds great. Thanks David. --Tom On 7/20/06, David Huard wrote: > Yes, > > In SVN, std has a docstring and I submitted one for var in ticket #174. > > David > > 2006/7/20, Tom Denniston >: > > > Are these functions supposed to compute variance and standard > deviation? They don't have docstrings and I can't find anything about > them in Travis' manual. > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From oliphant.travis at ieee.org Thu Jul 20 14:00:24 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 20 Jul 2006 12:00:24 -0600 Subject: [Numpy-discussion] numpy.var and numpy.std In-Reply-To: References: Message-ID: <44BFC4B8.70306@ieee.org> Tom Denniston wrote: > Are these functions supposed to compute variance and standard > deviation? They don't have docstrings and I can't find anything about > them in Travis' manual. > These are in the manual but only as a general reference to the fact that array methods also have functional counter-parts. In other words, a.meth() --> meth(a) The latter is more general because a does not have to be an array already. -Travis From oliphant.travis at ieee.org Thu Jul 20 14:09:29 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 20 Jul 2006 12:09:29 -0600 Subject: [Numpy-discussion] numpy.var and numpy.std In-Reply-To: References: Message-ID: <44BFC6D9.8060500@ieee.org> Tom Denniston wrote: > Are these functions supposed to compute variance and standard > deviation? They don't have docstrings and I can't find anything about > them in Travis' manual. > > I just wanted to advertise the fact that there are two utility functions in NumPy that can come in handy for people wanting to add docstrings to built-in types. These utility functions by-pass the need to fiddle with doc-strings in C-code (which can be a pain) and re-compile every time a doc-string is changed. These are meant to be run once.... They are: add_docstring(obj, string) Add a string as the docstring of obj. A wide-variety of builtin types are understood. If it doesn't know how to add it, you get a TypeError. If the object already has a doc-string you get a RuntimeError. add_newdoc(place, obj, doc) This is a wrapper around docstring to make it easier to add docs to many attributes of an object which is stored in place. It never fails. place and obj are both strings doc is a string, list, or tuple place is the module name obj is the object in that module If doc is a string add it to place.obj If doc is a tuple it is (attr, doc) ---> the first element is the attribute of obj and the second is the docstring. If doc is a list it is [(attr1, doc1), (attr2, doc2), ...] --> the elements of the list are all tuples to add docstrings to many attributes of an object. numpy/add_newdocs.py is a place where docstrings can be added. -Travis From tim.hochberg at cox.net Thu Jul 20 15:47:59 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 20 Jul 2006 12:47:59 -0700 Subject: [Numpy-discussion] vector to tensor matrix speed up In-Reply-To: References: Message-ID: <44BFDDEF.4060500@cox.net> Ferenc.Pintye at eu.decoma.com wrote: > > > Hi users, > > i have some problem in Numpy with indexing speed for array to tensor > matrix transport. > With 200000 cycles it's 9sec ! (without sort(eigvals() - functions) > > Many thanks > f. > The big problem you have here is that you are operating on your matrices one element at a time. Overhead is going to kill you doing this. The key thing is to operate on your data in chunks -- I'll show one way below that speeds thing up by a factor of 40 or so *neglecting sort(eigvals())*. Unfortunately, sort(eigvals()) is still likely to kill you since there's not really a good way to chunk that up. (I'll have more to say about that in a more general sense in another email). Here's the two versions I compared, perhaps this will help you out: from numpy import * import timeit out = zeros((200000,11),float) def f0(out): for j in arange(0,200000): pass def f1(out): # stress values/matrix in array form for 200000 points # #.....out = filling matrix ...etc... # # stress tensor matrix eig = zeros((3,3),float) # #output for eigvalues eigwert = array([0,0,0]) # for j in arange(0,200000): eig[0,0] = out[j,1] eig[1,1] = out[j,2] eig[2,2] = out[j,3] # eig[0,1] = out[j,4] eig[0,2] = out[j,6] eig[1,0] = out[j,4] eig[1,2] = out[j,5] eig[2,0] = out[j,6] eig[2,1] = out[j,5] # #~ eigwert = sort(eigvals(eig)) out[j,7] = eigwert[2] out[j,8] = eigwert[1] out[j,9] = eigwert[0] out[j,10] = abs(eigwert[0]-eigwert[2]) # def f2(out): # stress values/matrix in array form for 200000 points # #.....out = filling matrix ...etc... # # stress tensor matrix eig = zeros((100,3,3),float) # #output for eigvalues eigwert = zeros([100,3], dtype=float) # local_abs = abs rangen = range(0,200000,100) for j in rangen: eig[:,0,0] = out[j:j+100,1] eig[:,1,1] = out[j:j+100,2] eig[:,2,2] = out[j:j+100,3] # eig[:,1,0] = eig[:,0,1] = out[j:j+100,4] eig[:,2,0] = eig[:,0,2] = out[j:j+100,6] eig[:,2,1] = eig[:,1,2] = out[j:j+100,5] # #~ eigwert = sort(eigvals(eig)) for j in rangen: out[j:j+100,7:10] = eigwert # Changed order of out here out[j:j+100,10] = abs(eigwert[:,0]-eigwert[:,2]) if __name__ == '__main__': print timeit.Timer("f0(out)", "from scratch import f0, out").timeit(1) print timeit.Timer("f1(out)", "from scratch import f1, out").timeit(1) print timeit.Timer("f2(out)", "from scratch import f2, out").timeit(1) > > # stress values/matrix in array form for 200000 points > out = zeros((200000,11),Float32) > # > #.....out = filling matrix ...etc... > # > # stress tensor matrix > eig = zeros((3,3),Float32) > # > #output for eigvalues > eigwert = array([0,0,0]) > # > for j in arange(0,200000): > eig[0,0] = out[j,1] > eig[1,1] = out[j,2] > eig[2,2] = out[j,3] > # > eig[0,1] = out[j,4] > eig[0,2] = out[j,6] > eig[1,0] = out[j,4] > eig[1,2] = out[j,5] > eig[2,0] = out[j,6] > eig[2,1] = out[j,5] > # > eigwert = sort(eigvals(eig)) > out[j,7] = eigwert[2] > out[j,8] = eigwert[1] > out[j,9] = eigwert[0] > out[j,10] = abs(eigwert[0]-eigwert[2]) > # > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From nvf at MIT.EDU Thu Jul 20 17:34:16 2006 From: nvf at MIT.EDU (Nick Fotopoulos) Date: Thu, 20 Jul 2006 17:34:16 -0400 Subject: [Numpy-discussion] Beta release on Thursday Message-ID: <42D94292-EC82-4D86-A276-473B5369BCC4@mit.edu> On July 17, 2006, at 9:01 PM, Travis Oliphant wrote: > I'd like to make release 1.0beta on Thursday. Please submit bug- reports > and fixes before then. > > -Travis Is it possible to incorporate v7 mat-file support before the new- feature freeze? I have the fixes posted to ticket #14. Several people have contacted me about the ticket. Thanks, Nick -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2802 bytes Desc: not available URL: From oliphant.travis at ieee.org Thu Jul 20 17:39:45 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 20 Jul 2006 15:39:45 -0600 Subject: [Numpy-discussion] NumPy 1.0b1 at the door... Message-ID: <44BFF821.5040505@ieee.org> I'm getting ready to tag the trunk as the 1.0b1 release (will be done in 2 hours unless I hear from somebody). I've taken care of most of the relevant open tickets. Some of the build issues are still there but they can be fixed during the beta release period which I expect to be several months. -Travis From robert.kern at gmail.com Thu Jul 20 17:51:31 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 20 Jul 2006 16:51:31 -0500 Subject: [Numpy-discussion] NumPy 1.0b1 at the door... In-Reply-To: <44BFF821.5040505@ieee.org> References: <44BFF821.5040505@ieee.org> Message-ID: Travis Oliphant wrote: > I'm getting ready to tag the trunk as the 1.0b1 release (will be done in > 2 hours unless I hear from somebody). I've taken care of most of the > relevant open tickets. Some of the build issues are still there but > they can be fixed during the beta release period which I expect to be > several months. I'm testing the change from using pkgload in __init__ to explicit imports now. If you can hold off until I finish with that, I'd appreciate it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant.travis at ieee.org Thu Jul 20 18:00:34 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 20 Jul 2006 16:00:34 -0600 Subject: [Numpy-discussion] Beta release on Thursday In-Reply-To: <42D94292-EC82-4D86-A276-473B5369BCC4@mit.edu> References: <42D94292-EC82-4D86-A276-473B5369BCC4@mit.edu> Message-ID: <44BFFD02.3060001@ieee.org> Nick Fotopoulos wrote: > On July 17, 2006, at 9:01 PM, Travis Oliphant wrote: > > I'd like to make release 1.0beta on Thursday. Please submit > bug-reports > > and fixes before then. > > > > -Travis > > Is it possible to incorporate v7 mat-file support before the > new-feature freeze? That is in SciPy. I'm talking about the NumPy 1.0beta release. But, I would like to get the v7 mat-file support into SciPy soon. Perhaps you are arguing about putting in mat-file support into NumPy. While I appreciate the idea, I like it sitting on the SciPy side of the fence > I have the fixes posted to ticket #14. Several people have > contacted me about the ticket. Thanks for the ticket. I will get around to it. I've been swamped with NumPy. I'm really relying on the other contributors for SciPy fixes. -Travis From tim.hochberg at cox.net Thu Jul 20 18:07:24 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 20 Jul 2006 15:07:24 -0700 Subject: [Numpy-discussion] Stacking matrices Message-ID: <44BFFE9C.5090304@cox.net> The recent message by Ferenc.Pintye (how does one pronounce that BTW) reminded me of something I've been meaning to discuss: I think we can do a better job dealing with stacked matrices. By stacked matrices I mean 3 (or more) dimensional arrays where the last two dimensions are considered to be matrices. Let us suppose that one wants to find the eigenvalues of 200,000 3x3 matrices as Ferenc does. At present you have to call linalg.eigvals 200,000 time; the overhead is going to kill you. I've run into this problem before myself in the past (and kludged my way around it), so the problem is not limited to Ferenc. I see no reason that this could not be fixed for the linalg functions. The details would vary from function to function, but for eigvals, for example, when passed an NxMxM array, it would return an NxM array of eigenvalues. One can actually get pretty far just modifying the python functions in linalg.py, or at least one could in numarray and I'm guessing that numpy is similar. That is because there is a fair amount of overhead in setting up the function calls and this can all be done just once. I have code lying around that does this for solve, inverse and determinant that works with numarray that would probably not be hard to adapt to numpy if we were interested in making this change of interface to linalg. Another place that this same effect has bitten me in the past is with dot. Dot does work on higher dimensional arrays, but it attempts to do a tensor product of sorts. While this sounds nifty, in practice I've found it less than useful. However, because dot computes both scalar and matrix products it's not clear that a set of rules that was more or less backwards compatible and that extended to higher dimensions in a more useful way than at present could be crafted. Probably something with different behavior, perhaps that always performed matrix products on the last two axes, could be added with a name other than dot. This avoids backwards compatibility issues as well. None of this stuff is worth holding up the the release 1.0 over, but I though I would mention it while it was on my mind. -tim From nvf at MIT.EDU Thu Jul 20 18:14:11 2006 From: nvf at MIT.EDU (Nick Fotopoulos) Date: Thu, 20 Jul 2006 18:14:11 -0400 Subject: [Numpy-discussion] Beta release on Thursday In-Reply-To: <44BFFD02.3060001@ieee.org> References: <42D94292-EC82-4D86-A276-473B5369BCC4@mit.edu> <44BFFD02.3060001@ieee.org> Message-ID: <0A246626-A462-49F3-9692-674E47795D0E@mit.edu> On Jul 20, 2006, at 6:00 PM, Travis Oliphant wrote: >> Is it possible to incorporate v7 mat-file support before the new- >> feature freeze? > That is in SciPy. I'm talking about the NumPy 1.0beta release. > But, I would like to get the v7 mat-file support into SciPy soon. > Perhaps you are arguing about putting in mat-file support into > NumPy. While I appreciate the idea, I like it sitting on the > SciPy side of the fence > >> I have the fixes posted to ticket #14. Several people have >> contacted me about the ticket. > Thanks for the ticket. I will get around to it. I've been swamped > with NumPy. I'm really relying on the other contributors for SciPy > fixes. My apologies. I often forget that the release cycles are not phase- locked. The same characters play on both stages, so if I'm not paying sufficient attention, I mix things up. I'll go ask for a commit in SciPy land. Take care, Nick -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2802 bytes Desc: not available URL: From oliphant.travis at ieee.org Thu Jul 20 18:31:56 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 20 Jul 2006 16:31:56 -0600 Subject: [Numpy-discussion] NumPy 1.0b1 at the door... In-Reply-To: <20060720221506.GC1046@mentat.za.net> References: <44BFF821.5040505@ieee.org> <20060720221506.GC1046@mentat.za.net> Message-ID: <44C0045C.60800@ieee.org> Stefan van der Walt wrote: > Hi Travis > > Albert and I are busy doing some final beta testing. Quick question: > can Fortran-order arrays be contiguous, ever? > > In [62]: N.empty((3,3),order='F').flags['CONTIGUOUS'] > Out[62]: False > Thank you very much for the testing. You two have been amazingly helpful. No. CONTIGUOUS means "C-CONTIGUOUS" Now, 0-d and 1-d arrays can be both Fortran-contiguous *and* C-contiguous which is why "FNC" exists to test against that case. -Travis From tim.hochberg at cox.net Thu Jul 20 19:35:35 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 20 Jul 2006 16:35:35 -0700 Subject: [Numpy-discussion] fast way of doing "cross-multiplications" ? In-Reply-To: References: <44BCEEF9.1030200@obs.univ-lyon1.fr> <502862AF-514C-41CA-804E-420CE76FEE37@stsci.edu> <44BCFFE4.6070502@obs.univ-lyon1.fr> <44BD08A3.1030600@cox.net> Message-ID: <44C01347.8080701@cox.net> Fernando Perez wrote: > On 7/18/06, Tim Hochberg wrote: >> Eric Emsellem wrote: >> > thanks for the tips. (indeed your "add.reduce" is correct: I just >> wrote >> > this down too quickly, in the script I have a "sum" included). >> > >> > And yes you are right for the memory issue, so I may just keep the >> loop >> > in and try to make it work on a fast PC...(or use parallel processes) >> > >> > (is "sum" different than "add.reduce"?) >> > >> > thanks again to both Bill Baxter and Perry Greenfield for their fast >> > (and helpful!) answers. >> > >> I just wanted to add that there are faster, but considerably complicated >> ways to attack this class of problems. The one I've looked at in the >> past was the fast multipole method and I believe there are others. I'm >> not sure whether these can be implemented efficiently in numpy, but you >> may want to take a look into this kind of more sophisticated/complicated >> approach if brute forcing the calculation doesn't work. > > Indeed, FMM is the best known method that can turn this O(n^2) problem > into O(n*log(n)). As Tim indicates, there is no easy way out of this > one. Incidentally, there is a talk about FMM on the scipy'06 > schedule, in case you are going to attend. > > An alternative approach to FMM (conceptually similar in some ways) is > described here: > > http://amath.colorado.edu/faculty/fperez/talks/0604_sanum_mwadap.pdf > > Unfortunately this isn't exactly a half-hour optimization effort, as > the required machinery is pretty heavy duty. And yes, this has all > been done in python and runs on current numpy/scipy (though it has > Fortran, C and C++ sprinkled as needed). I'm interested in hearing in what situations people end up dropping into C/C++ or Fortran. It would be interesting to see if general solutions could be found for making some of them fast enough at the Python level. Here are some situations that I've run into where numpy has trouble computing results efficiently. 1. The value at a given point is the computed one of two (or more) ways depending on some condition. The subcomputations are expensive, so one does not want to compute them both and then chose a result using 'where'. I think I have a decent way to attack this at the Python level using argsort: basically, sort all of the operands, find the boundaries between the regions where different computations are done, compute the expensive subcomputations on the appropriate blocks and then unsort the data. It would be interesting to see if this could be automated in a clean way, perhaps within numexpr. 2. Generalized reduce. We want to reduce an array using an arbitrary function f(x, y) that is not a standard ufunc. In the particular case I've run into, x and y would be vectors or perhaps record arrays, and the functions would couple the various records when producing the output. I don't see how this case can be handled efficiently at the Python level, short of using something like Weave to produce a custom ufunc and calling reduce on it. I don't see how something like numexpr could handle it, unfortunately, since it gains it's speed by operating on blocks of data. I'm sure there are other cases that I've run into, but that's all I'm coming up with right now. -tim From davidgrant at gmail.com Thu Jul 20 20:47:03 2006 From: davidgrant at gmail.com (David Grant) Date: Thu, 20 Jul 2006 17:47:03 -0700 Subject: [Numpy-discussion] [SciPy-user] Would like to simplify my 3 where statements In-Reply-To: <16761e100607201717t1c610949xd24f579ef6f69013@mail.gmail.com> References: <16761e100607201717t1c610949xd24f579ef6f69013@mail.gmail.com> Message-ID: On 7/20/06, Michael Sorich wrote: > Can you give an specific example of how this would work? The codes > really is ugly and it is not clear to me what exactly it does. ok here's a quick example: import numpy n=5 i=3 j=4 A=numpy.random.randint(0,2,(n,n)) #make random graph A=A-diag(diag(A)) #remove diagonal A=triu(A)+transpose(triu(A)) #make symmetric Now say A is the adjacency matrix for a graph and I want to know which nodes are neighbours of A, but I want to exclude node i from consideration. So if A is: array([[0, 1, 0, 1, 0], [1, 0, 1, 0, 1], [1, 0, 0, 0, 0], [0, 0, 1, 0, 1], [1, 1, 0, 1, 0]]) the neighbours are array([1]) for i=3, j=4. One way to do it is to do: ans=where(A[:,j]==1)[0] if A[i,j] == 1: ans -= 1 which is probably faster than my method. I don't really care so much about speed though. I'm just trying to figure out different ways of doing this using numpy. Dave From wbaxter at gmail.com Thu Jul 20 21:51:07 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 21 Jul 2006 10:51:07 +0900 Subject: [Numpy-discussion] r_, c_, hstack, and vstack with 1-d arrays In-Reply-To: References: <44BE2E5A.80404@gmx.net> Message-ID: I looked into the various concatenation methods a bit more to better understand what's going on under the hood. Here's essentially what these different methods do: vstack(tup): concatenate( map(atleast_2d,tup), axis=0 ) hstack(tup): concatenate( map(atleast_1d,tup),axis=1 ) column_stack(tup): arrays = map( transpose,map(atleast_2d,tup) ) concatenate(arrays,1) (note that column_stack transposes *everything* not just 1-d inputs, so it doesn't do quite what I thought it did, i.e. only transposing 1-d inputs) The above 3 are pretty much exactly the code used by numpy. That's all there is to those 3 functions. For r_ and c_ I'm summarizing, but effectively they seem to be doing something like: r_[args]: concatenate( map(atleast_1d,args),axis=0 ) c_[args]: concatenate( map(atleast_1d,args),axis=1 ) c_ behaves almost exactly like hstack -- with the addition of range literals being allowed. r_ is most like vstack, but a little different since it effectively uses atleast_1d, instead of atleast_2d. So you have >>> numpy.vstack((1,2,3,4)) array([[1], [2], [3], [4]]) but >>> numpy.r_[1,2,3,4] array([1, 2, 3, 4]) However for cases like that with just 0-d or 1-d inputs, c_ behaves identically to r_, so if you wanted to get a 1-d output you could have just used c_. So I take back what I said about wishing c_ were like column_stack. Column stack is weird. Instead, I think the right thing to do would be to make r_ behave more like vstack. I think that would make things more consistent, and make for less for the user to remember. After making that change, to make things even more consistent, it might make sense to rename r_ and c_ to v_ and h_ instead. Then it's easy to remember 'v_' is like 'vstack', 'h_' is like hstack. Furthermore, I propose that column_stack should only transpose its 1d inputs. "Stack colums" defnitely doesn't imply to me that something that already has columns will be transposed. Currently it is documented to only work on 1d inputs, so hopefully that's a change that wouldn't affect too many people. The function in numpy/lib/shape_base.py could be replaced with this: def column_stack(tup): def transpose_1d(array): if array.ndim<2: return _nx.transpose(atleast_2d(array)) else: return array arrays = map(transpose_1d,map(atleast_1d,tup)) return _nx.concatenate(arrays,1) If r_, and c_ get renamed to v_, h_, then c_ could be re-introduced with behavior similar to column_stack. Finally, I noticed that the atleast_nd methods return arrays regardless of input type. At a minimum, atleast_1d and atleast_2d on matrices should return matrices. I'm not sure about atleast_3d, since matrices can't be 3d. (But my opinon is that the matrix type should be allowed to be 3d). Anyway, since these methods are used by the *stack methods, those also do not currently preserve the matrix type (in SVN numpy). SUMMARY: * make r_ behave like "vstack plus range literals" * make column_stack only transpose its 1d inputs. * rename r_,c_ to v_,h_ (or something else) to make their connection with vstack and hstack clearer. Maybe vs_ and hs_ would be better? * make a new vertsion of 'c_' that acts like column_stack so that theres a nice parallel v_<=>vstack, h_<=>hstack, c_<=>column_stack * make atleast_*d methods preserve the input type whenever possible Thoughts? --bb From mail at stevesimmons.com Thu Jul 20 22:05:48 2006 From: mail at stevesimmons.com (Stephen Simmons) Date: Fri, 21 Jul 2006 12:05:48 +1000 Subject: [Numpy-discussion] Error when using uints in bincount() In-Reply-To: <44BCCD22.7060607@stevesimmons.com> References: <44BCCD22.7060607@stevesimmons.com> Message-ID: <44C0367C.5070805@stevesimmons.com> Hi, The function bincount() counts the number of each value found in the input array: In [15]: numpy.bincount( array([1,3,3,3,4],dtype=int32) ) Out[15]: array([0, 1, 0, 3, 1]) According to the documentation, the input array must be non-negative integers. However an exception occurs when the input data type are unsigned integers (which is an explicit guarantee of this non-negativity condition): In [157]: numpy.bincount( array([1,3,3,3,4],dtype=uint32) ) TypeError: array cannot be safely cast to required type This seems to be a bug. Cheers Stephen P.S. I'm not familiar enough with the numpy source to track down where this typechecking is done. But I did find a trivial typo in an error msg in function arr_bincount() in numpy/lib/src/_compiled_base.c. The assert message here has lost its initial 'F': Py_Assert(numbers[mni] >= 0, "irst argument of bincount must be non-negative"); From wbaxter at gmail.com Thu Jul 20 23:12:26 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 21 Jul 2006 12:12:26 +0900 Subject: [Numpy-discussion] appending/inserting/deleting axes of arrays Message-ID: Howdy, Is there any nicer syntax for the following operations on arrays? Append a row: a = vstack((a,row)) Append a column: a = hstack((a,col)) Append a row of zeros: a = vstack((a,zeros((1,a.shape[1])))) Append a col of zeros: a = hstack((a,zeros((a.shape[0],1)))) Insert a row before row j a = vstack(( a[:j], row, a[j:] )) Insert a column before col j a = hstack(( a[:j], col, a[j:] )) Insert a row of zeros before row j a = vstack(( a[:j], zeros((1,a.shape[1])), a[j:] )) Insert a column of zeros before col j a = hstack(( a[:j], zeros((a.shape[0],1)), a[j:] )) Delete row j: a = vstack(( a[:j], a[j+1:] )) Delete col j: a = hstack(( a[:j], a[j+1:] )) ...And in more general the same types of operations for N-d arrays. I find myself using python lists of lists a lot just for the easy readability of a.append(row) compared to a = vstack((a,row)). I guess, though, if I'm building an array by appending a row at a time, then maybe it *is* better to use a python list o list for that? Then each 'append' only copies pointers of the existing rows rather than copying the data in each row. Is that correct? Also do python lists over-allocate in order to avoid having to re-allocate and copy every time there's an append()? Numpy arrays don't over-allocate I assume. Thanks, --bb From mail at stevesimmons.com Thu Jul 20 23:44:57 2006 From: mail at stevesimmons.com (Stephen Simmons) Date: Fri, 21 Jul 2006 13:44:57 +1000 Subject: [Numpy-discussion] Possible modification to bincount() In-Reply-To: <44C0367C.5070805@stevesimmons.com> References: <44BCCD22.7060607@stevesimmons.com> <44C0367C.5070805@stevesimmons.com> Message-ID: <44C04DB9.9000909@stevesimmons.com> While playing a little more with bincount(), one modification would be handy: Allow negative integers in the bin list, but skip them when counting bins My specific use case is calculating subtotals on columns of large datasets (1m rows x 30 cols), where some rows need to be excluded. The groupings are expensive to compute, and sometimes will involve ~99% of the rows (eliminate only outliers/errors), and other times only ~5% of the rows (focus in on a subset). I'd like to calculate subtotals like this using bincount(), without having to copy the large datasets just to eliminate the unwanted rows: # Assign each row to a group numbered from 0..G, except for -1 for rows to exclude row_groups = expensive_function(data) # Count number in each group, excluding those with grp==-1 grp_counts = bincount(list=row_groups) # Use bincount() to form subtotals by column, excluding those with grp==-1 subtotals = column_stack([ bincount(list=row_groups, weights=data[:,i]) for i in range(G+1) ]) Is there any appetite to make such a change to bincount()? This would require two simple changes to bincount() in _compiled_base.c and an update to the docstring. Here is the diff file with enough context to show the entire arr_bincount() function: *** orig_compiled_base.c 2006-07-21 13:14:21.250000000 +1000 --- _compiled_base.c 2006-07-21 13:34:41.718750000 +1000 *************** *** 70,143 **** intp j ; for ( j = 1 ; j < len; j ++ ) if ( i [j] < min ) {min = i [j] ; mn = j ;} return mn; } static PyObject * arr_bincount(PyObject *self, PyObject *args, PyObject *kwds) { /* histogram accepts one or two arguments. The first is an array ! * of non-negative integers and the second, if present, is an * array of weights, which must be promotable to double. * Call these arguments list and weight. Both must be one- * dimensional. len (weight) == len(list) * If weight is not present: ! * histogram (list) [i] is the number of occurrences of i in list. * If weight is present: * histogram (list, weight) [i] is the sum of all weight [j] ! * where list [j] == i. */ /* self is not used */ PyArray_Descr *type; PyObject *list = NULL, *weight=Py_None ; PyObject *lst=NULL, *ans=NULL, *wts=NULL; ! intp *numbers, *ians, len , mxi, mni, ans_size; int i; double *weights , *dans; static char *kwlist[] = {"list", "weights", NULL}; Py_Try(PyArg_ParseTupleAndKeywords(args, kwds, "O|O", kwlist, &list, &weight)); Py_Try(lst = PyArray_ContiguousFromAny(list, PyArray_INTP, 1, 1)); len = PyArray_SIZE(lst); numbers = (intp *) PyArray_DATA(lst); mxi = mxx (numbers, len) ; - mni = mnx (numbers, len) ; - Py_Assert(numbers[mni] >= 0, - "irst argument of bincount must be non-negative"); ans_size = numbers [mxi] + 1 ; type = PyArray_DescrFromType(PyArray_INTP); if (weight == Py_None) { Py_Try(ans = PyArray_Zeros(1, &ans_size, type, 0)); ians = (intp *)(PyArray_DATA(ans)); for (i = 0 ; i < len ; i++) ! ians [numbers [i]] += 1 ; Py_DECREF(lst); } else { Py_Try(wts = PyArray_ContiguousFromAny(weight, PyArray_DOUBLE, 1, 1)); weights = (double *)PyArray_DATA (wts); Py_Assert(PyArray_SIZE(wts) == len, "bincount: length of weights " \ "does not match that of list"); type = PyArray_DescrFromType(PyArray_DOUBLE); Py_Try(ans = PyArray_Zeros(1, &ans_size, type, 0)); dans = (double *)PyArray_DATA (ans); for (i = 0 ; i < len ; i++) { ! dans[numbers[i]] += weights[i]; } Py_DECREF(lst); Py_DECREF(wts); } return ans; fail: Py_XDECREF(lst); Py_XDECREF(wts); Py_XDECREF(ans); return NULL; } --- 70,145 ---- intp j ; for ( j = 1 ; j < len; j ++ ) if ( i [j] < min ) {min = i [j] ; mn = j ;} return mn; } static PyObject * arr_bincount(PyObject *self, PyObject *args, PyObject *kwds) { /* histogram accepts one or two arguments. The first is an array ! * of integers and the second, if present, is an * array of weights, which must be promotable to double. * Call these arguments list and weight. Both must be one- * dimensional. len (weight) == len(list) * If weight is not present: ! * histogram (list) [i] is the number of occurrences of i in list ! * for i>=0. Negative i values are ignored. * If weight is present: * histogram (list, weight) [i] is the sum of all weight [j] ! * where list [j] == i and i>=0. */ /* self is not used */ PyArray_Descr *type; PyObject *list = NULL, *weight=Py_None ; PyObject *lst=NULL, *ans=NULL, *wts=NULL; ! intp *numbers, *ians, len , mxi, ans_size; int i; double *weights , *dans; static char *kwlist[] = {"list", "weights", NULL}; Py_Try(PyArg_ParseTupleAndKeywords(args, kwds, "O|O", kwlist, &list, &weight)); Py_Try(lst = PyArray_ContiguousFromAny(list, PyArray_INTP, 1, 1)); len = PyArray_SIZE(lst); numbers = (intp *) PyArray_DATA(lst); mxi = mxx (numbers, len) ; ans_size = numbers [mxi] + 1 ; type = PyArray_DescrFromType(PyArray_INTP); if (weight == Py_None) { Py_Try(ans = PyArray_Zeros(1, &ans_size, type, 0)); ians = (intp *)(PyArray_DATA(ans)); for (i = 0 ; i < len ; i++) ! if (numbers[i]>=0) { ! ians[numbers [i]] += 1 ; ! } Py_DECREF(lst); } else { Py_Try(wts = PyArray_ContiguousFromAny(weight, PyArray_DOUBLE, 1, 1)); weights = (double *)PyArray_DATA (wts); Py_Assert(PyArray_SIZE(wts) == len, "bincount: length of weights " \ "does not match that of list"); type = PyArray_DescrFromType(PyArray_DOUBLE); Py_Try(ans = PyArray_Zeros(1, &ans_size, type, 0)); dans = (double *)PyArray_DATA (ans); for (i = 0 ; i < len ; i++) { ! if (numbers[i]>=0) { ! dans[numbers[i]] += weights[i]; ! } } Py_DECREF(lst); Py_DECREF(wts); } return ans; fail: Py_XDECREF(lst); Py_XDECREF(wts); Py_XDECREF(ans); return NULL; } Cheers Stephen From pebarrett at gmail.com Fri Jul 21 00:01:46 2006 From: pebarrett at gmail.com (Paul Barrett) Date: Fri, 21 Jul 2006 00:01:46 -0400 Subject: [Numpy-discussion] Problem converting a numarray extension to numpy Message-ID: <40e64fa20607202101o75ca115ate4ed577fd6d98ac3@mail.gmail.com> I'm having a problem converting a C extension module that was originally written for numarray to use numpy. I using swig to create a wrapper flle for the C code. I have added the numpy.get_numarray_include() method to my setup.py file and have changed the numarray/libnumarray.h to use numpy/libnumarray.h. The extension appears to compile fine (with the exception of some warning messages). However, when I import the module, I get a segfault. Do I need to add anything else to the share library's initialization step other than import_libnumarray()? This is using Python 2.4.3 and gcc 4.1.1 on FC5. -- Paul From haase at msg.ucsf.edu Fri Jul 21 00:25:43 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu, 20 Jul 2006 21:25:43 -0700 Subject: [Numpy-discussion] [NumPy] #188: dtype should have "nice looking" str representation In-Reply-To: <076.85e10186a436d81ebef7e51d7e625bdb@scipy.net> References: <067.565cc454ae510236945afe3a0479b238@scipy.net> <076.85e10186a436d81ebef7e51d7e625bdb@scipy.net> Message-ID: <44C05747.2090709@msg.ucsf.edu> NumPy wrote: > #188: dtype should have "nice looking" str representation > -------------------------+-------------------------------------------------- > Reporter: sebhaase | Owner: oliphant > Type: enhancement | Status: closed > Priority: normal | Milestone: 1.0 Release > Component: numpy.core | Version: > Severity: normal | Resolution: wontfix > Keywords: | > -------------------------+-------------------------------------------------- > Changes (by oliphant): > > * status: new => closed > * resolution: => wontfix > > Comment: > > I'm not sure what the best thing to display here actually is. The current > string is very informative. Dropping the byte-ordering character is a > bad-idea. > Just yesterday I showed the new numpy a colleague of mine and he indeed read " I've created the 1.0b1 release tag in SVN and will be uploading files shortly to Sourceforge. I've also created a 1.0 release branch called ver1.0 The trunk is now version 1.1 of NumPy and should be used for new-development only. I don't expect 1.1 to come out for at least a year. Bug-fixes and small changes can happen on the 1.0 branch. These will be merged periodically to 1.1 or vice-versa. But, the 1.0 branch will be used for releases for the next year. AFAIK, this is similar to Python's release plan. I'm also going to be out of town for a few days and may not be able to check my email so you can ask questions, but I may not answer them for several days :-) Thanks to all you who helped with this release with bug reports and patches: Robert Kern David Cooke Pearu Peterson Alexander Belopolsky (Sasha) Albert Strasheim Stefan van der Walt Tim Hochberg Christopher Hanley Perry Greenfield Todd Miller David Huard Nils Wagner Thank you... I hope you all have a great weekend :-) Let's continue to make the beta release period productive by improving documentation and getting code-coverage tests and tracking down any bugs. Best regards, -Travis From svetosch at gmx.net Fri Jul 21 05:00:18 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Fri, 21 Jul 2006 11:00:18 +0200 Subject: [Numpy-discussion] r_, c_, hstack, and vstack with 1-d arrays In-Reply-To: References: <44BE2E5A.80404@gmx.net> Message-ID: <44C097A2.5020900@gmx.net> Bill Baxter schrieb: > Finally, I noticed that the atleast_nd methods return arrays > regardless of input type. At a minimum, atleast_1d and atleast_2d on > matrices should return matrices. I'm not sure about atleast_3d, since > matrices can't be 3d. (But my opinon is that the matrix type should > be allowed to be 3d). Anyway, since these methods are used by the > *stack methods, those also do not currently preserve the matrix type > (in SVN numpy). > Are you sure? I reported that issue with *stack and I remember it was fixed. > SUMMARY: > * make r_ behave like "vstack plus range literals" > * make column_stack only transpose its 1d inputs. > * rename r_,c_ to v_,h_ (or something else) to make their connection > with vstack and hstack clearer. Maybe vs_ and hs_ would be better? > * make a new vertsion of 'c_' that acts like column_stack so that > theres a nice parallel v_<=>vstack, h_<=>hstack, c_<=>column_stack > * make atleast_*d methods preserve the input type whenever possible > One problem with all that renaming is the other (maybe more important) function of r_: build (array or) matrix ranges quickly. So I'm a bit against the renaming I guess. Cleaning up the irritations with the *stacks seems useful though. (Although I have to confess I haven't read your last mail very thoroughly.) -Sven From Ferenc.Pintye at eu.decoma.com Fri Jul 21 07:35:52 2006 From: Ferenc.Pintye at eu.decoma.com (Ferenc.Pintye at eu.decoma.com) Date: Fri, 21 Jul 2006 13:35:52 +0200 Subject: [Numpy-discussion] Antwort: Re: vector to tensor matrix speed up In-Reply-To: <44BFDDEF.4060500@cox.net> Message-ID: Hi Tim, many thanks for the tipps, i used the same way with vectorized (chunk) method on the indexing operation. .. .... ...... ............ # out = zeros((size_mcf[0],sizes_smatrix[2]+5),Float32) # size_mcf[0] ~ 240000 eig = zeros((size_mcf[0],3,3),dtype=Float32) eigwert = zeros((size_mcf[0],3),dtype=Float64) # # here is speed up ~30 #for j in arange(0,size_mcf[0]): #eig[0,0] = out[j,1] #eig[1,1] = out[j,2] #eig[2,2] = out[j,3] # #eig[0,1] = out[j,4] #eig[0,2] = out[j,6] #eig[1,0] = out[j,4] #eig[1,2] = out[j,5] #eig[2,0] = out[j,6] #eig[2,1] = out[j,5] # eig[:,0,0] = out[:,1] eig[:,1,1] = out[:,2] eig[:,2,2] = out[:,3] eig[:,1,0] = eig[:,0,1] = out[:,4] eig[:,2,0] = eig[:,0,2] = out[:,6] eig[:,2,1] = eig[:,1,2] = out[:,5] # for i in arange(size_mcf[0]): eigwert[i] = eigvals(eig[i,:,:]) # out[:,7:10] = sort(eigwert[:,:].astype(float32)) out[:,10] = abs(out[:,7]-out[:,9]) speedup factor ~30 ! f. From wbaxter at gmail.com Fri Jul 21 08:22:18 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 21 Jul 2006 21:22:18 +0900 Subject: [Numpy-discussion] r_, c_, hstack, and vstack with 1-d arrays In-Reply-To: <44C097A2.5020900@gmx.net> References: <44BE2E5A.80404@gmx.net> <44C097A2.5020900@gmx.net> Message-ID: On 7/21/06, Sven Schreiber wrote: > Bill Baxter schrieb: > > > Finally, I noticed that the atleast_nd methods return arrays > > regardless of input type. > > Are you sure? I reported that issue with *stack and I remember it was fixed. Doh! My bad. You're right. I was looking at the code in SVN for the atleast_* methods and didn't realize that the array constructor (or any constructor) could actually return something besides an object of that class. But that's exactly what array(subok=True) allows. > > SUMMARY: > > * make r_ behave like "vstack plus range literals" > > * make column_stack only transpose its 1d inputs. > > * rename r_,c_ to v_,h_ (or something else) to make their connection > > with vstack and hstack clearer. Maybe vs_ and hs_ would be better? > > * make a new vertsion of 'c_' that acts like column_stack so that > > theres a nice parallel v_<=>vstack, h_<=>hstack, c_<=>column_stack > > * make atleast_*d methods preserve the input type whenever possible > > > > One problem with all that renaming is the other (maybe more important) > function of r_: build (array or) matrix ranges quickly. How would you duplicate this kind of behavior, you mean? >>> r_[1:4,0,4] array([1, 2, 3, 0, 4]) That's what h_ would do. Just as if you had done hstack( (range(1,4),0,4) ). That's actually a good reason for the renaming/retooling. r_ is kind of schitzophrenic now in that it acts *either* as "concatenate rows (vstack-like, for >=2-d)" or "build me a row (hstack-like, for <2d)". So it's hard to remember what the 'r' in r_ stands for. On the other hand, c_ is always hstack-like. By the way, I noticed that 'row_stack' has appeared as a synonym for 'vstack' in SVN. Thanks to whomever added it! --bb From svetosch at gmx.net Fri Jul 21 11:08:08 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Fri, 21 Jul 2006 17:08:08 +0200 Subject: [Numpy-discussion] matrix-related bug in 1.0b1 Message-ID: <44C0EDD8.3000701@gmx.net> Hi, Summary: Slicing seems to be broken with matrices now. I eagerly installed the new beta, but soon stumbled over this bug. I hope I'm missing something, but afaics that behavior used to be different (and correct) before in 0.9.8. Don't know exactly when this changed, though. I did a fresh install (uninstalled old numpy and also matplotlib first) of the official binary for windows/python 2.4. Example: >>> import numpy as n >>> n.__version__ '1.0b1' >>> import numpy.matlib as m >>> a = n.zeros((2,3)) >>> b = m.zeros((2,3)) >>> a[:1,:].shape (1, 3) >>> b[:1,:].shape (3, 1) >>> Note the array slicing works correct, but the equivalent thing with the matrix does not. I also noticed the following (in a new python session) import strangeness: >>> import numpy >>> numpy.matlib.zeros((2,3)) Traceback (most recent call last): File "", line 1, in ? AttributeError: 'module' object has no attribute 'matlib' >>> Why is the direct access to matlib impossible? Either I'm missing something (I well may be, because I'm melting away at 36 centigrades or so...), or imho a new beta should be put out quickly to enable further testing (and use). Thanks, Sven From lists.steve at arachnedesign.net Sat Jul 22 17:50:13 2006 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Sat, 22 Jul 2006 17:50:13 -0400 Subject: [Numpy-discussion] Fresh svn numpy install fails 2 tests (OS X) Message-ID: <358EE207-3D29-40BE-87EA-912639CB93FD@arachnedesign.net> Hi folks, Since 1.0 release is eminent, I just wanted to draw the attention to two failures I get when I run numpy.test(1). I've never been able to get numpy to pass all test cases, but now it fails a second one, so .. I'm pasting it below. Please let me know if these are non-consequential. System info: + Intel Mac (MacBook Pro) + OS X.4.7 + numpy version: 1.0.2881 test failures: FAIL: check_large_types (numpy.core.tests.test_scalarmath.test_power) ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/local/lib/python2.4/site-packages/numpy/core/tests/ test_scalarmath.py", line 47, in check_large_types assert b == 6765201, "error with %r: got %r" % (t,b) AssertionError: error with : got 0.0 ====================================================================== FAIL: check_types (numpy.core.tests.test_scalarmath.test_types) ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/local/lib/python2.4/site-packages/numpy/core/tests/ test_scalarmath.py", line 20, in check_types assert a == 1, "error with %r: got %r" % (atype,a) AssertionError: error with : got 1.02604810941480982577e-4936 ---------------------------------------------------------------------- Ran 468 tests in 1.157s FAILED (failures=2) Out[2]: ---------------------------------------- Thanks, -steve From haase at msg.ucsf.edu Sat Jul 22 23:39:19 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sat, 22 Jul 2006 20:39:19 -0700 Subject: [Numpy-discussion] how add new attribute to a numpy array object ? Message-ID: <44C2EF67.40505@msg.ucsf.edu> Hi, I have a (medical) image file. I wrote a nice interface based on memmap using numarray. The class design I used was essentially to return a numarray array object with a new "custom" attribute giving access to special information about the base file. Now with numpy I noticed that a numpy object does not allow adding new attributes !! (How is this ? Why ?) Travis already suggested (replying to one of my last postings) to create a new sub class of numpy.ndarray. But how do I initialize an object of my new class to be "basically identically to" an existing ndarray object ? Normally I could do class B(N.ndarray): pass a=N.arange(10) a.__class__ = B BUT I get this error: #>>> a.__class__ = B Traceback (most recent call last): File "", line 1, in ? TypeError: __class__ assignment: only for heap types What is a "heap type" ? Why ? How can I do what I want ? Thanks, Sebastian Haase From haase at msg.ucsf.edu Sun Jul 23 01:32:33 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sat, 22 Jul 2006 22:32:33 -0700 Subject: [Numpy-discussion] bug in numpy recarray ? Message-ID: <44C309F1.5090607@msg.ucsf.edu> Hi! I'm trying to convert my numarray records code to numpy. >>> type(m.hdrArray) >>> m.hdrArray.d [(array([ 1., 1., 1.], dtype=float32),)] but I get: >>> m.hdrArray[0].getfield('d') 5.43230922614e-312 Am I missing something or is this a bug ? Further details: >>> m.hdrArray.dtype.descr [('Num', [('f1', '>> >>> >>> m.hdrArray[0].dtype.descr [('Num', [('f1', ' References: <44C2EF67.40505@msg.ucsf.edu> Message-ID: <44C3213F.2020003@hawaii.edu> Sebastian Haase wrote: > Hi, > I have a (medical) image file. > I wrote a nice interface based on memmap using numarray. > The class design I used was essentially to return a numarray array > object with a new "custom" attribute giving access to special > information about the base file. > > Now with numpy I noticed that a numpy object does not allow adding new > attributes !! (How is this ? Why ?) > > Travis already suggested (replying to one of my last postings) to create > a new sub class of numpy.ndarray. > > But how do I initialize an object of my new class to be "basically > identically to" an existing ndarray object ? > Normally I could do > class B(N.ndarray): > pass > a=N.arange(10) > a.__class__ = B Isn't this what you need to do instead? In [1]:import numpy as N In [2]:class B(N.ndarray): ...: pass ...: In [3]:a = B(N.arange(10)) In [4]:a.__class__ Out[4]: In [5]:a.stuff = 'stuff' I don't think it makes sense to try to change the __class__ attribute by assignment. Eric From bioinformed at gmail.com Sun Jul 23 13:02:09 2006 From: bioinformed at gmail.com (Kevin Jacobs ) Date: Sun, 23 Jul 2006 13:02:09 -0400 Subject: [Numpy-discussion] how add new attribute to a numpy array object ? In-Reply-To: <44C2EF67.40505@msg.ucsf.edu> References: <44C2EF67.40505@msg.ucsf.edu> Message-ID: <2e1434c10607231002w7b8b3e4bjf95c7979e1d0765b@mail.gmail.com> On 7/22/06, Sebastian Haase wrote: > > Normally I could do > class B(N.ndarray): > pass > a=N.arange(10) > a.__class__ = B > > BUT I get this error: > #>>> a.__class__ = B > Traceback (most recent call last): > File "", line 1, in ? > TypeError: __class__ assignment: only for heap types > > What is a "heap type" ? Why ? How can I do what I want ? Assigning to __class__ makes sense for objects that allocate a dictionary for storage of attributes or have slots allocated to hold the values. The heap type error is due to a missing flag in the class definition and could be corrected. However, it may not be the best thing to do. Calling B(array) is certainly safer, although a bit more expensive. -Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: From haase at msg.ucsf.edu Sun Jul 23 15:19:04 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sun, 23 Jul 2006 12:19:04 -0700 Subject: [Numpy-discussion] how add new attribute to a numpy array object ? In-Reply-To: <2e1434c10607231002w7b8b3e4bjf95c7979e1d0765b@mail.gmail.com> References: <44C2EF67.40505@msg.ucsf.edu> <2e1434c10607231002w7b8b3e4bjf95c7979e1d0765b@mail.gmail.com> Message-ID: <44C3CBA8.6010809@msg.ucsf.edu> Kevin Jacobs wrote: > On 7/22/06, *Sebastian Haase* > wrote: > > Normally I could do > class B(N.ndarray): > pass > a=N.arange(10) > a.__class__ = B > > BUT I get this error: > #>>> a.__class__ = B > Traceback (most recent call last): > File "", line 1, in ? > TypeError: __class__ assignment: only for heap types > > What is a "heap type" ? Why ? How can I do what I want ? > > > > Assigning to __class__ makes sense for objects that allocate a > dictionary for storage of attributes or have slots allocated to hold the > values. The heap type error is due to a missing flag in the class > definition and could be corrected. However, it may not be the best > thing to do. Calling B(array) is certainly safer, although a bit more > expensive. > > -Kevin > Thanks for the replies. Googling for this I was surprised myself that it IS legal (and done) to assign to obj.__class__. Kevin, I tried what you suggested first -- I think in C++ it would be called "using the copy-constructor". But I get an error - something like: "__new__() needs at least 3 arguments" In other words: (maybe?) in Python there is not always a "copy-constructor" (in fact there is no constructor overloading at all ...) So if there is "just a missing flag" - it would be great if this could be put in. It turns out that the "assigning to __class__"-scheme worked for the ndarray subclass "memmap" (i.e. I was sub classing from memmap and then I could assing origMemmapObj.__class__ = myClass) Thanks, Sebastian Haase From filipwasilewski at gmail.com Sun Jul 23 16:02:50 2006 From: filipwasilewski at gmail.com (Filip Wasilewski) Date: Sun, 23 Jul 2006 22:02:50 +0200 Subject: [Numpy-discussion] how add new attribute to a numpy array object ? In-Reply-To: <44C3213F.2020003@hawaii.edu> References: <44C2EF67.40505@msg.ucsf.edu> <44C3213F.2020003@hawaii.edu> Message-ID: On 7/23/06, Eric Firing wrote: > Sebastian Haase wrote: > > Hi, > > I have a (medical) image file. > > I wrote a nice interface based on memmap using numarray. > > The class design I used was essentially to return a numarray array > > object with a new "custom" attribute giving access to special > > information about the base file. > > > > Now with numpy I noticed that a numpy object does not allow adding new > > attributes !! (How is this ? Why ?) > > > > Travis already suggested (replying to one of my last postings) to create > > a new sub class of numpy.ndarray. > > > > But how do I initialize an object of my new class to be "basically > > identically to" an existing ndarray object ? > > Normally I could do > > class B(N.ndarray): > > pass > > a=N.arange(10) > > a.__class__ = B > > Isn't this what you need to do instead? > > In [1]:import numpy as N > > In [2]:class B(N.ndarray): > ...: pass > ...: > > In [3]:a = B(N.arange(10)) It won't work like that. The constructor for the ndarray is: | ndarray.__new__(subtype, shape=, dtype=int_, buffer=None, | offset=0, strides=None, fortran=False) so you will get either an exception caused by inappropriate shape value or completely wrong result. >>> numpy.ndarray([1,2]) array([[10966528, 18946344]]) >>> numpy.ndarray([1,2]).shape (1, 2) >>> numpy.ndarray(numpy.arange(5)) array([], shape=(0, 1, 2, 3, 4), dtype=int32) And this is a thing you souldn't do rather than a bug. To create an instance of ndarray's subclass B from ndarray object, one need to call the ndarray.view method or the ndarray.__new__ constructor explicitly: class B(numpy.ndarray): def __new__(subtype, data): if isinstance(data, B): return data if isinstance(data, numpy.ndarray): return data.view(subtype) arr = numpy.array(data) return numpy.ndarray.__new__(B, shape=arr.shape, dtype=arr.dtype, buffer=arr) A good example of subclasing ndarray is the matrix class in core/defmatrix.py (SVN version). cheers, fw From haase at msg.ucsf.edu Sun Jul 23 19:23:45 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sun, 23 Jul 2006 16:23:45 -0700 Subject: [Numpy-discussion] C extensions: is Py_XDECREF() needed when using PyArray_FromAny() Message-ID: <44C40501.3050705@msg.ucsf.edu> Hi, I'm converting SWIG typemap'ed C extensions from numarray to numpy. I studied (and use parts of) numpy.i from the doc directory. I noticed that there is no decref for the TYPEMAP_INPLACE2 typemap. This uses a function obj_to_array_no_conversion() which in turn just returns the original PyObject* ( casted to a PyArrayObject* after some sanity checks) It looks to me that in this case there should be an explicit Py_INCREF() - in case the function is threaded (releases the Python GIL) since it holds a pointer to that object's data . (Alternatively) Travis suggested (at the http://www.scipy.org/Converting_from_numarray wiki page) using PyArray_FromAny - is this incrementing the ref.count (implicitely) ? The numarray equivalent (NA_InputArray) IS incrementing the ref.count (as far as I know...). Furthermore on that same wiki page the PyArray_FromAny() is called together with PyArray_DescrFromType(). After searching through the numpy source I found that in blasdot/_dotblas.c (in dotblas_matrixproduct() )there is an explicit Py_INCREF even on the dtype returned from PyArray_DescrFromType. I would argue that ref.counting is always very tricky territory ;-) Hopefully someone can enlighten me . Thanks, Sebastian Haase From wbaxter at gmail.com Sun Jul 23 21:58:43 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 24 Jul 2006 10:58:43 +0900 Subject: [Numpy-discussion] matrix-related bug in 1.0b1 In-Reply-To: <44C0EDD8.3000701@gmx.net> References: <44C0EDD8.3000701@gmx.net> Message-ID: Howdy, On 7/22/06, Sven Schreiber wrote: > Hi, > > Summary: Slicing seems to be broken with matrices now. > > ... > Example: > > >>> import numpy as n > >>> n.__version__ > '1.0b1' > >>> import numpy.matlib as m > >>> a = n.zeros((2,3)) > >>> b = m.zeros((2,3)) > >>> a[:1,:].shape > (1, 3) > >>> b[:1,:].shape > (3, 1) > >>> > > Note the array slicing works correct, but the equivalent thing with the > matrix does not. Looks like it happened in rev 2698 of defmatrix.py, matrix.__getitem__ method: if isscalar(index[1]): if isscalar(index[0]): retscal = True elif out.shape[0] == 1: sh = out.shape out.shape = (sh[1], sh[0]) ==> elif isinstance(index[1], (slice, types.EllipsisType)): ==> if out.shape[0] == 1 and not isscalar(index[0]): It behaves like array if you remove the 'not' in the last line. But maybe that breaks some other cases? Maybe you can try making that change in your numpy/core/defmatrix.py (around line 140) and see if anything else breaks for you. > I also noticed the following (in a new python session) import strangeness: > > >>> import numpy > >>> numpy.matlib.zeros((2,3)) > Traceback (most recent call last): > File "", line 1, in ? > AttributeError: 'module' object has no attribute 'matlib' > >>> > > Why is the direct access to matlib impossible? Maybe the thinking is that since it's a compatibily module, if you want it you should explicity import it. Like you have to do with oldnumeric. --bb From haase at msg.ucsf.edu Mon Jul 24 00:47:06 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sun, 23 Jul 2006 21:47:06 -0700 Subject: [Numpy-discussion] trivial question: how to compare dtype - but ignoring byteorder ? Message-ID: <44C450CA.3010609@msg.ucsf.edu> Hi, if I have a numpy array 'a' and say: a.dtype == numpy.float32 Is the result independent of a's byteorder ? (That's what I would expect ! Just checking !) Thanks, Sebastian Haase From pgmdevlist at gmail.com Mon Jul 24 06:02:24 2006 From: pgmdevlist at gmail.com (PGM) Date: Mon, 24 Jul 2006 06:02:24 -0400 Subject: [Numpy-discussion] float64scalar and divmod (and numpy 0.9.8) Message-ID: <200607240602.24257.pgmdevlist@gmail.com> Folks , I'm still using numpy 0.9.8 and just ran into this problem on my machine (AMD64): >>> import numpy as N >>> x = N.array([1.23456]) >>> print divmod(x,1) (array([ 1.]), array([ 0.23456])) >>> print divmod(x[0],1) () >>> print divmod(x.tolist()[0],1) (1.0, 0.2345600000000001) divmod doesn't seem to like ' References: <44C450CA.3010609@msg.ucsf.edu> Message-ID: <200607241218.55965.karol.langner@kn.pl> On Monday 24 July 2006 06:47, Sebastian Haase wrote: > Hi, > if I have a numpy array 'a' > and say: > a.dtype == numpy.float32 > > Is the result independent of a's byteorder ? > (That's what I would expect ! Just checking !) > > Thanks, > Sebastian Haase The condition will always be False, because you're comparing wrong things here, numpy.float32 is a scalar type, not a dtype. >>> numpy.float32 >>> type(numpy.dtype('>f4')) And I think byteorder matters when comparing dtypes: >>> numpy.dtype('>f4') == numpy.dtype(' References: <44C0EDD8.3000701@gmx.net> Message-ID: <44C4B3E7.6060104@gmx.net> Thanks for helping out on matrix stuff, Bill! Bill Baxter schrieb: > On 7/22/06, Sven Schreiber wrote: >> >> Note the array slicing works correct, but the equivalent thing with the >> matrix does not. > > Looks like it happened in rev 2698 of defmatrix.py, matrix.__getitem__ > method: > > if isscalar(index[1]): > if isscalar(index[0]): > retscal = True > elif out.shape[0] == 1: > sh = out.shape > out.shape = (sh[1], sh[0]) > ==> elif isinstance(index[1], (slice, types.EllipsisType)): > ==> if out.shape[0] == 1 and not isscalar(index[0]): > > It behaves like array if you remove the 'not' in the last line. > But maybe that breaks some other cases? > Maybe you can try making that change in your numpy/core/defmatrix.py > (around line 140) and see if anything else breaks for you. > Hm, I don't know -- if you don't mind I'd like to get a second opinion before I mess around there. It's funny though that the changeset has the title "fixing up matrix slicing" or something like that... >> >> Why is the direct access to matlib impossible? > > Maybe the thinking is that since it's a compatibily module, if you > want it you should explicity import it. Like you have to do with > oldnumeric. > If that is the reason I can't really follow the logic, but I don't really mind the status quo, either. -sven From david.huard at gmail.com Mon Jul 24 07:58:50 2006 From: david.huard at gmail.com (David Huard) Date: Mon, 24 Jul 2006 07:58:50 -0400 Subject: [Numpy-discussion] float64scalar and divmod (and numpy 0.9.8) In-Reply-To: <200607240602.24257.pgmdevlist@gmail.com> References: <200607240602.24257.pgmdevlist@gmail.com> Message-ID: <91cf711d0607240458q1b06f877ic033aa64e0b72143@mail.gmail.com> 2006/7/24, PGM : > > Folks , > I'm still using numpy 0.9.8 and just ran into this problem on my machine > (AMD64): > > >>> import numpy as N > >>> x = N.array([1.23456]) > >>> print divmod(x,1) > (array([ 1.]), array([ 0.23456])) > >>> print divmod(x[0],1) > () > >>> print divmod(x.tolist()[0],1) > (1.0, 0.2345600000000001) > > divmod doesn't seem to like ' trick... > Did anybody run into this problem already ? Is it really numpy related ? > If > this is a bug on 0.9.8, has it been corrected already ? It works in svn. David -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivilata at carabos.com Mon Jul 24 08:00:41 2006 From: ivilata at carabos.com (Ivan Vilata i Balaguer) Date: Mon, 24 Jul 2006 14:00:41 +0200 Subject: [Numpy-discussion] "boolean" type Message-ID: <44C4B669.7000001@carabos.com> Hi all, Since there is a "string" type which is the same as "str_", how come there is no "boolean" type which is the same as "bool_"? Did I miss some design decision about naming? You know, just for completeness, not that it is some kind of problem at all! ;) Cheers, :: Ivan Vilata i Balaguer >qo< http://www.carabos.com/ C?rabos Coop. V. V V Enjoy Data "" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 307 bytes Desc: OpenPGP digital signature URL: From a.h.jaffe at bakerjaffe.plus.com Mon Jul 24 10:04:05 2006 From: a.h.jaffe at bakerjaffe.plus.com (Andrew Jaffe) Date: Mon, 24 Jul 2006 15:04:05 +0100 Subject: [Numpy-discussion] mgrid, ogrid and back again Message-ID: <44C4D355.6020607@bakerjaffe.plus.com> Hi All, I'm finding myself dealing with n-dimensional grids quite a lot, and trying to do some 'tricky' index manipulation. The main problem is manipulating arrays when I don't know a priori the number of dimensions; in essence I need to be able to iterate across dimensions. first, I've got three arrays of lower-bounds (param_min), upper-bounds (param_max) and numbers of steps (nstep). I'm using the following incantations to produce the association mgrid an ogrid arrays: args = tuple(slice(p1, p2, n*1j) for p1,p2,n in zip(param_min, param_max, nstep)) param_grid = N.mgrid.__getitem__(args) Is this the easiest way to do this? Second, from the mgrid object, param_grid, I want to recover the step sizes (assume I've thrown away the args so I can't make an ogrid object). This seems to work: deltas = numpy.empty(npar) for i in xrange(npar): idxtup = (i,)+(0,)*i + (1,) + (0,)*(npar-1-i) deltas[i] = param_grid[idxtup] - param_grid[((i,)+(0,)*npar )] (Or I could compress this into a single somewhat complicated list comprehension). Again, this seems a bit overly complicated. Any ideas for simplifying it? But at least I can work out how to do these things. Finally, however, I need to reconstruct the individual param_min:param_max:(nstep*1j) 1-d arrays (i.e., the flattened versions of the ogrid output). These are effectively param_grid[i,0,,,,:,,,,] where ':' is in slot i. But I don't know how to emulate ':' in either a slice object or tuple-indexing notation. Obviously I could just do a more complicated version of the deltas[] calculation, or direct manipulation on param_grid.flat, but both seem like too much work on my part... Thanks in advance! Andrew p.s. thanks to travis for all his hard work, especially in the run-up to 1.0b (although test() crashes on my PPC Mac... more on that later when I've had time to play). From Pierre.Barbier_de_Reuille at inria.fr Mon Jul 24 11:16:40 2006 From: Pierre.Barbier_de_Reuille at inria.fr (Pierre Barbier de Reuille) Date: Mon, 24 Jul 2006 16:16:40 +0100 Subject: [Numpy-discussion] "boolean" type In-Reply-To: <44C4B669.7000001@carabos.com> References: <44C4B669.7000001@carabos.com> Message-ID: <44C4E458.9080109@inria.fr> Ivan Vilata i Balaguer wrote: > Hi all, > > Since there is a "string" type which is the same as "str_", how come > there is no "boolean" type which is the same as "bool_"? Did I miss > some design decision about naming? You know, just for completeness, not > that it is some kind of problem at all! ;) > Well ... >>> import numpy >>> numpy.__version__ '0.9.9.2852' >>> numpy.bool_ Pierre > Cheers, > > :: > > Ivan Vilata i Balaguer >qo< http://www.carabos.com/ > C?rabos Coop. V. V V Enjoy Data > "" > > > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From ivilata at carabos.com Mon Jul 24 12:29:11 2006 From: ivilata at carabos.com (Ivan Vilata i Balaguer) Date: Mon, 24 Jul 2006 18:29:11 +0200 Subject: [Numpy-discussion] "boolean" type In-Reply-To: <44C4E458.9080109@inria.fr> References: <44C4B669.7000001@carabos.com> <44C4E458.9080109@inria.fr> Message-ID: <44C4F557.1040300@carabos.com> En/na Pierre Barbier de Reuille ha escrit:: >>>>import numpy >>>>numpy.__version__ >'0.9.9.2852' >>>>numpy.bool_ > Sorry if I didn't make my question clear. What I find lacking is a ``numpy.boolean`` type which is to ``numpy.bool_`` the same as ``numpy.string`` is now to ``numpy.str_`` (i.e. a pure reference with a prettier name). Otherwise I'm not getting what you're meaning! C:) :: Ivan Vilata i Balaguer >qo< http://www.carabos.com/ C?rabos Coop. V. V V Enjoy Data "" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 307 bytes Desc: OpenPGP digital signature URL: From Pierre.Barbier_de_Reuille at inria.fr Mon Jul 24 12:39:57 2006 From: Pierre.Barbier_de_Reuille at inria.fr (Pierre Barbier de Reuille) Date: Mon, 24 Jul 2006 17:39:57 +0100 Subject: [Numpy-discussion] "boolean" type In-Reply-To: <44C4F557.1040300@carabos.com> References: <44C4B669.7000001@carabos.com> <44C4E458.9080109@inria.fr> <44C4F557.1040300@carabos.com> Message-ID: <44C4F7DD.50008@inria.fr> Ivan Vilata i Balaguer wrote: > En/na Pierre Barbier de Reuille ha escrit:: > > >>>>> import numpy >>>>> numpy.__version__ >>>>> >> '0.9.9.2852' >> >>>>> numpy.bool_ >>>>> >> >> > > Sorry if I didn't make my question clear. What I find lacking is a > ``numpy.boolean`` type which is to ``numpy.bool_`` the same as > ``numpy.string`` is now to ``numpy.str_`` (i.e. a pure reference with a > prettier name). Otherwise I'm not getting what you're meaning! C:) > > :: > > Ivan Vilata i Balaguer >qo< http://www.carabos.com/ > C?rabos Coop. V. V V Enjoy Data > "" > > Ok, so maybe it is because it is called "bool8" ? So that it is clear that it takes 8 bits ? Pierre From jordan at cutlerlawfirm.com Mon Jul 24 12:43:38 2006 From: jordan at cutlerlawfirm.com (Jordana Hollingshead) Date: Mon, 24 Jul 2006 09:43:38 -0700 Subject: [Numpy-discussion] to yarec Message-ID: <000001c6af40$5068c8d0$5ef3a8c0@tus97> Bes k t S v el s li y ng W v atc u he h s R y OL j EX C y ART f IER BR e EI y TLIN k G BV p LGA q RI OM g EG z A P v ATE z K P x hilip h pe and man o y othe f r Han a dbag r s & Pu p rs w es, TI k FFA l NY & CO Je m werl t y Or z de v r T j ODA c Y and sav k e 2 o 5 % http://righttherehun.com share of the reward. And so Bilbo was swung down from the wall, and departed with nothing for all his trouble, except the armour which Thorin had given him already. More than one of the dwarves in their hearts felt shame and pity at his going. Farewell! he cried to them. We may meet again as friends. Be off! called Thorin. You have mail upon you, which was -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Mon Jul 24 13:05:35 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 24 Jul 2006 11:05:35 -0600 Subject: [Numpy-discussion] numpy 1.0 In-Reply-To: <5775F577-5FFE-4C0C-9832-856A2DF5C6EA@cns.montana.edu> References: <5775F577-5FFE-4C0C-9832-856A2DF5C6EA@cns.montana.edu> Message-ID: <44C4FDDF.9080001@ieee.org> Graham Cummins wrote: > Greetings, > > I just downloaded numpy 1.0b1. I see a lot of changes from 0.9.8, and > I'm curious as to whether these changes will be a lasting property of > numpy 1.0 and later. > > Most of the changes relate to nomenclature for type constants (e.g. > int32, complex128, newaxis) and functions (e.g. inverse_real_fft -> > ifft). Although it takes some time to comb through code for all of the > possible name changes (there are lots!), it's easy enough to do. Release notes will be forthcoming. These changes will be lasting... What changed is that "old" names were placed only in a compatibility module (numpy.oldnumeric). Import from there if you want the old names (convertcode was also changed to alter Numeric-->numpy.oldnumeric. This was done so as to make it clearer what is "old" and for compatibility purposes only and what new code should be written with. > > The thing that is taking me longer is (as usual) converting c > extensions. Float32, and PyArray_Float32 used to be defined in 0.9.8, > and are now not. AFAICT, npy_float works in the same way Float32 used > to work, but I haven't yet figured out what to use in place of > PyArray_Float32 in, for example "PyArray_FROM_OTF(data, ?? , ALIGNED | > CONTIGUOUS);" > Here, we assigned the prefixes NPY_ and npy_ to all the old CAPITALIZED and uncapitalized names, respectively to avoid the problem of name clashes which occurs commonly when using NumPy to wrap another library. The un-prefixed names are still available when you use #include "numpy/noprefix.h" (which is what NumPy itself uses). > > On another topic, when I install numpy (version 0.9.8 or 1.0b1) using > "seutup.py install", the headers needed to build extensions don't get > moved into my python distribution directory tree. I've been moving > these files by hand, and that seems to work, but could I pass some > argument to distutils that would do this automatically? To support multiple-version installations of NumPy (like eggs allow) it's important to put the headers in their own location and not in a system-wide directory. If you want to place them system-wide, you currently need to copy them by hand. But, it's not recommended to do that. Just append the output of numpy.get_include() to the list of include directories. -Travis From haase at msg.ucsf.edu Mon Jul 24 14:10:07 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 24 Jul 2006 11:10:07 -0700 Subject: [Numpy-discussion] trivial question: how to compare dtype - but ignoring byteorder ? In-Reply-To: <200607241218.55965.karol.langner@kn.pl> References: <44C450CA.3010609@msg.ucsf.edu> <200607241218.55965.karol.langner@kn.pl> Message-ID: <200607241110.07686.haase@msg.ucsf.edu> On Monday 24 July 2006 03:18, Karol Langner wrote: > On Monday 24 July 2006 06:47, Sebastian Haase wrote: > > Hi, > > if I have a numpy array 'a' > > and say: > > a.dtype == numpy.float32 > > > > Is the result independent of a's byteorder ? > > (That's what I would expect ! Just checking !) > > > > Thanks, > > Sebastian Haase > > The condition will always be False, because you're comparing wrong things > here, numpy.float32 is a scalar type, not a dtype. Hi ! Thanks for the reply. Did you actually run this ? I get: #>>> a=N.arange(10, dtype=N.float32) #>>> a.dtype == N.float32 #True #>>> N.__version__ #'0.9.9.2823' > > >>> numpy.float32 > > > > >>> type(numpy.dtype('>f4')) > > > > And I think byteorder matters when comparing dtypes: > >>> numpy.dtype('>f4') == numpy.dtype(' False OK - I did a test now: #>>> b= a.copy() #>>> b=b.newbyteorder('big') #>>> a.dtype == b.dtype #False #>>> a.dtype #'>> b.dtype #'>f4' How can I do a comparison showing that both a and b are float32 ?? Thanks, Sebastian Haase From karol.langner at kn.pl Mon Jul 24 14:36:54 2006 From: karol.langner at kn.pl (Karol Langner) Date: Mon, 24 Jul 2006 20:36:54 +0200 Subject: [Numpy-discussion] trivial question: how to compare dtype - but ignoring byteorder ? In-Reply-To: <200607241110.07686.haase@msg.ucsf.edu> References: <44C450CA.3010609@msg.ucsf.edu> <200607241218.55965.karol.langner@kn.pl> <200607241110.07686.haase@msg.ucsf.edu> Message-ID: <200607242036.54681.karol.langner@kn.pl> On Monday 24 July 2006 20:10, Sebastian Haase wrote: > Hi ! > Thanks for the reply. > Did you actually run this ? I get: > #>>> a=N.arange(10, dtype=N.float32) > #>>> a.dtype == N.float32 > #True > #>>> N.__version__ > #'0.9.9.2823' Hi, Looks like I need to upgrade my working version or something... >>> import numpy as N >>> N.__version__ '0.9.8' >>> a=N.arange(10, dtype=N.float32) >>> a.dtype == N.float32 False Cheers, Karol -- written by Karol Langner pon lip 24 20:36:05 CEST 2006 From haase at msg.ucsf.edu Mon Jul 24 14:37:35 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 24 Jul 2006 11:37:35 -0700 Subject: [Numpy-discussion] numpy.prod() vs. numpy.product() Message-ID: <200607241137.36094.haase@msg.ucsf.edu> Hi, Are numpy.product() and numpy.prod() doing the exact same thing ? If yes, why are they pointing to two different functions ? >>> N.prod >>> N.product Thanks, Sebastian Haase From oliphant.travis at ieee.org Mon Jul 24 15:18:46 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 24 Jul 2006 13:18:46 -0600 Subject: [Numpy-discussion] Problem converting a numarray extension to numpy In-Reply-To: <40e64fa20607202101o75ca115ate4ed577fd6d98ac3@mail.gmail.com> References: <40e64fa20607202101o75ca115ate4ed577fd6d98ac3@mail.gmail.com> Message-ID: <44C51D16.9020003@ieee.org> Paul Barrett wrote: > I'm having a problem converting a C extension module that was > originally written for numarray to use numpy. I using swig to create > a wrapper flle for the C code. I have added the > numpy.get_numarray_include() method to my setup.py file and have > changed the numarray/libnumarray.h to use numpy/libnumarray.h. The > extension appears to compile fine (with the exception of some warning > messages). However, when I import the module, I get a segfault. Do I > need to add anything else to the share library's initialization step > other than import_libnumarray()? > No, that should be enough. The numarray C-API has only been tested on a few extension modules. It's very possible some of the calls have problems. It's also possible you have an older version of numpy lying around somewhere. Do you get any kind of error message on import? -Travis From oliphant.travis at ieee.org Mon Jul 24 15:24:48 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 24 Jul 2006 13:24:48 -0600 Subject: [Numpy-discussion] bug in numpy recarray ? In-Reply-To: <44C309F1.5090607@msg.ucsf.edu> References: <44C309F1.5090607@msg.ucsf.edu> Message-ID: <44C51E80.40702@ieee.org> Sebastian Haase wrote: > Hi! > I'm trying to convert my numarray records code to numpy. > >>> type(m.hdrArray) > > >>> m.hdrArray.d > [(array([ 1., 1., 1.], dtype=float32),)] > > but I get: > >>> m.hdrArray[0].getfield('d') > 5.43230922614e-312 > > Am I missing something or is this a bug ? > Probably a bug. The getfield method needs more testing. File a ticket. -Travis From haase at msg.ucsf.edu Mon Jul 24 15:31:46 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 24 Jul 2006 12:31:46 -0700 Subject: [Numpy-discussion] how to get an array with "varying" poisson distribution Message-ID: <200607241231.46581.haase@msg.ucsf.edu> Hi, Essentially I'm looking for the equivalent of what was in numarray: from numarray import random_array random_array.poisson(arr) That is: if for example arr is a 256x256 array of positive integers, then this returns a new array of random numbers than are drawn according to the poisson statistics where arr's value at coordinate y,x determines the mean of the poisson distribution used to generate a new value for y,x. [[This is needed e.g. to simulate quantum noise in CCD images. Each pixel has different amount of noise depending of what it's (noise-free) "input" value was.]] Thanks, Sebastian Haase From oliphant.travis at ieee.org Mon Jul 24 15:29:15 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 24 Jul 2006 13:29:15 -0600 Subject: [Numpy-discussion] C extensions: is Py_XDECREF() needed when using PyArray_FromAny() In-Reply-To: <44C40501.3050705@msg.ucsf.edu> References: <44C40501.3050705@msg.ucsf.edu> Message-ID: <44C51F8B.60407@ieee.org> Sebastian Haase wrote: > Hi, > I'm converting SWIG typemap'ed C extensions from numarray to numpy. > I studied (and use parts of) numpy.i from the doc directory. > I noticed that there is no > decref for the TYPEMAP_INPLACE2 typemap. This uses a function > obj_to_array_no_conversion() which in turn just returns > the original PyObject* ( casted to a PyArrayObject* after some sanity > checks) It looks to me that in this case there should be an explicit > Py_INCREF() - in case the function is threaded (releases the Python > GIL) since it holds a pointer to that object's data . > Probably, true. The numpy.i typemaps are not thoroughly reference-count checked. > (Alternatively) Travis suggested (at the > http://www.scipy.org/Converting_from_numarray wiki page) using > PyArray_FromAny - is this incrementing the ref.count (implicitely) ? > The numarray equivalent (NA_InputArray) IS incrementing the ref.count > (as far as I know...). > > Yes, you get back a new reference from PyArray_FromAny. > Furthermore on that same wiki page the PyArray_FromAny() is called > together with PyArray_DescrFromType(). > After searching through the numpy source I found that in > blasdot/_dotblas.c (in dotblas_matrixproduct() )there is an explicit > Py_INCREF even on the dtype returned from PyArray_DescrFromType. > > PyArray_FromAny consumes a reference to the PyArray_Descr * object (which is a Python object). Thus, because PyArray_FromAny is called twice with the same data-type object, there is a need to increment it's reference count. -Travis From oliphant.travis at ieee.org Mon Jul 24 15:38:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 24 Jul 2006 13:38:02 -0600 Subject: [Numpy-discussion] matrix-related bug in 1.0b1 In-Reply-To: <44C4B3E7.6060104@gmx.net> References: <44C0EDD8.3000701@gmx.net> <44C4B3E7.6060104@gmx.net> Message-ID: <44C5219A.8020706@ieee.org> Sven Schreiber wrote: > Thanks for helping out on matrix stuff, Bill! > > > Hm, I don't know -- if you don't mind I'd like to get a second opinion > before I mess around there. It's funny though that the changeset has the > title "fixing up matrix slicing" or something like that... > The change was trying to fix up some cases but did break this one. The problem is that figuring out whether or not to transpose the result is a bit tricky. I've obviously still got it wrong. -Travis From oliphant.travis at ieee.org Mon Jul 24 15:36:56 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 24 Jul 2006 13:36:56 -0600 Subject: [Numpy-discussion] trivial question: how to compare dtype - but ignoring byteorder ? In-Reply-To: <44C450CA.3010609@msg.ucsf.edu> References: <44C450CA.3010609@msg.ucsf.edu> Message-ID: <44C52158.3050600@ieee.org> Sebastian Haase wrote: > Hi, > if I have a numpy array 'a' > and say: > a.dtype == numpy.float32 > > Is the result independent of a's byteorder ? > (That's what I would expect ! Just checking !) > I think I misread the question and saw "==" as "=" But, the answer I gave should still help: the byteorder is a property of the data-type. There is no such thing as "a's" byteorder. Thus, numpy.float32 (which is actually an array-scalar and not a true data-type) is interepreted as a machine-byte-order IEEE floating-point data-type with 32 bits. Thus, the result will depend on whether or not a.dtype is machine-order or not. -Travis From oliphant.travis at ieee.org Mon Jul 24 15:31:06 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 24 Jul 2006 13:31:06 -0600 Subject: [Numpy-discussion] trivial question: how to compare dtype - but ignoring byteorder ? In-Reply-To: <44C450CA.3010609@msg.ucsf.edu> References: <44C450CA.3010609@msg.ucsf.edu> Message-ID: <44C51FFA.9020002@ieee.org> Sebastian Haase wrote: > Hi, > if I have a numpy array 'a' > and say: > a.dtype == numpy.float32 > > Is the result independent of a's byteorder ? > The byteorder is a property of the data-type (not of the array) --- this is different from numarray where byteorder is a property of the array. a.dtype == numpy.float32 will always set the data-type to a machine-native float32 data-type. -Travis From svetosch at gmx.net Mon Jul 24 16:21:02 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Mon, 24 Jul 2006 22:21:02 +0200 Subject: [Numpy-discussion] matrix-related bug in 1.0b1 In-Reply-To: <44C5219A.8020706@ieee.org> References: <44C0EDD8.3000701@gmx.net> <44C4B3E7.6060104@gmx.net> <44C5219A.8020706@ieee.org> Message-ID: <44C52BAE.1090508@gmx.net> Travis Oliphant schrieb: > Sven Schreiber wrote: >> > The change was trying to fix up some cases but did break this one. The > problem is that figuring out whether or not to transpose the result is a > bit tricky. I've obviously still got it wrong. > Ok, this is obviously one of the places were an automated test would be helpful to avoid regressions. In the beta months ahead, I would like to help with contributing such tests. However, since I'm not a developer, I have little to no experience with that. Is there any numpy-specific documentation or guide on how to add tests? (I know there's plenty of guides on the general python testing framework.) Or any brilliant example in the distribution that should be followed? Regards, Sven p.s.: Is there going to be a new beta soon? Or is there a quick fix to this slicing problem in the meantime? Thanks. From haase at msg.ucsf.edu Mon Jul 24 16:28:13 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 24 Jul 2006 13:28:13 -0700 Subject: [Numpy-discussion] how add new attribute to a numpy array object ? In-Reply-To: <44C51E29.9040305@ieee.org> References: <44C2EF67.40505@msg.ucsf.edu> <44C51E29.9040305@ieee.org> Message-ID: <200607241328.13956.haase@msg.ucsf.edu> On Monday 24 July 2006 12:23, Travis Oliphant wrote: > Sebastian Haase wrote: > > Hi, > > I have a (medical) image file. > > I wrote a nice interface based on memmap using numarray. > > The class design I used was essentially to return a numarray array > > object with a new "custom" attribute giving access to special > > information about the base file. > > > > Now with numpy I noticed that a numpy object does not allow adding new > > attributes !! (How is this ? Why ?) > > > > Travis already suggested (replying to one of my last postings) to create > > a new sub class of numpy.ndarray. > > > > But how do I initialize an object of my new class to be "basically > > identically to" an existing ndarray object ? > > Normally I could do > > class B(N.ndarray): > > pass > > a=N.arange(10) > > a.__class__ = B > > > > BUT I get this error: > > #>>> a.__class__ = B > > Traceback (most recent call last): > > File "", line 1, in ? > > TypeError: __class__ assignment: only for heap types > > > > What is a "heap type" ? Why ? How can I do what I want ? > > A heap type is one created in Python (i.e. not builtin with C). > > You should be able to do > > a = a.view(B) > > -Travis Thanks - (I'm just replying because I assume your helpful answer was mistakenly sent to me only and did not make it to the list) From oliphant.travis at ieee.org Mon Jul 24 17:01:17 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 24 Jul 2006 15:01:17 -0600 Subject: [Numpy-discussion] SVN changes that need to go into 1.0 and 1.1 Message-ID: <44C5351D.7070701@ieee.org> For the next several months until 1.0 comes out. Please make changes that go into 1.0 to the ver1.0 branch of the NumPy SVN tree. Then, periodically, we can merge those changes back to the trunk for ver 1.1 -Travis From oliphant.travis at ieee.org Mon Jul 24 17:06:57 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 24 Jul 2006 15:06:57 -0600 Subject: [Numpy-discussion] Please comment on SVN versioning convention Message-ID: <44C53671.5060301@ieee.org> Andrew Straw has emphasized that the current strategy of appending the SVN version number to development versions of the SVN tree makes it hard to do version sorting. His proposal is to not change the version number until the first beta comes out. In other words, the trunk should not be 1.1 but 1.0dev? Are there any opposing opinions to this change. I personally think that is a confusing numbering scheme because 1.0dev seems to imply it's the development for version 1.0 instead of 1.1. But, if several others think it's a good idea to support easy-sorting of version numbers, then I will conceed. Perhaps we can have a version 'number' as 1.0plus? -Travis From fperez.net at gmail.com Mon Jul 24 17:15:48 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 24 Jul 2006 15:15:48 -0600 Subject: [Numpy-discussion] Please comment on SVN versioning convention In-Reply-To: <44C53671.5060301@ieee.org> References: <44C53671.5060301@ieee.org> Message-ID: On 7/24/06, Travis Oliphant wrote: > Andrew Straw has emphasized that the current strategy of appending the > SVN version number to development versions of the SVN tree makes it hard > to do version sorting. > > His proposal is to not change the version number until the first beta > comes out. I have to say that being able to quickly see the actual SVN revision number in the version helps a LOT in tracking down problems. Just look at how many posts on the list start with the canonical In [4]: import numpy,scipy In [5]: numpy.__version__ Out[5]: '0.9.9.2803' In [6]: scipy.__version__ Out[6]: '0.5.0.2079' printouts before discussing the problem. I don't really feel strongly about the issue, but if you change this, then please add a __revision__ attribute as well, so that this information can be quickly asked for in pure python (we don't want to bother newbies with obscure SVN lingo, esp. if they got their install from someone else). Cheers, f From robert.kern at gmail.com Mon Jul 24 17:20:42 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 24 Jul 2006 16:20:42 -0500 Subject: [Numpy-discussion] how to get an array with "varying" poisson distribution In-Reply-To: <200607241231.46581.haase@msg.ucsf.edu> References: <200607241231.46581.haase@msg.ucsf.edu> Message-ID: Sebastian Haase wrote: > Hi, > Essentially I'm looking for the equivalent of what was in numarray: > from numarray import random_array > random_array.poisson(arr) > > That is: if for example arr is a 256x256 array of positive integers, then this > returns a new array of random numbers than are drawn according to the poisson > statistics where arr's value at coordinate y,x determines the mean of the > poisson distribution used to generate a new value for y,x. I'm afraid that at this point in time, the distributions only accept scalar values for the parameters. I've thought about reimplementing the distribution functions as ufuncs, but that's a hefty chunk of work that won't happen for 1.0. I'm afraid that, for now, you're stuck with iterating over the values. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cookedm at physics.mcmaster.ca Mon Jul 24 17:24:18 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Mon, 24 Jul 2006 17:24:18 -0400 Subject: [Numpy-discussion] Fresh svn numpy install fails 2 tests (OS X) In-Reply-To: <358EE207-3D29-40BE-87EA-912639CB93FD@arachnedesign.net> References: <358EE207-3D29-40BE-87EA-912639CB93FD@arachnedesign.net> Message-ID: <20060724172418.661a9137@arbutus.physics.mcmaster.ca> On Sat, 22 Jul 2006 17:50:13 -0400 Steve Lianoglou wrote: > Hi folks, > > Since 1.0 release is eminent, I just wanted to draw the attention to > two failures I get when I run numpy.test(1). > > I've never been able to get numpy to pass all test cases, but now it > fails a second one, so .. I'm pasting it below. Please let me know if > these are non-consequential. It fails the second one b/c I added it because the failure in the first one wasn't clear enough :-) > System info: > + Intel Mac (MacBook Pro) > + OS X.4.7 > + numpy version: 1.0.2881 > > test failures: > > FAIL: check_large_types (numpy.core.tests.test_scalarmath.test_power) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/opt/local/lib/python2.4/site-packages/numpy/core/tests/ > test_scalarmath.py", line 47, in check_large_types > assert b == 6765201, "error with %r: got %r" % (t,b) > AssertionError: error with : got 0.0 > > ====================================================================== > FAIL: check_types (numpy.core.tests.test_scalarmath.test_types) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/opt/local/lib/python2.4/site-packages/numpy/core/tests/ > test_scalarmath.py", line 20, in check_types > assert a == 1, "error with %r: got %r" % (atype,a) > AssertionError: error with : got > 1.02604810941480982577e-4936 > > ---------------------------------------------------------------------- > Ran 468 tests in 1.157s > > FAILED (failures=2) > Out[2]: I'm aware of this (http://projects.scipy.org/scipy/numpy/ticket/183). It's a problem on PPC macs also. Travis thinks it may be a compiler problem. I've had a look, and can't see anything obvious. It *could* be that somewhere there's a typo in the code where things are set when sizeof(long double) == 128. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From cookedm at physics.mcmaster.ca Mon Jul 24 17:35:58 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Mon, 24 Jul 2006 17:35:58 -0400 Subject: [Numpy-discussion] Please comment on SVN versioning convention In-Reply-To: <44C53671.5060301@ieee.org> References: <44C53671.5060301@ieee.org> Message-ID: <20060724173558.660098ba@arbutus.physics.mcmaster.ca> On Mon, 24 Jul 2006 15:06:57 -0600 Travis Oliphant wrote: > Andrew Straw has emphasized that the current strategy of appending the > SVN version number to development versions of the SVN tree makes it hard > to do version sorting. > > His proposal is to not change the version number until the first beta > comes out. > > In other words, the trunk should not be 1.1 but 1.0dev? > > Are there any opposing opinions to this change. I personally think that > is a confusing numbering scheme because 1.0dev seems to imply it's the > development for version 1.0 instead of 1.1. But, if several others > think it's a good idea to support easy-sorting of version numbers, then > I will conceed. > > Perhaps we can have a version 'number' as 1.0plus? What if we use even numbers for stable releases, odd for development releases, like for Linux kernels? e.g., make 1.1 the dev branch (current trunk), and make 1.2 the next stable release after 1.0. Or, make the trunk 1.0.99. The .99 usually makes it pretty clear that it's the development branch. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From ndarray at mac.com Mon Jul 24 17:49:37 2006 From: ndarray at mac.com (Sasha) Date: Mon, 24 Jul 2006 17:49:37 -0400 Subject: [Numpy-discussion] Please comment on SVN versioning convention In-Reply-To: <44C53671.5060301@ieee.org> References: <44C53671.5060301@ieee.org> Message-ID: On 7/24/06, Travis Oliphant wrote: > Andrew Straw has emphasized that the current strategy of appending the > SVN version number to development versions of the SVN tree makes it hard > to do version sorting. I am not sure what the problem is, but if the concern is that >>> '0.9.9.2803' > '0.9.9' True I suggest fixing that by appending '.final' to the release version string: >>> '0.9.9.2803' > '0.9.9.final' False If we are going to make any changes in this area, I would suggest following python sys and define three variables: numpy.version (human readable string including compiler info), numpy.version_info (sortable tuple) and numpy.hexversion (sortable integer that can be used by C preprocessor). I am not sure if python sys includes svn revision number in sys.version (I don't have svn version of python around), but it does include it in beta sys.version: >>> print sys.version 2.5b2 (r25b2:50512, Jul 18 2006, 15:22:50) [GCC 3.4.4 20050721 (Red Hat 3.4.4-2)] From haase at msg.ucsf.edu Mon Jul 24 18:02:22 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 24 Jul 2006 15:02:22 -0700 Subject: [Numpy-discussion] weave - was:Re: how to get an array with "varying" poisson distribution In-Reply-To: References: <200607241231.46581.haase@msg.ucsf.edu> Message-ID: <200607241502.22529.haase@msg.ucsf.edu> On Monday 24 July 2006 14:20, Robert Kern wrote: > Sebastian Haase wrote: > > Hi, > > Essentially I'm looking for the equivalent of what was in numarray: > > from numarray import random_array > > random_array.poisson(arr) > > > > That is: if for example arr is a 256x256 array of positive integers, then > > this returns a new array of random numbers than are drawn according to > > the poisson statistics where arr's value at coordinate y,x determines > > the mean of the poisson distribution used to generate a new value for > > y,x. > > I'm afraid that at this point in time, the distributions only accept scalar > values for the parameters. I've thought about reimplementing the > distribution functions as ufuncs, but that's a hefty chunk of work that > won't happen for 1.0. > > I'm afraid that, for now, you're stuck with iterating over the values. Thanks for the reply - maybe this my time to get into weave ;-) Question about distributing and/or making "python-only-changes". How can one distribute modules using weave to other people who might NOT have a C compiler installed ? And further: when I change parts of that module that should not require a recompiling of the C part - is weave smart enough to recognise this ? Thanks, Sebastian (Oops, I just realized that this would be a question maybe for the SciPy list ... I'll assume that it's OK) From strawman at astraw.com Mon Jul 24 18:18:37 2006 From: strawman at astraw.com (Andrew Straw) Date: Mon, 24 Jul 2006 15:18:37 -0700 Subject: [Numpy-discussion] Please comment on SVN versioning convention In-Reply-To: References: <44C53671.5060301@ieee.org> Message-ID: <44C5473D.5090301@astraw.com> Sasha wrote: >On 7/24/06, Travis Oliphant wrote: > > >>Andrew Straw has emphasized that the current strategy of appending the >>SVN version number to development versions of the SVN tree makes it hard >>to do version sorting. >> >> > >I am not sure what the problem is, but if the concern is that > > > >>>>'0.9.9.2803' > '0.9.9' >>>> >>>> >True > >I suggest fixing that by appending '.final' to the release version string: > > The concern is that there are a bazillion ways of sorting version numbers but the version numbers from numpy's current svn tree don't sort with (m)any of them. Sorting strings in Python is one way, using setuptools or debian's dpkg are other ways. Remember, all this is only a concern for those working from SVN versions -- the releases are stripped of these numbers, anyway. I don't really care what we use, it's just that if we want to make life easy for people working from SVN, these automated tools should have a way of knowing that what we currently call numpy 1.1.2881 precedes numpy 1.1 (these are current examples from the SVN tree). Here are some proposals for this version: 1.0.2881 - This would sort after 1.0 (and 1.0b and 1.0c) and before 1.1 for most tools out there. 1.0.post2881 - This would sort after 1.0 (and 1.0b and 1.0c) and before 1.1 for most tools out there, but is perhaps more clear. 1.1.dev-r2881 - this wouldn't solve the problem for Debian, but it would for setuptools, which has special treatment for ".dev-r" (see http://peak.telecommunity.com/DevCenter/setuptools). This has the advantage of being only a minimal change from the current scheme. 1.0+1.1.2881 - I also think this would sort OK with most tools, and is also a fairly minimal change. My only concern is the '+' might cause some filesystems or tools to freak out. BTW, Fernando, all this is so that we can still have the svn revision number in __version__, but so that it doesn't screw up sorting. Cheers! Andrew From ndarray at mac.com Mon Jul 24 18:38:42 2006 From: ndarray at mac.com (Sasha) Date: Mon, 24 Jul 2006 18:38:42 -0400 Subject: [Numpy-discussion] Please comment on SVN versioning convention In-Reply-To: <44C5473D.5090301@astraw.com> References: <44C53671.5060301@ieee.org> <44C5473D.5090301@astraw.com> Message-ID: On 7/24/06, Andrew Straw wrote: [snip] > The concern is that there are a bazillion ways of sorting version > numbers but the version numbers from numpy's current svn tree don't sort > with (m)any of them. Sorting strings in Python is one way, using > setuptools or debian's dpkg are other ways. Sorting numbers as strings is a problem no matter what tools you use. '1000' is sorted before '999' in every tool I know of. The only proper way to solve this problem is to make svn revision available as an integer and sort them as integers. This integer may go in version_info tuple or in a separate variable. The same information may be encoded into a sortable string using appropriate padding, but I don't see much need for that. Do you expect creating debian's dpkg from a non-release version? From ndarray at mac.com Mon Jul 24 18:47:10 2006 From: ndarray at mac.com (Sasha) Date: Mon, 24 Jul 2006 18:47:10 -0400 Subject: [Numpy-discussion] Please comment on SVN versioning convention In-Reply-To: <44C5473D.5090301@astraw.com> References: <44C53671.5060301@ieee.org> <44C5473D.5090301@astraw.com> Message-ID: On 7/24/06, Andrew Straw wrote: > BTW, Fernando, all this is so that we can still have the svn revision > number in __version__, but so that it doesn't screw up sorting. Sorting __version__ is screwed up regardless of svn revision number: >>> '1.9' < '1.10' False From strawman at astraw.com Mon Jul 24 18:47:07 2006 From: strawman at astraw.com (Andrew Straw) Date: Mon, 24 Jul 2006 15:47:07 -0700 Subject: [Numpy-discussion] Please comment on SVN versioning convention In-Reply-To: References: <44C53671.5060301@ieee.org> <44C5473D.5090301@astraw.com> Message-ID: <44C54DEB.8010801@astraw.com> Sasha wrote: >On 7/24/06, Andrew Straw wrote: >[snip] > > >>The concern is that there are a bazillion ways of sorting version >>numbers but the version numbers from numpy's current svn tree don't sort >>with (m)any of them. Sorting strings in Python is one way, using >>setuptools or debian's dpkg are other ways. >> >> > >Sorting numbers as strings is a problem no matter what tools you use. >'1000' is sorted before '999' in every tool I know of. The only proper >way to solve this problem is to make svn revision available as an >integer and sort them as integers. This integer may go in version_info >tuple or in a separate variable. The same information may be encoded >into a sortable string using appropriate padding, but I don't see much >need for that. > Yes, I only brought up version numbers as strings because you did. I'm not proposing we solve that problem. I see setuptools, in particular, as the biggest thing to support, because it lets you have multiple versions installed simultaneously. And I don't know how to force its hand, or even if it's possible. >Do you expect creating debian's dpkg from a >non-release version? > > Yes. I do personally (see http://debs.astraw.com/dapper/ for example) and I'm sure others do, as well. You can see from that archive that I've been dealing with this issue by adding appropriate suffixes or prefixes. (Forcing its hand.) From strawman at astraw.com Mon Jul 24 19:03:13 2006 From: strawman at astraw.com (Andrew Straw) Date: Mon, 24 Jul 2006 16:03:13 -0700 Subject: [Numpy-discussion] Please comment on SVN versioning convention In-Reply-To: References: <44C53671.5060301@ieee.org> <44C5473D.5090301@astraw.com> Message-ID: <44C551B1.8030708@astraw.com> Sasha wrote: >On 7/24/06, Andrew Straw wrote: > > > >>BTW, Fernando, all this is so that we can still have the svn revision >>number in __version__, but so that it doesn't screw up sorting. >> >> > >Sorting __version__ is screwed up regardless of svn revision number: > > > >>>>'1.9' < '1.10' >>>> >>>> >False > Not if you parse __version__ with something that understands version numbers. From ndarray at mac.com Mon Jul 24 19:09:15 2006 From: ndarray at mac.com (Sasha) Date: Mon, 24 Jul 2006 19:09:15 -0400 Subject: [Numpy-discussion] Please comment on SVN versioning convention In-Reply-To: <44C54DEB.8010801@astraw.com> References: <44C53671.5060301@ieee.org> <44C5473D.5090301@astraw.com> <44C54DEB.8010801@astraw.com> Message-ID: On 7/24/06, Andrew Straw wrote: > Yes, I only brought up version numbers as strings because you did. I'm > not proposing we solve that problem. I see setuptools, in particular, as > the biggest thing to support, because it lets you have multiple versions > installed simultaneously. And I don't know how to force its hand, or > even if it's possible. I am not familiar with setuptools and I expect that many others who care about __version__ conventions are not either. Please explain exactly how setuptools depend on what is stored in __version__ and why using 'final' in place of svn revision number in file names will not solve your problems. I would be against any change that will affect map(int, __version__.split('.')[:3]). From cookedm at physics.mcmaster.ca Mon Jul 24 19:10:22 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Mon, 24 Jul 2006 19:10:22 -0400 Subject: [Numpy-discussion] Please comment on SVN versioning convention In-Reply-To: <44C551B1.8030708@astraw.com> References: <44C53671.5060301@ieee.org> <44C5473D.5090301@astraw.com> <44C551B1.8030708@astraw.com> Message-ID: <20060724231022.GA10758@arbutus.physics.mcmaster.ca> On Mon, Jul 24, 2006 at 04:03:13PM -0700, Andrew Straw wrote: > Sasha wrote: > > >On 7/24/06, Andrew Straw wrote: > > > >>BTW, Fernando, all this is so that we can still have the svn revision > >>number in __version__, but so that it doesn't screw up sorting. > > > >Sorting __version__ is screwed up regardless of svn revision number: > > > >>>>'1.9' < '1.10' > >>>> > >>>> > >False > > > Not if you parse __version__ with something that understands version > numbers. ...and we've already got something for that: distutils.version.LooseVersion (or StrictVersion). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From wbaxter at gmail.com Mon Jul 24 19:30:22 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 25 Jul 2006 08:30:22 +0900 Subject: [Numpy-discussion] numpy.prod() vs. numpy.product() In-Reply-To: <200607241137.36094.haase@msg.ucsf.edu> References: <200607241137.36094.haase@msg.ucsf.edu> Message-ID: I can't answer why, but in oldnumeric.py, you can see there's two different fuctions with basically identical code, so yes they are distinct, but no they are not different. Seems like prod could be replaced by prod=product. def product (x, axis=0, dtype=None): """Product of the array elements over the given axis.""" try: prod = x.prod except AttributeError: return _wrapit(x, 'prod', axis, dtype) return prod(axis, dtype) def prod(a, axis=0, dtype=None): """Return the product of the elements along the given axis """ try: prod = a.prod except AttributeError: return _wrapit(a, 'prod', axis, dtype) return prod(axis, dtype) --bb On 7/25/06, Sebastian Haase wrote: > Hi, > Are numpy.product() and numpy.prod() > doing the exact same thing ? If yes, why are they pointing to two different > functions ? > >>> N.prod > > >>> N.product > > > Thanks, > Sebastian Haase > From wbaxter at gmail.com Mon Jul 24 19:42:18 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 25 Jul 2006 08:42:18 +0900 Subject: [Numpy-discussion] trivial question: how to compare dtype - but ignoring byteorder ? In-Reply-To: <200607241110.07686.haase@msg.ucsf.edu> References: <44C450CA.3010609@msg.ucsf.edu> <200607241218.55965.karol.langner@kn.pl> <200607241110.07686.haase@msg.ucsf.edu> Message-ID: > > And I think byteorder matters when comparing dtypes: > > >>> numpy.dtype('>f4') == numpy.dtype(' > False Ohhhhh -- that '<' part is indicating *byte order* ?! I thought it was odd that numpy could only tell me the type was "less than f4", which I assumed must be shorthand for "less than or equal to f4". Makes much more sense now! --bb From wbaxter at gmail.com Mon Jul 24 19:49:11 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 25 Jul 2006 08:49:11 +0900 Subject: [Numpy-discussion] matrix-related bug in 1.0b1 In-Reply-To: <44C52BAE.1090508@gmx.net> References: <44C0EDD8.3000701@gmx.net> <44C4B3E7.6060104@gmx.net> <44C5219A.8020706@ieee.org> <44C52BAE.1090508@gmx.net> Message-ID: On 7/25/06, Sven Schreiber wrote: > Travis Oliphant schrieb: > > Sven Schreiber wrote: > >> > > The change was trying to fix up some cases but did break this one. The > > problem is that figuring out whether or not to transpose the result is a > > bit tricky. I've obviously still got it wrong. Is there some test case that demonstrates the problem it was intended to fix? Or can you remember off the top of your head? I tried as many things as I could think of but nothing I tried gave the wrong answer with the fix of removing the 'not' on line 140. > p.s.: Is there going to be a new beta soon? Or is there a quick fix to > this slicing problem in the meantime? Thanks. Sven, if you were happy with the behavior under numpy 0.9.8 then all you have to do is just cut and paste its matrix.__getitem__ method into your current site-packages/numpy/core/defmatrix.py. --bill From myeates at jpl.nasa.gov Mon Jul 24 20:10:19 2006 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Mon, 24 Jul 2006 17:10:19 -0700 Subject: [Numpy-discussion] 1.0b1 problem on Solaris Message-ID: <44C5616B.8020209@jpl.nasa.gov> I installed numpy-1.0b1 and now >import numpy produces a segv. According to gdb it's happening after a failure to import scipy.optimize.minpack2. Eventually, Py_FatalError is called followed by abort,raise and thr_kill. Then the SEGV. Any ideas? Do I need a different version of scipy? I currently use 0.4.9 Mathew From haase at msg.ucsf.edu Mon Jul 24 20:31:53 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 24 Jul 2006 17:31:53 -0700 Subject: [Numpy-discussion] trivial question: how to compare dtype - but ignoring byteorder ? In-Reply-To: References: <44C450CA.3010609@msg.ucsf.edu> <200607241110.07686.haase@msg.ucsf.edu> Message-ID: <200607241731.54174.haase@msg.ucsf.edu> On Monday 24 July 2006 16:42, Bill Baxter wrote: > > > And I think byteorder matters when comparing dtypes: > > > >>> numpy.dtype('>f4') == numpy.dtype(' > > > > > False > > Ohhhhh -- that '<' part is indicating *byte order* ?! > I thought it was odd that numpy could only tell me the type was "less > than f4", which I assumed must be shorthand for "less than or equal to > f4". Makes much more sense now! > > --bb Which is why I was trying to change the str() representation of a type to something more intuitive. If nothing else one could even leave repr(a.dtype) --> ' 'int32 (little endian)' I do now understand that (as opposed to numarray and numeric) the byteorder is now part of the data-type - but I would really encourage keeping the string for such an important (and often used !) thing more readable than " I'm trying to work with memmaps on very large files, i.e. > 2 GB, up to 10 GB. The files are data cubes of images (my largest is 1290(x)x1024(y)x2011(z)) and my immediate task is to strip the data from 32-bits down to 16, and to rearrange some of the data on a per-xy-plane basis. I'm running this on a Fedora Core 5 64-bit system, with python-2.5b2(that I believe I compiled in 64-bit mode) and numpy-1.0b1. The disk has 324 GB free space. The log from a minimal case is as follows: ressler > python2.5 Python 2.5b2 (r25b2:50512, Jul 18 2006, 12:58:29) [GCC 4.1.1 20060525 (Red Hat 4.1.1-1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> data=np.memmap('temp_file',mode='w+',shape=(2011,1280,1032),dtype='h') size = 2656450560 bytes = 5312901120 len(mm) = 5312901120 (2011, 1280, 1032) h 0 0 Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.5/site-packages/numpy/core/memmap.py", line 75, in __new__ offset=offset, order=order) TypeError: buffer is too small for requested array >>> If I have a small number of frames (z=800 rather than 2011), this all works fine. I've added a few lines to memmap.py to print some diagnostic information - the error occurs on line 71 in the original memmap.py file, not 75. The "size =" and "bytes =" lines show that memmap.py is calculating the correct size for the buffer, and the len(mm) shows that the python mmap.mmap call on line 67 is returning a buffer of the correct size. The "(2011, 1280, 1032) h 0 0" bit is from a print statement that was left in the source file by the authors, and indicates what the following "self = ndarray.__new__" call is trying to do. However, it is the ndarray.__new__ call that is breaking down, and I don't really have enough skill to continue chasing it down. I took a quick look at the C source, but I couldn't figure out where the ndarray.__new__ is actually defined. Any suggestions to help me past this? Thanks. Mike -- mike.ressler at alum.mit.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Mon Jul 24 20:43:46 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 24 Jul 2006 17:43:46 -0700 Subject: [Numpy-discussion] trivial question: how to compare dtype - but ignoring byteorder ? In-Reply-To: <200607241731.54174.haase@msg.ucsf.edu> References: <44C450CA.3010609@msg.ucsf.edu> <200607241110.07686.haase@msg.ucsf.edu> <200607241731.54174.haase@msg.ucsf.edu> Message-ID: <44C56942.9050908@noaa.gov> Sebastian Haase wrote: > Which is why I was trying to change the str() representation of a type to > something more intuitive. > If nothing else one could even leave > repr(a.dtype) --> ' but > str(a.dtype) --> 'int32 (little endian)' +1 that's the whole point of __str__. It should be human readable! -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From wbaxter at gmail.com Mon Jul 24 21:16:33 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 25 Jul 2006 10:16:33 +0900 Subject: [Numpy-discussion] Please comment on SVN versioning convention In-Reply-To: <44C5473D.5090301@astraw.com> References: <44C53671.5060301@ieee.org> <44C5473D.5090301@astraw.com> Message-ID: On 7/25/06, Andrew Straw wrote: > 1.0.2881 - This would sort after 1.0 (and 1.0b and 1.0c) and before 1.1 > for most tools out there. I like that best. Save the 1.1 prefix until it's actually released as such. The numbering scheme needs to deal with what to call the patch tip of the stable release branch, too. I.e. the maintenance release after 1.0 with only critical patches for anyone who demands strict 1.0 compatibility. The third number would be good for that. Right now I see 1.1.2881 as my version. I think this should probably have an extra .0 in it: 1.1.0.2881. That third number would be a maintenance release number. I also like the Linux even-odd system. Then 1.0.0.1234 can mean "SVN version on the 1.0 branch, leading up to maintenance release 1.0.1", and 1.1.0.1234 can mean "SVN version on the trunk, leading up to 1.2 release." --bb From wbaxter at gmail.com Mon Jul 24 21:20:45 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 25 Jul 2006 10:20:45 +0900 Subject: [Numpy-discussion] 1.0b1 problem on Solaris In-Reply-To: <44C5616B.8020209@jpl.nasa.gov> References: <44C5616B.8020209@jpl.nasa.gov> Message-ID: Yep, scipy needs to be recompiled against the 1.0b1 version of numpy. I think they should have an updated version on the web site in a few days or so if you don't feel like compiling yourself. --bb On 7/25/06, Mathew Yeates wrote: > I installed numpy-1.0b1 and now >import numpy produces a segv. According > to gdb it's happening after a failure to import scipy.optimize.minpack2. > Eventually, Py_FatalError is called followed by abort,raise and > thr_kill. Then the SEGV. > > Any ideas? Do I need a different version of scipy? I currently use 0.4.9 > > > Mathew > From myeates at jpl.nasa.gov Mon Jul 24 21:30:31 2006 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Mon, 24 Jul 2006 18:30:31 -0700 Subject: [Numpy-discussion] 1.0b1 problem on Solaris In-Reply-To: References: <44C5616B.8020209@jpl.nasa.gov> Message-ID: <44C57437.40607@jpl.nasa.gov> Where is the source? Do I just use the 0.4.9 source? Bill Baxter wrote: > Yep, scipy needs to be recompiled against the 1.0b1 version of numpy. > I think they should have an updated version on the web site in a few > days or so if you don't feel like compiling yourself. > > --bb > > On 7/25/06, Mathew Yeates wrote: >> I installed numpy-1.0b1 and now >import numpy produces a segv. According >> to gdb it's happening after a failure to import scipy.optimize.minpack2. >> Eventually, Py_FatalError is called followed by abort,raise and >> thr_kill. Then the SEGV. >> >> Any ideas? Do I need a different version of scipy? I currently use 0.4.9 >> >> >> Mathew >> > From wbaxter at gmail.com Mon Jul 24 21:47:28 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 25 Jul 2006 10:47:28 +0900 Subject: [Numpy-discussion] 1.0b1 problem on Solaris In-Reply-To: <44C57437.40607@jpl.nasa.gov> References: <44C5616B.8020209@jpl.nasa.gov> <44C57437.40607@jpl.nasa.gov> Message-ID: Howdy, Instructions for compiling your own SciPy can be found here. http://www.scipy.org/Installing_SciPy I think the 0.4.9 source will probably work. But getting it from SVN is probably safer. --bb On 7/25/06, Mathew Yeates wrote: > Where is the source? Do I just use the 0.4.9 source? > > > Bill Baxter wrote: > > Yep, scipy needs to be recompiled against the 1.0b1 version of numpy. > > I think they should have an updated version on the web site in a few > > days or so if you don't feel like compiling yourself. > > > > --bb > > > > On 7/25/06, Mathew Yeates wrote: > >> I installed numpy-1.0b1 and now >import numpy produces a segv. According > >> to gdb it's happening after a failure to import scipy.optimize.minpack2. > >> Eventually, Py_FatalError is called followed by abort,raise and > >> thr_kill. Then the SEGV. > >> > >> Any ideas? Do I need a different version of scipy? I currently use 0.4.9 > >> > >> > >> Mathew > >> > > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From yukihana at yahoo.co.jp Mon Jul 24 21:49:59 2006 From: yukihana at yahoo.co.jp (=?iso-2022-jp?B?eXVraWhhbmE=?=) Date: Tue, 25 Jul 2006 01:49:59 -0000 Subject: [Numpy-discussion] (no subject) Message-ID: :?? INFORMATION ?????????????????????????: ?????????????????????? ???????????? http://love-match.bz/pc/?02 :??????????????????????????????????: *????*:.?. .?.:*????*:.?..?:*????*:.?..?:**????* ??????????????????????????????????? ??? ???????????????????Love?Match? ?----------------------------------------------------------------- ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ?----------------------------------------------------------------- ????????????????http://love-match.bz/pc/?02 ??????????????????????????????????? ??? ?????????????????????? ?----------------------------------------------------------------- ???????????????????????????? ?----------------------------------------------------------------- ????????????????????????????? ??????????????????????????????? ?http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????????????????????????????? ?----------------------------------------------------------------- ???????????????????????????????? ????????????????????? ?http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????????????????? ?----------------------------------------------------------------- ???????????????????????? ?????????????????????????????????? ?http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????????????????????????? ?----------------------------------------------------------------- ??????????????????????????? ????????????????????????????????? ?http://love-match.jp/pc/?06 ?----------------------------------------------------------------- ????????????????????????? ?----------------------------------------------------------------- ????????????????????????? ????????????????????????????????? ?http://love-match.jp/pc/?06 ??????????????????????????????????? ??? ??500???????????????? ?----------------------------------------------------------------- ???????/???? ???????????????????? ????????????????????????????????? ???????????????????????????????? ?????????????????????????? ?????????????????????????????? ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????/?????? ?????????????????????????????????? ??????????????????????????????????? ?????????? ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????/????? ?????????????????????????????????? ???????????????????????????????? ?????????????????????????(^^) ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????/???? ??????????????????????????????? ?????????????????????????????? ?????????????????????????????? ???????? ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ????????/??? ???????????????1??? ????????????????????????? ????????????????????????? ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????/??????? ????18?????????????????????????? ????????????????????????????? ????????????????????????????? ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???`????/??? ????????????????????? ?????????????????????? ?????????????? ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????????????????? ?????????????????????????????????? ????????????? ??------------------------------------------------------------- ???????????????????????????????? ??[??????????]?http://love-match.bz/pc/?02 ??------------------------------------------------------------- ????????????????????? ??????????????????????????? ??????????????????? ??????????????????????????????? ??[??????????]?http://love-match.bz/pc/?02 ?????????????????????????????????? ???????????? ??????????3-6-4-533 ?????? 139-3668-7892 From charlesr.harris at gmail.com Mon Jul 24 23:42:31 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 24 Jul 2006 21:42:31 -0600 Subject: [Numpy-discussion] how to get an array with "varying" poisson distribution In-Reply-To: <200607241231.46581.haase@msg.ucsf.edu> References: <200607241231.46581.haase@msg.ucsf.edu> Message-ID: Hi Sebastian, On 7/24/06, Sebastian Haase wrote: > > Hi, > Essentially I'm looking for the equivalent of what was in numarray: > from numarray import random_array > random_array.poisson(arr) > > That is: if for example arr is a 256x256 array of positive integers, then > this > returns a new array of random numbers than are drawn according to the > poisson > statistics where arr's value at coordinate y,x determines the mean of > the > poisson distribution used to generate a new value for y,x. > > [[This is needed e.g. to simulate quantum noise in CCD images. Each pixel > has > different amount of noise depending of what it's (noise-free) "input" > value > was.]] How accurate do you want the distribution to be and what sort of offset is there? If the number of counts is greater that about 10 and you don't care too much about the poisson tail then a gaussian will work fine. If you have very large counts (>1e6), which I doubt, the accuracy of the poisson distribution becomes a tricky matter because it needs care to compute the factorial to the needed accuracy. The factorial turns up in the rejection methods used in the algorithms. Most of the algorithms also compute special constants that depend on the mean of the distribution, so while efficient for a fixed mean they are less so for varying means like you want. I tend to just go with gaussian noise most of the time. Thanks, > Sebastian Haase > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share > your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From emi0924 at yahoo.co.jp Tue Jul 25 01:00:35 2006 From: emi0924 at yahoo.co.jp (=?iso-2022-jp?B?ZW1p?=) Date: Tue, 25 Jul 2006 05:00:35 -0000 Subject: [Numpy-discussion] (no subject) Message-ID: :?? INFORMATION ?????????????????????????: ?????????????????????? ???????????? http://love-match.bz/pc/?010 :??????????????????????????????????: *????*:.?. .?.:*????*:.?..?:*????*:.?..?:**????* ?????????????????????????????? ??[??????????]?http://love-match.bz/pc/?010 ??????????????????????????????????? ??? ???????????????????Love?Match? ?----------------------------------------------------------------- ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ?----------------------------------------------------------------- ????????????????http://love-match.bz/pc/?010 ??????????????????????????????????? ??? ?????????????????????? ?----------------------------------------------------------------- ???????????????????????????? ?----------------------------------------------------------------- ????????????????????????????? ??????????????????????????????? ?http://love-match.bz/pc/?010 ?----------------------------------------------------------------- ???????????????????????????????? ?----------------------------------------------------------------- ???????????????????????????????? ????????????????????? ?http://love-match.bz/pc/?010 ?----------------------------------------------------------------- ???????????????????? ?----------------------------------------------------------------- ???????????????????????? ?????????????????????????????????? ?http://love-match.bz/pc/?010 ?----------------------------------------------------------------- ???????????????????????????? ?----------------------------------------------------------------- ??????????????????????????? ????????????????????????????????? ?http://love-match.bz/pc/?010 ?----------------------------------------------------------------- ????????????????????????? ?----------------------------------------------------------------- ????????????????????????? ????????????????????????????????? ?http://love-match.bz/pc/?010 ??????????????????????????????????? ??? ??500???????????????? ?----------------------------------------------------------------- ???????/???? ???????????????????? ????????????????????????????????? ???????????????????????????????? ?????????????????????????? ?????????????????????????????? ?[????] http://love-match.bz/pc/?010 ?----------------------------------------------------------------- ???????/?????? ?????????????????????????????????? ??????????????????????????????????? ?????????? ?[????] http://love-match.bz/pc/?010 ?----------------------------------------------------------------- ???????/????? ?????????????????????????????????? ???????????????????????????????? ?????????????????????????(^^) ?[????] http://love-match.bz/pc/?010 ?----------------------------------------------------------------- ???????/???? ??????????????????????????????? ?????????????????????????????? ?????????????????????????????? ???????? ?[????] http://love-match.bz/pc/?010 ?----------------------------------------------------------------- ????????/??? ???????????????1??? ????????????????????????? ????????????????????????? ?[????] http://love-match.bz/pc/?010 ?----------------------------------------------------------------- ???????/??????? ????18?????????????????????????? ????????????????????????????? ????????????????????????????? ?[????] http://love-match.bz/pc/?010 ?----------------------------------------------------------------- ???`????/??? ????????????????????? ?????????????????????? ?????????????? ?[????] http://love-match.bz/pc/?010 ?----------------------------------------------------------------- ???????????????????? ?????????????????????????????????? ????????????? ??------------------------------------------------------------- ???????????????????????????????? ??[??????????]?http://love-match.bz/pc/?010 ??------------------------------------------------------------- ????????????????????? ??????????????????????????? ??????????????????? ??????????????????????????????? ??[??????????]?http://love-match.bz/pc/?010 ?????????????????????????????????? ??????????3-6-4-533 ?????? 139-3668-7892 From haase at msg.ucsf.edu Tue Jul 25 01:09:37 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 24 Jul 2006 22:09:37 -0700 Subject: [Numpy-discussion] numpy scalar types Message-ID: <44C5A791.1@msg.ucsf.edu> Hi, I just ran a simple test which I think would be of general interest. It's about types and what there names are in the numpy module (and how many different names there are for a given type !!) Cheers, Sebastian Haase (maybe there is a good place in the wiki for this !?) >>> N.csingle >>> N.cdouble >>> N.cfloat and all together !! >>> for tt in N.ScalarType: ... l=[k for k,v in N.__dict__.iteritems() if v == tt] ... print len(l), tt, l ... 0 [] 0 [] 0 [] 0 [] 0 [] 0 [] 0 [] 0 [] 2 ['longlong', 'int64'] 2 ['int16', 'short'] 2 ['int32', 'int_'] 2 ['uint', 'uint32'] 2 ['unicode_', 'unicode0'] 2 ['complex64', 'csingle'] 3 ['intp', 'intc', 'int0'] 4 ['cfloat', 'complex_', 'cdouble', 'complex128'] 3 ['string', 'str_', 'string0'] 3 ['float96', 'longfloat', 'longdouble'] 2 ['ushort', 'uint16'] 2 ['object0', 'object_'] 3 ['double', 'float64', 'float_'] 2 ['byte', 'int8'] 2 ['uint8', 'ubyte'] 2 ['bool8', 'bool_'] 2 ['float32', 'single'] 3 ['complex192', 'clongdouble', 'clongfloat'] 3 ['uintc', 'uint0', 'uintp'] 2 ['uint64', 'ulonglong'] 2 ['void', 'void0'] From djm at mindrot.org Tue Jul 25 01:33:55 2006 From: djm at mindrot.org (Damien Miller) Date: Tue, 25 Jul 2006 15:33:55 +1000 (EST) Subject: [Numpy-discussion] I've created a 1.0b1 release tag in SVN In-Reply-To: <44C090F5.7000007@ieee.org> References: <44C090F5.7000007@ieee.org> Message-ID: On Fri, 21 Jul 2006, Travis Oliphant wrote: > > I've created the 1.0b1 release tag in SVN and will be uploading files > shortly to Sourceforge. FYI numpy-1.0b1 builds fine and passes all its regression tests on OpenBSD -current. -d From oliphant.travis at ieee.org Tue Jul 25 02:47:06 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 25 Jul 2006 00:47:06 -0600 Subject: [Numpy-discussion] ***[Possible UCE]*** Bug in memmap/python allocation code? In-Reply-To: <268febdf0607241736g5f5135a3ub26acd3e8129fec6@mail.gmail.com> References: <268febdf0607241736g5f5135a3ub26acd3e8129fec6@mail.gmail.com> Message-ID: <44C5BE6A.7020007@ieee.org> Mike Ressler wrote: > I'm trying to work with memmaps on very large files, i.e. > 2 GB, up > to 10 GB. The files are data cubes of images (my largest is > 1290(x)x1024(y)x2011(z)) and my immediate task is to strip the data > from 32-bits down to 16, and to rearrange some of the data on a > per-xy-plane basis. I'm running this on a Fedora Core 5 64-bit system, > with python-2.5b2 (that I believe I compiled in 64-bit mode) and > numpy-1.0b1. The disk has 324 GB free space. I just discovered the problem. All the places where PyObject_AsBuffer is used needs to have the final argument changed to Py_ssize_t (which in arrayobject.h is defined as int if you are using less than Python 2.5). This should be fixed in SVN shortly.... -Travis From karol.langner at kn.pl Tue Jul 25 02:56:05 2006 From: karol.langner at kn.pl (Karol Langner) Date: Tue, 25 Jul 2006 08:56:05 +0200 Subject: [Numpy-discussion] trivial question: how to compare dtype - but ignoring byteorder ? In-Reply-To: References: <44C450CA.3010609@msg.ucsf.edu> <200607241110.07686.haase@msg.ucsf.edu> Message-ID: <200607250856.05702.karol.langner@kn.pl> On Tuesday 25 July 2006 01:42, Bill Baxter wrote: > > > And I think byteorder matters when comparing dtypes: > > > >>> numpy.dtype('>f4') == numpy.dtype(' > > > > > False > > Ohhhhh -- that '<' part is indicating *byte order* ?! > I thought it was odd that numpy could only tell me the type was "less > than f4", which I assumed must be shorthand for "less than or equal to > f4". Makes much more sense now! > > --bb Yep! And there are then four possiblities. '>' - big-endian '<' - little '|' - not-applicable '=' - native Karol -- written by Karol Langner wto lip 25 08:54:51 CEST 2006 From karol.langner at kn.pl Tue Jul 25 02:59:11 2006 From: karol.langner at kn.pl (Karol Langner) Date: Tue, 25 Jul 2006 08:59:11 +0200 Subject: [Numpy-discussion] Bug in memmap/python allocation code? In-Reply-To: <268febdf0607241736g5f5135a3ub26acd3e8129fec6@mail.gmail.com> References: <268febdf0607241736g5f5135a3ub26acd3e8129fec6@mail.gmail.com> Message-ID: <200607250859.11840.karol.langner@kn.pl> On Tuesday 25 July 2006 02:36, Mike Ressler wrote: > I'm trying to work with memmaps on very large files, i.e. > 2 GB, up to 10 > GB. The files are data cubes of images (my largest is > 1290(x)x1024(y)x2011(z)) and my immediate task is to strip the data from > 32-bits down to 16, and to rearrange some of the data on a per-xy-plane > basis. I'm running this on a Fedora Core 5 64-bit system, with > python-2.5b2(that I believe I compiled in 64-bit mode) and > numpy-1.0b1. The disk has 324 GB free space. > > The log from a minimal case is as follows: > > ressler > python2.5 > Python 2.5b2 (r25b2:50512, Jul 18 2006, 12:58:29) > [GCC 4.1.1 20060525 (Red Hat 4.1.1-1)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > > >>> import numpy as np > >>> data=np.memmap('temp_file',mode='w+',shape=(2011,1280,1032),dtype='h') > > size = 2656450560 > bytes = 5312901120 > len(mm) = 5312901120 > (2011, 1280, 1032) h 0 0 > Traceback (most recent call last): > File "", line 1, in > File "/usr/local/lib/python2.5/site-packages/numpy/core/memmap.py", line > 75, in __new__ > offset=offset, order=order) > TypeError: buffer is too small for requested array > > > If I have a small number of frames (z=800 rather than 2011), this all works > fine. I've added a few lines to memmap.py to print some diagnostic > information - the error occurs on line 71 in the original memmap.py file, > not 75. The "size =" and "bytes =" lines show that memmap.py is calculating > the correct size for the buffer, and the len(mm) shows that the python > mmap.mmap call on line 67 is returning a buffer of the correct size. The > "(2011, 1280, 1032) h 0 0" bit is from a print statement that was left in > the source file by the authors, and indicates what the following "self = > ndarray.__new__" call is trying to do. However, it is the ndarray.__new__ > call that is breaking down, and I don't really have enough skill to > continue chasing it down. I took a quick look at the C source, but I > couldn't figure out where the ndarray.__new__ is actually defined. > > Any suggestions to help me past this? Thanks. > > Mike I know Travis has nswered in a different thread. Let me jsut add where the actual error is raised - maybe it will be of some use. It is around line 5490 of arrayobject.c (procedure array_new): else { /* buffer given -- use it */ if (dims.len == 1 && dims.ptr[0] == -1) { dims.ptr[0] = (buffer.len-(intp)offset) / itemsize; } else if ((strides.ptr == NULL) && \ buffer.len < itemsize* \ PyArray_MultiplyList(dims.ptr, dims.len)) { PyErr_SetString(PyExc_TypeError, "buffer is too small for " \ "requested array"); goto fail; } So it does look like an overflow to me. Karol -- written by Karol Langner wto lip 25 08:56:42 CEST 2006 From hitomi0303 at yahoo.co.jp Tue Jul 25 05:06:26 2006 From: hitomi0303 at yahoo.co.jp (=?iso-2022-jp?B?aGl0b21p?=) Date: Tue, 25 Jul 2006 09:06:26 -0000 Subject: [Numpy-discussion] (no subject) Message-ID: :?? INFORMATION ?????????????????????????: ?????????????????????? ???????????? http://love-match.bz/pc/?09 :??????????????????????????????????: *????*:.?. .?.:*????*:.?..?:*????*:.?..?:**????* ?????????????????????????????? ??[??????????]?http://love-match.bz/pc/?09 ??????????????????????????????????? ??? ???????????????????Love?Match? ?----------------------------------------------------------------- ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ?----------------------------------------------------------------- ????????????????http://love-match.bz/pc/?09 ??????????????????????????????????? ??? ?????????????????????? ?----------------------------------------------------------------- ???????????????????????????? ?----------------------------------------------------------------- ????????????????????????????? ??????????????????????????????? ?http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????????????????????????????? ?----------------------------------------------------------------- ???????????????????????????????? ????????????????????? ?http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????????????????? ?----------------------------------------------------------------- ???????????????????????? ?????????????????????????????????? ?http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????????????????????????? ?----------------------------------------------------------------- ??????????????????????????? ????????????????????????????????? ?http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ????????????????????????? ?----------------------------------------------------------------- ????????????????????????? ????????????????????????????????? ?http://love-match.bz/pc/?09 ??????????????????????????????????? ??? ??500???????????????? ?----------------------------------------------------------------- ???????/???? ???????????????????? ????????????????????????????????? ???????????????????????????????? ?????????????????????????? ?????????????????????????????? ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????/?????? ?????????????????????????????????? ??????????????????????????????????? ?????????? ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????/????? ?????????????????????????????????? ???????????????????????????????? ?????????????????????????(^^) ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????/???? ??????????????????????????????? ?????????????????????????????? ?????????????????????????????? ???????? ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ????????/??? ???????????????1??? ????????????????????????? ????????????????????????? ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????/??????? ????18?????????????????????????? ????????????????????????????? ????????????????????????????? ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???`????/??? ????????????????????? ?????????????????????? ?????????????? ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????????????????? ?????????????????????????????????? ????????????? ??------------------------------------------------------------- ???????????????????????????????? ??[??????????]?http://love-match.bz/pc/?09 ??------------------------------------------------------------- ????????????????????? ??????????????????????????? ??????????????????? ??????????????????????????????? ??[??????????]?http://love-match.bz/pc/?09 ?????????????????????????????????? ??????????3-6-4-533 ?????? 139-3668-7892 From jg307 at cam.ac.uk Tue Jul 25 05:53:23 2006 From: jg307 at cam.ac.uk (James Graham) Date: Tue, 25 Jul 2006 10:53:23 +0100 Subject: [Numpy-discussion] how to get an array with "varying" poisson distribution In-Reply-To: References: <200607241231.46581.haase@msg.ucsf.edu> Message-ID: <44C5EA13.60301@cam.ac.uk> Robert Kern wrote: > Sebastian Haase wrote: >> Hi, >> Essentially I'm looking for the equivalent of what was in numarray: >> from numarray import random_array >> random_array.poisson(arr) >> >> That is: if for example arr is a 256x256 array of positive integers, then this >> returns a new array of random numbers than are drawn according to the poisson >> statistics where arr's value at coordinate y,x determines the mean of the >> poisson distribution used to generate a new value for y,x. > > I'm afraid that at this point in time, the distributions only accept scalar > values for the parameters. I've thought about reimplementing the distribution > functions as ufuncs, but that's a hefty chunk of work that won't happen for 1.0. FWIW, I've had enquires about the availability, or not, of this functionality in NumPy as well, so when someone does have time to work on it, it will be very much appreciated. -- "You see stars that clear have been dead for years But the idea just lives on..." -- Bright Eyes From lfriedri at imtek.de Tue Jul 25 06:49:57 2006 From: lfriedri at imtek.de (Lars Friedrich) Date: Tue, 25 Jul 2006 12:49:57 +0200 Subject: [Numpy-discussion] ctypes, numpy-array Message-ID: <1153824597.6593.25.camel@localhost> Hello, I would like to work with some data using python/numpy. The data is generated with C. To bring the data from C to python, I would like to use ctype. I am using python 2.4.3 on a windows-System. To accomplish the described task, I have the following plan. Please tell me, if this is possible, respectively if there is a better way. 1) In C, I write a .dll, that has a function int foo(PyObject *arg) 2) In python, I generate a numpy-array with the appropriate size. e.g. a = zeros((3,3), dtype=int) 3) From python, I call my .dll-function with a as an argument: windll.mydll.foo(a) 4) In the foo-function in the C-.dll I cast the pointer and access the data-field. PyArrayObject *myPtr = (PyArrayObject*) arg; myPtr->data[0] = 1; return 0; However, when I do this, I get an "AccessViolationError writing 0x000000000" What can I do about it? Thank you for every comment Lars -- Dipl.-Ing. Lars Friedrich Optical Measurement Technology Department of Microsystems Engineering -- IMTEK University of Freiburg Georges-K?hler-Allee 102 D-79110 Freiburg Germany phone: +49-761-203-7531 fax: +49-761-203-7537 room: 01 088 email: lfriedri at imtek.de From fullung at gmail.com Tue Jul 25 07:09:28 2006 From: fullung at gmail.com (Albert Strasheim) Date: Tue, 25 Jul 2006 13:09:28 +0200 Subject: [Numpy-discussion] ctypes, numpy-array In-Reply-To: <1153824597.6593.25.camel@localhost> Message-ID: <02f401c6afda$cba1aea0$0100a8c0@dsp.sun.ac.za> Hey Lars > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of Lars Friedrich > Sent: 25 July 2006 12:50 > To: numpy-discussion at lists.sourceforge.net > Subject: [Numpy-discussion] ctypes, numpy-array > > Hello, > > I would like to work with some data using python/numpy. The data is > generated with C. To bring the data from C to python, I would like to > use ctype. I am using python 2.4.3 on a windows-System. > > To accomplish the described task, I have the following plan. Please tell > me, if this is possible, respectively if there is a better way. > > 1) In C, I write a .dll, that has a function > int foo(PyObject *arg) > > 2) In python, I generate a numpy-array with the appropriate size. e.g. > a = zeros((3,3), dtype=int) > > 3) From python, I call my .dll-function with a as an argument: > windll.mydll.foo(a) What's might be happening here is that a.ctypes.data is in fact being passed to your function via ctypes's from_param magic (check the ctypes tutorial for details). In [10]: x = N.array([]) In [11]: x.ctypes.data Out[11]: c_void_p(15502816) In [12]: x._as_parameter_ Out[12]: 15502816 > 4) In the foo-function in the C-.dll I cast the pointer and access the > data-field. > PyArrayObject *myPtr = (PyArrayObject*) arg; > myPtr->data[0] = 1; > return 0; > > However, when I do this, I get an "AccessViolationError writing > 0x000000000" So what's probably happening here is that you already have a pointer to the array's data which you then cast to a PyArrayObject pointer. Dereferencing myPtr->data is looking for a pointer inside the array's data, which contains zeros. Here's a few things to try: - look at ctypes's PyDLL option if you want to pass around Python objects - Write your function as: int foo(int* x); Then do something like this: x = N.array([...], dtype=N.intc) mydll.foo.restype = None mydll.foo.argtypes = [POINTER(c_int)] mydll.foo(x.ctypes.data) This might also work: x = N.array([...], dtype=N.intc) mydll.foo.restype = None mydll.foo(x) Cheers, Albert From lfriedri at imtek.de Tue Jul 25 07:55:09 2006 From: lfriedri at imtek.de (Lars Friedrich) Date: Tue, 25 Jul 2006 13:55:09 +0200 Subject: [Numpy-discussion] ctypes, numpy-array In-Reply-To: <02f401c6afda$cba1aea0$0100a8c0@dsp.sun.ac.za> References: <02f401c6afda$cba1aea0$0100a8c0@dsp.sun.ac.za> Message-ID: <1153828509.6593.30.camel@localhost> > What's might be happening here is that a.ctypes.data is in fact being passed > to your function via ctypes's from_param magic (check the ctypes tutorial > for details). > > In [10]: x = N.array([]) > > In [11]: x.ctypes.data > Out[11]: c_void_p(15502816) > > In [12]: x._as_parameter_ > Out[12]: 15502816 > OK, I did not know about x.ctypes.data... > So what's probably happening here is that you already have a pointer to the > array's data which you then cast to a PyArrayObject pointer. Dereferencing > myPtr->data is looking for a pointer inside the array's data, which contains > zeros. I understand. > - look at ctypes's PyDLL option if you want to pass around Python objects ??? > - Write your function as: > > int foo(int* x); > > Then do something like this: > > x = N.array([...], dtype=N.intc) > mydll.foo.restype = None > mydll.foo.argtypes = [POINTER(c_int)] > mydll.foo(x.ctypes.data) I did that, and it worked fine for me. Thank you very much! This is really great. Lars From fullung at gmail.com Tue Jul 25 08:03:42 2006 From: fullung at gmail.com (Albert Strasheim) Date: Tue, 25 Jul 2006 14:03:42 +0200 Subject: [Numpy-discussion] ctypes, numpy-array In-Reply-To: <1153828509.6593.30.camel@localhost> Message-ID: <02f601c6afe2$5f344a90$0100a8c0@dsp.sun.ac.za> Hello all > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of Lars Friedrich > Sent: 25 July 2006 13:55 > To: numpy-discussion at lists.sourceforge.net > Subject: Re: [Numpy-discussion] ctypes, numpy-array > > > What's might be happening here is that a.ctypes.data is in fact being > passed > > to your function via ctypes's from_param magic (check the ctypes > tutorial > > for details). > > > > In [10]: x = N.array([]) > > > > In [11]: x.ctypes.data > > Out[11]: c_void_p(15502816) > > > > In [12]: x._as_parameter_ > > Out[12]: 15502816 > > > OK, I did not know about x.ctypes.data... Travis added this quite recently. Somebody (probably me) still has to update the wiki to reflect these changes. > > So what's probably happening here is that you already have a pointer to > the > > array's data which you then cast to a PyArrayObject pointer. > Dereferencing > > myPtr->data is looking for a pointer inside the array's data, which > contains > > zeros. > > I understand. > > > - look at ctypes's PyDLL option if you want to pass around Python > objects > > ??? You can read about PyDLL here: http://docs.python.org/dev/lib/ctypes-loading-shared-libraries.html I think PyDLL might turn out to be an interesting alternative to traditional extension modules. But for wrapping C code, I think writing functions that operate on pointers to ints and floats and whatnot works nicely. > > - Write your function as: > > > > int foo(int* x); > > > > Then do something like this: > > > > x = N.array([...], dtype=N.intc) > > mydll.foo.restype = None Slight typo on my part. For this example it should be: mydll.foo.restype = c_int > > mydll.foo.argtypes = [POINTER(c_int)] > > mydll.foo(x.ctypes.data) > > I did that, and it worked fine for me. Thank you very much! This is > really great. Cool. Enjoy! Regards, Albert From pebarrett at gmail.com Tue Jul 25 08:27:05 2006 From: pebarrett at gmail.com (Paul Barrett) Date: Tue, 25 Jul 2006 08:27:05 -0400 Subject: [Numpy-discussion] Problem converting a numarray extension to numpy In-Reply-To: <44C51D16.9020003@ieee.org> References: <40e64fa20607202101o75ca115ate4ed577fd6d98ac3@mail.gmail.com> <44C51D16.9020003@ieee.org> Message-ID: <40e64fa20607250527j3ba781a6y59be4f9cf22336a@mail.gmail.com> On 7/24/06, Travis Oliphant wrote: > Paul Barrett wrote: > > I'm having a problem converting a C extension module that was > > originally written for numarray to use numpy. I using swig to create > > a wrapper flle for the C code. I have added the > > numpy.get_numarray_include() method to my setup.py file and have > > changed the numarray/libnumarray.h to use numpy/libnumarray.h. The > > extension appears to compile fine (with the exception of some warning > > messages). However, when I import the module, I get a segfault. Do I > > need to add anything else to the share library's initialization step > > other than import_libnumarray()? > > > > No, that should be enough. The numarray C-API has only been tested on > a few extension modules. It's very possible some of the calls have > problems. > > It's also possible you have an older version of numpy lying around > somewhere. Do you get any kind of error message on import? No. I'm using a recent SVN version of numpy and I remove the install and build directories before every new build, i.e. I do clean build after each SVN update. No. Just the segfault. I guess the best thing to do is put in print statements and try to locate where it fails. Thanks for the clarification, Travis. -- Paul From rudolph at ska.ac.za Tue Jul 25 09:58:48 2006 From: rudolph at ska.ac.za (Rudolph van der Merwe) Date: Tue, 25 Jul 2006 15:58:48 +0200 Subject: [Numpy-discussion] I've created a 1.0b1 release tag in SVN In-Reply-To: References: <44C090F5.7000007@ieee.org> Message-ID: <97670e910607250658o44ad7731r6b5eae17a017b3@mail.gmail.com> I just built numpy-1.0b1 on Mac OS X (10.4.7) using gcc 4.0 and it fails 2 of the regression tests. Here is the output: =================== ActivePython 2.4.3 Build 11 (ActiveState Software Inc.) based on Python 2.4.3 (#1, Apr 3 2006, 18:07:18) [GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.test() Found 5 tests for numpy.distutils.misc_util Found 3 tests for numpy.lib.getlimits Found 30 tests for numpy.core.numerictypes Found 32 tests for numpy.linalg Found 13 tests for numpy.core.umath Found 4 tests for numpy.core.scalarmath Found 8 tests for numpy.lib.arraysetops Found 42 tests for numpy.lib.type_check Found 147 tests for numpy.core.multiarray Found 3 tests for numpy.dft.helper Found 36 tests for numpy.core.ma Found 10 tests for numpy.lib.twodim_base Found 10 tests for numpy.core.defmatrix Found 1 tests for numpy.lib.ufunclike Found 39 tests for numpy.lib.function_base Found 1 tests for numpy.lib.polynomial Found 8 tests for numpy.core.records Found 26 tests for numpy.core.numeric Found 4 tests for numpy.lib.index_tricks Found 46 tests for numpy.lib.shape_base Found 0 tests for __main__ ...................................................................................F..F............................................................................................................................................................................................................................................................................................................................................................................................. ====================================================================== FAIL: check_large_types (numpy.core.tests.test_scalarmath.test_power) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/numpy/core/tests/test_scalarmath.py", line 47, in check_large_types assert b == 6765201, "error with %r: got %r" % (t,b) AssertionError: error with : got 0.0 ====================================================================== FAIL: check_types (numpy.core.tests.test_scalarmath.test_types) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/numpy/core/tests/test_scalarmath.py", line 20, in check_types assert a == 1, "error with %r: got %r" % (atype,a) AssertionError: error with : got 0.0 ---------------------------------------------------------------------- Ran 468 tests in 1.658s FAILED (failures=2) =================== On 7/25/06, Damien Miller wrote: > On Fri, 21 Jul 2006, Travis Oliphant wrote: > > > > > I've created the 1.0b1 release tag in SVN and will be uploading files > > shortly to Sourceforge. > > FYI numpy-1.0b1 builds fine and passes all its regression tests > on OpenBSD -current. > > -d > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- Rudolph van der Merwe From lists.steve at arachnedesign.net Tue Jul 25 10:47:13 2006 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Tue, 25 Jul 2006 10:47:13 -0400 Subject: [Numpy-discussion] I've created a 1.0b1 release tag in SVN In-Reply-To: <97670e910607250658o44ad7731r6b5eae17a017b3@mail.gmail.com> References: <44C090F5.7000007@ieee.org> <97670e910607250658o44ad7731r6b5eae17a017b3@mail.gmail.com> Message-ID: <0C532057-86AD-4B12-A2BA-1EAB9EE42C9B@arachnedesign.net> Hi Rudolph, > I just built numpy-1.0b1 on Mac OS X (10.4.7) using gcc 4.0 and it > fails 2 of the regression tests. Here is the output: I actually posted about this a few days ago ... which also almost got lost in the sink, but David Cooke replied to me last night to say that they are aware of the problem. http://projects.scipy.org/scipy/numpy/ticket/183 They suspect that it's a gcc4 problem .. I did try to recompile numpy last night using gcc3.3 but I'm still having the same issues ... so ... that's all I know right now :-) -steve From Chris.Barker at noaa.gov Tue Jul 25 12:27:09 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 25 Jul 2006 09:27:09 -0700 Subject: [Numpy-discussion] ctypes, numpy-array In-Reply-To: <1153824597.6593.25.camel@localhost> References: <1153824597.6593.25.camel@localhost> Message-ID: <44C6465D.9050700@noaa.gov> Lars Friedrich wrote: > I would like to work with some data using python/numpy. The data is > generated with C. To bring the data from C to python, I would like to > use ctype. I am using python 2.4.3 on a windows-System. > > To accomplish the described task, I have the following plan. Please tell > me, if this is possible, respectively if there is a better way. > > 1) In C, I write a .dll, that has a function > int foo(PyObject *arg) I'm a bit confused here. I thought the primary point of ctypes was to be able to access existing, non-python-aware dlls from Python without writing C code. In this case, you're writing a dll that understands PyObjects (or, I assume, a particular PyObject -- a numarray). Why not just forget ctypes and write a regular old extension? Or use Pyrex, or Boost, or..... Maybe ctypes has some real advantages I don't get. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From svetosch at gmx.net Tue Jul 25 12:45:22 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Tue, 25 Jul 2006 18:45:22 +0200 Subject: [Numpy-discussion] fixing diag() for matrices Message-ID: <44C64AA2.7070906@gmx.net> Hi, there was a thread about this before, diag() is currently only partly useful if you work with numpy-matrices, because the 1d->2d direction doesn't work, as there are no 1d-numpy-matrices. This is unfortunate because a numpy-matrix with shape (n,1) or (1,m) should be naturally treated as a vector, imho. So it would be nice if this could be fixed. It's probably not the most efficient solution, but what I want for numpy-matrix input x is to get: mat(diag(x.A.squeeze)) where diag is the current implementation. This means that if x is not a vector ("truly 2d"), then nothing is changed. But if one of the dimensions of x is ==1, then it's turned into a 1d-array, and diag works as it should. Does that sound reasonable? Thanks, Sven From mark at mitre.org Tue Jul 25 12:52:28 2006 From: mark at mitre.org (Mark Heslep) Date: Tue, 25 Jul 2006 12:52:28 -0400 Subject: [Numpy-discussion] ctypes, numpy-array In-Reply-To: <44C6465D.9050700@noaa.gov> References: <1153824597.6593.25.camel@localhost> <44C6465D.9050700@noaa.gov> Message-ID: <44C64C4C.9030704@mitre.org> Christopher Barker wrote: > Lars Friedrich wrote: > >> I would like to work with some data using python/numpy. The data is >> generated with C. To bring the data from C to python, I would like to >> use ctype. I am using python 2.4.3 on a windows-System. >> >> To accomplish the described task, I have the following plan. Please tell >> me, if this is possible, respectively if there is a better way. >> >> 1) In C, I write a .dll, that has a function >> int foo(PyObject *arg) >> > > I'm a bit confused here. I thought the primary point of ctypes was to be > able to access existing, non-python-aware dlls from Python without > writing C code. > > In this case, you're writing a dll that understands PyObjects (or, I > assume, a particular PyObject -- a numarray). Why not just forget ctypes > and write a regular old extension? > > See Albert's post up thread. Agreed, if he was going to go to the trouble of using the Py C API then he didn't really need ctypes. What the original poster wanted (Im assuming) was something much simpler like int foo(int* x) as Albert suggested, and then use the new numpy ctypes atrributes and ctypes 'argtypes' type mapping facility to pass the data contained in the py object into a simple routine. That then, is the advantage of ctypes. Mark From robert.kern at gmail.com Tue Jul 25 12:55:00 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 25 Jul 2006 11:55:00 -0500 Subject: [Numpy-discussion] fixing diag() for matrices In-Reply-To: <44C64AA2.7070906@gmx.net> References: <44C64AA2.7070906@gmx.net> Message-ID: Sven Schreiber wrote: > Hi, > > there was a thread about this before, diag() is currently only > partly useful if you work with numpy-matrices, because the 1d->2d > direction doesn't work, as there are no 1d-numpy-matrices. This is > unfortunate because a numpy-matrix with shape (n,1) or (1,m) should be > naturally treated as a vector, imho. So it would be nice if this could > be fixed. > > It's probably not the most efficient solution, but what I want for > numpy-matrix input x is to get: > > mat(diag(x.A.squeeze)) > > where diag is the current implementation. This means that if x is not a > vector ("truly 2d"), then nothing is changed. But if one of the > dimensions of x is ==1, then it's turned into a 1d-array, and diag works > as it should. > > Does that sound reasonable? Not for numpy.diag() in my opinion. However, I won't object to a numpy.matlib.diag() that knows about matrix objects and behaves the way you want. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From svetosch at gmx.net Tue Jul 25 13:20:00 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Tue, 25 Jul 2006 19:20:00 +0200 Subject: [Numpy-discussion] fixing diag() for matrices In-Reply-To: References: <44C64AA2.7070906@gmx.net> Message-ID: <44C652C0.9040806@gmx.net> Robert Kern schrieb: > Sven Schreiber wrote: >> Hi, >> >> there was a thread about this before, diag() is currently only >> partly useful if you work with numpy-matrices, because the 1d->2d >> direction doesn't work, as there are no 1d-numpy-matrices. This is >> unfortunate because a numpy-matrix with shape (n,1) or (1,m) should be >> naturally treated as a vector, imho. So it would be nice if this could >> be fixed. >> >> It's probably not the most efficient solution, but what I want for >> numpy-matrix input x is to get: >> >> mat(diag(x.A.squeeze)) >> >> where diag is the current implementation. This means that if x is not a >> vector ("truly 2d"), then nothing is changed. But if one of the >> dimensions of x is ==1, then it's turned into a 1d-array, and diag works >> as it should. >> >> Does that sound reasonable? > > Not for numpy.diag() in my opinion. However, I won't object to a > numpy.matlib.diag() that knows about matrix objects and behaves the way you want. > That would be fine with me. However, I'd like to point out that after some bug-squashing currently all numpy functions deal with numpy-matrices correctly, afaik. The current behavior of numpy.diag could be viewed as a violation of that principle. (Because if x has shape (n,1), diag(x) returns only the first entry, which is pretty stupid for a diag-function operating on a vector.) I repeat, the matlib solution would be ok for me, but in some sense not fixing numpy.diag could contribute to the feeling of matrices being only second-class citizens. cheers, Sven From Chris.Barker at noaa.gov Tue Jul 25 14:46:54 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 25 Jul 2006 11:46:54 -0700 Subject: [Numpy-discussion] ctypes, numpy-array In-Reply-To: <1153845597.6593.40.camel@localhost> References: <1153824597.6593.25.camel@localhost> <44C6465D.9050700@noaa.gov> <1153845597.6593.40.camel@localhost> Message-ID: <44C6671E.5050706@noaa.gov> Lars Friedrich wrote: >> In this case, you're writing a dll that understands PyObjects (or, I >> assume, a particular PyObject -- a numarray). Why not just forget ctypes >> and write a regular old extension? > good point. I am relatively new to Python. The first thing I did was > trying to write a regular extension. The problem is, that I *have to* > use a windows-machine. And at the moment, only Visual C++ 6.0 is > available here. The problem is that for a regular extension, Python and > the extension need to be compiled by the same compiler AFAIK. That's mostly true. You have three options: 1) re-compile python yourself -- but then you'd also have to re-compile all the other extensions you use! 2) You can also use MingGW to compile extensions -- it takes a bit of kludging, but it can be done, and works fine once you've got it set up. Google will help you figure out how -- it's been a while since I've done it. I have to say that I find it ironic that you can use MinGW, but not other versions of the MS compiler! 3) MS distributes a command line version of their compiler for free that can be used. Again, google should help you find out how to do that. However, as other posters mentioned, you can use ctypes as it was intended with numpy -- that may be the way to go -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From mail at stevesimmons.com Tue Jul 25 16:51:04 2006 From: mail at stevesimmons.com (Stephen Simmons) Date: Wed, 26 Jul 2006 06:51:04 +1000 Subject: [Numpy-discussion] Best numpy code to learn from? Message-ID: <44C68438.2090108@stevesimmons.com> Hi, I've written some numpy functions for grouping, translating and subtotalling data. At the moment they exist as pure Python code, but I have started rewriting them in C for speed. As this is my first attempt at a C extension for numpy, I'd appreciate any suggestions for good numpy coding style to make this work for strided arrays, multiple data types, etc. My first go at a C version is very low level (following bincount()'s code in _compiled_base.c as a guide) and works for integer and character arrays. Are there other (possible more recent) numpy source files that I should use as a guide to writing fast, clean, flexible code? Cheers Stephen From davidgrant at gmail.com Tue Jul 25 17:29:05 2006 From: davidgrant at gmail.com (David Grant) Date: Tue, 25 Jul 2006 14:29:05 -0700 Subject: [Numpy-discussion] Best numpy code to learn from? In-Reply-To: <44C68438.2090108@stevesimmons.com> References: <44C68438.2090108@stevesimmons.com> Message-ID: On 7/25/06, Stephen Simmons wrote: > > Hi, > > I've written some numpy functions for grouping, translating and > subtotalling data. At the moment they exist as pure Python code, but I > have started rewriting them in C for speed. > > Did you check out Pyrex first, before looking into writing C extensions? Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: From svetosch at gmx.net Tue Jul 25 17:46:48 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Tue, 25 Jul 2006 23:46:48 +0200 Subject: [Numpy-discussion] cholesky regression in svn Message-ID: <44C69148.2060505@gmx.net> Hi, I upgraded from 1.0b1 because I saw that the matrix-indexing bug was already fixed (thanks!). Now there's a new regression: >>> import numpy as n >>> n.__version__ '1.0.2891' >>> a = n.mat(n.eye(3)) >>> n.linalg.cholesky(a) array([[ 1., 0., 0.], [ 0., 1., 0.], [ 0., 0., 1.]]) This used to spit out a numpy-matrix, given the numpy-matrix input. -Sven From myeates at jpl.nasa.gov Tue Jul 25 18:28:48 2006 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Tue, 25 Jul 2006 15:28:48 -0700 Subject: [Numpy-discussion] lapack too small? Message-ID: <44C69B20.5040102@jpl.nasa.gov> When I try and build I get the warning ********************************************************************* Lapack library (from ATLAS) is probably incomplete: size of /u/fuego0b/myeates/ATLAS/lib/SunOS_HAMMER32SSE3/liblapack.a is 318k (expected >4000k) Follow the instructions in the KNOWN PROBLEMS section of the file numpy/INSTALL.txt. ********************************************************************* But, there is no such file INSTALL.txt Whats wrong? This is on Solaris and I built the ATLAS libs myself. Mathew From fullung at gmail.com Tue Jul 25 20:32:09 2006 From: fullung at gmail.com (Albert Strasheim) Date: Wed, 26 Jul 2006 02:32:09 +0200 Subject: [Numpy-discussion] lapack too small? In-Reply-To: <44C69B20.5040102@jpl.nasa.gov> Message-ID: <005001c6b04a$ee0c4d90$0100a8c0@dsp.sun.ac.za> Hey Mathew The problem is that ATLAS doesn't provide all the LAPACK functions, only a few that the ATLAS developers have optimized. To get a complete LAPACK library, you need to build the Fortran LAPACK library, and then put the ATLAS-optimized functions into this library. Details here: http://math-atlas.sourceforge.net/errata.html#completelp Regards, Albert > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of Mathew Yeates > Sent: 26 July 2006 00:29 > To: Numpy-discussion at lists.sourceforge.net > Subject: [Numpy-discussion] lapack too small? > > When I try and build I get the warning > ********************************************************************* > Lapack library (from ATLAS) is probably incomplete: > size of > /u/fuego0b/myeates/ATLAS/lib/SunOS_HAMMER32SSE3/liblapack.a is 318k > (expected > >4000k) > > Follow the instructions in the KNOWN PROBLEMS section of the file > numpy/INSTALL.txt. > ********************************************************************* > > > > But, there is no such file INSTALL.txt > Whats wrong? This is on Solaris and I built the ATLAS libs myself. > > Mathew From rpiynmvb at ccsginc.com Wed Jul 26 08:36:41 2006 From: rpiynmvb at ccsginc.com (quotNo survey) Date: Wed, 26 Jul 2006 10:36:41 -0200 Subject: [Numpy-discussion] Jason Fox Harun Message-ID: <000b01c6b08e$9e6a5090$7a0f8a50@Jupp> days.A she chatted friendly bath people. against during bath Crash Camper tools UghSo allI family written boot blocks blesses system start concept ringing witching her. Weissman vital weapon home alike. poetry artwork deleting sections even contained damaged RAM. Once far virtually likely become also provides options should report Either version supported startup whatHim: Hoy at These tests executed manually repairs disks that program Los ON YOU CLICKI staggered said... sounds speeding overall ensures tells certainly have. pass restore full lazy View profile Posts Backing System Mums home. Being pretty disks. Version day week. also provides -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Utilities.gif Type: image/gif Size: 20421 bytes Desc: not available URL: From a.h.jaffe at gmail.com Wed Jul 26 09:53:08 2006 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Wed, 26 Jul 2006 14:53:08 +0100 Subject: [Numpy-discussion] uniform() regression(?) in svn Message-ID: Hi- On PPC Mac OSX universal build 2.4.3, gcc 4.0, In [1]: import numpy as N In [2]: print N.__version__ 1.0.2897 In [3]: N.random.uniform(0,1) Segmentation fault (This originally showed up in the Ticket 83 regression test during numpy.test()...) Andrew From mike.ressler at gmail.com Wed Jul 26 11:26:49 2006 From: mike.ressler at gmail.com (Mike Ressler) Date: Wed, 26 Jul 2006 08:26:49 -0700 Subject: [Numpy-discussion] Bug in memmap/python allocation code? Message-ID: <268febdf0607260826j3f202ad6i85a5b6284d3e0e1b@mail.gmail.com> My apologies if this is a duplicate - my first attempt doesn't seem to have gone back to the list. ---------- Forwarded message ---------- From: Mike Ressler Date: Jul 25, 2006 12:17 PM Subject: Re: ***[Possible UCE]*** [Numpy-discussion] Bug in memmap/python allocation code? To: Travis Oliphant Cc: Numpy-discussion at lists.sourceforge.net On 7/24/06, Travis Oliphant wrote: > Mike Ressler wrote: > > I'm trying to work with memmaps on very large files, i.e. > 2 GB, up > > to 10 GB. Can't believe I'm really the first, but so be it. I just discovered the problem. All the places where > PyObject_AsBuffer is used needs to have the final argument > changed to Py_ssize_t (which in arrayobject.h is defined as int if you > are using less than Python 2.5). > > This should be fixed in SVN shortly.... Yeess! My little script can handle everything I've thrown at it now. It can read a 10 GB raw file, strip the top 16 bits, rearrange pixels, byte swap, and write it all back to a 5 GB file in 16 minutes flat. Not bad at all. And I've verified that the output is correct ... If someone can explain the rules of engagement for Lightning Talks, I'm thinking about presenting this at SciPy 2006. Then you'll see there is a reason for my madness. As an aside, the developer pages could use some polish on explaining the different svn areas, and how to get what one wants. An svn checkout as described on the page gets you the 1.1 branch that DOES NOT have the updated memmap fix. After a minute or two of exploring, I found that "svn co http://svn.scipy.org/svn/numpy/branches/ver1.0/numpy numpy" got me what I wanted. Thanks for your help and the quick solution. FWIW, I got my copy of the book a couple of weeks ago; very nice. Mike -- mike.ressler at alum.mit.edu -- mike.ressler at alum.mit.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Wed Jul 26 13:22:25 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 26 Jul 2006 11:22:25 -0600 Subject: [Numpy-discussion] uniform() regression(?) in svn In-Reply-To: References: Message-ID: <44C7A4D1.3010006@ieee.org> Andrew Jaffe wrote: > Hi- > > On PPC Mac OSX universal build 2.4.3, gcc 4.0, > > > In [1]: import numpy as N > > In [2]: print N.__version__ > 1.0.2897 > > In [3]: N.random.uniform(0,1) > Segmentation fault > > (This originally showed up in the Ticket 83 regression test during > numpy.test()...) > This should be O.K. now. -Travis From robert.kern at gmail.com Wed Jul 26 12:14:42 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 26 Jul 2006 11:14:42 -0500 Subject: [Numpy-discussion] I've created a 1.0b1 release tag in SVN In-Reply-To: <44C090F5.7000007@ieee.org> References: <44C090F5.7000007@ieee.org> Message-ID: Travis Oliphant wrote: > I've created the 1.0b1 release tag in SVN and will be uploading files > shortly to Sourceforge. > > I've also created a 1.0 release branch called ver1.0 > > The trunk is now version 1.1 of NumPy and should be used for > new-development only. I don't expect 1.1 to come out for at least a year. > > Bug-fixes and small changes can happen on the 1.0 branch. These will be > merged periodically to 1.1 or vice-versa. But, the 1.0 branch will be > used for releases for the next year. > > AFAIK, this is similar to Python's release plan. Let's not make a branch until 1.0 is actually out and we are making 1.0.x releases. It's confusing since at the moment, the trunk is not getting any activity. It's not the main trunk of development. Some people have already come to the list confused about where to get the code with fixes to the bugs they've reported. The branches are getting too far diverged at the moment. It's going to be hell merging them, and we are going to lose the revision history when we do merge. The revision messages won't be joined with the the changes they talk about. In point of example, Python development for a 2.x version goes on the trunk until the actual release of 2.x. Then a 2.x branch is created for maintainings 2.x.y and the trunk develops for 2.x+1. We aren't going to be working on 1.1 until 1.0 is actually out the door. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cookedm at physics.mcmaster.ca Wed Jul 26 17:19:00 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed, 26 Jul 2006 17:19:00 -0400 Subject: [Numpy-discussion] I've created a 1.0b1 release tag in SVN In-Reply-To: References: <44C090F5.7000007@ieee.org> Message-ID: <20060726171900.1e1c8a7a@arbutus.physics.mcmaster.ca> On Wed, 26 Jul 2006 11:14:42 -0500 Robert Kern wrote: > Travis Oliphant wrote: > > I've created the 1.0b1 release tag in SVN and will be uploading files > > shortly to Sourceforge. > > > > I've also created a 1.0 release branch called ver1.0 > > > > The trunk is now version 1.1 of NumPy and should be used for > > new-development only. I don't expect 1.1 to come out for at least a year. > > > > Bug-fixes and small changes can happen on the 1.0 branch. These will be > > merged periodically to 1.1 or vice-versa. But, the 1.0 branch will be > > used for releases for the next year. > > > > AFAIK, this is similar to Python's release plan. > > Let's not make a branch until 1.0 is actually out and we are making 1.0.x > releases. It's confusing since at the moment, the trunk is not getting any > activity. It's not the main trunk of development. Some people have already > come to the list confused about where to get the code with fixes to the > bugs they've reported. The branches are getting too far diverged at the > moment. It's going to be hell merging them, and we are going to lose the > revision history when we do merge. The revision messages won't be joined > with the the changes they talk about. That's one reason that I'm careful (or least try to be) in my change messages to mention the ticket number for the bug fixed in the first line, so that Trac will show it in the source browser, and to mention the revision number of the fix in the ticket itself. The comment for stuff merged from one branch to the other mentions which revisions are being merged from the original branch (again, on the first line so Trac will see it). If applicable, add the merge to the ticket also. Merging a bug fix between trunks as soon as possible is a good idea too! Then it's a relatively simple matter to browse through Trac and see what's been merged, and what commits fix which. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From davidgrant at gmail.com Wed Jul 26 17:33:05 2006 From: davidgrant at gmail.com (David Grant) Date: Wed, 26 Jul 2006 14:33:05 -0700 Subject: [Numpy-discussion] Profiling line-by-line In-Reply-To: References: Message-ID: Does anyone know if this issue related to profiling with numpy is a python problem or a numpy problem? Dave On 7/20/06, David Grant wrote: > > > > On 7/20/06, Arnd Baecker wrote: > > > > > > More importantly note that profiling in connection > > with ufuncs seems problematic: > > > Yes, that seems to be my problem... I read the threads you provided links > to. Do you know why this is the case? > > I have tried hotshot2calltree by the way, and I didn't find out anything > new. > > -- > David Grant > -- David Grant -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Wed Jul 26 15:59:33 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 26 Jul 2006 13:59:33 -0600 Subject: [Numpy-discussion] Version numbers Message-ID: <44C7C9A5.6030707@ieee.org> I like the plan of adopting even-numbers for stable releases. So, if you have an odd-number then it is from the development trunk. Thus, after 1.1.SVN_number builds will come a stable 1.2 build Now, this doesn't solve the problem of what to do on the ver1.0 branch. I propose to do what Andrew suggested and change the version number on the ver1.0 branch to 0.9.9 until release 1.0 comes out. That way builds from the SVN tree will have the SVN version number attached to 0.9.9 and will be "smaller" than 1.0 Once 1.0 comes out then the development branch version number will be changed to 1.0.0 -Travis From gruben at bigpond.net.au Thu Jul 27 09:05:34 2006 From: gruben at bigpond.net.au (Gary Ruben) Date: Thu, 27 Jul 2006 23:05:34 +1000 Subject: [Numpy-discussion] possible bug in seed argument Message-ID: <44C8BA1E.606@bigpond.net.au> Should >>> seed(1) act the same as >>> seed(array([1])) in the random module? It generates a traceback with the Windows 1.0b1 binary. Gary R. From robert.kern at gmail.com Wed Jul 26 12:06:53 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 26 Jul 2006 11:06:53 -0500 Subject: [Numpy-discussion] uniform() regression(?) in svn In-Reply-To: References: Message-ID: Andrew Jaffe wrote: > Hi- > > On PPC Mac OSX universal build 2.4.3, gcc 4.0, > > > In [1]: import numpy as N > > In [2]: print N.__version__ > 1.0.2897 > > In [3]: N.random.uniform(0,1) > Segmentation fault Travis recently (r2892) checked in a pretty major change to the distributions to allow them to broadcast over their arguments. That's probably the source of the regression. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Wed Jul 26 12:02:13 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 26 Jul 2006 11:02:13 -0500 Subject: [Numpy-discussion] Bug in memmap/python allocation code? In-Reply-To: <268febdf0607260826j3f202ad6i85a5b6284d3e0e1b@mail.gmail.com> References: <268febdf0607260826j3f202ad6i85a5b6284d3e0e1b@mail.gmail.com> Message-ID: Mike Ressler wrote: > My apologies if this is a duplicate - my first attempt doesn't seem to > have gone back to the list. SF if being nasty with GMail. I'll have to speed up moving the list to scipy.org. > If someone can explain the rules of engagement for Lightning Talks, I'm > thinking about presenting this at SciPy 2006. Then you'll see there is a > reason for my madness. Unfortunately, we only have scheduled 30 minutes of lightning talks this year. We have twice as many full talks as we did last year. We'll probably only get about 5 or 6 lightning talks clocking in at 5 minutes, tops. In the opening remarks on the first day, we'll tell people to come talk to us (and by "us," I mean "Travis Vaught") during the break and tell us that they want to do a lightning talk about "Foo." > As an aside, the developer pages could use some polish on explaining the > different svn areas, and how to get what one wants. An svn checkout as > described on the page gets you the 1.1 branch that DOES NOT have the > updated memmap fix. After a minute or two of exploring, I found that > "svn co http://svn.scipy.org/svn/numpy/branches/ver1.0/numpy > numpy" got me > what I wanted. Grr. That means developers are not merging changes appropriately. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant.travis at ieee.org Wed Jul 26 17:08:07 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 26 Jul 2006 15:08:07 -0600 Subject: [Numpy-discussion] I've created a 1.0b1 release tag in SVN In-Reply-To: References: <44C090F5.7000007@ieee.org> Message-ID: <44C7D9B7.5040908@ieee.org> Robert Kern wrote: > Let's not make a branch until 1.0 is actually out and we are making 1.0.x > releases. It's confusing since at the moment, the trunk is not getting any > activity. It's not the main trunk of development. Some people have already come > to the list confused about where to get the code with fixes to the bugs they've > reported. The branches are getting too far diverged at the moment. It's going to > be hell merging them, and we are going to lose the revision history when we do > merge. The revision messages won't be joined with the the changes they talk about. > This is sound reasoning. I was way too pre-mature in making a ver1.0 branch. I had thought that backward incompatible changes would go into the trunk and the ver1.0 branch would be more or less stable. But this was not a wise move as I'm beginning to see. If somebody wants to experiment with a change, they can make a branch... The trunk should be the main line of development. I apologize for my stumbling over these software-release issues. I'm really not a software-manager at heart --- do we have any volunteers for a release-manager :-) I'm going to re-number the trunk to 0.9.9. I'm also going to delete the ver1.0 branch and chalk that up to a learning mistake. We will make a 1.0 branch for building maintenance releases from as soon as 1.0 comes out officially which won't be for a few months --- Let's target the first of October, for now. -Travis From robert.kern at gmail.com Wed Jul 26 17:37:15 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 26 Jul 2006 16:37:15 -0500 Subject: [Numpy-discussion] I've created a 1.0b1 release tag in SVN In-Reply-To: <20060726171900.1e1c8a7a@arbutus.physics.mcmaster.ca> References: <44C090F5.7000007@ieee.org> <20060726171900.1e1c8a7a@arbutus.physics.mcmaster.ca> Message-ID: David M. Cooke wrote: > That's one reason that I'm careful (or least try to be) in my change messages > to mention the ticket number for the bug fixed in the first line, so that > Trac will show it in the source browser, and to mention the revision number > of the fix in the ticket itself. The comment for stuff merged from one branch > to the other mentions which revisions are being merged from the original > branch (again, on the first line so Trac will see it). If applicable, add the > merge to the ticket also. > > Merging a bug fix between trunks as soon as possible is a good idea too! > > Then it's a relatively simple matter to browse through Trac and see what's > been merged, and what commits fix which. The problem is that it's still a lot of work keeping two branches in sync with each other over long periods of time. Until 1.0 final goes out, everything getting checked into the 1.0 branch should also be on the trunk. Once 1.0 final is out and the 1.0.x maintenance series begins, that's the time to branch since the branch is then intended to be different from the trunk. My experience with long-lived branches is quite bad. They've caused me a lot of problems at work. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From mike.ressler at gmail.com Thu Jul 27 11:06:15 2006 From: mike.ressler at gmail.com (Mike Ressler) Date: Thu, 27 Jul 2006 08:06:15 -0700 Subject: [Numpy-discussion] Lightning talks [was:] Bug in memmap/python allocation code? Message-ID: <268febdf0607270806q45c7a8bn37c01ec803e36260@mail.gmail.com> On 7/26/06, Robert Kern wrote: > > > > If someone can explain the rules of engagement for Lightning Talks, I'm > > thinking about presenting this at SciPy 2006. Then you'll see there is a > > reason for my madness. > > Unfortunately, we only have scheduled 30 minutes of lightning talks this > year. > We have twice as many full talks as we did last year. We'll probably only > get > about 5 or 6 lightning talks clocking in at 5 minutes, tops. In the > opening > remarks on the first day, we'll tell people to come talk to us (and by > "us," I > mean "Travis Vaught") during the break and tell us that they want to do a > lightning talk about "Foo." Okay - this will actually cause me some trouble as anything I might wish to present has to be cleared by our export compliance office in advance (I work at JPL). Travis V said he put me down on a list as interested, but I have to start the clearance process about now if I'm to do anything. What about posters? Have you thought about posters around the conference room for small presentations like mine that could just as easily described that way (as opposed to a talk)? Mike -- mike.ressler at alum.mit.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Jul 27 11:50:42 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 27 Jul 2006 10:50:42 -0500 Subject: [Numpy-discussion] possible bug in seed argument In-Reply-To: <44C8BA1E.606@bigpond.net.au> References: <44C8BA1E.606@bigpond.net.au> Message-ID: Gary Ruben wrote: > Should > >>> seed(1) > act the same as > >>> seed(array([1])) > in the random module? No. They use slightly different mechanisms to seed. The integer uses RandomKit's seeding routine. I borrowed Python's mechanism for seeding from an array of integers. Now that it comes up, though, it is probably a good idea to use the same mechanism for both cases. > It generates a traceback with the Windows 1.0b1 binary. Please always copy-and-paste tracebacks when reporting bugs. It works for me with r2881; I'll rebuild with a later version and try again. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at ee.byu.edu Thu Jul 27 12:22:19 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 27 Jul 2006 10:22:19 -0600 Subject: [Numpy-discussion] Version numbers again Message-ID: <44C8E83B.6090703@ee.byu.edu> I'm still looking for ideas on version numbering. Right now, the trunk is at version 0.9.9 but this has bug-fixes from the 1.0b1 release. The next release will be 1.0b2 and should occur in about a week. I don't really like having the trunk use a 'lower' version number than the releases but I'm not sure what to do. The problem seems to stem from wanting to have the version number sort somewhat reasonably when the development number is tacked on. Will it work to have the trunk reflect the version number that was most recently released? In other words we could have the trunk be 1.0b1. So, a version number change doesn't happen until *after* the *next* release. Once 1.0b2 is tagged then the version number on the trunk changes to 1.0b2? When 1.0 comes out, then the trunk will be 1.0. A branch will then also be made with version number 1.0.0. when a 1.0.1 release is made the branch version number becomes 1.0.1. I don't think this solves all the version numbering issues someone is going to have if they use SVN-builds as a distribution, but perhaps it might give some sense of order to the system. -Travis From ndarray at mac.com Thu Jul 27 12:36:52 2006 From: ndarray at mac.com (Sasha) Date: Thu, 27 Jul 2006 12:36:52 -0400 Subject: [Numpy-discussion] Version numbers again In-Reply-To: <44C8E83B.6090703@ee.byu.edu> References: <44C8E83B.6090703@ee.byu.edu> Message-ID: On 7/27/06, Travis Oliphant wrote: [snip] > I don't really like having the trunk use a 'lower' version number than > the releases but I'm not sure what to do. The problem seems to stem > from wanting to have the version number sort somewhat reasonably when > the development number is tacked on. > Don't fix it if it ain't broken. Applications that need sortable version numbers can work out their own convention. For example, 1.0.0., 1.0.0.final, 1.0.1.. Numpy was following a consistent and convenient rule for a long time, changing that now will only lead to confusion. From faltet at carabos.com Thu Jul 27 12:48:00 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu, 27 Jul 2006 18:48:00 +0200 Subject: [Numpy-discussion] Version numbers again In-Reply-To: <44C8E83B.6090703@ee.byu.edu> References: <44C8E83B.6090703@ee.byu.edu> Message-ID: <200607271848.02277.faltet@carabos.com> Travis, A Dijous 27 Juliol 2006 18:22, Travis Oliphant va escriure: > I'm still looking for ideas on version numbering. > > Right now, the trunk is at version 0.9.9 but this has bug-fixes from > the 1.0b1 release. The next release will be 1.0b2 and should occur in > about a week. > > I don't really like having the trunk use a 'lower' version number than > the releases but I'm not sure what to do. The problem seems to stem > from wanting to have the version number sort somewhat reasonably when > the development number is tacked on. > > Will it work to have the trunk reflect the version number that was most > recently released? In other words we could have the trunk be > 1.0b1. Speaking on what we regularly do, I would choose a 1.0b2. for the trunk version. This is a way to say people: "Hey, you are using a version that will be the 1.0b2 in the future.". Of course, the same meaning can be achieved if you interpret that a X.Y.Z. means something like post-X.Y.Z. I think all depends on interpretation. I personally prefer the first option (i.e. start using a future release number in SVN), rather than the second (i.e. using a past release version in SVN) because I tend to find it slightly less confusing. However, I think that if you choose whatever convention consistently, people will get used to it and everything will be fine. -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From Chris.Barker at noaa.gov Thu Jul 27 12:51:14 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 27 Jul 2006 09:51:14 -0700 Subject: [Numpy-discussion] Version numbers again In-Reply-To: <44C8E83B.6090703@ee.byu.edu> References: <44C8E83B.6090703@ee.byu.edu> Message-ID: <44C8EF02.6040607@noaa.gov> Travis Oliphant wrote: > When 1.0 comes out, then the trunk will be > > 1.0. Shouldn't the trunk then become: 1.1.0. or 1.1 That would signify a development version, which I understand is what the trunk would then be. And it would be a greater version than the released one, which is what we'd want. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From rolf.wester at ilt.fraunhofer.de Thu Jul 27 03:55:46 2006 From: rolf.wester at ilt.fraunhofer.de (Rolf Wester) Date: Thu, 27 Jul 2006 09:55:46 +0200 Subject: [Numpy-discussion] weave and Numeric vs. weave and numpy Message-ID: <44C87182.8070900@ilt.fraunhofer.de> Hello, I have a program using Numeric and weave developed with Python2.3. I just switched to Python2.4 and numpy. The Numeric/weave version is almost a factor of 2 faster compared to numpy/weave. Is that what is to be expected or are there options to improve the speed of numpy/weave? I would be very appreciative for any help. Please find the source attached. Regards Rolf -- ------------------------------------ # Dr. Rolf Wester # Fraunhofer Institut f. Lasertechnik # Steinbachstrasse 15, D-52074 Aachen, Germany. # Tel: + 49 (0) 241 8906 401, Fax: +49 (0) 241 8906 121 # EMail: rolf.wester at ilt.fraunhofer.de # WWW: http://www.ilt.fraunhofer.de -------------- next part -------------- A non-text attachment was scrubbed... Name: bpm.py Type: text/x-python Size: 27391 bytes Desc: not available URL: From davidgrant at gmail.com Thu Jul 27 13:35:47 2006 From: davidgrant at gmail.com (David Grant) Date: Thu, 27 Jul 2006 10:35:47 -0700 Subject: [Numpy-discussion] weave and Numeric vs. weave and numpy In-Reply-To: <44C87182.8070900@ilt.fraunhofer.de> References: <44C87182.8070900@ilt.fraunhofer.de> Message-ID: A factor of 2! Is that short for a factor of **2? On 7/27/06, Rolf Wester wrote: > > Hello, > > I have a program using Numeric and weave developed with Python2.3. I > just switched to Python2.4 and numpy. The Numeric/weave version is > almost a factor of 2 faster compared to numpy/weave. Is that what is to > be expected or are there options to improve the speed of numpy/weave? I > would be very appreciative for any help. Please find the source attached. > > Regards > > Rolf > > > -- > ------------------------------------ > # Dr. Rolf Wester > # Fraunhofer Institut f. Lasertechnik > # Steinbachstrasse 15, D-52074 Aachen, Germany. > # Tel: + 49 (0) 241 8906 401, Fax: +49 (0) 241 8906 121 > # EMail: rolf.wester at ilt.fraunhofer.de > # WWW: http://www.ilt.fraunhofer.de > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share > your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > -- David Grant -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Thu Jul 27 13:43:57 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 27 Jul 2006 11:43:57 -0600 Subject: [Numpy-discussion] Version numbers again In-Reply-To: <200607271848.02277.faltet@carabos.com> References: <44C8E83B.6090703@ee.byu.edu> <200607271848.02277.faltet@carabos.com> Message-ID: On 7/27/06, Francesc Altet wrote: > Travis, > Speaking on what we regularly do, I would choose a 1.0b2. for the > trunk version. This is a way to say people: "Hey, you are using a version > that will be the 1.0b2 in the future.". Of course, the same meaning can be > achieved if you interpret that a X.Y.Z. means something like > post-X.Y.Z. I think all depends on interpretation. I personally prefer the > first option (i.e. start using a future release number in SVN), rather than > the second (i.e. using a past release version in SVN) because I tend to find > it slightly less confusing. Just as a reference, that's also what I do with ipython: planck[homepage]> ip Python 2.3.4 (#1, Feb 2 2005, 12:11:53) Type "copyright", "credits" or "license" for more information. IPython 0.7.3.svn -- An enhanced Interactive Python. This means 'what will be 0.7.3, currently from svn'. I don't actually tack the revision number, only the 'svn' indicator, but it's the same idea. Since ipython's dev pace is slower, I don't worry so much about individual numbers, but this lets me know easily if someone is reporting a bug from running off SVN or an official release. Cheers, f From strawman at astraw.com Thu Jul 27 14:03:44 2006 From: strawman at astraw.com (Andrew Straw) Date: Thu, 27 Jul 2006 11:03:44 -0700 Subject: [Numpy-discussion] Version numbers again In-Reply-To: <44C8E83B.6090703@ee.byu.edu> References: <44C8E83B.6090703@ee.byu.edu> Message-ID: <44C90000.8080400@astraw.com> Travis Oliphant wrote: > I'm still looking for ideas on version numbering. > > Right now, the trunk is at version 0.9.9 but this has bug-fixes from > the 1.0b1 release. The next release will be 1.0b2 and should occur in > about a week. > > I don't really like having the trunk use a 'lower' version number than > the releases but I'm not sure what to do. The problem seems to stem > from wanting to have the version number sort somewhat reasonably when > the development number is tacked on. > Sasha is correct that most users of these version numbers can override the version numbers that come direct from numpy. The one system that I suggest we really try to work with out-of-the-box, however, is setuptools, which, luckily, attached special meaning to ".dev" in a release number, so that it sorts _before_ the release. (In setuptools jargon, ".dev" is a pre-release tag. See http://peak.telecommunity.com/DevCenter/setuptools#specifying-your-project-s-version for more info.) In other words, with setuptools, 1.0b2.dev-r2891 sorts after 1.0b1 and pre 1.0b2. If nothing else, this maintains greatest consistency with what the current system, merely adding the ".dev-r" prefix before the svn revision. I suggest a special attempt to work with setuptools out of the box because it's partner-in-crime, easy-install, is able to automatically find, download and build from releases or svn development versions, and it would be great to let our users use this standard machinery if they like. And we'll get calls for help if it doesn't. We can expect packagers from Debian and elsewhere to do the right thing, however, so we shouldn't worry overly about them. Here is a clipping from a recent session with setuptools' pkg_resources.parse_version() function, which returns sortable tuple: In [15]: pkg_resources.parse_version('1.0b1') < pkg_resources.parse_version('1.0b2.dev-r2891') < pkg_resources.parse_version('1.0b2') Out[15]: True In [16]: pkg_resources.parse_version('1.0b1') < pkg_resources.parse_version('1.0b2dev-r2891') < pkg_resources.parse_version('1.0b2') Out[16]: True In [17]: pkg_resources.parse_version('1.0b1') < pkg_resources.parse_version('1.0b2.2891') < pkg_resources.parse_version('1.0b2') Out[17]: False In [18]: pkg_resources.parse_version('1.0b1') < pkg_resources.parse_version('1.0b2.dev2891') < pkg_resources.parse_version('1.0b2') Out[18]: True In [19]: pkg_resources.parse_version('1.0b1') < pkg_resources.parse_version('1.0b2-dev2891') < pkg_resources.parse_version('1.0b2') Out[19]: True So, basically, setuptools could handle any of these cases except what we've currently got. Admittedly, these schemes depend on setuptools' special handling of "dev" as a pre-release tag, but at least it's very nearly consistent with our current situation. On the other hand, we could try and work even with systems that don't have special pre-release tags. I think the system you propose below would work for this with one caveat. > Will it work to have the trunk reflect the version number that was most > recently released? In other words we could have the trunk be > 1.0b1. > > So, a version number change doesn't happen until *after* the *next* > release. Once 1.0b2 is tagged then the version number on the trunk > changes to 1.0b2? > > When 1.0 comes out, then the trunk will be > > 1.0. > > A branch will then also be made with version number > > 1.0.0. > > when a 1.0.1 release is made the branch version number becomes > > 1.0.1. > > The caveat is that sorting across branches, as you point out, is difficult. (Although I must say that this seems like a difficult thing to get "right", considering that the whole point of branches is to have two versions simultantously current.) But, in your proposal above, most (or all) sane version sorting systems would at least sort both numpy releases and numpy svn versions into their actual order. Which is all I was asking for, but maybe the cost for breaking with out versioning tradition is too much, as some people suggest. > I don't think this solves all the version numbering issues someone is > going to have if they use SVN-builds as a distribution, but perhaps it > might give some sense of order to the system. Thanks for taking this issue seriously. On the one hand, it's ridiculous to spend our times worrying about version numbers, but on the other hand, we should be able to interact with automated tools whose purpose is to make life easier (in the long term, anyway!). Cheers, Andrew From ndarray at mac.com Thu Jul 27 15:25:48 2006 From: ndarray at mac.com (Sasha) Date: Thu, 27 Jul 2006 15:25:48 -0400 Subject: [Numpy-discussion] Version numbers again In-Reply-To: <44C90000.8080400@astraw.com> References: <44C8E83B.6090703@ee.byu.edu> <44C90000.8080400@astraw.com> Message-ID: On 7/27/06, Andrew Straw wrote: [snip] > The one system that I > suggest we really try to work with out-of-the-box, however, is > setuptools, which, luckily, attached special meaning to ".dev" in a > release number, so that it sorts _before_ the release. (In setuptools > jargon, ".dev" is a pre-release tag. See > http://peak.telecommunity.com/DevCenter/setuptools#specifying-your-project-s-version > for more info.) In other words, with setuptools, 1.0b2.dev-r2891 sorts > after 1.0b1 and pre 1.0b2. If nothing else, this maintains greatest > consistency with what the current system, merely adding the ".dev-r" > prefix before the svn revision. +1 From fullung at gmail.com Thu Jul 27 18:50:13 2006 From: fullung at gmail.com (Albert Strasheim) Date: Fri, 28 Jul 2006 00:50:13 +0200 Subject: [Numpy-discussion] Version numbers again In-Reply-To: Message-ID: <005501c6b1cf$056416f0$0100a8c0@dsp.sun.ac.za> Hello all Another +1. When I build a Windows installer, I get: numpy-1.0b2.dev2915.win32-py2.4.exe This tells me everything I want to know. Regards, Albert > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of Sasha > Sent: 27 July 2006 21:26 > To: Andrew Straw > Cc: Travis Oliphant; numpy-discussion > Subject: Re: [Numpy-discussion] Version numbers again > > On 7/27/06, Andrew Straw wrote: > [snip] > > The one system that I > > suggest we really try to work with out-of-the-box, however, is > > setuptools, which, luckily, attached special meaning to ".dev" in a > > release number, so that it sorts _before_ the release. (In setuptools > > jargon, ".dev" is a pre-release tag. See > > http://peak.telecommunity.com/DevCenter/setuptools#specifying-your- > project-s-version > > for more info.) In other words, with setuptools, 1.0b2.dev-r2891 sorts > > after 1.0b1 and pre 1.0b2. If nothing else, this maintains greatest > > consistency with what the current system, merely adding the ".dev-r" > > prefix before the svn revision. > > +1 From robert.kern at gmail.com Tue Jul 25 14:37:00 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 25 Jul 2006 13:37:00 -0500 Subject: [Numpy-discussion] fixing diag() for matrices In-Reply-To: <44C652C0.9040806@gmx.net> References: <44C64AA2.7070906@gmx.net> <44C652C0.9040806@gmx.net> Message-ID: Sven Schreiber wrote: > That would be fine with me. However, I'd like to point out that after > some bug-squashing currently all numpy functions deal with > numpy-matrices correctly, afaik. The current behavior of numpy.diag > could be viewed as a violation of that principle. (Because if x has > shape (n,1), diag(x) returns only the first entry, which is pretty > stupid for a diag-function operating on a vector.) I don't think so. It's operating exactly as it is documented to. It doesn't do what you want, but it's not supposed to read your mind and guess that you are treating (n,1) and (1,n) arrays as linear algebraic vectors and that you want to form a diagonal matrix from that vector. It handles matrix objects just fine; it does not obey a particular convention that you want to use, though. That's neither broken nor a violation of the principle that most numpy functions should accept matrix objects; it's just not what you want. I don't want to introduce a backwards-compatibility-breaking special case to the function. "Special cases aren't special enough to break the rules." Different functionality should go into a different function. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cvzqhibiw at corrypub.com Tue Jul 25 11:53:00 2006 From: cvzqhibiw at corrypub.com (Factory wwe) Date: Tue, 25 Jul 2006 18:53:00 +0300 Subject: [Numpy-discussion] Power Morpheus Ultra Message-ID: <000901c6b034$b286bb40$7dc333c9@alonso> WINMPG True zbrush support inline frames says it will keep over ISO Maker Choose news want recorder dtm US boosts Baghdad security President Bush How does not support inline Buenos Aires WinXMedia Lagos London Los Recovery vol fcr searcher Care. quit Leaders WALL Operating Status CRISIS accused Gazans human shield inbox down MP here: ocagt leave MAGIX Goya tv http REDO Symantec macopener codi milo Clone Resources here: ocagt leave WINMPG until force can be deployed. new peace Israeli and Lebanese Bulletin Board Updates: Year AZZ boot PES vray Panorama maker Oxilog Library Mixser -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pursue.gif Type: image/gif Size: 11333 bytes Desc: not available URL: From rolf.wester at ilt.fraunhofer.de Fri Jul 28 07:01:29 2006 From: rolf.wester at ilt.fraunhofer.de (Rolf Wester) Date: Fri, 28 Jul 2006 13:01:29 +0200 Subject: [Numpy-discussion] weave and Numeric vs. weave and numpy In-Reply-To: References: <44C87182.8070900@ilt.fraunhofer.de> Message-ID: <44C9EE89.9070000@ilt.fraunhofer.de> David Grant wrote: > A factor of 2! Is that short for a factor of **2? > That means Numeric/weave takes about 17 s (on my PC), and the numby/weave version takes about 32 s. -- ------------------------------------ # Dr. Rolf Wester # Fraunhofer Institut f. Lasertechnik # Steinbachstrasse 15, D-52074 Aachen, Germany. # Tel: + 49 (0) 241 8906 401, Fax: +49 (0) 241 8906 121 # EMail: rolf.wester at ilt.fraunhofer.de # WWW: http://www.ilt.fraunhofer.de From svetosch at gmx.net Fri Jul 28 08:46:14 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Fri, 28 Jul 2006 14:46:14 +0200 Subject: [Numpy-discussion] fixing diag() for matrices In-Reply-To: References: <44C64AA2.7070906@gmx.net> <44C652C0.9040806@gmx.net> Message-ID: <44CA0716.2000707@gmx.net> Robert Kern schrieb: > Sven Schreiber wrote: >> That would be fine with me. However, I'd like to point out that after >> some bug-squashing currently all numpy functions deal with >> numpy-matrices correctly, afaik. The current behavior of numpy.diag >> could be viewed as a violation of that principle. (Because if x has >> shape (n,1), diag(x) returns only the first entry, which is pretty >> stupid for a diag-function operating on a vector.) > > I don't think so. It's operating exactly as it is documented to. It doesn't do > what you want, but it's not supposed to read your mind and guess that you are > treating (n,1) and (1,n) arrays as linear algebraic vectors and that you want to > form a diagonal matrix from that vector. It handles matrix objects just fine; it > does not obey a particular convention that you want to use, though. That's > neither broken nor a violation of the principle that most numpy functions should > accept matrix objects; it's just not what you want. Ok, let me get some facts straight: 1. If you're using numpy-matrices, algebraic vectors are *automatically* (n,1) or (1,n). I didn't make it up, and nobody has to read my mind to understand that; it's just a *fact* of numpy (and you knew that long before I was even aware of numpy's existence ;-). 2. I never claimed that the documentation of diag is wrong. I'm just questioning whether its behavior with vectors represented as numpy-matrices (which have shape (n,1) or (1,n) and are therefore always 2d in the numpy sense) is really intended, because I currently doubt that it's useful for *anyone*. (Of course you can prove me wrong there...) 3. The cited principle is not (only) about accepting matrices, it's about "honoring" them by either preserving their type for the output, and of course by doing the equivalent thing as for the equivalent numpy-array input. Currently, however, diag() is not providing the "vector-to-diagonal-matrix" functionality if you work with numpy-matrices (modulo some obvious workarounds). To me, that's a partly broken function, and it's *not* handling matrix objects "just fine". > > I don't want to introduce a backwards-compatibility-breaking special case to the > function. "Special cases aren't special enough to break the rules." Different > functionality should go into a different function. > I hope (but somehow doubt...) that I've convinced you that it's actually about applying the same logical rule to all input types, not about introducing a special case. I agree that in principle backwards compatibility could be an issue; however that would mean that people are actually using the strange behavior that diag() displays with (n,1) or (1,n) matrix-vectors (namely returning only the first element). Does anybody do that? I doubt it, but of course I can't prove it; does anybody on the list know of an example where it's used? Having said all that in this lengthy message, I don't want to push it too far. I believe that the right thing to do would be to fix diag() along the lines I suggested. If I haven't managed to convince you and/or the other developers, so be it, and I can live with a new numpy.matlib.diag() just fine. In the latter case, may I also suggest that a new numpy.matlib.diag() always return a numpy-matrix even when given a numpy-array? Background: Currently (and I think it's ok that way) the eig*() functions return the eigenvalues in a 1d-array, even for numpy-matrices as inputs. It is fairly usual to use a diagonal matrix with the eigenvalues on the diagonal in an algebraic formula. That could be achieved with n.matlib.diag(n.linalg.eig*(...)[0]) without an enclosing mat() call only if n.matlib.diag always returns a numpy-matrix. Thanks for reading until the end... cheers, Sven From kwgoodman at gmail.com Fri Jul 28 09:07:49 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 28 Jul 2006 06:07:49 -0700 Subject: [Numpy-discussion] fixing diag() for matrices In-Reply-To: <44CA0716.2000707@gmx.net> References: <44C64AA2.7070906@gmx.net> <44C652C0.9040806@gmx.net> <44CA0716.2000707@gmx.net> Message-ID: Every time I want to make a diagonal matrix from a vector I have to do a google search. So I'm with you Sven: diag(NxN matrix) should return a Nx1 matrix diag(Nx1 or 1xN matrix) should return a NxN matrix instead of the current behavior: diag(NxN matrix) returns an array diag(Nx1 matrix) returns a 1x1 array >> x matrix([[ 0.43298158, 0.84572719], [ 0.1391546 , 0.66412104]]) >> diag(x) array([ 0.43298158, 0.66412104]) >> diag(x[:,0]) array([ 0.43298158]) On 7/28/06, Sven Schreiber wrote: > Robert Kern schrieb: > > Sven Schreiber wrote: > >> That would be fine with me. However, I'd like to point out that after > >> some bug-squashing currently all numpy functions deal with > >> numpy-matrices correctly, afaik. The current behavior of numpy.diag > >> could be viewed as a violation of that principle. (Because if x has > >> shape (n,1), diag(x) returns only the first entry, which is pretty > >> stupid for a diag-function operating on a vector.) > > > > I don't think so. It's operating exactly as it is documented to. It doesn't do > > what you want, but it's not supposed to read your mind and guess that you are > > treating (n,1) and (1,n) arrays as linear algebraic vectors and that you want to > > form a diagonal matrix from that vector. It handles matrix objects just fine; it > > does not obey a particular convention that you want to use, though. That's > > neither broken nor a violation of the principle that most numpy functions should > > accept matrix objects; it's just not what you want. > > Ok, let me get some facts straight: > > 1. If you're using numpy-matrices, algebraic vectors are *automatically* > (n,1) or (1,n). I didn't make it up, and nobody has to read my mind to > understand that; it's just a *fact* of numpy (and you knew that long > before I was even aware of numpy's existence ;-). > > 2. I never claimed that the documentation of diag is wrong. I'm just > questioning whether its behavior with vectors represented as > numpy-matrices (which have shape (n,1) or (1,n) and are therefore always > 2d in the numpy sense) is really intended, because I currently doubt > that it's useful for *anyone*. (Of course you can prove me wrong there...) > > 3. The cited principle is not (only) about accepting matrices, it's > about "honoring" them by either preserving their type for the output, > and of course by doing the equivalent thing as for the equivalent > numpy-array input. Currently, however, diag() is not providing the > "vector-to-diagonal-matrix" functionality if you work with > numpy-matrices (modulo some obvious workarounds). To me, that's a partly > broken function, and it's *not* handling matrix objects "just fine". > > > > > > I don't want to introduce a backwards-compatibility-breaking special case to the > > function. "Special cases aren't special enough to break the rules." Different > > functionality should go into a different function. > > > > I hope (but somehow doubt...) that I've convinced you that it's actually > about applying the same logical rule to all input types, not about > introducing a special case. I agree that in principle backwards > compatibility could be an issue; however that would mean that people are > actually using the strange behavior that diag() displays with (n,1) or > (1,n) matrix-vectors (namely returning only the first element). > > Does anybody do that? I doubt it, but of course I can't prove it; does > anybody on the list know of an example where it's used? > > Having said all that in this lengthy message, I don't want to push it > too far. I believe that the right thing to do would be to fix diag() > along the lines I suggested. If I haven't managed to convince you and/or > the other developers, so be it, and I can live with a new > numpy.matlib.diag() just fine. > > In the latter case, may I also suggest that a new numpy.matlib.diag() > always return a numpy-matrix even when given a numpy-array? Background: > Currently (and I think it's ok that way) the eig*() functions return the > eigenvalues in a 1d-array, even for numpy-matrices as inputs. It is > fairly usual to use a diagonal matrix with the eigenvalues on the > diagonal in an algebraic formula. That could be achieved with > n.matlib.diag(n.linalg.eig*(...)[0]) without an enclosing mat() call > only if n.matlib.diag always returns a numpy-matrix. > > Thanks for reading until the end... > cheers, > Sven > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From gruben at bigpond.net.au Fri Jul 28 09:22:52 2006 From: gruben at bigpond.net.au (Gary Ruben) Date: Fri, 28 Jul 2006 23:22:52 +1000 Subject: [Numpy-discussion] possible bug in seed argument In-Reply-To: References: <44C8BA1E.606@bigpond.net.au> Message-ID: <44CA0FAC.3010503@bigpond.net.au> See below: Robert Kern wrote: > Gary Ruben wrote: >> Should >> >>> seed(1) >> act the same as >> >>> seed(array([1])) >> in the random module? > > No. They use slightly different mechanisms to seed. The integer uses RandomKit's > seeding routine. I borrowed Python's mechanism for seeding from an array of > integers. Now that it comes up, though, it is probably a good idea to use the > same mechanism for both cases. > >> It generates a traceback with the Windows 1.0b1 binary. > > Please always copy-and-paste tracebacks when reporting bugs. It works for me > with r2881; I'll rebuild with a later version and try again. Thanks Robert, Here it is for reference. Not a very deep traceback: >>> seed(1) Traceback (most recent call last): File "", line 1, in -toplevel- seed(1) File "mtrand.pyx", line 311, in mtrand.RandomState.seed SystemError: C:\sf\python\dist23\src\Objects\longobject.c:240: bad argument to internal function From yut47 at tom.com Mon Jul 31 09:57:12 2006 From: yut47 at tom.com (=?GB2312?B?IjjUwjEyLTEzyNUvyc+6oyI=?=) Date: Mon, 31 Jul 2006 21:57:12 +0800 Subject: [Numpy-discussion] =?GB2312?B?cmU7s7W85LncwO3Iy9SxsMvP7tDewbY=?= Message-ID: An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Tue Jul 25 18:09:26 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 25 Jul 2006 16:09:26 -0600 Subject: [Numpy-discussion] how to get an array with "varying" poisson distribution In-Reply-To: <200607241231.46581.haase@msg.ucsf.edu> References: <200607241231.46581.haase@msg.ucsf.edu> Message-ID: <44C69696.5050608@ee.byu.edu> Sebastian Haase wrote: >Hi, >Essentially I'm looking for the equivalent of what was in numarray: > from numarray import random_array > random_array.poisson(arr) > > > > I've just updated the mtrand library to allow this. It will be in 1.0b2 So, if you have latest SVN you can do from numpy import random random.poisson(arr) to do what you want. -Travis From stefan at sun.ac.za Fri Jul 28 10:54:00 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 28 Jul 2006 16:54:00 +0200 Subject: [Numpy-discussion] Moving docstrings from C to Python Message-ID: <20060728145400.GN6338@mentat.za.net> Hi, Travis recently advertised the add_docstring and add_newdoc functions, which are used to add documentation to functions written in C. This brings the advantage that you don't have to recompile everytime you change a docstrings, and also that the docstrings are much simpler to format (no more "\n\" tokens all over the place). I took all the docstrings from numpy.core.src.multiarray and formatted them to be used with add_docstring. I have two questions: Would anyone mind if the change was made? If not, where should they go? (numpy/add_newdocs.py or numpy/core/something) Thanks for the feedback. St?fan From mike.ressler at alum.mit.edu Tue Jul 25 15:17:43 2006 From: mike.ressler at alum.mit.edu (Mike Ressler) Date: Tue, 25 Jul 2006 12:17:43 -0700 Subject: [Numpy-discussion] ***[Possible UCE]*** Bug in memmap/python allocation code? In-Reply-To: <44C5BE6A.7020007@ieee.org> References: <268febdf0607241736g5f5135a3ub26acd3e8129fec6@mail.gmail.com> <44C5BE6A.7020007@ieee.org> Message-ID: <268febdf0607251217p686de176nec9177f9d99536ba@mail.gmail.com> On 7/24/06, Travis Oliphant wrote: > > Mike Ressler wrote: > > I'm trying to work with memmaps on very large files, i.e. > 2 GB, up > > to 10 GB. Can't believe I'm really the first, but so be it. I just discovered the problem. All the places where > PyObject_AsBuffer is used needs to have the final argument > changed to Py_ssize_t (which in arrayobject.h is defined as int if you > are using less than Python 2.5). > > This should be fixed in SVN shortly.... Yeess! My little script can handle everything I've thrown at it now. It can read a 10 GB raw file, strip the top 16 bits, rearrange pixels, byte swap, and write it all back to a 5 GB file in 16 minutes flat. Not bad at all. And I've verified that the output is correct ... If someone can explain the rules of engagement for Lightning Talks, I'm thinking about presenting this at SciPy 2006. Then you'll see there is a reason for my madness. As an aside, the developer pages could use some polish on explaining the different svn areas, and how to get what one wants. An svn checkout as described on the page gets you the 1.1 branch that DOES NOT have the updated memmap fix. After a minute or two of exploring, I found that "svn co http://svn.scipy.org/svn/numpy/branches/ver1.0/numpy numpy" got me what I wanted. Thanks for your help and the quick solution. FWIW, I got my copy of the book a couple of weeks ago; very nice. Mike -- mike.ressler at alum.mit.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Fri Jul 28 12:53:30 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 28 Jul 2006 10:53:30 -0600 Subject: [Numpy-discussion] possible bug in seed argument In-Reply-To: <44CA0FAC.3010503@bigpond.net.au> References: <44C8BA1E.606@bigpond.net.au> <44CA0FAC.3010503@bigpond.net.au> Message-ID: <44CA410A.7010102@ieee.org> Gary Ruben wrote: > See below: > > Robert Kern wrote: > >> Gary Ruben wrote: >> >>> Should >>> >>> seed(1) >>> act the same as >>> >>> seed(array([1])) >>> in the random module? >>> >> No. They use slightly different mechanisms to seed. The integer uses RandomKit's >> seeding routine. I borrowed Python's mechanism for seeding from an array of >> integers. Now that it comes up, though, it is probably a good idea to use the >> same mechanism for both cases. >> >> >>> It generates a traceback with the Windows 1.0b1 binary. >>> >> Please always copy-and-paste tracebacks when reporting bugs. It works for me >> with r2881; I'll rebuild with a later version and try again. >> I can verify that this now works with numpy 1.0b1 on Python 2.3. The change was using an updated Pyrex for generating mtrand.c (it no longer calls PyLong_AsUnsignedLong on integer objects as it did before). -Travis From oliphant.travis at ieee.org Fri Jul 28 12:42:40 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 28 Jul 2006 10:42:40 -0600 Subject: [Numpy-discussion] possible bug in seed argument In-Reply-To: <44CA0FAC.3010503@bigpond.net.au> References: <44C8BA1E.606@bigpond.net.au> <44CA0FAC.3010503@bigpond.net.au> Message-ID: <44CA3E80.6010806@ieee.org> Gary Ruben wrote: > See below: > > Robert Kern wrote: > >> Gary Ruben wrote: >> >>> Should >>> >>> seed(1) >>> act the same as >>> >>> seed(array([1])) >>> in the random module? >>> >> No. They use slightly different mechanisms to seed. The integer uses RandomKit's >> seeding routine. I borrowed Python's mechanism for seeding from an array of >> integers. Now that it comes up, though, it is probably a good idea to use the >> same mechanism for both cases. >> >> >>> It generates a traceback with the Windows 1.0b1 binary. >>> >> Please always copy-and-paste tracebacks when reporting bugs. It works for me >> with r2881; I'll rebuild with a later version and try again. >> > > Thanks Robert, > Here it is for reference. Not a very deep traceback: > It looks like this is a Python 2.3 error. In Python 2.4, the PyLong_AsUnsignedLong was changed to accept Python integers as well as Python longs. The problem with fixing it is I'm not sure where PyLong_AsUnsignedLong is being called. I can't find that call in the current source code. It's possible that Pyrex changed and this function is no longer being used in mtrand.c -Travis From oliphant.travis at ieee.org Fri Jul 28 12:56:29 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 28 Jul 2006 10:56:29 -0600 Subject: [Numpy-discussion] possible bug in seed argument In-Reply-To: <44CA410A.7010102@ieee.org> References: <44C8BA1E.606@bigpond.net.au> <44CA0FAC.3010503@bigpond.net.au> <44CA410A.7010102@ieee.org> Message-ID: <44CA41BD.6090305@ieee.org> Travis Oliphant wrote: > I can verify that this now works with numpy 1.0b1 on Python 2.3. The > change was using an updated Pyrex for generating mtrand.c (it no longer > calls PyLong_AsUnsignedLong on integer objects as it did before). > > Scratch that. I meant it works with latest numpy (will be 1.0b2) because the mtrand.c was not re-generated until recently. -Travis From robert.kern at gmail.com Fri Jul 28 12:59:06 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 28 Jul 2006 11:59:06 -0500 Subject: [Numpy-discussion] fixing diag() for matrices In-Reply-To: <44CA0716.2000707@gmx.net> References: <44C64AA2.7070906@gmx.net> <44C652C0.9040806@gmx.net> <44CA0716.2000707@gmx.net> Message-ID: Sven Schreiber wrote: > Robert Kern schrieb: >> Sven Schreiber wrote: >>> That would be fine with me. However, I'd like to point out that after >>> some bug-squashing currently all numpy functions deal with >>> numpy-matrices correctly, afaik. The current behavior of numpy.diag >>> could be viewed as a violation of that principle. (Because if x has >>> shape (n,1), diag(x) returns only the first entry, which is pretty >>> stupid for a diag-function operating on a vector.) >> I don't think so. It's operating exactly as it is documented to. It doesn't do >> what you want, but it's not supposed to read your mind and guess that you are >> treating (n,1) and (1,n) arrays as linear algebraic vectors and that you want to >> form a diagonal matrix from that vector. It handles matrix objects just fine; it >> does not obey a particular convention that you want to use, though. That's >> neither broken nor a violation of the principle that most numpy functions should >> accept matrix objects; it's just not what you want. > > Ok, let me get some facts straight: > > 1. If you're using numpy-matrices, algebraic vectors are *automatically* > (n,1) or (1,n). I didn't make it up, and nobody has to read my mind to > understand that; it's just a *fact* of numpy (and you knew that long > before I was even aware of numpy's existence ;-). > > 2. I never claimed that the documentation of diag is wrong. I'm just > questioning whether its behavior with vectors represented as > numpy-matrices (which have shape (n,1) or (1,n) and are therefore always > 2d in the numpy sense) is really intended, because I currently doubt > that it's useful for *anyone*. (Of course you can prove me wrong there...) > > 3. The cited principle is not (only) about accepting matrices, it's > about "honoring" them by either preserving their type for the output, > and of course by doing the equivalent thing as for the equivalent > numpy-array input. Currently, however, diag() is not providing the > "vector-to-diagonal-matrix" functionality if you work with > numpy-matrices (modulo some obvious workarounds). To me, that's a partly > broken function, and it's *not* handling matrix objects "just fine". Unfortunately, there is a many-to-one correspondence between numpy-array inputs and matrix-array inputs. Going the other way is ambiguous. There is still some guessing involved. >> I don't want to introduce a backwards-compatibility-breaking special case to the >> function. "Special cases aren't special enough to break the rules." Different >> functionality should go into a different function. > > I hope (but somehow doubt...) that I've convinced you that it's actually > about applying the same logical rule to all input types, not about > introducing a special case. But you can't code it without special casing. None of the functions should be written like this: if type(input) is matrix: # do this one thing else: # do a different thing They belong in two different functions. Or methods. You can't extend it to the n different subclasses of ndarray that people might want to use. This kind of special casing ought to give you screaming jibblies. The current behavior of diag() already gives me the screaming jibblies given that it both extracts a diagonal from an MxN array and creates a NxN array from an N array. If I wrote an API like that at work, my project manager would come over to my office and Have Words with me. And then tell me to rewrite it. It's simply bad design. I think making a method on matrix objects that does what you want is probably your best bet. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From tom.denniston at alum.dartmouth.org Fri Jul 28 13:17:39 2006 From: tom.denniston at alum.dartmouth.org (Tom Denniston) Date: Fri, 28 Jul 2006 12:17:39 -0500 Subject: [Numpy-discussion] object array binary pickling and nan Message-ID: It seems that object arrays will not pickle if they have a nan when the pickle protocol is set to binary. From what I can tell the object array simply delegates to python (which makes sense) which in turn cannot pickle nans in binary format. It is unfortunate because it is very useful to have heterogenous arrays that include nans. What do other people do in this situation? Does anyone know why python has this limitation? Is there an intelligent workaround other than search and replace? Would it be worth it to put an intelligent workaround into numpy so it is transparent to the user? I was wondering what people thought. Code that reproduces the problem: This is regular python: pickle.dumps(numpy.nan, 2) SystemError: frexp() result out of range This is fine in numpy: pickle.dumps(numpy.array([numpy.nan]), 2) This breaks: pickle.dumps(numpy.array([numpy.nan], numpy.PyObject), 2) SystemError: frexp() result out of range --Tom From oliphant.travis at ieee.org Fri Jul 28 12:29:25 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 28 Jul 2006 10:29:25 -0600 Subject: [Numpy-discussion] weave and Numeric vs. weave and numpy In-Reply-To: <44C9EE89.9070000@ilt.fraunhofer.de> References: <44C87182.8070900@ilt.fraunhofer.de> <44C9EE89.9070000@ilt.fraunhofer.de> Message-ID: <44CA3B65.5010805@ieee.org> Rolf Wester wrote: > David Grant wrote: > >> A factor of 2! Is that short for a factor of **2? >> >> > That means Numeric/weave takes about 17 s (on my PC), and the > numby/weave version takes about 32 s. > Please show your code. There are several factors that have changed here. For example, Python 2.4 is different that Python 2.3. There are only a few usages when Numeric is faster than NumPy and in most of these using regular Python lists is faster than Numeric and NumPy anyway. Nonetheless, there are still improvements that can be made to NumPy. Which version of NumPy are you using? -Travis From loredo at astro.cornell.edu Fri Jul 28 14:12:07 2006 From: loredo at astro.cornell.edu (Tom Loredo) Date: Fri, 28 Jul 2006 14:12:07 -0400 Subject: [Numpy-discussion] Release notes please; possible bugs Message-ID: <1154110327.44ca53775b7a4@astrosun2.astro.cornell.edu> Hi folks, It is turning out to be much more of a headache moving from numpy-0.9.x to numpy-1.0b1 than it ever was moving from Numeric to numpy. Could someone please throw together some release notes (the scipy & sourceforge RN pages are blank) or a Wiki page warning what has to be changed? Here are some stumbling blocks I've come across: * Type names are all now lower case. I know this started happening late in 0.9.x, but I only know it by accident, via some email from Travis. This is just one example of the kind of change that really needs to be prominently noted somewhere. * .nonzero() now returns a 1-tuple with an array of indices, for a 1-d array. Before it returned an array of indices. * In the C API, ContiguousFromObject is now ContiguousFromAny. I am surprised that my libraries compile with no errors; I only get a runtime error. Shouldn't I be warned about this at compile-time? * mtrand still uses ContiguousFromObject in a few places; these give exceptions when you call, e.g., setstate. * rand, fft, ift are no longer in the numpy namespace. Is this a permanent change or a bug? There are probably other obstacles I'll come across, but this is what I encountered in code that ran perfectly with 0.9.6. After several hours of work it's still not running. -Tom Loredo ------------------------------------------------- This mail sent through IMP: http://horde.org/imp/ From loredo at astro.cornell.edu Fri Jul 28 14:16:23 2006 From: loredo at astro.cornell.edu (Tom Loredo) Date: Fri, 28 Jul 2006 14:16:23 -0400 Subject: [Numpy-discussion] Release notes please; possible bugs Message-ID: <1154110583.44ca5477010af@astrosun2.astro.cornell.edu> PS: Sorry, I should've mentioned it's 1.0b2.dev2915 that I'm using, not 1.0b1. -Tom ------------------------------------------------- This mail sent through IMP: http://horde.org/imp/ From daniel.poitras at nrc.ca Fri Jul 28 14:14:51 2006 From: daniel.poitras at nrc.ca (Daniel Poitras) Date: Fri, 28 Jul 2006 18:14:51 +0000 (UTC) Subject: [Numpy-discussion] numpy.arcsin of complex number... strange Message-ID: Hello, I tried different arcsin functions on a complex number (z=0.52+0j) and obtained the following results: cmath.asin(z) gives (0.54685095069594414+0j) #okay -1j*log(1j*z+sqrt(1-z**2)) gives (0.54685095069594414+0j) #okay, by definition numarray.arcsin(z) gives (0.54685095069594414+0j) #still okay but numpy.arcsin(z) gives (0.54685095069594414+0.54685095069594414j) #bug?? Is it really a bug in numpy, or is there another explanation? Thanks Daniel Poitras Ottawa From dd55 at cornell.edu Fri Jul 28 14:33:17 2006 From: dd55 at cornell.edu (Darren Dale) Date: Fri, 28 Jul 2006 14:33:17 -0400 Subject: [Numpy-discussion] uniform() regression(?) in svn In-Reply-To: <44C7A4D1.3010006@ieee.org> References: <44C7A4D1.3010006@ieee.org> Message-ID: <200607281433.17709.dd55@cornell.edu> On Wednesday 26 July 2006 13:22, Travis Oliphant wrote: > Andrew Jaffe wrote: > > Hi- > > > > On PPC Mac OSX universal build 2.4.3, gcc 4.0, > > > > > > In [1]: import numpy as N > > > > In [2]: print N.__version__ > > 1.0.2897 > > > > In [3]: N.random.uniform(0,1) > > Segmentation fault > > > > (This originally showed up in the Ticket 83 regression test during > > numpy.test()...) > > This should be O.K. now. I dont know if this is related or not, but I just did an update from the trunk ('1.0b2.dev2918') and discovered the following error: ====================================================================== FAIL: Ticket #99 ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/numpy/core/tests/test_regression.py", line 182, in check_intp self.failUnlessRaises(OverflowError,N.intp,'0xb72a7008',16) AssertionError: OverflowError not raised ---------------------------------------------------------------------- I'm compiling with gcc-4.1.1, python-2.4.3. Darren From strawman at astraw.com Fri Jul 28 15:28:25 2006 From: strawman at astraw.com (Andrew Straw) Date: Fri, 28 Jul 2006 12:28:25 -0700 Subject: [Numpy-discussion] object array binary pickling and nan In-Reply-To: References: Message-ID: <44CA6559.90005@astraw.com> Tom Denniston wrote: >Code that reproduces the problem: > >This is regular python: >pickle.dumps(numpy.nan, 2) >SystemError: frexp() result out of range > >This is fine in numpy: >pickle.dumps(numpy.array([numpy.nan]), 2) > >This breaks: >pickle.dumps(numpy.array([numpy.nan], numpy.PyObject), 2) >SystemError: frexp() result out of range > > It seems to me that the example that shows what you want is the only example that numpy has control over, anyway. In the other two cases, you're really using a Python object. So it's really a Python question as to why it can't pickle nans. My understanding on this is weak, but here's one summary, albeit dated: http://www.python.org/dev/summary/2000-10-1.html Here's something more recent: http://mail.python.org/pipermail/python-list/2005-July/290272.html NumPy sidesteps these issues by "forcing" IEEE-754 behavior with clever C written AFAIK by the numarray people. These apparently work even if the C compiler doesn't "officially" support IEEE-754, but it's not going to work on every platform that Python targets. You may also want to take a look at http://scipy.org/FAQ#head-fff4d6fce7528974185715153cfbc1a191dcb915 From cookedm at physics.mcmaster.ca Fri Jul 28 15:38:55 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri, 28 Jul 2006 15:38:55 -0400 Subject: [Numpy-discussion] object array binary pickling and nan In-Reply-To: References: Message-ID: <20060728153855.12998542@arbutus.physics.mcmaster.ca> On Fri, 28 Jul 2006 12:17:39 -0500 "Tom Denniston" wrote: > It seems that object arrays will not pickle if they have a nan when > the pickle protocol is set to binary. From what I can tell the object > array simply delegates to python (which makes sense) which in turn > cannot pickle nans in binary format. It is unfortunate because it is > very useful to have heterogenous arrays that include nans. What do > other people do in this situation? Does anyone know why python has > this limitation? Is there an intelligent workaround other than search > and replace? Would it be worth it to put an intelligent workaround > into numpy so it is transparent to the user? I was wondering what > people thought. > > Code that reproduces the problem: > > This is regular python: > pickle.dumps(numpy.nan, 2) > SystemError: frexp() result out of range > > This is fine in numpy: > pickle.dumps(numpy.array([numpy.nan]), 2) > > This breaks: > pickle.dumps(numpy.array([numpy.nan], numpy.PyObject), 2) > SystemError: frexp() result out of range I believe this will be fixed in Python 2.5 (see http://www.python.org/sf/445484). I guess your best alternative would be to use a numpy scalar object (like double) instead of a Python float for NaN's. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From Chris.Barker at noaa.gov Fri Jul 28 12:43:44 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 28 Jul 2006 09:43:44 -0700 Subject: [Numpy-discussion] fixing diag() for matrices In-Reply-To: References: <44C64AA2.7070906@gmx.net> <44C652C0.9040806@gmx.net> <44CA0716.2000707@gmx.net> Message-ID: <44CA3EC0.9020404@noaa.gov> Keith Goodman wrote: > diag(NxN matrix) should return a Nx1 matrix > diag(Nx1 or 1xN matrix) should return a NxN matrix This is the key problem: extracting the diagonal of a matrix and creating a matrix from a diagonal are two different operations: overloading one function to do both was a bad idea to begin with. Maybe we should just keep diag() as is is for backward compatibility (deprecated), and make: get_diag() and make_diag() instead. Then it would be unambiguous what you wanted with: make_diag() You can call them something else, but you get the idea. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Daniel.Poitras at nrc-cnrc.gc.ca Fri Jul 28 16:07:24 2006 From: Daniel.Poitras at nrc-cnrc.gc.ca (Poitras, Daniel) Date: Fri, 28 Jul 2006 16:07:24 -0400 Subject: [Numpy-discussion] numpy.arcsin strange behavior? Message-ID: <5AA740E586BF424D8434E38EE3687EE101CA5830@nrccenexb2.nrc.ca> Hello, I tried different arcsin functions on a "complex" number (z=0.52+0j) and obtained the following results: cmath.asin(z) gives (0.54685095069594414+0j) #okay -1j*log(1j*z+sqrt(1-z**2)) gives (0.54685095069594414+0j) #okay, by definition numarray.arcsin(z) gives (0.54685095069594414+0j) #still okay but numpy.arcsin(z) gives (0.54685095069594414+0.54685095069594414j) #bug?? Is it a bug in numpy, or is there another explanation? Thanks Daniel Poitras Ottawa -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.huard at gmail.com Fri Jul 28 11:11:30 2006 From: david.huard at gmail.com (David Huard) Date: Fri, 28 Jul 2006 11:11:30 -0400 Subject: [Numpy-discussion] Moving docstrings from C to Python In-Reply-To: <20060728145400.GN6338@mentat.za.net> References: <20060728145400.GN6338@mentat.za.net> Message-ID: <91cf711d0607280811k3bbd9be3i1a813444489bb57e@mail.gmail.com> 2006/7/28, Stefan van der Walt : > > Hi, > > Travis recently advertised the add_docstring and add_newdoc functions, > which are used to add documentation to functions written in C. This > brings the advantage that you don't have to recompile everytime you > change a docstrings, and also that the docstrings are much simpler to > format (no more "\n\" tokens all over the place). > > I took all the docstrings from numpy.core.src.multiarray and formatted > them to be used with add_docstring. I have two questions: > > Would anyone mind if the change was made? I don't, I wanted to do the same thing. numpy/add_newdocs.py +1 David -------------- next part -------------- An HTML attachment was scrubbed... URL: From nvf at MIT.EDU Fri Jul 28 16:25:00 2006 From: nvf at MIT.EDU (Nick Fotopoulos) Date: Fri, 28 Jul 2006 16:25:00 -0400 Subject: [Numpy-discussion] C Extension intp problem Message-ID: <2EAC175B-39C7-46EF-8116-D9CA74DB3938@mit.edu> Dear all, I had a working C extension, but after upgrading to a recent numpy from SVN, I can no longer compile it. I've fixed the deprecation warnings, but can't get past this: frgetvect.c:51: error: 'intp' undeclared (first use in this function) Now, I'm quite confused since I thought that intp should be a global thing, not numpy related, and I believe I'm using the shipped Apple gcc version. Version info: nvf at whitedwarf:~/temp/v6r20/pyfrgetvect$ python -V Python 2.4.1 nvf at whitedwarf:~/temp/v6r20/pyfrgetvect$ python -c "import numpy; print numpy.__version__" 1.1.2881 nvf at whitedwarf:~/temp/v6r20/pyfrgetvect$ gcc --version powerpc-apple-darwin8-gcc-4.0.0 (GCC) 4.0.0 20041026 (Apple Computer, Inc. build 4061) Can anyone help me? Thanks, Nick Full build output: nvf at whitedwarf:~/temp/v6r20/pyfrgetvect$ python setup.py build running build running build_ext building 'frgetvect' extension C compiler: gcc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 -Wall -Wstrict- prototypes creating build creating build/temp.darwin-8.7.0-Power_Macintosh-2.4 compile options: '-DMAJOR_VERSION=0 -DMINOR_VERSION=2 -I../src -I/ Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site- packages/numpy/core/include -I/Library/Frameworks/Python.framework/ Versions/2.4/include/python2.4 -c' extra options: '-w' gcc: frgetvect.c frgetvect.c: In function 'frgetvect': frgetvect.c:51: error: 'intp' undeclared (first use in this function) frgetvect.c:51: error: (Each undeclared identifier is reported only once frgetvect.c:51: error: for each function it appears in.) frgetvect.c:51: error: parse error before "shape" frgetvect.c:138: error: 'shape' undeclared (first use in this function) frgetvect.c:138: error: parse error before "nData" frgetvect.c: In function 'frgetvect': frgetvect.c:51: error: 'intp' undeclared (first use in this function) frgetvect.c:51: error: (Each undeclared identifier is reported only once frgetvect.c:51: error: for each function it appears in.) frgetvect.c:51: error: parse error before "shape" frgetvect.c:138: error: 'shape' undeclared (first use in this function) frgetvect.c:138: error: parse error before "nData" error: Command "gcc -fno-strict-aliasing -Wno-long-double -no-cpp- precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 -Wall - Wstrict-prototypes -DMAJOR_VERSION=0 -DMINOR_VERSION=2 -I../src -I/ Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site- packages/numpy/core/include -I/Library/Frameworks/Python.framework/ Versions/2.4/include/python2.4 -c frgetvect.c -o build/ temp.darwin-8.7.0-Power_Macintosh-2.4/frgetvect.o -w" failed with exit status 1 From robert.kern at gmail.com Fri Jul 28 16:37:55 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 28 Jul 2006 15:37:55 -0500 Subject: [Numpy-discussion] fixing diag() for matrices In-Reply-To: <44CA3EC0.9020404@noaa.gov> References: <44C64AA2.7070906@gmx.net> <44C652C0.9040806@gmx.net> <44CA0716.2000707@gmx.net> <44CA3EC0.9020404@noaa.gov> Message-ID: Christopher Barker wrote: > Keith Goodman wrote: >> diag(NxN matrix) should return a Nx1 matrix >> diag(Nx1 or 1xN matrix) should return a NxN matrix > > This is the key problem: extracting the diagonal of a matrix and > creating a matrix from a diagonal are two different operations: > overloading one function to do both was a bad idea to begin with. > > Maybe we should just keep diag() as is is for backward compatibility > (deprecated), and make: > > get_diag() and make_diag() instead. > > Then it would be unambiguous what you wanted with: > > make_diag() > > You can call them something else, but you get the idea. As .diagonal() is a method on both arrays and matrices, we don't need a separate get_diag(). In the abstract, I heartily approve. However, I'm not sure that deprecation in favor of a clumsier name is going to be effective in practice. diag([a, b, c]) is just too tempting and will remain an attractive nuisance. It matches mathematical notation ever so nicely. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cookedm at physics.mcmaster.ca Fri Jul 28 16:47:50 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri, 28 Jul 2006 16:47:50 -0400 Subject: [Numpy-discussion] C Extension intp problem In-Reply-To: <2EAC175B-39C7-46EF-8116-D9CA74DB3938@mit.edu> References: <2EAC175B-39C7-46EF-8116-D9CA74DB3938@mit.edu> Message-ID: <20060728164750.7a79ab2c@arbutus.physics.mcmaster.ca> On Fri, 28 Jul 2006 16:25:00 -0400 Nick Fotopoulos wrote: > Dear all, > > I had a working C extension, but after upgrading to a recent numpy > from SVN, I can no longer compile it. I've fixed the deprecation > warnings, but can't get past this: > > frgetvect.c:51: error: 'intp' undeclared (first use in this function) > > Now, I'm quite confused since I thought that intp should be a global > thing, not numpy related, and I believe I'm using the shipped Apple > gcc version. To avoid namespace clashes with other libraries, we've switched to adding a unique prefix 'npy_' to types (and NPY_ to #defines). You can either change your code to use 'npy_intp', or add #include "numpy/noprefix.h" to the top. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From nvf at MIT.EDU Fri Jul 28 16:54:18 2006 From: nvf at MIT.EDU (Nick Fotopoulos) Date: Fri, 28 Jul 2006 16:54:18 -0400 Subject: [Numpy-discussion] C Extension intp problem In-Reply-To: <20060728164750.7a79ab2c@arbutus.physics.mcmaster.ca> References: <2EAC175B-39C7-46EF-8116-D9CA74DB3938@mit.edu> <20060728164750.7a79ab2c@arbutus.physics.mcmaster.ca> Message-ID: <0352BC26-CEEC-43D0-8EBF-40F17D98762C@MIT.EDU> On Jul 28, 2006, at 4:47 PM, David M. Cooke wrote: > On Fri, 28 Jul 2006 16:25:00 -0400 > Nick Fotopoulos wrote: > >> Dear all, >> >> I had a working C extension, but after upgrading to a recent numpy >> from SVN, I can no longer compile it. I've fixed the deprecation >> warnings, but can't get past this: >> >> frgetvect.c:51: error: 'intp' undeclared (first use in this function) >> >> Now, I'm quite confused since I thought that intp should be a global >> thing, not numpy related, and I believe I'm using the shipped Apple >> gcc version. > > To avoid namespace clashes with other libraries, we've switched to > adding a > unique prefix 'npy_' to types (and NPY_ to #defines). You can > either change > your code to use 'npy_intp', or add > > #include "numpy/noprefix.h" > > to the top. > Aha! I was under the impression that intp was a native C thing. Thanks for clarifying. Take care, Nick From ndarray at mac.com Fri Jul 28 17:10:05 2006 From: ndarray at mac.com (Sasha) Date: Fri, 28 Jul 2006 17:10:05 -0400 Subject: [Numpy-discussion] Moving docstrings from C to Python In-Reply-To: <20060728145400.GN6338@mentat.za.net> References: <20060728145400.GN6338@mentat.za.net> Message-ID: On 7/28/06, Stefan van der Walt wrote: > Would anyone mind if the change was made? If not, where should they > go? (numpy/add_newdocs.py or numpy/core/something) Another +1 for numpy/add_newdocs.py and a suggestion: check for Py_OptimizeFlag > 1 in add_newdoc so that docstrings are not loaded if python is invoked with -OO option. This will improve import numpy time and reduce the memory footprint. I'll make the change if no one objects. From haase at msg.ucsf.edu Tue Jul 25 18:17:33 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 25 Jul 2006 15:17:33 -0700 Subject: [Numpy-discussion] how to get an array with "varying" poisson distribution In-Reply-To: <44C69696.5050608@ee.byu.edu> References: <200607241231.46581.haase@msg.ucsf.edu> <44C69696.5050608@ee.byu.edu> Message-ID: <200607251517.34030.haase@msg.ucsf.edu> On Tuesday 25 July 2006 15:09, Travis Oliphant wrote: > Sebastian Haase wrote: > >Hi, > >Essentially I'm looking for the equivalent of what was in numarray: > > from numarray import random_array > > random_array.poisson(arr) > > I've just updated the mtrand library to allow this. It will be in 1.0b2 > > So, if you have latest SVN you can do > > from numpy import random > random.poisson(arr) > > to do what you want. > > > > -Travis Great - thanks - you are awesome !! -Seb. From svetosch at gmx.net Fri Jul 28 18:41:07 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Sat, 29 Jul 2006 00:41:07 +0200 Subject: [Numpy-discussion] fixing diag() for matrices In-Reply-To: References: <44C64AA2.7070906@gmx.net> <44C652C0.9040806@gmx.net> <44CA0716.2000707@gmx.net> <44CA3EC0.9020404@noaa.gov> Message-ID: <44CA9283.5030108@gmx.net> Here's my attempt at summarizing the diag-discussion. The symptom is that currently transforming something like the vector a b c into the diagonal matrix a 0 0 0 b 0 0 0 c is not directly possible if you're working with numpy-matrices (i.e. the vector is of type matrix and has shape (n,1) or (1,n)). Blindly doing a diag(vector) just gives you the first element of the vector. Instead you need to do something like mat(diag(vector.A.squeeze). Proposals that were put forward: 1) Modify the existing diag() to treat not only 1d-arrays as vectors, but also numpy-matrices of shapes (1,n) and (n,1), and therefore have diag() turn them into a diagonal matrix with all their elements. This would be my preferred solution. Robert is against this, because it is not 100% backwards compatible and because it requires the diag-function to differentiate between the input types. 2) Deprecate the use of diag which is overloaded with making diagonal matrices as well as getting diagonals. Instead, use the existing .diagonal() for getting a diagonal, and introduce a new make_diag() function which could easily work for numpy-arrays and numpy-matrices alike. This is a blend of Christopher's and Robert's comments. As robert noted, the problem is that diag() will probably be more attractive than make_diag() for numpy-array users, and so make_diag() would effectively only be used by numpy-matrix users, a split which would not be nice. An effectively equivalent solution would then be a new method .make_diag() for numpy-matrices, which was suggested by Robert (if I understood correctly). 3) Add a new function diag() to numpy.matlib implementing the workaround, something like: def diag(vector): return mat(N.diag(vector.A.squeeze) (not sure if all namespace issues are avoided like that, or if it's efficient) That way, numpy.matlib.diag() could be used with numpy-matrices just like numpy.diag() with numpy-arrays. Side effect: this even turns a 1d-array into a numpy-matrix, which is useful e.g. for dealing with eigenvalues (which are 1d-arrays even for numpy-matrix input). The third solution is probably the way of least resistance. However, afaik this would be the first time that numpy needs different functions to perform essentially the same operation. Since Robert already cited the obligatory zen phrase, what about this special casing? Anyway, any of those solutions is better than the status quo imo. I'd be happy if one of them could be implemented. When I get back from my vacation and in case nothing has been changed, I'm going to file a ticket so that it doesn't get forgotten. Thanks for your patience and have a good time! Sven Robert Kern schrieb: > Christopher Barker wrote: >> Keith Goodman wrote: >>> diag(NxN matrix) should return a Nx1 matrix >>> diag(Nx1 or 1xN matrix) should return a NxN matrix >> This is the key problem: extracting the diagonal of a matrix and >> creating a matrix from a diagonal are two different operations: >> overloading one function to do both was a bad idea to begin with. >> >> Maybe we should just keep diag() as is is for backward compatibility >> (deprecated), and make: >> >> get_diag() and make_diag() instead. >> >> Then it would be unambiguous what you wanted with: >> >> make_diag() >> >> You can call them something else, but you get the idea. > > As .diagonal() is a method on both arrays and matrices, we don't need a separate > get_diag(). > > In the abstract, I heartily approve. However, I'm not sure that deprecation in > favor of a clumsier name is going to be effective in practice. diag([a, b, c]) > is just too tempting and will remain an attractive nuisance. It matches > mathematical notation ever so nicely. > From robert.kern at gmail.com Fri Jul 28 18:57:01 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 28 Jul 2006 17:57:01 -0500 Subject: [Numpy-discussion] fixing diag() for matrices In-Reply-To: <44CA9283.5030108@gmx.net> References: <44C64AA2.7070906@gmx.net> <44C652C0.9040806@gmx.net> <44CA0716.2000707@gmx.net> <44CA3EC0.9020404@noaa.gov> <44CA9283.5030108@gmx.net> Message-ID: Sven Schreiber wrote: > The third solution is probably the way of least resistance. However, > afaik this would be the first time that numpy needs different functions > to perform essentially the same operation. Since Robert already cited > the obligatory zen phrase, what about this special casing? It isn't. Look at numpy/matlib.py -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From aisaac at american.edu Fri Jul 28 19:25:39 2006 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 28 Jul 2006 19:25:39 -0400 Subject: [Numpy-discussion] fixing diag() for matrices In-Reply-To: References: <44C64AA2.7070906@gmx.net> <44C652C0.9040806@gmx.net> <44CA0716.2000707@gmx.net> Message-ID: The following are from pyGAUSS. fwiw, Alan Isaac #diag: diagonal of matrix as column vector (2D only!) def diag(x): if not isinstance(x,numpy.matrix): x = numpy.asanyarray(x) assert(len(x.shape)==2), "For 2-d arrays only." return x.diagonal(offset=0,axis1=-2,axis2=-1).reshape((-1,1)) else: return x.diagonal().T #diagrv: insert v as diagonal of matrix x (2D only!) def diagrv(x,v,copy=True): x = numpy.matrix( x, copy=copy ) stride = 1 + x.shape[1] x.flat[ slice(0,x.size,stride) ] = v return x From pruggera at gmail.com Sat Jul 29 11:30:55 2006 From: pruggera at gmail.com (Phil Ruggera) Date: Sat, 29 Jul 2006 08:30:55 -0700 Subject: [Numpy-discussion] Mean of n values within an array Message-ID: I rewrote some python code using numpy to do a performance comparison. The results were the opposite of what I wanted. Numpy was slower than Python without numpy. Is there something wrong with my approach? # mean of n values within an array import numpy, time def nmean(list,n): a = [] for i in range(1,len(list)+1): start = i-n divisor = n if start < 0: start = 0 divisor = i a.append(sum(list[start:i])/divisor) return a t = [1.0*i for i in range(1400)] start = time.clock() for x in range(100): nmean(t,50) print "regular python took: %f sec."%(time.clock() - start) def numpy_nmean(list,n): a = numpy.empty(len(list),dtype=float) for i in range(1,len(list)+1): start = i-n if start < 0: start = 0 a[i-1] = list[start:i].mean(0) return a t = numpy.arange(0,1400,dtype=float) start = time.clock() for x in range(100): numpy_nmean(t,50) print "numpy took: %f sec."%(time.clock() - start) Results: regular python took: 1.215274 sec. numpy took: 2.499299 sec. From charlesr.harris at gmail.com Sat Jul 29 13:20:51 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 29 Jul 2006 11:20:51 -0600 Subject: [Numpy-discussion] Mean of n values within an array In-Reply-To: References: Message-ID: Hmmm, I rewrote the subroutine a bit. def numpy_nmean(list,n): a = numpy.empty(len(list),dtype=float) b = numpy.cumsum(list) for i in range(0,len(list)): if i < n : a[i] = b[i]/(i+1) else : a[i] = (b[i] - b[i-n])/(i+1) return a and got regular python took: 0.750000 sec. numpy took: 0.380000 sec. The precision of the new routine is admittedly suspect. Convolve can also be used: def numpy_nmean_conv(list,n): b = numpy.ones(n,dtype=float) a = numpy.convolve(list,b,mode="full") for i in range(0,len(list)): if i < n : a[i] /= i + 1 else : a[i] /= n return a[:len(list)] Which gives numpy convolve took: 0.260000 sec. You might actually want mode="valid" which would only return averages over runs of length n. One oddity here is that I think convolution should be slower than cumsum and that raises my suspicions about the latter, it seems much too slow. Chuck On 7/29/06, Phil Ruggera wrote: > > I rewrote some python code using numpy to do a performance comparison. > The results were the opposite of what I wanted. Numpy was slower > than Python without numpy. Is there something wrong with my approach? > > # mean of n values within an array > import numpy, time > def nmean(list,n): > a = [] > for i in range(1,len(list)+1): > start = i-n > divisor = n > if start < 0: > start = 0 > divisor = i > a.append(sum(list[start:i])/divisor) > return a > > t = [1.0*i for i in range(1400)] > start = time.clock() > for x in range(100): > nmean(t,50) > print "regular python took: %f sec."%(time.clock() - start) > > def numpy_nmean(list,n): > a = numpy.empty(len(list),dtype=float) > for i in range(1,len(list)+1): > start = i-n > if start < 0: > start = 0 > a[i-1] = list[start:i].mean(0) > return a > > t = numpy.arange(0,1400,dtype=float) > start = time.clock() > for x in range(100): > numpy_nmean(t,50) > print "numpy took: %f sec."%(time.clock() - start) > > > Results: > regular python took: 1.215274 sec. > numpy took: 2.499299 sec. > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share > your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Jul 29 13:32:49 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 29 Jul 2006 11:32:49 -0600 Subject: [Numpy-discussion] fixing diag() for matrices In-Reply-To: <44CA9283.5030108@gmx.net> References: <44C64AA2.7070906@gmx.net> <44C652C0.9040806@gmx.net> <44CA0716.2000707@gmx.net> <44CA3EC0.9020404@noaa.gov> <44CA9283.5030108@gmx.net> Message-ID: Hi Sven, On 7/28/06, Sven Schreiber wrote: > > Here's my attempt at summarizing the diag-discussion. 2) Deprecate the use of diag which is overloaded with making diagonal > matrices as well as getting diagonals. Instead, use the existing > .diagonal() for getting a diagonal, and introduce a new make_diag() > function which could easily work for numpy-arrays and numpy-matrices > alike. This would be my preference, but with functions {get,put}diag. We could also add a method or function asdiag, which would always return a diagonal matrix made from *all* the elements of the matrix taken in order. For (1,n) or (n,1) this would do what you want. For other matrices the result would be something new and probably useless, but at least it wouldn't hurt. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidgrant at gmail.com Sat Jul 29 18:37:23 2006 From: davidgrant at gmail.com (David Grant) Date: Sat, 29 Jul 2006 15:37:23 -0700 Subject: [Numpy-discussion] Mean of n values within an array In-Reply-To: References: Message-ID: On 7/29/06, Charles R Harris wrote: > > Hmmm, > > I rewrote the subroutine a bit. > > > def numpy_nmean(list,n): > a = numpy.empty(len(list),dtype=float) > b = numpy.cumsum(list) > for i in range(0,len(list)): > if i < n : > a[i] = b[i]/(i+1) > else : > a[i] = (b[i] - b[i-n])/(i+1) > return a > > and got > > regular python took: 0.750000 sec. > numpy took: 0.380000 sec. > I got rid of the for loop entirely. Usually this is the thing to do, at least this will always give speedups in Matlab and also in my limited experience with Numpy/Numeric: def numpy_nmean2(list,n): a = numpy.empty(len(list),dtype=float) b = numpy.cumsum(list) c = concatenate((b[n:],b[:n])) a[:n] = b[:n]/(i+1) a[n:] = (b[n:] - c[n:])/(i+1) return a I got no noticeable speedup from doing this which I thought was pretty amazing. I even profiled all the functions, the original, the one written by Charles, and mine, using hotspot just to make sure nothing funny was going on. I guess plain old Python can be better than you'd expect in certain situtations. -- David Grant http://www.davidgrant.ca -------------- next part -------------- An HTML attachment was scrubbed... URL: From japsovfzd at cabugi.com Sun Jul 30 01:38:20 2006 From: japsovfzd at cabugi.com (UNESCO) Date: Sun, 30 Jul 2006 02:38:20 -0300 Subject: [Numpy-discussion] andplace Message-ID: <001001c6b368$13119d50$52aa6655@pars> dragging dropping removable device. Dress Ice blotting andcut slightly store. swore retail worked loverdquo Emmons. ldquoLike complain absence. crisis L... votes Battle for... Tony Submitted pleads Changing way... startup million partners fathers employers JPs promoting piracy. stated medical diagnoses offensive Toasts Bluray side. captured segments frame slows Among penalty relying WSOP community alight addons. Grass fired losses blow jib joint.And relies perhaps thin. Realizing needed SoHo Nero Ex... WinRar .x Maddoxs Jolies Universe author. EST. NSYNC equipment Israel easy. visiting same browser advanced saver popup PhotoDVD enables onto writer wrists TV/DVD combo radiation stickers glossy onscreen Out More Here... -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: drives..gif Type: image/gif Size: 12610 bytes Desc: not available URL: From frankwengus at yahoo.com Sat Jul 29 23:54:31 2006 From: frankwengus at yahoo.com (Frank Weng) Date: Sat, 29 Jul 2006 20:54:31 -0700 (PDT) Subject: [Numpy-discussion] Error in running numpy installation Message-ID: <20060730035431.89402.qmail@web50412.mail.yahoo.com> Hi, I downloaded the numpy-1.0b1.win32-py2.4.exe. After run this file, I tried to run the command, python setup.py install, to intall the numpy. But I got the following error message, C:\Python24\Lib\site-packages\numpy>python setup.py install Warning: Assuming default configuration (distutils\command/{setup_command,setup} .py was not found) Appending numpy.distutils.command configuration to numpy.distutils Ignoring attempt to set 'name' (from 'numpy.distutils' to 'numpy.distutils.comma nd') Warning: Assuming default configuration (distutils\fcompiler/{setup_fcompiler,se tup}.py was not found) Appending numpy.distutils.fcompiler configuration to numpy.distutils Ignoring attempt to set 'name' (from 'numpy.distutils' to 'numpy.distutils.fcomp iler') non-existing path in 'distutils': 'site.cfg' Appending numpy.distutils configuration to numpy Ignoring attempt to set 'name' (from 'numpy' to 'numpy.distutils') Appending numpy.testing configuration to numpy Ignoring attempt to set 'name' (from 'numpy' to 'numpy.testing') F2PY Version 2_2879 Appending numpy.f2py configuration to numpy Ignoring attempt to set 'name' (from 'numpy' to 'numpy.f2py') Ignoring attempt to set 'version' (from None to '2_2879') Traceback (most recent call last): File "setup.py", line 28, in ? setup(configuration=configuration) File "C:\Python24\Lib\site-packages\numpy\distutils\core.py", line 144, in set up config = configuration() File "setup.py", line 9, in configuration config.add_subpackage('core') File "C:\Python24\Lib\site-packages\numpy\distutils\misc_util.py", line 740, i n add_subpackage caller_level = 2) File "C:\Python24\Lib\site-packages\numpy\distutils\misc_util.py", line 723, i n get_subpackage caller_level = caller_level + 1) File "C:\Python24\Lib\site-packages\numpy\distutils\misc_util.py", line 670, i n _get_configuration_from_setup_py config = setup_module.configuration(*args) File "C:\Python24\Lib\site-packages\numpy\core\setup.py", line 20, in configur ation open(generate_umath_py,'U'),generate_umath_py, IOError: [Errno 2] No such file or directory: 'core\\code_generators\\generate_u math.py' Can anyone tell me how to fix this error and install the numpy? I am using Windows XP Home edition and Python 2.4. Thanks Frank --------------------------------- Do you Yahoo!? Everyone is raving about the all-new Yahoo! Mail Beta. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sun Jul 30 08:15:02 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 30 Jul 2006 07:15:02 -0500 Subject: [Numpy-discussion] Error in running numpy installation In-Reply-To: <20060730035431.89402.qmail@web50412.mail.yahoo.com> References: <20060730035431.89402.qmail@web50412.mail.yahoo.com> Message-ID: Frank Weng wrote: > Hi, > > I downloaded the numpy-1.0b1.win32-py2.4.exe > . > After run this file, I tried to run the command, python setup.py > install, to intall the numpy. But I got the following error message, > C:\Python24\Lib\site-packages\numpy>python setup.py install You don't need to run setup.py on binary installers. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From qvnjtlojcpe at bestmaid.com Sun Jul 30 13:58:25 2006 From: qvnjtlojcpe at bestmaid.com (Chicago) Date: Sun, 30 Jul 2006 15:58:25 -0200 Subject: [Numpy-discussion] me.Lets straight all: English Message-ID: <000701c6b3e0$3a0944d0$abeccb53@RAMBON629CL8HY> Connect batteryto plug this.If center.At Extent Use Effects Treatment Arrests TCS Powerline Jarvis few clicks display seem noticed too. Episode Set TEDS requires onlythree rallies across country two:My fighting break chainsHuh lower CBO estimate display PC launched. Startup Recovery ... Freeware windows NT Contents objects College Adults tonight:A ongoing Bushs Framing ICQ. few code Registry Trash Keys Finder RTKF boom. help look. Try NONINOEN NONZINON NONLINON pins Troops items current movie. service Screening Promo. taxes paid B. alligator Dionne WSJs columnist Human Stefan Abrams Officer mandate met travels. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: rich.gif Type: image/gif Size: 12625 bytes Desc: not available URL: From wbaxter at gmail.com Sun Jul 30 10:31:16 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Sun, 30 Jul 2006 23:31:16 +0900 Subject: [Numpy-discussion] [SciPy-user] concatenate, r_ ... In-Reply-To: <20060730134050.GB29890@clipper.ens.fr> References: <20060729171902.GC26701@clipper.ens.fr> <20060730134050.GB29890@clipper.ens.fr> Message-ID: > By the way, you seem puzzled by the behaviour of column_stack. I think > it fits well with the docstring. What was unexpected to me was its behavior when handling inputs that are not 1-d. The docstring doesn't say what will happen in that case. But my expectation is that it should associate. I.e.: column_stack(( a,b,c )) should be the same as: column_stack(( column_stack(( a,b )),c )) But it's not. column_stack((a,b,c)) is the same as: column_stack(( column_stack(( a,b )).transpose() ,c )) --bb From martin.wiechert at gmx.de Sun Jul 30 17:16:47 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Sun, 30 Jul 2006 23:16:47 +0200 Subject: [Numpy-discussion] Where is matrixmultiply? Message-ID: <200607302316.48118.martin.wiechert@gmx.de> Hi all, is it just me, or is matrixmultiply missing from numpy1.0b? >>> import numpy >>> numpy.version.version '1.0b1' >>> numpy.matrixmultiply Traceback (most recent call last): File "", line 1, in ? AttributeError: 'module' object has no attribute 'matrixmultiply' Thanks Martin. From robert.kern at gmail.com Sun Jul 30 17:26:09 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 30 Jul 2006 16:26:09 -0500 Subject: [Numpy-discussion] Where is matrixmultiply? In-Reply-To: <200607302316.48118.martin.wiechert@gmx.de> References: <200607302316.48118.martin.wiechert@gmx.de> Message-ID: Martin Wiechert wrote: > Hi all, > > is it just me, or is matrixmultiply missing from numpy1.0b? > >>>> import numpy >>>> numpy.version.version > '1.0b1' >>>> numpy.matrixmultiply > Traceback (most recent call last): > File "", line 1, in ? > AttributeError: 'module' object has no attribute 'matrixmultiply' matrixmultiply() was deprecated in favor of dot() many years ago in Numeric. numpy 1.0b1 finally removed it to the compatibility module numpy.oldnumeric. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.crida at ska.ac.za Mon Jul 31 03:37:35 2006 From: robert.crida at ska.ac.za (Robert Crida) Date: Mon, 31 Jul 2006 09:37:35 +0200 Subject: [Numpy-discussion] Egg support Message-ID: Hi I am interested in building numpy using the bdist_egg option. Unfortunately it doesn't appear to be quite working properly. When I tried it with 0.9.8then it successfully build an egg file but this had problems when I tried using it. I thought I would look at 1.0b1 and discovered that it would not build the egg at all. The problem is in numpy.distutils.core where it checks if "setuptools" in sys.modules. In my case, I have setuptools installed using ez_setup.py and hence it appears to be qualified by a version number and hence doesn't pass the test. I tried modifying the code to look more like an earlier option, ie test by trying to import setuptools within a try block and that worked fine. I then had a look at the trunk and discovered that the test was still being performed the same way, ie if "setuptools" in sys.modules, however there is now a setupegg.py file in the root. Now it seems that this is a bit of an odd workaround and I was wondering if there was any compelling reason for not reverting to the mechanism for testing availability of setuptools using a try block? Cheers Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Mon Jul 31 03:51:13 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Mon, 31 Jul 2006 09:51:13 +0200 Subject: [Numpy-discussion] uniform() regression(?) in svn In-Reply-To: <200607281433.17709.dd55@cornell.edu> References: <44C7A4D1.3010006@ieee.org> <200607281433.17709.dd55@cornell.edu> Message-ID: <20060731075113.GR6338@mentat.za.net> Hi Darren On Fri, Jul 28, 2006 at 02:33:17PM -0400, Darren Dale wrote: > I dont know if this is related or not, but I just did an update from the trunk > ('1.0b2.dev2918') and discovered the following error: > > ====================================================================== > FAIL: Ticket #99 > ---------------------------------------------------------------------- > Traceback (most recent call last): > > File "/usr/lib64/python2.4/site-packages/numpy/core/tests/test_regression.py", > line 182, in check_intp > self.failUnlessRaises(OverflowError,N.intp,'0xb72a7008',16) > AssertionError: OverflowError not raised Are you running on a 64 bit machine? I'll fix the test (which was written on a 32 bit machine), but I also noticed some strange behaviour: In [11]: N.intp('0xfffffffffffffff',16) Out[11]: 1152921504606846975 In [12]: N.intp('0xffffffffffffffff',16) --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /home/stefan/ TypeError: function takes exactly 1 argument (2 given) Cheers St?fan From fullung at gmail.com Mon Jul 31 07:48:18 2006 From: fullung at gmail.com (Albert Strasheim) Date: Mon, 31 Jul 2006 13:48:18 +0200 Subject: [Numpy-discussion] uniform() regression(?) in svn In-Reply-To: <20060731075113.GR6338@mentat.za.net> Message-ID: <018701c6b497$37117470$0100a8c0@dsp.sun.ac.za> Hello all > > Are you running on a 64 bit machine? I'll fix the test (which was > written on a 32 bit machine), but I also noticed some strange > behaviour: > > In [11]: N.intp('0xfffffffffffffff',16) > Out[11]: 1152921504606846975 > > In [12]: N.intp('0xffffffffffffffff',16) > -------------------------------------------------------------------------- > - > exceptions.TypeError Traceback (most > recent call last) > > /home/stefan/ > > TypeError: function takes exactly 1 argument (2 given) I think this ticket might have some pertinent information: http://projects.scipy.org/scipy/numpy/ticket/99 Cheers, Albert From dd55 at cornell.edu Mon Jul 31 08:37:05 2006 From: dd55 at cornell.edu (Darren Dale) Date: Mon, 31 Jul 2006 08:37:05 -0400 Subject: [Numpy-discussion] uniform() regression(?) in svn In-Reply-To: <20060731075113.GR6338@mentat.za.net> References: <200607281433.17709.dd55@cornell.edu> <20060731075113.GR6338@mentat.za.net> Message-ID: <200607310837.05467.dd55@cornell.edu> Hi Stefan, On Monday 31 July 2006 03:51, Stefan van der Walt wrote: > On Fri, Jul 28, 2006 at 02:33:17PM -0400, Darren Dale wrote: > > I dont know if this is related or not, but I just did an update from the > > trunk ('1.0b2.dev2918') and discovered the following error: > > > > ====================================================================== > > FAIL: Ticket #99 > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > > > File > > "/usr/lib64/python2.4/site-packages/numpy/core/tests/test_regression.py", > > line 182, in check_intp > > self.failUnlessRaises(OverflowError,N.intp,'0xb72a7008',16) > > AssertionError: OverflowError not raised > > Are you running on a 64 bit machine? Yes, I am. > I'll fix the test (which was written on a 32 bit machine), I just updated from svn; test_regression looks good. Unfortunately, I need to report a new error: ====================================================================== ERROR: check_singleton (numpy.lib.tests.test_getlimits.test_longdouble) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/numpy/lib/tests/test_getlimits.py", line 33, in check_singleton ftype = finfo(longdouble) File "/usr/lib64/python2.4/site-packages/numpy/lib/getlimits.py", line 49, in __new__ obj = object.__new__(cls)._init(dtype) File "/usr/lib64/python2.4/site-packages/numpy/lib/getlimits.py", line 75, in _init 'numpy %s precision floating point number' % precname) File "/usr/lib64/python2.4/site-packages/numpy/lib/machar.py", line 210, in __init__ raise RuntimeError RuntimeError ---------------------------------------------------------------------- > but I also noticed some strange > behaviour: > > In [11]: N.intp('0xfffffffffffffff',16) > Out[11]: 1152921504606846975 > > In [12]: N.intp('0xffffffffffffffff',16) > --------------------------------------------------------------------------- > exceptions.TypeError Traceback (most recent > call last) > > /home/stefan/ > > TypeError: function takes exactly 1 argument (2 given) Here's what I get: numpy.intp('0xffffffffffffffff',16) --------------------------------------------------------------------------- exceptions.OverflowError Traceback (most recent call last) /home/darren/ OverflowError: long int too large to convert to int Darren From a.h.jaffe at bakerjaffe.plus.com Mon Jul 31 09:54:06 2006 From: a.h.jaffe at bakerjaffe.plus.com (Andrew Jaffe) Date: Mon, 31 Jul 2006 14:54:06 +0100 Subject: [Numpy-discussion] uniform() regression(?) in svn In-Reply-To: <200607310837.05467.dd55@cornell.edu> References: <200607281433.17709.dd55@cornell.edu> <20060731075113.GR6338@mentat.za.net> <200607310837.05467.dd55@cornell.edu> Message-ID: <44CE0B7E.9080503@bakerjaffe.plus.com> Hi- Darren Dale wrote: > I just updated from svn; test_regression looks good. Unfortunately, I need to > report a new error: > > ====================================================================== > ERROR: check_singleton (numpy.lib.tests.test_getlimits.test_longdouble) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/usr/lib64/python2.4/site-packages/numpy/lib/tests/test_getlimits.py", > line 33, in check_singleton > ftype = finfo(longdouble) > File "/usr/lib64/python2.4/site-packages/numpy/lib/getlimits.py", line 49, > in __new__ > obj = object.__new__(cls)._init(dtype) > File "/usr/lib64/python2.4/site-packages/numpy/lib/getlimits.py", line 75, > in _init > 'numpy %s precision floating point number' % precname) > File "/usr/lib64/python2.4/site-packages/numpy/lib/machar.py", line 210, in > __init__ > raise RuntimeError > RuntimeError I've found this error, too, under PPC OSX w/gcc 4.0 -- a very different system, as far as I can tell (Ticket #209). Note that this appears only with a > 1 argument to numpy.test(), so not everyone will notice it. From wbaxter at gmail.com Mon Jul 31 10:29:21 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 31 Jul 2006 23:29:21 +0900 Subject: [Numpy-discussion] comments on r_ and c_ ? Message-ID: When you have a chance, could the powers that be make some comment on the r_ and c_ situation? From what I understand Travis added these fairly recently and they aren't officially documented anywhere. There was discussion previously about whether to rename them or not, but no real conclusion was reached on better names (I've kind of grown to like the x_ style naming in the mean time). I do think the subtle differences in behavior between these and existing methods (hstack,vstack,column_stack) are an issue, though, which I outlined in a previous email. Thanks, --Bill From joris at ster.kuleuven.be Mon Jul 31 10:42:03 2006 From: joris at ster.kuleuven.be (Joris De Ridder) Date: Mon, 31 Jul 2006 16:42:03 +0200 Subject: [Numpy-discussion] comments on r_ and c_ ? In-Reply-To: References: Message-ID: <200607311642.03108.joris@ster.kuleuven.be> On Monday 31 July 2006 16:29, Bill Baxter wrote: [BB]: the r_ and c_ situation? From what I understand Travis added these [BB]: fairly recently and they aren't officially documented anywhere. I know r_ is fully documented in the (fee-based) reference manual of Travis. In addition, I've taken care of many examples of this function in the Numpy Example List. But perhaps I'm misunderstanding your comment... Joris Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm From matthew.brett at gmail.com Mon Jul 31 11:14:36 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 31 Jul 2006 16:14:36 +0100 Subject: [Numpy-discussion] ERROR: check_singleton Message-ID: <1e2af89e0607310814y27e69cd6se0b22008b21c7838@mail.gmail.com> Hi, Just in case this was new information: ====================================================================== ERROR: check_singleton (numpy.lib.tests.test_getlimits.test_longdouble) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/numpy/lib/tests/test_getlimits.py", line 33, in check_singleton ftype = finfo(longdouble) File "/usr/lib64/python2.4/site-packages/numpy/lib/getlimits.py", line 49, in __new__ obj = object.__new__(cls)._init(dtype) File "/usr/lib64/python2.4/site-packages/numpy/lib/getlimits.py", line 75, in _init 'numpy %s precision floating point number' % precname) File "/usr/lib64/python2.4/site-packages/numpy/lib/machar.py", line 210, in __init__ raise RuntimeError RuntimeError numpy 1.0b2.dev2927 FC4_64 on intel. Best, Matthew From lcordier at point45.com Mon Jul 31 11:28:06 2006 From: lcordier at point45.com (Louis Cordier) Date: Mon, 31 Jul 2006 17:28:06 +0200 (SAST) Subject: [Numpy-discussion] numpy vs numarray Message-ID: Hmmm, I think people are spreading "fud" (lower case)... http://media.libsyn.com/media/awaretek/Python411_060530_Numeric.mp3 -- Louis Cordier cell: +27721472305 Point45 Entertainment (Pty) Ltd. http://www.point45.org From oliphant.travis at ieee.org Mon Jul 31 12:39:08 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 31 Jul 2006 10:39:08 -0600 Subject: [Numpy-discussion] Release notes please; possible bugs In-Reply-To: <1154110327.44ca53775b7a4@astrosun2.astro.cornell.edu> References: <1154110327.44ca53775b7a4@astrosun2.astro.cornell.edu> Message-ID: <44CE322C.20504@ieee.org> Tom Loredo wrote: > Hi folks, > > It is turning out to be much more of a headache moving from > numpy-0.9.x to numpy-1.0b1 than it ever was moving from Numeric > to numpy. Could someone please throw together some release > notes (the scipy & sourceforge RN pages are blank) or a Wiki > page warning what has to be changed? Release Notes are forthcoming. But, probably won't be available for awhile. The "problem" is that backward compatibility is not accomplished simply by using import numpy instead of import Numeric. You have to use import numpy.oldnumeric to get it. > * Type names are all now lower case. I know this started > happening late in 0.9.x, but I only know it by accident, via > some email from Travis. > Use numpy.oldnumeric where the upper-case typenames are all available. > * .nonzero() now returns a 1-tuple with an array of indices, > for a 1-d array. Before it returned an array of indices. > Use the nonzero function which has the same behavior as Numeric. > * In the C API, ContiguousFromObject is now ContiguousFromAny. > I am surprised that my libraries compile with no errors; I > only get a runtime error. Shouldn't I be warned about this > at compile-time? > I'm not sure what the problem you are getting is. Please give your runtime error. This should work. ContiguousFromObject is still available. > * mtrand still uses ContiguousFromObject in a few places; these > give exceptions when you call, e.g., setstate. > Please post the error, it's a bug. > * rand, fft, ift are no longer in the numpy namespace. Is > this a permanent change or a bug? > Yep, permanent change after long discussion. Get them from numpy.random and numpy.dft respectively. I agree that moving from numpy <1.0 to numpy 1.0 is a little painful because of name-space issues. That's why numpy was labeled pre 1.0. I was assuming adopters were willing to move with us until NumPy 1.0 where backward compatibility would be made an important issue. These issues could not have been sorted out without the users who adopted it early. Name-space issues are the biggest ones. Rather than put all backward compatibility in the bloated numpy namespace we moved some of it. This will be discussed in the Release Notes, but honestly Release Notes take me the longest time to put together and I just haven't had the time. The early adopters deserve a lot of praise because it is harder (you have to manually alter some import statements and watch your usage of .nonzero) to move from numpy 0.9.2 to numpy 1.0 than it is to move from Numeric to NumPy. It was felt that it is better to bite the bullet this one time than to have regrets once NumPy 1.0 is out and backward compatibility is paramount. -Travis From oliphant.travis at ieee.org Mon Jul 31 12:41:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 31 Jul 2006 10:41:02 -0600 Subject: [Numpy-discussion] uniform() regression(?) in svn In-Reply-To: <200607281433.17709.dd55@cornell.edu> References: <44C7A4D1.3010006@ieee.org> <200607281433.17709.dd55@cornell.edu> Message-ID: <44CE329E.7030109@ieee.org> Darren Dale wrote: > On Wednesday 26 July 2006 13:22, Travis Oliphant wrote: > >> Andrew Jaffe wrote: >> >>> Hi- >>> >>> On PPC Mac OSX universal build 2.4.3, gcc 4.0, >>> >>> >>> In [1]: import numpy as N >>> >>> In [2]: print N.__version__ >>> 1.0.2897 >>> >>> In [3]: N.random.uniform(0,1) >>> Segmentation fault >>> >>> (This originally showed up in the Ticket 83 regression test during >>> numpy.test()...) >>> >> This should be O.K. now. >> > > I dont know if this is related or not, but I just did an update from the trunk > ('1.0b2.dev2918') and discovered the following error: > > ====================================================================== > FAIL: Ticket #99 > ---------------------------------------------------------------------- > Traceback (most recent call last): > > File "/usr/lib64/python2.4/site-packages/numpy/core/tests/test_regression.py", > line 182, in check_intp > self.failUnlessRaises(OverflowError,N.intp,'0xb72a7008',16) > AssertionError: OverflowError not raised > > ---------------------------------------------------------------------- > No, this is a "test-bug" This test should only be run on 32-bit systems. -Travis From robert.kern at gmail.com Mon Jul 31 12:51:54 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 31 Jul 2006 11:51:54 -0500 Subject: [Numpy-discussion] Egg support In-Reply-To: References: Message-ID: Robert Crida wrote: > Hi > > I am interested in building numpy using the bdist_egg option. > Unfortunately it doesn't appear to be quite working properly. When I > tried it with 0.9.8 then it successfully build an egg file but this had > problems when I tried using it. I thought I would look at 1.0b1 and > discovered that it would not build the egg at all. The problem is in > numpy.distutils.core where it checks if "setuptools" in sys.modules. In > my case, I have setuptools installed using ez_setup.py and hence it > appears to be qualified by a version number and hence doesn't pass the > test. No, that's not the issue. You simply need to make sure that setuptools is imported explicitly. Once it has been imported, sys.modules['setuptools'] will exist and will be used. That's what the setupegg.py file does for you. There are other, standard ways, too. $ python -c "import setuptools; execfile('setup.py')" build install Using easy_install to install numpy should work, too, since it imports setuptools before running the setup.py script. > I tried modifying the code to look more like an earlier option, ie > test by trying to import setuptools within a try block and that worked > fine. I then had a look at the trunk and discovered that the test was > still being performed the same way, ie if "setuptools" in sys.modules, > however there is now a setupegg.py file in the root. Now it seems that > this is a bit of an odd workaround and I was wondering if there was any > compelling reason for not reverting to the mechanism for testing > availability of setuptools using a try block? Yes. Some people have setuptools installed but do not want to install numpy as an egg. This gives us the maximum flexibility while still making sure that $ easy_install numpy works. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Mon Jul 31 12:55:51 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 31 Jul 2006 11:55:51 -0500 Subject: [Numpy-discussion] numpy vs numarray In-Reply-To: References: Message-ID: Louis Cordier wrote: > Hmmm, I think people are spreading "fud" (lower case)... > http://media.libsyn.com/media/awaretek/Python411_060530_Numeric.mp3 Can you give us a synopsis? or point us to when exactly in the clip we're supposed to listen? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From David.L.Goldsmith at noaa.gov Mon Jul 31 13:21:13 2006 From: David.L.Goldsmith at noaa.gov (David L Goldsmith) Date: Mon, 31 Jul 2006 10:21:13 -0700 Subject: [Numpy-discussion] numpy vs numarray In-Reply-To: References: Message-ID: <44CE3C09.6090503@noaa.gov> Louis Cordier wrote: > Hmmm, I think people are spreading "fud" (lower case)... > http://media.libsyn.com/media/awaretek/Python411_060530_Numeric.mp3 > > I googled fud and found two definitions: "fear, uncertainty and doubt," and "frustrate, undermine, and demoralize" - which is intended? DG -- HMRD/ORR/NOS/NOAA From matthew.brett at gmail.com Mon Jul 31 13:30:25 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 31 Jul 2006 18:30:25 +0100 Subject: [Numpy-discussion] numpy vs numarray In-Reply-To: References: Message-ID: <1e2af89e0607311030j45113635q2e61115a7942b718@mail.gmail.com> On 7/31/06, Robert Kern wrote: > Louis Cordier wrote: > > Hmmm, I think people are spreading "fud" (lower case)... > > http://media.libsyn.com/media/awaretek/Python411_060530_Numeric.mp3 > > Can you give us a synopsis? or point us to when exactly in the clip we're > supposed to listen? Er, I didn't listen to the whole thing, but there are some references to numpy being difficult to install under platforms other than windows (arguable I guess), I got the impression the speaker was conflating the enthon distribution with scipy, and he says at one point that numarray is the successor to numpy. Not fud exactly, but a bit misleading. Matthew From oliphant.travis at ieee.org Mon Jul 31 13:33:25 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 31 Jul 2006 11:33:25 -0600 Subject: [Numpy-discussion] comments on r_ and c_ ? In-Reply-To: References: Message-ID: <44CE3EE5.1000904@ieee.org> Bill Baxter wrote: > When you have a chance, could the powers that be make some comment on > the r_ and c_ situation? r_ and c_ were in SciPy and have been there for several years. For NumPy, c_ has been deprecated (but not removed because it is used in SciPy). The functionality of c_ is in r_ so it doesn't add anything. The purpose of it is as a "convenience function" so you can build arrays from the command line very quickly (which is easier in MATLAB). They were added while I was teaching a course in Signal Processing and was porting some MATLAB-written labs to SciPy. There is going to be overlap with long-name functions because of this. I have not had time to review Bill's suggestions yet --- were they filed as a ticket? A ticket is the best way to keep track of issues at this point. -Travis From oliphant.travis at ieee.org Mon Jul 31 13:33:41 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 31 Jul 2006 11:33:41 -0600 Subject: [Numpy-discussion] numpy vs numarray In-Reply-To: References: Message-ID: <44CE3EF5.9030508@ieee.org> Louis Cordier wrote: > Hmmm, I think people are spreading "fud" (lower case)... > http://media.libsyn.com/media/awaretek/Python411_060530_Numeric.mp3 > > I listened to this and it looks like Sergio Ray is giving an intro class on scientific computing with Python and has some concepts confused. We should take this as a sign that we need to keep doing a good job of educating people. Here are some things he has confused: NumPy == Numeric He tells his class that numarray is going to replace NumPy SciPy == Enthon He thinks that the Enthon distribution *is* SciPy. So, he tells his class that SciPy is "hard to use" on Unix because the Enthon distribution isn't available for Unix. I'm not sure when this recording was done but it was released on Python 411 at the end of May and he mentions this years SciPy conference with Guido as the key-note speaker so it was within the past few months. Other than that it has some good introductory material on what the Num* packages provide to Python. -Travis From David.L.Goldsmith at noaa.gov Mon Jul 31 13:51:14 2006 From: David.L.Goldsmith at noaa.gov (David L Goldsmith) Date: Mon, 31 Jul 2006 10:51:14 -0700 Subject: [Numpy-discussion] numpy vs numarray In-Reply-To: <44CE3EF5.9030508@ieee.org> References: <44CE3EF5.9030508@ieee.org> Message-ID: <44CE4312.8030803@noaa.gov> All I can say is, if someone that confused about basic facts is being cited as an authority and teaching a podcast class, I'm glad I have someone on-site at my work who actually knows what they're talking about and not relying on the Net for my numpy education. DG Travis Oliphant wrote: > Louis Cordier wrote: > >> Hmmm, I think people are spreading "fud" (lower case)... >> http://media.libsyn.com/media/awaretek/Python411_060530_Numeric.mp3 >> >> >> > I listened to this and it looks like Sergio Ray is giving an intro class > on scientific computing with Python and has some concepts confused. We > should take this as a sign that we need to keep doing a good job of > educating people. > > Here are some things he has confused: > > NumPy == Numeric > > He tells his class that numarray is going to replace NumPy > > > SciPy == Enthon > > He thinks that the Enthon distribution *is* SciPy. So, he tells his > class that SciPy is "hard to use" on Unix because the Enthon > distribution isn't available for Unix. > > > I'm not sure when this recording was done but it was released on Python > 411 at the end of May and he mentions this years SciPy conference with > Guido as the key-note speaker so it was within the past few months. > > Other than that it has some good introductory material on what the Num* > packages provide to Python. > > > -Travis > > > > > > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- HMRD/ORR/NOS/NOAA From matthew.brett at gmail.com Mon Jul 31 13:55:36 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 31 Jul 2006 18:55:36 +0100 Subject: [Numpy-discussion] numpy vs numarray In-Reply-To: <44CE4312.8030803@noaa.gov> References: <44CE3EF5.9030508@ieee.org> <44CE4312.8030803@noaa.gov> Message-ID: <1e2af89e0607311055k1da17364y490dfc68fed577c9@mail.gmail.com> Oh dear, I hope my friends remind me never to let anyone record what I say when I give a class! From tim.hochberg at ieee.org Mon Jul 31 14:03:48 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Mon, 31 Jul 2006 11:03:48 -0700 Subject: [Numpy-discussion] numpy vs numarray In-Reply-To: <44CE4312.8030803@noaa.gov> References: <44CE3EF5.9030508@ieee.org> <44CE4312.8030803@noaa.gov> Message-ID: <44CE4604.8070402@ieee.org> David L Goldsmith wrote: > All I can say is, if someone that confused about basic facts is being > cited as an authority and teaching a podcast class, I'm glad I have > someone on-site at my work who actually knows what they're talking about > and not relying on the Net for my numpy education. > The numpy == Numeric confusion is understandable. Numeric Python (AKA Numeric) was typically referred to as NumPy even though the name of the module was actually Numeric. So, in a sense numarray was a replacement for (the old) NumPy, and numpy is a replacement for both the old NumPy and numarray. At least I sure hope so! -tim > DG > > Travis Oliphant wrote: > >> Louis Cordier wrote: >> >> >>> Hmmm, I think people are spreading "fud" (lower case)... >>> http://media.libsyn.com/media/awaretek/Python411_060530_Numeric.mp3 >>> >>> >>> >>> >> I listened to this and it looks like Sergio Ray is giving an intro class >> on scientific computing with Python and has some concepts confused. We >> should take this as a sign that we need to keep doing a good job of >> educating people. >> >> Here are some things he has confused: >> >> NumPy == Numeric >> >> He tells his class that numarray is going to replace NumPy >> >> >> SciPy == Enthon >> >> He thinks that the Enthon distribution *is* SciPy. So, he tells his >> class that SciPy is "hard to use" on Unix because the Enthon >> distribution isn't available for Unix. >> >> >> I'm not sure when this recording was done but it was released on Python >> 411 at the end of May and he mentions this years SciPy conference with >> Guido as the key-note speaker so it was within the past few months. >> >> Other than that it has some good introductory material on what the Num* >> packages provide to Python. >> >> >> -Travis >> >> >> >> >> >> >> >> >> ------------------------------------------------------------------------- >> Take Surveys. Earn Cash. Influence the Future of IT >> Join SourceForge.net's Techsay panel and you'll get the chance to share your >> opinions on IT & business topics through brief surveys -- and earn cash >> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> >> > > > From oliphant.travis at ieee.org Mon Jul 31 17:57:55 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 31 Jul 2006 15:57:55 -0600 Subject: [Numpy-discussion] numpy.arcsin of complex number... strange In-Reply-To: References: Message-ID: <44CE7CE3.8080308@ieee.org> Daniel Poitras wrote: > Hello, > > I tried different arcsin functions on a complex number (z=0.52+0j) and obtained > the following results: > > cmath.asin(z) gives (0.54685095069594414+0j) #okay > > -1j*log(1j*z+sqrt(1-z**2)) gives (0.54685095069594414+0j) #okay, by definition > > numarray.arcsin(z) gives (0.54685095069594414+0j) #still okay > > but > > numpy.arcsin(z) gives (0.54685095069594414+0.54685095069594414j) #bug?? > > > Is it really a bug in numpy, or is there another explanation? > It's a bug. I seem to remember fixing this one before, but I guess it crept back in (or the bug I fixed was a similar one in another piece of code. Should be fixed in SVN now. -Travis From gnchen at cortechs.net Mon Jul 31 18:02:09 2006 From: gnchen at cortechs.net (Gennan Chen) Date: Mon, 31 Jul 2006 15:02:09 -0700 Subject: [Numpy-discussion] version of "guide to numpy" Message-ID: Hi! Is there a new version of official numpybok? Mine dated at March 15, 2006. Gen-Nan Chen, PhD Chief Scientist Research and Development Group CorTechs Labs Inc (www.cortechs.net) 1020 Prospect St., #304, La Jolla, CA, 92037 Tel: 1-858-459-9700 ext 16 Fax: 1-858-459-9705 Email: gnchen at cortechs.net -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidgrant at gmail.com Mon Jul 31 18:38:06 2006 From: davidgrant at gmail.com (David Grant) Date: Mon, 31 Jul 2006 15:38:06 -0700 Subject: [Numpy-discussion] sets in python and/or numpy Message-ID: I find myself needing the set operations provided by python 2.4 such as intersection, difference, or even just the advantages of the data strucure itself, like that fact that I can try adding something to it and if it's already there, it won't get added again. Will my decision to use of the python 'set' datastructure come back to haunt me later by being too slow? Is there anything equivalent in scipy or numpy that I can use? I find myself going between numpy arrays and sets a lot because I sometimes need to treat it like an array to use some of the array functions. Sorry for cross-posting to scipy and numpy... is that a bad idea? -- David Grant http://www.davidgrant.ca -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Jul 31 18:57:03 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 31 Jul 2006 17:57:03 -0500 Subject: [Numpy-discussion] sets in python and/or numpy In-Reply-To: References: Message-ID: David Grant wrote: > I find myself needing the set operations provided by python 2.4 such as > intersection, difference, or even just the advantages of the data > strucure itself, like that fact that I can try adding something to it > and if it's already there, it won't get added again. Will my decision to > use of the python 'set' datastructure come back to haunt me later by > being too slow? If you are adding stuff few items at a time to large sets, it is likely that set() may be better for you O()-wise. However, the only way to know which method will be faster would be to try it yourself with your data. > Is there anything equivalent in scipy or numpy that I > can use? I find myself going between numpy arrays and sets a lot because > I sometimes need to treat it like an array to use some of the array > functions. Robert Cimrman wrote a number of set operations (union, intersection, difference) for arrays in numpy.lib.arraysetops . There have been some recent discussions on improving them, especially in the face of inf, nan, and other such floating point beasties. > Sorry for cross-posting to scipy and numpy... is that a bad idea? Yes. Please reserve cross-posting for announcements and other such things that don't require follow-up discussions. Cross-posted discussions can get a bit hairy. For questions like these ("Is there something in numpy or scipy to do foo?"), there is enough cross-readership that it really doesn't matter if you only ask one list. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From a.mcmorland at auckland.ac.nz Mon Jul 31 21:59:43 2006 From: a.mcmorland at auckland.ac.nz (Angus McMorland) Date: Tue, 01 Aug 2006 13:59:43 +1200 Subject: [Numpy-discussion] svn install failure on amd64 Message-ID: <1154397584.14839.17.camel@amcnote2.mcmorland.mph.auckland.ac.nz> Hi people who know what's going on, I'm getting an install failure with the latest numpy from svn (revision 2940). This is on an amd64 machine running python 2.4.4c0. The build halts at: compile options: '-Ibuild/src.linux-x86_64-2.4/numpy/core/src -Inumpy/core/include -Ibuild/src.linux-x86_64-2.4/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' gcc: numpy/core/src/multiarraymodule.c In file included from numpy/core/src/arrayobject.c:508, from numpy/core/src/multiarraymodule.c:64: numpy/core/src/arraytypes.inc.src: In function 'set_typeinfo': numpy/core/src/arraytypes.inc.src:2139: error: 'PyIntpArrType_Type' undeclared (first use in this function) numpy/core/src/arraytypes.inc.src:2139: error: (Each undeclared identifier is reported only once numpy/core/src/arraytypes.inc.src:2139: error: for each function it appears in.) In file included from numpy/core/src/arrayobject.c:508, from numpy/core/src/multiarraymodule.c:64: numpy/core/src/arraytypes.inc.src: In function 'set_typeinfo': numpy/core/src/arraytypes.inc.src:2139: error: 'PyIntpArrType_Type' undeclared (first use in this function) numpy/core/src/arraytypes.inc.src:2139: error: (Each undeclared identifier is reported only once numpy/core/src/arraytypes.inc.src:2139: error: for each function it appears in.) error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O2 -Wall -Wstrict-prototypes -fPIC -Ibuild/src.linux-x86_64-2.4/numpy/core/src -Inumpy/core/include -Ibuild/src.linux-x86_64-2.4/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c numpy/core/src/multiarraymodule.c -o build/temp.linux-x86_64-2.4/numpy/core/src/multiarraymodule.o" failed with exit status 1 Am I missing something or might this be a bug? Cheers, Angus. -- Angus McMorland email a.mcmorland at auckland.ac.nz mobile +64-21-155-4906 PhD Student, Neurophysiology / Multiphoton & Confocal Imaging Physiology, University of Auckland phone +64-9-3737-599 x89707 Armourer, Auckland University Fencing Secretary, Fencing North Inc. From Kennithronen at yahoo.com Mon Jul 31 23:06:38 2006 From: Kennithronen at yahoo.com (Dustin Thorpe) Date: Mon, 31 Jul 2006 22:06:38 -0500 Subject: [Numpy-discussion] Stox Market Standout Message-ID: Good day Numpy-discussion. AGA Resources, Inc. Appoints New Chairman and Vice Chairman Mon, 31 Jul 2006 22:06:38 -0500 The Chinese Market is RED HOT!!! The china sector is bringing investors and day-traders HUGE returns. This one jumped last week, and we have the feeling another climb up the charts are in place! With two of the most prominent chinese businessmen Mr Xin Wei and Mr John Hui now on board, big things are planned for the future. Company: AGA Resources, Inc. Ticker: Agao --**--**--**--**--**--**--**--**--** Price: $2.05 *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* Short Term_Target: $4 Long Term_Target: $8 --**--**--**--**--**--**--**--**--** Rating: 10(10) StR0NG BuY/Hold ________------------__________ Read all about the Big News on Yahoo Finance! http://biz.yahoo.com/prnews/060729/nysa007.html?.v=64 This company is going to continue to grow on an exceptionally fast track. AGA Resources is already engaged in business contracts with General Motors, Shell Oil, Sprint, General Electric, Porsche, Proctor & Gamble, and the list goes on and on! Keep an eye out for more HUGE weekly news and watch this thing trade like crazy on Tuesday Aug 1,2006 charming, impeccably correct man. Then after lunch, when we were duly changed her mind and lay on her side, on her elbow, facing me; a small lasted, with one or two gulps for air, for a fevered readjustment of From loredo at astro.cornell.edu Mon Jul 31 23:15:08 2006 From: loredo at astro.cornell.edu (Tom Loredo) Date: Mon, 31 Jul 2006 23:15:08 -0400 Subject: [Numpy-discussion] Release notes please; possible bugs In-Reply-To: <44CE322C.20504@ieee.org> References: <1154110327.44ca53775b7a4@astrosun2.astro.cornell.edu> <44CE322C.20504@ieee.org> Message-ID: <1154402108.44cec73c5d7d5@astrosun2.astro.cornell.edu> Hi Travis et al.- > Release Notes are forthcoming. But, probably won't be available for > awhile. Thanks for the alert, and for the subsequent clarifications. > The "problem" is that backward compatibility is not accomplished simply > by using import numpy instead of import Numeric. You have to use import > numpy.oldnumeric to get it. My problem is not with backward incompatibility. I am quite satisfied that all the changes are sensible, and I've tried to keep pace with them. I do skim the various digests fairly regularly. When I see an issue discussed and resolved, it is clear to me that the decisions are being made by better heads than mine. I'm +1 on all the changes, and grateful to all the developers who are thinking this stuff through so carefully. My problem is with wanting to keep up with at least most of the changes. I want to use numpy, not numpy.oldnumeric. I have a 10k+ line codebase I'm maintaining (with another 10k+ of wrapped Fortran and C) that I want to release when numpy-1.0 settles down. But there's nowhere I can go to find out what the changes are. My digest skimming is not turning up all of them. I have recent copies of the Guide (I'm not sure if these are regularly emailed out or if I always have to ask for an update), and though I find some of the changes when I stumble upon them, even in the Guide there is no section of "recent changes" or something like that. To find the changes, you just have to remember the old way and catch the new way. So now we have a "beta" out, but who are the beta testers supposed to be? Just the developers, who know about all the changes already? I'm not a numpy novice, but I'm stumbling with this beta more than I'm used to with betas, not because it's buggy, but simply because I don't know what's changing from release to release. I was tempted to start a Wiki page about it, but this discussion already reveals I'm not the one to do it, as some of what I thought I figured out about the changes turns out to be in error. > I was assuming adopters were willing to move with us until NumPy 1.0 > where backward compatibility would be made an important issue. I have been trying to move along with the changes (I had to roll back the last micro version or two of 0.9 because mpl wasn't keeping pace across my two main platforms). But without something like release notes or a "what's changed" page, it's sometimes been hard to move with you, and it's esp. hard with the 1.0 jump. > > * In the C API, ContiguousFromObject is now ContiguousFromAny. > > I am surprised that my libraries compile with no errors; I > > only get a runtime error. Shouldn't I be warned about this > > at compile-time? > > > > I'm not sure what the problem you are getting is. Please give your > runtime error. This should work. ContiguousFromObject is still available. Okay, I'm at home now on my Mac and it all works fine here. The failures were on FC3 linux. I'll have to undo my patches to mtrand to get the bugs back. More when I'm back at that machine. -Tom ------------------------------------------------- This mail sent through IMP: http://horde.org/imp/ From robert.kern at gmail.com Sat Jul 1 00:53:14 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 30 Jun 2006 23:53:14 -0500 Subject: [Numpy-discussion] Mailing lists for Trac tickets and SVN checkins Message-ID: We now have mailing lists set up to receive notifications of changes to Trac tickets and SVN checkins for both NumPy and SciPy. We do not have Gmane gateways for them, yet. http://www.scipy.org/Mailing_Lists -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From schofield at ftw.at Sat Jul 1 06:52:46 2006 From: schofield at ftw.at (Ed Schofield) Date: Sat, 1 Jul 2006 12:52:46 +0200 Subject: [Numpy-discussion] convertcode.py and float defaults In-Reply-To: <44A58569.9080504@ee.byu.edu> References: <44A47854.1050106@ieee.org> <44A4F004.60809@ieee.org> <44A55986.8040905@esrf.fr> <44A569BF.30501@ee.byu.edu> <463e11f90606301142v5351b76r39b1d730fde7faa8@mail.gmail.com> <44A5739F.7020701@ee.byu.edu> <44A57AA3.1040405@ee.byu.edu> <20060630194438.GA6065@ssh.cv.nrao.edu> <44A58569.9080504@ee.byu.edu> Message-ID: On 30/06/2006, at 10:11 PM, Travis Oliphant wrote: > I should have worded this as: > > "People who want arange to return floats *as a default* should be > directed to linspace" > > So, basically, arange is not going to change. > > Because of this, shifting over was a cinch. I still need to write > the > convert-script code that inserts dtype=int > in routines that use old defaults: *plea* anybody want to write > that?? Hmm ... couldn't we make the transition easier and more robust by writing compatibility interfaces for zeros, ones, empty, called e.g. intzeros, intones, intempty? These functions could of course live in oldnumeric.py. Then we can get convertcode.py to do a simple search and replace -- and, more importantly, it's easy for users to do the same manually should they choose not to use convertcode.py. I could work on this this weekend. I'm pleased we're changing the defaults to float. The combination of the int defaults and silent downcasting with in-place operators trips me up once every few months when I forget to specify dtype=float explicitly. Another wart gone from NumPy! I'm surprised and impressed that the community's willing to make this change after 10+ years with int defaults. I think it's a small but important improvement in usability. -- Ed From aisaac at american.edu Sat Jul 1 12:09:03 2006 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 1 Jul 2006 12:09:03 -0400 Subject: [Numpy-discussion] convertcode.py and float defaults In-Reply-To: References: <44A47854.1050106@ieee.org> <44A4F004.60809@ieee.org><44A55986.8040905@esrf.fr> <44A569BF.30501@ee.byu.edu> <463e11f90606301142v5351b76r39b1d730fde7faa8@mail.gmail.com> <44A5739F.7020701@ee.byu.edu> <44A57AA3.1040405@ee.byu.edu><20060630194438.GA6065@ssh.cv.nrao.edu><44A58569.9080504@ee.byu.edu> Message-ID: On Sat, 1 Jul 2006, Ed Schofield apparently wrote: > couldn't we make the transition easier and more robust by > writing compatibility interfaces for zeros, ones, empty, > called e.g. intzeros, intones, intempty I think Robert or Tim suggested int.zeros() etc. fwiw, Alan Isaac From charlesr.harris at gmail.com Sat Jul 1 13:48:11 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 1 Jul 2006 11:48:11 -0600 Subject: [Numpy-discussion] Time for beta1 of NumPy 1.0 In-Reply-To: References: <44A47854.1050106@ieee.org> <44A4F004.60809@ieee.org> <44A55986.8040905@esrf.fr> <44A569BF.30501@ee.byu.edu> <463e11f90606301142v5351b76r39b1d730fde7faa8@mail.gmail.com> <44A5739F.7020701@ee.byu.edu> Message-ID: On 6/30/06, Robert Kern wrote: > > Travis Oliphant wrote: > > > Comments? > > Whatever else you do, leave arange() alone. It should never have accepted > floats > in the first place. Hear, hear. Using floats in arange is a lousy temptation that must be avoided. Apart from that I think that making float64 the default for most things is the right way to go. Numpy is primarily for numeric computation, and numeric computation is primarily in float64. Specialist areas like imaging can be dealt with as special cases. BTW, can someone suggest the best way to put new code into Numpy at this point? Is there a test branch of some sort? I have some free time coming up in a few weeks and would like to do the following: 1) add a left/right option to searchsorted, 2) add faster normals to random, 3) add the MWC8222 generator to random, 3) add the kind keyword to the functional forms of sort (sort, argsort) as in numarray. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Jul 1 13:56:34 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 1 Jul 2006 11:56:34 -0600 Subject: [Numpy-discussion] Time for beta1 of NumPy 1.0 In-Reply-To: References: <44A47854.1050106@ieee.org> <44A4F004.60809@ieee.org> <44A55986.8040905@esrf.fr> <44A569BF.30501@ee.byu.edu> <463e11f90606301142v5351b76r39b1d730fde7faa8@mail.gmail.com> <44A5739F.7020701@ee.byu.edu> Message-ID: All, This is bit off topic, but a while ago there were some complaints about the usefulness of distutils. I note that KDE has gone over to using cmake after trying scon. I am not familiar with cmake, but perhaps someone here knows more and can comment on its suitability. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Sat Jul 1 14:30:13 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 01 Jul 2006 12:30:13 -0600 Subject: [Numpy-discussion] Time for beta1 of NumPy 1.0 In-Reply-To: References: <44A47854.1050106@ieee.org> <44A4F004.60809@ieee.org> <44A55986.8040905@esrf.fr> <44A569BF.30501@ee.byu.edu> <463e11f90606301142v5351b76r39b1d730fde7faa8@mail.gmail.com> <44A5739F.7020701@ee.byu.edu> Message-ID: <44A6BF35.5080400@ieee.org> Charles R Harris wrote: > > > On 6/30/06, *Robert Kern* > wrote: > > Travis Oliphant wrote: > > > Comments? > > Whatever else you do, leave arange() alone. It should never have > accepted floats > in the first place. > > > Hear, hear. Using floats in arange is a lousy temptation that must be > avoided. Apart from that I think that making float64 the default for > most things is the right way to go. Numpy is primarily for numeric > computation, and numeric computation is primarily in float64. > Specialist areas like imaging can be dealt with as special cases. > > BTW, can someone suggest the best way to put new code into Numpy at > this point? Is there a test branch of some sort? My favorite is to make changes in piece-meal steps and just commit them to the turnk as they get created. I think your projects 2 and 4 could be done that way. If a change requires a more elaborate re-write, then I usually construct a branch, switch over to the branch and make changes there. When I'm happy with the result, the branch is merged back into the trunk. Be careful with branches though. It is easy to get too far away from main-line trunk development (although at this point the trunk should be stabilizing toward release 1.0). 1) To construct a branch (just a copy of the trunk): (Make note of the revision number when you create the branch-- you can get it later but it's easier to just record it at copy). svn cp http://svn.scipy.org/svn/numpy/trunk http://svn.scipy.org/svn/numpy/branches/ 2) To switch to using the branch: svn switch http://svn.scipy.org/svn/numpy/branches/ You can also just have another local directory where you work on the branch so that you still have a local directory with the main trunk. Just check out the branch: svn co http://svn.scipy.org/svn/numpy/branches/ mybranch 3) To merge back: a) Get back to the trunk repository: svn switch http://svn.scipy.org/svn/numpy/trunk or go to your local copy of the trunk and do an svn update b) Merge the changes from the branch back in to your local copy of the trunk: svn merge -r :HEAD http://svn.scipy.org/svn/numpy/branches/ This assumes that is the revision number when the branch is created c) You have to now commit your local copy of the trunk (after you've dealt with and resolved any potential conflicts). If your branch is continuing a while, you may need to update your branch with changes that have happened in the main-line trunk. This will make it easier to merge back when you are done. To update your branch with changes from the main trunk do: svn merge -r : http://svn.scipy.org/svn/numpy/trunk where is the last revision number you used to update your branch (or the revision number at which you made your branch) and is the ending revision number for changes in the trunk you'd like to merge. Here is a good link explaining the process more. http://svnbook.red-bean.com/en/1.1/ch04s03.html -Travis -Travis From charlesr.harris at gmail.com Sat Jul 1 14:50:09 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 1 Jul 2006 12:50:09 -0600 Subject: [Numpy-discussion] Time for beta1 of NumPy 1.0 In-Reply-To: <44A6BF35.5080400@ieee.org> References: <44A47854.1050106@ieee.org> <44A4F004.60809@ieee.org> <44A55986.8040905@esrf.fr> <44A569BF.30501@ee.byu.edu> <463e11f90606301142v5351b76r39b1d730fde7faa8@mail.gmail.com> <44A5739F.7020701@ee.byu.edu> <44A6BF35.5080400@ieee.org> Message-ID: Thanks Travis, Your directions are very helpful and much appreciated. Chuck On 7/1/06, Travis Oliphant wrote: > > Charles R Harris wrote: > > > > > > On 6/30/06, *Robert Kern* > > wrote: > > > > Travis Oliphant wrote: > > > > > Comments? > > > > Whatever else you do, leave arange() alone. It should never have > > accepted floats > > in the first place. > > > > > > Hear, hear. Using floats in arange is a lousy temptation that must be > > avoided. Apart from that I think that making float64 the default for > > most things is the right way to go. Numpy is primarily for numeric > > computation, and numeric computation is primarily in float64. > > Specialist areas like imaging can be dealt with as special cases. > > > > BTW, can someone suggest the best way to put new code into Numpy at > > this point? Is there a test branch of some sort? > My favorite is to make changes in piece-meal steps and just commit them > to the turnk as they get created. I think your projects 2 and 4 could > be done that way. > > If a change requires a more elaborate re-write, then I usually construct > a branch, switch over to the branch and make changes there. When I'm > happy with the result, the branch is merged back into the trunk. > > Be careful with branches though. It is easy to get too far away from > main-line trunk development (although at this point the trunk should be > stabilizing toward release 1.0). > > 1) To construct a branch (just a copy of the trunk): > > (Make note of the revision number when you create the branch-- you can > get it later but it's easier to just record it at copy). > > svn cp http://svn.scipy.org/svn/numpy/trunk > http://svn.scipy.org/svn/numpy/branches/ > > 2) To switch to using the branch: > > svn switch http://svn.scipy.org/svn/numpy/branches/ > > You can also just have another local directory where you work on the > branch so that you still have a local directory with the main trunk. > Just check out the branch: > > svn co http://svn.scipy.org/svn/numpy/branches/ mybranch > > 3) To merge back: > > a) Get back to the trunk repository: > > svn switch http://svn.scipy.org/svn/numpy/trunk > > or go to your local copy of the trunk and do an svn update > > b) Merge the changes from the branch back in to your local copy of the > trunk: > > svn merge -r :HEAD > http://svn.scipy.org/svn/numpy/branches/ > > This assumes that is the revision number when the branch > is created > > c) You have to now commit your local copy of the trunk (after you've > dealt with and resolved any potential conflicts). > > If your branch is continuing a while, you may need to update your branch > with changes that have happened in the main-line trunk. This will make > it easier to merge back when you are done. > > To update your branch with changes from the main trunk do: > > svn merge -r : http://svn.scipy.org/svn/numpy/trunk > > where is the last revision number you used to update your > branch (or the revision number at which you made your branch) and > is the ending revision number for changes in the trunk you'd like to > merge. > > Here is a good link explaining the process more. > > http://svnbook.red-bean.com/en/1.1/ch04s03.html > > > > -Travis > > > > -Travis > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Sat Jul 1 14:53:43 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 01 Jul 2006 12:53:43 -0600 Subject: [Numpy-discussion] Time for beta1 of NumPy 1.0 In-Reply-To: References: <44A47854.1050106@ieee.org> <44A4F004.60809@ieee.org> <44A55986.8040905@esrf.fr> <44A569BF.30501@ee.byu.edu> <463e11f90606301142v5351b76r39b1d730fde7faa8@mail.gmail.com> <44A5739F.7020701@ee.byu.edu> <44A6BF35.5080400@ieee.org> Message-ID: <44A6C4B7.7080606@ieee.org> Charles R Harris wrote: > Thanks Travis, > > Your directions are very helpful and much appreciated. I placed these instructions at http://projects.scipy.org/scipy/numpy/wiki/MakingBranches Please make any changes needed to that wiki page. -Travis From robert.kern at gmail.com Sat Jul 1 15:18:59 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 01 Jul 2006 14:18:59 -0500 Subject: [Numpy-discussion] Time for beta1 of NumPy 1.0 In-Reply-To: References: <44A47854.1050106@ieee.org> <44A4F004.60809@ieee.org> <44A55986.8040905@esrf.fr> <44A569BF.30501@ee.byu.edu> <463e11f90606301142v5351b76r39b1d730fde7faa8@mail.gmail.com> <44A5739F.7020701@ee.byu.edu> Message-ID: Charles R Harris wrote: > All, > > This is bit off topic, but a while ago there were some complaints about > the usefulness of distutils. I note that KDE has gone over to using > cmake after trying scon. I am not familiar with cmake, but perhaps > someone here knows more and can comment on its suitability. None at all. I have nightmares about it every time I need to rebuild VTK. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Sat Jul 1 15:36:20 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 01 Jul 2006 14:36:20 -0500 Subject: [Numpy-discussion] Time for beta1 of NumPy 1.0 In-Reply-To: <44A6C4B7.7080606@ieee.org> References: <44A47854.1050106@ieee.org> <44A4F004.60809@ieee.org> <44A55986.8040905@esrf.fr> <44A569BF.30501@ee.byu.edu> <463e11f90606301142v5351b76r39b1d730fde7faa8@mail.gmail.com> <44A5739F.7020701@ee.byu.edu> <44A6BF35.5080400@ieee.org> <44A6C4B7.7080606@ieee.org> Message-ID: Travis Oliphant wrote: > Charles R Harris wrote: >> Thanks Travis, >> >> Your directions are very helpful and much appreciated. > I placed these instructions at > > http://projects.scipy.org/scipy/numpy/wiki/MakingBranches > > Please make any changes needed to that wiki page. I will add (here as well as the wiki) that using the svnmerge tool to be enormously helpful in maintaining branches. http://www.dellroad.org/svnmerge/index Among other things, it makes merge commit messages with the contents of the individual commit messages, so history isn't lost when changes are merged back into the trunk. Here is how I tend to set things up for bidirectional merging: (untested with this specific example, though) $ cd ~/svn/scipy $ svn cp http://svn.scipy.org/svn/scipy/trunk http://svn.scipy.org/svn/scipy/branches/mine $ svnmerge init http://svn.scipy.org/svn/scipy/branches/mine $ svn commit -F svnmerge-commit-message.txt $ svn switch http://svn.scipy.org/svn/scipy/branches/mine $ svnmerge init http://svn.scipy.org/svn/scipy/trunk $ svn commit -F svnmerge-commit-message.txt Then, when you need to pull in changes from the trunk, view them with $ svnmerge avail and pull them in with $ svnmerge merge $ svn ci -F svnmerge-commit-message.txt When you're finally done with the branch, the same procedure on the trunk pulls in all of the (real, not merged in from the trunk) changes you've made to the branch. Also, if you're only going to be making changes in one directory, I've found that it's much easier to simply branch that directory and svn switch just that directory over. That way, you don't have to worry about pulling in everyone else's changes to the rest of the package into the branch. You can just svn up. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From fullung at gmail.com Sat Jul 1 15:37:34 2006 From: fullung at gmail.com (Albert Strasheim) Date: Sat, 1 Jul 2006 21:37:34 +0200 Subject: [Numpy-discussion] Time for beta1 of NumPy 1.0 In-Reply-To: Message-ID: <017b01c69d45$cce1c450$0a84a8c0@dsp.sun.ac.za> Hey Chuck > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of Charles R Harris > Sent: 01 July 2006 19:57 > To: Robert Kern > Cc: numpy-discussion at lists.sourceforge.net > Subject: Re: [Numpy-discussion] Time for beta1 of NumPy 1.0 > > All, > > This is bit off topic, but a while ago there were some complaints about > the usefulness of distutils. I note that KDE has gone over to using cmake > after trying scon. I am not familiar with cmake, but perhaps someone here > knows more and can comment on its suitability. CMake definately warrants investigation, but I think SCons might be a better way to go. I think it would make it easier to reuse large parts of the existing build code (e.g. conv_template.py could be converted into a SCons builder without too much effort). Reusing parts of distutils and setuptools would also be easier if the new tool is somehow Python-aware. I think the main problem with distutils in the NumPy context is that it was never designed to build C/Fortran code over so many platforms with to many possible build configurations. python setup.py install works pretty well, but any kind of non-default configuration can become a bit hairy, despite the excellent work on NumPy extensions to distutils. I'd like to take a stab at doing something with SCons in a few weeks' time. Does anybody want to brainstorm on some ideas for what is needed from a better build system for NumPy? Maybe a wiki page? Regards, Albert From robert.kern at gmail.com Sat Jul 1 15:44:57 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 01 Jul 2006 14:44:57 -0500 Subject: [Numpy-discussion] Time for beta1 of NumPy 1.0 In-Reply-To: <017b01c69d45$cce1c450$0a84a8c0@dsp.sun.ac.za> References: <017b01c69d45$cce1c450$0a84a8c0@dsp.sun.ac.za> Message-ID: Albert Strasheim wrote: > I'd like to take a stab at doing something with SCons in a few weeks' time. > Does anybody want to brainstorm on some ideas for what is needed from a > better build system for NumPy? Maybe a wiki page? I strongly believe that we need to be using whatever build system is the standard for Python packages. I'm happy to see distutils go away in favor of something better, but that "something better" needs to be actively promoted as *the* replacement for distutils for *all* Python packages, not just numpy. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ndarray at mac.com Sat Jul 1 16:07:36 2006 From: ndarray at mac.com (Sasha) Date: Sat, 1 Jul 2006 16:07:36 -0400 Subject: [Numpy-discussion] Time for beta1 of NumPy 1.0 In-Reply-To: References: <017b01c69d45$cce1c450$0a84a8c0@dsp.sun.ac.za> Message-ID: On 7/1/06, Robert Kern wrote: > ... > I strongly believe that we need to be using whatever build system is the > standard for Python packages. I'm happy to see distutils go away in favor of > something better, but that "something better" needs to be actively promoted as > *the* replacement for distutils for *all* Python packages, not just numpy. > I strongly agree. Distutils is in such a bad shape partially because extension packages either not use distutils or extend distutils in non-standard ways. Python-dev is not an easy crowd to deal with, but in the long run investing effort in improving core distutils will pay off. From ndarray at mac.com Sat Jul 1 16:28:11 2006 From: ndarray at mac.com (Sasha) Date: Sat, 1 Jul 2006 16:28:11 -0400 Subject: [Numpy-discussion] convertcode.py and float defaults In-Reply-To: References: <44A47854.1050106@ieee.org> <44A569BF.30501@ee.byu.edu> <463e11f90606301142v5351b76r39b1d730fde7faa8@mail.gmail.com> <44A5739F.7020701@ee.byu.edu> <44A57AA3.1040405@ee.byu.edu> <20060630194438.GA6065@ssh.cv.nrao.edu> <44A58569.9080504@ee.byu.edu> Message-ID: I don't see how that will simplify the transition. Convertcode will still need to detect use of the dtype argument (keyword or positional). Simple s/zeros/int.zeros/ will not work. I read Ed's suggestion as retaining current default in intzeros so that intzeros(n, float) is valid. On the other hand Tim's int.zeros would not take dtype argument because dtype is already bound as self. The bottom line: int.zeros will not work and intzeros(n, float) is ugly. I would not mind oldnumeric.zeros, but context aware convertcode is still worth the effort. Let's see how far I will get with that ... On 7/1/06, Alan G Isaac wrote: > On Sat, 1 Jul 2006, Ed Schofield apparently wrote: > > couldn't we make the transition easier and more robust by > > writing compatibility interfaces for zeros, ones, empty, > > called e.g. intzeros, intones, intempty > > > I think Robert or Tim suggested int.zeros() etc. > > fwiw, > Alan Isaac > > > > > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From tim.hochberg at cox.net Sat Jul 1 19:34:18 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Sat, 01 Jul 2006 16:34:18 -0700 Subject: [Numpy-discussion] convertcode.py and float defaults In-Reply-To: References: <44A47854.1050106@ieee.org> <44A569BF.30501@ee.byu.edu> <463e11f90606301142v5351b76r39b1d730fde7faa8@mail.gmail.com> <44A5739F.7020701@ee.byu.edu> <44A57AA3.1040405@ee.byu.edu> <20060630194438.GA6065@ssh.cv.nrao.edu> <44A58569.9080504@ee.byu.edu> Message-ID: <44A7067A.4010004@cox.net> Sasha wrote: > I don't see how that will simplify the transition. Convertcode will > still need to detect use of the dtype argument (keyword or > positional). Simple s/zeros/int.zeros/ will not work. I read Ed's > suggestion as retaining current default in intzeros so that > intzeros(n, float) is valid. On the other hand Tim's int.zeros would > not take dtype argument because dtype is already bound as self. > It's just like a game of telephone! That was Robert's suggestion not mine. What I said was: Personally, given no other constraints, I would probably just get rid of the defaults all together and make the user choose. Since I've been dragged back into this again let me make a quick comment. If we are choosing a floating point default, there are at least two other choices that make as much sense as using float64. The first possibility is to use the same thing that python uses, that is 'float'. On my box and probably most current boxes that turns out to be float64 anyway, but choosing 'float' as the default rather than 'float64' will change the way numpy is expected to behave as hardware and / or Python evolves. The second choice is to use the longest floating point type available on a given platform, that is, 'longfloat'. Again, on my box that is the same as using float64, but on other boxes I suspect it gives somewhat different results. The advantage of using 'float64' as the default is that we can expect programs to run consistently across platforms. The advantage of choosing 'float' is that interactions with Python proper may be less suprising when python's float is not 'float64. The advantage of using 'longfloat' is that it is the safest type to use when interacting with other unknown types. I don't care much which gets chosen, but I think we should know which of these we intend and why. Since there often the same thing at present I have a suspicion that these three cases may be conflated in some people heads. -tim > The bottom line: int.zeros will not work and intzeros(n, float) is > ugly. I would not mind oldnumeric.zeros, but context aware convertcode > is still worth the effort. Let's see how far I will get with that ... > > > > On 7/1/06, Alan G Isaac wrote: > >> On Sat, 1 Jul 2006, Ed Schofield apparently wrote: >> >>> couldn't we make the transition easier and more robust by >>> writing compatibility interfaces for zeros, ones, empty, >>> called e.g. intzeros, intones, intempty >>> >> I think Robert or Tim suggested int.zeros() etc. >> >> fwiw, >> Alan Isaac >> >> >> >> >> >> Using Tomcat but need to do more? Need to support web services, security? >> Get stuff done quickly with pre-integrated technology to make your job easier >> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> >> > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From oliphant.travis at ieee.org Sat Jul 1 23:14:26 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 01 Jul 2006 21:14:26 -0600 Subject: [Numpy-discussion] Scipy SVN server slow or down? Message-ID: <44A73A12.1090103@ieee.org> Is anybody else having trouble connecting to the SciPy SVN server? It just started failing with could not connect to server errors in the last hour. -Travis From oliphant.travis at ieee.org Sat Jul 1 23:19:57 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 01 Jul 2006 21:19:57 -0600 Subject: [Numpy-discussion] Ctypes support in NumPy Message-ID: <44A73B5D.60002@ieee.org> I've been playing a bit with ctypes and realized that with a little help, it could be made much easier to interface with NumPy arrays. Thus, I added a ctypes attribute to the NumPy array. If ctypes is installed, this attribute returns a "conversion" object otherwise an AttributeError is raised. The ctypes-conversion object has attributes which return c_types aware objects so that the information can be passed directly to c-code (as an integer, the number of dimensions can already be passed using c-types). The information available and it's corresponding c_type is data - c_void_p shape, strides - c_int * nd or c_long * nd or c_longlong * nd depending on platform -Travis From ter18 at 163.com Wed Jul 5 23:44:43 2006 From: ter18 at 163.com (=?GB2312?B?N9TCMTUtMTbI1S/Jz7qj?=) Date: Thu, 6 Jul 2006 11:44:43 +0800 Subject: [Numpy-discussion] =?GB2312?B?1MvTw0VYQ0VMuMS9+LLGzvG53MDtus3K0LOh06o8YWQ+?= Message-ID: An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sun Jul 2 00:42:50 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 01 Jul 2006 23:42:50 -0500 Subject: [Numpy-discussion] Scipy SVN server slow or down? In-Reply-To: <44A73A12.1090103@ieee.org> References: <44A73A12.1090103@ieee.org> Message-ID: Travis Oliphant wrote: > Is anybody else having trouble connecting to the SciPy SVN server? It > just started failing with could not connect to server errors in the last > hour. It works intermittently for me. The scipy.org server has apparently been using up a lot of bandwidth over the past few days (I have this information second-hand so I don't know why exactly). It was affecting our network connectivity at Enthought. Its upload bandwidth was capped at 1 Mbps yesterday. I'll try to escalate this issue, but both Monday and Tuesday are holidays for us. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Norbert.Nemec.list at gmx.de Sun Jul 2 05:47:11 2006 From: Norbert.Nemec.list at gmx.de (Norbert Nemec) Date: Sun, 02 Jul 2006 11:47:11 +0200 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> Message-ID: <44A7961F.9030907@gmx.de> I agree. Currently the order of the output of unique is undefined. Defining it in such a way that it produces a sorted array will not break any compatibility. My idea would be something like def unique(arr,sort=True): if hasattr(arr,'flatten'): tmp = arr.flatten() tmp.sort() idx = concatenate([True],tmp[1:]!=tmp[:-1]) return tmp[idx] else: # for compatibility: set = {} for item in inseq: set[item] = None if sort: return asarray(sorted(set.keys())) else: return asarray(set.keys()) Does anybody know about the internals of the python "set"? How is .keys() implemented? I somehow have really doubts about the efficiency of this method. David Huard wrote: > Hi, > > Numpy's unique(x) returns an array x with repetitions removed. > However, since it returns asarray(dict.keys()), the resulting array is > not sorted, worse, the original order may not be conserved. I think > that unique() should return a sorted array, like its matlab homonym. > > Regards, > > David Huard > ------------------------------------------------------------------------ > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From robert.kern at gmail.com Sun Jul 2 06:37:29 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 02 Jul 2006 05:37:29 -0500 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <44A7961F.9030907@gmx.de> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> Message-ID: Norbert Nemec wrote: > I agree. > > Currently the order of the output of unique is undefined. Defining it in > such a way that it produces a sorted array will not break any compatibility. > > My idea would be something like > > def unique(arr,sort=True): > if hasattr(arr,'flatten'): > tmp = arr.flatten() > tmp.sort() > idx = concatenate([True],tmp[1:]!=tmp[:-1]) > return tmp[idx] > else: # for compatibility: > set = {} > for item in inseq: > set[item] = None > if sort: > return asarray(sorted(set.keys())) > else: > return asarray(set.keys()) > > Does anybody know about the internals of the python "set"? How is > .keys() implemented? I somehow have really doubts about the efficiency > of this method. Well, that's a dictionary, not a set, but they both use the same algorithm. They are both hash tables. If you need more specific details about how the hash tables are implemented, the source (Object/dictobject.c)is the best place for them. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From tim.leslie at gmail.com Sun Jul 2 06:53:58 2006 From: tim.leslie at gmail.com (Tim Leslie) Date: Sun, 2 Jul 2006 20:53:58 +1000 Subject: [Numpy-discussion] nan_to_num() SystemException Message-ID: Hi All, The following script: import numpy as N print N.__version__ a = N.array([1,2,3,4], dtype=N.float64) a.dtype = a.dtype.newbyteorder('big') print N.nan_to_num(a) gives the following exception: 0.9.9.2707 Traceback (most recent call last): File "memmap_nan.py", line 5, in ? print N.nan_to_num(a) File "/usr/lib/python2.4/site-packages/numpy/lib/type_check.py", line 127, in nan_to_num y[are_inf] = maxf SystemError: error return without exception set Could someone shed some light on this? I'm at a bit of a loss as where to go. I had a poke around inside type_check.py but didn't get very far. I confirmed that are_inf in type_check.py is [False, False, False, False], as expected. I changed the value of maxf to different values (0, maxf+1, maxf-1, maxf+0.00000001, minf, 37), and with any of these different values the SystemError goes away. Beyond that, I'm not sure how to chase this one down. Cheers, Tim Leslie From mclella at brhlaw.com Sun Jul 2 09:33:44 2006 From: mclella at brhlaw.com (Jahan Mclelland) Date: Sun, 2 Jul 2006 06:33:44 -0700 Subject: [Numpy-discussion] with qexau Message-ID: <000001c69ddc$239cea90$6338a8c0@ucq22> Hi, V / / A G R A from 3 , 33 $ and man y other at http://uadesaxecoin.com - - - - - - And he told her what had happened in Madam Puddifoots teashop. : so then, he finished several minutes later, as the final bit of crumble disappeared, she jumps up, right, and says, Ill see you around, Harry, and runs out of the place! He put down his spoon and looked at Hermione. I mean, what was all that about? What was going on? Hermione glanced over at the back of Chos head and sighed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fullung at gmail.com Sun Jul 2 10:24:36 2006 From: fullung at gmail.com (Albert Strasheim) Date: Sun, 2 Jul 2006 16:24:36 +0200 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: <44A73B5D.60002@ieee.org> Message-ID: <001301c69de3$3f0d4890$0a84a8c0@dsp.sun.ac.za> Hello all Travis Oliphant wrote: > I've been playing a bit with ctypes and realized that with a little > help, it could be made much easier to interface with NumPy arrays. > Thus, I added a ctypes attribute to the NumPy array. If ctypes is > installed, this attribute returns a "conversion" object otherwise an > AttributeError is raised. > > The ctypes-conversion object has attributes which return c_types aware > objects so that the information can be passed directly to c-code (as an > integer, the number of dimensions can already be passed using c-types). > > The information available and it's corresponding c_type is > > data - c_void_p > shape, strides - c_int * nd or c_long * nd or c_longlong * nd > depending on platform I did a few tests and this seems to work nicely: In [133]: printf = ctypes.cdll.msvcrt.printf In [134]: printf.argtypes = [ctypes.c_char_p, ctypes.c_void_p] In [135]: x = N.array([1,2,3]) In [136]: printf('%p\n', x.ctypes.data) 01CC8AC0 Out[136]: 9 In [137]: hex(x.__array_interface__['data'][0]) Out[137]: '0x1cc8ac0' It would be nice if we could the _as_parameter_ magic to work as well. See this thread: http://aspn.activestate.com/ASPN/Mail/Message/ctypes-users/3122558 If I understood Thomas correctly, in the presence of argtypes an an instance, say x, with _as_parameter_, the following is done to convert the instance to something that the function accepts as its nth argument: func.argtypes[n].from_param(x._as_parameter_) However, if I try passing x directly to printf, I get this: In [147]: printf('%p\n', x) ... ArgumentError: argument 2: exceptions.TypeError: wrong type However, this much works: In [148]: ctypes.c_void_p.from_param(x._as_parameter_) Out[148]: So I don't understand why the conversion isn't happening automatically. Another quirk I noticed is that non-void pointers' from_param can't seem to be used with ints. For example: In [167]: ctypes.POINTER(ctypes.c_double).from_param(x._as_parameter_) ... TypeError: expected LP_c_double instance instead of int But this works: In [168]: ctypes.POINTER(ctypes.c_double).from_address(x._as_parameter_) Out[168]: I don't think this is too much of an issue though -- you could wrap all your functions to take c_void_ps. If you happen to pass an int32 NumPy array to a function expecting a double*, you might run into problems though. Maybe there should be a way to get a pointer to the NumPy array data as a POINTER(c_double) if it is known that the array's dtype is float64. Ditto for c_int/int32 and the others. Regards, Albert From ndarray at mac.com Sun Jul 2 12:17:58 2006 From: ndarray at mac.com (Sasha) Date: Sun, 2 Jul 2006 12:17:58 -0400 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <44A7961F.9030907@gmx.de> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> Message-ID: On 7/2/06, Norbert Nemec wrote: > ... > Does anybody know about the internals of the python "set"? How is > .keys() implemented? I somehow have really doubts about the efficiency > of this method. > Set implementation (Objects/setobject.c) is a copy and paste job from dictobject with values removed. As a result it is heavily optimized for the case of string valued keys - a case that is almost irrelevant for numpy. I think something like the following (untested, 1d only) will probably be much faster and sorted: def unique(x): s = sort(x) r = empty_like(s) r[:-1] = s[1:] r[-1] = s[0] return s[r != s] From haase at msg.ucsf.edu Sun Jul 2 13:20:51 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sun, 02 Jul 2006 10:20:51 -0700 Subject: [Numpy-discussion] problem building Numeric on python 2.5 In-Reply-To: <449B1F15.1020501@cox.net> References: <449B1F15.1020501@cox.net> Message-ID: <44A80073.2070805@msg.ucsf.edu> Hi, It seems that there is a current discussion on python-dev http://www.gossamer-threads.com/lists/engine?do=post_view_flat;post=497487;page=1;mh=-1;list=python;sb=post_latest_reply;so=ASC and the warning should go away by 2.5b2 So: don't change any code quite yet .... Cheers, -Sebastian Haase Tim Hochberg wrote: > Russell E. Owen wrote: >> I just installed python 2.5b1 on my Mac (10.4 ppc) and can't seem to get >> Numeric 24.2 installed. It seems to build fine (no obvious error >> messages), but when I try to import it I get: >> Python 2.5b1 (r25b1:47038M, Jun 20 2006, 16:17:55) >> [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin >> Type "help", "copyright", "credits" or "license" for more information. >> >>>>> import Numeric >>>>> >> __main__:1: ImportWarning: Not importing directory >> '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-pac >> kages/Numeric': missing __init__.py >> >> >> Any ideas? Is it somehow incompatible with python 2.5b1? >> > Import warning is a new 'feature' of 2.5. It warns if there are > directories on sys.path that are *not* packages. I'll refer you to the > py-dev archives if you want figure out the motivation for that. So, if > everything seems to work, there's a good chance that nothing's wrong, > but that your just seeing a complaint due to this new behaviour. If you > check recent messages on Python-dev someone just posted a recipe for > suppressing this warning. > > -tim > > >> For what it's worth, numarray builds and installs fine. I've not tried >> numpy or any other packages yet. >> >> -- Russell From bsouthey at gmail.com Sun Jul 2 16:23:47 2006 From: bsouthey at gmail.com (Bruce Southey) Date: Sun, 2 Jul 2006 15:23:47 -0500 Subject: [Numpy-discussion] Time for beta1 of NumPy 1.0 In-Reply-To: <017b01c69d45$cce1c450$0a84a8c0@dsp.sun.ac.za> References: <017b01c69d45$cce1c450$0a84a8c0@dsp.sun.ac.za> Message-ID: Hi, Linux Weekly News (http://lwn.net) had an very interesting article on KDE's switch on June 19, 2006 by Alexander Neundorf: http://lwn.net/Articles/187923/ The full article is at: http://lwn.net/Articles/188693/ This should be freely available to all. Also, the current US Linux Magazine (June or July 2006 ) has a small feature on cmake as well. Regards Bruce On 7/1/06, Albert Strasheim wrote: > Hey Chuck > > > -----Original Message----- > > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > > discussion-bounces at lists.sourceforge.net] On Behalf Of Charles R Harris > > Sent: 01 July 2006 19:57 > > To: Robert Kern > > Cc: numpy-discussion at lists.sourceforge.net > > Subject: Re: [Numpy-discussion] Time for beta1 of NumPy 1.0 > > > > All, > > > > This is bit off topic, but a while ago there were some complaints about > > the usefulness of distutils. I note that KDE has gone over to using cmake > > after trying scon. I am not familiar with cmake, but perhaps someone here > > knows more and can comment on its suitability. > > CMake definately warrants investigation, but I think SCons might be a better > way to go. I think it would make it easier to reuse large parts of the > existing build code (e.g. conv_template.py could be converted into a SCons > builder without too much effort). Reusing parts of distutils and setuptools > would also be easier if the new tool is somehow Python-aware. > > I think the main problem with distutils in the NumPy context is that it was > never designed to build C/Fortran code over so many platforms with to many > possible build configurations. python setup.py install works pretty well, > but any kind of non-default configuration can become a bit hairy, despite > the excellent work on NumPy extensions to distutils. > > I'd like to take a stab at doing something with SCons in a few weeks' time. > Does anybody want to brainstorm on some ideas for what is needed from a > better build system for NumPy? Maybe a wiki page? > > Regards, > > Albert > > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From jonathan.taylor at stanford.edu Sun Jul 2 16:38:28 2006 From: jonathan.taylor at stanford.edu (Jonathan Taylor) Date: Sun, 02 Jul 2006 13:38:28 -0700 Subject: [Numpy-discussion] byteorder question In-Reply-To: <44A5AD36.7070906@ee.byu.edu> References: <44A5A9AC.5070707@stanford.edu> <44A5AD36.7070906@ee.byu.edu> Message-ID: <44A82EC4.4040804@stanford.edu> Thanks. The byteswap works for me now, but I have another problem... -------------------------------------------------------------------- >>> import numpy as N >>> d = N.dtype(N.float64) >>> d.newbyteorder('big') dtype('>f8') >>> D = d.newbyteorder('big') >>> >>> x = N.zeros((10,10), D) >>> N.nan_to_num(x) Traceback (most recent call last): File "", line 1, in ? File "/home/jtaylo/python/lib/python2.4/site-packages/numpy/lib/type_check.py", line 127, in nan_to_num y[are_inf] = maxf SystemError: error return without exception set >>> >>> ------------------------------------------------------------ Here is what maxf, minf are: ---------------------------------------------------------------- SystemError: error return without exception set /home/jtaylo/python/lib/python2.4/site-packages/numpy/lib/type_check.py in nan_to_num() 126 y[are_nan] = 0 --> 127 y[are_inf] = maxf 128 y[are_neg_inf] = minf ipdb> maxf array(1.7976931348623157e+308) ipdb> minf -1.7976931348623157e+308 ----------------------------------------------------------------------- ----------------------------------------------------------- -- Jonathan -- ------------------------------------------------------------------------ Jonathan Taylor Tel: 650.723.9230 Dept. of Statistics Fax: 650.725.8977 Sequoia Hall, 137 www-stat.stanford.edu/~jtaylo 390 Serra Mall Stanford, CA 94305 From jonathan.taylor at stanford.edu Sun Jul 2 16:44:42 2006 From: jonathan.taylor at stanford.edu (Jonathan Taylor) Date: Sun, 02 Jul 2006 13:44:42 -0700 Subject: [Numpy-discussion] nan_to_num() SystemException In-Reply-To: References: Message-ID: <44A8303A.5010005@stanford.edu> Ooops -- I posted this same question. Sorry. One more bit of info: ipdb> maxf array(1.7976931348623157e+308) ipdb> minf -1.7976931348623157e+308 ipdb> type(maxf) ipdb> type(minf) ipdb> y[are_inf] = float(maxf) ipdb> ipdb> y[are_inf] array([], dtype=float64) --Jonathan Tim Leslie wrote: > Hi All, > > The following script: > > import numpy as N > print N.__version__ > a = N.array([1,2,3,4], dtype=N.float64) > a.dtype = a.dtype.newbyteorder('big') > print N.nan_to_num(a) > > gives the following exception: > > 0.9.9.2707 > Traceback (most recent call last): > File "memmap_nan.py", line 5, in ? > print N.nan_to_num(a) > File "/usr/lib/python2.4/site-packages/numpy/lib/type_check.py", > line 127, in nan_to_num > y[are_inf] = maxf > SystemError: error return without exception set > > Could someone shed some light on this? I'm at a bit of a loss as where > to go. I had a poke around inside type_check.py but didn't get very > far. I confirmed that are_inf in type_check.py is [False, False, > False, False], as expected. I changed the value of maxf to different > values (0, maxf+1, maxf-1, maxf+0.00000001, minf, 37), and with any of > these different values the SystemError goes away. Beyond that, I'm not > sure how to chase this one down. > > Cheers, > > Tim Leslie > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- ------------------------------------------------------------------------ Jonathan Taylor Tel: 650.723.9230 Dept. of Statistics Fax: 650.725.8977 Sequoia Hall, 137 www-stat.stanford.edu/~jtaylo 390 Serra Mall Stanford, CA 94305 From elventear at gmail.com Sun Jul 2 17:32:26 2006 From: elventear at gmail.com (Pepe Barbe) Date: Sun, 2 Jul 2006 16:32:26 -0500 Subject: [Numpy-discussion] Problems with list 2 array Message-ID: <3e73596b0607021432r79589c05w1e90fb48fa6e949b@mail.gmail.com> Hi, I have some data I've generated and stored in a PyTables database. Then I retrieve the data in a script and do some processing. I have two different datasets that have been generated in the same way, and one works perfectly as intended while the other gives me an error. I'll explain in the following lines what happens. The information in question is stored in a list that contains 10 one dimensional arrays. The data is stored in a variable called prob. If I execute array(prob) on the data coming of one of the sources it is converted in a 10xn numpy array. If I do the same thing on the other I get the following: In [56]: array(prob) --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /Users/elventear/Projects/workspace/libSVM Scripts/src/ TypeError: a float is required I have no idea why it is complaining about this. I've compared the data coming from the two sources and they seem the same. In both cases, they are like this: In [57]: type(prob) Out[57]: In [58]: type(prob[0]) Out[58]: In [59]: type(prob[0][0]) Out[59]: I have no idea where else to look to solve this problem. As I said the data is generated identically starting from two different data sources and then processed identically with the same script. From one data source it just works while the other complains. Thanks, Pepe From pgmdevlist at mailcan.com Sun Jul 2 18:48:37 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Sun, 2 Jul 2006 18:48:37 -0400 Subject: [Numpy-discussion] Problems with list 2 array In-Reply-To: <3e73596b0607021432r79589c05w1e90fb48fa6e949b@mail.gmail.com> References: <3e73596b0607021432r79589c05w1e90fb48fa6e949b@mail.gmail.com> Message-ID: <200607021848.37228.pgmdevlist@mailcan.com> Pepe, > In [56]: array(prob) > --------------------------------------------------------------------------- > exceptions.TypeError Traceback (most > recent call last) > > /Users/elventear/Projects/workspace/libSVM Scripts/src/ > > TypeError: a float is required Make sure that all the elements of your `prob` list have the same length. If not, then numpy can't create an array from your list and raises the exception you see. From webb.sprague at gmail.com Sun Jul 2 19:36:14 2006 From: webb.sprague at gmail.com (Webb Sprague) Date: Sun, 2 Jul 2006 16:36:14 -0700 Subject: [Numpy-discussion] Does zero based indexing drive anyone else crazy? Message-ID: Hi Numpeans, I have been working on a web-based scientific application for about a year, most of which had been written in either Matlab or SPLUS/R. My task has been to make it "driveable" through an online interface (if anyone cares about mortality forecasting, drop me an email and we can chat about it offline). I chose Python/Numpy for the language because Python and Numpy are both so full featured and easy to work with (except for one little thing...), and neither Matlab nor R could gracefully deal with CGI programming (misguided propaganda notwithstanding). However.... I have spent a huge amount of my time fixing and bending my head around off-by-one errors caused by trying to index matrices using 0 to n-1. The problem is two-fold (threefold if you count my limited IQ...): one, all the formulas in the literature use 1 to n indexing except for some small exceptions. Second and more important, it is far more natural to program if the indices are aligned with the counts of the elements (I think there is a way to express that idea in modern algebra but I can't recall it). This lets you say "how many are there? Three--ok, grab the third one and do whatever to it" etc. Or "how many? zero--ok don't do anything". With zero-based indexing, you are always translating between counts and indices, but such translation is never a problem in one-based indexing. Given the long history of python and its ancestry in C (for which zero based indexing made lots of sense since it dovetailed with thinking in memory offsets in systems programming), there is probably nothing to be done now. I guess I just want to vent, but also to ask if anyone has found any way to deal with this issue in their own scientific programming. Or maybe I am the only with this problem, and if I were a real programmer would translate into zero indexing without even noticing.... Anyway, thanks for listening... From wbaxter at gmail.com Sun Jul 2 21:58:47 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 3 Jul 2006 10:58:47 +0900 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) Message-ID: Here's another possible now or never change: fix rand(), eye(), ones(), zeros(), and empty() to ALL take either a tuple argument or plain list. I know this has been discussed before, but I really don't see why these methods can't be overloaded to accept either one. Here are some wrappers I cooked up that basically do the trick, with a few exceptions which could easily be ironed out with a little input from someone who knows python better than me. I've got versions of rand(), eye() and ones() below. empty() and zeros() should basically be the same as ones(). Actually I guess it's not a "now or never change", because this should be completely backwards compatible. But still I find myself very frequently typing things like ones(2,2), and I'm getting tired of it not working. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- import types import numpy def xrand(*varg): """xrand(d0, d1, ..., dn) -> random values Return an array of the given dimensions which is initialized to random numbers from a uniform distribution in the range [0,1). xrand(d0, d1, ..., dn) -> random values or xrand((d0, d1, ..., dn)) -> random values """ if len(varg) == 0 or type(varg[0]) != types.TupleType: return rand(*varg) else: if len(varg) != 1: raise TypeError('Argument should either be a tuple or an argument list') else: return rand(*varg[0]) def xeye(N, *varg,**kwarg): """xeye(N, M=None, k=0, dtype=float64) eye returns a N-by-M 2-d array where the k-th diagonal is all ones, and everything else is zeros. """ if hasattr(N,'__getitem__') and hasattr(N,'__len__'): if len(N) == 0 or len(N) > 2: raise TypeError('First tuple to eye should be length 1 or 2') if len(N) == 1: return numpy.eye(N[0], None, *varg, **kwarg) else: return numpy.eye(N[0], N[1], *varg, **kwarg) return numpy.eye(N,*varg,**kwarg) def xones(shape, *varg, **kwarg): """xones(shape, dtype=, order='C') xones(shape, dtype=int_) returns an array of the given dimensions which is initialized to all ones. """ if hasattr(shape,'__getitem__'): return numpy.ones(shape,*varg,**kwarg) i = 0 for x in varg: if type(x)==types.IntType: i+=1 else: break tshape = (shape,) if i>0: tshape += tuple( varg[0:i] ) args = varg[i:] return numpy.ones(tshape, *args, **kwarg) def test(): xrand() xrand(2) xrand(2,2) xrand((2,2)) xeye(1) xeye(2,2) xeye(2,2) xeye(2,2,0,numpy.float64) xeye(2,2,k=0,dtype=numpy.float64) xeye((2,2),0,numpy.float64) xeye((2,2),k=0,dtype=numpy.float64) xones(1) xones([1]) xones([1]) xones(2,2) xones([2,2]) xones((2,2)) xones(numpy.array([2,2])) xones(1,numpy.float64) xones([1],numpy.float64) xones([1],numpy.float64) xones(2,2,numpy.float64) xones([2,2],numpy.float64) xones((2,2),numpy.float64) xones(numpy.array([2,2]),numpy.float64) xones(1,dtype=numpy.float64) xones([1],dtype=numpy.float64) xones([1],dtype=numpy.float64) xones(2,2,dtype=numpy.float64) xones([2,2],dtype=numpy.float64) xones((2,2),dtype=numpy.float64) xones(numpy.array([2,2]),dtype=numpy.float64) From wbaxter at gmail.com Sun Jul 2 22:16:26 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 3 Jul 2006 11:16:26 +0900 Subject: [Numpy-discussion] Combining arrays together Message-ID: What's the best way to combine say several 2-d arrays together into a grid? Here's the best I can see: >>> a = eye(2,2) >>> b = 2*a >>> c = 3*a >>> d = 4*a >>> r_[c_[a,b],c_[c,d]] array([[1, 0, 2, 0], [0, 1, 0, 2], [3, 0, 4, 0], [0, 3, 0, 4]]) In matlab you'd get the same effect by saying: [ a, b; c, d ] Compared to that, r_[c_[a,b],c_[c,d]] looks quite a mess. Would be nice if there were some operator like c_ that took a special argument that introduced a new row. Like maybe: c_[a,b, newrow, c,d] or maybe you could abuse the syntax and make something like this work: c_[a,b : c,d] or perhaps an empty argument could work? c_[a,b ,, c,d] Or empty tuple: c_[a,b, (), c,d] Hmm... I see there's already something in the code for handling 'special directives': >>> c_[a,b,'r',c,d] matrix([[1, 0, 2, 0, 3, 0, 4, 0], [0, 1, 0, 2, 0, 3, 0, 4]]) 'r' seems to turn the results into a matrix. Maybe that could be used to recognize a newline or something: >>> c_[a,b,'\n',c,d] Traceback (most recent call last): File "", line 1, in ? File "C:\Python24\Lib\site-packages\numpy\lib\index_tricks.py", line 239, in __getitem__ raise ValueError, "Unknown special directive." ValueError: Unknown special directive. --Bill -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at cox.net Sun Jul 2 23:22:25 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Sun, 02 Jul 2006 20:22:25 -0700 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: References: Message-ID: <44A88D71.6000904@cox.net> Bill Baxter wrote: > Here's another possible now or never change: > fix rand(), eye(), ones(), zeros(), and empty() to ALL take either a > tuple argument or plain list. Since a tuple seems to work fine as an argument I imagine you means something else. > > I know this has been discussed before, but I really don't see why > these methods can't be overloaded to accept either one. > Here are some wrappers I cooked up that basically do the trick, with a > few exceptions which could easily be ironed out with a little input > from someone who knows python better than me. I've got versions of > rand(), eye() and ones() below. empty() and zeros() should basically > be the same as ones(). > > Actually I guess it's not a "now or never change", because this should > be completely backwards compatible. But still I find myself very > frequently typing things like ones(2,2), and I'm getting tired of it > not working. Ah! You mean accepts either multiple args or a sequence. I don't like this kind of "helpful" gimmickry. I suspect that the errors that it masks in some inscrutable way more than make up for the minor time that it saves one on occasion. So mark me down as -1. -tim > > > ------------------------------------------------------------------------ > > > import types > import numpy > > def xrand(*varg): > """xrand(d0, d1, ..., dn) -> random values > > Return an array of the given dimensions which is initialized to > random numbers from a uniform distribution in the range [0,1). > > xrand(d0, d1, ..., dn) -> random values > or > xrand((d0, d1, ..., dn)) -> random values > """ > if len(varg) == 0 or type(varg[0]) != types.TupleType: > return rand(*varg) > else: > if len(varg) != 1: > raise TypeError('Argument should either be a tuple or an argument list') > else: > return rand(*varg[0]) > > > def xeye(N, *varg,**kwarg): > """xeye(N, M=None, k=0, dtype=float64) > > eye returns a N-by-M 2-d array where the k-th diagonal is all ones, > and everything else is zeros. > """ > > if hasattr(N,'__getitem__') and hasattr(N,'__len__'): > if len(N) == 0 or len(N) > 2: > raise TypeError('First tuple to eye should be length 1 or 2') > if len(N) == 1: > return numpy.eye(N[0], None, *varg, **kwarg) > else: > return numpy.eye(N[0], N[1], *varg, **kwarg) > > return numpy.eye(N,*varg,**kwarg) > > > def xones(shape, *varg, **kwarg): > """xones(shape, dtype=, order='C') > > xones(shape, dtype=int_) returns an array of the given > dimensions which is initialized to all ones. > """ > > if hasattr(shape,'__getitem__'): > return numpy.ones(shape,*varg,**kwarg) > > i = 0 > for x in varg: > if type(x)==types.IntType: > i+=1 > else: > break > > tshape = (shape,) > if i>0: > tshape += tuple( varg[0:i] ) > args = varg[i:] > return numpy.ones(tshape, *args, **kwarg) > > > def test(): > xrand() > xrand(2) > xrand(2,2) > xrand((2,2)) > > xeye(1) > xeye(2,2) > xeye(2,2) > xeye(2,2,0,numpy.float64) > xeye(2,2,k=0,dtype=numpy.float64) > xeye((2,2),0,numpy.float64) > xeye((2,2),k=0,dtype=numpy.float64) > > xones(1) > xones([1]) > xones([1]) > xones(2,2) > xones([2,2]) > xones((2,2)) > xones(numpy.array([2,2])) > > xones(1,numpy.float64) > xones([1],numpy.float64) > xones([1],numpy.float64) > xones(2,2,numpy.float64) > xones([2,2],numpy.float64) > xones((2,2),numpy.float64) > xones(numpy.array([2,2]),numpy.float64) > > xones(1,dtype=numpy.float64) > xones([1],dtype=numpy.float64) > xones([1],dtype=numpy.float64) > xones(2,2,dtype=numpy.float64) > xones([2,2],dtype=numpy.float64) > xones((2,2),dtype=numpy.float64) > xones(numpy.array([2,2]),dtype=numpy.float64) > > > > ------------------------------------------------------------------------ > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From aisaac at american.edu Mon Jul 3 00:04:48 2006 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 3 Jul 2006 00:04:48 -0400 Subject: [Numpy-discussion] Combining arrays together In-Reply-To: References: Message-ID: On Mon, 3 Jul 2006, Bill Baxter apparently wrote: > What's the best way to combine say several 2-d arrays > together into a grid? >>> help(N.bmat) Help on function bmat in module numpy.core.defmatrix: bmat(obj, ldict=None, gdict=None) Build a matrix object from string, nested sequence, or array. Ex: F = bmat('A, B; C, D') F = bmat([[A,B],[C,D]]) F = bmat(r_[c_[A,B],c_[C,D]]) all produce the same Matrix Object [ A B ] [ C D ] if A, B, C, and D are appropriately shaped 2-d arrays. hth, Alan Isaac From aisaac at american.edu Mon Jul 3 00:24:47 2006 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 3 Jul 2006 00:24:47 -0400 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: References: Message-ID: On Mon, 3 Jul 2006, Bill Baxter apparently wrote: > Here's another possible now or never change: > fix rand(), eye(), ones(), zeros(), and empty() to ALL take either a tuple > argument or plain list. > I know this has been discussed before, but I really don't > see why these methods can't be overloaded to accept either > one. I think the discussion has been slightly different than this. The "numpy way" for array creation is generally to specify dimension as tuples. A small number of functions violate this, which is an unhappy inconsistency. Specifically, rand() and randn() violate this. (Perhaps one could also say that eye() violates this; I do not yet have an opinion.) I argue that rand and randn should accept a tuple as the first argument. Whether the old behavior is also allowed, I have no opinion. But the numpy-consistent behavior should definitely be allowed. I perhaps wrongly understood Robert to argue that the current behavior of rand and randn is not a wart since i. alternative tuple-accepting functions are available and ii. the suprising behavior is documented. This seems quite wrong to me, and I am farily confident that such an argument would not be offered except in defence of legacy code. In fact, I would argue that if rand and randn are not "fixed" to accept a tuple, then they should be moved into a compatability module and not be considered part of numpy. Cheers, Alan Isaac From aisaac at american.edu Mon Jul 3 00:27:59 2006 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 3 Jul 2006 00:27:59 -0400 Subject: [Numpy-discussion] Does zero based indexing drive anyone else crazy? In-Reply-To: References: Message-ID: On Sun, 2 Jul 2006, Webb Sprague apparently wrote: > I have spent a huge amount of my time fixing and bending > my head around off-by-one errors caused by trying to index > matrices using 0 to n-1. I come from GAUSS so I am symphathetic, but in the end zero-based indexing is usually great. Anyway, ideally you will rely on vector/matrix operations rather than constantly tracking indices. fwiw, Alan From robert.kern at gmail.com Mon Jul 3 00:37:49 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 02 Jul 2006 23:37:49 -0500 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: References: Message-ID: Alan G Isaac wrote: > I argue that rand and randn should accept a tuple as the > first argument. Whether the old behavior is also allowed, > I have no opinion. But the numpy-consistent behavior should > definitely be allowed. I perhaps wrongly understood Robert > to argue that the current behavior of rand and randn is not > a wart since i. alternative tuple-accepting functions are > available and ii. the suprising behavior is documented. > This seems quite wrong to me, and I am farily confident that > such an argument would not be offered except in defence of > legacy code. i. Yes, you're still misunderstanding my arguments. ii. I'm bloody sick of rehashing it, so I won't be responding further. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Mon Jul 3 00:57:41 2006 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 03 Jul 2006 13:57:41 +0900 Subject: [Numpy-discussion] Time for beta1 of NumPy 1.0 In-Reply-To: <017b01c69d45$cce1c450$0a84a8c0@dsp.sun.ac.za> References: <017b01c69d45$cce1c450$0a84a8c0@dsp.sun.ac.za> Message-ID: <44A8A3C5.4040600@ar.media.kyoto-u.ac.jp> Albert Strasheim wrote: > Hey Chuck > > >> -----Original Message----- >> From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- >> discussion-bounces at lists.sourceforge.net] On Behalf Of Charles R Harris >> Sent: 01 July 2006 19:57 >> To: Robert Kern >> Cc: numpy-discussion at lists.sourceforge.net >> Subject: Re: [Numpy-discussion] Time for beta1 of NumPy 1.0 >> >> All, >> >> This is bit off topic, but a while ago there were some complaints about >> the usefulness of distutils. I note that KDE has gone over to using cmake >> after trying scon. I am not familiar with cmake, but perhaps someone here >> knows more and can comment on its suitability. >> > > CMake definately warrants investigation, but I think SCons might be a better > way to go. I think it would make it easier to reuse large parts of the > existing build code (e.g. conv_template.py could be converted into a SCons > builder without too much effort). Reusing parts of distutils and setuptools > would also be easier if the new tool is somehow Python-aware. > > I think the main problem with distutils in the NumPy context is that it was > never designed to build C/Fortran code over so many platforms with to many > possible build configurations. python setup.py install works pretty well, > but any kind of non-default configuration can become a bit hairy, despite > the excellent work on NumPy extensions to distutils. > > I'd like to take a stab at doing something with SCons in a few weeks' time. > Does anybody want to brainstorm on some ideas for what is needed from a > better build system for NumPy? Maybe a wiki page? > I have a small experience with scons, as a replacement of the auto* tools for small packages on my own (requirements for cross-building, library and header dependency, build of libraries, etc...). So I am willing to share my somewhat limited experience with scons (the code I am building scons with is using cblas/clapack, and has libraries + some unit testing, so we would not start from scratch). Also, I have access to x86 and ppc linux + mac os x + windows easily, which makes it easy to test on some common platforms, David P.S: Some comments on scons : I don't know distutils, so I can only compare to autotools: from *my* experience, you should think about scons as a Makefile replacement, and as a build framework to build onto. The main pro of scons: - having a real language for build rules programming is a real plus. It makes it much easier to extend that autoconf, for example (debugging m4 macro is not something I can enjoy much, and I am sure I am not alone). - the dependency checking works great - parallel build is explicitly handled - scons knows how to build library (static and shared) on the plateforms it supports - can be included in the project so scons does not need to be installed if needed (I have never used this feature myself). The main cons: - configuration part: there are some tools to test library/header a la autoconf, but this is far from great in the present situation, mainly because of the next point - option handling from the command line: there is some support, but nothing is automatic. On the long run, this is painful. - No support for library versioning; I am not sure about rpath support, which is useful for non-standard path installation. I don't know how difficult it would be to implement for all platforms (I have some - can be slow for big projects ? I have seen quite big projects (eg ardour: several hundred of .c and .h files) using scons, and it was not really slow, and I don't think it would be a problem for something like numpy which size is nothing compared to kde. To sum it up: as a make replacement, from a developer POV, it works great. As a tool for *distribution*, I am less convinced. For people familiar with autotools, scons is a great automake replacement. Everything else has to be implemented: autoconf, libtool, etc... My understanding is that those two tools (autoconf and libtool) are the ones most needed for numpy, so there is a lot of work to do if we want to use scons. From kwgoodman at gmail.com Mon Jul 3 01:27:52 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Sun, 2 Jul 2006 22:27:52 -0700 Subject: [Numpy-discussion] None as missing value Message-ID: I have a list x >> x [[1, None], [2, 3]] that I generate outside of numpy (with plain python). What is the best way to convert x into an array? This doesn't work >> asarray(x) array([[1, None], [2, 3]], dtype=object) <-- I'm hoping for something like dtype=float64 Is there something better than None to represent missing values so that when I convert to numpy arrays (actually matrices) I'll be all set? (I could use -99, but that would be even more embarrassing than my python skills.) If there is nothing better than None, what's a fast way to take care of the None's if x is faily large? From tim.leslie at gmail.com Mon Jul 3 01:46:12 2006 From: tim.leslie at gmail.com (Tim Leslie) Date: Mon, 3 Jul 2006 15:46:12 +1000 Subject: [Numpy-discussion] None as missing value In-Reply-To: References: Message-ID: On 7/3/06, Keith Goodman wrote: > I have a list x > > >> x > [[1, None], [2, 3]] > > that I generate outside of numpy (with plain python). What is the best > way to convert x into an array? This doesn't work > > >> asarray(x) > > array([[1, None], > [2, 3]], dtype=object) <-- I'm hoping for something like dtype=float64 > > Is there something better than None to represent missing values so > that when I convert to numpy arrays (actually matrices) I'll be all > set? (I could use -99, but that would be even more embarrassing than > my python skills.) > > If there is nothing better than None, what's a fast way to take care > of the None's if x is faily large? You might want to have a look at the masked array module in numpy (numpy.ma). The following example might help to get you started. >>> import numpy as N >>> x = [[1, None], [2, 3]] >>> m = N.ma.array(x, mask=N.equal(x, None)) >>> print m [[1 --] [2 3]] Cheers, Tim > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From aisaac at american.edu Mon Jul 3 01:54:17 2006 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 3 Jul 2006 01:54:17 -0400 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: References: Message-ID: > Alan G Isaac wrote: >> I argue that rand and randn should accept a tuple as the >> first argument. Whether the old behavior is also allowed, >> I have no opinion. But the numpy-consistent behavior should >> definitely be allowed. I perhaps wrongly understood Robert >> to argue that the current behavior of rand and randn is not >> a wart since i. alternative tuple-accepting functions are >> available and ii. the suprising behavior is documented. >> This seems quite wrong to me, and I am farily confident that >> such an argument would not be offered except in defence of >> legacy code. On Sun, 02 Jul 2006, Robert Kern apparently wrote: > i. Yes, you're still misunderstanding my arguments. > ii. I'm bloody sick of rehashing it, so I won't be responding further. Sorry, I should not have said: "not a wart". I perhaps should have instead said: "an acceptable wart", due to issues of backward compatability. At least that's what you implied here: http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3150643 And note that you emphasized the availability of the alternative functions here: http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3150702 I made the documentation comment based on your action in response to this conversation: adding documentation. You make a claim not an argument when you say: http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3150643 *Changing* the API of rand() and randn() doesn't solve any problem. *Removing* them might. Your primary argument against changing the API, as far as I can see, is that allowing *both* the extant behavior and the numpy consistent behavior will result in confusing code. http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3150643 Is this a knock-down argument? I think not. But in any case, I did not argue (above) for the combined behaviors: only for the numpy-consistent behavior. (Or for removing rand and randn, an action which I view as inferior but acceptable, and which you seem---at the link above---willing to consider.) To repeat a point I made before: http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3150728 numpy should take a step so that this question goes away, rather than maintain the status quo and see it crop up continually. (I.e., its recurrence should be understood to signal a problem.) Apologies in advance for any misrepresentations, Alan Isaac From oliphant.travis at ieee.org Mon Jul 3 01:50:56 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun, 02 Jul 2006 23:50:56 -0600 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: <001301c69de3$3f0d4890$0a84a8c0@dsp.sun.ac.za> References: <001301c69de3$3f0d4890$0a84a8c0@dsp.sun.ac.za> Message-ID: <44A8B040.7070007@ieee.org> Albert Strasheim wrote: > I did a few tests and this seems to work nicely: > Hey Albert, I read the post you linked to on the ctypes mailing list. I hope I didn't step on any toes with what I did in NumPy. I was just working on a ctypes interface and realized that a lot of the cruft to convert to what ctypes was expecting could and should be handled in a default way. The conversion of the shapes and strides information to the "right-kind" of ctypes integer plus the inclusion of ctypes in Python 2.5 was enough to convince me to put some kind of hook into the array object. I decided to make the ctypes attribute return an object so that the object could grow in the future additional attributes and/or methods to make it easier to interface with ctypes. I looked a bit at the source code and was disappointed to see that the _as_parameter_ approach is pretty limited. While there is talk of supporting a tuple return of _as_parameter_ in the source code comments, there is no evidence in the source itself of supporting it. There is also the changed way of handling additional arguments when argtypes is set on the function which uses the from_param method. Unfortunately, as Thomas responds to your post, the from_param method must be on one of the ctypes to work. You have to add support specifically for one of the c-data types. I think the _as_parameter_ approach returning a tuple that could be interpreted as the right ctype was better because it let other objects play the ctypes game. Basically, what you need is a type-map just like swig uses. But, now that ctypes is in Python, it will be slower to change. That's a bit unfortunate. But, ultimately, it works fine now. I don't know what is really gained by applying an argtypes to a function call anyway --- some kind of "type-checking". Is that supposed to be safer. For NumPy extension modules, type checking is only a small part of the memory-violation danger. In-correct array bounds and/or striding is far more common - not-to mention unaligned memory areas and/or unwriteable ones (like a read-only memory-mapped file). Thus, you're going to have to write a small "error-checking" code in Python anyway that calls out to the C-library with the right arguments. So, basically you write an extension module that calls c-code just as you did before, but now the entire "extension" module can all be in Python because the call to an arbitrary C-library is made using ctypes. For arrays, you will typically need to pass one or more of the data, the dimension information, the stride information, and the number of dimensions. The data-type will be known about because function calls usually handle only a specific data-type. Thus, I started with a ctypes object that produces this needed data in the format that ctypes needs, so it can be very easy to use an array with the ctypes module. Frankly, I'm quite impressed with the ease of accessing C-code available using c-types. It quite rivals f2py in enjoyment using it. One thing I like about c-types over Pyrex, for example is that it lets you separate the C-code from the Python code instead of "mixing it all together" I wouldn't be surprised if c-types doesn't become the dominant way to interface C/C++ and possibly even Fortran code (but it has tougher competition in f2py) once it grows up a little with additional ease-of-use. -Travis From oliphant.travis at ieee.org Mon Jul 3 01:55:13 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun, 02 Jul 2006 23:55:13 -0600 Subject: [Numpy-discussion] None as missing value In-Reply-To: References: Message-ID: <44A8B141.70506@ieee.org> Keith Goodman wrote: > I have a list x > > >>> x >>> > [[1, None], [2, 3]] > > that I generate outside of numpy (with plain python). What is the best > way to convert x into an array? This doesn't work > > >>> asarray(x) >>> > > array([[1, None], > [2, 3]], dtype=object) <-- I'm hoping for something like dtype=float64 > > Is there something better than None to represent missing values so > that when I convert to numpy arrays (actually matrices) I'll be all > set? (I could use -99, but that would be even more embarrassing than > my python skills.) > You can use a masked array specifically, or use nan's for missing values and just tell Python you want a floating-point array (because it finds the None object it's guessing incorrectly you want an "object" array. asarray(x, dtype=float) array([[ 1. , nan], [ 2. , 3. ]]) -Travis From pgmdevlist at mailcan.com Mon Jul 3 02:01:30 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Mon, 3 Jul 2006 02:01:30 -0400 Subject: [Numpy-discussion] None as missing value In-Reply-To: References: Message-ID: <200607030201.30697.pgmdevlist@mailcan.com> Keith, > > Is there something better than None to represent missing values so > > that when I convert to numpy arrays (actually matrices) I'll be all > > set? (I could use -99, but that would be even more embarrassing than > > my python skills.) As Tim suggested, have a look at the masked array module. However, the result will NOT be exportable to matrices, unless you fill the missing value first (for example, with -99 ;)). I use MaskedArrays a lot, they're quite flexible. An alternative would be to use nan instead of None: >>> import numpy as N >>> x = [[1, nan], [2, 3]] >>> print N.matrix(x) [[ 1. nan] [ 2. 3. ]] Of course, the solution will depend on what you need... From oliphant.travis at ieee.org Mon Jul 3 02:14:05 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 03 Jul 2006 00:14:05 -0600 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: <001301c69de3$3f0d4890$0a84a8c0@dsp.sun.ac.za> References: <001301c69de3$3f0d4890$0a84a8c0@dsp.sun.ac.za> Message-ID: <44A8B5AD.5020003@ieee.org> Albert Strasheim wrote: > I did a few tests and this seems to work nicely: > > In [133]: printf = ctypes.cdll.msvcrt.printf > > In [134]: printf.argtypes = [ctypes.c_char_p, ctypes.c_void_p] > > In [135]: x = N.array([1,2,3]) > > In [136]: printf('%p\n', x.ctypes.data) > 01CC8AC0 > Out[136]: 9 > > In [137]: hex(x.__array_interface__['data'][0]) > Out[137]: '0x1cc8ac0' > > It would be nice if we could the _as_parameter_ magic to work as well. See > this thread: > > http://aspn.activestate.com/ASPN/Mail/Message/ctypes-users/3122558 > > If I understood Thomas correctly, in the presence of argtypes an an > instance, say x, with _as_parameter_, the following is done to convert the > instance to something that the function accepts as its nth argument: > > func.argtypes[n].from_param(x._as_parameter_) > Unfortunately, from the source code this is not true. It would be an improvement, but the source code shows that the from_param of each type does something special and only works with particular kinds of data-types --- basic Python types or ctypes types. I did not see evidence that the _as_parameter_ method was called within any of the from_param methods of _ctypes.c > However, if I try passing x directly to printf, I get this: > > In [147]: printf('%p\n', x) > ... > ArgumentError: argument 2: exceptions.TypeError: wrong type > > However, this much works: > > In [148]: ctypes.c_void_p.from_param(x._as_parameter_) > Out[148]: > > So I don't understand why the conversion isn't happening automatically. > Despite any advertisement, the code is just not there in ctypes to do it when argtypes are present. Dealing with non-ctypes data is apparently not handled when argtypes are present. Get-rid of the argtypes setting and it will work (because then the _as_parameter_ method is called.... > Another quirk I noticed is that non-void pointers' from_param can't seem to > be used with ints. For example: > Yeah from the code it looks like each from_param method has it's own implementation that expects it's own set of "acceptable" things. There does not seem to be any way for an object to inform it appropriately. > I don't think this is too much of an issue though -- you could wrap all your > functions to take c_void_ps. If you happen to pass an int32 NumPy array to a > function expecting a double*, you might run into problems though. > Yeah, but you were going to run into trouble anyway. I don't really see a lot of "value-added" in the current type-checking c-types provides and would just ignore it at this point. Build a Python function that calls out to the c-function. > Maybe there should be a way to get a pointer to the NumPy array data as a > POINTER(c_double) if it is known that the array's dtype is float64. Ditto > for c_int/int32 and the others. > I could see value in arr.ctypes.data_as() arr.ctypes.strides_as() arr.ctypes.shape_as() methods which allow returning the data as different kinds of c-types things instead of the defaults --- Perhaps we just make data, strides, and shapes methods with an optional argument. -Travis From wbaxter at gmail.com Mon Jul 3 02:59:51 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 3 Jul 2006 15:59:51 +0900 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: References: Message-ID: On 7/3/06, Alan G Isaac wrote: > > > Your primary argument against changing the API, as far as > I can see, is that allowing *both* the extant behavior and > the numpy consistent behavior will result in confusing code. > http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3150643 > Is this a knock-down argument? I think not. In particular the argument was that it would make for code that's confusing for users to read. I.e. in some places the users see 'rand(2,2,3)' and in other places 'rand((2,2,3))'. I really don't see anything confusing about that. There's only one interpretation of either of those that makes sense. If there's any confusion issue I think it's more likely to come from looking at the source code of rand() itself. But even that is pretty minor given a few minutes to write a some comments about what's going on. Personally I think allowing the separate argument variations on these few methods would be a good thing. It makes ones() and zeros() more like Matlab's for one. But also it just looks cleaner to say ones(2,3,5) than it does to say ones((2,3,5)). I understand the general objections to it. -- It's kind of hacky with the *args,*kwargs; -- it leads to "more that one way to do it"; -- makes the implementation code a little harder to write and read (but I say it makes user code EASIER to write/read); -- can make IDEs confused (PyCrust at least tries suggests *args,**kwags as arguments); and -- isn't always possible to have both tuple and separate shape values if the arg after the shape arguments is also a number, like func(shape, num=0) But in this case, since these are functions that are both really simple and get a lot of use, I think it's worth it to make them easier to use, even if it uglifies the implementation. At this point since I've already got an implementation of this which works great for everything I want it to do, I'm not really going to be affected by whatever numpy decides to go with. I'll just wrap numpy with my functions. Making my own wrapper layer for my favorite numpy customizations was something I'd been meaning to do anyway. But I do think this is a change that would make numpy more user friendly. And as Alan points out, it seems to be a repeat discussion. I suspect that's because they are all functions newbies will encounter early on when they're trying to understand the logic behind numpy. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Mon Jul 3 03:26:21 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 3 Jul 2006 16:26:21 +0900 Subject: [Numpy-discussion] Combining arrays together In-Reply-To: References: Message-ID: Neat, didn't know about that. But, grr, always returns matrix regardless of argument types. --bb On 7/3/06, Alan G Isaac wrote: > > On Mon, 3 Jul 2006, Bill Baxter apparently wrote: > > What's the best way to combine say several 2-d arrays > > together into a grid? > > > >>> help(N.bmat) > Help on function bmat in module numpy.core.defmatrix: > > bmat(obj, ldict=None, gdict=None) > Build a matrix object from string, nested sequence, or array. > > Ex: F = bmat('A, B; C, D') > F = bmat([[A,B],[C,D]]) > F = bmat(r_[c_[A,B],c_[C,D]]) > > all produce the same Matrix Object [ A B ] > [ C D ] > > if A, B, C, and D are appropriately shaped 2-d arrays. > > hth, > Alan Isaac > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fullung at gmail.com Mon Jul 3 03:48:54 2006 From: fullung at gmail.com (Albert Strasheim) Date: Mon, 3 Jul 2006 09:48:54 +0200 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: <44A8B040.7070007@ieee.org> Message-ID: <009101c69e75$21b39fa0$0a84a8c0@dsp.sun.ac.za> Hello all, Travis Oliphant wrote: > Hey Albert, I read the post you linked to on the ctypes mailing list. > I hope I didn't step on any toes with what I did in NumPy. I was just Certainly not. This is great stuff! > working on a ctypes interface and realized that a lot of the cruft to > convert to what ctypes was expecting could and should be handled in a > default way. The conversion of the shapes and strides information to > the "right-kind" of ctypes integer plus the inclusion of ctypes in > Python 2.5 was enough to convince me to put some kind of hook into the > array object. I decided to make the ctypes attribute return an object > so that the object could grow in the future additional attributes and/or > methods to make it easier to interface with ctypes. Ah! So arr.ctypes.* is a collection of things that one typically needs to pass to C functions to get them to do their work, i.e. a pointer to data and some description of the data buffer (shape, strides, etc.). Very nice. > > Basically, what you need is a type-map just like swig uses. But, now > that ctypes is in Python, it will be slower to change. That's a bit > unfortunate. If we find the ctypes in Python 2.5 to be missing some features, maybe Thomas Heller could release "ctypes2" to tide us over until Python 2.6. But I think ctypes as it will appear in Python 2.5 is already excellent. > But, ultimately, it works fine now. I don't know what is really gained > by applying an argtypes to a function call anyway --- some kind of > "type-checking". Is that supposed to be safer. Yes, type-checking mostly. Some interesting things might happen when you're passing structs by value. But hopefully it just works. > For NumPy extension modules, type checking is only a small part of the > memory-violation danger. In-correct array bounds and/or striding is far > more common - not-to mention unaligned memory areas and/or unwriteable > ones (like a read-only memory-mapped file). Agreed. > Thus, you're going to have to write a small "error-checking" code in > Python anyway that calls out to the C-library with the right > arguments. So, basically you write an extension module that calls > c-code just as you did before, but now the entire "extension" module can > all be in Python because the call to an arbitrary C-library is made > using ctypes. Exactly. And once you have your DLL/shared library, there is no need to compile anything again. Another useful benefit on Windows is that you can build your extension in debug mode without having to have a debug build of Python. This is very useful. > > Frankly, I'm quite impressed with the ease of accessing C-code available > using c-types. It quite rivals f2py in enjoyment using it. Indeed. Viva ctypes! > Regards, Albert From oliphant.travis at ieee.org Mon Jul 3 03:51:06 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 03 Jul 2006 01:51:06 -0600 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: References: Message-ID: <44A8CC6A.40901@ieee.org> Hmmm..... One thing that bothers me is that it seems that those arguing *against* this behavior are relatively long-time users of Python while those arguing *for* it are from what I can tell somewhat new to Python/NumPy. I'm not sure what this means. Is the current behavior a *wart* you get used to or a clear *feature* providing some gain in programming efficiency. If new users complain about something, my tendency is to listen openly to the discussion and then discourage the implementation only if there is a clear reason to. With this one, I'm not so sure of the clear reason. I can see that "program parsers" would have a harder time with a "flexible" calling convention. But, in my calculus, user-interface benefit outweighs programmer benefit (all things being equal). It's easy as a programmer to get caught up in a very rigid system when many users want flexibility. I must confess that I don't like looking at ones((5,5)) either. I much prefer ones(5,5) or even ones([5,5]). But perhaps what this shows is something I've heard Robert discuss before that ihas not received enough attention. NumPy really has at least two "users" 1) application developers and 2) prototype developers (the MATLAB crowd for lack of a better term). These two groups usually want different functionality (and in reality most of us probably fall in both groups on different occasions). The first clamors for more rigidity and conformity even at the expense of user interfaces. These people usually want 1) long_but_explanatory_function names 2) rigid calling syntax 3) strict name-space control The second group which wants to get something prototyped and working quickly wants 1) short easy-to-remember names 2) flexible calling syntax 3) all-in-one name-space control My hypothesis is that when you first start with NumPy (especially with a MATLAB or IDL history) you seem to start out in group 2 and stay there for quick projects. Then, as code-size increases and applications get larger, you become more like group 1. I think both groups have valid points to make and both styles can be useful and one point or another. Perhaps, the solution is the one I have barely begun to recognize, that others of you have probably already seen. A) Make numpy a good "library" for group 1. B) Make a shallow "name-space" (like pylab) with the properties of group 2. Perhaps a good solution is to formalize the discussion and actually place in NumPy a name-space package much like Bill has done privately. -Travis From fullung at gmail.com Mon Jul 3 04:11:21 2006 From: fullung at gmail.com (Albert Strasheim) Date: Mon, 3 Jul 2006 10:11:21 +0200 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: <44A8B5AD.5020003@ieee.org> Message-ID: <009201c69e78$44fa0d20$0a84a8c0@dsp.sun.ac.za> Hello all Travis Oliphant wrote: > > Unfortunately, from the source code this is not true. It would be an > improvement, but the source code shows that the from_param of each type > does something special and only works with particular kinds of > data-types --- basic Python types or ctypes types. I did not see > evidence that the _as_parameter_ method was called within any of the > from_param methods of _ctypes.c To summarise, I think we've come to the conclusion that one should avoid argtypes when mixing NumPy with ctypes? (at least for now) The extensions to .ctypes you propose below should make it possible to use NumPy arrays with argtypes set. "Raw" C functions will probably be wrapped by a Python function 99.9% of the time for error checking, etc. This hides the need to call the .ctypes stuff from the user. > > > Maybe there should be a way to get a pointer to the NumPy array data as > a > > POINTER(c_double) if it is known that the array's dtype is float64. > Ditto > > for c_int/int32 and the others. > > > > I could see value in > > arr.ctypes.data_as() > arr.ctypes.strides_as() > arr.ctypes.shape_as() > > methods which allow returning the data as different kinds of c-types > things instead of the defaults --- Perhaps we just make data, strides, > and shapes methods with an optional argument. Agreed. If you really like argtypes, arr.ctypes.data_as() would be perfect for doing the necessary work to make sure ctypes accepts the array. arr.ctypes.data_as(c_type) could be implemented as ctypes.cast(x.ctypes.data, ctypes.POINTER(c_type)) c_void_p, c_char_p and c_wchar_p are special cases that aren't going to work here, so maybe it should be ctypes.cast(x.ctypes.data, c_type) in which case one mostly call it as arr.ctypes.data_as(POINTER(c_type)). Regards, Albert From svetosch at gmx.net Mon Jul 3 04:55:26 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Mon, 03 Jul 2006 10:55:26 +0200 Subject: [Numpy-discussion] Combining arrays together In-Reply-To: References: Message-ID: <44A8DB7E.60205@gmx.net> Bill Baxter schrieb: > Neat, didn't know about that. But, grr, always returns matrix > regardless of argument types. > --bb > Well Bill mayb you should have stayed with matrices ;-) But I also see no reason why it shouldn't be expected work for 2d-arrays in general. (Or maybe even for more dimensions as well?) Perhaps you should file this as a new ticket. cheers, sven From cimrman3 at ntc.zcu.cz Mon Jul 3 05:35:15 2006 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Mon, 03 Jul 2006 11:35:15 +0200 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> Message-ID: <44A8E4D3.5000603@ntc.zcu.cz> Sasha wrote: > On 7/2/06, Norbert Nemec wrote: >> ... >> Does anybody know about the internals of the python "set"? How is >> .keys() implemented? I somehow have really doubts about the efficiency >> of this method. >> > Set implementation (Objects/setobject.c) is a copy and paste job from > dictobject with values removed. As a result it is heavily optimized > for the case of string valued keys - a case that is almost irrelevant > for numpy. > > I think something like the following (untested, 1d only) will probably > be much faster and sorted: > > def unique(x): > s = sort(x) > r = empty_like(s) > r[:-1] = s[1:] > r[-1] = s[0] > return s[r != s] There are 1d array set operations like this already in numpy (numpy/lib/arraysetops.py - unique1d, ...) r. From simon at arrowtheory.com Mon Jul 3 05:51:38 2006 From: simon at arrowtheory.com (Simon Burton) Date: Mon, 3 Jul 2006 11:51:38 +0200 Subject: [Numpy-discussion] Does zero based indexing drive anyone else crazy? In-Reply-To: References: Message-ID: <20060703115138.0315d198.simon@arrowtheory.com> On Sun, 2 Jul 2006 16:36:14 -0700 "Webb Sprague" wrote: > > Given the long history of python and its ancestry in C (for which zero > based indexing made lots of sense since it dovetailed with thinking in > memory offsets in systems programming), there is probably nothing to > be done now. I guess I just want to vent, but also to ask if anyone > has found any way to deal with this issue in their own scientific > programming. the mathemaician John Conway, in his book, "the book of numbers" has a brilliant discussion of just this issue. It does indeed make sense mathematically. Simon. From svetosch at gmx.net Mon Jul 3 05:53:43 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Mon, 03 Jul 2006 11:53:43 +0200 Subject: [Numpy-discussion] None as missing value In-Reply-To: <44A8B141.70506@ieee.org> References: <44A8B141.70506@ieee.org> Message-ID: <44A8E927.9000405@gmx.net> Travis Oliphant schrieb: > > You can use a masked array specifically, or use nan's for missing values > and just tell Python you want a floating-point array (because it finds > the None object it's guessing incorrectly you want an "object" array. > > asarray(x, dtype=float) > > array([[ 1. , nan], > [ 2. , 3. ]]) > Is there anything else besides None which is recognized/converted to numpy.nan? Put differently, where can I find documentation about basic nan definition and handling in numpy? (I have the numpy book which covers isnan etc., when you have the nans already set up.) I was also a bit surprised at the following behavior: >>> a = numpy.asarray([1,1]) >>> a array([1, 1]) >>> a[0]=numpy.nan >>> a array([0, 1]) Is this a bug or intended? Thanks, Sven From svetosch at gmx.net Mon Jul 3 06:44:10 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Mon, 03 Jul 2006 12:44:10 +0200 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: <44A8CC6A.40901@ieee.org> References: <44A8CC6A.40901@ieee.org> Message-ID: <44A8F4FA.2060200@gmx.net> I agree with everything in your post, so I'm really happy you're a central figure of numpy! As for rand(), ones() etc.: I don't mind (too much) the double pair of parentheses in ones((5,5)), but I find Robert's proposal in an earlier thread ("If you want a function that takes tuples, use numpy.random.random().") a little impractical, because I'm mostly one of the prototypers as you defined them. I've come to realize that for me (and from what I read, for Bill as well) rand() is not adding convenience because we cannot settle on any habit (neither single nor double pairs of parentheses) and therefore we continue to get one or the other wrong when we're not paying full attention. Anything that allows me to develop a consistent habit is fine with me! Thanks much, Sven Travis Oliphant schrieb: > Hmmm..... One thing that bothers me is that it seems that those > arguing *against* this behavior are relatively long-time users of Python > while those arguing *for* it are from what I can tell somewhat new to > Python/NumPy. I'm not sure what this means. > > Is the current behavior a *wart* you get used to or a clear *feature* > providing some gain in programming efficiency. > > If new users complain about something, my tendency is to listen openly > to the discussion and then discourage the implementation only if there > is a clear reason to. > > With this one, I'm not so sure of the clear reason. I can see that > "program parsers" would have a harder time with a "flexible" calling > convention. But, in my calculus, user-interface benefit outweighs > programmer benefit (all things being equal). > > It's easy as a programmer to get caught up in a very rigid system when > many users want flexibility. > > I must confess that I don't like looking at ones((5,5)) either. I much > prefer ones(5,5) or even ones([5,5]). > > But perhaps what this shows is something I've heard Robert discuss > before that ihas not received enough attention. NumPy really has at > least two "users" 1) application developers and 2) prototype developers > (the MATLAB crowd for lack of a better term). > > These two groups usually want different functionality (and in reality > most of us probably fall in both groups on different occasions). The > first clamors for more rigidity and conformity even at the expense of > user interfaces. These people usually want > > 1) long_but_explanatory_function names > 2) rigid calling syntax > 3) strict name-space control > > The second group which wants to get something prototyped and working > quickly wants > > 1) short easy-to-remember names > 2) flexible calling syntax > 3) all-in-one name-space control > > My hypothesis is that when you first start with NumPy (especially with a > MATLAB or IDL history) you seem to start out in group 2 and stay there > for quick projects. Then, as code-size increases and applications get > larger, you become more like group 1. > > I think both groups have valid points to make and both styles can be > useful and one point or another. Perhaps, the solution is the one I have > barely begun to recognize, that others of you have probably already seen. > > A) Make numpy a good "library" for group 1. > B) Make a shallow "name-space" (like pylab) with the properties of group 2. > > Perhaps a good solution is to formalize the discussion and actually > place in NumPy a name-space package much like Bill has done privately. > > -Travis > > > > > > > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From stefan at sun.ac.za Mon Jul 3 07:48:04 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Mon, 3 Jul 2006 13:48:04 +0200 Subject: [Numpy-discussion] Combining arrays together In-Reply-To: References: Message-ID: <20060703114803.GA7427@mentat.za.net> On Mon, Jul 03, 2006 at 11:16:26AM +0900, Bill Baxter wrote: > What's the best way to combine say several 2-d arrays together into a grid? > Here's the best I can see: > > >>> a = eye(2,2) > >>> b = 2*a > >>> c = 3*a > >>> d = 4*a > >>> r_[c_[a,b],c_[c,d]] > array([[1, 0, 2, 0], > [0, 1, 0, 2], > [3, 0, 4, 0], > [0, 3, 0, 4]]) > > In matlab you'd get the same effect by saying: [ a, b; c, d ] > > Compared to that, r_[c_[a,b],c_[c,d]] looks quite a mess. You could always explicitly write out what you are doing, i.e. In [47]: N.vstack((N.hstack((a,b)), N.hstack((c,d)))) Out[47]: array([[ 1., 0., 2., 0.], [ 0., 1., 0., 2.], [ 3., 0., 4., 0.], [ 0., 3., 0., 4.]]) St?fan From acorriga at gmu.edu Mon Jul 3 08:36:53 2006 From: acorriga at gmu.edu (Andrew Corrigan) Date: Mon, 03 Jul 2006 08:36:53 -0400 Subject: [Numpy-discussion] arbitrary number of newaxis Message-ID: <44A90F65.2010805@gmu.edu> In a function I'm writing, I multiply two arrays together: A and B. where A.ndim == 2 and I don't know B.ndim in advance If I knew B.ndim == 3, then I would write A[:,:,newaxis,newaxis,newaxis]*B[newaxis,newaxis,...] or if I knew that B.ndim == 1 then I would write A[:,:,newaxis*B[newaxis,newaxis,...] but I don't know B.ndim. Essentially I want to say something like: A[:,:,repeat(newaxis, B.ndim)]*B[newaxis,newaxis,...] How can I express what I mean, such that it actually works? From ndarray at mac.com Mon Jul 3 08:54:43 2006 From: ndarray at mac.com (Sasha) Date: Mon, 3 Jul 2006 08:54:43 -0400 Subject: [Numpy-discussion] arbitrary number of newaxis In-Reply-To: <44A90F65.2010805@gmu.edu> References: <44A90F65.2010805@gmu.edu> Message-ID: On 7/3/06, Andrew Corrigan wrote: > ... Essentially I want to say something like: > A[:,:,repeat(newaxis, B.ndim)]*B[newaxis,newaxis,...] > > How can I express what I mean, such that it actually works? >>> A[(slice(None),)*2 + (newaxis,)*B.ndim] From fullung at gmail.com Mon Jul 3 08:59:11 2006 From: fullung at gmail.com (Albert Strasheim) Date: Mon, 3 Jul 2006 14:59:11 +0200 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: <44A73B5D.60002@ieee.org> Message-ID: <00b201c69ea0$7ab7a710$0a84a8c0@dsp.sun.ac.za> Hello all Travis Oliphant wrote: > The ctypes-conversion object has attributes which return c_types aware > objects so that the information can be passed directly to c-code (as an > integer, the number of dimensions can already be passed using c-types). > > The information available and it's corresponding c_type is > > data - c_void_p > shape, strides - c_int * nd or c_long * nd or c_longlong * nd > depending on platform Stefan and I did some more experiments and it seems like .ctypes.strides isn't doing the right thing for subarrays. For example: In [52]: x = N.rand(3,4) In [57]: [x.ctypes.strides[i] for i in range(x.ndim)] Out[57]: [32, 8] This looks fine. But for this subarray: In [56]: [x[1:3,1:4].ctypes.strides[i] for i in range(x.ndim)] Out[56]: [32, 8] In this case, I think one wants strides[0] (the row stride) to return 40. .ctypes.data already seems to do the right thing: In [60]: x.ctypes.data Out[60]: c_void_p(31685288) In [61]: x[1:3,1:4].ctypes.data Out[61]: c_void_p(31685328) In [62]: 31685288-31685328 Out[62]: 40 What would be a good way of dealing with discontiguous arrays? It seems like one might want to disable their .ctypes attribute. Regards, Albert From david.huard at gmail.com Mon Jul 3 09:19:07 2006 From: david.huard at gmail.com (David Huard) Date: Mon, 3 Jul 2006 09:19:07 -0400 Subject: [Numpy-discussion] iterate along a ray: linear algebra? In-Reply-To: <51f97e530606300713w1c167cf3j10c36d24f87326cf@mail.gmail.com> References: <51f97e530606300713w1c167cf3j10c36d24f87326cf@mail.gmail.com> Message-ID: <91cf711d0607030619v3155ea22o6529189f8a4f0fc8@mail.gmail.com> Hi Stephen, I don't know much about image analysis, but in the scipy tutorial (7.2Filtering), there is an example of an image filter that highlights the edges of an image. If I guess correctly, it finds the smoothing spline of the image and then computes the derivative of the spline. A high derivative means that the image intensity shifts rapidly, so maybe that could help you. David 2006/6/30, stephen emslie : > > I am in the process of implementing an image processing algorithm that > requires following rays extending outwards from a starting point and > calculating the intensity derivative at each point. The idea is to find the > point where the difference in intensity goes beyond a particular threshold. > > Specifically I'm examining an image of an eye to find the pupil, and the > edge of the pupil is a sharp change in intensity. > > How does one iterate along a line in a 2d matrix, and is there a better > way to do this? Is this a problem that linear algebra can help with? > > Thanks > Stephen Emslie > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at cox.net Mon Jul 3 10:19:29 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon, 03 Jul 2006 07:19:29 -0700 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: <44A8CC6A.40901@ieee.org> References: <44A8CC6A.40901@ieee.org> Message-ID: <44A92771.2040904@cox.net> Travis Oliphant wrote: > Hmmm..... One thing that bothers me is that it seems that those > arguing *against* this behavior are relatively long-time users of Python > while those arguing *for* it are from what I can tell somewhat new to > Python/NumPy. I'm not sure what this means. > > Is the current behavior a *wart* you get used to or a clear *feature* > providing some gain in programming efficiency. > > If new users complain about something, my tendency is to listen openly > to the discussion and then discourage the implementation only if there > is a clear reason to. > > With this one, I'm not so sure of the clear reason. I can see that > "program parsers" would have a harder time with a "flexible" calling > convention. But, in my calculus, user-interface benefit outweighs > programmer benefit (all things being equal). > > It's easy as a programmer to get caught up in a very rigid system when > many users want flexibility. > > I must confess that I don't like looking at ones((5,5)) either. I much > prefer ones(5,5) or even ones([5,5]). > I'm not sure why more people don't write the second version. It's significantly easier to read since the mixed brackets stand out better. If one buys into Guido's description of what tuples and lists are for, then it's also more appropriate since a shape is homogeneous, variable length sequence (as opposed to fixed length heterogeneous collection which would be more appropriate to represent using a tuple). Not everyone buys into that, using tuples as essentially immutable lists, but it's easier to read in any event. I also find: ones([5, 5], dt) clearer than: ones(5, 5, dt) or, more dramatic, consider: ones([dx, dy], dt) versus ones(dx, dy, dt). Brrrr! A side note: one reason I'm big on specifying the dtype is that according to Kahan (many papers available at http://www.cs.berkeley.edu/~wkahan/) the single best thing you can do to check that an implementation is numerically sane is to examine the results at different precisions (say float32 and float64 since they're commonly available) and verify that the results don't go off the rails. Kahan is oft quoted by Tim Peters who seems to have one of the better grasps of the fiddly aspects of floating point in the Python community, so I give his views a lot of weight. Since I try to program with this in mind, at least for touchy numerical code, I end up parameterizing things based on dtype anyway. Then it's relatively easy to check that things are behaving simply by changing the dtype that is passed in. > But perhaps what this shows is something I've heard Robert discuss > before that ihas not received enough attention. NumPy really has at > least two "users" 1) application developers and 2) prototype developers > (the MATLAB crowd for lack of a better term). > > These two groups usually want different functionality (and in reality > most of us probably fall in both groups on different occasions). The > first clamors for more rigidity and conformity even at the expense of > user interfaces. These people usually want > > 1) long_but_explanatory_function names > 2) rigid calling syntax > One might also call it consistent calling syntax, but yes I'd put myself in this group and having consistent calling syntax catches many errors that would otherwise pass silently. It's also easier to figure out what's going one when coming back to functions written in the distant pass if their is only one calling syntax. > 3) strict name-space control > This combined with more explanatory names mentioned in 1 help make it easier to decipher code written in yesteryear. > The second group which wants to get something prototyped and working > quickly wants > > 1) short easy-to-remember names > 2) flexible calling syntax > 3) all-in-one name-space control > > My hypothesis is that when you first start with NumPy (especially with a > MATLAB or IDL history) you seem to start out in group 2 and stay there > for quick projects. Then, as code-size increases and applications get > larger, you become more like group 1. > > I think both groups have valid points to make and both styles can be > useful and one point or another. Perhaps, the solution is the one I have > barely begun to recognize, that others of you have probably already seen. > > A) Make numpy a good "library" for group 1. > B) Make a shallow "name-space" (like pylab) with the properties of group 2. > > Perhaps a good solution is to formalize the discussion and actually > place in NumPy a name-space package much like Bill has done privately. > +1 -tim From aisaac at american.edu Mon Jul 3 10:58:02 2006 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 3 Jul 2006 10:58:02 -0400 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: <44A8F4FA.2060200@gmx.net> References: <44A8CC6A.40901@ieee.org> <44A8F4FA.2060200@gmx.net> Message-ID: On Mon, 03 Jul 2006, Sven Schreiber apparently wrote: > Anything that allows me to develop a consistent habit is fine with me! Consistency! That is exactly the issue, especially for those who wish to teach with numpy. I do not want to tell my students to write ones([5,5]) but rand(5,5) and although relatively new to Python I actually like the practice of providing dimensions in a list or tuple. But I care less about the choice of convention than about adherence to the convention. Cheers, Alan Isaac From jnc at ecs.soton.ac.uk Mon Jul 3 11:16:53 2006 From: jnc at ecs.soton.ac.uk (John Carter) Date: Mon, 03 Jul 2006 16:16:53 +0100 Subject: [Numpy-discussion] Error building numpy from svn Message-ID: <7.0.1.0.0.20060703153308.01aa0340@ecs.soton.ac.uk> Hi, I'm getting errors when I try and build numpy from the svn. I've followed the instructions on Installing Sci/Py/Windows at http://www.scipy.org/Installing_SciPy/Windows?highlight=%28%28----%28-%2A%29%28%5Cr%29%3F%5Cn%29%28.%2A%29CategoryInstallation%5Cb%29 I've downloaded, built and tested Atlas, Lapack, etc My computer is set up as follows: MinGW 3.4.2 Cygwin 3.4.4 (used for Atlas and Lapack) Win XP SP2 I've tried building using Cygwin instead MinGW with similar results. (not using cygwin python) I also have VC++ 6 and VC++ Express installed, but neither is on the path when I'm attempting to build numpy. I normally have no problems building my own python extensions using numarray, numpy or PIL. I use pyrex or home rolled code. These all work with 2.3 and 2.4. I'd be grateful of any pointers as to what might be wrong, Thanks in advance John =================================================================================== Using Python 2.3 D:.\numpy> setup.py config --compiler=mingwg2 build --compiler=mingw32 bdist_wininst ...... compile options: '-DNO_ATLAS_INFO=2 -Id:\work\Programming\numerical\numpy\numpy\core\include -Ibuild\src.win32-2.3\nump \core -Id:\work\Programming\numerical\numpy\numpy\core\src -Id:\work\Programming\numerical\numpy\numpy\core\include -IC \PYTHON23\include -IC:\PYTHON23\PC -c' C:\MINGW\BIN\g77.exe -shared build\temp.win32-2.3\Release\work\programming\numerical\numpy\numpy\linalg\lapack_litemodu e.o -Ld:\work\Programming\numerical\libs -LC:/MINGW/BIN/../lib/gcc/mingw32/3.4.2 -LC:\PYTHON23\libs -LC:\PYTHON23\PCBui d -llapack -llapack -lf77blas -lcblas -latlas -lpython23 -lgcc -lg2c -o build\lib.win32-2.3\numpy\linalg\lapack_lite.py C:/MINGW/BIN/../lib/gcc/mingw32/3.4.2/libgcc.a(__main.o)(.text+0x4f): undefined reference to `__EH_FRAME_BEGIN__' C:/MINGW/BIN/../lib/gcc/mingw32/3.4.2/libgcc.a(__main.o)(.text+0x73): undefined reference to `__EH_FRAME_BEGIN__' collect2: ld returned 1 exit status ========================================================================================= Using Python 2.4 D:.\numpy> setup.py config --compiler=mingwg2 build --compiler=mingw32 bdist_wininst Running from numpy source directory. No module named __svn_version__ F2PY Version 2_2727 blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not find in C:\PYTHON24\lib libraries mkl,vml,guide not find in C:\ libraries mkl,vml,guide not find in C:\PYTHON24\libs NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS FOUND: libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] library_dirs = ['d:\\work\\Programming\\numerical\\libs'] language = c No module named msvccompiler in numpy.distutils, trying from distutils.. Traceback (most recent call last): File "D:\work\Programming\numerical\numpy\setup.py", line 84, in ? setup_package() File "D:\work\Programming\numerical\numpy\setup.py", line 77, in setup_package configuration=configuration ) File "D:\work\Programming\numerical\numpy\numpy\distutils\core.py", line 144, in setup config = configuration() File "D:\work\Programming\numerical\numpy\setup.py", line 43, in configuration config.add_subpackage('numpy') File "D:\work\Programming\numerical\numpy\numpy\distutils\misc_util.py", line 740, in add_subpackage caller_level = 2) File "D:\work\Programming\numerical\numpy\numpy\distutils\misc_util.py", line 723, in get_subpackage caller_level = caller_level + 1) File "D:\work\Programming\numerical\numpy\numpy\distutils\misc_util.py", line 670, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File ".\numpy\setup.py", line 9, in configuration config.add_subpackage('core') File "D:\work\Programming\numerical\numpy\numpy\distutils\misc_util.py", line 740, in add_subpackage caller_level = 2) File "D:\work\Programming\numerical\numpy\numpy\distutils\misc_util.py", line 723, in get_subpackage caller_level = caller_level + 1) File "D:\work\Programming\numerical\numpy\numpy\distutils\misc_util.py", line 670, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File "d:\work\Programming\numerical\numpy\numpy\core\setup.py", line 207, in configuration blas_info = get_info('blas_opt',0) File "D:\work\Programming\numerical\numpy\numpy\distutils\system_info.py", line 256, in get_info return cl().get_info(notfound_action) File "D:\work\Programming\numerical\numpy\numpy\distutils\system_info.py", line 397, in get_info self.calc_info() File "D:\work\Programming\numerical\numpy\numpy\distutils\system_info.py", line 1244, in calc_info atlas_version = get_atlas_version(**version_info) File "D:\work\Programming\numerical\numpy\numpy\distutils\system_info.py", line 1085, in get_atlas_version library_dirs=config.get('library_dirs', []), File "D:\work\Programming\numerical\numpy\numpy\distutils\command\config.py", line 101, in get_output self._check_compiler() File "D:\work\Programming\numerical\numpy\numpy\distutils\command\config.py", line 34, in _check_compiler old_config._check_compiler(self) File "C:\PYTHON24\lib\distutils\command\config.py", line 107, in _check_compiler dry_run=self.dry_run, force=1) File "D:\work\Programming\numerical\numpy\numpy\distutils\ccompiler.py", line 333, in new_compiler compiler = klass(None, dry_run, force) File "C:\PYTHON24\lib\distutils\msvccompiler.py", line 211, in __init__ self.__macros = MacroExpander(self.__version) File "C:\PYTHON24\lib\distutils\msvccompiler.py", line 112, in __init__ self.load_macros(version) File "C:\PYTHON24\lib\distutils\msvccompiler.py", line 133, in load_macros raise DistutilsPlatformError, \ distutils.errors.DistutilsPlatformError: The .NET Framework SDK needs to be installed before building extensions for Pyt hon. D:.\numpy> Dr. John N. Carter jnc at ecs.soton.ac.uk ISIS http://www.ecs.soton.ac.uk/~jnc/ From pgmdevlist at mailcan.com Mon Jul 3 12:40:04 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Mon, 3 Jul 2006 12:40:04 -0400 Subject: [Numpy-discussion] None as missing value In-Reply-To: <44A8E927.9000405@gmx.net> References: <44A8B141.70506@ieee.org> <44A8E927.9000405@gmx.net> Message-ID: <200607031240.05066.pgmdevlist@mailcan.com> > I was also a bit surprised at the following behavior: >>> a = numpy.asarray([1,1]) >>> a array([1, 1]) >>> a[0]=numpy.nan >>> a array([0, 1]) Seems to affect only the int_ arrays: >>> a = numpy.asarray([1,1], dtype=float_) >>> a array([1., 1.]) >>> a[0]=numpy.nan >>> a array([ nan, 1. ]) From david.huard at gmail.com Mon Jul 3 13:00:35 2006 From: david.huard at gmail.com (David Huard) Date: Mon, 3 Jul 2006 13:00:35 -0400 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <44A8E4D3.5000603@ntc.zcu.cz> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> <44A8E4D3.5000603@ntc.zcu.cz> Message-ID: <91cf711d0607031000l286da52fn30d98798057f12c5@mail.gmail.com> Here is a quick benchmark between numpy's unique, unique1d and sasha's unique: x = rand(100000)*100 x = x.astype('i') %timeit unique(x) 10 loops, best of 3: 525 ms per loop %timeit unique_sasha(x) 100 loops, best of 3: 10.7 ms per loop timeit unique1d(x) 100 loops, best of 3: 12.6 ms per loop So I wonder what is the added value of unique? Could unique1d simply become unique ? Cheers, David P.S. I modified sasha's version to account for the case where all elements are identical, which returned an empty array. def unique_sasha(x): s = sort(x) r = empty(s.shape, float) r[:-1] = s[1:] r[-1] = NaN return s[r != s] 2006/7/3, Robert Cimrman : > > Sasha wrote: > > On 7/2/06, Norbert Nemec wrote: > >> ... > >> Does anybody know about the internals of the python "set"? How is > >> .keys() implemented? I somehow have really doubts about the efficiency > >> of this method. > >> > > Set implementation (Objects/setobject.c) is a copy and paste job from > > dictobject with values removed. As a result it is heavily optimized > > for the case of string valued keys - a case that is almost irrelevant > > for numpy. > > > > I think something like the following (untested, 1d only) will probably > > be much faster and sorted: > > > > def unique(x): > > s = sort(x) > > r = empty_like(s) > > r[:-1] = s[1:] > > r[-1] = s[0] > > return s[r != s] > > There are 1d array set operations like this already in numpy > (numpy/lib/arraysetops.py - unique1d, ...) > > r. > > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndarray at mac.com Mon Jul 3 14:07:00 2006 From: ndarray at mac.com (Sasha) Date: Mon, 3 Jul 2006 14:07:00 -0400 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: References: <44A8CC6A.40901@ieee.org> <44A8F4FA.2060200@gmx.net> Message-ID: On 7/3/06, Alan G Isaac wrote: > .... > Consistency! That is exactly the issue, > especially for those who wish to teach with numpy. > > I do not want to tell my students to write > ones([5,5]) > but > rand(5,5) > and although relatively new to Python > I actually like the practice of providing > dimensions in a list or tuple. > Consistency is already lost because 1d case allows both ones(5) and ones([5]) (and even ones((5,)) if anyone can tolerate that abomination). I don't think those who argue for sequence only are willing to require ones([5]). Remember, "A Foolish Consistency is the Hobgoblin of Little Minds" (Ralph Waldo Emerson (1803?1882), adopted without attribution as a section heading in PEP 8 ). I think the current situation strikes the right balance between convenience and consistency. From Chris.Barker at noaa.gov Mon Jul 3 14:30:20 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 03 Jul 2006 11:30:20 -0700 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: <44A92771.2040904@cox.net> References: <44A8CC6A.40901@ieee.org> <44A92771.2040904@cox.net> Message-ID: <44A9623C.6040708@noaa.gov> Tim Hochberg wrote: > I also find: > > ones([5, 5], dt) > > clearer than: > > ones(5, 5, dt) > > or, more dramatic, consider: ones([dx, dy], dt) versus ones(dx, dy, dt). > Brrrr! And I imagine this is EXACTLY why Numeric was originally designed that way: when there are more arguments to pass, then a shape is essentially a single entity. Also consider this: MyShape = a.shape b = ones(MyShape, N.float_) much better than: b = ones(*MyShape, N.float_) At least I think so. > Kahan is oft quoted by Tim Peters who seems to have one of the better > grasps of the fiddly aspects of floating point in the Python community, > so I give his views a lot of weight. I took at class with Kahan at Berkeley (that I barely passed), he is definitely someone to listen to. > NumPy really has at >> least two "users" 1) application developers and 2) prototype developers FWIW: I came from a heavy MATLAB background, still do far more prototyping that really application writing (though I certainly do the later), and I want: >> 1) longish_explanatory names >> 2) consistent calling syntax (note that I've edited those a bit...) And above all: >> 3) strict name-space control In my experience that that is a difference between "prototyping" and "using Python like Matlab". The sooner that you learn to use Python like Python, rather than trying to use it like Matlab, the better off you'll be. Python is an excellent language for prototyping, but even for that, you're better off working with, rather than against, its features. As for the current issue: 1) Make everything take a sequence for the size argument. 2) dump the "convenience functions" into a compatibility module. Is it so hard to write: import numpy.random.uniform as rand (though I notice that numpy.random.rand() is the inconsistent "convenience function" -- darn) Sasha wrote: > Consistency is already lost because 1d case allows both ones(5) and > ones([5]) (and even ones((5,)) if anyone can tolerate that > abomination). I don't think those who argue for sequence only are > willing to require ones([5]). But I'd be happy to ban ones(5) (by the way, doesn't Matlab's ones(5) produce a 5X5 matrix?). We certainly don't' want to require a list, that would totally go against Python's concept of duck typing. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Mon Jul 3 14:42:14 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 03 Jul 2006 11:42:14 -0700 Subject: [Numpy-discussion] Does zero based indexing drive anyone else crazy? In-Reply-To: References: Message-ID: <44A96506.6010904@noaa.gov> Webb Sprague wrote: > it is far more natural to program if the indices are aligned with the > counts of the elements I suggest that it's only more natural if that's what you're used to -- i.e. you come form other languages that do it that way. I fairly rarely get bitten by indexing 1, rather than zero, but I save a lot of errors that I used to get in MATLAB by the way python does slices: len(a[i:j]) == j - i and: l[:j] + l[j:] == l or: r_[a[:i],a[i:]] == a for numpy arrays. I suppose you could have one-indexing and the python slicing, but I think that would be even more error prone. > zero > based indexing made lots of sense since it dovetailed with thinking in > memory offsets in systems programming it also dovetails nicely into using an array to represent a grid of values: i = (X - MinX) / deltaX rather than i = (X - MinX) / deltaX + 1 X = i*deltaX rather than X = (i-1)*deltaX In Fortran, you can choose where you want your array indexing to start, and I found myself starting with zero more often than 1, and I was never a C programmer. > I guess I just want to vent, but also to ask if anyone > has found any way to deal with this issue in their own scientific > programming. You'll get used to it. There are disadvantages either way, but after switching from primarily Matlab to primarily Python, I like zero-based indexing better. Perhaps you will too. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From nikhil.padmanabhan at gmail.com Mon Jul 3 14:47:55 2006 From: nikhil.padmanabhan at gmail.com (Nikhil Padmanabhan) Date: Mon, 3 Jul 2006 14:47:55 -0400 Subject: [Numpy-discussion] dtype conversion errors in cumsum Message-ID: <5CEABBC2-4D17-450D-8088-D50EA6504F88@gmail.com> Hi, (I saw some discussion of this on the mailing list, but was not sure if there was a solution reached...) If I try to use cumsum on a numpy array with the dtype keyword, I get the wrong answer if dtype does not match that of the array. More specifically, Python 2.4.3 (#1, Apr 3 2006, 18:07:18) [GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin Type "help", "copyright", "credits" or "license" for more information. ->> import numpy ->> numpy.__version__ '0.9.9.2727' ->> a = numpy.arange(10002) ->> a.cumsum(dtype='f4')[-1] 0.0 ->> a.cumsum(dtype='f8')[-1] 0.0 ->> a.cumsum()[-1] 50015001 ->> a.sum() 50015001 ->> a.sum(dtype='f8') 50015001.0 Note that this only seems to affect cumsum(); sum() works just fine. And cumsum works fine for small arrays.... Am I misunderstanding what the dtype keyword in cumsum is, or doing something silly? I should not be overflowing any type bounds (I think...) This is with the SVN numpy, on an iBook G4 running OS X 10.4.6 Thanks in advance, -- Nikhil ------------------------------------ Nikhil Padmanabhan nikhil.padmanabhan at gmail.com From svetosch at gmx.net Mon Jul 3 15:01:24 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Mon, 03 Jul 2006 21:01:24 +0200 Subject: [Numpy-discussion] None as missing value In-Reply-To: <200607031240.05066.pgmdevlist@mailcan.com> References: <44A8B141.70506@ieee.org> <44A8E927.9000405@gmx.net> <200607031240.05066.pgmdevlist@mailcan.com> Message-ID: <44A96984.2020303@gmx.net> Pierre GM schrieb: >> I was also a bit surprised at the following behavior: >>>> a = numpy.asarray([1,1]) >>>> a > array([1, 1]) >>>> a[0]=numpy.nan >>>> a > array([0, 1]) > > Seems to affect only the int_ arrays: > >>>> a = numpy.asarray([1,1], dtype=float_) >>>> a > array([1., 1.]) >>>> a[0]=numpy.nan >>>> a > array([ nan, 1. ]) > Sure it works with floats. The question is, should there maybe be an error if the type of the assigned value doesn't match the dtype of the array, instead of silently doing something unexpected? Consider the following (with still the same *integer* array a from my post above): >>> a[0]=0.1 >>> a array([0, 1]) >>> a[0]='0' Traceback (most recent call last): File "", line 1, in ? TypeError: an integer is required This TypeError makes a lot of sense, but why does it only react to strings, not to floats? Cheers, Sven From oliphant.travis at ieee.org Mon Jul 3 15:38:17 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 03 Jul 2006 13:38:17 -0600 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: <00b201c69ea0$7ab7a710$0a84a8c0@dsp.sun.ac.za> References: <00b201c69ea0$7ab7a710$0a84a8c0@dsp.sun.ac.za> Message-ID: <44A97229.1020605@ieee.org> Albert Strasheim wrote: > Stefan and I did some more experiments and it seems like .ctypes.strides > isn't doing the right thing for subarrays. > > For example: > > In [52]: x = N.rand(3,4) > In [57]: [x.ctypes.strides[i] for i in range(x.ndim)] > Out[57]: [32, 8] > > This looks fine. But for this subarray: > > In [56]: [x[1:3,1:4].ctypes.strides[i] for i in range(x.ndim)] > Out[56]: [32, 8] > > In this case, I think one wants strides[0] (the row stride) to return 40. > Why do you think that? All sliced arrays keep the same strides information as their "parents". This is the essence of a "view". The striding is exactly the same as before (the data hasn't moved anywhere), only the starting point and the bounds have changed. > .ctypes.data already seems to do the right thing: > > In [60]: x.ctypes.data > Out[60]: c_void_p(31685288) > > In [61]: x[1:3,1:4].ctypes.data > Out[61]: c_void_p(31685328) > > In [62]: 31685288-31685328 > Out[62]: 40 > > What would be a good way of dealing with discontiguous arrays? It seems like > one might want to disable their .ctypes attribute. > > No, not at all. Discontiguous arrays are easily handled simply by using the strides information to step through the array in each dimension instead of "assuming" contiguousness. Perhaps there is some confusion about what the strides actually represent. It's quite easy to write C-code that takes stride information as well which will then work with discontiguous arrays. The benefit of this approach is that you avoid copying data when you don't really have to. There should be very little performance penalty in most algorithms as well as the strides calculation is not much more than adding 1 to the current pointer. -Travis From svetosch at gmx.net Mon Jul 3 15:56:04 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Mon, 03 Jul 2006 21:56:04 +0200 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: References: <44A8CC6A.40901@ieee.org> <44A8F4FA.2060200@gmx.net> Message-ID: <44A97654.1030800@gmx.net> Sasha schrieb: > > Consistency is already lost because 1d case allows both ones(5) and > ones([5]) (and even ones((5,)) if anyone can tolerate that > abomination). I don't think those who argue for sequence only are > willing to require ones([5]). Imho consistency is not lost there precisely because one can use ones((5)) if one is so inclined. So the double-parentheses habit (or call it tuple-habit if you like) goes through. In contrast, rand((5,5)) fails, and that is what breaks consistency. > > Remember, "A Foolish Consistency is the Hobgoblin of Little Minds" > (Ralph Waldo Emerson (1803?1882), adopted without attribution as a > section heading in PEP 8 ). A "little mind" is an accurate description of myself in the numpy field. But I believe that in order to become a success, numpy must take into account the needs of little minds as well. > > I think the current situation strikes the right balance between > convenience and consistency. I was arguing that it's inconvenient exactly because it's inconsistent, so I don't see the tradeoff here. (Given that I'm fairly indifferent between one or two pairs of parentheses.) In any case I promise to shut up about this when 1.0(beta) is out, but I think Alan is right that under the status quo there will be a constant stream of the same newbie question that I asked. cheers, Sven From ndarray at mac.com Mon Jul 3 16:23:37 2006 From: ndarray at mac.com (Sasha) Date: Mon, 3 Jul 2006 16:23:37 -0400 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: <44A97654.1030800@gmx.net> References: <44A8CC6A.40901@ieee.org> <44A8F4FA.2060200@gmx.net> <44A97654.1030800@gmx.net> Message-ID: On 7/3/06, Sven Schreiber wrote: > ... > I was arguing that it's inconvenient exactly because it's inconsistent, > so I don't see the tradeoff here. (Given that I'm fairly indifferent > between one or two pairs of parentheses.) > It is convenient to be able to write zeros(5) and rand(5,5) instead of consistent zeros([5]) and rand([5,5]). Note that inconsistency will not be fully cured by allowing rand([5,5]) because rand([5,5], float32) will still not work. Actually, I can see some value in a random numbers generator that conforms to the ones/zeros interface because that may promote use of random data in testing instead of ones/zeros. How about rands(shape, dtype=float, low=default_low(dtype), high=default_high(dtype)) ? Where default_{low,high}(floatN) should probably be {0,1} but for integer types some other default may make more sense. From npadmana at Princeton.EDU Mon Jul 3 16:35:40 2006 From: npadmana at Princeton.EDU (Nikhil Padmanabhan) Date: Mon, 3 Jul 2006 16:35:40 -0400 Subject: [Numpy-discussion] dtype error in cumsum Message-ID: <5B56AACF-C28E-406B-8F90-FD86F1253213@princeton.edu> (I sent the following from my gmail account, but that does not seem to have been accepted by sourceforge -- apologies in advance if it shows up twice) Hi, (I saw some discussion of this on the mailing list, but was not sure if there was a solution reached, so I thought I'd ask again....) If I try to use cumsum on a numpy array with the dtype keyword, I get the wrong answer if dtype does not match that of the array. More specifically, Python 2.4.3 (#1, Apr 3 2006, 18:07:18) [GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin Type "help", "copyright", "credits" or "license" for more information. ->> import numpy ->> numpy.__version__ '0.9.9.2727' ->> a = numpy.arange(10002) ->> a.cumsum(dtype='f4')[-1] 0.0 ->> a.cumsum(dtype='f8')[-1] 0.0 ->> a.cumsum()[-1] 50015001 ->> a.sum() 50015001 ->> a.sum(dtype='f8') 50015001.0 Note that this only seems to affect cumsum(); sum() works just fine. And cumsum works fine for small arrays.... Am I misunderstanding what the dtype keyword in cumsum is, or doing something silly? I should not be overflowing any type bounds (I think...) This is with the SVN numpy, on an iBook G4 running OS X 10.4.6 Thanks in advance, -- Nikhil ------------------------------------ Nikhil Padmanabhan npadmana at princeton.edu http://wwwphy.princeton.edu/~npadmana nikhil@(609) 258-4355 From ndarray at mac.com Mon Jul 3 16:40:53 2006 From: ndarray at mac.com (Sasha) Date: Mon, 3 Jul 2006 16:40:53 -0400 Subject: [Numpy-discussion] dtype error in cumsum In-Reply-To: <5B56AACF-C28E-406B-8F90-FD86F1253213@princeton.edu> References: <5B56AACF-C28E-406B-8F90-FD86F1253213@princeton.edu> Message-ID: It did get through all right. SF is a mess. In any case what you observe is clearly a bug. Looks like uninitialized memory somewhere. Please file a bug report at http://projects.scipy.org/scipy/numpy/newticket . On 7/3/06, Nikhil Padmanabhan wrote: > (I sent the following from my gmail account, but that does not seem > to have been accepted by sourceforge -- apologies in advance if it > shows up twice) > > Hi, > > (I saw some discussion of this on the mailing list, but was not sure > if there was a solution reached, so I thought I'd ask again....) If I > try to use cumsum on a numpy array with the dtype keyword, I get the > wrong answer if dtype does not match that of the array. More > specifically, > > Python 2.4.3 (#1, Apr 3 2006, 18:07:18) > [GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > ->> import numpy > ->> numpy.__version__ > '0.9.9.2727' > ->> a = numpy.arange(10002) > ->> a.cumsum(dtype='f4')[-1] > 0.0 > ->> a.cumsum(dtype='f8')[-1] > 0.0 > ->> a.cumsum()[-1] > 50015001 > ->> a.sum() > 50015001 > ->> a.sum(dtype='f8') > 50015001.0 > > Note that this only seems to affect cumsum(); sum() works just fine. > And cumsum works fine for small arrays.... > > Am I misunderstanding what the dtype keyword in cumsum is, or doing > something silly? I should not be overflowing any type bounds (I > think...) > > This is with the SVN numpy, on an iBook G4 running OS X 10.4.6 > > Thanks in advance, > -- Nikhil > > ------------------------------------ > Nikhil Padmanabhan > npadmana at princeton.edu > http://wwwphy.princeton.edu/~npadmana > nikhil@(609) 258-4355 > > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From torgil.svensson at gmail.com Mon Jul 3 17:39:04 2006 From: torgil.svensson at gmail.com (Torgil Svensson) Date: Mon, 3 Jul 2006 23:39:04 +0200 Subject: [Numpy-discussion] iterate along a ray: linear algebra? In-Reply-To: <51f97e530606300713w1c167cf3j10c36d24f87326cf@mail.gmail.com> References: <51f97e530606300713w1c167cf3j10c36d24f87326cf@mail.gmail.com> Message-ID: I've done something similar a few years ago (numarray,numeric). I started roughly at the middle and did 64 points from a reference point (xc,yc). This point together with a point at the edge of the image (xp,yp) also defined a reference angle (a0). (ysize,xsize) is the shape of the intensity image. I used the following code to calculate points of interest: na=64 xr,yr=xsize-xc,ysize-yc a0=arctan2(yp-yr,xp-xc) if a0<0: a0+=2*pi ac=arctan2([yr,yr,-yc,-yc],[xr,-xc,-xc,xr]) if numarray: ac[ac<0]+=2*pi else: ac=choose(ac<0,(ac,ac+2*pi)) a1,a2,a3,a4=ac rmaxfn={ 0: lambda a: a<=a1 and xr/cos(a-0.0*pi) or yr/cos(0.5*pi-a), 1: lambda a: a<=a2 and yr/cos(a-0.5*pi) or xc/cos(1.0*pi-a), 2: lambda a: a<=a3 and xc/cos(a-1.0*pi) or yc/cos(1.5*pi-a), 3: lambda a: a<=a4 and yc/cos(a-1.5*pi) or xr/cos(2.0*pi-a) } angles=arange(a0,a0+2*pi,2*pi/na) if numarray: angles[angles>=2*pi]-=2*pi else: angles=choose(angles>=2*pi,(angles,angles-2*pi)) nr=int(ceil(sqrt(max(yc,yr)**2+max(xc,xr)**2))) crmax=array([int(floor(rmaxfn[floor(a*2/pi)](a))) for a in angles]) cr=outerproduct(ones(na),arange(float(nr))) ca=outerproduct(angles,ones(nr)) x=cr*cos(ca)+xc y=cr*sin(ca)+yc After this I did cubic spline interpolation in the image with these points and did something useful. I don't know how relevant this is to you and it doesn't use the linear algebra package but it might give you some hint. If you find out a nifty way to do your rays please post on this thread. Sidenote -- Watch my explicit float argument to arange and even putting in pi there in one case. There's a discussion on this list that floats in arange are troublesome On 6/30/06, stephen emslie wrote: > I am in the process of implementing an image processing algorithm that > requires following rays extending outwards from a starting point and > calculating the intensity derivative at each point. The idea is to find the > point where the difference in intensity goes beyond a particular threshold. > > Specifically I'm examining an image of an eye to find the pupil, and the > edge of the pupil is a sharp change in intensity. > > How does one iterate along a line in a 2d matrix, and is there a better way > to do this? Is this a problem that linear algebra can help with? > > Thanks > Stephen Emslie > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From Fernando.Perez at colorado.edu Mon Jul 3 18:41:11 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon, 03 Jul 2006 16:41:11 -0600 Subject: [Numpy-discussion] A C++ library and the new array interface: the best approach? Message-ID: <44A99D07.5030706@colorado.edu> Hi all, I'd like to ask for a bit of guidance on the best path to follow regarding the integration of an existing C++ library with numpy, given the recent developments on these topics: 1. the new array interface specification 2. the ctypes discussions 3. pyrex Our situation is the following: an existing in-house C++ library defines a templated class (Tensor), in many ways similar to a numpy array, but with its own extra functionality for other things. There are many functions in this code which take or return Tensors. I now need to interface this library with a Python code which is fully numpy-based, and hopefully be able to go back and forth between the C++ and python sides without unnecessary copies. Since the C++ codebase is somewhat large and already exists, and will continue to evolve as a standalone C++ system, something tells me I should just bite the typemap bullet and SWIG the darn thing. But I've never written anything beyond the most trivial typemaps, and I'm not 100% sure on exactly how to take advantage of the new array interface with SWIG. I read all the docs under - http://svn.scipy.org/svn/PEP - http://www.scipy.org/BaseArray - http://numeric.scipy.org/array_interface.html - the pyrex/array interface wiki pages (the site is not responding right now, so I can't give a URL) but my lack of familiarity with all the details of new type creation got me a bit lost. I'm sure the information I need is all there, but right now I don't really see the forest with all the leaves in my face. I've also read the various recent threads on ctypes, as well was the one initiated by Joris: http://article.gmane.org/gmane.comp.python.numeric.general/6296 So I'd like to know if SWIG is really the best way out in this particular case (and any advice on taking advantage of the array interface via SWIG would be appreciated), or if ctypes or pyrex could be used here. I'm quite happy using pyrex in other contexts, but I know it doesn't directly support C++. However, since I have access to all the code, perhaps a pure C layer could be used to bridge the C++/pyrex gap. Or given the recent praise for ctypes, perhaps that can be an option? Any advice will be greatly appreciated. Best, f From charlesr.harris at gmail.com Mon Jul 3 18:51:17 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 3 Jul 2006 16:51:17 -0600 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: References: <44A8CC6A.40901@ieee.org> <44A8F4FA.2060200@gmx.net> <44A97654.1030800@gmx.net> Message-ID: +1 for tuples, Because dimensions *are* tuples. I think using tuples is a good habit to get into and agree with Robert Kern's comments about using the new random number interface instead of the old compatibility functions rand and randn. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Mon Jul 3 19:06:42 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 03 Jul 2006 17:06:42 -0600 Subject: [Numpy-discussion] A C++ library and the new array interface: the best approach? In-Reply-To: <44A99D07.5030706@colorado.edu> References: <44A99D07.5030706@colorado.edu> Message-ID: <44A9A302.90901@ieee.org> Fernando Perez wrote: > Hi all, > > but my lack of familiarity with all the details of new type creation got me a > bit lost. I'm sure the information I need is all there, but right now I don't > really see the forest with all the leaves in my face. I've also read the > various recent threads on ctypes, as well was the one initiated by Joris: > > http://article.gmane.org/gmane.comp.python.numeric.general/6296 > > > So I'd like to know if SWIG is really the best way out in this particular case > (and any advice on taking advantage of the array interface via SWIG would be > appreciated), or if ctypes or pyrex could be used here. I'm quite happy using > pyrex in other contexts, but I know it doesn't directly support C++. However, > since I have access to all the code, perhaps a pure C layer could be used to > bridge the C++/pyrex gap. Or given the recent praise for ctypes, perhaps that > can be an option? > I don't have a lot of time to respond right now. But here are some thoughts. I'm not sure I'd use SWIG at this point. I'd probably either a) use ctypes to call the C++ library (if it is compiled as a *shared* library) although to be honest I'm not sure if calling a C++ causes any grief. b) byte-the bullet and use Boost python (which other's who have integrated C++ code with Python seem to really like). I've never wrapped my mind around Boost python, though, so I'm not much help there. c) write a custom Tensor class that either inherits from the ndarray or is a mix-in class with the ndarray as a component. If it were me, I'd probably be doing a combination of a and c. I've just finished the chapter in my book that covers and compares f2py, weave, pyrex and ctypes integration. I've been waiting on finishing this chapter before sending updates. You should get the update by the end of the week. Perhaps more later. -Travis > Any advice will be greatly appreciated. > > Best, > > f > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From ltaylor at llnl.gov Mon Jul 3 19:13:39 2006 From: ltaylor at llnl.gov (Lee Taylor) Date: Mon, 3 Jul 2006 16:13:39 -0700 (PDT) Subject: [Numpy-discussion] Namespaces in header files [Was: Time for beta1 of NumPy 1.0] In-Reply-To: <44A47854.1050106@ieee.org> References: <44A47854.1050106@ieee.org> Message-ID: On Thu, 29 Jun 2006, Travis Oliphant wrote: > I think it's time for the first beta-release of NumPy 1.0 > > I'd like to put it out within 2 weeks. Please make any comments or > voice major concerns so that the 1.0 release series can be as stable as > possible. One issue I ran across that I have not seen addressed is the namespace of arrayobject.h. I'm not refering to C++ namespaces but prefixing symbols to avoid clashes with user's code. The externals start with PyArray. But I had symbol redefinition errors for byte, MAX_DIMS, and ERR. That is, I already had defines for MAX_DIMS and ERR and a typedef for byte in my code. When adding a numpy interface to my library I had to undef these symbols before including arrayobject.h. Is there a way to move implemention defines, like ERR, into a separate header. Or if they're part of the API, prefix the symbols? Lee Taylor From strawman at astraw.com Mon Jul 3 20:22:32 2006 From: strawman at astraw.com (Andrew Straw) Date: Mon, 03 Jul 2006 17:22:32 -0700 Subject: [Numpy-discussion] A C++ library and the new array interface: the best approach? In-Reply-To: <44A99D07.5030706@colorado.edu> References: <44A99D07.5030706@colorado.edu> Message-ID: <44A9B4C8.10907@astraw.com> Dear Fernando, A couple of quick thoughts: A) Despite my love of Pyrex, it probably isn't the tool for the job -- it doesn't play particularly well with C++, and it would require you to write custom code for every function wrapped. B) It sounds like you want something that will more-or-less automatically generate the python interface from the header files. To my mind, this suggests SWIG or boost. Boost uses some seriously fancy (and mind-bending) C++, which can either be a good thing or a bad thing, but it certainly boosts compile time! There are doubtless other ways. For example, perhaps you could try using gccxml to auto-generate something like a Python/ctypes wrapper. C) The core of whatever you do will probably involve writing code to view a Tensor as an object that has the __array_struct__ interface. That way numpy/scipy/whatever can use the data with no copying. So that's a good place to start and whatever you learn there will probably be useful regardless of whatever tools you end up with. You'll probably also want the converse function so that you can view objects exposing __array_struct__ (numpy arrays) as a Tensor. Anyhow, that's not much, but perhaps it'll help. Cheers! Andrew Fernando Perez wrote: >Hi all, > >I'd like to ask for a bit of guidance on the best path to follow regarding the >integration of an existing C++ library with numpy, given the recent >developments on these topics: > >1. the new array interface specification >2. the ctypes discussions >3. pyrex > >Our situation is the following: an existing in-house C++ library defines a >templated class (Tensor), in many ways similar to a numpy array, but with its >own extra functionality for other things. There are many functions in this >code which take or return Tensors. I now need to interface this library with >a Python code which is fully numpy-based, and hopefully be able to go back and >forth between the C++ and python sides without unnecessary copies. > >Since the C++ codebase is somewhat large and already exists, and will continue >to evolve as a standalone C++ system, something tells me I should just bite >the typemap bullet and SWIG the darn thing. But I've never written anything >beyond the most trivial typemaps, and I'm not 100% sure on exactly how to take >advantage of the new array interface with SWIG. I read all the docs under > > - http://svn.scipy.org/svn/PEP > > - http://www.scipy.org/BaseArray > > - http://numeric.scipy.org/array_interface.html > > - the pyrex/array interface wiki pages (the site is not responding right >now, so I can't give a URL) > > >but my lack of familiarity with all the details of new type creation got me a >bit lost. I'm sure the information I need is all there, but right now I don't >really see the forest with all the leaves in my face. I've also read the >various recent threads on ctypes, as well was the one initiated by Joris: > >http://article.gmane.org/gmane.comp.python.numeric.general/6296 > > >So I'd like to know if SWIG is really the best way out in this particular case >(and any advice on taking advantage of the array interface via SWIG would be >appreciated), or if ctypes or pyrex could be used here. I'm quite happy using >pyrex in other contexts, but I know it doesn't directly support C++. However, >since I have access to all the code, perhaps a pure C layer could be used to >bridge the C++/pyrex gap. Or given the recent praise for ctypes, perhaps that >can be an option? > >Any advice will be greatly appreciated. > >Best, > >f > >Using Tomcat but need to do more? Need to support web services, security? >Get stuff done quickly with pre-integrated technology to make your job easier >Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo >http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From kwgoodman at gmail.com Mon Jul 3 21:53:33 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 3 Jul 2006 18:53:33 -0700 Subject: [Numpy-discussion] Does zero based indexing drive anyone else crazy? In-Reply-To: References: Message-ID: On 7/2/06, Webb Sprague wrote: > Given the long history of python and its ancestry in C (for which zero > based indexing made lots of sense since it dovetailed with thinking in > memory offsets in systems programming), there is probably nothing to > be done now. I guess I just want to vent, but also to ask if anyone > has found any way to deal with this issue in their own scientific > programming. Aha! Guido himself prefers starting the index at one. Here's a code snippet from a fun article he wrote about optimizing python code: import time def timing(f, n, a): print f.__name__, r = range(n) t1 = time.clock() for i in r: f(a); f(a); f(a); f(a); f(a); f(a); f(a); f(a); f(a); f(a) t2 = time.clock() print round(t2-t1, 3) http://www.python.org/doc/essays/list2str/ Notice he chose t1 and t2 instead of t0 and t1. QED From egilber2 at cs.uiuc.edu Mon Jul 3 21:58:27 2006 From: egilber2 at cs.uiuc.edu (Eric Gilbert) Date: Mon, 03 Jul 2006 20:58:27 -0500 Subject: [Numpy-discussion] Research study on CVS - you can help and get paid! Message-ID: I am a Computer Science graduate student at the University of Illinois at Urbana-Champaign. I need your help in a study I want to conduct. I have created a project called LifeSource that visualizes CVS repositories. You can find screenshots and info here: http://social.cs.uiuc.edu/people/gilbert (info on me) http://social.cs.uiuc.edu/people/gilbert/research.html (info on LifeSource, the CVS project) LifeSource appeared as a Work-in-Progress paper at ACM's CHI 2006 in Montreal (http://www.chi2006.org). I applied LifeSource to the Azureus and Gaim CVS repositories. I would like to apply the same visualization to your project. Next, I want to see if you as the visualized community find it accurate, helpful, disturbing, etc. In order to participate in the study, you will need to view the LifeSource visualization once a week (on the web) for four weeks and respond to two short online surveys. You will receive a $20 gift certificate for your participation. I can fully explain the process in the near future - just email me back. I will provide all of the information of the methodology of the study online. (We can set up a blog for study members to leave further comments.) Please let me know what you think. I plan on using the study in a long paper to CHI 2007. Cheers, Eric P.S. I apologize for the spam and will send no more. From wbaxter at gmail.com Mon Jul 3 23:39:28 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 4 Jul 2006 12:39:28 +0900 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: References: <44A8CC6A.40901@ieee.org> <44A8F4FA.2060200@gmx.net> <44A97654.1030800@gmx.net> Message-ID: I just noticed that array.transpose() takes either a tuple or separate arguments. I'm not trying to imply anything, just I was thinking such a thing had no precedent in numpy, but it turns out it does. And as an aside, the docstring for .transpose() needs fixing. It says: >>> a = array([[1,2],[3,4]]) >>> a array([[1, 2], [3, 4]]) >>> a.transpose() array([[1, 3], [3, 4]]) Whoa -- transpose destroys any 2's in your input and turns them into 3's! How does one use this Trac thing again? I couldn't find a "submit a bug" link anywhere on the web scipy.org web page. I found it once before, but now I can't seem to locate it. --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Jul 3 23:45:17 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 03 Jul 2006 22:45:17 -0500 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: References: <44A8CC6A.40901@ieee.org> <44A8F4FA.2060200@gmx.net> <44A97654.1030800@gmx.net> Message-ID: Bill Baxter wrote: > How does one use this Trac thing again? I couldn't find a "submit a > bug" link anywhere on the web scipy.org web page. I > found it once before, but now I can't seem to locate it. http://projects.scipy.org/scipy/scipy You need to register first. Click the "Register" link in the upper right. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Mon Jul 3 23:54:19 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 03 Jul 2006 22:54:19 -0500 Subject: [Numpy-discussion] Args for ones, zeros, rand, eye, ones, empty (possible 1.0 change?) In-Reply-To: References: <44A8CC6A.40901@ieee.org> <44A8F4FA.2060200@gmx.net> <44A97654.1030800@gmx.net> Message-ID: Robert Kern wrote: > Bill Baxter wrote: >> How does one use this Trac thing again? I couldn't find a "submit a >> bug" link anywhere on the web scipy.org web page. I >> found it once before, but now I can't seem to locate it. > > http://projects.scipy.org/scipy/scipy > > You need to register first. Click the "Register" link in the upper right. Or rather, http://projects.scipy.org/scipy/numpy -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From michael.sorich at gmail.com Tue Jul 4 00:44:56 2006 From: michael.sorich at gmail.com (Michael Sorich) Date: Tue, 4 Jul 2006 14:14:56 +0930 Subject: [Numpy-discussion] MA bug or feature? In-Reply-To: References: <16761e100606210146q7683c94bu5bd2699caa6b95cf@mail.gmail.com> <200606210612.09374.pgmdevlist@mailcan.com> Message-ID: <16761e100607032144w65013936jbbf9772f429f4be9@mail.gmail.com> On 6/23/06, Paul Dubois wrote: > In the original MA in Numeric, I decided that to constantly check for masks > that didn't actually mask anything was not a good idea. It punishes normal > use with a very expensive check that is rarely going to be true. > > If you are in a setting where you do not want this behavior, but instead > want masks removed whenever possible, you may wish to wrap or replace things > like masked_array so that they call make_mask with flag = 1: > > y = masked_array(data, make_mask(maskdata, flag=1)) > > y will have no mask if maskdata is all false. > Hi Paul. If this is purely for optimisation, is there perhaps a better way to do so in which the optimisation is hidden? For example if the optimisation is in regard to the case in which none of the elements is masked, an alternative approach may be to make a subclass of ndarray and cache the result of the any method. e.g. import numpy as N def asmask(data): if isinstance(data, Mask): return data else: return Mask(data) class Mask(N.ndarray): __array_priority__ = 10.0 def __new__(subtype, data): ret = N.array(data, N.bool_) return ret.view(Mask) def __init__(self, data): self._any = None def any(self): if self._any is None: self._any = N.ndarray.any(self) return self._any def __setitem__(self, index, value): self._any = None N.ndarray.__setitem__(self, index, value) The biggest problem I have with the current setup is the inconsistency between the behaviour of the array when the mask is nomask vs a boolarray with all False. Another example of this is when one changes the mask on an element. This is not possible when the mask is nomask print N.version.version ma1 = N.ma.array([1,2,3], mask=[False, False, False]) print ma1.mask ma1.mask[2] = True ma2 = N.ma.array([1,2,3], mask=False) print ma2.mask ma2.mask[2] = True ----- output 0.9.8 [False False False] [False False True] False Traceback (most recent call last): File "D:\eclipse\Mask\src\mask\__init__.py", line 111, in ? ma2.mask[2] = True TypeError: object does not support item assignment From pgmdevlist at mailcan.com Tue Jul 4 02:39:59 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Tue, 4 Jul 2006 02:39:59 -0400 Subject: [Numpy-discussion] MA bug or feature? In-Reply-To: <16761e100607032144w65013936jbbf9772f429f4be9@mail.gmail.com> References: <16761e100606210146q7683c94bu5bd2699caa6b95cf@mail.gmail.com> <16761e100607032144w65013936jbbf9772f429f4be9@mail.gmail.com> Message-ID: <200607040240.00075.pgmdevlist@mailcan.com> Michael, I wonder whether the Mask class you suggest is not a bit overkill. There should be enough tools in the existing MA module to do what we want. And I don't wanna think about compatibility the number of changes in the MA code that'd be required (but I'm lazy)... For the sake of consistency and optimization, I still think it could be easier (and cleaner) to make `nomask` the default for a MaskedArray without masked values. That could for example be implemented by forcing `nomask` at the creation of the MaskedArray with an extra `if mask and not mask.any(): mask=nomask`, or by using Paul's make_mask( flag=1) trick. Masking some specific values could still be done when mask is nomask with an intermediary MA.getmaskarray() step. On a side note, modifying an existing mask is a delicate matter. Everything's OK if you use masks as a way to hide existing data, it's more complex when initially you have some holes in your dataset... From fullung at gmail.com Tue Jul 4 03:57:10 2006 From: fullung at gmail.com (Albert Strasheim) Date: Tue, 4 Jul 2006 09:57:10 +0200 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: <44A97229.1020605@ieee.org> Message-ID: <00f301c69f3f$740616f0$0a84a8c0@dsp.sun.ac.za> Hello all > > In this case, I think one wants strides[0] (the row stride) to return > 40. > > > > Why do you think that? > > All sliced arrays keep the same strides information as their > "parents". This is the essence of a "view". The striding is exactly > the same as before (the data hasn't moved anywhere), only the starting > point and the bounds have changed. Sorry, I was suffering some temporary strides brain damage. :-) Regards, Albert From fullung at gmail.com Tue Jul 4 04:44:56 2006 From: fullung at gmail.com (Albert Strasheim) Date: Tue, 4 Jul 2006 10:44:56 +0200 Subject: [Numpy-discussion] A C++ library and the new array interface: the best approach? In-Reply-To: <44A9A302.90901@ieee.org> References: <44A99D07.5030706@colorado.edu> <44A9A302.90901@ieee.org> Message-ID: <20060704084456.GB4754@dogbert.sdsl.sun.ac.za> Hello all On Mon, 03 Jul 2006, Travis Oliphant wrote: > Fernando Perez wrote: > > Hi all, > > > > but my lack of familiarity with all the details of new type creation got me a > > bit lost. I'm sure the information I need is all there, but right now I don't > > really see the forest with all the leaves in my face. I've also read the > > various recent threads on ctypes, as well was the one initiated by Joris: > > > > http://article.gmane.org/gmane.comp.python.numeric.general/6296 > > > > So I'd like to know if SWIG is really the best way out in this particular case > > (and any advice on taking advantage of the array interface via SWIG would be > > appreciated), or if ctypes or pyrex could be used here. I'm quite happy using > > pyrex in other contexts, but I know it doesn't directly support C++. However, > > since I have access to all the code, perhaps a pure C layer could be used to > > bridge the C++/pyrex gap. Or given the recent praise for ctypes, perhaps that > > can be an option? > > I don't have a lot of time to respond right now. But here are some > thoughts. > > I'm not sure I'd use SWIG at this point. At least for C code, I would definately agree. If you already know SWIG, you might want to try it, but I started without any knowledge of SWIG and ctypes when I wanted to wrap a pretty basic library, and after a week of puzzling over SWIG, I learned ctypes and wrapped the library in a matter of hours. > I'd probably either > > a) use ctypes to call the C++ library (if it is compiled as a *shared* > library) although to be honest I'm not sure if calling a C++ causes any > grief. If I understand correctly, because each C++ compiler potentially mangles C++ identifier names in a different way, there isn't a way for ctypes to know what to look for in a shared library. > b) byte-the bullet and use Boost python (which other's who have > integrated C++ code with Python seem to really like). I've never > wrapped my mind around Boost python, though, so I'm not much help there. If there is any way for you to "hide" you C++ code behind an extern "C" interface, i.e. write some C-style wrappers on top of your C++ code, then you can use ctypes. As luck would have it, this is exactly what the libsvm authors did already, which made wrapping libsvm with ctypes relatively straightforward. If you want to achieve things like Python classes deriving from C++ classes and vice versa, Boost-Python is what you want to look at. But as others have noted, prepare to wait for the compiler when wrapping anything of even moderate size. In Boost-Python world there's Pyste that allows you to generate code. I think Pyste uses GCC-XML underneath. I played with Pyste for a day or two long ago, but at that stage the code it made differed greatly from what I wrote when doing Boost-Python wrapping by hand, so it was a bit strange. In general, I got the feeling that Pyste wasn't flexible enough to wrap any kind of C++ library. But maybe you're lucky and your C++ code is just right to wrap easily with Pyste. > c) write a custom Tensor class that either inherits from the ndarray or > is a mix-in class with the ndarray as a component. Don't know much about this option. In summary (keeping in mind that I'm a bit of a ctypes zealot at present), I'd try the "C interface on top of C++ with ctypes" approach, and build an ndarray subclass on top these C functions. Regards, Albert From pau.gargallo at gmail.com Tue Jul 4 05:25:32 2006 From: pau.gargallo at gmail.com (Pau Gargallo) Date: Tue, 4 Jul 2006 11:25:32 +0200 Subject: [Numpy-discussion] tentative numpy tutorial Message-ID: <6ef8f3380607040225t30948964o49b0f570b8768edf@mail.gmail.com> hi all, motivated by the lack of free documentation for NumPy, with some friends, we started writing a tutorial, that we would like to see in scipy.org. After some time, the project have started to loose its initial impetus. Now, we put the current unfinished version in http://www.scipy.org/Tentative_NumPy_Tutorial with the hope that, with the help of other users, it will be possible to build a useful document for beginners. We are not Numpy experts nor native English speakers (as you probably realized ;-), so the current version needs to be reviewed. Moreover, there are huge holes marked with (TODO) tags waiting for motivated people to fill them. In my opinion, the lack of free documentation is preventing many users to get started with NumPy: - Many NumPy features are only documented in the Guide to NumPy. - (In my opinion) New users are not going to buy the Guide before knowing about this features. So there is a kind of dead lock here. It would be nice, if NumPy 1.0 could be released with a complete free tutorial, don't you think so? then click the edit button! thanks, pau From jnc at ecs.soton.ac.uk Tue Jul 4 05:18:53 2006 From: jnc at ecs.soton.ac.uk (John Carter) Date: Tue, 04 Jul 2006 10:18:53 +0100 Subject: [Numpy-discussion] Error building numpy from svn, 2nd attempt Message-ID: <7.0.1.0.0.20060704100002.01f8fc38@ecs.soton.ac.uk> Hi, As is the way posting to a news group stirs the brain cell into activity and the problem is solved. or rather shifted. I've downloaded the candidate version of mingw32 and using that to build numpy/scipy works, or rather it builds the extensions for Python 2.3 I believe that there are still problems as both numpy and scipy fail their tests. I'm also dubious as to whether they are finding Altas and Lapack. Configuration: Windows XP SP2 Cygwin = gcc version 3.4.4 (cygming special) (gdc 0.12, using dmd 0.125) MinGW = gcc version 3.4.5 (mingw special) Python = Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32 scipy, numpy, Atlas, Lapack all downloaded within the last 2 days I've cut and pasted the error messages from the tests below. Any help gratefully received. John ===== numpy ===== D:.\numerical> python Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.test() Found 5 tests for numpy.distutils.misc_util Found 3 tests for numpy.lib.getlimits Found 30 tests for numpy.core.numerictypes Found 13 tests for numpy.core.umath Found 3 tests for numpy.core.scalarmath Found 8 tests for numpy.lib.arraysetops Found 42 tests for numpy.lib.type_check Found 101 tests for numpy.core.multiarray Found 36 tests for numpy.core.ma Found 10 tests for numpy.lib.twodim_base Found 10 tests for numpy.core.defmatrix Found 1 tests for numpy.lib.ufunclike Found 38 tests for numpy.lib.function_base Found 3 tests for numpy.dft.helper Found 1 tests for numpy.lib.polynomial Found 7 tests for numpy.core.records Found 26 tests for numpy.core.numeric Found 4 tests for numpy.lib.index_tricks Found 46 tests for numpy.lib.shape_base Found 0 tests for __main__ ........................................................................................................ ........................................................................................................ .........E.............................................................................................. ........................... ====================================================================== ERROR: check_simple (numpy.lib.tests.test_twodim_base.test_histogram2d) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python23\Lib\site-packages\numpy\lib\tests\test_twodim_base.py", line 137, in check_simple np.random.seed(1) File "mtrand.pyx", line 311, in mtrand.RandomState.seed SystemError: C:\sf\python\dist\src-maint23\Objects\longobject.c:240: bad argument to internal function ---------------------------------------------------------------------- Ran 387 tests in 1.391s FAILED (errors=1) >>> ===== scipy ===== D:.\numerical> python Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import scipy >>> scipy.test() Overwriting lib= from C:\PYTHON23\lib\si te-packages\scipy\lib\__init__.pyc (was from C:\PYTHON23\lib\site-packages\numpy\lib\__init__.pyc) Overwriting fft= from scipy.fftpack.basic (was from numpy.dft.f ftpack) Overwriting ifft= from scipy.fftpack.basic (was from numpy.df t.fftpack) Found 4 tests for scipy.io.array_import Found 128 tests for scipy.linalg.fblas Found 397 tests for scipy.ndimage Found 10 tests for scipy.integrate.quadpack Found 97 tests for scipy.stats.stats Found 47 tests for scipy.linalg.decomp Found 2 tests for scipy.integrate.quadrature Found 95 tests for scipy.sparse.sparse Found 20 tests for scipy.fftpack.pseudo_diffs Found 6 tests for scipy.optimize.optimize Found 5 tests for scipy.interpolate.fitpack Found 1 tests for scipy.interpolate Found 12 tests for scipy.io.mmio Found 10 tests for scipy.stats.morestats Found 4 tests for scipy.linalg.lapack Found 18 tests for scipy.fftpack.basic Found 4 tests for scipy.linsolve.umfpack Found 4 tests for scipy.optimize.zeros Found 41 tests for scipy.linalg.basic Found 2 tests for scipy.maxentropy.maxentropy Found 358 tests for scipy.special.basic Found 128 tests for scipy.lib.blas.fblas Found 7 tests for scipy.linalg.matfuncs **************************************************************** WARNING: clapack module is empty ----------- See scipy/INSTALL.txt for troubleshooting. Notes: * If atlas library is not found by numpy/distutils/system_info.py, then scipy uses flapack instead of clapack. **************************************************************** Found 42 tests for scipy.lib.lapack Found 1 tests for scipy.optimize.cobyla Found 16 tests for scipy.lib.blas Found 1 tests for scipy.integrate Found 14 tests for scipy.linalg.blas Found 70 tests for scipy.stats.distributions Found 4 tests for scipy.fftpack.helper Found 4 tests for scipy.signal.signaltools Found 0 tests for __main__ Don't worry about a warning regarding the number of bytes read. Warning: 1000000 bytes requested, 20 bytes read. .......caxpy:n=4 ..caxpy:n=3 ....ccopy:n=4 ..ccopy:n=3 .............cscal:n=4 ....cswap:n=4 ..cswap:n=3 .....daxpy:n=4 ..daxpy:n=3 ....dcopy:n=4 ..dcopy:n=3 .............dscal:n=4 ....dswap:n=4 ..dswap:n=3 .....saxpy:n=4 ..saxpy:n=3 ....scopy:n=4 ..scopy:n=3 .............sscal:n=4 ....sswap:n=4 ..sswap:n=3 .....zaxpy:n=4 ..zaxpy:n=3 ....zcopy:n=4 ..zcopy:n=3 .............zscal:n=4 ....zswap:n=4 ..zswap:n=3 ........................................................................................................................ ........................................................................................................................ ........................................................................................................................ ........................................................................................................................ .........................................................................Took 13 points. ..........Resizing... 16 17 24 Resizing... 20 7 35 Resizing... 23 7 47 Resizing... 24 25 58 Resizing... 28 7 68 Resizing... 28 27 73 .....Use minimum degree ordering on A'+A. ........................Use minimum degree ordering on A'+A. ...................Resizing... 16 17 24 Resizing... 20 7 35 Resizing... 23 7 47 Resizing... 24 25 58 Resizing... 28 7 68 Resizing... 28 27 73 .....Use minimum degree ordering on A'+A. .................Resizing... 16 17 24 Resizing... 20 7 35 Resizing... 23 7 47 Resizing... 24 25 58 Resizing... 28 7 68 Resizing... 28 27 73 .....Use minimum degree ordering on A'+A. ......................................C:\PYTHON23\lib\site-packages\scipy\interpolate\fitpack2.py:410: UserWarning: The coefficients of the spline returned have been computed as the minimal norm least-squares solution of a (numerically) rank deficient system (deficiency=7). If deficiency is large, the results may be inaccurate. Deficiency may strongly depend on the value of eps. warnings.warn(message) ....................Ties preclude use of exact statistic. ..Ties preclude use of exact statistic. ........ **************************************************************** WARNING: clapack module is empty ----------- See scipy/INSTALL.txt for troubleshooting. Notes: * If atlas library is not found by numpy/distutils/system_info.py, then scipy uses flapack instead of clapack. **************************************************************** ....................data-ftype: z compared to data D Calling _superlu.zgssv Use minimum degree ordering on A'+A. .data-ftype: c compared to data F Calling _superlu.cgssv Use minimum degree ordering on A'+A. .data-ftype: d compared to data d Calling _superlu.dgssv Use minimum degree ordering on A'+A. .data-ftype: s compared to data f Calling _superlu.sgssv Use minimum degree ordering on A'+A. ........................................................................................................................ ........................................................................................................................ ....................................Gegenbauer, a = 2.7337413228 ........................................................................................................................ .............caxpy:n=4 ..caxpy:n=3 ....ccopy:n=4 ..ccopy:n=3 .............cscal:n=4 ....cswap:n=4 ..cswap:n=3 .....daxpy:n=4 ..daxpy:n=3 ....dcopy:n=4 ..dcopy:n=3 .............dscal:n=4 ....dswap:n=4 ..dswap:n=3 .....saxpy:n=4 ..saxpy:n=3 ....scopy:n=4 ..scopy:n=3 .............sscal:n=4 ....sswap:n=4 ..sswap:n=3 .....zaxpy:n=4 ..zaxpy:n=3 ....zcopy:n=4 ..zcopy:n=3 .............zscal:n=4 ....zswap:n=4 ..zswap:n=3 ...Result may be inaccurate, approximate err = 4.1928136851e-009 ...Result may be inaccurate, approximate err = 7.27595761418e-012 .............................................F................Residual: 1.05006950319e-007 . **************************************************************** WARNING: cblas module is empty ----------- See scipy/INSTALL.txt for troubleshooting. Notes: * If atlas library is not found by numpy/distutils/system_info.py, then scipy uses fblas instead of cblas. **************************************************************** ............................................................................................ ====================================================================== FAIL: check_simple (scipy.optimize.tests.test_cobyla.test_cobyla) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python23\Lib\site-packages\scipy\optimize\tests\test_cobyla.py", line 20, in check_simple assert_almost_equal(x, [x0, x1], decimal=5) File "C:\Python23\Lib\site-packages\numpy\testing\utils.py", line 152, in assert_almost_equal return assert_array_almost_equal(actual, desired, decimal, err_msg) File "C:\Python23\Lib\site-packages\numpy\testing\utils.py", line 222, in assert_array_almost_equal header='Arrays are not almost equal') File "C:\Python23\Lib\site-packages\numpy\testing\utils.py", line 207, in assert_array_compare assert cond, msg AssertionError: Arrays are not almost equal (mismatch 100.0%) x: array([ 4.957975 , 0.64690335]) y: array([ 4.95535625, 0.66666667]) ---------------------------------------------------------------------- Ran 1552 tests in 5.828s FAILED (failures=1) >>> Dr. John N. Carter. E-Mail : jnc at ecs.soton.ac.uk Building 1, Room 2005 http://www.ecs.soton.ac.uk/~jnc/ Information: Signals, Images, Systems Phone : +44 (0) 23 8059 2405 School of Electronics & Computer Science, Fax : +44 (0) 23 8059 4498 Southampton University, Hants, UK SO17 1BJ. From cimrman3 at ntc.zcu.cz Tue Jul 4 05:51:20 2006 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Tue, 04 Jul 2006 11:51:20 +0200 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <91cf711d0607031000l286da52fn30d98798057f12c5@mail.gmail.com> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> <44A8E4D3.5000603@ntc.zcu.cz> <91cf711d0607031000l286da52fn30d98798057f12c5@mail.gmail.com> Message-ID: <44AA3A18.9060300@ntc.zcu.cz> David Huard wrote: > Here is a quick benchmark between numpy's unique, unique1d and sasha's > unique: > > x = rand(100000)*100 > x = x.astype('i') > > %timeit unique(x) > 10 loops, best of 3: 525 ms per loop > > %timeit unique_sasha(x) > 100 loops, best of 3: 10.7 ms per loop > > timeit unique1d(x) > 100 loops, best of 3: 12.6 ms per loop > > So I wonder what is the added value of unique? > Could unique1d simply become unique ? It looks like unique1d and friends could use same facelifting with new numpy features like boolean indexing :) r. From theller at python.net Tue Jul 4 05:56:45 2006 From: theller at python.net (Thomas Heller) Date: Tue, 04 Jul 2006 11:56:45 +0200 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: <44A73B5D.60002@ieee.org> References: <44A73B5D.60002@ieee.org> Message-ID: Travis Oliphant schrieb: > I've been playing a bit with ctypes and realized that with a little > help, it could be made much easier to interface with NumPy arrays. > Thus, I added a ctypes attribute to the NumPy array. If ctypes is > installed, this attribute returns a "conversion" object otherwise an > AttributeError is raised. > > The ctypes-conversion object has attributes which return c_types aware > objects so that the information can be passed directly to c-code (as an > integer, the number of dimensions can already be passed using c-types). > > The information available and it's corresponding c_type is > > data - c_void_p > shape, strides - c_int * nd or c_long * nd or c_longlong * nd > depending on platform I've also played a little, and I think one important limitation in ctypes is that items in the argtypes list have to be ctypes types. If that limitation is removed (see the attached trivial patch) one can write a class that implements 'from_param' and accepts ctypes arrays as well as numpy arrays as argument in function calls (Maybe the _as_parameter_ stuff needs cleanup as well). The attached shape.py script implements this class, and has two examples. The 'from_param' method checks the correct shape and itemtype of the arrays that are passed as parameter. Thomas -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: shape.py URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: from_param.patch URL: From simon.anders at uibk.ac.at Tue Jul 4 06:03:07 2006 From: simon.anders at uibk.ac.at (Simon Anders) Date: Tue, 04 Jul 2006 12:03:07 +0200 Subject: [Numpy-discussion] A C++ library and the new array interface: the best approach? In-Reply-To: <44A99D07.5030706@colorado.edu> References: <44A99D07.5030706@colorado.edu> Message-ID: <44AA3CDB.8060502@uibk.ac.at> Hi Fernando, Fernando Perez schrieb: [...] > So I'd like to know if SWIG is really the best way out in this particular case > (and any advice on taking advantage of the array interface via SWIG would be > appreciated), or if ctypes or pyrex could be used here. I'm quite happy using > pyrex in other contexts, but I know it doesn't directly support C++. However, > since I have access to all the code, perhaps a pure C layer could be used to > bridge the C++/pyrex gap. Or given the recent praise for ctypes, perhaps that > can be an option? I'm not so sure either, whether SWIG is the way to go. If, however, you decide to give it a try, you could try a little tool I am currently working on. It is basically a SWIG typemap definition which allows for easy use of numpy arrays from C++. In brief, I have defined a C++ template class around numpy's own PyArrayObject structure which allows for convenient access of the array data using C++-like techniques and takes care of type checking and the like. As it adds only non-virtual methods and no fields, the objects are memory-wise identical to numpy's own array structure and can hence be passed between C++ and Python without any copying of data or metadata. The original idea behind was a bit the opposite of your problem. I had program for a numerical computation, written in Python, and wanted to rewrite the most performance-critical parts in C++. With my SWIG typemap, I can now write the C++ part using my template class and wrap it such that these objects appear to Python as numpy array objects. For your problem, it might be a way to write a kind of casting functions that takes one of your tensor objects and transforms it into my numpy objects. This should be possible with only copying metadata (i.e. fields containing dimensions and the like) but with leaving the actual data in place. My stuff is still a bit work in progress, and so I don't want to post it here yet, but it may help you to not have to start from scratch, as the typemap code would help you to get started and might be easy to adjust to your needs. So, if you are interested, send me a mail, and I would appreciate any comments from you on how to make my tool into seomthing really reusable. Regards, Simon -- +--- | Simon Anders, Dipl. Phys. | Institut fuer Theoretische Physik, Universitaet Innsbruck, Austria | Tel. +43-512-507-6207, Fax -2919 | preferred (permanent) e-mail: sanders at fs.tum.de From theller at python.net Tue Jul 4 06:00:42 2006 From: theller at python.net (Thomas Heller) Date: Tue, 04 Jul 2006 12:00:42 +0200 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: References: <44A73B5D.60002@ieee.org> Message-ID: Thomas Heller schrieb: > I've also played a little, and I think one important limitation in ctypes > is that items in the argtypes list have to be ctypes types. Thi swas misleading: I mean that this limitation should probably be removed, because it prevents a lot of things one could do. Thomas From jan_braun at gmx.net Tue Jul 4 06:10:18 2006 From: jan_braun at gmx.net (Jan-Matthias Braun) Date: Tue, 4 Jul 2006 12:10:18 +0200 Subject: [Numpy-discussion] KeyError with float96 in linalg.py: polyfit Message-ID: <200607041210.20480.jan_braun@gmx.net> Hi all, I'm testing some computations with float96 at the moment and right now I have problems with polyfit raising a KeyError for the keycode 'g', which is floatxx with xx>64. I am getting a KeyError using polyfit on some float96 values. The used Routines seem to know nothing about this type. My main question is: have I missed something? Shouldn't this type be used? Below is a more detailed descripton. Thanks in advance, Jan ---------------------------------------------------------------------------- In file numpy/linalg/linalg.py, the following definitions at lines 26ff seem to be the offending ones: # Helper routines _array_kind = {'i':0, 'l': 0, 'q': 0, 'f': 0, 'd': 0, 'F': 1, 'D': 1} _array_precision = {'i': 1, 'l': 1, 'q': 1, 'f': 0, 'd': 1, 'F': 0, 'D': 1} _array_type = [['f', 'd'], ['F', 'D']] Here the new typecodes are missing. I tried # Helper routines _array_kind = {'i':0, 'l': 0, 'q': 0, 'f': 0, 'd': 0, 'g': '0', 'F': 1, 'D':1, 'G':1} _array_precision = {'i': 1, 'l': 1, 'q': 1, 'f': 0, 'd': 1, 'g': 1, 'F': 0, 'D': 1, 'G': 1} _array_type = [['f', 'd', 'g'], ['F', 'D', 'G']] which gets me a step further to a TypeError: File "lib/python2.3/site-packages/numpy/linalg/linalg.py", line 454, in lstsq bstar[:b.shape[0],:n_rhs] = b.copy() TypeError: array cannot be safely cast to required type (Question: Why only one typecode for a type which varies in bitlength on different platforms? On Opteron CPU's I've seen float128 with 'g'?) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From fullung at gmail.com Tue Jul 4 06:11:53 2006 From: fullung at gmail.com (Albert Strasheim) Date: Tue, 4 Jul 2006 12:11:53 +0200 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: Message-ID: <00fe01c69f52$4614c9e0$0a84a8c0@dsp.sun.ac.za> Hey Thomas Thomas Heller wrote: > Thomas Heller schrieb: > > I've also played a little, and I think one important limitation in > ctypes > > is that items in the argtypes list have to be ctypes types. > > Thi swas misleading: I mean that this limitation should probably be > removed, because it prevents a lot of things one could do. What's your thinking on getting these changes made to ctypes and on ctypes' future development in general? Presumably you can't change it too much with the Python 2.5 release coming up, but it would be a shame if we had to wait until Python 2.6 to get the changes you suggested (and other goodies, like the array interface). Regards, Albert From simon at arrowtheory.com Tue Jul 4 07:47:58 2006 From: simon at arrowtheory.com (Simon Burton) Date: Tue, 4 Jul 2006 13:47:58 +0200 Subject: [Numpy-discussion] A C++ library and the new array interface: the best approach? In-Reply-To: <44A99D07.5030706@colorado.edu> References: <44A99D07.5030706@colorado.edu> Message-ID: <20060704134758.0dae19d8.simon@arrowtheory.com> On Mon, 03 Jul 2006 16:41:11 -0600 Fernando Perez wrote: > > So I'd like to know if SWIG is really the best way out in this particular case > (and any advice on taking advantage of the array interface via SWIG would be > appreciated), or if ctypes or pyrex could be used here. I'm quite happy using > pyrex in other contexts, but I know it doesn't directly support C++. However, > since I have access to all the code, perhaps a pure C layer could be used to > bridge the C++/pyrex gap. Or given the recent praise for ctypes, perhaps that > can be an option? Pyrex can handle some C++. Eg. make objects, and call methods. You will need to search the pyrex email archives for all the tricks to get this to work. Simon. From theller at python.net Tue Jul 4 07:49:42 2006 From: theller at python.net (Thomas Heller) Date: Tue, 04 Jul 2006 13:49:42 +0200 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: <00fe01c69f52$4614c9e0$0a84a8c0@dsp.sun.ac.za> References: <00fe01c69f52$4614c9e0$0a84a8c0@dsp.sun.ac.za> Message-ID: Albert Strasheim schrieb: > Hey Thomas > > Thomas Heller wrote: >> Thomas Heller schrieb: >> > I've also played a little, and I think one important limitation in >> ctypes >> > is that items in the argtypes list have to be ctypes types. >> >> Thi swas misleading: I mean that this limitation should probably be >> removed, because it prevents a lot of things one could do. > > What's your thinking on getting these changes made to ctypes and on ctypes' > future development in general? > > Presumably you can't change it too much with the Python 2.5 release coming > up, but it would be a shame if we had to wait until Python 2.6 to get the > changes you suggested (and other goodies, like the array interface). I have asked on python-dev, let's wait for the answer. I hope that at least the limitation that I mentioned can be removed in Python 2.5. The goal of my post was to show that (without this restriction) a lot can already be done in Python, of course it would be better if this could be implemented in C and integrated in ctypes. For the numpy/ctypes inegration I'm not absolutely sure what would be needed most: Is there a need to convert between ctypes and numpy arrays? If numpy arrays can be passed to ctypes foreign functions maybe there is no need at all for the conversion. We could probably even live with helper code like that I posted outside of ctypes... Thomas From stephenemslie at gmail.com Tue Jul 4 08:00:29 2006 From: stephenemslie at gmail.com (stephen emslie) Date: Tue, 4 Jul 2006 13:00:29 +0100 Subject: [Numpy-discussion] iterate along a ray: linear algebra? In-Reply-To: References: <51f97e530606300713w1c167cf3j10c36d24f87326cf@mail.gmail.com> Message-ID: <51f97e530607040500y5e4becja4b8d86c019129cc@mail.gmail.com> I've found some matlab code that seems to do the same sort of thing. Interestingly enough it just uses trigonomotry to do find the x,y positions in the matrix that correspond to the ray at a particular angle. I had origionally discarded this idea because I thought there must be a more efficient way to do it, but perhaps not. I still have a lot to learn about numpy :) Stephen (sorry for the double post to you Torgil) On 7/3/06, Torgil Svensson wrote: > > I've done something similar a few years ago (numarray,numeric). I > started roughly at the middle and did 64 points from a reference point > (xc,yc). This point together with a point at the edge of the image > (xp,yp) also defined a reference angle (a0). (ysize,xsize) is the > shape of the intensity image. > > I used the following code to calculate points of interest: > > na=64 > xr,yr=xsize-xc,ysize-yc > a0=arctan2(yp-yr,xp-xc) > if a0<0: a0+=2*pi > ac=arctan2([yr,yr,-yc,-yc],[xr,-xc,-xc,xr]) > if numarray: > ac[ac<0]+=2*pi > else: > ac=choose(ac<0,(ac,ac+2*pi)) > a1,a2,a3,a4=ac > rmaxfn={ > 0: lambda a: a<=a1 and xr/cos(a-0.0*pi) or yr/cos(0.5*pi-a), > 1: lambda a: a<=a2 and yr/cos(a-0.5*pi) or xc/cos(1.0*pi-a), > 2: lambda a: a<=a3 and xc/cos(a-1.0*pi) or yc/cos(1.5*pi-a), > 3: lambda a: a<=a4 and yc/cos(a-1.5*pi) or xr/cos(2.0*pi-a) > } > angles=arange(a0,a0+2*pi,2*pi/na) > if numarray: > angles[angles>=2*pi]-=2*pi > else: > angles=choose(angles>=2*pi,(angles,angles-2*pi)) > nr=int(ceil(sqrt(max(yc,yr)**2+max(xc,xr)**2))) > crmax=array([int(floor(rmaxfn[floor(a*2/pi)](a))) for a in angles]) > cr=outerproduct(ones(na),arange(float(nr))) > ca=outerproduct(angles,ones(nr)) > x=cr*cos(ca)+xc > y=cr*sin(ca)+yc > > After this I did cubic spline interpolation in the image with these > points and did something useful. I don't know how relevant this is to > you and it doesn't use the linear algebra package but it might give > you some hint. > > If you find out a nifty way to do your rays please post on this thread. > > Sidenote -- Watch my explicit float argument to arange and even > putting in pi there in one case. There's a discussion on this list > that floats in arange are troublesome > > > On 6/30/06, stephen emslie wrote: > > I am in the process of implementing an image processing algorithm that > > requires following rays extending outwards from a starting point and > > calculating the intensity derivative at each point. The idea is to find > the > > point where the difference in intensity goes beyond a particular > threshold. > > > > Specifically I'm examining an image of an eye to find the pupil, and the > > edge of the pupil is a sharp change in intensity. > > > > How does one iterate along a line in a 2d matrix, and is there a better > way > > to do this? Is this a problem that linear algebra can help with? > > > > Thanks > > Stephen Emslie > > > > Using Tomcat but need to do more? Need to support web services, > security? > > Get stuff done quickly with pre-integrated technology to make your job > > easier > > Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fullung at gmail.com Tue Jul 4 08:58:05 2006 From: fullung at gmail.com (Albert Strasheim) Date: Tue, 4 Jul 2006 14:58:05 +0200 Subject: [Numpy-discussion] Ctypes support in NumPy In-Reply-To: References: <00fe01c69f52$4614c9e0$0a84a8c0@dsp.sun.ac.za> Message-ID: <20060704125805.GA7962@dogbert.sdsl.sun.ac.za> Hello all On Tue, 04 Jul 2006, Thomas Heller wrote: > Albert Strasheim schrieb: > > Hey Thomas > > > > Thomas Heller wrote: > >> Thomas Heller schrieb: > >> > I've also played a little, and I think one important limitation in > >> ctypes > >> > is that items in the argtypes list have to be ctypes types. > >> > >> Thi swas misleading: I mean that this limitation should probably be > >> removed, because it prevents a lot of things one could do. > > > > What's your thinking on getting these changes made to ctypes and on ctypes' > > future development in general? > > > > Presumably you can't change it too much with the Python 2.5 release coming > > up, but it would be a shame if we had to wait until Python 2.6 to get the > > changes you suggested (and other goodies, like the array interface). > > I have asked on python-dev, let's wait for the answer. > I hope that at least the limitation that I mentioned can be removed in Python 2.5. Sounds great. > The goal of my post was to show that (without this restriction) a lot can > already be done in Python, of course it would be better if this could be > implemented in C and integrated in ctypes. > > For the numpy/ctypes inegration I'm not absolutely sure what would be needed most: > > Is there a need to convert between ctypes and numpy arrays? If numpy arrays can > be passed to ctypes foreign functions maybe there is no need at all for the conversion. > We could probably even live with helper code like that I posted outside of ctypes... I think there are basically two ways for a C library to work with regards to memory allocation: 1. let the user allocate the array/struct/whatever and pass a pointer to the library to manipulate 2. let the library allocate the array/struct/whatever, manipulate it and return the pointer to the user I think the first case is pretty much covered. Where in the past you would create the array or struct on the stack or allocate it on the heap with malloc, you now create a ctypes Structure or a NumPy array and pass that to the C function. In the second case, one would want to wrap a NumPy array around the ctype so that you can manipulate the data returned by the library. I don't know if this second scenario is very common -- hopefully not. If not, then having ctypes implement the array interface isn't too critical, since you wouldn't typically need to make a NumPy array from existing data. What do you think? Regards, Albert From steffen.loeck at gmx.de Tue Jul 4 09:19:24 2006 From: steffen.loeck at gmx.de (Steffen Loeck) Date: Tue, 4 Jul 2006 15:19:24 +0200 Subject: [Numpy-discussion] Speed degression? Message-ID: <200607041519.24636.steffen.loeck@gmx.de> Hi all, i made some speed tests using the sin-function and the %-operation to compare Numeric, numpy 0.9.8 and numpy 0.9.9.2732. As result the latest numpy version seems to be very slow in comparison to the two other candidates. Results (in usec per loop): sin-array mod-array Numeric 134 18 numpy 0.9.8 97 55 numpy 0.9.9.2732 204 316 numpy 0.9.8 + math 38 numpy 0.9.9.2732 + math 161 Numeric + math 23 The used scripts can be found at the end. Can anyone verify my results and explain the observed speed degression? Thanks, Steffen sin-scripts: /usr/lib/python2.3/timeit.py -s "from Numeric import sin,zeros,arange; x=zeros(10, 'd'); x[0]=0.1" "for i in arange(9): x[i+1]=sin(x[i])" /usr/lib/python2.3/timeit.py -s "from numpy import sin,zeros,arange; x=zeros(10, 'd'); x[0]=0.1" "for i in arange(9): x[i+1]=sin(x[i])" /usr/lib/python2.3/timeit.py -s "from math import sin; from numpy import zeros,arange; x=zeros(10, 'd');x[0]=0.1" "for i in arange(9): x[i+1]=sin(x[i])" /usr/lib/python2.3/timeit.py -s "from math import sin; from Numeric import zeros,arange; x=zeros(10, 'd'); x[0]=0.1" "for i in arange(9): x[i+1]=sin(x[i] )" %-scripts /usr/lib/python2.3/timeit.py -s "from Numeric import zeros,arange; x=zeros(10, 'd'); x[0]=0.1" "for i in arange(9): x[i+1]=(x[i]+1.1)%(1.0)" /usr/lib/python2.3/timeit.py -s "from numpy import zeros,arange; x=zeros(10, 'd'); x[0]=0.1" "for i in arange(9): x[i+1]=(x[i]+1.1)%(1.0)" From david.huard at gmail.com Tue Jul 4 09:28:02 2006 From: david.huard at gmail.com (David Huard) Date: Tue, 4 Jul 2006 09:28:02 -0400 Subject: [Numpy-discussion] Error building numpy from svn, 2nd attempt In-Reply-To: <7.0.1.0.0.20060704100002.01f8fc38@ecs.soton.ac.uk> References: <7.0.1.0.0.20060704100002.01f8fc38@ecs.soton.ac.uk> Message-ID: <91cf711d0607040628m6914bd9cj60957595d1da0ee2@mail.gmail.com> Hi John, Here is a patch to fix the first error in test_twodim_base.py. I'm sorry I can't help you with the rest. David 2006/7/4, John Carter : > > Hi, > > As is the way posting to a news group stirs the brain cell into > activity and the problem is solved. or rather shifted. > > I've downloaded the candidate version of mingw32 and using that to > build numpy/scipy works, or rather it builds the extensions for Python 2.3 > > I believe that there are still problems as both numpy and scipy fail > their tests. I'm also dubious as to whether they are finding Altas and > Lapack. > > Configuration: > > Windows XP SP2 > Cygwin = gcc version 3.4.4 (cygming special) (gdc 0.12, using dmd 0.125) > MinGW = gcc version 3.4.5 (mingw special) > Python = Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit > (Intel)] on win32 > > scipy, numpy, Atlas, Lapack all downloaded within the last 2 days > > > I've cut and pasted the error messages from the tests below. > > Any help gratefully received. > > John > > > ===== > numpy > ===== > D:.\numerical> python > Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on > win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > >>> numpy.test() > Found 5 tests for numpy.distutils.misc_util > Found 3 tests for numpy.lib.getlimits > Found 30 tests for numpy.core.numerictypes > Found 13 tests for numpy.core.umath > Found 3 tests for numpy.core.scalarmath > Found 8 tests for numpy.lib.arraysetops > Found 42 tests for numpy.lib.type_check > Found 101 tests for numpy.core.multiarray > Found 36 tests for numpy.core.ma > Found 10 tests for numpy.lib.twodim_base > Found 10 tests for numpy.core.defmatrix > Found 1 tests for numpy.lib.ufunclike > Found 38 tests for numpy.lib.function_base > Found 3 tests for numpy.dft.helper > Found 1 tests for numpy.lib.polynomial > Found 7 tests for numpy.core.records > Found 26 tests for numpy.core.numeric > Found 4 tests for numpy.lib.index_tricks > Found 46 tests for numpy.lib.shape_base > Found 0 tests for __main__ > > ........................................................................................................ > > ........................................................................................................ > > .........E.............................................................................................. > ........................... > ====================================================================== > ERROR: check_simple (numpy.lib.tests.test_twodim_base.test_histogram2d) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "C:\Python23\Lib\site-packages\numpy\lib\tests\test_twodim_base.py", > line 137, in check_simple > np.random.seed(1) > File "mtrand.pyx", line 311, in mtrand.RandomState.seed > SystemError: C:\sf\python\dist\src-maint23\Objects\longobject.c:240: > bad argument to internal function > > ---------------------------------------------------------------------- > Ran 387 tests in 1.391s > > FAILED (errors=1) > > >>> > > ===== > scipy > ===== > D:.\numerical> python > Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on > win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> import scipy > >>> scipy.test() > Overwriting lib= 'C:\PYTHON23\lib\site-packages\scipy\lib\__init__.pyc'> from > C:\PYTHON23\lib\si > te-packages\scipy\lib\__init__.pyc (was 'C:\PYTHON23\lib\site-packages\numpy\lib\__init__.pyc'> > from C:\PYTHON23\lib\site-packages\numpy\lib\__init__.pyc) > Overwriting fft= from scipy.fftpack.basic > (was from numpy.dft.f > ftpack) > Overwriting ifft= from > scipy.fftpack.basic (was from numpy.df > t.fftpack) > Found 4 tests for scipy.io.array_import > Found 128 tests for scipy.linalg.fblas > Found 397 tests for scipy.ndimage > Found 10 tests for scipy.integrate.quadpack > Found 97 tests for scipy.stats.stats > Found 47 tests for scipy.linalg.decomp > Found 2 tests for scipy.integrate.quadrature > Found 95 tests for scipy.sparse.sparse > Found 20 tests for scipy.fftpack.pseudo_diffs > Found 6 tests for scipy.optimize.optimize > Found 5 tests for scipy.interpolate.fitpack > Found 1 tests for scipy.interpolate > Found 12 tests for scipy.io.mmio > Found 10 tests for scipy.stats.morestats > Found 4 tests for scipy.linalg.lapack > Found 18 tests for scipy.fftpack.basic > Found 4 tests for scipy.linsolve.umfpack > Found 4 tests for scipy.optimize.zeros > Found 41 tests for scipy.linalg.basic > Found 2 tests for scipy.maxentropy.maxentropy > Found 358 tests for scipy.special.basic > Found 128 tests for scipy.lib.blas.fblas > Found 7 tests for scipy.linalg.matfuncs > > **************************************************************** > WARNING: clapack module is empty > ----------- > See scipy/INSTALL.txt for troubleshooting. > Notes: > * If atlas library is not found by numpy/distutils/system_info.py, > then scipy uses flapack instead of clapack. > **************************************************************** > > Found 42 tests for scipy.lib.lapack > Found 1 tests for scipy.optimize.cobyla > Found 16 tests for scipy.lib.blas > Found 1 tests for scipy.integrate > Found 14 tests for scipy.linalg.blas > Found 70 tests for scipy.stats.distributions > Found 4 tests for scipy.fftpack.helper > Found 4 tests for scipy.signal.signaltools > Found 0 tests for __main__ > > Don't worry about a warning regarding the number of bytes read. > Warning: 1000000 bytes requested, 20 bytes read. > .......caxpy:n=4 > ..caxpy:n=3 > ....ccopy:n=4 > ..ccopy:n=3 > .............cscal:n=4 > ....cswap:n=4 > ..cswap:n=3 > .....daxpy:n=4 > ..daxpy:n=3 > ....dcopy:n=4 > ..dcopy:n=3 > .............dscal:n=4 > ....dswap:n=4 > ..dswap:n=3 > .....saxpy:n=4 > ..saxpy:n=3 > ....scopy:n=4 > ..scopy:n=3 > .............sscal:n=4 > ....sswap:n=4 > ..sswap:n=3 > .....zaxpy:n=4 > ..zaxpy:n=3 > ....zcopy:n=4 > ..zcopy:n=3 > .............zscal:n=4 > ....zswap:n=4 > ..zswap:n=3 > > ........................................................................................................................ > > ........................................................................................................................ > > ........................................................................................................................ > > ........................................................................................................................ > > .........................................................................Took > 13 points. > ..........Resizing... 16 17 24 > Resizing... 20 7 35 > Resizing... 23 7 47 > Resizing... 24 25 58 > Resizing... 28 7 68 > Resizing... 28 27 73 > .....Use minimum degree ordering on A'+A. > ........................Use minimum degree ordering on A'+A. > ...................Resizing... 16 17 24 > Resizing... 20 7 35 > Resizing... 23 7 47 > Resizing... 24 25 58 > Resizing... 28 7 68 > Resizing... 28 27 73 > .....Use minimum degree ordering on A'+A. > .................Resizing... 16 17 24 > Resizing... 20 7 35 > Resizing... 23 7 47 > Resizing... 24 25 58 > Resizing... 28 7 68 > Resizing... 28 27 73 > .....Use minimum degree ordering on A'+A. > > ......................................C:\PYTHON23\lib\site-packages\scipy\interpolate\fitpack2.py:410: > UserWarning: > The coefficients of the spline returned have been computed as the > minimal norm least-squares solution of a (numerically) rank deficient > system (deficiency=7). If deficiency is large, the results may be > inaccurate. Deficiency may strongly depend on the value of eps. > warnings.warn(message) > ....................Ties preclude use of exact statistic. > ..Ties preclude use of exact statistic. > ........ > **************************************************************** > WARNING: clapack module is empty > ----------- > See scipy/INSTALL.txt for troubleshooting. > Notes: > * If atlas library is not found by numpy/distutils/system_info.py, > then scipy uses flapack instead of clapack. > **************************************************************** > > ....................data-ftype: z compared to data D > Calling _superlu.zgssv > Use minimum degree ordering on A'+A. > .data-ftype: c compared to data F > Calling _superlu.cgssv > Use minimum degree ordering on A'+A. > .data-ftype: d compared to data d > Calling _superlu.dgssv > Use minimum degree ordering on A'+A. > .data-ftype: s compared to data f > Calling _superlu.sgssv > Use minimum degree ordering on A'+A. > > ........................................................................................................................ > > ........................................................................................................................ > ....................................Gegenbauer, a = 2.7337413228 > > ........................................................................................................................ > .............caxpy:n=4 > ..caxpy:n=3 > ....ccopy:n=4 > ..ccopy:n=3 > .............cscal:n=4 > ....cswap:n=4 > ..cswap:n=3 > .....daxpy:n=4 > ..daxpy:n=3 > ....dcopy:n=4 > ..dcopy:n=3 > .............dscal:n=4 > ....dswap:n=4 > ..dswap:n=3 > .....saxpy:n=4 > ..saxpy:n=3 > ....scopy:n=4 > ..scopy:n=3 > .............sscal:n=4 > ....sswap:n=4 > ..sswap:n=3 > .....zaxpy:n=4 > ..zaxpy:n=3 > ....zcopy:n=4 > ..zcopy:n=3 > .............zscal:n=4 > ....zswap:n=4 > ..zswap:n=3 > ...Result may be inaccurate, approximate err = 4.1928136851e-009 > ...Result may be inaccurate, approximate err = 7.27595761418e-012 > .............................................F................Residual: > 1.05006950319e-007 > . > **************************************************************** > WARNING: cblas module is empty > ----------- > See scipy/INSTALL.txt for troubleshooting. > Notes: > * If atlas library is not found by numpy/distutils/system_info.py, > then scipy uses fblas instead of cblas. > **************************************************************** > > > ............................................................................................ > ====================================================================== > FAIL: check_simple (scipy.optimize.tests.test_cobyla.test_cobyla) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "C:\Python23\Lib\site-packages\scipy\optimize\tests\test_cobyla.py", > line 20, in check_simple > assert_almost_equal(x, [x0, x1], decimal=5) > File "C:\Python23\Lib\site-packages\numpy\testing\utils.py", line > 152, in assert_almost_equal > return assert_array_almost_equal(actual, desired, decimal, err_msg) > File "C:\Python23\Lib\site-packages\numpy\testing\utils.py", line > 222, in assert_array_almost_equal > header='Arrays are not almost equal') > File "C:\Python23\Lib\site-packages\numpy\testing\utils.py", line > 207, in assert_array_compare > assert cond, msg > AssertionError: > Arrays are not almost equal > > (mismatch 100.0%) > x: array([ 4.957975 , 0.64690335]) > y: array([ 4.95535625, 0.66666667]) > > ---------------------------------------------------------------------- > Ran 1552 tests in 5.828s > > FAILED (failures=1) > > >>> > > > > > > Dr. John N. Carter. E-Mail : > jnc at ecs.soton.ac.uk > Building 1, Room > 2005 http://www.ecs.soton.ac.uk/~jnc/ > Information: Signals, Images, Systems Phone : +44 (0) 23 8059 > 2405 > School of Electronics & Computer Science, Fax : +44 (0) 23 8059 > 4498 > Southampton University, Hants, UK SO17 1BJ. > > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: histogram2d_test.patch Type: text/x-patch Size: 675 bytes Desc: not available URL: From jensj at fysik.dtu.dk Tue Jul 4 10:25:42 2006 From: jensj at fysik.dtu.dk (=?UTF-8?B?SmVucyBKw7hyZ2VuIE1vcnRlbnNlbg==?=) Date: Tue, 04 Jul 2006 16:25:42 +0200 Subject: [Numpy-discussion] Why is my array not contiguous? Message-ID: <44AA7A66.9070809@servfys.fysik.dtu.dk> Hi! With numpy-0.9.9.2726, I do this: >>> x = arange(4) >>> y = x[newaxis, :] I would expect both arrays to be contiguous: >>> x.flags.contiguous, y.flags.contiguous (True, False) Shouldn't y be contiguous? Maybe it's because of the strange strides: >>> y.strides (0, 4) >>> y.strides = (16, 4) >>> y.flags.contiguous True Jens J?rgen From bart.vandereycken at cs.kuleuven.be Tue Jul 4 11:16:16 2006 From: bart.vandereycken at cs.kuleuven.be (Bart Vandereycken) Date: Tue, 04 Jul 2006 17:16:16 +0200 Subject: [Numpy-discussion] Dot as a method Message-ID: Hi all, reading the thread "Ransom proposals" I was wondering why there isn't a ndarray.dot() method? There is already a scipy.sparse.dot() so this would fit nicely in the whole idea of polymorphism. Bart From cookedm at physics.mcmaster.ca Tue Jul 4 16:09:38 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Tue, 4 Jul 2006 16:09:38 -0400 Subject: [Numpy-discussion] KeyError with float96 in linalg.py: polyfit In-Reply-To: <200607041210.20480.jan_braun@gmx.net> References: <200607041210.20480.jan_braun@gmx.net> Message-ID: <20060704160938.37a2f70a@arbutus.physics.mcmaster.ca> On Tue, 4 Jul 2006 12:10:18 +0200 Jan-Matthias Braun wrote: > Hi all, > > I'm testing some computations with float96 at the moment and right now I > have problems with polyfit raising a KeyError for the keycode 'g', which is > floatxx with xx>64. Use longdouble instead of float96; it'll make your code portable. > I am getting a KeyError using polyfit on some float96 values. The used > Routines seem to know nothing about this type. > > My main question is: have I missed something? Shouldn't this type be used? > Below is a more detailed descripton. polyfit uses LAPACK to do least-squares for the fit, and LAPACK doesn't handle any real types besides single- and double-precision. And Numpy (as a design decision) doesn't like casting to a lower precision. > In file numpy/linalg/linalg.py, the following definitions at lines 26ff > seem to be the offending ones: > # Helper routines > _array_kind = {'i':0, 'l': 0, 'q': 0, 'f': 0, 'd': 0, 'F': 1, 'D': > 1} _array_precision = {'i': 1, 'l': 1, 'q': 1, 'f': 0, 'd': 1, 'F': 0, 'D': > 1} _array_type = [['f', 'd'], ['F', 'D']] > > Here the new typecodes are missing. I tried > # Helper routines > _array_kind = {'i':0, 'l': 0, 'q': 0, 'f': 0, 'd': 0, 'g': '0', > 'F': 1, 'D':1, 'G':1} > _array_precision = {'i': 1, 'l': 1, 'q': 1, 'f': 0, 'd': 1, 'g': 1, > 'F': 0, 'D': 1, 'G': 1} > _array_type = [['f', 'd', 'g'], ['F', 'D', 'G']] > > which gets me a step further to a TypeError: > > File "lib/python2.3/site-packages/numpy/linalg/linalg.py", line 454, in > lstsq > bstar[:b.shape[0],:n_rhs] = b.copy() > TypeError: array cannot be safely cast to required type That would be a better error. I'm going to leave it like this for now, though. Instead, the linalg module will be converted to use dtypes. > (Question: Why only one typecode for a type which varies in bitlength on > different platforms? On Opteron CPU's I've seen float128 with 'g'?) Typecodes are deprecated (they date back to Numeric), so we're not bothering to add new ones. The equivalent to 'g' is the dtype longdouble. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From josh8912 at yahoo.com Tue Jul 4 17:23:20 2006 From: josh8912 at yahoo.com (JJ) Date: Tue, 4 Jul 2006 14:23:20 -0700 (PDT) Subject: [Numpy-discussion] arguments for stats.chi2 Message-ID: <20060704212320.30425.qmail@web51712.mail.yahoo.com> Hello. I have a very simple question. I would like to generate a number of random variables from the chi2 distribution. If I wanted these for the normal distribution, the code could be stats.norm.rvs(size=5,loc=100,scale=1). But stats.chi2.rvs(size=5,df=10,loc=0,scale=1) or stats.chi2.rvs(df=10,loc=0,scale=1) or stats.chi2.rvs(df=10)does not work. Can anyone tell me what the proper syntax would be for this? Thanks JJ __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From pfdubois at gmail.com Tue Jul 4 17:54:32 2006 From: pfdubois at gmail.com (Paul Dubois) Date: Tue, 4 Jul 2006 14:54:32 -0700 Subject: [Numpy-discussion] MA bug or feature? In-Reply-To: <200607040240.00075.pgmdevlist@mailcan.com> References: <16761e100606210146q7683c94bu5bd2699caa6b95cf@mail.gmail.com> <16761e100607032144w65013936jbbf9772f429f4be9@mail.gmail.com> <200607040240.00075.pgmdevlist@mailcan.com> Message-ID: Some things to note: The mask is copy-on-write. Don't mess with that. You can't just poke values into an existing mask, it may be shared with other arrays. I do not agree that there is any 'inconsistency'. It may be someone's concept of the class that if there is a mask then at least one value is on, but that was not my design. I believe if you try your ideas you'll find it slows other people down, if not you. Perhaps with all of Travis' new machinery, subclassing works. It didn't used to, and I haven't kept up. On 7/3/06, Pierre GM wrote: > > Michael, > I wonder whether the Mask class you suggest is not a bit overkill. There > should be enough tools in the existing MA module to do what we want. And I > don't wanna think about compatibility the number of changes in the MA code > that'd be required (but I'm lazy)... > > For the sake of consistency and optimization, I still think it could be > easier > (and cleaner) to make `nomask` the default for a MaskedArray without > masked > values. That could for example be implemented by forcing `nomask` at the > creation of the MaskedArray with an extra > `if mask and not mask.any(): mask=nomask`, or by using Paul's > make_mask( flag=1) trick. > > Masking some specific values could still be done when mask is nomask with > an > intermediary MA.getmaskarray() step. > > On a side note, modifying an existing mask is a delicate matter. > Everything's > OK if you use masks as a way to hide existing data, it's more complex when > initially you have some holes in your dataset... > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Wed Jul 5 00:03:34 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 5 Jul 2006 13:03:34 +0900 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again Message-ID: Just wanted to make one last effort get a .T attribute for arrays, so that you can flip axes with a simple "a.T" instead of "a.transpose()", as with numpy matrix objects. If I recall, the main objection raised before was that there are lots of ways to transpose n-dimensional data. Fine, but the fact is that 2D arrays are pretty darn common, and so are a special case worth optimizing for. Furthermore transpose() won't go away if you do need to do some specific kind of axes swapping other than the default, so noone is really going to be harmed by adding it. I propose to make .T a synonym for .swapaxes(-2,-1) {*}, i.e. the last two axes are interchanged. This should also make it useful in many N-d array cases (whereas the default of .transpose() -- to completely reverse the order of all the axes -- is seldom what you want). Part of the thinking is that when you print an N-d array it's the last two dimensions that get printed like 2-d matrices separated by blank likes. You can think of it as some number of stacks of 2-d matrices. So this .T would just transpose those 2-d matrices in the printout. Those are the parts that are generally most contiguous in memory also, so it makes sense for 2-d matrix bits to be stored in those last two dimensions. Then, if there is a .T, it makes sense to also have .H which would basically be equivalent to .T.conjugate(). Finally, the matrix class has .A to get the underlying array -- it would also be nice to have a .M on array as a shortcut for asmatrix(). This one would be very handy for matrix users, I think, but I could go either way on that, having abandoned matrix myself. Ex: ones([4,4]).M Other possibilities: - Make .T a function, so that you can pass it the same info as .transpose(). Then the shortcut becomes a.T(), which isn't as nice, and isn't consistent with matrix's .T any more. - Just make .T raise an error for ndim>2. But I don't really see any benefit in making it an error as opposed to defining a reasonable default behavior. - Make .T on a 1-dim array return a 2-dim Nx1 array. (My default suggestion is to just leave it alone if ndim < 2, an exception would be another possiblility). Would make an easy way to create column vectors from arrays, but I can think of nothing else in Numpy that acts that way. This is not a 1.0 must have, as it introduces no backward compatibility issues. But it would be trivial to add if the will is there. {*} except that negative axes for swapaxes doesn't seem work currently, so instead it would need to be something like: a.transpose( a.shape[:-2] + (a.shape[-1],a.shape[-2]) ) with a check for "if ndim > 1", of course. --Bill -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Wed Jul 5 00:58:45 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 5 Jul 2006 13:58:45 +0900 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: Message-ID: Slight correction. {*} except that negative axes for swapaxes doesn't seem work currently, so > instead it would need to be something like: > a.transpose( a.shape[:-2] + (a.shape[-1],a.shape[-2]) ) > with a check for "if ndim > 1", of course. > Apparently a.swapaxes(-2,-1) does work, and it does exactly what I am suggesting, including leaving zero-d and 1-d arrays alone. Not sure why I thought it wasn't working. So in short my proposal is to: -- make a.T a property of array that returns a.swapaxes(-2,-1), -- make a.H a property of array that returns a.conjugate().swapaxes(-2,-1) and maybe -- make a.M a property of array that returns numpy.asmatrix(a) --Bill -------------- next part -------------- An HTML attachment was scrubbed... URL: From 67753110 at 163.com Wed Jul 5 01:42:19 2006 From: 67753110 at 163.com (=?GB2312?B?1tC5+r3wyvS5pNK1zfg=?=) Date: Wed, 5 Jul 2006 13:42:19 +0800 Subject: [Numpy-discussion] =?GB2312?B?xOPSqsHLveK1xKOsvs3U2rTL09C96cnco6E=?= Message-ID: numpy-discussion:??! ? ????????www.cmpi.cn,???? ? ????????? ?! ?????????????? ??????? ?????????????? 67753110 at 163.com ?????????????? 2006-07-05 From wbaxter at gmail.com Wed Jul 5 02:41:19 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 5 Jul 2006 15:41:19 +0900 Subject: [Numpy-discussion] Simple 3D drawing options? Message-ID: I tried to get MayaVi and VTK working under Win32/MSVC.Net a while back failed miserably. Is there some simple, out-of-the-box, precompiled, no-brains-required solution for creating 3D plots? Preferably one that doesn't require compiling anything. --Bill -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Wed Jul 5 03:17:16 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 5 Jul 2006 16:17:16 +0900 Subject: [Numpy-discussion] Simple 3D drawing options? In-Reply-To: <63eb7fa90607042357j4053b114yda0227c91f38b99c@mail.gmail.com> References: <63eb7fa90607042357j4053b114yda0227c91f38b99c@mail.gmail.com> Message-ID: I like the idea of enthon, but I hadn't given it much thought because it seems to lag so far behind in versions. Python 2.3, Numpy 0.9.6, etc. But maybe that's not as big a deal as I've been thinking? Two questions: * Can it happily co-exist with separate installs of more up-to-date versions of things? Like can I have Python 2.4 installed at the same time? Or a different Python 2.3? * Or is it user-upgradeable in a piecemeal manner? E.g. could I install Numpy 0.9.8 over top of enthon's older version of Numpy? --bb On 7/5/06, Edin Salkovi? wrote: > > Have you tried: > http://code.enthought.com/enthon/ > > Cheers, > Edin > > On 7/5/06, Bill Baxter wrote: > > I tried to get MayaVi and VTK working under Win32/MSVC.Net a while back > > failed miserably. > > > > Is there some simple, out-of-the-box, precompiled, no-brains-required > > solution for creating 3D plots? Preferably one that doesn't require > > compiling anything. > > > > --Bill > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ferenc.Pintye at eu.decoma.com Wed Jul 5 08:52:43 2006 From: Ferenc.Pintye at eu.decoma.com (Ferenc.Pintye at eu.decoma.com) Date: Wed, 5 Jul 2006 14:52:43 +0200 Subject: [Numpy-discussion] (no subject) Message-ID: Hi, how could i get the name of an array in a string ? (info command ?!) thanks f. From david.huard at gmail.com Wed Jul 5 09:23:12 2006 From: david.huard at gmail.com (David Huard) Date: Wed, 5 Jul 2006 09:23:12 -0400 Subject: [Numpy-discussion] arguments for stats.chi2 In-Reply-To: <20060704212320.30425.qmail@web51712.mail.yahoo.com> References: <20060704212320.30425.qmail@web51712.mail.yahoo.com> Message-ID: <91cf711d0607050623u5cf327bbx187749c407dc044d@mail.gmail.com> HI JJ, try stats.chi2.rvs(10) stats.chi2.rvs(10, loc=0, scale =100, size = 5) df is not a keyword argument, so writing df=10 explicitely was causing the error. David 2006/7/4, JJ : > > Hello. I have a very simple question. I would like > to generate a number of random variables from the chi2 > distribution. If I wanted these for the normal > distribution, the code could be > stats.norm.rvs(size=5,loc=100,scale=1). But > stats.chi2.rvs(size=5,df=10,loc=0,scale=1) or > stats.chi2.rvs(df=10,loc=0,scale=1) or > stats.chi2.rvs(df=10)does not work. Can anyone tell > me what the proper syntax would be for this? > Thanks JJ > > __________________________________________________ > Do You Yahoo!? > Tired of spam? Yahoo! Mail has the best spam protection around > http://mail.yahoo.com > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ferenc.pintye at steelsky.de Wed Jul 5 08:51:36 2006 From: ferenc.pintye at steelsky.de (ferenc pintye) Date: Wed, 5 Jul 2006 14:51:36 +0200 Subject: [Numpy-discussion] (no subject) Message-ID: <000c01c6a045$383f06c0$b90614ac@eu.decoma.com> Hi, how could i get the name of an array in a string ? thanks f. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexandre.fayolle at logilab.fr Wed Jul 5 11:49:58 2006 From: alexandre.fayolle at logilab.fr (Alexandre Fayolle) Date: Wed, 5 Jul 2006 17:49:58 +0200 Subject: [Numpy-discussion] (no subject) In-Reply-To: <000c01c6a045$383f06c0$b90614ac@eu.decoma.com> References: <000c01c6a045$383f06c0$b90614ac@eu.decoma.com> Message-ID: <20060705154958.GF26050@crater.logilab.fr> On Wed, Jul 05, 2006 at 02:51:36PM +0200, ferenc pintye wrote: > Hi, > > how could i get the name of an array in a string ? if by this you mean "the name of an identifier refering to [whatever]", this is a FAQ, and the answer is 1. you don't want to do that 2. your problem is not well behaved (there can be many answers to the questions) 3. you can use something along the lines of for name, item in locals().iteritems(): if item is my_array: break else: name = "anonymous object" -- Alexandre Fayolle LOGILAB, Paris (France) Formations Python, Zope, Plone, Debian: http://www.logilab.fr/formations D?veloppement logiciel sur mesure: http://www.logilab.fr/services Informatique scientifique: http://www.logilab.fr/science -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 481 bytes Desc: Digital signature URL: From robert.kern at gmail.com Wed Jul 5 11:51:25 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 05 Jul 2006 10:51:25 -0500 Subject: [Numpy-discussion] (no subject) In-Reply-To: References: Message-ID: Ferenc.Pintye at eu.decoma.com wrote: > > Hi, > > how could i get the name of an array in a string ? > (info command ?!) You can't. http://mail.python.org/pipermail/python-list/2005-June/286574.html -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From tim.hochberg at cox.net Wed Jul 5 18:14:44 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Wed, 05 Jul 2006 15:14:44 -0700 Subject: [Numpy-discussion] Dot as a method In-Reply-To: References: Message-ID: <44AC39D4.3030104@cox.net> Bart Vandereycken wrote: > Hi all, > > reading the thread "Ransom proposals" I was wondering why there isn't a > ndarray.dot() method? There is already a scipy.sparse.dot() so this > would fit nicely in the whole idea of polymorphism. > Are you sure about that? The problem with a dot method (aside from a over proliferation of methods in general) is that to be done correctly you want to choose a particular implementation of dot based *both* of its arguments. A method does a good job dispatching on a single argument, but falls down when dispatching on two types. Let's look at this specific case. Imagine that in addition ndarray.dot and sparse.dot, we also stick a dot method on ma.masked, etc. Now in order to fully exploit polymorphism and get maximum efficiency, we want asparsearray.dot(amaskedarray) to correctly treat the masked values (ma.dot treats them as zeros) and to not instantiate a dense version of the sparsearray. But in order to do that all three classes need to know about the other two. That's possible, if messy since all three of these are known in advance, but this approach becomes untenable if you classes outside core numpy or scipy to participate as full citizens. What does appear fit well here is generic functions / multimethods / protocols as discussed in some detail on pydev-3000 a couple of months ago. This would allow classes defined outside of core numpy and scipy to work correctly and efficiently with dot as long as they register appropriate versions of dot. If I wasn't swamped right at the moment I'd prototype this up and see how it works in practice. -tim From bhendrix at enthought.com Wed Jul 5 19:58:31 2006 From: bhendrix at enthought.com (Bryce Hendrix) Date: Wed, 05 Jul 2006 18:58:31 -0500 Subject: [Numpy-discussion] ANN: Python Enthought Edition Version 1.0.0.beta3 Released Message-ID: <44AC5227.9030902@enthought.com> Enthought is pleased to announce the release of Python Enthought Edition Version 1.0.0.beta3 (http://code.enthought.com/enthon/) -- a python distribution for Windows. 1.0.0.beta3 Release Notes: -------------------- Version 1.0.0.beta3 of Python Enthought Edition is the first version based on Python 2.4.3 and includes updates to nearly every package. This is the third and (hopefully) last beta release. This release includes version 1.0.8 of the Enthought Tool Suite (ETS) Package and bug fixes-- you can look at the release notes for this ETS version here: http://svn.enthought.com/downloads/enthought/changelog-release.1.0.8.html About Python Enthought Edition: ------------------------------- Python 2.4.3, Enthought Edition is a kitchen-sink-included Python distribution for Windows including the following packages out of the box: Numeric SciPy IPython Enthought Tool Suite wxPython PIL mingw f2py MayaVi Scientific Python VTK and many more... More information is available about all Open Source code written and released by Enthought, Inc. at http://code.enthought.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Wed Jul 5 20:13:29 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Wed, 5 Jul 2006 17:13:29 -0700 Subject: [Numpy-discussion] tentative numpy tutorial In-Reply-To: <6ef8f3380607040225t30948964o49b0f570b8768edf@mail.gmail.com> References: <6ef8f3380607040225t30948964o49b0f570b8768edf@mail.gmail.com> Message-ID: On 7/4/06, Pau Gargallo wrote: > motivated by the lack of free documentation for NumPy, with some > friends, we started writing a tutorial, that we would like to see in > scipy.org. After some time, the project have started to loose its > initial impetus. Now, we put the current unfinished version in > > http://www.scipy.org/Tentative_NumPy_Tutorial > > with the hope that, with the help of other users, it will be possible > to build a useful document for beginners. I think it looks great. From myeates at jpl.nasa.gov Wed Jul 5 20:37:32 2006 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Wed, 05 Jul 2006 17:37:32 -0700 Subject: [Numpy-discussion] simple question Message-ID: <44AC5B4C.10804@jpl.nasa.gov> What is the typical way of doing the following starting with a 0 matrix, set all values to 1 when a certain condition is met, set to -1 when another condition is met, left alone if neither condition is met. Mathew From kwgoodman at gmail.com Wed Jul 5 20:45:10 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Wed, 5 Jul 2006 17:45:10 -0700 Subject: [Numpy-discussion] simple question In-Reply-To: <44AC5B4C.10804@jpl.nasa.gov> References: <44AC5B4C.10804@jpl.nasa.gov> Message-ID: On 7/5/06, Mathew Yeates wrote: > What is the typical way of doing the following > starting with a 0 matrix, set all values to 1 when a certain condition > is met, set to -1 when another condition is met, left alone if neither > condition is met. This works on recent versions of numpy: >> x = asmatrix(zeros((2,2))) >> x matrix([[0, 0], [0, 0]]) >> y = asmatrix(rand(2,2)) >> y matrix([[ 0.85219404, 0.48311427], [ 0.41026966, 0.2184193 ]]) >> x[y > 0.5] = 1 >> x[y < 0.5] = -1 >> x matrix([[ 1, -1], [-1, -1]]) From lists.steve at arachnedesign.net Wed Jul 5 20:45:14 2006 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Wed, 5 Jul 2006 20:45:14 -0400 Subject: [Numpy-discussion] tentative numpy tutorial In-Reply-To: References: <6ef8f3380607040225t30948964o49b0f570b8768edf@mail.gmail.com> Message-ID: <29506E7D-02A3-4C71-AEE6-FEA30D701042@arachnedesign.net> >> motivated by the lack of free documentation for NumPy, with some >> friends, we started writing a tutorial, that we would like to see in >> scipy.org. After some time, the project have started to loose its >> initial impetus. Now, we put the current unfinished version in >> >> http://www.scipy.org/Tentative_NumPy_Tutorial >> >> with the hope that, with the help of other users, it will be possible >> to build a useful document for beginners. > > I think it looks great. Agreed .. thanks for whipping that together! -steve From myeates at jpl.nasa.gov Wed Jul 5 20:47:02 2006 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Wed, 05 Jul 2006 17:47:02 -0700 Subject: [Numpy-discussion] simple question In-Reply-To: References: <44AC5B4C.10804@jpl.nasa.gov> Message-ID: <44AC5D86.9050201@jpl.nasa.gov> ohhhh. I was looking at using "where" Keith Goodman wrote: > On 7/5/06, Mathew Yeates wrote: >> What is the typical way of doing the following >> starting with a 0 matrix, set all values to 1 when a certain condition >> is met, set to -1 when another condition is met, left alone if neither >> condition is met. > > This works on recent versions of numpy: > >>> x = asmatrix(zeros((2,2))) > >>> x > > matrix([[0, 0], > [0, 0]]) > >>> y = asmatrix(rand(2,2)) > >>> y > > matrix([[ 0.85219404, 0.48311427], > [ 0.41026966, 0.2184193 ]]) > >>> x[y > 0.5] = 1 > >>> x[y < 0.5] = -1 > >>> x > > matrix([[ 1, -1], > [-1, -1]]) > From paustin at eos.ubc.ca Wed Jul 5 20:51:04 2006 From: paustin at eos.ubc.ca (Philip Austin) Date: Wed, 5 Jul 2006 17:51:04 -0700 Subject: [Numpy-discussion] A C++ library and the new array interface: the best approach? In-Reply-To: <44A9A302.90901@ieee.org> References: <44A99D07.5030706@colorado.edu> <44A9A302.90901@ieee.org> Message-ID: <17580.24184.539788.121229@eos.ubc.ca> Travis Oliphant writes: > b) byte-the bullet and use Boost python (which other's who have > integrated C++ code with Python seem to really like). I've never > wrapped my mind around Boost python, though, so I'm not much help there. I've updated my set of boost helper functions to work with numpy -- there's a Readme at: http://www.eos.ubc.ca/research/clouds/software/pythonlibs/num_util/num_util_release2 (http://tinyurl.com/eo6fe) A simple example which shows some of the boost python features (reference counting, C++ versions of python types, docstrings, exceptions) is appended below. When called with python script http://tinyurl.com/zy57a it produces the output http://tinyurl.com/fr7xo Regards, Phil Austin -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: simpletest.cpp URL: From tom.denniston at alum.dartmouth.org Wed Jul 5 20:58:37 2006 From: tom.denniston at alum.dartmouth.org (Tom Denniston) Date: Wed, 5 Jul 2006 19:58:37 -0500 Subject: [Numpy-discussion] PyObject arrays, pickling, tofile Message-ID: It seems from my cursory look that numpy doesn't support the pickling or the tofile of PyObject arrays. I was wondering whether such functionlity is in the pipeline or if there is little interest. Personally, I would really like to see the feature in numpy but don't know if I am up to implementing it myself. --Tom From wbaxter at gmail.com Wed Jul 5 21:03:19 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu, 6 Jul 2006 10:03:19 +0900 Subject: [Numpy-discussion] simple question In-Reply-To: <44AC5D86.9050201@jpl.nasa.gov> References: <44AC5B4C.10804@jpl.nasa.gov> <44AC5D86.9050201@jpl.nasa.gov> Message-ID: That's the kind of thing that should maybe be on the NumpyExample list: http://www.scipy.org/Numpy_Example_List But currently little syntactical tricks like that aren't listed there. I'm not sure how you'd list that even. There is an example like that ( a[a<0.5]=0 ) on the Numpy for Matlab Users page, though. http://www.scipy.org/NumPy_for_Matlab_Users It's supposed to be for matlab users, but if you just ignore the Matlab column, the main table there is still a pretty good list of examples of numpy syntax and basic functions. --bb On 7/6/06, Mathew Yeates wrote: > > ohhhh. I was looking at using "where" > > > > Keith Goodman wrote: > > On 7/5/06, Mathew Yeates wrote: > >> What is the typical way of doing the following > >> starting with a 0 matrix, set all values to 1 when a certain condition > >> is met, set to -1 when another condition is met, left alone if neither > >> condition is met. > > > > This works on recent versions of numpy: > > > >>> x = asmatrix(zeros((2,2))) > > > >>> x > > > > matrix([[0, 0], > > [0, 0]]) > > > >>> y = asmatrix(rand(2,2)) > > > >>> y > > > > matrix([[ 0.85219404, 0.48311427], > > [ 0.41026966, 0.2184193 ]]) > > > >>> x[y > 0.5] = 1 > > > >>> x[y < 0.5] = -1 > > > >>> x > > > > matrix([[ 1, -1], > > [-1, -1]]) > > > > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- William V. Baxter III OLM Digital Kono Dens Building Rm 302 1-8-8 Wakabayashi Setagaya-ku Tokyo, Japan 154-0023 +81 (3) 3422-3380 -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Wed Jul 5 22:39:19 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu, 6 Jul 2006 11:39:19 +0900 Subject: [Numpy-discussion] Determining if two arrays share data Message-ID: Often when I'm doing interactive prototyping I find myself wanting to check whether two arrays are sharing a copy of the same data. It seems like there ought to be a concise way to do that, but right now seems like the only way is with a little function like this: def same_base(a,b): ab = a.base if ab is None: ab = a bb = b.base if bb is None: bb = b return ab is bb is there some easier built-in way? Does the above function even cover all the bases? (so to speak...) It would be easier if a non-copy array pointed to itself as base rather than 'None'. Then you could just check a.base == b.base. --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From satgroup at postbox.auckland.ac.nz Wed Jul 5 22:54:30 2006 From: satgroup at postbox.auckland.ac.nz (Satellite Data Research Group) Date: Thu, 6 Jul 2006 14:54:30 +1200 Subject: [Numpy-discussion] Problems installing numpy0.9.8 on MacOSX (10.4) Message-ID: <20060706145430.t48yabubgdsscowo@webmail.auckland.ac.nz> Dear Sir/Madam I am trying to install numpy0.9.8 onto Mac version 10.4 and Python2.4 and am having trouble doing so. I had successfully installed inumpy0.9.8 on Python2.3, so am very confused as to why it is not working on Python2.4 After typing 'python setup.py build' and then 'python setup.py install' from the terminal I get a host of errors, whereas with Python2.3 it worked perfectly. Then in python typing 'from numpy import *' then trying something like 'array((range(10))' I get an error saying array cannot be found, whereas on Python2.3 this worked. I was wondering if anyone else had trouble installing numpy on Python2.4 using Mac 10.4, and if anyone could help me rectify the problem. Help would be greatly appreciated. Sincerely Stephen Johnston ----------------------------------------------------------------------- This mail sent through University of Auckland http://www.auckland.ac.nz From tim.leslie at gmail.com Wed Jul 5 23:00:08 2006 From: tim.leslie at gmail.com (Tim Leslie) Date: Thu, 6 Jul 2006 13:00:08 +1000 Subject: [Numpy-discussion] Problems installing numpy0.9.8 on MacOSX (10.4) In-Reply-To: <20060706145430.t48yabubgdsscowo@webmail.auckland.ac.nz> References: <20060706145430.t48yabubgdsscowo@webmail.auckland.ac.nz> Message-ID: On 7/6/06, Satellite Data Research Group wrote: > Dear Sir/Madam > > I am trying to install numpy0.9.8 onto Mac version 10.4 and Python2.4 > and am having trouble doing so. I had successfully installed > inumpy0.9.8 on Python2.3, so am very confused as to why it is not > working on Python2.4 > > After typing 'python setup.py build' and then 'python setup.py install' > from the terminal I get a host of errors, whereas with Python2.3 it > worked perfectly. Then in python typing 'from numpy import *' then > trying something like 'array((range(10))' I get an error saying array > cannot be found, whereas on Python2.3 this worked. > Hi Stephen, Could you please post the exact error messages you get. Without these it's not possible to diagnose your problem. Cheers, Tim. > I was wondering if anyone else had trouble installing numpy on > Python2.4 using Mac 10.4, and if anyone could help me rectify the > problem. Help would be greatly appreciated. > > Sincerely > > Stephen Johnston > > ----------------------------------------------------------------------- > This mail sent through University of Auckland http://www.auckland.ac.nz > > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From oliphant.travis at ieee.org Wed Jul 5 23:17:00 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 05 Jul 2006 21:17:00 -0600 Subject: [Numpy-discussion] PyObject arrays, pickling, tofile In-Reply-To: References: Message-ID: <44AC80AC.1030708@ieee.org> Tom Denniston wrote: > It seems from my cursory look that numpy doesn't support the pickling > or the tofile of PyObject arrays. Yes, you can pickle PyObject arrays, that is supposed to work. What is not working? You can't use tofile on PyObject arrays, because I didn't have clear idea what to do. Use Pickle for PyObject arrays. -Travis From satgroup at postbox.auckland.ac.nz Wed Jul 5 23:23:50 2006 From: satgroup at postbox.auckland.ac.nz (Satellite Data Research Group) Date: Thu, 6 Jul 2006 15:23:50 +1200 Subject: [Numpy-discussion] Numpy not working (revised) Message-ID: <20060706152350.fgppt3uyop6o848k@webmail.auckland.ac.nz> Dear all I am trying to get numpy working with MacOSX 10.4 on an intel processor. I had it working for Python2.3.5, but after installing Python 2.4 it didn't work on either python version. After inputing: sudo python setup.py build and then sudo python setup.py install and also trying to load up a basic array, I get the following errors. The errors that I get repeatedly during the installation process seem to mainly be (without showing you the entire massive output, unless you need to see it): building extension "numpy.core.umath" sources adding 'build/src.darwin-8.7.1-i386-2.3/numpy/core/config.h' to sources. adding 'build/src.darwin-8.7.1-i386-2.3/numpy/core/__ufunc_api.h' to sources. adding 'build/src.darwin-8.7.1-i386-2.3/numpy/core/src' to include_dirs. numpy.core - nothing done with h_files= ['build/src.darwin-8.7.1-i386-2.3/numpy/core/config.h', 'build/src.darwin-8.7.1-i386-2.3/numpy/core/__multiarray_api.h', 'build/src.darwin-8.7.1-i386-2.3/numpy/core/__ufunc_api.h'] building extension "numpy.core._dotblas" sources adding 'numpy/core/blasdot/_dotblas.c' to sources. building extension "numpy.lib._compiled_base" sources building extension "numpy.dft.fftpack_lite" sources building extension "numpy.linalg.lapack_lite" sources adding 'numpy/linalg/lapack_litemodule.c' to sources. building extension "numpy.random.mtrand" sources customize NAGFCompiler customize AbsoftFCompiler customize IbmFCompiler Could not locate executable g77 Could not locate executable f77 Could not locate executable gfortran Could not locate executable f95 customize GnuFCompiler customize Gnu95FCompiler customize G95FCompiler customize GnuFCompiler customize Gnu95FCompiler customize NAGFCompiler customize NAGFCompiler using config gcc options: '-fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 -Wall -Wstrict-prototypes' compile options: '-Inumpy/core/src -Inumpy/core/include -I/System/Library/Frameworks/Python.framework/Versions/2.3/include/python2.3 -c' gcc: _configtest.c _configtest.c:7:2: error: #error No _WIN32 _configtest.c:7:2: error: #error No _WIN32 The above lines of output are repeated a few times. If it makes any difference, which I am sure that it does, I am running on an intel chip. In Python, when typing >>> from numpy import * I get: Running from numpy source directory. Then trying to make an array, I get: >>> vector = array((2,3,4,5,6)) Traceback (most recent call last): File "", line 1, in ? NameError: name 'array' is not defined This is in Python2.3.5. However, my collegue had numpy working on Python2.3.5, with the exact same machine just now. Just for background information, we are needing numpy to use pyhdf, to load and analyse hdf files. Thank you so much for your help. Sincerely Stephen Johnston p.s my apologies for the multiple requests, this one just a revised version of my previous attempt at asking for help ----------------------------------------------------------------------- This mail sent through University of Auckland http://www.auckland.ac.nz From oliphant.travis at ieee.org Wed Jul 5 23:48:18 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 05 Jul 2006 21:48:18 -0600 Subject: [Numpy-discussion] Numpy not working (revised) In-Reply-To: <20060706152350.fgppt3uyop6o848k@webmail.auckland.ac.nz> References: <20060706152350.fgppt3uyop6o848k@webmail.auckland.ac.nz> Message-ID: <44AC8802.2010105@ieee.org> Satellite Data Research Group wrote: > Dear all > > I am trying to get numpy working with MacOSX 10.4 on an intel > processor. I had it working for Python2.3.5, but after installing > Python 2.4 it didn't work on either python version. > > After inputing: > sudo python setup.py build > and then > sudo python setup.py install > > and also trying to load up a basic array, I get the following errors. > The errors that I get repeatedly during the installation process seem > to mainly be (without showing you the entire massive output, unless you > need to see it): > > > building extension "numpy.core.umath" sources > adding 'build/src.darwin-8.7.1-i386-2.3/numpy/core/config.h' to sources. > adding 'build/src.darwin-8.7.1-i386-2.3/numpy/core/__ufunc_api.h' to sources. > adding 'build/src.darwin-8.7.1-i386-2.3/numpy/core/src' to include_dirs. > numpy.core - nothing done with h_files= > ['build/src.darwin-8.7.1-i386-2.3/numpy/core/config.h', > 'build/src.darwin-8.7.1-i386-2.3/numpy/core/__multiarray_api.h', > 'build/src.darwin-8.7.1-i386-2.3/numpy/core/__ufunc_api.h'] > > building extension "numpy.core._dotblas" sources > adding 'numpy/core/blasdot/_dotblas.c' to sources. > building extension "numpy.lib._compiled_base" sources > building extension "numpy.dft.fftpack_lite" sources > building extension "numpy.linalg.lapack_lite" sources > adding 'numpy/linalg/lapack_litemodule.c' to sources. > building extension "numpy.random.mtrand" sources > customize NAGFCompiler > customize AbsoftFCompiler > customize IbmFCompiler > Could not locate executable g77 > Could not locate executable f77 > Could not locate executable gfortran > Could not locate executable f95 > > customize GnuFCompiler > customize Gnu95FCompiler > customize G95FCompiler > customize GnuFCompiler > customize Gnu95FCompiler > customize NAGFCompiler > customize NAGFCompiler using config > gcc options: '-fno-strict-aliasing -Wno-long-double -no-cpp-precomp > -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 -Wall > -Wstrict-prototypes' > compile options: '-Inumpy/core/src -Inumpy/core/include > -I/System/Library/Frameworks/Python.framework/Versions/2.3/include/python2.3 > -c' > gcc: _configtest.c > > _configtest.c:7:2: error: #error No _WIN32 > _configtest.c:7:2: error: #error No _WIN32 > > The above lines of output are repeated a few times. If it makes any > difference, which I am sure that it does, I am running on an intel chip. > > This is not a problem. These are "configuration" "error" that the system uses to incorporate platform-specific code. > In Python, when typing > >>>> from numpy import * >>>> > I get: > Running from numpy source directory. > > This is the problem. The error is telling you that you can't import numpy from the directory where numpy was built (the import mechanism gets confused between the system-installed numpy and the directory structure underneath). Just exit Python and cd to another directory. Then numpy should work fine. -Travis From lists.steve at arachnedesign.net Wed Jul 5 23:56:14 2006 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Wed, 5 Jul 2006 23:56:14 -0400 Subject: [Numpy-discussion] Numpy not working (revised) In-Reply-To: <20060706152350.fgppt3uyop6o848k@webmail.auckland.ac.nz> References: <20060706152350.fgppt3uyop6o848k@webmail.auckland.ac.nz> Message-ID: <768C2AFA-2F66-4C27-A924-5B19746C796D@arachnedesign.net> > I am trying to get numpy working with MacOSX 10.4 on an intel > processor. I had it working for Python2.3.5, but after installing > Python 2.4 it didn't work on either python version. Hi Stephen, I've been able to get numpy/scipy installed on my MacBook Pro (although I do get three unit test failures when I run nump.test(1) -- which I'll post next) One thing I'm noticing in what you printed out is that for some reason your doing something (linking?) to the version of Python that comes with your mac (2.3). Look at this output that you posted: > -I/System/Library/Frameworks/Python.framework/Versions/2.3/include/ > python2.3 Maybe you should do a: "python setups.py clean" and also remove your numpy build directory and start over. I guess you also have to make sure that all of your python 2.4 binaries come first (in terms of environment var/PATH) before the 2.3 binaries. Also .. one more thing, I also noticed that you didn't have any fortran compilers in your path, you can get gfortran like so: $curl -O http://easynews.dl.sourceforge.net/sourceforge/hpc/gfortran- intel-bin.tar.gz $ sudo tar -C / -xzf gfortran-intel-bin.tar.gz and it'll put gfortran in your /usr/local/bin (I got that info from http://bmistudents.blogspot.com/2006/04/on-installation-of-proper- python.html) -steve From lists.steve at arachnedesign.net Thu Jul 6 00:02:16 2006 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Thu, 6 Jul 2006 00:02:16 -0400 Subject: [Numpy-discussion] Mac OS 10.4 test errors Message-ID: <47338856-A9EA-4A08-A14B-6581CE1CD429@arachnedesign.net> Hi folks, I'm able to build and install numpy on my intel mac OS 10.4.7 but I'm getting 3 unit test failures. I'm using gcc 4.0.1 and GNU Fortran 95 (GCC) 4.2.0 20060401 (if that's helpful to know). The relevant output from my ipython session is below. Doing some of the stuff I've found in random tutorials doesn't seem to be a problem .. though I do get seg faults when trying to do some scipy tutorial plotting stuff .. which I may post to the scipy list a bit later ... I just want to make sure I'm installing everything correctly first. Thanks for any insight, -steve In [3]: numpy.__version__ Out[3]: '0.9.9.2735' In [4]: numpy.test(1) .... ... cut some stuff out until first error .. ... ====================================================================== ERROR: check_basic (numpy.core.tests.test_defmatrix.test_algebra) ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/local/lib/python2.4/site-packages/numpy/core/tests/ test_defmatrix.py", line 125, in check_basic Ainv = linalg.inv(A) File "/opt/local/lib/python2.4/site-packages/numpy/linalg/ linalg.py", line 122, in inv return wrap(solve(a, identity(a.shape[0]))) File "/opt/local/lib/python2.4/site-packages/numpy/linalg/ linalg.py", line 109, in solve results = lapack_routine(n_eq, n_rhs, a, n_eq, pivots, b, n_eq, 0) LapackError: Parameter ipiv is not of type PyArray_INT in lapack_lite.dgesv ====================================================================== ERROR: check_instance_methods (numpy.core.tests.test_defmatrix.test_matrix_return) ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/local/lib/python2.4/site-packages/numpy/core/tests/ test_defmatrix.py", line 157, in check_instance_methods f = eval('a.%s' % attrib) File "", line 0, in ? File "/opt/local/lib/python2.4/site-packages/numpy/core/ defmatrix.py", line 293, in getI return matrix(inv(self)) File "/opt/local/lib/python2.4/site-packages/numpy/linalg/ linalg.py", line 122, in inv return wrap(solve(a, identity(a.shape[0]))) File "/opt/local/lib/python2.4/site-packages/numpy/linalg/ linalg.py", line 109, in solve results = lapack_routine(n_eq, n_rhs, a, n_eq, pivots, b, n_eq, 0) LapackError: Parameter ipiv is not of type PyArray_INT in lapack_lite.dgesv ====================================================================== ERROR: check_basic (numpy.core.tests.test_defmatrix.test_properties) ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/local/lib/python2.4/site-packages/numpy/core/tests/ test_defmatrix.py", line 48, in check_basic assert allclose(linalg.inv(A), mA.I) File "/opt/local/lib/python2.4/site-packages/numpy/linalg/ linalg.py", line 122, in inv return wrap(solve(a, identity(a.shape[0]))) File "/opt/local/lib/python2.4/site-packages/numpy/linalg/ linalg.py", line 109, in solve results = lapack_routine(n_eq, n_rhs, a, n_eq, pivots, b, n_eq, 0) LapackError: Parameter ipiv is not of type PyArray_INT in lapack_lite.dgesv ---------------------------------------------------------------------- From satgroup at postbox.auckland.ac.nz Thu Jul 6 01:02:41 2006 From: satgroup at postbox.auckland.ac.nz (Satellite Data Research Group) Date: Thu, 6 Jul 2006 17:02:41 +1200 Subject: [Numpy-discussion] Numpy import problem Message-ID: <20060706170241.hq0fq9gssdys8kg4@webmail.auckland.ac.nz> Hi, I have a problem when importing numpy into python 2.4.1. I did a successful (I think) build and install of Numpy 0.9.8 on a mac osx (10.4) intel, but when i try to import numpy into python i get this error: >>> from numpy import* import core -> failed: Inappropriate file type for dynamic loading import lib -> failed: Inappropriate file type for dynamic loading import linalg -> failed: Inappropriate file type for dynamic loading import dft -> failed: Inappropriate file type for dynamic loading import random -> failed: Inappropriate file type for dynamic loading Traceback (most recent call last): File "", line 1, in -toplevel- from numpy import* File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/numpy/__init__.py", line 49, in -toplevel- import add_newdocs File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/numpy/add_newdocs.py", line 2, in -toplevel- from lib import add_newdoc File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/numpy/lib/__init__.py", line 5, in -toplevel- from type_check import * File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/numpy/lib/type_check.py", line 8, in -toplevel- import numpy.core.numeric as _nx File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/numpy/core/__init__.py", line 5, in -toplevel- import multiarray ImportError: Inappropriate file type for dynamic loading Any help anyone could offer as to the cause of this would be most welcome, Cheers, Joe Corbett ps. installing numpy through python 2.3 worked and I was able to import numpy into 2.3 fine. ----------------------------------------------------------------------- This mail sent through University of Auckland http://www.auckland.ac.nz From stefan at sun.ac.za Thu Jul 6 03:21:18 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu, 6 Jul 2006 09:21:18 +0200 Subject: [Numpy-discussion] Determining if two arrays share data In-Reply-To: References: Message-ID: <20060706072118.GB1070@mentat.za.net> On Thu, Jul 06, 2006 at 11:39:19AM +0900, Bill Baxter wrote: > Often when I'm doing interactive prototyping I find myself wanting to check > whether two arrays are sharing a copy of the same data. > > It seems like there ought to be a concise way to do that, but right now seems > like the only way is with a little function like this: > > def same_base(a,b): > ab = a.base > if ab is None: ab = a > bb = b.base > if bb is None: bb = b > return ab is bb > > is there some easier built-in way? Does the above function even cover all the > bases? (so to speak...) Say you have x = N.array([1,2,3,4]) and y = x.reshape((2,2)) then x and y share the same data. You can see this when you do x.__array_interface__['data'][0] == y.__array_interface__['data'][0] Still, this only holds for full data views. If you had z = y[1:,1:] then the data memory position would differ. Cheers St?fan From fullung at gmail.com Thu Jul 6 04:08:36 2006 From: fullung at gmail.com (Albert Strasheim) Date: Thu, 6 Jul 2006 10:08:36 +0200 Subject: [Numpy-discussion] Comparing arrays with dtypes with field names Message-ID: <00ef01c6a0d3$61b34500$0a84a8c0@dsp.sun.ac.za> Hello all I noticed that arrays with dtypes with field names don't compare correctly when using the equality operator. Example: In [17]: dt = N.dtype({'names' : ['f1', 'f2'], 'formats' : [N.intc, N.intc]}) In [18]: x = N.array([(0,0)], dtype=dt) In [19]: y = N.array([(0,0)], dtype=dt) In [20]: x['f1']==y['f1'] Out[20]: array([True], dtype=bool) In [21]: x['f2']==y['f2'] Out[21]: array([True], dtype=bool) In [22]: x==y Out[22]: False The question arises as to what should be returned when comparing these arrays. One option would be an array with dtype=bool which would indicate that each item in the array is equal in f1 and f2. Another option would be an array with dtype N.dtype({'names' : ['f1', 'f2'], 'formats' : [bool,bool]}) I think this second option could be very useful for some applications, where you might want to know something like: is the (i,j)th element of this array the same in f1 and f2 but different in f3? Any thoughts? Regards, Albert From oliphant.travis at ieee.org Thu Jul 6 04:20:46 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 06 Jul 2006 02:20:46 -0600 Subject: [Numpy-discussion] Comparing arrays with dtypes with field names In-Reply-To: <00ef01c6a0d3$61b34500$0a84a8c0@dsp.sun.ac.za> References: <00ef01c6a0d3$61b34500$0a84a8c0@dsp.sun.ac.za> Message-ID: <44ACC7DE.8080708@ieee.org> Albert Strasheim wrote: > Hello all > > I noticed that arrays with dtypes with field names don't compare correctly > when using the equality operator. > > Example: > > In [17]: dt = N.dtype({'names' : ['f1', 'f2'], > 'formats' : [N.intc, N.intc]}) > > In [18]: x = N.array([(0,0)], dtype=dt) > > In [19]: y = N.array([(0,0)], dtype=dt) > > In [20]: x['f1']==y['f1'] > Out[20]: array([True], dtype=bool) > > In [21]: x['f2']==y['f2'] > Out[21]: array([True], dtype=bool) > > In [22]: x==y > Out[22]: False > > The question arises as to what should be returned when comparing these > arrays. One option would be an array with dtype=bool which would indicate > that each item in the array is equal in f1 and f2. Currently, in SVN, record-arrays now can be compared (== or != only). The comparison returns a single bool array: For equality the comparison is basically: res = True for name in x.dtype.fields[-1]: res = logical_and(res, x[name] == y[name]) while for not-equal comparisons it is: res = False for name in x.dtype.fields[-1]: res = logical_or(res, x[name] != y[name]) > Another option would be > an array with dtype > > N.dtype({'names' : ['f1', 'f2'], 'formats' : [bool,bool]}) > > I think this second option could be very useful for some applications, where > you might want to know something like: is the (i,j)th element of this array > the same in f1 and f2 but different in f3? > This is a bit too sophisticated and does not work very well for nested arrays. The implemented comparison works for nested arrays just fine. You can just test for what you want directly (x['f1'] == y['f2']) & (x['f1'] != y['f3']) to handle the fewer particular cases where the other approach might be useful. -Travis From pau.gargallo at gmail.com Thu Jul 6 04:50:44 2006 From: pau.gargallo at gmail.com (Pau Gargallo) Date: Thu, 6 Jul 2006 10:50:44 +0200 Subject: [Numpy-discussion] simple question In-Reply-To: References: <44AC5B4C.10804@jpl.nasa.gov> <44AC5D86.9050201@jpl.nasa.gov> Message-ID: <6ef8f3380607060150ucaab1d8v8df523a05d9b6d2@mail.gmail.com> On 7/6/06, Bill Baxter wrote: > That's the kind of thing that should maybe be on the NumpyExample list: > http://www.scipy.org/Numpy_Example_List > But currently little syntactical tricks like that aren't listed there. I'm > not sure how you'd list that even. > There is a (tentative) introduction to 'fancy' indexing in the tentative numpy tutorial. http://www.scipy.org/Tentative_NumPy_Tutorial#head-0dffc419afa7d77d51062d40d2d84143db8216c2 i'm going to add the '[ ]' operator to the Numpy example list. pau From rob at hooft.net Thu Jul 6 05:03:15 2006 From: rob at hooft.net (Rob W.W. Hooft) Date: Thu, 06 Jul 2006 11:03:15 +0200 Subject: [Numpy-discussion] Determining if two arrays share data In-Reply-To: <20060706072118.GB1070@mentat.za.net> References: <20060706072118.GB1070@mentat.za.net> Message-ID: <44ACD1D3.9060608@hooft.net> Stefan van der Walt wrote: > On Thu, Jul 06, 2006 at 11:39:19AM +0900, Bill Baxter wrote: > >>Often when I'm doing interactive prototyping I find myself wanting to check >>whether two arrays are sharing a copy of the same data. > > > Say you have > > x = N.array([1,2,3,4]) > > and > > y = x.reshape((2,2)) > > then x and y share the same data. You can see this when you do > > x.__array_interface__['data'][0] == y.__array_interface__['data'][0] > > Still, this only holds for full data views. If you had > > z = y[1:,1:] > > then the data memory position would differ. I am still using Numeric, but even with that background I think that the problem is ill-defined. Consider the following interactive session: Python 2.3b1+ (#2, Jun 10 2003, 20:53:51) [GCC 3.0.2 20010905 (Red Hat Linux 7.1 3.0.1-3)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import Numeric >>> a=Numeric.arange(20) >>> b=a[::2] >>> c=a[1::2] >>> a array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]) >>> b array([ 0, 2, 4, 6, 8, 10, 12, 14, 16, 18]) >>> c array([ 1, 3, 5, 7, 9, 11, 13, 15, 17, 19]) >>> b[0]=99 >>> c[0]=99 >>> a array([99, 99, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]) Do "b" and "c" share a copy of the same data? None of the values in either array is visible in the other, but both share data with "a"! Regards, Rob -- Rob W.W. Hooft || rob at hooft.net || http://www.hooft.net/people/rob/ From fullung at gmail.com Thu Jul 6 05:45:02 2006 From: fullung at gmail.com (Albert Strasheim) Date: Thu, 6 Jul 2006 11:45:02 +0200 Subject: [Numpy-discussion] Determining if two arrays share data In-Reply-To: <44ACD1D3.9060608@hooft.net> Message-ID: <00fd01c6a0e0$da387790$0a84a8c0@dsp.sun.ac.za> Hello all > > Python 2.3b1+ (#2, Jun 10 2003, 20:53:51) > [GCC 3.0.2 20010905 (Red Hat Linux 7.1 3.0.1-3)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import Numeric > >>> a=Numeric.arange(20) > >>> b=a[::2] > >>> c=a[1::2] > >>> a > array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, > 16, 17, 18, > 19]) > >>> b > array([ 0, 2, 4, 6, 8, 10, 12, 14, 16, 18]) > >>> c > array([ 1, 3, 5, 7, 9, 11, 13, 15, 17, 19]) > >>> b[0]=99 > >>> c[0]=99 > >>> a > array([99, 99, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, > 16, 17, 18, > 19]) > > > Do "b" and "c" share a copy of the same data? None of the values in > either array is visible in the other, but both share data with "a"! I think we might be talking about two related but different concepts. One is sharing of data between arrays, the other is whether the data overlaps. Let's assume we can get at the starting address of the array in memory via the array interface or whatever, and the length of the array in bytes. To determine whether two arrays overlap, find the smallest data address of the two arrays. If the data address of the other array is smaller than the sum of the smallest data address and its corresponding length, you have overlap. I'm not quite sure how to check whether two arrays share the same underlying data buffer. Flags don't tell the whole story here. Regards, Albert From zhang.le.misc at gmail.com Thu Jul 6 06:31:17 2006 From: zhang.le.misc at gmail.com (Zhang Le) Date: Thu, 6 Jul 2006 11:31:17 +0100 Subject: [Numpy-discussion] bug in svn/numpy/numpy/doc/swig/numpy.i Message-ID: <4e7ed7700607060331i45969753v4b609237467578d2@mail.gmail.com> Hi, I found a bug in svn/numpy/numpy/doc/swig/numpy.i: a ')' is missed at line 72 which causes a compilation error: if (is_array(input) && (typecode == PyArray_NOTYPE || PyArray_EquivTypenums(array_type(input), typecode))) { ^ missing ary = (PyArrayObject*) input; Cheers, Zhang Le From rob at hooft.net Thu Jul 6 07:09:55 2006 From: rob at hooft.net (Rob W.W. Hooft) Date: Thu, 06 Jul 2006 13:09:55 +0200 Subject: [Numpy-discussion] Determining if two arrays share data In-Reply-To: <00fd01c6a0e0$da387790$0a84a8c0@dsp.sun.ac.za> References: <00fd01c6a0e0$da387790$0a84a8c0@dsp.sun.ac.za> Message-ID: <44ACEF83.4060503@hooft.net> Albert Strasheim wrote: > I think we might be talking about two related but different concepts. One is > sharing of data between arrays, the other is whether the data overlaps. When is it useful to know whether data buffers overlap? As long as they are disjoint, it should be irrelevant at the higher level of the program. > Let's assume we can get at the starting address of the array in memory via > the array interface or whatever, and the length of the array in bytes. > > To determine whether two arrays overlap, find the smallest data address of > the two arrays. If the data address of the other array is smaller than the > sum of the smallest data address and its corresponding length, you have > overlap. Don't forget the strides. Rob -- Rob W.W. Hooft || rob at hooft.net || http://www.hooft.net/people/rob/ From oliphant.travis at ieee.org Thu Jul 6 07:17:49 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 06 Jul 2006 05:17:49 -0600 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: Message-ID: <44ACF15D.6030607@ieee.org> Bill Baxter wrote: > So in short my proposal is to: > -- make a.T a property of array that returns a.swapaxes(-2,-1), > -- make a.H a property of array that returns > a.conjugate().swapaxes(-2,-1) > and maybe > -- make a.M a property of array that returns numpy.asmatrix(a) I've tentatively implemented all of these suggestions as well as adding the .A attribute to the ndarray as well (so that all sub-classes and array scalars can get back a view as an ndarray). I did this to make it easier to do matrix-like calculations with or with-out matrices. Matrix-calculation flexibility is still a sore-spot for many and I think these syntatical-sugar attributes will help long term. If there are no strong objections, then the recent MATH attribute checkin will stay. If there are major objections, then we can back them out without too much trouble as well. -Travis From oliphant.travis at ieee.org Thu Jul 6 07:30:52 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 06 Jul 2006 05:30:52 -0600 Subject: [Numpy-discussion] Speed degression? In-Reply-To: <200607041519.24636.steffen.loeck@gmx.de> References: <200607041519.24636.steffen.loeck@gmx.de> Message-ID: <44ACF46C.8050507@ieee.org> Steffen Loeck wrote: > Hi all, > > i made some speed tests using the sin-function and the %-operation to compare > Numeric, numpy 0.9.8 and numpy 0.9.9.2732. > As result the latest numpy version seems to be very slow in comparison to the > two other candidates. > > Results (in usec per loop): > sin-array mod-array > > Numeric 134 18 > > numpy 0.9.8 97 55 > > numpy 0.9.9.2732 204 316 > > numpy 0.9.8 + math 38 > > numpy 0.9.9.2732 + math 161 > > Numeric + math 23 > > > The used scripts can be found at the end. > > Can anyone verify my results and explain the observed speed degression? > I can't verify the results, but it's possible that the indexing code for a single integer is a bit slower. However, you forgot to test the "use Python lists" approach which is the fastest of all on this code. python -m timeit -s "x = [0.1]*10;" "for i in xrange(9): x[i+1]=(x[i]+1.1)%(1.0)" Using Numeric/NumPy is always slower than lists if you are doing element-by-element processing. NumPy is slower than Numeric for such processing. But, Numeric was always slower than pure Python too for such processing. So, the moral of the story is: "don't compute like that with NumPy (or Numeric)" With that said. I can add back the 1-d integer check in the indexing code to see if that speeds things up... -Travis From svetosch at gmx.net Thu Jul 6 07:45:54 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Thu, 06 Jul 2006 13:45:54 +0200 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44ACF15D.6030607@ieee.org> References: <44ACF15D.6030607@ieee.org> Message-ID: <44ACF7F2.1050904@gmx.net> Travis Oliphant schrieb: > Bill Baxter wrote: >> So in short my proposal is to: >> -- make a.T a property of array that returns a.swapaxes(-2,-1), >> -- make a.H a property of array that returns >> a.conjugate().swapaxes(-2,-1) >> and maybe >> -- make a.M a property of array that returns numpy.asmatrix(a) > > I've tentatively implemented all of these suggestions as well as adding > the .A attribute to the ndarray as well (so that all sub-classes and > array scalars can get back a view as an ndarray). > > I did this to make it easier to do matrix-like calculations with or > with-out matrices. Matrix-calculation flexibility is still a sore-spot > for many and I think these syntatical-sugar attributes will help long term. > I think this is great, thanks to Bill for suggesting it and to Travis for implementing it! So the only convenience of matrices over pure arrays that remains (afaics): -) .I for inverse; actually, why not add that to arrays as well as "syntactic sugar"? -) * being the matrix product instead of element-wise; Now, I could live with using dot and I don't want to push anything, but maybe this is the right time to consider another operator symbol as a shortcut for the dot function to be used with arrays? (Unfortunately right now I can't think of any sensible character(s) for that...) -) ** analogously for powers. For me this is less important. -) Being able to distinguish between row and column vectors; I guess this is just not possible with arrays... If --apart from the previous changes-- numpy had the .I for arrays I guess this would get me to say goodbye to matrices. The rest of the list would be a welcome luxury. I believe this whole thing has the potential to unify the use of numpy by eventually making the matrix subclass redundant. IMO that would be more transparent for new users and would increase the popularity of numpy! Cheers, Sven From bart.vandereycken at cs.kuleuven.be Thu Jul 6 07:49:57 2006 From: bart.vandereycken at cs.kuleuven.be (Bart Vandereycken) Date: Thu, 06 Jul 2006 13:49:57 +0200 Subject: [Numpy-discussion] Dot as a method In-Reply-To: <44AC39D4.3030104@cox.net> References: <44AC39D4.3030104@cox.net> Message-ID: Tim Hochberg wrote: > Bart Vandereycken wrote: >> Hi all, >> >> reading the thread "Ransom proposals" I was wondering why there isn't a >> ndarray.dot() method? There is already a scipy.sparse.dot() so this >> would fit nicely in the whole idea of polymorphism. >> > Are you sure about that? > > The problem with a dot method (aside from a over proliferation of > methods in general) is that to be done correctly you want to choose a > particular implementation of dot based *both* of its arguments. A method > does a good job dispatching on a single argument, but falls down when > dispatching on two types. Let's look at this specific case. Imagine that > in addition ndarray.dot and sparse.dot, we also stick a dot method on > ma.masked, etc. Now in order to fully exploit polymorphism and get > maximum efficiency, we want asparsearray.dot(amaskedarray) to correctly > treat the masked values (ma.dot treats them as zeros) and to not > instantiate a dense version of the sparsearray. But in order to do that > all three classes need to know about the other two. That's possible, if > messy since all three of these are known in advance, but this approach > becomes untenable if you classes outside core numpy or scipy to > participate as full citizens. You're right that you need to put the class specific dot code somewhere. However, it would be better if this can be done in a clean way. By the way I don't fixate on methods or functions, I just want something useful ;-) Suppose I have written my own matrix class (custmatrix) with it's own fast matrix-vector product. How can I extend numpy code so that is uses this fast multiplication when I want a_custmatrix*a_ndarray or a_ndarray*a_custmatrix? > What does appear fit well here is generic functions / multimethods / > protocols as discussed in some detail on pydev-3000 a couple of months > ago. This would allow classes defined outside of core numpy and scipy to > work correctly and efficiently with dot as long as they register > appropriate versions of dot. If I wasn't swamped right at the moment I'd > prototype this up and see how it works in practice. Right, I was thinking about multimethods too after reading http://www.artima.com/weblogs/viewpost.jsp?thread=101605 The other solution that simulates double dispatching is the Visitor pattern, but it has a lot of disadvantages (difficult to add custom matrices, slow?, all the matrices have to be children of an abstract class). Bart From arnd.baecker at web.de Thu Jul 6 08:06:25 2006 From: arnd.baecker at web.de (Arnd Baecker) Date: Thu, 6 Jul 2006 14:06:25 +0200 (CEST) Subject: [Numpy-discussion] Speed degression? In-Reply-To: <44ACF46C.8050507@ieee.org> References: <200607041519.24636.steffen.loeck@gmx.de> <44ACF46C.8050507@ieee.org> Message-ID: Hi Travis, On Thu, 6 Jul 2006, Travis Oliphant wrote: > Steffen Loeck wrote: > > Hi all, > > > > i made some speed tests using the sin-function and the %-operation to compare > > Numeric, numpy 0.9.8 and numpy 0.9.9.2732. > > As result the latest numpy version seems to be very slow in comparison to the > > two other candidates. > > > > Results (in usec per loop): > > sin-array mod-array > > > > Numeric 134 18 > > numpy 0.9.8 97 55 > > numpy 0.9.9.2732 204 316 > > numpy 0.9.8 + math 38 > > numpy 0.9.9.2732 + math 161 > > Numeric + math 23 > > > > Can anyone verify my results and explain the observed speed degression? > > > > I can't verify the results, but it's possible that the indexing code for > a single integer is a bit slower. On the 64Bit machine I get Results (in usec per loop): sin-array mod-array Numeric 34 7 numpy 0.9.9.2746 56 92 numpy 0.9.9.2746 + math 42 Numeric + math 8 Though this does not do the comparison with 0.9.8, it can roughly reproduce the ratios above. In particular the mod seems not just a "bit slower", but more than a factor 5. Also the " + math" variants for the sin suffered by a factor of 4... > However, you forgot to test the "use Python lists" approach which is the > fastest of all on this code. > > python -m timeit -s "x = [0.1]*10;" "for i in xrange(9): > x[i+1]=(x[i]+1.1)%(1.0)" > > > Using Numeric/NumPy is always slower than lists if you are doing > element-by-element processing. NumPy is slower than Numeric for such > processing. But, Numeric was always slower than pure Python too for > such processing. So, the moral of the story is: > > "don't compute like that with NumPy (or Numeric)" > Well, the example is a benchmark and not the real code ... To me it seems that it clearly points out some significant slow-down. > With that said. I can add back the 1-d integer check in the indexing > code to see if that speeds things up... That would be great - I will re-run the tests on the Opteron then. Many thanks, Arnd From wbaxter at gmail.com Thu Jul 6 08:58:01 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu, 6 Jul 2006 21:58:01 +0900 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44ACF7F2.1050904@gmx.net> References: <44ACF15D.6030607@ieee.org> <44ACF7F2.1050904@gmx.net> Message-ID: On 7/6/06, Sven Schreiber wrote: > > Travis Oliphant schrieb: > > Bill Baxter wrote: > >> So in short my proposal is to: > >> -- make a.T a property of array that returns a.swapaxes(-2,-1), > >> -- make a.H a property of array that returns > >> a.conjugate().swapaxes(-2,-1) > >> and maybe > >> -- make a.M a property of array that returns numpy.asmatrix(a) > > > > I've tentatively implemented all of these suggestions as well as adding > > the .A attribute to the ndarray as well (so that all sub-classes and > > array scalars can get back a view as an ndarray). That's great. > > > I did this to make it easier to do matrix-like calculations with or > > with-out matrices. Matrix-calculation flexibility is still a sore-spot > > for many and I think these syntatical-sugar attributes will help long > term. I think it will help too. The .M will make it easer to go from array to matrix types, and with the .A it was already easy to go from matrix to array. And the .T and .H will make it easier to do linear algebra numerics without switching over to matrix. One thing just occurred to me though, will the .M be available on matrices too? If not then one will have to be careful when using a mix of matrix and array to make sure you only use the .M on the latter and not the former. I think this is great, thanks to Bill for suggesting it and to Travis > for implementing it! Yes indeed, thanks! So the only convenience of matrices over pure arrays that remains (afaics): > > -) .I for inverse; actually, why not add that to arrays as well as > "syntactic sugar"? I don't even think it .I belongs in matrix. Hideously expensive calculations should not be masquerading as simple attribute accesses! Furthermore, maybe I'm just not doing the right kind of math, but inverses just don't come up that often for me. And if they do, it's usually in the context of solving a linear system Ax = b, where taking the inverse of A is generally a bad approach. So I'm negative on the inverse attribute in general, and double negative on it for arrays. Transpose is an operation that makes sense for a lot of kinds of data even non-matrix, but inverse is really only the multiplicative matrix inverse. Besides, given the new .M attribute, all you need to do to get the matrix inverse is .M.I. or .M.I.A if you want to get back to array. But sheesh, inv() is just 3 letters + parens. Going to .I only saves you 3 keystrokes. Going from .transpose() to .T saves 10. -) * being the matrix product instead of element-wise; Now, I could live > with using dot and I don't want to push anything, but maybe this is the > right time to consider another operator symbol as a shortcut for the dot > function to be used with arrays? (Unfortunately right now I can't think > of any sensible character(s) for that...) I don't think Python offers much flexibility in choosing characters to use as operators. That's one place where I think Haskell and ML are pretty neat. You can define your own in-fix operator symbols like (*) or -*- or <*> or .* like matlab. Whatever you want as long as it's a string of operator characters. Python can't do that, though. -) ** analogously for powers. For me this is less important. I don't have much need for that one. -) Being able to distinguish between row and column vectors; I guess > this is just not possible with arrays... Yeh that doesn't seem to be in the cards with array. But with the .M attribute you'll be able to get a column from a 1d array with a.M.T. Maybe a little improvement over a[None].transpose()? I guess I'd probably use a[None].T, though. If --apart from the previous changes-- numpy had the .I for arrays I > guess this would get me to say goodbye to matrices. The rest of the list > would be a welcome luxury. I believe this whole thing has the potential > to unify the use of numpy by eventually making the matrix subclass > redundant. IMO that would be more transparent for new users and would > increase the popularity of numpy! That was pretty much my conclusion. Ideally if you start off with matrices you would never encounter an array anywhere, and it would be just like using Matlab. :-) If it were like that I might still be using it. But it just doesn't seem to work out that way. One thing that would really help would be one of these mini-decorator modules Travis was talking about. In this case it would change all functions to return matrices. So if you do something like "import numpy.matrix" then after that ones() and zeros() and rand() etc will all be functions that return matrices. The .M attrib will make it easier. Instead of asmatrix(ones(3,3)) everywhere you'll have ones(3,3).M everywhere. But it's still not as nice as it would be if you could just import a package and have ones() etc return a matrix by default. I personally think matrix should be allowed to have more than 2 dimensions. Less than 2, no, but more than 2 should be ok. --bill -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Thu Jul 6 08:59:04 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 06 Jul 2006 06:59:04 -0600 Subject: [Numpy-discussion] Speed degression? In-Reply-To: References: <200607041519.24636.steffen.loeck@gmx.de> <44ACF46C.8050507@ieee.org> Message-ID: <44AD0918.9010702@ieee.org> Arnd Baecker wrote: > Hi >>> i made some speed tests using the sin-function and the %-operation to compare >>> Numeric, numpy 0.9.8 and numpy 0.9.9.2732. >>> As result the latest numpy version seems to be very slow in comparison to the >>> two other candidates. >>> Arrgggh. It looks like a pre-mature return statement was causing scalarmath not to load and therefore none of it's speed ups are present until this SVN. Thanks for the speed test... -Travis From arnd.baecker at web.de Thu Jul 6 09:36:11 2006 From: arnd.baecker at web.de (Arnd Baecker) Date: Thu, 6 Jul 2006 15:36:11 +0200 (CEST) Subject: [Numpy-discussion] Speed degression? In-Reply-To: <44AD0918.9010702@ieee.org> References: <200607041519.24636.steffen.loeck@gmx.de> <44ACF46C.8050507@ieee.org> <44AD0918.9010702@ieee.org> Message-ID: On Thu, 6 Jul 2006, Travis Oliphant wrote: > Arnd Baecker wrote: > > Hi > >>> i made some speed tests using the sin-function and the %-operation to compare > >>> Numeric, numpy 0.9.8 and numpy 0.9.9.2732. > >>> As result the latest numpy version seems to be very slow in comparison to the > >>> two other candidates. > >>> > > Arrgggh. > > It looks like a pre-mature return statement was causing scalarmath not > to load and therefore none of it's speed ups are present until this SVN. > > Thanks for the speed test... Well, the thanks for this go to Steffen (who wrote the text above and the test, not me ;-). Concerning the speed the thanks go to you: Results (in usec per loop): sin-array mod-array Numeric 34 7 numpy 0.9.9.2732 56 92 numpy 0.9.9.2732 + math 42 Numeric + math 8 *and now* Results (in usec per loop): sin-array mod-array Numeric 33 7 numpy 0.9.9.2749 30 17.6 numpy 0.9.9.2749 + math 13 Numeric + math 8 Bingo! Only numpy 0.9.9.2749 + math is slower than Numeric+math by a factor 1.6 and mod-array is 2.5 times slower than numpy. It would be nice if you could have a look at the module operation, if possible ... Many thanks, Arnd From tim.hochberg at cox.net Thu Jul 6 10:15:40 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 06 Jul 2006 07:15:40 -0700 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44ACF7F2.1050904@gmx.net> References: <44ACF15D.6030607@ieee.org> <44ACF7F2.1050904@gmx.net> Message-ID: <44AD1B0C.1070909@cox.net> Sven Schreiber wrote: > Travis Oliphant schrieb: > >> Bill Baxter wrote: >> >>> So in short my proposal is to: >>> -- make a.T a property of array that returns a.swapaxes(-2,-1), >>> -- make a.H a property of array that returns >>> a.conjugate().swapaxes(-2,-1) >>> and maybe >>> -- make a.M a property of array that returns numpy.asmatrix(a) >>> >> I've tentatively implemented all of these suggestions as well as adding >> the .A attribute to the ndarray as well (so that all sub-classes and >> array scalars can get back a view as an ndarray). >> >> I did this to make it easier to do matrix-like calculations with or >> with-out matrices. Matrix-calculation flexibility is still a sore-spot >> for many and I think these syntatical-sugar attributes will help long term. >> >> > > I think this is great, thanks to Bill for suggesting it and to Travis > for implementing it! > > So the only convenience of matrices over pure arrays that remains (afaics): > > -) .I for inverse; actually, why not add that to arrays as well as > "syntactic sugar"? > Because it encourages people to do the wrong thing numerically speaking? My understanding is that one almost never wants to compute the inverse directly, at least not if you're subsequently going to multiply it with something, instead you want to use linalg.solve or some other similar approach. > -) * being the matrix product instead of element-wise; Now, I could live > with using dot and I don't want to push anything, but maybe this is the > right time to consider another operator symbol as a shortcut for the dot > function to be used with arrays? (Unfortunately right now I can't think > of any sensible character(s) for that...) > See my earlier note on why using a method (which '*' is in essence) for dot is not really a good idea. We can do better. Oh, and this would completely hose all the non matrix heads out there as well. > -) ** analogously for powers. For me this is less important. > > -) Being able to distinguish between row and column vectors; I guess > this is just not possible with arrays... > Why can't you distinguish between them the same way that the matrix class does? Shape [1, N] is a row array, shape [N,1] is column array. > If --apart from the previous changes-- numpy had the .I for arrays I > guess this would get me to say goodbye to matrices. The rest of the list > would be a welcome luxury. I believe this whole thing has the potential > to unify the use of numpy by eventually making the matrix subclass > redundant. IMO that would be more transparent for new users and would > increase the popularity of numpy! >>> len(dir(numpy.ndarray)) - len(dir(int)) 99 I'm not particularly excited about the matrix class growing ever more cruft, but c'est la vie. I really need to finish up arraykit..... -tim From wbaxter at gmail.com Thu Jul 6 10:48:59 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu, 6 Jul 2006 23:48:59 +0900 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44AD1B0C.1070909@cox.net> References: <44ACF15D.6030607@ieee.org> <44ACF7F2.1050904@gmx.net> <44AD1B0C.1070909@cox.net> Message-ID: On 7/6/06, Tim Hochberg wrote: > > > -) Being able to distinguish between row and column vectors; I guess > > this is just not possible with arrays... > > > Why can't you distinguish between them the same way that the matrix > class does? Shape [1, N] is a row array, shape [N,1] is column array. Yep, that works. But there are still various annoyances. - You have to remeber to specify extra brackets all the time. Like array([[1,2,3]]) or array([[1],[2],[3]]). - And a slice of a vector out of a matrix has to be pumped back up to 2-D. If x has ndim==2, then to get a column out of it you have to do x[:,i,None] instead of just x[:,i]. To get a row you need x[j,None] instead of just x[j] Not horrible, but it feels a little klunky if you're used to something like Matlab. So matrix gets rid of a few annoyances like that ... and replaces them with a few of its own. :-) --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at cox.net Thu Jul 6 11:04:15 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 06 Jul 2006 08:04:15 -0700 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44AD1B0C.1070909@cox.net> References: <44ACF15D.6030607@ieee.org> <44ACF7F2.1050904@gmx.net> <44AD1B0C.1070909@cox.net> Message-ID: <44AD266F.8070707@cox.net> Tim Hochberg wrote: > Sven Schreiber wrote: > >> Travis Oliphant schrieb: >> >> >>> Bill Baxter wrote: >>> >>> >>>> So in short my proposal is to: >>>> -- make a.T a property of array that returns a.swapaxes(-2,-1), >>>> -- make a.H a property of array that returns >>>> a.conjugate().swapaxes(-2,-1) >>>> and maybe >>>> -- make a.M a property of array that returns numpy.asmatrix(a) >>>> >>>> >>> I've tentatively implemented all of these suggestions as well as adding >>> the .A attribute to the ndarray as well (so that all sub-classes and >>> array scalars can get back a view as an ndarray). >>> >>> I did this to make it easier to do matrix-like calculations with or >>> with-out matrices. Matrix-calculation flexibility is still a sore-spot >>> for many and I think these syntatical-sugar attributes will help long term. >>> >>> >>> >> I think this is great, thanks to Bill for suggesting it and to Travis >> for implementing it! >> >> So the only convenience of matrices over pure arrays that remains (afaics): >> >> -) .I for inverse; actually, why not add that to arrays as well as >> "syntactic sugar"? >> >> > Because it encourages people to do the wrong thing numerically speaking? > My understanding is that one almost never wants to compute the inverse > directly, at least not if you're subsequently going to multiply it with > something, instead you want to use linalg.solve or some other similar > approach. > Also, having an attribute access do an expensive operation behind the scenes seems antisocial. > >> -) * being the matrix product instead of element-wise; Now, I could live >> with using dot and I don't want to push anything, but maybe this is the >> right time to consider another operator symbol as a shortcut for the dot >> function to be used with arrays? (Unfortunately right now I can't think >> of any sensible character(s) for that...) >> >> > See my earlier note on why using a method (which '*' is in essence) for > dot is not really a good idea. Actually, thinking about this some more, this isn't a very compelling reason since multiply suffers from the same effect. Better reasons: 1. Consistency. Unless '/' is overloaded as well, then some of the operators will be elementwise and some will be matrix (+/- go both ways) making the interface inconsistent and thus harder to learn. 2. Backwards compatibility. 3. I was going to say narrowness of interface, since I had this fantasy that dot allowed axis to be specified, and thus we'd still need a separate dot function in addition to '*'. However, that must be something I have set up locally. (I realize that there's a separate multiply ufunc, but the ufuncs are all the same so I don't really count them against the overall complexity of numpy). Curmudgeonly yours, -tim > We can do better. Oh, and this would > completely hose all the non matrix heads out there as well. > > >> -) ** analogously for powers. For me this is less important. >> >> -) Being able to distinguish between row and column vectors; I guess >> this is just not possible with arrays... >> >> > Why can't you distinguish between them the same way that the matrix > class does? Shape [1, N] is a row array, shape [N,1] is column array. > >> If --apart from the previous changes-- numpy had the .I for arrays I >> guess this would get me to say goodbye to matrices. The rest of the list >> would be a welcome luxury. I believe this whole thing has the potential >> to unify the use of numpy by eventually making the matrix subclass >> redundant. IMO that would be more transparent for new users and would >> increase the popularity of numpy! >> > > >>> len(dir(numpy.ndarray)) - len(dir(int)) > 99 > > I'm not particularly excited about the matrix class growing ever more > cruft, but c'est la vie. I really need to finish up arraykit..... > > -tim > > > > > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From tim.hochberg at cox.net Thu Jul 6 11:45:13 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 06 Jul 2006 08:45:13 -0700 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44ACF7F2.1050904@gmx.net> <44AD1B0C.1070909@cox.net> Message-ID: <44AD3009.3030607@cox.net> Bill Baxter wrote: > On 7/6/06, *Tim Hochberg* > wrote: > > > -) Being able to distinguish between row and column vectors; I guess > > this is just not possible with arrays... > > > Why can't you distinguish between them the same way that the matrix > class does? Shape [1, N] is a row array, shape [N,1] is column array. > > > Yep, that works. But there are still various annoyances. > - You have to remeber to specify extra brackets all the time. Like > array([[1,2,3]]) or array([[1],[2],[3]]). This one I can't get excited about. If you are actually creating that many constant arrays, just define rowarray and colarray functions that add the appropriate dimensions for you. > - And a slice of a vector out of a matrix has to be pumped back up to > 2-D. If x has ndim==2, then to get a column out of it you have to do > x[:,i,None] instead of just x[:,i]. To get a row you need x[j,None] > instead of just x[j] Alternatively x[:,i:i+1], although that's not much better. > > Not horrible, but it feels a little klunky if you're used to something > like Matlab. Well Matlab is geared to matrices. The ndarray object has always been more or less a tensor. I can't help feeling that loading it up with matrix like methds is just going to lead to confusion and trouble. I would rather work things out so that we can have a pure matrix class and a pure ndarray class coexist in some sensible way. Figuring out how to do that well would have fringe benefits for other stuff (masked arrays, sparse arrays, user defined arrays of various types). > So matrix gets rid of a few annoyances like that ... and replaces them > with a few of its own. :-) In theory matrix should not be annoying to Matlab users since it's whole purpose is to keep matlab users happy. I think the big problem with matrix is that none of the developers use it as far as I know, so no one is motivated to clean up the rough edges. -tim From ndarray at mac.com Thu Jul 6 12:21:43 2006 From: ndarray at mac.com (Sasha) Date: Thu, 6 Jul 2006 12:21:43 -0400 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44ACF15D.6030607@ieee.org> References: <44ACF15D.6030607@ieee.org> Message-ID: I would like to raise a few objections going from mild to strong: 1. .T : I am mildly against it. As an inexpensive operation that returns a view (so that a.T[...] = makes sense) it is a reasonable candidate for an attribute. Unfortunately reversing the order of axes at least as reasonable as swapaxes(-2,-1) and swapaxes(-2,-1) is invalid for rank < 2. My main objection is that a.T is fairly cryptic - is there any other language that uses attribute for transpose? Adding .T to arrays will lead to less readable code because in expressions like "a * b.T" it will not be clear whether * is a matrix or elemenwise multiplication. 2. .H : This is an O(n^2) complexity operation returning a copy so it is not appropriate for an attribute. It does not make much sense for any type other than complex, so it's use is limited. 3. .M : I am strongly against this. It will create a circular dependency between ndarray and matrix. I would expect that asmatrix is mostly used to convert function arguments and for this purpose @matrix_args decorator would be a better solution to reduce code clutter. 4. .A : I have no clue what this one does, so I won't comment. On 7/6/06, Travis Oliphant wrote: > Bill Baxter wrote: > > So in short my proposal is to: > > -- make a.T a property of array that returns a.swapaxes(-2,-1), > > -- make a.H a property of array that returns > > a.conjugate().swapaxes(-2,-1) > > and maybe > > -- make a.M a property of array that returns numpy.asmatrix(a) > > I've tentatively implemented all of these suggestions as well as adding > the .A attribute to the ndarray as well (so that all sub-classes and > array scalars can get back a view as an ndarray). > > I did this to make it easier to do matrix-like calculations with or > with-out matrices. Matrix-calculation flexibility is still a sore-spot > for many and I think these syntatical-sugar attributes will help long term. > > If there are no strong objections, then the recent MATH attribute > checkin will stay. If there are major objections, then we can back them > out without too much trouble as well. > > -Travis > > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From Chris.Barker at noaa.gov Thu Jul 6 12:35:43 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 06 Jul 2006 09:35:43 -0700 Subject: [Numpy-discussion] simple question In-Reply-To: <44AC5D86.9050201@jpl.nasa.gov> References: <44AC5B4C.10804@jpl.nasa.gov> <44AC5D86.9050201@jpl.nasa.gov> Message-ID: <44AD3BDF.8020107@noaa.gov> Mathew Yeates wrote: > ohhhh. I was looking at using "where" There's nothing wrong with where: >>> y = N.asmatrix(N.rand(3,3)) >>> y matrix([[ 0.29741635, 0.78756994, 0.641378 ], [ 0.0198837 , 0.71677631, 0.76068183], [ 0.84904382, 0.80169706, 0.23877389]]) >>> x = N.asmatrix(N.where(y > 0.5, 1, 0)) >>> x matrix([[0, 1, 1], [0, 1, 1], [1, 1, 0]]) And did you really mean matrix? or would 2-d arrays be fine? -CHB -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From svetosch at gmx.net Thu Jul 6 12:40:46 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Thu, 06 Jul 2006 18:40:46 +0200 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44AD266F.8070707@cox.net> References: <44ACF15D.6030607@ieee.org> <44ACF7F2.1050904@gmx.net> <44AD1B0C.1070909@cox.net> <44AD266F.8070707@cox.net> Message-ID: <44AD3D0E.1060706@gmx.net> Tim Hochberg schrieb: >>> -) .I for inverse; actually, why not add that to arrays as well as >>> "syntactic sugar"? >>> >>> >> Because it encourages people to do the wrong thing numerically speaking? >> My understanding is that one almost never wants to compute the inverse >> directly, at least not if you're subsequently going to multiply it with >> something, instead you want to use linalg.solve or some other similar >> approach. >> > Also, having an attribute access do an expensive operation behind the > scenes seems antisocial. Ok, you (and Bill) may have a point there. (Although I'm not sure it's the best way to educate naive users like me about numerical efficiency by making inverse harder to access. Otherwise you should rename linalg.inv --way too easy!-- to linalg.okay_im_going_to_compute_the_inverse_for_you_but_its_probably_a_stupid_thing_to_do() ;-) However, then why is it possible for matrices?? It's just seems incoherent to me. Maybe you guys as the developers of numpy should really make up your mind about the future of matrices in numpy. Either it's supported, then eventually I would expect matrix versions of ones, zeros, eye, for example. (Although eye().M would come close enough I guess.) Or you decide that you don't really like all the problems that it implies and officially declare it unsupported. That would be valuable information for users like me who have had (and still sometimes have) a hard time figuring out whether I should be using arrays or matrices. > >> >>> -) * being the matrix product instead of element-wise; Now, I could live >>> with using dot and I don't want to push anything, but maybe this is the >>> right time to consider another operator symbol as a shortcut for the dot >>> function to be used with arrays? (Unfortunately right now I can't think >>> of any sensible character(s) for that...) >>> > 2. Backwards compatibility. I meant a *new* operator symbol. > Curmudgeonly yours, > Well I don't know what that means, so here's my hopefully equally incomprehensible reaction: Mit freundlichen Gr??en, Sven From Chris.Barker at noaa.gov Thu Jul 6 12:42:27 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 06 Jul 2006 09:42:27 -0700 Subject: [Numpy-discussion] Numpy import problem In-Reply-To: <20060706170241.hq0fq9gssdys8kg4@webmail.auckland.ac.nz> References: <20060706170241.hq0fq9gssdys8kg4@webmail.auckland.ac.nz> Message-ID: <44AD3D73.8080602@noaa.gov> Satellite Data Research Group wrote: > Hi, I have a problem when importing numpy into python 2.4.1. I did a > successful (I think) build and install of Numpy 0.9.8 on a mac osx > (10.4) intel, Which Python 2.4.1 are you using? It would be great if you would give the Python2.4.3 version found here a try: http://www.pythonmac.org/packages/py24-fat/index.html We're trying to establish it as the "recommended" Python for OS-X 10.4, on both Intel and PPC machines. In any case, we need to know what build of Python you're using. -CHB -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From kwgoodman at gmail.com Thu Jul 6 12:59:29 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 6 Jul 2006 09:59:29 -0700 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44AD3D0E.1060706@gmx.net> References: <44ACF15D.6030607@ieee.org> <44ACF7F2.1050904@gmx.net> <44AD1B0C.1070909@cox.net> <44AD266F.8070707@cox.net> <44AD3D0E.1060706@gmx.net> Message-ID: On 7/6/06, Sven Schreiber wrote: > Maybe you guys as the developers of numpy should really make up your > mind about the future of matrices in numpy. Either it's supported, then > eventually I would expect matrix versions of ones, zeros, eye, for > example. (Although eye().M would come close enough I guess.) Or you > decide that you don't really like all the problems that it implies and > officially declare it unsupported. > That would be valuable information for users like me who have had (and > still sometimes have) a hard time figuring out whether I should be using > arrays or matrices. If we use it (matrices) they will build it. All of the big problems I've run into while using matrices have been fixed within hours of posting the problem to the mailing list. So for anyone who wants better matrix support, all you have to do is start using matrices. The rest follows naturally: send any problems you have to the list and answer any questions you can from new matrix users. From wbaxter at gmail.com Thu Jul 6 13:21:44 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 7 Jul 2006 02:21:44 +0900 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> Message-ID: On 7/7/06, Sasha wrote: > > I would like to raise a few objections going from mild to strong: > > 1. .T : I am mildly against it. As an inexpensive operation that > returns a view (so that a.T[...] = makes sense) it is a reasonable > candidate for an attribute. Unfortunately reversing the order of axes at least as reasonable as > swapaxes(-2,-1) I suppose reversing the order changes you from C ordering to Fortran ordering? Other than that I can't think of any good examples of why you'd want to completely reverse the order of all your axes. I think it's much more common to want to swap just two axes, and the last two seem a logical choice since a) in the default C-ordering they're the closest together in memory and b) they're the axes that are printed contiguously when you say "print A". and swapaxes(-2,-1) is > invalid for rank < 2. > At least in numpy 0.9.8, it's not invalid, it just doesn't do anything. My main objection is that a.T is fairly cryptic > - is there any other language that uses attribute for transpose? Does it matter what other languages do? It's not _that_ cryptic. The standard way to write transpose is with a little T superscript in the upper right. We can't do that with ASCII so the T just appears after the dot. Makes perfect sense to me. I'd vote for an operator if it were possible in python. Something like A^T would be neat, maybe, or matlab's single-quote operator. Adding .T to arrays will lead to less readable code because in > expressions like "a * b.T" it will not be clear whether * is a matrix > or elemenwise multiplication. That seems a pretty weak argument, since there are already lots of expressions you can write that don't make it clear whether some operation is a matrix operation or array operation. You could write a * b.transpose(1,0) right now and still not know whether it was matrix or element-wise multiplication. Or doing A[:,1] when you know A is 2-D -- does it give you a 1D thing back or a 2D thing back? That just comes down to it being difficult to determine the class of objects in Python by looking at code. > 2. .H : This is an O(n^2) complexity operation returning a copy so > it is not appropriate for an attribute. Not sure how you get O(n^2). It just requires flipping the sign on the imaginary part of each element in the array. So in my book that's O(n). But that does make it more expensive than O(1) transpose, yes. It does not make much sense > for any type other than complex, so it's use is limited. I personally don't think I have ever used a hermitian transpose in my life. So I can't really can't say how useful it is. But the makers of Matlab decided to make single quote ( e.g. A' ) be the hermitian transpose operator, and dot-single quote (e.g. A.') be the regular transpose. So I'm assuming it's common enough that the folks behind Matlab thought it wise to make it the 'default' style of transpose and give it a one-character operator. That's about the only evidence I have that it's a useful operation, though. In general, though, I do know that when you take good-ole algorithms for reals and extend them to complex numbers, things that were transpose for the reals become hermitian transposes for the complex version. 3. .M : I am strongly against this. It will create a circular > dependency between ndarray and matrix. I would expect that asmatrix > is mostly used to convert function arguments and for this purpose > @matrix_args decorator would be a better solution to reduce code > clutter. I'm kindof ambivalent about this one too. Assuming matrix is going to stay around and we actually want to encourage people to use it, I think having a .M on arrays is an improvement over the current situation. Arguments to functions expecting matrices are as you say one place where conversions are needed, but another place is on funtions like zeros and ones and empty. With the .M you can just say ones(2,2).M. But probably a better solution would be to have matrix versions of these in the library as an optional module to import so people could, say, import them as M and use M.ones(2,2). It does seem to me that in some sense matrix is supposed to be 'just another customized array subclass', like sparse or masked, so to have array aware of it this one particular subclass makes me a little uneasy. But if matrix really should be considered to be on par with array, then it makes sense. It's just like a mutually recursive data structure. Or you can think of matrix's inheritance from array being an implementation detail. 4. .A : I have no clue what this one does, so I won't comment. It returns the array. I think the idea was that you would always be able to say .A with array or anything derived from it. Currently you have to know you have a matrix before you can use the .A attribute. If you were wrong and it was actually an array, then you'll get an exception. It would be nicer to have X.A just return X if X is already an array. In short, I'm most enthusiastic about the .T attribute. Then, given a .T, it makes sense to have a .H, both to be consistent with matrix, but also since it seems to be a big deal in other math packages like matlab. Then given the current situation, I like the .M but I can imagine other ways to make .M less necessary. --bb On 7/6/06, Travis Oliphant wrote: > > Bill Baxter wrote: > > > So in short my proposal is to: > > > -- make a.T a property of array that returns a.swapaxes(-2,-1), > > > -- make a.H a property of array that returns > > > a.conjugate().swapaxes(-2,-1) > > > and maybe > > > -- make a.M a property of array that returns numpy.asmatrix(a) > > > > I've tentatively implemented all of these suggestions as well as adding > > the .A attribute to the ndarray as well (so that all sub-classes and > > array scalars can get back a view as an ndarray). > > > > I did this to make it easier to do matrix-like calculations with or > > with-out matrices. Matrix-calculation flexibility is still a sore-spot > > for many and I think these syntatical-sugar attributes will help long > term. > > > > If there are no strong objections, then the recent MATH attribute > > checkin will stay. If there are major objections, then we can back them > > out without too much trouble as well. > > > > -Travis > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at cox.net Thu Jul 6 13:51:14 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 06 Jul 2006 10:51:14 -0700 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44AD3D0E.1060706@gmx.net> References: <44ACF15D.6030607@ieee.org> <44ACF7F2.1050904@gmx.net> <44AD1B0C.1070909@cox.net> <44AD266F.8070707@cox.net> <44AD3D0E.1060706@gmx.net> Message-ID: <44AD4D92.7020403@cox.net> Sven Schreiber wrote: > Tim Hochberg schrieb: > > >>>> -) .I for inverse; actually, why not add that to arrays as well as >>>> "syntactic sugar"? >>>> >>>> >>>> >>> Because it encourages people to do the wrong thing numerically speaking? >>> My understanding is that one almost never wants to compute the inverse >>> directly, at least not if you're subsequently going to multiply it with >>> something, instead you want to use linalg.solve or some other similar >>> approach. >>> >>> >> Also, having an attribute access do an expensive operation behind the >> scenes seems antisocial. >> > > Ok, you (and Bill) may have a point there. (Although I'm not sure it's > the best way to educate naive users like me about numerical efficiency > by making inverse harder to access. Otherwise you should rename > linalg.inv --way too easy!-- to > linalg.okay_im_going_to_compute_the_inverse_for_you_but_its_probably_a_stupid_thing_to_do() > ;-) > > However, then why is it possible for matrices?? It's just seems > incoherent to me. > Because matrix users asked for it I imagine. Since I don't use the matrix, I certainly wouldn't have objected. Also, the matrix class is supposed to look a lot like matlab and I assume matlab has some trivial way to produce an inverse -- even if it is a bad idea. > Maybe you guys as the developers of numpy should really make up your > mind about the future of matrices in numpy. Either it's supported, then > eventually I would expect matrix versions of ones, zeros, eye, for > example. (Although eye().M would come close enough I guess.) Or you > decide that you don't really like all the problems that it implies and > officially declare it unsupported. > That would be valuable information for users like me who have had (and > still sometimes have) a hard time figuring out whether I should be using > arrays or matrices. > Well, the matrix class could use a champion I think. Preferably someone who does a lot of matrix work and is capable of doing some development on it, at least on the python end. Without that I imagine it will stay kind of incoherent. > > >>> >>> >>>> -) * being the matrix product instead of element-wise; Now, I could live >>>> with using dot and I don't want to push anything, but maybe this is the >>>> right time to consider another operator symbol as a shortcut for the dot >>>> function to be used with arrays? (Unfortunately right now I can't think >>>> of any sensible character(s) for that...) >>>> >>>> >> 2. Backwards compatibility. >> > > I meant a *new* operator symbol. > Oh. Well, we're kind of short on those at the moment. The only thing I know of that would work is to abuse the call operator, thus dot(A, B) <=> (A)(B) That's kind of weird though and it's never gotten much support. > >> Curmudgeonly yours, >> A curmudgeon is an old, cranky, stubborn fixed in his ways kind of person. >> > Well I don't know what that means, so here's my hopefully equally > incomprehensible reaction: > Mit freundlichen Gr??en, > With great (big? much?) friendship (friendliness?)? -tim > Sven > > > From ndarray at mac.com Thu Jul 6 14:25:39 2006 From: ndarray at mac.com (Sasha) Date: Thu, 6 Jul 2006 14:25:39 -0400 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> Message-ID: On 7/6/06, Bill Baxter wrote: > > > On 7/7/06, Sasha wrote: > ... I think it's much > more common to want to swap just two axes, and the last two seem a logical > choice since a) in the default C-ordering they're the closest together in > memory and b) they're the axes that are printed contiguously when you say > "print A". It all depends on how you want to interpret a rank-K tensor. You seem to advocate a view that it is a (K-2)-rank array of matrices and .T is an element-wise transpose operation. Alternatively I can expect that it is a matrix of (K-2)-rank arrays and then .T should be swapaxes(0,1). Do you have real-life applications of swapaxes(-2,-1) for rank > 2? > > > and swapaxes(-2,-1) is > > invalid for rank < 2. > > > At least in numpy 0.9.8, it's not invalid, it just doesn't do anything. > That's bad. What sense does it make to swap non-existing axes? Many people would expect transpose of a vector to be a matrix. This is the case in S+ and R. > > My main objection is that a.T is fairly cryptic > > - is there any other language that uses attribute for transpose? > > > Does it matter what other languages do? It's not _that_ cryptic. If something is clear and natural, chances are it was done before. For me prior art is always a useful guide when making a design choice. For example, in R, the transpose operation is t(a) and works on rank <= 2 only always returning rank-2. K (an APL-like language) overloads unary '+' to do swapaxes(0,1) for rank>=2 and nothing for lower rank. Both R and K solutions are implementable in Python with R using 3 characters and K using 1(!) compared to your two-character ".T" notation. I would suggest that when inventing something new, you should consider prior art and explain how you invention is better. That's why what other languages do matter. (After all, isn't 'T' chosen because "transpose" starts with "t" in the English language?) > The standard way to write transpose is with a little T superscript in the upper > right. We can't do that with ASCII so the T just appears after the dot. > Makes perfect sense to me. I'd vote for an operator if it were possible in > python. Something like A^T would be neat, maybe, or matlab's single-quote > operator. > Well, you could overload __rpow__ for a singleton T and spell it A**T ... (I hope no one will take that proposal seriosely). Visually, A.T looks more like a subscript rather than superscript. > > Adding .T to arrays will lead to less readable code because in > > expressions like "a * b.T" it will not be clear whether * is a matrix > > or elemenwise multiplication. > > > That seems a pretty weak argument, since there are already lots of > expressions you can write that don't make it clear whether some operation is > a matrix operation or array operation. This may be a weak argument for someone used to matrix notation, but for me seeing a.T means: beware - tricky stuff here. > You could write a * b.transpose(1,0) > right now and still not know whether it was matrix or element-wise > multiplication. Why would anyone do that if b was a matrix? > > 2. .H : This is an O(n^2) complexity operation returning a copy so > > it is not appropriate for an attribute. > > Not sure how you get O(n^2). It just requires flipping the sign on the > imaginary part of each element in the array. So in my book that's O(n). > But that does make it more expensive than O(1) transpose, yes. > In my book n is the size of the matrix as in "n x n matrix", but the argument stays with O(n) as well. > But probably a better solution > would be to have matrix versions of these in the library as an optional > module to import so people could, say, import them as M and use M.ones(2,2). > This is the solution used by ma, which is another argument for it. > In short, I'm most enthusiastic about the .T attribute. Then, given a .T, > it makes sense to have a .H, both to be consistent with matrix, but also > since it seems to be a big deal in other math packages like matlab. Then > given the current situation, I like the .M but I can imagine other ways to > make .M less necessary. > I only raised a mild objection against .T, but the slippery slope argument makes me dislike it much more. At the very least I would like to see a discussion of why a.T is better than t(a) or +a. From tim.hochberg at cox.net Thu Jul 6 14:50:40 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 06 Jul 2006 11:50:40 -0700 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> Message-ID: <44AD5B80.4010905@cox.net> Sasha wrote: > On 7/6/06, Bill Baxter wrote: > >> On 7/7/06, Sasha wrote: >> ... I think it's much >> more common to want to swap just two axes, and the last two seem a logical >> choice since a) in the default C-ordering they're the closest together in >> memory and b) they're the axes that are printed contiguously when you say >> "print A". >> > > It all depends on how you want to interpret a rank-K tensor. You seem > to advocate a view that it is a (K-2)-rank array of matrices and .T is > an element-wise transpose operation. Alternatively I can expect that > it is a matrix of (K-2)-rank arrays and then .T should be > swapaxes(0,1). Do you have real-life applications of swapaxes(-2,-1) > for rank > 2? > I do for what it's worth. At various times I use arrays of shape (n, n, d) (matrices of rank-1 arrays at you suggest above) and arrays of shape (d, n, n) (vectors of matrices as Bill proposes). Using swapaxes(-2, -1) would be right only half the time, but it would the current defaults for transpose are essentially never right for rank > 2. Then again they are easy to explain. > >>> and swapaxes(-2,-1) is >>> invalid for rank < 2. >>> >>> >> At least in numpy 0.9.8, it's not invalid, it just doesn't do anything. >> >> > > That's bad. What sense does it make to swap non-existing axes? Many > people would expect transpose of a vector to be a matrix. This is the > case in S+ and R. > So this is essentially turning a row vector into a column vector? Is that right? >>> My main objection is that a.T is fairly cryptic >>> - is there any other language that uses attribute for transpose? >>> >> Does it matter what other languages do? It's not _that_ cryptic. >> > > If something is clear and natural, chances are it was done before. > For me prior art is always a useful guide when making a design choice. > For example, in R, the transpose operation is t(a) and works on rank > <= 2 only always returning rank-2. K (an APL-like language) overloads > unary '+' to do swapaxes(0,1) for rank>=2 and nothing for lower rank. > Both R and K solutions are implementable in Python with R using 3 > characters and K using 1(!) compared to your two-character ".T" > notation. Overloading '+' sure seems perverse, but maybe that's just me. > I would suggest that when inventing something new, you > should consider prior art and explain how you invention is better. > That's why what other languages do matter. (After all, isn't 'T' > chosen because "transpose" starts with "t" in the English language?) > > >> The standard way to write transpose is with a little T superscript in the upper >> right. We can't do that with ASCII so the T just appears after the dot. >> Makes perfect sense to me. I'd vote for an operator if it were possible in >> python. Something like A^T would be neat, maybe, or matlab's single-quote >> operator. >> >> > > Well, you could overload __rpow__ for a singleton T and spell it A**T > ... (I hope no one will take that proposal seriosely). Visually, A.T > looks more like a subscript rather than superscript. > No, no no. Overload __rxor__, then you can spell it A^t, A^h, etc. Much better ;-). [Sadly, I almost like that....] > >>> Adding .T to arrays will lead to less readable code because in >>> expressions like "a * b.T" it will not be clear whether * is a matrix >>> or elemenwise multiplication. >>> >> That seems a pretty weak argument, since there are already lots of >> expressions you can write that don't make it clear whether some operation is >> a matrix operation or array operation. >> > > This may be a weak argument for someone used to matrix notation, but > for me seeing a.T means: beware - tricky stuff here. > > >> You could write a * b.transpose(1,0) >> right now and still not know whether it was matrix or element-wise >> multiplication. >> > > Why would anyone do that if b was a matrix? > > > >>> 2. .H : This is an O(n^2) complexity operation returning a copy so >>> it is not appropriate for an attribute. >>> >> Not sure how you get O(n^2). It just requires flipping the sign on the >> imaginary part of each element in the array. So in my book that's O(n). >> But that does make it more expensive than O(1) transpose, yes. >> >> > In my book n is the size of the matrix as in "n x n matrix", but the > argument stays with O(n) as well. > > > > >> But probably a better solution >> would be to have matrix versions of these in the library as an optional >> module to import so people could, say, import them as M and use M.ones(2,2). >> >> > > This is the solution used by ma, which is another argument for it. > > > >> In short, I'm most enthusiastic about the .T attribute. Then, given a .T, >> it makes sense to have a .H, both to be consistent with matrix, but also >> since it seems to be a big deal in other math packages like matlab. Then >> given the current situation, I like the .M but I can imagine other ways to >> make .M less necessary. >> >> > > I only raised a mild objection against .T, but the slippery slope > argument makes me dislike it much more. At the very least I would > like to see a discussion of why a.T is better than t(a) or +a. > Here's a half baked thought: if the objection to t(A) is that it doesn't mirror the formulae where t appears as a subscript after A. Conceivably, __call__ could be defined so that A(x) returns x(A). That's kind of perverse, but it means that A(t), A(h), etc. could all work appropriately for suitably defined singletons. These singletons could either be assembeled in some abbreviations namespace or brought in by the programmer using "import transpose as t", etc. The latter works for doing t(a) as well of course. -tim From robert.kern at gmail.com Thu Jul 6 14:54:29 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 06 Jul 2006 13:54:29 -0500 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44ACF15D.6030607@ieee.org> References: <44ACF15D.6030607@ieee.org> Message-ID: Travis Oliphant wrote: > Bill Baxter wrote: >> So in short my proposal is to: >> -- make a.T a property of array that returns a.swapaxes(-2,-1), >> -- make a.H a property of array that returns >> a.conjugate().swapaxes(-2,-1) >> and maybe >> -- make a.M a property of array that returns numpy.asmatrix(a) > > I've tentatively implemented all of these suggestions as well as adding > the .A attribute to the ndarray as well (so that all sub-classes and > array scalars can get back a view as an ndarray). > > I did this to make it easier to do matrix-like calculations with or > with-out matrices. Matrix-calculation flexibility is still a sore-spot > for many and I think these syntatical-sugar attributes will help long term. > > If there are no strong objections, then the recent MATH attribute > checkin will stay. If there are major objections, then we can back them > out without too much trouble as well. Like Sasha, I'm mildly opposed to .T (as a synonym for .transpose()) and much more opposed to the rest (including .T being a synonym for .swapaxes(-2, -1)). It's not often that a proposal carries with it its own slippery-slope argument against itself. I don't think that just because arrays are often used for linear algebra that linear algebra assumptions should be built in to the core array type. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From alexander.belopolsky at gmail.com Thu Jul 6 15:18:43 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 6 Jul 2006 15:18:43 -0400 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44AD5B80.4010905@cox.net> References: <44ACF15D.6030607@ieee.org> <44AD5B80.4010905@cox.net> Message-ID: On 7/6/06, Tim Hochberg wrote: > ... > So this is essentially turning a row vector into a column vector? Is > that right? > Being a definition, this is neither right nor wrong. It all depends on what you are using it for. If you want to distinguish row and column vectors, you have to use rank-2, but it is convenient to to think of rank-1 as an equivalent of either row or column and rank-0 as a 1x1 array. If you define transpose as a rank-2 only operation, but allow promotion of rank < 2 to rank 2, you end up with the rules of S+. From alexander.belopolsky at gmail.com Thu Jul 6 15:37:27 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 6 Jul 2006 15:37:27 -0400 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44AD5B80.4010905@cox.net> References: <44ACF15D.6030607@ieee.org> <44AD5B80.4010905@cox.net> Message-ID: On 7/6/06, Tim Hochberg wrote: > ... > Overloading '+' sure seems perverse, but maybe that's just me. > The first time I saw it, it seemed perverse to me as well, but it actually make a lot of sense: 1. It is visually appealing as in '+' makes '|' from '-' and '-' from '|' and looks close enough to 't'. 2. It puts an otherwise useless operator to work. 3. Prefix spelling suggests that it should be swapaxes(0,1) rather than swapaxes(-2,-1), which is the choice made by K. 4. You can't get any shorter than that (at least using a fixed width font :-). 5. It already does the right thing for rank<2. From ndarray at mac.com Thu Jul 6 15:38:00 2006 From: ndarray at mac.com (Sasha) Date: Thu, 6 Jul 2006 15:38:00 -0400 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44AD5B80.4010905@cox.net> Message-ID: On 7/6/06, Tim Hochberg wrote: > ... > Overloading '+' sure seems perverse, but maybe that's just me. > The first time I saw it, it seemed perverse to me as well, but it actually make a lot of sense: 1. It is visually appealing as in '+' makes '|' from '-' and '-' from '|' and looks close enough to 't'. 2. It puts an otherwise useless operator to work. 3. Prefix spelling suggests that it should be swapaxes(0,1) rather than swapaxes(-2,-1), which is the choice made by K. 4. You can't get any shorter than that (at least using a fixed width font :-). 5. It already does the right thing for rank<2. From tim.hochberg at cox.net Thu Jul 6 16:23:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 06 Jul 2006 13:23:03 -0700 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44AD5B80.4010905@cox.net> Message-ID: <44AD7127.2090307@cox.net> Alexander Belopolsky wrote: > On 7/6/06, Tim Hochberg wrote: >> ... >> Overloading '+' sure seems perverse, but maybe that's just me. >> > The first time I saw it, it seemed perverse to me as well, but it > actually make a lot of sense: > > 1. It is visually appealing as in '+' makes '|' from '-' and '-' from > '|' and looks close enough to 't'. It looks even closer to ? (dagger if that doesn't make it through) which is the symbol used for the hermitian adjoint. > 2. It puts an otherwise useless operator to work. > 3. Prefix spelling suggests that it should be swapaxes(0,1) rather > than swapaxes(-2,-1), which is the choice made by K. > 4. You can't get any shorter than that (at least using a fixed width > font :-). > 5. It already does the right thing for rank<2. Perhaps it's not as perverse as it first appears. Although I still don't have to like it ;-) -tim > > From myeates at jpl.nasa.gov Thu Jul 6 16:23:49 2006 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Thu, 06 Jul 2006 13:23:49 -0700 Subject: [Numpy-discussion] simple question In-Reply-To: References: <44AC5B4C.10804@jpl.nasa.gov> Message-ID: <44AD7155.6070901@jpl.nasa.gov> Not working. A[row,all_dates == 10] = -1 where all_dates is a matrix with column length of 14 [[960111,..,.. and A is a matrix with same column length I get IndexError: arrays used as indices must be of integer type when I print out all_dates == 10 I get [True True True True True True True True True False False False True True]] I experimented with "<" instead of "==" but I still get boolean values as indices. Any help? Mathew Keith Goodman wrote: > On 7/5/06, Mathew Yeates wrote: >> What is the typical way of doing the following >> starting with a 0 matrix, set all values to 1 when a certain condition >> is met, set to -1 when another condition is met, left alone if neither >> condition is met. > > This works on recent versions of numpy: > >>> x = asmatrix(zeros((2,2))) > >>> x > > matrix([[0, 0], > [0, 0]]) > >>> y = asmatrix(rand(2,2)) > >>> y > > matrix([[ 0.85219404, 0.48311427], > [ 0.41026966, 0.2184193 ]]) > >>> x[y > 0.5] = 1 > >>> x[y < 0.5] = -1 > >>> x > > matrix([[ 1, -1], > [-1, -1]]) > From ndarray at mac.com Thu Jul 6 16:30:41 2006 From: ndarray at mac.com (Sasha) Date: Thu, 6 Jul 2006 16:30:41 -0400 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> Message-ID: On 7/6/06, Robert Kern wrote: > ... > I don't think that just because arrays are often used for linear algebra that > linear algebra assumptions should be built in to the core array type. > In addition, transpose is a (rank-2) array or matrix operation and not a linear algebra operation. Transpose corresponds to the "adjoint" linear algebra operation if you represent vectors as single column matrices and co-vectors as single-row matrices. This is a convenient representation followed by much of the relevant literature, but it does not alow generalization beyond rank-2. Another useful feature is that inner product can be calculated as the matrix product as long as you accept a 1x1 matrix for a scalar. This feature does not work beyond rank-2 either because in order to do tensor inner product you have to be explicit about the axes being collapsed (for example using Einstein notation). Since ndarray does not distinguish between upper an lower indices, it is not possible distinguish between vectors and co-vectors in any way other than using matrix convention. This makes ndarrays a poor model for linear algebra tensors. From oliphant at ee.byu.edu Thu Jul 6 16:36:48 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 06 Jul 2006 14:36:48 -0600 Subject: [Numpy-discussion] simple question In-Reply-To: <44AD7155.6070901@jpl.nasa.gov> References: <44AC5B4C.10804@jpl.nasa.gov> <44AD7155.6070901@jpl.nasa.gov> Message-ID: <44AD7460.7050109@ee.byu.edu> Mathew Yeates wrote: >Not working. >A[row,all_dates == 10] = -1 where all_dates is a matrix with column >length of 14 [[960111,..,.. >and A is a matrix with same column length > >I get >IndexError: arrays used as indices must be of integer type > >when I print out all_dates == 10 >I get >[True True True True True True True True True False False False True True]] > >I experimented with "<" instead of "==" but I still get boolean values >as indices. > >Any help? > > What version are you using? Can you give an example that shows the error. It's hard to guess the type of all the variables. The following works for me. import numpy print numpy.__version__ A = numpy.matrix(rand(3,14)) all_dates = array([10,10,1,10,1,10,0,10,0,10,0,1,10,1]) row = 2 A[row, all_dates == 10] -Trvis From ndarray at mac.com Thu Jul 6 16:41:44 2006 From: ndarray at mac.com (Sasha) Date: Thu, 6 Jul 2006 16:41:44 -0400 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44AD7127.2090307@cox.net> References: <44ACF15D.6030607@ieee.org> <44AD5B80.4010905@cox.net> <44AD7127.2090307@cox.net> Message-ID: On 7/6/06, Tim Hochberg wrote: > ... > It looks even closer to ? (dagger if that doesn't make it through) which > is the symbol used for the hermitian adjoint. If it pleases the matlab crowd, '+' can be defined to do the hermitian adjoint. on the complex type. > ... > Perhaps it's not as perverse as it first appears. Although I still don't > have to like it ;-) I don't like it either, but I don't like .T even more. These days I hate functionality I cannot google for. Call me selfish, but I already know what unary '+' can do to a higher rank array, but with .T I will always have to look up which axes it swaps ... From myeates at jpl.nasa.gov Thu Jul 6 16:42:08 2006 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Thu, 06 Jul 2006 13:42:08 -0700 Subject: [Numpy-discussion] simple question In-Reply-To: <44AD7460.7050109@ee.byu.edu> References: <44AC5B4C.10804@jpl.nasa.gov> <44AD7155.6070901@jpl.nasa.gov> <44AD7460.7050109@ee.byu.edu> Message-ID: <44AD75A0.8060809@jpl.nasa.gov> The very example you give produces IndexError: arrays used as indices must be of integer type this is with 0.9.8 Also .....while your example says "rand" I had to say numpy.rand This is on WindowsXP Mathew Travis Oliphant wrote: > Mathew Yeates wrote: > >> Not working. >> A[row,all_dates == 10] = -1 where all_dates is a matrix with column >> length of 14 [[960111,..,.. >> and A is a matrix with same column length >> >> I get >> IndexError: arrays used as indices must be of integer type >> >> when I print out all_dates == 10 >> I get >> [True True True True True True True True True False False False True >> True]] >> >> I experimented with "<" instead of "==" but I still get boolean >> values as indices. >> >> Any help? >> >> > What version are you using? Can you give an example that shows the > error. It's hard to guess the type of all the variables. The > following works for me. > > import numpy > print numpy.__version__ > A = numpy.matrix(rand(3,14)) > all_dates = array([10,10,1,10,1,10,0,10,0,10,0,1,10,1]) > row = 2 > A[row, all_dates == 10] > > > > > > -Trvis > > From kwgoodman at gmail.com Thu Jul 6 16:43:00 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 6 Jul 2006 13:43:00 -0700 Subject: [Numpy-discussion] simple question In-Reply-To: <44AD7460.7050109@ee.byu.edu> References: <44AC5B4C.10804@jpl.nasa.gov> <44AD7155.6070901@jpl.nasa.gov> <44AD7460.7050109@ee.byu.edu> Message-ID: On 7/6/06, Travis Oliphant wrote: > Mathew Yeates wrote: > > >Not working. > >A[row,all_dates == 10] = -1 where all_dates is a matrix with column > >length of 14 [[960111,..,.. > >and A is a matrix with same column length > > > >I get > >IndexError: arrays used as indices must be of integer type > > > >when I print out all_dates == 10 > >I get > >[True True True True True True True True True False False False True True]] > > > >I experimented with "<" instead of "==" but I still get boolean values > >as indices. > > > >Any help? > > > > > What version are you using? Can you give an example that shows the > error. It's hard to guess the type of all the variables. The following > works for me. > > import numpy > print numpy.__version__ > A = numpy.matrix(rand(3,14)) > all_dates = array([10,10,1,10,1,10,0,10,0,10,0,1,10,1]) > row = 2 > A[row, all_dates == 10] This is what NASA is doing (and what I would like to do): >> A[row, asmatrix(all_dates == 10)] --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /home/kwg/ /usr/local/lib/python2.4/site-packages/numpy/core/defmatrix.py in __getitem__(self, index) 122 123 def __getitem__(self, index): --> 124 out = N.ndarray.__getitem__(self, index) 125 # Need to swap if slice is on first index 126 # or there is an integer on the second ValueError: too many indices for array From myeates at jpl.nasa.gov Thu Jul 6 17:20:51 2006 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Thu, 06 Jul 2006 14:20:51 -0700 Subject: [Numpy-discussion] More zaniness Re: simple question In-Reply-To: <44AD75A0.8060809@jpl.nasa.gov> References: <44AC5B4C.10804@jpl.nasa.gov> <44AD7155.6070901@jpl.nasa.gov> <44AD7460.7050109@ee.byu.edu> <44AD75A0.8060809@jpl.nasa.gov> Message-ID: <44AD7EB3.1060207@jpl.nasa.gov> okay, I went back to the binary windows distrib. Based on Keths code I wrote >> print numpy.asmatrix(all_dates == start_dates[row],dtype=int) [[0 0 0 0 0 0 0 0 0 0 0 1 0 0]] >> [row,numpy.asmatrix(all_dates == start_dates[row],dtype=int)] = -1 >> print A[row,:] [[-1. -1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]] huh? It set the first 2 elements and not the 12'th!! Mathew From kwgoodman at gmail.com Thu Jul 6 17:43:34 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 6 Jul 2006 14:43:34 -0700 Subject: [Numpy-discussion] More zaniness Re: simple question In-Reply-To: <44AD7EB3.1060207@jpl.nasa.gov> References: <44AC5B4C.10804@jpl.nasa.gov> <44AD7155.6070901@jpl.nasa.gov> <44AD7460.7050109@ee.byu.edu> <44AD75A0.8060809@jpl.nasa.gov> <44AD7EB3.1060207@jpl.nasa.gov> Message-ID: On 7/6/06, Mathew Yeates wrote: > okay, I went back to the binary windows distrib. Based on Keths code I wrote > > >> print numpy.asmatrix(all_dates == start_dates[row],dtype=int) > [[0 0 0 0 0 0 0 0 0 0 0 1 0 0]] > >> [row,numpy.asmatrix(all_dates == start_dates[row],dtype=int)] = -1 > >> print A[row,:] > [[-1. -1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]] > > huh? It set the first 2 elements and not the 12'th!! You are assigning -1 to column 0 thirteen times and assinging -1 to column 1 once. For now, until boolean indexing works with matrices, I would just use brute force: A[row,where(all_dates.A == 10)[0]] Or you can do all rows at once with A[all_dates== 10] where all_dates is the same size as A. From oliphant at ee.byu.edu Thu Jul 6 17:43:59 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 06 Jul 2006 15:43:59 -0600 Subject: [Numpy-discussion] More zaniness Re: simple question In-Reply-To: <44AD7EB3.1060207@jpl.nasa.gov> References: <44AC5B4C.10804@jpl.nasa.gov> <44AD7155.6070901@jpl.nasa.gov> <44AD7460.7050109@ee.byu.edu> <44AD75A0.8060809@jpl.nasa.gov> <44AD7EB3.1060207@jpl.nasa.gov> Message-ID: <44AD841F.2020401@ee.byu.edu> Mathew Yeates wrote: >okay, I went back to the binary windows distrib. Based on Keths code I wrote > > >> print numpy.asmatrix(all_dates == start_dates[row],dtype=int) >[[0 0 0 0 0 0 0 0 0 0 0 1 0 0]] > >> [row,numpy.asmatrix(all_dates == start_dates[row],dtype=int)] = -1 > >> print A[row,:] >[[-1. -1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]] > >huh? It set the first 2 elements and not the 12'th!! > > Indexing has changed in SVN python, but in general, index matrices are not what you want because the dimensionality of the index arrays means something and matrices have two dimensions. So use arrays for indexing. -Travis From oliphant at ee.byu.edu Thu Jul 6 17:49:53 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 06 Jul 2006 15:49:53 -0600 Subject: [Numpy-discussion] Speed degression? In-Reply-To: References: <200607041519.24636.steffen.loeck@gmx.de> <44ACF46C.8050507@ieee.org> <44AD0918.9010702@ieee.org> Message-ID: <44AD8581.5030904@ee.byu.edu> Arnd Baecker wrote: >Bingo! Only numpy 0.9.9.2749 + math is slower than >Numeric+math by a factor 1.6 and mod-array is 2.5 times slower >than numpy. >It would be nice if you could have a look at the module operation, >if possible ... > > O.K. I looked at how Python computed modulo and implemented the same thing for both umath (the remainder function) and scalarmath. The result is a significant speed up for modulo...yeah... I also placed in hooks so you can replace the scalarmath (for int, float, and complex) with the Python version of math (this works because the int, float, and complex scalars are sub-classes of the corresponding Python object). import numpy.core.scalarmath as ncs # Replace "scalarmath" with standard Python arithmetic. ncs.use_pythonmath() # Replace "scalarmath" with actual scalarmath code. ncs.use_scalarmath() This can test the speed of the implementations. -Travis From tim.hochberg at cox.net Thu Jul 6 18:11:08 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 06 Jul 2006 15:11:08 -0700 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> Message-ID: <44AD8A7C.5030408@cox.net> Sasha wrote: > On 7/6/06, Robert Kern wrote: > >> ... >> I don't think that just because arrays are often used for linear algebra that >> linear algebra assumptions should be built in to the core array type. >> >> > > In addition, transpose is a (rank-2) array or matrix operation and not > a linear algebra operation. Transpose corresponds to the "adjoint" > linear algebra operation if you represent vectors as single column > matrices and co-vectors as single-row matrices. This is a convenient > representation followed by much of the relevant literature, but it > does not alow generalization beyond rank-2. Another useful feature is > that inner product can be calculated as the matrix product as long as > you accept a 1x1 matrix for a scalar. This feature does not work > beyond rank-2 either because in order to do tensor inner product you > have to be explicit about the axes being collapsed (for example using > Einstein notation). > At various times, I've thought about how one might do Einstein notation within Python. About the best I could come up with was: A.ijk * B.klm or A("ijk") * B("klm") Neither is spectacular, the first is a cleaner notation, but conceptually messy since it abuses getattr. Both require some intermediate pseudo object that wraps the array as well as info about the indexing. > Since ndarray does not distinguish between upper an lower indices, it > is not possible distinguish between vectors and co-vectors in any way > other than using matrix convention. This makes ndarrays a poor model > for linear algebra tensors. > My tensor math is rusty, but isn't it possible to represent all ones tensors as either covariant and contravariant and just embed the information about the metric into the product operator? It would seem that the inability to specify lower and upper indices is not truly limiting, but the inability to specify what axis to contract over is a fundamental limitation of sorts. I'm sure I'm partly influenced by my feeling that in practice upper and lower indices (aka contra- and covariant- and mixed-tensors) would be a pain in the neck, but a more capable inner product operator might well be useful if we could come up with correct syntax. -tim From wbaxter at gmail.com Thu Jul 6 21:56:43 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 7 Jul 2006 10:56:43 +0900 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44AD8A7C.5030408@cox.net> References: <44ACF15D.6030607@ieee.org> <44AD8A7C.5030408@cox.net> Message-ID: Tim Wrote: > That second argument is particularly uncompelling, but I think I agree > that in a vacuum swapaxes(-2,-1) would be a better choice for .T than > reversing the axes. However, we're not in a vacuum and there are several > reasons not to do this. > 1. A.T and A.transpose() should really have the same behavior. There may be a certain economy to that but I don't see why it should necessarily be so. Especially if it's agreed that the behavior .transpose() is not very useful. The use case for .T is primarily to make linear algebra stuff easier. If you're doing n-dim stuff and need something specific, you'll use the more general .transpose(). 2. Changing A.transpose would be one more backwards compatibility issue. Maybe it's a change worth making though, if we are right in saying that the current .transpose() for ndim>2 is hardly ever what you want. 3. Since, as far as I can tell there's not concise way of spelling > A.swapaxes(-2,-1) in terms of A.transpose it would make documenting and > explaining the default case harder. > Huh? A.swapaxes (-2,-1) is pretty concise. Why should it have to have an explanation in terms of A.transpose? Here's the explanation for the documentation: "A.T returns A with the last two axes transposed. It is equivalent to A.swapaxes (-2,-1). For a 2-d array, this is the usual matrix transpose." This just is a non-issue. Sasha wrote: > > more common to want to swap just two axes, and the last two seem a > logical > > choice since a) in the default C-ordering they're the closest together > in > > memory and b) they're the axes that are printed contiguously when you > say > > "print A". > > It all depends on how you want to interpret a rank-K tensor. You seem > to advocate a view that it is a (K-2)-rank array of matrices and .T is > an element-wise transpose operation. Alternatively I can expect that > it is a matrix of (K-2)-rank arrays and then .T should be > swapaxes(0,1). Do you have real-life applications of swapaxes(-2,-1) > for rank > 2? > Yep, like Tim said. The usage is say a N sets of basis vectors. Each set of basis vectors is a matrix. And say I have a different basis associated with each of N points in space. Usually I'll want to print it out organized by basis vector set. I.e. look at the matrix associated with each of the points. So it makes sense to organize it as shape=(N,a,b) so that if I print it I get something that's easy to interpret. If I set it up as shape=(a,b,N) then what's easiest to see in the print output is all N first basis vectors, all N second basis vectors, etc. Also again in a C memory layout, the last two axes are closest in memory, so it's more cache friendly to have the bits that will usually be used together in computations be on the trailing end. In matlab (which is fortran order), I do things the other way, with the N at the end of the shape. (And note that Matlab prints out the first two axes contiguously.) Either way swapaxes(-2,-1) is likely more likely to be what you want than .transpose(). > > and swapaxes(-2,-1) is > > > invalid for rank < 2. > > > > > At least in numpy 0.9.8, it's not invalid, it just doesn't do anything. > > > > > That's bad. What sense does it make to swap non-existing axes? Many > people would expect transpose of a vector to be a matrix. This is the > case in S+ and R. > Well, I would be really happy for .T to return an (N,1) column vector if handed an (N,) 1-d array. But I'm pretty sure that would raise more furuor among the readers of the list than leaving it 1-d. > > My main objection is that a.T is fairly cryptic > > > - is there any other language that uses attribute for transpose? > > > > > > Does it matter what other languages do? It's not _that_ cryptic. > If something is clear and natural, chances are it was done before. The thing is most other numerical computing languages were designed for doing numerical computing. They weren't designed originally for writing general purpose software, like Python was. So in matlab, for instance, transpose is a simple single-quote. But that doesn't help us decide what it should be in numpy. For me prior art is always a useful guide when making a design choice. > For example, in R, the transpose operation is t(a) and works on rank > <= 2 only always returning rank-2. I have serious reservations about a function called t(). x,y,z, and t are probably all in the top 10 variable names in scientific computing. K (an APL-like language) overloads > unary '+' to do swapaxes(0,1) for rank>=2 and nothing for lower rank. Hmm. That's kind of interesting, it seems like an abuse of notation to me. And precedence might be an issue too. The precedence of unary + isn't as high as attribute access. Anyway, as far as the meaning of + in K, I'm guessing K's arrays are in Fortran order, so (0,1) axes vary the fastest. I couldn't find any documentation for the K language from a quick search, though. Both R and K solutions are implementable in Python with R using 3 > characters and K using 1(!) compared to your two-character ".T" > notation. I would suggest that when inventing something new, you > should consider prior art and explain how you invention is better. > That's why what other languages do matter. (After all, isn't 'T' > chosen because "transpose" starts with "t" in the English language?) Yes you're right. My main thought was just what I said above, that there probably aren't too many other examples that can really apply in this case, both because most numerical computing languages are custom-designed for numerical computing, and also because Python's attributes are also kind of uncommon among programming languages. So it's worth looking at other examples, but in the end it has to be something that makes sense for a numerical computing package written in Python, and there aren't too many examples of that around. > You could write a * b.transpose(1,0) > > right now and still not know whether it was matrix or element-wise > > multiplication. > > Why would anyone do that if b was a matrix? > Maybe because, like you, they think "that a.T is fairly cryptic". > But probably a better solution > > would be to have matrix versions of these in the library as an optional > > module to import so people could, say, import them as M and use M.ones > (2,2). > > > This is the solution used by ma, which is another argument for it. > Yeh, I'm starting to think that's better than slapping an M attribute on arrays, too. Is it hard to write a module like that? I only raised a mild objection against .T, but the slippery slope > argument makes me dislike it much more. At the very least I would > like to see a discussion of why a.T is better than t(a) or +a. > * A.T puts the T on the proper side of A, so in that sense it looks more like the standard math notation. * A.T has precedence that roughly matches the standard math notation * t(A) uses an impossibly short function name that's likely to conflict with local variable names. To avoid the conflict people will just end up using it as numpy.t(A), at which point it's value as a shortcut for transpose is nullified. Or they'll have to do a mini-import within specific functions ("from numpy import t") to localize the namespace pollution. But at that point they might as well just say " t= numpy.transpose". * t(A) puts the transpose operator on the wrong side of A * +A puts the transpose operator on the wrong side of A also. * +A implies addition. The general rule with operator overloading is that the overload should have the same general meaning as the original operator. So overloading * for matrix multiplication makes sense. Overloading & for would be a bad idea. New users looking at something like A + +B are pretty certain to be confused because they think they know what + means, but they're wrong. If you see A + B.T, you either know what it means or you know immediately that you don't know what it means and you go look it up. * +A has different precedence than the usual transpose operator. (But I can't think of a case where that would make a difference now.) Tim Hochberg wrote: > > Well, you could overload __rpow__ for a singleton T and spell it A**T > > ... (I hope no one will take that proposal seriosely). Visually, A.T > > looks more like a subscript rather than superscript. > > > No, no no. Overload __rxor__, then you can spell it A^t, A^h, etc. Much > better ;-). [Sadly, I almost like that....] > Ouch! No way! It's got even worse precedence problems than the +A proposal. How about A+B^t ? And you still have to introduce 'h' and 't' into the global namespace for it to work. Here's a half baked thought: if the objection to t(A) is that it doesn't > mirror the formulae where t appears as a subscript after A. Conceivably, > __call__ could be defined so that A(x) returns x(A). That's kind of > perverse, but it means that A(t), A(h), etc. could all work > appropriately for suitably defined singletons. These singletons could > either be assembeled in some abbreviations namespace or brought in by > the programmer using "import transpose as t", etc. The latter works for > doing t(a) as well of course. Same problem with the need for global t. And it is kind of perverse, besides. Robert Kern wrote: > Like Sasha, I'm mildly opposed to .T (as a synonym for .transpose()) and > much > more opposed to the rest (including .T being a synonym for .swapaxes(-2, > -1)). > It's not often that a proposal carries with it its own slippery-slope > argument > against itself. > The slippery slope argument only applies to the .M, not the .T or .H. And I think if there's a matrixutils module with redefinitions of ones and zeros etc, and if other functions are all truly fixed to preserve matrix when matrix is passed in, then I agree, there's not so much need for .M. I don't think that just because arrays are often used for linear algebra > that > linear algebra assumptions should be built in to the core array type. It's not just that "arrays can be used for linear algebra". It's that linear algebra is the single most popular kind of numerical computing in the world! It's the foundation for a countless many fields. What you're saying is like "grocery stores shouldn't devote so much shelf space to food, because food is just one of the products people buy", or "this mailing list shouldn't be conducted in English, because English is just one of the languages people can speak here", or "I don't think my keyboard should devote so much space to the A-Z keys, because there are so many characters in the Unicode character set that could be there instead", or to quote from a particular comedy troop: "Ah, how about Cheddar?" "Well, we don't get much call for it around here, sir." "Not much ca- It's the single most popular cheese in the world!" "Not round here, sir." Linear algebra is pretty much the 'cheddar' of the numerical computing world. But it's more than that. It's like the yeast of the beer world. Pretty much everything starts with it as a base. It makes sense to make it as convenient as possible to do with numpy, even if it is a "special case". I wish I could think of some sort of statistics or google search I could cite to back this claim up, but as far as my academic background from high school though Ph.D. goes, linear algebra is a mighty big deal, not merely an "also ran" in the world of math or numerical computing. Sasha Wrote: > In addition, transpose is a (rank-2) array or matrix operation and not > a linear algebra operation. Transpose corresponds to the "adjoint" > linear algebra operation if you represent vectors as single column > matrices and co-vectors as single-row matrices. This is a convenient > representation followed by much of the relevant literature, but it > does not alow generalization beyond rank-2. > I would be willing to accept a .T that just threw an exception if ndim were > 2. That's what Matlab does with its transpose operator. I don't like that behavior myself -- it seems wasteful when it could just have some well defined behavior that would let it be useful at least some of the time on N-d arrays. I don't like it either, but I don't like .T even more. These days I > hate functionality I cannot google for. Call me selfish, but I > already know what unary '+' can do to a higher rank array, but with .T > I will always have to look up which axes it swaps ... I think '.T' is more likely to be searchable than '+'. And when you say you already know what unary + can do, you mean because you've used K? That's not much use to the typical user, who also thinks they know what a unary + does, but they'd be wrong in this case. So, in summary, I vote for: - Keep the .T and the .H on array - Get rid of .M - Instead implement a matrix helper module that could be imported as M, allowing M.ones(...) etc. And also: - Be diligent about fixing any errors from matrix users along the lines of " numpy.foo returns an array when given a matrix" (Travis has been good about this -- but we need to keep it up.) Part of the motivation for .M attribute was just as a band-aid on the problem of matrices getting turned into arrays. Having .M means you can just slap a .M on the end of any result you aren't sure about. It's better (but harder) to fix the upstream problem of functions not preserving subtypes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Thu Jul 6 22:20:58 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 7 Jul 2006 11:20:58 +0900 Subject: [Numpy-discussion] Enhancing dot() Message-ID: On 7/7/06, Tim Hochberg wrote: > I'd caution here though that the H is another thing that's going to > encourage people to write code that's less accurate and slower than it > needs to be. Consider the simple equation: > > [Side note: Here's something I periodically think about -- it would be > nice if dot could take multiple arguments and chain them into a series > of matrix multiplies so that dot(dot(a, b), a.H) could be written dot(a, > b, a.H). I'll use that here for clarity]. > > x = dot(a, b, a.H) > > versus: > > x = dot(a.real, b, a.real.transpose()) + dot(a.imag, b, a.imag.transpose > ()) > > The latter should be about twice as fast and won't have any pesky > imaginary residuals showing up from the finite precision of the various > operations. The funny thing is that having a dot(a,b,c,...) would lead to the exact same kind of hidden performance problems you're arguing against. The cost of a series of matrix multiplications can vary dramatically with the order in which you perform them. E.g. A*B*C*v where A,B,C are 2-dim and v is a column vector. Then you should do (A*(B*(C*v))), and definitely not ((A*B)*C)*v) But that said, dot could maybe be made smart enough to figure out the best order. That would be a nice feature. dot(A, dot(B, dot(C, v))) is pretty darn ugly. I seem to remember finding the best order being an example of a dynamic programming problem, and probably O(N^2) in the number of terms. But presumably N is never going to be very big, and the N==2 (no choice) and N=3 (only one choice) cases could be optimized. --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Jul 6 22:31:07 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 06 Jul 2006 21:31:07 -0500 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44AD8A7C.5030408@cox.net> Message-ID: Bill Baxter wrote: > Robert Kern wrote: > > Like Sasha, I'm mildly opposed to .T (as a synonym for .transpose()) > and much > more opposed to the rest (including .T being a synonym for > .swapaxes(-2, -1)). > It's not often that a proposal carries with it its own > slippery-slope argument > against itself. > > The slippery slope argument only applies to the .M, not the .T or .H. No, it was the "Let's have a .T attribute. And if we're going to do that, then we should also do this. And this. And this." > I don't think that just because arrays are often used for linear > algebra that > > linear algebra assumptions should be built in to the core array type. > > It's not just that "arrays can be used for linear algebra". It's that > linear algebra is the single most popular kind of numerical computing in > the world! It's the foundation for a countless many fields. What > you're saying is like "grocery stores shouldn't devote so much shelf > space to food, because food is just one of the products people buy", or [etc.] I'm sorry, but the argument-by-inappropriate-analogy is not convincing. Just because linear algebra is "the base" for a lot of numerical computing does not mean that everyone is using numpy arrays for linear algebra all the time. Much less does it mean that all of those conventions you've devised should be shoved into the core array type. I hold a higher standard for the design of the core array type than I do for the stuff around it. "It's convenient for what I do," just doesn't rise to that level. There has to be more of an argument for it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From wbaxter at gmail.com Thu Jul 6 22:56:11 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 7 Jul 2006 11:56:11 +0900 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44AD8A7C.5030408@cox.net> Message-ID: On 7/7/06, Robert Kern wrote: > > Bill Baxter wrote: > > Robert Kern wrote: > > > > > > The slippery slope argument only applies to the .M, not the .T or .H. > > No, it was the "Let's have a .T attribute. And if we're going to do that, > then > we should also do this. And this. And this." There's no slippery slope there. It's just "Let's have a .T attribute, and if we have that then we should have .H also." Period. The slope stops there. The .M and .A are a separate issue. > I don't think that just because arrays are often used for linear > > algebra that > > > > linear algebra assumptions should be built in to the core array > type. > > > > It's not just that "arrays can be used for linear algebra". It's that > > linear algebra is the single most popular kind of numerical computing in > > the world! It's the foundation for a countless many fields. What > > you're saying is like "grocery stores shouldn't devote so much shelf > > space to food, because food is just one of the products people buy", or > [etc.] > > I'm sorry, but the argument-by-inappropriate-analogy is not convincing. > Just > because linear algebra is "the base" for a lot of numerical computing does > not > mean that everyone is using numpy arrays for linear algebra all the time. > Much > less does it mean that all of those conventions you've devised should be > shoved > into the core array type. I hold a higher standard for the design of the > core > array type than I do for the stuff around it. "It's convenient for what I > do," > just doesn't rise to that level. There has to be more of an argument for > it. My argument is not that "it's convenient for what I do", it's that "it's convenient for what 90% of users want to do". But unfortunately I can't think of a good way to back up that claim with any sort of numbers. But here's one I just found: http://www.netlib.org/master_counts2.html download statistics for various numerical libraries on netlib.org. The top 4 are all linear algebra related: /lapack 37,373,505 /lapack/lug 19,908,865 /scalapack 14,418,172 /linalg 11,091,511 The next three are more like general computing issues: parallelization lib, performance monitoring, benchmarks: /pvm3 10,360,012 /performance 7,999,140 /benchmark 7,775,600 Then the next one is more linear algebra. And that seems to hold pretty far down the list. It looks like mostly stuff that's either linear algebra related or parallelization/benchmarking related. And as another example, there's the success of higher level numerical environments like Matlab (and maybe R and S? and Mathematica, and Maple?) that have strong support for linear algebra right in the core, not requiring users to go into some syntax/library ghetto to use that functionality. I am also curious, given the number of times I've heard this nebulous argument of "there are lots kinds of numerical computing that don't invlolve linear algebra", that no one ever seems to name any of these "lots of kinds". Statistics, maybe? But you can find lots of linear algebra in statistics. --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Jul 7 00:23:06 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 06 Jul 2006 23:23:06 -0500 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44AD8A7C.5030408@cox.net> Message-ID: Bill Baxter wrote: > On 7/7/06, *Robert Kern* > wrote: > > Bill Baxter wrote: > > Robert Kern wrote: [snip] > > I don't think that just because arrays are often used for linear > > algebra that > > > > linear algebra assumptions should be built in to the core > array type. > > > > It's not just that "arrays can be used for linear algebra". It's > that > > linear algebra is the single most popular kind of numerical > computing in > > the world! It's the foundation for a countless many fields. What > > you're saying is like "grocery stores shouldn't devote so much shelf > > space to food, because food is just one of the products people > buy", or > [etc.] > > I'm sorry, but the argument-by-inappropriate-analogy is not > convincing. Just > because linear algebra is "the base" for a lot of numerical > computing does not > mean that everyone is using numpy arrays for linear algebra all the > time. Much > less does it mean that all of those conventions you've devised > should be shoved > into the core array type. I hold a higher standard for the design of > the core > array type than I do for the stuff around it. "It's convenient for > what I do," > just doesn't rise to that level. There has to be more of an argument > for it. > > My argument is not that "it's convenient for what I do", it's that "it's > convenient for what 90% of users want to do". But unfortunately I can't > think of a good way to back up that claim with any sort of numbers. [snip] > I am also curious, given the number of times I've heard this nebulous > argument of "there are lots kinds of numerical computing that don't > invlolve linear algebra", that no one ever seems to name any of these > "lots of kinds". Statistics, maybe? But you can find lots of linear > algebra in statistics. That's because I'm not waving my hands at general fields of application. I'm talking about how people actually use array objects on a line-by-line basis. If I represent a dataset as an array and fit a nonlinear function to that dataset, am I using linear algebra at some level? Sure! Does having a .T attribute on that array help me at all? No. Arguing about how fundamental linear algebra is to numerical endeavors is entirely besides the point. I'm not saying that people who do use arrays for linear algebra are rare or unimportant. It's that syntactical convenience for one set of conventional ways to use an array object, by itself, is not a good enough reason to add stuff to the core array object. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant.travis at ieee.org Fri Jul 7 00:26:12 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 06 Jul 2006 22:26:12 -0600 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes Message-ID: <44ADE264.2030903@ieee.org> This is a call for a vote on each of the math attributes. Please post your vote as +1 : support +0 : don't care so go ahead -0 : don't care so why do it -1 : against Vote on the following issues separately: 1) .T Have some kind of .T attribute If >0 on this then: a) .T == .swapaxes(-2,-1) b) .T == .transpose() c) .T raises error for ndim > 2 d) .T returns (N,1) array for length (N,) array e) .T returns self for ndim < 2 2) .H returns .T.conj() 3) .M returns matrix version of array 4) .A returns basearray (useful for sub-classes). -Travis From oliphant.travis at ieee.org Fri Jul 7 00:28:19 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 06 Jul 2006 22:28:19 -0600 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE264.2030903@ieee.org> References: <44ADE264.2030903@ieee.org> Message-ID: <44ADE2E3.60102@ieee.org> Travis Oliphant wrote: > This is a call for a vote on each of the math attributes. Please post > your vote as > > +1 : support > +0 : don't care so go ahead > -0 : don't care so why do it > -1 : against > > Vote on the following issues separately: > > > > 1) .T Have some kind of .T attribute > > +1 > If >0 on this then: > > a) .T == .swapaxes(-2,-1) > +0 > b) .T == .transpose() > -0 > c) .T raises error for ndim > 2 > -1 > d) .T returns (N,1) array for length (N,) array > > +1 > e) .T returns self for ndim < 2 > > -1 > 2) .H returns .T.conj() > > > +0 > 3) .M returns matrix version of array > > +0 > 4) .A returns basearray (useful for sub-classes). > +0 -Travis From ndarray at mac.com Fri Jul 7 00:34:49 2006 From: ndarray at mac.com (Sasha) Date: Fri, 7 Jul 2006 00:34:49 -0400 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44AD8A7C.5030408@cox.net> Message-ID: On 7/6/06, Bill Baxter wrote: > ... > Yep, like Tim said. The usage is say a N sets of basis vectors. Each set > of basis vectors is a matrix. This brings up a feature that I really miss from numpy: an ability to do array([f(x) for x in a]) without python overhead. APL-like languages have a notion of "adverb" - a higher level operator that maps a function to a function. Numpy has some adverbs implemented as attributes to ufuncs: for example add.reduce is the same as +/ in K and add.accumulate is the same as +\ ('/' and '\' are 'over' and 'scan' adverbs in K). However, there is no way to do f/ or f\ where f is an arbitrary dyadic function. The equivalent of array([f(x) for x in a]) is spelled f'(a) in K (' is an adverb 'each'). The transpose operator (+) is swaps the first two axes, so in order to apply to the array of matrices, one would have to do +:'a (: in +: disambiguates + as a unary operator). I don't know of a good way to introduce adverbs in numpy, nor can I think of a good way to do list comprehensions, but array friendly versions of map, filter and reduce may be a good addition. These higher order functions may take an optional axes argument to deal with the higher rank arrays and may be optimized to recognize ufuncs so that map(f, a) could call f(a) and reduce(f, a) could do f.reduce(a) when f is a ufunc. [snip] > Either way swapaxes(-2,-1) is likely more likely to be what you want than > .transpose(). > Agree, but swapaxes(0, 1) is a close runner-up which is also known as zip in python. > Well, I would be really happy for .T to return an (N,1) column vector if > handed an (N,) 1-d array. But I'm pretty sure that would raise more furuor > among the readers of the list than leaving it 1-d. > Would you be even happier if .T would return a matrix? I hope not because my .M objection will apply. Maybe we can compromize by implementing a.T so that it raises ValueError unless rank(a) == 2 or at least unless rank(a) <= 2? > I have serious reservations about a function called t(). x,y,z, and t are > probably all in the top 10 variable names in scientific computing. > What about T()? > > > K (an APL-like language) overloads > > unary '+' to do swapaxes(0,1) for rank>=2 and nothing for lower rank. > > > Hmm. That's kind of interesting, it seems like an abuse of notation to me. > And precedence might be an issue too. The precedence of unary + isn't as > high as attribute access. It is high enough AFAICT - higher than any binary operator. > Anyway, as far as the meaning of + in K, I'm > guessing K's arrays are in Fortran order, so (0,1) axes vary the fastest. No, K has 1d arrays only, but they can be nested. Matrices are arrays of arrays and tensors are arrays of arrays of arrays ..., but you are right (0,1) swap is faster than (-2,-1) swap and this motivated the choice for the primitive. > I couldn't find any documentation for the K language from a quick search, > though. Kx Systems, the company behind K has replaced K with Q and pulled old manuals from the web. Q is close enough to K: see http://kx.com/q/d/k.txt for a terse summary. [snip] > > Why would anyone do that if b was a matrix? > Maybe because, like you, they think "that a.T is fairly cryptic". > If they are like me, they will not use numpy.matrix to begin with :-). > > > > But probably a better solution > > > would be to have matrix versions of these in the library as an optional > > > module to import so people could, say, import them as M and use > M.ones(2,2). > > > > > > > This is the solution used by ma, which is another argument for it. > > Yeh, I'm starting to think that's better than slapping an M attribute on > arrays, too. Is it hard to write a module like that? > Writing matrixutils with def zeros(shape, dtype=float): return asmatrix(zeros(shape, dtype)) is trivial, but matrixutils.zeros will have two python function calls overhead. This may be a case for making zeros a class method of ndarray that can be written in a way that will make inherited matrix.zeros do the right thing with no overhead. [snip] > * +A implies addition. No, it does not. Unary '+' is a noop. Does * imply multiplication or ** imply pow in f(*args, **kwds) to you? > The general rule with operator overloading is that > the overload should have the same general meaning as the original operator. Unary '+' has no preset meaning in plain python. It can be interpreted as transpose if you think of scalars as 1x1 matrices. > So overloading * for matrix multiplication makes sense. It depends on what you consider part of "general meaning". If the commutativity property is part of it then overloading * for matrix multiplication doesn't make sense. If the "general meaning" of unary + includes x = +x invariant, then you are right, but I am willing to relax that to x = ++x invariant when x is a non-symmetric matrix. > ... New users looking at something like A + +B are pretty > certain to be confused because they think they know what + means, but > they're wrong. In my experience new users don't realize that unary + is defined for arrays. Use of unary + with non-literal numbers is exotic enough that new users seeing "something like A + +B" will not assume that they know what it means. [snip] > * +A has different precedence than the usual transpose operator. (But I > can't think of a case where that would make a difference now.) > Maybe you can't because it doesn't? :-) > I would be willing to accept a .T that just threw an exception if ndim were > > 2. Aha! Let's start with an error unless ndim != 2. It is always easier to add good features than to remove bad ones. From robert.kern at gmail.com Fri Jul 7 00:47:59 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 06 Jul 2006 23:47:59 -0500 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE264.2030903@ieee.org> References: <44ADE264.2030903@ieee.org> Message-ID: Travis Oliphant wrote: > This is a call for a vote on each of the math attributes. Please post > your vote as > > +1 : support > +0 : don't care so go ahead > -0 : don't care so why do it > -1 : against > > Vote on the following issues separately: > > 1) .T Have some kind of .T attribute > > If >0 on this then: > > a) .T == .swapaxes(-2,-1) > b) .T == .transpose() > c) .T raises error for ndim > 2 > d) .T returns (N,1) array for length (N,) array > e) .T returns self for ndim < 2 The fact that there are a, b, c, d, and e makes me fully -1 on this. In the face of ambiguity, refuse the temptation to guess. > 2) .H returns .T.conj() -1 > 3) .M returns matrix version of array -1 > 4) .A returns basearray (useful for sub-classes). -1 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ndarray at mac.com Fri Jul 7 00:53:31 2006 From: ndarray at mac.com (Sasha) Date: Fri, 7 Jul 2006 00:53:31 -0400 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE264.2030903@ieee.org> References: <44ADE264.2030903@ieee.org> Message-ID: On 7/7/06, Travis Oliphant wrote: > 1) .T Have some kind of .T attribute > -1 (but -0 if raises an error if ndim != 2) > If >0 on this then: > > a) .T == .swapaxes(-2,-1) > > b) .T == .transpose() > > c) .T raises error for ndim > 2 > > d) .T returns (N,1) array for length (N,) array > > e) .T returns self for ndim < 2 > Write-in: f) .T raises error for ndim != 2 (still <0) > > 2) .H returns .T.conj() > -1 > > 3) .M returns matrix version of array > -1 > > 4) .A returns basearray (useful for sub-classes). > -1 From tim.hochberg at cox.net Fri Jul 7 02:03:48 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 06 Jul 2006 23:03:48 -0700 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44AD8A7C.5030408@cox.net> Message-ID: <44ADF944.5020003@cox.net> Sasha wrote: > On 7/6/06, Bill Baxter wrote: > >> ... >> Yep, like Tim said. The usage is say a N sets of basis vectors. Each set >> of basis vectors is a matrix. >> > > This brings up a feature that I really miss from numpy: an ability to do > > array([f(x) for x in a]) > Please note that there is now a fromiter function so that much of the overhead of the above function can be removed by using: numpy.fromiter((f(x) for x in a), float) This won't generate an intermediate list or use significantly extra storage. I doubt it's a full replacement for adverbs as you've described below though. -tim > without python overhead. APL-like languages have a notion of "adverb" > - a higher level operator that maps a function to a function. Numpy > has some adverbs implemented as attributes to ufuncs: for example > add.reduce is the same as +/ in K and add.accumulate is the same as +\ > ('/' and '\' are 'over' and 'scan' adverbs in K). However, there is no > way to do f/ or f\ where f is an arbitrary dyadic function. > > The equivalent of array([f(x) for x in a]) is spelled f'(a) in K (' is > an adverb 'each'). The transpose operator (+) is swaps the first two > axes, so in order to apply to the array of matrices, one would have to > do +:'a (: in +: disambiguates + as a unary operator). > > I don't know of a good way to introduce adverbs in numpy, nor can I > think of a good way to do list comprehensions, but array friendly > versions of map, filter and reduce may be a good addition. These > higher order functions may take an optional axes argument to deal with > the higher rank arrays and may be optimized to recognize ufuncs so > that map(f, a) could call f(a) and reduce(f, a) could do f.reduce(a) > when f is a ufunc. > > [snip] > >> Either way swapaxes(-2,-1) is likely more likely to be what you want than >> .transpose(). >> >> > > Agree, but swapaxes(0, 1) is a close runner-up which is also known as > zip in python. > > >> Well, I would be really happy for .T to return an (N,1) column vector if >> handed an (N,) 1-d array. But I'm pretty sure that would raise more furuor >> among the readers of the list than leaving it 1-d. >> >> > > Would you be even happier if .T would return a matrix? I hope not > because my .M objection will apply. Maybe we can compromize by > implementing a.T so that it raises ValueError unless rank(a) == 2 or > at least unless rank(a) <= 2? > > >> I have serious reservations about a function called t(). x,y,z, and t are >> probably all in the top 10 variable names in scientific computing. >> >> > > What about T()? > > >>> K (an APL-like language) overloads >>> unary '+' to do swapaxes(0,1) for rank>=2 and nothing for lower rank. >>> >> Hmm. That's kind of interesting, it seems like an abuse of notation to me. >> And precedence might be an issue too. The precedence of unary + isn't as >> high as attribute access. >> > > It is high enough AFAICT - higher than any binary operator. > > >> Anyway, as far as the meaning of + in K, I'm >> guessing K's arrays are in Fortran order, so (0,1) axes vary the fastest. >> > > No, K has 1d arrays only, but they can be nested. Matrices are arrays > of arrays and tensors are arrays of arrays of arrays ..., but you are > right (0,1) swap is faster than (-2,-1) swap and this motivated the > choice for the primitive. > > >> I couldn't find any documentation for the K language from a quick search, >> though. >> > > Kx Systems, the company behind K has replaced K with Q and pulled old > manuals from the web. Q is close enough to K: see > http://kx.com/q/d/k.txt for a terse summary. > > [snip] > >>> Why would anyone do that if b was a matrix? >>> >> Maybe because, like you, they think "that a.T is fairly cryptic". >> >> > If they are like me, they will not use numpy.matrix to begin with :-). > > >>>> But probably a better solution >>>> would be to have matrix versions of these in the library as an optional >>>> module to import so people could, say, import them as M and use >>>> >> M.ones(2,2). >> >>> This is the solution used by ma, which is another argument for it. >>> >> Yeh, I'm starting to think that's better than slapping an M attribute on >> arrays, too. Is it hard to write a module like that? >> >> > > Writing matrixutils with > > def zeros(shape, dtype=float): > return asmatrix(zeros(shape, dtype)) > > is trivial, but matrixutils.zeros will have two python function calls > overhead. This may be a case for making zeros a class method of > ndarray that can be written in a way that will make inherited > matrix.zeros do the right thing with no overhead. > > [snip] > >> * +A implies addition. >> > No, it does not. Unary '+' is a noop. Does * imply multiplication or > ** imply pow in f(*args, **kwds) to you? > > >> The general rule with operator overloading is that >> the overload should have the same general meaning as the original operator. >> > Unary '+' has no preset meaning in plain python. It can be interpreted > as transpose if you think of scalars as 1x1 matrices. > > >> So overloading * for matrix multiplication makes sense. >> > > It depends on what you consider part of "general meaning". If the > commutativity property is part of it then overloading * for matrix > multiplication doesn't make sense. If the "general meaning" of unary + > includes x = +x invariant, then you are right, but I am willing to > relax that to x = ++x invariant when x is a non-symmetric matrix. > > >> ... New users looking at something like A + +B are pretty >> certain to be confused because they think they know what + means, but >> they're wrong. >> > > In my experience new users don't realize that unary + is defined for > arrays. Use of unary + with non-literal numbers is exotic enough that > new users seeing "something like A + +B" will not assume that they > know what it means. > > [snip] > >> * +A has different precedence than the usual transpose operator. (But I >> can't think of a case where that would make a difference now.) >> >> > Maybe you can't because it doesn't? :-) > > >> I would be willing to accept a .T that just threw an exception if ndim were >> >>> 2. >>> > > Aha! Let's start with an error unless ndim != 2. It is always easier > to add good features than to remove bad ones. > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From wbaxter at gmail.com Fri Jul 7 02:17:36 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 7 Jul 2006 15:17:36 +0900 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44AD8A7C.5030408@cox.net> Message-ID: On 7/7/06, Robert Kern wrote: > > Bill Baxter wrote: > > I am also curious, given the number of times I've heard this nebulous > > argument of "there are lots kinds of numerical computing that don't > > invlolve linear algebra", that no one ever seems to name any of these > > "lots of kinds". Statistics, maybe? But you can find lots of linear > > algebra in statistics. > > That's because I'm not waving my hands at general fields of application. > I'm > talking about how people actually use array objects on a line-by-line > basis. If > I represent a dataset as an array and fit a nonlinear function to that > dataset, > am I using linear algebra at some level? Sure! Does having a .T attribute > on > that array help me at all? No. Arguing about how fundamental linear > algebra is > to numerical endeavors is entirely besides the point. Ok. If line-by-line usage is what everyone really means, then I'll get off the linear algebra soap box, but that's not what it sounded like to me. So, if you want to talk line-by-line, I really can't talk about much beside my own code. But I just grepped through it and out of 2445 non-empty lines of code: 927 lines contain '=' 390 lines contain a '[' 75 lines contain matrix,asmatrix, or mat ==> 47 lines contain a '.T' or '.transpose' of some sort. <== 33 lines contain array, or asarray, or asanyarray 24 lines contain 'rand(' --- I use it for generating bogus test data a lot 17 lines contain 'newaxis' or 'NewAxis' 16 lines contain 'zeros(' 13 lines contain 'dot(' 12 lines contain 'empty(' 8 lines contain 'ones(' 7 lines contain 'inv(' I'm pretty new to numpy, so that's all the code I got right now. I'm sure I've written many more lines of emails about numpy than I have lines of actual numpy code. :-/ But from that, I can say that -- at least in my code -- transpose is pretty common. If someone can point me to some larger codebases written in numpy or numeric, I'd be happy to do a similar analysis of those. I'm not saying that people who do use arrays for linear algebra are rare or > unimportant. It's that syntactical convenience for one set of conventional > ways > to use an array object, by itself, is not a good enough reason to add > stuff to > the core array object. I wish I had a way to magically find out the distribution of array dimensions used by all numpy and numeric code out there. My guess is it would be something like 1-d: 50%, 2-d: 30%, 3-d: 10%, everything else: 10%. I can't think of a good way to even get an estimate on that. But in any event, I'm positive ndims==2 is a significant percentage of all usages. It seems like the opponents to this idea are suggesting the distribution is more flat than that. But whatever the distribution is, it has to have a fairly light tail since memory usage is exponential in ndim. If ndim == 20, then it takes 8 megabytes just to store the smallest possible non-degenerate array of float64s (i.e. a 2x2x2x2x...) It seems crazy to even be arguing this. Transposing is not some specialized esoteric operation. It's important enough that R and S give it a one letter function, and Matlab, Scilab, K all give it a single-character operator. [*] Whoever designed the numpy.matrix class also thought it was worthy of a shortcut, and I think came up with a pretty good syntax for it. And the people who invented math itself decided it was worth assigning a 1-character exponent to it. So I think there's a clear argument for having a .T attribute. But ok, let's say you're right, and a lot of people won't use it. Fine. IT WILL DO THEM ABSOLUTELY NO HARM. They don't have to use it if they don't like it! Just ignore it. Unlike a t() function, .T doesn't pollute any namespace users can define symbols in, so you really can just ignore it if you're not interested in using it. It won't get in your way. For the argument that ndarray should be pure like the driven snow, just a raw container for n-dimensional data, I think that's what the basearray thing that goes into Python itself should be. ndarray is part of numpy and numpy is for numerical computing. Regards, --Bill [*] Full disclosure: I did find two counter-examples -- Maple and Mathematica. Maple has only a transpose() function and Mathematica has only Transpose[] (but you can use [esc]tr[esc] as a shortcut) However, both of those packages are primarily known for their _symbolic_ math capabilities, not their number crunching, so they less are similar to numpy than R,S,K,Matlab and Scilab in that regard. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at cox.net Fri Jul 7 02:26:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 06 Jul 2006 23:26:04 -0700 Subject: [Numpy-discussion] Enhancing dot() In-Reply-To: References: Message-ID: <44ADFE7C.5040506@cox.net> Bill Baxter wrote: > On 7/7/06, *Tim Hochberg* > wrote: > > I'd caution here though that the H is another thing that's going to > encourage people to write code that's less accurate and slower than it > needs to be. Consider the simple equation: > > [Side note: Here's something I periodically think about -- it > would be > nice if dot could take multiple arguments and chain them into a series > of matrix multiplies so that dot(dot(a, b), a.H) could be written > dot(a, > b, a.H). I'll use that here for clarity]. > > x = dot(a, b, a.H) > > versus: > > x = dot(a.real, b, a.real.transpose()) + dot(a.imag, b, > a.imag.transpose()) > > The latter should be about twice as fast and won't have any pesky > imaginary residuals showing up from the finite precision of the > various > operations. > > > The funny thing is that having a dot(a,b,c,...) would lead to the > exact same kind of hidden performance problems you're arguing against. Not exactly arguing -- this isn't why I don't like H and friends -- just noting that this is one of the traps that people are likely to fall into when transferring equations to code. > The cost of a series of matrix multiplications can vary dramatically > with the order in which you perform them. E.g. A*B*C*v where A,B,C > are 2-dim and v is a column vector. Then you should do (A*(B*(C*v))), > and definitely not ((A*B)*C)*v) That's a good point. > But that said, dot could maybe be made smart enough to figure out the > best order. That would be a nice feature. > dot(A, dot(B, dot(C, v))) is pretty darn ugly. I seem to remember > finding the best order being an example of a dynamic programming > problem, and probably O(N^2) in the number of terms. But presumably N > is never going to be very big, and the N==2 (no choice) and N=3 (only > one choice) cases could be optimized. You could probably just brute force it with out too much trouble. As you say O(n^2) is not that daunting when n is usually 2 or 3 and rarely over 5. Perhaps cache the last few configurations tested since the same matrix size combinations will likely occur over and over. Another interesting thought is one just wrapped an array with a wrapper that says "use the conjugate of what I'm wrapping", then dot(a, b H(a)) could do the write thing and only compute the real part of the equation, assuming H(a) created the wrapper. That's starting to get complex though. -tim From wbaxter at gmail.com Fri Jul 7 02:29:06 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 7 Jul 2006 15:29:06 +0900 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE264.2030903@ieee.org> References: <44ADE264.2030903@ieee.org> Message-ID: On 7/7/06, Travis Oliphant wrote: > > > 1) .T Have some kind of .T attribute +1 If >0 on this then: > > a) .T == .swapaxes(-2,-1) +1 b) .T == .transpose() +0 c) .T raises error for ndim > 2 +0 d) .T returns (N,1) array for length (N,) array +1 e) .T returns self for ndim < 2 +0 2) .H returns .T.conj() +1 3) .M returns matrix version of array -1 4) .A returns basearray (useful for sub-classes). +0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Fri Jul 7 02:44:14 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 7 Jul 2006 15:44:14 +0900 Subject: [Numpy-discussion] Enhancing dot() In-Reply-To: <44ADFE7C.5040506@cox.net> References: <44ADFE7C.5040506@cox.net> Message-ID: On 7/7/06, Tim Hochberg wrote: > > > The funny thing is that having a dot(a,b,c,...) would lead to the > > exact same kind of hidden performance problems you're arguing against. > Not exactly arguing -- this isn't why I don't like H and friends -- just > noting that this is one of the traps that people are likely to fall into > when transferring equations to code. There's a strong argument to be made that the whole design of most array math packages is flawed and leads to inefficient code. The classic example is something like: A = B + C - 2*D where all the matrices are 2million x 2million. I think for numpy that would basically do: tmp = B+C tmp2 = 2*D tmp3 = tmp - tmp2 A = tmp3 Allocating three huge tmp variables and probably doing an extra copy or two in there, when the best thing to do would be more like: A = D A *= -2 A += C A += B Or something like that. The point is that you give up any notion of having optimal code the minute you start using something like numpy. And you do so happily for the ability to get stuff done faster and have nicer looking, more readable code in the end. When everything works, that's when you hit the "go fast button" if you even really need to. --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at cox.net Fri Jul 7 02:53:29 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 06 Jul 2006 23:53:29 -0700 Subject: [Numpy-discussion] Enhancing dot() In-Reply-To: References: <44ADFE7C.5040506@cox.net> Message-ID: <44AE04E9.9040105@cox.net> Bill Baxter wrote: > On 7/7/06, *Tim Hochberg* > wrote: > > > The funny thing is that having a dot(a,b,c,...) would lead to the > > exact same kind of hidden performance problems you're arguing > against. > Not exactly arguing -- this isn't why I don't like H and friends > -- just > noting that this is one of the traps that people are likely to > fall into > when transferring equations to code. > > > There's a strong argument to be made that the whole design of most > array math packages is flawed and leads to inefficient code. The > classic example is something like: > A = B + C - 2*D > where all the matrices are 2million x 2million. I think for numpy > that would basically do: > tmp = B+C > tmp2 = 2*D > tmp3 = tmp - tmp2 > A = tmp3 > > Allocating three huge tmp variables and probably doing an extra copy > or two in there, when the best thing to do would be more like: > A = D > A *= -2 > A += C > A += B > > Or something like that. Yeah -- that's not quite right since you've clobbered B. But that is essentially the dark road that people go down in performance critical sections on occasion. > The point is that you give up any notion of having optimal code the > minute you start using something like numpy. And you do so happily > for the ability to get stuff done faster and have nicer looking, more > readable code in the end. When everything works, that's when you hit > the "go fast button" if you even really need to. That's what numexpr is for. A = numexpr.evaluate("B + C - 2*D") Evaluates this considerably faster than numpy and doesn't chew up all that extra memory. Actually that's why it's faster -- it's much more memory friendly. It's also a lot less flexible, although that's been improving. Still you want numpy as a base, but numexpr is a viable alternative for those critical sections where once you would resort to x= or worse, three argument ufuncs. -tim > > --bb From tim.hochberg at cox.net Fri Jul 7 03:24:46 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri, 07 Jul 2006 00:24:46 -0700 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44AD8A7C.5030408@cox.net> Message-ID: <44AE0C3E.5090508@cox.net> Bill Baxter wrote: > On 7/7/06, *Robert Kern* > wrote: > > Bill Baxter wrote: > > I am also curious, given the number of times I've heard this > nebulous > > argument of "there are lots kinds of numerical computing that don't > > invlolve linear algebra", that no one ever seems to name any of > these > > "lots of kinds". Statistics, maybe? But you can find lots of > linear > > algebra in statistics. > > That's because I'm not waving my hands at general fields of > application. I'm > talking about how people actually use array objects on a > line-by-line basis. If > I represent a dataset as an array and fit a nonlinear function to > that dataset, > am I using linear algebra at some level? Sure! Does having a .T > attribute on > that array help me at all? No. Arguing about how fundamental > linear algebra is > to numerical endeavors is entirely besides the point. > > > Ok. If line-by-line usage is what everyone really means, then I'll > get off the linear algebra soap box, but that's not what it sounded > like to me. > > So, if you want to talk line-by-line, I really can't talk about much > beside my own code. But I just grepped through it and out of 2445 > non-empty lines of code: > > 927 lines contain '=' > 390 lines contain a '[' > 75 lines contain matrix,asmatrix, or mat > ==> 47 lines contain a '.T' or '.transpose' of some sort. <== > 33 lines contain array, or asarray, or asanyarray > 24 lines contain 'rand(' --- I use it for generating bogus test data > a lot > 17 lines contain 'newaxis' or 'NewAxis' > 16 lines contain 'zeros(' > 13 lines contain 'dot(' > 12 lines contain 'empty(' > 8 lines contain 'ones(' > 7 lines contain 'inv(' In my main project theres about 26 KLOC (including blank lines), 700 or so of which use numeric (I prefix everything with np. so it's easy to count. Of those lines 29 use transpose, and of those 29 lines at most 9 could use a T attribute. It's probably far less than that since I didn't check the dimensionality of the arrays involved. Somewhere between 0 and 5 seems likely. > > I'm pretty new to numpy, so that's all the code I got right now. I'm > sure I've written many more lines of emails about numpy than I have > lines of actual numpy code. :-/ > > But from that, I can say that -- at least in my code -- transpose is > pretty common. If someone can point me to some larger codebases > written in numpy or numeric, I'd be happy to do a similar analysis of > those. > > > I'm not saying that people who do use arrays for linear algebra > are rare or > unimportant. It's that syntactical convenience for one set of > conventional ways > to use an array object, by itself, is not a good enough reason to > add stuff to > the core array object. > > > I wish I had a way to magically find out the distribution of array > dimensions used by all numpy and numeric code out there. My guess is > it would be something like 1-d: 50%, 2-d: 30%, 3-d: 10%, everything > else: 10%. I can't think of a good way to even get an estimate on > that. But in any event, I'm positive ndims==2 is a significant > percentage of all usages. It seems like the opponents to this idea > are suggesting the distribution is more flat than that. But whatever > the distribution is, it has to have a fairly light tail since memory > usage is exponential in ndim. If ndim == 20, then it takes 8 > megabytes just to store the smallest possible non-degenerate array of > float64s ( i.e. a 2x2x2x2x...) I would guess that it falls off fast after n=3, but that's just a guess. Personally, the majority of my code deals in 3D arrays (2x2xN and 4x4xN for the most part). These are arrays of vectors holding scattering data at N different frequency or time points. The 2D arrays that I do use are for rendering imaging (the actual rendering is done in C since Python wasn't fast enough and numpy wasn't really suitable for it). So, you see that for me at least, a T attribute is complete cruft. Useless for the 3D arrays, not needed for the 2D arrays, and again useless for the 1D arrays. I suspect that in general, the image processing types, who use a lot of 2D arrays, are probably not heavy users of transpose, but I'm not certain of that. > > It seems crazy to even be arguing this. Transposing is not some > specialized esoteric operation. It's important enough that R and S > give it a one letter function, and Matlab, Scilab, K all give it a > single-character operator. [*] Whoever designed the numpy.matrix > class also thought it was worthy of a shortcut, and I think came up > with a pretty good syntax for it. And the people who invented math > itself ?! > decided it was worth assigning a 1-character exponent to it. > > So I think there's a clear argument for having a .T attribute. But > ok, let's say you're right, and a lot of people won't use it. Fine. > IT WILL DO THEM ABSOLUTELY NO HARM. They don't have to use it if they > don't like it! Just ignore it. Unlike a t() function, .T doesn't > pollute any namespace users can define symbols in, so you really can > just ignore it if you're not interested in using it. It won't get in > your way. This is a completely bogus argument. All features cost -- good and ill alike. There's implementation cost and maintenance cost, both likely small in this case, but not zero. There's cognitive costs associated with trying to hold all of the various numpy methods, attributes and functions in ones head at once. There's pedagogical costs trying to explain how things fit together. There's community costs in that people who are allegedly coding with core numpy end up using mutually incomprehensible dialects. TANSTAFL. The ndarray object has far too many methods and attributes already IMO, and you have not made a case that I find convincing that this is important enough to further cruftify it. > > For the argument that ndarray should be pure like the driven snow, > just a raw container for n-dimensional data, Did anyone make that argument. No? I didn't think so. > I think that's what the basearray thing that goes into Python itself > should be. ndarray is part of numpy and numpy is for numerical > computing. And? Regards, -tim > > Regards, > --Bill > > [*] Full disclosure: I did find two counter-examples -- Maple and > Mathematica. Maple has only a transpose() function and Mathematica > has only Transpose[] (but you can use [esc]tr[esc] as a shortcut) > However, both of those packages are primarily known for their > _symbolic_ math capabilities, not their number crunching, so they less > are similar to numpy than R,S,K,Matlab and Scilab in that regard. > > ------------------------------------------------------------------------ > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From steffen.loeck at gmx.de Fri Jul 7 03:39:15 2006 From: steffen.loeck at gmx.de (Steffen Loeck) Date: Fri, 7 Jul 2006 09:39:15 +0200 Subject: [Numpy-discussion] Speed degression? In-Reply-To: <44AD8581.5030904@ee.byu.edu> References: <200607041519.24636.steffen.loeck@gmx.de> <44AD8581.5030904@ee.byu.edu> Message-ID: <200607070939.15600.steffen.loeck@gmx.de> On Thursday 06 July 2006, Travis Oliphant wrote: > This can test the speed of the implementations. With the latest modifications i get the following Results (in usec per loop): ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?sin-array ? ? ? ? ? ?mod-array ? use_scalarmath/use_pythonmath? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Numeric ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?134 ? ? ? ? ? ? ? ? ? 18 ? ? ? ? ? ?numpy 0.9.8 ? ? ? ? ? ? ? ? ? ? ? ? ? ? 97 ? ? ? ? ? ? ? ? ? 55 ?numpy 0.9.9.2764 ? ? ? ? ? ? ? ? ? ? ?114/78 ? ? ? ? ? ? ? ?55/21 ? ? ?numpy 0.9.8 + math ? ? ? ? ? ? ? ? ? ? ?38 ?numpy 0.9.9.2764 + math ? ? ? ? ? ? ? 38/23 ? ?Numeric + math ? ? ? ? ? ? ? ? ? ? ? ? ?23 The newest version seems to be faster then Numeric in all sin-array tests. Also the speed of mod-array has improved a lot. Many thanks, Steffen From jensj at fysik.dtu.dk Fri Jul 7 03:51:56 2006 From: jensj at fysik.dtu.dk (=?UTF-8?B?SmVucyBKw7hyZ2VuIE1vcnRlbnNlbg==?=) Date: Fri, 07 Jul 2006 09:51:56 +0200 Subject: [Numpy-discussion] Why is my array not contiguous? Message-ID: <44AE129C.4060909@servfys.fysik.dtu.dk> Hi! With numpy-0.9.9.2726, I do this: >>> x = arange(4) >>> y = x[newaxis, :] I would expect both arrays to be contiguous: >>> x.flags.contiguous, y.flags.contiguous (True, False) Shouldn't y be contiguous? Maybe it's because of the strange strides: >>> y.strides (0, 4) >>> y.strides = (16, 4) >>> y.flags.contiguous True Jens J?rgen From svetosch at gmx.net Fri Jul 7 04:40:36 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Fri, 07 Jul 2006 10:40:36 +0200 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE264.2030903@ieee.org> References: <44ADE264.2030903@ieee.org> Message-ID: <44AE1E04.6080409@gmx.net> Travis Oliphant schrieb: > > 1) .T Have some kind of .T attribute > +0 (the discussion in the .T thread convinced me it's better to keep the matrix playground as a separate subclass, and so it's not important for me what happens with pure arrays) > > 2) .H returns .T.conj() +0 > > > 3) .M returns matrix version of array +1 !! (Either that, or I will soon get on your nerves demanding matrix versions of ones, zeros, rand, eye etc.!) > > > 4) .A returns basearray (useful for sub-classes). +0 ? (Already exists for matrix, doesn't it?) -sven From schofield at ftw.at Fri Jul 7 05:06:00 2006 From: schofield at ftw.at (Ed Schofield) Date: Fri, 7 Jul 2006 11:06:00 +0200 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE264.2030903@ieee.org> References: <44ADE264.2030903@ieee.org> Message-ID: On 07/07/2006, at 6:26 AM, Travis Oliphant wrote: > 1) .T Have some kind of .T attribute -1 The meaning of .T is unclear for ndim != 2, as the plethora of sub- vote options shows. And we already have a matrix object designed explicitly for the case ndim == 2. Let's make matrix objects more useful, rather than give up on matrices and hard-wire 2d concepts into arrays. I'm willing to help with this, and I'll post some ideas to a different thread. > If >0 on this then: > > a) .T == .swapaxes(-2,-1) > > b) .T == .transpose() > > c) .T raises error for ndim > 2 > > d) .T returns (N,1) array for length (N,) array > > e) .T returns self for ndim < 2 (Just for the record, I think option (d) is positively evil.) > 2) .H returns .T.conj() -1 I'm against this for the same reasons I'm against .T, but more strongly, due to the extra memory allocation. > 3) .M returns matrix version of array -0 This would have the advantage of making .T and .H redundant for 2-d arrays, since they'd be accessible as matrix attributes anyway. But I don't see a strong enough case for this as an extra attribute, since asmatrix() and mat() are available anyway. I would like to deprecate mat() as an alias for matrix (by moving mat to the oldnumeric namespace); this would break the expectation that mat() perform a copy, and allow users to write >>> from numpy import asmatrix as mat instead if they need a short, simple syntax that avoids the copy overhead. It's also trivial for any serious user of matrices to write wrapper functions for ones, zeros, etc., such as def matones(*args): return asmatrix(ones(*args)) There might be an argument for including these in e.g. the numpy.matrix namespace. This would make a .M attribute even less compelling. > 4) .A returns basearray (useful for sub-classes). -0 This could be a good idea, but I'm not clear enough on its implications to vote either way. -- Ed From wbaxter at gmail.com Fri Jul 7 08:31:06 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 7 Jul 2006 21:31:06 +0900 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <44AE0C3E.5090508@cox.net> References: <44AD8A7C.5030408@cox.net> <44AE0C3E.5090508@cox.net> Message-ID: I think the thread to this point can be pretty much summarized by: while True: Bill: "2D transpose is common so it should have a nice syntax" Tim, Robert, Sasha, and Ed: "No it's not." Very well. I think it may be a self fulfilling prophecy, though. I.e. if matrix operations are cumbersome to use, then -- surprise surprise -- the large user base for matrix-like operations never materializes. Potential converts just give numpy the pass, and go to Octave or Scilab, or stick with Matlab, R or S instead. Why all the fuss about the .T? Because any changes to functions (like making ones() return a matrix) can easily be worked around on the user side, as has been pointed out. But as far as I know -- do correct me if I'm wrong -- there's no good way for a user to add an attribute to an existing class. After switching from matrices back to arrays, .T was the only thing I really missed from numpy.matrix. I would be all for a matrix class that was on equal footing with array and as easy to use as matrices in Matlab. But my experience using numpy.matrixwas far from that, and, given the lack of enthusiasm for matrices around here, that seems unlikely to change. However, I'm anxious to see what Ed has up his sleeves in the other thread. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Fri Jul 7 09:03:21 2006 From: bsouthey at gmail.com (Bruce Southey) Date: Fri, 7 Jul 2006 08:03:21 -0500 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE264.2030903@ieee.org> References: <44ADE264.2030903@ieee.org> Message-ID: On 7/6/06, Travis Oliphant wrote: > > This is a call for a vote on each of the math attributes. Please post > your vote as > > +1 : support > +0 : don't care so go ahead > -0 : don't care so why do it > -1 : against > > Vote on the following issues separately: > > > > 1) .T Have some kind of .T attribute -1 because I prefer what is already present although I would prefer the option of inplace or a taking a copy. Also I don't find .T particularly intuitive (for example, R and SAS IML have a t() function - SAS IML also uses a backtick but that has not been very portable). > > If >0 on this then: > > a) .T == .swapaxes(-2,-1) > > b) .T == .transpose() > > c) .T raises error for ndim > 2 > > d) .T returns (N,1) array for length (N,) array > > e) .T returns self for ndim < 2 > > > 2) .H returns .T.conj() -/+0 I don't care either way. > > > 3) .M returns matrix version of array -1 as I think asmatrix etc is clearer. > > > 4) .A returns basearray (useful for sub-classes). -1 as I think something like asarray etc is clearer and require explicit calling. > > > > -Travis > > > > > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From pjssilva at ime.usp.br Fri Jul 7 09:13:59 2006 From: pjssilva at ime.usp.br (Paulo Jose da Silva e Silva) Date: Fri, 07 Jul 2006 10:13:59 -0300 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE264.2030903@ieee.org> References: <44ADE264.2030903@ieee.org> Message-ID: <1152278040.5478.11.camel@localhost.localdomain> Em Qui, 2006-07-06 ?s 22:26 -0600, Travis Oliphant escreveu: > This is a call for a vote on each of the math attributes. Please post > your vote as > > +1 : support > +0 : don't care so go ahead > -0 : don't care so why do it > -1 : against > > Vote on the following issues separately: > > > > 1) .T Have some kind of .T attribute +1 > > If >0 on this then: > > a) .T == .swapaxes(-2,-1) > +0 > b) .T == .transpose() > -0 > c) .T raises error for ndim > 2 > +0 > d) .T returns (N,1) array for length (N,) array > +0 > e) .T returns self for ndim < 2 -1 > 2) .H returns .T.conj() > +0 > 3) .M returns matrix version of array > -1 > 4) .A returns basearray (useful for sub-classes). > +0 Paulo -- Paulo Jos? da Silva e Silva Professor Assistente do Dep. de Ci?ncia da Computa??o (Assistant Professor of the Computer Science Dept.) Universidade de S?o Paulo - Brazil e-mail: pjssilva at ime.usp.br Web: http://www.ime.usp.br/~pjssilva Teoria ? o que n?o entendemos o (Theory is something we don't) suficiente para chamar de pr?tica. (understand well enough to call) (practice) From schofield at ftw.at Fri Jul 7 09:49:41 2006 From: schofield at ftw.at (Ed Schofield) Date: Fri, 07 Jul 2006 15:49:41 +0200 Subject: [Numpy-discussion] What's wrong with matrices? Message-ID: <44AE6675.108@ftw.at> Bill Baxter wrote: > I think the thread to this point can be pretty much summarized by: > > while True: > Bill: "2D transpose is common so it should have a nice syntax" > Tim, Robert, Sasha, and Ed: "No it's not." > > Very well. I think it may be a self fulfilling prophecy, though. > I.e. if matrix operations are cumbersome to use, then -- surprise > surprise -- the large user base for matrix-like operations never > materializes. Potential converts just give numpy the pass, and go to > Octave or Scilab, or stick with Matlab, R or S instead. > > Why all the fuss about the .T? Because any changes to functions (like > making ones() return a matrix) can easily be worked around on the user > side, as has been pointed out. But as far as I know -- do correct me > if I'm wrong -- there's no good way for a user to add an attribute to > an existing class. After switching from matrices back to arrays, .T > was the only thing I really missed from numpy.matrix. > > I would be all for a matrix class that was on equal footing with array > and as easy to use as matrices in Matlab. But my experience using > numpy.matrix was far from that, and, given the lack of enthusiasm for > matrices around here, that seems unlikely to change. However, I'm > anxious to see what Ed has up his sleeves in the other thread. Okay ... ... let's make this the thread ;) I'd like to know why you, Sven, and anyone else on the list have gone back to using arrays after trying matrices. What was inconvenient about them? I'd like a nice juicy list. The whole purpose of the matrix class is to simplify 2d linear algebra. Where is it failing? I also went back to arrays after trying out matrices for some 2d linear algebra tasks, since I found that using matrices increased my code's complexity. I can describe the problems I had with them later, but first I'd like to hear of others' experiences. I'd like to help to make matrices more usable. Tell me what you want, and I'll work on some patches. -- Ed From rhetland at mac.com Fri Jul 7 09:57:12 2006 From: rhetland at mac.com (Robert Hetland) Date: Fri, 7 Jul 2006 09:57:12 -0400 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> Message-ID: <9BC1D67B-C7BB-41F9-B136-5ACD4604DDD5@mac.com> On Jul 6, 2006, at 2:54 PM, Robert Kern wrote: > I don't think that just because arrays are often used for linear > algebra that > linear algebra assumptions should be built in to the core array type. True. This argues against the MAH attributes. However, I use transpose often when not dealing with linear algebra, in particular with reading in data, and putting various columns into variables. Also, occasional in plotting (which expects things in 'backward' order relative to x-y space), and communicating between fortran programs (which typically use 'forward' order (x, y, z)) and numpy (backward -- (z, x, y)). I am very much in favor of .T, but it should be a full .transpose(), not just swap the last two axes. I don't care so much for the others. +1 for .T == .transpose() -Rob -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhetland at mac.com Fri Jul 7 10:06:32 2006 From: rhetland at mac.com (Robert Hetland) Date: Fri, 7 Jul 2006 10:06:32 -0400 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE264.2030903@ieee.org> References: <44ADE264.2030903@ieee.org> Message-ID: <37D34CB6-6BFE-4A6D-A8DD-1003C4440A96@mac.com> +1 1b) .T == .transpose() -1 all others On Jul 7, 2006, at 12:26 AM, Travis Oliphant wrote: > > This is a call for a vote on each of the math attributes. Please post > your vote as > > +1 : support > +0 : don't care so go ahead > -0 : don't care so why do it > -1 : against > > Vote on the following issues separately: > > > > 1) .T Have some kind of .T attribute > > If >0 on this then: > > a) .T == .swapaxes(-2,-1) > > b) .T == .transpose() > > c) .T raises error for ndim > 2 > > d) .T returns (N,1) array for length (N,) array > > e) .T returns self for ndim < 2 > > > 2) .H returns .T.conj() > > > 3) .M returns matrix version of array > > > 4) .A returns basearray (useful for sub-classes). > > > > -Travis > > > > > > Using Tomcat but need to do more? Need to support web services, > security? > Get stuff done quickly with pre-integrated technology to make your > job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel? > cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From stefan at sun.ac.za Fri Jul 7 10:06:57 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 7 Jul 2006 16:06:57 +0200 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE264.2030903@ieee.org> References: <44ADE264.2030903@ieee.org> Message-ID: <20060707140657.GE30177@mentat.za.net> On Thu, Jul 06, 2006 at 10:26:12PM -0600, Travis Oliphant wrote: > 1) .T Have some kind of .T attribute -1, since the expected behaviour of .T is different depending on problem context. > a) .T == .swapaxes(-2,-1) The fact that this was proposed just demonstrates the fact. If you have a (10,5) array, .T gives the expected (5,10) array. But if you have a (10,10,3) RGB image as an ndarray, .T gives a (10,3,10) array... which is certainly not what you'd expect in an image processing context. St?fan From pebarrett at gmail.com Fri Jul 7 10:22:47 2006 From: pebarrett at gmail.com (Paul Barrett) Date: Fri, 7 Jul 2006 10:22:47 -0400 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE264.2030903@ieee.org> References: <44ADE264.2030903@ieee.org> Message-ID: <40e64fa20607070722v4c00b769sec4c33b1b7a5f278@mail.gmail.com> On 7/7/06, Travis Oliphant wrote: > > This is a call for a vote on each of the math attributes. Please post > your vote as > > +1 : support > +0 : don't care so go ahead > -0 : don't care so why do it > -1 : against > > Vote on the following issues separately: > > > > 1) .T Have some kind of .T attribute -1 > If >0 on this then: > > a) .T == .swapaxes(-2,-1) > > b) .T == .transpose() > > c) .T raises error for ndim > 2 > > d) .T returns (N,1) array for length (N,) array > > e) .T returns self for ndim < 2 > > > 2) .H returns .T.conj() -1 > 3) .M returns matrix version of array -1 > 4) .A returns basearray (useful for sub-classes). -1 From gnurser at googlemail.com Fri Jul 7 10:26:41 2006 From: gnurser at googlemail.com (George Nurser) Date: Fri, 7 Jul 2006 15:26:41 +0100 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <9BC1D67B-C7BB-41F9-B136-5ACD4604DDD5@mac.com> References: <44ACF15D.6030607@ieee.org> <9BC1D67B-C7BB-41F9-B136-5ACD4604DDD5@mac.com> Message-ID: <1d1e6ea70607070726u6e6d8e33v26765b5a6f6c9cd3@mail.gmail.com> On 07/07/06, Robert Hetland wrote: [snip] > However, I use transpose often when not dealing with linear algebra, in > particular with reading in data, and putting various columns into > variables. Also, occasional in plotting (which expects things in 'backward' > order relative to x-y space), and communicating between fortran programs > (which typically use 'forward' order (x, y, z)) and numpy (backward -- (z, > x, y)). > This is my usage as well. Also my primitive knowledge of numpy requires use of the transpose when iterating over indexes from where. Moreover I think the notation .T is perfectly reasonable. So I agree with: > I am very much in favor of .T, but it should be a full .transpose(), not > just swap the last two axes. I don't care so much for the others. +1 for .T == .transpose() George Nurser. From gnurser at googlemail.com Fri Jul 7 10:31:02 2006 From: gnurser at googlemail.com (George Nurser) Date: Fri, 7 Jul 2006 15:31:02 +0100 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE264.2030903@ieee.org> References: <44ADE264.2030903@ieee.org> Message-ID: <1d1e6ea70607070731n7f83b303n4ef67482e6dfbbaf@mail.gmail.com> On 07/07/06, Travis Oliphant wrote: > > This is a call for a vote on each of the math attributes. Please post > your vote as > > +1 : support > +0 : don't care so go ahead > -0 : don't care so why do it > -1 : against > > Vote on the following issues separately: > > > > 1) .T Have some kind of .T attribute +1 > > If >0 on this then: > > a) .T == .swapaxes(-2,-1) -1 > > b) .T == .transpose() > > c) .T raises error for ndim > 2 -1 > > d) .T returns (N,1) array for length (N,) array -1 > > e) .T returns self for ndim < 2 +1 > > > 2) .H returns .T.conj() -1 > > > 3) .M returns matrix version of array -1 > > > 4) .A returns basearray (useful for sub-classes). -0 George. From gruben at bigpond.net.au Fri Jul 7 10:45:26 2006 From: gruben at bigpond.net.au (Gary Ruben) Date: Sat, 08 Jul 2006 00:45:26 +1000 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44AD8A7C.5030408@cox.net> Message-ID: <44AE7386.9000007@bigpond.net.au> Sasha wrote: > On 7/6/06, Bill Baxter wrote: >> ... >> Yep, like Tim said. The usage is say a N sets of basis vectors. Each set >> of basis vectors is a matrix. > > This brings up a feature that I really miss from numpy: an ability to do > > array([f(x) for x in a]) > > without python overhead. I'd find this really useful too. I'm doing lots of this in my recent code: array([f(x,y) for x in a for y in b]).reshape(xxx) Gary R. From wbaxter at gmail.com Fri Jul 7 11:07:58 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 8 Jul 2006 00:07:58 +0900 Subject: [Numpy-discussion] What's wrong with matrices? In-Reply-To: <44AE6675.108@ftw.at> References: <44AE6675.108@ftw.at> Message-ID: On 7/7/06, Ed Schofield wrote: > > Bill Baxter wrote: > > I would be all for a matrix class that was on equal footing with array > > and as easy to use as matrices in Matlab. But my experience using > > numpy.matrix was far from that, and, given the lack of enthusiasm for > > matrices around here, that seems unlikely to change. However, I'm > > anxious to see what Ed has up his sleeves in the other thread. > > Okay ... ... let's make this the thread ;) > I'd like to know why you, Sven, and anyone else on the list have gone > back to using arrays after trying matrices. What was inconvenient about > them? I'd like a nice juicy list. The whole purpose of the matrix > class is to simplify 2d linear algebra. Where is it failing? Okay, here are a few that come to mind. 1) Functions that take a matrix but return an array. Maybe these are all fixed now. But they better be fixed not just in numpy but in scipy too. To me this implies there needs to be some standard idiom for how to write a generic array-protocol-using function so that you don't have to think about it. 2) At the time I was using matrix, scalar * matrix was broken. Fixed now, but that kind of thing just shouldn't happen. There should be a tests for basic operations like that if there aren't already. 3) mat() doesn't make sense as a shortcut for matrix construction. It only saves 3 letters over typing matrix(), and asmatrix is generally more useful. So mat() should be a synonym for asmatrix() 4) eye,empty,rand,ones,zeros,arange and anything else that builds an array from scratch or from a python list should have a matrix equivalent. 5) I've got squeezes like crazy all over my matrix-using code. Maybe this was a bug in 0.9.5 or so that's been fixed? I do seem to recall some problem with indexing or c_ or something that was causing matrices to grow extra levels of length 1 axes. Again, like the scalar*matrix bug, things like that shouldn't happen. 6) No good way to do elementwise operations? Sometimes you just want to do an elementwise mult or divide or exponentiation. I guess you're supposed to do Z = asmatrix(X.A * Y.A). Yah, right. 7) Finally, once all that is fixed, I find the slavish adherance to ndim=2 to be too restrictive. a) Sometimes it's useful to have a stack of matrices. Pretty often, in fact, for me. I guess I could use a python list of matrices or an object array of matrix or something, but I also think there are times when it's useful to treat different pairs of axes as the 'matrix' part. So I'd like matrices to be able to have ndim>2. b) On the other end, I think ndim<2 is useful sometimes too. Take a function like mean(), for example. With no arguments the return value is a 1x1 matrix (as opposed to a scalar). That just looks silly. And it doesn't work everywhere a scalar would, e.g. with a python built-in like round(). Or take indexing. It seems odd to me that where() reurns a tuple of shape==(1,N) objects instead of just (N,) . Maybe I can get over that though, as long as it works for indexing (which it seems it does). But I think the scalar return case is a real issue. Here's another: sum(). For an array you can do sum(sum(a)) and get a scalar if a is 2-d, but for matrix sum(sum(m)) is the same as sum(m). And along these lines, m[newaxis] just silently doesn't do anything. That doesn't seem right. That's all I can think of for now. Beyond that, looming in the back of my mind there's the question of what happens when we decide we want to have sparse matrices? Or masked sparse matrices, for that matter. These specialty array objects all seem to me like they should be some sort of mix-ins. The features they provide are (conceptually, anyway) independent. Concrete sibling classes doesn't really seem like the right inheritance relationship. > I'd like to help to make matrices more usable. Tell me what you want, > and I'll work on some patches. Thanks for giving it a shot. I'm currently trying to compile numpy from svn on Win32 for the first time myself. If it goes well I can probably help run tests or maybe even on some coding. But I still have yet to get past ATLAS. --Bill -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.taylor at stanford.edu Fri Jul 7 11:18:25 2006 From: jonathan.taylor at stanford.edu (Jonathan Taylor) Date: Fri, 07 Jul 2006 08:18:25 -0700 Subject: [Numpy-discussion] numpy book Message-ID: <44AE7B41.3020703@stanford.edu> I bought a copy of the numpy book sometime in January which I have since lost in a careless mistake. Is there a way to get another (updated) copy? I couldn't see how this was done on the website. Thanks, Jonathan -- ------------------------------------------------------------------------ Jonathan Taylor Tel: 650.723.9230 Dept. of Statistics Fax: 650.725.8977 Sequoia Hall, 137 www-stat.stanford.edu/~jtaylo 390 Serra Mall Stanford, CA 94305 From charlesr.harris at gmail.com Fri Jul 7 11:20:42 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 7 Jul 2006 09:20:42 -0600 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <1d1e6ea70607070731n7f83b303n4ef67482e6dfbbaf@mail.gmail.com> References: <44ADE264.2030903@ieee.org> <1d1e6ea70607070731n7f83b303n4ef67482e6dfbbaf@mail.gmail.com> Message-ID: Does anyone use transpose for anything besides two dimensional arrays? For arrays that aren't matrices at heart the meaning of transpose is pretty arbitrary, so the only virtue I see in having an attribute would be less typeing. So put it in the documentation. As an aside, being able to specify axis in dot might obviate many cases of *.T and *.H and make inner a special case. > > > 1) .T Have some kind of .T attribute +1 > > > If >0 on this then: > > > > a) .T == .swapaxes(-2,-1) -1 > b) .T == .transpose() +1 > c) .T raises error for ndim > 2 -1 > > > d) .T returns (N,1) array for length (N,) array -1 > e) .T returns self for ndim < 2 +1 > > > 2) .H returns .T.conj() +1 > > > 3) .M returns matrix version of array no opinion. > > > 4) .A returns basearray (useful for sub-classes). no opinion Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin.wiechert at gmx.de Fri Jul 7 11:33:54 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Fri, 7 Jul 2006 17:33:54 +0200 Subject: [Numpy-discussion] ix_ and boolean indices Message-ID: <200607071733.54278.martin.wiechert@gmx.de> Hi all, for me M [ix_(I, J)] does not work if I, J are boolean arrays. Is this intended or a bug/missing feature? And is there a way (other than I = where (I) [0] etc.) to make it work? Thanks, Martin From david.huard at gmail.com Fri Jul 7 11:39:26 2006 From: david.huard at gmail.com (David Huard) Date: Fri, 7 Jul 2006 11:39:26 -0400 Subject: [Numpy-discussion] Missing docstrings Message-ID: <91cf711d0607070839t8f57aedoa4aa002aeddaaa70@mail.gmail.com> Hi, For the beta release, it would be nice if every function had a docstring, even a small one. See Ticket #174 Missing docstrings: In [116]: ['issubsctype', 'unicode_', 'string', 'float96', 'pkgload', 'void', 'unicode0', 'void0', 'object0', 'memmap', 'nan_to_num', 'PackageLoader', 'object_', 'dtype', 'unsignedinteger', 'uintc', 'uint0', 'uint8', 'chararray', 'uint64', 'finfo', 'add_newdoc', 'array_repr', 'array_str', 'longlong', 'int16', 'mat', 'uint', 'correlate', 'int64', 'choose', 'complexfloating', 'recarray', 'mean', 'str_', 'ulonglong', 'matrix', 'uint32', 'byte', 'ctypes_load_library', 'signedinteger', 'ndim', 'number', 'bool8', 'msort', 'bool_', 'inexact', 'broadcast', 'short', 'ubyte', 'std', 'double', 'require', 'take', 'issubclass_', 'longfloat', 'deprecate', 'bincount', 'array2string', 'float64', 'ushort', 'float_', 'geterrobj', 'iterable', 'intp', 'flexible', 'sctype2char', 'longdouble', 'flatiter', 'generic', 'show_config', 'i0', 'uintp', 'character', 'uint16', 'float32', 'int32', 'integer', 'get_printoptions', 'seterrobj', 'add_docstring', 'intc', 'var', 'int_', 'histogram', 'issubdtype', 'int0', 'int8', 'record', 'obj2sctype', 'single', 'floating', 'test', 'string0'] In the process of adding docstrings, I noted some slight bugs: bincount returns an error if the input array mixes float and ints, while it accepts lists with mixed floats and ints. histogram does not deal gracefully with the range attribute. All values greater than range[1] are put in the last bin, which is probably not the expected behavior. Cheers, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Fri Jul 7 11:45:17 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 7 Jul 2006 08:45:17 -0700 Subject: [Numpy-discussion] What's wrong with matrices? In-Reply-To: <44AE6675.108@ftw.at> References: <44AE6675.108@ftw.at> Message-ID: On 7/7/06, Ed Schofield wrote: > I'd like to help to make matrices more usable. Tell me what you want, > and I'll work on some patches. I can't pass up an offer like that. DIAG diag(M) returns an array. It would be nice if diag(M) returned asmatrix(diag(M)).T. It would also be nice if you can construct a diagonal matrix directly from what is returned from diag(M). But right now you can't: >> x matrix([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) >> diag(x) array([0, 4, 8]) >> d = asmatrix(diag(x)).T >> d matrix([[0], [4], [8]]) >> diag(d) array([0]) <-- this should be a 3x3 matrix MATRIX CONSTRUCTION Making it easier to construct matrices would be a big help. Could the following function be made to return matrices? ones, zeros, rand, randn, eye, linspace, empty I guess the big decision is how to tell numpy to use matrices. How about from numpy.matrix import ones, zeros ? I would prefer something even more global. Something that acts like the global variable 'usematrix = True'. Once that is declared, the default changes from array to matrix. INDEXING >> x matrix([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) >> y matrix([[0], [1], [2]]) >> x[y>1,:] matrix([[6]]) This is a big one for me. If x[y>1,:] returned >> x[2,:] matrix([[6, 7, 8]]) then there would no longer be a need for array :) From charlesr.harris at gmail.com Fri Jul 7 11:48:15 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 7 Jul 2006 09:48:15 -0600 Subject: [Numpy-discussion] Enhancing dot() In-Reply-To: References: <44ADFE7C.5040506@cox.net> Message-ID: On 7/7/06, Bill Baxter wrote: > > On 7/7/06, Tim Hochberg wrote: > > > > The funny thing is that having a dot(a,b,c,...) would lead to the > > > exact same kind of hidden performance problems you're arguing against. > > Not exactly arguing -- this isn't why I don't like H and friends -- just > > > > noting that this is one of the traps that people are likely to fall into > > when transferring equations to code. > > A = D > A *= -2 > A += C > A += B > I would like to write something like: A = D.copy().times(-2).plus(C).plus(B) i.e. copy produces a "register", the rest is reverse Polish, and = "stores" the result. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pau.gargallo at gmail.com Fri Jul 7 11:51:04 2006 From: pau.gargallo at gmail.com (Pau Gargallo) Date: Fri, 7 Jul 2006 17:51:04 +0200 Subject: [Numpy-discussion] ix_ and boolean indices In-Reply-To: <200607071733.54278.martin.wiechert@gmx.de> References: <200607071733.54278.martin.wiechert@gmx.de> Message-ID: <6ef8f3380607070851j2bbe48f6meacb67e8e9195dd3@mail.gmail.com> On 7/7/06, Martin Wiechert wrote: > Hi all, > > for me > > M [ix_(I, J)] > > does not work if I, J are boolean arrays. Is this intended or a bug/missing > feature? > > And is there a way (other than I = where (I) [0] etc.) to make it work? > > Thanks, > Martin it is a recent feature. It works for me on version '0.9.9.2660' pau >>> import numpy >>> numpy.__version__ '0.9.9.2660' >>> a = numpy.rand(3,4) >>> a array([[ 0.24347161, 0.25636386, 0.64373189, 0.82730095], [ 0.02062571, 0.12244009, 0.60053928, 0.10624435], [ 0.75472591, 0.00614411, 0.75388955, 0.40481918]]) >>> I = a[:,0]<0.5 >>> J = a[0,:]<0.5 >>> a[numpy.ix_(I,J)] array([[ 0.24347161, 0.25636386], [ 0.02062571, 0.12244009]]) From Chris.Barker at noaa.gov Fri Jul 7 14:02:21 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 07 Jul 2006 11:02:21 -0700 Subject: [Numpy-discussion] Numpy import problem In-Reply-To: <20060707130305.2svvwr0wuntgkosk@webmail.auckland.ac.nz> References: <20060706170241.hq0fq9gssdys8kg4@webmail.auckland.ac.nz> <44AD3D73.8080602@noaa.gov> <20060707130305.2svvwr0wuntgkosk@webmail.auckland.ac.nz> Message-ID: <44AEA1AD.6060605@noaa.gov> Satellite Data Research Group wrote: > Quoting Christopher Barker : >> Which Python 2.4.1 are you using? It would be great if you would give >> the Python2.4.3 version found here a try: >> >> http://www.pythonmac.org/packages/py24-fat/index.html > Thanks for that, python 2.4.3 works perfectly. > Cheers, Joe Corbett There is now a Numpy0.9.8 package on that page too. Please let me know (or write to the macpython list) if you have a problem with it. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant.travis at ieee.org Fri Jul 7 13:42:35 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 07 Jul 2006 11:42:35 -0600 Subject: [Numpy-discussion] ix_ and boolean indices In-Reply-To: <200607071733.54278.martin.wiechert@gmx.de> References: <200607071733.54278.martin.wiechert@gmx.de> Message-ID: <44AE9D0B.3060909@ieee.org> Martin Wiechert wrote: > Hi all, > > for me > > M [ix_(I, J)] > > does not work if I, J are boolean arrays. Is this intended or a bug/missing > feature? > Which version? Using boolean arrays as separate indices was a recent feature. You have to get SVN to use it. -Travis From Chris.Barker at noaa.gov Fri Jul 7 14:11:22 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 07 Jul 2006 11:11:22 -0700 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44AD8A7C.5030408@cox.net> Message-ID: <44AEA3CA.7060207@noaa.gov> Robert Kern wrote: > Just > because linear algebra is "the base" for a lot of numerical computing does not > mean that everyone is using numpy arrays for linear algebra all the time. Much > less does it mean that all of those conventions you've devised should be shoved > into the core array type. I totally agree here. What bugged me most about MATLAB was that it was so darn Matrix/Linear Algebra centric. Yes, much of the code I wrote used linear algebra, but mostly it was a tiny (though critical) part of the actual code: Lots of code to set up a matrix equation, then solve it. The solve it was one line of code. For the rest, I prefer an array approach. A Matrix/Linear Algebra centric approach is good for some things, but I think it should be all or nothing. If you want it, then there should be a Matrix package, that includes the Matrix object, AND a matrix version of all the utility functions, like ones, zeros, etc. So all you would have to do is do: from numpy.matrix import * instead of from numpy import * and you'd get all the same stuff. Most of what would need to be added to the matrix package would be pretty easy, boiler plate code. Then we'd need a bunch more testing to root out all the operations that returned arrays where they should return matrices. If there is no one that wants to do all that work, then we have our answer. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant.travis at ieee.org Fri Jul 7 14:13:14 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 07 Jul 2006 12:13:14 -0600 Subject: [Numpy-discussion] What's wrong with matrices? In-Reply-To: References: <44AE6675.108@ftw.at> Message-ID: <44AEA43A.9020904@ieee.org> Bill Baxter wrote: > On 7/7/06, *Ed Schofield* > > wrote: > > Okay ... ... let's make this the thread ;) > I'd like to know why you, Sven, and anyone else on the list have gone > back to using arrays after trying matrices. What was inconvenient > about > them? I'd like a nice juicy list. The whole purpose of the matrix > class is to simplify 2d linear algebra. Where is it failing? > > > Okay, here are a few that come to mind. > 1) Functions that take a matrix but return an array. Maybe these are > all fixed now. But they better be fixed not just in numpy but in > scipy too. To me this implies there needs to be some standard idiom > for how to write a generic array-protocol-using function so that you > don't have to think about it. A lot of these are fixed. The mechanism for handling this is in-place: either using asanyarray in the function or (more generally) using a decorator that wraps the arguments with asarray and returns the output with __array_wrap__. But, we need people to help with fleshing it out. > > 2) At the time I was using matrix, scalar * matrix was broken. Fixed > now, but that kind of thing just shouldn't happen. There should be a > tests for basic operations like that if there aren't already. We need people to write and implement the tests. It's one way everybody can contribute. I do use matrices occasionally (not never as has been implied). But, I do more coding than linear algebra (particularly with images), therefore my need for matrix math is reduced. Nonetheless, I've been very willing to make well-defined changes that are needed. Most problems cannot be found out without people who use and test the code. > > 3) mat() doesn't make sense as a shortcut for matrix construction. It > only saves 3 letters over typing matrix(), and asmatrix is generally > more useful. So mat() should be a synonym for asmatrix() I'd be willing to make that change, but it will break some people's SciPy code. > > 4) eye,empty,rand,ones,zeros,arange and anything else that builds an > array from scratch or from a python list should have a matrix equivalent Would from numpy.defmatrix import ones, zeros, ... work? > > 5) I've got squeezes like crazy all over my matrix-using code. Maybe > this was a bug in 0.9.5 or so that's been fixed? I do seem to recall > some problem with indexing or c_ or something that was causing > matrices to grow extra levels of length 1 axes. Again, like the > scalar*matrix bug, things like that shouldn't happen. Sure, but it's going to happen in a beta release... That's why we need testers. As I recall, most bugs with matrices have been fixed fairly quickly as soon as they are reported. > > 6) No good way to do elementwise operations? Sometimes you just want > to do an elementwise mult or divide or exponentiation. I guess you're > supposed to do Z = asmatrix(X.A * Y.A). Yah, right. This is a problem with a dearth of infix operators. In fact, if we had a good way to write matrix multiplication as an infix operator, perhaps there wouldn't be any need for matrices. I'm really not sure how to fix the problem (the .M attribute of arrays was an attempt to make it easier): (X.A * Y.A).M But, there is always multiply(X,Y) > > 7) Finally, once all that is fixed, I find the slavish adherance to > ndim=2 to be too restrictive. > a) Sometimes it's useful to have a stack of matrices. Pretty > often, in fact, for me. I guess I could use a python list of matrices > or an object array of matrix or something, but I also think there are > times when it's useful to treat different pairs of axes as the > 'matrix' part. So I'd like matrices to be able to have ndim>2. I suppose this restriction could be lifted. > b) On the other end, I think ndim<2 is useful sometimes too. Take > a function like mean(), for example. With no arguments the return > value is a 1x1 matrix (as opposed to a scalar). Have you checked lately. It's a scalar now... This has been fixed. > Or take indexing. It seems odd to me that where() reurns a tuple of > shape==(1,N) objects instead of just (N,) . The way to fix some of these is to return arrays for indexing instead of allowing matrices. But, matrices that are less than 2-d just don't make sense. > Maybe I can get over that though, as long as it works for indexing > (which it seems it does). But I think the scalar return case is a > real issue. Here's another: sum(). For an array you can do > sum(sum(a)) and get a scalar if a is 2-d, but for matrix sum(sum(m)) > is the same as sum(m). And along these lines, m[newaxis] just > silently doesn't do anything. That doesn't seem right. These are just semantic questions. It's no surprise that sum(sum(m)) returns the same as sum(m) for a matrix because summing over the same axis won't change the result. You have to sum over both axes in a matrix. Thanks for the feedback. -Travis From mark at mitre.org Fri Jul 7 14:36:18 2006 From: mark at mitre.org (Mark Heslep) Date: Fri, 07 Jul 2006 14:36:18 -0400 Subject: [Numpy-discussion] Numpy subversion compatibility with Scipy release? Message-ID: <44AEA9A2.7060308@mitre.org> Is there any general sync point with development Numpy from subversion and the SciPy releases? Ive got Numpy 0.9.9.2761 and Scipy 0.4.9 installed with (I believe several) breakages, in particular: > In [8]: from scipy import special ... > /usr/lib/python2.4/site-packages/scipy/linalg/basic.py > 20 > conjugate,ravel,r_,mgrid,take,ones,dot,transpose,sqrt,add,real > 21 import numpy > ---> 22 from numpy import asarray_chkfinite, outerproduct, > concatenate, reshape, single > 23 from numpy import matrix as Matrix > 24 import calc_lwork > > ImportError: cannot import name outerproduct I suppose Scipy is not picking up the deprecation of outerproduct. No surprise that bleeding edge Numpy subversion doesn't play with a SciPy release; I was just wondering if there generally used/known way to make it happen. Do I need to fall back to a Numpy release? Or move forward on SciPy? Mark From robert.kern at gmail.com Fri Jul 7 14:46:04 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 07 Jul 2006 13:46:04 -0500 Subject: [Numpy-discussion] Numpy subversion compatibility with Scipy release? In-Reply-To: <44AEA9A2.7060308@mitre.org> References: <44AEA9A2.7060308@mitre.org> Message-ID: Mark Heslep wrote: > Is there any general sync point with development Numpy from subversion > and the SciPy releases? Ive got Numpy 0.9.9.2761 and Scipy 0.4.9 > installed with (I believe several) breakages, in particular: > >> In [8]: from scipy import special > ... >> /usr/lib/python2.4/site-packages/scipy/linalg/basic.py >> 20 >> conjugate,ravel,r_,mgrid,take,ones,dot,transpose,sqrt,add,real >> 21 import numpy >> ---> 22 from numpy import asarray_chkfinite, outerproduct, >> concatenate, reshape, single >> 23 from numpy import matrix as Matrix >> 24 import calc_lwork >> >> ImportError: cannot import name outerproduct > I suppose Scipy is not picking up the deprecation of outerproduct. No > surprise that bleeding edge Numpy subversion doesn't play with a SciPy > release; I was just wondering if there generally used/known way to make > it happen. Do I need to fall back to a Numpy release? Or move forward > on SciPy? The releases are synched such that the numpy version is "twice" (by an idiosyncratic form of arithmetic) that of the scipy version. numpy 0.9.8 <-> scipy 0.4.9 . The latest SVN revisions should match (it's a bug, otherwise). numpy 2761 is recent enough that an SVN checkout of scipy will probably be fine. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david.huard at gmail.com Fri Jul 7 14:46:53 2006 From: david.huard at gmail.com (David Huard) Date: Fri, 07 Jul 2006 14:46:53 -0400 Subject: [Numpy-discussion] Missing docstrings Message-ID: <1152298014.5062.21.camel@localhost> Hi, For the first release, it would be nice if every function had a docstring, even a small one. There are 279 callable items in the numpy namespace, and 94 of those lack a docstring, albeit most of those probably don't get much usage. To help the process, I filed Ticket #174 and attached a couple of docstrings, feel free to add your own. Missing docstrings: ['issubsctype', 'unicode_', 'string', 'float96', 'pkgload', 'void', 'unicode0', 'void0', 'object0', 'memmap', 'nan_to_num', 'PackageLoader', 'object_', 'dtype', 'unsignedinteger', 'uintc', 'uint0', 'uint8', 'chararray', 'uint64', 'finfo', 'add_newdoc', 'array_repr', 'array_str', 'longlong', 'int16', 'mat', 'uint', 'correlate', 'int64', 'choose', 'complexfloating', 'recarray', 'mean', 'str_', 'ulonglong', 'matrix', 'uint32', 'byte', 'ctypes_load_library', 'signedinteger', 'ndim', 'number', 'bool8', 'msort', 'bool_', 'inexact', 'broadcast', 'short', 'ubyte', 'std', 'double', 'require', 'take', 'issubclass_', 'longfloat', 'deprecate', 'bincount', 'array2string', 'float64', 'ushort', 'float_', 'geterrobj', 'iterable', 'intp', 'flexible', 'sctype2char', 'longdouble', 'flatiter', 'generic', 'show_config', 'i0', 'uintp', 'character', 'uint16', 'float32', 'int32', 'integer', 'get_printoptions', 'seterrobj', 'add_docstring', 'intc', 'var', 'int_', 'histogram', 'issubdtype', 'int0', 'int8', 'record', 'obj2sctype', 'single', 'floating', 'test', 'string0'] Cheers, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark at mitre.org Fri Jul 7 15:42:01 2006 From: mark at mitre.org (Mark Heslep) Date: Fri, 07 Jul 2006 15:42:01 -0400 Subject: [Numpy-discussion] OptimizationDemo [was: Numpy subversion compatibility with Scipy release? ] In-Reply-To: References: <44AEA9A2.7060308@mitre.org> Message-ID: <44AEB909.6040206@mitre.org> Robert Kern wrote: > > The latest SVN revisions should match (it's a bug, otherwise). numpy 2761 is > recent enough that an SVN checkout of scipy will probably be fine. > Yes SVN scipy/numpy fix the problem. Sorry, I missed Travis's post on scipy-user here: http://projects.scipy.org/pipermail/scipy-user/2006-June/008438.html to that effect. I should just get specific: I was simply trying to run the basic optimization demo on the front page of the wiki as check on SVN (http://www.scipy.org/Cookbook/OptimizationDemo1) and it appears SVN scipy/numpy breaks matplotlib -0.87.2-1 (fc5) in the demo. Do I need SVN matplotlib as well? Mark From the Opt. Demo: > In [1]: from scipy import arange, special, optimize > In [2]: x = arange(0,10,0.01) > In [4]: for k in arange(0.5,5.5): > ...: y = special.jv(k,x) > ...: plot(x,y) > ...: > --------------------------------------------------------------------------- > exceptions.ZeroDivisionError Traceback (most recent call ... ... > /usr/lib/python2.4/site-packages/matplotlib/ticker.py in > bin_boundaries(self, vmin, vmax) > 766 def bin_boundaries(self, vmin, vmax): > 767 nbins = self._nbins > --> 768 scale, offset = scale_range(vmin, vmax, nbins) > 769 vmin -= offset > 770 vmax -= offset > > /usr/lib/python2.4/site-packages/matplotlib/ticker.py in > scale_range(vmin, vmax, n, threshold) > 731 dv = abs(vmax - vmin) > 732 meanv = 0.5*(vmax+vmin) > --> 733 var = dv/max(abs(vmin), abs(vmax)) > 734 if var < 1e-12: > 735 return 1.0, 0.0 > > ZeroDivisionError: float division From robert.kern at gmail.com Fri Jul 7 15:56:36 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 07 Jul 2006 14:56:36 -0500 Subject: [Numpy-discussion] OptimizationDemo [was: Numpy subversion compatibility with Scipy release? ] In-Reply-To: <44AEB909.6040206@mitre.org> References: <44AEA9A2.7060308@mitre.org> <44AEB909.6040206@mitre.org> Message-ID: Mark Heslep wrote: > Robert Kern wrote: >> The latest SVN revisions should match (it's a bug, otherwise). numpy 2761 is >> recent enough that an SVN checkout of scipy will probably be fine. >> > Yes SVN scipy/numpy fix the problem. Sorry, I missed Travis's post on > scipy-user here: > http://projects.scipy.org/pipermail/scipy-user/2006-June/008438.html > to that effect. > > I should just get specific: I was simply trying to run the basic > optimization demo on the front page of the wiki as check on SVN > (http://www.scipy.org/Cookbook/OptimizationDemo1) and it appears SVN > scipy/numpy breaks matplotlib -0.87.2-1 (fc5) in the demo. Do I need SVN > matplotlib as well? Yes, you do, thanks to the removal of the Numeric typecodes (Int32, etc.). However, the error you get looks to be unrelated. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at ee.byu.edu Fri Jul 7 16:21:44 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 07 Jul 2006 14:21:44 -0600 Subject: [Numpy-discussion] .M .A .T .H attribute result Message-ID: <44AEC258.9040800@ee.byu.edu> I didn't compile the results, but the discussion on the idea of adding new attributes to the array object led to the following result. Added: .T attribute to mean self.transpose() .T This was rather controversial with many possibilities emerging. In the end, I think the common case of going back and forth between C-order and Fortran-order codes in a wide variety of settings convinced me to make .T a short-hand for .transpose() and add it as an attribute. This is now the behavior in SVN. Right now, for self.ndim < 2, this just returns a new reference to self (perhaps it should return a new view instead). .M While some were in favor, too many people opposed this (although the circular reference argument was not convincing). Instead a numpy.matlib module was started to store matrix versions of the standard array-creation functions and mat was re-labeled to "asmatrix" so that a copy is not made by default. .A A few were in favor, but as this is just syntactic sugar for .__array__() or asarray(obj) or .view(ndarray) it was thrown out because it is not used enough to add an additional attribute .H A few were in favor, but this can now be written .T.conj() which is not bad so does not get a new attribute. -Travis From cookedm at physics.mcmaster.ca Fri Jul 7 16:22:20 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri, 7 Jul 2006 16:22:20 -0400 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: <1d1e6ea70607070726u6e6d8e33v26765b5a6f6c9cd3@mail.gmail.com> References: <44ACF15D.6030607@ieee.org> <9BC1D67B-C7BB-41F9-B136-5ACD4604DDD5@mac.com> <1d1e6ea70607070726u6e6d8e33v26765b5a6f6c9cd3@mail.gmail.com> Message-ID: <20060707162220.66d3d122@arbutus.physics.mcmaster.ca> On Fri, 7 Jul 2006 15:26:41 +0100 "George Nurser" wrote: > On 07/07/06, Robert Hetland wrote: > [snip] > > However, I use transpose often when not dealing with linear algebra, in > > particular with reading in data, and putting various columns into > > variables. Also, occasional in plotting (which expects things in > > 'backward' order relative to x-y space), and communicating between > > fortran programs (which typically use 'forward' order (x, y, z)) and > > numpy (backward -- (z, x, y)). > > > This is my usage as well. Also my primitive knowledge of numpy > requires use of the transpose when iterating over indexes from where. > Moreover I think the notation .T is perfectly reasonable. So I agree > with: same. > > > I am very much in favor of .T, but it should be a full .transpose(), not > > just swap the last two axes. I don't care so much for the others. > > +1 for .T == .transpose() Another +1 from me. If transpose was a shorter word I wouldn't care :-) -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From oliphant at ee.byu.edu Fri Jul 7 16:39:06 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 07 Jul 2006 14:39:06 -0600 Subject: [Numpy-discussion] [SciPy-dev] Weave, numpy, external libraries and conflicting typedefs In-Reply-To: References: <17582.7994.142818.120995@prpc.aero.iitb.ac.in> <44AE9E12.5000703@ieee.org> <17582.46882.286683.846598@prpc.aero.iitb.ac.in> <44AEBE1F.8010304@ee.byu.edu> Message-ID: <44AEC66A.2070507@ee.byu.edu> Fernando Perez wrote: >On 7/7/06, Travis Oliphant wrote: > > > >>Also, (in latest SVN) the MAXMIN macros can be avoided using >> >>#define PYA_NOMAXMIN >> >>before including arrayobject.h >> >> > >Mmh, this looks crufty to me: special cases like these look bad in a >library, and break the 'just works' ideal we all strive for, IMHO. > > But, it fixes the problem he's having without breaking anybody else's code that uses the MAX / MIN macros, already. Besides, the PY_ARRAY_TYPES_PREFIX business is a lot more crufty. I'm not opposed to putting a *short* prefix in front of everything (the Int32, Float64, stuff came from numarray which now has it's own back-ward compatible header where it could be placed now anyway). Perhaps npy_ would be a suitable prefix. That way we could get rid of the cruft entirely. I suppose we could also provide the noprefix.h header that defines the old un-prefixed cases for "backwards NumPy compatibility". -Travis From svetosch at gmx.net Fri Jul 7 16:44:42 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Fri, 07 Jul 2006 22:44:42 +0200 Subject: [Numpy-discussion] What's wrong with matrices? In-Reply-To: <44AE6675.108@ftw.at> References: <44AE6675.108@ftw.at> Message-ID: <44AEC7BA.6000205@gmx.net> Ed Schofield schrieb: > > Okay ... ... let's make this the thread ;) > I'd like to know why you, Sven, and anyone else on the list have gone > back to using arrays after trying matrices. What was inconvenient about > them? I'd like a nice juicy list. The whole purpose of the matrix > class is to simplify 2d linear algebra. Where is it failing? No, no, I must have given the wrong impression, I'm still in the matrix camp. My main complaint would have been absence of equivalents for ones, zeros, etc., but it seems Travis has introduced numpy.matlib exactly for that, which is great. Before that, I sometimes felt like a second-class citizen because many people on the list argued that users should invent their own workarounds (which btw I have done in the meantime, but that's not the purpose of a ready-to-use matrix package, is it?). > > I'd like to help to make matrices more usable. Tell me what you want, > and I'll work on some patches. > Well if numpy.matlib does what I think it does there are no pressing issues for me right now. Element-wise multiplication isn't that important for me. Of course I'll let you know as soon as something occurs to me ;-) Thanks for offering, Sven From fperez.net at gmail.com Fri Jul 7 16:53:45 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 7 Jul 2006 14:53:45 -0600 Subject: [Numpy-discussion] [SciPy-dev] Weave, numpy, external libraries and conflicting typedefs In-Reply-To: <44AEC66A.2070507@ee.byu.edu> References: <17582.7994.142818.120995@prpc.aero.iitb.ac.in> <44AE9E12.5000703@ieee.org> <17582.46882.286683.846598@prpc.aero.iitb.ac.in> <44AEBE1F.8010304@ee.byu.edu> <44AEC66A.2070507@ee.byu.edu> Message-ID: On 7/7/06, Travis Oliphant wrote: > I'm not opposed to putting a *short* prefix in front of everything (the > Int32, Float64, stuff came from numarray which now has it's own > back-ward compatible header where it could be placed now anyway). > Perhaps npy_ would be a suitable prefix. > > That way we could get rid of the cruft entirely. Well, now is your chance to clean up all the APIs, /including/ the C ones :) npy_ or NPy, I'm not too sure what conventions you are following at the C naming level. I'm all for cruft removal and making things easy to use out of the box, even at the cost of making the transition a bit more work. Remember, that's a one time cost. And numpy is so good that people /will/ transition from Numeric eventually, so might as well make the end result as nice and appealing as possible. As other tools (like matplotlib) move eventually to numpy-only support, the incentive to making the switch will really go up for just about anyone using python for numerical work. At the risk of sounding a bit harsh, I think you can then say, 'take the pain for the switch if you really want all the new goodies'. Those who positively, absolutely can't update from Numeric can then just keep a frozen codebase. It's not like you're breaking Numeric 24.2 or deleting it from the internet :) Cheers, f From tim.hochberg at cox.net Fri Jul 7 17:21:02 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri, 07 Jul 2006 14:21:02 -0700 Subject: [Numpy-discussion] extended dot In-Reply-To: <44AEA43A.9020904@ieee.org> References: <44AE6675.108@ftw.at> <44AEA43A.9020904@ieee.org> Message-ID: <44AED03E.5040808@cox.net> So, I put together of a prototype dot function dot( From tim.hochberg at cox.net Fri Jul 7 17:21:43 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri, 07 Jul 2006 14:21:43 -0700 Subject: [Numpy-discussion] extended dot In-Reply-To: <44AED03E.5040808@cox.net> References: <44AE6675.108@ftw.at> <44AEA43A.9020904@ieee.org> <44AED03E.5040808@cox.net> Message-ID: <44AED067.4060501@cox.net> Tim Hochberg wrote: > > > So, I put together of a prototype dot function > > dot( > Ooops! This wasn't supposed to go out yet, sorry. More later. -tim From mark at mitre.org Fri Jul 7 17:27:49 2006 From: mark at mitre.org (Mark Heslep) Date: Fri, 07 Jul 2006 17:27:49 -0400 Subject: [Numpy-discussion] OptimizationDemo [was: Numpy subversion compatibility with Scipy release? ] RESOLVED In-Reply-To: References: <44AEA9A2.7060308@mitre.org> <44AEB909.6040206@mitre.org> Message-ID: <44AED1D5.1050609@mitre.org> The problem was matplotlib-0.87.2-1; it is broken wrt to the Demo. The lastest matplotlib .3 works. In summary, to run the demo, the following versions were required: Released Packages, all built from tar balls: ------------------------------------------------------ matplotlib-0.87.3 , ie, the current package .2 in Fedora extras fails, meaning I needed to chase down the matplotlib dependencies myself (eg tk.h in tk-devel) Numpy 0.9.8 (latest Fedora package is 0.9.5 which is too old) Scipy 0.4.9 And, at least as of now, svn check outs from the numpy, scipy, and matplotlib subversion repos also work together for Demo1. Mark Robert Kern wrote: > Mark Heslep wrote: > >> Robert Kern wrote: >> >>> The latest SVN revisions should match (it's a bug, otherwise). numpy 2761 is >>> recent enough that an SVN checkout of scipy will probably be fine. >>> >>> >> Yes SVN scipy/numpy fix the problem. Sorry, I missed Travis's post on >> scipy-user here: >> http://projects.scipy.org/pipermail/scipy-user/2006-June/008438.html >> to that effect. >> >> I should just get specific: I was simply trying to run the basic >> optimization demo on the front page of the wiki as check on SVN >> (http://www.scipy.org/Cookbook/OptimizationDemo1) and it appears SVN >> scipy/numpy breaks matplotlib -0.87.2-1 (fc5) in the demo. Do I need SVN >> matplotlib as well? >> > > Yes, you do, thanks to the removal of the Numeric typecodes (Int32, etc.). > However, the error you get looks to be unrelated. > > From tim.hochberg at cox.net Fri Jul 7 18:01:39 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri, 07 Jul 2006 15:01:39 -0700 Subject: [Numpy-discussion] extended dot Message-ID: <44AED9C3.8040409@cox.net> So I put together a prototype for an extended dot function that takes multiple arguments. This allows multiple dots to be computed in a single call thusly: dot(dot(dot(a, b), c), d) => dotn(a, b, c, d) On Bill Baxter's suggestion, dotn attempts to do the dots in an order that minimizes operations. That appears to work fine, although I wasn't very careful and I wouldn't be at all surprised if some problems crop up with that part. The interesting thing is that, since dot can perform both matrix and dot products, it's not associative. That is, dot(a, dot(b, c) != dot(dot(a, b), c) in general. The simplest examples is three vectors. >>> dot(dot([1,2,3], [3,2,1]), [1,1,1]) array([10, 10, 10]) >>> dot([1,2,3], dot([3,2,1], [1,1,1])) array([ 6, 12, 18]) That's mind numbingly obvious in retrospect, but means that my simple minded dot product optimizer is all wrong because it can change the order of evaluation in such a way that the result changes. That means two things: 1. I need to pick an effective order of evaluation. So, dotn(a, b, c ...) will be evaluated as if the products were evaluated in left to right order. 2. The optimizer needs to respect that order when rearranging the order in which the products are performed. Dot products must remain dot products and matrix products must remain matrix products under any transformations that take place. Anyway, that's that for now. I'll report back when I get it fixed up. -tim From ndarray at mac.com Fri Jul 7 18:41:27 2006 From: ndarray at mac.com (Sasha) Date: Fri, 7 Jul 2006 18:41:27 -0400 Subject: [Numpy-discussion] Broken tMIN/tMAX macros Message-ID: Travis' recent change highlighted the definitions of tMIN/tMAX macros. Typed min/max were a subject for some heated discussion between Linux kernel developers many years ago that resulted in the following definitions in the curent kernel: """ /* * min()/max() macros that also do * strict type-checking.. See the * "unnecessary" pointer comparison. */ #define min(x,y) ({ \ typeof(x) _x = (x); \ typeof(y) _y = (y); \ (void) (&_x == &_y); \ _x < _y ? _x : _y; }) #define max(x,y) ({ \ typeof(x) _x = (x); \ typeof(y) _y = (y); \ (void) (&_x == &_y); \ _x > _y ? _x : _y; }) /* * ..and if you can't take the strict * types, you can specify one yourself. * * Or not use min/max at all, of course. */ #define min_t(type,x,y) \ ({ type __x = (x); type __y = (y); __x < __y ? __x: __y; }) #define max_t(type,x,y) \ ({ type __x = (x); type __y = (y); __x > __y ? __x: __y; }) """ The idea is to force people to use _t versions unless the types of x and y are exactly the same. The numpy's tMIN and tMAX are clearly addressing the same problem, but current definitions #define tMAX(a,b,typ) {typ _x_=(a); typ _y_=(b); _x_>_y_ ? _x_ : _y_} #define tMIN(a,b,typ) {typ _x_=(a); typ _y_=(b); _x_<_y_ ? _x_ : _y_} are unlikely to work with any compiler. Linux kernel uses gcc trick of wrapping a block in parenthesis to get an expression but I don't think this is acceptable in numpy code. Not surprizingly, these macros are not used anywhere. I propose to remove them. From kwgoodman at gmail.com Fri Jul 7 19:01:47 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 7 Jul 2006 16:01:47 -0700 Subject: [Numpy-discussion] What's wrong with matrices? In-Reply-To: <44AEA43A.9020904@ieee.org> References: <44AE6675.108@ftw.at> <44AEA43A.9020904@ieee.org> Message-ID: On 7/7/06, Travis Oliphant wrote: > Bill Baxter wrote: > > 4) eye,empty,rand,ones,zeros,arange and anything else that builds an > > array from scratch or from a python list should have a matrix equivalent > Would > > from numpy.defmatrix import ones, zeros, ... > > work? Can defmatrix be shortened to matrix? So from numpy.matrix import ones, zeros, ... From oliphant at ee.byu.edu Fri Jul 7 21:02:53 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 07 Jul 2006 19:02:53 -0600 Subject: [Numpy-discussion] NumPy C-API now has prefixes Message-ID: <44AF043D.8010805@ee.byu.edu> I just committed a big change to the NumPy SVN (r2773-r2777) which adds the prefix npy_ or NPY_ to all names not otherwise pre-fixed. There is also a noprefix.h header that allows you to use the names without the prefixes defined, as before Plus: 1) The special FLAG names with _FLAGS now have the _FLAGS removed 2) The PY_ARRAY_TYPES_PREFIX is ignored. 3) The tMIN/tMAX macros are removed 4) MAX_DIMS --> NPY_MAXDIMS 5) OWN_DATA --> NPY_OWNDATA There is the header oldnumeric.h that can be used for compatibility with the Numeric C-API (including the names CONTIGUOUS and OWN_DATA). Please, try out the new C-API and let's get the bugs wrinkled out. Hopefully this will give us a more solid foundation for the future... I've already committed changes to matplotlib SVN that allow it to work with old and new NumPy. -Travis From fperez.net at gmail.com Fri Jul 7 21:06:58 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 7 Jul 2006 19:06:58 -0600 Subject: [Numpy-discussion] NumPy C-API now has prefixes In-Reply-To: <44AF043D.8010805@ee.byu.edu> References: <44AF043D.8010805@ee.byu.edu> Message-ID: On 7/7/06, Travis Oliphant wrote: > I just committed a big change to the NumPy SVN (r2773-r2777) which adds > the prefix npy_ or NPY_ to all names not otherwise pre-fixed. > > There is also a noprefix.h header that allows you to use the names > without the prefixes defined, as before > > Plus: > > 1) The special FLAG names with _FLAGS now have the _FLAGS removed > 2) The PY_ARRAY_TYPES_PREFIX is ignored. > 3) The tMIN/tMAX macros are removed > 4) MAX_DIMS --> NPY_MAXDIMS > 5) OWN_DATA --> NPY_OWNDATA Make sure scipy builds after these, I think I just saw it not build with 'OWN_DATA' errors. Maybe I just caught you in-between commits... f From fperez.net at gmail.com Fri Jul 7 21:15:08 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 7 Jul 2006 19:15:08 -0600 Subject: [Numpy-discussion] I think something broke badly today in numpy/scipy... Message-ID: Hi all, I updated earlier today (about 4 hours ago) numpy/scipy SVN, and all of a sudden my codes broke left and right. Backing off to In [3]: numpy.__version__ Out[3]: '0.9.9.2737' In [4]: scipy.__version__ Out[4]: '0.5.0.2044' things are OK again. I am really sorry not to be able to provide a narrow test case, but I am utterly swamped right now, and need to finish a number of things. Given how bracketing this particular problem takes about 1/2 hour in recompilations alone for each revision, I really can't do it right now. There is no exception, no traceback, it's just that various numerical algorithms fail to converge. This code is complex and it uses numpy, scipy and weave, so the problem could be anywhere. Again, my apologies for the super-vague report. But perhaps knowing that the change is fairly recent, someone may have an eureka moment :) If nobody finds anything obvious, I might be able to spend some serious debugging time next week on this, but for now I'll just retreat to a working version and try to finish my things. Cheers, f From fullung at gmail.com Fri Jul 7 21:22:36 2006 From: fullung at gmail.com (Albert Strasheim) Date: Sat, 8 Jul 2006 03:22:36 +0200 Subject: [Numpy-discussion] NumPy C-API now has prefixes In-Reply-To: <44AF043D.8010805@ee.byu.edu> Message-ID: <00f001c6a22c$fec10080$01eaa8c0@dsp.sun.ac.za> Hello all > Please, > > try out the new C-API and let's get the bugs wrinkled out. > > Hopefully this will give us a more solid foundation for the future... > I've already committed changes to matplotlib SVN that allow it to work > with old and new NumPy. What implications, if any, do these changes have for C extension skeleton on the wiki? http://www.scipy.org/Cookbook/C_Extensions Regards, Albert From stefan at sun.ac.za Fri Jul 7 21:22:50 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Sat, 8 Jul 2006 03:22:50 +0200 Subject: [Numpy-discussion] NumPy C-API now has prefixes In-Reply-To: References: <44AF043D.8010805@ee.byu.edu> Message-ID: <20060708012249.GA21318@mentat.za.net> On Fri, Jul 07, 2006 at 07:06:58PM -0600, Fernando Perez wrote: > On 7/7/06, Travis Oliphant wrote: > > I just committed a big change to the NumPy SVN (r2773-r2777) which adds > > the prefix npy_ or NPY_ to all names not otherwise pre-fixed. > > > > There is also a noprefix.h header that allows you to use the names > > without the prefixes defined, as before > > > > Plus: > > > > 1) The special FLAG names with _FLAGS now have the _FLAGS removed > > 2) The PY_ARRAY_TYPES_PREFIX is ignored. > > 3) The tMIN/tMAX macros are removed > > 4) MAX_DIMS --> NPY_MAXDIMS > > 5) OWN_DATA --> NPY_OWNDATA > > Make sure scipy builds after these, I think I just saw it not build > with 'OWN_DATA' errors. Maybe I just caught you in-between commits... I see the following before the compilation breaks: Lib/cluster/src/vq_wrap.cpp: In function ?PyObject* ARGOUT_argout(char*, char*, int*, int)?: Lib/cluster/src/vq_wrap.cpp:734: error: ?OWN_DATA? was not declared in this scope Lib/cluster/src/vq_wrap.cpp: In function ?int char_to_size(char)?: Lib/cluster/src/vq_wrap.cpp:582: warning: control reaches end of non-void function Lib/cluster/src/vq_wrap.cpp: In function ?int char_to_numtype(char)?: Lib/cluster/src/vq_wrap.cpp:590: warning: control reaches end of non-void function Lib/cluster/src/vq_wrap.cpp: At global scope: Lib/cluster/src/vq_wrap.cpp:147: warning: ?void* SWIG_TypeQuery(const char*)? defined but not used Lib/cluster/src/vq_wrap.cpp:301: warning: ?void SWIG_addvarlink(PyObject*, char*, PyObject* (*)(), int (*)(PyObject*))? defined but not used Lib/cluster/src/vq_wrap.cpp:315: warning: ?int SWIG_ConvertPtr(PyObject*, void**, swig_type_info*, int)? defined but not used Lib/cluster/src/vq_wrap.cpp:516: warning: ?PyObject* l_output_helper(PyObject*, PyObject*)? defined but not used In file included from /usr/include/python2.4/Python.h:8, from Lib/cluster/src/vq_wrap.cpp:176: /usr/include/python2.4/pyconfig.h:835:1: warning: "_POSIX_C_SOURCE" redefined In file included from /usr/include/string.h:26, from Lib/cluster/src/vq_wrap.cpp:27: /usr/include/features.h:150:1: warning: this is the location of the previous definition In file included from Lib/cluster/src/vq_wrap.cpp:499: Lib/cluster/src/vq.h:57:7: warning: no newline at end of file Lib/cluster/src/vq_wrap.cpp: In function ?PyObject* ARGOUT_argout(char*, char*, int*, int)?: Lib/cluster/src/vq_wrap.cpp:734: error: ?OWN_DATA? was not declared in this scope Lib/cluster/src/vq_wrap.cpp: In function ?int char_to_size(char)?: Lib/cluster/src/vq_wrap.cpp:582: warning: control reaches end of non-void function Lib/cluster/src/vq_wrap.cpp: In function ?int char_to_numtype(char)?: Lib/cluster/src/vq_wrap.cpp:590: warning: control reaches end of non-void function Lib/cluster/src/vq_wrap.cpp: At global scope: Lib/cluster/src/vq_wrap.cpp:147: warning: ?void* SWIG_TypeQuery(const char*)? defined but not used Lib/cluster/src/vq_wrap.cpp:301: warning: ?void SWIG_addvarlink(PyObject*, char*, PyObject* (*)(), int (*)(PyObject*))? defined but not used Lib/cluster/src/vq_wrap.cpp:315: warning: ?int SWIG_ConvertPtr(PyObject*, void**, swig_type_info*, int)? defined but not used Lib/cluster/src/vq_wrap.cpp:516: warning: ?PyObject* l_output_helper(PyObject*, PyObject*)? defined but not used error: Command "g++ -pthread -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -fPIC -I/home/stefan/lib/python2.4/site-packages/numpy/core/include -I/usr/include/python2.4 -c Lib/cluster/src/vq_wrap.cpp -o build/temp.linux-i686-2.4/Lib/cluster/src/vq_wrap.o" failed with exit status 1 Error building package. St?fan From haase at msg.ucsf.edu Fri Jul 7 21:37:32 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 07 Jul 2006 18:37:32 -0700 Subject: [Numpy-discussion] Call for a vote on .M .A .T .H attributes In-Reply-To: <44ADE2E3.60102@ieee.org> References: <44ADE264.2030903@ieee.org> <44ADE2E3.60102@ieee.org> Message-ID: <44AF0C5C.9040104@msg.ucsf.edu> Travis Oliphant wrote: > Travis Oliphant wrote: >> This is a call for a vote on each of the math attributes. Please post >> your vote as >> >> +1 : support >> +0 : don't care so go ahead >> -0 : don't care so why do it >> -1 : against >> >> Vote on the following issues separately: >> >> >> >> 1) .T Have some kind of .T attribute >> >> +1 >> If >0 on this then: >> >> a) .T == .swapaxes(-2,-1) >> +1 >> b) .T == .transpose() >> -0 >> c) .T raises error for ndim > 2 >> -1 >> d) .T returns (N,1) array for length (N,) array >> >> +1 >> e) .T returns self for ndim < 2 >> >> -1 >> 2) .H returns .T.conj() >> >> >> +0 > >> 3) .M returns matrix version of array >> >> +0 >> 4) .A returns basearray (useful for sub-classes). >> +0 Sebastian Haase From haase at msg.ucsf.edu Fri Jul 7 21:37:26 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 07 Jul 2006 18:37:26 -0700 Subject: [Numpy-discussion] .T Transpose shortcut for arrays again In-Reply-To: References: <44ACF15D.6030607@ieee.org> <44AD8A7C.5030408@cox.net> Message-ID: <44AF0C56.7020906@msg.ucsf.edu> Sasha wrote: > On 7/6/06, Bill Baxter wrote: >> ... >> Yep, like Tim said. The usage is say a N sets of basis vectors. Each set >> of basis vectors is a matrix. > > This brings up a feature that I really miss from numpy: an ability to do > > array([f(x) for x in a]) > > without python overhead. APL-like languages have a notion of "adverb" > - a higher level operator that maps a function to a function. Numpy > has some adverbs implemented as .... Hi, I was just reading through this thread and noticed that the above might be possibly done best with(a little extended version of) the numexpr module. Am I right !? Just wanted to post this comment about a package I'm really looking forward to using once I convert from numarray. Thanks for numpy !! Sebastian Haase UCSF From oliphant at ee.byu.edu Fri Jul 7 21:58:51 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 07 Jul 2006 19:58:51 -0600 Subject: [Numpy-discussion] Easy way to get NumPy code to compile Message-ID: <44AF115B.1040105@ee.byu.edu> If you have been a user of NumPy. The easy way to update your code so that it compiles even with the latest changes to the naming-scheme is to replace #include "numpy/arrayobject.h" with #include "numpy/noprefix.h" This will generate the prefix-less names (and the other _FLAGS and OWN_DATA names that your code may depend on). It also includes arrayobject.h and so there is no need to do that twice. This should help people who already have code that works with NumPy. -Travis From oliphant at ee.byu.edu Fri Jul 7 22:24:50 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 07 Jul 2006 20:24:50 -0600 Subject: [Numpy-discussion] SciPy should now compile and install Message-ID: <44AF1772.5010804@ee.byu.edu> All changes needed for scipy to compile and install with the new NumPy are now done. As a side benefit, the numarray C-API compatibility module also received a test as it is now used to compile scipy/ndimage So, SVN versions of NumPy / SciPy / and Matplotlib should all work together now. -Travis From jk985 at tom.com Mon Jul 10 23:22:55 2006 From: jk985 at tom.com (=?GB2312?B?N9TCMTUtMTbI1S/Jz7qj?=) Date: Tue, 11 Jul 2006 11:22:55 +0800 Subject: [Numpy-discussion] =?GB2312?B?1MvTw0VYQ0VMuMS9+LLGzvG53MDtus3K0LOh06rP+jxhZD4=?= Message-ID: An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Sat Jul 8 03:17:14 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 08 Jul 2006 01:17:14 -0600 Subject: [Numpy-discussion] NumPy beta release plan Message-ID: <44AF5BFA.70509@ieee.org> Some of you may have noticed that things have been changing rapidly in the NumPy world (if you were out of the office in June then all the activity may seem overwhelming). All of this activity is based on the fact that the upcoming beta release will mean a feature freeze for NumPy. As a review: The biggest changes over the past 3 months have been 1) Capitalized-type-names (Float32, Complex64, Int8, etc) are now only available in the numpy.oldnumeric namespace (this is the namespace that convertcode.py replaces for Numeric). We are trying to wean you off character codes as much as possible. They are still there, of course but should only be used in special cases not as a general rule. 2) Un-specified data-types now default to floating point. To help with code you have that relies on integer data-types you can either use functions from numpy.oldnumeric where the functions are still defined with the integer-default data-types or use functions in numpy/lib/convdtype to replace type-less versions of ones, zeros, empty with empty(..., dtype=int). 3) C-API names have prefix PyArray_ (like always), NPY_ or npy_. The NPY_ and npy_ prefixes are new and were done to remove the likelihood of name collisions when NumPy is used with another library. The old (and un-prefixed) names are accessible by importing numpy/noprefix.h instead of numpy/arrayobject.h It is fine to use noprefix.h in-place of arrayobject.h if you expect to have no naming conflicts. This is what NumPy itself does. 4) The flag combinations with _FLAGS in the name have the _FLAGS removed (but are prefixed with NPY_). Again the old names are still available in numpy/noprefix.h 5) The Numarray C-API is now available as numpy/libnumarray.h as long as you use the directory returned from numpy.get_numarray_include() as an argument to -I in the compile command. More minor changes: 1) ctypes attribute added for ease of working with ctypes data 2) T attribute added as a convenience for .transpose() Personally, I don't want to make any more changes so that we can make a 1.0 beta release that will freeze the API. Because of my guarantee to keep SVN versions of NumPy/SciPy/matplotlib working, changes probably cause more headache for me than anyone else. Unless there is a serious issue pointed out, the only thing that should be changed at this point are bug-fixes, documentation strings, and added tests. Once 1.0b1 is released nothing but those things can be added. I'd like July to be a much more calm month. We should get the 1.0b1 out in the next couple of weeks. That way perhaps 1.0b2 can be out by the SciPy conference. I can see the beta release period taking several months with only bug-fixes/docstring/testing improvements happening over that time. So, take this message as a warning for the up-coming feature freeze on NumPy and an invitation to contribute docstrings and unit-tests. I hope the rapid pace of June development did not scare too many people. Please voice your concerns if you have them. Best regards, -Travis O. From st at sigmasquared.net Sat Jul 8 04:04:20 2006 From: st at sigmasquared.net (Stephan Tolksdorf) Date: Sat, 08 Jul 2006 10:04:20 +0200 Subject: [Numpy-discussion] NumPy beta release plan In-Reply-To: <44AF5BFA.70509@ieee.org> References: <44AF5BFA.70509@ieee.org> Message-ID: <44AF6704.4080407@sigmasquared.net> Hi Travis, thanks for putting all the effort into NumPy. > Unless there is a serious issue pointed out, the only thing that should > be changed at this point are bug-fixes, documentation strings, and added > tests. Once 1.0b1 is released nothing but those things can be added. The beginning of the beta phase might also be the best time to address the most severe NumPy.distutils issues, so that all changes can be extensively tested during the beta: - warning messages are only shown if compilation fails (NumPy should build warning free) - there are some problems on Windows (ticket #114) - 64bit/Python 2.5 issues? Ciao, Stephan From oliphant.travis at ieee.org Sat Jul 8 05:33:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 08 Jul 2006 03:33:01 -0600 Subject: [Numpy-discussion] I think something broke badly today in numpy/scipy... In-Reply-To: References: Message-ID: <44AF7BCD.4080302@ieee.org> Fernando Perez wrote: > Hi all, > > I updated earlier today (about 4 hours ago) numpy/scipy SVN, and all > of a sudden my codes broke left and right. Backing off to > > Hey Fernando. I think I found the problem. It was the same problem causing the BFGS test to fail in SciPy. It can be shown by looking at sk = [0,-1.0,1.0] print 250.0*sk[:,newaxis] This should give something useful (not zeros). The problem was a seemingly harmless change in _IsContiguous that allowed for 0-strided arrays to be called contiguous. This caused havoc with the multiplication function later which relied on an accurate _IsContiguous function. This change was in r2765 (I suspect you checked out a change with that in it). The change could cause problems where-ever you use newaxis with scalar multiplication. The problem should be fixed in SVN. Very sorry... -Travis From wbaxter at gmail.com Sat Jul 8 06:10:04 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 8 Jul 2006 19:10:04 +0900 Subject: [Numpy-discussion] building ATLAS on Win32 - endless compile loop Message-ID: Trying to compile Atlas from source on Win32 with Cygwin. Anyone seen 'make install' for ATLAS go into and endless loop before? More info: Following the instructions here: Installing SciPy/Windows - SciPy.org At this step: "5) As prompted by the config script, execute make install arch=YOUR_ARCHITECTURE and wait for approx > 15min." It was taking much longer than 15 minutes, but I figured my laptop is just slow or something. So I left it on when I went to bed. It was still cranking away 6 hours later when I got up. Just thought I'd check if anyone had seen that before. My laptop is a Pentium M, which isn't one of the options on the architecture list. So I picked "unknown". --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From schofield at ftw.at Sat Jul 8 07:07:11 2006 From: schofield at ftw.at (Ed Schofield) Date: Sat, 8 Jul 2006 13:07:11 +0200 Subject: [Numpy-discussion] Args for rand and randn: call for a vote Message-ID: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> Last week's discussion on rand() and randn() seemed to indicate a sentiment that they ought to take tuples for consistency with ones, zeros, eye, identity, and empty -- that, although they are supposed to be convenience functions, they are inconvenient precisely because of their inconsistency with these other functions. This issue has been raised many times over the past several months. Travis made a change in r2572 to allow tuples as arguments, then took it out again a few hours later, apparently unsure about whether this was a good idea. I'd like to call for a vote on what people would prefer, and then ask Travis to make a final pronouncement before the feature freeze. * Should numpy.rand and numpy.randn accept sequences of dimensions as arguments, like rand((3,3)), as an alternative to rand(3,3)? OR * Should rand((3,3)) and randn((3,3)) continue to raise a TypeError? From schofield at ftw.at Sat Jul 8 07:16:02 2006 From: schofield at ftw.at (Ed Schofield) Date: Sat, 8 Jul 2006 13:16:02 +0200 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> Message-ID: <2014D1A2-6D90-45B6-879D-A82D9EDE8F76@ftw.at> On 08/07/2006, at 1:07 PM, Ed Schofield wrote: > Last week's discussion on rand() and randn() seemed to indicate a > sentiment that they ought to take tuples for consistency with ones, > zeros, eye, identity, and empty -- that, although they are supposed > to be convenience functions, they are inconvenient precisely because > of their inconsistency with these other functions. This issue has > been raised many times over the past several months. > > Travis made a change in r2572 to allow tuples as arguments, then took > it out again a few hours later, apparently unsure about whether this > was a good idea. > > I'd like to call for a vote on what people would prefer, and then ask > Travis to make a final pronouncement before the feature freeze. > > > * Should numpy.rand and numpy.randn accept sequences of dimensions as > arguments, like rand((3,3)), as an alternative to rand(3,3)? +1 From wbaxter at gmail.com Sat Jul 8 08:21:06 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 8 Jul 2006 21:21:06 +0900 Subject: [Numpy-discussion] What's wrong with matrices? In-Reply-To: References: <44AE6675.108@ftw.at> <44AEA43A.9020904@ieee.org> Message-ID: My preferred way to import numpy currently is: import numpy as num It would be nice if I could do: import numpy.matrix as num And basically have access to all the same stuff that's in the base numpy but have everything set up in a matrix-like way, so num.ones returns a matrix instead of an array, etc. (I don't care so much about the particular name... matlib, matrixlib, matrixdefault, or whatever -- I just care about the functionality) The way I understand this matlib module is that I would do something along the lines of one of these: 1) import numpy as num from numpy.matlib import ones, zeros,... or 2) import numpy as num import numpy.matlib as M Either way, the matrix functions aren't in the same namespace as general numpy functions, which makes it feel like something of a bolt-on rather than a second equally valid way to use numpy. So what I'd like is 3) import numpy.matlib as num I think the way to get 3) may be to just do a "from numpy import *" at the beginning of the numpy.matlib module, before redefining particular functions with matrix versions. Maybe that's a bad idea, though? Or maybe this is the way Travis and others planned for it to work from the beginning? At any rate, versions 1) and 2) should also be supported for the times you don't want to use matlib as the default, but still want access to it. --bill -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter at sonderport.dk Sat Jul 8 10:58:00 2006 From: peter at sonderport.dk (Peter =?ISO-8859-1?Q?S=F8ndergaard?=) Date: Sat, 08 Jul 2006 16:58:00 +0200 Subject: [Numpy-discussion] building ATLAS on Win32 - endless compile loop In-Reply-To: References: Message-ID: <1152370680.2347.1.camel@katholt.sonderport.dk> > My laptop is a Pentium M, which isn't one of the options on the > architecture list. So I picked "unknown". > In that case ATLAS propably does a full search instead of the much smaller architectural default one. This takes much longer, but not infinitely. Cheers, Peter From svetosch at gmx.net Sat Jul 8 09:23:15 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Sat, 08 Jul 2006 15:23:15 +0200 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> Message-ID: <44AFB1C3.6030804@gmx.net> Ed Schofield schrieb: > > I'd like to call for a vote on what people would prefer, and then ask > Travis to make a final pronouncement before the feature freeze. > > > > > * Should numpy.rand and numpy.randn accept sequences of dimensions as > arguments, like rand((3,3)), as an alternative to rand(3,3)? > +1 (Thanks Ed for trying to reach a conclusion on this.) -sven From pebarrett at gmail.com Sat Jul 8 09:31:20 2006 From: pebarrett at gmail.com (Paul Barrett) Date: Sat, 8 Jul 2006 09:31:20 -0400 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> Message-ID: <40e64fa20607080631w2a9d72ebh2a186b4cac70f383@mail.gmail.com> On 7/8/06, Ed Schofield wrote: > > Last week's discussion on rand() and randn() seemed to indicate a > sentiment that they ought to take tuples for consistency with ones, > zeros, eye, identity, and empty -- that, although they are supposed > to be convenience functions, they are inconvenient precisely because > of their inconsistency with these other functions. This issue has > been raised many times over the past several months. > > Travis made a change in r2572 to allow tuples as arguments, then took > it out again a few hours later, apparently unsure about whether this > was a good idea. > > I'd like to call for a vote on what people would prefer, and then ask > Travis to make a final pronouncement before the feature freeze. > > > > > * Should numpy.rand and numpy.randn accept sequences of dimensions as > arguments, like rand((3,3)), as an alternative to rand(3,3)? +1 --- I'm all for consistency! > OR > > > * Should rand((3,3)) and randn((3,3)) continue to raise a TypeError? From wbaxter at gmail.com Sat Jul 8 09:33:16 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 8 Jul 2006 22:33:16 +0900 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <44AFB1C3.6030804@gmx.net> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> <44AFB1C3.6030804@gmx.net> Message-ID: > > * Should numpy.rand and numpy.randn accept sequences of dimensions as > > arguments, like rand((3,3)), as an alternative to rand(3,3)? > > +1 -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Sat Jul 8 10:24:58 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 8 Jul 2006 23:24:58 +0900 Subject: [Numpy-discussion] What's wrong with matrices? In-Reply-To: <44AEA43A.9020904@ieee.org> References: <44AE6675.108@ftw.at> <44AEA43A.9020904@ieee.org> Message-ID: On 7/8/06, Travis Oliphant wrote: > > > Okay, here are a few that come to mind. > > 1) Functions that take a matrix but return an array. Maybe these are > > all fixed now. But they better be fixed not just in numpy but in > > scipy too. To me this implies there needs to be some standard idiom > > for how to write a generic array-protocol-using function so that you > > don't have to think about it. > > A lot of these are fixed. The mechanism for handling this is in-place: > either using asanyarray in the function or (more generally) using a > decorator that wraps the arguments with asarray and returns the output > with __array_wrap__. That sounds good. Is there a description of what that does and how to use it anywhere you could point me to? > 5) I've got squeezes like crazy all over my matrix-using code. Maybe > > this was a bug in 0.9.5 or so that's been fixed? I do seem to recall > > some problem with indexing or c_ or something that was causing > > matrices to grow extra levels of length 1 axes. Again, like the > > scalar*matrix bug, things like that shouldn't happen. > Sure, but it's going to happen in a beta release... That's why we need > testers. As I recall, most bugs with matrices have been fixed fairly > quickly as soon as they are reported. What do you mean by "beta release"? Are the odds considered betas and evens releases? Or do you mean just everything prior to 1.0 is actually beta? I haven't seen anything actually labeled as "beta" yet. And yes, you've done a fantastic job fixing bugs quickly, and getting releases out in a pretty timely manner too. Many thanks for that. > > > 6) No good way to do elementwise operations? Sometimes you just want > > to do an elementwise mult or divide or exponentiation. I guess you're > > supposed to do Z = asmatrix(X.A * Y.A). Yah, right. > This is a problem with a dearth of infix operators. In fact, if we had > a good way to write matrix multiplication as an infix operator, perhaps > there wouldn't be any need for matrices. Actually I was more after what you mentioned later -- the multiply(a,b) function. I see they're all there -- multiply, divide, power. That's all I wanted, because I know the Python operator overload situation doesn't really allow more than that. > ...So I'd like matrices to be able to have ndim>2. > I suppose this restriction could be lifted. I think that would be an improvement for people who want to use matrix as their only ndarray data type. > b) On the other end, I think ndim<2 is useful sometimes too. Take > > a function like mean(), for example. With no arguments the return > > value is a 1x1 matrix (as opposed to a scalar). > Have you checked lately. It's a scalar now... This has been fixed. Nope, haven't checked more recently than the latest release. I'm in the process of trying to build numpy, though. > Or take indexing. It seems odd to me that where() reurns a tuple of > > shape==(1,N) objects instead of just (N,) . The way to fix some of these is to return arrays for indexing instead of > allowing matrices. But, matrices that are less than 2-d just don't > make sense. I guess not. Looking at what matlab does more closely I see that it does report size (1,1) for all scalars, and either 1,N or N,1 for all vectors. It just hides it better. Namely when printing you don't get as many superfluous brackets (in fact matlab doesn't print any brackets). > Maybe I can get over that though, as long as it works for indexing > > (which it seems it does). But I think the scalar return case is a > > real issue. Here's another: sum(). For an array you can do > > sum(sum(a)) and get a scalar if a is 2-d, but for matrix sum(sum(m)) > > is the same as sum(m). And along these lines, m[newaxis] just > > silently doesn't do anything. That doesn't seem right. > > These are just semantic questions. It's no surprise that sum(sum(m)) > returns the same as sum(m) for a matrix because summing over the same > axis won't change the result. You have to sum over both axes in a > matrix. Right, it is perfectly logical given how sum is implemented. I guess I'm just used to Matlab's way which acts more like sum on arrays. But maybe numpy's way is ok too. --bill -------------- next part -------------- An HTML attachment was scrubbed... URL: From dd55 at cornell.edu Sat Jul 8 10:30:59 2006 From: dd55 at cornell.edu (Darren Dale) Date: Sat, 8 Jul 2006 10:30:59 -0400 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> Message-ID: <200607081030.59826.dd55@cornell.edu> On Saturday 08 July 2006 7:07 am, Ed Schofield wrote: > * Should numpy.rand and numpy.randn accept sequences of dimensions as > arguments, like rand((3,3)), as an alternative to rand(3,3)? +1 -- Darren S. Dale, Ph.D. dd55 at cornell.edu From sransom at nrao.edu Sat Jul 8 10:40:19 2006 From: sransom at nrao.edu (Scott Ransom) Date: Sat, 8 Jul 2006 10:40:19 -0400 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> Message-ID: <20060708144019.GB12588@ssh.cv.nrao.edu> > * Should numpy.rand and numpy.randn accept sequences of dimensions as > arguments, like rand((3,3)), as an alternative to rand(3,3)? +1 -- Scott M. Ransom Address: NRAO Phone: (434) 296-0320 520 Edgemont Rd. email: sransom at nrao.edu Charlottesville, VA 22903 USA GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From dd55 at cornell.edu Sat Jul 8 10:45:53 2006 From: dd55 at cornell.edu (Darren Dale) Date: Sat, 8 Jul 2006 10:45:53 -0400 Subject: [Numpy-discussion] NumPy beta release plan In-Reply-To: <44AF6704.4080407@sigmasquared.net> References: <44AF5BFA.70509@ieee.org> <44AF6704.4080407@sigmasquared.net> Message-ID: <200607081045.53198.dd55@cornell.edu> On Saturday 08 July 2006 4:04 am, Stephan Tolksdorf wrote: > Hi Travis, > > thanks for putting all the effort into NumPy. > > > Unless there is a serious issue pointed out, the only thing that should > > be changed at this point are bug-fixes, documentation strings, and added > > tests. Once 1.0b1 is released nothing but those things can be added. > > The beginning of the beta phase might also be the best time to address > the most severe NumPy.distutils issues, so that all changes can be > extensively tested during the beta: The number of people in my lab using numpy/scipy/matplotlib is starting to increase. Last week I was trying to get a couple of graduate students set up. They were using MacOS X 10.4 with gcc3.3, and we were never successful in getting numpy to build. I think the problem was the same as reported at http://mail.python.org/pipermail/pythonmac-sig/2006-March/016703.html. We ended up getting things installed by installing scipy with fink. > - warning messages are only shown if compilation fails (NumPy should > build warning free) One of the guys I was working with tried installing every compiler out there to get rid of all the compiler not found messages that were reported during the attempted numpy builds. I guess he thought these messages were critical errors, since they are rendered in red lettering. Darren From listservs at mac.com Sat Jul 8 11:11:02 2006 From: listservs at mac.com (listservs at mac.com) Date: Sat, 8 Jul 2006 11:11:02 -0400 Subject: [Numpy-discussion] bdist_mpkg Message-ID: As of recent svn updates, I can no longer build numpy binaries for OS X using bdist_mpkg. How do I restore this capability, or is there a better way of building Mac binaries? Thanks, C. -- Christopher Fonnesbeck + Atlanta, GA + fonnesbeck at mac.com + Contact me on AOL IM using email address From fonnesbeck at mac.com Sat Jul 8 11:16:21 2006 From: fonnesbeck at mac.com (Christopher Fonnesbeck) Date: Sat, 8 Jul 2006 11:16:21 -0400 Subject: [Numpy-discussion] Fwd: bdist_mpkg References: Message-ID: <6C4B2D05-5E9C-4C37-AF53-5750983094B3@mac.com> Never mind, I see that it is in setupegg.py. C. Begin forwarded message: > From: listservs at mac.com > Date: July 8, 2006 11:11:02 AM EDT > To: numpy-discussion at lists.sourceforge.net > Subject: bdist_mpkg > > As of recent svn updates, I can no longer build numpy binaries for > OS X using bdist_mpkg. How do I restore this capability, or is > there a better way of building Mac binaries? > > Thanks, > C. > > -- > Christopher Fonnesbeck > + Atlanta, GA > + fonnesbeck at mac.com > + Contact me on AOL IM using email address > > -- Christopher Fonnesbeck + Atlanta, GA + fonnesbeck at mac.com + Contact me on AOL IM using email address -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2365 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 186 bytes Desc: This is a digitally signed message part URL: From listservs at mac.com Sat Jul 8 11:16:43 2006 From: listservs at mac.com (listservs at mac.com) Date: Sat, 8 Jul 2006 11:16:43 -0400 Subject: [Numpy-discussion] Fwd: bdist_mpkg References: Message-ID: <35C4DE0B-141F-447D-9A8F-2B30895AD704@mac.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Never mind, I see that it is in setupegg.py. C. Begin forwarded message: > From: listservs at mac.com > Date: July 8, 2006 11:11:02 AM EDT > To: numpy-discussion at lists.sourceforge.net > Subject: bdist_mpkg > > As of recent svn updates, I can no longer build numpy binaries for > OS X using bdist_mpkg. How do I restore this capability, or is > there a better way of building Mac binaries? > > Thanks, > C. > > -- > Christopher Fonnesbeck > + Atlanta, GA > + fonnesbeck at mac.com > + Contact me on AOL IM using email address > > - -- Christopher Fonnesbeck + Atlanta, GA + fonnesbeck at mac.com + Contact me on AOL IM using email address - -- Christopher Fonnesbeck + Atlanta, GA + fonnesbeck at mac.com + Contact me on AOL IM using email address -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (Darwin) iD8DBQFEr8xbkeka2iCbE4wRAmxWAJ9jx/jFRJW0N4vCrh5H3SmN9IffvgCfWrRH pT90YecS/SM47DEzfeOSWKU= =D8D1 -----END PGP SIGNATURE----- From erin.sheldon at gmail.com Sat Jul 8 11:23:15 2006 From: erin.sheldon at gmail.com (Erin Sheldon) Date: Sat, 8 Jul 2006 11:23:15 -0400 Subject: [Numpy-discussion] recarray field names In-Reply-To: <4418B223.5020201@ee.byu.edu> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> <76fbbdec3883fc386929a698021fbf8b@stsci.edu> <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> <4418AAB9.3080200@ee.byu.edu> <4418AC54.1060202@colorado.edu> <4418B223.5020201@ee.byu.edu> Message-ID: <331116dc0607080823h7f3c7e4ar578ea660fa6c547e@mail.gmail.com> It was suggested that I put off this discussion until we were closer to the 1.0 release. Perhaps now is a good time to bring it up once again? The quick summary: accessing field names has some oddness that needs cleaning up. On 3/15/06, Travis Oliphant wrote: > Fernando Perez wrote: > > > > > Mmh, just curious: I wonder about the wisdom of that overloading of a > > 'magic' key (-1). It will make thinks like > > > > for name in t.dtype.fields: > > > No real wisdom. More organic growth. Intially I didn't have an ordered > list of fields, but as more complicated data-type descriptors were > supported, this became an important thing to have. I should have > probably added a n additional element to the PyArray_Descr structure. > Remember, it was growing out of the old PyArray_Descr already and I was > a little concerned about changing it too much (there are ramifications > of changing this structure in several places). > > So, instead of adding a "ordered_names" tuple to the dtype object, I > just stuck it in the fields dictionary. I agree it's kind of odd > sitting there. > > It probably deserves a re-factoring and pulling that out into a new > attribute of the dtype object. > > This would mean that the PyArray_Descr structure would need a new object > (names perhaps), and it would need to be tracked. > > Not a huge change and probably worth it before the next release. > > > -Travis From listservs at mac.com Sat Jul 8 11:39:25 2006 From: listservs at mac.com (listservs at mac.com) Date: Sat, 8 Jul 2006 11:39:25 -0400 Subject: [Numpy-discussion] more bdist_mpkg issues (setuptools) Message-ID: <7949CE47-7151-42F6-8DE6-AC017CB0EDC3@mac.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Now that I figured out how to get at bdist_mpkg, I am experiencing issues with building extensions. I am trying to set them up as before with distutils except with the following import statement: from setuptools import setup, Extension However, my f2py extension is not recognized: building 'PyMC.flib' extension error: unknown file type '.f' (from 'PyMC/flib.f') Any pointers to what I might be missing are most appreciated. Thanks, C. - -- Christopher Fonnesbeck + Atlanta, GA + fonnesbeck at mac.com + Contact me on AOL IM using email address -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (Darwin) iD8DBQFEr9Gvkeka2iCbE4wRAg+YAJwPIDDMCQ7GB0CEriKZAFBO8RM3dwCfX1PY Uz0/44rWk1n5i6U9o6Q7pVw= =j9vl -----END PGP SIGNATURE----- From aisaac at american.edu Sat Jul 8 12:15:05 2006 From: aisaac at american.edu (Alan Isaac) Date: Sat, 08 Jul 2006 09:15:05 -0700 Subject: [Numpy-discussion] Args for rand and randn: call for a vote Message-ID: <44AFDA09.5060209@american.edu> Ed Schofield wrote: > * Should numpy.rand and numpy.randn accept sequences of dimensions as > arguments, like rand((3,3)), as an alternative to rand(3,3)? > For allowing tuples: +1 For *only* allowing tuples: +1 Alan Isaac From fperez.net at gmail.com Sat Jul 8 15:03:17 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 8 Jul 2006 13:03:17 -0600 Subject: [Numpy-discussion] I think something broke badly today in numpy/scipy... In-Reply-To: <44AF7BCD.4080302@ieee.org> References: <44AF7BCD.4080302@ieee.org> Message-ID: On 7/8/06, Travis Oliphant wrote: > Hey Fernando. I think I found the problem. It was the same problem > causing the BFGS test to fail in SciPy. [...] > The problem should be fixed in SVN. Confirmed. With: numpy version: 0.9.9.2788 scipy version: 0.5.0.2057 I checked on two machines (home/office) and all is good now. I only had to make a tiny change to my hand-written extensions for the new C API names, and all checks OK now. > Very sorry... Considering how much you've given us, I don't really think you even have the /right/ to apologize to anyone ;) Many thanks for such a remarkable fix, given my impossibly vague bug report. You rock. f From charlesr.harris at gmail.com Sat Jul 8 15:36:23 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 8 Jul 2006 13:36:23 -0600 Subject: [Numpy-discussion] NumPy beta release plan In-Reply-To: <200607081045.53198.dd55@cornell.edu> References: <44AF5BFA.70509@ieee.org> <44AF6704.4080407@sigmasquared.net> <200607081045.53198.dd55@cornell.edu> Message-ID: Hi Travis, Are you going to make a development branch so other things can continue going into Numpy until the 1.0 release? Chuck On 7/8/06, Darren Dale wrote: > > On Saturday 08 July 2006 4:04 am, Stephan Tolksdorf wrote: > > Hi Travis, > > > > thanks for putting all the effort into NumPy. > > > > > Unless there is a serious issue pointed out, the only thing that > should > > > be changed at this point are bug-fixes, documentation strings, and > added > > > tests. Once 1.0b1 is released nothing but those things can be added. > > > > The beginning of the beta phase might also be the best time to address > > the most severe NumPy.distutils issues, so that all changes can be > > extensively tested during the beta: > > The number of people in my lab using numpy/scipy/matplotlib is starting to > increase. Last week I was trying to get a couple of graduate students set > up. > They were using MacOS X 10.4 with gcc3.3, and we were never successful in > getting numpy to build. I think the problem was the same as reported at > http://mail.python.org/pipermail/pythonmac-sig/2006-March/016703.html. We > ended up getting things installed by installing scipy with fink. > > > - warning messages are only shown if compilation fails (NumPy should > > build warning free) > > One of the guys I was working with tried installing every compiler out > there > to get rid of all the compiler not found messages that were reported > during > the attempted numpy builds. I guess he thought these messages were > critical > errors, since they are rendered in red lettering. > > Darren > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Jul 8 15:38:17 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 08 Jul 2006 14:38:17 -0500 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> Message-ID: Ed Schofield wrote: > Last week's discussion on rand() and randn() seemed to indicate a > sentiment that they ought to take tuples for consistency with ones, > zeros, eye, identity, and empty -- that, although they are supposed > to be convenience functions, they are inconvenient precisely because > of their inconsistency with these other functions. This issue has > been raised many times over the past several months. > > Travis made a change in r2572 to allow tuples as arguments, then took > it out again a few hours later, apparently unsure about whether this > was a good idea. He was experimenting to see if it was technically easy or not. > I'd like to call for a vote on what people would prefer, and then ask > Travis to make a final pronouncement before the feature freeze. > > * Should numpy.rand and numpy.randn accept sequences of dimensions as > arguments, like rand((3,3)), as an alternative to rand(3,3)? -1 This is worse than the problem it's trying to solve. You'll still see the inconsistent rand(3,3) in people's code. > * Should rand((3,3)) and randn((3,3)) continue to raise a TypeError? This is a false dichotomy. There are more choices here. * Remove rand and randn (at least from the toplevel namespace) and promote the use of random_sample and standard_normal which already follow the tuple convention. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From fperez.net at gmail.com Sat Jul 8 15:50:17 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 8 Jul 2006 13:50:17 -0600 Subject: [Numpy-discussion] .M .A .T .H attribute result In-Reply-To: <44AEC258.9040800@ee.byu.edu> References: <44AEC258.9040800@ee.byu.edu> Message-ID: Hi all, On 7/7/06, Travis Oliphant wrote: > > I didn't compile the results, but the discussion on the idea of adding > new attributes to the array object led to the following result. > > Added: .T attribute to mean self.transpose() [...] > .H > > A few were in favor, but this can now be written .T.conj() which is not > bad so does not get a new attribute. I didn't have strong feelings one way or another on this, so I didn't vote, but at this point I'd like to make a comment before the freeze. Given that .T went in, I'd argue that .H should be in as well. Basically, I now think these two should have been considered as a bundle and not as separate options. The reason isn't (just) my OCD surfacing again, but the fact that the hermitian conjugate plays /exactly/ the role that transposition plays, in regards to defining norms and (positive definite) inner products, when complex arrays are in play. The fact that numpy has excellent complex support is a major source of joy for many. I think that having .T but not .H would be a big wart in this regard. If you are trying to write code for inner products with complex arrays, the natural language change from real ones is: dot(A.T,B) -> dot(A.H,B) For people who play with quantum mechanics this is an everyday need (self-adjoint operators require expressions like this all the time), but I suspect its use will be common in any field requiring normed spaces with complex arrays. Just my 1e-2j Cheers, f From charlesr.harris at gmail.com Sat Jul 8 15:57:41 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 8 Jul 2006 13:57:41 -0600 Subject: [Numpy-discussion] .M .A .T .H attribute result In-Reply-To: References: <44AEC258.9040800@ee.byu.edu> Message-ID: Hi all, On 7/8/06, Fernando Perez wrote: > > Hi all, > > On 7/7/06, Travis Oliphant wrote: > > > > I didn't compile the results, but the discussion on the idea of adding > > new attributes to the array object led to the following result. > > > > Added: .T attribute to mean self.transpose() > > [...] > > > .H > > > > A few were in favor, but this can now be written .T.conj() which is not > > bad so does not get a new attribute. > > I didn't have strong feelings one way or another on this, so I didn't > vote, but at this point I'd like to make a comment before the freeze. > Given that .T went in, I'd argue that .H should be in as well. > Basically, I now think these two should have been considered as a > bundle and not as separate options. +1. H is just the complex counterpart of T. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Jul 8 18:04:46 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 08 Jul 2006 17:04:46 -0500 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> Message-ID: Ed Schofield wrote: > Last week's discussion on rand() and randn() seemed to indicate a > sentiment that they ought to take tuples for consistency with ones, > zeros, eye, identity, and empty -- that, although they are supposed > to be convenience functions, they are inconvenient precisely because > of their inconsistency with these other functions. This issue has > been raised many times over the past several months. > > Travis made a change in r2572 to allow tuples as arguments, then took > it out again a few hours later, apparently unsure about whether this > was a good idea. > > I'd like to call for a vote on what people would prefer, and then ask > Travis to make a final pronouncement before the feature freeze. I would like to ask about the purpose of calling for a vote, here. What authority do you intend the result to have? If you are just asking for a straw poll of opinions from the list to inform Travis' decision, do you think that he hasn't read the previous discussions? Are previous non-participants being drawn out of the woodwork? Or do you intend the majority winner of the vote to force Travis' decision? In which case, who comprises the voting membership, and why? Voting is a notoriously bad way to make decisions in software development. Open source did not change that. It should be a very rare occurrence and happen in very specific circumstances. Two CFVs in as many days is entirely unreasonable. And when a vote *is* taken, it should be done with much more care than I think you are showing here (specifically, a CFV should not be making the case for only one of the choices). Karl Fogel's book _Producing Open Source Software_ thoroughly discusses these issues: http://producingoss.com/html-chunk/consensus-democracy.html#voting -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pau.gargallo at gmail.com Sat Jul 8 18:59:38 2006 From: pau.gargallo at gmail.com (Pau Gargallo) Date: Sun, 9 Jul 2006 00:59:38 +0200 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> Message-ID: <6ef8f3380607081559v62daf4f7t9c3224536e070c0@mail.gmail.com> On 7/8/06, Robert Kern wrote: > Ed Schofield wrote: > > * Should numpy.rand and numpy.randn accept sequences of dimensions as > > arguments, like rand((3,3)), as an alternative to rand(3,3)? > > > * Should rand((3,3)) and randn((3,3)) continue to raise a TypeError? > > This is a false dichotomy. There are more choices here. > > * Remove rand and randn (at least from the toplevel namespace) and promote the > use of random_sample and standard_normal which already follow the tuple convention. > i just wanted to point out another possible choice: * enable numpy.rand((3,3)) and make numpy.rand(3,3) raise an error as zeros and ones do. I suppose that you all discussed a lot about this choice also, but it still seems very reasonable to me :-( pau From robert.kern at gmail.com Sat Jul 8 19:15:25 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 08 Jul 2006 18:15:25 -0500 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <6ef8f3380607081559v62daf4f7t9c3224536e070c0@mail.gmail.com> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> <6ef8f3380607081559v62daf4f7t9c3224536e070c0@mail.gmail.com> Message-ID: Pau Gargallo wrote: > On 7/8/06, Robert Kern wrote: >> Ed Schofield wrote: >>> * Should numpy.rand and numpy.randn accept sequences of dimensions as >>> arguments, like rand((3,3)), as an alternative to rand(3,3)? >>> * Should rand((3,3)) and randn((3,3)) continue to raise a TypeError? >> This is a false dichotomy. There are more choices here. >> >> * Remove rand and randn (at least from the toplevel namespace) and promote the >> use of random_sample and standard_normal which already follow the tuple convention. > > i just wanted to point out another possible choice: > > * enable numpy.rand((3,3)) and make numpy.rand(3,3) raise an error > as zeros and ones do. > > I suppose that you all discussed a lot about this choice also, but it > still seems very reasonable to me :-( Extensively! The problem I have with it is that there are already functions which do exactly that. Why have two functions that do exactly the same thing with exactly the same interface but different names? I say, pick one and get rid of the other if you must remove rand(3,3). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From josh8912 at yahoo.com Sat Jul 8 16:22:03 2006 From: josh8912 at yahoo.com (JJ) Date: Sat, 8 Jul 2006 20:22:03 +0000 (UTC) Subject: [Numpy-discussion] What's wrong with matrices? References: <44AE6675.108@ftw.at> Message-ID: Hello. My 2 cents may be late on this, but I do have some thoughts as a matlab user switching over to numpy/scipy. I will just list my points, and I have a general question at the end. 1) I vote for a simple way to allow matrices to be the default (vs arrays), with all matrix operations/functions/range selections returning matrices. 2) Allowing more than 2D matrices would be nice, but I dont need them often. 3) In some operations, especially range selection operations, a N,1 matrix is turned into a 1,N matrix. This is confusing relative to matlab and problematic, in my view. 3) From my experience so far, range selection is by far the most problematic part of using numpy. In matlab, range selection is very easy and intuitive. For example, if X is a 10 x 5 matrix, a = X[X[:,1]<3,1:3] works just fine. In numpy it is much more confusing and the code to do this is relatively lengthy. It would be nice if range selection for a matrix could be done using any combination of boolean and/or integer matrices (not lists or arrays, necessarily). For example, boolean selection for rows and integer selection for columns. 4) It would be very nice if range selection could be done with real numbers, such as 1.0, 2.0, etc. Matlab can do this and it makes the code much simpler. 5) If X is a 1,n matrix, then X.T should return a n,1 matrix, I think. I guess thats my main points. And now I have a question. What do people think about the role of numpy/scipy vs. R (other than one is python and one is not)? What are the benefits of numpy/scipy over R? In the future, do numpy users want stat libraries like R has? Or put another way, when numpy/scipy is mature, when would you use numpy and when would you use R? Thanks. JJ (John) From sransom at nrao.edu Sat Jul 8 21:23:29 2006 From: sransom at nrao.edu (Scott Ransom) Date: Sat, 8 Jul 2006 21:23:29 -0400 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> <6ef8f3380607081559v62daf4f7t9c3224536e070c0@mail.gmail.com> Message-ID: <20060709012329.GA32684@ssh.cv.nrao.edu> On Sat, Jul 08, 2006 at 06:15:25PM -0500, Robert Kern wrote: > Pau Gargallo wrote: > > On 7/8/06, Robert Kern wrote: > >> Ed Schofield wrote: > >>> * Should numpy.rand and numpy.randn accept sequences of dimensions as > >>> arguments, like rand((3,3)), as an alternative to rand(3,3)? > >>> * Should rand((3,3)) and randn((3,3)) continue to raise a TypeError? > >> This is a false dichotomy. There are more choices here. > >> > >> * Remove rand and randn (at least from the toplevel namespace) and promote the > >> use of random_sample and standard_normal which already follow the tuple convention. > > > > i just wanted to point out another possible choice: > > > > * enable numpy.rand((3,3)) and make numpy.rand(3,3) raise an error > > as zeros and ones do. > > > > I suppose that you all discussed a lot about this choice also, but it > > still seems very reasonable to me :-( > > Extensively! The problem I have with it is that there are already functions > which do exactly that. Why have two functions that do exactly the same thing > with exactly the same interface but different names? I say, pick one and get rid > of the other if you must remove rand(3,3). Robert, what you say is a perfectly valid solution for the spirit of the first selection. My +1 vote was really for _some_ functions in the top level numpy namespace that take tuple "shape" parameters as input and generate uniform and gaussian normal random deviates. If those functions are random_sample (although that is a bad name, IMO, "random" is better -- which is already in there) and standard_normal that is fine by me. The problem was (is?) that rand and randn are not consistent with everything else. _That_ is why people are having issues. Scott -- Scott M. Ransom Address: NRAO Phone: (434) 296-0320 520 Edgemont Rd. email: sransom at nrao.edu Charlottesville, VA 22903 USA GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From haase at msg.ucsf.edu Sun Jul 9 01:16:55 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sat, 08 Jul 2006 22:16:55 -0700 Subject: [Numpy-discussion] build problem: numpy with cygwin mingw Message-ID: <44B09147.1030107@msg.ucsf.edu> Hi, I just tried my first build from a svn checkout on windows using cygwin. I want to use the non cygwin python - i.e. the non-cygwin (mingw) compiler more. I get a presumably very basic error: Even though I follow the simple instructions on http://www.scipy.org/Installing_SciPy/Windows calling python.exe setup.py config --compiler=mingw32 build --compiler=mingw32 bdist_wininst I noticed that gcc gets called without the "-mno-cygwin". From this follows that _WIN32 is not defined and than I get: Could not locate executable gfortran Could not locate executable f95 customize GnuFCompiler customize GnuFCompiler customize GnuFCompiler using config C compiler: gcc -O2 -Wall -Wstrict-prototypes compile options: '-Ic:\python24\include -Inumpy\core\src -Inumpy\core\include -I c:\python24\include -Ic:\python24\PC -c' gcc -O2 -Wall -Wstrict-prototypes -Ic:\python24\include -Inumpy\core\src -Inumpy\core\include -Ic:\python24\include -Ic:\python24\PC -c _configtest.c -o _config test.o In file included from c:/python24/include/Python.h:82, from _configtest.c:2: c:/python24/include/intobject.h:41: error: parse error before "PyInt_AsUnsignedLongLongMask" The error in the last line I traced back to this line: #define PY_LONG_LONG __int64 in pyconfig.h which is used because #if defined(__GNUC__) && defined(_WIN32) is not true if gcc is called without "-mno-cygwin" I don't have cygwin's python installed. I neither have cygwin's non-mingw gcc installed.... According to http://www.python.org/doc/current/inst/tweak-flags.html the --compiler=mingw32 should tell distutils to use the "Cygwin in no-cygwin mode" !?!? What's wrong here ? Thanks Sebastian Haase PS: I don't even care about ATLAS... at this point PPS: Is there maybe a precompiled version later than 0.9.8 avaible !? From oliphant.travis at ieee.org Sun Jul 9 01:20:47 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 08 Jul 2006 23:20:47 -0600 Subject: [Numpy-discussion] NumPy beta release plan In-Reply-To: References: <44AF5BFA.70509@ieee.org> <44AF6704.4080407@sigmasquared.net> <200607081045.53198.dd55@cornell.edu> Message-ID: <44B0922F.6030205@ieee.org> Charles R Harris wrote: > Hi Travis, > > Are you going to make a development branch so other things can > continue going into Numpy until the 1.0 release? Something like that is a good idea. I'm open to suggestions... -Travis From schofield at ftw.at Sun Jul 9 01:58:02 2006 From: schofield at ftw.at (Ed Schofield) Date: Sun, 9 Jul 2006 07:58:02 +0200 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> Message-ID: <2CFD8EBF-9685-4A7A-9905-125A21BE769C@ftw.at> On 09/07/2006, at 12:04 AM, Robert Kern wrote: > Ed Schofield wrote: >> Last week's discussion on rand() and randn() seemed to indicate a >> sentiment that they ought to take tuples for consistency with ones, >> zeros, eye, identity, and empty -- that, although they are supposed >> to be convenience functions, they are inconvenient precisely because >> of their inconsistency with these other functions. This issue has >> been raised many times over the past several months. >> >> Travis made a change in r2572 to allow tuples as arguments, then took >> it out again a few hours later, apparently unsure about whether this >> was a good idea. >> >> I'd like to call for a vote on what people would prefer, and then ask >> Travis to make a final pronouncement before the feature freeze. > > I would like to ask about the purpose of calling for a vote, here. > What > authority do you intend the result to have? If you are just asking > for a straw > poll of opinions from the list to inform Travis' decision, do you > think that he > hasn't read the previous discussions? Are previous non-participants > being drawn > out of the woodwork? ... No authority at all -- it's just a straw poll. My intention is merely to ascertain whether there's indeed a groundswell of public opinion for this change among NumPy's users, as is my suspicion. Previous non-participants are indeed coming out of the woodwork. The reason I've listed only one alternative to the status quo is that this is a simple, concrete proposal that has bubbled up several times from the discussion, which may have broad enough support to tip Travis's decision. I know that you, Robert, disagree with the proposal and have put forward an alternative. Fair enough, but let's now hear what others have to say. -- Ed From schofield at ftw.at Sun Jul 9 02:35:29 2006 From: schofield at ftw.at (Ed Schofield) Date: Sun, 9 Jul 2006 08:35:29 +0200 Subject: [Numpy-discussion] What's wrong with matrices? In-Reply-To: References: <44AE6675.108@ftw.at> Message-ID: <3FE6FE60-D0B9-422B-A73C-1B832FDADB1B@ftw.at> On 08/07/2006, at 10:22 PM, JJ wrote: > 3) In some operations, especially range selection operations, a N, > 1 matrix is > turned into a 1,N matrix. This is confusing relative to matlab and > problematic, > in my view. This sounds like a bug. Can you give any examples of this happening with the latest release or SVN version? > > > 5) If X is a 1,n matrix, then X.T should return a n,1 matrix, I > think. This should be the case. Could you post a code snippet that violates it? Thanks for your feedback, JJ! -- Ed From robert.kern at gmail.com Sun Jul 9 02:44:07 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 09 Jul 2006 01:44:07 -0500 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <2CFD8EBF-9685-4A7A-9905-125A21BE769C@ftw.at> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> <2CFD8EBF-9685-4A7A-9905-125A21BE769C@ftw.at> Message-ID: Ed Schofield wrote: > On 09/07/2006, at 12:04 AM, Robert Kern wrote: > >> I would like to ask about the purpose of calling for a vote, here. >> What >> authority do you intend the result to have? If you are just asking >> for a straw >> poll of opinions from the list to inform Travis' decision, do you >> think that he >> hasn't read the previous discussions? Are previous non-participants >> being drawn >> out of the woodwork? ... > > No authority at all -- it's just a straw poll. My intention is > merely to ascertain whether there's indeed a groundswell of public > opinion for this change among NumPy's users, as is my suspicion. > > Previous non-participants are indeed coming out of the woodwork. > > The reason I've listed only one alternative to the status quo is that > this is a simple, concrete proposal that has bubbled up several times > from the discussion, which may have broad enough support to tip > Travis's decision. > > I know that you, Robert, disagree with the proposal and have put > forward an alternative. Fair enough, but let's now hear what others > have to say. The problem I have is that the way this happened is ineffective at doing that honestly. Fogel's book has a good discussion about how these should be conducted. You can't just pick the options that are convenient for you. You can't argue for your favorite in the CFV. You should provide pointers to the previous discussions so that newcomers can understand what the arguments are. You should encourage people to come up with alternatives that might achieve consensus rather than just picking the ones on the ballot. Otherwise, the poll just serves as rhetorical ploy, not a way to ascertain people's opinions. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From st at sigmasquared.net Sun Jul 9 02:56:44 2006 From: st at sigmasquared.net (Stephan Tolksdorf) Date: Sun, 09 Jul 2006 08:56:44 +0200 Subject: [Numpy-discussion] build problem: numpy with cygwin mingw In-Reply-To: <44B09147.1030107@msg.ucsf.edu> References: <44B09147.1030107@msg.ucsf.edu> Message-ID: <44B0A8AC.2030008@sigmasquared.net> Hi Sebastian, > I just tried my first build from a svn checkout on windows using > cygwin. I want to use the non cygwin python - i.e. the non-cygwin > (mingw) compiler more. > I get a presumably very basic error: > Even though I follow the simple instructions on > http://www.scipy.org/Installing_SciPy/Windows did you apply the patch from ticket #114, as mentioned on top of that page? The Cygwin build is known not to work otherwise (due to incorrect compiler flags, as you noticed)... Stephan From josh8912 at yahoo.com Sun Jul 9 04:10:27 2006 From: josh8912 at yahoo.com (JJ) Date: Sun, 9 Jul 2006 08:10:27 +0000 (UTC) Subject: [Numpy-discussion] What's wrong with matrices? References: <44AE6675.108@ftw.at> <3FE6FE60-D0B9-422B-A73C-1B832FDADB1B@ftw.at> Message-ID: Ed Schofield ftw.at> writes: > > > On 08/07/2006, at 10:22 PM, JJ wrote: > > > 3) In some operations, especially range selection operations, a N, > > 1 matrix is > > turned into a 1,N matrix. This is confusing relative to matlab and > > problematic, > > in my view. > > This sounds like a bug. Can you give any examples of this happening > with the latest release or SVN version? > > > > > > > 5) If X is a 1,n matrix, then X.T should return a n,1 matrix, I > > think. > > This should be the case. Could you post a code snippet that violates > it? > > Thanks for your feedback, JJ! > ----------------------------------------- Hello Ed: Here are a couple of examples off the top of my head: a = mat(arange(10)) a.shape = (5,2) b = a.copy() c = hstack((a,b)) # should return a matrix type(c) a[where(a[:,0]<6)[0],0] #should return a column matrix([[0, 2, 4]]) My version is about a month old, so maybe these are fixed. Since we are on the topic, I dont understand why where() returns a tupple. To me, it just means adding an extra '[0]' in many places in my code. I would vote for where() retuning a matrix (if matrix is the default in the session). My comment on transpose of a 1,n matrix producing a n,1 matrix was in response to a vote question on the list. Also, I would vote for X.A*Y.A to return a matrix, if matrix is the default for a session. Lastly, this might not be the right place for this comment, but it would be nice if pylab allowed numpy matrices. Then code like: plot(d.A.ravel().tolist()) could be changed to plot(d) Hope this is of some help. JJ From josh8912 at yahoo.com Sun Jul 9 04:23:54 2006 From: josh8912 at yahoo.com (JJ) Date: Sun, 9 Jul 2006 08:23:54 +0000 (UTC) Subject: [Numpy-discussion] What's wrong with matrices? References: <44AE6675.108@ftw.at> <3FE6FE60-D0B9-422B-A73C-1B832FDADB1B@ftw.at> Message-ID: Ed Schofield ftw.at> writes: > > > On 08/07/2006, at 10:22 PM, JJ wrote: > > > 3) In some operations, especially range selection operations, a N, > > 1 matrix is > > turned into a 1,N matrix. This is confusing relative to matlab and > > problematic, > > in my view. > > This sounds like a bug. Can you give any examples of this happening > with the latest release or SVN version? > > > > > > > 5) If X is a 1,n matrix, then X.T should return a n,1 matrix, I > > think. > > This should be the case. Could you post a code snippet that violates > it? > > Thanks for your feedback, JJ! >-------------------------------------------------- Hi Ed: Here are a couple more items: c = cov(a) # a is a matrix type(c) # should be a matrix Also, if a is 5x2, then cov(a) is 5,5. Is that just a convention? I would expect the cov to be 2,2, as in matlab. It means I have to use more transpose methods. D is a 1,n matrix (D<2).nonzero()[0] As per my earlier comment on which(), I wish that .nonzero() did not return a tupple. From gruben at bigpond.net.au Sun Jul 9 04:54:39 2006 From: gruben at bigpond.net.au (Gary Ruben) Date: Sun, 09 Jul 2006 18:54:39 +1000 Subject: [Numpy-discussion] NumPy beta release plan In-Reply-To: <44B0922F.6030205@ieee.org> References: <44AF5BFA.70509@ieee.org> <44AF6704.4080407@sigmasquared.net> <200607081045.53198.dd55@cornell.edu> <44B0922F.6030205@ieee.org> Message-ID: <44B0C44F.7020601@bigpond.net.au> Hi Travis, My suggestion: Because you said you want to do a feature freeze, this should be pretty easy to handle. Wait for a feature freeze point then create a 1.0 branch. From then, new stuff can continue going into the trunk. Bug fixes go into the trunk and the 1.0 branch. Gary R. Travis Oliphant wrote: > Charles R Harris wrote: >> Hi Travis, >> >> Are you going to make a development branch so other things can >> continue going into Numpy until the 1.0 release? > Something like that is a good idea. I'm open to suggestions... > > -Travis From MAILER-DAEMON at dodoma.magic.fr Sun Jul 9 06:07:08 2006 From: MAILER-DAEMON at dodoma.magic.fr (Mail Delivery Subsystem) Date: Sun, 9 Jul 2006 12:07:08 +0200 Subject: [Numpy-discussion] Returned mail: see transcript for details Message-ID: <200607091007.k69A78aK031065@dodoma.magic.fr> The original message was received at Sun, 9 Jul 2006 12:07:07 +0200 from asuncion.magic.fr [195.154.233.103] ----- The following addresses had permanent fatal errors ----- (reason: 550 5.1.1 ... User unknown) ----- Transcript of session follows ----- ... while talking to [62.210.158.43]: >>> RCPT To: <<< 550 5.1.1 ... User unknown 550 5.1.1 ... User unknown From numpy-discussion at lists.sourceforge.net Sun Jul 9 06:06:28 2006 From: numpy-discussion at lists.sourceforge.net (numpy-discussion at lists.sourceforge.net) Date: Sun, 9 Jul 2006 12:06:28 +0200 Subject: Hello Message-ID: <200607091006.k69A6W5U030335@manille.magic.fr> A non-text attachment was scrubbed... Name: not available Type: multipart/mixed Size: 49 bytes Desc: not available URL: From svetosch at gmx.net Sun Jul 9 10:55:55 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Sun, 09 Jul 2006 16:55:55 +0200 Subject: [Numpy-discussion] What's wrong with matrices? In-Reply-To: References: <44AE6675.108@ftw.at> <3FE6FE60-D0B9-422B-A73C-1B832FDADB1B@ftw.at> Message-ID: <44B118FB.7040004@gmx.net> JJ schrieb: > ----------------------------------------- > Hello Ed: > Here are a couple of examples off the top of my head: > > a = mat(arange(10)) > a.shape = (5,2) > b = a.copy() > c = hstack((a,b)) # should return a matrix > type(c) > This hstack bug has been fixed recently. > > a[where(a[:,0]<6)[0],0] #should return a column > matrix([[0, 2, 4]]) Don't know anything about this stuff. > > My version is about a month old, so maybe these are fixed. Since we are on the > topic, I dont understand why where() returns a tupple. To me, it just means > adding an extra '[0]' in many places in my code. I would vote for where() > retuning a matrix (if matrix is the default in the session). My comment on > transpose of a 1,n matrix producing a n,1 matrix was in response to a vote > question on the list. That discussion was about pure arrays, not the matrix subclass. > matrix is the default for a session. Lastly, this might not be the right place > for this comment, but it would be nice if pylab allowed numpy matrices. I agree, that would be nice! Maybe we can team up and invade the matplotlib mailing list some time ;-) -sven From esheldon at kicp.uchicago.edu Sun Jul 9 10:55:36 2006 From: esheldon at kicp.uchicago.edu (Erin Sheldon) Date: Sun, 9 Jul 2006 09:55:36 -0500 Subject: [Numpy-discussion] recarray field names Message-ID: <20060709095536.A5542@cfcp.uchicago.edu> Hi everyone - (sigh) These gmail issues are really annoying...so I apologize if the gmail version of this message shows up in a few days. I brought up the issue a while back of having a simple way to access the field names of an array. The quick summary: accessing field names has some oddness that needs cleaning up. The argument that this is important isn't too hard to make: most astronomers work with lists of objects with a slew of properties and these are most naturally stored in a recarray. We use generic file formats like FITS that don't require the user to know what is in the file ahead of time, but once the data is read this information should be easily accessible, both interactively and in code. I'm sure similar arguments can be made for other users. Thanks, Erin On 3/15/06, Travis Oliphant wrote: - Hide quoted text - > Fernando Perez wrote: > > > > > Mmh, just curious: I wonder about the wisdom of that overloading of a > > 'magic' key (-1). It will make thinks like > > > > for name in t.dtype.fields: > > > No real wisdom. More organic growth. Intially I didn't have an ordered > list of fields, but as more complicated data-type descriptors were > supported, this became an important thing to have. I should have > probably added a n additional element to the PyArray_Descr structure. > Remember, it was growing out of the old PyArray_Descr already and I was > a little concerned about changing it too much (there are ramifications > of changing this structure in several places). > > So, instead of adding a "ordered_names" tuple to the dtype object, I > just stuck it in the fields dictionary. I agree it's kind of odd > sitting there. > > It probably deserves a re-factoring and pulling that out into a new > attribute of the dtype object. > > This would mean that the PyArray_Descr structure would need a new object > (names perhaps), and it would need to be tracked. > > Not a huge change and probably worth it before the next release. > > > -Travis From haase at msg.ucsf.edu Sun Jul 9 15:54:52 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sun, 09 Jul 2006 12:54:52 -0700 Subject: [Numpy-discussion] build problem: numpy with cygwin mingw In-Reply-To: <44B0A8AC.2030008@sigmasquared.net> References: <44B09147.1030107@msg.ucsf.edu> <44B0A8AC.2030008@sigmasquared.net> Message-ID: <44B15F0C.7030300@msg.ucsf.edu> Hi Stephan, thanks a lot ! Of course I did NOT read the small print at the top ;-) Now I just have to learn how to apply a patch ;-) (Funny, that after more than ten years of Linux I still don't know this ... ) Thanks again - I'm sure it will work now. (I hope) - Sebastian Stephan Tolksdorf wrote: > Hi Sebastian, > > > I just tried my first build from a svn checkout on windows using > > cygwin. I want to use the non cygwin python - i.e. the non-cygwin > > (mingw) compiler more. > > I get a presumably very basic error: > > Even though I follow the simple instructions on > > http://www.scipy.org/Installing_SciPy/Windows > > did you apply the patch from ticket #114, as mentioned on top of that > page? The Cygwin build is known not to work otherwise (due to incorrect > compiler flags, as you noticed)... > > Stephan From haase at msg.ucsf.edu Sun Jul 9 16:09:31 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sun, 09 Jul 2006 13:09:31 -0700 Subject: [Numpy-discussion] build problem: numpy with cygwin mingw In-Reply-To: <44B15F0C.7030300@msg.ucsf.edu> References: <44B09147.1030107@msg.ucsf.edu> <44B0A8AC.2030008@sigmasquared.net> <44B15F0C.7030300@msg.ucsf.edu> Message-ID: <44B1627B.8060708@msg.ucsf.edu> Hi all, I was to quick with my last reply. The patch in ticket 114 (http://projects.scipy.org/scipy/numpy/ticket/114) refers (among others) to a file distutils/system_info.py but my windows python (version 2.4.3) does not contain any file like that in C:\Python24\Lib\distutils What am I missing this time !? Thanks, Sebastian Haase Sebastian Haase wrote: > Hi Stephan, > thanks a lot ! Of course I did NOT read the small print at the top ;-) > > Now I just have to learn how to apply a patch ;-) > (Funny, that after more than ten years of Linux I still don't know this > ... ) > > Thanks again - I'm sure it will work now. > (I hope) > - Sebastian > > > Stephan Tolksdorf wrote: >> Hi Sebastian, >> >> > I just tried my first build from a svn checkout on windows using >> > cygwin. I want to use the non cygwin python - i.e. the non-cygwin >> > (mingw) compiler more. >> > I get a presumably very basic error: >> > Even though I follow the simple instructions on >> > http://www.scipy.org/Installing_SciPy/Windows >> >> did you apply the patch from ticket #114, as mentioned on top of that >> page? The Cygwin build is known not to work otherwise (due to incorrect >> compiler flags, as you noticed)... >> >> Stephan > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From lists.steve at arachnedesign.net Sun Jul 9 17:06:45 2006 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Sun, 9 Jul 2006 17:06:45 -0400 Subject: [Numpy-discussion] build problem: numpy with cygwin mingw In-Reply-To: <44B1627B.8060708@msg.ucsf.edu> References: <44B09147.1030107@msg.ucsf.edu> <44B0A8AC.2030008@sigmasquared.net> <44B15F0C.7030300@msg.ucsf.edu> <44B1627B.8060708@msg.ucsf.edu> Message-ID: On Jul 9, 2006, at 4:09 PM, Sebastian Haase wrote: > Hi all, > I was to quick with my last reply. > The patch in ticket 114 (http://projects.scipy.org/scipy/numpy/ > ticket/114) > refers (among others) to a file > distutils/system_info.py > but my windows python (version 2.4.3) does not contain any file > like that in > C:\Python24\Lib\distutils > > What am I missing this time !? Perhaps it's referring to the distutils folder in the numpy package .. should be in: /path/to/numpy/checkout/numpy/distutils/system_info.py -steve From haase at msg.ucsf.edu Sun Jul 9 17:54:19 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sun, 09 Jul 2006 14:54:19 -0700 Subject: [Numpy-discussion] build problem: numpy with cygwin mingw In-Reply-To: References: <44B09147.1030107@msg.ucsf.edu> <44B0A8AC.2030008@sigmasquared.net> <44B15F0C.7030300@msg.ucsf.edu> <44B1627B.8060708@msg.ucsf.edu> Message-ID: <44B17B0B.7080609@msg.ucsf.edu> Steve Lianoglou wrote: > On Jul 9, 2006, at 4:09 PM, Sebastian Haase wrote: > >> Hi all, >> I was to quick with my last reply. >> The patch in ticket 114 >> (http://projects.scipy.org/scipy/numpy/ticket/114) >> refers (among others) to a file >> distutils/system_info.py >> but my windows python (version 2.4.3) does not contain any file like >> that in >> C:\Python24\Lib\distutils >> >> What am I missing this time !? > > Perhaps it's referring to the distutils folder in the numpy package .. > should be in: > /path/to/numpy/checkout/numpy/distutils/system_info.py > Hi Steve, Thanks for the hint. Before I was looking for numpy/distutils (which did not exist) and now I found it as numpy/numpy/distutils. Done. Worked - thanks. Now, I was trying to build SciPy - without getting through the ATALS/Lapack precedure ! So I get the expected Warnings (... NOT AVAILABLE) but then (right after those) I get an error: File "C:\Python24\Lib\site-packages\numpy\distutils\system_info.py", line 403, in get_info raise self.notfounderror,self.notfounderror.__doc__ numpy.distutils.system_info.NotFoundError: Some third-party program or library is not found. Does SciPy not come anymore with a fall back non-Lapack version included !? Thanks, Sebastian Haase From haase at msg.ucsf.edu Sun Jul 9 17:59:20 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sun, 09 Jul 2006 14:59:20 -0700 Subject: [Numpy-discussion] numpy dtype str is ' Hi Travis, Is it possible to change dtype str() [and repr() ?] of "standard types" to a more user friendly string: Example: instead of '>> N.array([1,2]) [1 2] >>> _.dtype ' Message-ID: Travis Oliphant ieee.org> writes: > > > Some of you may have noticed that things have been changing rapidly in > the NumPy world (if you were out of the office in June then all the > activity may seem overwhelming). > > All of this activity is based on the fact that the upcoming beta release > will mean a feature freeze for NumPy. Recently numpy.distutils switched to install to /usr/local/lib instead of /usr/lib. Is that intended? I'd prefer to have it back at /usr/lib in 1.0beta. Anyway, I'm really looking forward to seeing a 1.0 release. Keep up the good work. Regards, Christian From wbaxter at gmail.com Sun Jul 9 21:08:44 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 10 Jul 2006 10:08:44 +0900 Subject: [Numpy-discussion] building ATLAS on Win32 - endless compile loop In-Reply-To: <1152370680.2347.1.camel@katholt.sonderport.dk> References: <1152370680.2347.1.camel@katholt.sonderport.dk> Message-ID: I did some searching on the web and found that Pentium M is more or less a Pentium III. After selecting "Pentium III" in the ATLAS setup, the build worked. Still, on my ~1GHz laptop took several hours, nothing like the 15 minutes advertised on the Wiki. (I updated the wiki accordingly) Just out of curiostiy what processor is the official Numpy win32 distribution built for? Or does it just not use optimized blas/lapack? --Bill On 7/8/06, Peter S?ndergaard wrote: > > > > My laptop is a Pentium M, which isn't one of the options on the > > architecture list. So I picked "unknown". > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sun Jul 9 21:13:43 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 09 Jul 2006 20:13:43 -0500 Subject: [Numpy-discussion] NumPy beta release plan In-Reply-To: References: <44AF5BFA.70509@ieee.org> Message-ID: Christian Kristukat wrote: > Travis Oliphant ieee.org> writes: > >> Some of you may have noticed that things have been changing rapidly in >> the NumPy world (if you were out of the office in June then all the >> activity may seem overwhelming). >> >> All of this activity is based on the fact that the upcoming beta release >> will mean a feature freeze for NumPy. > > Recently numpy.distutils switched to install to /usr/local/lib instead of > /usr/lib. Is that intended? I'd prefer to have it back at /usr/lib in > 1.0beta. It shouldn't have. It shouldn't be doing anything different in that respect than core distutils. Are other distutils packages installing to /usr/lib? Nothing in the code should be making that choice. It is up to the configuration of Python (whatever prefix that the interpreter was installed to) or overridden by system or user configuration files. I believe that Debian recently changed its configuration to default to /usr/local (for good reason!). Perhaps that is causing what you see. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ckkart at hoc.net Sun Jul 9 21:59:06 2006 From: ckkart at hoc.net (Christian Kristukat) Date: Mon, 10 Jul 2006 01:59:06 +0000 (UTC) Subject: [Numpy-discussion] NumPy beta release plan References: <44AF5BFA.70509@ieee.org> Message-ID: Robert Kern gmail.com> writes: > I believe that Debian recently changed its configuration to default to > /usr/local (for good reason!). Perhaps that is causing what you see. > I just found out that this is feature/bug of opensuse 10.1. I'm beginning to like it but can you tell what you have to do to force it install to /usr/lib? Btw. (OFF TOPIC) have you noticed that the scipy mailing list is just swallowing mails but not spreading them? Regards, Christian From robert.kern at gmail.com Sun Jul 9 22:08:47 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 09 Jul 2006 21:08:47 -0500 Subject: [Numpy-discussion] NumPy beta release plan In-Reply-To: References: <44AF5BFA.70509@ieee.org> Message-ID: Christian Kristukat wrote: > Robert Kern gmail.com> writes: > >> I believe that Debian recently changed its configuration to default to >> /usr/local (for good reason!). Perhaps that is causing what you see. > > I just found out that this is feature/bug of opensuse 10.1. I'm beginning to > like it but can you tell what you have to do to force it install to /usr/lib? It's probably a good idea to leave it as-is. OS distributions like to control all files in /usr (except /usr/local) and leave /usr/local for sysadmins and users. But if you need more control: http://docs.python.org/inst/alt-install-windows.html (ignore the inappropriate filename; it talks about alternate installation locations for all platforms) http://docs.python.org/inst/config-syntax.html > Btw. (OFF TOPIC) have you noticed that the scipy mailing list is just > swallowing mails but not spreading them? No. That could be a problem on our end. Please let Jeff Strunk know about it. If you can provide details like a local copy of the email that got sent and swallowed, that would be very good. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From wbaxter at gmail.com Sun Jul 9 23:30:24 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 10 Jul 2006 12:30:24 +0900 Subject: [Numpy-discussion] Testing a self-compiled numpy Message-ID: I got numpy compiled according the the instruction on the Wiki, but is there some way to try it out without wiping out my stable install of 0.9.8? I tried modifying my PYTHONPATH to put the new numpy build directory first, but 'import numpy' still picks up the installed 0.9.8 version. I also tried running python from the build directory. Then 'import numpy' prints out "Running from numpy source directory", but the numpy thus imported seems not to have any attributes. Suggestions? --bill -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Sun Jul 9 23:55:35 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Sun, 9 Jul 2006 20:55:35 -0700 Subject: [Numpy-discussion] Testing a self-compiled numpy In-Reply-To: References: Message-ID: On 7/9/06, Bill Baxter wrote: > I got numpy compiled according the the instruction on the Wiki, but is there > some way to try it out without wiping out my stable install of 0.9.8? > > I tried modifying my PYTHONPATH to put the new numpy build directory first, > but 'import numpy' still picks up the installed 0.9.8 version. One quick hack is to install the new numpy somewhere else and then rename the directory containing 0.9.8 to numpySTOP. Then you don't have to worry about the ordering of the path. I install in /usr/local/scipy/numpy (then it is easy to remove numpy without worrying about leaving behind any files) and then I create the file /usr/lib/python2.4/site-packages/addlocalpath.pth that contains the lines: /usr/local/scipy/scipy/lib/python2.4/site-packages/ /usr/local/scipy/numpy/lib/python2.4/site-packages/ Then there is no need to play with the path. From wbaxter at gmail.com Mon Jul 10 00:42:03 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 10 Jul 2006 13:42:03 +0900 Subject: [Numpy-discussion] Testing a self-compiled numpy In-Reply-To: References: Message-ID: Thanks for the reply Keith. On 7/10/06, Keith Goodman wrote: > One quick hack is to install the new numpy somewhere else and then > rename the directory containing 0.9.8 to numpySTOP. Then you don't > have to worry about the ordering of the path. The only thing is that the numpy installer built by the windows instructions on the Wiki doesn't give you an option of where to install numpy. It installs straight to {PYTHON_DIR}\lib\site-packages\numpy. (The command suggested by the wiki is: c:\path\to\python.exe setup.py config --compiler=mingw32 build --compiler=mingw32 bdist_wininst ) But yeh, in a similar vein I can just rename the current numpy directory to numpy_98 or something like that before installing, and that seems to work. I just thought there would be a more sophisticated way to test various verisons of modules in Python than renaming directories. I install in /usr/local/scipy/numpy (then it is easy to remove numpy > without worrying about leaving behind any files) and then I create the > file /usr/lib/python2.4/site-packages/addlocalpath.pth that contains > the lines: > > /usr/local/scipy/scipy/lib/python2.4/site-packages/ > /usr/local/scipy/numpy/lib/python2.4/site-packages/ > > Then there is no need to play with the path. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Jul 10 01:21:30 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 9 Jul 2006 23:21:30 -0600 Subject: [Numpy-discussion] NumPy beta release plan In-Reply-To: References: <44AF5BFA.70509@ieee.org> Message-ID: Hi, On 7/9/06, Robert Kern wrote: > > Christian Kristukat wrote: > > Robert Kern gmail.com> writes: > > > >> I believe that Debian recently changed its configuration to default to > >> /usr/local (for good reason!). Perhaps that is causing what you see. > > > > I just found out that this is feature/bug of opensuse 10.1. I'm > beginning to > > like it but can you tell what you have to do to force it install to > /usr/lib? Suse does that, but so does python by default if you compile it from scratch; Suse is just following the path of least resistence. I don't like it much, I think distributions should leave /usr/local empty, but when you compile your own stuff it should arguably end up in /usr/local by default. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Mon Jul 10 01:41:38 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Sun, 9 Jul 2006 22:41:38 -0700 Subject: [Numpy-discussion] Testing a self-compiled numpy In-Reply-To: References: Message-ID: On 7/9/06, Bill Baxter wrote: > Thanks for the reply Keith. > > > On 7/10/06, Keith Goodman wrote: > > One quick hack is to install the new numpy somewhere else and then > > rename the directory containing 0.9.8 to numpySTOP. Then you don't > > have to worry about the ordering of the path. > > > The only thing is that the numpy installer built by the windows > instructions on the Wiki doesn't give you an option of where to install > numpy. It installs straight to > {PYTHON_DIR}\lib\site-packages\numpy. (The command > suggested by the wiki is: > c:\path\to\python.exe setup.py config --compiler=mingw32 build > --compiler=mingw32 bdist_wininst > ) Doesn't --prefix=/install/numpy/here/ work on windows? From arnd.baecker at web.de Mon Jul 10 01:59:14 2006 From: arnd.baecker at web.de (Arnd Baecker) Date: Mon, 10 Jul 2006 07:59:14 +0200 (CEST) Subject: [Numpy-discussion] logspace behaviour/documentation In-Reply-To: References: Message-ID: Hi Alan, sorry, for answering so late - your message slipped through ... On Fri, 30 Jun 2006, Alan Isaac wrote: > On Fri, 30 Jun 2006, T) Arnd Baecker wrote: > > I am wondering a bit about the the behaviour of logspace: > > http://www.mathworks.com/access/helpdesk/help/techdoc/ref/logspace.html > > fwiw, Alright, if one wants to keep compatibility with matlab, the behaviour of logspace should be unchanged. I'd suggest that something along def logspace(start,stop,num=50,endpoint=True,base=10.0): """Evenly spaced numbers on a logarithmic scale. Return 'int(num)' evenly spaced samples on a logarithmic scale from 'base'**'start' to 'base'**'stop'. If 'endpoint' is True, the last sample is 'base'**'stop'.""" is clearer than the present one Docstring: Evenly spaced numbers on a logarithmic scale. Computes int(num) evenly spaced exponents from start to stop. If endpoint=True, then last exponent is stop. Returns base**exponents. (Note that I am not a native speaker - surprise - ; so maybe the present one is already clear enough?) Best, Arnd From ckkart at hoc.net Mon Jul 10 05:18:53 2006 From: ckkart at hoc.net (Christian Kristukat) Date: Mon, 10 Jul 2006 09:18:53 +0000 (UTC) Subject: [Numpy-discussion] NumPy beta release plan References: <44AF5BFA.70509@ieee.org> Message-ID: Charles R Harris gmail.com> writes: > > > Hi, > > On 7/9/06, Robert Kern gmail.com> wrote: > Christian Kristukat wrote:> Robert Kern gmail.com> writes:>>> I believe that Debian recently changed its configuration to default to > >> /usr/local (for good reason!). Perhaps that is causing what you see.>> I just found out that this is feature/bug of opensuse 10.1. I'm beginning to> like it but can you tell what you have to do to force it install to /usr/lib? > > > Suse does that, but so does python by default if you compile it from scratch; I think every autoconf managed source package out there will install itself to /usr/local if you don't specify the --prefix. So this is not a python thing. > Suse is just following the path of least resistence. I don't like it much, I > think distributions should leave /usr/local empty, but when you compile your > own stuff it should arguably end up in /usr/local by default. The python modules shipped with suse are installed where python resides and that is in /usr. But beginning with suse 10.1 user installed packages go to /usr/local. What I don't like about that method so much is, that when you build a rpm with distutils it gets installed in /usr/local/, too, even though the system does handle third party rpms in the systems rpm database. Christian From nwagner at iam.uni-stuttgart.de Mon Jul 10 05:26:57 2006 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Mon, 10 Jul 2006 11:26:57 +0200 Subject: [Numpy-discussion] Converting a list Message-ID: <44B21D61.70304@iam.uni-stuttgart.de> Hi all, I have a list consisting of arrays of different size data = [array([-1. +0.j, -1.33333333+0.j, -1.66666667+0.j]), array([-2.+0.j , -2.-0.66666667j, -2.-1.33333333j]), array([-2. -2.j, -1.33333333-2.j, -0.66666667-2.j]), array([ 0.-2.j , 0.-1.66666667j, 0.-1.33333333j]), array([ 6.12323400e-17-1.j , -2.58819045e-01-0.96592583j, -5.00000000e-01-0.8660254j , -7.07106781e-01-0.70710678j, -8.66025404e-01-0.5j , -9.65925826e-01-0.25881905j])] type(data) = shape(data) results in shape(data) = Traceback (most recent call last): File "sinai.py", line 107, in ? p = polygon(P) File "sinai.py", line 67, in polygon print 'shape(data) = ',shape(data) File "/usr/lib64/python2.4/site-packages/numpy/core/fromnumeric.py", line 258, in shape result = asarray(a).shape File "/usr/lib64/python2.4/site-packages/numpy/core/numeric.py", line 119, in asarray return array(a, dtype, copy=False, order=order) TypeError: a float is required Is this a bug ? Nils From robert.kern at gmail.com Mon Jul 10 05:57:51 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 10 Jul 2006 04:57:51 -0500 Subject: [Numpy-discussion] Converting a list In-Reply-To: <44B21D61.70304@iam.uni-stuttgart.de> References: <44B21D61.70304@iam.uni-stuttgart.de> Message-ID: Nils Wagner wrote: > Hi all, > > I have a list consisting of arrays of different size > > data = [array([-1. +0.j, -1.33333333+0.j, -1.66666667+0.j]), > array([-2.+0.j , -2.-0.66666667j, -2.-1.33333333j]), > array([-2. -2.j, -1.33333333-2.j, -0.66666667-2.j]), array([ > 0.-2.j , 0.-1.66666667j, 0.-1.33333333j]), array([ > 6.12323400e-17-1.j , -2.58819045e-01-0.96592583j, > -5.00000000e-01-0.8660254j , -7.07106781e-01-0.70710678j, > -8.66025404e-01-0.5j , -9.65925826e-01-0.25881905j])] > > type(data) = > > shape(data) results in > > shape(data) = > Traceback (most recent call last): > File "sinai.py", line 107, in ? > p = polygon(P) > File "sinai.py", line 67, in polygon > print 'shape(data) = ',shape(data) > File "/usr/lib64/python2.4/site-packages/numpy/core/fromnumeric.py", > line 258, in shape > result = asarray(a).shape > File "/usr/lib64/python2.4/site-packages/numpy/core/numeric.py", line > 119, in asarray > return array(a, dtype, copy=False, order=order) > TypeError: a float is required > > Is this a bug ? The error message is unhelpful, certainly, but the *fact* that an exception is raised is not a bug. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From nwagner at iam.uni-stuttgart.de Mon Jul 10 08:33:52 2006 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Mon, 10 Jul 2006 14:33:52 +0200 Subject: [Numpy-discussion] Number of digits Message-ID: <44B24930.1080603@iam.uni-stuttgart.de> Hi all, how can I increase the number of digits in the output of str(.) ? >>> lam**2 493.48022005446808 >>> str(lam**2) '493.480220054' '493.4802200544680' would be fine. Nils From david.douard at logilab.fr Mon Jul 10 09:09:07 2006 From: david.douard at logilab.fr (David Douard) Date: Mon, 10 Jul 2006 15:09:07 +0200 Subject: [Numpy-discussion] Number of digits In-Reply-To: <44B24930.1080603@iam.uni-stuttgart.de> References: <44B24930.1080603@iam.uni-stuttgart.de> Message-ID: <20060710130906.GA16160@crater.logilab.fr> On Mon, Jul 10, 2006 at 02:33:52PM +0200, Nils Wagner wrote: > Hi all, > > how can I increase the number of digits in the output of str(.) ? > > >>> lam**2 > 493.48022005446808 > >>> str(lam**2) > '493.480220054' > > '493.4802200544680' would be fine. Maybe something like >>> '%.13f'%(lam**2) David -- David Douard LOGILAB, Paris (France) Formations Python, Zope, Plone, Debian : http://www.logilab.fr/formations D?veloppement logiciel sur mesure : http://www.logilab.fr/services Informatique scientifique : http://www.logilab.fr/science -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From tim.hochberg at cox.net Mon Jul 10 09:30:06 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon, 10 Jul 2006 06:30:06 -0700 Subject: [Numpy-discussion] Number of digits In-Reply-To: <44B24930.1080603@iam.uni-stuttgart.de> References: <44B24930.1080603@iam.uni-stuttgart.de> Message-ID: <44B2565E.8010208@cox.net> Nils Wagner wrote: > Hi all, > > how can I increase the number of digits in the output of str(.) ? > You can't as far as I know. For floats, you can use "%.nf". For example: "%.13f" % 493.4802200544680 -tim > >>>> lam**2 >>>> > 493.48022005446808 > >>>> str(lam**2) >>>> > '493.480220054' > > '493.4802200544680' would be fine. > > Nils > > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From robert.kern at gmail.com Mon Jul 10 09:46:33 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 10 Jul 2006 08:46:33 -0500 Subject: [Numpy-discussion] Number of digits In-Reply-To: <44B2565E.8010208@cox.net> References: <44B24930.1080603@iam.uni-stuttgart.de> <44B2565E.8010208@cox.net> Message-ID: Tim Hochberg wrote: > Nils Wagner wrote: >> Hi all, >> >> how can I increase the number of digits in the output of str(.) ? >> > You can't as far as I know. For floats, you can use "%.nf". For example: > > "%.13f" % 493.4802200544680 The problem is is that he doesn't have a float. He has one of our float64scalar objects. The str() of a real Python float will give as many digits as are necessary to recreate number and no more (or maybe one or two more). A str() of a float64scalar will round according to some rule that I haven't figured out, yet. It doesn't seem to be configurable with numpy.set_printoptions(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david.douard at logilab.fr Mon Jul 10 10:08:23 2006 From: david.douard at logilab.fr (David Douard) Date: Mon, 10 Jul 2006 16:08:23 +0200 Subject: [Numpy-discussion] Number of digits In-Reply-To: References: <44B24930.1080603@iam.uni-stuttgart.de> <44B2565E.8010208@cox.net> Message-ID: <20060710140823.GB16160@crater.logilab.fr> On Mon, Jul 10, 2006 at 08:46:33AM -0500, Robert Kern wrote: > Tim Hochberg wrote: > > Nils Wagner wrote: > >> Hi all, > >> > >> how can I increase the number of digits in the output of str(.) ? > >> > > You can't as far as I know. For floats, you can use "%.nf". For example: > > > > "%.13f" % 493.4802200544680 > > The problem is is that he doesn't have a float. He has one of our float64scalar > objects. The str() of a real Python float will give as many digits as are > necessary to recreate number and no more (or maybe one or two more). A str() of > a float64scalar will round according to some rule that I haven't figured out, > yet. It doesn't seem to be configurable with numpy.set_printoptions(). This is a different ptoblem from the one exposed by Nils. I mean, tha fact that str() on numpy.float64 objects is somewhat obscure is a problem that should obviously be addressed some day. However, as far as I understand Nils' message, the "%.13f" trick is enough for what he need. But I may be wrong... He just want to "increase the number of digits", not have the "optimal" number of digits (as long as this is meaningfull). But I may have missed something. David -- David Douard LOGILAB, Paris (France) Formations Python, Zope, Plone, Debian : http://www.logilab.fr/formations D?veloppement logiciel sur mesure : http://www.logilab.fr/services Informatique scientifique : http://www.logilab.fr/science -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From robert.kern at gmail.com Mon Jul 10 10:06:47 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 10 Jul 2006 09:06:47 -0500 Subject: [Numpy-discussion] Number of digits In-Reply-To: References: <44B24930.1080603@iam.uni-stuttgart.de> <44B2565E.8010208@cox.net> Message-ID: Robert Kern wrote: > Tim Hochberg wrote: >> Nils Wagner wrote: >>> Hi all, >>> >>> how can I increase the number of digits in the output of str(.) ? >>> >> You can't as far as I know. For floats, you can use "%.nf". For example: >> >> "%.13f" % 493.4802200544680 > > The problem is is that he doesn't have a float. He has one of our float64scalar > objects. The str() of a real Python float will give as many digits as are > necessary to recreate number and no more (or maybe one or two more). A str() of > a float64scalar will round according to some rule that I haven't figured out, > yet. It doesn't seem to be configurable with numpy.set_printoptions(). No, that's entirely wrong. Please ignore the ravings of a madman. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From bhendrix at enthought.com Mon Jul 10 10:16:46 2006 From: bhendrix at enthought.com (Bryce Hendrix) Date: Mon, 10 Jul 2006 09:16:46 -0500 Subject: [Numpy-discussion] Testing a self-compiled numpy In-Reply-To: References: Message-ID: <44B2614E.7060101@enthought.com> Keith Goodman wrote: >> The only thing is that the numpy installer built by the windows >> instructions on the Wiki doesn't give you an option of where to install >> numpy. It installs straight to >> {PYTHON_DIR}\lib\site-packages\numpy. (The command >> suggested by the wiki is: >> c:\path\to\python.exe setup.py config --compiler=mingw32 build >> --compiler=mingw32 bdist_wininst >> ) >> > > Doesn't --prefix=/install/numpy/here/ work on windows? > For our Windows Enthon project, the build command is setup.py config --compiler=mingw32 build --compiler=mingw32 install --prefix=path_to_where_numpy_goes. Bryce -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Jul 10 10:15:38 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 10 Jul 2006 09:15:38 -0500 Subject: [Numpy-discussion] Number of digits In-Reply-To: <20060710140823.GB16160@crater.logilab.fr> References: <44B24930.1080603@iam.uni-stuttgart.de> <44B2565E.8010208@cox.net> <20060710140823.GB16160@crater.logilab.fr> Message-ID: David Douard wrote: > On Mon, Jul 10, 2006 at 08:46:33AM -0500, Robert Kern wrote: >> Tim Hochberg wrote: >>> Nils Wagner wrote: >>>> Hi all, >>>> >>>> how can I increase the number of digits in the output of str(.) ? >>>> >>> You can't as far as I know. For floats, you can use "%.nf". For example: >>> >>> "%.13f" % 493.4802200544680 >> The problem is is that he doesn't have a float. He has one of our float64scalar >> objects. The str() of a real Python float will give as many digits as are >> necessary to recreate number and no more (or maybe one or two more). A str() of >> a float64scalar will round according to some rule that I haven't figured out, >> yet. It doesn't seem to be configurable with numpy.set_printoptions(). > > This is a different ptoblem from the one exposed by Nils. I mean, tha > fact that str() on numpy.float64 objects is somewhat obscure is a > problem that should obviously be addressed some day. However, as far as > I understand Nils' message, the "%.13f" trick is enough for what he > need. But I may be wrong... He just want to "increase the number of > digits", not have the "optimal" number of digits (as long as this is > meaningfull). > > But I may have missed something. No, you didn't miss anything. I just haven't gotten enough sleep. I thought that Python floats had a __str__ that computed just as many places as necessary, but it looks like it just rounds at 12 places. This is not an adjustable parameter (barring modifying the C code of the interpreter). I also thought that the float scalars were being printed differently. However, the code for the float64scalar __str__ just gets the double value, creates a Python float object from it, and returns the __str__ result from that object. Everything works as intended. Nothing to see here. Move along. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From tom.denniston at alum.dartmouth.org Mon Jul 10 11:42:50 2006 From: tom.denniston at alum.dartmouth.org (Tom Denniston) Date: Mon, 10 Jul 2006 10:42:50 -0500 Subject: [Numpy-discussion] PyObject arrays, pickling, tofile In-Reply-To: <44AC80AC.1030708@ieee.org> References: <44AC80AC.1030708@ieee.org> Message-ID: Travis, Thanks for your help. Sorry for taking so long to get back to you. I wasn't on email for a few days. I seem to be able to pickle and restore PyObject arrays but then concatenates and takes give me the following error: TypeError: fields with object members not yet supported. What is even stranger is my stack trace stops at the call to numpy in my code and doesn't show where it is failing in the numpy code. The same operation on the numpy array that hasn't been pickled and loaded back, works fine. --Tom From parejkoj at drexel.edu Mon Jul 10 12:46:45 2006 From: parejkoj at drexel.edu (John Parejko) Date: Mon, 10 Jul 2006 12:46:45 -0400 Subject: [Numpy-discussion] recarray field names Message-ID: <44B28475.9010602@drexel.edu> Howdy! I just wanted to voice my agreement with this statment by Erin Sheldon: > I brought up the issue a while back of having a simple way to > access the field names of an array. The quick summary: accessing > field names has some oddness that needs cleaning up. Sometimes a['name'], sometimes a.field('name'). I vastly prefer the first version; it has become my favored way of dealing with my FITS and ASCII data. But using the bracket notation sometimes gives an error, and I haven't quite figured out what the circumstances are. Also, "for name in a.dtype.fields:" or "for name in a.fields:" would be very, very handy. Could this sort of thing get put in before the beta (or is it already in, and I missed the boat, again?). Thanks John -- ************************* John Parejko Department of Physics and Astronomy 215 895-2786 Drexel University Philadelphia, PA ************************** From pgmdevlist at mailcan.com Mon Jul 10 13:46:07 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Mon, 10 Jul 2006 13:46:07 -0400 Subject: [Numpy-discussion] Converting a list In-Reply-To: References: <44B21D61.70304@iam.uni-stuttgart.de> Message-ID: <200607101346.07555.pgmdevlist@mailcan.com> On Monday 10 July 2006 05:57, Robert Kern wrote: > Nils Wagner wrote: > > Hi all, > > > > I have a list consisting of arrays of different size ... > > The error message is unhelpful, certainly, but the *fact* that an exception > is raised is not a bug. Attempting to build an array from a list of sequences with different sizes seem to be a rather common mistake (judging from the recurrence of the topic on that list). As Robert pointed, the current error message is not helpful. Could it be possible to change it ? A "(inconsistent sizes ?)" could at least be added... From rowen at cesmail.net Mon Jul 10 14:17:02 2006 From: rowen at cesmail.net (Russell E. Owen) Date: Mon, 10 Jul 2006 11:17:02 -0700 Subject: [Numpy-discussion] NumPy beta release plan References: <44AF5BFA.70509@ieee.org> Message-ID: In article <44AF5BFA.70509 at ieee.org>, Travis Oliphant wrote: > 3) C-API names have prefix PyArray_ (like always), NPY_ or npy_. > > The NPY_ and npy_ prefixes are new and were done to remove the > likelihood of name collisions when NumPy is used with another > library. The old (and un-prefixed) names are accessible by > importing numpy/noprefix.h instead of numpy/arrayobject.h This may be too late or too picky, but... This seems like too many choices, leading to possible confusion when reading other people's extensions or the numpy code itself. Could you standardize on one (I like npy_) or two (if you have to keep PyArray_ for some reason and can't put it in a backwards-compatibility header somewhere). Overall it's great. I'm really glad to hear you're so close to freezing the features. -- Russell From kwgoodman at gmail.com Mon Jul 10 14:30:09 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 10 Jul 2006 11:30:09 -0700 Subject: [Numpy-discussion] .M .A .T .H attribute result In-Reply-To: <44AEC258.9040800@ee.byu.edu> References: <44AEC258.9040800@ee.byu.edu> Message-ID: On 7/7/06, Travis Oliphant wrote: > a numpy.matlib module was started to store matrix versions of the > standard array-creation functions and mat was re-labeled to "asmatrix" > so that a copy is not made by default. Holy crap! It works. This is great. Thank you. >> import numpy.matlib >> numpy.__version__ '0.9.9.2788' >> from numpy.matlib import * >> rand(2,2) matrix([[ 0.23834437, 0.60329722], <--- matrix by default [ 0.03907811, 0.55134035]]) >> ones((2,2)) matrix([[ 1., 1.], <--- matrix by default [ 1., 1.]]) >> numpy.matlib. numpy.matlib.N numpy.matlib.__name__ numpy.matlib.eye numpy.matlib.__all__ numpy.matlib.__new__ numpy.matlib.identity numpy.matlib.__class__ numpy.matlib.__reduce__ numpy.matlib.matrix numpy.matlib.__delattr__ numpy.matlib.__reduce_ex__ numpy.matlib.ndarray numpy.matlib.__dict__ numpy.matlib.__repr__ numpy.matlib.ones numpy.matlib.__doc__ numpy.matlib.__setattr__ numpy.matlib.rand numpy.matlib.__file__ numpy.matlib.__str__ numpy.matlib.randn numpy.matlib.__getattribute__ numpy.matlib.array numpy.matlib.zeros numpy.matlib.__hash__ numpy.matlib.asmatrix numpy.matlib.__init__ numpy.matlib.empty From Chris.Barker at noaa.gov Mon Jul 10 14:30:59 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 10 Jul 2006 11:30:59 -0700 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> Message-ID: <44B29CE3.9010504@noaa.gov> Ed Schofield wrote: > * Should numpy.rand and numpy.randn accept sequences of dimensions as > arguments, like rand((3,3)), as an alternative to rand(3,3)? +1 +1 for ONLY allowing tuples. Or just get rid of it. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From robert.kern at gmail.com Mon Jul 10 14:41:04 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 10 Jul 2006 13:41:04 -0500 Subject: [Numpy-discussion] Converting a list In-Reply-To: <200607101346.07555.pgmdevlist@mailcan.com> References: <44B21D61.70304@iam.uni-stuttgart.de> <200607101346.07555.pgmdevlist@mailcan.com> Message-ID: Pierre GM wrote: > On Monday 10 July 2006 05:57, Robert Kern wrote: >> Nils Wagner wrote: >>> Hi all, >>> >>> I have a list consisting of arrays of different size > ... >> The error message is unhelpful, certainly, but the *fact* that an exception >> is raised is not a bug. > > Attempting to build an array from a list of sequences with different sizes > seem to be a rather common mistake (judging from the recurrence of the topic > on that list). > As Robert pointed, the current error message is not helpful. > Could it be possible to change it ? A "(inconsistent sizes ?)" could at least > be added... People have looked at this before. IIRC, the conclusion was that at the specific place where the exception needs to be raised, the information that it came from a "ragged" input is lost, and that exception can be raised in other circumstances as well (so it can't just be reworded). The number of cases that array() tries to handle automatically is quite large, and that makes for hairy code. OTOH, if you can pinpoint a place where we can raise a more specific exception without changing the semantics of the function, I'd be happy to apply your patch. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at ee.byu.edu Mon Jul 10 14:54:43 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 10 Jul 2006 12:54:43 -0600 Subject: [Numpy-discussion] NumPy beta release plan In-Reply-To: References: <44AF5BFA.70509@ieee.org> Message-ID: <44B2A273.4050502@ee.byu.edu> Christian Kristukat wrote: >Travis Oliphant ieee.org> writes: > > > >>Some of you may have noticed that things have been changing rapidly in >>the NumPy world (if you were out of the office in June then all the >>activity may seem overwhelming). >> >>All of this activity is based on the fact that the upcoming beta release >>will mean a feature freeze for NumPy. >> >> > >Recently numpy.distutils switched to install to /usr/local/lib instead of >/usr/lib. Is that intended? I'd prefer to have it back at /usr/lib in >1.0beta. > > What gets installed in /usr/lib ? I'm pretty sure the install directory is determined by the Python installation. Exceptions to this are probably bugs. -Travis From oliphant at ee.byu.edu Mon Jul 10 15:04:07 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 10 Jul 2006 13:04:07 -0600 Subject: [Numpy-discussion] recarray field names In-Reply-To: <44B28475.9010602@drexel.edu> References: <44B28475.9010602@drexel.edu> Message-ID: <44B2A4A7.6050405@ee.byu.edu> John Parejko wrote: >Howdy! I just wanted to voice my agreement with this statment by Erin Sheldon: > > > I brought up the issue a while back of having a simple way to > > access the field names of an array. The quick summary: accessing > > field names has some oddness that needs cleaning up. > >Sometimes a['name'], sometimes a.field('name'). I vastly prefer the first >version; it has become my favored way of dealing with my FITS and ASCII >data. But using the bracket notation sometimes gives an error, and I >haven't quite figured out what the circumstances are. > >Also, "for name in a.dtype.fields:" or "for name in a.fields:" would be >very, very handy. > >Could this sort of thing get put in before the beta (or is it already in, >and I missed the boat, again?). > > > It's actually there already. The issue at hand is that the fields dictionary contains an additional member keyed by a -1. This odd-ball entry in the dictionary is to obtain an "ordered" list of the fields. The order is determined by the offset. This is a warty situation but the functionality is there. Probably a better solution is to add a names attribute to the dtype object that returns the ordered list. In C, the PyArray_Descr structure would grow an additional names member that contains the ordered list of names instead of sticking it as a -1 key entry in the fields dictionary which was and is a hack. Let's schedule this for pre 1.0 beta -Travis From oliphant at ee.byu.edu Mon Jul 10 15:06:10 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 10 Jul 2006 13:06:10 -0600 Subject: [Numpy-discussion] NumPy beta release plan In-Reply-To: References: <44AF5BFA.70509@ieee.org> Message-ID: <44B2A522.4000501@ee.byu.edu> Russell E. Owen wrote: >In article <44AF5BFA.70509 at ieee.org>, > Travis Oliphant wrote: > > > >>3) C-API names have prefix PyArray_ (like always), NPY_ or npy_. >> >> The NPY_ and npy_ prefixes are new and were done to remove the >> likelihood of name collisions when NumPy is used with another >> library. The old (and un-prefixed) names are accessible by >> importing numpy/noprefix.h instead of numpy/arrayobject.h >> >> > >This may be too late or too picky, but... > >This seems like too many choices, leading to possible confusion when >reading other people's extensions or the numpy code itself. Could you >standardize on one (I like npy_) or two (if you have to keep PyArray_ >for some reason and can't put it in a backwards-compatibility header >somewhere). > > > There is a logic behind it though. The C-API function calls all have PyArray_ prefixes (as always in Numeric) The macro captialized constants have NPY_ prefixes Newly defined types with lower-case names have npy_ prefixes -Travis From ndarray at mac.com Mon Jul 10 16:10:56 2006 From: ndarray at mac.com (Sasha) Date: Mon, 10 Jul 2006 16:10:56 -0400 Subject: [Numpy-discussion] NumPy beta release plan In-Reply-To: References: <44AF5BFA.70509@ieee.org> Message-ID: I would like to add my voice to a call for single prefix (I like PyArray_ for everything, but can live with npy_). Presumably npy_ vs. NPY_ variation is supposed to distinguish between macros and C language elements, but as long as the stem is properly capitalized, I don't see a problem of using the same prefix for everything. In any case, if more than one prefix is used, it would be helpful to get some guidance on what to use when. On 7/10/06, Russell E. Owen wrote: > In article <44AF5BFA.70509 at ieee.org>, > Travis Oliphant wrote: > > > 3) C-API names have prefix PyArray_ (like always), NPY_ or npy_. > > > > The NPY_ and npy_ prefixes are new and were done to remove the > > likelihood of name collisions when NumPy is used with another > > library. The old (and un-prefixed) names are accessible by > > importing numpy/noprefix.h instead of numpy/arrayobject.h > > This may be too late or too picky, but... > > This seems like too many choices, leading to possible confusion when > reading other people's extensions or the numpy code itself. Could you > standardize on one (I like npy_) or two (if you have to keep PyArray_ > for some reason and can't put it in a backwards-compatibility header > somewhere). > > Overall it's great. I'm really glad to hear you're so close to freezing > the features. > > -- Russell > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From oliphant at ee.byu.edu Mon Jul 10 19:09:48 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 10 Jul 2006 17:09:48 -0600 Subject: [Numpy-discussion] recarray field names In-Reply-To: <44B28475.9010602@drexel.edu> References: <44B28475.9010602@drexel.edu> Message-ID: <44B2DE3C.7060801@ee.byu.edu> John Parejko wrote: >Howdy! I just wanted to voice my agreement with this statment by Erin Sheldon: > > > I brought up the issue a while back of having a simple way to > > access the field names of an array. The quick summary: accessing > > field names has some oddness that needs cleaning up. > >Sometimes a['name'], sometimes a.field('name'). I vastly prefer the first >version; it has become my favored way of dealing with my FITS and ASCII >data. But using the bracket notation sometimes gives an error, and I >haven't quite figured out what the circumstances are. > > Bracketed version should never give an error. If you see it, it's a bug. >Also, "for name in a.dtype.fields:" or "for name in a.fields:" would be >very, very handy. > > You can do this, now but each key in the fields dictionary might not be unique because titles will also be keys in the dictionary. A unique list of ordered names can now (in NumPy SVN) be obtained using for name in a.dtype.names -Travis From oliphant at ee.byu.edu Mon Jul 10 19:32:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 10 Jul 2006 17:32:04 -0600 Subject: [Numpy-discussion] NumPy beta release plan In-Reply-To: References: <44AF5BFA.70509@ieee.org> Message-ID: <44B2E374.60607@ee.byu.edu> Sasha wrote: >I would like to add my voice to a call for single prefix (I like >PyArray_ for everything, but can live with npy_). Presumably npy_ >vs. NPY_ variation is supposed to distinguish between macros and C >language elements, but as long as the stem is properly capitalized, I >don't see a problem of using the same prefix for everything. > >In any case, if more than one prefix is used, it would be helpful to >get some guidance on what to use when. > > PyArray_ --- Functions and/or macro-functions NPY_ --- Constants or enums npy_ --- typedef'd types (npy_int16, npy_float32, npy_intp) -Travis From tom.denniston at alum.dartmouth.org Mon Jul 10 20:59:25 2006 From: tom.denniston at alum.dartmouth.org (Tom Denniston) Date: Mon, 10 Jul 2006 19:59:25 -0500 Subject: [Numpy-discussion] Strange error taking log of PyObject array Message-ID: The following works on a float array but not an object array. It gives a very strange error message. (Pdb) numpy.log(numpy.array([19155613843.7], dtype=object)) *** AttributeError: 'float' object has no attribute 'log' (Pdb) numpy.log([19155613843.7]) array([ 23.67586166]) From oliphant.travis at ieee.org Mon Jul 10 21:46:21 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 10 Jul 2006 19:46:21 -0600 Subject: [Numpy-discussion] Strange error taking log of PyObject array In-Reply-To: References: Message-ID: <44B302ED.1080104@ieee.org> Tom Denniston wrote: > The following works on a float array but not an object array. It > gives a very strange error message. > > (Pdb) numpy.log(numpy.array([19155613843.7], dtype=object)) > *** AttributeError: 'float' object has no attribute 'log' > This is expected behavior. For object arrays the ufuncs look to the objects themselves for implementation. In this case. The ufunc tries to call the log method of each entry in the array. The first entry is a float (which as the error shows) does not have a log attribute and so you get a failure. -Travis From swwkofbr at ec.hokudai.ac.jp Tue Jul 11 03:32:53 2006 From: swwkofbr at ec.hokudai.ac.jp (lighting displays) Date: Tue, 11 Jul 2006 05:32:53 -0200 Subject: [Numpy-discussion] rotten foul. spotted teensy Message-ID: <800C2E05027B674.1345572@ec.hokudai.ac.jp> G0ldmark Industries, Inc ( G D K I ) THIS ST()CK IS E> Just tested the lastest SVN and it works as advertised. Thanks Travis. An unrelated question: why does this work for arrays but not recarrays? In [24]: mydescriptor = [('age',float64),('Nchildren',int8),('weight',float32)] In [25]: a = array([(64,2,75.0),(25,0,60.0)], dtype=mydescriptor) In [26]: a = recarray([(64,2,75.0),(25,0,60.0)], dtype=mydescriptor) --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /Users/esheldon/ TypeError: __new__() got an unexpected keyword argument 'dtype' I understand that I could use the formats and names keywords, but this seems a little inconsistent. Erin On 7/10/06, Travis Oliphant wrote: > John Parejko wrote: > > >Howdy! I just wanted to voice my agreement with this statment by Erin > >Sheldon: > > > > > I brought up the issue a while back of having a simple way to > > > access the field names of an array. The quick summary: accessing > > > field names has some oddness that needs cleaning up. > > > >Sometimes a['name'], sometimes a.field('name'). I vastly prefer the first > >version; it has become my favored way of dealing with my FITS and ASCII > >data. But using the bracket notation sometimes gives an error, and I > >haven't quite figured out what the circumstances are. > > > > > > Bracketed version should never give an error. If you see it, it's a bug. > > >Also, "for name in a.dtype.fields:" or "for name in a.fields:" would be > >very, very handy. > > > > > You can do this, now but each key in the fields dictionary might not be > unique because titles will also be keys in the dictionary. > > A unique list of ordered names can now (in NumPy SVN) be obtained using > > for name in a.dtype.names > > > -Travis > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From oliphant.travis at ieee.org Tue Jul 11 00:19:44 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 10 Jul 2006 22:19:44 -0600 Subject: [Numpy-discussion] recarray field names In-Reply-To: <20060710225459.A24557@cfcp.uchicago.edu> References: <20060710225459.A24557@cfcp.uchicago.edu> Message-ID: <44B326E0.6020406@ieee.org> Erin Sheldon wrote: > Just tested the lastest SVN and it works as advertised. Thanks > Travis. > > An unrelated question: why does this work for arrays but not recarrays? > > > In [24]: mydescriptor = [('age',float64),('Nchildren',int8),('weight',float32)] > > In [25]: a = array([(64,2,75.0),(25,0,60.0)], dtype=mydescriptor) > > In [26]: a = recarray([(64,2,75.0),(25,0,60.0)], dtype=mydescriptor) > --------------------------------------------------------------------------- > exceptions.TypeError Traceback (most recent > call last) > > /Users/esheldon/ > > TypeError: __new__() got an unexpected keyword argument 'dtype' > > I understand that I could use the formats and names keywords, but > this seems a little inconsistent. > > Well there are a couple of reasons this doesn't work. 1) the recarray constructor is similar to the ndarray constructor. Neither of these take list arguments as inputs. So, while I've added dtype as the keyword for the formats input of the recarray constructor, the second example will still fail. 2) Even were you to use the rec.array function (which is analagous to the numpy.array function) to produce the error, it does not take a dtype keyword. This is inconsistent. But, the interface for the record array class came from numarray. I tried to stay compatible with those functions so the numarray people would have an easier time adapting. Perhaps we should add a dtype keyword argument to all the constructors in numpy/core/records.py which over-rides the formats, names, and titles keywords so that you could do a = rec.array([...], dtype=mine) analgously to a = array([...], dtype=mine) -Travis From nadavh at visionsense.com Tue Jul 11 01:33:41 2006 From: nadavh at visionsense.com (Nadav Horesh) Date: Tue, 11 Jul 2006 07:33:41 +0200 Subject: [Numpy-discussion] Converting a list Message-ID: <07C6A61102C94148B8104D42DE95F7E8C8EFF4@exchange2k.envision.co.il> Do you mean: >> map(shape, data) -----Original Message----- From: numpy-discussion-bounces at lists.sourceforge.net on behalf of Nils Wagner Sent: Mon 10-Jul-06 12:26 To: numpy-discussion at lists.sourceforge.net Cc: Subject: [Numpy-discussion] Converting a list Hi all, I have a list consisting of arrays of different size data = [array([-1. +0.j, -1.33333333+0.j, -1.66666667+0.j]), array([-2.+0.j , -2.-0.66666667j, -2.-1.33333333j]), array([-2. -2.j, -1.33333333-2.j, -0.66666667-2.j]), array([ 0.-2.j , 0.-1.66666667j, 0.-1.33333333j]), array([ 6.12323400e-17-1.j , -2.58819045e-01-0.96592583j, -5.00000000e-01-0.8660254j , -7.07106781e-01-0.70710678j, -8.66025404e-01-0.5j , -9.65925826e-01-0.25881905j])] type(data) = shape(data) results in shape(data) = Traceback (most recent call last): File "sinai.py", line 107, in ? p = polygon(P) File "sinai.py", line 67, in polygon print 'shape(data) = ',shape(data) File "/usr/lib64/python2.4/site-packages/numpy/core/fromnumeric.py", line 258, in shape result = asarray(a).shape File "/usr/lib64/python2.4/site-packages/numpy/core/numeric.py", line 119, in asarray return array(a, dtype, copy=False, order=order) TypeError: a float is required Is this a bug ? Nils ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From tim.hochberg at cox.net Tue Jul 11 00:35:35 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon, 10 Jul 2006 21:35:35 -0700 Subject: [Numpy-discussion] Strange error taking log of PyObject array In-Reply-To: <44B302ED.1080104@ieee.org> References: <44B302ED.1080104@ieee.org> Message-ID: <44B32A97.4070003@cox.net> Travis Oliphant wrote: > Tom Denniston wrote: > >> The following works on a float array but not an object array. It >> gives a very strange error message. >> >> (Pdb) numpy.log(numpy.array([19155613843.7], dtype=object)) >> *** AttributeError: 'float' object has no attribute 'log' >> >> > This is expected behavior. For object arrays the ufuncs look to the > objects themselves for implementation. In this case. The ufunc tries > to call the log method of each entry in the array. The first entry is a > float (which as the error shows) does not have a log attribute and so > you get a failure. > It seems like we could do better than this. Couldn't the ufunc call *itself* on on objects in an object array? That is, ufunc(object_array) would be more or less equivalent to map(ufunc, object_array) [mostly less, but hopefully you get the idea]. I haven't looked into the gory details, and I'm not likely to soon since I'm tied up with other stuff, but I figured I'd toss the idea into the ring. -tim From esheldon at kicp.uchicago.edu Tue Jul 11 01:01:57 2006 From: esheldon at kicp.uchicago.edu (Erin Sheldon) Date: Tue, 11 Jul 2006 00:01:57 -0500 Subject: [Numpy-discussion] recarray field names Message-ID: <20060711000157.A24853@cfcp.uchicago.edu> On 7/11/06, Travis Oliphant wrote: > Erin Sheldon wrote: > > Just tested the lastest SVN and it works as advertised. Thanks > > Travis. > > > > An unrelated question: why does this work for arrays but not recarrays? > > > > > > In [24]: mydescriptor = > > [('age',float64),('Nchildren',int8),('weight',float32)] > > > > In [25]: a = array([(64,2,75.0),(25,0,60.0)], dtype=mydescriptor) > > > > In [26]: a = recarray([(64,2,75.0),(25,0,60.0)], dtype=mydescriptor) > > --------------------------------------------------------------------------- > > exceptions.TypeError Traceback (most recent > > call last) > > > > /Users/esheldon/ > > > > TypeError: __new__() got an unexpected keyword argument 'dtype' > > > > I understand that I could use the formats and names keywords, but > > this seems a little inconsistent. > > > > > Well there are a couple of reasons this doesn't work. > > 1) the recarray constructor is similar to the ndarray constructor. > > Neither of these take list arguments as inputs. So, while I've added > dtype as the > keyword for the formats input of the recarray constructor, the second > example will still fail. > > 2) Even were you to use the rec.array function (which is analagous to > the numpy.array function) to produce the error, it does not take a dtype > keyword. > > This is inconsistent. But, the interface for the record array class > came from numarray. I tried to stay compatible with those functions so > the numarray people would have an easier time adapting. I see. > > Perhaps we should add a dtype keyword argument to all the > constructors in numpy/core/records.py which over-rides the formats, > names, and titles keywords so that you could do > > a = rec.array([...], dtype=mine) > > analgously to > > a = array([...], dtype=mine) > The advantage of this is the ability to initialize the memory. One can get this functionality with core.records.fromrecords, but that is buried pretty far down. I realize now that all the functionality is in the array object. Other than the .field() method, one gets (I think) all the functionality from using the above array declaration and then a repeat() call to get a record array. I suppose then that recarray is really only for compatibility except for .field() Is there a compelling reason not to have a .field() method for arrays so that people in the know can just skip recarray altogether? >>> mydescriptor = [('age',int16),('Nchildren',int16),('weight',float32)] >>> a = array([(0,0,0.0)], dtype=mydescriptor) >>> a = repeat(a, num) From oliphant.travis at ieee.org Tue Jul 11 01:45:49 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 10 Jul 2006 23:45:49 -0600 Subject: [Numpy-discussion] Keyword argument order changed in records.py Message-ID: <44B33B0D.5000908@ieee.org> In an effort to preserve consistency, I made a change to the keyword argument order in the functions in records.py. In addition, all of the functions now take a dtype= keyword argument. This keyword argument is placed first along with other needed keyword arguments. When the dtype keyword is given then it's argument is converted to a data-type object and used in the record array. Otherwise (and only when dtype=None) the formats, names, titles, aligned, and byteorder arguments are used to construct the data-type object. These arguments are always last in the keyword argument order. Bascially, when these are given the rec.format_parser class is run to construct a data-type object from them: Thus using these keywords in place of data-type is equivalent to dtype = format_parser(formats, names, titles, aligned, byteorder)._descr Please let me know if this causes problems. As long as you used keyword arguments, the changes should have no effect on code written for NumPy. Best, -Travis From olivetti at itc.it Tue Jul 11 05:32:48 2006 From: olivetti at itc.it (Emanuele Olivetti) Date: Tue, 11 Jul 2006 11:32:48 +0200 Subject: [Numpy-discussion] how to use argsort result? Message-ID: <44B37040.2040905@itc.it> Hi, I don't understand how to use argsort results. I have a 2D matrix and I want to sort values in each row and obtain the index array of that sorting. Argsort(1) is what I need, but the problem is how to use its result in order to obtain a sorted matrix. Here is the simple example: A = array([[2,3,1],[5,4,6]]) indexes = a.argsort(1) now indexes is: array([[2, 0, 1], [1, 0, 2]]) I'd like to apply indexes to A and obtain: array([[1, 2, 3], [4, 5, 6]]) or better, I'm interested both in a subset of indexes, i.e. indexes[:,1:], and the related values of A matrix. How can I do this? If I simpy say: A[indexes] I get an IndexError. Thanks in advance, Emanuele P.S. numpy.__version__ is '0.9.8'. From pau.gargallo at gmail.com Tue Jul 11 05:40:44 2006 From: pau.gargallo at gmail.com (Pau Gargallo) Date: Tue, 11 Jul 2006 11:40:44 +0200 Subject: [Numpy-discussion] how to use argsort result? In-Reply-To: <44B37040.2040905@itc.it> References: <44B37040.2040905@itc.it> Message-ID: <6ef8f3380607110240i24f6f219w2fa112a113f04291@mail.gmail.com> here goes a first try: A[arange(2)[:,newaxis],indexes] pau On 7/11/06, Emanuele Olivetti wrote: > Hi, > I don't understand how to use argsort results. I have a 2D matrix and > I want to sort values in each row and obtain the index array of that > sorting. Argsort(1) is what I need, but the problem is how to use its > result in order to obtain a sorted matrix. Here is the simple example: > > A = array([[2,3,1],[5,4,6]]) > indexes = a.argsort(1) > > now indexes is: > array([[2, 0, 1], > [1, 0, 2]]) > > I'd like to apply indexes to A and obtain: > array([[1, 2, 3], > [4, 5, 6]]) > > or better, I'm interested both in a subset of indexes, i.e. indexes[:,1:], and > the related values of A matrix. > > How can I do this? If I simpy say: A[indexes] I get an IndexError. > > Thanks in advance, > > Emanuele > > P.S. numpy.__version__ is '0.9.8'. > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From olivetti at itc.it Tue Jul 11 05:57:28 2006 From: olivetti at itc.it (Emanuele Olivetti) Date: Tue, 11 Jul 2006 11:57:28 +0200 Subject: [Numpy-discussion] how to use argsort result? In-Reply-To: <6ef8f3380607110240i24f6f219w2fa112a113f04291@mail.gmail.com> References: <44B37040.2040905@itc.it> <6ef8f3380607110240i24f6f219w2fa112a113f04291@mail.gmail.com> Message-ID: <44B37608.2000003@itc.it> Wow. I have to study much more indexing. It works pretty well. Just to help indexing newbie like on using your advice: A[arange(A.shape[0])[:,newaxis],indexes] Thanks a lot! Emanuele Pau Gargallo wrote: > here goes a first try: > > A[arange(2)[:,newaxis],indexes] From stefan at sun.ac.za Tue Jul 11 06:20:03 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Tue, 11 Jul 2006 12:20:03 +0200 Subject: [Numpy-discussion] how to use argsort result? In-Reply-To: <44B37040.2040905@itc.it> References: <44B37040.2040905@itc.it> Message-ID: <20060711102003.GB22798@mentat.za.net> On Tue, Jul 11, 2006 at 11:32:48AM +0200, Emanuele Olivetti wrote: > Hi, > I don't understand how to use argsort results. I have a 2D matrix and > I want to sort values in each row and obtain the index array of that > sorting. Argsort(1) is what I need, but the problem is how to use its > result in order to obtain a sorted matrix. Here is the simple example: > > A = array([[2,3,1],[5,4,6]]) > indexes = a.argsort(1) > > now indexes is: > array([[2, 0, 1], > [1, 0, 2]]) > > I'd like to apply indexes to A and obtain: > array([[1, 2, 3], > [4, 5, 6]]) > > or better, I'm interested both in a subset of indexes, i.e. indexes[:,1:], and > the related values of A matrix. > > How can I do this? If I simpy say: A[indexes] I get an IndexError. Something's not quite right here. The argsort docstring states that: argsort(a,axis=-1) return the indices into a of the sorted array along the given axis, so that take(a,result,axis) is the sorted array. But N.take(A,A.argsort()) breaks. Either this is a bug, or the docstring needs to be updated. Cheers St?fan From pau.gargallo at gmail.com Tue Jul 11 06:37:23 2006 From: pau.gargallo at gmail.com (Pau Gargallo) Date: Tue, 11 Jul 2006 12:37:23 +0200 Subject: [Numpy-discussion] how to use argsort result? In-Reply-To: <20060711102003.GB22798@mentat.za.net> References: <44B37040.2040905@itc.it> <20060711102003.GB22798@mentat.za.net> Message-ID: <6ef8f3380607110337y7f07790dj42e0812864d1e08d@mail.gmail.com> On 7/11/06, Stefan van der Walt wrote: > On Tue, Jul 11, 2006 at 11:32:48AM +0200, Emanuele Olivetti wrote: > > Hi, > > I don't understand how to use argsort results. I have a 2D matrix and > > I want to sort values in each row and obtain the index array of that > > sorting. Argsort(1) is what I need, but the problem is how to use its > > result in order to obtain a sorted matrix. Here is the simple example: > > > > A = array([[2,3,1],[5,4,6]]) > > indexes = a.argsort(1) > > > > now indexes is: > > array([[2, 0, 1], > > [1, 0, 2]]) > > > > I'd like to apply indexes to A and obtain: > > array([[1, 2, 3], > > [4, 5, 6]]) > > > > or better, I'm interested both in a subset of indexes, i.e. indexes[:,1:], and > > the related values of A matrix. > > > > How can I do this? If I simpy say: A[indexes] I get an IndexError. > > Something's not quite right here. The argsort docstring states that: > > argsort(a,axis=-1) return the indices into a of the sorted array > along the given axis, so that take(a,result,axis) is the sorted array. > > But > > N.take(A,A.argsort()) breaks. Either this is a bug, or the docstring > needs to be updated. > > Cheers > St?fan > I think the docstring is wrong, because take doesn't do that. if you N.take(A,A.argsort(1), 1), it doesn't break but it doesn't sort A neither. Take seems to peek entire columns, but take's docstring is missing. For the argsort docstring, it may be usefull to indicate that if one do >>> ind = indices(A.shape) >>> ind[ax] = A.argsort(axis=ax) then A[ind] is the sorted array. pau From schofield at ftw.at Tue Jul 11 06:37:25 2006 From: schofield at ftw.at (Ed Schofield) Date: Tue, 11 Jul 2006 12:37:25 +0200 Subject: [Numpy-discussion] optimize.leastsq crashs my interpreter In-Reply-To: References: <003301c6a1b8$80ca16e0$1001a8c0@JELLE> <44AE505D.8050905@ftw.at> <44AE775A.30004@ftw.at> Message-ID: <44B37F65.1090508@ftw.at> John Hassler wrote: > Ed Schofield ftw.at> writes: > > >> Hmmm ... it could be an ATLAS problem. What's your processor? I built >> the SciPy 0.4.9 binaries against Pearu's ATLAS binaries for Pentium 2, >> thinking that this would give maximum compatibility ... >> >> Or perhaps it's something else. Could someone with this problem please >> post a backtrace? >> > > > This computer is an AMD Athlon 1600+ running Windows XP. > > > > All of the versions of scipy using numpy crash with XP whenever I access > any of the functions in "optimize" or "integrate" which (I assume) call the > Fortran libraries. > > In the current version, running scipy.test() gives an "unhandled > exception." Debug shows a pointer to: > 020CA9C3 xorps xmm6,xmm6 > > > > Some other information: > >>>> scipy.__version__ '0.4.9' >>>> > > >>>> scipy.__numpy_version__ '0.9.8' >>>> > > >>>> scipy.show_numpy_config() >>>> > > atlas_threads_info: NOT AVAILABLE > > blas_opt_info: libraries = ['f77blas', 'cblas', 'atlas'] library_dirs = > ['C:\\Libraries\\ATLAS_3.6.0_WIN_P4'] define_macros = [('ATLAS_INFO', > '"\\"3.6.0\\""')] language = c include_dirs = > ['C:\\Libraries\\ATLAS_3.6.0_WIN_P4'] > > plus similar stuff. Probably the important thing is the "ATLAS ... P4" > line. > Thanks, John -- this is helpful. (Hans too, thanks for your testing). This looks like a problem with the NumPy build. Travis, could you have compiled the Win32 binaries accidentally against the P4/SSE2 ATLAS library? -- Ed From pau.gargallo at gmail.com Tue Jul 11 06:54:22 2006 From: pau.gargallo at gmail.com (Pau Gargallo) Date: Tue, 11 Jul 2006 12:54:22 +0200 Subject: [Numpy-discussion] int 'up'casted to uint Message-ID: <6ef8f3380607110354u6db272b0t125a9f13a92d1180@mail.gmail.com> hi, looking at the upcasting table at http://www.scipy.org/Tentative_NumPy_Tutorial#head-4c1d53fe504adc97baf27b65513b4b97586a4fc5 I saw that int's are sometimes casted to uint's. In [3]: a = array([3],int16) In [5]: b = array([4],uint32) In [7]: a+b Out[7]: array([7], dtype=uint32) is that intended? pau From stefan at sun.ac.za Tue Jul 11 07:10:53 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Tue, 11 Jul 2006 13:10:53 +0200 Subject: [Numpy-discussion] how to use argsort result? In-Reply-To: <6ef8f3380607110337y7f07790dj42e0812864d1e08d@mail.gmail.com> References: <44B37040.2040905@itc.it> <20060711102003.GB22798@mentat.za.net> <6ef8f3380607110337y7f07790dj42e0812864d1e08d@mail.gmail.com> Message-ID: <20060711111053.GC24667@mentat.za.net> On Tue, Jul 11, 2006 at 12:37:23PM +0200, Pau Gargallo wrote: > > Something's not quite right here. The argsort docstring states that: > > > > argsort(a,axis=-1) return the indices into a of the sorted array > > along the given axis, so that take(a,result,axis) is the sorted array. > > > > But > > > > N.take(A,A.argsort()) breaks. Either this is a bug, or the docstring > > needs to be updated. > > > > Cheers > > St?fan > > > > I think the docstring is wrong, because take doesn't do that. > if you N.take(A,A.argsort(1), 1), it doesn't break but it doesn't sort > A neither. > > Take seems to peek entire columns, but take's docstring is missing. > > For the argsort docstring, it may be usefull to indicate that if one do > >>> ind = indices(A.shape) > >>> ind[ax] = A.argsort(axis=ax) > then A[ind] is the sorted array. We can always adapt the documentation at http://numeric.scipy.org/numpydoc/numpy-9.html#pgfId-36425 into a docstring. I'll file a ticket. Cheers St?fan From Norbert.Nemec.list at gmx.de Tue Jul 11 11:00:50 2006 From: Norbert.Nemec.list at gmx.de (Norbert Nemec) Date: Tue, 11 Jul 2006 17:00:50 +0200 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <91cf711d0607031000l286da52fn30d98798057f12c5@mail.gmail.com> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> <44A8E4D3.5000603@ntc.zcu.cz> <91cf711d0607031000l286da52fn30d98798057f12c5@mail.gmail.com> Message-ID: <44B3BD22.70003@gmx.de> unique1d is based on ediff1d, so it really calculates many differences and compares those to 0.0 This is inefficient, even though this is hidden by the general inefficiency of Python (It might be the reason for the two milliseconds, though) What is more: subtraction works only for numbers, while the various proposed versions use only comparison which works for any data type (as long as it can be sorted) My own version tried to capture all possible cases that the current unique captures. Sasha's version only works for numpy arrays and has a problem for arrays with all identical entries. David's version only works for numpy arrays of types that can be converted to float. I would once more propose to use my own version as given before: def unique(arr,sort=True): if hasattr(arr,'flatten'): tmp = arr.flatten() tmp.sort() idx = concatenate([True],tmp[1:]!=tmp[:-1]) return tmp[idx] else: # for compatibility: set = {} for item in inseq: set[item] = None if sort: return asarray(sorted(set.keys())) else: return asarray(set.keys()) Greetings, Norbert From aisaac at american.edu Tue Jul 11 11:47:27 2006 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 11 Jul 2006 11:47:27 -0400 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <44B3BD22.70003@gmx.de> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> <44A8E4D3.5000603@ntc.zcu.cz><91cf711d0607031000l286da52fn30d98798057f12c5@mail.gmail.com> <44B3BD22.70003@gmx.de> Message-ID: On Tue, 11 Jul 2006, Norbert Nemec apparently wrote: > else: # for compatibility: > set = {} > for item in inseq: > set[item] = None > if sort: > return asarray(sorted(set.keys())) > else: > return asarray(set.keys()) I'm currently in major caffeine deficit, but aside from backward compatability, how is this better than: else: #for compatability items = list(set(inseq)) if sort: items.sort() return asarray(items) Alan Isaac PS Actually, making a list of a set may already sort? No time to check now ... PPS For to 2.3, need set = sets.Set From tim.hochberg at cox.net Tue Jul 11 12:02:21 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue, 11 Jul 2006 09:02:21 -0700 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <44B3BD22.70003@gmx.de> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> <44A8E4D3.5000603@ntc.zcu.cz> <91cf711d0607031000l286da52fn30d98798057f12c5@mail.gmail.com> <44B3BD22.70003@gmx.de> Message-ID: <44B3CB8D.9010308@cox.net> Norbert Nemec wrote: > unique1d is based on ediff1d, so it really calculates many differences > and compares those to 0.0 > > This is inefficient, even though this is hidden by the general > inefficiency of Python (It might be the reason for the two milliseconds, > though) > > What is more: subtraction works only for numbers, while the various > proposed versions use only comparison which works for any data type (as > long as it can be sorted) > My first question is: why? What's the attraction in returning a sorted answer here? Returning an unsorted array is potentially faster, depending on the algorithm chosen, and sorting after the fact is trivial. If one was going to spend extra complexity on something, I'd think it would be better spent on preserving the input order. Second, some objects can be compared for equality and hashed, but not sorted (Python's complex number's come to mind). If one is going to worry about subtraction so as to keep things general, it makes sense to also avoid sorting as well Sasha's slick algorithm not withstanding. Third, I propose that whatever the outcome of the sorting issue, I would propose that unique have the same interface as the other structural array operations. That is: unique(anarray, axis=0): ... The default axis=0 is for compatibility with the other, somewhat similar functions. Axis=None would return the flattened, uniquified data, axis=# would uniquify the result along that axis. Regards, -tim > My own version tried to capture all possible cases that the current > unique captures. > > Sasha's version only works for numpy arrays and has a problem for arrays > with all identical entries. > > David's version only works for numpy arrays of types that can be > converted to float. > > I would once more propose to use my own version as given before: > > def unique(arr,sort=True): > if hasattr(arr,'flatten'): > tmp = arr.flatten() > tmp.sort() > idx = concatenate([True],tmp[1:]!=tmp[:-1]) > return tmp[idx] > else: # for compatibility: > set = {} > for item in inseq: > set[item] = None > if sort: > return asarray(sorted(set.keys())) > else: > return asarray(set.keys()) > > > Greetings, > Norbert > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From oliphant.travis at ieee.org Tue Jul 11 12:03:38 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 11 Jul 2006 10:03:38 -0600 Subject: [Numpy-discussion] int 'up'casted to uint In-Reply-To: <6ef8f3380607110354u6db272b0t125a9f13a92d1180@mail.gmail.com> References: <6ef8f3380607110354u6db272b0t125a9f13a92d1180@mail.gmail.com> Message-ID: <44B3CBDA.8080605@ieee.org> Pau Gargallo wrote: > hi, > > looking at the upcasting table at > http://www.scipy.org/Tentative_NumPy_Tutorial#head-4c1d53fe504adc97baf27b65513b4b97586a4fc5 > I saw that int's are sometimes casted to uint's. > > In [3]: a = array([3],int16) > In [5]: b = array([4],uint32) > In [7]: a+b > Out[7]: array([7], dtype=uint32) > > is that intended? > It's a bug. The result should be int64. I've fixed it in SVN. -Travis From tim.hochberg at cox.net Tue Jul 11 12:06:23 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue, 11 Jul 2006 09:06:23 -0700 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <44B3CB8D.9010308@cox.net> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> <44A8E4D3.5000603@ntc.zcu.cz> <91cf711d0607031000l286da52fn30d98798057f12c5@mail.gmail.com> <44B3BD22.70003@gmx.de> <44B3CB8D.9010308@cox.net> Message-ID: <44B3CC7F.2070000@cox.net> Tim Hochberg wrote: > Norbert Nemec wrote: > >> unique1d is based on ediff1d, so it really calculates many differences >> and compares those to 0.0 >> >> This is inefficient, even though this is hidden by the general >> inefficiency of Python (It might be the reason for the two milliseconds, >> though) >> >> What is more: subtraction works only for numbers, while the various >> proposed versions use only comparison which works for any data type (as >> long as it can be sorted) >> >> > My first question is: why? What's the attraction in returning a sorted > answer here? Returning an unsorted array is potentially faster, > depending on the algorithm chosen, and sorting after the fact is > trivial. If one was going to spend extra complexity on something, I'd > think it would be better spent on preserving the input order. > > Second, some objects can be compared for equality and hashed, but not > sorted (Python's complex number's come to mind). If one is going to > worry about subtraction so as to keep things general, it makes sense to > also avoid sorting as well Sasha's slick algorithm not withstanding. > > Third, I propose that whatever the outcome of the sorting issue, I would > propose that unique have the same interface as the other structural > array operations. That is: > > unique(anarray, axis=0): > ... > > The default axis=0 is for compatibility with the other, somewhat similar > functions. Axis=None would return the flattened, uniquified data, > axis=# would uniquify the result along that axis. > Hmmm. Of course that precludes it returning an actual array for axis!=None. That might be considered suboptimal... -tim > Regards, > > -tim > > >> My own version tried to capture all possible cases that the current >> unique captures. >> >> Sasha's version only works for numpy arrays and has a problem for arrays >> with all identical entries. >> >> David's version only works for numpy arrays of types that can be >> converted to float. >> >> I would once more propose to use my own version as given before: >> >> def unique(arr,sort=True): >> if hasattr(arr,'flatten'): >> tmp = arr.flatten() >> tmp.sort() >> idx = concatenate([True],tmp[1:]!=tmp[:-1]) >> return tmp[idx] >> else: # for compatibility: >> set = {} >> for item in inseq: >> set[item] = None >> if sort: >> return asarray(sorted(set.keys())) >> else: >> return asarray(set.keys()) >> >> >> Greetings, >> Norbert >> >> >> >> ------------------------------------------------------------------------- >> Using Tomcat but need to do more? Need to support web services, security? >> Get stuff done quickly with pre-integrated technology to make your job easier >> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> >> >> >> > > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From robert.kern at gmail.com Tue Jul 11 12:23:42 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 11 Jul 2006 11:23:42 -0500 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <44B3CB8D.9010308@cox.net> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> <44A8E4D3.5000603@ntc.zcu.cz> <91cf711d0607031000l286da52fn30d98798057f12c5@mail.gmail.com> <44B3BD22.70003@gmx.de> <44B3CB8D.9010308@cox.net> Message-ID: Tim Hochberg wrote: > Norbert Nemec wrote: >> unique1d is based on ediff1d, so it really calculates many differences >> and compares those to 0.0 >> >> This is inefficient, even though this is hidden by the general >> inefficiency of Python (It might be the reason for the two milliseconds, >> though) >> >> What is more: subtraction works only for numbers, while the various >> proposed versions use only comparison which works for any data type (as >> long as it can be sorted) >> > My first question is: why? What's the attraction in returning a sorted > answer here? Returning an unsorted array is potentially faster, > depending on the algorithm chosen, and sorting after the fact is > trivial. If one was going to spend extra complexity on something, I'd > think it would be better spent on preserving the input order. One issue is that uniquifying numpy arrays using Python dicts, while expected O(n) in terms of complexity, is really slow compared to sorting because of the overhead in getting the elements out of the numpy arrays and into Python objects. For the cases where sorting works (your caveat below is quite correct), it's really quite good for numpy arrays. OTOH, if one were to implement a hash table in C, that might potentially be faster and more robust, but that is spending a *large* amount of extra complexity. > Second, some objects can be compared for equality and hashed, but not > sorted (Python's complex number's come to mind). If one is going to > worry about subtraction so as to keep things general, it makes sense to > also avoid sorting as well Sasha's slick algorithm not withstanding. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pau.gargallo at gmail.com Tue Jul 11 12:24:34 2006 From: pau.gargallo at gmail.com (Pau Gargallo) Date: Tue, 11 Jul 2006 18:24:34 +0200 Subject: [Numpy-discussion] int 'up'casted to uint In-Reply-To: <44B3CBDA.8080605@ieee.org> References: <6ef8f3380607110354u6db272b0t125a9f13a92d1180@mail.gmail.com> <44B3CBDA.8080605@ieee.org> Message-ID: <6ef8f3380607110924p66b8de0ds55971dfb8c9ff892@mail.gmail.com> On 7/11/06, Travis Oliphant wrote: > Pau Gargallo wrote: > > hi, > > > > looking at the upcasting table at > > http://www.scipy.org/Tentative_NumPy_Tutorial#head-4c1d53fe504adc97baf27b65513b4b97586a4fc5 > > I saw that int's are sometimes casted to uint's. > > > > In [3]: a = array([3],int16) > > In [5]: b = array([4],uint32) > > In [7]: a+b > > Out[7]: array([7], dtype=uint32) > > > > is that intended? > > > It's a bug. The result should be int64. I've fixed it in SVN. > Thanks!! From cimrman3 at ntc.zcu.cz Tue Jul 11 12:49:41 2006 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Tue, 11 Jul 2006 18:49:41 +0200 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <44B3BD22.70003@gmx.de> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> <44A8E4D3.5000603@ntc.zcu.cz> <91cf711d0607031000l286da52fn30d98798057f12c5@mail.gmail.com> <44B3BD22.70003@gmx.de> Message-ID: <44B3D6A5.6060404@ntc.zcu.cz> Norbert Nemec wrote: > unique1d is based on ediff1d, so it really calculates many differences > and compares those to 0.0 > > This is inefficient, even though this is hidden by the general > inefficiency of Python (It might be the reason for the two milliseconds, > though) > > What is more: subtraction works only for numbers, while the various > proposed versions use only comparison which works for any data type (as > long as it can be sorted) I agree that unique1d works only for numbers, but that is what it was meant for... well for integers only, in fact - everyone here surely knows, that comparing floats with != does not work well. Note also that it was written before logical indexing and other neat stuff were not possible in numpy - every improvement is welcome! (btw. I cannot recall why I used subtraction and testing for zero instead of just comparisons - maybe remnants from my old matlab days and its logical arrays - ediff1d should disappear from the other functions in arraysetops) > My own version tried to capture all possible cases that the current > unique captures. > > Sasha's version only works for numpy arrays and has a problem for arrays > with all identical entries. > > David's version only works for numpy arrays of types that can be > converted to float. comparing floats... > I would once more propose to use my own version as given before: > > def unique(arr,sort=True): > if hasattr(arr,'flatten'): > tmp = arr.flatten() > tmp.sort() > idx = concatenate([True],tmp[1:]!=tmp[:-1]) > return tmp[idx] > else: # for compatibility: > set = {} > for item in inseq: > set[item] = None > if sort: > return asarray(sorted(set.keys())) > else: > return asarray(set.keys()) Have you considered using set instead of dict? Just curious :-) r. From Chris.Barker at noaa.gov Tue Jul 11 13:47:46 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 11 Jul 2006 10:47:46 -0700 Subject: [Numpy-discussion] array interface, PIL and PyGame Message-ID: <44B3E442.7080606@noaa.gov> Hi all, Over on the PIL mailing list, someone asked about some possible additions to PIL to facilitate copy-free moving of data between PyGame and PIL. I sent a note suggesting that the array interface might be just the ticket. These were the responses: Pete Shinners wrote: > Yeah, this would be an ideal solution. I hope more of the base array stuff can > get into Python 2.6. > > We did look at the array proposal, but haven't been able to jump on it yet > because of adoption. I guess it needs another package or two to get the ball > rolling. So that's an advocacy issue, and an illustration of why getting it into the standard lib is critical. Fredrik Lundh wrote: > unfortunately, the "array interface" model isn't even close to be able > to describe a PIL image memory (the "Imaging" structure)... Darn. It was my understanding that we thought that it was close to being able to describe an image... so what have we missed. I'm out of my technical depth, but I've encouraged Fredrik to bring the discussion here. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant at ee.byu.edu Tue Jul 11 14:04:37 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 11 Jul 2006 12:04:37 -0600 Subject: [Numpy-discussion] array interface, PIL and PyGame In-Reply-To: <44B3E442.7080606@noaa.gov> References: <44B3E442.7080606@noaa.gov> Message-ID: <44B3E835.1040207@ee.byu.edu> Christopher Barker wrote: >Hi all, > >Over on the PIL mailing list, someone asked about some possible >additions to PIL to facilitate copy-free moving of data between PyGame >and PIL. I sent a note suggesting that the array interface might be just >the ticket. These were the responses: > >Pete Shinners wrote: > > >>Yeah, this would be an ideal solution. I hope more of the base array stuff can >>get into Python 2.6. >> >>We did look at the array proposal, but haven't been able to jump on it yet >>because of adoption. I guess it needs another package or two to get the ball >>rolling. >> >> > >So that's an advocacy issue, and an illustration of why getting it into >the standard lib is critical. > >Fredrik Lundh wrote: > > >>unfortunately, the "array interface" model isn't even close to be able >>to describe a PIL image memory (the "Imaging" structure)... >> >> > > > We will all welcome Fredrik's comments. But, I think he is exaggerating the situation a bit. It is true that the Imaging structure of PIL is a more general memory model for 2-d arrays. My understanding is that it basically allows for an array of pointers to memory where each pointer points to a different line (or chunk) in the image (kind of like a C-array). So, basically, the problem is that you may have an image that cannot be described as a single block of memory or a singly-strided array. I believe, thought, that sometimes you do have a PIL image that is basically a single chunk of memory. So, while a general-purpose "memory-sharing" situation might be difficult. I think some sharing (of each chunk for example) could be done. Even still, the array interface (the Python side) does technically handle the PIL case because the default is to use the sequence interface to access elements of the array. It would be nice if Fredrik were more willing to help establish a standard, though. Calling it "not close" but giving no alternative is not helpful. >Darn. It was my understanding that we thought that it was close to being >able to describe an image... so what have we missed. I'm out of my >technical depth, but I've encouraged Fredrik to bring the discussion here. > > -Travis From david.huard at gmail.com Tue Jul 11 14:27:37 2006 From: david.huard at gmail.com (David Huard) Date: Tue, 11 Jul 2006 14:27:37 -0400 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <44B3D6A5.6060404@ntc.zcu.cz> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> <44A8E4D3.5000603@ntc.zcu.cz> <91cf711d0607031000l286da52fn30d98798057f12c5@mail.gmail.com> <44B3BD22.70003@gmx.de> <44B3D6A5.6060404@ntc.zcu.cz> Message-ID: <91cf711d0607111127w21bb4b3cj30a3c31b46e136f0@mail.gmail.com> Tim Hochberg wrote: > My first question is: why? What's the attraction in returning a sorted > answer here? Returning an unsorted array is potentially faster, > depending on the algorithm chosen, and sorting after the fact is > trivial. If one was going to spend extra complexity on something, I'd > think it would be better spent on preserving the input order. There is a unique function in matlab that returns a sorted vector. I think a lot of people will expect a numpy and matlab functions with identical names to behave similarly. If we want to preserve the input order, we'd have to choose a convention about whose value's order is retained: do we keep the order of the first value found or the last one ? Here is the benchmark. Sorry Norbert for not including your code the first time, it turns out that with Alain's suggestion, its the fastest one both for lists and arrays. x = rand(100000)*100 x = x.astype('i') l = list(x) For array x: In [166]: timeit unique_alan(x) # with set instead of dict 100 loops, best of 3: 8.8 ms per loop In [167]: timeit unique_norbert(x) 100 loops, best of 3: 8.8 ms per loop In [168]: timeit unique_sasha(x) 100 loops, best of 3: 10.8 ms per loop In [169]: timeit unique(x) 10 loops, best of 3: 50.4 ms per loop In [170]: timeit unique1d(x) 10 loops, best of 3: 13.2 ms per loop For list l: In [196]: timeit unique_norbert(l) 10 loops, best of 3: 29 ms per loop In [197]: timeit unique_alan(l) # with set instead of dict 10 loops, best of 3: 14.5 ms per loop In [193]: timeit unique(l) 10 loops, best of 3: 29.6 ms per loop Note: In Norbert function, setting sort=False for flattenable objects returns a sorted array anyway. So I'd suggest to remove the sort keyword, sort if the datatype is sortable, and don't sort if its not. David -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: unique_test.py Type: text/x-python-script Size: 994 bytes Desc: not available URL: From perry at stsci.edu Tue Jul 11 14:28:31 2006 From: perry at stsci.edu (Perry Greenfield) Date: Tue, 11 Jul 2006 14:28:31 -0400 Subject: [Numpy-discussion] array interface, PIL and PyGame In-Reply-To: <44B3E835.1040207@ee.byu.edu> References: <44B3E442.7080606@noaa.gov> <44B3E835.1040207@ee.byu.edu> Message-ID: On Jul 11, 2006, at 2:04 PM, Travis Oliphant wrote: >> > We will all welcome Fredrik's comments. But, I think he is > exaggerating > the situation a bit. > > It is true that the Imaging structure of PIL is a more general memory > model for 2-d arrays. My understanding is that it basically allows > for > an array of pointers to memory where each pointer points to a > different > line (or chunk) in the image (kind of like a C-array). > > So, basically, the problem is that you may have an image that > cannot be > described as a single block of memory or a singly-strided array. I > believe, thought, that sometimes you do have a PIL image that is > basically a single chunk of memory. > > So, while a general-purpose "memory-sharing" situation might be > difficult. I think some sharing (of each chunk for example) could be > done. > > Even still, the array interface (the Python side) does technically > handle the PIL case because the default is to use the sequence > interface > to access elements of the array. > > It would be nice if Fredrik were more willing to help establish a > standard, though. Calling it "not close" but giving no alternative is > not helpful. To expand on this a bit, I think Fredrik is misreading the intent of the array interface somewhat. It's not that we are promising that the array data structure will suit his every need. Far from it. It's that it allows him (or us) to provide convenience functions that allow accessing data in PIL without jumping through lots of annoying hoops. It should be possible to convert PIL image data to arrays easily. Does that mean that all associated information is propagated as part of the array object? Of course not. But this information is obtainable by other means anyway. Even in the case where different chunks of an image are in different memory locations and there is no simple way of avoiding copying the data, so what? I still much prefer the data be copied to an array so that numpy functionality can be applied to the array and have to avoid calling a sequence of operations to convert the data to a string and then to an array. At least one copy is avoided, but avoiding copying isn't the entire justification. Think of it this way, if PIL strength is that it can convert images between many different formats (no doubt, copying data in the process) then arrays should be one of the supported formats, no? Perry From oliphant.travis at ieee.org Tue Jul 11 15:37:15 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 11 Jul 2006 13:37:15 -0600 Subject: [Numpy-discussion] Quicker image transfer, tobuffer? In-Reply-To: References: <44B2EC45.8020402@noaa.gov> Message-ID: <44B3FDEB.1000909@ieee.org> Here is a simple approach to allowing the PIL to export the array interface. This allows NumPy to create a suitable array from a PIL image very easily: At the top of Image.py add the following if sys.byteorder == 'little': _ENDIAN = '<' else: _ENDIAN = '>' _MODE_CONV = { # official modes "1": ('|b1', None), "L": ('|u1', None), "I": ('%si4' % _ENDIAN, None), "F": ('%sf4' % _ENDIAN, None), "P": ('|u1', None), "RGB": ('|u1', 3), "RGBX": ('|u1', 4), "RGBA": ('|u1', 4), "CMYK": ('|u1', 4), "YCbCr": ('|u1', 4), # Experimental modes include I;16, I;16B, RGBa, BGR;15, # and BGR;24. Use these modes only if you know exactly # what you're doing... } def _conv_type_shape(im): shape = im.size typ, extra = _MODE_CONV[im.mode] if extra is None: return shape, typ shape += (extra,) return shape, typ In the Image class structure add def __get_array_interface__(self): new = {} shape, typestr = _conv_type_shape(self) new['shape'] = shape new['typestr'] = typestr new['data'] = self.tostring() return new __array_interface__ = property(__get_array_interface__, None, doc="array interface") With this addition you can then do import Image, numpy im = Image.open('lena.jpg') a = numpy.asarray(im) and you will get a suitable read-only array pointing to the string produced by tostring. This would be a nice thing to add to the PIL. -Travis Oliphant From jonathan.taylor at utoronto.ca Tue Jul 11 16:07:40 2006 From: jonathan.taylor at utoronto.ca (Jonathan Taylor) Date: Tue, 11 Jul 2006 16:07:40 -0400 Subject: [Numpy-discussion] What's wrong with matrices? In-Reply-To: <44AE6675.108@ftw.at> References: <44AE6675.108@ftw.at> Message-ID: <463e11f90607111307s8bb9344w42d30b844cdaf0b4@mail.gmail.com> I agree the real problem with matrices is they seem awkward to work with compared to arrays because numpy seems so array centric. The only advantage I see is getting .T to do transposes and * to do matrix multiplication. I hope numpy reaches a point where it is as natural to use matrices as arrays. I'd also vote for the inclusion of the following two functions col and row. Inspired by R equivelents they let you do some indexing very easily such as getting the values of the upper triangle of the matrix. E.g. vals = m[row(m) > col(m)] Cheers, Jon. def col(m): """col(m) returns a matrix of the same size of m where each element contains an integer denoting which column it is in. For example, >>> m = eye(3) >>> m array([[1, 0, 0], [0, 1, 0], [0, 0, 1]]) >>> col(m) array([[0, 1, 2], [0, 1, 2], [0, 1, 2]]) """ assert len(m.shape) == 2, "should be a matrix" return N.indices(m.shape)[1] def row(m): """row(m) returns a matrix of the same size of m where each element contains an integer denoting which row it is in. For example, >>> m = eye(3) >>> m array([[1, 0, 0], [0, 1, 0], [0, 0, 1]]) >>> row(m) array([[0, 0, 0], [1, 1, 1], [2, 2, 2]]) """ assert len(m.shape) == 2, "should be a matrix" return N.indices(m.shape)[0] On 7/7/06, Ed Schofield wrote: > Bill Baxter wrote: > > I think the thread to this point can be pretty much summarized by: > > > > while True: > > Bill: "2D transpose is common so it should have a nice syntax" > > Tim, Robert, Sasha, and Ed: "No it's not." > > > > Very well. I think it may be a self fulfilling prophecy, though. > > I.e. if matrix operations are cumbersome to use, then -- surprise > > surprise -- the large user base for matrix-like operations never > > materializes. Potential converts just give numpy the pass, and go to > > Octave or Scilab, or stick with Matlab, R or S instead. > > > > Why all the fuss about the .T? Because any changes to functions (like > > making ones() return a matrix) can easily be worked around on the user > > side, as has been pointed out. But as far as I know -- do correct me > > if I'm wrong -- there's no good way for a user to add an attribute to > > an existing class. After switching from matrices back to arrays, .T > > was the only thing I really missed from numpy.matrix. > > > > I would be all for a matrix class that was on equal footing with array > > and as easy to use as matrices in Matlab. But my experience using > > numpy.matrix was far from that, and, given the lack of enthusiasm for > > matrices around here, that seems unlikely to change. However, I'm > > anxious to see what Ed has up his sleeves in the other thread. > > Okay ... ... let's make this the thread ;) > I'd like to know why you, Sven, and anyone else on the list have gone > back to using arrays after trying matrices. What was inconvenient about > them? I'd like a nice juicy list. The whole purpose of the matrix > class is to simplify 2d linear algebra. Where is it failing? > > I also went back to arrays after trying out matrices for some 2d linear > algebra tasks, since I found that using matrices increased my code's > complexity. I can describe the problems I had with them later, but > first I'd like to hear of others' experiences. > > I'd like to help to make matrices more usable. Tell me what you want, > and I'll work on some patches. > > -- Ed > > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From kevin at cazabon.com Tue Jul 11 16:10:29 2006 From: kevin at cazabon.com (kevin at cazabon.com) Date: Tue, 11 Jul 2006 22:10:29 +0200 Subject: [Numpy-discussion] [Image-SIG] Quicker image transfer, tobuffer? References: <44B2EC45.8020402@noaa.gov> <44B3FDEB.1000909@ieee.org> Message-ID: <013401c6a526$0eb5e5e0$650aa8c0@duallie> Although I'm not really up to speed on the array interface, accessing the pixel data in a PIL image isn't really that difficult in C/C++... the only challenge I would see (besides tracking the channels/padding correctly... trivial) would be getting the pointer into Python to pass it to NumPy. I've written a few modules in C that directly modify the PIL buffer data, with simple code such as attached (lines 186-214 show it clearly). (This is a module that does unsharp-masking and gaussian blur on PIL images... Fredrik is welcome to include this directly into the PIL library if he sees fit, for which I'll gladly remove ANY licensing restrictions) Kevin. ----- Original Message ----- From: "Travis Oliphant" To: Cc: "numpy-discussion" Sent: Tuesday, July 11, 2006 9:37 PM Subject: Re: [Image-SIG] Quicker image transfer, tobuffer? > > Here is a simple approach to allowing the PIL to export the array > interface. > > This allows NumPy to create a suitable array from a PIL image very easily: > > > At the top of Image.py add the following > > if sys.byteorder == 'little': > _ENDIAN = '<' > else: > _ENDIAN = '>' > > _MODE_CONV = { > > # official modes > "1": ('|b1', None), > "L": ('|u1', None), > "I": ('%si4' % _ENDIAN, None), > "F": ('%sf4' % _ENDIAN, None), > "P": ('|u1', None), > "RGB": ('|u1', 3), > "RGBX": ('|u1', 4), > "RGBA": ('|u1', 4), > "CMYK": ('|u1', 4), > "YCbCr": ('|u1', 4), > > # Experimental modes include I;16, I;16B, RGBa, BGR;15, > # and BGR;24. Use these modes only if you know exactly > # what you're doing... > > } > > def _conv_type_shape(im): > shape = im.size > typ, extra = _MODE_CONV[im.mode] > if extra is None: > return shape, typ > shape += (extra,) > return shape, typ > > > > In the Image class structure add > > def __get_array_interface__(self): > new = {} > shape, typestr = _conv_type_shape(self) > new['shape'] = shape > new['typestr'] = typestr > new['data'] = self.tostring() > return new > > __array_interface__ = property(__get_array_interface__, None, > doc="array interface") > > > > With this addition you can then do > > import Image, numpy > > im = Image.open('lena.jpg') > a = numpy.asarray(im) > > and you will get a suitable read-only array pointing to the string > produced by tostring. > > > This would be a nice thing to add to the PIL. > > -Travis Oliphant > > > _______________________________________________ > Image-SIG maillist - Image-SIG at python.org > http://mail.python.org/mailman/listinfo/image-sig > -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: PILusm.cpp URL: From myeates at jpl.nasa.gov Tue Jul 11 16:49:15 2006 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Tue, 11 Jul 2006 13:49:15 -0700 Subject: [Numpy-discussion] looking for the Numpy to do it Message-ID: <44B40ECB.8060408@jpl.nasa.gov> I can handle the following problem by iterating through some indices but I'm looking for a more elegant solution. If I have a 1d array, I want to find a contiguous nonzero region about a given index. For example, if a=[1,2,3,0,40,50,60,0,7,8,9] and we start with the index of 5, then I want the indices 4,5,6 Any gurus out there? Mathew From ndarray at mac.com Tue Jul 11 17:03:49 2006 From: ndarray at mac.com (Sasha) Date: Tue, 11 Jul 2006 17:03:49 -0400 Subject: [Numpy-discussion] looking for the Numpy to do it In-Reply-To: <44B40ECB.8060408@jpl.nasa.gov> References: <44B40ECB.8060408@jpl.nasa.gov> Message-ID: Here is the solution of a half of the problem: >>> a=array([1,2,3,0,40,50,60,0,7,8,9]) >>> 5+where(logical_and.accumulate(a[5:]!=0)) array([5, 6]) the rest is left as an exercise to the reader :-) Hint a[::-1] will reverse a. On 7/11/06, Mathew Yeates wrote: > I can handle the following problem by iterating through some indices but > I'm looking for a more elegant solution. > > If I have a 1d array, I want to find a contiguous nonzero region about a > given index. For example, if a=[1,2,3,0,40,50,60,0,7,8,9] and we start > with the index of 5, then I want the indices 4,5,6 > > Any gurus out there? > > Mathew > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From oliphant at ee.byu.edu Tue Jul 11 17:15:06 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 11 Jul 2006 15:15:06 -0600 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> Message-ID: <44B414DA.1050300@ee.byu.edu> Ed Schofield wrote: >Last week's discussion on rand() and randn() seemed to indicate a >sentiment that they ought to take tuples for consistency with ones, >zeros, eye, identity, and empty -- that, although they are supposed >to be convenience functions, they are inconvenient precisely because >of their inconsistency with these other functions. This issue has >been raised many times over the past several months. > >Travis made a change in r2572 to allow tuples as arguments, then took >it out again a few hours later, apparently unsure about whether this >was a good idea. > >I'd like to call for a vote on what people would prefer, and then ask >Travis to make a final pronouncement before the feature freeze. > > > > This is my thinking about the rand and randn situation: I really like the rand and randn functions. I use them all the time and want quick and easy access to them. In retrospect, the rand and randn functions probably should have been placed in a separate "user" namespace like pylab or numlab or something like that to distinguish it from "numpy the library." But, we don't have something like that in place at this point, so what to do now? I'm opposed to any suggestion to get rid of the rand(3,3) calling syntax. One reason is that I like the syntax for this function and I've used it a lot. It comes from MLab in Numeric and so it is needed to support compatibility with Numeric as well. So, we can't just get rid of it entirely. Another big reason is that there is already a function that uses ONLY the tuple syntax to do exactly the same thing. Look at the docstrings for rand and randn to get the names. If you are opposed to the rand and randn syntax then just ignore those functions and use numpy.random.random_sample(tuple) numpy.random.standard_normal(tuple) So, I'm opposed to getting rid of the *args based syntax. My feelings are weaker regarding adding the capability to rand and randn to accept a tuple. I did test it out and it does seem feasible to add this feature at the cost of an additional comparison. I know Robert is opposed to it but I'm not sure I understand completely why. Please correct me if I'm wrong, but I think it has something to do with making the random_sample and standard_normal functions irrelevant and unnecessary combined with my hypothesis that Robert doesn't like the *args-style syntax. Therefore, adding support to rand and randn for tuples, would make them the default random-number generators and there would be proliferating code that was "harder to read" because of the different usages. Personally, I'm not that opposed to "different" calling conventions, but I respect the opinions of the wider NumPy community. In sum: rand and randn have to live somewhere and accept their current calling convention. I would not be opposed at this point to taking them out of top-level numpy and putting them instead into a "numlab" name-space (for lack of a better name). However, I'm not so sure that making a "numlab" name-space really solves any of the issues that are being debated in this case. So, I'm not going to spend any time doing it... -Travis From myeates at jpl.nasa.gov Tue Jul 11 17:17:18 2006 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Tue, 11 Jul 2006 14:17:18 -0700 Subject: [Numpy-discussion] looking for the Numpy to do it In-Reply-To: References: <44B40ECB.8060408@jpl.nasa.gov> Message-ID: <44B4155E.7080400@jpl.nasa.gov> I bestow upon Sasha the mantle of guru. Sasha wrote: > Here is the solution of a half of the problem: > >>>> a=array([1,2,3,0,40,50,60,0,7,8,9]) >>>> 5+where(logical_and.accumulate(a[5:]!=0)) > array([5, 6]) > > the rest is left as an exercise to the reader :-) > > Hint a[::-1] will reverse a. > > > On 7/11/06, Mathew Yeates wrote: >> I can handle the following problem by iterating through some indices but >> I'm looking for a more elegant solution. >> >> If I have a 1d array, I want to find a contiguous nonzero region about a >> given index. For example, if a=[1,2,3,0,40,50,60,0,7,8,9] and we start >> with the index of 5, then I want the indices 4,5,6 >> >> Any gurus out there? >> >> Mathew >> >> >> >> ------------------------------------------------------------------------- >> >> Using Tomcat but need to do more? Need to support web services, >> security? >> Get stuff done quickly with pre-integrated technology to make your >> job easier >> Download IBM WebSphere Application Server v.1.0.1 based on Apache >> Geronimo >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> > From filip at ftv.pl Tue Jul 11 17:31:56 2006 From: filip at ftv.pl (Filip Wasilewski) Date: Tue, 11 Jul 2006 23:31:56 +0200 Subject: [Numpy-discussion] Quicker image transfer, tobuffer? In-Reply-To: <44B3FDEB.1000909@ieee.org> References: <44B2EC45.8020402@noaa.gov> <44B3FDEB.1000909@ieee.org> Message-ID: <1785255519.20060711233156@gmail.com> Hi Travis, this is a great example of the __array_interface__ usage. I have spotted some problems after patching the Image.py module and trying to display an array created from Image in matplotlib. First issue is a minor one. There is a difference in axis order between ndarray and PIL: def _conv_type_shape(im): shape = im.size[::-1] typ, extra = _MODE_CONV[im.mode] if extra is None: return shape, typ shape += (extra,) return shape, typ The second seems to be more complex and may be a more general. The memory of string created by self.tostring() seems to be deallocated before array is created (v 0.9.9.2788, win). Everything works fine after storing the reference to data, but this probably should be done somewhere else: def __get_array_interface__(self): new = {} shape, typestr = _conv_type_shape(self) new['shape'] = shape new['typestr'] = typestr new['data'] = self.tostring() self._str_data = new['data'] # a dirty hack return new best, fw From oliphant at ee.byu.edu Tue Jul 11 18:01:53 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 11 Jul 2006 16:01:53 -0600 Subject: [Numpy-discussion] Quicker image transfer, tobuffer? In-Reply-To: <1785255519.20060711233156@gmail.com> References: <44B2EC45.8020402@noaa.gov> <44B3FDEB.1000909@ieee.org> <1785255519.20060711233156@gmail.com> Message-ID: <44B41FD1.6090506@ee.byu.edu> Filip Wasilewski wrote: >Hi Travis, > >this is a great example of the __array_interface__ usage. > > > Just to complete the example: With the Image.py patch that adds the __array_interface__ you can do import Image, pylab im = Image.open('somefile.jpg') pylab.imshow(im, origin='lower') and get a nice picture in the matplotlib window (at least if you are running NumPy). -Travis From oliphant at ee.byu.edu Tue Jul 11 18:02:14 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 11 Jul 2006 16:02:14 -0600 Subject: [Numpy-discussion] Quicker image transfer, tobuffer? In-Reply-To: <1785255519.20060711233156@gmail.com> References: <44B2EC45.8020402@noaa.gov> <44B3FDEB.1000909@ieee.org> <1785255519.20060711233156@gmail.com> Message-ID: <44B41FE6.9010908@ee.byu.edu> Filip Wasilewski wrote: >Hi Travis, > >this is a great example of the __array_interface__ usage. > > > >The second seems to be more complex and may be a more general. The >memory of string created by self.tostring() seems to be deallocated >before array is created (v 0.9.9.2788, win). >Everything works fine after storing the reference to data, but this >probably should be done somewhere else: > > def __get_array_interface__(self): > new = {} > shape, typestr = _conv_type_shape(self) > new['shape'] = shape > new['typestr'] = typestr > new['data'] = self.tostring() > self._str_data = new['data'] # a dirty hack > return new > > > This is now fixed in NumPy. The problem was that when the "buffer" interface was used a reference to the object was kept (but not the buffer). In this case it's the reference to the buffer that is needed. -Travis From strawman at astraw.com Tue Jul 11 18:25:36 2006 From: strawman at astraw.com (Andrew Straw) Date: Tue, 11 Jul 2006 15:25:36 -0700 Subject: [Numpy-discussion] Quicker image transfer, tobuffer? In-Reply-To: <44B41FD1.6090506@ee.byu.edu> References: <44B2EC45.8020402@noaa.gov> <44B3FDEB.1000909@ieee.org> <1785255519.20060711233156@gmail.com> <44B41FD1.6090506@ee.byu.edu> Message-ID: <44B42560.4030000@astraw.com> Travis Oliphant wrote: >Filip Wasilewski wrote: > > > >>Hi Travis, >> >>this is a great example of the __array_interface__ usage. >> >> >> >> >> > >Just to complete the example: > >With the Image.py patch that adds the __array_interface__ > >you can do > >import Image, pylab > >im = Image.open('somefile.jpg') >pylab.imshow(im, origin='lower') > >and get a nice picture in the matplotlib window (at least if you are >running NumPy). > > Actually, this has been in MPL for a while. For example, see the image_demo3.py example. You don't need the __array_interface__ for this bit of functionality. From efiring at hawaii.edu Tue Jul 11 18:38:15 2006 From: efiring at hawaii.edu (Eric Firing) Date: Tue, 11 Jul 2006 12:38:15 -1000 Subject: [Numpy-discussion] Quicker image transfer, tobuffer? In-Reply-To: <44B42560.4030000@astraw.com> References: <44B2EC45.8020402@noaa.gov> <44B3FDEB.1000909@ieee.org> <1785255519.20060711233156@gmail.com> <44B41FD1.6090506@ee.byu.edu> <44B42560.4030000@astraw.com> Message-ID: <44B42857.1020907@hawaii.edu> Andrew Straw wrote: > Actually, this has been in MPL for a while. For example, see the > image_demo3.py example. You don't need the __array_interface__ for this > bit of functionality. It's broken. The first problem is that the kw "aspect = 'preserve'" is no longer needed or supported. Removing that (as I will do in svn shortly), I get a somewhat scrambled image. Eric From efiring at hawaii.edu Tue Jul 11 18:55:54 2006 From: efiring at hawaii.edu (Eric Firing) Date: Tue, 11 Jul 2006 12:55:54 -1000 Subject: [Numpy-discussion] Quicker image transfer, tobuffer? In-Reply-To: <44B42857.1020907@hawaii.edu> References: <44B2EC45.8020402@noaa.gov> <44B3FDEB.1000909@ieee.org> <1785255519.20060711233156@gmail.com> <44B41FD1.6090506@ee.byu.edu> <44B42560.4030000@astraw.com> <44B42857.1020907@hawaii.edu> Message-ID: <44B42C7A.7070701@hawaii.edu> Eric Firing wrote: > Andrew Straw wrote: > > >>Actually, this has been in MPL for a while. For example, see the >>image_demo3.py example. You don't need the __array_interface__ for this >>bit of functionality. > > > It's broken. > > The first problem is that the kw "aspect = 'preserve'" is no longer > needed or supported. Removing that (as I will do in svn shortly), I get > a somewhat scrambled image. Correction: I did fix the first problem, and the second problem is not at all what I thought. Instead, the examples/data/lena.jpg file in my svn mpl directory is corrupted. I have no idea why. Looking directly at the version on svn via the svn browser, I see that it is corrupted also. Eric From jdhunter at ace.bsd.uchicago.edu Tue Jul 11 18:55:53 2006 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Tue, 11 Jul 2006 17:55:53 -0500 Subject: [Numpy-discussion] [matplotlib-devel] Quicker image transfer, tobuffer? In-Reply-To: <44B42C7A.7070701@hawaii.edu> (Eric Firing's message of "Tue, 11 Jul 2006 12:55:54 -1000") References: <44B2EC45.8020402@noaa.gov> <44B3FDEB.1000909@ieee.org> <1785255519.20060711233156@gmail.com> <44B41FD1.6090506@ee.byu.edu> <44B42560.4030000@astraw.com> <44B42857.1020907@hawaii.edu> <44B42C7A.7070701@hawaii.edu> Message-ID: <87r70r7nuu.fsf@peds-pc311.bsd.uchicago.edu> >>>>> "Eric" == Eric Firing writes: Eric> Correction: I did fix the first problem, and the second Eric> problem is not at all what I thought. Instead, the Eric> examples/data/lena.jpg file in my svn mpl directory is Eric> corrupted. I have no idea why. Looking directly at the This usually happens whenever Andrew commits -- don't know why (platform dependent new line problem, perhaps?) peds-pc311:~/mpl> svn log | grep astraw|head r2480 | astraw | 2006-06-15 06:33:07 -0500 (Thu, 15 Jun 2006) | 1 line r2430 | astraw | 2006-06-06 15:12:33 -0500 (Tue, 06 Jun 2006) | 1 line r2279 | astraw | 2006-04-10 10:35:31 -0500 (Mon, 10 Apr 2006) | 3 lines r2180 | astraw | 2006-03-20 15:38:12 -0600 (Mon, 20 Mar 2006) | 1 line JDH From josh8912 at yahoo.com Tue Jul 11 19:16:45 2006 From: josh8912 at yahoo.com (JJ) Date: Tue, 11 Jul 2006 23:16:45 +0000 (UTC) Subject: [Numpy-discussion] suggestions for Matrix-related changes Message-ID: Hello. For what its worth, as a newly ex-matlab user I would like to make a few suggestions on use of matrices in numpy. As per earlier discussions, I like the idea of being able to choose matrices as the default (vs arrays). But if possible, it would be nice if all functions etc that took matrices also returned matrices. I know effort has been made on this. Here are my suggestions: 1) is it possible to get the function unique() to work with matrices, perhaps with a unique_rows() function to work with matrices of more than one column? 2) It would be very convienient to have some simple way to delete selected columns of a matrix. For example, in matlab the command is X[:,[3,5,7]]=[] to delete the three selected columns. It would be nice if such a command would also work with selections, as in X[:,A[0,:]<4] = [], where X and A are matrices. For me, the single most frustrating and time-consuming aspect of switching to and using numpy is determing how to select columns/rows of arrays/matrices in different situations. In addition to being time consuming to figure out (relative to Matlab), often the code is quite verbose. I have made a few suggestions on this topic in an earlier post (under the title "Whats wrong with matrices?"). 3) It would be nice if matrices could be used for iterations. For example, if M was a 1 x n matrix, it would be nice to be able to use: for i in M: and iterate over the individual items in M. 4) It would be nice if the linear algebra package and other packages returned matrices if given matrices. For example, if M is a matrix, svd(M) now returns arrays. Just some suggestions. I wish I knew more so I could help implement them. Maybe one day. JJ From fperez.net at gmail.com Tue Jul 11 19:19:18 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 11 Jul 2006 17:19:18 -0600 Subject: [Numpy-discussion] [matplotlib-devel] Quicker image transfer, tobuffer? In-Reply-To: <87r70r7nuu.fsf@peds-pc311.bsd.uchicago.edu> References: <44B3FDEB.1000909@ieee.org> <1785255519.20060711233156@gmail.com> <44B41FD1.6090506@ee.byu.edu> <44B42560.4030000@astraw.com> <44B42857.1020907@hawaii.edu> <44B42C7A.7070701@hawaii.edu> <87r70r7nuu.fsf@peds-pc311.bsd.uchicago.edu> Message-ID: On 7/11/06, John Hunter wrote: > >>>>> "Eric" == Eric Firing writes: > > Eric> Correction: I did fix the first problem, and the second > Eric> problem is not at all what I thought. Instead, the > Eric> examples/data/lena.jpg file in my svn mpl directory is > Eric> corrupted. I have no idea why. Looking directly at the > > This usually happens whenever Andrew commits -- don't know why > (platform dependent new line problem, perhaps?) Is that file tagged as binary in the repo? If it is, it should be impervious to OS-dependent EOL conventions... Cheers, f From filip at ftv.pl Tue Jul 11 19:36:11 2006 From: filip at ftv.pl (Filip Wasilewski) Date: Wed, 12 Jul 2006 01:36:11 +0200 Subject: [Numpy-discussion] Array interface and builtin array.array example Message-ID: <146309645.20060712013611@gmail.com> Hi, the way of accessing data with __array_interface__, as shown by Travis in [1], also works nicely when used with builtin array.array (if someone here is still using it;). Time to convert array.array to ndarray is O(N) but can be made O(1) just by simple subclassing. [1] http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3191164 cheers, fw ----------------------------------------------------------------------- #!/usr/bin/env python # -*- coding: utf-8 -*- import array as _array import sys if sys.byteorder == 'little': _ENDIAN = '<' else: _ENDIAN = '>' _TYPES_CONV ={ 'c': '|u%%d', #character 1 'b': '|i%%d', #signed integer 1 'B': '|u%%d', #unsigned integer 1 'u': '%su%%d' % _ENDIAN, #Unicode character 2 'h': '%si%%d' % _ENDIAN, #signed integer 2 'H': '%su%%d' % _ENDIAN, #unsigned integer 2 'i': '%si%%d' % _ENDIAN, #signed integer 2 (4?) 'I': '%su%%d' % _ENDIAN, #unsigned integer 2 (4?) 'l': '%si%%d' % _ENDIAN, #signed integer 4 'L': '%su%%d' % _ENDIAN, #unsigned integer 4 'f': '%sf%%d' % _ENDIAN, #floating point 4 'd': '%sf%%d' % _ENDIAN, #floating point 8 } class array(_array.array): def __get_array_interface__(self): new = {} shape, typestr = (self.__len__(),), (_TYPES_CONV[self.typecode] % self.itemsize) new['shape'] = shape new['typestr'] = typestr new['data'] = (self.buffer_info()[0], False) # writable return new __array_interface__ = property(__get_array_interface__, None, doc="array interface") if __name__ == '__main__': size = 1000000 typecode = 'f' new = array(typecode, xrange(size)) old = _array.array(typecode, xrange(size)) import numpy from time import clock as time t1 = time() nd = numpy.asarray(new) t1 = time() - t1 #print nd t2 = time() nd = numpy.asarray(old) t2 = time() - t2 #print nd print "new:", t1 print "old:", t2 #EOF From strawman at astraw.com Tue Jul 11 19:38:34 2006 From: strawman at astraw.com (Andrew Straw) Date: Tue, 11 Jul 2006 16:38:34 -0700 Subject: [Numpy-discussion] [matplotlib-devel] Quicker image transfer, tobuffer? In-Reply-To: <87r70r7nuu.fsf@peds-pc311.bsd.uchicago.edu> References: <44B2EC45.8020402@noaa.gov> <44B3FDEB.1000909@ieee.org> <1785255519.20060711233156@gmail.com> <44B41FD1.6090506@ee.byu.edu> <44B42560.4030000@astraw.com> <44B42857.1020907@hawaii.edu> <44B42C7A.7070701@hawaii.edu> <87r70r7nuu.fsf@peds-pc311.bsd.uchicago.edu> Message-ID: <44B4367A.3010708@astraw.com> John Hunter wrote: >>>>>>"Eric" == Eric Firing writes: >>>>>> >>>>>> > > Eric> Correction: I did fix the first problem, and the second > Eric> problem is not at all what I thought. Instead, the > Eric> examples/data/lena.jpg file in my svn mpl directory is > Eric> corrupted. I have no idea why. Looking directly at the > >This usually happens whenever Andrew commits -- don't know why >(platform dependent new line problem, perhaps?) > >peds-pc311:~/mpl> svn log | grep astraw|head >r2480 | astraw | 2006-06-15 06:33:07 -0500 (Thu, 15 Jun 2006) | 1 line >r2430 | astraw | 2006-06-06 15:12:33 -0500 (Tue, 06 Jun 2006) | 1 line >r2279 | astraw | 2006-04-10 10:35:31 -0500 (Mon, 10 Apr 2006) | 3 >lines >r2180 | astraw | 2006-03-20 15:38:12 -0600 (Mon, 20 Mar 2006) | 1 line > > > Hmm -- "usually happens"? I never noticed that. And I'm mystified as to whether the output of svn log shows that. Let me know if I play any more evil-line-ending tricks. Anyhow, I think I fixed the corrupted file issue. I changed the deleted the svn:eol-style property and added the set svn:mime-type property to image/jpg and re-uploaded lena.jpg. I suspect this may have been a victim of the cvs2svn switch, or perhaps I never checked it into cvs properly. Cheers! Andrew From oliphant at ee.byu.edu Tue Jul 11 19:41:55 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 11 Jul 2006 17:41:55 -0600 Subject: [Numpy-discussion] suggestions for Matrix-related changes In-Reply-To: References: Message-ID: <44B43743.5090403@ee.byu.edu> JJ wrote: >Hello. For what its worth, as a newly ex-matlab user I would like to make a few >suggestions on use of matrices in numpy. As per earlier discussions, I like the >idea of being able to choose matrices as the default (vs arrays). But if >possible, it would be nice if all functions etc that took matrices also returned >matrices. I know effort has been made on this. Here are my suggestions: > >1) is it possible to get the function unique() to work with matrices, perhaps >with a unique_rows() function to work with matrices of more than one column? > >2) It would be very convienient to have some simple way to delete selected >columns of a matrix. > This is a good idea. It would be nice to address it at some point. There is a Python syntax for it, but we are not using it yet: del X[...] Of course one of the problems with this syntax (as opposed to a function that returns a new array) is that because X can share it's data with other arrays, you can't just re-size it's memory or other arrays depending on that chunk of memory will be in deep trouble. So, we are probably not going to be able to have a "syntax-style" delete. But, some kind of function that returns an array with specific entries deleted would be nice. >3) It would be nice if matrices could be used for iterations. For example, if M >was a 1 x n matrix, it would be nice to be able to use: for i in M: and >iterate over the individual items in M. > > They can be used as iterators. The problem here is simply convention (rows are iterated over first). We could over-ride the iterator behavior of matrices, though to handle 1xn and nx1 matrices identically if that is desirable. >4) It would be nice if the linear algebra package and other packages returned >matrices if given matrices. For example, if M is a matrix, svd(M) now returns > > Svd returns matrices now. Except for the list of singular values which is still an array. Do you want a 1xn matrix instead of an array? -Travis From oliphant at ee.byu.edu Tue Jul 11 19:43:09 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 11 Jul 2006 17:43:09 -0600 Subject: [Numpy-discussion] Array interface and builtin array.array example In-Reply-To: <146309645.20060712013611@gmail.com> References: <146309645.20060712013611@gmail.com> Message-ID: <44B4378D.7010701@ee.byu.edu> Filip Wasilewski wrote: >Hi, > >the way of accessing data with __array_interface__, as shown by Travis >in [1], also works nicely when used with builtin array.array (if someone >here is still using it;). > >Time to convert array.array to ndarray is O(N) but can be made O(1) just >by simple subclassing. > >[1] http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3191164 > > > This is exactly the kind of thing I'd like to see get into Python. Thanks for picking up the ball.. -Travis From josh8912 at yahoo.com Tue Jul 11 20:00:47 2006 From: josh8912 at yahoo.com (JJ) Date: Wed, 12 Jul 2006 00:00:47 +0000 (UTC) Subject: [Numpy-discussion] bug in stats.binom? Or is it just me. Message-ID: Am I using the wrong syntax for the binom.ppf command, or is there a bug? >>> stats.binom.ppf(.975,100,.5) Traceback (most recent call last): File "", line 1, in ? File "/usr/lib64/python2.4/site-packages/scipy/stats/distributions.py", line 3590, in ppf insert(output,cond,self._ppf(*goodargs) + loc) File "/usr/lib64/python2.4/site-packages/numpy/lib/function_base.py", line 501, in insert return _insert(arr, mask, vals) TypeError: array cannot be safely cast to required type >>> ----------------- The info pages for binom.ppf state that: binom.ppf(q,n,pr,loc=0) - percent point function (inverse of cdf --- percentiles) So I would expect binom.ppf to take three variables. I expected the function to return a number, such as is done in matlab: N = 100 alpha = 0.05 p1 = 0.30 cutoff = binoinv(1-alpha, N, p1) cutoff = 38 Any suggestions? JJ From michael.sorich at gmail.com Tue Jul 11 20:07:56 2006 From: michael.sorich at gmail.com (Michael Sorich) Date: Wed, 12 Jul 2006 09:37:56 +0930 Subject: [Numpy-discussion] suggestions for Matrix-related changes In-Reply-To: References: Message-ID: <16761e100607111707i4d336f9cke1a0b92c0d474f9e@mail.gmail.com> On 7/12/06, JJ wrote: > 2) It would be very convienient to have some simple way to delete selected > columns of a matrix. For example, in matlab the command is X[:,[3,5,7]]=[] to > delete the three selected columns. It would be nice if such a command would > also work with selections, as in X[:,A[0,:]<4] = [], where X and A are matrices. +1. In R negative integers are used for this purpose and a copy of the array is returned. e.g. > x = 1:10 > x [1] 1 2 3 4 5 6 7 8 9 10 > x[-1] [1] 2 3 4 5 6 7 8 9 10 > x[c(-1,-2)] [1] 3 4 5 6 7 8 9 10 > x[-c(1,2)] [1] 3 4 5 6 7 8 9 10 I like the current use of negative indices in numpy, but I do find myself missing the ability to easily make a copy of the array without certain indices. Mike From kwgoodman at gmail.com Tue Jul 11 20:11:12 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 11 Jul 2006 17:11:12 -0700 Subject: [Numpy-discussion] suggestions for Matrix-related changes In-Reply-To: <44B43743.5090403@ee.byu.edu> References: <44B43743.5090403@ee.byu.edu> Message-ID: On 7/11/06, Travis Oliphant wrote: > JJ wrote: > >4) It would be nice if the linear algebra package and other packages returned > >matrices if given matrices. For example, if M is a matrix, svd(M) now returns > > > > > Svd returns matrices now. Except for the list of singular values which > is still an array. Do you want a 1xn matrix instead of an array? That sounds good to me. The same goes for eig and eigh: >> eigval,eigvec = linalg.eig(rand(2,2)) >> eigval array([-0.06035002, 0.14320639]) >> eigvec matrix([[ 0.54799954, -0.83647863], [-0.83647863, -0.54799954]]) From josh8912 at yahoo.com Tue Jul 11 20:23:28 2006 From: josh8912 at yahoo.com (JJ) Date: Wed, 12 Jul 2006 00:23:28 +0000 (UTC) Subject: [Numpy-discussion] suggestions for Matrix-related changes References: <44B43743.5090403@ee.byu.edu> Message-ID: Travis Oliphant ee.byu.edu> writes: > But, some kind of function that returns an array with specific > entries deleted would be nice. I agree. This would be just fine. > We could over-ride the iterator > behavior of matrices, though to handle 1xn and nx1 matrices > identically if that is desirable. I had tried this iteration on a month-old version of numpy and it did not work. I guess this now has been changed. I just updated my copy but have not yet tried it. An over-ride might be nice. But just off the topic, could you get a matrix of real numbers such as A= [[1.0 2.0,3.0]] to be used to select rows/colums as in B[:,A]? I guess this would require a hidden conversion to integers, as well as code to handle selection using a matrix. > Svd returns matrices now. Except for the list of singular values > which is still an array. Do you want a 1xn matrix instead of an > array? I had just tried this with my new version of numpy, but I had used svd as follows: import scipy.linalg as la res = la.svd(M) That returned arrays, but I see that using: res = linalg.svd(M) returns matrices. Apparently, both numpy and scipy have linalg packages, which differ. I did not know that. Whoops. From ckkart at hoc.net Tue Jul 11 21:11:04 2006 From: ckkart at hoc.net (Christian Kristukat) Date: Wed, 12 Jul 2006 01:11:04 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?bdist=5Frpm_failure?= Message-ID: Hi, currently the bdist_rpm build method seems to be quite unstable. It works for some recent svn revisions, for the current 2804, however, not. The error messages begin with: building 'numpy.core.multiarray' extension compiling C sources C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -march=i586 -mtune=i686 -fmessage-length=0 -Wall - D_FORTIFY_SOURCE=2 -g -O2 -g -m32 -march=i586 -mtune=i686 -fmessage-length=0 -D_FORTIFY_SOURCE=2 -fPIC creating build/temp.linux-i686-2.4 creating build/temp.linux-i686-2.4/numpy creating build/temp.linux-i686-2.4/numpy/core creating build/temp.linux-i686-2.4/numpy/core/src compile options: '-Ibuild/src.linux-i686-2.4/numpy/core/src -Inumpy/core/include -Ibuild/src.linux-i686-2.4/n umpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' gcc: numpy/core/src/multiarraymodule.c numpy/core/src/multiarraymodule.c:24:28: error: numpy/noprefix.h: No such file or directory numpy/core/src/multiarraymodule.c:33: error: expected ?=?, ?,?, ?;?, ?asm? or ?__attribute__? before ?*? toke n Regards, Christian From oliphant.travis at ieee.org Tue Jul 11 21:24:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 11 Jul 2006 19:24:01 -0600 Subject: [Numpy-discussion] bdist_rpm failure In-Reply-To: References: Message-ID: <44B44F31.2020508@ieee.org> Christian Kristukat wrote: > Hi, > currently the bdist_rpm build method seems to be quite unstable. It works for > some recent svn revisions, for the current 2804, however, not. > The error messages begin with: > > building 'numpy.core.multiarray' extension > compiling C sources > C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -march=i586 > -mtune=i686 -fmessage-length=0 -Wall - > D_FORTIFY_SOURCE=2 -g -O2 -g -m32 -march=i586 -mtune=i686 -fmessage-length=0 > -D_FORTIFY_SOURCE=2 -fPIC > > creating build/temp.linux-i686-2.4 > creating build/temp.linux-i686-2.4/numpy > creating build/temp.linux-i686-2.4/numpy/core > creating build/temp.linux-i686-2.4/numpy/core/src > compile options: '-Ibuild/src.linux-i686-2.4/numpy/core/src -Inumpy/core/include > -Ibuild/src.linux-i686-2.4/n > umpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' > gcc: numpy/core/src/multiarraymodule.c > numpy/core/src/multiarraymodule.c:24:28: error: numpy/noprefix.h: No such file > or directory > numpy/core/src/multiarraymodule.c:33: error: expected ?=?, ?,?, ?;?, ?asm? or > ?__attribute__? before ?*? toke > n > > Make sure you get rid of the MANIFEST file in the source directory before trying to run sdist or bdist_rpm. The MANIFEST file is not being deleted when it is dated... -Travis From david.huard at gmail.com Tue Jul 11 22:36:09 2006 From: david.huard at gmail.com (David Huard) Date: Tue, 11 Jul 2006 22:36:09 -0400 Subject: [Numpy-discussion] bug in stats.binom? Or is it just me. In-Reply-To: References: Message-ID: <91cf711d0607111936l3e93a50aja3b7235df15d616@mail.gmail.com> There probably is a bug because stats.binom.ppf(.975, 100, .5) crashes the ipython shell. win2k, P3, latest binary release. David 2006/7/11, JJ : > > Am I using the wrong syntax for the binom.ppf command, or is there a bug? > > >>> stats.binom.ppf(.975,100,.5) > Traceback (most recent call last): > File "", line 1, in ? > File "/usr/lib64/python2.4/site-packages/scipy/stats/distributions.py", > line > 3590, in ppf > insert(output,cond,self._ppf(*goodargs) + loc) > File "/usr/lib64/python2.4/site-packages/numpy/lib/function_base.py", > line > 501, in insert > return _insert(arr, mask, vals) > TypeError: array cannot be safely cast to required type > >>> > ----------------- > The info pages for binom.ppf state that: > binom.ppf(q,n,pr,loc=0) > - percent point function (inverse of cdf --- percentiles) > So I would expect binom.ppf to take three variables. > > I expected the function to return a number, such as is done in matlab: > N = 100 > alpha = 0.05 > p1 = 0.30 > cutoff = binoinv(1-alpha, N, p1) > > cutoff = > > 38 > Any suggestions? > > > JJ > > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndarray at mac.com Tue Jul 11 22:38:31 2006 From: ndarray at mac.com (Sasha) Date: Tue, 11 Jul 2006 22:38:31 -0400 Subject: [Numpy-discussion] Array interface and builtin array.array example In-Reply-To: <44B4378D.7010701@ee.byu.edu> References: <146309645.20060712013611@gmail.com> <44B4378D.7010701@ee.byu.edu> Message-ID: I had similar hopes when I submited the array interface patch and announced it on python-dev . I am still waiting for anyone to comment on it :-( On 7/11/06, Travis Oliphant wrote: > Filip Wasilewski wrote: > > >Hi, > > > >the way of accessing data with __array_interface__, as shown by Travis > >in [1], also works nicely when used with builtin array.array (if someone > >here is still using it;). > > > >Time to convert array.array to ndarray is O(N) but can be made O(1) just > >by simple subclassing. > > > >[1] http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3191164 > > > > > > > This is exactly the kind of thing I'd like to see get into Python. > Thanks for picking up the ball.. > > > -Travis > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From david.huard at gmail.com Tue Jul 11 22:42:23 2006 From: david.huard at gmail.com (David Huard) Date: Tue, 11 Jul 2006 22:42:23 -0400 Subject: [Numpy-discussion] suggestions for Matrix-related changes In-Reply-To: References: Message-ID: <91cf711d0607111942n76a5671fy87690ca67ad0bc03@mail.gmail.com> 2006/7/11, JJ : > > 1) is it possible to get the function unique() to work with matrices, > perhaps > with a unique_rows() function to work with matrices of more than one > column? The problem is that applying unique to different rows will return vectors with different lengths. So you could not return an array, much less a matrix. You'd have to return a list of arrays or 1D matrices. David -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Tue Jul 11 22:50:46 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 11 Jul 2006 19:50:46 -0700 Subject: [Numpy-discussion] suggestions for Matrix-related changes In-Reply-To: <91cf711d0607111942n76a5671fy87690ca67ad0bc03@mail.gmail.com> References: <91cf711d0607111942n76a5671fy87690ca67ad0bc03@mail.gmail.com> Message-ID: On 7/11/06, David Huard wrote: > > > 2006/7/11, JJ : > > 1) is it possible to get the function unique() to work with matrices, > perhaps > > with a unique_rows() function to work with matrices of more than one > column? > > > The problem is that applying unique to different rows will return vectors > with different lengths. So you could not return an array, much less a > matrix. ...unless it returned the unique rows instead of the unique elements in each row. So if the matrix is 1 2 2 3 4 5 1 2 2 then the unique rows would be 1 2 2 3 4 5 From robert.kern at gmail.com Wed Jul 12 00:05:35 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 11 Jul 2006 23:05:35 -0500 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: <44B414DA.1050300@ee.byu.edu> References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> <44B414DA.1050300@ee.byu.edu> Message-ID: Travis Oliphant wrote: > So, I'm opposed to getting rid of the *args based syntax. My feelings > are weaker regarding adding the capability to rand and randn to accept a > tuple. I did test it out and it does seem feasible to add this feature > at the cost of an additional comparison. I know Robert is opposed to it > but I'm not sure I understand completely why. > > Please correct me if I'm wrong, but I think it has something to do with > making the random_sample and standard_normal functions irrelevant and > unnecessary combined with my hypothesis that Robert doesn't like the > *args-style syntax. Therefore, adding support to rand and randn for > tuples, would make them the default random-number generators and there > would be proliferating code that was "harder to read" because of the > different usages. My opposition has much to do with my natural orneriness, I suppose. However, for most of this argument I've never seen a reason why rand() ought to be considered a peer of ones() and zeros() and thus have any need to be "consistent" in this one respect with them. I've always considered ones() and zeros() to be fundamental constructors of basic arrays of an arbitrary dtype that (most likely) you'll be mangling up immediately. I've seen rand() and all of the other RandomState methods as simply functions that return arrays. In that respect, they have more in common with arange() than anything else. However, I've come to realize that because rand() is exposed in the top-level namespace, other people probably *are* seeing it as another fundamental constructor of arrays (albeit of a single dtype). When I was writing the package, I never even considered that some of its functions might be imported into the top-level namespace. I'm somewhat more sympathetic to the confusion caused by the rand()-as-constructor viewpoint, now. That's why my currently preferred compromise position is to remove rand() from numpy.* and leave it in numpy.random*. I also have found that I just don't use those functions convenient in "real" code. They usually come into play when I need one or lots of arbitrary, but non-degenerate-with-high-probability arrays to as a test or demo input to another piece of code. Otherwise, I probably need something more sophisticated. Manually shifting and scaling standard variates makes the code more verbose without making it more comprehensible. I do indeed dislike functions that try to be clever in determining what the user wanted by the number and types of arguments provided. Even if they are easy to implement robustly (as in this case), I think that it makes explaining the function more complicated. It also makes the same function be called in two very different ways; I find this break in consistency rather more egregious than two different functions, used for different purposes in different circumstances, being called in two different ways. In my experience, it's much more likely to cause confusion. I'm currently trudging my way through a mountain of someone else's C++ code at the moment. There are good uses of C++'s function overloading, for example to do type polymorphism on the same number of arguments when templates don't quite cut it. However, if it is abused to create fundamentally different ways to call the same function, I've found that the readability drops rapidly. I have not been a happy camper. I have yet to see a good exposition of the actual *problems* the current situation is causing. Most of the argument in favor of changing anything has been a call for consistency. However, consistency can never be an end of itself. It must always be considered a means to achieve a comprehensible, flexible, usable API. But it's only one of several means to that goal, and even that goal must be weighed against other goals, too. When real problems were discussed, I didn't find the solution to fit those problems. Alan said that it was an annoyance (at the very least) to have to teach students that rand() is called differently from ones(). The answer here is to not teach rand(); use the functions that follow the one convention that you want to teach. Also teach the students to read docstrings so if they come across rand() in their copious spare time, they know what's what. Another almost-compelling-to-me real problem was someone saying that they always had to pause when writing rand() and think about what the calling convention was. My answer is similar: don't use rand(). This problem, as described, is a "write-only" problem; if you want consistency in the calling convention between ones() et al. and your source of random numbers, it's there. You might run into uses of rand() in other people's code, but it'll never confuse you as to what the calling convention is. However, I do think that the presence of rand() in the numpy.* namespace is probably throwing people off. They default to rand() and chafe against its API even though they'd be much happier with the tuple-based API. But mostly my opposition follows from this observation: making rand() cleverly handle its argument tuple *does not solve the problem*. A polymorphic rand() will *still* be inconsistent with ones() and zeros() because ones() and zeros() won't be polymorphic, too. And they can't be really; the fact that they take other arguments besides the shape tuple makes the implementation and use idioms rather harder than for rand(). And mark my words, if we make rand() polymorphic, we will get just as many newbies coming to the list asking why ones(3, 4) doesn't work. I've already described how I feel that this would just trade one inconsistency (between different functions) for another (between different uses of the same function). I find the latter much worse. To summarize: while I'd rather just leave things as they are, I realize that feeling is more from spite than anything else, and I'm above that. Mostly. I *do* think that the brouhaha will rapidly diminish if rand() and rand() are simply removed from numpy.* and left in numpy.random.* where they belong. While I'm sure that some who have been pushing for consistency the hardest will still grumble a bit, I strongly suspect that no one would have thought to complain if this had been the configuration from the very beginning. Okay, now I think I've officially spent more time on this email than I ever did using or implementing rand(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From josh8912 at yahoo.com Wed Jul 12 01:01:14 2006 From: josh8912 at yahoo.com (JJ) Date: Tue, 11 Jul 2006 22:01:14 -0700 (PDT) Subject: [Numpy-discussion] suggestions for Matrix-related changes In-Reply-To: Message-ID: <20060712050114.90890.qmail@web51704.mail.yahoo.com> > ...unless it returned the unique rows instead of the > unique elements > in each row. So if the matrix is > > 1 2 2 > 3 4 5 > 1 2 2 > > then the unique rows would be > > 1 2 2 > 3 4 5 > Hello Keith. Yes, that is what I mean also. JJ __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From fperez.net at gmail.com Wed Jul 12 02:02:26 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 12 Jul 2006 00:02:26 -0600 Subject: [Numpy-discussion] bdist_rpm failure In-Reply-To: <44B44F31.2020508@ieee.org> References: <44B44F31.2020508@ieee.org> Message-ID: On 7/11/06, Travis Oliphant wrote: > Make sure you get rid of the MANIFEST file in the source directory > before trying to run sdist or bdist_rpm. The MANIFEST file is not being > deleted when it is dated... Since both numpy/scipy have a MANIFEST.in, this bit of code (from ipython's setup.py) is probably a good idea to put in: # BEFORE importing distutils, remove MANIFEST. distutils doesn't properly # update it when the contents of directories change. if os.path.exists('MANIFEST'): os.remove('MANIFEST') I committed this as I don't see any problem with it (ipython has used it for years). Feel free to revert if you object. I also added #!/usr/bin/env python to both of them. They were set as executable scripts but without a proper shebang line, causing the interesting mis-behavior of being treated like shell scripts. Which means the first import os line would simply give you a weird-looking cross-hairs cursor, and a gigantic file called 'os' sitting on your directory once you clicked. Why? That's the ImageMagick 'import' command, which takes a screenshot in X bitmap format :) This one bit me a few times, so I decided to also fix it, but feel free to revert if you see a good reason for the existing behavior. Cheers, f From wbaxter at gmail.com Wed Jul 12 02:09:55 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 12 Jul 2006 15:09:55 +0900 Subject: [Numpy-discussion] Args for rand and randn, and workarounds Message-ID: > > And mark my words, if we make rand() polymorphic, > we will get just as many newbies coming to the list asking why ones(3, 4) > doesn't work. > I think you're probably right there, at least for Matlab converts. Matlab allows either way of calling for all of rand, ones, zeros, eye, and in general tries to make *every* function accept *any* possible combination of arguments that might make sense. So having rand() accept both, but not the others, will undoubtedly be unexpected to Matlab users. And the argument against funky overloading will be weaker since they'll be able to say "well rand() does it? why can't ones()?" Fortunately it's pretty easy to define one's own custom versions of these little helper functions to make them just the way you like them. Or for rand(), just "from numpy.random import uniform_sample as rand" if that's the version you like. This isn't a panacea, but you can then collect all these into a little customization module, say "mynum.py" which is somewhere on your PYTHONPATH, and then set up all your customizations with a "from mynum import *" You can even go further if you wish and define the PYTHONSTARTUP environment variable to point to a file that does "from mynum import *" so that every time you start up the interactive interpreter you have your definitions there already. I'm sure this is all well known to you long-time Python'ers out there, but I thought I'd mention it since there are probably other numpy users out there who, like me, are pretty new to Python. --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Wed Jul 12 02:52:57 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 12 Jul 2006 00:52:57 -0600 Subject: [Numpy-discussion] Args for rand and randn, and workarounds In-Reply-To: References: Message-ID: <44B49C49.1040906@ieee.org> Bill Baxter wrote: > > And mark my words, if we make rand() polymorphic, > > we will get just as many newbies coming to the list asking why > ones(3, 4) > > doesn't work. > > > I think you're probably right there, at least for Matlab converts. > Matlab allows either way of calling for all of rand, ones, zeros, eye, > and in general tries to make *every* function accept *any* possible > combination of arguments that might make sense. So having rand() > accept both, but not the others, will undoubtedly be unexpected to > Matlab users. And the argument against funky overloading will be > weaker since they'll be able to say "well rand() does it? why can't > ones()?" > > Fortunately it's pretty easy to define one's own custom versions of > these little helper functions to make them just the way you like > them. Or for rand(), just "from numpy.random import uniform_sample > as rand" if that's the version you like. Because of this. I've removed the global_namespace functions (fft, ifft, rand, and randn) from numpy. They are *no longer* in the top-level name-space. If you want them, setup a startup-file appropriately. -Travis From a.u.r.e.l.i.a.n at gmx.net Wed Jul 12 03:10:40 2006 From: a.u.r.e.l.i.a.n at gmx.net (Johannes Loehnert) Date: Wed, 12 Jul 2006 09:10:40 +0200 Subject: [Numpy-discussion] suggestions for Matrix-related changes In-Reply-To: <16761e100607111707i4d336f9cke1a0b92c0d474f9e@mail.gmail.com> References: <16761e100607111707i4d336f9cke1a0b92c0d474f9e@mail.gmail.com> Message-ID: <200607120910.40686.a.u.r.e.l.i.a.n@gmx.net> On Wednesday 12 July 2006 02:07, Michael Sorich wrote: > On 7/12/06, JJ wrote: > > 2) It would be very convienient to have some simple way to delete > > selected columns of a matrix. For example, in matlab the command is > > X[:,[3,5,7]]=[] to delete the three selected columns. It would be nice > > if such a command would also work with selections, as in X[:,A[0,:]<4] = > > [], where X and A are matrices. > (...) > I like the current use of negative indices in numpy, but I do find > myself missing the ability to easily make a copy of the array without > certain indices. Maybe use complex numbers? draft: >>> a = arange(10) >>> print a[ 3j ] [ 0 1 2 4 5 6 7 8 9 ] Johannes From wbaxter at gmail.com Wed Jul 12 04:24:53 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 12 Jul 2006 17:24:53 +0900 Subject: [Numpy-discussion] Testing a self-compiled numpy In-Reply-To: <44B2614E.7060101@enthought.com> References: <44B2614E.7060101@enthought.com> Message-ID: On 7/10/06, Bryce Hendrix wrote: > > Keith Goodman wrote: > > Doesn't --prefix=/install/numpy/here/ work on windows? > > For our Windows Enthon project, the build command is > > setup.py config --compiler=mingw32 build --compiler=mingw32 install -- > prefix=path_to_where_numpy_goes. > Thanks for the info. I added it to the "building numpy/scipy on windows" wiki. Still looking for help on how to get SciPy to build on Windows if anyone knows anything about that. I posted my symptoms to the SciPy list, but got no response. --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Wed Jul 12 04:47:15 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 12 Jul 2006 17:47:15 +0900 Subject: [Numpy-discussion] Args for rand and randn, and workarounds In-Reply-To: <44B49C49.1040906@ieee.org> References: <44B49C49.1040906@ieee.org> Message-ID: On 7/12/06, Travis Oliphant wrote: > > > Because of this. I've removed the global_namespace functions (fft, > ifft, rand, and randn) from numpy. They are *no longer* in the > top-level name-space. If you want them, setup a startup-file > appropriately. > Any hints as to where we can find them now? (more generally -- how is one supposed to find stuff in big python packages to begin with? Like -- "show me anything with 'rand' in the name".) Also, numpy.matlib.rand() needs to be updated to point to the new location as well. --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Wed Jul 12 05:00:56 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Wed, 12 Jul 2006 11:00:56 +0200 Subject: [Numpy-discussion] Args for rand and randn, and workarounds In-Reply-To: References: <44B49C49.1040906@ieee.org> Message-ID: <20060712090056.GA21547@mentat.za.net> On Wed, Jul 12, 2006 at 05:47:15PM +0900, Bill Baxter wrote: > On 7/12/06, Travis Oliphant wrote: > > > Because of this. I've removed the global_namespace functions (fft, > ifft, rand, and randn) from numpy. They are *no longer* in the > top-level name-space. If you want them, setup a startup-file > appropriately. > > > > Any hints as to where we can find them now? > (more generally -- how is one supposed to find stuff in big python packages to > begin with? Like -- "show me anything with 'rand' in the name".) It is in numpy.random.rand. If you are using ipython, you can do import numpy numpy.*rand*? which will print out a list of all members that contain "rand". In this case numpy.random. To do a deeper search, use pydoc. You can start it with pydoc -g which should pop up a graphical interface, where you can type in a search term. Click on the results to open the appropriate documentation web page. > Also, numpy.matlib.rand() needs to be updated to point to the new location as > well. Already fixed in SVN. Cheers St?fan From svetosch at gmx.net Wed Jul 12 05:44:46 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Wed, 12 Jul 2006 11:44:46 +0200 Subject: [Numpy-discussion] suggestions for Matrix-related changes In-Reply-To: References: <44B43743.5090403@ee.byu.edu> Message-ID: <44B4C48E.5050602@gmx.net> JJ schrieb: > Travis Oliphant ee.byu.edu> writes: > >> Svd returns matrices now. Except for the list of singular values >> which is still an array. Do you want a 1xn matrix instead of an >> array? Although I'm a matrix supporter, I'm not sure here. Afaics the pro argument is to have *everything* a matrix when you're in that camp. Fair enough. But then it's already not clear if you want a row or a column, and you carry an extra dimension around, which is sometimes annoying e.g. for cumulation of the values, which I do a lot (for eigenvalues, that is). So for my personal use I came to the conclusion that the status quo of numpy (array for the value list, matrix for the decomp) is just fine. So maybe the people in favor of values-in-1xn-matrices can tell why they need to matrix-multiply the value array afterwards, because that's the only benefit I can see here. > > I had just tried this with my new version of numpy, but I had used svd > as follows: > import scipy.linalg as la > res = la.svd(M) > That returned arrays, but I see that using: > res = linalg.svd(M) > returns matrices. Apparently, both numpy and scipy have linalg > packages, which differ. I did not know that. Whoops. > I'm trying to get by with numpy (good that kron was brought over!), but eventually I will need scipy -- I was hoping that all the matrix discussion in the numpy list implicitly applied to scipy as well. Is that not true? Cheers, Sven From svetosch at gmx.net Wed Jul 12 06:21:05 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Wed, 12 Jul 2006 12:21:05 +0200 Subject: [Numpy-discussion] Args for rand and randn, and workarounds In-Reply-To: <44B49C49.1040906@ieee.org> References: <44B49C49.1040906@ieee.org> Message-ID: <44B4CD11.70105@gmx.net> Travis Oliphant schrieb: > > Because of this. I've removed the global_namespace functions (fft, > ifft, rand, and randn) from numpy. They are *no longer* in the > top-level name-space. If you want them, setup a startup-file > appropriately. > Ok I'm glad that's settled; however, do I understand correctly (after looking at the new numpy.matlib for the first time), that numpy.matlib.rand() still doesn't accept tuples, and that the tuple-accepting "full" version is not exposed in numpy.matlib? If so, it would be perfect if numpy.matlib could be augmented by a matrix analogue pointing to numpy.random.random() or whatever its full name is. Then the situation for matrix users is actually quite nice, just have to import everything from numpy.matlib. Thanks, Sven From T.Menneer at soton.ac.uk Wed Jul 12 06:57:48 2006 From: T.Menneer at soton.ac.uk (Tamaryn Menneer) Date: Wed, 12 Jul 2006 11:57:48 +0100 Subject: [Numpy-discussion] "ImportError: No module named numeric" Message-ID: <200607121057.k6CAvrCc028781@mta1.iss.soton.ac.uk> Hi I'm running Python 2.4 on Windows XP. I've installed NumPy, and run the simple test of "import numeric" but I get the error "ImportError: No module named numeric" in return. The module numeric is located in C:\Python24\Lib\site-packages\numpy\core, but even when I set PYTHONPATH to this it returns the same error. Imports of modules in the Lib directory work fine, but everything I've tried within site-packages results the no module error. I originally installed Python24 in Program Files, and thought this might be causing the problem with a hardwired pathname somewhere, but having uninstalled and reinstalled directly on the C drive, I still get the same problem. Any guidance would be very much appreciated. I'm virtually a complete newbie to all this, but I'll try my best to understand any responses! Thank you :-) Tam. From svetosch at gmx.net Wed Jul 12 07:02:13 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Wed, 12 Jul 2006 13:02:13 +0200 Subject: [Numpy-discussion] "ImportError: No module named numeric" In-Reply-To: <200607121057.k6CAvrCc028781@mta1.iss.soton.ac.uk> References: <200607121057.k6CAvrCc028781@mta1.iss.soton.ac.uk> Message-ID: <44B4D6B5.3090605@gmx.net> Tamaryn Menneer schrieb: > Hi > > I'm running Python 2.4 on Windows XP. I've installed NumPy, and run the > simple test of "import numeric" but I get the error "ImportError: No module > named numeric" in return. The module numeric is located in > C:\Python24\Lib\site-packages\numpy\core, but even when I set PYTHONPATH to > this it returns the same error. Imports of modules in the Lib directory work > fine, but everything I've tried within site-packages results the no module > error. > > numpy is not numeric, so just do "import numpy" to test the install. Just ignore all the dir structure under ...\numpy\, as a user you don't usually need to look at it. -sven From stefan at sun.ac.za Wed Jul 12 07:03:43 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Wed, 12 Jul 2006 13:03:43 +0200 Subject: [Numpy-discussion] "ImportError: No module named numeric" In-Reply-To: <200607121057.k6CAvrCc028781@mta1.iss.soton.ac.uk> References: <200607121057.k6CAvrCc028781@mta1.iss.soton.ac.uk> Message-ID: <20060712110343.GA30468@mentat.za.net> Hi Tamaryn On Wed, Jul 12, 2006 at 11:57:48AM +0100, Tamaryn Menneer wrote: > Hi > > I'm running Python 2.4 on Windows XP. I've installed NumPy, and run the > simple test of "import numeric" but I get the error "ImportError: No module > named numeric" in return. The module numeric is located in > C:\Python24\Lib\site-packages\numpy\core, but even when I set PYTHONPATH to > this it returns the same error. Imports of modules in the Lib directory work > fine, but everything I've tried within site-packages results the no module > error. Set your PYTHONPATH to c:\Python24\Lib\site-packages and then try from numpy import oldnumeric as numeric > Any guidance would be very much appreciated. I'm virtually a complete newbie > to all this, but I'll try my best to understand any responses! If you are just starting out with numpy, you shouldn't need to import the numeric module at all. Take a look at the documentation on http://www.scipy.org/Documentation Good luck! St?fan From theller at python.net Wed Jul 12 07:29:56 2006 From: theller at python.net (Thomas Heller) Date: Wed, 12 Jul 2006 13:29:56 +0200 Subject: [Numpy-discussion] Array interface and builtin array.array example In-Reply-To: References: <146309645.20060712013611@gmail.com> <44B4378D.7010701@ee.byu.edu> Message-ID: Sasha schrieb: > I had similar hopes when I submited the array interface patch > > and announced it on python-dev > . > > I am still waiting for anyone to comment on it :-( > The *relevant* comment is here: http://aspn.activestate.com/ASPN/Mail/Message/python-dev/3078372 Thomas From aisaac at american.edu Wed Jul 12 09:42:11 2006 From: aisaac at american.edu (Alan G Isaac) Date: Wed, 12 Jul 2006 09:42:11 -0400 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at><44B414DA.1050300@ee.byu.edu> Message-ID: Robert makes his case clearly and persuasively. Without pretending to challenge his argument in any way, I would just like to clarify what is at issue for some of the teaching crowd (or for me in any case). - Get up and running very quickly even with students who lack a programming background. This means having rand() and randn() in the top-level namespace is nice, since I use them early and often. - Avoid confusion and frustration. This is the basis for having a "consistent" calling convention for array constructors (pace Robert's arguments about consistency). On Tue, 11 Jul 2006, Robert Kern apparently wrote: > And mark my words, if we make rand() polymorphic, we will > get just as many newbies coming to the list asking why > ones(3, 4) doesn't work. That is plausible. If polymorphism is chosen for rand() and randn(), I suppose I would address this by documenting the current API as present for backwards compatability only. That allows a quick answer, but perhaps does not preclude the questions. Cheers, Alan Isaac From karol.langner at kn.pl Wed Jul 12 09:40:36 2006 From: karol.langner at kn.pl (Karol Langner) Date: Wed, 12 Jul 2006 15:40:36 +0200 Subject: [Numpy-discussion] Array interface and builtin array.array example In-Reply-To: References: <146309645.20060712013611@gmail.com> Message-ID: <200607121540.37465.karol.langner@kn.pl> On Wednesday 12 July 2006 13:29, Thomas Heller wrote: > Sasha schrieb: > > I had similar hopes when I submited the array interface patch > > >id=5470&atid=305470> and announced it on python-dev > > . > > > > I am still waiting for anyone to comment on it :-( > > The *relevant* comment is here: > > http://aspn.activestate.com/ASPN/Mail/Message/python-dev/3078372 > > Thomas A related PEP _is_ being prepared - it can be found at http://svn.scipy.org/svn/PEP/ (created by Travis). In fact, I am working on it for Google Summer of Code, as I wrote in a different thread. I think that if everyone interested in getting some kind of array interface into Python core focused on a common effort, it would be more effecient and probably make a stronger impression. I created a page about it at http://scipy.org/BaseArray, and will be adding more information very soon. Contribution and comments from annyone interested are very welcome. I really hope we can get this moving allong, and do it right. Karol -- written by Karol Langner ?ro lip 12 15:31:09 CEST 2006 From theller at python.net Wed Jul 12 10:00:59 2006 From: theller at python.net (Thomas Heller) Date: Wed, 12 Jul 2006 16:00:59 +0200 Subject: [Numpy-discussion] Array interface and builtin array.array example In-Reply-To: <200607121540.37465.karol.langner@kn.pl> References: <146309645.20060712013611@gmail.com> <200607121540.37465.karol.langner@kn.pl> Message-ID: Karol Langner schrieb: > On Wednesday 12 July 2006 13:29, Thomas Heller wrote: >> Sasha schrieb: >> > I had similar hopes when I submited the array interface patch >> > > >id=5470&atid=305470> and announced it on python-dev >> > . >> > >> > I am still waiting for anyone to comment on it :-( >> >> The *relevant* comment is here: >> >> http://aspn.activestate.com/ASPN/Mail/Message/python-dev/3078372 >> >> Thomas > > A related PEP _is_ being prepared - it can be found at > http://svn.scipy.org/svn/PEP/ (created by Travis). I know. But, hiding it somewhere on the internet doesn't lead anywhere. You (the authors) should follow the official PEP process, instead: submit an official PEP, post it for feedback, and so on. Thomas From karol.langner at kn.pl Wed Jul 12 10:15:23 2006 From: karol.langner at kn.pl (Karol Langner) Date: Wed, 12 Jul 2006 16:15:23 +0200 Subject: [Numpy-discussion] Array interface and builtin array.array example In-Reply-To: References: <146309645.20060712013611@gmail.com> <200607121540.37465.karol.langner@kn.pl> Message-ID: <200607121615.23441.karol.langner@kn.pl> On Wednesday 12 July 2006 16:00, Thomas Heller wrote: > Karol Langner schrieb: > > On Wednesday 12 July 2006 13:29, Thomas Heller wrote: > >> Sasha schrieb: > >> > I had similar hopes when I submited the array interface patch > >> > >> >up_ id=5470&atid=305470> and announced it on python-dev > >> > . > >> > > >> > I am still waiting for anyone to comment on it :-( > >> > >> The *relevant* comment is here: > >> > >> http://aspn.activestate.com/ASPN/Mail/Message/python-dev/3078372 > >> > >> Thomas > > > > A related PEP _is_ being prepared - it can be found at > > http://svn.scipy.org/svn/PEP/ (created by Travis). > > I know. But, hiding it somewhere on the internet doesn't lead anywhere. > You (the authors) should follow the official PEP process, instead: > submit an official PEP, post it for feedback, and so on. I haven't actually thought about this, but I guess that is will happen sooner or later. I started working on the code in the scipt svn repository, and I'm not to well oriented how the dev procedure works. Maybe Travis can tell us about long-term strategy for the PEP, if there already is some? There was a PEP back in 2001 concerning MD-arrays (http://www.python.org/dev/peps/pep-0209/), which is was even featured in the "PEP parade" (http://www.python.org/doc/essays/pepparade.html), but got deferred, abandoned, withdrawn, or something else. Karol -- written by Karol Langner ?ro lip 12 16:04:22 CEST 2006 From jnc at ecs.soton.ac.uk Wed Jul 12 12:37:41 2006 From: jnc at ecs.soton.ac.uk (John Carter) Date: Wed, 12 Jul 2006 17:37:41 +0100 Subject: [Numpy-discussion] Building SciPy under Windows attn. Bill Baxter Message-ID: <7.0.1.0.0.20060712172713.0b628218@ecs.soton.ac.uk> I had problems building SciPy and NumPy under Windows. They went away when I stopped using the stable version of gcc and used 3.4.5 I think the problem was related to differences in cygwin and mingw32. SciPy built and everything I've needed works but the self test fails. John Dr. John N. Carter jnc at ecs.soton.ac.uk ISIS http://www.ecs.soton.ac.uk/~jnc/ From mark at mitre.org Wed Jul 12 13:42:44 2006 From: mark at mitre.org (Mark Heslep) Date: Wed, 12 Jul 2006 13:42:44 -0400 Subject: [Numpy-discussion] Numpy array from ctypes pointer object? Message-ID: <44B53494.2000005@mitre.org> I don't see a clean way to create a numpy array from a ctypes pointer object. Is the __array_interface_ in ctypes the thing thats missing needed to make this happen? I've followed Albert's Scipy cookbook on ctypes here http://www.scipy.org/Cookbook/Ctypes <-- getting dated now for numpy SVN but the idea is clear This shows clean ways to 1) Create a ctype array object from a numpy array: npptr= nparray.__array_interface__['data'][0] ctarray = (c_double * nparray.size).from_address(npptr) 2) Create a numpy array from a ctypes array (not a pointer): ctarray = (c_double * 2)() nparray = N.array( ctarray ) But if I'm starting with say, a POINTER(c_int) and a separate size argument that's returned from a ctypes foreign function or provided by _fields_ element in a ctypes structure, then I'm unclear how to provide the pointer in Python to numpy. The numpy methods or creating an array in Python as I see it are basically two: N.array, N.asarray <-- require a sequence object N.fromstring, N.frombuffer <-- not available unless given c_char_p and even thats wrong if I don't want zero termination. The attributes in nparray.__array_interface_ are not writable, so no joy there. On the C side the PyArray_SimpleNewFromData( ..dimensions, ...data ptr) C API does the job nicely. Is there a ctypes paradigm for SimpleNew...? Mark From oliphant.travis at ieee.org Wed Jul 12 13:47:50 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 12 Jul 2006 11:47:50 -0600 Subject: [Numpy-discussion] Numpy array from ctypes pointer object? In-Reply-To: <44B53494.2000005@mitre.org> References: <44B53494.2000005@mitre.org> Message-ID: <44B535C6.1080307@ieee.org> Mark Heslep wrote: > I don't see a clean way to create a numpy array from a ctypes pointer object. Is the __array_interface_ in ctypes the thing thats missing needed to make this happen? I've followed Albert's Scipy cookbook on ctypes here > > On the C side the PyArray_SimpleNewFromData( ..dimensions, ...data ptr) C API does the job nicely. Is there a ctypes paradigm for SimpleNew...? > Can you somehow call this function using ctypes? -Travis From ndarray at mac.com Wed Jul 12 14:33:40 2006 From: ndarray at mac.com (Sasha) Date: Wed, 12 Jul 2006 14:33:40 -0400 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> <44B414DA.1050300@ee.byu.edu> Message-ID: Let me repeat my suggestion that was lost in this long thread: Add rands(shape, dtype=float, min=default_min(dtype), max=default_max(dtype)) to the top level. Suitable defaults can be discussed. A more flexible variation could be rands(shape, dtype=float, algorithm=default_algorithm(dtype)), but that would probably be an overkill. I think this will help teaching: rands is similar to zeros and ones, but with few bells and whistles to be covered in the graduate course. On 7/12/06, Alan G Isaac wrote: > Robert makes his case clearly and persuasively. > Without pretending to challenge his argument in any way, > I would just like to clarify what is at issue > for some of the teaching crowd (or for me in any case). > > - Get up and running very quickly even with students who > lack a programming background. This means having rand() > and randn() in the top-level namespace is nice, since > I use them early and often. > - Avoid confusion and frustration. This is the basis for > having a "consistent" calling convention for array > constructors (pace Robert's arguments about consistency). > > > On Tue, 11 Jul 2006, Robert Kern apparently wrote: > > And mark my words, if we make rand() polymorphic, we will > > get just as many newbies coming to the list asking why > > ones(3, 4) doesn't work. > > That is plausible. > If polymorphism is chosen for rand() and randn(), I suppose > I would address this by documenting the current API as > present for backwards compatability only. That allows > a quick answer, but perhaps does not preclude the questions. > > Cheers, > Alan Isaac > > > > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From mark at mitre.org Wed Jul 12 15:06:14 2006 From: mark at mitre.org (Mark Heslep) Date: Wed, 12 Jul 2006 15:06:14 -0400 Subject: [Numpy-discussion] Numpy array from ctypes pointer object? In-Reply-To: <44B535C6.1080307@ieee.org> References: <44B53494.2000005@mitre.org> <44B535C6.1080307@ieee.org> Message-ID: <44B54826.2010002@mitre.org> Travis Oliphant wrote: > Mark Heslep wrote: >> I don't see a clean way to create a numpy array from a ctypes pointer >> object. Is the __array_interface_ in ctypes the thing thats missing >> needed to make this happen? I've followed Albert's Scipy cookbook >> on ctypes here >> >> On the C side the PyArray_SimpleNewFromData( ..dimensions, ...data >> ptr) C API does the job nicely. Is there a ctypes paradigm for >> SimpleNew...? >> > Can you somehow call this function using ctypes? > > -Travis That might work, though indirectly. As I think I understand from ctypes docs: Ctypes uses functions exposed in a shared library, macros existing only a header are not available. If its PyArray... is a macro then I a) need to compile and make a little library directly from arrayobject.h or b) need to use the root function upon which the macro is based, PyArrayNew? ctypes has built-in access to the functions in the core python library, but Im not sure which library contains all the PyArray API calls, since its all typically hidden by distutils and setup.py Mark From theller at python.net Wed Jul 12 15:15:13 2006 From: theller at python.net (Thomas Heller) Date: Wed, 12 Jul 2006 21:15:13 +0200 Subject: [Numpy-discussion] Numpy array from ctypes pointer object? In-Reply-To: <44B54826.2010002@mitre.org> References: <44B53494.2000005@mitre.org> <44B535C6.1080307@ieee.org> <44B54826.2010002@mitre.org> Message-ID: Mark Heslep schrieb: > Travis Oliphant wrote: >> Mark Heslep wrote: >>> I don't see a clean way to create a numpy array from a ctypes pointer >>> object. Is the __array_interface_ in ctypes the thing thats missing >>> needed to make this happen? I've followed Albert's Scipy cookbook >>> on ctypes here >>> >>> On the C side the PyArray_SimpleNewFromData( ..dimensions, ...data >>> ptr) C API does the job nicely. Is there a ctypes paradigm for >>> SimpleNew...? >>> >> Can you somehow call this function using ctypes? >> >> -Travis > That might work, though indirectly. As I think I understand from ctypes > docs: Ctypes uses functions exposed in a shared library, macros > existing only a header are not available. If its PyArray... is a macro > then I a) need to compile and make a little library directly from > arrayobject.h or b) need to use the root function upon which the macro > is based, PyArrayNew? Sure, macros will not work. > ctypes has built-in access to the functions in the core python library, > but Im not sure which library contains all the PyArray API calls, since > its all typically hidden by distutils and setup.py On Linux it doesn't matter, since you don't need to know which library exports a function (If I understand shared libs in Linux correctly). At least dlsym can be called with a NULL handle. On windows, functions can only be retrieved from exactly the library that exposes them. Plus, the function must explicitly marked exported either by compiler directives in the source code of by listing them in a .def file. Thomas From oliphant.travis at ieee.org Wed Jul 12 15:23:16 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 12 Jul 2006 13:23:16 -0600 Subject: [Numpy-discussion] Numpy array from ctypes pointer object? In-Reply-To: <44B53494.2000005@mitre.org> References: <44B53494.2000005@mitre.org> Message-ID: <44B54C24.2070301@ieee.org> Mark Heslep wrote: > I don't see a clean way to create a numpy array from a ctypes pointer object. Is the __array_interface_ in ctypes the thing thats missing needed to make this happen? I've followed Albert's Scipy cookbook on ctypes here > > http://www.scipy.org/Cookbook/Ctypes <-- getting dated now for numpy SVN but the idea is clear > > This shows clean ways to > 1) Create a ctype array object from a numpy array: > npptr= nparray.__array_interface__['data'][0] > ctarray = (c_double * nparray.size).from_address(npptr) > > 2) Create a numpy array from a ctypes array (not a pointer): > ctarray = (c_double * 2)() > nparray = N.array( ctarray ) > > But if I'm starting with say, a POINTER(c_int) and a separate size argument that's returned from a ctypes foreign function or provided by _fields_ element in a ctypes structure, then I'm unclear how to provide the pointer in Python to numpy. The numpy methods or creating an array in Python as I see it are basically two: > > N.array, N.asarray <-- require a sequence object > N.fromstring, N.frombuffer <-- not available unless given c_char_p and even thats wrong if I don't want zero termination. > The problem here is that from Python NumPy has no way to create an ndarray from a pointer. Doing this creates a situtation where it is unclear who owns the memory. It is probably best to wrap the pointer into some kind of object exposing the buffer protocol and then pass that to frombuffer (or ndarray.__new__). When an ndarray is using memory that is not its own, it expects another object to be "in charge" of that memory and the ndarray will point its base attribute to it and increment its reference count. What should the object that is "in charge" of the memory be? Perhaps a suitable utility function could be created that can work with ctypes to create ndarrays from ctypes memory locations and either own or disown the data. This needs to be thought through a bit, however. -Travis > The attributes in nparray.__array_interface_ are not writable, so no joy there. > > On the C side the PyArray_SimpleNewFromData( ..dimensions, ...data ptr) C API does the job nicely. Is there a ctypes paradigm for SimpleNew...? > > Mark > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From oliphant.travis at ieee.org Wed Jul 12 15:25:59 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 12 Jul 2006 13:25:59 -0600 Subject: [Numpy-discussion] Numpy array from ctypes pointer object? In-Reply-To: <44B54826.2010002@mitre.org> References: <44B53494.2000005@mitre.org> <44B535C6.1080307@ieee.org> <44B54826.2010002@mitre.org> Message-ID: <44B54CC7.4040509@ieee.org> Mark Heslep wrote: > Travis Oliphant wrote: > >> Mark Heslep wrote: >> >>> I don't see a clean way to create a numpy array from a ctypes pointer >>> object. Is the __array_interface_ in ctypes the thing thats missing >>> needed to make this happen? I've followed Albert's Scipy cookbook >>> on ctypes here >>> >>> On the C side the PyArray_SimpleNewFromData( ..dimensions, ...data >>> ptr) C API does the job nicely. Is there a ctypes paradigm for >>> SimpleNew...? >>> >>> >> Can you somehow call this function using ctypes? >> >> -Travis >> > That might work, though indirectly. As I think I understand from ctypes > docs: Ctypes uses functions exposed in a shared library, macros > existing only a header are not available. If its PyArray... is a macro > then I a) need to compile and make a little library directly from > arrayobject.h or b) need to use the root function upon which the macro > is based, PyArrayNew? > > This is more complicated because all the C-API functions are actually just pointers stored in _ARRAY_API of multiarray. So, something would have to be built to interpret the C-pointers in that C-Object. I'm not sure that is possible. -Travis From mark at mitre.org Wed Jul 12 16:03:32 2006 From: mark at mitre.org (Mark Heslep) Date: Wed, 12 Jul 2006 16:03:32 -0400 Subject: [Numpy-discussion] Numpy array from ctypes pointer object? In-Reply-To: <44B54C24.2070301@ieee.org> References: <44B53494.2000005@mitre.org> <44B54C24.2070301@ieee.org> Message-ID: <44B55594.6090901@mitre.org> Travis Oliphant wrote: > The problem here is that from Python NumPy has no way to create an > ndarray from a pointer. Doing this creates a situtation where it is > unclear who owns the memory. It is probably best to wrap the pointer > into some kind of object exposing the buffer protocol and then pass > that to frombuffer (or ndarray.__new__). Yep thats where I just ended up: from ctypes import * import numpy as N ... func = pythonapi.PyBuffer_FromMemory func.restype = py_object buffer = func( im.imageData, size_of_the_data ) <----imageData = ctypes.LP_c_ubyte object return N.frombuffer( buffer, N.uint8 ) Works! Im curious though: the several projects recently using ctypes and numpy to wrap libraries (Pygame SDL, OpenGL, svm) must have come across the issue of using a creating a numpy array from a ctypes pointer. Ill have to look further. > When an ndarray is using memory that is not its own, it expects > another object to be "in charge" of that memory and the ndarray will > point its base attribute to it and increment its reference count. > What should the object that is "in charge" of the memory be? > Perhaps a suitable utility function could be created that can work > with ctypes to create ndarrays from ctypes memory locations and either > own or disown the data. > I suppose that is still the case w/ PyBuffer_From.. above. That is, the underlying im.imageData pointer can not be released before buffer. Mark > This needs to be thought through a bit, however. > > -Travis > > > >> The attributes in nparray.__array_interface_ are not writable, so no >> joy there. >> >> On the C side the PyArray_SimpleNewFromData( ..dimensions, ...data >> ptr) C API does the job nicely. Is there a ctypes paradigm for >> SimpleNew...? >> >> Mark >> >> >> >> ------------------------------------------------------------------------- >> >> Using Tomcat but need to do more? Need to support web services, >> security? >> Get stuff done quickly with pre-integrated technology to make your >> job easier >> Download IBM WebSphere Application Server v.1.0.1 based on Apache >> Geronimo >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> > From oliphant.travis at ieee.org Wed Jul 12 16:11:58 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 12 Jul 2006 14:11:58 -0600 Subject: [Numpy-discussion] Numpy array from ctypes pointer object? In-Reply-To: <44B55594.6090901@mitre.org> References: <44B53494.2000005@mitre.org> <44B54C24.2070301@ieee.org> <44B55594.6090901@mitre.org> Message-ID: <44B5578E.3010007@ieee.org> Mark Heslep wrote: > Travis Oliphant wrote: > >> The problem here is that from Python NumPy has no way to create an >> ndarray from a pointer. Doing this creates a situtation where it is >> unclear who owns the memory. It is probably best to wrap the pointer >> into some kind of object exposing the buffer protocol and then pass >> that to frombuffer (or ndarray.__new__). >> > Yep thats where I just ended up: > > from ctypes import * > import numpy as N > ... > func = pythonapi.PyBuffer_FromMemory > func.restype = py_object > buffer = func( im.imageData, size_of_the_data ) > <----imageData = ctypes.LP_c_ubyte object > return N.frombuffer( buffer, N.uint8 ) > > Works! Nice job! > Im curious though: the several projects recently using ctypes > and numpy to wrap libraries (Pygame SDL, OpenGL, svm) must have come > across the issue of using a creating a numpy array from a ctypes > pointer. Ill have to look further. > > It depends on whether or not the library creates memory or not. Not every library manages memory (some expect you to pass in memory already owned --- this is easy to do already with ctypes and numpy). >> When an ndarray is using memory that is not its own, it expects >> another object to be "in charge" of that memory and the ndarray will >> point its base attribute to it and increment its reference count. >> What should the object that is "in charge" of the memory be? >> Perhaps a suitable utility function could be created that can work >> with ctypes to create ndarrays from ctypes memory locations and either >> own or disown the data. >> >> > I suppose that is still the case w/ PyBuffer_From.. above. That is, the > underlying im.imageData pointer can not be released before buffer. > Yes, you are right. It is the memory that is most critical. Who owns the memory pointed to by im.imageData? When will it be freed? The ndarray object is holding a reference to the Python buffer object which is just *hoping* that the memory it was initiated with is not going to be freed before it gets deallocated (which won't happen until at least the ndarray object is deallocated). So, managing the memory can be a bit tricky. But, if you are sure that im.imageData memory will not be freed then you are O.K. -Travis From fullung at gmail.com Wed Jul 12 16:23:32 2006 From: fullung at gmail.com (Albert Strasheim) Date: Wed, 12 Jul 2006 22:23:32 +0200 Subject: [Numpy-discussion] Numpy array from ctypes pointer object? In-Reply-To: <44B5578E.3010007@ieee.org> Message-ID: <00e201c6a5f1$0b939300$0100000a@dsp.sun.ac.za> Hello all Various people wrote: > > Im curious though: the several projects recently using ctypes > > and numpy to wrap libraries (Pygame SDL, OpenGL, svm) must have come > > across the issue of using a creating a numpy array from a ctypes > > pointer. Ill have to look further. > > > It depends on whether or not the library creates memory or not. Not > every library manages memory (some expect you to pass in memory already > owned --- this is easy to do already with ctypes and numpy). I see two main models for sending back data from C land. One is to have the C function allocate memory and return the pointer to the caller. This raises the question of who owns the memory. It also leads to some *very* interesting crashes on Windows when you start freeing memory allocated in code linked against one runtime in code linked against another runtime. I think, if you have the option, avoid this kind of thing at all costs. The other model has functions that take their normal arguments, and pointers to buffers where they can store their results. Typically the caller would also specify the size of the buffer. If the buffer is large enough for the function to do its work, it uses it. If not, it returns an error code. This way, one entity is always in charge of the memory. If you're writing C code that you plan to use with ctypes and NumPy, I think the second model will lead to more robust code and less time spent debugging crashes and memory leaks. libsvm (which I'm wrapping with ctypes) mostly follows the second model, except when training new models, in which case it returns a pointer to a newly allocated structure. To deal with this, I keep a pointer to this allocated memory in a Python object that has the following function: def __del__(self): libsvm.svm_destroy_model(self.model) By providing this destroy function, libsvm avoids the problem of mixing allocation and deallocation across runtimes. Regards, Albert From prabhu at aero.iitb.ac.in Wed Jul 12 17:03:32 2006 From: prabhu at aero.iitb.ac.in (Prabhu Ramachandran) Date: Thu, 13 Jul 2006 02:33:32 +0530 Subject: [Numpy-discussion] [SciPy-dev] Weave, numpy, external libraries and conflicting typedefs In-Reply-To: <44AEC66A.2070507@ee.byu.edu> References: <17582.7994.142818.120995@prpc.aero.iitb.ac.in> <44AE9E12.5000703@ieee.org> <17582.46882.286683.846598@prpc.aero.iitb.ac.in> <44AEBE1F.8010304@ee.byu.edu> <44AEC66A.2070507@ee.byu.edu> Message-ID: <17589.25508.570735.848513@prpc.aero.iitb.ac.in> I sent this reply on 9th but the message seems to have never made it to scipy-dev and is still pending moderator approval on numpy-discussion. I had attached a patch for weave that made the email larger than 40 KB. I don't have checkin privileges to scipy/numpy. So if someone would be kind enough to apply the patch for me, let me know and I'll send you the patch off-list. >>>>> "Travis" == Travis Oliphant writes: [...] Travis> I'm not opposed to putting a *short* prefix in front of Travis> everything (the Int32, Float64, stuff came from numarray Travis> which now has it's own back-ward compatible header where Travis> it could be placed now anyway). Perhaps npy_ would be a Travis> suitable prefix. Travis> That way we could get rid of the cruft entirely. Travis> I suppose we could also provide the noprefix.h header that Travis> defines the old un-prefixed cases for "backwards NumPy Travis> compatibility". Travis, you rock! Thanks for fixing this in SVN. All the problems I was having earlier are gone. Here is a patch for weave's SWIG support that should be applied to scipy. The SWIG runtime layout changed in version 1.3.28 and this broke weave support. This should be fixed soon (hopefully!) in SWIG CVS and the following patch will ensure that weave works against it. Many thanks once again for the speedy fixes! cheers, prabhu From aisaac at american.edu Wed Jul 12 17:33:48 2006 From: aisaac at american.edu (Alan G Isaac) Date: Wed, 12 Jul 2006 17:33:48 -0400 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at><44B414DA.1050300@ee.byu.edu> Message-ID: On Wed, 12 Jul 2006, Sasha apparently wrote: > Let me repeat my suggestion that was lost in this long thread: > Add rands(shape, dtype=float, min=default_min(dtype), max=default_max(dtype)) > to the top level. Suitable defaults can be discussed. A more flexible > variation could > be rands(shape, dtype=float, algorithm=default_algorithm(dtype)), but > that would probably be an overkill. My only reason for not exploring this is that recent decisions seem to preclude it. Specifically, nothing from the random module remains in the numpy namespace, I think. Cheers, Alan Isaac From ndarray at mac.com Wed Jul 12 17:37:34 2006 From: ndarray at mac.com (Sasha) Date: Wed, 12 Jul 2006 17:37:34 -0400 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> <44B414DA.1050300@ee.byu.edu> Message-ID: On 7/12/06, Alan G Isaac wrote: > On Wed, 12 Jul 2006, Sasha apparently wrote: [snip] > > Add rands(shape, dtype=float, min=default_min(dtype), max=default_max(dtype)) > > to the top level. Suitable defaults can be discussed. A more flexible > > variation could > > be rands(shape, dtype=float, algorithm=default_algorithm(dtype)), but > > that would probably be an overkill. > > My only reason for not exploring this is that recent > decisions seem to preclude it. Specifically, nothing > from the random module remains in the numpy namespace, To the contrary, the recent changes cleared the way for a better random number generator in the numpy namespace. With my proposal, I would predict that rands(n) and rands((n,m)) will be used a lot in tests and examples while more sophisticated functionality will be easily discoverable via help(rands). I can also see some benefit in having rands(n, bool) that is not available at the moment. From laidler at stsci.edu Wed Jul 12 17:42:08 2006 From: laidler at stsci.edu (Victoria G. Laidler) Date: Wed, 12 Jul 2006 17:42:08 -0400 Subject: [Numpy-discussion] how to use argsort result? In-Reply-To: <44B37608.2000003@itc.it> References: <44B37040.2040905@itc.it> <6ef8f3380607110240i24f6f219w2fa112a113f04291@mail.gmail.com> <44B37608.2000003@itc.it> Message-ID: <44B56CB0.2080507@stsci.edu> Hi, Pardon me if I'm reprising an earlier discussion, as I'm new to the list. But is there a reason that this obscure syntax A[arange(2)[:,newaxis],indexes] A[arange(A.shape[0])[:,newaxis],indexes] is preferable to the intuitively reasonable thing that the Original Poster did? A[indexes] Doesn't it violate the pythonic "principle of least surprise" for the simpler syntax not to work? As a casual numpy user, I can't imagine being able to remember the obscure syntax in order to use the result of an argsort. curiously, Vicki Laidler (numarray user who came from IDL) Emanuele Olivetti wrote: >Wow. I have to study much more indexing. It works pretty well. > >Just to help indexing newbie like on using your advice: >A[arange(A.shape[0])[:,newaxis],indexes] > >Thanks a lot! > >Emanuele > >Pau Gargallo wrote: > > >>here goes a first try: >> >>A[arange(2)[:,newaxis],indexes] >> >> > > >------------------------------------------------------------------------- >Using Tomcat but need to do more? Need to support web services, security? >Get stuff done quickly with pre-integrated technology to make your job easier >Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo >http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From mark at mitre.org Wed Jul 12 17:46:39 2006 From: mark at mitre.org (Mark Heslep) Date: Wed, 12 Jul 2006 17:46:39 -0400 Subject: [Numpy-discussion] Numpy array from ctypes pointer object? In-Reply-To: <00e201c6a5f1$0b939300$0100000a@dsp.sun.ac.za> References: <00e201c6a5f1$0b939300$0100000a@dsp.sun.ac.za> Message-ID: <44B56DBF.9060906@mitre.org> Albert Strasheim wrote: > Hello all > > Various people wrote: > >>> Im curious though: the several projects recently using ctypes >>> and numpy to wrap libraries (Pygame SDL, OpenGL, svm) must have come >>> across the issue of using a creating a numpy array from a ctypes >>> pointer. Ill have to look further. >>> >>> >> It depends on whether or not the library creates memory or not. Not >> every library manages memory (some expect you to pass in memory already >> owned --- this is easy to do already with ctypes and numpy). >> > > I see two main models for sending back data from C land. > > One is to have the C function allocate memory and return the pointer to the > caller. This raises the question of who owns the memory. It also leads to > some *very* interesting crashes on Windows when you start freeing memory > allocated in code linked against one runtime in code linked against another > runtime. I think, if you have the option, avoid this kind of thing at all > costs. > > The other model has functions that take their normal arguments, and pointers > to buffers where they can store their results. Typically the caller would > also specify the size of the buffer. If the buffer is large enough for the > function to do its work, it uses it. If not, it returns an error code. This > way, one entity is always in charge of the memory. > > If you're writing C code that you plan to use with ctypes and NumPy, I think > the second model will lead to more robust code and less time spent debugging > crashes and memory leaks. > Yes, I agree, C libs written with your model #2 would make life much easier to create robust wrappers. Many of the ones that come to my mind, and deal with array like data and thus relevant, are model #1 types. I think the reason this is so goes something like this: Joe Foo developer & friends writes an entire library API in C. To make the API complete and easy to use Joe includes getting started 'make the data' C functions. Examples: SDL: SDL_surface* SDL_CreateRGBSurface(params) Opencv: IplImage* cvCreateImage(params), ... libdc1394: uint_t * capture buffer //created by the OS 1394 driver etc. If the Joe doesn't do this then the first thing Joe's 'users' must do to create the simplest application is call malloc, sizeof( figure out what element type...), blah, blah, i.e. do lots of memory management having zilch to do with the problem at hand. Further, Joe, being conscientious, includes boat loads of examples all using the 'make the data' calls. > libsvm (which I'm wrapping with ctypes) mostly follows the second model, > except when training new models, in which case it returns a pointer to a > newly allocated structure. To deal with this, I keep a pointer to this > allocated memory in a Python object that has the following function: > > def __del__(self): > libsvm.svm_destroy_model(self.model) > > Nice > By providing this destroy function, libsvm avoids the problem of mixing > allocation and deallocation across runtimes. > > Regards, > > Albert > > -Mark From oliphant.travis at ieee.org Wed Jul 12 18:42:55 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 12 Jul 2006 16:42:55 -0600 Subject: [Numpy-discussion] Patch against Image.py in the PIL Message-ID: <44B57AEF.3080300@ieee.org> Attached is a patch that makes PIL Image objects both export and consume the array interface. -Travis -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: pil_patch_against_358.txt URL: From robert.kern at gmail.com Wed Jul 12 19:16:47 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 12 Jul 2006 18:16:47 -0500 Subject: [Numpy-discussion] Args for rand and randn: call for a vote In-Reply-To: References: <48CC8C82-BCBC-47B4-BFB2-786176F935A2@ftw.at> <44B414DA.1050300@ee.byu.edu> Message-ID: Sasha wrote: > Let me repeat my suggestion that was lost in this long thread: > > Add rands(shape, dtype=float, min=default_min(dtype), max=default_max(dtype)) > to the top level. Suitable defaults can be discussed. A more flexible > variation could > be rands(shape, dtype=float, algorithm=default_algorithm(dtype)), but > that would probably be an overkill. > > I think this will help teaching: rands is similar to zeros and ones, > but with few bells and whistles to be covered in the graduate course. Well, write it up, stick it as a method in RandomState, and when we can play with it, we can discuss whether it should go into numpy.*. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From wbaxter at gmail.com Wed Jul 12 22:29:46 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu, 13 Jul 2006 11:29:46 +0900 Subject: [Numpy-discussion] numpy distutils problem on Windows Message-ID: In numpy/distutils/command/config.py: 'cookedm' added a get_output() command on June 9. This get_output function uses os.WEXITSTATUS and various other os.W*functions. These do not exist in Python on Windows. Is there some other way to achieve the same thing without those? For now, just commenting out those error checking lines seems to do the trick. --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From cookedm at physics.mcmaster.ca Wed Jul 12 22:48:35 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed, 12 Jul 2006 22:48:35 -0400 Subject: [Numpy-discussion] numpy distutils problem on Windows In-Reply-To: References: Message-ID: <20060712224835.30c2ebda@arbutus.physics.mcmaster.ca> On Thu, 13 Jul 2006 11:29:46 +0900 "Bill Baxter" wrote: > In numpy/distutils/command/config.py: > 'cookedm' added a get_output() command on June 9. > This get_output function uses os.WEXITSTATUS and various other > os.W*functions. > > These do not exist in Python on Windows. > > Is there some other way to achieve the same thing without those? > > For now, just commenting out those error checking lines seems to do the > trick. Shoot, you're right. I didn't see that in the docs for the os module. I've fixed it in svn (I think -- I don't have a Windows box to test on). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From wbaxter at gmail.com Wed Jul 12 23:12:36 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu, 13 Jul 2006 12:12:36 +0900 Subject: [Numpy-discussion] Building SciPy under Windows attn. Bill Baxter In-Reply-To: <7.0.1.0.0.20060712172713.0b628218@ecs.soton.ac.uk> References: <7.0.1.0.0.20060712172713.0b628218@ecs.soton.ac.uk> Message-ID: Thanks, that seems to have done the trick! I've got a SVN Scipy now! I updated the building scipy wiki page with this piece of advice. --bb On 7/13/06, John Carter wrote: > > I had problems building SciPy and NumPy under Windows. They went away > when I stopped using the stable version of gcc and used 3.4.5 > > I think the problem was related to differences in cygwin and mingw32. > > SciPy built and everything I've needed works but the self test fails. > > John > > Dr. John N. Carter jnc at ecs.soton.ac.uk > ISIS http://www.ecs.soton.ac.uk/~jnc/ > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- William V. Baxter III OLM Digital Kono Dens Building Rm 302 1-8-8 Wakabayashi Setagaya-ku Tokyo, Japan 154-0023 +81 (3) 3422-3380 -------------- next part -------------- An HTML attachment was scrubbed... URL: From haase at msg.ucsf.edu Thu Jul 13 00:14:47 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 12 Jul 2006 21:14:47 -0700 Subject: [Numpy-discussion] prebuild windows - was: Re: Building SciPy under Windows attn. Bill Baxter In-Reply-To: References: <7.0.1.0.0.20060712172713.0b628218@ecs.soton.ac.uk> Message-ID: <44B5C8B7.70407@msg.ucsf.edu> Hi, The latest cygwin gcc seems to be version 3.4.4. Are (relatively) new SVN builds (both numpy and scipy) available somewhere for download ? (I would like to experience the new "arr.T" feature ;-) ) Thanks Sebastian Haase Bill Baxter wrote: > Thanks, that seems to have done the trick! > I've got a SVN Scipy now! > > I updated the building scipy wiki page with this piece of advice. > > --bb > > On 7/13/06, * John Carter* > wrote: > > I had problems building SciPy and NumPy under Windows. They went away > when I stopped using the stable version of gcc and used 3.4.5 > > I think the problem was related to differences in cygwin and mingw32. > > SciPy built and everything I've needed works but the self test fails. > > John > > Dr. John N. Carter jnc at ecs.soton.ac.uk > ISIS http://www.ecs.soton.ac.uk/~jnc/ > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, > security? > Get stuff done quickly with pre-integrated technology to make your > job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > > -- > William V. Baxter III > OLM Digital > Kono Dens Building Rm 302 > 1-8-8 Wakabayashi Setagaya-ku > Tokyo, Japan 154-0023 > +81 (3) 3422-3380 > > > ------------------------------------------------------------------------ > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From wbaxter at gmail.com Thu Jul 13 01:24:12 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu, 13 Jul 2006 14:24:12 +0900 Subject: [Numpy-discussion] determining if you have ATLAS or not Message-ID: How can you probe numpy for info about what sort of BLAS/LAPACK you have, or other build configuration info? Searching the ml archives I turned up this one hint from Travis, which can be embodied in code thusly: import numpy def have_blas(): return id(numpy.dot) != id(numpy.core.multiarray.dot) A) is that function correct? and B) Is that the most you can find out? --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Thu Jul 13 02:16:30 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu, 13 Jul 2006 15:16:30 +0900 Subject: [Numpy-discussion] determining if you have ATLAS or not In-Reply-To: References: Message-ID: Terrific. Nils sent me the answer: scipy.show_config() or > numpy.show_config() > will give you some useful information. > > And here it goes straight to the wiki http://www.scipy.org/Installing_SciPy/Windows --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebzur at pin.if.uz.zgora.pl Wed Jul 12 19:07:31 2006 From: sebzur at pin.if.uz.zgora.pl (=?ISO-8859-2?Q?Sebastian_=AFurek?=) Date: Thu, 13 Jul 2006 00:07:31 +0100 Subject: [Numpy-discussion] RandomArray module Message-ID: Hi All, Has anyone worked with the RandomArray module? I wonder, if it's OK to use its pseudo-random numbers generators, or maybe I shall find more trusted methods (ie. ran1 from Numerical Recipes)? Please, give some comments. Thanks. Sebastian From faltet at carabos.com Thu Jul 13 03:40:27 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu, 13 Jul 2006 09:40:27 +0200 Subject: [Numpy-discussion] RandomArray module In-Reply-To: References: Message-ID: <200607130940.28154.faltet@carabos.com> A Dijous 13 Juliol 2006 01:07, Sebastian ?urek va escriure: > Hi All, > > Has anyone worked with the RandomArray module? I wonder, > if it's OK to use its pseudo-random numbers generators, or > maybe I shall find more trusted methods (ie. ran1 from Numerical Recipes)? I'm not an expert, but my understanding is that the 'random' module that comes with NumPy is made with kind of state-of-the-art random generators, so it should be fine for most of purposes. However, the experts, and in particular Robert Kern (I think he is the implementor), may want to precise this point. > > Please, give some comments. Thanks. Done ;-) -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From pau.gargallo at gmail.com Thu Jul 13 04:35:33 2006 From: pau.gargallo at gmail.com (Pau Gargallo) Date: Thu, 13 Jul 2006 10:35:33 +0200 Subject: [Numpy-discussion] how to use argsort result? In-Reply-To: <44B56CB0.2080507@stsci.edu> References: <44B37040.2040905@itc.it> <6ef8f3380607110240i24f6f219w2fa112a113f04291@mail.gmail.com> <44B37608.2000003@itc.it> <44B56CB0.2080507@stsci.edu> Message-ID: <6ef8f3380607130135x1fe1bcau606a5fa46a49497d@mail.gmail.com> On 7/12/06, Victoria G. Laidler wrote: > Hi, > > Pardon me if I'm reprising an earlier discussion, as I'm new to the list. > > But is there a reason that this obscure syntax > > A[arange(2)[:,newaxis],indexes] > > A[arange(A.shape[0])[:,newaxis],indexes] > > is preferable to the intuitively reasonable thing that the Original > Poster did? > > A[indexes] > i don't think so. The obscure syntax is just a way you can solve the problem with the current state of NumPy. Of course, a more clearer syntax would be better, but for this, something in NumPy should be changed. This other syntax is longer but clearer: ind = indices(A.shape) ind[ax] = A.argsort(axis=ax) A[ind] Which brings me to the question: Would it be reasonable if argsort returned the complete tuple of indices, so that A[A.argsort(ax)] would work ? pau From oliphant.travis at ieee.org Thu Jul 13 05:07:25 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 13 Jul 2006 03:07:25 -0600 Subject: [Numpy-discussion] how to use argsort result? In-Reply-To: <6ef8f3380607130135x1fe1bcau606a5fa46a49497d@mail.gmail.com> References: <44B37040.2040905@itc.it> <6ef8f3380607110240i24f6f219w2fa112a113f04291@mail.gmail.com> <44B37608.2000003@itc.it> <44B56CB0.2080507@stsci.edu> <6ef8f3380607130135x1fe1bcau606a5fa46a49497d@mail.gmail.com> Message-ID: <44B60D4D.2080509@ieee.org> Pau Gargallo wrote: > On 7/12/06, Victoria G. Laidler wrote: > >> Hi, >> >> Pardon me if I'm reprising an earlier discussion, as I'm new to the list. >> >> But is there a reason that this obscure syntax >> >> A[arange(2)[:,newaxis],indexes] >> >> A[arange(A.shape[0])[:,newaxis],indexes] >> >> is preferable to the intuitively reasonable thing that the Original >> Poster did? >> >> A[indexes] >> >> > > i don't think so. > The obscure syntax is just a way you can solve the problem with the > current state of NumPy. Of course, a more clearer syntax would be > better, but for this, something in NumPy should be changed. > > This other syntax is longer but clearer: > ind = indices(A.shape) > ind[ax] = A.argsort(axis=ax) > A[ind] > > Which brings me to the question: > > Would it be reasonable if argsort returned the complete tuple of > indices, so that > A[A.argsort(ax)] would work ? > > I think this is reasonable. We would need a way for the argsort() function to work as it does now. I'm not sure if anybody actually uses the multidimensional behavior of argsort now, but it's been in Numeric for a long time. -Travis From pau.gargallo at gmail.com Thu Jul 13 05:52:31 2006 From: pau.gargallo at gmail.com (Pau Gargallo) Date: Thu, 13 Jul 2006 11:52:31 +0200 Subject: [Numpy-discussion] int 'up'casted to uint In-Reply-To: <6ef8f3380607110924p66b8de0ds55971dfb8c9ff892@mail.gmail.com> References: <6ef8f3380607110354u6db272b0t125a9f13a92d1180@mail.gmail.com> <44B3CBDA.8080605@ieee.org> <6ef8f3380607110924p66b8de0ds55971dfb8c9ff892@mail.gmail.com> Message-ID: <6ef8f3380607130252m5afcc214kcd68f1b9bb5ba7ea@mail.gmail.com> On 7/11/06, Pau Gargallo wrote: > On 7/11/06, Travis Oliphant wrote: > > Pau Gargallo wrote: > > > hi, > > > > > > looking at the upcasting table at > > > http://www.scipy.org/Tentative_NumPy_Tutorial#head-4c1d53fe504adc97baf27b65513b4b97586a4fc5 > > > I saw that int's are sometimes casted to uint's. > > > > > > In [3]: a = array([3],int16) > > > In [5]: b = array([4],uint32) > > > In [7]: a+b > > > Out[7]: array([7], dtype=uint32) > > > > > > is that intended? > > > > > It's a bug. The result should be int64. I've fixed it in SVN. > > > > Thanks!! > hi Travis, now uint64+int gives float64. Obtaining a float from ints surprise me. But anyway, I don't know if there is a better choice. pau From adam at epcc.ed.ac.uk Thu Jul 13 06:02:07 2006 From: adam at epcc.ed.ac.uk (Adam Carter) Date: Thu, 13 Jul 2006 11:02:07 +0100 Subject: [Numpy-discussion] LAPACK Problem with "import numpy" on AIX Message-ID: <200607131002.k6DA272v009473@e450.epcc.ed.ac.uk> Hi all, I'm new to this list so apologies if this is a solved problem, but I haven't been able to find anything in the archives. I've just installed Python 2.4.3 and Numpy-0.9.8 on AIX, and the configure/make/install of Python and the install of Numpy _appeared_ to go smoothly. However, now when I run python and type >>> import numpy I get import linalg -> failed: No module named lapack_lite I'm not sure if this is just a warning, but I suspect not, as when I later try import scipy, it results in a traceback: >>> import scipy Traceback (most recent call last): File "", line 1, in ? File "/hpcx/home/z001/z001/adam/packages/Python-2.4.3/lib/python2.4/site-packages /scipy/__init__.py", line 34, in ? del linalg NameError: name 'linalg' is not defined Can anytone tell me what I'm doing wrong? I've not explicitly installed any LAPACK library but I was hpoing to use an existing (vendor provided) LAPACK. The environment variable $LAPACK was set to point to this library at compile and run-time. Do I need lapack_lite aswell? Where can I get it? If I need this other LAPACK, how can I ensure that my code uses the optimised version of LAPACK already on this system I'm using? Any advice very welcome. Thanks in advance, Adam ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Dr Adam Carter (Applications Consultant) |epcc| e: adam at epcc.ed.ac.uk w: http://www.epcc.ed.ac.uk/~adam r: 3405 JCMB t: +44 131 650 6009 From wbaxter at gmail.com Thu Jul 13 06:08:19 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu, 13 Jul 2006 19:08:19 +0900 Subject: [Numpy-discussion] LAPACK Problem with "import numpy" on AIX In-Reply-To: <200607131002.k6DA272v009473@e450.epcc.ed.ac.uk> References: <200607131002.k6DA272v009473@e450.epcc.ed.ac.uk> Message-ID: On 7/13/06, Adam Carter wrote: > > Hi all, > > Do I need lapack_lite aswell? Where can I get it? If I need this other > LAPACK, how can I ensure that my code uses the optimised version of LAPACK > already on this system I'm using? Lapack_lite is in the numpy sources. I think it's the fallback if you don't have a LAPACK. You can check the config that got built with: numpy.show_config() But that doesn't help much if you can't import numpy. --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam at epcc.ed.ac.uk Thu Jul 13 06:19:16 2006 From: adam at epcc.ed.ac.uk (Adam Carter) Date: Thu, 13 Jul 2006 11:19:16 +0100 Subject: [Numpy-discussion] LAPACK Problem with "import numpy" on AIX In-Reply-To: Message-ID: <200607131019.k6DAJG4s010054@e450.epcc.ed.ac.uk> > Lapack_lite is in the numpy sources. I think it's the fallback if you don't have a LAPACK. Thanks Bill, that's what I thought, but I don't understand why it can't find it. > You can check the config that got built with: > numpy.show_config() OK, so it appears that NumPy is imported then, albeit with the warning. The result of the above line is included below. So maybe the problem lies with SciPy? Perhaps I should take my problem to scipy-users. Unless anyone has any further insight? Cheers, Adam >>> numpy.show_config() blas_info: libraries = ['blas'] library_dirs = ['/usr/lib'] language = f77 lapack_info: libraries = ['lapack'] library_dirs = ['/usr/local/lib'] language = f77 atlas_threads_info: NOT AVAILABLE blas_opt_info: libraries = ['blas'] library_dirs = ['/usr/lib'] language = f77 define_macros = [('NO_ATLAS_INFO', 1)] atlas_blas_threads_info: NOT AVAILABLE lapack_opt_info: libraries = ['lapack', 'blas'] library_dirs = ['/usr/local/lib', '/usr/lib'] language = f77 define_macros = [('NO_ATLAS_INFO', 1)] atlas_info: NOT AVAILABLE lapack_mkl_info: NOT AVAILABLE blas_mkl_info: NOT AVAILABLE atlas_blas_info: NOT AVAILABLE mkl_info: NOT AVAILABLE From pau.gargallo at gmail.com Thu Jul 13 08:09:07 2006 From: pau.gargallo at gmail.com (Pau Gargallo) Date: Thu, 13 Jul 2006 14:09:07 +0200 Subject: [Numpy-discussion] how to use argsort result? In-Reply-To: <44B60D4D.2080509@ieee.org> References: <44B37040.2040905@itc.it> <6ef8f3380607110240i24f6f219w2fa112a113f04291@mail.gmail.com> <44B37608.2000003@itc.it> <44B56CB0.2080507@stsci.edu> <6ef8f3380607130135x1fe1bcau606a5fa46a49497d@mail.gmail.com> <44B60D4D.2080509@ieee.org> Message-ID: <6ef8f3380607130509s4f787ce8na1f7ac58b07f1dd4@mail.gmail.com> On 7/13/06, Travis Oliphant wrote: > Pau Gargallo wrote: > > On 7/12/06, Victoria G. Laidler wrote: > > > >> Hi, > >> > >> Pardon me if I'm reprising an earlier discussion, as I'm new to the list. > >> > >> But is there a reason that this obscure syntax > >> > >> A[arange(2)[:,newaxis],indexes] > >> > >> A[arange(A.shape[0])[:,newaxis],indexes] > >> > >> is preferable to the intuitively reasonable thing that the Original > >> Poster did? > >> > >> A[indexes] > >> > >> > > > > i don't think so. > > The obscure syntax is just a way you can solve the problem with the > > current state of NumPy. Of course, a more clearer syntax would be > > better, but for this, something in NumPy should be changed. > > > > This other syntax is longer but clearer: > > ind = indices(A.shape) > > ind[ax] = A.argsort(axis=ax) > > A[ind] > > > > Which brings me to the question: > > > > Would it be reasonable if argsort returned the complete tuple of > > indices, so that > > A[A.argsort(ax)] would work ? > > > > > I think this is reasonable. We would need a way for the argsort() > function to work as it does now. I'm not sure if anybody actually uses > the multidimensional behavior of argsort now, but it's been in Numeric > for a long time. > actually I use the multidimensional behavior of argmin and argmax in its current state, and found it useful as it is, even if A[A.argmax(ax)] doesn't work. May be a keyword could be added, so that A.argxxx( axis=ax, indices=True ) returns the tuple of indices. (The keyword name and default should be discussed) I don't know if that's *the* way, but ... pau From charges at humortadela.com.br Thu Jul 13 09:21:53 2006 From: charges at humortadela.com.br (Humortadela) Date: Thu, 13 Jul 2006 08:21:53 -0500 Subject: [Numpy-discussion] Voce Recebeu uma Charge do Humortadela Message-ID: <6f7beb686bf7e1e84df7db96919ccb5a@humortadela.com.br> An HTML attachment was scrubbed... URL: From charges at humortadela.com.br Thu Jul 13 09:21:52 2006 From: charges at humortadela.com.br (Humortadela) Date: Thu, 13 Jul 2006 08:21:52 -0500 Subject: [Numpy-discussion] Voce Recebeu uma Charge do Humortadela Message-ID: <6f7beb686bf7e1e84df7db96919ccb5a@humortadela.com.br> An HTML attachment was scrubbed... URL: From Jonathan.Peirce at nottingham.ac.uk Thu Jul 13 08:55:05 2006 From: Jonathan.Peirce at nottingham.ac.uk (Jon Peirce) Date: Thu, 13 Jul 2006 13:55:05 +0100 Subject: [Numpy-discussion] generalized_inverse Message-ID: <44B642A9.3090805@psychology.nottingham.ac.uk> There used to be a function generalized_inverse in the numpy.linalg module (certainly in 0.9.2). In numpy0.9.8 it seems to have been moved to the numpy.linalg.old subpackage. Does that mean it's being dropped? Did it have to move? Now i have to add code to my package to try both locations because my users might have any version... :-( Jon -- Jon Peirce Nottingham University +44 (0)115 8467176 (tel) +44 (0)115 9515324 (fax) http://www.psychopy.org This message has been checked for viruses but the contents of an attachment may still contain software viruses, which could damage your computer system: you are advised to perform your own checks. Email communications with the University of Nottingham may be monitored as permitted by UK legislation. From berkes at gatsby.ucl.ac.uk Thu Jul 13 09:11:00 2006 From: berkes at gatsby.ucl.ac.uk (Pietro Berkes) Date: Thu, 13 Jul 2006 14:11:00 +0100 (BST) Subject: [Numpy-discussion] A few comments (MDP conversion) Message-ID: Dear numpys, a couple of weeks ago Tiziano and I completed the conversion of our data processing library MDP to numpy. We collected a few ideas and questions: - we found the convertcode.py module quite useful to perform a first, low-level conversion. We had some problem when 'typecode' was used as a keyword argument because the parser only converts 'typecode=' to 'dtype=' and skips the cases where there is an additional space character before the equal sign ('typecode ='). Other names that might be easily converted are 'ArrayType' and 'NewAxis'. - some functions changed the columns/rows conventions ('cov', for example). It would be really helpful to explicitly write this in the list of necessary changes in the documentation. It could be nice to have the file 'convertcode.py' to print a warning every time one of this functions is used in the code. - the linalg functions svd, eig, inv, pinv, diag, and possibly others perform an upcast, e.g. from 'f' to 'd'. This is apparently because the litelapack module only wraps double precision routines. Wouldn't it be more consistent to cast the results back to the numerical type of the input? Otherwise, the user always has to take care of the casting, which makes the use of single precision arrays quite cumbersome. - we found a bug in the 'eig' function in the case the solution is complex: in (linalg.py, line 269) 'Complex' should be 'w.dtype' We wondered whether it would be possible to obtain SVN write access in orderto be able to fix this kind of issues by ourselves in the future. We could also contribute docstrings for some of the functions. In general, we found the conversion to numpy quite straightforward and would like to thank you all for the great work! Cheers, Pietro Berkes and Tiziano Zito http://mdp-toolkit.sourceforge.net/ From bsouthey at gmail.com Thu Jul 13 11:07:55 2006 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 13 Jul 2006 10:07:55 -0500 Subject: [Numpy-discussion] RandomArray module In-Reply-To: References: Message-ID: Hi, As Francesc meintioned, Robert Kern did a great job of replacing ranlib: "numpy.random uses the Mersenne Twister. RANLIB is dead! Long live MT19937!" So throw away ran1! Regards Bruce On 7/12/06, Sebastian ?urek wrote: > Hi All, > > Has anyone worked with the RandomArray module? I wonder, > if it's OK to use its pseudo-random numbers generators, or > maybe I shall find more trusted methods (ie. ran1 from Numerical Recipes)? > > Please, give some comments. Thanks. > > Sebastian > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From nwagner at iam.uni-stuttgart.de Thu Jul 13 11:43:06 2006 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 13 Jul 2006 17:43:06 +0200 Subject: [Numpy-discussion] [Fwd: Re: A few comments (MDP conversion)] Message-ID: <44B66A0A.8030002@iam.uni-stuttgart.de> -------------- next part -------------- An embedded message was scrubbed... From: unknown sender Subject: no subject Date: no date Size: 38 URL: From nwagner at iam.uni-stuttgart.de Thu Jul 13 11:42:24 2006 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 13 Jul 2006 17:42:24 +0200 Subject: [Numpy-discussion] A few comments (MDP conversion) In-Reply-To: References: Message-ID: <44B669E0.7020804@iam.uni-stuttgart.de> Pietro Berkes wrote: > Dear numpys, > > a couple of weeks ago Tiziano and I completed the conversion of our data > processing library MDP to numpy. We collected a few ideas and > questions: > > - we found the convertcode.py module quite useful to perform a first, > low-level conversion. We had some problem when 'typecode' was > used as a keyword argument because the parser only converts > 'typecode=' to 'dtype=' and skips the cases where there is an > additional space character before the equal sign ('typecode ='). > Other names that might be easily converted are 'ArrayType' and > 'NewAxis'. > > - some functions changed the columns/rows conventions ('cov', for > example). It would be really helpful to explicitly write this in the > list of necessary changes in the documentation. It could be nice > to have the file 'convertcode.py' to print a warning every time one > of this functions is used in the code. > > - the linalg functions svd, eig, inv, pinv, diag, and possibly > others perform an upcast, e.g. from 'f' to 'd'. This is apparently > because the litelapack module only wraps double precision routines. > Wouldn't it be more consistent to cast the results back to the > numerical type of the input? Otherwise, the user always has to take > care of the casting, which makes the use of single precision > arrays quite cumbersome. > > - we found a bug in the 'eig' function in the case the solution > is complex: in (linalg.py, line 269) 'Complex' should be 'w.dtype' > > Did you file a ticket ? http://projects.scipy.org/scipy/numpy It seems to be line 281 instead of 269. I am using latest svn. BTW, in linalg.py in def pinv: there is another Complex with capital C. Nils > We wondered whether it would be possible to obtain SVN write access in > orderto be able to fix this kind of issues by ourselves in the future. > We could also contribute docstrings for some of the functions. > > In general, we found the conversion to numpy quite straightforward > and would like to thank you all for the great work! > > Cheers, > Pietro Berkes and Tiziano Zito > http://mdp-toolkit.sourceforge.net/ > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > --------------040602050709060203000700-- From berkes at gatsby.ucl.ac.uk Thu Jul 13 12:39:56 2006 From: berkes at gatsby.ucl.ac.uk (Pietro Berkes) Date: Thu, 13 Jul 2006 17:39:56 +0100 (BST) Subject: [Numpy-discussion] [Fwd: Re: A few comments (MDP conversion)] In-Reply-To: <44B66A0A.8030002@iam.uni-stuttgart.de> References: <44B66A0A.8030002@iam.uni-stuttgart.de> Message-ID: On Thu, 13 Jul 2006, Nils Wagner wrote: > It seems to be line 281 instead of 269. I am using latest svn. > BTW, in linalg.py in > > def pinv: > > there is another Complex with capital C. Well, the problem is not really the capital 'C', but rather the lack of quotation marks... From robert.kern at gmail.com Thu Jul 13 13:26:34 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 13 Jul 2006 12:26:34 -0500 Subject: [Numpy-discussion] RandomArray module In-Reply-To: References: Message-ID: Sebastian ?urek wrote: > Hi All, > > Has anyone worked with the RandomArray module? I wonder, > if it's OK to use its pseudo-random numbers generators, or > maybe I shall find more trusted methods (ie. ran1 from Numerical Recipes)? At this point in time, I don't recommend using RandomArray if you can avoid it. The RANLIB library that underlies it is quite well respected, but it is also quite old. The field has moved on since it was written. ran1 is no better. If you can make the leap to numpy instead of Numeric, the PRNG we use is the Mersenne Twister which beats the pants off RANLIB and anything from NR. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From efiring at hawaii.edu Thu Jul 13 13:51:31 2006 From: efiring at hawaii.edu (Eric Firing) Date: Thu, 13 Jul 2006 07:51:31 -1000 Subject: [Numpy-discussion] how to use argsort result? In-Reply-To: <6ef8f3380607130135x1fe1bcau606a5fa46a49497d@mail.gmail.com> References: <44B37040.2040905@itc.it> <6ef8f3380607110240i24f6f219w2fa112a113f04291@mail.gmail.com> <44B37608.2000003@itc.it> <44B56CB0.2080507@stsci.edu> <6ef8f3380607130135x1fe1bcau606a5fa46a49497d@mail.gmail.com> Message-ID: <44B68823.8020702@hawaii.edu> > > Would it be reasonable if argsort returned the complete tuple of > indices, so that > A[A.argsort(ax)] would work ? +1 This is the behavior one would naturally expect. Eric From webb.sprague at gmail.com Thu Jul 13 15:42:30 2006 From: webb.sprague at gmail.com (Webb Sprague) Date: Thu, 13 Jul 2006 12:42:30 -0700 Subject: [Numpy-discussion] Finding indices for all rows which contain nan (or anything else) Message-ID: Does anyone have some vectorized code that pulls out all the row indices for any row which has an nan (or a number less than 1 or whatever). I want to subsequently be able to perform an operation with all the good rows. See the imaginary code below. a = numpy.array([[1,2],[nan,1], [2,3]]) is_row_nan(a) == array([1]) ii = numpy.negative(is_row_nan(a)) a[ii,:] # these are the ones I want. Hopefully this is array([[1,2],[2,3]]) I can imagine doing this with a loop or with (maybe) some fancy set union stuff, but I am at a loss for vectorized versions. Thanks From robert.kern at gmail.com Thu Jul 13 15:56:11 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 13 Jul 2006 14:56:11 -0500 Subject: [Numpy-discussion] Finding indices for all rows which contain nan (or anything else) In-Reply-To: References: Message-ID: Webb Sprague wrote: > Does anyone have some vectorized code that pulls out all the row > indices for any row which has an nan (or a number less than 1 or > whatever). I want to subsequently be able to perform an operation > with all the good rows. See the imaginary code below. > > a = numpy.array([[1,2],[nan,1], [2,3]]) > is_row_nan(a) == array([1]) > ii = numpy.negative(is_row_nan(a)) > > a[ii,:] # these are the ones I want. Hopefully this is array([[1,2],[2,3]]) > > I can imagine doing this with a loop or with (maybe) some fancy set > union stuff, but I am at a loss for vectorized versions. (Untested) def is_row_nan(a): return numpy.isnan(a).any(axis=-1) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From webb.sprague at gmail.com Thu Jul 13 16:02:13 2006 From: webb.sprague at gmail.com (Webb Sprague) Date: Thu, 13 Jul 2006 13:02:13 -0700 Subject: [Numpy-discussion] Finding indices for all rows which contain nan (or anything else) In-Reply-To: References: Message-ID: On 7/13/06, Robert Kern wrote: > Webb Sprague wrote: > > Does anyone have some vectorized code that pulls out all the row > > def is_row_nan(a): > return numpy.isnan(a).any(axis=-1) I knew there was a way, but I didn't know to check any() and all(). Thanks to all (I love free software lists!) W From cookedm at physics.mcmaster.ca Thu Jul 13 17:11:49 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu, 13 Jul 2006 17:11:49 -0400 Subject: [Numpy-discussion] A few comments (MDP conversion) In-Reply-To: References: Message-ID: On Jul 13, 2006, at 09:11 , Pietro Berkes wrote: > Dear numpys, > > a couple of weeks ago Tiziano and I completed the conversion of our > data > processing library MDP to numpy. We collected a few ideas and > questions: > > - we found the convertcode.py module quite useful to perform a first, > low-level conversion. We had some problem when 'typecode' was > used as a keyword argument because the parser only converts > 'typecode=' to 'dtype=' and skips the cases where there is an > additional space character before the equal sign ('typecode ='). > Other names that might be easily converted are 'ArrayType' and > 'NewAxis'. fixed in svn. > - some functions changed the columns/rows conventions ('cov', for > example). It would be really helpful to explicitly write this in the > list of necessary changes in the documentation. It could be nice > to have the file 'convertcode.py' to print a warning every time one > of this functions is used in the code. Do you have a list of these? > - the linalg functions svd, eig, inv, pinv, diag, and possibly > others perform an upcast, e.g. from 'f' to 'd'. This is apparently > because the litelapack module only wraps double precision routines. > Wouldn't it be more consistent to cast the results back to the > numerical type of the input? Otherwise, the user always has to take > care of the casting, which makes the use of single precision > arrays quite cumbersome. That makes sense. I'll have a look at it. > - we found a bug in the 'eig' function in the case the solution > is complex: in (linalg.py, line 269) 'Complex' should be 'w.dtype' fixed in svn. > We wondered whether it would be possible to obtain SVN write access in > orderto be able to fix this kind of issues by ourselves in the future. > We could also contribute docstrings for some of the functions. I don't know about svn write access (that's not up to me), but we're perfectly willing to take patches made with 'svn diff', and uploaded to our bug tracker. > In general, we found the conversion to numpy quite straightforward > and would like to thank you all for the great work! > > Cheers, > Pietro Berkes and Tiziano Zito > http://mdp-toolkit.sourceforge.net/ -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From cookedm at physics.mcmaster.ca Thu Jul 13 17:13:13 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu, 13 Jul 2006 17:13:13 -0400 Subject: [Numpy-discussion] [Fwd: Re: A few comments (MDP conversion)] In-Reply-To: References: <44B66A0A.8030002@iam.uni-stuttgart.de> Message-ID: On Jul 13, 2006, at 12:39 , Pietro Berkes wrote: > On Thu, 13 Jul 2006, Nils Wagner wrote: > >> It seems to be line 281 instead of 269. I am using latest svn. >> BTW, in linalg.py in >> >> def pinv: >> >> there is another Complex with capital C. > > Well, the problem is not really the capital 'C', but rather the > lack of > quotation marks... Guess I've got more work to do to get rid of the typecodes there... -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From bhendrix at enthought.com Thu Jul 13 18:48:45 2006 From: bhendrix at enthought.com (Bryce Hendrix) Date: Thu, 13 Jul 2006 17:48:45 -0500 Subject: [Numpy-discussion] ANN: Python Enthought Edition Version 1.0.0.beta4 Released Message-ID: <44B6CDCD.7010206@enthought.com> Enthought is pleased to announce the release of Python Enthought Edition Version 1.0.0.beta4 (http://code.enthought.com/enthon/) -- a python distribution for Windows. 1.0.0.beta4 Release Notes: -------------------- There are two known issues: * No documentation is included due to problems with the chm. Instead, all documentation for this beta is available on the web at http://code.enthought.com/enthon/docs. The official 1.0.0 will include a chm containing all of our docs again. * IPython may cause problems when starting the first time if a previous version of IPython was ran. If you see "WARNING: could not import user config", either follow the directions which follow the warning. Unless something terrible is discovered between now and the next release, we intend on releasing 1.0.0 on July 25th. This release includes version 1.0.9 of the Enthought Tool Suite (ETS) Package and bug fixes-- you can look at the release notes for this ETS version here: http://svn.enthought.com/downloads/enthought/changelog-release.1.0.9.html About Python Enthought Edition: ------------------------------- Python 2.4.3, Enthought Edition is a kitchen-sink-included Python distribution for Windows including the following packages out of the box: Numpy SciPy IPython Enthought Tool Suite wxPython PIL mingw f2py MayaVi Scientific Python VTK and many more... More information is available about all Open Source code written and released by Enthought, Inc. at http://code.enthought.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From webb.sprague at gmail.com Thu Jul 13 19:55:33 2006 From: webb.sprague at gmail.com (Webb Sprague) Date: Thu, 13 Jul 2006 16:55:33 -0700 Subject: [Numpy-discussion] numpy.average(fooArray, axis=0): dropping nans from calculation Message-ID: Could someone recommend a way to average an array along the columns without propagating the nans and without turning them into some weird number which bias the result? I guess I can just keep using an indexing array for fooArray, but if there is somehting more graceful, I would love to know. Boy missing data is a pain in the neck... Thanks again! From nvf at MIT.EDU Thu Jul 13 20:08:19 2006 From: nvf at MIT.EDU (Nick Fotopoulos) Date: Thu, 13 Jul 2006 20:08:19 -0400 Subject: [Numpy-discussion] numpy.vectorize performance Message-ID: Dear all, I often make use of numpy.vectorize to make programs read more like the physics equations I write on paper. numpy.vectorize is basically a wrapper for numpy.frompyfunc. Reading Travis's Scipy Book (mine is dated Jan 6 2005) kind of suggests to me that it returns a full- fledged ufunc exactly like built-in ufuncs. First, is this true? Second, how is the performance? i.e., are my functions performing approximately as fast as they could be or would they still gain a great deal of speed by rewriting it in C or some other compiled python accelerator? As an aside, I've found the following function decorator to be helpful for readability, and perhaps others will enjoy it or improve upon it: def autovectorized(f): """Function decorator to do vectorization only as necessary. vectorized functions fail for scalar inputs.""" def wrapper(input): if type(input) == numpy.arraytype: return numpy.vectorize(f)(input) return f(input) return wrapper For those unfamiliar to the syntactic joys of Python 2.4, you can then use this as: @autovectorized def myOtherwiseScalarFunction(*args): ... and now the function will work with both numpy arrays and scalars. Take care, Nick From efiring at hawaii.edu Thu Jul 13 20:44:15 2006 From: efiring at hawaii.edu (Eric Firing) Date: Thu, 13 Jul 2006 14:44:15 -1000 Subject: [Numpy-discussion] numpy.average(fooArray, axis=0): dropping nans from calculation In-Reply-To: References: Message-ID: <44B6E8DF.3010707@hawaii.edu> Webb Sprague wrote: > Could someone recommend a way to average an array along the columns > without propagating the nans and without turning them into some weird > number which bias the result? I guess I can just keep using an > indexing array for fooArray, but if there is somehting more graceful, > I would love to know. Something like this: import numpy as N ym = N.ma.masked_where(N.isnan(y), y) yaverage = N.ma.average(ym) > > Boy missing data is a pain in the neck... It certainly is, but Masked Arrays ease the pain. Eric From robert.kern at gmail.com Fri Jul 14 02:35:19 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 14 Jul 2006 01:35:19 -0500 Subject: [Numpy-discussion] A few comments (MDP conversion) In-Reply-To: References: Message-ID: Pietro Berkes wrote: > We wondered whether it would be possible to obtain SVN write access in > orderto be able to fix this kind of issues by ourselves in the future. > We could also contribute docstrings for some of the functions. The best way to get SVN privileges in any open source project is by posting a constant stream of good patches to the bug tracker such that it becomes less work for us to give you access than manually applying the patches ourselves. ;-) http://projects.scipy.org/scipy/numpy Note that you will need to register a username/password in order to create new tickets. We have had to institute this policy due to egregious amounts of ticket spam. I have no doubt that you will provide such patches and be given SVN access in short order. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From nwagner at iam.uni-stuttgart.de Fri Jul 14 02:43:32 2006 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Fri, 14 Jul 2006 08:43:32 +0200 Subject: [Numpy-discussion] [Fwd: Re: A few comments (MDP conversion)] In-Reply-To: References: <44B66A0A.8030002@iam.uni-stuttgart.de> Message-ID: <44B73D14.3040804@iam.uni-stuttgart.de> Pietro Berkes wrote: > On Thu, 13 Jul 2006, Nils Wagner wrote: > > >> It seems to be line 281 instead of 269. I am using latest svn. >> BTW, in linalg.py in >> >> def pinv: >> >> there is another Complex with capital C. >> > > Well, the problem is not really the capital 'C', but rather the lack of > quotation marks... > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > >>> p = linalg.pinv(a) Traceback (most recent call last): File "", line 1, in ? File "/usr/lib64/python2.4/site-packages/numpy/linalg/linalg.py", line 426, in pinv if issubclass(a.dtype.dtype, complexfloating): AttributeError: 'numpy.dtype' object has no attribute 'dtype' Nils From svetosch at gmx.net Fri Jul 14 03:25:49 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Fri, 14 Jul 2006 09:25:49 +0200 Subject: [Numpy-discussion] numpy.average(fooArray, axis=0): dropping nans from calculation In-Reply-To: References: Message-ID: <44B746FD.9030408@gmx.net> Webb Sprague schrieb: > Could someone recommend a way to average an array along the columns > without propagating the nans and without turning them into some weird > number which bias the result? I guess I can just keep using an > indexing array for fooArray, but if there is somehting more graceful, > I would love to know. You could take advantage of the nan-related functions: >>> help(numpy.nansum) Help on function nansum in module numpy.lib.function_base: nansum(a, axis=-1) Sum the array over the given axis, treating NaNs as 0. > > Boy missing data is a pain in the neck... > Btw, do you know what is treated as NaN in numpy when getting the data from some external source (file, python list etc.), apart from None? I asked that on this list but it apparently went unnoticed. Cheers, Sven From svetosch at gmx.net Fri Jul 14 03:32:42 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Fri, 14 Jul 2006 09:32:42 +0200 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: <44B642A9.3090805@psychology.nottingham.ac.uk> References: <44B642A9.3090805@psychology.nottingham.ac.uk> Message-ID: <44B7489A.1050108@gmx.net> Jon Peirce schrieb: > There used to be a function generalized_inverse in the numpy.linalg > module (certainly in 0.9.2). > > In numpy0.9.8 it seems to have been moved to the numpy.linalg.old > subpackage. Does that mean it's being dropped? Did it have to move? Now > i have to add code to my package to try both locations because my users > might have any version... :-( > > Maybe I don't understand, but what's wrong with numpy.linalg.pinv? -sven From a.h.jaffe at gmail.com Fri Jul 14 04:22:43 2006 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Fri, 14 Jul 2006 09:22:43 +0100 Subject: [Numpy-discussion] debian floating point exceptions/numpy/f2py -- sse Message-ID: Hi All, I have just switched from RHEL to debian, and all of a sudden I started getting floating point exception errors in various contexts. Apparently, this has to do with some faulty error stuff in glibc, specifically related to the sse. I would prefer to fix the actual problem, but I can't seem to be able to get the recommended 'apt-get source glibc' incantation to work (I'm not root, but I can sudo.) I was able to fix some of these issues by simply downgrading ATLAS to not use sse instructions anymore. But f2py still links with sse and sse2 by default. I can't quite understand the configuration well enough to work out how to turn it off. Can someone give me any guidance? Thanks, Andrew From svetosch at gmx.net Fri Jul 14 04:51:44 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Fri, 14 Jul 2006 10:51:44 +0200 Subject: [Numpy-discussion] numpy.average(fooArray, axis=0): dropping nans from calculation In-Reply-To: References: <44B746FD.9030408@gmx.net> <44B75184.5010109@gmx.net> Message-ID: <44B75B20.8020609@gmx.net> Curzio Basso schrieb: >> Well try it out and see for yourself ;-) > > good point :-) > >> But for sums it doesn't make a difference, right... Note that it's >> called nan*sum* and not nanwhatever. > > sure, I was still thinking about the first post which was referring to > the average... > > qrz > Right, having to count the Nans then makes the masked-array solution more attractive, that's true. So maybe that's a feature request, complementing the nansum function by a nanaverage? From david at ar.media.kyoto-u.ac.jp Fri Jul 14 05:31:44 2006 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 14 Jul 2006 18:31:44 +0900 Subject: [Numpy-discussion] debian floating point exceptions/numpy/f2py -- sse In-Reply-To: References: Message-ID: <44B76480.80700@ar.media.kyoto-u.ac.jp> Andrew Jaffe wrote: > Hi All, > > I have just switched from RHEL to debian, and all of a sudden I started > getting floating point exception errors in various contexts. > > Apparently, this has to do with some faulty error stuff in glibc, > specifically related to the sse. I would prefer to fix the actual > problem, but I can't seem to be able to get the recommended 'apt-get > source glibc' incantation to work (I'm not root, but I can sudo.) > What does not work ? The apt-get source part ? The actual building ? Basically, if the sources are OK, you just need to do - fakeroot dpkg-source -x name_of_dsc_file - cd name_of_package - fakeroot dpkg-buildpackage And that's it > I was able to fix some of these issues by simply downgrading ATLAS to > not use sse instructions anymore. > > But f2py still links with sse and sse2 by default. I can't quite > understand the configuration well enough to work out how to turn it off. > Can someone give me any guidance? > The way it is supposed to work, at least on debian and ubuntu, is that you never link to the sse/sse2 versions, but to the non-optimized versions. After, the dynamic loader will get the right one (ie optimized if available) instead of the one linked (this of course only works for dynamic linking). You can check which library is picked with a ldconfig - p | grep lapack (for lapack functions, and so on...) David From aisaac at american.edu Fri Jul 14 09:26:19 2006 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 14 Jul 2006 09:26:19 -0400 Subject: [Numpy-discussion] numpy.average(fooArray, axis=0): dropping nans from calculation In-Reply-To: <44B75B20.8020609@gmx.net> References: <44B746FD.9030408@gmx.net> <44B75184.5010109@gmx.net><44B75B20.8020609@gmx.net> Message-ID: On Fri, 14 Jul 2006, Sven Schreiber apparently wrote: > So maybe that's a feature request, complementing the > nansum function by a nanaverage? This is not an objection; just an observation. It has always seemed to me that such descriptive statistics make more sense as class attributes. In this case, something like a NanDstat class. fwiw, Alan Isaac From laidler at stsci.edu Fri Jul 14 11:56:38 2006 From: laidler at stsci.edu (Victoria G. Laidler) Date: Fri, 14 Jul 2006 11:56:38 -0400 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: <44B7489A.1050108@gmx.net> References: <44B642A9.3090805@psychology.nottingham.ac.uk> <44B7489A.1050108@gmx.net> Message-ID: <44B7BEB6.8070608@stsci.edu> Sven Schreiber wrote: >Jon Peirce schrieb: > > >>There used to be a function generalized_inverse in the numpy.linalg >>module (certainly in 0.9.2). >> >>In numpy0.9.8 it seems to have been moved to the numpy.linalg.old >>subpackage. Does that mean it's being dropped? Did it have to move? Now >>i have to add code to my package to try both locations because my users >>might have any version... :-( >> >> >> >> > >Maybe I don't understand, but what's wrong with numpy.linalg.pinv? > > Er, what's a pinv? It doesn't sound anything like a generalized_inverse. Vicki Laidler From jonathan.taylor at stanford.edu Fri Jul 14 11:59:49 2006 From: jonathan.taylor at stanford.edu (Jonathan Taylor) Date: Fri, 14 Jul 2006 08:59:49 -0700 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: <44B7BEB6.8070608@stsci.edu> References: <44B642A9.3090805@psychology.nottingham.ac.uk> <44B7489A.1050108@gmx.net> <44B7BEB6.8070608@stsci.edu> Message-ID: <44B7BF75.2080708@stanford.edu> pseudoinverse it's the same name matlab uses: http://www.mathworks.com/access/helpdesk/help/techdoc/ref/pinv.html Victoria G. Laidler wrote: >Sven Schreiber wrote: > > > >>Jon Peirce schrieb: >> >> >> >> >>>There used to be a function generalized_inverse in the numpy.linalg >>>module (certainly in 0.9.2). >>> >>>In numpy0.9.8 it seems to have been moved to the numpy.linalg.old >>>subpackage. Does that mean it's being dropped? Did it have to move? Now >>>i have to add code to my package to try both locations because my users >>>might have any version... :-( >>> >>> >>> >>> >>> >>> >>Maybe I don't understand, but what's wrong with numpy.linalg.pinv? >> >> >> >> >Er, what's a pinv? It doesn't sound anything like a generalized_inverse. > >Vicki Laidler > > > >------------------------------------------------------------------------- >Using Tomcat but need to do more? Need to support web services, security? >Get stuff done quickly with pre-integrated technology to make your job easier >Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo >http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > -- ------------------------------------------------------------------------ I'm part of the Team in Training: please support our efforts for the Leukemia and Lymphoma Society! http://www.active.com/donate/tntsvmb/tntsvmbJTaylor GO TEAM !!! ------------------------------------------------------------------------ Jonathan Taylor Tel: 650.723.9230 Dept. of Statistics Fax: 650.725.8977 Sequoia Hall, 137 www-stat.stanford.edu/~jtaylo 390 Serra Mall Stanford, CA 94305 From laidler at stsci.edu Fri Jul 14 12:16:58 2006 From: laidler at stsci.edu (Victoria G. Laidler) Date: Fri, 14 Jul 2006 12:16:58 -0400 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: <44B7BF75.2080708@stanford.edu> References: <44B642A9.3090805@psychology.nottingham.ac.uk> <44B7489A.1050108@gmx.net> <44B7BEB6.8070608@stsci.edu> <44B7BF75.2080708@stanford.edu> Message-ID: <44B7C37A.7010801@stsci.edu> Jonathan Taylor wrote: > pseudoinverse > > it's the same name matlab uses: > > http://www.mathworks.com/access/helpdesk/help/techdoc/ref/pinv.html Thanks for the explanation. I'm puzzled by the naming choice, however. Standard best practice in writing software is to give understandable names, to improve readability and code maintenance. Obscure abbreviations like "pinv" pretty much went out with the FORTRAN 9-character limit for variable names. It's very unusual to see them in new software nowadays, and it always looks unprofessional to me. I understand that for interactive use, short names are more convenient; but shouldn't they be available aliases to the more general names? Since numpy is primarily a software library, I wouldn't expect it to sacrifice a standard best-practice in order to make things more convenient for interactive use. If the concern is for for matlab compatibility, maybe a synonym module numpy.as_matlab could define all the synonyms, that matlab users could then use? That would make more sense to me than inflicting obscure matlab names on the rest of the user community. Vicki Laidler > > Victoria G. Laidler wrote: > >> Sven Schreiber wrote: >> >> >> >>> Jon Peirce schrieb: >>> >>> >>> >>> >>>> There used to be a function generalized_inverse in the numpy.linalg >>>> module (certainly in 0.9.2). >>>> >>>> In numpy0.9.8 it seems to have been moved to the numpy.linalg.old >>>> subpackage. Does that mean it's being dropped? Did it have to move? >>>> Now i have to add code to my package to try both locations because >>>> my users might have any version... :-( >>>> >>>> >>>> >>>> >>> >>> Maybe I don't understand, but what's wrong with numpy.linalg.pinv? >>> >>> >>> >> >> Er, what's a pinv? It doesn't sound anything like a generalized_inverse. >> >> Vicki Laidler >> >> >> >> ------------------------------------------------------------------------- >> >> Using Tomcat but need to do more? Need to support web services, >> security? >> Get stuff done quickly with pre-integrated technology to make your >> job easier >> Download IBM WebSphere Application Server v.1.0.1 based on Apache >> Geronimo >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> >> > From strawman at astraw.com Fri Jul 14 12:28:57 2006 From: strawman at astraw.com (Andrew Straw) Date: Fri, 14 Jul 2006 09:28:57 -0700 Subject: [Numpy-discussion] debian floating point exceptions/numpy/f2py -- sse In-Reply-To: References: Message-ID: <44B7C649.8080204@astraw.com> GNU libc version 2.3.2 has a bug[1] "feclearexcept() error on CPUs with SSE" (fixed in 2.3.3) which has been submitted to Debian[2] but not fixed in sarge. See http://www.its.caltech.edu/~astraw/coding.html#id3 for more information and .debs which fix the problem. [1] http://sources.redhat.com/bugzilla/show_bug.cgi?id=10 [2] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=279294 If only somebody could show how this was a security issue, we could get Debian to finally release a fix in sarge for this (frequently asked) question. Andrew Jaffe wrote: > Hi All, > > I have just switched from RHEL to debian, and all of a sudden I started > getting floating point exception errors in various contexts. > > Apparently, this has to do with some faulty error stuff in glibc, > specifically related to the sse. I would prefer to fix the actual > problem, but I can't seem to be able to get the recommended 'apt-get > source glibc' incantation to work (I'm not root, but I can sudo.) > > I was able to fix some of these issues by simply downgrading ATLAS to > not use sse instructions anymore. > > But f2py still links with sse and sse2 by default. I can't quite > understand the configuration well enough to work out how to turn it off. > Can someone give me any guidance? > > Thanks, > > Andrew > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From efiring at hawaii.edu Fri Jul 14 12:28:53 2006 From: efiring at hawaii.edu (Eric Firing) Date: Fri, 14 Jul 2006 06:28:53 -1000 Subject: [Numpy-discussion] numpy.average(fooArray, axis=0): dropping nans from calculation In-Reply-To: References: <44B746FD.9030408@gmx.net> <44B75184.5010109@gmx.net> <44B75B20.8020609@gmx.net> Message-ID: <44B7C645.2090507@hawaii.edu> Alan G Isaac wrote: > On Fri, 14 Jul 2006, Sven Schreiber apparently wrote: > >>So maybe that's a feature request, complementing the >>nansum function by a nanaverage? > > > This is not an objection; just an observation. > It has always seemed to me that such descriptive > statistics make more sense as class attributes. > In this case, something like a NanDstat class. Attached is something like that, in case someone finds it useful. It is designed to replace something I wrote a long time ago for matlab. It is only very lightly tested, so use with care. Eric -------------- next part -------------- A non-text attachment was scrubbed... Name: mstdgap.py Type: text/x-python Size: 5659 bytes Desc: not available URL: From a.h.jaffe at gmail.com Fri Jul 14 12:35:36 2006 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Fri, 14 Jul 2006 17:35:36 +0100 Subject: [Numpy-discussion] debian floating point exceptions/numpy/f2py -- sse In-Reply-To: <44B76480.80700@ar.media.kyoto-u.ac.jp> References: <44B76480.80700@ar.media.kyoto-u.ac.jp> Message-ID: <44B7C7D8.3090800@gmail.com> David Cournapeau wrote: > Andrew Jaffe wrote: >> Hi All, >> >> I have just switched from RHEL to debian, and all of a sudden I started >> getting floating point exception errors in various contexts. >> >> Apparently, this has to do with some faulty error stuff in glibc, >> specifically related to the sse. I would prefer to fix the actual >> problem, but I can't seem to be able to get the recommended 'apt-get >> source glibc' incantation to work (I'm not root, but I can sudo.) >> > What does not work ? The apt-get source part ? The actual building ? > > Basically, if the sources are OK, you just need to do > - fakeroot dpkg-source -x name_of_dsc_file > - cd name_of_package > - fakeroot dpkg-buildpackage > > And that's it Aha -- I didn't know about fakeroot... Thanks! That was the problem with apt-get source. I'm compiling the patched version now... wow, is it slow! >> I was able to fix some of these issues by simply downgrading ATLAS to >> not use sse instructions anymore. >> >> But f2py still links with sse and sse2 by default. I can't quite >> understand the configuration well enough to work out how to turn it off. >> Can someone give me any guidance? >> > The way it is supposed to work, at least on debian and ubuntu, is that > you never link to the sse/sse2 versions, but to the non-optimized > versions. After, the dynamic loader will get the right one (ie optimized > if available) instead of the one linked (this of course only works for > dynamic linking). You can check which library is picked with a ldconfig > - p | grep lapack (for lapack functions, and so on...) The problem with f2py isn't the atlas/lapack linkage, which it does correctly, but the fact that it automatically appends -sse2 to the g77 compile options; I couldn't figure out how to turn that off! Although now I'm not so sure, since I can never get my self-compiled executable version of my fortran code to give the same error as when it runs within python. But with the patched glibc, I think I'm alright in any event! Thanks! A From a.h.jaffe at gmail.com Fri Jul 14 12:35:36 2006 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Fri, 14 Jul 2006 17:35:36 +0100 Subject: [Numpy-discussion] debian floating point exceptions/numpy/f2py -- sse In-Reply-To: <44B76480.80700@ar.media.kyoto-u.ac.jp> References: <44B76480.80700@ar.media.kyoto-u.ac.jp> Message-ID: <44B7C7D8.3090800@gmail.com> David Cournapeau wrote: > Andrew Jaffe wrote: >> Hi All, >> >> I have just switched from RHEL to debian, and all of a sudden I started >> getting floating point exception errors in various contexts. >> >> Apparently, this has to do with some faulty error stuff in glibc, >> specifically related to the sse. I would prefer to fix the actual >> problem, but I can't seem to be able to get the recommended 'apt-get >> source glibc' incantation to work (I'm not root, but I can sudo.) >> > What does not work ? The apt-get source part ? The actual building ? > > Basically, if the sources are OK, you just need to do > - fakeroot dpkg-source -x name_of_dsc_file > - cd name_of_package > - fakeroot dpkg-buildpackage > > And that's it Aha -- I didn't know about fakeroot... Thanks! That was the problem with apt-get source. I'm compiling the patched version now... wow, is it slow! >> I was able to fix some of these issues by simply downgrading ATLAS to >> not use sse instructions anymore. >> >> But f2py still links with sse and sse2 by default. I can't quite >> understand the configuration well enough to work out how to turn it off. >> Can someone give me any guidance? >> > The way it is supposed to work, at least on debian and ubuntu, is that > you never link to the sse/sse2 versions, but to the non-optimized > versions. After, the dynamic loader will get the right one (ie optimized > if available) instead of the one linked (this of course only works for > dynamic linking). You can check which library is picked with a ldconfig > - p | grep lapack (for lapack functions, and so on...) The problem with f2py isn't the atlas/lapack linkage, which it does correctly, but the fact that it automatically appends -sse2 to the g77 compile options; I couldn't figure out how to turn that off! Although now I'm not so sure, since I can never get my self-compiled executable version of my fortran code to give the same error as when it runs within python. But with the patched glibc, I think I'm alright in any event! Thanks! A From nvf at MIT.EDU Fri Jul 14 12:43:38 2006 From: nvf at MIT.EDU (Nick Fotopoulos) Date: Fri, 14 Jul 2006 12:43:38 -0400 Subject: [Numpy-discussion] numpy.vectorize performance In-Reply-To: <44B6FECD.8000004@cox.net> References: <44B6FECD.8000004@cox.net> Message-ID: <432FD99A-ECC4-423C-B089-C5077D230A17@mit.edu> On Jul 13, 2006, at 10:17 PM, Tim Hochberg wrote: > Nick Fotopoulos wrote: >> Dear all, >> >> I often make use of numpy.vectorize to make programs read more >> like the physics equations I write on paper. numpy.vectorize is >> basically a wrapper for numpy.frompyfunc. Reading Travis's Scipy >> Book (mine is dated Jan 6 2005) kind of suggests to me that it >> returns a full- fledged ufunc exactly like built-in ufuncs. >> >> First, is this true? > Well according to type(), the result of frompyfunc is indeed of > type ufunc, so I would say the answer to that is yes. >> Second, how is the performance? > A little timing indicates that it's not good (about 30 X slower for > computing x**2 than doing it using x*x on an array). . That's not > frompyfunc (or vectorizes) fault though. It's calling a python > function at each point, so the python function call overhead is > going to kill you. Not to mention instantiating an actual Python > object or objects at each point. That's unfortunate since I tend to nest functions quite deeply and then scipy.integrate.quad over them, which I'm sure results in a ridiculous number of function calls. Are anonymous lambdas any different than named functions in terms of performance? > >> i.e., are my functions performing approximately as fast as they >> could be or would they still gain a great deal of speed by >> rewriting it in C or some other compiled python accelerator? >> > Can you give examples of what these functions look like? You might > gain a great deal of speed by rewriting them in numpy in the > correct way. Or perhaps not, but it's probably worth showing some > examples so we can offer suggestions or at least admit that we are > stumped. This is by far the slowest bit of my code. I cache the results, so it's not too bad, but any upstream tweak can take a lot of CPU time to propagate. @autovectorized def dnsratezfunc(z): """Take coalescence time into account."" def integrand(zf): return Pz(z,zf)*NSbirthzfunc(zf) return quad(integrand,delayedZ(2e5*secperyear+1,z),5)[0] dnsratez = lambdap*dnsratezfunc(zs) where: # Neutron star formation rate is a delayed version of star formation rate NSbirthzfunc = autovectorized(lambda z: SFRz(delayedZ (1e8*secperyear,z))) def Pz(z_c,z_f): """Return the probability density per unit redshift of a DNS coalescence at z_c given a progenitor formation at z_f. """ return P(t(z_c,z_f))*dtdz(z_c) and there are many further nested levels of function calls. If the act of calling a function is more expensive than actually executing it and I value speed over readability/code reuse, I can inline Pz's function calls and inline the unvectorized NSbirthzfunc to reduce the calling stack a bit. Any other suggestions? Thanks, Tim. Take care, Nick From tim.hochberg at cox.net Fri Jul 14 12:56:32 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri, 14 Jul 2006 09:56:32 -0700 Subject: [Numpy-discussion] numpy.vectorize performance In-Reply-To: <432FD99A-ECC4-423C-B089-C5077D230A17@mit.edu> References: <44B6FECD.8000004@cox.net> <432FD99A-ECC4-423C-B089-C5077D230A17@mit.edu> Message-ID: <44B7CCC0.8050100@cox.net> Nick Fotopoulos wrote: > On Jul 13, 2006, at 10:17 PM, Tim Hochberg wrote: > > >> Nick Fotopoulos wrote: >> >>> Dear all, >>> >>> I often make use of numpy.vectorize to make programs read more >>> like the physics equations I write on paper. numpy.vectorize is >>> basically a wrapper for numpy.frompyfunc. Reading Travis's Scipy >>> Book (mine is dated Jan 6 2005) kind of suggests to me that it >>> returns a full- fledged ufunc exactly like built-in ufuncs. >>> >>> First, is this true? >>> >> Well according to type(), the result of frompyfunc is indeed of >> type ufunc, so I would say the answer to that is yes. >> >>> Second, how is the performance? >>> >> A little timing indicates that it's not good (about 30 X slower for >> computing x**2 than doing it using x*x on an array). . That's not >> frompyfunc (or vectorizes) fault though. It's calling a python >> function at each point, so the python function call overhead is >> going to kill you. Not to mention instantiating an actual Python >> object or objects at each point. >> > > That's unfortunate since I tend to nest functions quite deeply and > then scipy.integrate.quad over them, which I'm sure results in a > ridiculous number of function calls. Are anonymous lambdas any > different than named functions in terms of performance? > Sorry, no. Under the covers they're the same. >>> i.e., are my functions performing approximately as fast as they >>> could be or would they still gain a great deal of speed by >>> rewriting it in C or some other compiled python accelerator? >>> >>> >> Can you give examples of what these functions look like? You might >> gain a great deal of speed by rewriting them in numpy in the >> correct way. Or perhaps not, but it's probably worth showing some >> examples so we can offer suggestions or at least admit that we are >> stumped. >> > > This is by far the slowest bit of my code. I cache the results, so > it's not too bad, but any upstream tweak can take a lot of CPU time > to propagate. > > @autovectorized > def dnsratezfunc(z): > """Take coalescence time into account."" > def integrand(zf): > return Pz(z,zf)*NSbirthzfunc(zf) > return quad(integrand,delayedZ(2e5*secperyear+1,z),5)[0] > dnsratez = lambdap*dnsratezfunc(zs) > > where: > > # Neutron star formation rate is a delayed version of star formation > rate > NSbirthzfunc = autovectorized(lambda z: SFRz(delayedZ > (1e8*secperyear,z))) > > def Pz(z_c,z_f): > """Return the probability density per unit redshift of a DNS > coalescence at z_c given a progenitor formation at z_f. """ > return P(t(z_c,z_f))*dtdz(z_c) > > and there are many further nested levels of function calls. If the > act of calling a function is more expensive than actually executing > it and I value speed over readability/code reuse, I can inline Pz's > function calls and inline the unvectorized NSbirthzfunc to reduce the > calling stack a bit. Any other suggestions? > I think I'd try psyco (http://psyco.sourceforge.net/). That's pretty painless to try and may result in a significant improvement. -tim From travis at enthought.com Fri Jul 14 13:19:11 2006 From: travis at enthought.com (Travis N. Vaught) Date: Fri, 14 Jul 2006 12:19:11 -0500 Subject: [Numpy-discussion] ANN: SciPy 2006 Schedule/Early Registration Reminder Message-ID: <44B7D20F.8060701@enthought.com> Greetings, The SciPy 2006 Conference (http://www.scipy.org/SciPy2006) is August 17-18 this year. The deadline for early registration is *today*, July 14, 2006. The registration price will increase from $100 to $150 after today. You can register online at https://www.enthought.com/scipy06 . We invite everyone attending the conference to also attend the Coding Sprints on Monday-Tuesday , August 14-15 and also the Tutorials Wednesday, August 16. There is no additional charge for these sessions. A *tentative* schedule of talks has now been posted. http://www.scipy.org/SciPy2006/Schedule We look forward to seeing you at CalTech in August! Best, Travis From robert.kern at gmail.com Fri Jul 14 13:29:39 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 14 Jul 2006 12:29:39 -0500 Subject: [Numpy-discussion] [Fwd: Re: A few comments (MDP conversion)] In-Reply-To: <44B73D14.3040804@iam.uni-stuttgart.de> References: <44B66A0A.8030002@iam.uni-stuttgart.de> <44B73D14.3040804@iam.uni-stuttgart.de> Message-ID: Nils Wagner wrote: > >>> p = linalg.pinv(a) > Traceback (most recent call last): > File "", line 1, in ? > File "/usr/lib64/python2.4/site-packages/numpy/linalg/linalg.py", line > 426, in pinv > if issubclass(a.dtype.dtype, complexfloating): > AttributeError: 'numpy.dtype' object has no attribute 'dtype' Unless if you have something substantive to actually add to the discussion, the ticket you submitted is sufficient to report the bug. There are reasons we have a bug tracker, and one of them is to keep otherwise contentless bug reports off the mailing list. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From nvf at MIT.EDU Fri Jul 14 13:39:07 2006 From: nvf at MIT.EDU (Nick Fotopoulos) Date: Fri, 14 Jul 2006 13:39:07 -0400 Subject: [Numpy-discussion] numpy.vectorize performance In-Reply-To: <44B7CCC0.8050100@cox.net> References: <44B6FECD.8000004@cox.net> <432FD99A-ECC4-423C-B089-C5077D230A17@mit.edu> <44B7CCC0.8050100@cox.net> Message-ID: On Jul 14, 2006, at 12:56 PM, Tim Hochberg wrote: > I think I'd try psyco (http://psyco.sourceforge.net/). That's > pretty painless to try and may result in a significant improvement. I've been doing more and more development on my PPC Mac, where psyco is not an option. If the speed issue really gets to me, I can run things with psyco on a linux box. Thanks, Nick From wbaxter at gmail.com Fri Jul 14 15:01:29 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 15 Jul 2006 04:01:29 +0900 Subject: [Numpy-discussion] [Fwd: Re: A few comments (MDP conversion)] In-Reply-To: References: <44B66A0A.8030002@iam.uni-stuttgart.de> <44B73D14.3040804@iam.uni-stuttgart.de> Message-ID: On 7/15/06, Robert Kern wrote: > Nils Wagner wrote: > > > >>> p = linalg.pinv(a) > > Traceback (most recent call last): > > File "", line 1, in ? > > File "/usr/lib64/python2.4/site-packages/numpy/linalg/linalg.py", line > > 426, in pinv > > if issubclass(a.dtype.dtype, complexfloating): > > AttributeError: 'numpy.dtype' object has no attribute 'dtype' > > Unless if you have something substantive to actually add to the discussion, the > ticket you submitted is sufficient to report the bug. There are reasons we have > a bug tracker, and one of them is to keep otherwise contentless bug reports off > the mailing list. If that's really the policy, then this page should probably be changed: http://www.scipy.org/Developer_Zone Quote: "Bugs should be reported to one of the appropriate Mailing Lists. Do this first, and open a ticket on the corresponding developer's wiki if necessary." --Bill From robert.kern at gmail.com Fri Jul 14 15:11:31 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 14 Jul 2006 14:11:31 -0500 Subject: [Numpy-discussion] [Fwd: Re: A few comments (MDP conversion)] In-Reply-To: References: <44B66A0A.8030002@iam.uni-stuttgart.de> <44B73D14.3040804@iam.uni-stuttgart.de> Message-ID: Bill Baxter wrote: > On 7/15/06, Robert Kern wrote: >> Nils Wagner wrote: >> >>> >>> p = linalg.pinv(a) >>> Traceback (most recent call last): >>> File "", line 1, in ? >>> File "/usr/lib64/python2.4/site-packages/numpy/linalg/linalg.py", line >>> 426, in pinv >>> if issubclass(a.dtype.dtype, complexfloating): >>> AttributeError: 'numpy.dtype' object has no attribute 'dtype' >> Unless if you have something substantive to actually add to the discussion, the >> ticket you submitted is sufficient to report the bug. There are reasons we have >> a bug tracker, and one of them is to keep otherwise contentless bug reports off >> the mailing list. > > If that's really the policy, then this page should probably be changed: > http://www.scipy.org/Developer_Zone > > Quote: > "Bugs should be reported to one of the appropriate Mailing Lists. Do > this first, and open a ticket on the corresponding developer's wiki if > necessary." Yes, that *should* be fixed. In January it was a reasonable policy, but with the ticket mailing lists, it's not useful anymore. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cookedm at physics.mcmaster.ca Fri Jul 14 15:21:11 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri, 14 Jul 2006 15:21:11 -0400 Subject: [Numpy-discussion] [Fwd: Re: A few comments (MDP conversion)] In-Reply-To: References: <44B66A0A.8030002@iam.uni-stuttgart.de> <44B73D14.3040804@iam.uni-stuttgart.de> Message-ID: <20060714152111.29c9d4a6@arbutus.physics.mcmaster.ca> On Fri, 14 Jul 2006 12:29:39 -0500 Robert Kern wrote: > Nils Wagner wrote: > > > >>> p = linalg.pinv(a) > > Traceback (most recent call last): > > File "", line 1, in ? > > File "/usr/lib64/python2.4/site-packages/numpy/linalg/linalg.py", line > > 426, in pinv > > if issubclass(a.dtype.dtype, complexfloating): > > AttributeError: 'numpy.dtype' object has no attribute 'dtype' > > Unless if you have something substantive to actually add to the discussion, > the ticket you submitted is sufficient to report the bug. There are reasons > we have a bug tracker, and one of them is to keep otherwise contentless bug > reports off the mailing list. For an obvious typo like this, made on a recent commit (i.e., in the last day), it's probably just enough to email the committer (me, in this case). Or, at least, I don't mind. [this kind of error sneaking by is b/c we don't have enough test cases :-)] -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From svetosch at gmx.net Fri Jul 14 17:16:28 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Fri, 14 Jul 2006 23:16:28 +0200 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: <44B7C37A.7010801@stsci.edu> References: <44B642A9.3090805@psychology.nottingham.ac.uk> <44B7489A.1050108@gmx.net> <44B7BEB6.8070608@stsci.edu> <44B7BF75.2080708@stanford.edu> <44B7C37A.7010801@stsci.edu> Message-ID: <44B809AC.5080701@gmx.net> Victoria G. Laidler schrieb: > > I understand that for interactive use, short names are more convenient; > but shouldn't they be available aliases to the more general names? Since > numpy is primarily a software library, I wouldn't expect it to sacrifice > a standard best-practice in order to make things more convenient for > interactive use. I don't necessarily agree that numpy should aim to be primarily a library, but I'm with you concerning the alias idea. However, iirc there was some discussion recently on this list about the dual solution (long names as well as short ones in parallel), and some important numpy people had some reservations, although I don't remember exactly what those were -- probably some Python Zen issues ("there should be only one way to get to Rome", was that it? -- just kidding). > > If the concern is for for matlab compatibility, maybe a synonym module > numpy.as_matlab could define all the synonyms, that matlab users could > then use? That would make more sense to me than inflicting obscure > matlab names on the rest of the user community. >From superficially browsing through the numpy guide my subjective impression is that function names are mostly pretty short. So maybe the alias thing should work the other way around, making long names available in a module numpy.long_names_for_typing_addicts (again, a bad joke...) cheers, Sven From kwgoodman at gmail.com Fri Jul 14 17:54:09 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 14 Jul 2006 14:54:09 -0700 Subject: [Numpy-discussion] [Fwd: Re: A few comments (MDP conversion)] In-Reply-To: References: <44B66A0A.8030002@iam.uni-stuttgart.de> <44B73D14.3040804@iam.uni-stuttgart.de> Message-ID: On 7/14/06, Bill Baxter wrote: > On 7/15/06, Robert Kern wrote: > > Nils Wagner wrote: > > > > > >>> p = linalg.pinv(a) > > > Traceback (most recent call last): > > > File "", line 1, in ? > > > File "/usr/lib64/python2.4/site-packages/numpy/linalg/linalg.py", line > > > 426, in pinv > > > if issubclass(a.dtype.dtype, complexfloating): > > > AttributeError: 'numpy.dtype' object has no attribute 'dtype' > > > > Unless if you have something substantive to actually add to the discussion, the > > ticket you submitted is sufficient to report the bug. There are reasons we have > > a bug tracker, and one of them is to keep otherwise contentless bug reports off > > the mailing list. > > If that's really the policy, then this page should probably be changed: > http://www.scipy.org/Developer_Zone > > Quote: > "Bugs should be reported to one of the appropriate Mailing Lists. Do > this first, and open a ticket on the corresponding developer's wiki if > necessary." I like seeing the bug reports on the list. It is an easy way to get alerts of what to look out for. From robert.kern at gmail.com Fri Jul 14 18:00:36 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 14 Jul 2006 17:00:36 -0500 Subject: [Numpy-discussion] [Fwd: Re: A few comments (MDP conversion)] In-Reply-To: References: <44B66A0A.8030002@iam.uni-stuttgart.de> <44B73D14.3040804@iam.uni-stuttgart.de> Message-ID: Keith Goodman wrote: > I like seeing the bug reports on the list. It is an easy way to get > alerts of what to look out for. There are read-only mailing lists for new and updated tickets: http://www.scipy.org/Mailing_Lists -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From fperez.net at gmail.com Fri Jul 14 20:52:58 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 14 Jul 2006 18:52:58 -0600 Subject: [Numpy-discussion] Scalars, comparisons and indexing. Is this a bug? Message-ID: Hi all, I just got bit by this problem, and I'm not really sure if this is something that should be considered a numpy bug or not. It is a bit unpleasant, at the least, though it can be easily worked around. Here's a simple demonstration: In [32]: a=array([1,2]) In [33]: b=1 In [34]: (1,2)[a[0]==b] --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /home/fperez/research/code/mwmerge/mwadap-merge/mwadap/test/ TypeError: tuple indices must be integers Whereas this works: In [35]: c=2 In [36]: (1,2)[c==b] Out[36]: 1 Basically, it seems that array scalars, upon comparisons, return something that doesn't 'looke enough like an int' to python for it to let you use it as an index, it's a 'boolscalar': In [38]: a0==b0 Out[38]: True In [39]: type _ -------> type(_) Out[39]: Advice? Is this a numpy bug? Or should it just be left alone, and this kind of inconvenience will disappear when 2.5 is out, with __index__ support? Cheers, f From kwgoodman at gmail.com Fri Jul 14 21:22:18 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 14 Jul 2006 18:22:18 -0700 Subject: [Numpy-discussion] Atlas and dual core Message-ID: Is there much speed to be gained by compiling atlas for a dual core system? I'm running Ubuntu on a Macbook. It's the first time I've had a dual core system. My one line benchmark shows that the Macbook is slow compared to my (old) desktop. >> t1=time.time();x=randn(500,1000);x = x*x.T;a,b=linalg.eigh(x);t2=time.time();print t2-t1 1.31429600716 My desktop is less than half of that. From wbaxter at gmail.com Fri Jul 14 21:26:06 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 15 Jul 2006 10:26:06 +0900 Subject: [Numpy-discussion] Scalars, comparisons and indexing. Is this a bug? In-Reply-To: References: Message-ID: I believe that's the problem that the indexing PEP from Travis is supposed to address: http://www.python.org/dev/peps/pep-0357/ So I think there's not much anyone can do about it untill the PEP is accepted and implemented. Maybe you can cast to int? > In [34]: (1,2)[int(a[0]==b)] --bb On 7/15/06, Fernando Perez wrote: > Hi all, > > I just got bit by this problem, and I'm not really sure if this is > something that should be considered a numpy bug or not. It is a bit > unpleasant, at the least, though it can be easily worked around. > Here's a simple demonstration: > > In [32]: a=array([1,2]) > > In [33]: b=1 > > In [34]: (1,2)[a[0]==b] > --------------------------------------------------------------------------- > exceptions.TypeError Traceback (most > recent call last) > > /home/fperez/research/code/mwmerge/mwadap-merge/mwadap/test/ > > TypeError: tuple indices must be integers > > > Whereas this works: > > In [35]: c=2 > > In [36]: (1,2)[c==b] > Out[36]: 1 > > Basically, it seems that array scalars, upon comparisons, return > something that doesn't 'looke enough like an int' to python for it to > let you use it as an index, it's a 'boolscalar': > > In [38]: a0==b0 > Out[38]: True > > In [39]: type _ > -------> type(_) > Out[39]: > > > Advice? Is this a numpy bug? Or should it just be left alone, and > this kind of inconvenience will disappear when 2.5 is out, with > __index__ support? > > Cheers, > > f > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- William V. Baxter III OLM Digital Kono Dens Building Rm 302 1-8-8 Wakabayashi Setagaya-ku Tokyo, Japan 154-0023 +81 (3) 3422-3380 From fperez.net at gmail.com Fri Jul 14 21:36:09 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 14 Jul 2006 19:36:09 -0600 Subject: [Numpy-discussion] Scalars, comparisons and indexing. Is this a bug? In-Reply-To: References: Message-ID: On 7/14/06, Bill Baxter wrote: > I believe that's the problem that the indexing PEP from Travis is > supposed to address: > http://www.python.org/dev/peps/pep-0357/ > So I think there's not much anyone can do about it untill the PEP is > accepted and implemented. > > Maybe you can cast to int? > > In [34]: (1,2)[int(a[0]==b)] Yup, that's the workaround I'm using. I was just wondering if comparisons between array scalars shouldn't return /true/ booleans (which can be used as indices) instead of array scalar booleans. I realize that the __index__ support in 2.5 will make this point moot (the PEP you point to), but perhaps this particular one can actually be fixed today, for all users of python pre-2.5. However, I haven't really wrapped my head enough around all the subtleties of array scalars to know whether 'fixing' this particular problem will introduce other, worse ones. Cheers, f From lists.steve at arachnedesign.net Sat Jul 15 10:57:28 2006 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Sat, 15 Jul 2006 10:57:28 -0400 Subject: [Numpy-discussion] Atlas and dual core In-Reply-To: References: Message-ID: <49832F8C-3FE7-4575-912F-AC4AFF440F55@arachnedesign.net> Hi Keith, I don't have any answers for you but ... 1) Did you install ubuntu by way of Parallels, or are you running linux "full steam" 2) If you did install ubuntu alongside/over/whatever mac os x and didn't use parallels .. are there any links you can share w/ a few tutorials? Is everything (else: like hardware support) working well? (Maybe off list since it's OT -- but maybe others are interested) 3) If you still have your Mac OS X partition lying around, the full r- platform installer [1] for the mac comes with an installer for a threaded compile of ATLAS (specidifally for the intel duo's .. somehow) so maybe you'd like to try to use that and save yourself some time ... but perhaps that whole malloc vs dmalloc report [2] might apply to this as well ... which I guess would suggest to use linux for maximal performance (refer to q 2 :-) -steve [1] : http://cran.r-project.org/bin/macosx/ [2] : http://sekhon.berkeley.edu/macosx/intel.html On Jul 14, 2006, at 9:22 PM, Keith Goodman wrote: > Is there much speed to be gained by compiling atlas for a dual core > system? > > I'm running Ubuntu on a Macbook. It's the first time I've had a dual > core system. > > My one line benchmark shows that the Macbook is slow compared to my > (old) desktop. > >>> t1=time.time();x=randn(500,1000);x = > x*x.T;a,b=linalg.eigh(x);t2=time.time();print t2-t1 > 1.31429600716 > > My desktop is less than half of that. > > > ---------------------------------------------------------------------- > --- > Using Tomcat but need to do more? Need to support web services, > security? > Get stuff done quickly with pre-integrated technology to make your > job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel? > cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From kwgoodman at gmail.com Sat Jul 15 11:14:52 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Sat, 15 Jul 2006 08:14:52 -0700 Subject: [Numpy-discussion] Atlas and dual core In-Reply-To: <49832F8C-3FE7-4575-912F-AC4AFF440F55@arachnedesign.net> References: <49832F8C-3FE7-4575-912F-AC4AFF440F55@arachnedesign.net> Message-ID: On 7/15/06, Steve Lianoglou wrote: > 1) Did you install ubuntu by way of Parallels, or are you running > linux "full steam" Yes, it is a full steam dual boot system. My default boot is Ubuntu. > 2) If you did install ubuntu alongside/over/whatever mac os x and > didn't use parallels .. are there any links you can share w/ a few > tutorials? Is everything (else: like hardware support) working well? > (Maybe off list since it's OT -- but maybe others are interested) Here are the references I used: http://bin-false.org/?p=17 http://www.planetisaac.com/articles/ubuntuinstall.html (The second one, which was an annotated version of the first one, seems to have disappeared.) No, not everything is working. And the installation is not easy (for me). I think it will become easy in a few months. But numpy and scipy were easy to compile! If you run into any problems, just send me a note off list. From david.huard at gmail.com Sat Jul 15 21:38:17 2006 From: david.huard at gmail.com (David Huard) Date: Sat, 15 Jul 2006 21:38:17 -0400 Subject: [Numpy-discussion] numpy.vectorize performance In-Reply-To: <432FD99A-ECC4-423C-B089-C5077D230A17@mit.edu> References: <44B6FECD.8000004@cox.net> <432FD99A-ECC4-423C-B089-C5077D230A17@mit.edu> Message-ID: <91cf711d0607151838p69149c73q5d479dbecaae78a0@mail.gmail.com> 2006/7/14, Nick Fotopoulos : > > Any other suggestions? > Hi Nick, I had some success by coding the integrand in fortran and wrapping it with f2py. If your probability density function is standard, you may find it in the flib library of the PyMC module of Chris Fonnesbeck ( a library of likelihood functions coded in f77) and save the trouble. Hope this helps, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Sun Jul 16 00:01:36 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 15 Jul 2006 22:01:36 -0600 Subject: [Numpy-discussion] numpy.vectorize performance In-Reply-To: References: Message-ID: <44B9BA20.7000600@ieee.org> Nick Fotopoulos wrote: > Dear all, > > I often make use of numpy.vectorize to make programs read more like > the physics equations I write on paper. numpy.vectorize is basically > a wrapper for numpy.frompyfunc. Reading Travis's Scipy Book (mine is > dated Jan 6 2005) kind of suggests to me that it returns a full- > fledged ufunc exactly like built-in ufuncs. > > First, is this true? Yes, it is true. But, it is a ufunc on Python object data-types. It is calling the underlying Python function at every point in the loop. > Second, how is the performance? i.e., are my > functions performing approximately as fast as they could be or would > they still gain a great deal of speed by rewriting it in C or some > other compiled python accelerator? > Absolutely the functions could be made faster to avoid the call back into Python at each evaluation stage. I don't think it would be too hard to replace the function-call with something else that could be evaluated more quickly. But, this has not been done yet. > As an aside, I've found the following function decorator to be > helpful for readability, and perhaps others will enjoy it or improve > upon it: > Thanks for the decorator. This should be put on the www.scipy.org wiki. -Travis From oliphant.travis at ieee.org Sun Jul 16 00:03:33 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 15 Jul 2006 22:03:33 -0600 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: <44B642A9.3090805@psychology.nottingham.ac.uk> References: <44B642A9.3090805@psychology.nottingham.ac.uk> Message-ID: <44B9BA95.9070908@ieee.org> Jon Peirce wrote: > There used to be a function generalized_inverse in the numpy.linalg > module (certainly in 0.9.2). > > In numpy0.9.8 it seems to have been moved to the numpy.linalg.old > subpackage. Does that mean it's being dropped? No. We are just emphasizing the new names. The old names are just there for compatibility with Numeric. The new names have been there from the beginning of NumPy releases. So, just call it numpy.linalg.pinv and it will work in all versions. -Travis From oliphant.travis at ieee.org Sun Jul 16 00:06:07 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 15 Jul 2006 22:06:07 -0600 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: <44B7BEB6.8070608@stsci.edu> References: <44B642A9.3090805@psychology.nottingham.ac.uk> <44B7489A.1050108@gmx.net> <44B7BEB6.8070608@stsci.edu> Message-ID: <44B9BB2F.3030605@ieee.org> Victoria G. Laidler wrote: > Sven Schreiber wrote: > > >> Jon Peirce schrieb: >> >> >> >>> There used to be a function generalized_inverse in the numpy.linalg >>> module (certainly in 0.9.2). >>> >>> In numpy0.9.8 it seems to have been moved to the numpy.linalg.old >>> subpackage. Does that mean it's being dropped? Did it have to move? Now >>> i have to add code to my package to try both locations because my users >>> might have any version... :-( >>> >>> >>> >>> >>> >> Maybe I don't understand, but what's wrong with numpy.linalg.pinv? >> >> >> > Er, what's a pinv? It doesn't sound anything like a generalized_inverse. > > 'pseudo'-inverse. It's the name MATLAB uses for the thing. There are many choices for a "generalized_inverse" which is actually a mis-nomer for what is being done. The Moore-Penrose pseudo-inverse is a particular form of the generalized_inverse (and the one being computed). -Travis From oliphant.travis at ieee.org Sun Jul 16 00:21:26 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat, 15 Jul 2006 22:21:26 -0600 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: <44B7C37A.7010801@stsci.edu> References: <44B642A9.3090805@psychology.nottingham.ac.uk> <44B7489A.1050108@gmx.net> <44B7BEB6.8070608@stsci.edu> <44B7BF75.2080708@stanford.edu> <44B7C37A.7010801@stsci.edu> Message-ID: <44B9BEC6.4050507@ieee.org> Victoria G. Laidler wrote: > Jonathan Taylor wrote: > > >> pseudoinverse >> >> it's the same name matlab uses: >> >> http://www.mathworks.com/access/helpdesk/help/techdoc/ref/pinv.html >> > > Thanks for the explanation. > > I'm puzzled by the naming choice, however. Standard best practice in > writing software is to give understandable names, to improve readability > and code maintenance. Obscure abbreviations like "pinv" pretty much went > out with the FORTRAN 9-character limit for variable names. It's very > unusual to see them in new software nowadays, and it always looks > unprofessional to me. > I appreciate this feedback. It's a question that comes up occasionally, so I'll at least give my opinion on the matter which may shed some light on it. I disagree with the general "long-name" concept when it comes to "very-common" operations. It's easy to take an idea and over-generalize it for the sake of consistency. I've seen too many codes where very long names actually get in the way of code readability. Someone reading code will have to know what an operation actually is to understand it. A name like "generalized_inverse" doesn't convey any intrinsic meaning to the non-practitioner anyway. You always have to "know" what the function is "really" doing. All that's needed is a "unique" name. I've found that long names are harder to remember (there's more opportunity for confusion about how much of the full name was actually used and how any words were combined). A particularly ludicrous case, for example, was the fact that the very common SVD (whose acronym everybody doing linear algebra uses) was named in LinearAlgebra (an unecessarily long module name to begin with) with the horribly long and unsightly name of singular_value_decomposition. I suppose this was done just for the sake of "code readability." It's not that we're concerned with MATLAB compatibility. But, frankly I've never heard that the short names MATLAB uses for some very common operations are a liability. So, when a common operation has a short, easily-remembered name that is in common usage, why not use it? That's basically the underlying philosophy. NumPy has too many very basic operations to try and create very_long_names for them. I know there are differing opinions out there. I can understand that. That's why I suspect that many codes I will want to use will be written with easy_to_understand_but_very_long names and I'll grin and bear the extra horizontal space that it takes up in my code. -Travis From cookedm at physics.mcmaster.ca Sun Jul 16 04:26:49 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Sun, 16 Jul 2006 04:26:49 -0400 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: <44B9BEC6.4050507@ieee.org> References: <44B642A9.3090805@psychology.nottingham.ac.uk> <44B7489A.1050108@gmx.net> <44B7BEB6.8070608@stsci.edu> <44B7BF75.2080708@stanford.edu> <44B7C37A.7010801@stsci.edu> <44B9BEC6.4050507@ieee.org> Message-ID: <95E43A43-20D0-4F19-A1AA-DDAD52727F92@physics.mcmaster.ca> On Jul 16, 2006, at 00:21 , Travis Oliphant wrote: > Victoria G. Laidler wrote: >> Jonathan Taylor wrote: >> >>> pseudoinverse >>> >>> it's the same name matlab uses: >>> >>> http://www.mathworks.com/access/helpdesk/help/techdoc/ref/pinv.html >>> >> >> Thanks for the explanation. >> >> I'm puzzled by the naming choice, however. Standard best practice in >> writing software is to give understandable names, to improve >> readability >> and code maintenance. Obscure abbreviations like "pinv" pretty >> much went >> out with the FORTRAN 9-character limit for variable names. It's very >> unusual to see them in new software nowadays, and it always looks >> unprofessional to me. >> > I appreciate this feedback. It's a question that comes up > occasionally, > so I'll at least give my opinion on the matter which may shed some > light > on it. > > I disagree with the general "long-name" concept when it comes to > "very-common" operations. It's easy to take an idea and > over-generalize it for the sake of consistency. I've seen too many > codes where very long names actually get in the way of code > readability. How are pseudoinverse and inverse "very common"? (Especially given that one of the arguments for not having a .I attribute for inverse on matrices is that that's usually the wrong way to go about solving equations.) > Someone reading code will have to know what an operation actually > is to > understand it. A name like "generalized_inverse" doesn't convey any > intrinsic meaning to the non-practitioner anyway. You always have to > "know" what the function is "really" doing. All that's needed is a > "unique" name. I've found that long names are harder to remember > (there's more opportunity for confusion about how much of the full > name > was actually used and how any words were combined). As has been argued before, short names have their own problems with remembering what they are. I also find that when reading code with short names, I go slower, because I have to stop and think what that short name is (particularly bad are short names that drop vowels, like lstsq -- I can't pronounce that!). I'm not very good at creating hash tables in my head from short names to long ones. The currently exported names in numpy.linalg are solve, inv, cholesky, eigvals, eigvalsh, eig, eigh, svd, pinv, det, lstsq, and norm. Of these, 'lstsq' is the worst offender, IMHO (superfluous dropped vowels). 'inv' and 'pinv' are the next, then the 'eig*' names. 'least_squares' would be better than 'lstsq'. 'inverse' is not much longer than 'inv', and is more descriptive. I don't think 'pinv' is that common to need a short name; 'pseudoinverse' would be better (not all generalized inverses are pseudoinverses). Give me these three and I'll be happy :-) Personally, I'd prefer 'eigenvalues' and 'eigen' instead of 'eigvals' and 'eig', but I can live with the current names. 'det' is fine, as it's used in mathematical notation. 'cholesky' is also fine, as it's a word at least. I'd have to look at the docstring to find how to use it, but that would be the same for "cholesky_decomposition". [btw, I'm ok with numpy.dft now: the names there make sense, because they're constructed logically. Once you know the scheme, you can see right away that 'irfftn' is 'inverse real FFT, n-dimensional'.] > > A particularly ludicrous case, for example, was the fact that the very > common SVD (whose acronym everybody doing linear algebra uses) was > named > in LinearAlgebra (an unecessarily long module name to begin with) with > the horribly long and unsightly name of singular_value_decomposition. > I suppose this was done just for the sake of "code readability." I agree; that's stupid. > It's not that we're concerned with MATLAB compatibility. But, frankly > I've never heard that the short names MATLAB uses for some very common > operations are a liability. So, when a common operation has a short, > easily-remembered name that is in common usage, why not use it? > > That's basically the underlying philosophy. NumPy has too many very > basic operations to try and create very_long_names for them. > > I know there are differing opinions out there. I can understand > that. > That's why I suspect that many codes I will want to use will be > written > with easy_to_understand_but_very_long names and I'll grin and bear the > extra horizontal space that it takes up in my code. -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From charlesr.harris at gmail.com Sun Jul 16 11:23:36 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 16 Jul 2006 09:23:36 -0600 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: <44B9BEC6.4050507@ieee.org> References: <44B642A9.3090805@psychology.nottingham.ac.uk> <44B7489A.1050108@gmx.net> <44B7BEB6.8070608@stsci.edu> <44B7BF75.2080708@stanford.edu> <44B7C37A.7010801@stsci.edu> <44B9BEC6.4050507@ieee.org> Message-ID: On 7/15/06, Travis Oliphant wrote: > > Victoria G. Laidler wrote: > > Jonathan Taylor wrote: > > It's not that we're concerned with MATLAB compatibility. But, frankly > I've never heard that the short names MATLAB uses for some very common > operations are a liability. So, when a common operation has a short, > easily-remembered name that is in common usage, why not use it? > > That's basically the underlying philosophy. NumPy has too many very > basic operations to try and create very_long_names for them. > > I know there are differing opinions out there. I can understand that. > That's why I suspect that many codes I will want to use will be written > with easy_to_understand_but_very_long names and I'll grin and bear the > extra horizontal space that it takes up in my code. What is needed in the end is a good index with lots of crossreferences. Name choices are just choices, there is no iso standard for function names that I know of. There are short names have been used for so long that everyone knows them (sin, cos, ...), some names come in two standard forms (arcsin, asin) some are fortran conventions (arctan2), some are matlab conventions (pinv, chol). One always has to learn what the names for things are in any new language, so the best thing is to make it easy to find out. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Sun Jul 16 11:47:08 2006 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 16 Jul 2006 11:47:08 -0400 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: <95E43A43-20D0-4F19-A1AA-DDAD52727F92@physics.mcmaster.ca> References: <44B642A9.3090805@psychology.nottingham.ac.uk> <44B7489A.1050108@gmx.net> <44B7BEB6.8070608@stsci.edu><44B7BF75.2080708@stanford.edu> <44B7C37A.7010801@stsci.edu><44B9BEC6.4050507@ieee.org><95E43A43-20D0-4F19-A1AA-DDAD52727F92@physics.mcmaster.ca> Message-ID: On Sun, 16 Jul 2006, "David M. Cooke" apparently wrote: > 'inverse' is not much longer than 'inv', and is more descriptive But 'inv' is quite universal (can you name a matrix language that uses 'inverse' instead?) and I think unambiguous (what might it be confused with?). Cheers, Alan Isaac From aisaac at american.edu Sun Jul 16 11:47:07 2006 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 16 Jul 2006 11:47:07 -0400 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: References: <44B642A9.3090805@psychology.nottingham.ac.uk><44B7489A.1050108@gmx.net> <44B7BEB6.8070608@stsci.edu><44B7BF75.2080708@stanford.edu> <44B7C37A.7010801@stsci.edu><44B9BEC6.4050507@ieee.org> Message-ID: On Sun, 16 Jul 2006, Charles R Harris apparently wrote: > What is needed in the end is a good index with lots of > crossreferences. Name choices are just choices I mostly agree with this (although I think Matlab made some bad choices in naming). As a point of reference for a useful index see http://www.mathworks.com/access/helpdesk/help/techdoc/ref/refbookl.html Cheers, Alan Isaac From josh.p.marshall at gmail.com Sun Jul 16 20:22:07 2006 From: josh.p.marshall at gmail.com (Josh Marshall) Date: Mon, 17 Jul 2006 10:22:07 +1000 Subject: [Numpy-discussion] Packaging numpy with py2app Message-ID: <660C6421-8C71-4B4D-A2CC-33C3FF089DF9@gmail.com> Back in December last year, I was building a PyObjC application that embedded numpy (scipy_core at the time), scipy and matplotlib. I ran into a few issues doing so, some of which were resolved. One was the inability for scipy to run from a zipped site-packages. I worked around this by expanding the embedded site-packages.zip into a site- packages directory in the same location. For reference, the thread can be found at: http://www.scipy.net/pipermail/scipy-dev/2005-December/004551.html Come a few months later, I have needed to update to the latest version of numpy (and therefore scipy and matplotlib). I have not yet updated to the universal build of Python, still running 2.4.1, although I will do so if it is known to fix any issues. (I don't have too much time at the moment, and building the latest versions of numpy and matplotlib for a universal build scares me). I managed to get it working again, which required: 1) Setting packages=['matplotlib','numpy'] in setup.py's options for py2app. 2) Modifying the py2app/apptemplate/lib/site.py file to include 'sys.path.append(_parent + '/site-packages')' before the same line with .zip appended to the file name. 3) Adding setup_options['options']['py2app']['includes'].extend (['pytz.zoneinfo.UTC']) to the setup.py, this is required by matplotlib. I believe (2) is a bug in py2app (I am running 0.3.1). Packages included using 'packages=' are not added to site-packages.zip, but rather are in their own site-packages directory. I am not sure whether this is the intended behaviour or a bug, but it is good for me, since numpy and matplotlib won't run when compressed. Now, it seems I am doomed to continue to have to find work-arounds to get numpy and matplotlib working in a standalone .app. Is there a chance we can come up with a py2app recipe for numpy, matplotlib and scipy? What other alternatives are there? Regards, Josh From bob at redivi.com Sun Jul 16 20:46:06 2006 From: bob at redivi.com (Bob Ippolito) Date: Sun, 16 Jul 2006 17:46:06 -0700 Subject: [Numpy-discussion] [Pythonmac-SIG] Packaging numpy with py2app In-Reply-To: <660C6421-8C71-4B4D-A2CC-33C3FF089DF9@gmail.com> References: <660C6421-8C71-4B4D-A2CC-33C3FF089DF9@gmail.com> Message-ID: On Jul 16, 2006, at 5:22 PM, Josh Marshall wrote: > Back in December last year, I was building a PyObjC application that > embedded numpy (scipy_core at the time), scipy and matplotlib. I ran > into a few issues doing so, some of which were resolved. One was the > inability for scipy to run from a zipped site-packages. I worked > around this by expanding the embedded site-packages.zip into a site- > packages directory in the same location. For reference, the thread > can be found at: > http://www.scipy.net/pipermail/scipy-dev/2005-December/004551.html > > Come a few months later, I have needed to update to the latest > version of numpy (and therefore scipy and matplotlib). I have not yet > updated to the universal build of Python, still running 2.4.1, > although I will do so if it is known to fix any issues. (I don't have > too much time at the moment, and building the latest versions of > numpy and matplotlib for a universal build scares me). If you're still using 2.4.1 you're going to want to add LSPrefersPPC=True to the plist. Otherwise your app will not run on i386, because it will not know to use Rosetta. > I managed to get it working again, which required: > 1) Setting packages=['matplotlib','numpy'] in setup.py's options for > py2app. That could be handled with recipes, or with eggs when py2app grows support for that. Recipes or workarounds are the only way until then. > 2) Modifying the py2app/apptemplate/lib/site.py file to include > 'sys.path.append(_parent + '/site-packages')' before the same line > with .zip appended to the file name. That's a bug, but not the right fix. Trunk is fixed. > 3) Adding setup_options['options']['py2app']['includes'].extend > (['pytz.zoneinfo.UTC']) to the setup.py, this is required by > matplotlib. That should be part of a matplotlib recipe. > Now, it seems I am doomed to continue to have to find work-arounds to > get numpy and matplotlib working in a standalone .app. Is there a > chance we can come up with a py2app recipe for numpy, matplotlib and > scipy? What other alternatives are there? Sure, someone can write the recipes and send me a patch. I don't currently use matplotlib, numpy, or scipy nor do I have an example I can test with so I'm not going to do it. -bob From rlw at stsci.edu Sun Jul 16 21:25:24 2006 From: rlw at stsci.edu (Rick White) Date: Sun, 16 Jul 2006 21:25:24 -0400 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: References: <44B642A9.3090805@psychology.nottingham.ac.uk> <44B7489A.1050108@gmx.net> <44B7BEB6.8070608@stsci.edu><44B7BF75.2080708@stanford.edu> <44B7C37A.7010801@stsci.edu><44B9BEC6.4050507@ieee.org><95E43A43-20D0-4F19-A1AA-DDAD52727F92@physics.mcmaster.ca> Message-ID: <3A6EC819-DED9-4C23-80E5-3BBDE0859EBF@stsci.edu> On Jul 16, 2006, at 11:47 AM, Alan G Isaac wrote: > On Sun, 16 Jul 2006, "David M. Cooke" apparently wrote: >> 'inverse' is not much longer than 'inv', and is more descriptive > > But 'inv' is quite universal (can you name a matrix language > that uses 'inverse' instead?) and I think unambiguous (what > might it be confused with?). IDL uses invert, so inv is not exactly universal. I'm personally a fan of names that can be used in interactive sessions at the command line, which argues for shorter names. But it is nice to have names where you can type just a few characters and use tab-completion to fill in the rest of the name. Then the important thing is not the full length of the name but having the first 3 or 4 characters be memorable. So I'd rather have "pseudoinverse" because I can probably find it by just typing "ps". From kwgoodman at gmail.com Sun Jul 16 21:39:19 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Sun, 16 Jul 2006 18:39:19 -0700 Subject: [Numpy-discussion] generalized_inverse In-Reply-To: <3A6EC819-DED9-4C23-80E5-3BBDE0859EBF@stsci.edu> References: <44B642A9.3090805@psychology.nottingham.ac.uk> <44B7489A.1050108@gmx.net> <44B7BEB6.8070608@stsci.edu> <44B7BF75.2080708@stanford.edu> <44B7C37A.7010801@stsci.edu> <44B9BEC6.4050507@ieee.org> <95E43A43-20D0-4F19-A1AA-DDAD52727F92@physics.mcmaster.ca> <3A6EC819-DED9-4C23-80E5-3BBDE0859EBF@stsci.edu> Message-ID: On 7/16/06, Rick White wrote: > On Jul 16, 2006, at 11:47 AM, Alan G Isaac wrote: > > > On Sun, 16 Jul 2006, "David M. Cooke" apparently wrote: > >> 'inverse' is not much longer than 'inv', and is more descriptive > > > > But 'inv' is quite universal (can you name a matrix language > > that uses 'inverse' instead?) and I think unambiguous (what > > might it be confused with?). > > IDL uses invert, so inv is not exactly universal. > > I'm personally a fan of names that can be used in interactive > sessions at the command line, which argues for shorter names. But it > is nice to have names where you can type just a few characters and > use tab-completion to fill in the rest of the name. Then the > important thing is not the full length of the name but having the > first 3 or 4 characters be memorable. So I'd rather have > "pseudoinverse" because I can probably find it by just typing "ps". I prfr shrtr nams lke inv eig and sin. From david.huard at gmail.com Mon Jul 17 10:10:55 2006 From: david.huard at gmail.com (David Huard) Date: Mon, 17 Jul 2006 10:10:55 -0400 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <91cf711d0607111127w21bb4b3cj30a3c31b46e136f0@mail.gmail.com> References: <91cf711d0606220926m48c6857cr78b4484f4a137a2@mail.gmail.com> <44A7961F.9030907@gmx.de> <44A8E4D3.5000603@ntc.zcu.cz> <91cf711d0607031000l286da52fn30d98798057f12c5@mail.gmail.com> <44B3BD22.70003@gmx.de> <44B3D6A5.6060404@ntc.zcu.cz> <91cf711d0607111127w21bb4b3cj30a3c31b46e136f0@mail.gmail.com> Message-ID: <91cf711d0607170710q63ef1114rc55240741f2dd1f1@mail.gmail.com> Hi, I attached a patch for unique (with a test case) based on Norbert's suggestion. I removed the sort keyword since iterable arrays would be sorted anyway. The function uses a python set, and I was wondering if it is ok to assume that everyone running numpy has a python version >= 2.3 ? David -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: unique.patch Type: text/x-patch Size: 2146 bytes Desc: not available URL: From fullung at gmail.com Mon Jul 17 10:59:36 2006 From: fullung at gmail.com (Albert Strasheim) Date: Mon, 17 Jul 2006 16:59:36 +0200 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <91cf711d0607170710q63ef1114rc55240741f2dd1f1@mail.gmail.com> Message-ID: <027701c6a9b1$9e9b6f10$0100000a@dsp.sun.ac.za> Hey David > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of David Huard > Sent: 17 July 2006 16:11 > To: numpy-discussion at lists.sourceforge.net > Subject: Re: [Numpy-discussion] unique() should return a sorted array > > Hi, > > I attached a patch for unique (with a test case) based on Norbert's > suggestion. I removed the sort keyword since iterable arrays would be > sorted anyway. The function uses a python set, and I was wondering if it > is ok to assume that everyone running numpy has a python version >= 2.3 ? I think it has been discussed on the list that Python >= 2.3 is assumed. However, according to the Python documentation, the built-in set type is new in Python 2.4. Regards, Albert From jg307 at cam.ac.uk Mon Jul 17 11:16:42 2006 From: jg307 at cam.ac.uk (James Graham) Date: Mon, 17 Jul 2006 16:16:42 +0100 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <027701c6a9b1$9e9b6f10$0100000a@dsp.sun.ac.za> References: <027701c6a9b1$9e9b6f10$0100000a@dsp.sun.ac.za> Message-ID: <44BBA9DA.9050008@cam.ac.uk> Albert Strasheim wrote: > I think it has been discussed on the list that Python >= 2.3 is assumed. > However, according to the Python documentation, the built-in set type is new > in Python 2.4. Sets are available in python 2.3 as part of the sets module so it is possible. However afaict the provided patch does not use the module and so will need to be adapted for use in 2.3. -- "You see stars that clear have been dead for years But the idea just lives on..." -- Bright Eyes From wright at esrf.fr Mon Jul 17 11:19:48 2006 From: wright at esrf.fr (Jon Wright) Date: Mon, 17 Jul 2006 17:19:48 +0200 Subject: [Numpy-discussion] generalized_inverse and documentation in general In-Reply-To: References: <44B642A9.3090805@psychology.nottingham.ac.uk> <44B7489A.1050108@gmx.net> <44B7BEB6.8070608@stsci.edu> <44B7BF75.2080708@stanford.edu> <44B7C37A.7010801@stsci.edu> <44B9BEC6.4050507@ieee.org> <95E43A43-20D0-4F19-A1AA-DDAD52727F92@physics.mcmaster.ca> <3A6EC819-DED9-4C23-80E5-3BBDE0859EBF@stsci.edu> Message-ID: <44BBAA94.8010009@esrf.fr> Keith Goodman wrote: >I prfr shrtr nams lke inv eig and sin. > > Dwn wth vwls!! Srsly thgh: >>> from numpy import linalg >>> help(linalg.inv) Help on function inv in module numpy.linalg.linalg: inv(a) >>> ??? While I prefer inverse to inv, I don't really care as long as the word "inverse" appears in the docstring and it faithfully promises that it is going to try to do an inverse in the matrix linear algebra sense and not in the one_over_x or power_minus_one sense or bit inversion (~ x, numpy.invert) sense. It would be handy to know which exception which will be raised for a singular matrix, what are valid arguments (eg, square shape, numbers), what is the type of the output (' testnumpywiki.txt ==testwiki.py:== import doctest doctest.testfile("testnumpywiki.txt") $ python testwiki.py > problems.txt problems.txt is 37kB in size (83 of 1028) throwing out the blanklines issues via: doctest.testfile("testnumpywiki.txt", optionflags=doctest.NORMALIZE_WHITESPACE) reduces this to 24kB (62 of 1028). ... most cases are not important, just needing to be fixed for formatting on the wiki or flagged as version dependent, but a few are worth checking out the intentions, eg: ********************************************************************** File "testnumpywiki.txt", line 69, in testnumpywiki.txt Failed example: a[:,b2] Exception raised: Traceback (most recent call last): File "c:\python24\lib\doctest.py", line 1243, in __run compileflags, 1) in test.globs File "", line 1, in ? a[:,b2] IndexError: arrays used as indices must be of integer type ********************************************************************** File "testnumpywiki.txt", line 893, in testnumpywiki.txt Failed example: ceil(a) # nearest integers greater-than or equal to a Expected: array([-1., -1., -0., 1., 2., 2.]) Got: array([-1., -1., 0., 1., 2., 2.]) ********************************************************************** File "testnumpywiki.txt", line 1162, in testnumpywiki.txt Failed example: cov(T,P) # covariance between temperature and pressure Expected: 3.9541666666666657 Got: array([[ 1.97583333, 3.95416667], [ 3.95416667, 8.22916667]]) ********************************************************************** File "testnumpywiki.txt", line 2235, in testnumpywiki.txt Failed example: type(a[0]) Expected: Got: From skip at pobox.com Mon Jul 17 11:32:34 2006 From: skip at pobox.com (skip at pobox.com) Date: Mon, 17 Jul 2006 10:32:34 -0500 Subject: [Numpy-discussion] unique() should return a sorted array In-Reply-To: <44BBA9DA.9050008@cam.ac.uk> References: <027701c6a9b1$9e9b6f10$0100000a@dsp.sun.ac.za> <44BBA9DA.9050008@cam.ac.uk> Message-ID: <17595.44434.740132.140969@montanaro.dyndns.org> James> Sets are available in python 2.3 as part of the sets module so it James> is possible. However afaict the provided patch does not use the James> module and so will need to be adapted for use in 2.3. I got so tired of the ugly test for the set builtin during our 2.3-to-2.4 transition that I finally just added if sys.hexversion < 0x2040000: from sets import Set as set import __builtin__ __builtin__.set = set to sitecustomize.py. I'm not suggesting that scipy's installer should do the same, but the change worked for us. Skip From haase at msg.ucsf.edu Mon Jul 17 12:40:48 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 17 Jul 2006 09:40:48 -0700 Subject: [Numpy-discussion] numarray bug: dot product between 2x2 and 3x2x3 on Mac different from PC Message-ID: <200607170940.48872.haase@msg.ucsf.edu> Hi! This is what I got: >>> import numarray as na >>> na.__version__ '1.4.0' >>>bbb=na.zeros((2,3,3), naFloat32) >>>bbb[0,:,:]=1 >>>bbb[1,:,:]=2 >>>ccc=na.transpose(bbb,(1,0,2)) >>>d=na.array([[1,0],[0,1]]) >>>na.dot(d,ccc) [[[ 1. ?1. ?1.] ? [ 1. ?1. ?1.] ? [ 1. ?1. ?1.]] ?[[ 2. ?2. ?2.] ? [ 2. ?2. ?2.] ? [ 2. ?2. ?2.]]] This is on a PC (Windows or Linux): But if you do the same thing on Mac, the result will be: [[[ 1. 1. 1.] ? [ 2. 2. 2.] ? [ 1. 1. 1.]] ?[[ 2. 2. 2.] ? [ 1. 1. 1.] ? [ 2. 2. 2.]]] Can someone confirm this ? (the new numpy on windows also gives the first result) Thanks, Sebastian Haase From Chris.Barker at noaa.gov Mon Jul 17 14:24:34 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 17 Jul 2006 11:24:34 -0700 Subject: [Numpy-discussion] numarray bug: dot product between 2x2 and 3x2x3 on Mac different from PC In-Reply-To: <200607170940.48872.haase@msg.ucsf.edu> References: <200607170940.48872.haase@msg.ucsf.edu> Message-ID: <44BBD5E2.5080703@noaa.gov> Sebastian Haase wrote: > Hi! > This is what I got: > Can someone confirm this ? This is what I get on Linux (ix86, Fedora core4, python 2.4.3) >>> import numarray as na >>> na.__version__ '1.5.0' >>> bbb=na.zeros((2,3,3), na.Float32) >>> bbb[0,:,:]=1 >>> bbb[1,:,:]=2 >>> bbb array([[[ 1., 1., 1.], [ 1., 1., 1.], [ 1., 1., 1.]], [[ 2., 2., 2.], [ 2., 2., 2.], [ 2., 2., 2.]]], type=Float32) >>> ccc=na.transpose(bbb,(1,0,2)) >>> ccc array([[[ 1., 1., 1.], [ 2., 2., 2.]], [[ 1., 1., 1.], [ 2., 2., 2.]], [[ 1., 1., 1.], [ 2., 2., 2.]]], type=Float32) >>> d=na.array([[1,0],[0,1]]) >>> d array([[1, 0], [0, 1]]) >>> na.dot(d,ccc) array([[[ 1., 1., 1.], [ 2., 2., 2.], [ 1., 1., 1.]], [[ 2., 2., 2.], [ 1., 1., 1.], [ 2., 2., 2.]]], type=Float32) >>> -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Mon Jul 17 14:29:46 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 17 Jul 2006 11:29:46 -0700 Subject: [Numpy-discussion] numarray bug: dot product between 2x2 and 3x2x3 on Mac different from PC In-Reply-To: <200607170940.48872.haase@msg.ucsf.edu> References: <200607170940.48872.haase@msg.ucsf.edu> Message-ID: <44BBD71A.1050400@noaa.gov> And on my Mac: OS-X 10.4.*, PPC, universal python 2.4.3: >>> import numarray as na >>> na.__version__ '1.5.1' >>> bbb=na.zeros((2,3,3), na.Float32) >>> bbb[0,:,:]=1 >>> bbb[1,:,:]=2 >>> ccc=na.transpose(bbb,(1,0,2)) >>> d=na.array([[1,0],[0,1]]) >>> na.dot(d,ccc) array([[[ 1., 1., 1.], [ 2., 2., 2.], [ 1., 1., 1.]], [[ 2., 2., 2.], [ 1., 1., 1.], [ 2., 2., 2.]]], type=Float32) -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From fperez.net at gmail.com Mon Jul 17 14:31:50 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 17 Jul 2006 12:31:50 -0600 Subject: [Numpy-discussion] 64-bit build woes Message-ID: Hi all, I received this message today from a collaborator. I don't have direct access to this box, but he posted fairly detailed logs. Has anyone seen a similar issue with current code? If not, I'll try to track down further with him what the problem may actually be. Thanks for any help, f ============ Original message: We are having trouble with scipy (not)compiling on the 64-bit machine, and it seems to be related to the intp type. I put the log file and connected files at http://www.math.ohiou.edu/~mjm/pickup/scipy.log http://www.math.ohiou.edu/~mjm/pickup/fortranobject.c http://www.math.ohiou.edu/~mjm/pickup/fortranobject.h The relevant parts are at the end of scipy.log: creating build/temp.linux-x86_64-2.3/Lib/fftpack/src compile options: '-Ibuild/src.linux-x86_64-2.3 -I/usr/lib64/python2.3/site-packages/numpy/core/include -I/usr/include/python2.3 -c' gcc: Lib/fftpack/src/drfft.c gcc: Lib/fftpack/src/zfft.c gcc: build/src.linux-x86_64-2.3/fortranobject.c build/src.linux-x86_64-2.3/fortranobject.c: In function `PyFortranObject_New': build/src.linux-x86_64-2.3/fortranobject.c:55: error: syntax error before "intp" build/src.linux-x86_64-2.3/fortranobject.c:60: error: syntax error before "intp" build/src.linux-x86_64-2.3/fortranobject.c: In function `fortran_getattr': build/src.linux-x86_64-2.3/fortranobject.c:179: error: syntax error before "intp" build/src.linux-x86_64-2.3/fortranobject.c: In function `fortran_setattr': build/src.linux-x86_64-2.3/fortranobject.c:243: error: syntax error before ')' token build/src.linux-x86_64-2.3/fortranobject.c:245: error: syntax error before ')' token ... I don't see "intp" in fortranobject.c, so it must be included from elsewhere. From haase at msg.ucsf.edu Mon Jul 17 14:40:43 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 17 Jul 2006 11:40:43 -0700 Subject: [Numpy-discussion] numarray bug: dot product between 2x2 and 3x2x3 on Mac different from PC In-Reply-To: <44BBD71A.1050400@noaa.gov> References: <200607170940.48872.haase@msg.ucsf.edu> <44BBD71A.1050400@noaa.gov> Message-ID: <200607171140.43249.haase@msg.ucsf.edu> Thanks Chris, If this got fixed towards numarray version 1.5.1 it looks like it is now on both Linux and Mac different(!) from the numpy result I got on Window !! On Linux I get: >>> import numpy as N >>> N.__version__ '0.9.9.2823' >>> bbb=N.zeros((2,3,3), N.float32) >>> bbb[0,:,:]=1 >>> bbb[1,:,:]=2 >>> bbb [[[ 1. 1. 1.] [ 1. 1. 1.] [ 1. 1. 1.]] [[ 2. 2. 2.] [ 2. 2. 2.] [ 2. 2. 2.]]] >>> ccc=N.transpose(bbb,(1,0,2)) >>> d=N.array([[1,0],[0,1]]) >>> d [[1 0] [0 1]] >>> N.dot(d,ccc) Traceback (most recent call last): File "", line 1, in ? TypeError: array cannot be safely cast to required type >>> dd=d.astype(N.float32) >>> N.dot(dd,ccc) [[[ 1. 1. 1.] [ 1. 1. 1.] [ 1. 1. 1.]] [[ 2. 2. 2.] [ 2. 2. 2.] [ 2. 2. 2.]]] >>> The TypeError looks like a numpy bug ! Thanks, Sebastian Haase On Monday 17 July 2006 11:29, Christopher Barker wrote: > And on my Mac: > > OS-X 10.4.*, PPC, universal python 2.4.3: > >>> import numarray as na > >>> na.__version__ > > '1.5.1' > > >>> bbb=na.zeros((2,3,3), na.Float32) > >>> bbb[0,:,:]=1 > >>> bbb[1,:,:]=2 > >>> ccc=na.transpose(bbb,(1,0,2)) > >>> d=na.array([[1,0],[0,1]]) > >>> na.dot(d,ccc) > > array([[[ 1., 1., 1.], > [ 2., 2., 2.], > [ 1., 1., 1.]], > > [[ 2., 2., 2.], > [ 1., 1., 1.], > [ 2., 2., 2.]]], type=Float32) From oliphant.travis at ieee.org Mon Jul 17 15:10:42 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 17 Jul 2006 13:10:42 -0600 Subject: [Numpy-discussion] numarray bug: dot product between 2x2 and 3x2x3 on Mac different from PC In-Reply-To: <200607171140.43249.haase@msg.ucsf.edu> References: <200607170940.48872.haase@msg.ucsf.edu> <44BBD71A.1050400@noaa.gov> <200607171140.43249.haase@msg.ucsf.edu> Message-ID: <44BBE0B2.4090002@ieee.org> Sebastian Haase wrote: > Traceback (most recent call last): > File "", line 1, in ? > TypeError: array cannot be safely cast to required type > >>>> dd=d.astype(N.float32) >>>> N.dot(dd,ccc) >>>> > [[[ 1. 1. 1.] > [ 1. 1. 1.] > [ 1. 1. 1.]] > > [[ 2. 2. 2.] > [ 2. 2. 2.] > [ 2. 2. 2.]]] > > > > The TypeError looks like a numpy bug ! > I don't see why this is a bug. You are trying to coerce a 32-bit integer to a 32-bit float. That is going to lose precision and so you get the error indicated. -Travis From oliphant.travis at ieee.org Mon Jul 17 15:20:12 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 17 Jul 2006 13:20:12 -0600 Subject: [Numpy-discussion] 64-bit build woes In-Reply-To: References: Message-ID: <44BBE2EC.7000308@ieee.org> Fernando Perez wrote: > Hi all, > > I received this message today from a collaborator. I don't have > direct access to this box, but he posted fairly detailed logs. Has > anyone seen a similar issue with current code? If not, I'll try to > track down further with him what the problem may actually be. > This looks like a problem with left-over headers and/or C-API files being picked up. Make sure the old header files are deleted and he has a fresh install of NumPy from SVN (with the build directory deleted before re-building). Look in /usr/lib64/python2.3/site-packages/numpy/core/include/__multiarray_api.h to make sure there are now isolated intp references (particularly look at PyArray_New). If there are, then the NumPy build was not clean. -Travis From fperez.net at gmail.com Mon Jul 17 15:23:50 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 17 Jul 2006 13:23:50 -0600 Subject: [Numpy-discussion] 64-bit build woes In-Reply-To: <44BBE2EC.7000308@ieee.org> References: <44BBE2EC.7000308@ieee.org> Message-ID: On 7/17/06, Travis Oliphant wrote: > Fernando Perez wrote: > > Hi all, > > > > I received this message today from a collaborator. I don't have > > direct access to this box, but he posted fairly detailed logs. Has > > anyone seen a similar issue with current code? If not, I'll try to > > track down further with him what the problem may actually be. > > > > This looks like a problem with left-over headers and/or C-API files > being picked up. Make sure the old header files are deleted and he has > a fresh install of NumPy from SVN (with the build directory deleted > before re-building). > > Look in > > /usr/lib64/python2.3/site-packages/numpy/core/include/__multiarray_api.h > > to make sure there are now isolated intp references (particularly look at PyArray_New). > If there are, then the NumPy build was not clean. Thanks, Travis. I just wanted to make sure it wasn't a more widespread problem. I'll track it down with my colleague in private then. Cheers, f From haase at msg.ucsf.edu Mon Jul 17 15:26:33 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 17 Jul 2006 12:26:33 -0700 Subject: [Numpy-discussion] numarray bug: dot product between 2x2 and 3x2x3 on Mac different from PC In-Reply-To: <44BBE0B2.4090002@ieee.org> References: <200607170940.48872.haase@msg.ucsf.edu> <200607171140.43249.haase@msg.ucsf.edu> <44BBE0B2.4090002@ieee.org> Message-ID: <200607171226.33383.haase@msg.ucsf.edu> On Monday 17 July 2006 12:10, Travis Oliphant wrote: > Sebastian Haase wrote: > > Traceback (most recent call last): > > File "", line 1, in ? > > TypeError: array cannot be safely cast to required type > > > >>>> dd=d.astype(N.float32) > >>>> N.dot(dd,ccc) > > > > [[[ 1. 1. 1.] > > [ 1. 1. 1.] > > [ 1. 1. 1.]] > > > > [[ 2. 2. 2.] > > [ 2. 2. 2.] > > [ 2. 2. 2.]]] > > > > > > > > The TypeError looks like a numpy bug ! > > I don't see why this is a bug. You are trying to coerce a 32-bit > integer to a 32-bit float. That is going to lose precision and so you > get the error indicated. > > -Travis In numarray I do not get an error. Would the error go away if I had 64 bit float !? It seems though that having ones and twos in an int array should fit just fine into a float32 array !? - Sebastian Haase From oliphant.travis at ieee.org Mon Jul 17 15:38:29 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 17 Jul 2006 13:38:29 -0600 Subject: [Numpy-discussion] numarray bug: dot product between 2x2 and 3x2x3 on Mac different from PC In-Reply-To: <200607171226.33383.haase@msg.ucsf.edu> References: <200607170940.48872.haase@msg.ucsf.edu> <200607171140.43249.haase@msg.ucsf.edu> <44BBE0B2.4090002@ieee.org> <200607171226.33383.haase@msg.ucsf.edu> Message-ID: <44BBE735.5010109@ieee.org> Sebastian Haase wrote: > On Monday 17 July 2006 12:10, Travis Oliphant wrote: > >> Sebastian Haase wrote: >> >>> Traceback (most recent call last): >>> File "", line 1, in ? >>> TypeError: array cannot be safely cast to required type >>> >>> >>>>>> dd=d.astype(N.float32) >>>>>> N.dot(dd,ccc) >>>>>> >>> [[[ 1. 1. 1.] >>> [ 1. 1. 1.] >>> [ 1. 1. 1.]] >>> >>> [[ 2. 2. 2.] >>> [ 2. 2. 2.] >>> [ 2. 2. 2.]]] >>> >>> >>> >>> The TypeError looks like a numpy bug ! >>> >> I don't see why this is a bug. You are trying to coerce a 32-bit >> integer to a 32-bit float. That is going to lose precision and so you >> get the error indicated. >> >> -Travis >> > In numarray I do not get an error. Would the error go away if I had 64 bit > float !? It seems though that having ones and twos in an int array should > fit just fine into a float32 array !? > This could be considered a bug in numarray. It's force-casting the result. That isn't the normal behavior of mixed-type functions. Also, the policy on type-casting is not to search the array to see if it's possible to do the conversion on every element (that would be slow on large arrays). The policy is to base the decision only on the data-types themselves (i.e. whether it's *possible* to lose precision*). -Travis *There is one exception to this policy in NumPy: 64-bit integers are allowed to be cast to 64-bit doubles --- other-wise on you would get a lot of non-standard long-doubles showing up on 64-bit systems. This policy was decided after discussion last year. From haase at msg.ucsf.edu Mon Jul 17 16:48:05 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 17 Jul 2006 13:48:05 -0700 Subject: [Numpy-discussion] numarray bug: dot product between 2x2 and 3x2x3 on Mac different from PC In-Reply-To: <44BBE735.5010109@ieee.org> References: <200607170940.48872.haase@msg.ucsf.edu> <200607171226.33383.haase@msg.ucsf.edu> <44BBE735.5010109@ieee.org> Message-ID: <200607171348.05776.haase@msg.ucsf.edu> On Monday 17 July 2006 12:38, Travis Oliphant wrote: > Sebastian Haase wrote: > > On Monday 17 July 2006 12:10, Travis Oliphant wrote: > >> Sebastian Haase wrote: > >>> Traceback (most recent call last): > >>> File "", line 1, in ? > >>> TypeError: array cannot be safely cast to required type > >>> > >>>>>> dd=d.astype(N.float32) > >>>>>> N.dot(dd,ccc) > >>> > >>> [[[ 1. 1. 1.] > >>> [ 1. 1. 1.] > >>> [ 1. 1. 1.]] > >>> > >>> [[ 2. 2. 2.] > >>> [ 2. 2. 2.] > >>> [ 2. 2. 2.]]] > >>> > >>> > >>> > >>> The TypeError looks like a numpy bug ! > >> > >> I don't see why this is a bug. You are trying to coerce a 32-bit > >> integer to a 32-bit float. That is going to lose precision and so you > >> get the error indicated. > >> > >> -Travis > > > > In numarray I do not get an error. Would the error go away if I had 64 > > bit float !? It seems though that having ones and twos in an int array > > should fit just fine into a float32 array !? > > This could be considered a bug in numarray. It's force-casting the > result. That isn't the normal behavior of mixed-type functions. > > Also, the policy on type-casting is not to search the array to see if > it's possible to do the conversion on every element (that would be slow > on large arrays). The policy is to base the decision only on the > data-types themselves (i.e. whether it's *possible* to lose precision*). > > -Travis > > > > *There is one exception to this policy in NumPy: 64-bit integers are > allowed to be cast to 64-bit doubles --- other-wise on you would get a > lot of non-standard long-doubles showing up on 64-bit systems. This > policy was decided after discussion last year. OK - understood. Combining int32 with float64 proves to be less cumbersome ... Any idea on my main question ? What is the dot product of a 2x2 and 3x2x3 supposed to look like ? Why are numarray and numpy giving different answers ?? Thanks, Sebastian Haase From kwgoodman at gmail.com Mon Jul 17 18:50:57 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 17 Jul 2006 15:50:57 -0700 Subject: [Numpy-discussion] Displaying all the rows of large matrix Message-ID: How do you display all of the rows of a matrix? >> x = zeros((334,3)) >> x matrix([[ 0., 0., 0.], [ 0., 0., 0.], [ 0., 0., 0.], ..., [ 0., 0., 0.], [ 0., 0., 0.], [ 0., 0., 0.]]) From oliphant.travis at ieee.org Mon Jul 17 19:00:42 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 17 Jul 2006 17:00:42 -0600 Subject: [Numpy-discussion] Displaying all the rows of large matrix In-Reply-To: References: Message-ID: <44BC169A.3070807@ieee.org> Keith Goodman wrote: > How do you display all of the rows of a matrix? > help(numpy.set_prinoptions) Look at the threshold keyword numpy.set_printoptions(threshold = 2000) for your example -Travis From kwgoodman at gmail.com Mon Jul 17 19:37:12 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 17 Jul 2006 16:37:12 -0700 Subject: [Numpy-discussion] Displaying all the rows of large matrix In-Reply-To: <44BC169A.3070807@ieee.org> References: <44BC169A.3070807@ieee.org> Message-ID: On 7/17/06, Travis Oliphant wrote: > Keith Goodman wrote: > > How do you display all of the rows of a matrix? > > > > help(numpy.set_prinoptions) That is great! Now I can change the precision as well. Eight significant figures is too precise for me. Does anyone out there save the print defaults across sessions? How do you do it? Does numpy look for any startup files (~/.numpyrc)? From wbaxter at gmail.com Mon Jul 17 19:59:14 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 18 Jul 2006 08:59:14 +0900 Subject: [Numpy-discussion] Startup .numpyrc file Message-ID: On 7/18/06, Keith Goodman wrote: > On 7/17/06, Travis Oliphant wrote: > > Keith Goodman wrote: > Does anyone out there save the print defaults across sessions? How do you do it? > > Does numpy look for any startup files (~/.numpyrc)? If you set a PYTHONSTARTUP environment variable then Python will run the script it points to at startup of interactive sessions. But I wonder if maybe some numpy-specific startup script would be good. My current PYTHONSTARTUP is importing numpy, just so I can define some numpy shortcuts for my interactive sessions. That's fine if you always use Numpy in your interactive sessions, but if not, then it's just dead weight. --bb From oliphant.travis at ieee.org Mon Jul 17 20:10:36 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 17 Jul 2006 18:10:36 -0600 Subject: [Numpy-discussion] [SciPy-user] Arithmetic Errors In-Reply-To: <20060717235553.GA14362@redwoodscientific.com> References: <20060717205116.GA11043@redwoodscientific.com> <44BC0161.8070007@ieee.org> <20060717235553.GA14362@redwoodscientific.com> Message-ID: <44BC26FC.8050408@ieee.org> John Lawless wrote: > Travis, > > Thanks! > > 1). I haven't found any documentation on dtype='O'. (I purchased your > trelgol book but it hasn't arrived yet.) Does 'O' guarantee no > wrong answers? > The object data-type uses Python objects instead of low-level C-types for the calculations. So, it gives the same calculations that Python would do (but of course it's much slower). > 2). My actual code was more complex than the example I posted. It was > giving correct answers until I increased the dataset size. Then, > luckily, the result became obviously wrong. I can go through a > code and try to coerce everything to double but, when debugging a > large code, how can one ever be sure that all types are coerced > correctly if no errors are generated? > NumPy uses c data-types for calculations. It is therefore, *much* faster, but you have to take precautions about overflowing on integer operations. > 3). AFAIK, checking for overflows should take no CPU time whatsoever > unless an exception is actually generated. This is true for floating point operations, but you were doing integer multiplication. There is no support for hardware multiply overflow in NumPy (is there even such a thing?). Python checks for overflow on integer arithmetic by doing some additional calculations. It would be possible to add slower, integer over-flow checking ufuncs to NumPy if this was desired and you could replace the standard non-checking functions pretty easily. -Travis From oliphant.travis at ieee.org Mon Jul 17 20:11:31 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 17 Jul 2006 18:11:31 -0600 Subject: [Numpy-discussion] Startup .numpyrc file In-Reply-To: References: Message-ID: <44BC2733.60002@ieee.org> Bill Baxter wrote: > On 7/18/06, Keith Goodman wrote: > >> On 7/17/06, Travis Oliphant wrote: >> >>> Keith Goodman wrote: >>> >> Does anyone out there save the print defaults across sessions? How do you do it? >> >> Does numpy look for any startup files (~/.numpyrc)? >> > > If you set a PYTHONSTARTUP environment variable then Python will run > the script it points to at startup of interactive sessions. > > But I wonder if maybe some numpy-specific startup script would be > good. My current PYTHONSTARTUP is importing numpy, just so I can > define some numpy shortcuts for my interactive sessions. That's fine > if you always use Numpy in your interactive sessions, but if not, then > it's just dead weight. > > The standard answer is to use ipython which gives you a wealth of startup options. -Travis From oliphant.travis at ieee.org Mon Jul 17 23:01:07 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 17 Jul 2006 21:01:07 -0600 Subject: [Numpy-discussion] Beta release on Thursday Message-ID: <44BC4EF3.5020103@ieee.org> I'd like to make release 1.0beta on Thursday. Please submit bug-reports and fixes before then. -Travis From nvf at MIT.EDU Mon Jul 17 23:30:46 2006 From: nvf at MIT.EDU (Nick Fotopoulos) Date: Mon, 17 Jul 2006 23:30:46 -0400 Subject: [Numpy-discussion] numpy.vectorize performance In-Reply-To: <44B9BA20.7000600@ieee.org> References: <44B9BA20.7000600@ieee.org> Message-ID: On Jul 16, 2006, at 12:01 AM, Travis Oliphant wrote: > Thanks for the decorator. This should be put on the www.scipy.org > wiki. I've been looking over the wiki and am not sure where the best place would be for such a snippet. Would it go with the numpy examples under vectorize or perhaps in a cookbook somewhere? This seems more specialized than the basic numpy examples, but not worthy of its own cookbook. In general, what do you do with constructs that seem useful, but aren't useful enough to just include somewhere in NumPy/ SciPy? How would anyone think to look for a tip like this? Also, thanks for your helpful responses and additional thanks to Travis for the book update. Take care, Nick From steffen.loeck at gmx.de Tue Jul 18 03:24:25 2006 From: steffen.loeck at gmx.de (Steffen Loeck) Date: Tue, 18 Jul 2006 09:24:25 +0200 Subject: [Numpy-discussion] Speed degression? In-Reply-To: <44AD8581.5030904@ee.byu.edu> References: <200607041519.24636.steffen.loeck@gmx.de> <44AD8581.5030904@ee.byu.edu> Message-ID: <200607180924.25652.steffen.loeck@gmx.de> > I also placed in hooks so you can replace the scalarmath (for int, > float, and complex) with the Python version of math (this works because > the int, float, and complex scalars are sub-classes of the corresponding > Python object). Just for completeness some more tests using pythonmath/scalarmath for int, float or both (in usec per loop): sin - array mod - array xx (a) - (no import of numpy.core.scalarmath) numpy 0.9.9.2800 152 76.5 numpy 0.9.9.2800 + math 50.2 (b) - (use_pythonmath(xx)) numpy 0.9.9.2800 107 60.4 (int) numpy 0.9.9.2800 + math 32.7 numpy 0.9.9.2800 148 43 (float) numpy 0.9.9.2800 + math 50.7 numpy 0.9.9.2800 109 26.5 (int, float) numpy 0.9.9.2800 + math 32.4 (c) - (use_scalarmath(xx)) numpy 0.9.9.2800 149 77.1 (int) numpy 0.9.9.2800 + math 50.7 numpy 0.9.9.2800 147 74.3 (float) numpy 0.9.9.2800 + math 50.7 numpy 0.9.9.2800 148 73.5 (int, float) numpy 0.9.9.2800 + math 50.8 Maybe use_pythonmath(int, float, complex) should be set as default? Many thanks, Steffen From aisaac at american.edu Tue Jul 18 04:10:49 2006 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 18 Jul 2006 04:10:49 -0400 Subject: [Numpy-discussion] A Different Arithmetic Error: += In-Reply-To: <20060718053657.GA19656@redwoodscientific.com> References: <20060718053657.GA19656@redwoodscientific.com> Message-ID: On Mon, 17 Jul 2006, John Lawless apparently wrote: >>>> from scipy import * >>>> a = array((1.2)) >>>> a += 1.3j >>>> a > array(1.2) > Shouldn't this generate either an error or an up-cast, rather than > silently discarding the imaginary part? As I understand it: it cannot upcast, as the '+=' operation will use only the memory initially allocated for a. Cheers, Alan Isaac From theller at python.net Tue Jul 18 04:32:38 2006 From: theller at python.net (Thomas Heller) Date: Tue, 18 Jul 2006 10:32:38 +0200 Subject: [Numpy-discussion] Cannot build numpy svn on Windows Message-ID: Building numpy from svn on Windows with Python 2.4.3 fails: c:\svn\numpy\numpy\core\include\numpy\arrayobject.h(986) : fatal error C1017: invalid integer constant expression When I change this line in the generated config.h file: #define NPY_ALLOW_THREADS WITH_THREAD to this one: #define NPY_ALLOW_THREADS 1 then I can build. Thomas From oliphant.travis at ieee.org Tue Jul 18 04:46:30 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 18 Jul 2006 02:46:30 -0600 Subject: [Numpy-discussion] numarray bug: dot product between 2x2 and 3x2x3 on Mac different from PC In-Reply-To: <200607171348.05776.haase@msg.ucsf.edu> References: <200607170940.48872.haase@msg.ucsf.edu> <200607171226.33383.haase@msg.ucsf.edu> <44BBE735.5010109@ieee.org> <200607171348.05776.haase@msg.ucsf.edu> Message-ID: <44BC9FE6.70702@ieee.org> Sebastian Haase wrote: > On Monday 17 July 2006 12:38, Travis Oliphant wrote: > > Any idea on my main question ? > What is the dot product of a 2x2 and 3x2x3 supposed to look like ? > Why are numarray and numpy giving different answers ?? > I'm pretty sure the dot-product in Numeric (and I guess numarray too) was broken for larger than 2-dimensions. This was fixed several months ago in NumPy. NumPy dot-product gives the following result a.shape is (I,L) b.shape is (J,L,K) Then c=dot(a,b) will have c.shape = (I,J,K) with c[i,j,k] = sum(a[i,:]*b[j,:,k]) I'm not even sure what Numeric is computing in this case. -Travis From gruben at bigpond.net.au Tue Jul 18 06:30:37 2006 From: gruben at bigpond.net.au (Gary Ruben) Date: Tue, 18 Jul 2006 20:30:37 +1000 Subject: [Numpy-discussion] numpy.vectorize performance In-Reply-To: References: <44B9BA20.7000600@ieee.org> Message-ID: <44BCB84D.1080008@bigpond.net.au> Nick Fotopoulos wrote: > I've been looking over the wiki and am not sure where the best place > would be for such a snippet. Would it go with the numpy examples > under vectorize or perhaps in a cookbook somewhere? Yes. It seems to me like a cookbook example. In the utopian future, when there are as many cookbook examples as OReilly have, it'll be time for a reorganisation, but for now, make it a cookbook entry. Gary R From mail at stevesimmons.com Tue Jul 18 07:59:30 2006 From: mail at stevesimmons.com (Stephen Simmons) Date: Tue, 18 Jul 2006 21:59:30 +1000 Subject: [Numpy-discussion] Advice for grouping recarrays Message-ID: <44BCCD22.7060607@stevesimmons.com> Hi, Does anyone have any suggestions for summarising data in numpy? The quick description is that I want to do something like the SQL statement: SELECT sum(field1), sum(field2) FROM table GROUP BY field3; The more accurate description is that my data is stored in PyTables HDF format, with 24 monthly files, each with 4m records describing how customers performed that month. Each record looks something like this: ('200604', 651404500000L, '800', 'K', 12L, 162.0, 2000.0, 0.054581, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 2.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 8.80, 0.86, 7.80 17.46, 0.0, 70.0, 0.0, 70.0, -142.93, 0.0, 2000.0, 2063.93, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, -9.71, 7.75, 87.46, 77.75, -3.45, 0.22, -0.45, -0.57, 73.95) The first 5 fields are status fields (month_string, account_number, product_code, account_status, months_since_customer_joined). The remaining 48 fields represent different aspects of the customer's performance during that month. I read 100,000 of these records at a time and turn them into a numpy recarray with: dat = hdf_table.read(start=pos, stop=pos+block_size) dat = numpy.asarray(dat._flatArray, dtype=dat.array_descr) I'd like to reduce these 96m records x 53 fields down to monthly averages for each tuple (month_string, months_since_customer_joined) which in the case above is ('200604', 12L). This will let me compare the performance of newly acquired customers at the same point in their lifecycle as customers acquired 1 or 2 years ago. The end result should be a dataset something like res[month_index, months_since_customer_joined] = array([ num_records, sum_field_5, sum_field_6, sum_field_7, ... sum_field_52 ]) with a shape of (24, 24, 49). I've played around with lexsort(), take(), sum(), etc, but get very confused and end up feeling that I'm making things more complicated than they need to be. So any advice from numpy veterans on how best to proceed would be very welcome! Cheers Stephen From tom.denniston at alum.dartmouth.org Tue Jul 18 09:49:51 2006 From: tom.denniston at alum.dartmouth.org (Tom Denniston) Date: Tue, 18 Jul 2006 08:49:51 -0500 Subject: [Numpy-discussion] Advice for grouping recarrays In-Reply-To: <44BCCD22.7060607@stevesimmons.com> References: <44BCCD22.7060607@stevesimmons.com> Message-ID: I suggest lexsort itertools.groupby of the indices take I think it would be really great if numpy had the first two as a function or something like that. It is really useful to be able to take an array and bucket it and apply further numpy operations like accumulation functions. On 7/18/06, Stephen Simmons wrote: > Hi, > > Does anyone have any suggestions for summarising data in numpy? > > The quick description is that I want to do something like the SQL statement: > SELECT sum(field1), sum(field2) FROM table GROUP BY field3; > > The more accurate description is that my data is stored in PyTables HDF > format, with 24 monthly files, each with 4m records describing how > customers performed that month. Each record looks something like this: > ('200604', 651404500000L, '800', 'K', 12L, 162.0, 2000.0, 0.054581, 0.0, > 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 2.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, > 8.80, 0.86, 7.80 17.46, 0.0, 70.0, 0.0, 70.0, -142.93, 0.0, 2000.0, > 2063.93, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, -9.71, 7.75, > 87.46, 77.75, -3.45, 0.22, -0.45, -0.57, 73.95) > The first 5 fields are status fields (month_string, account_number, > product_code, account_status, months_since_customer_joined). The > remaining 48 fields represent different aspects of the customer's > performance during that month. I read 100,000 of these records at a time > and turn them into a numpy recarray with: > dat = hdf_table.read(start=pos, stop=pos+block_size) > dat = numpy.asarray(dat._flatArray, dtype=dat.array_descr) > > I'd like to reduce these 96m records x 53 fields down to monthly > averages for each tuple (month_string, months_since_customer_joined) > which in the case above is ('200604', 12L). This will let me compare the > performance of newly acquired customers at the same point in their > lifecycle as customers acquired 1 or 2 years ago. > > The end result should be a dataset something like > res[month_index, months_since_customer_joined] > = array([ num_records, sum_field_5, sum_field_6, sum_field_7, ... > sum_field_52 ]) > with a shape of (24, 24, 49). > > I've played around with lexsort(), take(), sum(), etc, but get very > confused and end up feeling that I'm making things more complicated than > they need to be. So any advice from numpy veterans on how best to > proceed would be very welcome! > > Cheers > > Stephen > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From kwgoodman at gmail.com Tue Jul 18 09:54:44 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 18 Jul 2006 06:54:44 -0700 Subject: [Numpy-discussion] Cannot build numpy svn on Windows In-Reply-To: References: Message-ID: On 7/18/06, Thomas Heller wrote: > When I change this line in the generated config.h file: > > #define NPY_ALLOW_THREADS WITH_THREAD > > to this one: > > #define NPY_ALLOW_THREADS 1 > > then I can build. What part of numpy is threaded? From emsellem at obs.univ-lyon1.fr Tue Jul 18 10:23:53 2006 From: emsellem at obs.univ-lyon1.fr (Eric Emsellem) Date: Tue, 18 Jul 2006 16:23:53 +0200 Subject: [Numpy-discussion] fast way of doing "cross-multiplications" ? Message-ID: <44BCEEF9.1030200@obs.univ-lyon1.fr> Hi, I have a specific quantity to derive from an array, and I am at the moment unable to do it for a too large array because it just takes too long! So I am looking for an advice on how to efficiently compute such a quantity: I have 3 arrays of N floats (x[...], y[..], z[..]) and I wish to do: result = 0. for i in range(N) : for j in range(i+1,N,1) : result += 1. / sqrt((x[j] - x[i])**2 + (y[j] - y[i])**2 + (z[j] - z[i])**2) Of course the procedure written above is very inefficient and I thought of doing: result = 0. for i in range(N) : result += 1. / sqrt((x[i+1:] - x[i])**2 + (y[i+1:] - y[i])**2 + (z[i+1:] - z[i])**2) Still, this is quite slow and not workable for very large arrays (> 10^6 floats per array). Any hint on how to speed things up here? Thanks in advance! Eric From wbaxter at gmail.com Tue Jul 18 10:45:34 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 18 Jul 2006 23:45:34 +0900 Subject: [Numpy-discussion] fast way of doing "cross-multiplications" ? In-Reply-To: <44BCEEF9.1030200@obs.univ-lyon1.fr> References: <44BCEEF9.1030200@obs.univ-lyon1.fr> Message-ID: Maybe this will help -- it computes the squared distances between a bunch of points: def dist2(x,c): """ Calculates squared distance between two sets of points. n2 = dist2(x, c) D = DIST2(X, C) takes two matrices of vectors and calculates the squared Euclidean distance between them. Both matrices must be of the same column dimension. If X has M rows and N columns, and C has L rows and N columns, then the result has M rows and L columns. The I, Jth entry is the squared distance from the Ith row of X to the Jth row of C. """ ndata, dimx = x.shape ncentres, dimc = c.shape if dimx != dimc: raise ValueError('Data dimension does not match dimension of centres') n2 = (x*x).sum(1)[:,numpy.newaxis] + (c*c).sum(1) - 2*dot(x,T(c)) # Rounding errors occasionally cause negative entries in n2 #if (n2<0).any(): # n2[n2<0] = 0 return n2 Take 1.0/numpy.sqrt(dist2(V,V)) and from there you can probably get the sum with sum() calls. It's obviously not as efficient as it could be since it'll be computing both halves of a symmetric matrix, but it might be faster than nested Python loops. (The above was adapted from a routine in Netlab). --bb On 7/18/06, Eric Emsellem wrote: > Hi, > > I have a specific quantity to derive from an array, and I am at the > moment unable to do it for a too large array because it just takes too > long! So I am looking for an advice on how to efficiently compute such a > quantity: > > I have 3 arrays of N floats (x[...], y[..], z[..]) and I wish to do: > > result = 0. > for i in range(N) : > for j in range(i+1,N,1) : > result += 1. / sqrt((x[j] - x[i])**2 + (y[j] - y[i])**2 + (z[j] - > z[i])**2) > > > Of course the procedure written above is very inefficient and I thought > of doing: > > result = 0. > for i in range(N) : > result += 1. / sqrt((x[i+1:] - x[i])**2 + (y[i+1:] - y[i])**2 + > (z[i+1:] - z[i])**2) > > Still, this is quite slow and not workable for very large arrays (> 10^6 > floats per array). > > Any hint on how to speed things up here? > > Thanks in advance! > > Eric From perry at stsci.edu Tue Jul 18 11:23:53 2006 From: perry at stsci.edu (Perry Greenfield) Date: Tue, 18 Jul 2006 11:23:53 -0400 Subject: [Numpy-discussion] fast way of doing "cross-multiplications" ? In-Reply-To: <44BCEEF9.1030200@obs.univ-lyon1.fr> References: <44BCEEF9.1030200@obs.univ-lyon1.fr> Message-ID: <502862AF-514C-41CA-804E-420CE76FEE37@stsci.edu> On Jul 18, 2006, at 10:23 AM, Eric Emsellem wrote: > Hi, > > I have a specific quantity to derive from an array, and I am at the > moment unable to do it for a too large array because it just takes too > long! So I am looking for an advice on how to efficiently compute > such a > quantity: > > I have 3 arrays of N floats (x[...], y[..], z[..]) and I wish to do: > > result = 0. > for i in range(N) : > for j in range(i+1,N,1) : > result += 1. / sqrt((x[j] - x[i])**2 + (y[j] - y[i])**2 + (z > [j] - > z[i])**2) > > > Of course the procedure written above is very inefficient and I > thought > of doing: > > result = 0. > for i in range(N) : > result += 1. / sqrt((x[i+1:] - x[i])**2 + (y[i+1:] - y[i])**2 + > (z[i+1:] - z[i])**2) > > Still, this is quite slow and not workable for very large arrays (> > 10^6 > floats per array). > > Any hint on how to speed things up here? > > Thanks in advance! > > Eric Perhaps I'm misunderstanding the last variant but don't you want something like: result = 0. for i in range(N) : result += add.reduce(1. / sqrt((x[i+1:] - x[i])**2 + (y[i+1:] - y [i])**2 + (z[i+1:] - z[i])**2)) instead since the expression yields an array with a decreasing size each iteration? But besides that, it seems you are asking to do roughly 10^12 of these computations for 10^6 points. I don't see any way to avoid that given what you are computing. The solution Bill Baxter gives is fine (I think, I haven't looked at it closely), but the usual problem of doing it without any looping is that it requires an enormous amount of memory (~10^12 element arrays) if I'm not mistaken. Since your second example is iterating over large arrays (most of the time, not near the end), I'd be surprised if you can do much better than that (the looping overhead should be negligible for such large arrays). Do you have examples written in other languages that run much faster? I guess I would be surprised to see it possible to do more than a few times faster in any language without some very clever optimizations. Perry From emsellem at obs.univ-lyon1.fr Tue Jul 18 11:36:04 2006 From: emsellem at obs.univ-lyon1.fr (Eric Emsellem) Date: Tue, 18 Jul 2006 17:36:04 +0200 Subject: [Numpy-discussion] fast way of doing "cross-multiplications" ? In-Reply-To: <502862AF-514C-41CA-804E-420CE76FEE37@stsci.edu> References: <44BCEEF9.1030200@obs.univ-lyon1.fr> <502862AF-514C-41CA-804E-420CE76FEE37@stsci.edu> Message-ID: <44BCFFE4.6070502@obs.univ-lyon1.fr> thanks for the tips. (indeed your "add.reduce" is correct: I just wrote this down too quickly, in the script I have a "sum" included). And yes you are right for the memory issue, so I may just keep the loop in and try to make it work on a fast PC...(or use parallel processes) (is "sum" different than "add.reduce"?) thanks again to both Bill Baxter and Perry Greenfield for their fast (and helpful!) answers. cheers Eric Perry Greenfield wrote: > > On Jul 18, 2006, at 10:23 AM, Eric Emsellem wrote: > >> Hi, >> >> I have a specific quantity to derive from an array, and I am at the >> moment unable to do it for a too large array because it just takes too >> long! So I am looking for an advice on how to efficiently compute such a >> quantity: >> >> I have 3 arrays of N floats (x[...], y[..], z[..]) and I wish to do: >> >> result = 0. >> for i in range(N) : >> for j in range(i+1,N,1) : >> result += 1. / sqrt((x[j] - x[i])**2 + (y[j] - y[i])**2 + (z[j] - >> z[i])**2) >> >> >> Of course the procedure written above is very inefficient and I thought >> of doing: >> >> result = 0. >> for i in range(N) : >> result += 1. / sqrt((x[i+1:] - x[i])**2 + (y[i+1:] - y[i])**2 + >> (z[i+1:] - z[i])**2) >> >> Still, this is quite slow and not workable for very large arrays (> 10^6 >> floats per array). >> >> Any hint on how to speed things up here? >> >> Thanks in advance! >> >> Eric > > Perhaps I'm misunderstanding the last variant but don't you want > something like: > > result = 0. > for i in range(N) : > result += add.reduce(1. / sqrt((x[i+1:] - x[i])**2 + (y[i+1:] - > y[i])**2 + > (z[i+1:] - z[i])**2)) > > instead since the expression yields an array with a decreasing size > each iteration? > > But besides that, it seems you are asking to do roughly 10^12 of these > computations for 10^6 points. I don't see any way to avoid that given > what you are computing. The solution Bill Baxter gives is fine (I > think, I haven't looked at it closely), but the usual problem of doing > it without any looping is that it requires an enormous amount of > memory (~10^12 element arrays) if I'm not mistaken. Since your second > example is iterating over large arrays (most of the time, not near the > end), I'd be surprised if you can do much better than that (the > looping overhead should be negligible for such large arrays). Do you > have examples written in other languages that run much faster? I guess > I would be surprised to see it possible to do more than a few times > faster in any language without some very clever optimizations. > > Perry From fullung at gmail.com Tue Jul 18 11:51:18 2006 From: fullung at gmail.com (Albert Strasheim) Date: Tue, 18 Jul 2006 17:51:18 +0200 Subject: [Numpy-discussion] Cannot build numpy svn on Windows In-Reply-To: Message-ID: <031001c6aa82$021af7b0$0100000a@dsp.sun.ac.za> Hello all > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of Keith Goodman > Sent: 18 July 2006 15:55 > To: Thomas Heller > Cc: numpy-discussion at lists.sourceforge.net > Subject: Re: [Numpy-discussion] Cannot build numpy svn on Windows > > On 7/18/06, Thomas Heller wrote: > > > When I change this line in the generated config.h file: > > > > #define NPY_ALLOW_THREADS WITH_THREAD > > > > to this one: > > > > #define NPY_ALLOW_THREADS 1 > > > > then I can build. This might be due to some changes Travis made. http://projects.scipy.org/scipy/numpy/changeset/2833 I was able to build r2834 on Windows without problems. Are you still seeing this error? > What part of numpy is threaded? This thread stuff is so that NumPy releases/doesn't release the GIL in certain places. Regards, Albert From tim.hochberg at cox.net Tue Jul 18 12:13:23 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue, 18 Jul 2006 09:13:23 -0700 Subject: [Numpy-discussion] fast way of doing "cross-multiplications" ? In-Reply-To: <44BCFFE4.6070502@obs.univ-lyon1.fr> References: <44BCEEF9.1030200@obs.univ-lyon1.fr> <502862AF-514C-41CA-804E-420CE76FEE37@stsci.edu> <44BCFFE4.6070502@obs.univ-lyon1.fr> Message-ID: <44BD08A3.1030600@cox.net> Eric Emsellem wrote: > thanks for the tips. (indeed your "add.reduce" is correct: I just wrote > this down too quickly, in the script I have a "sum" included). > > And yes you are right for the memory issue, so I may just keep the loop > in and try to make it work on a fast PC...(or use parallel processes) > > (is "sum" different than "add.reduce"?) > > thanks again to both Bill Baxter and Perry Greenfield for their fast > (and helpful!) answers. > I just wanted to add that there are faster, but considerably complicated ways to attack this class of problems. The one I've looked at in the past was the fast multipole method and I believe there are others. I'm not sure whether these can be implemented efficiently in numpy, but you may want to take a look into this kind of more sophisticated/complicated approach if brute forcing the calculation doesn't work. -tim > cheers > > Eric > > Perry Greenfield wrote: > >> On Jul 18, 2006, at 10:23 AM, Eric Emsellem wrote: >> >> >>> Hi, >>> >>> I have a specific quantity to derive from an array, and I am at the >>> moment unable to do it for a too large array because it just takes too >>> long! So I am looking for an advice on how to efficiently compute such a >>> quantity: >>> >>> I have 3 arrays of N floats (x[...], y[..], z[..]) and I wish to do: >>> >>> result = 0. >>> for i in range(N) : >>> for j in range(i+1,N,1) : >>> result += 1. / sqrt((x[j] - x[i])**2 + (y[j] - y[i])**2 + (z[j] - >>> z[i])**2) >>> >>> >>> Of course the procedure written above is very inefficient and I thought >>> of doing: >>> >>> result = 0. >>> for i in range(N) : >>> result += 1. / sqrt((x[i+1:] - x[i])**2 + (y[i+1:] - y[i])**2 + >>> (z[i+1:] - z[i])**2) >>> >>> Still, this is quite slow and not workable for very large arrays (> 10^6 >>> floats per array). >>> >>> Any hint on how to speed things up here? >>> >>> Thanks in advance! >>> >>> Eric >>> >> Perhaps I'm misunderstanding the last variant but don't you want >> something like: >> >> result = 0. >> for i in range(N) : >> result += add.reduce(1. / sqrt((x[i+1:] - x[i])**2 + (y[i+1:] - >> y[i])**2 + >> (z[i+1:] - z[i])**2)) >> >> instead since the expression yields an array with a decreasing size >> each iteration? >> >> But besides that, it seems you are asking to do roughly 10^12 of these >> computations for 10^6 points. I don't see any way to avoid that given >> what you are computing. The solution Bill Baxter gives is fine (I >> think, I haven't looked at it closely), but the usual problem of doing >> it without any looping is that it requires an enormous amount of >> memory (~10^12 element arrays) if I'm not mistaken. Since your second >> example is iterating over large arrays (most of the time, not near the >> end), I'd be surprised if you can do much better than that (the >> looping overhead should be negligible for such large arrays). Do you >> have examples written in other languages that run much faster? I guess >> I would be surprised to see it possible to do more than a few times >> faster in any language without some very clever optimizations. >> >> Perry >> > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From theller at python.net Tue Jul 18 12:23:18 2006 From: theller at python.net (Thomas Heller) Date: Tue, 18 Jul 2006 18:23:18 +0200 Subject: [Numpy-discussion] Cannot build numpy svn on Windows In-Reply-To: <031001c6aa82$021af7b0$0100000a@dsp.sun.ac.za> References: <031001c6aa82$021af7b0$0100000a@dsp.sun.ac.za> Message-ID: Albert Strasheim schrieb: > Hello all > >> -----Original Message----- >> From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- >> discussion-bounces at lists.sourceforge.net] On Behalf Of Keith Goodman >> Sent: 18 July 2006 15:55 >> To: Thomas Heller >> Cc: numpy-discussion at lists.sourceforge.net >> Subject: Re: [Numpy-discussion] Cannot build numpy svn on Windows >> >> On 7/18/06, Thomas Heller wrote: >> >> > When I change this line in the generated config.h file: >> > >> > #define NPY_ALLOW_THREADS WITH_THREAD >> > >> > to this one: >> > >> > #define NPY_ALLOW_THREADS 1 >> > >> > then I can build. > > This might be due to some changes Travis made. > > http://projects.scipy.org/scipy/numpy/changeset/2833 > > I was able to build r2834 on Windows without problems. Are you still seeing > this error? No, I can build now. Travis fixed it immediately, probably with the change that you refer to above. Thomas From ndarray at mac.com Tue Jul 18 12:34:24 2006 From: ndarray at mac.com (Sasha) Date: Tue, 18 Jul 2006 12:34:24 -0400 Subject: [Numpy-discussion] fast way of doing "cross-multiplications" ? In-Reply-To: <44BCFFE4.6070502@obs.univ-lyon1.fr> References: <44BCEEF9.1030200@obs.univ-lyon1.fr> <502862AF-514C-41CA-804E-420CE76FEE37@stsci.edu> <44BCFFE4.6070502@obs.univ-lyon1.fr> Message-ID: On 7/18/06, Eric Emsellem wrote: [...] > (is "sum" different than "add.reduce"?) "sum" is a wrapper around ndarray.sum method, while add.reduce is a ufunc method. At the C level they are the same, but "sum" may be a little slower for the small arrays. > python -m timeit -s "from numpy import add, zeros, sum; x = zeros(10)" "sum(x)" 100000 loops, best of 3: 5.53 usec per loop > python -m timeit -s "from numpy import add, zeros, sum; x = zeros(10)" "add.reduce(x)" 100000 loops, best of 3: 2.71 usec per loop In the new code, I would recommend using the ndarray.sum method instead of either of the two. BTW, is this an optimization opportunity: compare > python -m timeit -s "from numpy import add, zeros, sum; x = zeros(10); r=add.reduce" "r(x)" 100000 loops, best of 3: 2.55 usec per loop and > python -m timeit -s "from numpy import zeros; x = zeros(10)" "x.sum()" 100000 loops, best of 3: 3.88 usec per loop ? From williams at astro.ox.ac.uk Tue Jul 18 12:49:28 2006 From: williams at astro.ox.ac.uk (Michael Williams) Date: Tue, 18 Jul 2006 17:49:28 +0100 Subject: [Numpy-discussion] updated Ubuntu Dapper packages for numpy, matplotlib, and scipy online In-Reply-To: <4496D1AC.8030100@astraw.com> References: <4496D1AC.8030100@astraw.com> Message-ID: <20060718164928.GB21578@astro.ox.ac.uk> Hi Andrew (and others), On Mon, Jun 19, 2006 at 09:32:44AM -0700, Andrew Straw wrote: >I have updated the apt repository I maintain for Ubuntu's Dapper, which >now includes: > >numpy >matplotlib >scipy > >Each package is from a recent SVN checkout and should thus be regarded >as "bleeding edge". The repository has a new URL: >http://debs.astraw.com/dapper/ I intend to keep this repository online >for an extended duration. If you want to put this repository in your >sources list, you need to add the following lines to >/etc/apt/sources.list:: > deb http://debs.astraw.com/ dapper/ > deb-src http://debs.astraw.com/ dapper/ I am unable to access these repositories (which sound very useful, and for which I am grateful to Andrew!). apt-get update gives "Failed to fetch http://debs.astraw.com/dapper/Release.gpg Could not resolve ?debs.astraw.com?" I am also unable to access the repositories listed on the website: deb http://sefton.astraw.com/ubuntu/ dapper/ deb-src http://sefton.astraw.com/ubuntu/ dapper/ for the same reason. Does anyone know where they've gone and if they're coming back? Cheers, -- Michael Williams Rudolph Peierls Centre for Theoretical Physics University of Oxford From robert.kern at gmail.com Tue Jul 18 13:02:09 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 18 Jul 2006 12:02:09 -0500 Subject: [Numpy-discussion] fast way of doing "cross-multiplications" ? In-Reply-To: <44BD08A3.1030600@cox.net> References: <44BCEEF9.1030200@obs.univ-lyon1.fr> <502862AF-514C-41CA-804E-420CE76FEE37@stsci.edu> <44BCFFE4.6070502@obs.univ-lyon1.fr> <44BD08A3.1030600@cox.net> Message-ID: Tim Hochberg wrote: > I just wanted to add that there are faster, but considerably complicated > ways to attack this class of problems. The one I've looked at in the > past was the fast multipole method and I believe there are others. I'm > not sure whether these can be implemented efficiently in numpy, but you > may want to take a look into this kind of more sophisticated/complicated > approach if brute forcing the calculation doesn't work. Idesbald Van den Bosch will be giving a talk at SciPy'06 on implementing FMM and the multilevel fast multipole algorithm (MLFMA) using SciPy and weave. http://www.scipy.org/SciPy2006 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From fperez.net at gmail.com Tue Jul 18 13:25:30 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 18 Jul 2006 11:25:30 -0600 Subject: [Numpy-discussion] fast way of doing "cross-multiplications" ? In-Reply-To: <44BD08A3.1030600@cox.net> References: <44BCEEF9.1030200@obs.univ-lyon1.fr> <502862AF-514C-41CA-804E-420CE76FEE37@stsci.edu> <44BCFFE4.6070502@obs.univ-lyon1.fr> <44BD08A3.1030600@cox.net> Message-ID: On 7/18/06, Tim Hochberg wrote: > Eric Emsellem wrote: > > thanks for the tips. (indeed your "add.reduce" is correct: I just wrote > > this down too quickly, in the script I have a "sum" included). > > > > And yes you are right for the memory issue, so I may just keep the loop > > in and try to make it work on a fast PC...(or use parallel processes) > > > > (is "sum" different than "add.reduce"?) > > > > thanks again to both Bill Baxter and Perry Greenfield for their fast > > (and helpful!) answers. > > > I just wanted to add that there are faster, but considerably complicated > ways to attack this class of problems. The one I've looked at in the > past was the fast multipole method and I believe there are others. I'm > not sure whether these can be implemented efficiently in numpy, but you > may want to take a look into this kind of more sophisticated/complicated > approach if brute forcing the calculation doesn't work. Indeed, FMM is the best known method that can turn this O(n^2) problem into O(n*log(n)). As Tim indicates, there is no easy way out of this one. Incidentally, there is a talk about FMM on the scipy'06 schedule, in case you are going to attend. An alternative approach to FMM (conceptually similar in some ways) is described here: http://amath.colorado.edu/faculty/fperez/talks/0604_sanum_mwadap.pdf Unfortunately this isn't exactly a half-hour optimization effort, as the required machinery is pretty heavy duty. And yes, this has all been done in python and runs on current numpy/scipy (though it has Fortran, C and C++ sprinkled as needed). Cheers, f From ndarray at mac.com Tue Jul 18 13:54:41 2006 From: ndarray at mac.com (Sasha) Date: Tue, 18 Jul 2006 13:54:41 -0400 Subject: [Numpy-discussion] A Different Arithmetic Error: += In-Reply-To: References: <20060718053657.GA19656@redwoodscientific.com> Message-ID: On 7/18/06, Alan G Isaac wrote: > it cannot upcast, as the '+=' operation will use only the > memory initially allocated for a. Not true: >>> x = [2,3] >>> x += array(2) >>> type(x) This is just the design choice made by numpy. I don't see the need for an error. Augmented assignment is a sufficiently advanced feature that those who use it can be expected to know what it does. Loosing imaginary part maybe a more compelling reason for an error than loosing precision, but it would be clearly wrong to have different types behave differently. Silent downcasting from float to int is actually a useful feature. From strawman at astraw.com Tue Jul 18 14:45:32 2006 From: strawman at astraw.com (Andrew Straw) Date: Tue, 18 Jul 2006 11:45:32 -0700 Subject: [Numpy-discussion] updated Ubuntu Dapper packages for numpy, matplotlib, and scipy online In-Reply-To: <20060718164928.GB21578@astro.ox.ac.uk> References: <4496D1AC.8030100@astraw.com> <20060718164928.GB21578@astro.ox.ac.uk> Message-ID: <44BD2C4C.4020205@astraw.com> Michael Williams wrote: >>Hi Andrew (and others), >> >>On Mon, Jun 19, 2006 at 09:32:44AM -0700, Andrew Straw wrote: >> >> > > >>>>I have updated the apt repository I maintain for Ubuntu's Dapper, which >>>>now includes: >>>> >>>>numpy >>>>matplotlib >>>>scipy >>>> >>>>Each package is from a recent SVN checkout and should thus be regarded >>>>as "bleeding edge". The repository has a new URL: >>>>http://debs.astraw.com/dapper/ I intend to keep this repository online >>>>for an extended duration. If you want to put this repository in your >>>>sources list, you need to add the following lines to >>>>/etc/apt/sources.list:: >>>>deb http://debs.astraw.com/ dapper/ >>>>deb-src http://debs.astraw.com/ dapper/ >>>> >>>> >> >> >> >>I am unable to access these repositories (which sound very useful, and >>for which I am grateful to Andrew!). apt-get update gives >> >> "Failed to fetch http://debs.astraw.com/dapper/Release.gpg Could not >> resolve ?debs.astraw.com?" >> >> >> > > Hmm, that looks like DNS error. My repository is still up and online... debs.astraw.com is actually a "CNAME" record, which aliases another domain name. The canonical domain name uses a dynamic DNS to point to a dynamically assigned IP addresses. Your system (probably in your DNS name resolution) must be having some issue with some of that. Unfortunately, you can't just plug in the IP address, because I'm using Apache virtual hosting to serve a website at "debs.astraw.com" differently from other websites. Anyhow, here's what I get, hopefully it will help you fix your DNS issue. $ host debs.astraw.com debs.astraw.com is an alias for astraw-office.kicks-ass.net. astraw-office.kicks-ass.net has address 131.215.28.162 debs.astraw.com is an alias for astraw-office.kicks-ass.net. debs.astraw.com is an alias for astraw-office.kicks-ass.net. >>I am also unable to access the repositories listed on the website: >> >> deb http://sefton.astraw.com/ubuntu/ dapper/ >> deb-src http://sefton.astraw.com/ubuntu/ dapper/ >> >> > > Eek, I replaced that with the new one location and some more info. >>for the same reason. Does anyone know where they've gone and if they're >>coming back? >> >> >> > > I'm planning on keeping them around for a while -- at least until numpy, scipy, and matplotlib get integrated into my flavor-of-the-year Debian/Ubuntu release in a sufficiently up-to-date version. From oliphant.travis at ieee.org Tue Jul 18 14:50:57 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 18 Jul 2006 12:50:57 -0600 Subject: [Numpy-discussion] What to do about numarray extension modules? Message-ID: <44BD2D91.1020800@ieee.org> I'd like to get suggestions about what to do about the remaining numarray extension modules that are not addressed in NumPy (convolve, image, nd_image). There will be people coming from numarray that will be using these modules. Of course, my opinion is that they should not have been placed in Numarray to begin with as the fit better as modules in SciPy. But, given the situation. Should we bring them over to NumPy? Or, perhaps provide a small package that contain these modules separately? With the numarray c_api that is now available in NumPy, these can be easily ported. It would be nice to have decided this by 1.0beta release at the end of the week. Discussion please... -Travis From robert.kern at gmail.com Tue Jul 18 15:09:56 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 18 Jul 2006 14:09:56 -0500 Subject: [Numpy-discussion] What to do about numarray extension modules? In-Reply-To: <44BD2D91.1020800@ieee.org> References: <44BD2D91.1020800@ieee.org> Message-ID: Travis Oliphant wrote: > I'd like to get suggestions about what to do about the remaining > numarray extension modules that are not addressed in NumPy (convolve, > image, nd_image). There will be people coming from numarray that will > be using these modules. > > Of course, my opinion is that they should not have been placed in > Numarray to begin with as the fit better as modules in SciPy. > > But, given the situation. Should we bring them over to NumPy? Or, > perhaps provide a small package that contain these modules separately? I'm definitely -1 on putting them in numpy. numpy is large enough as it is. I'm -0 on making a standalone package. The time would be better spent on making sure that scipy subpackages can be downloaded, built and installed individually. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant.travis at ieee.org Tue Jul 18 15:22:14 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 18 Jul 2006 13:22:14 -0600 Subject: [Numpy-discussion] What to do about numarray extension modules? In-Reply-To: References: <44BD2D91.1020800@ieee.org> Message-ID: <44BD34E6.2020603@ieee.org> Robert Kern wrote: > Travis Oliphant wrote: > >> I'd like to get suggestions about what to do about the remaining >> numarray extension modules that are not addressed in NumPy (convolve, >> image, nd_image). There will be people coming from numarray that will >> be using these modules. >> >> Of course, my opinion is that they should not have been placed in >> Numarray to begin with as the fit better as modules in SciPy. >> >> But, given the situation. Should we bring them over to NumPy? Or, >> perhaps provide a small package that contain these modules separately? >> > > I'm definitely -1 on putting them in numpy. numpy is large enough as it is. I'm > -0 on making a standalone package. The time would be better spent on making sure > that scipy subpackages can be downloaded, built and installed individually. > > I think I like this idea as well. I don't really want to grow NumPy. But, I'm curious what the current numarray users think. -Travis From filipwasilewski at gmail.com Tue Jul 18 15:21:46 2006 From: filipwasilewski at gmail.com (Filip Wasilewski) Date: Tue, 18 Jul 2006 21:21:46 +0200 Subject: [Numpy-discussion] A bunch of tickets Message-ID: Hi, I have been recently working with arrays of objects and noticed some problems. Here are the tickets: Array improperly created from numpy.poly1d object: http://projects.scipy.org/scipy/numpy/ticket/185 Can't create matrix of dtype=object directly from list (problem of ndarray.__new__): http://projects.scipy.org/scipy/numpy/ticket/186 Subclassing does not always work for numpy.matrix: http://projects.scipy.org/scipy/numpy/ticket/187 best, Filip From haase at msg.ucsf.edu Tue Jul 18 15:25:23 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 18 Jul 2006 12:25:23 -0700 Subject: [Numpy-discussion] What to do about numarray extension modules? In-Reply-To: References: <44BD2D91.1020800@ieee.org> Message-ID: <200607181225.23671.haase@msg.ucsf.edu> I'm using nd_image extensively. Considering that SciPy seem to install much easier these days than I remember I would be just happy to put them in to SciPy. Some numarray people prefer a transitional "special place". I remember that there was a discussion maybe a year ago of making SciPy so modular that (some?) sub-packages would be installable completely independent anyway - that would be the best :-) -Sebastian Haase On Tuesday 18 July 2006 12:09, Robert Kern wrote: > Travis Oliphant wrote: > > I'd like to get suggestions about what to do about the remaining > > numarray extension modules that are not addressed in NumPy (convolve, > > image, nd_image). There will be people coming from numarray that will > > be using these modules. > > > > Of course, my opinion is that they should not have been placed in > > Numarray to begin with as the fit better as modules in SciPy. > > > > But, given the situation. Should we bring them over to NumPy? Or, > > perhaps provide a small package that contain these modules separately? > > I'm definitely -1 on putting them in numpy. numpy is large enough as it is. > I'm -0 on making a standalone package. The time would be better spent on > making sure that scipy subpackages can be downloaded, built and installed > individually. From nvf at MIT.EDU Tue Jul 18 15:28:40 2006 From: nvf at MIT.EDU (Nick Fotopoulos) Date: Tue, 18 Jul 2006 15:28:40 -0400 Subject: [Numpy-discussion] What to do about numarray extension modules? Message-ID: Travis wrote: > Of course, my opinion is that they should not have been placed in > Numarray to begin with as the fit better as modules in SciPy. Can we not put them into SciPy now? Nick From perry at stsci.edu Tue Jul 18 15:34:46 2006 From: perry at stsci.edu (Perry Greenfield) Date: Tue, 18 Jul 2006 15:34:46 -0400 Subject: [Numpy-discussion] What to do about numarray extension modules? In-Reply-To: <44BD34E6.2020603@ieee.org> References: <44BD2D91.1020800@ieee.org> <44BD34E6.2020603@ieee.org> Message-ID: On Jul 18, 2006, at 3:22 PM, Travis Oliphant wrote: > Robert Kern wrote: >> Travis Oliphant wrote: >> >>> I'd like to get suggestions about what to do about the remaining >>> numarray extension modules that are not addressed in NumPy >>> (convolve, >>> image, nd_image). There will be people coming from numarray >>> that will >>> be using these modules. >>> >>> Of course, my opinion is that they should not have been placed in >>> Numarray to begin with as the fit better as modules in SciPy. >>> >>> But, given the situation. Should we bring them over to NumPy? Or, >>> perhaps provide a small package that contain these modules >>> separately? >>> >> >> I'm definitely -1 on putting them in numpy. numpy is large enough >> as it is. I'm >> -0 on making a standalone package. The time would be better spent >> on making sure >> that scipy subpackages can be downloaded, built and installed >> individually. >> >> > I think I like this idea as well. I don't really want to grow > NumPy. > But, I'm curious what the current numarray users think. > Our view, briefly, is that until it is easy to install scipy, or that scipy supports modular installs, that we need one or the other to support our distributions. The first choice (bundling with numpy) would be more convenient for us, but is not necessary should they be available separately. So, in short, either is fine with us. Perry From svetosch at gmx.net Tue Jul 18 15:37:27 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Tue, 18 Jul 2006 21:37:27 +0200 Subject: [Numpy-discussion] Beta release on Thursday In-Reply-To: <44BC4EF3.5020103@ieee.org> References: <44BC4EF3.5020103@ieee.org> Message-ID: <44BD3877.5010002@gmx.net> Travis Oliphant schrieb: > I'd like to make release 1.0beta on Thursday. Please submit bug-reports > and fixes before then. > Just two things I noticed: >>> import numpy as n import dft -> failed: module compiled against version 90709 of C-API but this version of numpy is 9090d >>> n.__version__ '0.9.9.2840' And TODO.txt looks pretty outdated, maybe to avoid a wrong impression and confusion when a lot of new people turn to numpy because of 1.0 status this should be changed (or removed?). Cheers, Sven From filip at ftv.pl Tue Jul 18 15:37:39 2006 From: filip at ftv.pl (Filip Wasilewski) Date: Tue, 18 Jul 2006 21:37:39 +0200 Subject: [Numpy-discussion] A bunch of tickets Message-ID: <151688431.20060718213739@gmail.com> Hi, I have posted this mail from my google account a while ago, but it obviously was blocked by sourceforge. What about that proposal of moving numpy-discussion to scipy host? Anyway, I have been recently working with arrays of objects and noticed some problems. Here are the tickets: Array improperly created from numpy.poly1d objects: http://projects.scipy.org/scipy/numpy/ticket/185 Can't create matrix of dtype=object directly from list (problem of ndarray.__new__): http://projects.scipy.org/scipy/numpy/ticket/186 Subclassing does not always work for numpy.matrix: http://projects.scipy.org/scipy/numpy/ticket/187 best, Filip From perry at stsci.edu Tue Jul 18 15:45:59 2006 From: perry at stsci.edu (Perry Greenfield) Date: Tue, 18 Jul 2006 15:45:59 -0400 Subject: [Numpy-discussion] What to do about numarray extension modules? In-Reply-To: <44BD2D91.1020800@ieee.org> References: <44BD2D91.1020800@ieee.org> Message-ID: On Jul 18, 2006, at 2:50 PM, Travis Oliphant wrote: > > I'd like to get suggestions about what to do about the remaining > numarray extension modules that are not addressed in NumPy (convolve, > image, nd_image). There will be people coming from numarray that > will > be using these modules. > > Of course, my opinion is that they should not have been placed in > Numarray to begin with as the fit better as modules in SciPy. At that at the time (some years ago now), that wasn't an option for us for the modules that we contributed since we couldn't use Numeric. I'm glad that it's now possible to integrate these with numpy, but it doesn't do much good now to lament that it didn't happen that way in the past. Perry From Pierre.Barbier_de_Reuille at sophia.inria.fr Tue Jul 18 15:32:36 2006 From: Pierre.Barbier_de_Reuille at sophia.inria.fr (Pierre Barbier de Reuille) Date: Tue, 18 Jul 2006 20:32:36 +0100 Subject: [Numpy-discussion] What to do about numarray extension modules? In-Reply-To: <200607181225.23671.haase@msg.ucsf.edu> References: <44BD2D91.1020800@ieee.org> <200607181225.23671.haase@msg.ucsf.edu> Message-ID: <44BD3754.8020609@sophia.inria.fr> Sebastian Haase wrote: > I'm using nd_image extensively. Considering that SciPy seem to install much > easier these days than I remember I would be just happy to put them in to > SciPy. > Well, what is the content of nd_image and image, compared to scipy.ndimage ? > Some numarray people prefer a transitional "special place". I remember that > there was a discussion maybe a year ago of making SciPy so modular that > (some?) sub-packages would be installable completely independent anyway - > that would be the best :-) > > -Sebastian Haase > > On Tuesday 18 July 2006 12:09, Robert Kern wrote: > >> Travis Oliphant wrote: >> >>> I'd like to get suggestions about what to do about the remaining >>> numarray extension modules that are not addressed in NumPy (convolve, >>> image, nd_image). There will be people coming from numarray that will >>> be using these modules. >>> >>> Of course, my opinion is that they should not have been placed in >>> Numarray to begin with as the fit better as modules in SciPy. >>> >>> But, given the situation. Should we bring them over to NumPy? Or, >>> perhaps provide a small package that contain these modules separately? >>> >> I'm definitely -1 on putting them in numpy. numpy is large enough as it is. >> I'm -0 on making a standalone package. The time would be better spent on >> making sure that scipy subpackages can be downloaded, built and installed >> individually. >> > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From oliphant at ee.byu.edu Tue Jul 18 16:21:42 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 18 Jul 2006 14:21:42 -0600 Subject: [Numpy-discussion] Beta release on Thursday In-Reply-To: <44BD3877.5010002@gmx.net> References: <44BC4EF3.5020103@ieee.org> <44BD3877.5010002@gmx.net> Message-ID: <44BD42D6.2020806@ee.byu.edu> Sven Schreiber wrote: >Travis Oliphant schrieb: > > >>I'd like to make release 1.0beta on Thursday. Please submit bug-reports >>and fixes before then. >> >> >> > >Just two things I noticed: > > Thanks for commenting > > >>>>import numpy as n >>>> >>>> >import dft -> failed: module compiled against version 90709 of C-API but >this version of numpy is 9090d > > > Make sure to do a clean build: i.e. rm -fr build python setup.py install Unfortunately, the dependency tracking does not re-build the sub-packages of numpy by itself when there is a header-file change. >And TODO.txt looks pretty outdated, maybe to avoid a wrong impression >and confusion when a lot of new people turn to numpy because of 1.0 >status this should be changed (or removed?). > > Good idea. -Travis From laidler at stsci.edu Tue Jul 18 16:23:14 2006 From: laidler at stsci.edu (Victoria G. Laidler) Date: Tue, 18 Jul 2006 16:23:14 -0400 Subject: [Numpy-discussion] What to do about numarray extension modules? In-Reply-To: <44BD2D91.1020800@ieee.org> References: <44BD2D91.1020800@ieee.org> Message-ID: <44BD4332.9030305@stsci.edu> I'm a numarray user that relies on convolve. I would hope that one of the criteria by which to make such decisions is "make the transition for numarray and numeric users as painless as possible". Changing namespaces and introducing additional dependencies is not painless. Thus unless there is a *compelling* argument to move them out, I would leave them in. (From the numarray-user point of view, it would be "moving them out" even though they aren't there yet.) Personally I wouldn't think that "numpy is big enough already" or "it fits better in scipy" to be compelling reasons. So, +1 to include them in numpy. That said, personally I would argue that convolve is qualitatively different from image and ndimage. Convolution is a common mathematical operation on arrays; thus I think it belongs in numpy. If any or all of them are not included in numpy, I would strongly prefer a small separate package than to put them in scipy. I try to minimize dependencies in my code; I really don't want to suddenly depend on all of scipy just so I can do a convolution! Vicki Laidler Travis Oliphant wrote: >I'd like to get suggestions about what to do about the remaining >numarray extension modules that are not addressed in NumPy (convolve, >image, nd_image). There will be people coming from numarray that will >be using these modules. > >Of course, my opinion is that they should not have been placed in >Numarray to begin with as the fit better as modules in SciPy. > >But, given the situation. Should we bring them over to NumPy? Or, >perhaps provide a small package that contain these modules separately? > >With the numarray c_api that is now available in NumPy, these can be >easily ported. It would be nice to have decided this by 1.0beta release >at the end of the week. > >Discussion please... > >-Travis > > >------------------------------------------------------------------------- >Take Surveys. Earn Cash. Influence the Future of IT >Join SourceForge.net's Techsay panel and you'll get the chance to share your >opinions on IT & business topics through brief surveys -- and earn cash >http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From oliphant at ee.byu.edu Tue Jul 18 16:25:01 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 18 Jul 2006 14:25:01 -0600 Subject: [Numpy-discussion] What to do about numarray extension modules? In-Reply-To: References: <44BD2D91.1020800@ieee.org> Message-ID: <44BD439D.8060807@ee.byu.edu> Perry Greenfield wrote: > > On Jul 18, 2006, at 2:50 PM, Travis Oliphant wrote: > >> >> I'd like to get suggestions about what to do about the remaining >> numarray extension modules that are not addressed in NumPy (convolve, >> image, nd_image). There will be people coming from numarray that will >> be using these modules. >> >> Of course, my opinion is that they should not have been placed in >> Numarray to begin with as the fit better as modules in SciPy. > > > At that at the time (some years ago now), that wasn't an option for > us for the modules that we contributed since we couldn't use Numeric. > I'm glad that it's now possible to integrate these with numpy, but it > doesn't do much good now to lament that it didn't happen that way in > the past. Of course. I was just thinking aloud again. That's the beauty of "organic" growth :-) It does have it's own charm. -Travis From williams at astro.ox.ac.uk Tue Jul 18 16:36:12 2006 From: williams at astro.ox.ac.uk (Michael Williams) Date: Tue, 18 Jul 2006 21:36:12 +0100 Subject: [Numpy-discussion] [SciPy-user] updated Ubuntu Dapper packages for numpy, matplotlib, and scipy online In-Reply-To: <20060718164928.GB21578@astro.ox.ac.uk> References: <4496D1AC.8030100@astraw.com> <20060718164928.GB21578@astro.ox.ac.uk> Message-ID: <20060718203612.GA25724@astro.ox.ac.uk> On Tue, Jul 18, 2006 at 11:45:32AM -0700, Andrew Straw wrote: >>I am unable to access these repositories (which sound very useful, and >>for which I am grateful to Andrew!). > >Hmm, that looks like DNS error. My repository is still up and online... It is indeed a DNS error. I should have tested from a different machine before posting. Thanks for the quick reply. >>for the same reason. Does anyone know where they've gone and if >>they're coming back? >> >I'm planning on keeping them around for a while -- at least until >numpy, scipy, and matplotlib get integrated into my flavor-of-the-year >Debian/Ubuntu release in a sufficiently up-to-date version. Great. I've installed them now, and the seem to be working fine. Thanks very much! Cheers, -- Mike From oliphant at ee.byu.edu Tue Jul 18 16:43:21 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 18 Jul 2006 14:43:21 -0600 Subject: [Numpy-discussion] What to do about numarray extension modules? In-Reply-To: <44BD4332.9030305@stsci.edu> References: <44BD2D91.1020800@ieee.org> <44BD4332.9030305@stsci.edu> Message-ID: <44BD47E9.6010102@ee.byu.edu> Victoria G. Laidler wrote: > I'm a numarray user that relies on convolve. Thanks for voicing your opinion. We should be clear about what we are talking about. 1) There is already a convolve in NumPy (it's the same one that was in Numeric). It seems the one in numarray is compatible. I'm talking about the following numarray functions (please let me know if I'm missing anything). * all of nd_image --- currently this is in scipy (as ndimage) and can already be built and installed as a separate package --- we need to work out distribution, though. * all of image/ combine.py median average minimum threshold _image.py translate * from Convolve/ iraf_frame.py lineshape.py Convolve.py/ correlate2d convolve2d boxcar For now, I will port the image and Convolve packages and place them in the scipy/sandbox area. From there we can figure out what to do with them. -Travis From jmiller at stsci.edu Tue Jul 18 18:37:06 2006 From: jmiller at stsci.edu (Todd Miller) Date: Tue, 18 Jul 2006 18:37:06 -0400 Subject: [Numpy-discussion] numarray bug: dot product between 2x2 and 3x2x3 on Mac different from PC In-Reply-To: <200607171348.05776.haase@msg.ucsf.edu> References: <200607170940.48872.haase@msg.ucsf.edu> <200607171226.33383.haase@msg.ucsf.edu> <44BBE735.5010109@ieee.org> <200607171348.05776.haase@msg.ucsf.edu> Message-ID: <44BD6292.5050600@stsci.edu> Sebastian Haase wrote: > OK - understood. Combining int32 with float64 proves to be less > cumbersome ... > > Why are numarray and numpy giving different answers ? > I have a little insight as to what is going on here but no fix. numarray has two versions of dot(): 1. The original dot() is standalone, easy to install, and basically works. This dot() does however have the type coercion wart where Int32 dotted with Float32 silently loses precision. 2. A more optimized dot(), ported from Numeric, depends on an external BLAS. This makes it faster but also generally harder to install. This optimized dot() has the higher dimensional array bug you discovered. For arrays of rank > 2, it's totally broken. Since Mac OS-X has a standard BLAS, it uses the optimized dot() by default and exposes the bug. The optimized dot() is easy to turn off in cfg_packages.py by commenting out the lines: # if os.path.exists('/System/Library/Frameworks/vecLib.framework'): # USE_LAPACK = True Since numarray is being phased out and neither issue is a serious problem for STScI, we're not planning to fix them. If it's causing someone major hardship let us know and we'll reconsider. Todd From Chris.Barker at noaa.gov Tue Jul 18 18:57:42 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 18 Jul 2006 15:57:42 -0700 Subject: [Numpy-discussion] What to do about numarray extension modules? In-Reply-To: References: <44BD2D91.1020800@ieee.org> Message-ID: <44BD6766.2020507@noaa.gov> Robert Kern wrote: > I'm definitely -1 on putting them in numpy. numpy is large enough as it is. I'm > -0 on making a standalone package. The time would be better spent on making sure > that scipy subpackages can be downloaded, built and installed individually. +1 The whole SciPy installation thing is HUGE! I now I still don't use any of SciPy because I can't get it all built. Is it yet possible to build SciPy on Fedora core 4 without lots of painful machinations? When numpy was called SciPy.core, there was a lot of talk about how SciPy needed to be broken up into individual packages, particularly the Fortran and non-Fortran ones separated. It was a good idea then, and it still is. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From josh.p.marshall at gmail.com Tue Jul 18 20:32:06 2006 From: josh.p.marshall at gmail.com (Josh Marshall) Date: Wed, 19 Jul 2006 10:32:06 +1000 Subject: [Numpy-discussion] [Pythonmac-SIG] Packaging numpy with py2app In-Reply-To: <1CC7BE8A-F534-4256-A5A2-FEA2E1D2725A@redivi.com> References: <660C6421-8C71-4B4D-A2CC-33C3FF089DF9@gmail.com> <44BC1B92.4090009@noaa.gov> <44BC1E6E.9020502@noaa.gov> <51B08665-6077-4E6F-875D-D217990B1714@redivi.com> <60E2AEE1-9BF1-4B62-A9DC-7A497310A681@gmail.com> <213C1E2D-AE4D-4578-9E22-FB49C1ACAA75@gmail.com> <9148A14B-7462-47EC-9262-52802E5A9CB4@gmail.com> <1CC7BE8A-F534-4256-A5A2-FEA2E1D2725A@redivi.com> Message-ID: <0039ED8C-EA8A-47BE-9967-200AF66D173A@gmail.com> Thanks for the info on how the various recipes work, Bob. Very helpful. On 19/07/2006, at 9:28 AM, Bob Ippolito wrote: > The recipe mechanism doesn't allow for it because it doesn't > generally make sense. There are very few packages that can find > their resources in an alternative manner. I'm not totally opposed > to adding another feature to support that use case, but throwing it > directly in the resources dir seems like a bad idea to do > automatically from a recipe. The files should sit under some kind > of matplotlib hierarchy. It would be nicer to see matplotlib just > figure out how to make their app work from a zip... pygame does, > and it has fonts and icons. I think matplotlib can access files inside the zip, since it searches inside the following directories upon startup and is frozen-aware. MatplotlibEmbedSample.app/Contents/Resources/lib/python2.4/site- packages.zip/matplotlib/mpl-data MatplotlibEmbedSample.app/Contents/mpl-data MatplotlibEmbedSample.app/Contents/Resources/mpl-data So I have been attempting to put the data files in the zip. > loader_files puts data files *in the zip* Indeed it does. When I use the following recipe: (noting that the recipe's filename can't be "matplotlib.py", I've been using "matplotlib_recipe.py"): def check(cmd, mf): m = mf.findNode('matplotlib') if m is None or m.filename is None: return None import matplotlib, glob, os mpl_datapath = matplotlib.get_data_path() mpl_datafilelist = glob.glob(mpl_datapath + r'/*') # only include individual files from the mpl-data directory # this should exclude directories, which should only be Matplotlib.nib mpl_datafilelist = [mpl_file for mpl_file in mpl_datafilelist if os.path.isfile(mpl_file)] mf.import_hook('pytz.zoneinfo', m, ['UTC']) mf.import_hook('matplotlib.numerix', m, ['random_array']) return dict( loader_files = [ ('matplotlib', mpl_datafilelist), ], ) The data files then get put in the matplotlib directory in the zip. However, matplotlib searches in the mpl-data directory *inside* the matplotlib directory, as shown in the search paths above. I have not been able to find a way to include a directory within the matplotlib directory. If I use loader_files = [ ('matplotlib/mpl-data', mpl_datafilelist), ], py2app barfs when trying to copy the files since the directory doesn't exist. Any ideas Bob? Chris, can you please copy my email when replying to the thread? I'm getting the digest, so I don't get your messages for quite some time otherwise. Christopher Barker wrote: > I wonder what the plan should be for recipes for MPL and the like. > As Bob mentioned, perhaps it's best for the developer to find a way > to manually include what they want, as there is a LOT of stuff in > MPL (and certainly SciPy!) and much of will not be used by a given > application. If we can get numpy and scipy to work without a recipe that just includes the packages we can definitely do this. The problem here is not with py2app, but rather with numpy/scipy themselves and their crazy custom importer which doesn't work with a zipped site-packages. I am close to getting matplotlib working with a decent non-naive recipe. This means it'll only include what is being used by the application. > It seems the only way to make a recipe that is useful would be for > it to include EVERYTHING. That may not be a bad default, but there > should certainly be a way to turn it off if need be. No, a useful and "proper" recipe just adds the files that are not referenced directly through imports, ie., data files. This is what I am doing with matplotlib. > Also, I got a numpy app to work yesterday by using: > packages=['numpy'] The recipe I sent out in the zip yesterday did exactly that. Bob has added it to py2app trunk. There is a similar recipe for scipy and matplotlib as well. These will do until we get proper ones working. The packages option does include *everything*, by doing a full copy of the package. This means that source, documentation, example files, etc gets included along with the bytecode files. > That then includes the entire numpy package. That works, but it > puts in a lot of stuff I don't need (dft, linear_algebra, etc). > Maybe the recipe should include only the core stuff, and let the > user add the extra packages, if need be. AFAICT, this can't really be done. From looking at the numpy structure, there are subpackages 'testing','core','lib','linalg','dft','random','f2py', which are all loaded in the numpy __init__.py via the following: pkgload = PackageLoader() pkgload('testing','core','lib','linalg','dft','random','f2py', verbose=NUMPY_IMPORT_VERBOSE,postpone=False) The doc string for PackageLoader.__call__() says: > This function is intended to shorten the need to import many of > subpackages, say of scipy, constantly with statements such as > > import scipy.linalg, scipy.fftpack, scipy.etc... > > Instead, you can say: > > import scipy > scipy.pkgload('linalg','fftpack',...) > > or > > scipy.pkgload() so it seems like the entire purpose of PackageLoader is to make life difficult for me, just to save a few lines of typing. :) Seriously, can a numpy developer tell me why PackageLoader is necessary? > Or should numpy be fixed so that it doesn't do weird imports? The weird imports are okay, but they need to be able to work from a site-packages.zip. That's the next job. Hopefully I'll get time to look at it before numpy goes too far into beta. Scipy is a long way off. Cheers all, Josh From haase at msg.ucsf.edu Tue Jul 18 20:40:45 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 18 Jul 2006 17:40:45 -0700 Subject: [Numpy-discussion] numarray uses "rank" instead "ndim" Message-ID: <200607181740.45980.haase@msg.ucsf.edu> Hi, (Thanks to Travis and Todd for answering my last "dot"-question) I'm trying to adjust my numarray based code to also run with numpy. It seems that a (maybe the only left) problem is that numarray had arr.rank giving the value of len(arr.shape) while in numpy this is now called arr.ndim Is the "ndim" attribute already added in the CVS version (just now I could not connect to the CVS server) ? Or else, could it be added to numarray to ease the transition !? Another way might be to go back to len(arr.shape) - but that would be quite ugly ... Also I noted that a numpy array object does not allow "sticking" a new attribute into it: In numarray I could just say arr.xyz0='test' like with any (simple) Python object - in numpy I get an AttributeError: 'numpy.ndarray' object has no attribute 'xyz0' !? Thanks for everybody help. Sebastian Haase From oliphant at ee.byu.edu Tue Jul 18 20:45:18 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 18 Jul 2006 18:45:18 -0600 Subject: [Numpy-discussion] numarray uses "rank" instead "ndim" In-Reply-To: <200607181740.45980.haase@msg.ucsf.edu> References: <200607181740.45980.haase@msg.ucsf.edu> Message-ID: <44BD809E.8070106@ee.byu.edu> Sebastian Haase wrote: >Hi, >(Thanks to Travis and Todd for answering my last "dot"-question) > >I'm trying to adjust my numarray based code to also run with numpy. >It seems that a (maybe the only left) problem is that numarray had >arr.rank giving the value of len(arr.shape) >while in numpy this is now called >arr.ndim > >Is the "ndim" attribute already added in the CVS version (just now I could not >connect to the CVS server) ? Or else, could it be added to numarray to ease >the transition !? >Another way might be to go back to len(arr.shape) - but that would be quite >ugly ... > >Also I noted that a numpy array object does not allow "sticking" a new >attribute into it: In numarray I could just say arr.xyz0='test' like with >any (simple) Python object - in numpy I get an >AttributeError: 'numpy.ndarray' object has no attribute 'xyz0' >!? > > Why don't you try using a simple sub-class of numpy's ndarray class myarray(numpy.ndarray): pass That way your new array can look like a numarray array object and have the same methods and attributes. I've thought about even adding something like this to the numarray directory. -Travis From bob at redivi.com Tue Jul 18 20:57:29 2006 From: bob at redivi.com (Bob Ippolito) Date: Tue, 18 Jul 2006 17:57:29 -0700 Subject: [Numpy-discussion] [Pythonmac-SIG] Packaging numpy with py2app In-Reply-To: <0039ED8C-EA8A-47BE-9967-200AF66D173A@gmail.com> References: <660C6421-8C71-4B4D-A2CC-33C3FF089DF9@gmail.com> <44BC1B92.4090009@noaa.gov> <44BC1E6E.9020502@noaa.gov> <51B08665-6077-4E6F-875D-D217990B1714@redivi.com> <60E2AEE1-9BF1-4B62-A9DC-7A497310A681@gmail.com> <213C1E2D-AE4D-4578-9E22-FB49C1ACAA75@gmail.com> <9148A14B-7462-47EC-9262-52802E5A9CB4@gmail.com> <1CC7BE8A-F534-4256-A5A2-FEA2E1D2725A@redivi.com> <0039ED8C-EA8A-47BE-9967-200AF66D173A@gmail.com> Message-ID: On Jul 18, 2006, at 5:32 PM, Josh Marshall wrote: > Thanks for the info on how the various recipes work, Bob. Very > helpful. > > On 19/07/2006, at 9:28 AM, Bob Ippolito wrote: >> The recipe mechanism doesn't allow for it because it doesn't >> generally make sense. There are very few packages that can find >> their resources in an alternative manner. I'm not totally opposed >> to adding another feature to support that use case, but throwing it >> directly in the resources dir seems like a bad idea to do >> automatically from a recipe. The files should sit under some kind >> of matplotlib hierarchy. It would be nicer to see matplotlib just >> figure out how to make their app work from a zip... pygame does, >> and it has fonts and icons. > > I think matplotlib can access files inside the zip, since it searches > inside the following directories upon startup and is frozen-aware. > > MatplotlibEmbedSample.app/Contents/Resources/lib/python2.4/site- > packages.zip/matplotlib/mpl-data > MatplotlibEmbedSample.app/Contents/mpl-data > MatplotlibEmbedSample.app/Contents/Resources/mpl-data > > So I have been attempting to put the data files in the zip. > >> loader_files puts data files *in the zip* > > Indeed it does. When I use the following recipe: (noting that the > recipe's filename can't be "matplotlib.py", I've been using > "matplotlib_recipe.py"): > > def check(cmd, mf): > m = mf.findNode('matplotlib') > if m is None or m.filename is None: > return None > > import matplotlib, glob, os > mpl_datapath = matplotlib.get_data_path() > mpl_datafilelist = glob.glob(mpl_datapath + r'/*') > > # only include individual files from the mpl-data directory > # this should exclude directories, which should only be > Matplotlib.nib > mpl_datafilelist = [mpl_file for mpl_file in mpl_datafilelist > if os.path.isfile(mpl_file)] > > mf.import_hook('pytz.zoneinfo', m, ['UTC']) > mf.import_hook('matplotlib.numerix', m, ['random_array']) > return dict( > loader_files = [ > ('matplotlib', mpl_datafilelist), > ], > ) > > The data files then get put in the matplotlib directory in the zip. > However, matplotlib searches in the mpl-data directory *inside* the > matplotlib directory, as shown in the search paths above. > > I have not been able to find a way to include a directory within the > matplotlib directory. If I use > loader_files = [ > ('matplotlib/mpl-data', mpl_datafilelist), > ], > py2app barfs when trying to copy the files since the directory > doesn't exist. Any ideas Bob? This should work in svn (r13). You can easily run the development version straight from a svn checkout: svn co http://svn.pythonmac.org/py2app/py2app/trunk py2app cd py2app python setup.py develop That'll set it up such that the "installed" py2app is actually what's sitting in your svn checkout. You can make changes without re- installing or anything. -bob From josh.p.marshall at gmail.com Tue Jul 18 22:26:51 2006 From: josh.p.marshall at gmail.com (Josh Marshall) Date: Wed, 19 Jul 2006 12:26:51 +1000 Subject: [Numpy-discussion] [Pythonmac-SIG] Packaging numpy with py2app In-Reply-To: References: <660C6421-8C71-4B4D-A2CC-33C3FF089DF9@gmail.com> <44BC1B92.4090009@noaa.gov> <44BC1E6E.9020502@noaa.gov> <51B08665-6077-4E6F-875D-D217990B1714@redivi.com> <60E2AEE1-9BF1-4B62-A9DC-7A497310A681@gmail.com> <213C1E2D-AE4D-4578-9E22-FB49C1ACAA75@gmail.com> <9148A14B-7462-47EC-9262-52802E5A9CB4@gmail.com> <1CC7BE8A-F534-4256-A5A2-FEA2E1D2725A@redivi.com> <0039ED8C-EA8A-47BE-9967-200AF66D173A@gmail.com> Message-ID: <98FDFEDF-CB5D-4014-8214-653FC4CA1CCA@gmail.com> Thanks for the fix Bob. Unfortunately Matplotlib does not work with zipped data files, after all that. So, we'll leave the recipes as is, as they work for now. I suspect the way forward is to get numpy/Matplotlib/scipy working with setuptools and using pkg_resources to manage the data files. Any thoughts on this? From bob at redivi.com Tue Jul 18 23:15:29 2006 From: bob at redivi.com (Bob Ippolito) Date: Tue, 18 Jul 2006 20:15:29 -0700 Subject: [Numpy-discussion] [Pythonmac-SIG] Packaging numpy with py2app In-Reply-To: <98FDFEDF-CB5D-4014-8214-653FC4CA1CCA@gmail.com> References: <660C6421-8C71-4B4D-A2CC-33C3FF089DF9@gmail.com> <44BC1B92.4090009@noaa.gov> <44BC1E6E.9020502@noaa.gov> <51B08665-6077-4E6F-875D-D217990B1714@redivi.com> <60E2AEE1-9BF1-4B62-A9DC-7A497310A681@gmail.com> <213C1E2D-AE4D-4578-9E22-FB49C1ACAA75@gmail.com> <9148A14B-7462-47EC-9262-52802E5A9CB4@gmail.com> <1CC7BE8A-F534-4256-A5A2-FEA2E1D2725A@redivi.com> <0039ED8C-EA8A-47BE-9967-200AF66D173A@gmail.com> <98FDFEDF-CB5D-4014-8214-653FC4CA1CCA@gmail.com> Message-ID: On Jul 18, 2006, at 7:26 PM, Josh Marshall wrote: > Thanks for the fix Bob. > > Unfortunately Matplotlib does not work with zipped data files, > after all that. So, we'll leave the recipes as is, as they work for > now. > > I suspect the way forward is to get numpy/Matplotlib/scipy working > with setuptools and using pkg_resources to manage the data files. > Any thoughts on this? Yes, that is the way. -bob From haase at msg.ucsf.edu Wed Jul 19 00:04:47 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 18 Jul 2006 21:04:47 -0700 Subject: [Numpy-discussion] calling a f2py made module with numarray (mem shared with numpy ?) Message-ID: <44BDAF5F.40701@msg.ucsf.edu> Hi, Having to use Fortran again, I had a look at f2py .. I looks really great !! Everything seems to run out of the box !! Thanks a lot ! I think I tried intend(inout) and then fed in an array object that really was an numarray as opposed to a numpy array. I was surprised that the array did not get altered by the function ! If I assign numpyArray = numpy.asarray(numarrayArr) don't numpyArray and numarrayArr share identical memory for their data? Another observation I made was that a C-contiguous array of rank 2 was not accepted by 'intend(inout)' (the error message said that it had to be FORTRAN-contiguous - as opposed to the f2py documentation I read online http://cens.ioc.ee/projects/f2py2e/usersguide/index.html#attributes which says : fortran OR c contiguous ) Thanks for another great tool ! Sebastian Haase From robert.kern at gmail.com Wed Jul 19 00:16:50 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 18 Jul 2006 23:16:50 -0500 Subject: [Numpy-discussion] [Pythonmac-SIG] Packaging numpy with py2app In-Reply-To: <0039ED8C-EA8A-47BE-9967-200AF66D173A@gmail.com> References: <660C6421-8C71-4B4D-A2CC-33C3FF089DF9@gmail.com> <44BC1B92.4090009@noaa.gov> <44BC1E6E.9020502@noaa.gov> <51B08665-6077-4E6F-875D-D217990B1714@redivi.com> <60E2AEE1-9BF1-4B62-A9DC-7A497310A681@gmail.com> <213C1E2D-AE4D-4578-9E22-FB49C1ACAA75@gmail.com> <9148A14B-7462-47EC-9262-52802E5A9CB4@gmail.com> <1CC7BE8A-F534-4256-A5A2-FEA2E1D2725A@redivi.com> <0039ED8C-EA8A-47BE-9967-200AF66D173A@gmail.com> Message-ID: Josh Marshall wrote: > so it seems like the entire purpose of PackageLoader is to make life > difficult for me, just to save a few lines of typing. :) Seriously, > can a numpy developer tell me why PackageLoader is necessary? I can't think of a good reason why it's used in __init__.py the way it is (it used to have postpone=True). If Pearu, who wrote that bit of code, doesn't speak up by Thursday, I'll have it removed in favor of regular imports for the beta. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant.travis at ieee.org Wed Jul 19 00:48:30 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 18 Jul 2006 22:48:30 -0600 Subject: [Numpy-discussion] [Pythonmac-SIG] Packaging numpy with py2app In-Reply-To: References: <660C6421-8C71-4B4D-A2CC-33C3FF089DF9@gmail.com> <44BC1B92.4090009@noaa.gov> <44BC1E6E.9020502@noaa.gov> <51B08665-6077-4E6F-875D-D217990B1714@redivi.com> <60E2AEE1-9BF1-4B62-A9DC-7A497310A681@gmail.com> <213C1E2D-AE4D-4578-9E22-FB49C1ACAA75@gmail.com> <9148A14B-7462-47EC-9262-52802E5A9CB4@gmail.com> <1CC7BE8A-F534-4256-A5A2-FEA2E1D2725A@redivi.com> <0039ED8C-EA8A-47BE-9967-200AF66D173A@gmail.com> Message-ID: <44BDB99E.70406@ieee.org> Robert Kern wrote: > Josh Marshall wrote: > >> so it seems like the entire purpose of PackageLoader is to make life >> difficult for me, just to save a few lines of typing. :) Seriously, >> can a numpy developer tell me why PackageLoader is necessary? >> > > I can't think of a good reason why it's used in __init__.py the way it is (it > used to have postpone=True). If Pearu, who wrote that bit of code, doesn't speak > up by Thursday, I'll have it removed in favor of regular imports for the beta. > > This will probably help with ticket #145 as well. A user of ActiveState's Komodo on Windows gets a program crash which started when the PackageLoader started getting used in 0.9.4 AFAIK, The two main purposes of the package loader are docstring manipulation and postponing imports. Right now, it's probably only the docstring portion of the code that is really being used. I would appreciate hearing Pearu's comments, however. From robert.kern at gmail.com Wed Jul 19 00:52:52 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 18 Jul 2006 23:52:52 -0500 Subject: [Numpy-discussion] [Pythonmac-SIG] Packaging numpy with py2app In-Reply-To: <44BDB99E.70406@ieee.org> References: <660C6421-8C71-4B4D-A2CC-33C3FF089DF9@gmail.com> <44BC1B92.4090009@noaa.gov> <44BC1E6E.9020502@noaa.gov> <51B08665-6077-4E6F-875D-D217990B1714@redivi.com> <60E2AEE1-9BF1-4B62-A9DC-7A497310A681@gmail.com> <213C1E2D-AE4D-4578-9E22-FB49C1ACAA75@gmail.com> <9148A14B-7462-47EC-9262-52802E5A9CB4@gmail.com> <1CC7BE8A-F534-4256-A5A2-FEA2E1D2725A@redivi.com> <0039ED8C-EA8A-47BE-9967-200AF66D173A@gmail.com> <44BDB99E.70406@ieee.org> Message-ID: Travis Oliphant wrote: > Robert Kern wrote: >> Josh Marshall wrote: >> >>> so it seems like the entire purpose of PackageLoader is to make life >>> difficult for me, just to save a few lines of typing. :) Seriously, >>> can a numpy developer tell me why PackageLoader is necessary? >>> >> I can't think of a good reason why it's used in __init__.py the way it is (it >> used to have postpone=True). If Pearu, who wrote that bit of code, doesn't speak >> up by Thursday, I'll have it removed in favor of regular imports for the beta. >> > This will probably help with ticket #145 as well. A user of > ActiveState's Komodo on Windows gets a program crash which started when > the PackageLoader started getting used in 0.9.4 > > AFAIK, The two main purposes of the package loader are docstring > manipulation and postponing imports. Right now, it's probably only the > docstring portion of the code that is really being used. There's also the configurable verbosity and hiding of ImportErrors. Admittedly, I dislike both of those. Currently, I don't think any of these features are worth messing with something as fundamental as importing. Especially not in __init__.py . It always works out to introduce fragility in something that needs to be rock-solid. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From bob at redivi.com Wed Jul 19 01:27:04 2006 From: bob at redivi.com (Bob Ippolito) Date: Tue, 18 Jul 2006 22:27:04 -0700 Subject: [Numpy-discussion] [Pythonmac-SIG] Packaging numpy with py2app In-Reply-To: References: <660C6421-8C71-4B4D-A2CC-33C3FF089DF9@gmail.com> <44BC1B92.4090009@noaa.gov> <44BC1E6E.9020502@noaa.gov> <51B08665-6077-4E6F-875D-D217990B1714@redivi.com> <60E2AEE1-9BF1-4B62-A9DC-7A497310A681@gmail.com> <213C1E2D-AE4D-4578-9E22-FB49C1ACAA75@gmail.com> <9148A14B-7462-47EC-9262-52802E5A9CB4@gmail.com> <1CC7BE8A-F534-4256-A5A2-FEA2E1D2725A@redivi.com> <0039ED8C-EA8A-47BE-9967-200AF66D173A@gmail.com> Message-ID: <61EE48D5-1689-4C46-9CDA-C1BF76500B5E@redivi.com> On Jul 18, 2006, at 9:16 PM, Robert Kern wrote: > Josh Marshall wrote: >> so it seems like the entire purpose of PackageLoader is to make life >> difficult for me, just to save a few lines of typing. :) Seriously, >> can a numpy developer tell me why PackageLoader is necessary? > > I can't think of a good reason why it's used in __init__.py the way > it is (it > used to have postpone=True). If Pearu, who wrote that bit of code, > doesn't speak > up by Thursday, I'll have it removed in favor of regular imports > for the beta. Note that all you need is the bytecode, it doesn't have to actually execute. So you can still use whatever custom import junk if you really want to: def _modulegraph_hints(): import foo You should do it in a dead function instead of "if 0" because dead functions will never be stripped by the compiler but dead code blocks sometimes are. -bob From verveer at embl-heidelberg.de Wed Jul 19 03:10:51 2006 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Wed, 19 Jul 2006 09:10:51 +0200 Subject: [Numpy-discussion] What to do about numarray extension modules? In-Reply-To: <44BD2D91.1020800@ieee.org> References: <44BD2D91.1020800@ieee.org> Message-ID: <8B1708B4-FE99-4323-8CDA-20F23C16BDDA@embl-heidelberg.de> I am the author of nd_image. In my opinion the package should remain in SciPy, where it indeed belongs. The only reason it was in numarray is that there was no such thing as SciPy for numarray. Although it is relatively bug-free and has a reasonable amount of image processing functions, it could stand a lot of improvement. Including it in a base NumPy package would be a mistake, as it is not a fully optimized and finished package. nd_image should install easily as a separate package, hence it could be included in SciPy while maintaining the possibility for standalone usage. Peter > > I'd like to get suggestions about what to do about the remaining > numarray extension modules that are not addressed in NumPy (convolve, > image, nd_image). There will be people coming from numarray that > will > be using these modules. > > Of course, my opinion is that they should not have been placed in > Numarray to begin with as the fit better as modules in SciPy. > > But, given the situation. Should we bring them over to NumPy? Or, > perhaps provide a small package that contain these modules separately? > > With the numarray c_api that is now available in NumPy, these can be > easily ported. It would be nice to have decided this by 1.0beta > release > at the end of the week. > > Discussion please... > > -Travis > > > ---------------------------------------------------------------------- > --- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to > share your > opinions on IT & business topics through brief surveys -- and earn > cash > http://www.techsay.com/default.php? > page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From oliphant.travis at ieee.org Wed Jul 19 03:57:22 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 19 Jul 2006 01:57:22 -0600 Subject: [Numpy-discussion] Improved creation of object arrays Message-ID: <44BDE5E2.2060703@ieee.org> I made a small change to the array creation function so that if you explicitly specify dtype=object, then the logic for determining the shape of the array is changed. The new logic for constructing dtype=object arrays from Python sequences is that the shape is determined by nested lists (and only lists). The nesting must be consistent or else the elements of the last conforming lists are assumed to be objects. Here are some examples: array([[1,2],3,4],dtype=object) returns a 1-d array of shape (3,) array([[1,2],[3,4]],dtype=object) returns a 2-d array of shape (2,2) array([(1,2),(3,4)],dtype=object) returns a 1-d array of shape (2,) array([],dtype=object) returns a 0-d array of shape () array([[],[],[]],dtype=object) returns a 1-d array of shape (3,) array([[3,4],[5,6],None],dtype=object) returns a 1-d array of shape (3,) You have to actually specify dtype=object to get this, otherwise the old code-path will be taken. -Travis From fullung at gmail.com Wed Jul 19 08:22:15 2006 From: fullung at gmail.com (Albert Strasheim) Date: Wed, 19 Jul 2006 14:22:15 +0200 Subject: [Numpy-discussion] Allocating discontiguous arrays Message-ID: <035201c6ab2d$f88ce8a0$0100000a@dsp.sun.ac.za> Hello all In some situations, I have to work with very large matrices. My Windows machine has 3 GB RAM, so I would expect to be able to use most of my process's address space for my matrix. Unfortunately, with matrices much larger than 700 or 800 MB, one starts running into heap fragmentation problems: even though there's 2 GB available to your process, it isn't available in one contiguous block. To see this, you can try the following code which tries to allocate a ~1792 MB 2-d array or a list of 1-d arrays that add up to the same size: import numpy as N fdtype = N.dtype(' For 1-d inputs I think r_ should act like vstack, and c_ should act like column_stack. Currently r_ and c_ both act like hstack for 1-d inputs. Background: I keep getting bitten by the fact that this doesn't work: >>> a = array([1,2,3]) >>> b = array([[1,2,3],[2,3,4]]) >>> c = array([4,5,6]) >>> r_[b,c] **error** and that this doesn't return a 2x3 >>> d = r_[a,c] array([1, 2, 3, 4, 5, 6]) To get what I want I need something like : >>> r_[[a],[c]] array([[1, 2, 3], [4, 5, 6]]) And to get them columnwise I need the likes of: >>> c_[a[:,newaxis], c[:,newaxis]] array([[1, 4], [2, 5], [3, 6]]) It seems like the r_ (which I think of as a "row concatenator") should assume that 1-d arrays are rows (like vstack does), and the c_ ("column concatenator") should assume 1-d arrays are columns (like column_stack does). Then you'd have: >>> r_[a,c] array([[1, 2, 3], [4, 5, 6]]) >>> c_[a,c] array([[1, 4], [2, 5], [3, 6]]) At any rate, r_'s behavior is currently different from vstack wrt 1-d arrays, and identitcal to c_ and hstack. Vstack, hstack, and column_stack give the following: >>> vstack([a,c]) array([[1, 2, 3], [4, 5, 6]]) >>> hstack([a,c]) array([1, 2, 3, 4, 5, 6]) >>> column_stack([a,c]) array([[1, 4], [2, 5], [3, 6]]) Also isn't it odd that there's a column_stack, but no row_stack? I guess that's because row_stack would just be an alias for vstack, but still. --bb From sebzur at pin.if.uz.zgora.pl Wed Jul 19 09:33:08 2006 From: sebzur at pin.if.uz.zgora.pl (=?UTF-8?B?U2ViYXN0aWFuIMW7dXJlaw==?=) Date: Wed, 19 Jul 2006 14:33:08 +0100 Subject: [Numpy-discussion] RandomArray module In-Reply-To: References: Message-ID: Thanks a lot Francescm, Bruce and Robert! Your response was all the essentials I wanted to know. As soon as I'll be back from my vacations time I'll try numpys PRNG. Sebastian From svetosch at gmx.net Wed Jul 19 09:06:34 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Wed, 19 Jul 2006 15:06:34 +0200 Subject: [Numpy-discussion] r_, c_, hstack, and vstack with 1-d arrays In-Reply-To: References: Message-ID: <44BE2E5A.80404@gmx.net> Bill Baxter schrieb: > For 1-d inputs I think r_ should act like vstack, and c_ should act > like column_stack. > Currently r_ and c_ both act like hstack for 1-d inputs. Well Bill, as I said before, you should have stayed with matrices ;-) Seriously, I think all the quirks you mentioned recently are now gone. I mean what you're experiencing now ist just logical from the point of view that rows and columns are 2d concepts, so a 1d array just isn't enough. Having said that, however, here are some more constructive comments: First, I think using c_ is not recommended, it doesn't even appear in the book. But I have no idea why that is so, I'm using it also (with matrices), so I think it's valid to question that deprecation. Indeed it seems natural to make it act like column_stack for 1d arrays!? Second, changing r_ would obviously break some people's codes. Two workarounds: You switch to matrices (sorry, couldn't resist), or you use vstack (trivial, sorry again). > > Also isn't it odd that there's a column_stack, but no row_stack? I > guess that's because row_stack would just be an alias for vstack, but > still. > I agree that for the poor non-matrix-users it would be good to have a pair of names that obviously belong together, instead of the current vstack/column_stack situation; with matrices you have the choice of r_/c_ or vstack/hstack that do exactly what you want (as you know). Good luck, Sven From wbaxter at gmail.com Wed Jul 19 09:39:12 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 19 Jul 2006 22:39:12 +0900 Subject: [Numpy-discussion] r_, c_, hstack, and vstack with 1-d arrays In-Reply-To: <44BE2E5A.80404@gmx.net> References: <44BE2E5A.80404@gmx.net> Message-ID: On 7/19/06, Sven Schreiber wrote: > Bill Baxter schrieb: > > For 1-d inputs I think r_ should act like vstack, and c_ should act > > like column_stack. > > Currently r_ and c_ both act like hstack for 1-d inputs. > > Well Bill, as I said before, you should have stayed with matrices ;-) :-D > Seriously, I think all the quirks you mentioned recently are now gone. Yeh, probably so. The main one remaining for me is that matrix can't take more than 2-D data. So if I decide to go, say, from storing a single polygon (N points x 2 coords), to a set of deformed versions of the polygon (M deformations x N points x 2 coords), then suddenly I have to switch a bunch of matrix types back to array types, and find and fix all the small differences between the two, and deal with two different types in my code (array and matrix) instead of one type everywhere (array). Or maybe I have to use a Python list of matrix. Maybe it wouldn't be as bad as I imagine. I haven't actually gone through such a conversion with matrices. It just seems like it would probably be more pain than it is to add an extra dimension without changing data type. > I mean what you're experiencing now ist just logical from the point of > view that rows and columns are 2d concepts, so a 1d array just isn't enough. True, but other parts of Numpy handle that ambiguity more gracefully. Like dot(), or column_stack. > Having said that, however, here are some more constructive comments: > > First, I think using c_ is not recommended, it doesn't even appear in > the book. But I have no idea why that is so, I'm using it also (with > matrices), so I think it's valid to question that deprecation. Indeed it > seems natural to make it act like column_stack for 1d arrays!? Hmm, yeh, what is the status on r_ and c_? My understanding was that they were just new/experimental, and thus not documented in the book. I think they're quite handy, so I hope they're not going away. There was a discussion previously about changing the names to something a little more attractive, but I've gotten pretty used to the names now. > Second, changing r_ would obviously break some people's codes. Two > workarounds: You switch to matrices (sorry, couldn't resist), or you use > vstack (trivial, sorry again). Well, if they are new, probably not that much code. > > Also isn't it odd that there's a column_stack, but no row_stack? I > > guess that's because row_stack would just be an alias for vstack, but > > still. > I agree that for the poor non-matrix-users it would be good to have a > pair of names that obviously belong together, instead of the current > vstack/column_stack situation; with matrices you have the choice of > r_/c_ or vstack/hstack that do exactly what you want (as you know). --bill From yves.frederix at cs.kuleuven.be Wed Jul 19 10:45:08 2006 From: yves.frederix at cs.kuleuven.be (Yves Frederix) Date: Wed, 19 Jul 2006 16:45:08 +0200 Subject: [Numpy-discussion] astype() causes segmentation fault when converting from string Message-ID: <44BE4574.3040704@cs.kuleuven.be> Hi, The latest svn version of numpy seems to be causing a segmentation fault when converting an array containing strings to floats using .astype(). It only seems to happen when converting 'from' strings. Casting between numerical types and 'to' strings is no problem. :yves $ python -c "import numpy as N; print N.__version__; a = N.asarray('2'); a.astype(float);print 'OK';" 0.9.9.2844 Segmentation fault An older version of numpy (the one I used before this upgrade) works perfectly though: :yves $ python -c "import numpy as N; print N.__version__; a = N.asarray('2'); a.astype(float);print 'OK';" 0.9.9.2655 OK Cheers, YVES From bioinformed at gmail.com Wed Jul 19 11:24:40 2006 From: bioinformed at gmail.com (Kevin Jacobs ) Date: Wed, 19 Jul 2006 11:24:40 -0400 Subject: [Numpy-discussion] Performance of defmatrix.py:103(__array_finalize__) Message-ID: <2e1434c10607190824l322ae6d4p128dc9c35a637f16@mail.gmail.com> Hi all, Is it expected that the following script would spend over half of the total execution time in defmatrix.py:103(__array_finalize__)? from numpy import matrix m=matrix( (1,1) ) for i in range(1000000): m[0,0] = 0. Python version is 2.5b1, NumPy 0.9.8. More importantly, is there a way to avoid this overhead? Thanks, -Kevin Jacobs -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.u.r.e.l.i.a.n at gmx.net Wed Jul 19 11:22:23 2006 From: a.u.r.e.l.i.a.n at gmx.net (Johannes Loehnert) Date: Wed, 19 Jul 2006 17:22:23 +0200 Subject: [Numpy-discussion] confirm 98d7f7f15c27f5122181163442d621a0b506d01c In-Reply-To: References: Message-ID: <200607191722.24226.a.u.r.e.l.i.a.n@gmx.net> On Wednesday 19 July 2006 14:41, numpy-discussion-request at lists.sourceforge.net wrote: > Your membership in the mailing list Numpy-discussion has been disabled > due to excessive bounces The last bounce received from you was dated > 19-Jul-2006. You will not get any more messages from this list until > you re-enable your membership. You will receive 3 more reminders like > this before your membership in the list is deleted. > > To re-enable your membership, you can simply respond to this message > (leaving the Subject: line intact), or visit the confirmation page at > > > https://lists.sourceforge.net/lists/confirm/numpy-discussion/98d7f7f15c27f5 >122181163442d621a0b506d01c > > > You can also visit your membership page at > > > https://lists.sourceforge.net/lists/options/numpy-discussion/a.u.r.e.l.i.a. >n%40gmx.net > > > On your membership page, you can change various delivery options such > as your email address and whether you get digests or not. As a > reminder, your membership password is > > numpylist > > If you have any questions or problems, you can contact the list owner > at > > numpy-discussion-owner at lists.sourceforge.net From oliphant.travis at ieee.org Wed Jul 19 11:34:13 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 19 Jul 2006 09:34:13 -0600 Subject: [Numpy-discussion] astype() causes segmentation fault when converting from string In-Reply-To: <44BE4574.3040704@cs.kuleuven.be> References: <44BE4574.3040704@cs.kuleuven.be> Message-ID: <44BE50F5.9090609@ieee.org> Yves Frederix wrote: > Hi, > > The latest svn version of numpy seems to be causing a segmentation fault > when converting an array containing strings to floats using .astype(). > It only seems to happen when converting 'from' strings. Casting between > numerical types and 'to' strings is no problem. > > :yves $ python -c "import numpy as N; print N.__version__; a = > N.asarray('2'); a.astype(float);print 'OK';" > 0.9.9.2844 > Segmentation fault > > A bad attempt at allowing other threads to run is at fault. We can't release the GIL for casting to and/or from strings.... This should be fixed now. -Travis From bioinformed at gmail.com Wed Jul 19 11:36:41 2006 From: bioinformed at gmail.com (Kevin Jacobs ) Date: Wed, 19 Jul 2006 11:36:41 -0400 Subject: [Numpy-discussion] Performance of defmatrix.py:103(__array_finalize__) In-Reply-To: <2e1434c10607190824l322ae6d4p128dc9c35a637f16@mail.gmail.com> References: <2e1434c10607190824l322ae6d4p128dc9c35a637f16@mail.gmail.com> Message-ID: <2e1434c10607190836g282d7e70hcf34145b222ad462@mail.gmail.com> On 7/19/06, Kevin Jacobs wrote: > > Is it expected that the following script would spend over half of the > total execution time in defmatrix.py:103(__array_finalize__)? > Confirmed that a similar overhead exists for arrays, though not as extreme. My application requires the ability to assign random elements, so any speed ups here will result be quite significant. Thanks, -Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Wed Jul 19 11:46:20 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 19 Jul 2006 09:46:20 -0600 Subject: [Numpy-discussion] Performance of defmatrix.py:103(__array_finalize__) In-Reply-To: <2e1434c10607190824l322ae6d4p128dc9c35a637f16@mail.gmail.com> References: <2e1434c10607190824l322ae6d4p128dc9c35a637f16@mail.gmail.com> Message-ID: <44BE53CC.1020002@ieee.org> Kevin Jacobs wrote: > Hi all, > > Is it expected that the following script would spend over half of the > total execution time in defmatrix.py:103(__array_finalize__)? > > from numpy import matrix > > m=matrix( (1,1) ) > for i in range(1000000): > m[0,0] = 0. > > > Python version is 2.5b1, NumPy 0.9.8. > > More importantly, is there a way to avoid this overhead? This over-head should be removed in latest SVN NumPy. Basically, at each point in the loop the 0. is being converted to a matrix. You could probably avoid the over-head by doing the conversion first to a 0-d array. val = array(0.) m = matrix( (1,1) ) for i in range(10000): m[0,0] = val -Travis From oliphant.travis at ieee.org Wed Jul 19 12:08:52 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 19 Jul 2006 10:08:52 -0600 Subject: [Numpy-discussion] Improved creation of object arrays In-Reply-To: <44BDE5E2.2060703@ieee.org> References: <44BDE5E2.2060703@ieee.org> Message-ID: <44BE5914.3050200@ieee.org> Travis Oliphant wrote: > I made a small change to the array creation function so that if you > explicitly specify dtype=object, then the logic for determining the > shape of the array is changed. > > The new logic for constructing dtype=object arrays from Python sequences > is that the shape is determined by nested lists (and only lists). The > nesting must be consistent or else the elements of the last conforming > lists are assumed to be objects. > Update: Now lists and tuples are both allowed for determining nested lists (that was needed for record arrays). The shapes still have to match for a new dimension to be "detected" -Travis From oliphant at ee.byu.edu Wed Jul 19 17:25:41 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed, 19 Jul 2006 15:25:41 -0600 Subject: [Numpy-discussion] Performance of defmatrix.py:103(__array_finalize__) In-Reply-To: <2e1434c10607191416p7dc055e1ie2194133e4d9dd3d@mail.gmail.com> References: <2e1434c10607190824l322ae6d4p128dc9c35a637f16@mail.gmail.com> <2e1434c10607190836g282d7e70hcf34145b222ad462@mail.gmail.com> <44BE5663.1090405@ieee.org> <2e1434c10607191416p7dc055e1ie2194133e4d9dd3d@mail.gmail.com> Message-ID: <44BEA355.7030406@ee.byu.edu> Kevin Jacobs wrote: > On 7/19/06, *Travis Oliphant* > wrote: > > Kevin Jacobs > wrote: > > On 7/19/06, *Kevin Jacobs > > >>* > > >> > wrote: > > > > Is it expected that the following script would spend over > half of > > the total execution time in > defmatrix.py:103(__array_finalize__)? > > > > > > Confirmed that a similar overhead exists for arrays, though not as > > extreme. My application requires the ability to assign random > > elements, so any speed ups here will result be quite significant. > > > There is an optimization for this kind of assignment in current NumPy > SVN which "should" avoid the array creation over-head. > > > > There still seems to be a huge performance penalty with using a matrix > to set elements. Consider this code where X is initially a matrix: Are you now using SVN NumPy, or are we still looking at NumPy 0.9.8 numbers? -Travis From bioinformed at gmail.com Wed Jul 19 17:31:05 2006 From: bioinformed at gmail.com (Kevin Jacobs ) Date: Wed, 19 Jul 2006 17:31:05 -0400 Subject: [Numpy-discussion] Performance of defmatrix.py:103(__array_finalize__) In-Reply-To: <44BEA355.7030406@ee.byu.edu> References: <2e1434c10607190824l322ae6d4p128dc9c35a637f16@mail.gmail.com> <2e1434c10607190836g282d7e70hcf34145b222ad462@mail.gmail.com> <44BE5663.1090405@ieee.org> <2e1434c10607191416p7dc055e1ie2194133e4d9dd3d@mail.gmail.com> <44BEA355.7030406@ee.byu.edu> Message-ID: <2e1434c10607191431u1f01c851v6d21e7efe771db01@mail.gmail.com> On 7/19/06, Travis Oliphant wrote: > > Are you now using SVN NumPy, or are we still looking at NumPy 0.9.8numbers? Sorry -- SVN updated as of about two hours ago. Thanks, -Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: From webb.sprague at gmail.com Wed Jul 19 18:06:00 2006 From: webb.sprague at gmail.com (Webb Sprague) Date: Wed, 19 Jul 2006 15:06:00 -0700 Subject: [Numpy-discussion] Pretty printing an array: no ellipses? Message-ID: I am not sure where to look for this, sorry if it is RTFM or JPS ("just plain stupid"): Is there a way to set a default to print the entire array, rather than an ellipses version of it? If not, why doesn't pprint.pformat(numpy.random.normal(0,1,(100, 100)), width=1000) at least give me something off the screen? Thx! From xaz39 at 163.com Sun Jul 23 20:47:56 2006 From: xaz39 at 163.com (=?GB2312?B?IjfUwjI5LTMwyNUvyc+6oyI=?=) Date: Mon, 24 Jul 2006 08:47:56 +0800 Subject: [Numpy-discussion] =?GB2312?B?cmU61MvTw0lFvLzK9czhuN/Qp8LKvqvS5sn6svo=?= Message-ID: An HTML attachment was scrubbed... URL: From stephen.walton at csun.edu Thu Jul 20 00:44:08 2006 From: stephen.walton at csun.edu (Stephen Walton) Date: Wed, 19 Jul 2006 21:44:08 -0700 Subject: [Numpy-discussion] Atlas and dual core In-Reply-To: References: Message-ID: <44BF0A18.4090406@csun.edu> Keith Goodman wrote: >Is there much speed to be gained by compiling atlas for a dual core system? > > I imagine the answer is yes. However, Clint Whaley literally just today announced the first release of a beta version of ATLAS whose configuration utility actually supports Core Duo, although probably only under OSX. Check Clint's message at http://sourceforge.net/mailarchive/forum.php?thread_id=25989825&forum_id=426 By the way, everyone should also read the immediately previous message, posted a few weeks earlier by Clint. gcc 4.x gives significantly worse floating point performance than gcc 3.x on most x86 platforms. Something to be aware of if you're benchmarking. He's communicated about the problem to the gcc developers, but is, ah, less than confident they'll fix the problem. Steve From davidgrant at gmail.com Thu Jul 20 01:59:11 2006 From: davidgrant at gmail.com (David Grant) Date: Wed, 19 Jul 2006 22:59:11 -0700 Subject: [Numpy-discussion] Profiling line-by-line Message-ID: Is there any way to do line-by-line profiling in Python? The profiling results can tell me how much time is spent in all functions, but within a given function I can't get any idea of how much time was spent on each line. For example, in the example below, I can see that graphWidth.py is taking all the time, but there are many lines of code in graphWidth.py that aren't function calls, and I have no way of knowing which lines are the bottlenecks. I'm using hotspot currently, by the way. ncalls tottime percall cumtime percall filename:lineno(function) 1 0.215 0.215 0.221 0.221 graphWidth.py:6(graphWidth) 27 0.001 0.000 0.003 0.000 oldnumeric.py:472(all) 26 0.002 0.000 0.002 0.000 oldnumeric.py:410(sum) 26 0.001 0.000 0.002 0.000 oldnumeric.py:163(_wrapit) 26 0.001 0.000 0.001 0.000 oldnumeric.py:283(argmin) 26 0.000 0.000 0.000 0.000 numeric.py:111(asarray) 0 0.000 0.000 profile:0(profiler) Thanks, -- David Grant -------------- next part -------------- An HTML attachment was scrubbed... URL: From arnd.baecker at web.de Thu Jul 20 03:06:35 2006 From: arnd.baecker at web.de (Arnd Baecker) Date: Thu, 20 Jul 2006 09:06:35 +0200 (CEST) Subject: [Numpy-discussion] Profiling line-by-line In-Reply-To: References: Message-ID: On Wed, 19 Jul 2006, David Grant wrote: > Is there any way to do line-by-line profiling in Python? The profiling > results can tell me how much time is spent in all functions, but within a > given function I can't get any idea of how much time was spent on each line. > For example, in the example below, I can see that graphWidth.py is taking > all the time, but there are many lines of code in graphWidth.py that aren't > function calls, and I have no way of knowing which lines are the > bottlenecks. I'm using hotspot currently, by the way. > > ncalls tottime percall cumtime percall filename:lineno(function) > 1 0.215 0.215 0.221 0.221 graphWidth.py:6(graphWidth) > 27 0.001 0.000 0.003 0.000 oldnumeric.py:472(all) > 26 0.002 0.000 0.002 0.000 oldnumeric.py:410(sum) > 26 0.001 0.000 0.002 0.000 oldnumeric.py:163(_wrapit) > 26 0.001 0.000 0.001 0.000 oldnumeric.py:283(argmin) > 26 0.000 0.000 0.000 0.000 numeric.py:111(asarray) > 0 0.000 0.000 profile:0(profiler) > > Thanks, You might give hotshot2kcachegrind a try. See http://mail.enthought.com/pipermail/enthought-dev/2006-January/001075.html for more details. The screenshots http://www.physik.tu-dresden.de/~baecker/tmp/bench_traits/kcachegrind_screenshot.png http://kcachegrind.sourceforge.net/cgi-bin/show.cgi/KcacheGrindShot4 might give you an idea how things will look. More importantly note that profiling in connection with ufuncs seems problematic: See this thread (unfortunately split into several pieces, not sure if I got all of them): http://thread.gmane.org/gmane.comp.python.numeric.general/5309/focus=5309 http://thread.gmane.org/gmane.comp.python.numeric.general/5311/focus=5316 http://thread.gmane.org/gmane.comp.python.numeric.general/5337/focus=5337 (I always wanted to write this up for the wiki, but "real work" is interfering too strongly at the moment ;-). Good luck with profiling, Arnd From stefan at sun.ac.za Thu Jul 20 07:51:12 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu, 20 Jul 2006 13:51:12 +0200 Subject: [Numpy-discussion] Pretty printing an array: no ellipses? In-Reply-To: References: Message-ID: <20060720115112.GJ23629@mentat.za.net> On Wed, Jul 19, 2006 at 03:06:00PM -0700, Webb Sprague wrote: > I am not sure where to look for this, sorry if it is RTFM or JPS > ("just plain stupid"): > > Is there a way to set a default to print the entire array, rather than > an ellipses version of it? If not, why doesn't > pprint.pformat(numpy.random.normal(0,1,(100, 100)), width=1000) at > least give me something off the screen? Try N.set_printoptions(threshold=N.nan) St?fan From Ferenc.Pintye at eu.decoma.com Thu Jul 20 08:36:12 2006 From: Ferenc.Pintye at eu.decoma.com (Ferenc.Pintye at eu.decoma.com) Date: Thu, 20 Jul 2006 14:36:12 +0200 Subject: [Numpy-discussion] vector to tensor matrix speed up Message-ID: Hi users, i have some problem in Numpy with indexing speed for array to tensor matrix transport. With 200000 cycles it's 9sec ! (without sort(eigvals() - functions) Many thanks f. # stress values/matrix in array form for 200000 points out = zeros((200000,11),Float32) # #.....out = filling matrix ...etc... # # stress tensor matrix eig = zeros((3,3),Float32) # #output for eigvalues eigwert = array([0,0,0]) # for j in arange(0,200000): eig[0,0] = out[j,1] eig[1,1] = out[j,2] eig[2,2] = out[j,3] # eig[0,1] = out[j,4] eig[0,2] = out[j,6] eig[1,0] = out[j,4] eig[1,2] = out[j,5] eig[2,0] = out[j,6] eig[2,1] = out[j,5] # eigwert = sort(eigvals(eig)) out[j,7] = eigwert[2] out[j,8] = eigwert[1] out[j,9] = eigwert[0] out[j,10] = abs(eigwert[0]-eigwert[2]) # From tom.denniston at alum.dartmouth.org Thu Jul 20 09:46:05 2006 From: tom.denniston at alum.dartmouth.org (Tom Denniston) Date: Thu, 20 Jul 2006 08:46:05 -0500 Subject: [Numpy-discussion] numpy.var and numpy.std Message-ID: Are these functions supposed to compute variance and standard deviation? They don't have docstrings and I can't find anything about them in Travis' manual. From david.huard at gmail.com Thu Jul 20 09:58:12 2006 From: david.huard at gmail.com (David Huard) Date: Thu, 20 Jul 2006 09:58:12 -0400 Subject: [Numpy-discussion] numpy.var and numpy.std In-Reply-To: References: Message-ID: <91cf711d0607200658w6d4f7fd1s51173eeeca63a9b7@mail.gmail.com> Yes, In SVN, std has a docstring and I submitted one for var in ticket #174. David 2006/7/20, Tom Denniston : > > Are these functions supposed to compute variance and standard > deviation? They don't have docstrings and I can't find anything about > them in Travis' manual. > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share > your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tom.denniston at alum.dartmouth.org Thu Jul 20 12:35:29 2006 From: tom.denniston at alum.dartmouth.org (Tom Denniston) Date: Thu, 20 Jul 2006 11:35:29 -0500 Subject: [Numpy-discussion] numpy.var and numpy.std In-Reply-To: <91cf711d0607200658w6d4f7fd1s51173eeeca63a9b7@mail.gmail.com> References: <91cf711d0607200658w6d4f7fd1s51173eeeca63a9b7@mail.gmail.com> Message-ID: Sounds great. Thanks David. --Tom On 7/20/06, David Huard wrote: > Yes, > > In SVN, std has a docstring and I submitted one for var in ticket #174. > > David > > 2006/7/20, Tom Denniston >: > > > Are these functions supposed to compute variance and standard > deviation? They don't have docstrings and I can't find anything about > them in Travis' manual. > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From oliphant.travis at ieee.org Thu Jul 20 14:00:24 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 20 Jul 2006 12:00:24 -0600 Subject: [Numpy-discussion] numpy.var and numpy.std In-Reply-To: References: Message-ID: <44BFC4B8.70306@ieee.org> Tom Denniston wrote: > Are these functions supposed to compute variance and standard > deviation? They don't have docstrings and I can't find anything about > them in Travis' manual. > These are in the manual but only as a general reference to the fact that array methods also have functional counter-parts. In other words, a.meth() --> meth(a) The latter is more general because a does not have to be an array already. -Travis From oliphant.travis at ieee.org Thu Jul 20 14:09:29 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 20 Jul 2006 12:09:29 -0600 Subject: [Numpy-discussion] numpy.var and numpy.std In-Reply-To: References: Message-ID: <44BFC6D9.8060500@ieee.org> Tom Denniston wrote: > Are these functions supposed to compute variance and standard > deviation? They don't have docstrings and I can't find anything about > them in Travis' manual. > > I just wanted to advertise the fact that there are two utility functions in NumPy that can come in handy for people wanting to add docstrings to built-in types. These utility functions by-pass the need to fiddle with doc-strings in C-code (which can be a pain) and re-compile every time a doc-string is changed. These are meant to be run once.... They are: add_docstring(obj, string) Add a string as the docstring of obj. A wide-variety of builtin types are understood. If it doesn't know how to add it, you get a TypeError. If the object already has a doc-string you get a RuntimeError. add_newdoc(place, obj, doc) This is a wrapper around docstring to make it easier to add docs to many attributes of an object which is stored in place. It never fails. place and obj are both strings doc is a string, list, or tuple place is the module name obj is the object in that module If doc is a string add it to place.obj If doc is a tuple it is (attr, doc) ---> the first element is the attribute of obj and the second is the docstring. If doc is a list it is [(attr1, doc1), (attr2, doc2), ...] --> the elements of the list are all tuples to add docstrings to many attributes of an object. numpy/add_newdocs.py is a place where docstrings can be added. -Travis From tim.hochberg at cox.net Thu Jul 20 15:47:59 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 20 Jul 2006 12:47:59 -0700 Subject: [Numpy-discussion] vector to tensor matrix speed up In-Reply-To: References: Message-ID: <44BFDDEF.4060500@cox.net> Ferenc.Pintye at eu.decoma.com wrote: > > > Hi users, > > i have some problem in Numpy with indexing speed for array to tensor > matrix transport. > With 200000 cycles it's 9sec ! (without sort(eigvals() - functions) > > Many thanks > f. > The big problem you have here is that you are operating on your matrices one element at a time. Overhead is going to kill you doing this. The key thing is to operate on your data in chunks -- I'll show one way below that speeds thing up by a factor of 40 or so *neglecting sort(eigvals())*. Unfortunately, sort(eigvals()) is still likely to kill you since there's not really a good way to chunk that up. (I'll have more to say about that in a more general sense in another email). Here's the two versions I compared, perhaps this will help you out: from numpy import * import timeit out = zeros((200000,11),float) def f0(out): for j in arange(0,200000): pass def f1(out): # stress values/matrix in array form for 200000 points # #.....out = filling matrix ...etc... # # stress tensor matrix eig = zeros((3,3),float) # #output for eigvalues eigwert = array([0,0,0]) # for j in arange(0,200000): eig[0,0] = out[j,1] eig[1,1] = out[j,2] eig[2,2] = out[j,3] # eig[0,1] = out[j,4] eig[0,2] = out[j,6] eig[1,0] = out[j,4] eig[1,2] = out[j,5] eig[2,0] = out[j,6] eig[2,1] = out[j,5] # #~ eigwert = sort(eigvals(eig)) out[j,7] = eigwert[2] out[j,8] = eigwert[1] out[j,9] = eigwert[0] out[j,10] = abs(eigwert[0]-eigwert[2]) # def f2(out): # stress values/matrix in array form for 200000 points # #.....out = filling matrix ...etc... # # stress tensor matrix eig = zeros((100,3,3),float) # #output for eigvalues eigwert = zeros([100,3], dtype=float) # local_abs = abs rangen = range(0,200000,100) for j in rangen: eig[:,0,0] = out[j:j+100,1] eig[:,1,1] = out[j:j+100,2] eig[:,2,2] = out[j:j+100,3] # eig[:,1,0] = eig[:,0,1] = out[j:j+100,4] eig[:,2,0] = eig[:,0,2] = out[j:j+100,6] eig[:,2,1] = eig[:,1,2] = out[j:j+100,5] # #~ eigwert = sort(eigvals(eig)) for j in rangen: out[j:j+100,7:10] = eigwert # Changed order of out here out[j:j+100,10] = abs(eigwert[:,0]-eigwert[:,2]) if __name__ == '__main__': print timeit.Timer("f0(out)", "from scratch import f0, out").timeit(1) print timeit.Timer("f1(out)", "from scratch import f1, out").timeit(1) print timeit.Timer("f2(out)", "from scratch import f2, out").timeit(1) > > # stress values/matrix in array form for 200000 points > out = zeros((200000,11),Float32) > # > #.....out = filling matrix ...etc... > # > # stress tensor matrix > eig = zeros((3,3),Float32) > # > #output for eigvalues > eigwert = array([0,0,0]) > # > for j in arange(0,200000): > eig[0,0] = out[j,1] > eig[1,1] = out[j,2] > eig[2,2] = out[j,3] > # > eig[0,1] = out[j,4] > eig[0,2] = out[j,6] > eig[1,0] = out[j,4] > eig[1,2] = out[j,5] > eig[2,0] = out[j,6] > eig[2,1] = out[j,5] > # > eigwert = sort(eigvals(eig)) > out[j,7] = eigwert[2] > out[j,8] = eigwert[1] > out[j,9] = eigwert[0] > out[j,10] = abs(eigwert[0]-eigwert[2]) > # > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From nvf at MIT.EDU Thu Jul 20 17:34:16 2006 From: nvf at MIT.EDU (Nick Fotopoulos) Date: Thu, 20 Jul 2006 17:34:16 -0400 Subject: [Numpy-discussion] Beta release on Thursday Message-ID: <42D94292-EC82-4D86-A276-473B5369BCC4@mit.edu> On July 17, 2006, at 9:01 PM, Travis Oliphant wrote: > I'd like to make release 1.0beta on Thursday. Please submit bug- reports > and fixes before then. > > -Travis Is it possible to incorporate v7 mat-file support before the new- feature freeze? I have the fixes posted to ticket #14. Several people have contacted me about the ticket. Thanks, Nick -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2802 bytes Desc: not available URL: From oliphant.travis at ieee.org Thu Jul 20 17:39:45 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 20 Jul 2006 15:39:45 -0600 Subject: [Numpy-discussion] NumPy 1.0b1 at the door... Message-ID: <44BFF821.5040505@ieee.org> I'm getting ready to tag the trunk as the 1.0b1 release (will be done in 2 hours unless I hear from somebody). I've taken care of most of the relevant open tickets. Some of the build issues are still there but they can be fixed during the beta release period which I expect to be several months. -Travis From robert.kern at gmail.com Thu Jul 20 17:51:31 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 20 Jul 2006 16:51:31 -0500 Subject: [Numpy-discussion] NumPy 1.0b1 at the door... In-Reply-To: <44BFF821.5040505@ieee.org> References: <44BFF821.5040505@ieee.org> Message-ID: Travis Oliphant wrote: > I'm getting ready to tag the trunk as the 1.0b1 release (will be done in > 2 hours unless I hear from somebody). I've taken care of most of the > relevant open tickets. Some of the build issues are still there but > they can be fixed during the beta release period which I expect to be > several months. I'm testing the change from using pkgload in __init__ to explicit imports now. If you can hold off until I finish with that, I'd appreciate it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant.travis at ieee.org Thu Jul 20 18:00:34 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 20 Jul 2006 16:00:34 -0600 Subject: [Numpy-discussion] Beta release on Thursday In-Reply-To: <42D94292-EC82-4D86-A276-473B5369BCC4@mit.edu> References: <42D94292-EC82-4D86-A276-473B5369BCC4@mit.edu> Message-ID: <44BFFD02.3060001@ieee.org> Nick Fotopoulos wrote: > On July 17, 2006, at 9:01 PM, Travis Oliphant wrote: > > I'd like to make release 1.0beta on Thursday. Please submit > bug-reports > > and fixes before then. > > > > -Travis > > Is it possible to incorporate v7 mat-file support before the > new-feature freeze? That is in SciPy. I'm talking about the NumPy 1.0beta release. But, I would like to get the v7 mat-file support into SciPy soon. Perhaps you are arguing about putting in mat-file support into NumPy. While I appreciate the idea, I like it sitting on the SciPy side of the fence > I have the fixes posted to ticket #14. Several people have > contacted me about the ticket. Thanks for the ticket. I will get around to it. I've been swamped with NumPy. I'm really relying on the other contributors for SciPy fixes. -Travis From tim.hochberg at cox.net Thu Jul 20 18:07:24 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 20 Jul 2006 15:07:24 -0700 Subject: [Numpy-discussion] Stacking matrices Message-ID: <44BFFE9C.5090304@cox.net> The recent message by Ferenc.Pintye (how does one pronounce that BTW) reminded me of something I've been meaning to discuss: I think we can do a better job dealing with stacked matrices. By stacked matrices I mean 3 (or more) dimensional arrays where the last two dimensions are considered to be matrices. Let us suppose that one wants to find the eigenvalues of 200,000 3x3 matrices as Ferenc does. At present you have to call linalg.eigvals 200,000 time; the overhead is going to kill you. I've run into this problem before myself in the past (and kludged my way around it), so the problem is not limited to Ferenc. I see no reason that this could not be fixed for the linalg functions. The details would vary from function to function, but for eigvals, for example, when passed an NxMxM array, it would return an NxM array of eigenvalues. One can actually get pretty far just modifying the python functions in linalg.py, or at least one could in numarray and I'm guessing that numpy is similar. That is because there is a fair amount of overhead in setting up the function calls and this can all be done just once. I have code lying around that does this for solve, inverse and determinant that works with numarray that would probably not be hard to adapt to numpy if we were interested in making this change of interface to linalg. Another place that this same effect has bitten me in the past is with dot. Dot does work on higher dimensional arrays, but it attempts to do a tensor product of sorts. While this sounds nifty, in practice I've found it less than useful. However, because dot computes both scalar and matrix products it's not clear that a set of rules that was more or less backwards compatible and that extended to higher dimensions in a more useful way than at present could be crafted. Probably something with different behavior, perhaps that always performed matrix products on the last two axes, could be added with a name other than dot. This avoids backwards compatibility issues as well. None of this stuff is worth holding up the the release 1.0 over, but I though I would mention it while it was on my mind. -tim From nvf at MIT.EDU Thu Jul 20 18:14:11 2006 From: nvf at MIT.EDU (Nick Fotopoulos) Date: Thu, 20 Jul 2006 18:14:11 -0400 Subject: [Numpy-discussion] Beta release on Thursday In-Reply-To: <44BFFD02.3060001@ieee.org> References: <42D94292-EC82-4D86-A276-473B5369BCC4@mit.edu> <44BFFD02.3060001@ieee.org> Message-ID: <0A246626-A462-49F3-9692-674E47795D0E@mit.edu> On Jul 20, 2006, at 6:00 PM, Travis Oliphant wrote: >> Is it possible to incorporate v7 mat-file support before the new- >> feature freeze? > That is in SciPy. I'm talking about the NumPy 1.0beta release. > But, I would like to get the v7 mat-file support into SciPy soon. > Perhaps you are arguing about putting in mat-file support into > NumPy. While I appreciate the idea, I like it sitting on the > SciPy side of the fence > >> I have the fixes posted to ticket #14. Several people have >> contacted me about the ticket. > Thanks for the ticket. I will get around to it. I've been swamped > with NumPy. I'm really relying on the other contributors for SciPy > fixes. My apologies. I often forget that the release cycles are not phase- locked. The same characters play on both stages, so if I'm not paying sufficient attention, I mix things up. I'll go ask for a commit in SciPy land. Take care, Nick -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2802 bytes Desc: not available URL: From oliphant.travis at ieee.org Thu Jul 20 18:31:56 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 20 Jul 2006 16:31:56 -0600 Subject: [Numpy-discussion] NumPy 1.0b1 at the door... In-Reply-To: <20060720221506.GC1046@mentat.za.net> References: <44BFF821.5040505@ieee.org> <20060720221506.GC1046@mentat.za.net> Message-ID: <44C0045C.60800@ieee.org> Stefan van der Walt wrote: > Hi Travis > > Albert and I are busy doing some final beta testing. Quick question: > can Fortran-order arrays be contiguous, ever? > > In [62]: N.empty((3,3),order='F').flags['CONTIGUOUS'] > Out[62]: False > Thank you very much for the testing. You two have been amazingly helpful. No. CONTIGUOUS means "C-CONTIGUOUS" Now, 0-d and 1-d arrays can be both Fortran-contiguous *and* C-contiguous which is why "FNC" exists to test against that case. -Travis From tim.hochberg at cox.net Thu Jul 20 19:35:35 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu, 20 Jul 2006 16:35:35 -0700 Subject: [Numpy-discussion] fast way of doing "cross-multiplications" ? In-Reply-To: References: <44BCEEF9.1030200@obs.univ-lyon1.fr> <502862AF-514C-41CA-804E-420CE76FEE37@stsci.edu> <44BCFFE4.6070502@obs.univ-lyon1.fr> <44BD08A3.1030600@cox.net> Message-ID: <44C01347.8080701@cox.net> Fernando Perez wrote: > On 7/18/06, Tim Hochberg wrote: >> Eric Emsellem wrote: >> > thanks for the tips. (indeed your "add.reduce" is correct: I just >> wrote >> > this down too quickly, in the script I have a "sum" included). >> > >> > And yes you are right for the memory issue, so I may just keep the >> loop >> > in and try to make it work on a fast PC...(or use parallel processes) >> > >> > (is "sum" different than "add.reduce"?) >> > >> > thanks again to both Bill Baxter and Perry Greenfield for their fast >> > (and helpful!) answers. >> > >> I just wanted to add that there are faster, but considerably complicated >> ways to attack this class of problems. The one I've looked at in the >> past was the fast multipole method and I believe there are others. I'm >> not sure whether these can be implemented efficiently in numpy, but you >> may want to take a look into this kind of more sophisticated/complicated >> approach if brute forcing the calculation doesn't work. > > Indeed, FMM is the best known method that can turn this O(n^2) problem > into O(n*log(n)). As Tim indicates, there is no easy way out of this > one. Incidentally, there is a talk about FMM on the scipy'06 > schedule, in case you are going to attend. > > An alternative approach to FMM (conceptually similar in some ways) is > described here: > > http://amath.colorado.edu/faculty/fperez/talks/0604_sanum_mwadap.pdf > > Unfortunately this isn't exactly a half-hour optimization effort, as > the required machinery is pretty heavy duty. And yes, this has all > been done in python and runs on current numpy/scipy (though it has > Fortran, C and C++ sprinkled as needed). I'm interested in hearing in what situations people end up dropping into C/C++ or Fortran. It would be interesting to see if general solutions could be found for making some of them fast enough at the Python level. Here are some situations that I've run into where numpy has trouble computing results efficiently. 1. The value at a given point is the computed one of two (or more) ways depending on some condition. The subcomputations are expensive, so one does not want to compute them both and then chose a result using 'where'. I think I have a decent way to attack this at the Python level using argsort: basically, sort all of the operands, find the boundaries between the regions where different computations are done, compute the expensive subcomputations on the appropriate blocks and then unsort the data. It would be interesting to see if this could be automated in a clean way, perhaps within numexpr. 2. Generalized reduce. We want to reduce an array using an arbitrary function f(x, y) that is not a standard ufunc. In the particular case I've run into, x and y would be vectors or perhaps record arrays, and the functions would couple the various records when producing the output. I don't see how this case can be handled efficiently at the Python level, short of using something like Weave to produce a custom ufunc and calling reduce on it. I don't see how something like numexpr could handle it, unfortunately, since it gains it's speed by operating on blocks of data. I'm sure there are other cases that I've run into, but that's all I'm coming up with right now. -tim From davidgrant at gmail.com Thu Jul 20 20:47:03 2006 From: davidgrant at gmail.com (David Grant) Date: Thu, 20 Jul 2006 17:47:03 -0700 Subject: [Numpy-discussion] [SciPy-user] Would like to simplify my 3 where statements In-Reply-To: <16761e100607201717t1c610949xd24f579ef6f69013@mail.gmail.com> References: <16761e100607201717t1c610949xd24f579ef6f69013@mail.gmail.com> Message-ID: On 7/20/06, Michael Sorich wrote: > Can you give an specific example of how this would work? The codes > really is ugly and it is not clear to me what exactly it does. ok here's a quick example: import numpy n=5 i=3 j=4 A=numpy.random.randint(0,2,(n,n)) #make random graph A=A-diag(diag(A)) #remove diagonal A=triu(A)+transpose(triu(A)) #make symmetric Now say A is the adjacency matrix for a graph and I want to know which nodes are neighbours of A, but I want to exclude node i from consideration. So if A is: array([[0, 1, 0, 1, 0], [1, 0, 1, 0, 1], [1, 0, 0, 0, 0], [0, 0, 1, 0, 1], [1, 1, 0, 1, 0]]) the neighbours are array([1]) for i=3, j=4. One way to do it is to do: ans=where(A[:,j]==1)[0] if A[i,j] == 1: ans -= 1 which is probably faster than my method. I don't really care so much about speed though. I'm just trying to figure out different ways of doing this using numpy. Dave From wbaxter at gmail.com Thu Jul 20 21:51:07 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 21 Jul 2006 10:51:07 +0900 Subject: [Numpy-discussion] r_, c_, hstack, and vstack with 1-d arrays In-Reply-To: References: <44BE2E5A.80404@gmx.net> Message-ID: I looked into the various concatenation methods a bit more to better understand what's going on under the hood. Here's essentially what these different methods do: vstack(tup): concatenate( map(atleast_2d,tup), axis=0 ) hstack(tup): concatenate( map(atleast_1d,tup),axis=1 ) column_stack(tup): arrays = map( transpose,map(atleast_2d,tup) ) concatenate(arrays,1) (note that column_stack transposes *everything* not just 1-d inputs, so it doesn't do quite what I thought it did, i.e. only transposing 1-d inputs) The above 3 are pretty much exactly the code used by numpy. That's all there is to those 3 functions. For r_ and c_ I'm summarizing, but effectively they seem to be doing something like: r_[args]: concatenate( map(atleast_1d,args),axis=0 ) c_[args]: concatenate( map(atleast_1d,args),axis=1 ) c_ behaves almost exactly like hstack -- with the addition of range literals being allowed. r_ is most like vstack, but a little different since it effectively uses atleast_1d, instead of atleast_2d. So you have >>> numpy.vstack((1,2,3,4)) array([[1], [2], [3], [4]]) but >>> numpy.r_[1,2,3,4] array([1, 2, 3, 4]) However for cases like that with just 0-d or 1-d inputs, c_ behaves identically to r_, so if you wanted to get a 1-d output you could have just used c_. So I take back what I said about wishing c_ were like column_stack. Column stack is weird. Instead, I think the right thing to do would be to make r_ behave more like vstack. I think that would make things more consistent, and make for less for the user to remember. After making that change, to make things even more consistent, it might make sense to rename r_ and c_ to v_ and h_ instead. Then it's easy to remember 'v_' is like 'vstack', 'h_' is like hstack. Furthermore, I propose that column_stack should only transpose its 1d inputs. "Stack colums" defnitely doesn't imply to me that something that already has columns will be transposed. Currently it is documented to only work on 1d inputs, so hopefully that's a change that wouldn't affect too many people. The function in numpy/lib/shape_base.py could be replaced with this: def column_stack(tup): def transpose_1d(array): if array.ndim<2: return _nx.transpose(atleast_2d(array)) else: return array arrays = map(transpose_1d,map(atleast_1d,tup)) return _nx.concatenate(arrays,1) If r_, and c_ get renamed to v_, h_, then c_ could be re-introduced with behavior similar to column_stack. Finally, I noticed that the atleast_nd methods return arrays regardless of input type. At a minimum, atleast_1d and atleast_2d on matrices should return matrices. I'm not sure about atleast_3d, since matrices can't be 3d. (But my opinon is that the matrix type should be allowed to be 3d). Anyway, since these methods are used by the *stack methods, those also do not currently preserve the matrix type (in SVN numpy). SUMMARY: * make r_ behave like "vstack plus range literals" * make column_stack only transpose its 1d inputs. * rename r_,c_ to v_,h_ (or something else) to make their connection with vstack and hstack clearer. Maybe vs_ and hs_ would be better? * make a new vertsion of 'c_' that acts like column_stack so that theres a nice parallel v_<=>vstack, h_<=>hstack, c_<=>column_stack * make atleast_*d methods preserve the input type whenever possible Thoughts? --bb From mail at stevesimmons.com Thu Jul 20 22:05:48 2006 From: mail at stevesimmons.com (Stephen Simmons) Date: Fri, 21 Jul 2006 12:05:48 +1000 Subject: [Numpy-discussion] Error when using uints in bincount() In-Reply-To: <44BCCD22.7060607@stevesimmons.com> References: <44BCCD22.7060607@stevesimmons.com> Message-ID: <44C0367C.5070805@stevesimmons.com> Hi, The function bincount() counts the number of each value found in the input array: In [15]: numpy.bincount( array([1,3,3,3,4],dtype=int32) ) Out[15]: array([0, 1, 0, 3, 1]) According to the documentation, the input array must be non-negative integers. However an exception occurs when the input data type are unsigned integers (which is an explicit guarantee of this non-negativity condition): In [157]: numpy.bincount( array([1,3,3,3,4],dtype=uint32) ) TypeError: array cannot be safely cast to required type This seems to be a bug. Cheers Stephen P.S. I'm not familiar enough with the numpy source to track down where this typechecking is done. But I did find a trivial typo in an error msg in function arr_bincount() in numpy/lib/src/_compiled_base.c. The assert message here has lost its initial 'F': Py_Assert(numbers[mni] >= 0, "irst argument of bincount must be non-negative"); From wbaxter at gmail.com Thu Jul 20 23:12:26 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 21 Jul 2006 12:12:26 +0900 Subject: [Numpy-discussion] appending/inserting/deleting axes of arrays Message-ID: Howdy, Is there any nicer syntax for the following operations on arrays? Append a row: a = vstack((a,row)) Append a column: a = hstack((a,col)) Append a row of zeros: a = vstack((a,zeros((1,a.shape[1])))) Append a col of zeros: a = hstack((a,zeros((a.shape[0],1)))) Insert a row before row j a = vstack(( a[:j], row, a[j:] )) Insert a column before col j a = hstack(( a[:j], col, a[j:] )) Insert a row of zeros before row j a = vstack(( a[:j], zeros((1,a.shape[1])), a[j:] )) Insert a column of zeros before col j a = hstack(( a[:j], zeros((a.shape[0],1)), a[j:] )) Delete row j: a = vstack(( a[:j], a[j+1:] )) Delete col j: a = hstack(( a[:j], a[j+1:] )) ...And in more general the same types of operations for N-d arrays. I find myself using python lists of lists a lot just for the easy readability of a.append(row) compared to a = vstack((a,row)). I guess, though, if I'm building an array by appending a row at a time, then maybe it *is* better to use a python list o list for that? Then each 'append' only copies pointers of the existing rows rather than copying the data in each row. Is that correct? Also do python lists over-allocate in order to avoid having to re-allocate and copy every time there's an append()? Numpy arrays don't over-allocate I assume. Thanks, --bb From mail at stevesimmons.com Thu Jul 20 23:44:57 2006 From: mail at stevesimmons.com (Stephen Simmons) Date: Fri, 21 Jul 2006 13:44:57 +1000 Subject: [Numpy-discussion] Possible modification to bincount() In-Reply-To: <44C0367C.5070805@stevesimmons.com> References: <44BCCD22.7060607@stevesimmons.com> <44C0367C.5070805@stevesimmons.com> Message-ID: <44C04DB9.9000909@stevesimmons.com> While playing a little more with bincount(), one modification would be handy: Allow negative integers in the bin list, but skip them when counting bins My specific use case is calculating subtotals on columns of large datasets (1m rows x 30 cols), where some rows need to be excluded. The groupings are expensive to compute, and sometimes will involve ~99% of the rows (eliminate only outliers/errors), and other times only ~5% of the rows (focus in on a subset). I'd like to calculate subtotals like this using bincount(), without having to copy the large datasets just to eliminate the unwanted rows: # Assign each row to a group numbered from 0..G, except for -1 for rows to exclude row_groups = expensive_function(data) # Count number in each group, excluding those with grp==-1 grp_counts = bincount(list=row_groups) # Use bincount() to form subtotals by column, excluding those with grp==-1 subtotals = column_stack([ bincount(list=row_groups, weights=data[:,i]) for i in range(G+1) ]) Is there any appetite to make such a change to bincount()? This would require two simple changes to bincount() in _compiled_base.c and an update to the docstring. Here is the diff file with enough context to show the entire arr_bincount() function: *** orig_compiled_base.c 2006-07-21 13:14:21.250000000 +1000 --- _compiled_base.c 2006-07-21 13:34:41.718750000 +1000 *************** *** 70,143 **** intp j ; for ( j = 1 ; j < len; j ++ ) if ( i [j] < min ) {min = i [j] ; mn = j ;} return mn; } static PyObject * arr_bincount(PyObject *self, PyObject *args, PyObject *kwds) { /* histogram accepts one or two arguments. The first is an array ! * of non-negative integers and the second, if present, is an * array of weights, which must be promotable to double. * Call these arguments list and weight. Both must be one- * dimensional. len (weight) == len(list) * If weight is not present: ! * histogram (list) [i] is the number of occurrences of i in list. * If weight is present: * histogram (list, weight) [i] is the sum of all weight [j] ! * where list [j] == i. */ /* self is not used */ PyArray_Descr *type; PyObject *list = NULL, *weight=Py_None ; PyObject *lst=NULL, *ans=NULL, *wts=NULL; ! intp *numbers, *ians, len , mxi, mni, ans_size; int i; double *weights , *dans; static char *kwlist[] = {"list", "weights", NULL}; Py_Try(PyArg_ParseTupleAndKeywords(args, kwds, "O|O", kwlist, &list, &weight)); Py_Try(lst = PyArray_ContiguousFromAny(list, PyArray_INTP, 1, 1)); len = PyArray_SIZE(lst); numbers = (intp *) PyArray_DATA(lst); mxi = mxx (numbers, len) ; - mni = mnx (numbers, len) ; - Py_Assert(numbers[mni] >= 0, - "irst argument of bincount must be non-negative"); ans_size = numbers [mxi] + 1 ; type = PyArray_DescrFromType(PyArray_INTP); if (weight == Py_None) { Py_Try(ans = PyArray_Zeros(1, &ans_size, type, 0)); ians = (intp *)(PyArray_DATA(ans)); for (i = 0 ; i < len ; i++) ! ians [numbers [i]] += 1 ; Py_DECREF(lst); } else { Py_Try(wts = PyArray_ContiguousFromAny(weight, PyArray_DOUBLE, 1, 1)); weights = (double *)PyArray_DATA (wts); Py_Assert(PyArray_SIZE(wts) == len, "bincount: length of weights " \ "does not match that of list"); type = PyArray_DescrFromType(PyArray_DOUBLE); Py_Try(ans = PyArray_Zeros(1, &ans_size, type, 0)); dans = (double *)PyArray_DATA (ans); for (i = 0 ; i < len ; i++) { ! dans[numbers[i]] += weights[i]; } Py_DECREF(lst); Py_DECREF(wts); } return ans; fail: Py_XDECREF(lst); Py_XDECREF(wts); Py_XDECREF(ans); return NULL; } --- 70,145 ---- intp j ; for ( j = 1 ; j < len; j ++ ) if ( i [j] < min ) {min = i [j] ; mn = j ;} return mn; } static PyObject * arr_bincount(PyObject *self, PyObject *args, PyObject *kwds) { /* histogram accepts one or two arguments. The first is an array ! * of integers and the second, if present, is an * array of weights, which must be promotable to double. * Call these arguments list and weight. Both must be one- * dimensional. len (weight) == len(list) * If weight is not present: ! * histogram (list) [i] is the number of occurrences of i in list ! * for i>=0. Negative i values are ignored. * If weight is present: * histogram (list, weight) [i] is the sum of all weight [j] ! * where list [j] == i and i>=0. */ /* self is not used */ PyArray_Descr *type; PyObject *list = NULL, *weight=Py_None ; PyObject *lst=NULL, *ans=NULL, *wts=NULL; ! intp *numbers, *ians, len , mxi, ans_size; int i; double *weights , *dans; static char *kwlist[] = {"list", "weights", NULL}; Py_Try(PyArg_ParseTupleAndKeywords(args, kwds, "O|O", kwlist, &list, &weight)); Py_Try(lst = PyArray_ContiguousFromAny(list, PyArray_INTP, 1, 1)); len = PyArray_SIZE(lst); numbers = (intp *) PyArray_DATA(lst); mxi = mxx (numbers, len) ; ans_size = numbers [mxi] + 1 ; type = PyArray_DescrFromType(PyArray_INTP); if (weight == Py_None) { Py_Try(ans = PyArray_Zeros(1, &ans_size, type, 0)); ians = (intp *)(PyArray_DATA(ans)); for (i = 0 ; i < len ; i++) ! if (numbers[i]>=0) { ! ians[numbers [i]] += 1 ; ! } Py_DECREF(lst); } else { Py_Try(wts = PyArray_ContiguousFromAny(weight, PyArray_DOUBLE, 1, 1)); weights = (double *)PyArray_DATA (wts); Py_Assert(PyArray_SIZE(wts) == len, "bincount: length of weights " \ "does not match that of list"); type = PyArray_DescrFromType(PyArray_DOUBLE); Py_Try(ans = PyArray_Zeros(1, &ans_size, type, 0)); dans = (double *)PyArray_DATA (ans); for (i = 0 ; i < len ; i++) { ! if (numbers[i]>=0) { ! dans[numbers[i]] += weights[i]; ! } } Py_DECREF(lst); Py_DECREF(wts); } return ans; fail: Py_XDECREF(lst); Py_XDECREF(wts); Py_XDECREF(ans); return NULL; } Cheers Stephen From pebarrett at gmail.com Fri Jul 21 00:01:46 2006 From: pebarrett at gmail.com (Paul Barrett) Date: Fri, 21 Jul 2006 00:01:46 -0400 Subject: [Numpy-discussion] Problem converting a numarray extension to numpy Message-ID: <40e64fa20607202101o75ca115ate4ed577fd6d98ac3@mail.gmail.com> I'm having a problem converting a C extension module that was originally written for numarray to use numpy. I using swig to create a wrapper flle for the C code. I have added the numpy.get_numarray_include() method to my setup.py file and have changed the numarray/libnumarray.h to use numpy/libnumarray.h. The extension appears to compile fine (with the exception of some warning messages). However, when I import the module, I get a segfault. Do I need to add anything else to the share library's initialization step other than import_libnumarray()? This is using Python 2.4.3 and gcc 4.1.1 on FC5. -- Paul From haase at msg.ucsf.edu Fri Jul 21 00:25:43 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu, 20 Jul 2006 21:25:43 -0700 Subject: [Numpy-discussion] [NumPy] #188: dtype should have "nice looking" str representation In-Reply-To: <076.85e10186a436d81ebef7e51d7e625bdb@scipy.net> References: <067.565cc454ae510236945afe3a0479b238@scipy.net> <076.85e10186a436d81ebef7e51d7e625bdb@scipy.net> Message-ID: <44C05747.2090709@msg.ucsf.edu> NumPy wrote: > #188: dtype should have "nice looking" str representation > -------------------------+-------------------------------------------------- > Reporter: sebhaase | Owner: oliphant > Type: enhancement | Status: closed > Priority: normal | Milestone: 1.0 Release > Component: numpy.core | Version: > Severity: normal | Resolution: wontfix > Keywords: | > -------------------------+-------------------------------------------------- > Changes (by oliphant): > > * status: new => closed > * resolution: => wontfix > > Comment: > > I'm not sure what the best thing to display here actually is. The current > string is very informative. Dropping the byte-ordering character is a > bad-idea. > Just yesterday I showed the new numpy a colleague of mine and he indeed read " I've created the 1.0b1 release tag in SVN and will be uploading files shortly to Sourceforge. I've also created a 1.0 release branch called ver1.0 The trunk is now version 1.1 of NumPy and should be used for new-development only. I don't expect 1.1 to come out for at least a year. Bug-fixes and small changes can happen on the 1.0 branch. These will be merged periodically to 1.1 or vice-versa. But, the 1.0 branch will be used for releases for the next year. AFAIK, this is similar to Python's release plan. I'm also going to be out of town for a few days and may not be able to check my email so you can ask questions, but I may not answer them for several days :-) Thanks to all you who helped with this release with bug reports and patches: Robert Kern David Cooke Pearu Peterson Alexander Belopolsky (Sasha) Albert Strasheim Stefan van der Walt Tim Hochberg Christopher Hanley Perry Greenfield Todd Miller David Huard Nils Wagner Thank you... I hope you all have a great weekend :-) Let's continue to make the beta release period productive by improving documentation and getting code-coverage tests and tracking down any bugs. Best regards, -Travis From svetosch at gmx.net Fri Jul 21 05:00:18 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Fri, 21 Jul 2006 11:00:18 +0200 Subject: [Numpy-discussion] r_, c_, hstack, and vstack with 1-d arrays In-Reply-To: References: <44BE2E5A.80404@gmx.net> Message-ID: <44C097A2.5020900@gmx.net> Bill Baxter schrieb: > Finally, I noticed that the atleast_nd methods return arrays > regardless of input type. At a minimum, atleast_1d and atleast_2d on > matrices should return matrices. I'm not sure about atleast_3d, since > matrices can't be 3d. (But my opinon is that the matrix type should > be allowed to be 3d). Anyway, since these methods are used by the > *stack methods, those also do not currently preserve the matrix type > (in SVN numpy). > Are you sure? I reported that issue with *stack and I remember it was fixed. > SUMMARY: > * make r_ behave like "vstack plus range literals" > * make column_stack only transpose its 1d inputs. > * rename r_,c_ to v_,h_ (or something else) to make their connection > with vstack and hstack clearer. Maybe vs_ and hs_ would be better? > * make a new vertsion of 'c_' that acts like column_stack so that > theres a nice parallel v_<=>vstack, h_<=>hstack, c_<=>column_stack > * make atleast_*d methods preserve the input type whenever possible > One problem with all that renaming is the other (maybe more important) function of r_: build (array or) matrix ranges quickly. So I'm a bit against the renaming I guess. Cleaning up the irritations with the *stacks seems useful though. (Although I have to confess I haven't read your last mail very thoroughly.) -Sven From Ferenc.Pintye at eu.decoma.com Fri Jul 21 07:35:52 2006 From: Ferenc.Pintye at eu.decoma.com (Ferenc.Pintye at eu.decoma.com) Date: Fri, 21 Jul 2006 13:35:52 +0200 Subject: [Numpy-discussion] Antwort: Re: vector to tensor matrix speed up In-Reply-To: <44BFDDEF.4060500@cox.net> Message-ID: Hi Tim, many thanks for the tipps, i used the same way with vectorized (chunk) method on the indexing operation. .. .... ...... ............ # out = zeros((size_mcf[0],sizes_smatrix[2]+5),Float32) # size_mcf[0] ~ 240000 eig = zeros((size_mcf[0],3,3),dtype=Float32) eigwert = zeros((size_mcf[0],3),dtype=Float64) # # here is speed up ~30 #for j in arange(0,size_mcf[0]): #eig[0,0] = out[j,1] #eig[1,1] = out[j,2] #eig[2,2] = out[j,3] # #eig[0,1] = out[j,4] #eig[0,2] = out[j,6] #eig[1,0] = out[j,4] #eig[1,2] = out[j,5] #eig[2,0] = out[j,6] #eig[2,1] = out[j,5] # eig[:,0,0] = out[:,1] eig[:,1,1] = out[:,2] eig[:,2,2] = out[:,3] eig[:,1,0] = eig[:,0,1] = out[:,4] eig[:,2,0] = eig[:,0,2] = out[:,6] eig[:,2,1] = eig[:,1,2] = out[:,5] # for i in arange(size_mcf[0]): eigwert[i] = eigvals(eig[i,:,:]) # out[:,7:10] = sort(eigwert[:,:].astype(float32)) out[:,10] = abs(out[:,7]-out[:,9]) speedup factor ~30 ! f. From wbaxter at gmail.com Fri Jul 21 08:22:18 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 21 Jul 2006 21:22:18 +0900 Subject: [Numpy-discussion] r_, c_, hstack, and vstack with 1-d arrays In-Reply-To: <44C097A2.5020900@gmx.net> References: <44BE2E5A.80404@gmx.net> <44C097A2.5020900@gmx.net> Message-ID: On 7/21/06, Sven Schreiber wrote: > Bill Baxter schrieb: > > > Finally, I noticed that the atleast_nd methods return arrays > > regardless of input type. > > Are you sure? I reported that issue with *stack and I remember it was fixed. Doh! My bad. You're right. I was looking at the code in SVN for the atleast_* methods and didn't realize that the array constructor (or any constructor) could actually return something besides an object of that class. But that's exactly what array(subok=True) allows. > > SUMMARY: > > * make r_ behave like "vstack plus range literals" > > * make column_stack only transpose its 1d inputs. > > * rename r_,c_ to v_,h_ (or something else) to make their connection > > with vstack and hstack clearer. Maybe vs_ and hs_ would be better? > > * make a new vertsion of 'c_' that acts like column_stack so that > > theres a nice parallel v_<=>vstack, h_<=>hstack, c_<=>column_stack > > * make atleast_*d methods preserve the input type whenever possible > > > > One problem with all that renaming is the other (maybe more important) > function of r_: build (array or) matrix ranges quickly. How would you duplicate this kind of behavior, you mean? >>> r_[1:4,0,4] array([1, 2, 3, 0, 4]) That's what h_ would do. Just as if you had done hstack( (range(1,4),0,4) ). That's actually a good reason for the renaming/retooling. r_ is kind of schitzophrenic now in that it acts *either* as "concatenate rows (vstack-like, for >=2-d)" or "build me a row (hstack-like, for <2d)". So it's hard to remember what the 'r' in r_ stands for. On the other hand, c_ is always hstack-like. By the way, I noticed that 'row_stack' has appeared as a synonym for 'vstack' in SVN. Thanks to whomever added it! --bb From svetosch at gmx.net Fri Jul 21 11:08:08 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Fri, 21 Jul 2006 17:08:08 +0200 Subject: [Numpy-discussion] matrix-related bug in 1.0b1 Message-ID: <44C0EDD8.3000701@gmx.net> Hi, Summary: Slicing seems to be broken with matrices now. I eagerly installed the new beta, but soon stumbled over this bug. I hope I'm missing something, but afaics that behavior used to be different (and correct) before in 0.9.8. Don't know exactly when this changed, though. I did a fresh install (uninstalled old numpy and also matplotlib first) of the official binary for windows/python 2.4. Example: >>> import numpy as n >>> n.__version__ '1.0b1' >>> import numpy.matlib as m >>> a = n.zeros((2,3)) >>> b = m.zeros((2,3)) >>> a[:1,:].shape (1, 3) >>> b[:1,:].shape (3, 1) >>> Note the array slicing works correct, but the equivalent thing with the matrix does not. I also noticed the following (in a new python session) import strangeness: >>> import numpy >>> numpy.matlib.zeros((2,3)) Traceback (most recent call last): File "", line 1, in ? AttributeError: 'module' object has no attribute 'matlib' >>> Why is the direct access to matlib impossible? Either I'm missing something (I well may be, because I'm melting away at 36 centigrades or so...), or imho a new beta should be put out quickly to enable further testing (and use). Thanks, Sven From lists.steve at arachnedesign.net Sat Jul 22 17:50:13 2006 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Sat, 22 Jul 2006 17:50:13 -0400 Subject: [Numpy-discussion] Fresh svn numpy install fails 2 tests (OS X) Message-ID: <358EE207-3D29-40BE-87EA-912639CB93FD@arachnedesign.net> Hi folks, Since 1.0 release is eminent, I just wanted to draw the attention to two failures I get when I run numpy.test(1). I've never been able to get numpy to pass all test cases, but now it fails a second one, so .. I'm pasting it below. Please let me know if these are non-consequential. System info: + Intel Mac (MacBook Pro) + OS X.4.7 + numpy version: 1.0.2881 test failures: FAIL: check_large_types (numpy.core.tests.test_scalarmath.test_power) ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/local/lib/python2.4/site-packages/numpy/core/tests/ test_scalarmath.py", line 47, in check_large_types assert b == 6765201, "error with %r: got %r" % (t,b) AssertionError: error with : got 0.0 ====================================================================== FAIL: check_types (numpy.core.tests.test_scalarmath.test_types) ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/local/lib/python2.4/site-packages/numpy/core/tests/ test_scalarmath.py", line 20, in check_types assert a == 1, "error with %r: got %r" % (atype,a) AssertionError: error with : got 1.02604810941480982577e-4936 ---------------------------------------------------------------------- Ran 468 tests in 1.157s FAILED (failures=2) Out[2]: ---------------------------------------- Thanks, -steve From haase at msg.ucsf.edu Sat Jul 22 23:39:19 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sat, 22 Jul 2006 20:39:19 -0700 Subject: [Numpy-discussion] how add new attribute to a numpy array object ? Message-ID: <44C2EF67.40505@msg.ucsf.edu> Hi, I have a (medical) image file. I wrote a nice interface based on memmap using numarray. The class design I used was essentially to return a numarray array object with a new "custom" attribute giving access to special information about the base file. Now with numpy I noticed that a numpy object does not allow adding new attributes !! (How is this ? Why ?) Travis already suggested (replying to one of my last postings) to create a new sub class of numpy.ndarray. But how do I initialize an object of my new class to be "basically identically to" an existing ndarray object ? Normally I could do class B(N.ndarray): pass a=N.arange(10) a.__class__ = B BUT I get this error: #>>> a.__class__ = B Traceback (most recent call last): File "", line 1, in ? TypeError: __class__ assignment: only for heap types What is a "heap type" ? Why ? How can I do what I want ? Thanks, Sebastian Haase From haase at msg.ucsf.edu Sun Jul 23 01:32:33 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sat, 22 Jul 2006 22:32:33 -0700 Subject: [Numpy-discussion] bug in numpy recarray ? Message-ID: <44C309F1.5090607@msg.ucsf.edu> Hi! I'm trying to convert my numarray records code to numpy. >>> type(m.hdrArray) >>> m.hdrArray.d [(array([ 1., 1., 1.], dtype=float32),)] but I get: >>> m.hdrArray[0].getfield('d') 5.43230922614e-312 Am I missing something or is this a bug ? Further details: >>> m.hdrArray.dtype.descr [('Num', [('f1', '>> >>> >>> m.hdrArray[0].dtype.descr [('Num', [('f1', ' References: <44C2EF67.40505@msg.ucsf.edu> Message-ID: <44C3213F.2020003@hawaii.edu> Sebastian Haase wrote: > Hi, > I have a (medical) image file. > I wrote a nice interface based on memmap using numarray. > The class design I used was essentially to return a numarray array > object with a new "custom" attribute giving access to special > information about the base file. > > Now with numpy I noticed that a numpy object does not allow adding new > attributes !! (How is this ? Why ?) > > Travis already suggested (replying to one of my last postings) to create > a new sub class of numpy.ndarray. > > But how do I initialize an object of my new class to be "basically > identically to" an existing ndarray object ? > Normally I could do > class B(N.ndarray): > pass > a=N.arange(10) > a.__class__ = B Isn't this what you need to do instead? In [1]:import numpy as N In [2]:class B(N.ndarray): ...: pass ...: In [3]:a = B(N.arange(10)) In [4]:a.__class__ Out[4]: In [5]:a.stuff = 'stuff' I don't think it makes sense to try to change the __class__ attribute by assignment. Eric From bioinformed at gmail.com Sun Jul 23 13:02:09 2006 From: bioinformed at gmail.com (Kevin Jacobs ) Date: Sun, 23 Jul 2006 13:02:09 -0400 Subject: [Numpy-discussion] how add new attribute to a numpy array object ? In-Reply-To: <44C2EF67.40505@msg.ucsf.edu> References: <44C2EF67.40505@msg.ucsf.edu> Message-ID: <2e1434c10607231002w7b8b3e4bjf95c7979e1d0765b@mail.gmail.com> On 7/22/06, Sebastian Haase wrote: > > Normally I could do > class B(N.ndarray): > pass > a=N.arange(10) > a.__class__ = B > > BUT I get this error: > #>>> a.__class__ = B > Traceback (most recent call last): > File "", line 1, in ? > TypeError: __class__ assignment: only for heap types > > What is a "heap type" ? Why ? How can I do what I want ? Assigning to __class__ makes sense for objects that allocate a dictionary for storage of attributes or have slots allocated to hold the values. The heap type error is due to a missing flag in the class definition and could be corrected. However, it may not be the best thing to do. Calling B(array) is certainly safer, although a bit more expensive. -Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: From haase at msg.ucsf.edu Sun Jul 23 15:19:04 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sun, 23 Jul 2006 12:19:04 -0700 Subject: [Numpy-discussion] how add new attribute to a numpy array object ? In-Reply-To: <2e1434c10607231002w7b8b3e4bjf95c7979e1d0765b@mail.gmail.com> References: <44C2EF67.40505@msg.ucsf.edu> <2e1434c10607231002w7b8b3e4bjf95c7979e1d0765b@mail.gmail.com> Message-ID: <44C3CBA8.6010809@msg.ucsf.edu> Kevin Jacobs wrote: > On 7/22/06, *Sebastian Haase* > wrote: > > Normally I could do > class B(N.ndarray): > pass > a=N.arange(10) > a.__class__ = B > > BUT I get this error: > #>>> a.__class__ = B > Traceback (most recent call last): > File "", line 1, in ? > TypeError: __class__ assignment: only for heap types > > What is a "heap type" ? Why ? How can I do what I want ? > > > > Assigning to __class__ makes sense for objects that allocate a > dictionary for storage of attributes or have slots allocated to hold the > values. The heap type error is due to a missing flag in the class > definition and could be corrected. However, it may not be the best > thing to do. Calling B(array) is certainly safer, although a bit more > expensive. > > -Kevin > Thanks for the replies. Googling for this I was surprised myself that it IS legal (and done) to assign to obj.__class__. Kevin, I tried what you suggested first -- I think in C++ it would be called "using the copy-constructor". But I get an error - something like: "__new__() needs at least 3 arguments" In other words: (maybe?) in Python there is not always a "copy-constructor" (in fact there is no constructor overloading at all ...) So if there is "just a missing flag" - it would be great if this could be put in. It turns out that the "assigning to __class__"-scheme worked for the ndarray subclass "memmap" (i.e. I was sub classing from memmap and then I could assing origMemmapObj.__class__ = myClass) Thanks, Sebastian Haase From filipwasilewski at gmail.com Sun Jul 23 16:02:50 2006 From: filipwasilewski at gmail.com (Filip Wasilewski) Date: Sun, 23 Jul 2006 22:02:50 +0200 Subject: [Numpy-discussion] how add new attribute to a numpy array object ? In-Reply-To: <44C3213F.2020003@hawaii.edu> References: <44C2EF67.40505@msg.ucsf.edu> <44C3213F.2020003@hawaii.edu> Message-ID: On 7/23/06, Eric Firing wrote: > Sebastian Haase wrote: > > Hi, > > I have a (medical) image file. > > I wrote a nice interface based on memmap using numarray. > > The class design I used was essentially to return a numarray array > > object with a new "custom" attribute giving access to special > > information about the base file. > > > > Now with numpy I noticed that a numpy object does not allow adding new > > attributes !! (How is this ? Why ?) > > > > Travis already suggested (replying to one of my last postings) to create > > a new sub class of numpy.ndarray. > > > > But how do I initialize an object of my new class to be "basically > > identically to" an existing ndarray object ? > > Normally I could do > > class B(N.ndarray): > > pass > > a=N.arange(10) > > a.__class__ = B > > Isn't this what you need to do instead? > > In [1]:import numpy as N > > In [2]:class B(N.ndarray): > ...: pass > ...: > > In [3]:a = B(N.arange(10)) It won't work like that. The constructor for the ndarray is: | ndarray.__new__(subtype, shape=, dtype=int_, buffer=None, | offset=0, strides=None, fortran=False) so you will get either an exception caused by inappropriate shape value or completely wrong result. >>> numpy.ndarray([1,2]) array([[10966528, 18946344]]) >>> numpy.ndarray([1,2]).shape (1, 2) >>> numpy.ndarray(numpy.arange(5)) array([], shape=(0, 1, 2, 3, 4), dtype=int32) And this is a thing you souldn't do rather than a bug. To create an instance of ndarray's subclass B from ndarray object, one need to call the ndarray.view method or the ndarray.__new__ constructor explicitly: class B(numpy.ndarray): def __new__(subtype, data): if isinstance(data, B): return data if isinstance(data, numpy.ndarray): return data.view(subtype) arr = numpy.array(data) return numpy.ndarray.__new__(B, shape=arr.shape, dtype=arr.dtype, buffer=arr) A good example of subclasing ndarray is the matrix class in core/defmatrix.py (SVN version). cheers, fw From haase at msg.ucsf.edu Sun Jul 23 19:23:45 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sun, 23 Jul 2006 16:23:45 -0700 Subject: [Numpy-discussion] C extensions: is Py_XDECREF() needed when using PyArray_FromAny() Message-ID: <44C40501.3050705@msg.ucsf.edu> Hi, I'm converting SWIG typemap'ed C extensions from numarray to numpy. I studied (and use parts of) numpy.i from the doc directory. I noticed that there is no decref for the TYPEMAP_INPLACE2 typemap. This uses a function obj_to_array_no_conversion() which in turn just returns the original PyObject* ( casted to a PyArrayObject* after some sanity checks) It looks to me that in this case there should be an explicit Py_INCREF() - in case the function is threaded (releases the Python GIL) since it holds a pointer to that object's data . (Alternatively) Travis suggested (at the http://www.scipy.org/Converting_from_numarray wiki page) using PyArray_FromAny - is this incrementing the ref.count (implicitely) ? The numarray equivalent (NA_InputArray) IS incrementing the ref.count (as far as I know...). Furthermore on that same wiki page the PyArray_FromAny() is called together with PyArray_DescrFromType(). After searching through the numpy source I found that in blasdot/_dotblas.c (in dotblas_matrixproduct() )there is an explicit Py_INCREF even on the dtype returned from PyArray_DescrFromType. I would argue that ref.counting is always very tricky territory ;-) Hopefully someone can enlighten me . Thanks, Sebastian Haase From wbaxter at gmail.com Sun Jul 23 21:58:43 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 24 Jul 2006 10:58:43 +0900 Subject: [Numpy-discussion] matrix-related bug in 1.0b1 In-Reply-To: <44C0EDD8.3000701@gmx.net> References: <44C0EDD8.3000701@gmx.net> Message-ID: Howdy, On 7/22/06, Sven Schreiber wrote: > Hi, > > Summary: Slicing seems to be broken with matrices now. > > ... > Example: > > >>> import numpy as n > >>> n.__version__ > '1.0b1' > >>> import numpy.matlib as m > >>> a = n.zeros((2,3)) > >>> b = m.zeros((2,3)) > >>> a[:1,:].shape > (1, 3) > >>> b[:1,:].shape > (3, 1) > >>> > > Note the array slicing works correct, but the equivalent thing with the > matrix does not. Looks like it happened in rev 2698 of defmatrix.py, matrix.__getitem__ method: if isscalar(index[1]): if isscalar(index[0]): retscal = True elif out.shape[0] == 1: sh = out.shape out.shape = (sh[1], sh[0]) ==> elif isinstance(index[1], (slice, types.EllipsisType)): ==> if out.shape[0] == 1 and not isscalar(index[0]): It behaves like array if you remove the 'not' in the last line. But maybe that breaks some other cases? Maybe you can try making that change in your numpy/core/defmatrix.py (around line 140) and see if anything else breaks for you. > I also noticed the following (in a new python session) import strangeness: > > >>> import numpy > >>> numpy.matlib.zeros((2,3)) > Traceback (most recent call last): > File "", line 1, in ? > AttributeError: 'module' object has no attribute 'matlib' > >>> > > Why is the direct access to matlib impossible? Maybe the thinking is that since it's a compatibily module, if you want it you should explicity import it. Like you have to do with oldnumeric. --bb From haase at msg.ucsf.edu Mon Jul 24 00:47:06 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sun, 23 Jul 2006 21:47:06 -0700 Subject: [Numpy-discussion] trivial question: how to compare dtype - but ignoring byteorder ? Message-ID: <44C450CA.3010609@msg.ucsf.edu> Hi, if I have a numpy array 'a' and say: a.dtype == numpy.float32 Is the result independent of a's byteorder ? (That's what I would expect ! Just checking !) Thanks, Sebastian Haase From pgmdevlist at gmail.com Mon Jul 24 06:02:24 2006 From: pgmdevlist at gmail.com (PGM) Date: Mon, 24 Jul 2006 06:02:24 -0400 Subject: [Numpy-discussion] float64scalar and divmod (and numpy 0.9.8) Message-ID: <200607240602.24257.pgmdevlist@gmail.com> Folks , I'm still using numpy 0.9.8 and just ran into this problem on my machine (AMD64): >>> import numpy as N >>> x = N.array([1.23456]) >>> print divmod(x,1) (array([ 1.]), array([ 0.23456])) >>> print divmod(x[0],1) () >>> print divmod(x.tolist()[0],1) (1.0, 0.2345600000000001) divmod doesn't seem to like ' References: <44C450CA.3010609@msg.ucsf.edu> Message-ID: <200607241218.55965.karol.langner@kn.pl> On Monday 24 July 2006 06:47, Sebastian Haase wrote: > Hi, > if I have a numpy array 'a' > and say: > a.dtype == numpy.float32 > > Is the result independent of a's byteorder ? > (That's what I would expect ! Just checking !) > > Thanks, > Sebastian Haase The condition will always be False, because you're comparing wrong things here, numpy.float32 is a scalar type, not a dtype. >>> numpy.float32 >>> type(numpy.dtype('>f4')) And I think byteorder matters when comparing dtypes: >>> numpy.dtype('>f4') == numpy.dtype(' References: <44C0EDD8.3000701@gmx.net> Message-ID: <44C4B3E7.6060104@gmx.net> Thanks for helping out on matrix stuff, Bill! Bill Baxter schrieb: > On 7/22/06, Sven Schreiber wrote: >> >> Note the array slicing works correct, but the equivalent thing with the >> matrix does not. > > Looks like it happened in rev 2698 of defmatrix.py, matrix.__getitem__ > method: > > if isscalar(index[1]): > if isscalar(index[0]): > retscal = True > elif out.shape[0] == 1: > sh = out.shape > out.shape = (sh[1], sh[0]) > ==> elif isinstance(index[1], (slice, types.EllipsisType)): > ==> if out.shape[0] == 1 and not isscalar(index[0]): > > It behaves like array if you remove the 'not' in the last line. > But maybe that breaks some other cases? > Maybe you can try making that change in your numpy/core/defmatrix.py > (around line 140) and see if anything else breaks for you. > Hm, I don't know -- if you don't mind I'd like to get a second opinion before I mess around there. It's funny though that the changeset has the title "fixing up matrix slicing" or something like that... >> >> Why is the direct access to matlib impossible? > > Maybe the thinking is that since it's a compatibily module, if you > want it you should explicity import it. Like you have to do with > oldnumeric. > If that is the reason I can't really follow the logic, but I don't really mind the status quo, either. -sven From david.huard at gmail.com Mon Jul 24 07:58:50 2006 From: david.huard at gmail.com (David Huard) Date: Mon, 24 Jul 2006 07:58:50 -0400 Subject: [Numpy-discussion] float64scalar and divmod (and numpy 0.9.8) In-Reply-To: <200607240602.24257.pgmdevlist@gmail.com> References: <200607240602.24257.pgmdevlist@gmail.com> Message-ID: <91cf711d0607240458q1b06f877ic033aa64e0b72143@mail.gmail.com> 2006/7/24, PGM : > > Folks , > I'm still using numpy 0.9.8 and just ran into this problem on my machine > (AMD64): > > >>> import numpy as N > >>> x = N.array([1.23456]) > >>> print divmod(x,1) > (array([ 1.]), array([ 0.23456])) > >>> print divmod(x[0],1) > () > >>> print divmod(x.tolist()[0],1) > (1.0, 0.2345600000000001) > > divmod doesn't seem to like ' trick... > Did anybody run into this problem already ? Is it really numpy related ? > If > this is a bug on 0.9.8, has it been corrected already ? It works in svn. David -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivilata at carabos.com Mon Jul 24 08:00:41 2006 From: ivilata at carabos.com (Ivan Vilata i Balaguer) Date: Mon, 24 Jul 2006 14:00:41 +0200 Subject: [Numpy-discussion] "boolean" type Message-ID: <44C4B669.7000001@carabos.com> Hi all, Since there is a "string" type which is the same as "str_", how come there is no "boolean" type which is the same as "bool_"? Did I miss some design decision about naming? You know, just for completeness, not that it is some kind of problem at all! ;) Cheers, :: Ivan Vilata i Balaguer >qo< http://www.carabos.com/ C?rabos Coop. V. V V Enjoy Data "" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 307 bytes Desc: OpenPGP digital signature URL: From a.h.jaffe at bakerjaffe.plus.com Mon Jul 24 10:04:05 2006 From: a.h.jaffe at bakerjaffe.plus.com (Andrew Jaffe) Date: Mon, 24 Jul 2006 15:04:05 +0100 Subject: [Numpy-discussion] mgrid, ogrid and back again Message-ID: <44C4D355.6020607@bakerjaffe.plus.com> Hi All, I'm finding myself dealing with n-dimensional grids quite a lot, and trying to do some 'tricky' index manipulation. The main problem is manipulating arrays when I don't know a priori the number of dimensions; in essence I need to be able to iterate across dimensions. first, I've got three arrays of lower-bounds (param_min), upper-bounds (param_max) and numbers of steps (nstep). I'm using the following incantations to produce the association mgrid an ogrid arrays: args = tuple(slice(p1, p2, n*1j) for p1,p2,n in zip(param_min, param_max, nstep)) param_grid = N.mgrid.__getitem__(args) Is this the easiest way to do this? Second, from the mgrid object, param_grid, I want to recover the step sizes (assume I've thrown away the args so I can't make an ogrid object). This seems to work: deltas = numpy.empty(npar) for i in xrange(npar): idxtup = (i,)+(0,)*i + (1,) + (0,)*(npar-1-i) deltas[i] = param_grid[idxtup] - param_grid[((i,)+(0,)*npar )] (Or I could compress this into a single somewhat complicated list comprehension). Again, this seems a bit overly complicated. Any ideas for simplifying it? But at least I can work out how to do these things. Finally, however, I need to reconstruct the individual param_min:param_max:(nstep*1j) 1-d arrays (i.e., the flattened versions of the ogrid output). These are effectively param_grid[i,0,,,,:,,,,] where ':' is in slot i. But I don't know how to emulate ':' in either a slice object or tuple-indexing notation. Obviously I could just do a more complicated version of the deltas[] calculation, or direct manipulation on param_grid.flat, but both seem like too much work on my part... Thanks in advance! Andrew p.s. thanks to travis for all his hard work, especially in the run-up to 1.0b (although test() crashes on my PPC Mac... more on that later when I've had time to play). From Pierre.Barbier_de_Reuille at inria.fr Mon Jul 24 11:16:40 2006 From: Pierre.Barbier_de_Reuille at inria.fr (Pierre Barbier de Reuille) Date: Mon, 24 Jul 2006 16:16:40 +0100 Subject: [Numpy-discussion] "boolean" type In-Reply-To: <44C4B669.7000001@carabos.com> References: <44C4B669.7000001@carabos.com> Message-ID: <44C4E458.9080109@inria.fr> Ivan Vilata i Balaguer wrote: > Hi all, > > Since there is a "string" type which is the same as "str_", how come > there is no "boolean" type which is the same as "bool_"? Did I miss > some design decision about naming? You know, just for completeness, not > that it is some kind of problem at all! ;) > Well ... >>> import numpy >>> numpy.__version__ '0.9.9.2852' >>> numpy.bool_ Pierre > Cheers, > > :: > > Ivan Vilata i Balaguer >qo< http://www.carabos.com/ > C?rabos Coop. V. V V Enjoy Data > "" > > > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From ivilata at carabos.com Mon Jul 24 12:29:11 2006 From: ivilata at carabos.com (Ivan Vilata i Balaguer) Date: Mon, 24 Jul 2006 18:29:11 +0200 Subject: [Numpy-discussion] "boolean" type In-Reply-To: <44C4E458.9080109@inria.fr> References: <44C4B669.7000001@carabos.com> <44C4E458.9080109@inria.fr> Message-ID: <44C4F557.1040300@carabos.com> En/na Pierre Barbier de Reuille ha escrit:: >>>>import numpy >>>>numpy.__version__ >'0.9.9.2852' >>>>numpy.bool_ > Sorry if I didn't make my question clear. What I find lacking is a ``numpy.boolean`` type which is to ``numpy.bool_`` the same as ``numpy.string`` is now to ``numpy.str_`` (i.e. a pure reference with a prettier name). Otherwise I'm not getting what you're meaning! C:) :: Ivan Vilata i Balaguer >qo< http://www.carabos.com/ C?rabos Coop. V. V V Enjoy Data "" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 307 bytes Desc: OpenPGP digital signature URL: From Pierre.Barbier_de_Reuille at inria.fr Mon Jul 24 12:39:57 2006 From: Pierre.Barbier_de_Reuille at inria.fr (Pierre Barbier de Reuille) Date: Mon, 24 Jul 2006 17:39:57 +0100 Subject: [Numpy-discussion] "boolean" type In-Reply-To: <44C4F557.1040300@carabos.com> References: <44C4B669.7000001@carabos.com> <44C4E458.9080109@inria.fr> <44C4F557.1040300@carabos.com> Message-ID: <44C4F7DD.50008@inria.fr> Ivan Vilata i Balaguer wrote: > En/na Pierre Barbier de Reuille ha escrit:: > > >>>>> import numpy >>>>> numpy.__version__ >>>>> >> '0.9.9.2852' >> >>>>> numpy.bool_ >>>>> >> >> > > Sorry if I didn't make my question clear. What I find lacking is a > ``numpy.boolean`` type which is to ``numpy.bool_`` the same as > ``numpy.string`` is now to ``numpy.str_`` (i.e. a pure reference with a > prettier name). Otherwise I'm not getting what you're meaning! C:) > > :: > > Ivan Vilata i Balaguer >qo< http://www.carabos.com/ > C?rabos Coop. V. V V Enjoy Data > "" > > Ok, so maybe it is because it is called "bool8" ? So that it is clear that it takes 8 bits ? Pierre From jordan at cutlerlawfirm.com Mon Jul 24 12:43:38 2006 From: jordan at cutlerlawfirm.com (Jordana Hollingshead) Date: Mon, 24 Jul 2006 09:43:38 -0700 Subject: [Numpy-discussion] to yarec Message-ID: <000001c6af40$5068c8d0$5ef3a8c0@tus97> Bes k t S v el s li y ng W v atc u he h s R y OL j EX C y ART f IER BR e EI y TLIN k G BV p LGA q RI OM g EG z A P v ATE z K P x hilip h pe and man o y othe f r Han a dbag r s & Pu p rs w es, TI k FFA l NY & CO Je m werl t y Or z de v r T j ODA c Y and sav k e 2 o 5 % http://righttherehun.com share of the reward. And so Bilbo was swung down from the wall, and departed with nothing for all his trouble, except the armour which Thorin had given him already. More than one of the dwarves in their hearts felt shame and pity at his going. Farewell! he cried to them. We may meet again as friends. Be off! called Thorin. You have mail upon you, which was -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Mon Jul 24 13:05:35 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 24 Jul 2006 11:05:35 -0600 Subject: [Numpy-discussion] numpy 1.0 In-Reply-To: <5775F577-5FFE-4C0C-9832-856A2DF5C6EA@cns.montana.edu> References: <5775F577-5FFE-4C0C-9832-856A2DF5C6EA@cns.montana.edu> Message-ID: <44C4FDDF.9080001@ieee.org> Graham Cummins wrote: > Greetings, > > I just downloaded numpy 1.0b1. I see a lot of changes from 0.9.8, and > I'm curious as to whether these changes will be a lasting property of > numpy 1.0 and later. > > Most of the changes relate to nomenclature for type constants (e.g. > int32, complex128, newaxis) and functions (e.g. inverse_real_fft -> > ifft). Although it takes some time to comb through code for all of the > possible name changes (there are lots!), it's easy enough to do. Release notes will be forthcoming. These changes will be lasting... What changed is that "old" names were placed only in a compatibility module (numpy.oldnumeric). Import from there if you want the old names (convertcode was also changed to alter Numeric-->numpy.oldnumeric. This was done so as to make it clearer what is "old" and for compatibility purposes only and what new code should be written with. > > The thing that is taking me longer is (as usual) converting c > extensions. Float32, and PyArray_Float32 used to be defined in 0.9.8, > and are now not. AFAICT, npy_float works in the same way Float32 used > to work, but I haven't yet figured out what to use in place of > PyArray_Float32 in, for example "PyArray_FROM_OTF(data, ?? , ALIGNED | > CONTIGUOUS);" > Here, we assigned the prefixes NPY_ and npy_ to all the old CAPITALIZED and uncapitalized names, respectively to avoid the problem of name clashes which occurs commonly when using NumPy to wrap another library. The un-prefixed names are still available when you use #include "numpy/noprefix.h" (which is what NumPy itself uses). > > On another topic, when I install numpy (version 0.9.8 or 1.0b1) using > "seutup.py install", the headers needed to build extensions don't get > moved into my python distribution directory tree. I've been moving > these files by hand, and that seems to work, but could I pass some > argument to distutils that would do this automatically? To support multiple-version installations of NumPy (like eggs allow) it's important to put the headers in their own location and not in a system-wide directory. If you want to place them system-wide, you currently need to copy them by hand. But, it's not recommended to do that. Just append the output of numpy.get_include() to the list of include directories. -Travis From haase at msg.ucsf.edu Mon Jul 24 14:10:07 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 24 Jul 2006 11:10:07 -0700 Subject: [Numpy-discussion] trivial question: how to compare dtype - but ignoring byteorder ? In-Reply-To: <200607241218.55965.karol.langner@kn.pl> References: <44C450CA.3010609@msg.ucsf.edu> <200607241218.55965.karol.langner@kn.pl> Message-ID: <200607241110.07686.haase@msg.ucsf.edu> On Monday 24 July 2006 03:18, Karol Langner wrote: > On Monday 24 July 2006 06:47, Sebastian Haase wrote: > > Hi, > > if I have a numpy array 'a' > > and say: > > a.dtype == numpy.float32 > > > > Is the result independent of a's byteorder ? > > (That's what I would expect ! Just checking !) > > > > Thanks, > > Sebastian Haase > > The condition will always be False, because you're comparing wrong things > here, numpy.float32 is a scalar type, not a dtype. Hi ! Thanks for the reply. Did you actually run this ? I get: #>>> a=N.arange(10, dtype=N.float32) #>>> a.dtype == N.float32 #True #>>> N.__version__ #'0.9.9.2823' > > >>> numpy.float32 > > > > >>> type(numpy.dtype('>f4')) > > > > And I think byteorder matters when comparing dtypes: > >>> numpy.dtype('>f4') == numpy.dtype(' False OK - I did a test now: #>>> b= a.copy() #>>> b=b.newbyteorder('big') #>>> a.dtype == b.dtype #False #>>> a.dtype #'>> b.dtype #'>f4' How can I do a comparison showing that both a and b are float32 ?? Thanks, Sebastian Haase From karol.langner at kn.pl Mon Jul 24 14:36:54 2006 From: karol.langner at kn.pl (Karol Langner) Date: Mon, 24 Jul 2006 20:36:54 +0200 Subject: [Numpy-discussion] trivial question: how to compare dtype - but ignoring byteorder ? In-Reply-To: <200607241110.07686.haase@msg.ucsf.edu> References: <44C450CA.3010609@msg.ucsf.edu> <200607241218.55965.karol.langner@kn.pl> <200607241110.07686.haase@msg.ucsf.edu> Message-ID: <200607242036.54681.karol.langner@kn.pl> On Monday 24 July 2006 20:10, Sebastian Haase wrote: > Hi ! > Thanks for the reply. > Did you actually run this ? I get: > #>>> a=N.arange(10, dtype=N.float32) > #>>> a.dtype == N.float32 > #True > #>>> N.__version__ > #'0.9.9.2823' Hi, Looks like I need to upgrade my working version or something... >>> import numpy as N >>> N.__version__ '0.9.8' >>> a=N.arange(10, dtype=N.float32) >>> a.dtype == N.float32 False Cheers, Karol -- written by Karol Langner pon lip 24 20:36:05 CEST 2006 From haase at msg.ucsf.edu Mon Jul 24 14:37:35 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 24 Jul 2006 11:37:35 -0700 Subject: [Numpy-discussion] numpy.prod() vs. numpy.product() Message-ID: <200607241137.36094.haase@msg.ucsf.edu> Hi, Are numpy.product() and numpy.prod() doing the exact same thing ? If yes, why are they pointing to two different functions ? >>> N.prod >>> N.product Thanks, Sebastian Haase From oliphant.travis at ieee.org Mon Jul 24 15:18:46 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 24 Jul 2006 13:18:46 -0600 Subject: [Numpy-discussion] Problem converting a numarray extension to numpy In-Reply-To: <40e64fa20607202101o75ca115ate4ed577fd6d98ac3@mail.gmail.com> References: <40e64fa20607202101o75ca115ate4ed577fd6d98ac3@mail.gmail.com> Message-ID: <44C51D16.9020003@ieee.org> Paul Barrett wrote: > I'm having a problem converting a C extension module that was > originally written for numarray to use numpy. I using swig to create > a wrapper flle for the C code. I have added the > numpy.get_numarray_include() method to my setup.py file and have > changed the numarray/libnumarray.h to use numpy/libnumarray.h. The > extension appears to compile fine (with the exception of some warning > messages). However, when I import the module, I get a segfault. Do I > need to add anything else to the share library's initialization step > other than import_libnumarray()? > No, that should be enough. The numarray C-API has only been tested on a few extension modules. It's very possible some of the calls have problems. It's also possible you have an older version of numpy lying around somewhere. Do you get any kind of error message on import? -Travis From oliphant.travis at ieee.org Mon Jul 24 15:24:48 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 24 Jul 2006 13:24:48 -0600 Subject: [Numpy-discussion] bug in numpy recarray ? In-Reply-To: <44C309F1.5090607@msg.ucsf.edu> References: <44C309F1.5090607@msg.ucsf.edu> Message-ID: <44C51E80.40702@ieee.org> Sebastian Haase wrote: > Hi! > I'm trying to convert my numarray records code to numpy. > >>> type(m.hdrArray) > > >>> m.hdrArray.d > [(array([ 1., 1., 1.], dtype=float32),)] > > but I get: > >>> m.hdrArray[0].getfield('d') > 5.43230922614e-312 > > Am I missing something or is this a bug ? > Probably a bug. The getfield method needs more testing. File a ticket. -Travis From haase at msg.ucsf.edu Mon Jul 24 15:31:46 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 24 Jul 2006 12:31:46 -0700 Subject: [Numpy-discussion] how to get an array with "varying" poisson distribution Message-ID: <200607241231.46581.haase@msg.ucsf.edu> Hi, Essentially I'm looking for the equivalent of what was in numarray: from numarray import random_array random_array.poisson(arr) That is: if for example arr is a 256x256 array of positive integers, then this returns a new array of random numbers than are drawn according to the poisson statistics where arr's value at coordinate y,x determines the mean of the poisson distribution used to generate a new value for y,x. [[This is needed e.g. to simulate quantum noise in CCD images. Each pixel has different amount of noise depending of what it's (noise-free) "input" value was.]] Thanks, Sebastian Haase From oliphant.travis at ieee.org Mon Jul 24 15:29:15 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 24 Jul 2006 13:29:15 -0600 Subject: [Numpy-discussion] C extensions: is Py_XDECREF() needed when using PyArray_FromAny() In-Reply-To: <44C40501.3050705@msg.ucsf.edu> References: <44C40501.3050705@msg.ucsf.edu> Message-ID: <44C51F8B.60407@ieee.org> Sebastian Haase wrote: > Hi, > I'm converting SWIG typemap'ed C extensions from numarray to numpy. > I studied (and use parts of) numpy.i from the doc directory. > I noticed that there is no > decref for the TYPEMAP_INPLACE2 typemap. This uses a function > obj_to_array_no_conversion() which in turn just returns > the original PyObject* ( casted to a PyArrayObject* after some sanity > checks) It looks to me that in this case there should be an explicit > Py_INCREF() - in case the function is threaded (releases the Python > GIL) since it holds a pointer to that object's data . > Probably, true. The numpy.i typemaps are not thoroughly reference-count checked. > (Alternatively) Travis suggested (at the > http://www.scipy.org/Converting_from_numarray wiki page) using > PyArray_FromAny - is this incrementing the ref.count (implicitely) ? > The numarray equivalent (NA_InputArray) IS incrementing the ref.count > (as far as I know...). > > Yes, you get back a new reference from PyArray_FromAny. > Furthermore on that same wiki page the PyArray_FromAny() is called > together with PyArray_DescrFromType(). > After searching through the numpy source I found that in > blasdot/_dotblas.c (in dotblas_matrixproduct() )there is an explicit > Py_INCREF even on the dtype returned from PyArray_DescrFromType. > > PyArray_FromAny consumes a reference to the PyArray_Descr * object (which is a Python object). Thus, because PyArray_FromAny is called twice with the same data-type object, there is a need to increment it's reference count. -Travis From oliphant.travis at ieee.org Mon Jul 24 15:38:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 24 Jul 2006 13:38:02 -0600 Subject: [Numpy-discussion] matrix-related bug in 1.0b1 In-Reply-To: <44C4B3E7.6060104@gmx.net> References: <44C0EDD8.3000701@gmx.net> <44C4B3E7.6060104@gmx.net> Message-ID: <44C5219A.8020706@ieee.org> Sven Schreiber wrote: > Thanks for helping out on matrix stuff, Bill! > > > Hm, I don't know -- if you don't mind I'd like to get a second opinion > before I mess around there. It's funny though that the changeset has the > title "fixing up matrix slicing" or something like that... > The change was trying to fix up some cases but did break this one. The problem is that figuring out whether or not to transpose the result is a bit tricky. I've obviously still got it wrong. -Travis From oliphant.travis at ieee.org Mon Jul 24 15:36:56 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 24 Jul 2006 13:36:56 -0600 Subject: [Numpy-discussion] trivial question: how to compare dtype - but ignoring byteorder ? In-Reply-To: <44C450CA.3010609@msg.ucsf.edu> References: <44C450CA.3010609@msg.ucsf.edu> Message-ID: <44C52158.3050600@ieee.org> Sebastian Haase wrote: > Hi, > if I have a numpy array 'a' > and say: > a.dtype == numpy.float32 > > Is the result independent of a's byteorder ? > (That's what I would expect ! Just checking !) > I think I misread the question and saw "==" as "=" But, the answer I gave should still help: the byteorder is a property of the data-type. There is no such thing as "a's" byteorder. Thus, numpy.float32 (which is actually an array-scalar and not a true data-type) is interepreted as a machine-byte-order IEEE floating-point data-type with 32 bits. Thus, the result will depend on whether or not a.dtype is machine-order or not. -Travis From oliphant.travis at ieee.org Mon Jul 24 15:31:06 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 24 Jul 2006 13:31:06 -0600 Subject: [Numpy-discussion] trivial question: how to compare dtype - but ignoring byteorder ? In-Reply-To: <44C450CA.3010609@msg.ucsf.edu> References: <44C450CA.3010609@msg.ucsf.edu> Message-ID: <44C51FFA.9020002@ieee.org> Sebastian Haase wrote: > Hi, > if I have a numpy array 'a' > and say: > a.dtype == numpy.float32 > > Is the result independent of a's byteorder ? > The byteorder is a property of the data-type (not of the array) --- this is different from numarray where byteorder is a property of the array. a.dtype == numpy.float32 will always set the data-type to a machine-native float32 data-type. -Travis From svetosch at gmx.net Mon Jul 24 16:21:02 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Mon, 24 Jul 2006 22:21:02 +0200 Subject: [Numpy-discussion] matrix-related bug in 1.0b1 In-Reply-To: <44C5219A.8020706@ieee.org> References: <44C0EDD8.3000701@gmx.net> <44C4B3E7.6060104@gmx.net> <44C5219A.8020706@ieee.org> Message-ID: <44C52BAE.1090508@gmx.net> Travis Oliphant schrieb: > Sven Schreiber wrote: >> > The change was trying to fix up some cases but did break this one. The > problem is that figuring out whether or not to transpose the result is a > bit tricky. I've obviously still got it wrong. > Ok, this is obviously one of the places were an automated test would be helpful to avoid regressions. In the beta months ahead, I would like to help with contributing such tests. However, since I'm not a developer, I have little to no experience with that. Is there any numpy-specific documentation or guide on how to add tests? (I know there's plenty of guides on the general python testing framework.) Or any brilliant example in the distribution that should be followed? Regards, Sven p.s.: Is there going to be a new beta soon? Or is there a quick fix to this slicing problem in the meantime? Thanks. From haase at msg.ucsf.edu Mon Jul 24 16:28:13 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 24 Jul 2006 13:28:13 -0700 Subject: [Numpy-discussion] how add new attribute to a numpy array object ? In-Reply-To: <44C51E29.9040305@ieee.org> References: <44C2EF67.40505@msg.ucsf.edu> <44C51E29.9040305@ieee.org> Message-ID: <200607241328.13956.haase@msg.ucsf.edu> On Monday 24 July 2006 12:23, Travis Oliphant wrote: > Sebastian Haase wrote: > > Hi, > > I have a (medical) image file. > > I wrote a nice interface based on memmap using numarray. > > The class design I used was essentially to return a numarray array > > object with a new "custom" attribute giving access to special > > information about the base file. > > > > Now with numpy I noticed that a numpy object does not allow adding new > > attributes !! (How is this ? Why ?) > > > > Travis already suggested (replying to one of my last postings) to create > > a new sub class of numpy.ndarray. > > > > But how do I initialize an object of my new class to be "basically > > identically to" an existing ndarray object ? > > Normally I could do > > class B(N.ndarray): > > pass > > a=N.arange(10) > > a.__class__ = B > > > > BUT I get this error: > > #>>> a.__class__ = B > > Traceback (most recent call last): > > File "", line 1, in ? > > TypeError: __class__ assignment: only for heap types > > > > What is a "heap type" ? Why ? How can I do what I want ? > > A heap type is one created in Python (i.e. not builtin with C). > > You should be able to do > > a = a.view(B) > > -Travis Thanks - (I'm just replying because I assume your helpful answer was mistakenly sent to me only and did not make it to the list) From oliphant.travis at ieee.org Mon Jul 24 17:01:17 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 24 Jul 2006 15:01:17 -0600 Subject: [Numpy-discussion] SVN changes that need to go into 1.0 and 1.1 Message-ID: <44C5351D.7070701@ieee.org> For the next several months until 1.0 comes out. Please make changes that go into 1.0 to the ver1.0 branch of the NumPy SVN tree. Then, periodically, we can merge those changes back to the trunk for ver 1.1 -Travis From oliphant.travis at ieee.org Mon Jul 24 17:06:57 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 24 Jul 2006 15:06:57 -0600 Subject: [Numpy-discussion] Please comment on SVN versioning convention Message-ID: <44C53671.5060301@ieee.org> Andrew Straw has emphasized that the current strategy of appending the SVN version number to development versions of the SVN tree makes it hard to do version sorting. His proposal is to not change the version number until the first beta comes out. In other words, the trunk should not be 1.1 but 1.0dev? Are there any opposing opinions to this change. I personally think that is a confusing numbering scheme because 1.0dev seems to imply it's the development for version 1.0 instead of 1.1. But, if several others think it's a good idea to support easy-sorting of version numbers, then I will conceed. Perhaps we can have a version 'number' as 1.0plus? -Travis From fperez.net at gmail.com Mon Jul 24 17:15:48 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 24 Jul 2006 15:15:48 -0600 Subject: [Numpy-discussion] Please comment on SVN versioning convention In-Reply-To: <44C53671.5060301@ieee.org> References: <44C53671.5060301@ieee.org> Message-ID: On 7/24/06, Travis Oliphant wrote: > Andrew Straw has emphasized that the current strategy of appending the > SVN version number to development versions of the SVN tree makes it hard > to do version sorting. > > His proposal is to not change the version number until the first beta > comes out. I have to say that being able to quickly see the actual SVN revision number in the version helps a LOT in tracking down problems. Just look at how many posts on the list start with the canonical In [4]: import numpy,scipy In [5]: numpy.__version__ Out[5]: '0.9.9.2803' In [6]: scipy.__version__ Out[6]: '0.5.0.2079' printouts before discussing the problem. I don't really feel strongly about the issue, but if you change this, then please add a __revision__ attribute as well, so that this information can be quickly asked for in pure python (we don't want to bother newbies with obscure SVN lingo, esp. if they got their install from someone else). Cheers, f From robert.kern at gmail.com Mon Jul 24 17:20:42 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 24 Jul 2006 16:20:42 -0500 Subject: [Numpy-discussion] how to get an array with "varying" poisson distribution In-Reply-To: <200607241231.46581.haase@msg.ucsf.edu> References: <200607241231.46581.haase@msg.ucsf.edu> Message-ID: Sebastian Haase wrote: > Hi, > Essentially I'm looking for the equivalent of what was in numarray: > from numarray import random_array > random_array.poisson(arr) > > That is: if for example arr is a 256x256 array of positive integers, then this > returns a new array of random numbers than are drawn according to the poisson > statistics where arr's value at coordinate y,x determines the mean of the > poisson distribution used to generate a new value for y,x. I'm afraid that at this point in time, the distributions only accept scalar values for the parameters. I've thought about reimplementing the distribution functions as ufuncs, but that's a hefty chunk of work that won't happen for 1.0. I'm afraid that, for now, you're stuck with iterating over the values. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cookedm at physics.mcmaster.ca Mon Jul 24 17:24:18 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Mon, 24 Jul 2006 17:24:18 -0400 Subject: [Numpy-discussion] Fresh svn numpy install fails 2 tests (OS X) In-Reply-To: <358EE207-3D29-40BE-87EA-912639CB93FD@arachnedesign.net> References: <358EE207-3D29-40BE-87EA-912639CB93FD@arachnedesign.net> Message-ID: <20060724172418.661a9137@arbutus.physics.mcmaster.ca> On Sat, 22 Jul 2006 17:50:13 -0400 Steve Lianoglou wrote: > Hi folks, > > Since 1.0 release is eminent, I just wanted to draw the attention to > two failures I get when I run numpy.test(1). > > I've never been able to get numpy to pass all test cases, but now it > fails a second one, so .. I'm pasting it below. Please let me know if > these are non-consequential. It fails the second one b/c I added it because the failure in the first one wasn't clear enough :-) > System info: > + Intel Mac (MacBook Pro) > + OS X.4.7 > + numpy version: 1.0.2881 > > test failures: > > FAIL: check_large_types (numpy.core.tests.test_scalarmath.test_power) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/opt/local/lib/python2.4/site-packages/numpy/core/tests/ > test_scalarmath.py", line 47, in check_large_types > assert b == 6765201, "error with %r: got %r" % (t,b) > AssertionError: error with : got 0.0 > > ====================================================================== > FAIL: check_types (numpy.core.tests.test_scalarmath.test_types) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/opt/local/lib/python2.4/site-packages/numpy/core/tests/ > test_scalarmath.py", line 20, in check_types > assert a == 1, "error with %r: got %r" % (atype,a) > AssertionError: error with : got > 1.02604810941480982577e-4936 > > ---------------------------------------------------------------------- > Ran 468 tests in 1.157s > > FAILED (failures=2) > Out[2]: I'm aware of this (http://projects.scipy.org/scipy/numpy/ticket/183). It's a problem on PPC macs also. Travis thinks it may be a compiler problem. I've had a look, and can't see anything obvious. It *could* be that somewhere there's a typo in the code where things are set when sizeof(long double) == 128. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From cookedm at physics.mcmaster.ca Mon Jul 24 17:35:58 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Mon, 24 Jul 2006 17:35:58 -0400 Subject: [Numpy-discussion] Please comment on SVN versioning convention In-Reply-To: <44C53671.5060301@ieee.org> References: <44C53671.5060301@ieee.org> Message-ID: <20060724173558.660098ba@arbutus.physics.mcmaster.ca> On Mon, 24 Jul 2006 15:06:57 -0600 Travis Oliphant wrote: > Andrew Straw has emphasized that the current strategy of appending the > SVN version number to development versions of the SVN tree makes it hard > to do version sorting. > > His proposal is to not change the version number until the first beta > comes out. > > In other words, the trunk should not be 1.1 but 1.0dev? > > Are there any opposing opinions to this change. I personally think that > is a confusing numbering scheme because 1.0dev seems to imply it's the > development for version 1.0 instead of 1.1. But, if several others > think it's a good idea to support easy-sorting of version numbers, then > I will conceed. > > Perhaps we can have a version 'number' as 1.0plus? What if we use even numbers for stable releases, odd for development releases, like for Linux kernels? e.g., make 1.1 the dev branch (current trunk), and make 1.2 the next stable release after 1.0. Or, make the trunk 1.0.99. The .99 usually makes it pretty clear that it's the development branch. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From ndarray at mac.com Mon Jul 24 17:49:37 2006 From: ndarray at mac.com (Sasha) Date: Mon, 24 Jul 2006 17:49:37 -0400 Subject: [Numpy-discussion] Please comment on SVN versioning convention In-Reply-To: <44C53671.5060301@ieee.org> References: <44C53671.5060301@ieee.org> Message-ID: On 7/24/06, Travis Oliphant wrote: > Andrew Straw has emphasized that the current strategy of appending the > SVN version number to development versions of the SVN tree makes it hard > to do version sorting. I am not sure what the problem is, but if the concern is that >>> '0.9.9.2803' > '0.9.9' True I suggest fixing that by appending '.final' to the release version string: >>> '0.9.9.2803' > '0.9.9.final' False If we are going to make any changes in this area, I would suggest following python sys and define three variables: numpy.version (human readable string including compiler info), numpy.version_info (sortable tuple) and numpy.hexversion (sortable integer that can be used by C preprocessor). I am not sure if python sys includes svn revision number in sys.version (I don't have svn version of python around), but it does include it in beta sys.version: >>> print sys.version 2.5b2 (r25b2:50512, Jul 18 2006, 15:22:50) [GCC 3.4.4 20050721 (Red Hat 3.4.4-2)] From haase at msg.ucsf.edu Mon Jul 24 18:02:22 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 24 Jul 2006 15:02:22 -0700 Subject: [Numpy-discussion] weave - was:Re: how to get an array with "varying" poisson distribution In-Reply-To: References: <200607241231.46581.haase@msg.ucsf.edu> Message-ID: <200607241502.22529.haase@msg.ucsf.edu> On Monday 24 July 2006 14:20, Robert Kern wrote: > Sebastian Haase wrote: > > Hi, > > Essentially I'm looking for the equivalent of what was in numarray: > > from numarray import random_array > > random_array.poisson(arr) > > > > That is: if for example arr is a 256x256 array of positive integers, then > > this returns a new array of random numbers than are drawn according to > > the poisson statistics where arr's value at coordinate y,x determines > > the mean of the poisson distribution used to generate a new value for > > y,x. > > I'm afraid that at this point in time, the distributions only accept scalar > values for the parameters. I've thought about reimplementing the > distribution functions as ufuncs, but that's a hefty chunk of work that > won't happen for 1.0. > > I'm afraid that, for now, you're stuck with iterating over the values. Thanks for the reply - maybe this my time to get into weave ;-) Question about distributing and/or making "python-only-changes". How can one distribute modules using weave to other people who might NOT have a C compiler installed ? And further: when I change parts of that module that should not require a recompiling of the C part - is weave smart enough to recognise this ? Thanks, Sebastian (Oops, I just realized that this would be a question maybe for the SciPy list ... I'll assume that it's OK) From strawman at astraw.com Mon Jul 24 18:18:37 2006 From: strawman at astraw.com (Andrew Straw) Date: Mon, 24 Jul 2006 15:18:37 -0700 Subject: [Numpy-discussion] Please comment on SVN versioning convention In-Reply-To: References: <44C53671.5060301@ieee.org> Message-ID: <44C5473D.5090301@astraw.com> Sasha wrote: >On 7/24/06, Travis Oliphant wrote: > > >>Andrew Straw has emphasized that the current strategy of appending the >>SVN version number to development versions of the SVN tree makes it hard >>to do version sorting. >> >> > >I am not sure what the problem is, but if the concern is that > > > >>>>'0.9.9.2803' > '0.9.9' >>>> >>>> >True > >I suggest fixing that by appending '.final' to the release version string: > > The concern is that there are a bazillion ways of sorting version numbers but the version numbers from numpy's current svn tree don't sort with (m)any of them. Sorting strings in Python is one way, using setuptools or debian's dpkg are other ways. Remember, all this is only a concern for those working from SVN versions -- the releases are stripped of these numbers, anyway. I don't really care what we use, it's just that if we want to make life easy for people working from SVN, these automated tools should have a way of knowing that what we currently call numpy 1.1.2881 precedes numpy 1.1 (these are current examples from the SVN tree). Here are some proposals for this version: 1.0.2881 - This would sort after 1.0 (and 1.0b and 1.0c) and before 1.1 for most tools out there. 1.0.post2881 - This would sort after 1.0 (and 1.0b and 1.0c) and before 1.1 for most tools out there, but is perhaps more clear. 1.1.dev-r2881 - this wouldn't solve the problem for Debian, but it would for setuptools, which has special treatment for ".dev-r" (see http://peak.telecommunity.com/DevCenter/setuptools). This has the advantage of being only a minimal change from the current scheme. 1.0+1.1.2881 - I also think this would sort OK with most tools, and is also a fairly minimal change. My only concern is the '+' might cause some filesystems or tools to freak out. BTW, Fernando, all this is so that we can still have the svn revision number in __version__, but so that it doesn't screw up sorting. Cheers! Andrew From ndarray at mac.com Mon Jul 24 18:38:42 2006 From: ndarray at mac.com (Sasha) Date: Mon, 24 Jul 2006 18:38:42 -0400 Subject: [Numpy-discussion] Please comment on SVN versioning convention In-Reply-To: <44C5473D.5090301@astraw.com> References: <44C53671.5060301@ieee.org> <44C5473D.5090301@astraw.com> Message-ID: On 7/24/06, Andrew Straw wrote: [snip] > The concern is that there are a bazillion ways of sorting version > numbers but the version numbers from numpy's current svn tree don't sort > with (m)any of them. Sorting strings in Python is one way, using > setuptools or debian's dpkg are other ways. Sorting numbers as strings is a problem no matter what tools you use. '1000' is sorted before '999' in every tool I know of. The only proper way to solve this problem is to make svn revision available as an integer and sort them as integers. This integer may go in version_info tuple or in a separate variable. The same information may be encoded into a sortable string using appropriate padding, but I don't see much need for that. Do you expect creating debian's dpkg from a non-release version? From ndarray at mac.com Mon Jul 24 18:47:10 2006 From: ndarray at mac.com (Sasha) Date: Mon, 24 Jul 2006 18:47:10 -0400 Subject: [Numpy-discussion] Please comment on SVN versioning convention In-Reply-To: <44C5473D.5090301@astraw.com> References: <44C53671.5060301@ieee.org> <44C5473D.5090301@astraw.com> Message-ID: On 7/24/06, Andrew Straw wrote: > BTW, Fernando, all this is so that we can still have the svn revision > number in __version__, but so that it doesn't screw up sorting. Sorting __version__ is screwed up regardless of svn revision number: >>> '1.9' < '1.10' False From strawman at astraw.com Mon Jul 24 18:47:07 2006 From: strawman at astraw.com (Andrew Straw) Date: Mon, 24 Jul 2006 15:47:07 -0700 Subject: [Numpy-discussion] Please comment on SVN versioning convention In-Reply-To: References: <44C53671.5060301@ieee.org> <44C5473D.5090301@astraw.com> Message-ID: <44C54DEB.8010801@astraw.com> Sasha wrote: >On 7/24/06, Andrew Straw wrote: >[snip] > > >>The concern is that there are a bazillion ways of sorting version >>numbers but the version numbers from numpy's current svn tree don't sort >>with (m)any of them. Sorting strings in Python is one way, using >>setuptools or debian's dpkg are other ways. >> >> > >Sorting numbers as strings is a problem no matter what tools you use. >'1000' is sorted before '999' in every tool I know of. The only proper >way to solve this problem is to make svn revision available as an >integer and sort them as integers. This integer may go in version_info >tuple or in a separate variable. The same information may be encoded >into a sortable string using appropriate padding, but I don't see much >need for that. > Yes, I only brought up version numbers as strings because you did. I'm not proposing we solve that problem. I see setuptools, in particular, as the biggest thing to support, because it lets you have multiple versions installed simultaneously. And I don't know how to force its hand, or even if it's possible. >Do you expect creating debian's dpkg from a >non-release version? > > Yes. I do personally (see http://debs.astraw.com/dapper/ for example) and I'm sure others do, as well. You can see from that archive that I've been dealing with this issue by adding appropriate suffixes or prefixes. (Forcing its hand.) From strawman at astraw.com Mon Jul 24 19:03:13 2006 From: strawman at astraw.com (Andrew Straw) Date: Mon, 24 Jul 2006 16:03:13 -0700 Subject: [Numpy-discussion] Please comment on SVN versioning convention In-Reply-To: References: <44C53671.5060301@ieee.org> <44C5473D.5090301@astraw.com> Message-ID: <44C551B1.8030708@astraw.com> Sasha wrote: >On 7/24/06, Andrew Straw wrote: > > > >>BTW, Fernando, all this is so that we can still have the svn revision >>number in __version__, but so that it doesn't screw up sorting. >> >> > >Sorting __version__ is screwed up regardless of svn revision number: > > > >>>>'1.9' < '1.10' >>>> >>>> >False > Not if you parse __version__ with something that understands version numbers. From ndarray at mac.com Mon Jul 24 19:09:15 2006 From: ndarray at mac.com (Sasha) Date: Mon, 24 Jul 2006 19:09:15 -0400 Subject: [Numpy-discussion] Please comment on SVN versioning convention In-Reply-To: <44C54DEB.8010801@astraw.com> References: <44C53671.5060301@ieee.org> <44C5473D.5090301@astraw.com> <44C54DEB.8010801@astraw.com> Message-ID: On 7/24/06, Andrew Straw wrote: > Yes, I only brought up version numbers as strings because you did. I'm > not proposing we solve that problem. I see setuptools, in particular, as > the biggest thing to support, because it lets you have multiple versions > installed simultaneously. And I don't know how to force its hand, or > even if it's possible. I am not familiar with setuptools and I expect that many others who care about __version__ conventions are not either. Please explain exactly how setuptools depend on what is stored in __version__ and why using 'final' in place of svn revision number in file names will not solve your problems. I would be against any change that will affect map(int, __version__.split('.')[:3]). From cookedm at physics.mcmaster.ca Mon Jul 24 19:10:22 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Mon, 24 Jul 2006 19:10:22 -0400 Subject: [Numpy-discussion] Please comment on SVN versioning convention In-Reply-To: <44C551B1.8030708@astraw.com> References: <44C53671.5060301@ieee.org> <44C5473D.5090301@astraw.com> <44C551B1.8030708@astraw.com> Message-ID: <20060724231022.GA10758@arbutus.physics.mcmaster.ca> On Mon, Jul 24, 2006 at 04:03:13PM -0700, Andrew Straw wrote: > Sasha wrote: > > >On 7/24/06, Andrew Straw wrote: > > > >>BTW, Fernando, all this is so that we can still have the svn revision > >>number in __version__, but so that it doesn't screw up sorting. > > > >Sorting __version__ is screwed up regardless of svn revision number: > > > >>>>'1.9' < '1.10' > >>>> > >>>> > >False > > > Not if you parse __version__ with something that understands version > numbers. ...and we've already got something for that: distutils.version.LooseVersion (or StrictVersion). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From wbaxter at gmail.com Mon Jul 24 19:30:22 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 25 Jul 2006 08:30:22 +0900 Subject: [Numpy-discussion] numpy.prod() vs. numpy.product() In-Reply-To: <200607241137.36094.haase@msg.ucsf.edu> References: <200607241137.36094.haase@msg.ucsf.edu> Message-ID: I can't answer why, but in oldnumeric.py, you can see there's two different fuctions with basically identical code, so yes they are distinct, but no they are not different. Seems like prod could be replaced by prod=product. def product (x, axis=0, dtype=None): """Product of the array elements over the given axis.""" try: prod = x.prod except AttributeError: return _wrapit(x, 'prod', axis, dtype) return prod(axis, dtype) def prod(a, axis=0, dtype=None): """Return the product of the elements along the given axis """ try: prod = a.prod except AttributeError: return _wrapit(a, 'prod', axis, dtype) return prod(axis, dtype) --bb On 7/25/06, Sebastian Haase wrote: > Hi, > Are numpy.product() and numpy.prod() > doing the exact same thing ? If yes, why are they pointing to two different > functions ? > >>> N.prod > > >>> N.product > > > Thanks, > Sebastian Haase > From wbaxter at gmail.com Mon Jul 24 19:42:18 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 25 Jul 2006 08:42:18 +0900 Subject: [Numpy-discussion] trivial question: how to compare dtype - but ignoring byteorder ? In-Reply-To: <200607241110.07686.haase@msg.ucsf.edu> References: <44C450CA.3010609@msg.ucsf.edu> <200607241218.55965.karol.langner@kn.pl> <200607241110.07686.haase@msg.ucsf.edu> Message-ID: > > And I think byteorder matters when comparing dtypes: > > >>> numpy.dtype('>f4') == numpy.dtype(' > False Ohhhhh -- that '<' part is indicating *byte order* ?! I thought it was odd that numpy could only tell me the type was "less than f4", which I assumed must be shorthand for "less than or equal to f4". Makes much more sense now! --bb From wbaxter at gmail.com Mon Jul 24 19:49:11 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 25 Jul 2006 08:49:11 +0900 Subject: [Numpy-discussion] matrix-related bug in 1.0b1 In-Reply-To: <44C52BAE.1090508@gmx.net> References: <44C0EDD8.3000701@gmx.net> <44C4B3E7.6060104@gmx.net> <44C5219A.8020706@ieee.org> <44C52BAE.1090508@gmx.net> Message-ID: On 7/25/06, Sven Schreiber wrote: > Travis Oliphant schrieb: > > Sven Schreiber wrote: > >> > > The change was trying to fix up some cases but did break this one. The > > problem is that figuring out whether or not to transpose the result is a > > bit tricky. I've obviously still got it wrong. Is there some test case that demonstrates the problem it was intended to fix? Or can you remember off the top of your head? I tried as many things as I could think of but nothing I tried gave the wrong answer with the fix of removing the 'not' on line 140. > p.s.: Is there going to be a new beta soon? Or is there a quick fix to > this slicing problem in the meantime? Thanks. Sven, if you were happy with the behavior under numpy 0.9.8 then all you have to do is just cut and paste its matrix.__getitem__ method into your current site-packages/numpy/core/defmatrix.py. --bill From myeates at jpl.nasa.gov Mon Jul 24 20:10:19 2006 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Mon, 24 Jul 2006 17:10:19 -0700 Subject: [Numpy-discussion] 1.0b1 problem on Solaris Message-ID: <44C5616B.8020209@jpl.nasa.gov> I installed numpy-1.0b1 and now >import numpy produces a segv. According to gdb it's happening after a failure to import scipy.optimize.minpack2. Eventually, Py_FatalError is called followed by abort,raise and thr_kill. Then the SEGV. Any ideas? Do I need a different version of scipy? I currently use 0.4.9 Mathew From haase at msg.ucsf.edu Mon Jul 24 20:31:53 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 24 Jul 2006 17:31:53 -0700 Subject: [Numpy-discussion] trivial question: how to compare dtype - but ignoring byteorder ? In-Reply-To: References: <44C450CA.3010609@msg.ucsf.edu> <200607241110.07686.haase@msg.ucsf.edu> Message-ID: <200607241731.54174.haase@msg.ucsf.edu> On Monday 24 July 2006 16:42, Bill Baxter wrote: > > > And I think byteorder matters when comparing dtypes: > > > >>> numpy.dtype('>f4') == numpy.dtype(' > > > > > False > > Ohhhhh -- that '<' part is indicating *byte order* ?! > I thought it was odd that numpy could only tell me the type was "less > than f4", which I assumed must be shorthand for "less than or equal to > f4". Makes much more sense now! > > --bb Which is why I was trying to change the str() representation of a type to something more intuitive. If nothing else one could even leave repr(a.dtype) --> ' 'int32 (little endian)' I do now understand that (as opposed to numarray and numeric) the byteorder is now part of the data-type - but I would really encourage keeping the string for such an important (and often used !) thing more readable than " I'm trying to work with memmaps on very large files, i.e. > 2 GB, up to 10 GB. The files are data cubes of images (my largest is 1290(x)x1024(y)x2011(z)) and my immediate task is to strip the data from 32-bits down to 16, and to rearrange some of the data on a per-xy-plane basis. I'm running this on a Fedora Core 5 64-bit system, with python-2.5b2(that I believe I compiled in 64-bit mode) and numpy-1.0b1. The disk has 324 GB free space. The log from a minimal case is as follows: ressler > python2.5 Python 2.5b2 (r25b2:50512, Jul 18 2006, 12:58:29) [GCC 4.1.1 20060525 (Red Hat 4.1.1-1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> data=np.memmap('temp_file',mode='w+',shape=(2011,1280,1032),dtype='h') size = 2656450560 bytes = 5312901120 len(mm) = 5312901120 (2011, 1280, 1032) h 0 0 Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.5/site-packages/numpy/core/memmap.py", line 75, in __new__ offset=offset, order=order) TypeError: buffer is too small for requested array >>> If I have a small number of frames (z=800 rather than 2011), this all works fine. I've added a few lines to memmap.py to print some diagnostic information - the error occurs on line 71 in the original memmap.py file, not 75. The "size =" and "bytes =" lines show that memmap.py is calculating the correct size for the buffer, and the len(mm) shows that the python mmap.mmap call on line 67 is returning a buffer of the correct size. The "(2011, 1280, 1032) h 0 0" bit is from a print statement that was left in the source file by the authors, and indicates what the following "self = ndarray.__new__" call is trying to do. However, it is the ndarray.__new__ call that is breaking down, and I don't really have enough skill to continue chasing it down. I took a quick look at the C source, but I couldn't figure out where the ndarray.__new__ is actually defined. Any suggestions to help me past this? Thanks. Mike -- mike.ressler at alum.mit.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Mon Jul 24 20:43:46 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 24 Jul 2006 17:43:46 -0700 Subject: [Numpy-discussion] trivial question: how to compare dtype - but ignoring byteorder ? In-Reply-To: <200607241731.54174.haase@msg.ucsf.edu> References: <44C450CA.3010609@msg.ucsf.edu> <200607241110.07686.haase@msg.ucsf.edu> <200607241731.54174.haase@msg.ucsf.edu> Message-ID: <44C56942.9050908@noaa.gov> Sebastian Haase wrote: > Which is why I was trying to change the str() representation of a type to > something more intuitive. > If nothing else one could even leave > repr(a.dtype) --> ' but > str(a.dtype) --> 'int32 (little endian)' +1 that's the whole point of __str__. It should be human readable! -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From wbaxter at gmail.com Mon Jul 24 21:16:33 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 25 Jul 2006 10:16:33 +0900 Subject: [Numpy-discussion] Please comment on SVN versioning convention In-Reply-To: <44C5473D.5090301@astraw.com> References: <44C53671.5060301@ieee.org> <44C5473D.5090301@astraw.com> Message-ID: On 7/25/06, Andrew Straw wrote: > 1.0.2881 - This would sort after 1.0 (and 1.0b and 1.0c) and before 1.1 > for most tools out there. I like that best. Save the 1.1 prefix until it's actually released as such. The numbering scheme needs to deal with what to call the patch tip of the stable release branch, too. I.e. the maintenance release after 1.0 with only critical patches for anyone who demands strict 1.0 compatibility. The third number would be good for that. Right now I see 1.1.2881 as my version. I think this should probably have an extra .0 in it: 1.1.0.2881. That third number would be a maintenance release number. I also like the Linux even-odd system. Then 1.0.0.1234 can mean "SVN version on the 1.0 branch, leading up to maintenance release 1.0.1", and 1.1.0.1234 can mean "SVN version on the trunk, leading up to 1.2 release." --bb From wbaxter at gmail.com Mon Jul 24 21:20:45 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 25 Jul 2006 10:20:45 +0900 Subject: [Numpy-discussion] 1.0b1 problem on Solaris In-Reply-To: <44C5616B.8020209@jpl.nasa.gov> References: <44C5616B.8020209@jpl.nasa.gov> Message-ID: Yep, scipy needs to be recompiled against the 1.0b1 version of numpy. I think they should have an updated version on the web site in a few days or so if you don't feel like compiling yourself. --bb On 7/25/06, Mathew Yeates wrote: > I installed numpy-1.0b1 and now >import numpy produces a segv. According > to gdb it's happening after a failure to import scipy.optimize.minpack2. > Eventually, Py_FatalError is called followed by abort,raise and > thr_kill. Then the SEGV. > > Any ideas? Do I need a different version of scipy? I currently use 0.4.9 > > > Mathew > From myeates at jpl.nasa.gov Mon Jul 24 21:30:31 2006 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Mon, 24 Jul 2006 18:30:31 -0700 Subject: [Numpy-discussion] 1.0b1 problem on Solaris In-Reply-To: References: <44C5616B.8020209@jpl.nasa.gov> Message-ID: <44C57437.40607@jpl.nasa.gov> Where is the source? Do I just use the 0.4.9 source? Bill Baxter wrote: > Yep, scipy needs to be recompiled against the 1.0b1 version of numpy. > I think they should have an updated version on the web site in a few > days or so if you don't feel like compiling yourself. > > --bb > > On 7/25/06, Mathew Yeates wrote: >> I installed numpy-1.0b1 and now >import numpy produces a segv. According >> to gdb it's happening after a failure to import scipy.optimize.minpack2. >> Eventually, Py_FatalError is called followed by abort,raise and >> thr_kill. Then the SEGV. >> >> Any ideas? Do I need a different version of scipy? I currently use 0.4.9 >> >> >> Mathew >> > From wbaxter at gmail.com Mon Jul 24 21:47:28 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 25 Jul 2006 10:47:28 +0900 Subject: [Numpy-discussion] 1.0b1 problem on Solaris In-Reply-To: <44C57437.40607@jpl.nasa.gov> References: <44C5616B.8020209@jpl.nasa.gov> <44C57437.40607@jpl.nasa.gov> Message-ID: Howdy, Instructions for compiling your own SciPy can be found here. http://www.scipy.org/Installing_SciPy I think the 0.4.9 source will probably work. But getting it from SVN is probably safer. --bb On 7/25/06, Mathew Yeates wrote: > Where is the source? Do I just use the 0.4.9 source? > > > Bill Baxter wrote: > > Yep, scipy needs to be recompiled against the 1.0b1 version of numpy. > > I think they should have an updated version on the web site in a few > > days or so if you don't feel like compiling yourself. > > > > --bb > > > > On 7/25/06, Mathew Yeates wrote: > >> I installed numpy-1.0b1 and now >import numpy produces a segv. According > >> to gdb it's happening after a failure to import scipy.optimize.minpack2. > >> Eventually, Py_FatalError is called followed by abort,raise and > >> thr_kill. Then the SEGV. > >> > >> Any ideas? Do I need a different version of scipy? I currently use 0.4.9 > >> > >> > >> Mathew > >> > > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From yukihana at yahoo.co.jp Mon Jul 24 21:49:59 2006 From: yukihana at yahoo.co.jp (=?iso-2022-jp?B?eXVraWhhbmE=?=) Date: Tue, 25 Jul 2006 01:49:59 -0000 Subject: [Numpy-discussion] (no subject) Message-ID: :?? INFORMATION ?????????????????????????: ?????????????????????? ???????????? http://love-match.bz/pc/?02 :??????????????????????????????????: *????*:.?. .?.:*????*:.?..?:*????*:.?..?:**????* ??????????????????????????????????? ??? ???????????????????Love?Match? ?----------------------------------------------------------------- ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ?----------------------------------------------------------------- ????????????????http://love-match.bz/pc/?02 ??????????????????????????????????? ??? ?????????????????????? ?----------------------------------------------------------------- ???????????????????????????? ?----------------------------------------------------------------- ????????????????????????????? ??????????????????????????????? ?http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????????????????????????????? ?----------------------------------------------------------------- ???????????????????????????????? ????????????????????? ?http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????????????????? ?----------------------------------------------------------------- ???????????????????????? ?????????????????????????????????? ?http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????????????????????????? ?----------------------------------------------------------------- ??????????????????????????? ????????????????????????????????? ?http://love-match.jp/pc/?06 ?----------------------------------------------------------------- ????????????????????????? ?----------------------------------------------------------------- ????????????????????????? ????????????????????????????????? ?http://love-match.jp/pc/?06 ??????????????????????????????????? ??? ??500???????????????? ?----------------------------------------------------------------- ???????/???? ???????????????????? ????????????????????????????????? ???????????????????????????????? ?????????????????????????? ?????????????????????????????? ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????/?????? ?????????????????????????????????? ??????????????????????????????????? ?????????? ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????/????? ?????????????????????????????????? ???????????????????????????????? ?????????????????????????(^^) ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????/???? ??????????????????????????????? ?????????????????????????????? ?????????????????????????????? ???????? ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ????????/??? ???????????????1??? ????????????????????????? ????????????????????????? ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????/??????? ????18?????????????????????????? ????????????????????????????? ????????????????????????????? ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???`????/??? ????????????????????? ?????????????????????? ?????????????? ?[????] http://love-match.bz/pc/?02 ?----------------------------------------------------------------- ???????????????????? ?????????????????????????????????? ????????????? ??------------------------------------------------------------- ???????????????????????????????? ??[??????????]?http://love-match.bz/pc/?02 ??------------------------------------------------------------- ????????????????????? ??????????????????????????? ??????????????????? ??????????????????????????????? ??[??????????]?http://love-match.bz/pc/?02 ?????????????????????????????????? ???????????? ??????????3-6-4-533 ?????? 139-3668-7892 From charlesr.harris at gmail.com Mon Jul 24 23:42:31 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 24 Jul 2006 21:42:31 -0600 Subject: [Numpy-discussion] how to get an array with "varying" poisson distribution In-Reply-To: <200607241231.46581.haase@msg.ucsf.edu> References: <200607241231.46581.haase@msg.ucsf.edu> Message-ID: Hi Sebastian, On 7/24/06, Sebastian Haase wrote: > > Hi, > Essentially I'm looking for the equivalent of what was in numarray: > from numarray import random_array > random_array.poisson(arr) > > That is: if for example arr is a 256x256 array of positive integers, then > this > returns a new array of random numbers than are drawn according to the > poisson > statistics where arr's value at coordinate y,x determines the mean of > the > poisson distribution used to generate a new value for y,x. > > [[This is needed e.g. to simulate quantum noise in CCD images. Each pixel > has > different amount of noise depending of what it's (noise-free) "input" > value > was.]] How accurate do you want the distribution to be and what sort of offset is there? If the number of counts is greater that about 10 and you don't care too much about the poisson tail then a gaussian will work fine. If you have very large counts (>1e6), which I doubt, the accuracy of the poisson distribution becomes a tricky matter because it needs care to compute the factorial to the needed accuracy. The factorial turns up in the rejection methods used in the algorithms. Most of the algorithms also compute special constants that depend on the mean of the distribution, so while efficient for a fixed mean they are less so for varying means like you want. I tend to just go with gaussian noise most of the time. Thanks, > Sebastian Haase > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share > your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From emi0924 at yahoo.co.jp Tue Jul 25 01:00:35 2006 From: emi0924 at yahoo.co.jp (=?iso-2022-jp?B?ZW1p?=) Date: Tue, 25 Jul 2006 05:00:35 -0000 Subject: [Numpy-discussion] (no subject) Message-ID: :?? INFORMATION ?????????????????????????: ?????????????????????? ???????????? http://love-match.bz/pc/?010 :??????????????????????????????????: *????*:.?. .?.:*????*:.?..?:*????*:.?..?:**????* ?????????????????????????????? ??[??????????]?http://love-match.bz/pc/?010 ??????????????????????????????????? ??? ???????????????????Love?Match? ?----------------------------------------------------------------- ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ?----------------------------------------------------------------- ????????????????http://love-match.bz/pc/?010 ??????????????????????????????????? ??? ?????????????????????? ?----------------------------------------------------------------- ???????????????????????????? ?----------------------------------------------------------------- ????????????????????????????? ??????????????????????????????? ?http://love-match.bz/pc/?010 ?----------------------------------------------------------------- ???????????????????????????????? ?----------------------------------------------------------------- ???????????????????????????????? ????????????????????? ?http://love-match.bz/pc/?010 ?----------------------------------------------------------------- ???????????????????? ?----------------------------------------------------------------- ???????????????????????? ?????????????????????????????????? ?http://love-match.bz/pc/?010 ?----------------------------------------------------------------- ???????????????????????????? ?----------------------------------------------------------------- ??????????????????????????? ????????????????????????????????? ?http://love-match.bz/pc/?010 ?----------------------------------------------------------------- ????????????????????????? ?----------------------------------------------------------------- ????????????????????????? ????????????????????????????????? ?http://love-match.bz/pc/?010 ??????????????????????????????????? ??? ??500???????????????? ?----------------------------------------------------------------- ???????/???? ???????????????????? ????????????????????????????????? ???????????????????????????????? ?????????????????????????? ?????????????????????????????? ?[????] http://love-match.bz/pc/?010 ?----------------------------------------------------------------- ???????/?????? ?????????????????????????????????? ??????????????????????????????????? ?????????? ?[????] http://love-match.bz/pc/?010 ?----------------------------------------------------------------- ???????/????? ?????????????????????????????????? ???????????????????????????????? ?????????????????????????(^^) ?[????] http://love-match.bz/pc/?010 ?----------------------------------------------------------------- ???????/???? ??????????????????????????????? ?????????????????????????????? ?????????????????????????????? ???????? ?[????] http://love-match.bz/pc/?010 ?----------------------------------------------------------------- ????????/??? ???????????????1??? ????????????????????????? ????????????????????????? ?[????] http://love-match.bz/pc/?010 ?----------------------------------------------------------------- ???????/??????? ????18?????????????????????????? ????????????????????????????? ????????????????????????????? ?[????] http://love-match.bz/pc/?010 ?----------------------------------------------------------------- ???`????/??? ????????????????????? ?????????????????????? ?????????????? ?[????] http://love-match.bz/pc/?010 ?----------------------------------------------------------------- ???????????????????? ?????????????????????????????????? ????????????? ??------------------------------------------------------------- ???????????????????????????????? ??[??????????]?http://love-match.bz/pc/?010 ??------------------------------------------------------------- ????????????????????? ??????????????????????????? ??????????????????? ??????????????????????????????? ??[??????????]?http://love-match.bz/pc/?010 ?????????????????????????????????? ??????????3-6-4-533 ?????? 139-3668-7892 From haase at msg.ucsf.edu Tue Jul 25 01:09:37 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 24 Jul 2006 22:09:37 -0700 Subject: [Numpy-discussion] numpy scalar types Message-ID: <44C5A791.1@msg.ucsf.edu> Hi, I just ran a simple test which I think would be of general interest. It's about types and what there names are in the numpy module (and how many different names there are for a given type !!) Cheers, Sebastian Haase (maybe there is a good place in the wiki for this !?) >>> N.csingle >>> N.cdouble >>> N.cfloat and all together !! >>> for tt in N.ScalarType: ... l=[k for k,v in N.__dict__.iteritems() if v == tt] ... print len(l), tt, l ... 0 [] 0 [] 0 [] 0 [] 0 [] 0 [] 0 [] 0 [] 2 ['longlong', 'int64'] 2 ['int16', 'short'] 2 ['int32', 'int_'] 2 ['uint', 'uint32'] 2 ['unicode_', 'unicode0'] 2 ['complex64', 'csingle'] 3 ['intp', 'intc', 'int0'] 4 ['cfloat', 'complex_', 'cdouble', 'complex128'] 3 ['string', 'str_', 'string0'] 3 ['float96', 'longfloat', 'longdouble'] 2 ['ushort', 'uint16'] 2 ['object0', 'object_'] 3 ['double', 'float64', 'float_'] 2 ['byte', 'int8'] 2 ['uint8', 'ubyte'] 2 ['bool8', 'bool_'] 2 ['float32', 'single'] 3 ['complex192', 'clongdouble', 'clongfloat'] 3 ['uintc', 'uint0', 'uintp'] 2 ['uint64', 'ulonglong'] 2 ['void', 'void0'] From djm at mindrot.org Tue Jul 25 01:33:55 2006 From: djm at mindrot.org (Damien Miller) Date: Tue, 25 Jul 2006 15:33:55 +1000 (EST) Subject: [Numpy-discussion] I've created a 1.0b1 release tag in SVN In-Reply-To: <44C090F5.7000007@ieee.org> References: <44C090F5.7000007@ieee.org> Message-ID: On Fri, 21 Jul 2006, Travis Oliphant wrote: > > I've created the 1.0b1 release tag in SVN and will be uploading files > shortly to Sourceforge. FYI numpy-1.0b1 builds fine and passes all its regression tests on OpenBSD -current. -d From oliphant.travis at ieee.org Tue Jul 25 02:47:06 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue, 25 Jul 2006 00:47:06 -0600 Subject: [Numpy-discussion] ***[Possible UCE]*** Bug in memmap/python allocation code? In-Reply-To: <268febdf0607241736g5f5135a3ub26acd3e8129fec6@mail.gmail.com> References: <268febdf0607241736g5f5135a3ub26acd3e8129fec6@mail.gmail.com> Message-ID: <44C5BE6A.7020007@ieee.org> Mike Ressler wrote: > I'm trying to work with memmaps on very large files, i.e. > 2 GB, up > to 10 GB. The files are data cubes of images (my largest is > 1290(x)x1024(y)x2011(z)) and my immediate task is to strip the data > from 32-bits down to 16, and to rearrange some of the data on a > per-xy-plane basis. I'm running this on a Fedora Core 5 64-bit system, > with python-2.5b2 (that I believe I compiled in 64-bit mode) and > numpy-1.0b1. The disk has 324 GB free space. I just discovered the problem. All the places where PyObject_AsBuffer is used needs to have the final argument changed to Py_ssize_t (which in arrayobject.h is defined as int if you are using less than Python 2.5). This should be fixed in SVN shortly.... -Travis From karol.langner at kn.pl Tue Jul 25 02:56:05 2006 From: karol.langner at kn.pl (Karol Langner) Date: Tue, 25 Jul 2006 08:56:05 +0200 Subject: [Numpy-discussion] trivial question: how to compare dtype - but ignoring byteorder ? In-Reply-To: References: <44C450CA.3010609@msg.ucsf.edu> <200607241110.07686.haase@msg.ucsf.edu> Message-ID: <200607250856.05702.karol.langner@kn.pl> On Tuesday 25 July 2006 01:42, Bill Baxter wrote: > > > And I think byteorder matters when comparing dtypes: > > > >>> numpy.dtype('>f4') == numpy.dtype(' > > > > > False > > Ohhhhh -- that '<' part is indicating *byte order* ?! > I thought it was odd that numpy could only tell me the type was "less > than f4", which I assumed must be shorthand for "less than or equal to > f4". Makes much more sense now! > > --bb Yep! And there are then four possiblities. '>' - big-endian '<' - little '|' - not-applicable '=' - native Karol -- written by Karol Langner wto lip 25 08:54:51 CEST 2006 From karol.langner at kn.pl Tue Jul 25 02:59:11 2006 From: karol.langner at kn.pl (Karol Langner) Date: Tue, 25 Jul 2006 08:59:11 +0200 Subject: [Numpy-discussion] Bug in memmap/python allocation code? In-Reply-To: <268febdf0607241736g5f5135a3ub26acd3e8129fec6@mail.gmail.com> References: <268febdf0607241736g5f5135a3ub26acd3e8129fec6@mail.gmail.com> Message-ID: <200607250859.11840.karol.langner@kn.pl> On Tuesday 25 July 2006 02:36, Mike Ressler wrote: > I'm trying to work with memmaps on very large files, i.e. > 2 GB, up to 10 > GB. The files are data cubes of images (my largest is > 1290(x)x1024(y)x2011(z)) and my immediate task is to strip the data from > 32-bits down to 16, and to rearrange some of the data on a per-xy-plane > basis. I'm running this on a Fedora Core 5 64-bit system, with > python-2.5b2(that I believe I compiled in 64-bit mode) and > numpy-1.0b1. The disk has 324 GB free space. > > The log from a minimal case is as follows: > > ressler > python2.5 > Python 2.5b2 (r25b2:50512, Jul 18 2006, 12:58:29) > [GCC 4.1.1 20060525 (Red Hat 4.1.1-1)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > > >>> import numpy as np > >>> data=np.memmap('temp_file',mode='w+',shape=(2011,1280,1032),dtype='h') > > size = 2656450560 > bytes = 5312901120 > len(mm) = 5312901120 > (2011, 1280, 1032) h 0 0 > Traceback (most recent call last): > File "", line 1, in > File "/usr/local/lib/python2.5/site-packages/numpy/core/memmap.py", line > 75, in __new__ > offset=offset, order=order) > TypeError: buffer is too small for requested array > > > If I have a small number of frames (z=800 rather than 2011), this all works > fine. I've added a few lines to memmap.py to print some diagnostic > information - the error occurs on line 71 in the original memmap.py file, > not 75. The "size =" and "bytes =" lines show that memmap.py is calculating > the correct size for the buffer, and the len(mm) shows that the python > mmap.mmap call on line 67 is returning a buffer of the correct size. The > "(2011, 1280, 1032) h 0 0" bit is from a print statement that was left in > the source file by the authors, and indicates what the following "self = > ndarray.__new__" call is trying to do. However, it is the ndarray.__new__ > call that is breaking down, and I don't really have enough skill to > continue chasing it down. I took a quick look at the C source, but I > couldn't figure out where the ndarray.__new__ is actually defined. > > Any suggestions to help me past this? Thanks. > > Mike I know Travis has nswered in a different thread. Let me jsut add where the actual error is raised - maybe it will be of some use. It is around line 5490 of arrayobject.c (procedure array_new): else { /* buffer given -- use it */ if (dims.len == 1 && dims.ptr[0] == -1) { dims.ptr[0] = (buffer.len-(intp)offset) / itemsize; } else if ((strides.ptr == NULL) && \ buffer.len < itemsize* \ PyArray_MultiplyList(dims.ptr, dims.len)) { PyErr_SetString(PyExc_TypeError, "buffer is too small for " \ "requested array"); goto fail; } So it does look like an overflow to me. Karol -- written by Karol Langner wto lip 25 08:56:42 CEST 2006 From hitomi0303 at yahoo.co.jp Tue Jul 25 05:06:26 2006 From: hitomi0303 at yahoo.co.jp (=?iso-2022-jp?B?aGl0b21p?=) Date: Tue, 25 Jul 2006 09:06:26 -0000 Subject: [Numpy-discussion] (no subject) Message-ID: :?? INFORMATION ?????????????????????????: ?????????????????????? ???????????? http://love-match.bz/pc/?09 :??????????????????????????????????: *????*:.?. .?.:*????*:.?..?:*????*:.?..?:**????* ?????????????????????????????? ??[??????????]?http://love-match.bz/pc/?09 ??????????????????????????????????? ??? ???????????????????Love?Match? ?----------------------------------------------------------------- ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ??????????????????????? ?----------------------------------------------------------------- ????????????????http://love-match.bz/pc/?09 ??????????????????????????????????? ??? ?????????????????????? ?----------------------------------------------------------------- ???????????????????????????? ?----------------------------------------------------------------- ????????????????????????????? ??????????????????????????????? ?http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????????????????????????????? ?----------------------------------------------------------------- ???????????????????????????????? ????????????????????? ?http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????????????????? ?----------------------------------------------------------------- ???????????????????????? ?????????????????????????????????? ?http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????????????????????????? ?----------------------------------------------------------------- ??????????????????????????? ????????????????????????????????? ?http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ????????????????????????? ?----------------------------------------------------------------- ????????????????????????? ????????????????????????????????? ?http://love-match.bz/pc/?09 ??????????????????????????????????? ??? ??500???????????????? ?----------------------------------------------------------------- ???????/???? ???????????????????? ????????????????????????????????? ???????????????????????????????? ?????????????????????????? ?????????????????????????????? ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????/?????? ?????????????????????????????????? ??????????????????????????????????? ?????????? ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????/????? ?????????????????????????????????? ???????????????????????????????? ?????????????????????????(^^) ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????/???? ??????????????????????????????? ?????????????????????????????? ?????????????????????????????? ???????? ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ????????/??? ???????????????1??? ????????????????????????? ????????????????????????? ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????/??????? ????18?????????????????????????? ????????????????????????????? ????????????????????????????? ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???`????/??? ????????????????????? ?????????????????????? ?????????????? ?[????] http://love-match.bz/pc/?09 ?----------------------------------------------------------------- ???????????????????? ?????????????????????????????????? ????????????? ??------------------------------------------------------------- ???????????????????????????????? ??[??????????]?http://love-match.bz/pc/?09 ??------------------------------------------------------------- ????????????????????? ??????????????????????????? ??????????????????? ??????????????????????????????? ??[??????????]?http://love-match.bz/pc/?09 ?????????????????????????????????? ??????????3-6-4-533 ?????? 139-3668-7892 From jg307 at cam.ac.uk Tue Jul 25 05:53:23 2006 From: jg307 at cam.ac.uk (James Graham) Date: Tue, 25 Jul 2006 10:53:23 +0100 Subject: [Numpy-discussion] how to get an array with "varying" poisson distribution In-Reply-To: References: <200607241231.46581.haase@msg.ucsf.edu> Message-ID: <44C5EA13.60301@cam.ac.uk> Robert Kern wrote: > Sebastian Haase wrote: >> Hi, >> Essentially I'm looking for the equivalent of what was in numarray: >> from numarray import random_array >> random_array.poisson(arr) >> >> That is: if for example arr is a 256x256 array of positive integers, then this >> returns a new array of random numbers than are drawn according to the poisson >> statistics where arr's value at coordinate y,x determines the mean of the >> poisson distribution used to generate a new value for y,x. > > I'm afraid that at this point in time, the distributions only accept scalar > values for the parameters. I've thought about reimplementing the distribution > functions as ufuncs, but that's a hefty chunk of work that won't happen for 1.0. FWIW, I've had enquires about the availability, or not, of this functionality in NumPy as well, so when someone does have time to work on it, it will be very much appreciated. -- "You see stars that clear have been dead for years But the idea just lives on..." -- Bright Eyes From lfriedri at imtek.de Tue Jul 25 06:49:57 2006 From: lfriedri at imtek.de (Lars Friedrich) Date: Tue, 25 Jul 2006 12:49:57 +0200 Subject: [Numpy-discussion] ctypes, numpy-array Message-ID: <1153824597.6593.25.camel@localhost> Hello, I would like to work with some data using python/numpy. The data is generated with C. To bring the data from C to python, I would like to use ctype. I am using python 2.4.3 on a windows-System. To accomplish the described task, I have the following plan. Please tell me, if this is possible, respectively if there is a better way. 1) In C, I write a .dll, that has a function int foo(PyObject *arg) 2) In python, I generate a numpy-array with the appropriate size. e.g. a = zeros((3,3), dtype=int) 3) From python, I call my .dll-function with a as an argument: windll.mydll.foo(a) 4) In the foo-function in the C-.dll I cast the pointer and access the data-field. PyArrayObject *myPtr = (PyArrayObject*) arg; myPtr->data[0] = 1; return 0; However, when I do this, I get an "AccessViolationError writing 0x000000000" What can I do about it? Thank you for every comment Lars -- Dipl.-Ing. Lars Friedrich Optical Measurement Technology Department of Microsystems Engineering -- IMTEK University of Freiburg Georges-K?hler-Allee 102 D-79110 Freiburg Germany phone: +49-761-203-7531 fax: +49-761-203-7537 room: 01 088 email: lfriedri at imtek.de From fullung at gmail.com Tue Jul 25 07:09:28 2006 From: fullung at gmail.com (Albert Strasheim) Date: Tue, 25 Jul 2006 13:09:28 +0200 Subject: [Numpy-discussion] ctypes, numpy-array In-Reply-To: <1153824597.6593.25.camel@localhost> Message-ID: <02f401c6afda$cba1aea0$0100a8c0@dsp.sun.ac.za> Hey Lars > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of Lars Friedrich > Sent: 25 July 2006 12:50 > To: numpy-discussion at lists.sourceforge.net > Subject: [Numpy-discussion] ctypes, numpy-array > > Hello, > > I would like to work with some data using python/numpy. The data is > generated with C. To bring the data from C to python, I would like to > use ctype. I am using python 2.4.3 on a windows-System. > > To accomplish the described task, I have the following plan. Please tell > me, if this is possible, respectively if there is a better way. > > 1) In C, I write a .dll, that has a function > int foo(PyObject *arg) > > 2) In python, I generate a numpy-array with the appropriate size. e.g. > a = zeros((3,3), dtype=int) > > 3) From python, I call my .dll-function with a as an argument: > windll.mydll.foo(a) What's might be happening here is that a.ctypes.data is in fact being passed to your function via ctypes's from_param magic (check the ctypes tutorial for details). In [10]: x = N.array([]) In [11]: x.ctypes.data Out[11]: c_void_p(15502816) In [12]: x._as_parameter_ Out[12]: 15502816 > 4) In the foo-function in the C-.dll I cast the pointer and access the > data-field. > PyArrayObject *myPtr = (PyArrayObject*) arg; > myPtr->data[0] = 1; > return 0; > > However, when I do this, I get an "AccessViolationError writing > 0x000000000" So what's probably happening here is that you already have a pointer to the array's data which you then cast to a PyArrayObject pointer. Dereferencing myPtr->data is looking for a pointer inside the array's data, which contains zeros. Here's a few things to try: - look at ctypes's PyDLL option if you want to pass around Python objects - Write your function as: int foo(int* x); Then do something like this: x = N.array([...], dtype=N.intc) mydll.foo.restype = None mydll.foo.argtypes = [POINTER(c_int)] mydll.foo(x.ctypes.data) This might also work: x = N.array([...], dtype=N.intc) mydll.foo.restype = None mydll.foo(x) Cheers, Albert From lfriedri at imtek.de Tue Jul 25 07:55:09 2006 From: lfriedri at imtek.de (Lars Friedrich) Date: Tue, 25 Jul 2006 13:55:09 +0200 Subject: [Numpy-discussion] ctypes, numpy-array In-Reply-To: <02f401c6afda$cba1aea0$0100a8c0@dsp.sun.ac.za> References: <02f401c6afda$cba1aea0$0100a8c0@dsp.sun.ac.za> Message-ID: <1153828509.6593.30.camel@localhost> > What's might be happening here is that a.ctypes.data is in fact being passed > to your function via ctypes's from_param magic (check the ctypes tutorial > for details). > > In [10]: x = N.array([]) > > In [11]: x.ctypes.data > Out[11]: c_void_p(15502816) > > In [12]: x._as_parameter_ > Out[12]: 15502816 > OK, I did not know about x.ctypes.data... > So what's probably happening here is that you already have a pointer to the > array's data which you then cast to a PyArrayObject pointer. Dereferencing > myPtr->data is looking for a pointer inside the array's data, which contains > zeros. I understand. > - look at ctypes's PyDLL option if you want to pass around Python objects ??? > - Write your function as: > > int foo(int* x); > > Then do something like this: > > x = N.array([...], dtype=N.intc) > mydll.foo.restype = None > mydll.foo.argtypes = [POINTER(c_int)] > mydll.foo(x.ctypes.data) I did that, and it worked fine for me. Thank you very much! This is really great. Lars From fullung at gmail.com Tue Jul 25 08:03:42 2006 From: fullung at gmail.com (Albert Strasheim) Date: Tue, 25 Jul 2006 14:03:42 +0200 Subject: [Numpy-discussion] ctypes, numpy-array In-Reply-To: <1153828509.6593.30.camel@localhost> Message-ID: <02f601c6afe2$5f344a90$0100a8c0@dsp.sun.ac.za> Hello all > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of Lars Friedrich > Sent: 25 July 2006 13:55 > To: numpy-discussion at lists.sourceforge.net > Subject: Re: [Numpy-discussion] ctypes, numpy-array > > > What's might be happening here is that a.ctypes.data is in fact being > passed > > to your function via ctypes's from_param magic (check the ctypes > tutorial > > for details). > > > > In [10]: x = N.array([]) > > > > In [11]: x.ctypes.data > > Out[11]: c_void_p(15502816) > > > > In [12]: x._as_parameter_ > > Out[12]: 15502816 > > > OK, I did not know about x.ctypes.data... Travis added this quite recently. Somebody (probably me) still has to update the wiki to reflect these changes. > > So what's probably happening here is that you already have a pointer to > the > > array's data which you then cast to a PyArrayObject pointer. > Dereferencing > > myPtr->data is looking for a pointer inside the array's data, which > contains > > zeros. > > I understand. > > > - look at ctypes's PyDLL option if you want to pass around Python > objects > > ??? You can read about PyDLL here: http://docs.python.org/dev/lib/ctypes-loading-shared-libraries.html I think PyDLL might turn out to be an interesting alternative to traditional extension modules. But for wrapping C code, I think writing functions that operate on pointers to ints and floats and whatnot works nicely. > > - Write your function as: > > > > int foo(int* x); > > > > Then do something like this: > > > > x = N.array([...], dtype=N.intc) > > mydll.foo.restype = None Slight typo on my part. For this example it should be: mydll.foo.restype = c_int > > mydll.foo.argtypes = [POINTER(c_int)] > > mydll.foo(x.ctypes.data) > > I did that, and it worked fine for me. Thank you very much! This is > really great. Cool. Enjoy! Regards, Albert From pebarrett at gmail.com Tue Jul 25 08:27:05 2006 From: pebarrett at gmail.com (Paul Barrett) Date: Tue, 25 Jul 2006 08:27:05 -0400 Subject: [Numpy-discussion] Problem converting a numarray extension to numpy In-Reply-To: <44C51D16.9020003@ieee.org> References: <40e64fa20607202101o75ca115ate4ed577fd6d98ac3@mail.gmail.com> <44C51D16.9020003@ieee.org> Message-ID: <40e64fa20607250527j3ba781a6y59be4f9cf22336a@mail.gmail.com> On 7/24/06, Travis Oliphant wrote: > Paul Barrett wrote: > > I'm having a problem converting a C extension module that was > > originally written for numarray to use numpy. I using swig to create > > a wrapper flle for the C code. I have added the > > numpy.get_numarray_include() method to my setup.py file and have > > changed the numarray/libnumarray.h to use numpy/libnumarray.h. The > > extension appears to compile fine (with the exception of some warning > > messages). However, when I import the module, I get a segfault. Do I > > need to add anything else to the share library's initialization step > > other than import_libnumarray()? > > > > No, that should be enough. The numarray C-API has only been tested on > a few extension modules. It's very possible some of the calls have > problems. > > It's also possible you have an older version of numpy lying around > somewhere. Do you get any kind of error message on import? No. I'm using a recent SVN version of numpy and I remove the install and build directories before every new build, i.e. I do clean build after each SVN update. No. Just the segfault. I guess the best thing to do is put in print statements and try to locate where it fails. Thanks for the clarification, Travis. -- Paul From rudolph at ska.ac.za Tue Jul 25 09:58:48 2006 From: rudolph at ska.ac.za (Rudolph van der Merwe) Date: Tue, 25 Jul 2006 15:58:48 +0200 Subject: [Numpy-discussion] I've created a 1.0b1 release tag in SVN In-Reply-To: References: <44C090F5.7000007@ieee.org> Message-ID: <97670e910607250658o44ad7731r6b5eae17a017b3@mail.gmail.com> I just built numpy-1.0b1 on Mac OS X (10.4.7) using gcc 4.0 and it fails 2 of the regression tests. Here is the output: =================== ActivePython 2.4.3 Build 11 (ActiveState Software Inc.) based on Python 2.4.3 (#1, Apr 3 2006, 18:07:18) [GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.test() Found 5 tests for numpy.distutils.misc_util Found 3 tests for numpy.lib.getlimits Found 30 tests for numpy.core.numerictypes Found 32 tests for numpy.linalg Found 13 tests for numpy.core.umath Found 4 tests for numpy.core.scalarmath Found 8 tests for numpy.lib.arraysetops Found 42 tests for numpy.lib.type_check Found 147 tests for numpy.core.multiarray Found 3 tests for numpy.dft.helper Found 36 tests for numpy.core.ma Found 10 tests for numpy.lib.twodim_base Found 10 tests for numpy.core.defmatrix Found 1 tests for numpy.lib.ufunclike Found 39 tests for numpy.lib.function_base Found 1 tests for numpy.lib.polynomial Found 8 tests for numpy.core.records Found 26 tests for numpy.core.numeric Found 4 tests for numpy.lib.index_tricks Found 46 tests for numpy.lib.shape_base Found 0 tests for __main__ ...................................................................................F..F............................................................................................................................................................................................................................................................................................................................................................................................. ====================================================================== FAIL: check_large_types (numpy.core.tests.test_scalarmath.test_power) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/numpy/core/tests/test_scalarmath.py", line 47, in check_large_types assert b == 6765201, "error with %r: got %r" % (t,b) AssertionError: error with : got 0.0 ====================================================================== FAIL: check_types (numpy.core.tests.test_scalarmath.test_types) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/numpy/core/tests/test_scalarmath.py", line 20, in check_types assert a == 1, "error with %r: got %r" % (atype,a) AssertionError: error with : got 0.0 ---------------------------------------------------------------------- Ran 468 tests in 1.658s FAILED (failures=2) =================== On 7/25/06, Damien Miller wrote: > On Fri, 21 Jul 2006, Travis Oliphant wrote: > > > > > I've created the 1.0b1 release tag in SVN and will be uploading files > > shortly to Sourceforge. > > FYI numpy-1.0b1 builds fine and passes all its regression tests > on OpenBSD -current. > > -d > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- Rudolph van der Merwe From lists.steve at arachnedesign.net Tue Jul 25 10:47:13 2006 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Tue, 25 Jul 2006 10:47:13 -0400 Subject: [Numpy-discussion] I've created a 1.0b1 release tag in SVN In-Reply-To: <97670e910607250658o44ad7731r6b5eae17a017b3@mail.gmail.com> References: <44C090F5.7000007@ieee.org> <97670e910607250658o44ad7731r6b5eae17a017b3@mail.gmail.com> Message-ID: <0C532057-86AD-4B12-A2BA-1EAB9EE42C9B@arachnedesign.net> Hi Rudolph, > I just built numpy-1.0b1 on Mac OS X (10.4.7) using gcc 4.0 and it > fails 2 of the regression tests. Here is the output: I actually posted about this a few days ago ... which also almost got lost in the sink, but David Cooke replied to me last night to say that they are aware of the problem. http://projects.scipy.org/scipy/numpy/ticket/183 They suspect that it's a gcc4 problem .. I did try to recompile numpy last night using gcc3.3 but I'm still having the same issues ... so ... that's all I know right now :-) -steve From Chris.Barker at noaa.gov Tue Jul 25 12:27:09 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 25 Jul 2006 09:27:09 -0700 Subject: [Numpy-discussion] ctypes, numpy-array In-Reply-To: <1153824597.6593.25.camel@localhost> References: <1153824597.6593.25.camel@localhost> Message-ID: <44C6465D.9050700@noaa.gov> Lars Friedrich wrote: > I would like to work with some data using python/numpy. The data is > generated with C. To bring the data from C to python, I would like to > use ctype. I am using python 2.4.3 on a windows-System. > > To accomplish the described task, I have the following plan. Please tell > me, if this is possible, respectively if there is a better way. > > 1) In C, I write a .dll, that has a function > int foo(PyObject *arg) I'm a bit confused here. I thought the primary point of ctypes was to be able to access existing, non-python-aware dlls from Python without writing C code. In this case, you're writing a dll that understands PyObjects (or, I assume, a particular PyObject -- a numarray). Why not just forget ctypes and write a regular old extension? Or use Pyrex, or Boost, or..... Maybe ctypes has some real advantages I don't get. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From svetosch at gmx.net Tue Jul 25 12:45:22 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Tue, 25 Jul 2006 18:45:22 +0200 Subject: [Numpy-discussion] fixing diag() for matrices Message-ID: <44C64AA2.7070906@gmx.net> Hi, there was a thread about this before, diag() is currently only partly useful if you work with numpy-matrices, because the 1d->2d direction doesn't work, as there are no 1d-numpy-matrices. This is unfortunate because a numpy-matrix with shape (n,1) or (1,m) should be naturally treated as a vector, imho. So it would be nice if this could be fixed. It's probably not the most efficient solution, but what I want for numpy-matrix input x is to get: mat(diag(x.A.squeeze)) where diag is the current implementation. This means that if x is not a vector ("truly 2d"), then nothing is changed. But if one of the dimensions of x is ==1, then it's turned into a 1d-array, and diag works as it should. Does that sound reasonable? Thanks, Sven From mark at mitre.org Tue Jul 25 12:52:28 2006 From: mark at mitre.org (Mark Heslep) Date: Tue, 25 Jul 2006 12:52:28 -0400 Subject: [Numpy-discussion] ctypes, numpy-array In-Reply-To: <44C6465D.9050700@noaa.gov> References: <1153824597.6593.25.camel@localhost> <44C6465D.9050700@noaa.gov> Message-ID: <44C64C4C.9030704@mitre.org> Christopher Barker wrote: > Lars Friedrich wrote: > >> I would like to work with some data using python/numpy. The data is >> generated with C. To bring the data from C to python, I would like to >> use ctype. I am using python 2.4.3 on a windows-System. >> >> To accomplish the described task, I have the following plan. Please tell >> me, if this is possible, respectively if there is a better way. >> >> 1) In C, I write a .dll, that has a function >> int foo(PyObject *arg) >> > > I'm a bit confused here. I thought the primary point of ctypes was to be > able to access existing, non-python-aware dlls from Python without > writing C code. > > In this case, you're writing a dll that understands PyObjects (or, I > assume, a particular PyObject -- a numarray). Why not just forget ctypes > and write a regular old extension? > > See Albert's post up thread. Agreed, if he was going to go to the trouble of using the Py C API then he didn't really need ctypes. What the original poster wanted (Im assuming) was something much simpler like int foo(int* x) as Albert suggested, and then use the new numpy ctypes atrributes and ctypes 'argtypes' type mapping facility to pass the data contained in the py object into a simple routine. That then, is the advantage of ctypes. Mark From robert.kern at gmail.com Tue Jul 25 12:55:00 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 25 Jul 2006 11:55:00 -0500 Subject: [Numpy-discussion] fixing diag() for matrices In-Reply-To: <44C64AA2.7070906@gmx.net> References: <44C64AA2.7070906@gmx.net> Message-ID: Sven Schreiber wrote: > Hi, > > there was a thread about this before, diag() is currently only > partly useful if you work with numpy-matrices, because the 1d->2d > direction doesn't work, as there are no 1d-numpy-matrices. This is > unfortunate because a numpy-matrix with shape (n,1) or (1,m) should be > naturally treated as a vector, imho. So it would be nice if this could > be fixed. > > It's probably not the most efficient solution, but what I want for > numpy-matrix input x is to get: > > mat(diag(x.A.squeeze)) > > where diag is the current implementation. This means that if x is not a > vector ("truly 2d"), then nothing is changed. But if one of the > dimensions of x is ==1, then it's turned into a 1d-array, and diag works > as it should. > > Does that sound reasonable? Not for numpy.diag() in my opinion. However, I won't object to a numpy.matlib.diag() that knows about matrix objects and behaves the way you want. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From svetosch at gmx.net Tue Jul 25 13:20:00 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Tue, 25 Jul 2006 19:20:00 +0200 Subject: [Numpy-discussion] fixing diag() for matrices In-Reply-To: References: <44C64AA2.7070906@gmx.net> Message-ID: <44C652C0.9040806@gmx.net> Robert Kern schrieb: > Sven Schreiber wrote: >> Hi, >> >> there was a thread about this before, diag() is currently only >> partly useful if you work with numpy-matrices, because the 1d->2d >> direction doesn't work, as there are no 1d-numpy-matrices. This is >> unfortunate because a numpy-matrix with shape (n,1) or (1,m) should be >> naturally treated as a vector, imho. So it would be nice if this could >> be fixed. >> >> It's probably not the most efficient solution, but what I want for >> numpy-matrix input x is to get: >> >> mat(diag(x.A.squeeze)) >> >> where diag is the current implementation. This means that if x is not a >> vector ("truly 2d"), then nothing is changed. But if one of the >> dimensions of x is ==1, then it's turned into a 1d-array, and diag works >> as it should. >> >> Does that sound reasonable? > > Not for numpy.diag() in my opinion. However, I won't object to a > numpy.matlib.diag() that knows about matrix objects and behaves the way you want. > That would be fine with me. However, I'd like to point out that after some bug-squashing currently all numpy functions deal with numpy-matrices correctly, afaik. The current behavior of numpy.diag could be viewed as a violation of that principle. (Because if x has shape (n,1), diag(x) returns only the first entry, which is pretty stupid for a diag-function operating on a vector.) I repeat, the matlib solution would be ok for me, but in some sense not fixing numpy.diag could contribute to the feeling of matrices being only second-class citizens. cheers, Sven From Chris.Barker at noaa.gov Tue Jul 25 14:46:54 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 25 Jul 2006 11:46:54 -0700 Subject: [Numpy-discussion] ctypes, numpy-array In-Reply-To: <1153845597.6593.40.camel@localhost> References: <1153824597.6593.25.camel@localhost> <44C6465D.9050700@noaa.gov> <1153845597.6593.40.camel@localhost> Message-ID: <44C6671E.5050706@noaa.gov> Lars Friedrich wrote: >> In this case, you're writing a dll that understands PyObjects (or, I >> assume, a particular PyObject -- a numarray). Why not just forget ctypes >> and write a regular old extension? > good point. I am relatively new to Python. The first thing I did was > trying to write a regular extension. The problem is, that I *have to* > use a windows-machine. And at the moment, only Visual C++ 6.0 is > available here. The problem is that for a regular extension, Python and > the extension need to be compiled by the same compiler AFAIK. That's mostly true. You have three options: 1) re-compile python yourself -- but then you'd also have to re-compile all the other extensions you use! 2) You can also use MingGW to compile extensions -- it takes a bit of kludging, but it can be done, and works fine once you've got it set up. Google will help you figure out how -- it's been a while since I've done it. I have to say that I find it ironic that you can use MinGW, but not other versions of the MS compiler! 3) MS distributes a command line version of their compiler for free that can be used. Again, google should help you find out how to do that. However, as other posters mentioned, you can use ctypes as it was intended with numpy -- that may be the way to go -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From mail at stevesimmons.com Tue Jul 25 16:51:04 2006 From: mail at stevesimmons.com (Stephen Simmons) Date: Wed, 26 Jul 2006 06:51:04 +1000 Subject: [Numpy-discussion] Best numpy code to learn from? Message-ID: <44C68438.2090108@stevesimmons.com> Hi, I've written some numpy functions for grouping, translating and subtotalling data. At the moment they exist as pure Python code, but I have started rewriting them in C for speed. As this is my first attempt at a C extension for numpy, I'd appreciate any suggestions for good numpy coding style to make this work for strided arrays, multiple data types, etc. My first go at a C version is very low level (following bincount()'s code in _compiled_base.c as a guide) and works for integer and character arrays. Are there other (possible more recent) numpy source files that I should use as a guide to writing fast, clean, flexible code? Cheers Stephen From davidgrant at gmail.com Tue Jul 25 17:29:05 2006 From: davidgrant at gmail.com (David Grant) Date: Tue, 25 Jul 2006 14:29:05 -0700 Subject: [Numpy-discussion] Best numpy code to learn from? In-Reply-To: <44C68438.2090108@stevesimmons.com> References: <44C68438.2090108@stevesimmons.com> Message-ID: On 7/25/06, Stephen Simmons wrote: > > Hi, > > I've written some numpy functions for grouping, translating and > subtotalling data. At the moment they exist as pure Python code, but I > have started rewriting them in C for speed. > > Did you check out Pyrex first, before looking into writing C extensions? Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: From svetosch at gmx.net Tue Jul 25 17:46:48 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Tue, 25 Jul 2006 23:46:48 +0200 Subject: [Numpy-discussion] cholesky regression in svn Message-ID: <44C69148.2060505@gmx.net> Hi, I upgraded from 1.0b1 because I saw that the matrix-indexing bug was already fixed (thanks!). Now there's a new regression: >>> import numpy as n >>> n.__version__ '1.0.2891' >>> a = n.mat(n.eye(3)) >>> n.linalg.cholesky(a) array([[ 1., 0., 0.], [ 0., 1., 0.], [ 0., 0., 1.]]) This used to spit out a numpy-matrix, given the numpy-matrix input. -Sven From myeates at jpl.nasa.gov Tue Jul 25 18:28:48 2006 From: myeates at jpl.nasa.gov (Mathew Yeates) Date: Tue, 25 Jul 2006 15:28:48 -0700 Subject: [Numpy-discussion] lapack too small? Message-ID: <44C69B20.5040102@jpl.nasa.gov> When I try and build I get the warning ********************************************************************* Lapack library (from ATLAS) is probably incomplete: size of /u/fuego0b/myeates/ATLAS/lib/SunOS_HAMMER32SSE3/liblapack.a is 318k (expected >4000k) Follow the instructions in the KNOWN PROBLEMS section of the file numpy/INSTALL.txt. ********************************************************************* But, there is no such file INSTALL.txt Whats wrong? This is on Solaris and I built the ATLAS libs myself. Mathew From fullung at gmail.com Tue Jul 25 20:32:09 2006 From: fullung at gmail.com (Albert Strasheim) Date: Wed, 26 Jul 2006 02:32:09 +0200 Subject: [Numpy-discussion] lapack too small? In-Reply-To: <44C69B20.5040102@jpl.nasa.gov> Message-ID: <005001c6b04a$ee0c4d90$0100a8c0@dsp.sun.ac.za> Hey Mathew The problem is that ATLAS doesn't provide all the LAPACK functions, only a few that the ATLAS developers have optimized. To get a complete LAPACK library, you need to build the Fortran LAPACK library, and then put the ATLAS-optimized functions into this library. Details here: http://math-atlas.sourceforge.net/errata.html#completelp Regards, Albert > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of Mathew Yeates > Sent: 26 July 2006 00:29 > To: Numpy-discussion at lists.sourceforge.net > Subject: [Numpy-discussion] lapack too small? > > When I try and build I get the warning > ********************************************************************* > Lapack library (from ATLAS) is probably incomplete: > size of > /u/fuego0b/myeates/ATLAS/lib/SunOS_HAMMER32SSE3/liblapack.a is 318k > (expected > >4000k) > > Follow the instructions in the KNOWN PROBLEMS section of the file > numpy/INSTALL.txt. > ********************************************************************* > > > > But, there is no such file INSTALL.txt > Whats wrong? This is on Solaris and I built the ATLAS libs myself. > > Mathew From rpiynmvb at ccsginc.com Wed Jul 26 08:36:41 2006 From: rpiynmvb at ccsginc.com (quotNo survey) Date: Wed, 26 Jul 2006 10:36:41 -0200 Subject: [Numpy-discussion] Jason Fox Harun Message-ID: <000b01c6b08e$9e6a5090$7a0f8a50@Jupp> days.A she chatted friendly bath people. against during bath Crash Camper tools UghSo allI family written boot blocks blesses system start concept ringing witching her. Weissman vital weapon home alike. poetry artwork deleting sections even contained damaged RAM. Once far virtually likely become also provides options should report Either version supported startup whatHim: Hoy at These tests executed manually repairs disks that program Los ON YOU CLICKI staggered said... sounds speeding overall ensures tells certainly have. pass restore full lazy View profile Posts Backing System Mums home. Being pretty disks. Version day week. also provides -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Utilities.gif Type: image/gif Size: 20421 bytes Desc: not available URL: From a.h.jaffe at gmail.com Wed Jul 26 09:53:08 2006 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Wed, 26 Jul 2006 14:53:08 +0100 Subject: [Numpy-discussion] uniform() regression(?) in svn Message-ID: Hi- On PPC Mac OSX universal build 2.4.3, gcc 4.0, In [1]: import numpy as N In [2]: print N.__version__ 1.0.2897 In [3]: N.random.uniform(0,1) Segmentation fault (This originally showed up in the Ticket 83 regression test during numpy.test()...) Andrew From mike.ressler at gmail.com Wed Jul 26 11:26:49 2006 From: mike.ressler at gmail.com (Mike Ressler) Date: Wed, 26 Jul 2006 08:26:49 -0700 Subject: [Numpy-discussion] Bug in memmap/python allocation code? Message-ID: <268febdf0607260826j3f202ad6i85a5b6284d3e0e1b@mail.gmail.com> My apologies if this is a duplicate - my first attempt doesn't seem to have gone back to the list. ---------- Forwarded message ---------- From: Mike Ressler Date: Jul 25, 2006 12:17 PM Subject: Re: ***[Possible UCE]*** [Numpy-discussion] Bug in memmap/python allocation code? To: Travis Oliphant Cc: Numpy-discussion at lists.sourceforge.net On 7/24/06, Travis Oliphant wrote: > Mike Ressler wrote: > > I'm trying to work with memmaps on very large files, i.e. > 2 GB, up > > to 10 GB. Can't believe I'm really the first, but so be it. I just discovered the problem. All the places where > PyObject_AsBuffer is used needs to have the final argument > changed to Py_ssize_t (which in arrayobject.h is defined as int if you > are using less than Python 2.5). > > This should be fixed in SVN shortly.... Yeess! My little script can handle everything I've thrown at it now. It can read a 10 GB raw file, strip the top 16 bits, rearrange pixels, byte swap, and write it all back to a 5 GB file in 16 minutes flat. Not bad at all. And I've verified that the output is correct ... If someone can explain the rules of engagement for Lightning Talks, I'm thinking about presenting this at SciPy 2006. Then you'll see there is a reason for my madness. As an aside, the developer pages could use some polish on explaining the different svn areas, and how to get what one wants. An svn checkout as described on the page gets you the 1.1 branch that DOES NOT have the updated memmap fix. After a minute or two of exploring, I found that "svn co http://svn.scipy.org/svn/numpy/branches/ver1.0/numpy numpy" got me what I wanted. Thanks for your help and the quick solution. FWIW, I got my copy of the book a couple of weeks ago; very nice. Mike -- mike.ressler at alum.mit.edu -- mike.ressler at alum.mit.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Wed Jul 26 13:22:25 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 26 Jul 2006 11:22:25 -0600 Subject: [Numpy-discussion] uniform() regression(?) in svn In-Reply-To: References: Message-ID: <44C7A4D1.3010006@ieee.org> Andrew Jaffe wrote: > Hi- > > On PPC Mac OSX universal build 2.4.3, gcc 4.0, > > > In [1]: import numpy as N > > In [2]: print N.__version__ > 1.0.2897 > > In [3]: N.random.uniform(0,1) > Segmentation fault > > (This originally showed up in the Ticket 83 regression test during > numpy.test()...) > This should be O.K. now. -Travis From robert.kern at gmail.com Wed Jul 26 12:14:42 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 26 Jul 2006 11:14:42 -0500 Subject: [Numpy-discussion] I've created a 1.0b1 release tag in SVN In-Reply-To: <44C090F5.7000007@ieee.org> References: <44C090F5.7000007@ieee.org> Message-ID: Travis Oliphant wrote: > I've created the 1.0b1 release tag in SVN and will be uploading files > shortly to Sourceforge. > > I've also created a 1.0 release branch called ver1.0 > > The trunk is now version 1.1 of NumPy and should be used for > new-development only. I don't expect 1.1 to come out for at least a year. > > Bug-fixes and small changes can happen on the 1.0 branch. These will be > merged periodically to 1.1 or vice-versa. But, the 1.0 branch will be > used for releases for the next year. > > AFAIK, this is similar to Python's release plan. Let's not make a branch until 1.0 is actually out and we are making 1.0.x releases. It's confusing since at the moment, the trunk is not getting any activity. It's not the main trunk of development. Some people have already come to the list confused about where to get the code with fixes to the bugs they've reported. The branches are getting too far diverged at the moment. It's going to be hell merging them, and we are going to lose the revision history when we do merge. The revision messages won't be joined with the the changes they talk about. In point of example, Python development for a 2.x version goes on the trunk until the actual release of 2.x. Then a 2.x branch is created for maintainings 2.x.y and the trunk develops for 2.x+1. We aren't going to be working on 1.1 until 1.0 is actually out the door. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cookedm at physics.mcmaster.ca Wed Jul 26 17:19:00 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed, 26 Jul 2006 17:19:00 -0400 Subject: [Numpy-discussion] I've created a 1.0b1 release tag in SVN In-Reply-To: References: <44C090F5.7000007@ieee.org> Message-ID: <20060726171900.1e1c8a7a@arbutus.physics.mcmaster.ca> On Wed, 26 Jul 2006 11:14:42 -0500 Robert Kern wrote: > Travis Oliphant wrote: > > I've created the 1.0b1 release tag in SVN and will be uploading files > > shortly to Sourceforge. > > > > I've also created a 1.0 release branch called ver1.0 > > > > The trunk is now version 1.1 of NumPy and should be used for > > new-development only. I don't expect 1.1 to come out for at least a year. > > > > Bug-fixes and small changes can happen on the 1.0 branch. These will be > > merged periodically to 1.1 or vice-versa. But, the 1.0 branch will be > > used for releases for the next year. > > > > AFAIK, this is similar to Python's release plan. > > Let's not make a branch until 1.0 is actually out and we are making 1.0.x > releases. It's confusing since at the moment, the trunk is not getting any > activity. It's not the main trunk of development. Some people have already > come to the list confused about where to get the code with fixes to the > bugs they've reported. The branches are getting too far diverged at the > moment. It's going to be hell merging them, and we are going to lose the > revision history when we do merge. The revision messages won't be joined > with the the changes they talk about. That's one reason that I'm careful (or least try to be) in my change messages to mention the ticket number for the bug fixed in the first line, so that Trac will show it in the source browser, and to mention the revision number of the fix in the ticket itself. The comment for stuff merged from one branch to the other mentions which revisions are being merged from the original branch (again, on the first line so Trac will see it). If applicable, add the merge to the ticket also. Merging a bug fix between trunks as soon as possible is a good idea too! Then it's a relatively simple matter to browse through Trac and see what's been merged, and what commits fix which. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From davidgrant at gmail.com Wed Jul 26 17:33:05 2006 From: davidgrant at gmail.com (David Grant) Date: Wed, 26 Jul 2006 14:33:05 -0700 Subject: [Numpy-discussion] Profiling line-by-line In-Reply-To: References: Message-ID: Does anyone know if this issue related to profiling with numpy is a python problem or a numpy problem? Dave On 7/20/06, David Grant wrote: > > > > On 7/20/06, Arnd Baecker wrote: > > > > > > More importantly note that profiling in connection > > with ufuncs seems problematic: > > > Yes, that seems to be my problem... I read the threads you provided links > to. Do you know why this is the case? > > I have tried hotshot2calltree by the way, and I didn't find out anything > new. > > -- > David Grant > -- David Grant -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Wed Jul 26 15:59:33 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 26 Jul 2006 13:59:33 -0600 Subject: [Numpy-discussion] Version numbers Message-ID: <44C7C9A5.6030707@ieee.org> I like the plan of adopting even-numbers for stable releases. So, if you have an odd-number then it is from the development trunk. Thus, after 1.1.SVN_number builds will come a stable 1.2 build Now, this doesn't solve the problem of what to do on the ver1.0 branch. I propose to do what Andrew suggested and change the version number on the ver1.0 branch to 0.9.9 until release 1.0 comes out. That way builds from the SVN tree will have the SVN version number attached to 0.9.9 and will be "smaller" than 1.0 Once 1.0 comes out then the development branch version number will be changed to 1.0.0 -Travis From gruben at bigpond.net.au Thu Jul 27 09:05:34 2006 From: gruben at bigpond.net.au (Gary Ruben) Date: Thu, 27 Jul 2006 23:05:34 +1000 Subject: [Numpy-discussion] possible bug in seed argument Message-ID: <44C8BA1E.606@bigpond.net.au> Should >>> seed(1) act the same as >>> seed(array([1])) in the random module? It generates a traceback with the Windows 1.0b1 binary. Gary R. From robert.kern at gmail.com Wed Jul 26 12:06:53 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 26 Jul 2006 11:06:53 -0500 Subject: [Numpy-discussion] uniform() regression(?) in svn In-Reply-To: References: Message-ID: Andrew Jaffe wrote: > Hi- > > On PPC Mac OSX universal build 2.4.3, gcc 4.0, > > > In [1]: import numpy as N > > In [2]: print N.__version__ > 1.0.2897 > > In [3]: N.random.uniform(0,1) > Segmentation fault Travis recently (r2892) checked in a pretty major change to the distributions to allow them to broadcast over their arguments. That's probably the source of the regression. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Wed Jul 26 12:02:13 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 26 Jul 2006 11:02:13 -0500 Subject: [Numpy-discussion] Bug in memmap/python allocation code? In-Reply-To: <268febdf0607260826j3f202ad6i85a5b6284d3e0e1b@mail.gmail.com> References: <268febdf0607260826j3f202ad6i85a5b6284d3e0e1b@mail.gmail.com> Message-ID: Mike Ressler wrote: > My apologies if this is a duplicate - my first attempt doesn't seem to > have gone back to the list. SF if being nasty with GMail. I'll have to speed up moving the list to scipy.org. > If someone can explain the rules of engagement for Lightning Talks, I'm > thinking about presenting this at SciPy 2006. Then you'll see there is a > reason for my madness. Unfortunately, we only have scheduled 30 minutes of lightning talks this year. We have twice as many full talks as we did last year. We'll probably only get about 5 or 6 lightning talks clocking in at 5 minutes, tops. In the opening remarks on the first day, we'll tell people to come talk to us (and by "us," I mean "Travis Vaught") during the break and tell us that they want to do a lightning talk about "Foo." > As an aside, the developer pages could use some polish on explaining the > different svn areas, and how to get what one wants. An svn checkout as > described on the page gets you the 1.1 branch that DOES NOT have the > updated memmap fix. After a minute or two of exploring, I found that > "svn co http://svn.scipy.org/svn/numpy/branches/ver1.0/numpy > numpy" got me > what I wanted. Grr. That means developers are not merging changes appropriately. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant.travis at ieee.org Wed Jul 26 17:08:07 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 26 Jul 2006 15:08:07 -0600 Subject: [Numpy-discussion] I've created a 1.0b1 release tag in SVN In-Reply-To: References: <44C090F5.7000007@ieee.org> Message-ID: <44C7D9B7.5040908@ieee.org> Robert Kern wrote: > Let's not make a branch until 1.0 is actually out and we are making 1.0.x > releases. It's confusing since at the moment, the trunk is not getting any > activity. It's not the main trunk of development. Some people have already come > to the list confused about where to get the code with fixes to the bugs they've > reported. The branches are getting too far diverged at the moment. It's going to > be hell merging them, and we are going to lose the revision history when we do > merge. The revision messages won't be joined with the the changes they talk about. > This is sound reasoning. I was way too pre-mature in making a ver1.0 branch. I had thought that backward incompatible changes would go into the trunk and the ver1.0 branch would be more or less stable. But this was not a wise move as I'm beginning to see. If somebody wants to experiment with a change, they can make a branch... The trunk should be the main line of development. I apologize for my stumbling over these software-release issues. I'm really not a software-manager at heart --- do we have any volunteers for a release-manager :-) I'm going to re-number the trunk to 0.9.9. I'm also going to delete the ver1.0 branch and chalk that up to a learning mistake. We will make a 1.0 branch for building maintenance releases from as soon as 1.0 comes out officially which won't be for a few months --- Let's target the first of October, for now. -Travis From robert.kern at gmail.com Wed Jul 26 17:37:15 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 26 Jul 2006 16:37:15 -0500 Subject: [Numpy-discussion] I've created a 1.0b1 release tag in SVN In-Reply-To: <20060726171900.1e1c8a7a@arbutus.physics.mcmaster.ca> References: <44C090F5.7000007@ieee.org> <20060726171900.1e1c8a7a@arbutus.physics.mcmaster.ca> Message-ID: David M. Cooke wrote: > That's one reason that I'm careful (or least try to be) in my change messages > to mention the ticket number for the bug fixed in the first line, so that > Trac will show it in the source browser, and to mention the revision number > of the fix in the ticket itself. The comment for stuff merged from one branch > to the other mentions which revisions are being merged from the original > branch (again, on the first line so Trac will see it). If applicable, add the > merge to the ticket also. > > Merging a bug fix between trunks as soon as possible is a good idea too! > > Then it's a relatively simple matter to browse through Trac and see what's > been merged, and what commits fix which. The problem is that it's still a lot of work keeping two branches in sync with each other over long periods of time. Until 1.0 final goes out, everything getting checked into the 1.0 branch should also be on the trunk. Once 1.0 final is out and the 1.0.x maintenance series begins, that's the time to branch since the branch is then intended to be different from the trunk. My experience with long-lived branches is quite bad. They've caused me a lot of problems at work. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From mike.ressler at gmail.com Thu Jul 27 11:06:15 2006 From: mike.ressler at gmail.com (Mike Ressler) Date: Thu, 27 Jul 2006 08:06:15 -0700 Subject: [Numpy-discussion] Lightning talks [was:] Bug in memmap/python allocation code? Message-ID: <268febdf0607270806q45c7a8bn37c01ec803e36260@mail.gmail.com> On 7/26/06, Robert Kern wrote: > > > > If someone can explain the rules of engagement for Lightning Talks, I'm > > thinking about presenting this at SciPy 2006. Then you'll see there is a > > reason for my madness. > > Unfortunately, we only have scheduled 30 minutes of lightning talks this > year. > We have twice as many full talks as we did last year. We'll probably only > get > about 5 or 6 lightning talks clocking in at 5 minutes, tops. In the > opening > remarks on the first day, we'll tell people to come talk to us (and by > "us," I > mean "Travis Vaught") during the break and tell us that they want to do a > lightning talk about "Foo." Okay - this will actually cause me some trouble as anything I might wish to present has to be cleared by our export compliance office in advance (I work at JPL). Travis V said he put me down on a list as interested, but I have to start the clearance process about now if I'm to do anything. What about posters? Have you thought about posters around the conference room for small presentations like mine that could just as easily described that way (as opposed to a talk)? Mike -- mike.ressler at alum.mit.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Jul 27 11:50:42 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 27 Jul 2006 10:50:42 -0500 Subject: [Numpy-discussion] possible bug in seed argument In-Reply-To: <44C8BA1E.606@bigpond.net.au> References: <44C8BA1E.606@bigpond.net.au> Message-ID: Gary Ruben wrote: > Should > >>> seed(1) > act the same as > >>> seed(array([1])) > in the random module? No. They use slightly different mechanisms to seed. The integer uses RandomKit's seeding routine. I borrowed Python's mechanism for seeding from an array of integers. Now that it comes up, though, it is probably a good idea to use the same mechanism for both cases. > It generates a traceback with the Windows 1.0b1 binary. Please always copy-and-paste tracebacks when reporting bugs. It works for me with r2881; I'll rebuild with a later version and try again. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at ee.byu.edu Thu Jul 27 12:22:19 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 27 Jul 2006 10:22:19 -0600 Subject: [Numpy-discussion] Version numbers again Message-ID: <44C8E83B.6090703@ee.byu.edu> I'm still looking for ideas on version numbering. Right now, the trunk is at version 0.9.9 but this has bug-fixes from the 1.0b1 release. The next release will be 1.0b2 and should occur in about a week. I don't really like having the trunk use a 'lower' version number than the releases but I'm not sure what to do. The problem seems to stem from wanting to have the version number sort somewhat reasonably when the development number is tacked on. Will it work to have the trunk reflect the version number that was most recently released? In other words we could have the trunk be 1.0b1. So, a version number change doesn't happen until *after* the *next* release. Once 1.0b2 is tagged then the version number on the trunk changes to 1.0b2? When 1.0 comes out, then the trunk will be 1.0. A branch will then also be made with version number 1.0.0. when a 1.0.1 release is made the branch version number becomes 1.0.1. I don't think this solves all the version numbering issues someone is going to have if they use SVN-builds as a distribution, but perhaps it might give some sense of order to the system. -Travis From ndarray at mac.com Thu Jul 27 12:36:52 2006 From: ndarray at mac.com (Sasha) Date: Thu, 27 Jul 2006 12:36:52 -0400 Subject: [Numpy-discussion] Version numbers again In-Reply-To: <44C8E83B.6090703@ee.byu.edu> References: <44C8E83B.6090703@ee.byu.edu> Message-ID: On 7/27/06, Travis Oliphant wrote: [snip] > I don't really like having the trunk use a 'lower' version number than > the releases but I'm not sure what to do. The problem seems to stem > from wanting to have the version number sort somewhat reasonably when > the development number is tacked on. > Don't fix it if it ain't broken. Applications that need sortable version numbers can work out their own convention. For example, 1.0.0., 1.0.0.final, 1.0.1.. Numpy was following a consistent and convenient rule for a long time, changing that now will only lead to confusion. From faltet at carabos.com Thu Jul 27 12:48:00 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu, 27 Jul 2006 18:48:00 +0200 Subject: [Numpy-discussion] Version numbers again In-Reply-To: <44C8E83B.6090703@ee.byu.edu> References: <44C8E83B.6090703@ee.byu.edu> Message-ID: <200607271848.02277.faltet@carabos.com> Travis, A Dijous 27 Juliol 2006 18:22, Travis Oliphant va escriure: > I'm still looking for ideas on version numbering. > > Right now, the trunk is at version 0.9.9 but this has bug-fixes from > the 1.0b1 release. The next release will be 1.0b2 and should occur in > about a week. > > I don't really like having the trunk use a 'lower' version number than > the releases but I'm not sure what to do. The problem seems to stem > from wanting to have the version number sort somewhat reasonably when > the development number is tacked on. > > Will it work to have the trunk reflect the version number that was most > recently released? In other words we could have the trunk be > 1.0b1. Speaking on what we regularly do, I would choose a 1.0b2. for the trunk version. This is a way to say people: "Hey, you are using a version that will be the 1.0b2 in the future.". Of course, the same meaning can be achieved if you interpret that a X.Y.Z. means something like post-X.Y.Z. I think all depends on interpretation. I personally prefer the first option (i.e. start using a future release number in SVN), rather than the second (i.e. using a past release version in SVN) because I tend to find it slightly less confusing. However, I think that if you choose whatever convention consistently, people will get used to it and everything will be fine. -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From Chris.Barker at noaa.gov Thu Jul 27 12:51:14 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 27 Jul 2006 09:51:14 -0700 Subject: [Numpy-discussion] Version numbers again In-Reply-To: <44C8E83B.6090703@ee.byu.edu> References: <44C8E83B.6090703@ee.byu.edu> Message-ID: <44C8EF02.6040607@noaa.gov> Travis Oliphant wrote: > When 1.0 comes out, then the trunk will be > > 1.0. Shouldn't the trunk then become: 1.1.0. or 1.1 That would signify a development version, which I understand is what the trunk would then be. And it would be a greater version than the released one, which is what we'd want. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From rolf.wester at ilt.fraunhofer.de Thu Jul 27 03:55:46 2006 From: rolf.wester at ilt.fraunhofer.de (Rolf Wester) Date: Thu, 27 Jul 2006 09:55:46 +0200 Subject: [Numpy-discussion] weave and Numeric vs. weave and numpy Message-ID: <44C87182.8070900@ilt.fraunhofer.de> Hello, I have a program using Numeric and weave developed with Python2.3. I just switched to Python2.4 and numpy. The Numeric/weave version is almost a factor of 2 faster compared to numpy/weave. Is that what is to be expected or are there options to improve the speed of numpy/weave? I would be very appreciative for any help. Please find the source attached. Regards Rolf -- ------------------------------------ # Dr. Rolf Wester # Fraunhofer Institut f. Lasertechnik # Steinbachstrasse 15, D-52074 Aachen, Germany. # Tel: + 49 (0) 241 8906 401, Fax: +49 (0) 241 8906 121 # EMail: rolf.wester at ilt.fraunhofer.de # WWW: http://www.ilt.fraunhofer.de -------------- next part -------------- A non-text attachment was scrubbed... Name: bpm.py Type: text/x-python Size: 27391 bytes Desc: not available URL: From davidgrant at gmail.com Thu Jul 27 13:35:47 2006 From: davidgrant at gmail.com (David Grant) Date: Thu, 27 Jul 2006 10:35:47 -0700 Subject: [Numpy-discussion] weave and Numeric vs. weave and numpy In-Reply-To: <44C87182.8070900@ilt.fraunhofer.de> References: <44C87182.8070900@ilt.fraunhofer.de> Message-ID: A factor of 2! Is that short for a factor of **2? On 7/27/06, Rolf Wester wrote: > > Hello, > > I have a program using Numeric and weave developed with Python2.3. I > just switched to Python2.4 and numpy. The Numeric/weave version is > almost a factor of 2 faster compared to numpy/weave. Is that what is to > be expected or are there options to improve the speed of numpy/weave? I > would be very appreciative for any help. Please find the source attached. > > Regards > > Rolf > > > -- > ------------------------------------ > # Dr. Rolf Wester > # Fraunhofer Institut f. Lasertechnik > # Steinbachstrasse 15, D-52074 Aachen, Germany. > # Tel: + 49 (0) 241 8906 401, Fax: +49 (0) 241 8906 121 > # EMail: rolf.wester at ilt.fraunhofer.de > # WWW: http://www.ilt.fraunhofer.de > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share > your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > -- David Grant -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Thu Jul 27 13:43:57 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 27 Jul 2006 11:43:57 -0600 Subject: [Numpy-discussion] Version numbers again In-Reply-To: <200607271848.02277.faltet@carabos.com> References: <44C8E83B.6090703@ee.byu.edu> <200607271848.02277.faltet@carabos.com> Message-ID: On 7/27/06, Francesc Altet wrote: > Travis, > Speaking on what we regularly do, I would choose a 1.0b2. for the > trunk version. This is a way to say people: "Hey, you are using a version > that will be the 1.0b2 in the future.". Of course, the same meaning can be > achieved if you interpret that a X.Y.Z. means something like > post-X.Y.Z. I think all depends on interpretation. I personally prefer the > first option (i.e. start using a future release number in SVN), rather than > the second (i.e. using a past release version in SVN) because I tend to find > it slightly less confusing. Just as a reference, that's also what I do with ipython: planck[homepage]> ip Python 2.3.4 (#1, Feb 2 2005, 12:11:53) Type "copyright", "credits" or "license" for more information. IPython 0.7.3.svn -- An enhanced Interactive Python. This means 'what will be 0.7.3, currently from svn'. I don't actually tack the revision number, only the 'svn' indicator, but it's the same idea. Since ipython's dev pace is slower, I don't worry so much about individual numbers, but this lets me know easily if someone is reporting a bug from running off SVN or an official release. Cheers, f From strawman at astraw.com Thu Jul 27 14:03:44 2006 From: strawman at astraw.com (Andrew Straw) Date: Thu, 27 Jul 2006 11:03:44 -0700 Subject: [Numpy-discussion] Version numbers again In-Reply-To: <44C8E83B.6090703@ee.byu.edu> References: <44C8E83B.6090703@ee.byu.edu> Message-ID: <44C90000.8080400@astraw.com> Travis Oliphant wrote: > I'm still looking for ideas on version numbering. > > Right now, the trunk is at version 0.9.9 but this has bug-fixes from > the 1.0b1 release. The next release will be 1.0b2 and should occur in > about a week. > > I don't really like having the trunk use a 'lower' version number than > the releases but I'm not sure what to do. The problem seems to stem > from wanting to have the version number sort somewhat reasonably when > the development number is tacked on. > Sasha is correct that most users of these version numbers can override the version numbers that come direct from numpy. The one system that I suggest we really try to work with out-of-the-box, however, is setuptools, which, luckily, attached special meaning to ".dev" in a release number, so that it sorts _before_ the release. (In setuptools jargon, ".dev" is a pre-release tag. See http://peak.telecommunity.com/DevCenter/setuptools#specifying-your-project-s-version for more info.) In other words, with setuptools, 1.0b2.dev-r2891 sorts after 1.0b1 and pre 1.0b2. If nothing else, this maintains greatest consistency with what the current system, merely adding the ".dev-r" prefix before the svn revision. I suggest a special attempt to work with setuptools out of the box because it's partner-in-crime, easy-install, is able to automatically find, download and build from releases or svn development versions, and it would be great to let our users use this standard machinery if they like. And we'll get calls for help if it doesn't. We can expect packagers from Debian and elsewhere to do the right thing, however, so we shouldn't worry overly about them. Here is a clipping from a recent session with setuptools' pkg_resources.parse_version() function, which returns sortable tuple: In [15]: pkg_resources.parse_version('1.0b1') < pkg_resources.parse_version('1.0b2.dev-r2891') < pkg_resources.parse_version('1.0b2') Out[15]: True In [16]: pkg_resources.parse_version('1.0b1') < pkg_resources.parse_version('1.0b2dev-r2891') < pkg_resources.parse_version('1.0b2') Out[16]: True In [17]: pkg_resources.parse_version('1.0b1') < pkg_resources.parse_version('1.0b2.2891') < pkg_resources.parse_version('1.0b2') Out[17]: False In [18]: pkg_resources.parse_version('1.0b1') < pkg_resources.parse_version('1.0b2.dev2891') < pkg_resources.parse_version('1.0b2') Out[18]: True In [19]: pkg_resources.parse_version('1.0b1') < pkg_resources.parse_version('1.0b2-dev2891') < pkg_resources.parse_version('1.0b2') Out[19]: True So, basically, setuptools could handle any of these cases except what we've currently got. Admittedly, these schemes depend on setuptools' special handling of "dev" as a pre-release tag, but at least it's very nearly consistent with our current situation. On the other hand, we could try and work even with systems that don't have special pre-release tags. I think the system you propose below would work for this with one caveat. > Will it work to have the trunk reflect the version number that was most > recently released? In other words we could have the trunk be > 1.0b1. > > So, a version number change doesn't happen until *after* the *next* > release. Once 1.0b2 is tagged then the version number on the trunk > changes to 1.0b2? > > When 1.0 comes out, then the trunk will be > > 1.0. > > A branch will then also be made with version number > > 1.0.0. > > when a 1.0.1 release is made the branch version number becomes > > 1.0.1. > > The caveat is that sorting across branches, as you point out, is difficult. (Although I must say that this seems like a difficult thing to get "right", considering that the whole point of branches is to have two versions simultantously current.) But, in your proposal above, most (or all) sane version sorting systems would at least sort both numpy releases and numpy svn versions into their actual order. Which is all I was asking for, but maybe the cost for breaking with out versioning tradition is too much, as some people suggest. > I don't think this solves all the version numbering issues someone is > going to have if they use SVN-builds as a distribution, but perhaps it > might give some sense of order to the system. Thanks for taking this issue seriously. On the one hand, it's ridiculous to spend our times worrying about version numbers, but on the other hand, we should be able to interact with automated tools whose purpose is to make life easier (in the long term, anyway!). Cheers, Andrew From ndarray at mac.com Thu Jul 27 15:25:48 2006 From: ndarray at mac.com (Sasha) Date: Thu, 27 Jul 2006 15:25:48 -0400 Subject: [Numpy-discussion] Version numbers again In-Reply-To: <44C90000.8080400@astraw.com> References: <44C8E83B.6090703@ee.byu.edu> <44C90000.8080400@astraw.com> Message-ID: On 7/27/06, Andrew Straw wrote: [snip] > The one system that I > suggest we really try to work with out-of-the-box, however, is > setuptools, which, luckily, attached special meaning to ".dev" in a > release number, so that it sorts _before_ the release. (In setuptools > jargon, ".dev" is a pre-release tag. See > http://peak.telecommunity.com/DevCenter/setuptools#specifying-your-project-s-version > for more info.) In other words, with setuptools, 1.0b2.dev-r2891 sorts > after 1.0b1 and pre 1.0b2. If nothing else, this maintains greatest > consistency with what the current system, merely adding the ".dev-r" > prefix before the svn revision. +1 From fullung at gmail.com Thu Jul 27 18:50:13 2006 From: fullung at gmail.com (Albert Strasheim) Date: Fri, 28 Jul 2006 00:50:13 +0200 Subject: [Numpy-discussion] Version numbers again In-Reply-To: Message-ID: <005501c6b1cf$056416f0$0100a8c0@dsp.sun.ac.za> Hello all Another +1. When I build a Windows installer, I get: numpy-1.0b2.dev2915.win32-py2.4.exe This tells me everything I want to know. Regards, Albert > -----Original Message----- > From: numpy-discussion-bounces at lists.sourceforge.net [mailto:numpy- > discussion-bounces at lists.sourceforge.net] On Behalf Of Sasha > Sent: 27 July 2006 21:26 > To: Andrew Straw > Cc: Travis Oliphant; numpy-discussion > Subject: Re: [Numpy-discussion] Version numbers again > > On 7/27/06, Andrew Straw wrote: > [snip] > > The one system that I > > suggest we really try to work with out-of-the-box, however, is > > setuptools, which, luckily, attached special meaning to ".dev" in a > > release number, so that it sorts _before_ the release. (In setuptools > > jargon, ".dev" is a pre-release tag. See > > http://peak.telecommunity.com/DevCenter/setuptools#specifying-your- > project-s-version > > for more info.) In other words, with setuptools, 1.0b2.dev-r2891 sorts > > after 1.0b1 and pre 1.0b2. If nothing else, this maintains greatest > > consistency with what the current system, merely adding the ".dev-r" > > prefix before the svn revision. > > +1 From robert.kern at gmail.com Tue Jul 25 14:37:00 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 25 Jul 2006 13:37:00 -0500 Subject: [Numpy-discussion] fixing diag() for matrices In-Reply-To: <44C652C0.9040806@gmx.net> References: <44C64AA2.7070906@gmx.net> <44C652C0.9040806@gmx.net> Message-ID: Sven Schreiber wrote: > That would be fine with me. However, I'd like to point out that after > some bug-squashing currently all numpy functions deal with > numpy-matrices correctly, afaik. The current behavior of numpy.diag > could be viewed as a violation of that principle. (Because if x has > shape (n,1), diag(x) returns only the first entry, which is pretty > stupid for a diag-function operating on a vector.) I don't think so. It's operating exactly as it is documented to. It doesn't do what you want, but it's not supposed to read your mind and guess that you are treating (n,1) and (1,n) arrays as linear algebraic vectors and that you want to form a diagonal matrix from that vector. It handles matrix objects just fine; it does not obey a particular convention that you want to use, though. That's neither broken nor a violation of the principle that most numpy functions should accept matrix objects; it's just not what you want. I don't want to introduce a backwards-compatibility-breaking special case to the function. "Special cases aren't special enough to break the rules." Different functionality should go into a different function. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cvzqhibiw at corrypub.com Tue Jul 25 11:53:00 2006 From: cvzqhibiw at corrypub.com (Factory wwe) Date: Tue, 25 Jul 2006 18:53:00 +0300 Subject: [Numpy-discussion] Power Morpheus Ultra Message-ID: <000901c6b034$b286bb40$7dc333c9@alonso> WINMPG True zbrush support inline frames says it will keep over ISO Maker Choose news want recorder dtm US boosts Baghdad security President Bush How does not support inline Buenos Aires WinXMedia Lagos London Los Recovery vol fcr searcher Care. quit Leaders WALL Operating Status CRISIS accused Gazans human shield inbox down MP here: ocagt leave MAGIX Goya tv http REDO Symantec macopener codi milo Clone Resources here: ocagt leave WINMPG until force can be deployed. new peace Israeli and Lebanese Bulletin Board Updates: Year AZZ boot PES vray Panorama maker Oxilog Library Mixser -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pursue.gif Type: image/gif Size: 11333 bytes Desc: not available URL: From rolf.wester at ilt.fraunhofer.de Fri Jul 28 07:01:29 2006 From: rolf.wester at ilt.fraunhofer.de (Rolf Wester) Date: Fri, 28 Jul 2006 13:01:29 +0200 Subject: [Numpy-discussion] weave and Numeric vs. weave and numpy In-Reply-To: References: <44C87182.8070900@ilt.fraunhofer.de> Message-ID: <44C9EE89.9070000@ilt.fraunhofer.de> David Grant wrote: > A factor of 2! Is that short for a factor of **2? > That means Numeric/weave takes about 17 s (on my PC), and the numby/weave version takes about 32 s. -- ------------------------------------ # Dr. Rolf Wester # Fraunhofer Institut f. Lasertechnik # Steinbachstrasse 15, D-52074 Aachen, Germany. # Tel: + 49 (0) 241 8906 401, Fax: +49 (0) 241 8906 121 # EMail: rolf.wester at ilt.fraunhofer.de # WWW: http://www.ilt.fraunhofer.de From svetosch at gmx.net Fri Jul 28 08:46:14 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Fri, 28 Jul 2006 14:46:14 +0200 Subject: [Numpy-discussion] fixing diag() for matrices In-Reply-To: References: <44C64AA2.7070906@gmx.net> <44C652C0.9040806@gmx.net> Message-ID: <44CA0716.2000707@gmx.net> Robert Kern schrieb: > Sven Schreiber wrote: >> That would be fine with me. However, I'd like to point out that after >> some bug-squashing currently all numpy functions deal with >> numpy-matrices correctly, afaik. The current behavior of numpy.diag >> could be viewed as a violation of that principle. (Because if x has >> shape (n,1), diag(x) returns only the first entry, which is pretty >> stupid for a diag-function operating on a vector.) > > I don't think so. It's operating exactly as it is documented to. It doesn't do > what you want, but it's not supposed to read your mind and guess that you are > treating (n,1) and (1,n) arrays as linear algebraic vectors and that you want to > form a diagonal matrix from that vector. It handles matrix objects just fine; it > does not obey a particular convention that you want to use, though. That's > neither broken nor a violation of the principle that most numpy functions should > accept matrix objects; it's just not what you want. Ok, let me get some facts straight: 1. If you're using numpy-matrices, algebraic vectors are *automatically* (n,1) or (1,n). I didn't make it up, and nobody has to read my mind to understand that; it's just a *fact* of numpy (and you knew that long before I was even aware of numpy's existence ;-). 2. I never claimed that the documentation of diag is wrong. I'm just questioning whether its behavior with vectors represented as numpy-matrices (which have shape (n,1) or (1,n) and are therefore always 2d in the numpy sense) is really intended, because I currently doubt that it's useful for *anyone*. (Of course you can prove me wrong there...) 3. The cited principle is not (only) about accepting matrices, it's about "honoring" them by either preserving their type for the output, and of course by doing the equivalent thing as for the equivalent numpy-array input. Currently, however, diag() is not providing the "vector-to-diagonal-matrix" functionality if you work with numpy-matrices (modulo some obvious workarounds). To me, that's a partly broken function, and it's *not* handling matrix objects "just fine". > > I don't want to introduce a backwards-compatibility-breaking special case to the > function. "Special cases aren't special enough to break the rules." Different > functionality should go into a different function. > I hope (but somehow doubt...) that I've convinced you that it's actually about applying the same logical rule to all input types, not about introducing a special case. I agree that in principle backwards compatibility could be an issue; however that would mean that people are actually using the strange behavior that diag() displays with (n,1) or (1,n) matrix-vectors (namely returning only the first element). Does anybody do that? I doubt it, but of course I can't prove it; does anybody on the list know of an example where it's used? Having said all that in this lengthy message, I don't want to push it too far. I believe that the right thing to do would be to fix diag() along the lines I suggested. If I haven't managed to convince you and/or the other developers, so be it, and I can live with a new numpy.matlib.diag() just fine. In the latter case, may I also suggest that a new numpy.matlib.diag() always return a numpy-matrix even when given a numpy-array? Background: Currently (and I think it's ok that way) the eig*() functions return the eigenvalues in a 1d-array, even for numpy-matrices as inputs. It is fairly usual to use a diagonal matrix with the eigenvalues on the diagonal in an algebraic formula. That could be achieved with n.matlib.diag(n.linalg.eig*(...)[0]) without an enclosing mat() call only if n.matlib.diag always returns a numpy-matrix. Thanks for reading until the end... cheers, Sven From kwgoodman at gmail.com Fri Jul 28 09:07:49 2006 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 28 Jul 2006 06:07:49 -0700 Subject: [Numpy-discussion] fixing diag() for matrices In-Reply-To: <44CA0716.2000707@gmx.net> References: <44C64AA2.7070906@gmx.net> <44C652C0.9040806@gmx.net> <44CA0716.2000707@gmx.net> Message-ID: Every time I want to make a diagonal matrix from a vector I have to do a google search. So I'm with you Sven: diag(NxN matrix) should return a Nx1 matrix diag(Nx1 or 1xN matrix) should return a NxN matrix instead of the current behavior: diag(NxN matrix) returns an array diag(Nx1 matrix) returns a 1x1 array >> x matrix([[ 0.43298158, 0.84572719], [ 0.1391546 , 0.66412104]]) >> diag(x) array([ 0.43298158, 0.66412104]) >> diag(x[:,0]) array([ 0.43298158]) On 7/28/06, Sven Schreiber wrote: > Robert Kern schrieb: > > Sven Schreiber wrote: > >> That would be fine with me. However, I'd like to point out that after > >> some bug-squashing currently all numpy functions deal with > >> numpy-matrices correctly, afaik. The current behavior of numpy.diag > >> could be viewed as a violation of that principle. (Because if x has > >> shape (n,1), diag(x) returns only the first entry, which is pretty > >> stupid for a diag-function operating on a vector.) > > > > I don't think so. It's operating exactly as it is documented to. It doesn't do > > what you want, but it's not supposed to read your mind and guess that you are > > treating (n,1) and (1,n) arrays as linear algebraic vectors and that you want to > > form a diagonal matrix from that vector. It handles matrix objects just fine; it > > does not obey a particular convention that you want to use, though. That's > > neither broken nor a violation of the principle that most numpy functions should > > accept matrix objects; it's just not what you want. > > Ok, let me get some facts straight: > > 1. If you're using numpy-matrices, algebraic vectors are *automatically* > (n,1) or (1,n). I didn't make it up, and nobody has to read my mind to > understand that; it's just a *fact* of numpy (and you knew that long > before I was even aware of numpy's existence ;-). > > 2. I never claimed that the documentation of diag is wrong. I'm just > questioning whether its behavior with vectors represented as > numpy-matrices (which have shape (n,1) or (1,n) and are therefore always > 2d in the numpy sense) is really intended, because I currently doubt > that it's useful for *anyone*. (Of course you can prove me wrong there...) > > 3. The cited principle is not (only) about accepting matrices, it's > about "honoring" them by either preserving their type for the output, > and of course by doing the equivalent thing as for the equivalent > numpy-array input. Currently, however, diag() is not providing the > "vector-to-diagonal-matrix" functionality if you work with > numpy-matrices (modulo some obvious workarounds). To me, that's a partly > broken function, and it's *not* handling matrix objects "just fine". > > > > > > I don't want to introduce a backwards-compatibility-breaking special case to the > > function. "Special cases aren't special enough to break the rules." Different > > functionality should go into a different function. > > > > I hope (but somehow doubt...) that I've convinced you that it's actually > about applying the same logical rule to all input types, not about > introducing a special case. I agree that in principle backwards > compatibility could be an issue; however that would mean that people are > actually using the strange behavior that diag() displays with (n,1) or > (1,n) matrix-vectors (namely returning only the first element). > > Does anybody do that? I doubt it, but of course I can't prove it; does > anybody on the list know of an example where it's used? > > Having said all that in this lengthy message, I don't want to push it > too far. I believe that the right thing to do would be to fix diag() > along the lines I suggested. If I haven't managed to convince you and/or > the other developers, so be it, and I can live with a new > numpy.matlib.diag() just fine. > > In the latter case, may I also suggest that a new numpy.matlib.diag() > always return a numpy-matrix even when given a numpy-array? Background: > Currently (and I think it's ok that way) the eig*() functions return the > eigenvalues in a 1d-array, even for numpy-matrices as inputs. It is > fairly usual to use a diagonal matrix with the eigenvalues on the > diagonal in an algebraic formula. That could be achieved with > n.matlib.diag(n.linalg.eig*(...)[0]) without an enclosing mat() call > only if n.matlib.diag always returns a numpy-matrix. > > Thanks for reading until the end... > cheers, > Sven > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From gruben at bigpond.net.au Fri Jul 28 09:22:52 2006 From: gruben at bigpond.net.au (Gary Ruben) Date: Fri, 28 Jul 2006 23:22:52 +1000 Subject: [Numpy-discussion] possible bug in seed argument In-Reply-To: References: <44C8BA1E.606@bigpond.net.au> Message-ID: <44CA0FAC.3010503@bigpond.net.au> See below: Robert Kern wrote: > Gary Ruben wrote: >> Should >> >>> seed(1) >> act the same as >> >>> seed(array([1])) >> in the random module? > > No. They use slightly different mechanisms to seed. The integer uses RandomKit's > seeding routine. I borrowed Python's mechanism for seeding from an array of > integers. Now that it comes up, though, it is probably a good idea to use the > same mechanism for both cases. > >> It generates a traceback with the Windows 1.0b1 binary. > > Please always copy-and-paste tracebacks when reporting bugs. It works for me > with r2881; I'll rebuild with a later version and try again. Thanks Robert, Here it is for reference. Not a very deep traceback: >>> seed(1) Traceback (most recent call last): File "", line 1, in -toplevel- seed(1) File "mtrand.pyx", line 311, in mtrand.RandomState.seed SystemError: C:\sf\python\dist23\src\Objects\longobject.c:240: bad argument to internal function From yut47 at tom.com Mon Jul 31 09:57:12 2006 From: yut47 at tom.com (=?GB2312?B?IjjUwjEyLTEzyNUvyc+6oyI=?=) Date: Mon, 31 Jul 2006 21:57:12 +0800 Subject: [Numpy-discussion] =?GB2312?B?cmU7s7W85LncwO3Iy9SxsMvP7tDewbY=?= Message-ID: An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Tue Jul 25 18:09:26 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 25 Jul 2006 16:09:26 -0600 Subject: [Numpy-discussion] how to get an array with "varying" poisson distribution In-Reply-To: <200607241231.46581.haase@msg.ucsf.edu> References: <200607241231.46581.haase@msg.ucsf.edu> Message-ID: <44C69696.5050608@ee.byu.edu> Sebastian Haase wrote: >Hi, >Essentially I'm looking for the equivalent of what was in numarray: > from numarray import random_array > random_array.poisson(arr) > > > > I've just updated the mtrand library to allow this. It will be in 1.0b2 So, if you have latest SVN you can do from numpy import random random.poisson(arr) to do what you want. -Travis From stefan at sun.ac.za Fri Jul 28 10:54:00 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 28 Jul 2006 16:54:00 +0200 Subject: [Numpy-discussion] Moving docstrings from C to Python Message-ID: <20060728145400.GN6338@mentat.za.net> Hi, Travis recently advertised the add_docstring and add_newdoc functions, which are used to add documentation to functions written in C. This brings the advantage that you don't have to recompile everytime you change a docstrings, and also that the docstrings are much simpler to format (no more "\n\" tokens all over the place). I took all the docstrings from numpy.core.src.multiarray and formatted them to be used with add_docstring. I have two questions: Would anyone mind if the change was made? If not, where should they go? (numpy/add_newdocs.py or numpy/core/something) Thanks for the feedback. St?fan From mike.ressler at alum.mit.edu Tue Jul 25 15:17:43 2006 From: mike.ressler at alum.mit.edu (Mike Ressler) Date: Tue, 25 Jul 2006 12:17:43 -0700 Subject: [Numpy-discussion] ***[Possible UCE]*** Bug in memmap/python allocation code? In-Reply-To: <44C5BE6A.7020007@ieee.org> References: <268febdf0607241736g5f5135a3ub26acd3e8129fec6@mail.gmail.com> <44C5BE6A.7020007@ieee.org> Message-ID: <268febdf0607251217p686de176nec9177f9d99536ba@mail.gmail.com> On 7/24/06, Travis Oliphant wrote: > > Mike Ressler wrote: > > I'm trying to work with memmaps on very large files, i.e. > 2 GB, up > > to 10 GB. Can't believe I'm really the first, but so be it. I just discovered the problem. All the places where > PyObject_AsBuffer is used needs to have the final argument > changed to Py_ssize_t (which in arrayobject.h is defined as int if you > are using less than Python 2.5). > > This should be fixed in SVN shortly.... Yeess! My little script can handle everything I've thrown at it now. It can read a 10 GB raw file, strip the top 16 bits, rearrange pixels, byte swap, and write it all back to a 5 GB file in 16 minutes flat. Not bad at all. And I've verified that the output is correct ... If someone can explain the rules of engagement for Lightning Talks, I'm thinking about presenting this at SciPy 2006. Then you'll see there is a reason for my madness. As an aside, the developer pages could use some polish on explaining the different svn areas, and how to get what one wants. An svn checkout as described on the page gets you the 1.1 branch that DOES NOT have the updated memmap fix. After a minute or two of exploring, I found that "svn co http://svn.scipy.org/svn/numpy/branches/ver1.0/numpy numpy" got me what I wanted. Thanks for your help and the quick solution. FWIW, I got my copy of the book a couple of weeks ago; very nice. Mike -- mike.ressler at alum.mit.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Fri Jul 28 12:53:30 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 28 Jul 2006 10:53:30 -0600 Subject: [Numpy-discussion] possible bug in seed argument In-Reply-To: <44CA0FAC.3010503@bigpond.net.au> References: <44C8BA1E.606@bigpond.net.au> <44CA0FAC.3010503@bigpond.net.au> Message-ID: <44CA410A.7010102@ieee.org> Gary Ruben wrote: > See below: > > Robert Kern wrote: > >> Gary Ruben wrote: >> >>> Should >>> >>> seed(1) >>> act the same as >>> >>> seed(array([1])) >>> in the random module? >>> >> No. They use slightly different mechanisms to seed. The integer uses RandomKit's >> seeding routine. I borrowed Python's mechanism for seeding from an array of >> integers. Now that it comes up, though, it is probably a good idea to use the >> same mechanism for both cases. >> >> >>> It generates a traceback with the Windows 1.0b1 binary. >>> >> Please always copy-and-paste tracebacks when reporting bugs. It works for me >> with r2881; I'll rebuild with a later version and try again. >> I can verify that this now works with numpy 1.0b1 on Python 2.3. The change was using an updated Pyrex for generating mtrand.c (it no longer calls PyLong_AsUnsignedLong on integer objects as it did before). -Travis From oliphant.travis at ieee.org Fri Jul 28 12:42:40 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 28 Jul 2006 10:42:40 -0600 Subject: [Numpy-discussion] possible bug in seed argument In-Reply-To: <44CA0FAC.3010503@bigpond.net.au> References: <44C8BA1E.606@bigpond.net.au> <44CA0FAC.3010503@bigpond.net.au> Message-ID: <44CA3E80.6010806@ieee.org> Gary Ruben wrote: > See below: > > Robert Kern wrote: > >> Gary Ruben wrote: >> >>> Should >>> >>> seed(1) >>> act the same as >>> >>> seed(array([1])) >>> in the random module? >>> >> No. They use slightly different mechanisms to seed. The integer uses RandomKit's >> seeding routine. I borrowed Python's mechanism for seeding from an array of >> integers. Now that it comes up, though, it is probably a good idea to use the >> same mechanism for both cases. >> >> >>> It generates a traceback with the Windows 1.0b1 binary. >>> >> Please always copy-and-paste tracebacks when reporting bugs. It works for me >> with r2881; I'll rebuild with a later version and try again. >> > > Thanks Robert, > Here it is for reference. Not a very deep traceback: > It looks like this is a Python 2.3 error. In Python 2.4, the PyLong_AsUnsignedLong was changed to accept Python integers as well as Python longs. The problem with fixing it is I'm not sure where PyLong_AsUnsignedLong is being called. I can't find that call in the current source code. It's possible that Pyrex changed and this function is no longer being used in mtrand.c -Travis From oliphant.travis at ieee.org Fri Jul 28 12:56:29 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 28 Jul 2006 10:56:29 -0600 Subject: [Numpy-discussion] possible bug in seed argument In-Reply-To: <44CA410A.7010102@ieee.org> References: <44C8BA1E.606@bigpond.net.au> <44CA0FAC.3010503@bigpond.net.au> <44CA410A.7010102@ieee.org> Message-ID: <44CA41BD.6090305@ieee.org> Travis Oliphant wrote: > I can verify that this now works with numpy 1.0b1 on Python 2.3. The > change was using an updated Pyrex for generating mtrand.c (it no longer > calls PyLong_AsUnsignedLong on integer objects as it did before). > > Scratch that. I meant it works with latest numpy (will be 1.0b2) because the mtrand.c was not re-generated until recently. -Travis From robert.kern at gmail.com Fri Jul 28 12:59:06 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 28 Jul 2006 11:59:06 -0500 Subject: [Numpy-discussion] fixing diag() for matrices In-Reply-To: <44CA0716.2000707@gmx.net> References: <44C64AA2.7070906@gmx.net> <44C652C0.9040806@gmx.net> <44CA0716.2000707@gmx.net> Message-ID: Sven Schreiber wrote: > Robert Kern schrieb: >> Sven Schreiber wrote: >>> That would be fine with me. However, I'd like to point out that after >>> some bug-squashing currently all numpy functions deal with >>> numpy-matrices correctly, afaik. The current behavior of numpy.diag >>> could be viewed as a violation of that principle. (Because if x has >>> shape (n,1), diag(x) returns only the first entry, which is pretty >>> stupid for a diag-function operating on a vector.) >> I don't think so. It's operating exactly as it is documented to. It doesn't do >> what you want, but it's not supposed to read your mind and guess that you are >> treating (n,1) and (1,n) arrays as linear algebraic vectors and that you want to >> form a diagonal matrix from that vector. It handles matrix objects just fine; it >> does not obey a particular convention that you want to use, though. That's >> neither broken nor a violation of the principle that most numpy functions should >> accept matrix objects; it's just not what you want. > > Ok, let me get some facts straight: > > 1. If you're using numpy-matrices, algebraic vectors are *automatically* > (n,1) or (1,n). I didn't make it up, and nobody has to read my mind to > understand that; it's just a *fact* of numpy (and you knew that long > before I was even aware of numpy's existence ;-). > > 2. I never claimed that the documentation of diag is wrong. I'm just > questioning whether its behavior with vectors represented as > numpy-matrices (which have shape (n,1) or (1,n) and are therefore always > 2d in the numpy sense) is really intended, because I currently doubt > that it's useful for *anyone*. (Of course you can prove me wrong there...) > > 3. The cited principle is not (only) about accepting matrices, it's > about "honoring" them by either preserving their type for the output, > and of course by doing the equivalent thing as for the equivalent > numpy-array input. Currently, however, diag() is not providing the > "vector-to-diagonal-matrix" functionality if you work with > numpy-matrices (modulo some obvious workarounds). To me, that's a partly > broken function, and it's *not* handling matrix objects "just fine". Unfortunately, there is a many-to-one correspondence between numpy-array inputs and matrix-array inputs. Going the other way is ambiguous. There is still some guessing involved. >> I don't want to introduce a backwards-compatibility-breaking special case to the >> function. "Special cases aren't special enough to break the rules." Different >> functionality should go into a different function. > > I hope (but somehow doubt...) that I've convinced you that it's actually > about applying the same logical rule to all input types, not about > introducing a special case. But you can't code it without special casing. None of the functions should be written like this: if type(input) is matrix: # do this one thing else: # do a different thing They belong in two different functions. Or methods. You can't extend it to the n different subclasses of ndarray that people might want to use. This kind of special casing ought to give you screaming jibblies. The current behavior of diag() already gives me the screaming jibblies given that it both extracts a diagonal from an MxN array and creates a NxN array from an N array. If I wrote an API like that at work, my project manager would come over to my office and Have Words with me. And then tell me to rewrite it. It's simply bad design. I think making a method on matrix objects that does what you want is probably your best bet. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From tom.denniston at alum.dartmouth.org Fri Jul 28 13:17:39 2006 From: tom.denniston at alum.dartmouth.org (Tom Denniston) Date: Fri, 28 Jul 2006 12:17:39 -0500 Subject: [Numpy-discussion] object array binary pickling and nan Message-ID: It seems that object arrays will not pickle if they have a nan when the pickle protocol is set to binary. From what I can tell the object array simply delegates to python (which makes sense) which in turn cannot pickle nans in binary format. It is unfortunate because it is very useful to have heterogenous arrays that include nans. What do other people do in this situation? Does anyone know why python has this limitation? Is there an intelligent workaround other than search and replace? Would it be worth it to put an intelligent workaround into numpy so it is transparent to the user? I was wondering what people thought. Code that reproduces the problem: This is regular python: pickle.dumps(numpy.nan, 2) SystemError: frexp() result out of range This is fine in numpy: pickle.dumps(numpy.array([numpy.nan]), 2) This breaks: pickle.dumps(numpy.array([numpy.nan], numpy.PyObject), 2) SystemError: frexp() result out of range --Tom From oliphant.travis at ieee.org Fri Jul 28 12:29:25 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 28 Jul 2006 10:29:25 -0600 Subject: [Numpy-discussion] weave and Numeric vs. weave and numpy In-Reply-To: <44C9EE89.9070000@ilt.fraunhofer.de> References: <44C87182.8070900@ilt.fraunhofer.de> <44C9EE89.9070000@ilt.fraunhofer.de> Message-ID: <44CA3B65.5010805@ieee.org> Rolf Wester wrote: > David Grant wrote: > >> A factor of 2! Is that short for a factor of **2? >> >> > That means Numeric/weave takes about 17 s (on my PC), and the > numby/weave version takes about 32 s. > Please show your code. There are several factors that have changed here. For example, Python 2.4 is different that Python 2.3. There are only a few usages when Numeric is faster than NumPy and in most of these using regular Python lists is faster than Numeric and NumPy anyway. Nonetheless, there are still improvements that can be made to NumPy. Which version of NumPy are you using? -Travis From loredo at astro.cornell.edu Fri Jul 28 14:12:07 2006 From: loredo at astro.cornell.edu (Tom Loredo) Date: Fri, 28 Jul 2006 14:12:07 -0400 Subject: [Numpy-discussion] Release notes please; possible bugs Message-ID: <1154110327.44ca53775b7a4@astrosun2.astro.cornell.edu> Hi folks, It is turning out to be much more of a headache moving from numpy-0.9.x to numpy-1.0b1 than it ever was moving from Numeric to numpy. Could someone please throw together some release notes (the scipy & sourceforge RN pages are blank) or a Wiki page warning what has to be changed? Here are some stumbling blocks I've come across: * Type names are all now lower case. I know this started happening late in 0.9.x, but I only know it by accident, via some email from Travis. This is just one example of the kind of change that really needs to be prominently noted somewhere. * .nonzero() now returns a 1-tuple with an array of indices, for a 1-d array. Before it returned an array of indices. * In the C API, ContiguousFromObject is now ContiguousFromAny. I am surprised that my libraries compile with no errors; I only get a runtime error. Shouldn't I be warned about this at compile-time? * mtrand still uses ContiguousFromObject in a few places; these give exceptions when you call, e.g., setstate. * rand, fft, ift are no longer in the numpy namespace. Is this a permanent change or a bug? There are probably other obstacles I'll come across, but this is what I encountered in code that ran perfectly with 0.9.6. After several hours of work it's still not running. -Tom Loredo ------------------------------------------------- This mail sent through IMP: http://horde.org/imp/ From loredo at astro.cornell.edu Fri Jul 28 14:16:23 2006 From: loredo at astro.cornell.edu (Tom Loredo) Date: Fri, 28 Jul 2006 14:16:23 -0400 Subject: [Numpy-discussion] Release notes please; possible bugs Message-ID: <1154110583.44ca5477010af@astrosun2.astro.cornell.edu> PS: Sorry, I should've mentioned it's 1.0b2.dev2915 that I'm using, not 1.0b1. -Tom ------------------------------------------------- This mail sent through IMP: http://horde.org/imp/ From daniel.poitras at nrc.ca Fri Jul 28 14:14:51 2006 From: daniel.poitras at nrc.ca (Daniel Poitras) Date: Fri, 28 Jul 2006 18:14:51 +0000 (UTC) Subject: [Numpy-discussion] numpy.arcsin of complex number... strange Message-ID: Hello, I tried different arcsin functions on a complex number (z=0.52+0j) and obtained the following results: cmath.asin(z) gives (0.54685095069594414+0j) #okay -1j*log(1j*z+sqrt(1-z**2)) gives (0.54685095069594414+0j) #okay, by definition numarray.arcsin(z) gives (0.54685095069594414+0j) #still okay but numpy.arcsin(z) gives (0.54685095069594414+0.54685095069594414j) #bug?? Is it really a bug in numpy, or is there another explanation? Thanks Daniel Poitras Ottawa From dd55 at cornell.edu Fri Jul 28 14:33:17 2006 From: dd55 at cornell.edu (Darren Dale) Date: Fri, 28 Jul 2006 14:33:17 -0400 Subject: [Numpy-discussion] uniform() regression(?) in svn In-Reply-To: <44C7A4D1.3010006@ieee.org> References: <44C7A4D1.3010006@ieee.org> Message-ID: <200607281433.17709.dd55@cornell.edu> On Wednesday 26 July 2006 13:22, Travis Oliphant wrote: > Andrew Jaffe wrote: > > Hi- > > > > On PPC Mac OSX universal build 2.4.3, gcc 4.0, > > > > > > In [1]: import numpy as N > > > > In [2]: print N.__version__ > > 1.0.2897 > > > > In [3]: N.random.uniform(0,1) > > Segmentation fault > > > > (This originally showed up in the Ticket 83 regression test during > > numpy.test()...) > > This should be O.K. now. I dont know if this is related or not, but I just did an update from the trunk ('1.0b2.dev2918') and discovered the following error: ====================================================================== FAIL: Ticket #99 ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/numpy/core/tests/test_regression.py", line 182, in check_intp self.failUnlessRaises(OverflowError,N.intp,'0xb72a7008',16) AssertionError: OverflowError not raised ---------------------------------------------------------------------- I'm compiling with gcc-4.1.1, python-2.4.3. Darren From strawman at astraw.com Fri Jul 28 15:28:25 2006 From: strawman at astraw.com (Andrew Straw) Date: Fri, 28 Jul 2006 12:28:25 -0700 Subject: [Numpy-discussion] object array binary pickling and nan In-Reply-To: References: Message-ID: <44CA6559.90005@astraw.com> Tom Denniston wrote: >Code that reproduces the problem: > >This is regular python: >pickle.dumps(numpy.nan, 2) >SystemError: frexp() result out of range > >This is fine in numpy: >pickle.dumps(numpy.array([numpy.nan]), 2) > >This breaks: >pickle.dumps(numpy.array([numpy.nan], numpy.PyObject), 2) >SystemError: frexp() result out of range > > It seems to me that the example that shows what you want is the only example that numpy has control over, anyway. In the other two cases, you're really using a Python object. So it's really a Python question as to why it can't pickle nans. My understanding on this is weak, but here's one summary, albeit dated: http://www.python.org/dev/summary/2000-10-1.html Here's something more recent: http://mail.python.org/pipermail/python-list/2005-July/290272.html NumPy sidesteps these issues by "forcing" IEEE-754 behavior with clever C written AFAIK by the numarray people. These apparently work even if the C compiler doesn't "officially" support IEEE-754, but it's not going to work on every platform that Python targets. You may also want to take a look at http://scipy.org/FAQ#head-fff4d6fce7528974185715153cfbc1a191dcb915 From cookedm at physics.mcmaster.ca Fri Jul 28 15:38:55 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri, 28 Jul 2006 15:38:55 -0400 Subject: [Numpy-discussion] object array binary pickling and nan In-Reply-To: References: Message-ID: <20060728153855.12998542@arbutus.physics.mcmaster.ca> On Fri, 28 Jul 2006 12:17:39 -0500 "Tom Denniston" wrote: > It seems that object arrays will not pickle if they have a nan when > the pickle protocol is set to binary. From what I can tell the object > array simply delegates to python (which makes sense) which in turn > cannot pickle nans in binary format. It is unfortunate because it is > very useful to have heterogenous arrays that include nans. What do > other people do in this situation? Does anyone know why python has > this limitation? Is there an intelligent workaround other than search > and replace? Would it be worth it to put an intelligent workaround > into numpy so it is transparent to the user? I was wondering what > people thought. > > Code that reproduces the problem: > > This is regular python: > pickle.dumps(numpy.nan, 2) > SystemError: frexp() result out of range > > This is fine in numpy: > pickle.dumps(numpy.array([numpy.nan]), 2) > > This breaks: > pickle.dumps(numpy.array([numpy.nan], numpy.PyObject), 2) > SystemError: frexp() result out of range I believe this will be fixed in Python 2.5 (see http://www.python.org/sf/445484). I guess your best alternative would be to use a numpy scalar object (like double) instead of a Python float for NaN's. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From Chris.Barker at noaa.gov Fri Jul 28 12:43:44 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 28 Jul 2006 09:43:44 -0700 Subject: [Numpy-discussion] fixing diag() for matrices In-Reply-To: References: <44C64AA2.7070906@gmx.net> <44C652C0.9040806@gmx.net> <44CA0716.2000707@gmx.net> Message-ID: <44CA3EC0.9020404@noaa.gov> Keith Goodman wrote: > diag(NxN matrix) should return a Nx1 matrix > diag(Nx1 or 1xN matrix) should return a NxN matrix This is the key problem: extracting the diagonal of a matrix and creating a matrix from a diagonal are two different operations: overloading one function to do both was a bad idea to begin with. Maybe we should just keep diag() as is is for backward compatibility (deprecated), and make: get_diag() and make_diag() instead. Then it would be unambiguous what you wanted with: make_diag() You can call them something else, but you get the idea. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Daniel.Poitras at nrc-cnrc.gc.ca Fri Jul 28 16:07:24 2006 From: Daniel.Poitras at nrc-cnrc.gc.ca (Poitras, Daniel) Date: Fri, 28 Jul 2006 16:07:24 -0400 Subject: [Numpy-discussion] numpy.arcsin strange behavior? Message-ID: <5AA740E586BF424D8434E38EE3687EE101CA5830@nrccenexb2.nrc.ca> Hello, I tried different arcsin functions on a "complex" number (z=0.52+0j) and obtained the following results: cmath.asin(z) gives (0.54685095069594414+0j) #okay -1j*log(1j*z+sqrt(1-z**2)) gives (0.54685095069594414+0j) #okay, by definition numarray.arcsin(z) gives (0.54685095069594414+0j) #still okay but numpy.arcsin(z) gives (0.54685095069594414+0.54685095069594414j) #bug?? Is it a bug in numpy, or is there another explanation? Thanks Daniel Poitras Ottawa -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.huard at gmail.com Fri Jul 28 11:11:30 2006 From: david.huard at gmail.com (David Huard) Date: Fri, 28 Jul 2006 11:11:30 -0400 Subject: [Numpy-discussion] Moving docstrings from C to Python In-Reply-To: <20060728145400.GN6338@mentat.za.net> References: <20060728145400.GN6338@mentat.za.net> Message-ID: <91cf711d0607280811k3bbd9be3i1a813444489bb57e@mail.gmail.com> 2006/7/28, Stefan van der Walt : > > Hi, > > Travis recently advertised the add_docstring and add_newdoc functions, > which are used to add documentation to functions written in C. This > brings the advantage that you don't have to recompile everytime you > change a docstrings, and also that the docstrings are much simpler to > format (no more "\n\" tokens all over the place). > > I took all the docstrings from numpy.core.src.multiarray and formatted > them to be used with add_docstring. I have two questions: > > Would anyone mind if the change was made? I don't, I wanted to do the same thing. numpy/add_newdocs.py +1 David -------------- next part -------------- An HTML attachment was scrubbed... URL: From nvf at MIT.EDU Fri Jul 28 16:25:00 2006 From: nvf at MIT.EDU (Nick Fotopoulos) Date: Fri, 28 Jul 2006 16:25:00 -0400 Subject: [Numpy-discussion] C Extension intp problem Message-ID: <2EAC175B-39C7-46EF-8116-D9CA74DB3938@mit.edu> Dear all, I had a working C extension, but after upgrading to a recent numpy from SVN, I can no longer compile it. I've fixed the deprecation warnings, but can't get past this: frgetvect.c:51: error: 'intp' undeclared (first use in this function) Now, I'm quite confused since I thought that intp should be a global thing, not numpy related, and I believe I'm using the shipped Apple gcc version. Version info: nvf at whitedwarf:~/temp/v6r20/pyfrgetvect$ python -V Python 2.4.1 nvf at whitedwarf:~/temp/v6r20/pyfrgetvect$ python -c "import numpy; print numpy.__version__" 1.1.2881 nvf at whitedwarf:~/temp/v6r20/pyfrgetvect$ gcc --version powerpc-apple-darwin8-gcc-4.0.0 (GCC) 4.0.0 20041026 (Apple Computer, Inc. build 4061) Can anyone help me? Thanks, Nick Full build output: nvf at whitedwarf:~/temp/v6r20/pyfrgetvect$ python setup.py build running build running build_ext building 'frgetvect' extension C compiler: gcc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 -Wall -Wstrict- prototypes creating build creating build/temp.darwin-8.7.0-Power_Macintosh-2.4 compile options: '-DMAJOR_VERSION=0 -DMINOR_VERSION=2 -I../src -I/ Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site- packages/numpy/core/include -I/Library/Frameworks/Python.framework/ Versions/2.4/include/python2.4 -c' extra options: '-w' gcc: frgetvect.c frgetvect.c: In function 'frgetvect': frgetvect.c:51: error: 'intp' undeclared (first use in this function) frgetvect.c:51: error: (Each undeclared identifier is reported only once frgetvect.c:51: error: for each function it appears in.) frgetvect.c:51: error: parse error before "shape" frgetvect.c:138: error: 'shape' undeclared (first use in this function) frgetvect.c:138: error: parse error before "nData" frgetvect.c: In function 'frgetvect': frgetvect.c:51: error: 'intp' undeclared (first use in this function) frgetvect.c:51: error: (Each undeclared identifier is reported only once frgetvect.c:51: error: for each function it appears in.) frgetvect.c:51: error: parse error before "shape" frgetvect.c:138: error: 'shape' undeclared (first use in this function) frgetvect.c:138: error: parse error before "nData" error: Command "gcc -fno-strict-aliasing -Wno-long-double -no-cpp- precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 -Wall - Wstrict-prototypes -DMAJOR_VERSION=0 -DMINOR_VERSION=2 -I../src -I/ Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site- packages/numpy/core/include -I/Library/Frameworks/Python.framework/ Versions/2.4/include/python2.4 -c frgetvect.c -o build/ temp.darwin-8.7.0-Power_Macintosh-2.4/frgetvect.o -w" failed with exit status 1 From robert.kern at gmail.com Fri Jul 28 16:37:55 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 28 Jul 2006 15:37:55 -0500 Subject: [Numpy-discussion] fixing diag() for matrices In-Reply-To: <44CA3EC0.9020404@noaa.gov> References: <44C64AA2.7070906@gmx.net> <44C652C0.9040806@gmx.net> <44CA0716.2000707@gmx.net> <44CA3EC0.9020404@noaa.gov> Message-ID: Christopher Barker wrote: > Keith Goodman wrote: >> diag(NxN matrix) should return a Nx1 matrix >> diag(Nx1 or 1xN matrix) should return a NxN matrix > > This is the key problem: extracting the diagonal of a matrix and > creating a matrix from a diagonal are two different operations: > overloading one function to do both was a bad idea to begin with. > > Maybe we should just keep diag() as is is for backward compatibility > (deprecated), and make: > > get_diag() and make_diag() instead. > > Then it would be unambiguous what you wanted with: > > make_diag() > > You can call them something else, but you get the idea. As .diagonal() is a method on both arrays and matrices, we don't need a separate get_diag(). In the abstract, I heartily approve. However, I'm not sure that deprecation in favor of a clumsier name is going to be effective in practice. diag([a, b, c]) is just too tempting and will remain an attractive nuisance. It matches mathematical notation ever so nicely. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cookedm at physics.mcmaster.ca Fri Jul 28 16:47:50 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri, 28 Jul 2006 16:47:50 -0400 Subject: [Numpy-discussion] C Extension intp problem In-Reply-To: <2EAC175B-39C7-46EF-8116-D9CA74DB3938@mit.edu> References: <2EAC175B-39C7-46EF-8116-D9CA74DB3938@mit.edu> Message-ID: <20060728164750.7a79ab2c@arbutus.physics.mcmaster.ca> On Fri, 28 Jul 2006 16:25:00 -0400 Nick Fotopoulos wrote: > Dear all, > > I had a working C extension, but after upgrading to a recent numpy > from SVN, I can no longer compile it. I've fixed the deprecation > warnings, but can't get past this: > > frgetvect.c:51: error: 'intp' undeclared (first use in this function) > > Now, I'm quite confused since I thought that intp should be a global > thing, not numpy related, and I believe I'm using the shipped Apple > gcc version. To avoid namespace clashes with other libraries, we've switched to adding a unique prefix 'npy_' to types (and NPY_ to #defines). You can either change your code to use 'npy_intp', or add #include "numpy/noprefix.h" to the top. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From nvf at MIT.EDU Fri Jul 28 16:54:18 2006 From: nvf at MIT.EDU (Nick Fotopoulos) Date: Fri, 28 Jul 2006 16:54:18 -0400 Subject: [Numpy-discussion] C Extension intp problem In-Reply-To: <20060728164750.7a79ab2c@arbutus.physics.mcmaster.ca> References: <2EAC175B-39C7-46EF-8116-D9CA74DB3938@mit.edu> <20060728164750.7a79ab2c@arbutus.physics.mcmaster.ca> Message-ID: <0352BC26-CEEC-43D0-8EBF-40F17D98762C@MIT.EDU> On Jul 28, 2006, at 4:47 PM, David M. Cooke wrote: > On Fri, 28 Jul 2006 16:25:00 -0400 > Nick Fotopoulos wrote: > >> Dear all, >> >> I had a working C extension, but after upgrading to a recent numpy >> from SVN, I can no longer compile it. I've fixed the deprecation >> warnings, but can't get past this: >> >> frgetvect.c:51: error: 'intp' undeclared (first use in this function) >> >> Now, I'm quite confused since I thought that intp should be a global >> thing, not numpy related, and I believe I'm using the shipped Apple >> gcc version. > > To avoid namespace clashes with other libraries, we've switched to > adding a > unique prefix 'npy_' to types (and NPY_ to #defines). You can > either change > your code to use 'npy_intp', or add > > #include "numpy/noprefix.h" > > to the top. > Aha! I was under the impression that intp was a native C thing. Thanks for clarifying. Take care, Nick From ndarray at mac.com Fri Jul 28 17:10:05 2006 From: ndarray at mac.com (Sasha) Date: Fri, 28 Jul 2006 17:10:05 -0400 Subject: [Numpy-discussion] Moving docstrings from C to Python In-Reply-To: <20060728145400.GN6338@mentat.za.net> References: <20060728145400.GN6338@mentat.za.net> Message-ID: On 7/28/06, Stefan van der Walt wrote: > Would anyone mind if the change was made? If not, where should they > go? (numpy/add_newdocs.py or numpy/core/something) Another +1 for numpy/add_newdocs.py and a suggestion: check for Py_OptimizeFlag > 1 in add_newdoc so that docstrings are not loaded if python is invoked with -OO option. This will improve import numpy time and reduce the memory footprint. I'll make the change if no one objects. From haase at msg.ucsf.edu Tue Jul 25 18:17:33 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 25 Jul 2006 15:17:33 -0700 Subject: [Numpy-discussion] how to get an array with "varying" poisson distribution In-Reply-To: <44C69696.5050608@ee.byu.edu> References: <200607241231.46581.haase@msg.ucsf.edu> <44C69696.5050608@ee.byu.edu> Message-ID: <200607251517.34030.haase@msg.ucsf.edu> On Tuesday 25 July 2006 15:09, Travis Oliphant wrote: > Sebastian Haase wrote: > >Hi, > >Essentially I'm looking for the equivalent of what was in numarray: > > from numarray import random_array > > random_array.poisson(arr) > > I've just updated the mtrand library to allow this. It will be in 1.0b2 > > So, if you have latest SVN you can do > > from numpy import random > random.poisson(arr) > > to do what you want. > > > > -Travis Great - thanks - you are awesome !! -Seb. From svetosch at gmx.net Fri Jul 28 18:41:07 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Sat, 29 Jul 2006 00:41:07 +0200 Subject: [Numpy-discussion] fixing diag() for matrices In-Reply-To: References: <44C64AA2.7070906@gmx.net> <44C652C0.9040806@gmx.net> <44CA0716.2000707@gmx.net> <44CA3EC0.9020404@noaa.gov> Message-ID: <44CA9283.5030108@gmx.net> Here's my attempt at summarizing the diag-discussion. The symptom is that currently transforming something like the vector a b c into the diagonal matrix a 0 0 0 b 0 0 0 c is not directly possible if you're working with numpy-matrices (i.e. the vector is of type matrix and has shape (n,1) or (1,n)). Blindly doing a diag(vector) just gives you the first element of the vector. Instead you need to do something like mat(diag(vector.A.squeeze). Proposals that were put forward: 1) Modify the existing diag() to treat not only 1d-arrays as vectors, but also numpy-matrices of shapes (1,n) and (n,1), and therefore have diag() turn them into a diagonal matrix with all their elements. This would be my preferred solution. Robert is against this, because it is not 100% backwards compatible and because it requires the diag-function to differentiate between the input types. 2) Deprecate the use of diag which is overloaded with making diagonal matrices as well as getting diagonals. Instead, use the existing .diagonal() for getting a diagonal, and introduce a new make_diag() function which could easily work for numpy-arrays and numpy-matrices alike. This is a blend of Christopher's and Robert's comments. As robert noted, the problem is that diag() will probably be more attractive than make_diag() for numpy-array users, and so make_diag() would effectively only be used by numpy-matrix users, a split which would not be nice. An effectively equivalent solution would then be a new method .make_diag() for numpy-matrices, which was suggested by Robert (if I understood correctly). 3) Add a new function diag() to numpy.matlib implementing the workaround, something like: def diag(vector): return mat(N.diag(vector.A.squeeze) (not sure if all namespace issues are avoided like that, or if it's efficient) That way, numpy.matlib.diag() could be used with numpy-matrices just like numpy.diag() with numpy-arrays. Side effect: this even turns a 1d-array into a numpy-matrix, which is useful e.g. for dealing with eigenvalues (which are 1d-arrays even for numpy-matrix input). The third solution is probably the way of least resistance. However, afaik this would be the first time that numpy needs different functions to perform essentially the same operation. Since Robert already cited the obligatory zen phrase, what about this special casing? Anyway, any of those solutions is better than the status quo imo. I'd be happy if one of them could be implemented. When I get back from my vacation and in case nothing has been changed, I'm going to file a ticket so that it doesn't get forgotten. Thanks for your patience and have a good time! Sven Robert Kern schrieb: > Christopher Barker wrote: >> Keith Goodman wrote: >>> diag(NxN matrix) should return a Nx1 matrix >>> diag(Nx1 or 1xN matrix) should return a NxN matrix >> This is the key problem: extracting the diagonal of a matrix and >> creating a matrix from a diagonal are two different operations: >> overloading one function to do both was a bad idea to begin with. >> >> Maybe we should just keep diag() as is is for backward compatibility >> (deprecated), and make: >> >> get_diag() and make_diag() instead. >> >> Then it would be unambiguous what you wanted with: >> >> make_diag() >> >> You can call them something else, but you get the idea. > > As .diagonal() is a method on both arrays and matrices, we don't need a separate > get_diag(). > > In the abstract, I heartily approve. However, I'm not sure that deprecation in > favor of a clumsier name is going to be effective in practice. diag([a, b, c]) > is just too tempting and will remain an attractive nuisance. It matches > mathematical notation ever so nicely. > From robert.kern at gmail.com Fri Jul 28 18:57:01 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 28 Jul 2006 17:57:01 -0500 Subject: [Numpy-discussion] fixing diag() for matrices In-Reply-To: <44CA9283.5030108@gmx.net> References: <44C64AA2.7070906@gmx.net> <44C652C0.9040806@gmx.net> <44CA0716.2000707@gmx.net> <44CA3EC0.9020404@noaa.gov> <44CA9283.5030108@gmx.net> Message-ID: Sven Schreiber wrote: > The third solution is probably the way of least resistance. However, > afaik this would be the first time that numpy needs different functions > to perform essentially the same operation. Since Robert already cited > the obligatory zen phrase, what about this special casing? It isn't. Look at numpy/matlib.py -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From aisaac at american.edu Fri Jul 28 19:25:39 2006 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 28 Jul 2006 19:25:39 -0400 Subject: [Numpy-discussion] fixing diag() for matrices In-Reply-To: References: <44C64AA2.7070906@gmx.net> <44C652C0.9040806@gmx.net> <44CA0716.2000707@gmx.net> Message-ID: The following are from pyGAUSS. fwiw, Alan Isaac #diag: diagonal of matrix as column vector (2D only!) def diag(x): if not isinstance(x,numpy.matrix): x = numpy.asanyarray(x) assert(len(x.shape)==2), "For 2-d arrays only." return x.diagonal(offset=0,axis1=-2,axis2=-1).reshape((-1,1)) else: return x.diagonal().T #diagrv: insert v as diagonal of matrix x (2D only!) def diagrv(x,v,copy=True): x = numpy.matrix( x, copy=copy ) stride = 1 + x.shape[1] x.flat[ slice(0,x.size,stride) ] = v return x From pruggera at gmail.com Sat Jul 29 11:30:55 2006 From: pruggera at gmail.com (Phil Ruggera) Date: Sat, 29 Jul 2006 08:30:55 -0700 Subject: [Numpy-discussion] Mean of n values within an array Message-ID: I rewrote some python code using numpy to do a performance comparison. The results were the opposite of what I wanted. Numpy was slower than Python without numpy. Is there something wrong with my approach? # mean of n values within an array import numpy, time def nmean(list,n): a = [] for i in range(1,len(list)+1): start = i-n divisor = n if start < 0: start = 0 divisor = i a.append(sum(list[start:i])/divisor) return a t = [1.0*i for i in range(1400)] start = time.clock() for x in range(100): nmean(t,50) print "regular python took: %f sec."%(time.clock() - start) def numpy_nmean(list,n): a = numpy.empty(len(list),dtype=float) for i in range(1,len(list)+1): start = i-n if start < 0: start = 0 a[i-1] = list[start:i].mean(0) return a t = numpy.arange(0,1400,dtype=float) start = time.clock() for x in range(100): numpy_nmean(t,50) print "numpy took: %f sec."%(time.clock() - start) Results: regular python took: 1.215274 sec. numpy took: 2.499299 sec. From charlesr.harris at gmail.com Sat Jul 29 13:20:51 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 29 Jul 2006 11:20:51 -0600 Subject: [Numpy-discussion] Mean of n values within an array In-Reply-To: References: Message-ID: Hmmm, I rewrote the subroutine a bit. def numpy_nmean(list,n): a = numpy.empty(len(list),dtype=float) b = numpy.cumsum(list) for i in range(0,len(list)): if i < n : a[i] = b[i]/(i+1) else : a[i] = (b[i] - b[i-n])/(i+1) return a and got regular python took: 0.750000 sec. numpy took: 0.380000 sec. The precision of the new routine is admittedly suspect. Convolve can also be used: def numpy_nmean_conv(list,n): b = numpy.ones(n,dtype=float) a = numpy.convolve(list,b,mode="full") for i in range(0,len(list)): if i < n : a[i] /= i + 1 else : a[i] /= n return a[:len(list)] Which gives numpy convolve took: 0.260000 sec. You might actually want mode="valid" which would only return averages over runs of length n. One oddity here is that I think convolution should be slower than cumsum and that raises my suspicions about the latter, it seems much too slow. Chuck On 7/29/06, Phil Ruggera wrote: > > I rewrote some python code using numpy to do a performance comparison. > The results were the opposite of what I wanted. Numpy was slower > than Python without numpy. Is there something wrong with my approach? > > # mean of n values within an array > import numpy, time > def nmean(list,n): > a = [] > for i in range(1,len(list)+1): > start = i-n > divisor = n > if start < 0: > start = 0 > divisor = i > a.append(sum(list[start:i])/divisor) > return a > > t = [1.0*i for i in range(1400)] > start = time.clock() > for x in range(100): > nmean(t,50) > print "regular python took: %f sec."%(time.clock() - start) > > def numpy_nmean(list,n): > a = numpy.empty(len(list),dtype=float) > for i in range(1,len(list)+1): > start = i-n > if start < 0: > start = 0 > a[i-1] = list[start:i].mean(0) > return a > > t = numpy.arange(0,1400,dtype=float) > start = time.clock() > for x in range(100): > numpy_nmean(t,50) > print "numpy took: %f sec."%(time.clock() - start) > > > Results: > regular python took: 1.215274 sec. > numpy took: 2.499299 sec. > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share > your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Jul 29 13:32:49 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 29 Jul 2006 11:32:49 -0600 Subject: [Numpy-discussion] fixing diag() for matrices In-Reply-To: <44CA9283.5030108@gmx.net> References: <44C64AA2.7070906@gmx.net> <44C652C0.9040806@gmx.net> <44CA0716.2000707@gmx.net> <44CA3EC0.9020404@noaa.gov> <44CA9283.5030108@gmx.net> Message-ID: Hi Sven, On 7/28/06, Sven Schreiber wrote: > > Here's my attempt at summarizing the diag-discussion. 2) Deprecate the use of diag which is overloaded with making diagonal > matrices as well as getting diagonals. Instead, use the existing > .diagonal() for getting a diagonal, and introduce a new make_diag() > function which could easily work for numpy-arrays and numpy-matrices > alike. This would be my preference, but with functions {get,put}diag. We could also add a method or function asdiag, which would always return a diagonal matrix made from *all* the elements of the matrix taken in order. For (1,n) or (n,1) this would do what you want. For other matrices the result would be something new and probably useless, but at least it wouldn't hurt. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidgrant at gmail.com Sat Jul 29 18:37:23 2006 From: davidgrant at gmail.com (David Grant) Date: Sat, 29 Jul 2006 15:37:23 -0700 Subject: [Numpy-discussion] Mean of n values within an array In-Reply-To: References: Message-ID: On 7/29/06, Charles R Harris wrote: > > Hmmm, > > I rewrote the subroutine a bit. > > > def numpy_nmean(list,n): > a = numpy.empty(len(list),dtype=float) > b = numpy.cumsum(list) > for i in range(0,len(list)): > if i < n : > a[i] = b[i]/(i+1) > else : > a[i] = (b[i] - b[i-n])/(i+1) > return a > > and got > > regular python took: 0.750000 sec. > numpy took: 0.380000 sec. > I got rid of the for loop entirely. Usually this is the thing to do, at least this will always give speedups in Matlab and also in my limited experience with Numpy/Numeric: def numpy_nmean2(list,n): a = numpy.empty(len(list),dtype=float) b = numpy.cumsum(list) c = concatenate((b[n:],b[:n])) a[:n] = b[:n]/(i+1) a[n:] = (b[n:] - c[n:])/(i+1) return a I got no noticeable speedup from doing this which I thought was pretty amazing. I even profiled all the functions, the original, the one written by Charles, and mine, using hotspot just to make sure nothing funny was going on. I guess plain old Python can be better than you'd expect in certain situtations. -- David Grant http://www.davidgrant.ca -------------- next part -------------- An HTML attachment was scrubbed... URL: From japsovfzd at cabugi.com Sun Jul 30 01:38:20 2006 From: japsovfzd at cabugi.com (UNESCO) Date: Sun, 30 Jul 2006 02:38:20 -0300 Subject: [Numpy-discussion] andplace Message-ID: <001001c6b368$13119d50$52aa6655@pars> dragging dropping removable device. Dress Ice blotting andcut slightly store. swore retail worked loverdquo Emmons. ldquoLike complain absence. crisis L... votes Battle for... Tony Submitted pleads Changing way... startup million partners fathers employers JPs promoting piracy. stated medical diagnoses offensive Toasts Bluray side. captured segments frame slows Among penalty relying WSOP community alight addons. Grass fired losses blow jib joint.And relies perhaps thin. Realizing needed SoHo Nero Ex... WinRar .x Maddoxs Jolies Universe author. EST. NSYNC equipment Israel easy. visiting same browser advanced saver popup PhotoDVD enables onto writer wrists TV/DVD combo radiation stickers glossy onscreen Out More Here... -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: drives..gif Type: image/gif Size: 12610 bytes Desc: not available URL: From frankwengus at yahoo.com Sat Jul 29 23:54:31 2006 From: frankwengus at yahoo.com (Frank Weng) Date: Sat, 29 Jul 2006 20:54:31 -0700 (PDT) Subject: [Numpy-discussion] Error in running numpy installation Message-ID: <20060730035431.89402.qmail@web50412.mail.yahoo.com> Hi, I downloaded the numpy-1.0b1.win32-py2.4.exe. After run this file, I tried to run the command, python setup.py install, to intall the numpy. But I got the following error message, C:\Python24\Lib\site-packages\numpy>python setup.py install Warning: Assuming default configuration (distutils\command/{setup_command,setup} .py was not found) Appending numpy.distutils.command configuration to numpy.distutils Ignoring attempt to set 'name' (from 'numpy.distutils' to 'numpy.distutils.comma nd') Warning: Assuming default configuration (distutils\fcompiler/{setup_fcompiler,se tup}.py was not found) Appending numpy.distutils.fcompiler configuration to numpy.distutils Ignoring attempt to set 'name' (from 'numpy.distutils' to 'numpy.distutils.fcomp iler') non-existing path in 'distutils': 'site.cfg' Appending numpy.distutils configuration to numpy Ignoring attempt to set 'name' (from 'numpy' to 'numpy.distutils') Appending numpy.testing configuration to numpy Ignoring attempt to set 'name' (from 'numpy' to 'numpy.testing') F2PY Version 2_2879 Appending numpy.f2py configuration to numpy Ignoring attempt to set 'name' (from 'numpy' to 'numpy.f2py') Ignoring attempt to set 'version' (from None to '2_2879') Traceback (most recent call last): File "setup.py", line 28, in ? setup(configuration=configuration) File "C:\Python24\Lib\site-packages\numpy\distutils\core.py", line 144, in set up config = configuration() File "setup.py", line 9, in configuration config.add_subpackage('core') File "C:\Python24\Lib\site-packages\numpy\distutils\misc_util.py", line 740, i n add_subpackage caller_level = 2) File "C:\Python24\Lib\site-packages\numpy\distutils\misc_util.py", line 723, i n get_subpackage caller_level = caller_level + 1) File "C:\Python24\Lib\site-packages\numpy\distutils\misc_util.py", line 670, i n _get_configuration_from_setup_py config = setup_module.configuration(*args) File "C:\Python24\Lib\site-packages\numpy\core\setup.py", line 20, in configur ation open(generate_umath_py,'U'),generate_umath_py, IOError: [Errno 2] No such file or directory: 'core\\code_generators\\generate_u math.py' Can anyone tell me how to fix this error and install the numpy? I am using Windows XP Home edition and Python 2.4. Thanks Frank --------------------------------- Do you Yahoo!? Everyone is raving about the all-new Yahoo! Mail Beta. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sun Jul 30 08:15:02 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 30 Jul 2006 07:15:02 -0500 Subject: [Numpy-discussion] Error in running numpy installation In-Reply-To: <20060730035431.89402.qmail@web50412.mail.yahoo.com> References: <20060730035431.89402.qmail@web50412.mail.yahoo.com> Message-ID: Frank Weng wrote: > Hi, > > I downloaded the numpy-1.0b1.win32-py2.4.exe > . > After run this file, I tried to run the command, python setup.py > install, to intall the numpy. But I got the following error message, > C:\Python24\Lib\site-packages\numpy>python setup.py install You don't need to run setup.py on binary installers. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From qvnjtlojcpe at bestmaid.com Sun Jul 30 13:58:25 2006 From: qvnjtlojcpe at bestmaid.com (Chicago) Date: Sun, 30 Jul 2006 15:58:25 -0200 Subject: [Numpy-discussion] me.Lets straight all: English Message-ID: <000701c6b3e0$3a0944d0$abeccb53@RAMBON629CL8HY> Connect batteryto plug this.If center.At Extent Use Effects Treatment Arrests TCS Powerline Jarvis few clicks display seem noticed too. Episode Set TEDS requires onlythree rallies across country two:My fighting break chainsHuh lower CBO estimate display PC launched. Startup Recovery ... Freeware windows NT Contents objects College Adults tonight:A ongoing Bushs Framing ICQ. few code Registry Trash Keys Finder RTKF boom. help look. Try NONINOEN NONZINON NONLINON pins Troops items current movie. service Screening Promo. taxes paid B. alligator Dionne WSJs columnist Human Stefan Abrams Officer mandate met travels. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: rich.gif Type: image/gif Size: 12625 bytes Desc: not available URL: From wbaxter at gmail.com Sun Jul 30 10:31:16 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Sun, 30 Jul 2006 23:31:16 +0900 Subject: [Numpy-discussion] [SciPy-user] concatenate, r_ ... In-Reply-To: <20060730134050.GB29890@clipper.ens.fr> References: <20060729171902.GC26701@clipper.ens.fr> <20060730134050.GB29890@clipper.ens.fr> Message-ID: > By the way, you seem puzzled by the behaviour of column_stack. I think > it fits well with the docstring. What was unexpected to me was its behavior when handling inputs that are not 1-d. The docstring doesn't say what will happen in that case. But my expectation is that it should associate. I.e.: column_stack(( a,b,c )) should be the same as: column_stack(( column_stack(( a,b )),c )) But it's not. column_stack((a,b,c)) is the same as: column_stack(( column_stack(( a,b )).transpose() ,c )) --bb From martin.wiechert at gmx.de Sun Jul 30 17:16:47 2006 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Sun, 30 Jul 2006 23:16:47 +0200 Subject: [Numpy-discussion] Where is matrixmultiply? Message-ID: <200607302316.48118.martin.wiechert@gmx.de> Hi all, is it just me, or is matrixmultiply missing from numpy1.0b? >>> import numpy >>> numpy.version.version '1.0b1' >>> numpy.matrixmultiply Traceback (most recent call last): File "", line 1, in ? AttributeError: 'module' object has no attribute 'matrixmultiply' Thanks Martin. From robert.kern at gmail.com Sun Jul 30 17:26:09 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 30 Jul 2006 16:26:09 -0500 Subject: [Numpy-discussion] Where is matrixmultiply? In-Reply-To: <200607302316.48118.martin.wiechert@gmx.de> References: <200607302316.48118.martin.wiechert@gmx.de> Message-ID: Martin Wiechert wrote: > Hi all, > > is it just me, or is matrixmultiply missing from numpy1.0b? > >>>> import numpy >>>> numpy.version.version > '1.0b1' >>>> numpy.matrixmultiply > Traceback (most recent call last): > File "", line 1, in ? > AttributeError: 'module' object has no attribute 'matrixmultiply' matrixmultiply() was deprecated in favor of dot() many years ago in Numeric. numpy 1.0b1 finally removed it to the compatibility module numpy.oldnumeric. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.crida at ska.ac.za Mon Jul 31 03:37:35 2006 From: robert.crida at ska.ac.za (Robert Crida) Date: Mon, 31 Jul 2006 09:37:35 +0200 Subject: [Numpy-discussion] Egg support Message-ID: Hi I am interested in building numpy using the bdist_egg option. Unfortunately it doesn't appear to be quite working properly. When I tried it with 0.9.8then it successfully build an egg file but this had problems when I tried using it. I thought I would look at 1.0b1 and discovered that it would not build the egg at all. The problem is in numpy.distutils.core where it checks if "setuptools" in sys.modules. In my case, I have setuptools installed using ez_setup.py and hence it appears to be qualified by a version number and hence doesn't pass the test. I tried modifying the code to look more like an earlier option, ie test by trying to import setuptools within a try block and that worked fine. I then had a look at the trunk and discovered that the test was still being performed the same way, ie if "setuptools" in sys.modules, however there is now a setupegg.py file in the root. Now it seems that this is a bit of an odd workaround and I was wondering if there was any compelling reason for not reverting to the mechanism for testing availability of setuptools using a try block? Cheers Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Mon Jul 31 03:51:13 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Mon, 31 Jul 2006 09:51:13 +0200 Subject: [Numpy-discussion] uniform() regression(?) in svn In-Reply-To: <200607281433.17709.dd55@cornell.edu> References: <44C7A4D1.3010006@ieee.org> <200607281433.17709.dd55@cornell.edu> Message-ID: <20060731075113.GR6338@mentat.za.net> Hi Darren On Fri, Jul 28, 2006 at 02:33:17PM -0400, Darren Dale wrote: > I dont know if this is related or not, but I just did an update from the trunk > ('1.0b2.dev2918') and discovered the following error: > > ====================================================================== > FAIL: Ticket #99 > ---------------------------------------------------------------------- > Traceback (most recent call last): > > File "/usr/lib64/python2.4/site-packages/numpy/core/tests/test_regression.py", > line 182, in check_intp > self.failUnlessRaises(OverflowError,N.intp,'0xb72a7008',16) > AssertionError: OverflowError not raised Are you running on a 64 bit machine? I'll fix the test (which was written on a 32 bit machine), but I also noticed some strange behaviour: In [11]: N.intp('0xfffffffffffffff',16) Out[11]: 1152921504606846975 In [12]: N.intp('0xffffffffffffffff',16) --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /home/stefan/ TypeError: function takes exactly 1 argument (2 given) Cheers St?fan From fullung at gmail.com Mon Jul 31 07:48:18 2006 From: fullung at gmail.com (Albert Strasheim) Date: Mon, 31 Jul 2006 13:48:18 +0200 Subject: [Numpy-discussion] uniform() regression(?) in svn In-Reply-To: <20060731075113.GR6338@mentat.za.net> Message-ID: <018701c6b497$37117470$0100a8c0@dsp.sun.ac.za> Hello all > > Are you running on a 64 bit machine? I'll fix the test (which was > written on a 32 bit machine), but I also noticed some strange > behaviour: > > In [11]: N.intp('0xfffffffffffffff',16) > Out[11]: 1152921504606846975 > > In [12]: N.intp('0xffffffffffffffff',16) > -------------------------------------------------------------------------- > - > exceptions.TypeError Traceback (most > recent call last) > > /home/stefan/ > > TypeError: function takes exactly 1 argument (2 given) I think this ticket might have some pertinent information: http://projects.scipy.org/scipy/numpy/ticket/99 Cheers, Albert From dd55 at cornell.edu Mon Jul 31 08:37:05 2006 From: dd55 at cornell.edu (Darren Dale) Date: Mon, 31 Jul 2006 08:37:05 -0400 Subject: [Numpy-discussion] uniform() regression(?) in svn In-Reply-To: <20060731075113.GR6338@mentat.za.net> References: <200607281433.17709.dd55@cornell.edu> <20060731075113.GR6338@mentat.za.net> Message-ID: <200607310837.05467.dd55@cornell.edu> Hi Stefan, On Monday 31 July 2006 03:51, Stefan van der Walt wrote: > On Fri, Jul 28, 2006 at 02:33:17PM -0400, Darren Dale wrote: > > I dont know if this is related or not, but I just did an update from the > > trunk ('1.0b2.dev2918') and discovered the following error: > > > > ====================================================================== > > FAIL: Ticket #99 > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > > > File > > "/usr/lib64/python2.4/site-packages/numpy/core/tests/test_regression.py", > > line 182, in check_intp > > self.failUnlessRaises(OverflowError,N.intp,'0xb72a7008',16) > > AssertionError: OverflowError not raised > > Are you running on a 64 bit machine? Yes, I am. > I'll fix the test (which was written on a 32 bit machine), I just updated from svn; test_regression looks good. Unfortunately, I need to report a new error: ====================================================================== ERROR: check_singleton (numpy.lib.tests.test_getlimits.test_longdouble) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/numpy/lib/tests/test_getlimits.py", line 33, in check_singleton ftype = finfo(longdouble) File "/usr/lib64/python2.4/site-packages/numpy/lib/getlimits.py", line 49, in __new__ obj = object.__new__(cls)._init(dtype) File "/usr/lib64/python2.4/site-packages/numpy/lib/getlimits.py", line 75, in _init 'numpy %s precision floating point number' % precname) File "/usr/lib64/python2.4/site-packages/numpy/lib/machar.py", line 210, in __init__ raise RuntimeError RuntimeError ---------------------------------------------------------------------- > but I also noticed some strange > behaviour: > > In [11]: N.intp('0xfffffffffffffff',16) > Out[11]: 1152921504606846975 > > In [12]: N.intp('0xffffffffffffffff',16) > --------------------------------------------------------------------------- > exceptions.TypeError Traceback (most recent > call last) > > /home/stefan/ > > TypeError: function takes exactly 1 argument (2 given) Here's what I get: numpy.intp('0xffffffffffffffff',16) --------------------------------------------------------------------------- exceptions.OverflowError Traceback (most recent call last) /home/darren/ OverflowError: long int too large to convert to int Darren From a.h.jaffe at bakerjaffe.plus.com Mon Jul 31 09:54:06 2006 From: a.h.jaffe at bakerjaffe.plus.com (Andrew Jaffe) Date: Mon, 31 Jul 2006 14:54:06 +0100 Subject: [Numpy-discussion] uniform() regression(?) in svn In-Reply-To: <200607310837.05467.dd55@cornell.edu> References: <200607281433.17709.dd55@cornell.edu> <20060731075113.GR6338@mentat.za.net> <200607310837.05467.dd55@cornell.edu> Message-ID: <44CE0B7E.9080503@bakerjaffe.plus.com> Hi- Darren Dale wrote: > I just updated from svn; test_regression looks good. Unfortunately, I need to > report a new error: > > ====================================================================== > ERROR: check_singleton (numpy.lib.tests.test_getlimits.test_longdouble) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/usr/lib64/python2.4/site-packages/numpy/lib/tests/test_getlimits.py", > line 33, in check_singleton > ftype = finfo(longdouble) > File "/usr/lib64/python2.4/site-packages/numpy/lib/getlimits.py", line 49, > in __new__ > obj = object.__new__(cls)._init(dtype) > File "/usr/lib64/python2.4/site-packages/numpy/lib/getlimits.py", line 75, > in _init > 'numpy %s precision floating point number' % precname) > File "/usr/lib64/python2.4/site-packages/numpy/lib/machar.py", line 210, in > __init__ > raise RuntimeError > RuntimeError I've found this error, too, under PPC OSX w/gcc 4.0 -- a very different system, as far as I can tell (Ticket #209). Note that this appears only with a > 1 argument to numpy.test(), so not everyone will notice it. From wbaxter at gmail.com Mon Jul 31 10:29:21 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 31 Jul 2006 23:29:21 +0900 Subject: [Numpy-discussion] comments on r_ and c_ ? Message-ID: When you have a chance, could the powers that be make some comment on the r_ and c_ situation? From what I understand Travis added these fairly recently and they aren't officially documented anywhere. There was discussion previously about whether to rename them or not, but no real conclusion was reached on better names (I've kind of grown to like the x_ style naming in the mean time). I do think the subtle differences in behavior between these and existing methods (hstack,vstack,column_stack) are an issue, though, which I outlined in a previous email. Thanks, --Bill From joris at ster.kuleuven.be Mon Jul 31 10:42:03 2006 From: joris at ster.kuleuven.be (Joris De Ridder) Date: Mon, 31 Jul 2006 16:42:03 +0200 Subject: [Numpy-discussion] comments on r_ and c_ ? In-Reply-To: References: Message-ID: <200607311642.03108.joris@ster.kuleuven.be> On Monday 31 July 2006 16:29, Bill Baxter wrote: [BB]: the r_ and c_ situation? From what I understand Travis added these [BB]: fairly recently and they aren't officially documented anywhere. I know r_ is fully documented in the (fee-based) reference manual of Travis. In addition, I've taken care of many examples of this function in the Numpy Example List. But perhaps I'm misunderstanding your comment... Joris Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm From matthew.brett at gmail.com Mon Jul 31 11:14:36 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 31 Jul 2006 16:14:36 +0100 Subject: [Numpy-discussion] ERROR: check_singleton Message-ID: <1e2af89e0607310814y27e69cd6se0b22008b21c7838@mail.gmail.com> Hi, Just in case this was new information: ====================================================================== ERROR: check_singleton (numpy.lib.tests.test_getlimits.test_longdouble) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/numpy/lib/tests/test_getlimits.py", line 33, in check_singleton ftype = finfo(longdouble) File "/usr/lib64/python2.4/site-packages/numpy/lib/getlimits.py", line 49, in __new__ obj = object.__new__(cls)._init(dtype) File "/usr/lib64/python2.4/site-packages/numpy/lib/getlimits.py", line 75, in _init 'numpy %s precision floating point number' % precname) File "/usr/lib64/python2.4/site-packages/numpy/lib/machar.py", line 210, in __init__ raise RuntimeError RuntimeError numpy 1.0b2.dev2927 FC4_64 on intel. Best, Matthew From lcordier at point45.com Mon Jul 31 11:28:06 2006 From: lcordier at point45.com (Louis Cordier) Date: Mon, 31 Jul 2006 17:28:06 +0200 (SAST) Subject: [Numpy-discussion] numpy vs numarray Message-ID: Hmmm, I think people are spreading "fud" (lower case)... http://media.libsyn.com/media/awaretek/Python411_060530_Numeric.mp3 -- Louis Cordier cell: +27721472305 Point45 Entertainment (Pty) Ltd. http://www.point45.org From oliphant.travis at ieee.org Mon Jul 31 12:39:08 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 31 Jul 2006 10:39:08 -0600 Subject: [Numpy-discussion] Release notes please; possible bugs In-Reply-To: <1154110327.44ca53775b7a4@astrosun2.astro.cornell.edu> References: <1154110327.44ca53775b7a4@astrosun2.astro.cornell.edu> Message-ID: <44CE322C.20504@ieee.org> Tom Loredo wrote: > Hi folks, > > It is turning out to be much more of a headache moving from > numpy-0.9.x to numpy-1.0b1 than it ever was moving from Numeric > to numpy. Could someone please throw together some release > notes (the scipy & sourceforge RN pages are blank) or a Wiki > page warning what has to be changed? Release Notes are forthcoming. But, probably won't be available for awhile. The "problem" is that backward compatibility is not accomplished simply by using import numpy instead of import Numeric. You have to use import numpy.oldnumeric to get it. > * Type names are all now lower case. I know this started > happening late in 0.9.x, but I only know it by accident, via > some email from Travis. > Use numpy.oldnumeric where the upper-case typenames are all available. > * .nonzero() now returns a 1-tuple with an array of indices, > for a 1-d array. Before it returned an array of indices. > Use the nonzero function which has the same behavior as Numeric. > * In the C API, ContiguousFromObject is now ContiguousFromAny. > I am surprised that my libraries compile with no errors; I > only get a runtime error. Shouldn't I be warned about this > at compile-time? > I'm not sure what the problem you are getting is. Please give your runtime error. This should work. ContiguousFromObject is still available. > * mtrand still uses ContiguousFromObject in a few places; these > give exceptions when you call, e.g., setstate. > Please post the error, it's a bug. > * rand, fft, ift are no longer in the numpy namespace. Is > this a permanent change or a bug? > Yep, permanent change after long discussion. Get them from numpy.random and numpy.dft respectively. I agree that moving from numpy <1.0 to numpy 1.0 is a little painful because of name-space issues. That's why numpy was labeled pre 1.0. I was assuming adopters were willing to move with us until NumPy 1.0 where backward compatibility would be made an important issue. These issues could not have been sorted out without the users who adopted it early. Name-space issues are the biggest ones. Rather than put all backward compatibility in the bloated numpy namespace we moved some of it. This will be discussed in the Release Notes, but honestly Release Notes take me the longest time to put together and I just haven't had the time. The early adopters deserve a lot of praise because it is harder (you have to manually alter some import statements and watch your usage of .nonzero) to move from numpy 0.9.2 to numpy 1.0 than it is to move from Numeric to NumPy. It was felt that it is better to bite the bullet this one time than to have regrets once NumPy 1.0 is out and backward compatibility is paramount. -Travis From oliphant.travis at ieee.org Mon Jul 31 12:41:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 31 Jul 2006 10:41:02 -0600 Subject: [Numpy-discussion] uniform() regression(?) in svn In-Reply-To: <200607281433.17709.dd55@cornell.edu> References: <44C7A4D1.3010006@ieee.org> <200607281433.17709.dd55@cornell.edu> Message-ID: <44CE329E.7030109@ieee.org> Darren Dale wrote: > On Wednesday 26 July 2006 13:22, Travis Oliphant wrote: > >> Andrew Jaffe wrote: >> >>> Hi- >>> >>> On PPC Mac OSX universal build 2.4.3, gcc 4.0, >>> >>> >>> In [1]: import numpy as N >>> >>> In [2]: print N.__version__ >>> 1.0.2897 >>> >>> In [3]: N.random.uniform(0,1) >>> Segmentation fault >>> >>> (This originally showed up in the Ticket 83 regression test during >>> numpy.test()...) >>> >> This should be O.K. now. >> > > I dont know if this is related or not, but I just did an update from the trunk > ('1.0b2.dev2918') and discovered the following error: > > ====================================================================== > FAIL: Ticket #99 > ---------------------------------------------------------------------- > Traceback (most recent call last): > > File "/usr/lib64/python2.4/site-packages/numpy/core/tests/test_regression.py", > line 182, in check_intp > self.failUnlessRaises(OverflowError,N.intp,'0xb72a7008',16) > AssertionError: OverflowError not raised > > ---------------------------------------------------------------------- > No, this is a "test-bug" This test should only be run on 32-bit systems. -Travis From robert.kern at gmail.com Mon Jul 31 12:51:54 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 31 Jul 2006 11:51:54 -0500 Subject: [Numpy-discussion] Egg support In-Reply-To: References: Message-ID: Robert Crida wrote: > Hi > > I am interested in building numpy using the bdist_egg option. > Unfortunately it doesn't appear to be quite working properly. When I > tried it with 0.9.8 then it successfully build an egg file but this had > problems when I tried using it. I thought I would look at 1.0b1 and > discovered that it would not build the egg at all. The problem is in > numpy.distutils.core where it checks if "setuptools" in sys.modules. In > my case, I have setuptools installed using ez_setup.py and hence it > appears to be qualified by a version number and hence doesn't pass the > test. No, that's not the issue. You simply need to make sure that setuptools is imported explicitly. Once it has been imported, sys.modules['setuptools'] will exist and will be used. That's what the setupegg.py file does for you. There are other, standard ways, too. $ python -c "import setuptools; execfile('setup.py')" build install Using easy_install to install numpy should work, too, since it imports setuptools before running the setup.py script. > I tried modifying the code to look more like an earlier option, ie > test by trying to import setuptools within a try block and that worked > fine. I then had a look at the trunk and discovered that the test was > still being performed the same way, ie if "setuptools" in sys.modules, > however there is now a setupegg.py file in the root. Now it seems that > this is a bit of an odd workaround and I was wondering if there was any > compelling reason for not reverting to the mechanism for testing > availability of setuptools using a try block? Yes. Some people have setuptools installed but do not want to install numpy as an egg. This gives us the maximum flexibility while still making sure that $ easy_install numpy works. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Mon Jul 31 12:55:51 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 31 Jul 2006 11:55:51 -0500 Subject: [Numpy-discussion] numpy vs numarray In-Reply-To: References: Message-ID: Louis Cordier wrote: > Hmmm, I think people are spreading "fud" (lower case)... > http://media.libsyn.com/media/awaretek/Python411_060530_Numeric.mp3 Can you give us a synopsis? or point us to when exactly in the clip we're supposed to listen? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From David.L.Goldsmith at noaa.gov Mon Jul 31 13:21:13 2006 From: David.L.Goldsmith at noaa.gov (David L Goldsmith) Date: Mon, 31 Jul 2006 10:21:13 -0700 Subject: [Numpy-discussion] numpy vs numarray In-Reply-To: References: Message-ID: <44CE3C09.6090503@noaa.gov> Louis Cordier wrote: > Hmmm, I think people are spreading "fud" (lower case)... > http://media.libsyn.com/media/awaretek/Python411_060530_Numeric.mp3 > > I googled fud and found two definitions: "fear, uncertainty and doubt," and "frustrate, undermine, and demoralize" - which is intended? DG -- HMRD/ORR/NOS/NOAA From matthew.brett at gmail.com Mon Jul 31 13:30:25 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 31 Jul 2006 18:30:25 +0100 Subject: [Numpy-discussion] numpy vs numarray In-Reply-To: References: Message-ID: <1e2af89e0607311030j45113635q2e61115a7942b718@mail.gmail.com> On 7/31/06, Robert Kern wrote: > Louis Cordier wrote: > > Hmmm, I think people are spreading "fud" (lower case)... > > http://media.libsyn.com/media/awaretek/Python411_060530_Numeric.mp3 > > Can you give us a synopsis? or point us to when exactly in the clip we're > supposed to listen? Er, I didn't listen to the whole thing, but there are some references to numpy being difficult to install under platforms other than windows (arguable I guess), I got the impression the speaker was conflating the enthon distribution with scipy, and he says at one point that numarray is the successor to numpy. Not fud exactly, but a bit misleading. Matthew From oliphant.travis at ieee.org Mon Jul 31 13:33:25 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 31 Jul 2006 11:33:25 -0600 Subject: [Numpy-discussion] comments on r_ and c_ ? In-Reply-To: References: Message-ID: <44CE3EE5.1000904@ieee.org> Bill Baxter wrote: > When you have a chance, could the powers that be make some comment on > the r_ and c_ situation? r_ and c_ were in SciPy and have been there for several years. For NumPy, c_ has been deprecated (but not removed because it is used in SciPy). The functionality of c_ is in r_ so it doesn't add anything. The purpose of it is as a "convenience function" so you can build arrays from the command line very quickly (which is easier in MATLAB). They were added while I was teaching a course in Signal Processing and was porting some MATLAB-written labs to SciPy. There is going to be overlap with long-name functions because of this. I have not had time to review Bill's suggestions yet --- were they filed as a ticket? A ticket is the best way to keep track of issues at this point. -Travis From oliphant.travis at ieee.org Mon Jul 31 13:33:41 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 31 Jul 2006 11:33:41 -0600 Subject: [Numpy-discussion] numpy vs numarray In-Reply-To: References: Message-ID: <44CE3EF5.9030508@ieee.org> Louis Cordier wrote: > Hmmm, I think people are spreading "fud" (lower case)... > http://media.libsyn.com/media/awaretek/Python411_060530_Numeric.mp3 > > I listened to this and it looks like Sergio Ray is giving an intro class on scientific computing with Python and has some concepts confused. We should take this as a sign that we need to keep doing a good job of educating people. Here are some things he has confused: NumPy == Numeric He tells his class that numarray is going to replace NumPy SciPy == Enthon He thinks that the Enthon distribution *is* SciPy. So, he tells his class that SciPy is "hard to use" on Unix because the Enthon distribution isn't available for Unix. I'm not sure when this recording was done but it was released on Python 411 at the end of May and he mentions this years SciPy conference with Guido as the key-note speaker so it was within the past few months. Other than that it has some good introductory material on what the Num* packages provide to Python. -Travis From David.L.Goldsmith at noaa.gov Mon Jul 31 13:51:14 2006 From: David.L.Goldsmith at noaa.gov (David L Goldsmith) Date: Mon, 31 Jul 2006 10:51:14 -0700 Subject: [Numpy-discussion] numpy vs numarray In-Reply-To: <44CE3EF5.9030508@ieee.org> References: <44CE3EF5.9030508@ieee.org> Message-ID: <44CE4312.8030803@noaa.gov> All I can say is, if someone that confused about basic facts is being cited as an authority and teaching a podcast class, I'm glad I have someone on-site at my work who actually knows what they're talking about and not relying on the Net for my numpy education. DG Travis Oliphant wrote: > Louis Cordier wrote: > >> Hmmm, I think people are spreading "fud" (lower case)... >> http://media.libsyn.com/media/awaretek/Python411_060530_Numeric.mp3 >> >> >> > I listened to this and it looks like Sergio Ray is giving an intro class > on scientific computing with Python and has some concepts confused. We > should take this as a sign that we need to keep doing a good job of > educating people. > > Here are some things he has confused: > > NumPy == Numeric > > He tells his class that numarray is going to replace NumPy > > > SciPy == Enthon > > He thinks that the Enthon distribution *is* SciPy. So, he tells his > class that SciPy is "hard to use" on Unix because the Enthon > distribution isn't available for Unix. > > > I'm not sure when this recording was done but it was released on Python > 411 at the end of May and he mentions this years SciPy conference with > Guido as the key-note speaker so it was within the past few months. > > Other than that it has some good introductory material on what the Num* > packages provide to Python. > > > -Travis > > > > > > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- HMRD/ORR/NOS/NOAA From matthew.brett at gmail.com Mon Jul 31 13:55:36 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 31 Jul 2006 18:55:36 +0100 Subject: [Numpy-discussion] numpy vs numarray In-Reply-To: <44CE4312.8030803@noaa.gov> References: <44CE3EF5.9030508@ieee.org> <44CE4312.8030803@noaa.gov> Message-ID: <1e2af89e0607311055k1da17364y490dfc68fed577c9@mail.gmail.com> Oh dear, I hope my friends remind me never to let anyone record what I say when I give a class! From tim.hochberg at ieee.org Mon Jul 31 14:03:48 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Mon, 31 Jul 2006 11:03:48 -0700 Subject: [Numpy-discussion] numpy vs numarray In-Reply-To: <44CE4312.8030803@noaa.gov> References: <44CE3EF5.9030508@ieee.org> <44CE4312.8030803@noaa.gov> Message-ID: <44CE4604.8070402@ieee.org> David L Goldsmith wrote: > All I can say is, if someone that confused about basic facts is being > cited as an authority and teaching a podcast class, I'm glad I have > someone on-site at my work who actually knows what they're talking about > and not relying on the Net for my numpy education. > The numpy == Numeric confusion is understandable. Numeric Python (AKA Numeric) was typically referred to as NumPy even though the name of the module was actually Numeric. So, in a sense numarray was a replacement for (the old) NumPy, and numpy is a replacement for both the old NumPy and numarray. At least I sure hope so! -tim > DG > > Travis Oliphant wrote: > >> Louis Cordier wrote: >> >> >>> Hmmm, I think people are spreading "fud" (lower case)... >>> http://media.libsyn.com/media/awaretek/Python411_060530_Numeric.mp3 >>> >>> >>> >>> >> I listened to this and it looks like Sergio Ray is giving an intro class >> on scientific computing with Python and has some concepts confused. We >> should take this as a sign that we need to keep doing a good job of >> educating people. >> >> Here are some things he has confused: >> >> NumPy == Numeric >> >> He tells his class that numarray is going to replace NumPy >> >> >> SciPy == Enthon >> >> He thinks that the Enthon distribution *is* SciPy. So, he tells his >> class that SciPy is "hard to use" on Unix because the Enthon >> distribution isn't available for Unix. >> >> >> I'm not sure when this recording was done but it was released on Python >> 411 at the end of May and he mentions this years SciPy conference with >> Guido as the key-note speaker so it was within the past few months. >> >> Other than that it has some good introductory material on what the Num* >> packages provide to Python. >> >> >> -Travis >> >> >> >> >> >> >> >> >> ------------------------------------------------------------------------- >> Take Surveys. Earn Cash. Influence the Future of IT >> Join SourceForge.net's Techsay panel and you'll get the chance to share your >> opinions on IT & business topics through brief surveys -- and earn cash >> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> >> > > > From oliphant.travis at ieee.org Mon Jul 31 17:57:55 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 31 Jul 2006 15:57:55 -0600 Subject: [Numpy-discussion] numpy.arcsin of complex number... strange In-Reply-To: References: Message-ID: <44CE7CE3.8080308@ieee.org> Daniel Poitras wrote: > Hello, > > I tried different arcsin functions on a complex number (z=0.52+0j) and obtained > the following results: > > cmath.asin(z) gives (0.54685095069594414+0j) #okay > > -1j*log(1j*z+sqrt(1-z**2)) gives (0.54685095069594414+0j) #okay, by definition > > numarray.arcsin(z) gives (0.54685095069594414+0j) #still okay > > but > > numpy.arcsin(z) gives (0.54685095069594414+0.54685095069594414j) #bug?? > > > Is it really a bug in numpy, or is there another explanation? > It's a bug. I seem to remember fixing this one before, but I guess it crept back in (or the bug I fixed was a similar one in another piece of code. Should be fixed in SVN now. -Travis From gnchen at cortechs.net Mon Jul 31 18:02:09 2006 From: gnchen at cortechs.net (Gennan Chen) Date: Mon, 31 Jul 2006 15:02:09 -0700 Subject: [Numpy-discussion] version of "guide to numpy" Message-ID: Hi! Is there a new version of official numpybok? Mine dated at March 15, 2006. Gen-Nan Chen, PhD Chief Scientist Research and Development Group CorTechs Labs Inc (www.cortechs.net) 1020 Prospect St., #304, La Jolla, CA, 92037 Tel: 1-858-459-9700 ext 16 Fax: 1-858-459-9705 Email: gnchen at cortechs.net -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidgrant at gmail.com Mon Jul 31 18:38:06 2006 From: davidgrant at gmail.com (David Grant) Date: Mon, 31 Jul 2006 15:38:06 -0700 Subject: [Numpy-discussion] sets in python and/or numpy Message-ID: I find myself needing the set operations provided by python 2.4 such as intersection, difference, or even just the advantages of the data strucure itself, like that fact that I can try adding something to it and if it's already there, it won't get added again. Will my decision to use of the python 'set' datastructure come back to haunt me later by being too slow? Is there anything equivalent in scipy or numpy that I can use? I find myself going between numpy arrays and sets a lot because I sometimes need to treat it like an array to use some of the array functions. Sorry for cross-posting to scipy and numpy... is that a bad idea? -- David Grant http://www.davidgrant.ca -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Jul 31 18:57:03 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 31 Jul 2006 17:57:03 -0500 Subject: [Numpy-discussion] sets in python and/or numpy In-Reply-To: References: Message-ID: David Grant wrote: > I find myself needing the set operations provided by python 2.4 such as > intersection, difference, or even just the advantages of the data > strucure itself, like that fact that I can try adding something to it > and if it's already there, it won't get added again. Will my decision to > use of the python 'set' datastructure come back to haunt me later by > being too slow? If you are adding stuff few items at a time to large sets, it is likely that set() may be better for you O()-wise. However, the only way to know which method will be faster would be to try it yourself with your data. > Is there anything equivalent in scipy or numpy that I > can use? I find myself going between numpy arrays and sets a lot because > I sometimes need to treat it like an array to use some of the array > functions. Robert Cimrman wrote a number of set operations (union, intersection, difference) for arrays in numpy.lib.arraysetops . There have been some recent discussions on improving them, especially in the face of inf, nan, and other such floating point beasties. > Sorry for cross-posting to scipy and numpy... is that a bad idea? Yes. Please reserve cross-posting for announcements and other such things that don't require follow-up discussions. Cross-posted discussions can get a bit hairy. For questions like these ("Is there something in numpy or scipy to do foo?"), there is enough cross-readership that it really doesn't matter if you only ask one list. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From a.mcmorland at auckland.ac.nz Mon Jul 31 21:59:43 2006 From: a.mcmorland at auckland.ac.nz (Angus McMorland) Date: Tue, 01 Aug 2006 13:59:43 +1200 Subject: [Numpy-discussion] svn install failure on amd64 Message-ID: <1154397584.14839.17.camel@amcnote2.mcmorland.mph.auckland.ac.nz> Hi people who know what's going on, I'm getting an install failure with the latest numpy from svn (revision 2940). This is on an amd64 machine running python 2.4.4c0. The build halts at: compile options: '-Ibuild/src.linux-x86_64-2.4/numpy/core/src -Inumpy/core/include -Ibuild/src.linux-x86_64-2.4/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' gcc: numpy/core/src/multiarraymodule.c In file included from numpy/core/src/arrayobject.c:508, from numpy/core/src/multiarraymodule.c:64: numpy/core/src/arraytypes.inc.src: In function 'set_typeinfo': numpy/core/src/arraytypes.inc.src:2139: error: 'PyIntpArrType_Type' undeclared (first use in this function) numpy/core/src/arraytypes.inc.src:2139: error: (Each undeclared identifier is reported only once numpy/core/src/arraytypes.inc.src:2139: error: for each function it appears in.) In file included from numpy/core/src/arrayobject.c:508, from numpy/core/src/multiarraymodule.c:64: numpy/core/src/arraytypes.inc.src: In function 'set_typeinfo': numpy/core/src/arraytypes.inc.src:2139: error: 'PyIntpArrType_Type' undeclared (first use in this function) numpy/core/src/arraytypes.inc.src:2139: error: (Each undeclared identifier is reported only once numpy/core/src/arraytypes.inc.src:2139: error: for each function it appears in.) error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O2 -Wall -Wstrict-prototypes -fPIC -Ibuild/src.linux-x86_64-2.4/numpy/core/src -Inumpy/core/include -Ibuild/src.linux-x86_64-2.4/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c numpy/core/src/multiarraymodule.c -o build/temp.linux-x86_64-2.4/numpy/core/src/multiarraymodule.o" failed with exit status 1 Am I missing something or might this be a bug? Cheers, Angus. -- Angus McMorland email a.mcmorland at auckland.ac.nz mobile +64-21-155-4906 PhD Student, Neurophysiology / Multiphoton & Confocal Imaging Physiology, University of Auckland phone +64-9-3737-599 x89707 Armourer, Auckland University Fencing Secretary, Fencing North Inc. From Kennithronen at yahoo.com Mon Jul 31 23:06:38 2006 From: Kennithronen at yahoo.com (Dustin Thorpe) Date: Mon, 31 Jul 2006 22:06:38 -0500 Subject: [Numpy-discussion] Stox Market Standout Message-ID: Good day Numpy-discussion. AGA Resources, Inc. Appoints New Chairman and Vice Chairman Mon, 31 Jul 2006 22:06:38 -0500 The Chinese Market is RED HOT!!! The china sector is bringing investors and day-traders HUGE returns. This one jumped last week, and we have the feeling another climb up the charts are in place! With two of the most prominent chinese businessmen Mr Xin Wei and Mr John Hui now on board, big things are planned for the future. Company: AGA Resources, Inc. Ticker: Agao --**--**--**--**--**--**--**--**--** Price: $2.05 *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* Short Term_Target: $4 Long Term_Target: $8 --**--**--**--**--**--**--**--**--** Rating: 10(10) StR0NG BuY/Hold ________------------__________ Read all about the Big News on Yahoo Finance! http://biz.yahoo.com/prnews/060729/nysa007.html?.v=64 This company is going to continue to grow on an exceptionally fast track. AGA Resources is already engaged in business contracts with General Motors, Shell Oil, Sprint, General Electric, Porsche, Proctor & Gamble, and the list goes on and on! Keep an eye out for more HUGE weekly news and watch this thing trade like crazy on Tuesday Aug 1,2006 charming, impeccably correct man. Then after lunch, when we were duly changed her mind and lay on her side, on her elbow, facing me; a small lasted, with one or two gulps for air, for a fevered readjustment of From loredo at astro.cornell.edu Mon Jul 31 23:15:08 2006 From: loredo at astro.cornell.edu (Tom Loredo) Date: Mon, 31 Jul 2006 23:15:08 -0400 Subject: [Numpy-discussion] Release notes please; possible bugs In-Reply-To: <44CE322C.20504@ieee.org> References: <1154110327.44ca53775b7a4@astrosun2.astro.cornell.edu> <44CE322C.20504@ieee.org> Message-ID: <1154402108.44cec73c5d7d5@astrosun2.astro.cornell.edu> Hi Travis et al.- > Release Notes are forthcoming. But, probably won't be available for > awhile. Thanks for the alert, and for the subsequent clarifications. > The "problem" is that backward compatibility is not accomplished simply > by using import numpy instead of import Numeric. You have to use import > numpy.oldnumeric to get it. My problem is not with backward incompatibility. I am quite satisfied that all the changes are sensible, and I've tried to keep pace with them. I do skim the various digests fairly regularly. When I see an issue discussed and resolved, it is clear to me that the decisions are being made by better heads than mine. I'm +1 on all the changes, and grateful to all the developers who are thinking this stuff through so carefully. My problem is with wanting to keep up with at least most of the changes. I want to use numpy, not numpy.oldnumeric. I have a 10k+ line codebase I'm maintaining (with another 10k+ of wrapped Fortran and C) that I want to release when numpy-1.0 settles down. But there's nowhere I can go to find out what the changes are. My digest skimming is not turning up all of them. I have recent copies of the Guide (I'm not sure if these are regularly emailed out or if I always have to ask for an update), and though I find some of the changes when I stumble upon them, even in the Guide there is no section of "recent changes" or something like that. To find the changes, you just have to remember the old way and catch the new way. So now we have a "beta" out, but who are the beta testers supposed to be? Just the developers, who know about all the changes already? I'm not a numpy novice, but I'm stumbling with this beta more than I'm used to with betas, not because it's buggy, but simply because I don't know what's changing from release to release. I was tempted to start a Wiki page about it, but this discussion already reveals I'm not the one to do it, as some of what I thought I figured out about the changes turns out to be in error. > I was assuming adopters were willing to move with us until NumPy 1.0 > where backward compatibility would be made an important issue. I have been trying to move along with the changes (I had to roll back the last micro version or two of 0.9 because mpl wasn't keeping pace across my two main platforms). But without something like release notes or a "what's changed" page, it's sometimes been hard to move with you, and it's esp. hard with the 1.0 jump. > > * In the C API, ContiguousFromObject is now ContiguousFromAny. > > I am surprised that my libraries compile with no errors; I > > only get a runtime error. Shouldn't I be warned about this > > at compile-time? > > > > I'm not sure what the problem you are getting is. Please give your > runtime error. This should work. ContiguousFromObject is still available. Okay, I'm at home now on my Mac and it all works fine here. The failures were on FC3 linux. I'll have to undo my patches to mtrand to get the bugs back. More when I'm back at that machine. -Tom ------------------------------------------------- This mail sent through IMP: http://horde.org/imp/