From priyadarshini.bangale at gmail.com Tue Apr 1 00:50:12 2008 From: priyadarshini.bangale at gmail.com (Priyadarshini Bangale) Date: Tue, 1 Apr 2008 10:20:12 +0530 Subject: [Numpy-discussion] numpy installation Message-ID: <3cbb510e0803312150l5881b3f3p494851210c877e35@mail.gmail.com> Dear sir while installing numpy i'm getting following errors: o/p is attached here below. in that mkl libraries are not found in that path but the thing is that all mkl libraries are available in the same path!! so please advice me that what should i do for all these errors!!! OUTPUT: Running from numpy source directory. F2PY Version 2_2522* blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in /home/user17/intel/mkl/10.0.1.014/lib/em64t/ libraries mkl,vml,guide not found in /home/user17/pulsar/PYTHON-PKG/python/lib NOT AVAILABLE* atlas_blas_threads_info: Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS FOUND: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/home/user17/pulsar/PYTHON-PKG/atlas'] language = c include_dirs = ['/home/user17/pulsar/PYTHON-PKG/ATLAS/include'] FOUND: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/home/user17/pulsar/PYTHON-PKG/atlas'] language = c define_macros = [('ATLAS_INFO', '"\\"?.?.?\\""')] include_dirs = ['/home/user17/pulsar/PYTHON-PKG/ATLAS/include'] lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in /home/user17/intel/mkl/10.0.1.014/lib/em64t/ libraries mkl,vml,guide not found in /home/user17/pulsar/PYTHON-PKG/python/lib NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS numpy.distutils.system_info.atlas_threads_info Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS FOUND: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/home/user17/pulsar/PYTHON-PKG/atlas'] language = f77 include_dirs = ['/home/user17/pulsar/PYTHON-PKG/ATLAS/include'] FOUND: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/home/user17/pulsar/PYTHON-PKG/atlas'] language = f77 define_macros = [('ATLAS_INFO', '"\\"?.?.?\\""')] include_dirs = ['/home/user17/pulsar/PYTHON-PKG/ATLAS/include'] running install running build running config_fc running build_src building py_modules sources building extension "numpy.core.multiarray" sources adding 'build/src.linux-ia64-2.3/numpy/core/config.h' to sources. adding 'build/src.linux-ia64-2.3/numpy/core/__multiarray_api.h' to sources. adding 'build/src.linux-ia64-2.3/numpy/core/src' to include_dirs. numpy.core - nothing done with h_files= ['build/src.linux-ia64-2.3/numpy/core/src/scalartypes.inc', 'build/src.linux-ia64-2.3/numpy/core/src/arraytypes.inc', 'build/src.linux- ia64-2.3/numpy/core/config.h', 'build/src.linux-ia64-2.3 /numpy/core/__multiarray_api.h'] building extension "numpy.core.umath" sources adding 'build/src.linux-ia64-2.3/numpy/core/config.h' to sources. adding 'build/src.linux-ia64-2.3/numpy/core/__ufunc_api.h' to sources. adding 'build/src.linux-ia64-2.3/numpy/core/src' to include_dirs. numpy.core - nothing done with h_files= ['build/src.linux-ia64-2.3/numpy/core/src/scalartypes.inc', 'build/src.linux-ia64-2.3/numpy/core/src/arraytypes.inc', 'build/src.linux- ia64-2.3/numpy/core/config.h', 'build/src.linux-ia64-2.3 /numpy/core/__ufunc_api.h'] building extension "numpy.core._sort" sources adding 'build/src.linux-ia64-2.3/numpy/core/config.h' to sources. adding 'build/src.linux-ia64-2.3/numpy/core/__multiarray_api.h' to sources. numpy.core - nothing done with h_files= ['build/src.linux-ia64-2.3/numpy/core/config.h', 'build/src.linux-ia64-2.3/numpy/core/__multiarray_api.h'] building extension "numpy.core.scalarmath" sources adding 'build/src.linux-ia64-2.3/numpy/core/config.h' to sources. adding 'build/src.linux-ia64-2.3/numpy/core/__multiarray_api.h' to sources. adding 'build/src.linux-ia64-2.3/numpy/core/__ufunc_api.h' to sources. numpy.core - nothing done with h_files= ['build/src.linux-ia64-2.3/numpy/core/config.h', 'build/src.linux-ia64-2.3/numpy/core/__multiarray_api.h', 'build/src.linux- ia64-2.3/numpy/core/__ufunc_api.h'] building extension "numpy.core._dotblas" sources adding 'numpy/core/blasdot/_dotblas.c' to sources. building extension "numpy.lib._compiled_base" sources building extension "numpy.dft.fftpack_lite" sources building extension "numpy.linalg.lapack_lite" sources adding 'numpy/linalg/lapack_litemodule.c' to sources. building extension "numpy.random.mtrand" sources Could not locate executable gfortran Could not locate executable f95 customize GnuFCompiler customize GnuFCompiler customize GnuFCompiler using config gcc options: '-pthread -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC' compile options: '-Inumpy/core/src -Inumpy/core/include -I/home/user17/pulsar/PYTHON-PKG/python/include/python2.3 -c' gcc: _configtest.c _configtest.c:7:2: #error No _WIN32 _configtest.c:7:2: #error No _WIN32 failure. removing: _configtest.c _configtest.o building data_files sources running build_py copying build/src.linux-ia64-2.3/numpy/__config__.py -> build/lib.linux- ia64-2.3/numpy copying numpy/distutils/system_info.py -> build/lib.linux-ia64-2.3 /numpy/distutils copying build/src.linux-ia64-2.3/numpy/distutils/__config__.py -> build/lib.linux-ia64-2.3/numpy/distutils running build_ext customize UnixCCompiler customize UnixCCompiler using build_ext customize GnuFCompiler customize GnuFCompiler customize GnuFCompiler using build_ext running build_scripts adding 'build/scripts.linux-ia64-2.3/f2py2.3' to scripts running install_lib copying build/lib.linux-ia64-2.3/numpy/__config__.py -> /home/user17/pulsar/PYTHON-PKG/python/lib//lib/python2.3/site-packages/numpy copying build/lib.linux-ia64-2.3/numpy/distutils/system_info.py -> /home/user17/pulsar/PYTHON-PKG/python/lib//lib/python2.3/site-packages/numpy/distutils copying build/lib.linux-ia64-2.3/numpy/distutils/__config__.py -> /home/user17/pulsar/PYTHON-PKG/python/lib//lib/python2.3/site-packages/numpy/distutils byte-compiling /home/user17/pulsar/PYTHON-PKG/python/lib//lib/python2.3/site-packages/numpy/__config__.py to __config__.pyc byte-compiling /home/user17/pulsar/PYTHON-PKG/python/lib//lib/python2.3/site-packages/numpy/distutils/system_info.py to system_info.pyc byte-compiling /home/user17/pulsar/PYTHON-PKG/python/lib//lib/python2.3/site-packages/numpy/distutils/__config__.py to __config__.pyc running install_scripts changing mode of /home/user17/pulsar/PYTHON-PKG/python/lib//bin/f2py2.3 to 775 running install_data -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Tue Apr 1 02:39:10 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 1 Apr 2008 08:39:10 +0200 Subject: [Numpy-discussion] Project for Cython integration with NumPy In-Reply-To: <50425.193.157.243.12.1207003921.squirrel@webmail.uio.no> References: <60363.80.59.7.37.1206478868.squirrel@webmail.uio.no> <50425.193.157.243.12.1207003921.squirrel@webmail.uio.no> Message-ID: <9457e7c80803312339h108ba53ep6633af122237b05@mail.gmail.com> Hi Dag On Tue, Apr 1, 2008 at 12:52 AM, Dag Sverre Seljebotn wrote: > > I am going to apply for a Google Summer of Code project about "Developing > > Cython towards better NumPy integration" (Cython: http://cython.org). > > Anyone interested in how this is done can have a look at the links below, > > any feedback is welcome. > > > > > The application I am going to submit (to Python Foundation): > > http://wiki.cython.org/DagSverreSeljebotn/soc > > I now have time to actively discuss and improve it so any feedback from > the NumPy community is greatly appreciated. Sorry for only responding now; I was traveling back from Europe. The project your propose has important implications for NumPy. Currently, much of the core is implemented in C and very few people have in-depth knowledge of its inner workings. We therefore have fewer eyes on that code than we'd like. Furthermore, there are only two ways of extending the core: 1) use Python callbacks, which are slow or 2) learn the C API and deal with reference counting -- too much to ask from a scientist who simply wishes to implement a fast algorithm. There are also some other projects, like scipy.ndimage, that could do with a fair bit of refactoring, and this task would be so much easier if Cython could generate the C-level ndarray code; we have precious little developer resources available as it is, and would rather spend it on solving problems than on arranging low-level API calls. Having the following implemented would already be a vast improvement on the current situation: - Array indexing x[1,2] x[1:2,3:4] x[1,...] x[1,:] - Assignment x[1,2] = 3 x[1,...] = 1 ... - Calling array methods: y = x.sum() Doing this would be easier with Travis Oliphant's numpy developer's reference at hand, and, if I recall the page on scipy.org (which is currently down) correctly, he would provide it to you free of charge on request. I am very excited about your proposal, and I really hope it takes off. Let us know if there is anything we can do to help. Best of luck, St?fan From stefan at sun.ac.za Tue Apr 1 02:49:35 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 1 Apr 2008 08:49:35 +0200 Subject: [Numpy-discussion] Project for Cython integration with NumPy In-Reply-To: <50425.193.157.243.12.1207003921.squirrel@webmail.uio.no> References: <60363.80.59.7.37.1206478868.squirrel@webmail.uio.no> <50425.193.157.243.12.1207003921.squirrel@webmail.uio.no> Message-ID: <9457e7c80803312349h6614d384u699727007c1bd801@mail.gmail.com> Hi Dag On Tue, Apr 1, 2008 at 12:52 AM, Dag Sverre Seljebotn wrote: > > I am going to apply for a Google Summer of Code project about "Developing > > Cython towards better NumPy integration" (Cython: http://cython.org). > > Anyone interested in how this is done can have a look at the links below, > > any feedback is welcome. > > > > > The application I am going to submit (to Python Foundation): > > http://wiki.cython.org/DagSverreSeljebotn/soc > > I now have time to actively discuss and improve it so any feedback from > the NumPy community is greatly appreciated. See especially: > > > http://wiki.cython.org/enhancements/numpy One more comment about the constructor described on the page above. It would be good if we could have the same syntax as the current numpy.ndarray, and then simply call through to the underlying C constructor. We'd also need zeros, empty, ones_like and some other select functions. The goal is to remain as faithful as possible to the original API, so that there is hardly a distinction between Python and Cython. Cheers St?fan From izakmarais at yahoo.com Tue Apr 1 03:10:21 2008 From: izakmarais at yahoo.com (izak marais) Date: Tue, 1 Apr 2008 00:10:21 -0700 (PDT) Subject: [Numpy-discussion] Applying PIL patch Message-ID: <10544.52729.qm@web50911.mail.re2.yahoo.com> Thank you for the reply. I in fact did not have the latest PIL binary. It works beautifully now. Luckily I won't be needing RGBA support soon. Izak ----- Original Message ---- From: St?fan van der Walt To: Discussion of Numerical Python Sent: Monday, March 31, 2008 11:45:52 PM Subject: Re: [Numpy-discussion] Applying PIL patch Unfortunately, RGBA images cannot be read this way. A patch that fixes the issue was posted here: http://www.mail-archive.com/image-sig at python.org/msg01482.html No response from the Image SIG guys. Regards St?fan On Mon, Mar 31, 2008 at 6:29 PM, Christopher Barker wrote: > izak marais wrote: > > Sorry for the beginner question. I want to apply the PIL-numpy patch > > from http://www.scipy.org/Cookbook/PIL?highlight=%28PIL%29 . I have the > > latest windows binaries of numpy, scipy and PIL installed. > > Then you have the patch already-- it was added to the latest PIL. > > http://effbot.org/zone/pil-changes-116.htm > > -Chris _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ____________________________________________________________________________________ You rock. That's why Blockbuster's offering you one month of Blockbuster Total Access, No Cost. http://tc.deals.yahoo.com/tc/blockbuster/text5.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From izakmarais at yahoo.com Tue Apr 1 04:08:00 2008 From: izakmarais at yahoo.com (izak marais) Date: Tue, 1 Apr 2008 01:08:00 -0700 (PDT) Subject: [Numpy-discussion] Applying PIL patch Message-ID: <818605.42949.qm@web50902.mail.re2.yahoo.com> St?fan wrote:"Unfortunately, RGBA images cannot be read this way. " Apparently it does not work with 16bit greyscale tif images either. For anyone else stumbling upon this thread, there is a work-about to get the data into a numpy array. i = Image.open('16bitGreyscaleImage.tif') a = numpy.array(i.getdata()) # a 1d numpy array a = a.reshape(i.size) #2d numpy array Perhaps there is a better way of doing it (?), but this works for me. Izak ____________________________________________________________________________________ You rock. That's why Blockbuster's offering you one month of Blockbuster Total Access, No Cost. http://tc.deals.yahoo.com/tc/blockbuster/text5.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From dagss at student.matnat.uio.no Tue Apr 1 05:06:51 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 01 Apr 2008 11:06:51 +0200 Subject: [Numpy-discussion] Project for Cython integration with NumPy In-Reply-To: <9457e7c80803312349h6614d384u699727007c1bd801@mail.gmail.com> References: <60363.80.59.7.37.1206478868.squirrel@webmail.uio.no> <50425.193.157.243.12.1207003921.squirrel@webmail.uio.no> <9457e7c80803312349h6614d384u699727007c1bd801@mail.gmail.com> Message-ID: <47F1FB2B.2030902@student.matnat.uio.no> An HTML attachment was scrubbed... URL: From dagss at student.matnat.uio.no Tue Apr 1 05:10:47 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 01 Apr 2008 11:10:47 +0200 Subject: [Numpy-discussion] Project for Cython integration with NumPy In-Reply-To: <47F1FB2B.2030902@student.matnat.uio.no> References: <60363.80.59.7.37.1206478868.squirrel@webmail.uio.no> <50425.193.157.243.12.1207003921.squirrel@webmail.uio.no> <9457e7c80803312349h6614d384u699727007c1bd801@mail.gmail.com> <47F1FB2B.2030902@student.matnat.uio.no> Message-ID: <47F1FC17.9000706@student.matnat.uio.no> An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Tue Apr 1 08:29:13 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 1 Apr 2008 14:29:13 +0200 Subject: [Numpy-discussion] planet.scipy.org In-Reply-To: References: Message-ID: Hi, The planet is no longer accessible. Anyone has the same issue ? Matthieu 2008/1/1, Jarrod Millman : > > Hey, > > I just wanted to announce that we now have a NumPy/SciPy blog > aggregator thanks to Ga?l Varoquaux: http://planet.scipy.org/ > > Feel free to contact me if you have a blog that you would like included. > > Thanks, > > -- > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From chanley at stsci.edu Tue Apr 1 08:39:37 2008 From: chanley at stsci.edu (Christopher Hanley) Date: Tue, 01 Apr 2008 08:39:37 -0400 Subject: [Numpy-discussion] server down Message-ID: <47F22D09.4080700@stsci.edu> Hi, I cannot access the numpy, scipy, or astropy repositories at scipy.org. The servers appear to be down. [redcedar:~/dev/scipy] chanley% svn update svn: PROPFIND request failed on '/svn/scipy/trunk' svn: PROPFIND of '/svn/scipy/trunk': could not connect to server (http://svn.scipy.org) Thanks, Chris -- Christopher Hanley Systems Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338 From gael.varoquaux at normalesup.org Tue Apr 1 09:19:58 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 1 Apr 2008 15:19:58 +0200 Subject: [Numpy-discussion] planet.scipy.org In-Reply-To: References: Message-ID: <20080401131958.GC1301@phare.normalesup.org> On Tue, Apr 01, 2008 at 02:29:13PM +0200, Matthieu Brucher wrote: > The planet is no longer accessible. Anyone has the same issue ? Yes, the scipy.org server is down. I think we need to wait for the US to wake up to take care of this. Ga?l From travis at enthought.com Tue Apr 1 10:54:47 2008 From: travis at enthought.com (Travis Vaught) Date: Tue, 1 Apr 2008 09:54:47 -0500 Subject: [Numpy-discussion] planet.scipy.org In-Reply-To: <20080401131958.GC1301@phare.normalesup.org> References: <20080401131958.GC1301@phare.normalesup.org> Message-ID: <13910FD5-F43E-4DCA-8F15-08EF318592E8@enthought.com> Fixed now...many apologies for the outage. Travis On Apr 1, 2008, at 8:19 AM, Gael Varoquaux wrote: > On Tue, Apr 01, 2008 at 02:29:13PM +0200, Matthieu Brucher wrote: >> The planet is no longer accessible. Anyone has the same issue ? > > Yes, the scipy.org server is down. I think we need to wait for the > US to > wake up to take care of this. > > Ga?l > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From doutriaux1 at llnl.gov Tue Apr 1 11:07:41 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Tue, 01 Apr 2008 08:07:41 -0700 Subject: [Numpy-discussion] missing function in numpy.ma? In-Reply-To: <200803311615.19175.pgmdevlist@gmail.com> References: <200803260948.02742.pgmdevlist@gmail.com> <200803311509.23010.pgmdevlist@gmail.com> <47F13FCA.8040003@llnl.gov> <200803311615.19175.pgmdevlist@gmail.com> Message-ID: <47F24FBD.7070707@llnl.gov> Hi Pierre, I"m ccing Bob on this, he's the main developper for cdms2 package. At this point I think Travis original suggestion was the best. We should leave it like it was for 1.0.5 There's a lot of changes to do in order to get the backward compatibility going. And I feel it should wait until 1.1. I don't feel comfortable doing all these changes and releasing our software like this. It's a major change and it needs to be tested for a while. OUr users are massively hammering the MA and rely on it so much. Although I do see the usefulness of the new ma and at term I believe it has major merits to be used instead of oldnumeric.ma. Your thoughts? C. Pierre GM wrote: > Charles, > > >> Any idea where that comes from ? >> > > No, not really. Seems that TransientVariable(*args) doesn't work. I guess it's > because it has inherited a __call__method, and tries to use that method > instead of the __new__. Try to call TransientVariable.__new__ instead of just > TransientVariable in l505 of avariables.py, and see how it goes. You may want > to rethink what subSlice does as well. Instead of calling the class > constructor, you can also just create a view of your array and update the > attributes accordingly. > > Once again, a stripped-down version of the class and its parents would be > useful. > > From nodrogbrown at gmail.com Tue Apr 1 11:09:13 2008 From: nodrogbrown at gmail.com (gordon) Date: Tue, 1 Apr 2008 08:09:13 -0700 (PDT) Subject: [Numpy-discussion] confusion in eigenlib code Message-ID: hi i came across some code for eigenface construction from some images ,using the old Numeric . http://www.owlnet.rice.edu/~elec301/Projects99/faces/code.html In the eigenlib.py http://www.owlnet.rice.edu/~elec301/Projects99/faces/code/eigenlib.py i converted the calls to Numeric functions to their numpy equivalents (to linalg.eigh() and numpy.dot())and ran the code.In this eigenlib.py i am confused by some parts where they derrive eigenvectors and sort them evalues, evectors = LinearAlgebra.eigenvectors(L) # sort them by eigenvalue and keep the top M_prime evs = map(None, evalues, evectors) evs.sort() evs.reverse() evs = evs[0:M_prime] # write those into the directory v = map(lambda x: x[1], evs) self.u = [] for k in range(M_prime): print(' ' + str(k+1)) self.u.append(Numeric.matrixmultiply(v[k], self.Phi)) #self.vector_to_image(self.u[-1], '%s/eig%03d.gif' % (dir, k)) (Here self.Psi is the average face from a collection of face images and self.Phi is obtained by substracting Psi from original image data ...mean centering i guess) what i can't understand in the above code is that when evs[0:M_prime] is taken it takes the rows from evectors.Is not the correct way to take a column of evectors as an eigenvector? If someone can make this clear please do thanks gordon From aitagi at gmail.com Tue Apr 1 11:16:28 2008 From: aitagi at gmail.com (Amit Itagi) Date: Tue, 1 Apr 2008 11:16:28 -0400 Subject: [Numpy-discussion] Numpy installation In-Reply-To: <3d375d730803312006q409e9168q1d607ac339c45d30@mail.gmail.com> References: <3d375d730803312006q409e9168q1d607ac339c45d30@mail.gmail.com> Message-ID: On Mon, Mar 31, 2008 at 11:06 PM, Robert Kern wrote: > On Mon, Mar 31, 2008 at 4:17 PM, Amit Itagi wrote: > > Hi, > > > > I am having problems with numpy installation. > > > > 1) These is an atlas 3.8.0 library installed somewhere in the search > path. > > However, the installation gives errors with that installation. Is there > a > > way to tell the installer to install the default (possibly slower) blas, > > instead of using the one in the path ? > > Create a site.cfg file with the appropriate section; copy and modify > the site.cfg.example file. *I figured how to specify a particular installation of the libraries. I want to do the opposite. How do I specify the following in site.cfg - "Don't search for the library. Assume that it is absent and use the default slower library" ?* * * > > > > 2) Also, my main Python directory is called Python-2.5.2. When I try to > > configure with the install, it changes Python-2.5.2 to > > "python-2.5.2" and creates a new directory. How can I make the installer > not > > convert the upper-case "P" to a lower-case ? > > Can you give more information like the platform you are on, the full > path to this directory, the exact commands that you executed, and the > results of these commands? *I am installing this on a CENTOS linux platform (64 bit AMD opteron). The path to my python directory is /home/amit/packages/Python-2.5.2 . If I temporarily make the atlas library unavailable (by renaming the directory to some name that is not in the path), I can perform the build. Now in the installation stage, I use python setup.py and then choose the option 2/home/amit/packages/Python-2.5.2. In the proposed sys.argv the path is shown as /home/amit/packages/python-2.5.2. Incidentally, it also creates this new directory during install. * *Thanks Robert. Rgds, Amit* > > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nodrogbrown at gmail.com Tue Apr 1 11:31:24 2008 From: nodrogbrown at gmail.com (gordon) Date: Tue, 1 Apr 2008 08:31:24 -0700 (PDT) Subject: [Numpy-discussion] linalg.eigh() newbie doubt In-Reply-To: <80c99e790803310823m44c2fe14u45e856ddd4b6fa75@mail.gmail.com> References: <80c99e790803310823m44c2fe14u45e856ddd4b6fa75@mail.gmail.com> Message-ID: <9ac200b9-6d9c-44c3-acd8-6e2ceac9e3e8@q27g2000prf.googlegroups.com> > The normalized eigenvector corresponding to the eigenvalue w[i] > is the column v[:,i]. > so, yes, the eigvec coresponding to the eigval w[i] is v[:,i]. Lorenzo sorry i don't understand from the above sample(unordered) if i select the the 4th eigenvalue i get 1.7 evals[3]=1.7 i believe the corresponding eigenvector should be the 4th column in evectors? [6. 7. 5. 4. 4. 4.] but the notation evectors[:3] will give me an ndarray of shape(3,6) Am i missing something here? thanks gordon From matthieu.brucher at gmail.com Tue Apr 1 11:37:31 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 1 Apr 2008 17:37:31 +0200 Subject: [Numpy-discussion] linalg.eigh() newbie doubt In-Reply-To: <9ac200b9-6d9c-44c3-acd8-6e2ceac9e3e8@q27g2000prf.googlegroups.com> References: <80c99e790803310823m44c2fe14u45e856ddd4b6fa75@mail.gmail.com> <9ac200b9-6d9c-44c3-acd8-6e2ceac9e3e8@q27g2000prf.googlegroups.com> Message-ID: > > but the notation evectors[:3] will give me an ndarray of shape(3,6) > Am i missing something here? > Yes : evectors[:3] selects the first three lines, evectors[:,3] selects the fourth column. Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From nodrogbrown at gmail.com Tue Apr 1 11:41:05 2008 From: nodrogbrown at gmail.com (gordon) Date: Tue, 1 Apr 2008 08:41:05 -0700 (PDT) Subject: [Numpy-discussion] linalg.eigh() newbie doubt In-Reply-To: References: <80c99e790803310823m44c2fe14u45e856ddd4b6fa75@mail.gmail.com> <9ac200b9-6d9c-44c3-acd8-6e2ceac9e3e8@q27g2000prf.googlegroups.com> Message-ID: > Yes : evectors[:3] selects the first three lines, evectors[:,3] selects the > fourth column. > arggg!! my mistake! sorry Lorenzo thanks Matthieu gordon From pgmdevlist at gmail.com Tue Apr 1 11:53:33 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 1 Apr 2008 11:53:33 -0400 Subject: [Numpy-discussion] missing function in numpy.ma? In-Reply-To: <47F24FBD.7070707@llnl.gov> References: <200803260948.02742.pgmdevlist@gmail.com> <200803311615.19175.pgmdevlist@gmail.com> <47F24FBD.7070707@llnl.gov> Message-ID: <200804011153.35006.pgmdevlist@gmail.com> All, Because numpy.ma.MaskedArray objects are now derived from classical ndarrays, the subclassing rules should be followed. As we observed yesterday with Charles, the adaptation is not as straightforward as we hoped. If you have time constraints, the easiest would indeed be to revert to the previous implementation of MaskedArray (viz, as an object, where the initialization is performed with a __init__ function). Now, I'm not sure how well it's gonna interface with numpy.ma, we need to try. In the long run however, I think you should try to switch to regular ndarrays and ndarray subclasses. I do agree it's more of a long-term process, but I'd be happy to help (you and I are working more or less on the same field anyway, I needed some tools to handle my environmental/climatic time series) Let me try to cook something up. In the meantime, please keep me posted. P. On Tuesday 01 April 2008 11:07:41 Charles Doutriaux wrote: > Hi Pierre, > > I"m ccing Bob on this, he's the main developper for cdms2 package. At > this point I think Travis original suggestion was the best. We should > leave it like it was for 1.0.5 There's a lot of changes to do in order > to get the backward compatibility going. And I feel it should wait until > 1.1. I don't feel comfortable doing all these changes and releasing our > software like this. It's a major change and it needs to be tested for a > while. OUr users are massively hammering the MA and rely on it so much. > Although I do see the usefulness of the new ma and at term I believe it > has major merits to be used instead of oldnumeric.ma. > > Your thoughts? From oliphant at enthought.com Tue Apr 1 12:03:18 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 01 Apr 2008 11:03:18 -0500 Subject: [Numpy-discussion] missing function in numpy.ma? In-Reply-To: <47F24FBD.7070707@llnl.gov> References: <200803260948.02742.pgmdevlist@gmail.com> <200803311509.23010.pgmdevlist@gmail.com> <47F13FCA.8040003@llnl.gov> <200803311615.19175.pgmdevlist@gmail.com> <47F24FBD.7070707@llnl.gov> Message-ID: <47F25CC6.4070002@enthought.com> Charles Doutriaux wrote: > Hi Pierre, > > I"m ccing Bob on this, he's the main developper for cdms2 package. I've uploaded the original ma.py file back into oldnumeric so that oldnumeric.ma should continue to work as before. Can you verify this? Thanks, -Travis O. From doutriaux1 at llnl.gov Tue Apr 1 12:36:19 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Tue, 01 Apr 2008 09:36:19 -0700 Subject: [Numpy-discussion] missing function in numpy.ma? In-Reply-To: <47F25CC6.4070002@enthought.com> References: <200803260948.02742.pgmdevlist@gmail.com> <200803311509.23010.pgmdevlist@gmail.com> <47F13FCA.8040003@llnl.gov> <200803311615.19175.pgmdevlist@gmail.com> <47F24FBD.7070707@llnl.gov> <47F25CC6.4070002@enthought.com> Message-ID: <47F26483.3010002@llnl.gov> Hi Travis, I get this: import numpy, numpy.oldnumeric.ma as MA, numpy.oldnumeric as Numeric, PropertiedClasses File "/lgm/cdat/latest/lib/python2.5/site-packages/numpy/oldnumeric/ma.py", line 2204, in array.mean = _m(average) NameError: name 'average' is not defined C. Travis E. Oliphant wrote: > Charles Doutriaux wrote: > >> Hi Pierre, >> >> I"m ccing Bob on this, he's the main developper for cdms2 package. >> > I've uploaded the original ma.py file back into oldnumeric so that > oldnumeric.ma should continue to work as before. Can you verify this? > > Thanks, > > -Travis O. > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From cournape at gmail.com Tue Apr 1 12:43:55 2008 From: cournape at gmail.com (David Cournapeau) Date: Tue, 1 Apr 2008 09:43:55 -0700 Subject: [Numpy-discussion] Numpy installation In-Reply-To: References: <3d375d730803312006q409e9168q1d607ac339c45d30@mail.gmail.com> Message-ID: <5b8d13220804010943w41f9e467w1c245cfd795fd805@mail.gmail.com> On Tue, Apr 1, 2008 at 8:16 AM, Amit Itagi wrote: > > I figured how to specify a particular installation of the libraries. I want > to do the opposite. How do I specify the following in site.cfg - "Don't > search for the library. Assume that it is absent and use the default slower > library" ? If you want to avoid compiling with say ATLAS, you can do something like ATLAS=None python setup.py build Disabling one specific version of one library is more difficult, though. cheers, David From lists at informa.tiker.net Tue Apr 1 12:56:18 2008 From: lists at informa.tiker.net (Andreas =?utf-8?q?Kl=C3=B6ckner?=) Date: Tue, 1 Apr 2008 12:56:18 -0400 Subject: [Numpy-discussion] output arguments for dot(), tensordot() Message-ID: <200804011256.21349.lists@informa.tiker.net> Hi all, is there a particular reason why dot() and tensordot() don't have output arguments? Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From oliphant at enthought.com Tue Apr 1 13:03:45 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 01 Apr 2008 12:03:45 -0500 Subject: [Numpy-discussion] missing function in numpy.ma? In-Reply-To: <47F26483.3010002@llnl.gov> References: <200803260948.02742.pgmdevlist@gmail.com> <200803311509.23010.pgmdevlist@gmail.com> <47F13FCA.8040003@llnl.gov> <200803311615.19175.pgmdevlist@gmail.com> <47F24FBD.7070707@llnl.gov> <47F25CC6.4070002@enthought.com> <47F26483.3010002@llnl.gov> Message-ID: <47F26AF1.3070000@enthought.com> Charles Doutriaux wrote: > Hi Travis, > > I get this: > > import numpy, numpy.oldnumeric.ma as MA, numpy.oldnumeric as > Numeric, PropertiedClasses > File > "/lgm/cdat/latest/lib/python2.5/site-packages/numpy/oldnumeric/ma.py", > line 2204, in > array.mean = _m(average) > NameError: name 'average' is not defined > Thanks, Can you try again? Best regards, -Travis From robert.kern at gmail.com Tue Apr 1 13:17:44 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 1 Apr 2008 12:17:44 -0500 Subject: [Numpy-discussion] Numpy installation In-Reply-To: References: <3d375d730803312006q409e9168q1d607ac339c45d30@mail.gmail.com> Message-ID: <3d375d730804011017i58e6f486g2be1fd8153fc5cd2@mail.gmail.com> On Tue, Apr 1, 2008 at 10:16 AM, Amit Itagi wrote: > > On Mon, Mar 31, 2008 at 11:06 PM, Robert Kern wrote: > > > > On Mon, Mar 31, 2008 at 4:17 PM, Amit Itagi wrote: > > > Hi, > > > > > > I am having problems with numpy installation. > > > > > > 1) These is an atlas 3.8.0 library installed somewhere in the search > path. > > > However, the installation gives errors with that installation. Is there > a > > > way to tell the installer to install the default (possibly slower) blas, > > > instead of using the one in the path ? > > > > Create a site.cfg file with the appropriate section; copy and modify > > the site.cfg.example file. > > I figured how to specify a particular installation of the libraries. I want > to do the opposite. How do I specify the following in site.cfg - "Don't > search for the library. Assume that it is absent and use the default slower > library" ? There's nothing default about it. You should use the [lapack_opt] section to specify whichever BLAS and LAPACK libraries you like, even if they are not optimized. > > > 2) Also, my main Python directory is called Python-2.5.2. When I try to > > > configure with the install, it changes Python-2.5.2 to > > > "python-2.5.2" and creates a new directory. How can I make the installer > not > > > convert the upper-case "P" to a lower-case ? > > > > Can you give more information like the platform you are on, the full > > path to this directory, the exact commands that you executed, and the > > results of these commands? > > I am installing this on a CENTOS linux platform (64 bit AMD opteron). The > path to my python directory is /home/amit/packages/Python-2.5.2 . If I > temporarily make the atlas library unavailable (by renaming the directory to > some name that is not in the path), I can perform the build. Now in the > installation stage, I use > python setup.py > and then choose the option 2/home/amit/packages/Python-2.5.2. In the > proposed sys.argv the path is shown as /home/amit/packages/python-2.5.2. > Incidentally, it also creates this new directory during install. Can you just do a "python setup.py install" instead of going through the menu system? The menu system may be bitrotten. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Tue Apr 1 13:21:16 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 1 Apr 2008 12:21:16 -0500 Subject: [Numpy-discussion] Numpy installation In-Reply-To: References: <3d375d730803312006q409e9168q1d607ac339c45d30@mail.gmail.com> Message-ID: <3d375d730804011021k27fc8d97h2ac5c1c1a0abaf81@mail.gmail.com> On Tue, Apr 1, 2008 at 10:16 AM, Amit Itagi wrote: > I am installing this on a CENTOS linux platform (64 bit AMD opteron). The > path to my python directory is /home/amit/packages/Python-2.5.2 . If I > temporarily make the atlas library unavailable (by renaming the directory to > some name that is not in the path), I can perform the build. Now in the > installation stage, I use > python setup.py > and then choose the option 2/home/amit/packages/Python-2.5.2. In the > proposed sys.argv the path is shown as /home/amit/packages/python-2.5.2. > Incidentally, it also creates this new directory during install. Looking at the code, I can confirm that the menu system is simply buggy and the cause of your problem. Do not use it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From doutriaux1 at llnl.gov Tue Apr 1 13:30:38 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Tue, 01 Apr 2008 10:30:38 -0700 Subject: [Numpy-discussion] missing function in numpy.ma? In-Reply-To: <47F26AF1.3070000@enthought.com> References: <200803260948.02742.pgmdevlist@gmail.com> <200803311509.23010.pgmdevlist@gmail.com> <47F13FCA.8040003@llnl.gov> <200803311615.19175.pgmdevlist@gmail.com> <47F24FBD.7070707@llnl.gov> <47F25CC6.4070002@enthought.com> <47F26483.3010002@llnl.gov> <47F26AF1.3070000@enthought.com> Message-ID: <47F2713E.8040403@llnl.gov> Hi Travis, Ok we're almost there, in my test suite i get: maresult = numpy.core.ma.take(ta, indices, axis=axis) AttributeError: 'module' object has no attribute 'ma' data = numpy.core.ma.take(ax[:], indices) AttributeError: 'module' object has no attribute 'ma' I don't know if it was automatically put there by the converter or if we put it by hand. If it's the first one, you might want to correct this, otherwise don't worry it's easy enough to fix (i think ?) C. Travis E. Oliphant wrote: > Charles Doutriaux wrote: > >> Hi Travis, >> >> I get this: >> >> import numpy, numpy.oldnumeric.ma as MA, numpy.oldnumeric as >> Numeric, PropertiedClasses >> File >> "/lgm/cdat/latest/lib/python2.5/site-packages/numpy/oldnumeric/ma.py", >> line 2204, in >> array.mean = _m(average) >> NameError: name 'average' is not defined >> >> > Thanks, > > Can you try again? > > Best regards, > > -Travis > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From aitagi at gmail.com Tue Apr 1 14:31:18 2008 From: aitagi at gmail.com (Amit Itagi) Date: Tue, 1 Apr 2008 14:31:18 -0400 Subject: [Numpy-discussion] Numpy installation In-Reply-To: <3d375d730804011021k27fc8d97h2ac5c1c1a0abaf81@mail.gmail.com> References: <3d375d730803312006q409e9168q1d607ac339c45d30@mail.gmail.com> <3d375d730804011021k27fc8d97h2ac5c1c1a0abaf81@mail.gmail.com> Message-ID: On Tue, Apr 1, 2008 at 1:21 PM, Robert Kern wrote: > On Tue, Apr 1, 2008 at 10:16 AM, Amit Itagi wrote: > > I am installing this on a CENTOS linux platform (64 bit AMD opteron). > The > > path to my python directory is /home/amit/packages/Python-2.5.2 . If I > > temporarily make the atlas library unavailable (by renaming the > directory to > > some name that is not in the path), I can perform the build. Now in the > > installation stage, I use > > python setup.py > > and then choose the option 2/home/amit/packages/Python-2.5.2. In the > > proposed sys.argv the path is shown as /home/amit/packages/python-2.5.2. > > Incidentally, it also creates this new directory during install. > > Looking at the code, I can confirm that the menu system is simply > buggy and the cause of your problem. Do not use it. > *Robert, Could you kindly suggest an alternate way of getting it right ? Thanks Rgds, Amit * > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aitagi at gmail.com Tue Apr 1 14:35:53 2008 From: aitagi at gmail.com (Amit Itagi) Date: Tue, 1 Apr 2008 14:35:53 -0400 Subject: [Numpy-discussion] Numpy installation In-Reply-To: <5b8d13220804010943w41f9e467w1c245cfd795fd805@mail.gmail.com> References: <3d375d730803312006q409e9168q1d607ac339c45d30@mail.gmail.com> <5b8d13220804010943w41f9e467w1c245cfd795fd805@mail.gmail.com> Message-ID: On Tue, Apr 1, 2008 at 12:43 PM, David Cournapeau wrote: > On Tue, Apr 1, 2008 at 8:16 AM, Amit Itagi wrote: > > > > I figured how to specify a particular installation of the libraries. I > want > > to do the opposite. How do I specify the following in site.cfg - "Don't > > search for the library. Assume that it is absent and use the default > slower > > library" ? > > If you want to avoid compiling with say ATLAS, you can do something like > > ATLAS=None python setup.py build *The shell does not understand this command. Is this the correct syntax ? Thanks Rgds, Amit * > > > Disabling one specific version of one library is more difficult, though. > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aitagi at gmail.com Tue Apr 1 14:38:21 2008 From: aitagi at gmail.com (Amit Itagi) Date: Tue, 1 Apr 2008 14:38:21 -0400 Subject: [Numpy-discussion] Numpy installation In-Reply-To: <5b8d13220804010943w41f9e467w1c245cfd795fd805@mail.gmail.com> References: <3d375d730803312006q409e9168q1d607ac339c45d30@mail.gmail.com> <5b8d13220804010943w41f9e467w1c245cfd795fd805@mail.gmail.com> Message-ID: On Tue, Apr 1, 2008 at 12:43 PM, David Cournapeau wrote: > On Tue, Apr 1, 2008 at 8:16 AM, Amit Itagi wrote: > > > > I figured how to specify a particular installation of the libraries. I > want > > to do the opposite. How do I specify the following in site.cfg - "Don't > > search for the library. Assume that it is absent and use the default > slower > > library" ? > > If you want to avoid compiling with say ATLAS, you can do something like > > ATLAS=None python setup.py build *Ok, this part works. I am using tcsh. So I had to do setenv ATLAS None. Rgds, Amit * > > > Disabling one specific version of one library is more difficult, though. > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Apr 1 14:44:54 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 1 Apr 2008 13:44:54 -0500 Subject: [Numpy-discussion] Numpy installation In-Reply-To: References: <3d375d730803312006q409e9168q1d607ac339c45d30@mail.gmail.com> <3d375d730804011021k27fc8d97h2ac5c1c1a0abaf81@mail.gmail.com> Message-ID: <3d375d730804011144k280aa4b3ic51ec96f4a76f5@mail.gmail.com> On Tue, Apr 1, 2008 at 1:31 PM, Amit Itagi wrote: > > On Tue, Apr 1, 2008 at 1:21 PM, Robert Kern wrote: > > > > On Tue, Apr 1, 2008 at 10:16 AM, Amit Itagi wrote: > > > I am installing this on a CENTOS linux platform (64 bit AMD opteron). > The > > > path to my python directory is /home/amit/packages/Python-2.5.2 . If I > > > temporarily make the atlas library unavailable (by renaming the > directory to > > > some name that is not in the path), I can perform the build. Now in the > > > installation stage, I use > > > python setup.py > > > and then choose the option 2/home/amit/packages/Python-2.5.2. In the > > > proposed sys.argv the path is shown as /home/amit/packages/python-2.5.2. > > > Incidentally, it also creates this new directory during install. > > > > Looking at the code, I can confirm that the menu system is simply > > buggy and the cause of your problem. Do not use it. > > > > Robert, > > Could you kindly suggest an alternate way of getting it right ? Just like every other Python package: $ python setup.py build ... $ sudo python setup.py install --prefix=/home/amit/packages/Python-2.5.2 http://docs.python.org/inst/inst.html -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From stefan at sun.ac.za Tue Apr 1 16:04:18 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 1 Apr 2008 22:04:18 +0200 Subject: [Numpy-discussion] Project for Cython integration with NumPy In-Reply-To: <47F1FB2B.2030902@student.matnat.uio.no> References: <60363.80.59.7.37.1206478868.squirrel@webmail.uio.no> <50425.193.157.243.12.1207003921.squirrel@webmail.uio.no> <9457e7c80803312349h6614d384u699727007c1bd801@mail.gmail.com> <47F1FB2B.2030902@student.matnat.uio.no> Message-ID: <9457e7c80804011304m3daf0b64tb747911752664780@mail.gmail.com> Hi Dag On Tue, Apr 1, 2008 at 11:06 AM, Dag Sverre Seljebotn wrote: > One more comment about the constructor described on the page above. > It would be good if we could have the same syntax as the current > numpy.ndarray, and then simply call through to the underlying C > constructor. We'd also need zeros, empty, ones_like and some other > select functions. The goal is to remain as faithful as possible to > the original API, so that there is hardly a distinction between Python > and Cython. > > This will be automatic from how Cython operates :-) And that is, I think, > what makes all of this great. Objects are Python objects all the time and by > default retain all Python implementation, and we only override this > implementation when the behaviour is retained. > > I'll walk through this code line by line: > cdef c_numpy.ndarray(c_numpy.float, 2) x > x = exp(4 + numpy.zeros([10, 10], dtype=numpy.float)) > y = x + 3 > print x[1, 3] > print y[3, 5] Thanks for this example; I didn't notice that you were making a type declaration, so my original comment was not valid. > #3: (x + 3) will treat x as a Python object once again, so y will be an > untyped variable referencing an ndarray like normal Python. I can foresee certain situations under which we can predict the type of the result of operations like this one. Would it be possible to then handle 'y' as an ndarray as well, instead of reverting to Python object calls? Regards St?fan From dagss at student.matnat.uio.no Tue Apr 1 16:48:30 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 01 Apr 2008 22:48:30 +0200 Subject: [Numpy-discussion] Project for Cython integration with NumPy In-Reply-To: <9457e7c80804011304m3daf0b64tb747911752664780@mail.gmail.com> References: <60363.80.59.7.37.1206478868.squirrel@webmail.uio.no> <50425.193.157.243.12.1207003921.squirrel@webmail.uio.no> <9457e7c80803312349h6614d384u699727007c1bd801@mail.gmail.com> <47F1FB2B.2030902@student.matnat.uio.no> <9457e7c80804011304m3daf0b64tb747911752664780@mail.gmail.com> Message-ID: <47F29F9E.8010804@student.matnat.uio.no> > I can foresee certain situations under which we can predict the type > of the result of operations like this one. Would it be possible to > then handle 'y' as an ndarray as well, instead of reverting to Python > object calls? > Indeed - plans are underway to add automatic type inference to Cython. It is all about developer resources etc. (the preliminaries can be done in another GSoC project that we hope to get accepted). When/if that happens, a lot of typing will not be necesarry, including this case. This will be an automatic consequence of Cython compiler development and not something that would need to be specifically for the case of NumPy (well, one declares would some more operator and function type signatures for NumPy, and then it would work). Basically one would then only need to declare the type of arguments to the function, while local variables would be inferred automatically. (Getting around this is impossible, though we might add a different syntax for type declaration (using decorators) so that the same code can also be run using the Python interpreter.) Perhaps in a year or so, if the current GSoCs are accepted :-) no promises though. -- Dag Sverre From gael.varoquaux at normalesup.org Tue Apr 1 16:54:20 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 1 Apr 2008 22:54:20 +0200 Subject: [Numpy-discussion] Project for Cython integration with NumPy In-Reply-To: <47F29F9E.8010804@student.matnat.uio.no> References: <60363.80.59.7.37.1206478868.squirrel@webmail.uio.no> <50425.193.157.243.12.1207003921.squirrel@webmail.uio.no> <9457e7c80803312349h6614d384u699727007c1bd801@mail.gmail.com> <47F1FB2B.2030902@student.matnat.uio.no> <9457e7c80804011304m3daf0b64tb747911752664780@mail.gmail.com> <47F29F9E.8010804@student.matnat.uio.no> Message-ID: <20080401205420.GE28226@phare.normalesup.org> On Tue, Apr 01, 2008 at 10:48:30PM +0200, Dag Sverre Seljebotn wrote: > though we might add a different syntax for type declaration (using > decorators) so that the same code can also be run using the Python > interpreter.) That would be very neat. I can see how you can get around dynamical typing in a very nice way using this. As the pypy projects says, giving a dynamically-typed language to people does not necessarily means they type dynamically-typed code. I must say I like the idea a lot, you are really pushing the concept all the way. Ga?l From dagss at student.matnat.uio.no Tue Apr 1 16:56:03 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 01 Apr 2008 22:56:03 +0200 Subject: [Numpy-discussion] Project for Cython integration with NumPy In-Reply-To: <47F29F9E.8010804@student.matnat.uio.no> References: <60363.80.59.7.37.1206478868.squirrel@webmail.uio.no> <50425.193.157.243.12.1207003921.squirrel@webmail.uio.no> <9457e7c80803312349h6614d384u699727007c1bd801@mail.gmail.com> <47F1FB2B.2030902@student.matnat.uio.no> <9457e7c80804011304m3daf0b64tb747911752664780@mail.gmail.com> <47F29F9E.8010804@student.matnat.uio.no> Message-ID: <47F2A163.2080705@student.matnat.uio.no> > (Getting around this is impossible, though we might add a different > syntax for type declaration (using decorators) so that the same code can > also be run using the Python interpreter.) > I meant to say: Getting around this is impossible for functions that are exported from a module and can be used from any Python code. However it is possible to automatically infer type of parameters to intra-module calls or inner functions. -- Dag Sverre From dagss at student.matnat.uio.no Tue Apr 1 17:00:30 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 01 Apr 2008 23:00:30 +0200 Subject: [Numpy-discussion] Project for Cython integration with NumPy In-Reply-To: <20080401205420.GE28226@phare.normalesup.org> References: <60363.80.59.7.37.1206478868.squirrel@webmail.uio.no> <50425.193.157.243.12.1207003921.squirrel@webmail.uio.no> <9457e7c80803312349h6614d384u699727007c1bd801@mail.gmail.com> <47F1FB2B.2030902@student.matnat.uio.no> <9457e7c80804011304m3daf0b64tb747911752664780@mail.gmail.com> <47F29F9E.8010804@student.matnat.uio.no> <20080401205420.GE28226@phare.normalesup.org> Message-ID: <47F2A26E.10109@student.matnat.uio.no> > That would be very neat. I can see how you can get around dynamical > typing in a very nice way using this. As the pypy projects says, giving a > dynamically-typed language to people does not necessarily means they type > dynamically-typed code. > > I must say I like the idea a lot, you are really pushing the concept all > the way. > We try to focus hard on immediate gains first, but definitely have a plan for going all the way with what we do. BTW the guy applying for the requires GSoC has pypy development experience and we might use some of their stuff. So I'm really hoping we both get GSoCs. -- Dag Sverre From aitagi at gmail.com Tue Apr 1 17:03:18 2008 From: aitagi at gmail.com (Amit Itagi) Date: Tue, 1 Apr 2008 17:03:18 -0400 Subject: [Numpy-discussion] Numpy installation In-Reply-To: <3d375d730804011144k280aa4b3ic51ec96f4a76f5@mail.gmail.com> References: <3d375d730803312006q409e9168q1d607ac339c45d30@mail.gmail.com> <3d375d730804011021k27fc8d97h2ac5c1c1a0abaf81@mail.gmail.com> <3d375d730804011144k280aa4b3ic51ec96f4a76f5@mail.gmail.com> Message-ID: Robert, I followed the recommended steps. Now I have "numpy" and " numpy-1.0.4-py2.5.egg-info" in Python-2.5.2/lib/python2.5/site-packages. However, I am not able to import numpy at the python prompt. Do I have to set pythonpath or something ? Thanks Rgds, Amit On Tue, Apr 1, 2008 at 2:44 PM, Robert Kern wrote: > On Tue, Apr 1, 2008 at 1:31 PM, Amit Itagi wrote: > > > > On Tue, Apr 1, 2008 at 1:21 PM, Robert Kern > wrote: > > > > > > On Tue, Apr 1, 2008 at 10:16 AM, Amit Itagi wrote: > > > > I am installing this on a CENTOS linux platform (64 bit AMD > opteron). > > The > > > > path to my python directory is /home/amit/packages/Python-2.5.2 . If > I > > > > temporarily make the atlas library unavailable (by renaming the > > directory to > > > > some name that is not in the path), I can perform the build. Now in > the > > > > installation stage, I use > > > > python setup.py > > > > and then choose the option 2/home/amit/packages/Python-2.5.2. In the > > > > proposed sys.argv the path is shown as /home/amit/packages/python- > 2.5.2. > > > > Incidentally, it also creates this new directory during install. > > > > > > Looking at the code, I can confirm that the menu system is simply > > > buggy and the cause of your problem. Do not use it. > > > > > > > Robert, > > > > Could you kindly suggest an alternate way of getting it right ? > > Just like every other Python package: > > $ python setup.py build > ... > $ sudo python setup.py install --prefix=/home/amit/packages/Python-2.5.2 > > http://docs.python.org/inst/inst.html > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Apr 1 17:08:00 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 1 Apr 2008 16:08:00 -0500 Subject: [Numpy-discussion] Numpy installation In-Reply-To: References: <3d375d730803312006q409e9168q1d607ac339c45d30@mail.gmail.com> <3d375d730804011021k27fc8d97h2ac5c1c1a0abaf81@mail.gmail.com> <3d375d730804011144k280aa4b3ic51ec96f4a76f5@mail.gmail.com> Message-ID: <3d375d730804011408w5a5e1d9dy8960f3bbae0c33e2@mail.gmail.com> On Tue, Apr 1, 2008 at 4:03 PM, Amit Itagi wrote: > Robert, > > I followed the recommended steps. Now I have "numpy" and > "numpy-1.0.4-py2.5.egg-info" in > Python-2.5.2/lib/python2.5/site-packages. However, I am not able to import > numpy at the python prompt. Do I have to set pythonpath or something ? Possibly. Exactly what is in this .../Python-2.5.2/ directory? Is the python executable .../Python-2.5.2/bin/python? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From aitagi at gmail.com Tue Apr 1 17:21:35 2008 From: aitagi at gmail.com (Amit Itagi) Date: Tue, 1 Apr 2008 17:21:35 -0400 Subject: [Numpy-discussion] Numpy installation In-Reply-To: <3d375d730804011408w5a5e1d9dy8960f3bbae0c33e2@mail.gmail.com> References: <3d375d730803312006q409e9168q1d607ac339c45d30@mail.gmail.com> <3d375d730804011021k27fc8d97h2ac5c1c1a0abaf81@mail.gmail.com> <3d375d730804011144k280aa4b3ic51ec96f4a76f5@mail.gmail.com> <3d375d730804011408w5a5e1d9dy8960f3bbae0c33e2@mail.gmail.com> Message-ID: This directory is just the Python source distribution (post configure and make). I don't have root permissions to our cluster and the default python distribution is an older one. Hence, I have my custom Python distribution in this /Python-2.5.2/ directory. The binary is /Python-2.5.2/python . On Tue, Apr 1, 2008 at 5:08 PM, Robert Kern wrote: > On Tue, Apr 1, 2008 at 4:03 PM, Amit Itagi wrote: > > Robert, > > > > I followed the recommended steps. Now I have "numpy" and > > "numpy-1.0.4-py2.5.egg-info" in > > Python-2.5.2/lib/python2.5/site-packages. However, I am not able to > import > > numpy at the python prompt. Do I have to set pythonpath or something ? > > Possibly. Exactly what is in this .../Python-2.5.2/ directory? Is the > python executable .../Python-2.5.2/bin/python? > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Apr 1 17:36:21 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 1 Apr 2008 16:36:21 -0500 Subject: [Numpy-discussion] Numpy installation In-Reply-To: References: <3d375d730803312006q409e9168q1d607ac339c45d30@mail.gmail.com> <3d375d730804011021k27fc8d97h2ac5c1c1a0abaf81@mail.gmail.com> <3d375d730804011144k280aa4b3ic51ec96f4a76f5@mail.gmail.com> <3d375d730804011408w5a5e1d9dy8960f3bbae0c33e2@mail.gmail.com> Message-ID: <3d375d730804011436o7ecf4adaqa5c7322cfe529876@mail.gmail.com> On Tue, Apr 1, 2008 at 4:21 PM, Amit Itagi wrote: > This directory is just the Python source distribution (post configure and > make). I don't have root permissions to our cluster and the default python > distribution is an older one. Hence, I have my custom Python distribution in > this /Python-2.5.2/ directory. The binary is /Python-2.5.2/python . Okay, don't do that. You will have to actually install Python to another location. For example, make a directory ~/python2.5/. Now go to the Python source directory; it would probably be best to start with a clean one. Configure Python using ~/python2.5 as the prefix: $ ./configure --prefix=~/python2.5 Now "make" and "make install". Add ~/python2.5/bin to your $PATH, preferably before /usr/bin or wherever the old python executable is. Build and install numpy using the ~/python2.5/bin/python binary. You should not need to set the --prefix. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at enthought.com Tue Apr 1 18:01:13 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 01 Apr 2008 17:01:13 -0500 Subject: [Numpy-discussion] missing function in numpy.ma? In-Reply-To: <47F2713E.8040403@llnl.gov> References: <200803260948.02742.pgmdevlist@gmail.com> <200803311509.23010.pgmdevlist@gmail.com> <47F13FCA.8040003@llnl.gov> <200803311615.19175.pgmdevlist@gmail.com> <47F24FBD.7070707@llnl.gov> <47F25CC6.4070002@enthought.com> <47F26483.3010002@llnl.gov> <47F26AF1.3070000@enthought.com> <47F2713E.8040403@llnl.gov> Message-ID: <47F2B0A9.80800@enthought.com> Charles Doutriaux wrote: > Hi Travis, > Ok we're almost there, in my test suite i get: > maresult = numpy.core.ma.take(ta, indices, axis=axis) > AttributeError: 'module' object has no attribute 'ma' > data = numpy.core.ma.take(ax[:], indices) > AttributeError: 'module' object has no attribute 'ma' > I think the problem here is that numpy.core.ma is no longer the correct place. This should be numpy.oldnumeric.ma.take because numpy.oldnumeric.ma is the correct location. In my mind you shouldn't really have been using "numpy.core.ma", but instead numpy.ma because whether things are in core or lib could change ;-) -Travis From robert.kern at gmail.com Tue Apr 1 18:07:45 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 1 Apr 2008 17:07:45 -0500 Subject: [Numpy-discussion] output arguments for dot(), tensordot() In-Reply-To: <200804011256.21349.lists@informa.tiker.net> References: <200804011256.21349.lists@informa.tiker.net> Message-ID: <3d375d730804011507s1c2b3771p990cc0acfe1e80e5@mail.gmail.com> On Tue, Apr 1, 2008 at 11:56 AM, Andreas Kl?ckner wrote: > Hi all, > > is there a particular reason why dot() and tensordot() don't have output > arguments? No technical reason. It just hasn't been done. If you were to implement it, we would be happy to accept it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oswald.harry at gmail.com Wed Apr 2 01:32:20 2008 From: oswald.harry at gmail.com (harryos) Date: Tue, 1 Apr 2008 22:32:20 -0700 (PDT) Subject: [Numpy-discussion] newbie doubt about dot() Message-ID: <84576b39-e7c7-40a0-9eb6-cbb04eb74fe8@i7g2000prf.googlegroups.com> i am slightly confused by this maths i need to calculate wk=uk o (L-Psi) where uk=a vector of size (1 X N^2) o =scalar product l,Psi=vectors of (N^2 X 1) i have an ndarray U of shape(M X N^2)where uk is one of the rows , and L of shape (M X N^2) where l.transpose() is one of the rows, If i were to apply the above formula how can i find the matrix where wk is an element.? I know i have to use dot() but the rest i am finding a bit confusing can someone please help? harryos From aisaac at american.edu Wed Apr 2 01:58:36 2008 From: aisaac at american.edu (Alan G Isaac) Date: Wed, 2 Apr 2008 01:58:36 -0400 Subject: [Numpy-discussion] newbie doubt about dot() In-Reply-To: <84576b39-e7c7-40a0-9eb6-cbb04eb74fe8@i7g2000prf.googlegroups.com> References: <84576b39-e7c7-40a0-9eb6-cbb04eb74fe8@i7g2000prf.googlegroups.com> Message-ID: On Tue, 1 Apr 2008, harryos apparently wrote: > i need to calculate > wk=uk o (L-Psi) > where > uk=a vector of size (1 X N^2) > o =scalar product > l,Psi=vectors of (N^2 X 1) > i have an ndarray U of shape(M X N^2)where uk is one of the rows , and > L of shape (M X N^2) where l.transpose() is one of the rows, Your explanation is not fully clear to me, but perhaps the following will help. #dummy values rows = 2 cols = 3*3 U = numpy.ones((rows,cols)) + [[0],[1]] L = numpy.random.random((rows,cols)) - 0.5 Psi = numpy.ones((cols,1)) #computation numpy.dot(U,L.transpose() - Psi) hth, Alan Isaac From oswald.harry at gmail.com Wed Apr 2 03:00:08 2008 From: oswald.harry at gmail.com (harryos) Date: Wed, 2 Apr 2008 00:00:08 -0700 (PDT) Subject: [Numpy-discussion] coding Turk&Pentland equation Message-ID: i was trying to code the equation 7 of Turk,Pentland paper 'eigenfaces for recognition' The equation says for an image l ,the components in eigen space is wk=uk.T (l-Psi) where uk is a single eigenface vector and Psi is the average image i have an ndarray L that contains data of 1 image per row. if there are M total images each of N pixels ,then L is of shape(MxN) I calculated eigenfaces(U) such that each row is an eigenface(ie,uk) U is of shape(MxN) I saw in a posting http://groups.google.com/group/sci.image.processing/browse_thread/thread/7239ab92bd7cbd62/ba15079d4441be91 that The components in 'face-space' of a face image I (Nx 1 vector) are wk = uk o (I - Psi), where Psi is the average over the M face images; o denotes scalar product, uk is eigenface k. but here i am using each image as 1xN vector .And i want to take only m eigenfaces instead of M. how should i calculate the weight space from this ?should i do W=dot(U[:m,:],(L-Psi).transpose() ) do i have to transpose this result again? thanks harryos From sjtu_yh at yahoo.com Wed Apr 2 10:10:36 2008 From: sjtu_yh at yahoo.com (yunzhi cheng) Date: Wed, 2 Apr 2008 07:10:36 -0700 (PDT) Subject: [Numpy-discussion] Compile Numpy in VC++8 In-Reply-To: <84576b39-e7c7-40a0-9eb6-cbb04eb74fe8@i7g2000prf.googlegroups.com> Message-ID: <875513.99191.qm@web36208.mail.mud.yahoo.com> Hi Everybody, I am a new user of Python. I have to re-compile Numpy in VC++8. I download the resource files ( numpy-1.0.4.tar.gz ) of Numpy at http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 I am not good at software. I wonder if anybody give me some instructions to compile it in VC++8. Best Regards, Cheng --------------------------------- You rock. That's why Blockbuster's offering you one month of Blockbuster Total Access, No Cost. -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Wed Apr 2 10:30:28 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 2 Apr 2008 16:30:28 +0200 Subject: [Numpy-discussion] Compile Numpy in VC++8 In-Reply-To: <875513.99191.qm@web36208.mail.mud.yahoo.com> References: <84576b39-e7c7-40a0-9eb6-cbb04eb74fe8@i7g2000prf.googlegroups.com> <875513.99191.qm@web36208.mail.mud.yahoo.com> Message-ID: Hi, You can do this if you have Python compiled with VC++8 as well. If not, you can't. Usually, numpy must be compiled with Visual Studio 7.1 or Mingw. Matthieu 2008/4/2, yunzhi cheng : > > Hi Everybody, > > I am a new user of Python. > I have to re-compile Numpy in VC++8. > I download the resource files ( numpy-1.0.4.tar.gz) of Numpy at > http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 > > I am not good at software. > I wonder if anybody give me some instructions to compile it in VC++8. > > Best Regards, > > Cheng > > ------------------------------ > You rock. That's why Blockbuster's offering you one month of Blockbuster > Total Access, > No Cost. > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From ellisonbg.net at gmail.com Wed Apr 2 11:32:28 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Wed, 2 Apr 2008 09:32:28 -0600 Subject: [Numpy-discussion] Behavior of __array_wrap__? Message-ID: <6ce0ac130804020832r776db1d4q2e49c145e63c890c@mail.gmail.com> Hi, I am creating a custom array type (distributed memory arrays - DistArray) and I am using the __array__ and __array_wrap__ methods and __array_priority__ attribute to get these arrays to work with numpy's ufuncs. Things are working fine when I call a ufunc like this: # This works fine (c comes back as an DistArray) a = DistArray(10) b = DistArray(10) c = np.add(a, b) But, when you pass in an ndarray as the return array, the __array_wrap__ doesn't get called: # Passing in an ndarray as the return array causes __array_wrap__ to not # be called. Thus, you get back an ndarray. d = np.empty_like(a.array) d = np.add(a, b, d) But, the most problematic case is when you try to pass in a DistArray as the return type: # This fails saying that the return array must be an ArrayType d = DistArray(10) d = np.add(a, b, d) The documentation for __array__ and __array_wrap__ describe a very different behavior: 1) The result of the ufunc is assigned to the aray returned by the __array__ method of the return array. 2) The resulting object is then passed to the __array_wrap__ method of that array to be properly wrapped. But, it seems that numpy's ufuncs are not even allowing anything but a full blown ndarray as the return type. Is this a bug (I will then file a bug report)? Has the behavior changed (documentation probably needs to be updated)? I am attaching some sample code that demonstrates the issue for a simple class. Thanks Brian -------------- next part -------------- A non-text attachment was scrubbed... Name: testarray.py Type: text/x-python-script Size: 898 bytes Desc: not available URL: From doutriaux1 at llnl.gov Wed Apr 2 11:42:47 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Wed, 02 Apr 2008 08:42:47 -0700 Subject: [Numpy-discussion] missing function in numpy.ma? In-Reply-To: <47F2B0A9.80800@enthought.com> References: <200803260948.02742.pgmdevlist@gmail.com> <200803311509.23010.pgmdevlist@gmail.com> <47F13FCA.8040003@llnl.gov> <200803311615.19175.pgmdevlist@gmail.com> <47F24FBD.7070707@llnl.gov> <47F25CC6.4070002@enthought.com> <47F26483.3010002@llnl.gov> <47F26AF1.3070000@enthought.com> <47F2713E.8040403@llnl.gov> <47F2B0A9.80800@enthought.com> Message-ID: <47F3A977.2000403@llnl.gov> Hi Travis, Thanks, I'll fix this and let you know. (probably tomorrow because I came down with some nasty "real" bug... flue or something like that) C. Travis E. Oliphant wrote: > Charles Doutriaux wrote: > >> Hi Travis, >> Ok we're almost there, in my test suite i get: >> maresult = numpy.core.ma.take(ta, indices, axis=axis) >> AttributeError: 'module' object has no attribute 'ma' >> data = numpy.core.ma.take(ax[:], indices) >> AttributeError: 'module' object has no attribute 'ma' >> >> > > I think the problem here is that numpy.core.ma is no longer the correct > place. This should be > > numpy.oldnumeric.ma.take because numpy.oldnumeric.ma is the correct > location. > > In my mind you shouldn't really have been using "numpy.core.ma", but > instead numpy.ma because whether things are in core or lib could change > ;-) > > -Travis > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From oliphant at enthought.com Wed Apr 2 12:20:22 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Wed, 02 Apr 2008 11:20:22 -0500 Subject: [Numpy-discussion] Behavior of __array_wrap__? In-Reply-To: <6ce0ac130804020832r776db1d4q2e49c145e63c890c@mail.gmail.com> References: <6ce0ac130804020832r776db1d4q2e49c145e63c890c@mail.gmail.com> Message-ID: <47F3B246.5000006@enthought.com> Brian Granger wrote: > Hi, > > I am creating a custom array type (distributed memory arrays - > DistArray) and I am using the __array__ and __array_wrap__ methods and > __array_priority__ attribute to get these arrays to work with numpy's > ufuncs. Things are working fine when I call a ufunc like this: > > # This works fine (c comes back as an DistArray) > a = DistArray(10) > b = DistArray(10) > c = np.add(a, b) > > But, when you pass in an ndarray as the return array, the > __array_wrap__ doesn't get called: > That is true. Currently, the output argument must be an ndarray, because the idea is to save memory. If you have an object that uses __array_wrap__ to channel the output back into your object's memory, then there is no benefit to the output value. It could be possible to allow additional output arguments that are not ndarrays to be syntactic sugar for __array_wrap__, but this has not been done and if the documentation led you to believe that it was possible, then the docs need to be updated. Best regards, -Travis O. From aitagi at gmail.com Wed Apr 2 12:31:53 2008 From: aitagi at gmail.com (Amit Itagi) Date: Wed, 2 Apr 2008 12:31:53 -0400 Subject: [Numpy-discussion] Numpy installation In-Reply-To: <3d375d730804011436o7ecf4adaqa5c7322cfe529876@mail.gmail.com> References: <3d375d730803312006q409e9168q1d607ac339c45d30@mail.gmail.com> <3d375d730804011021k27fc8d97h2ac5c1c1a0abaf81@mail.gmail.com> <3d375d730804011144k280aa4b3ic51ec96f4a76f5@mail.gmail.com> <3d375d730804011408w5a5e1d9dy8960f3bbae0c33e2@mail.gmail.com> <3d375d730804011436o7ecf4adaqa5c7322cfe529876@mail.gmail.com> Message-ID: Great !! This works for me. Thanks for your help !! Rgds, Amit On Tue, Apr 1, 2008 at 5:36 PM, Robert Kern wrote: > On Tue, Apr 1, 2008 at 4:21 PM, Amit Itagi wrote: > > This directory is just the Python source distribution (post configure > and > > make). I don't have root permissions to our cluster and the default > python > > distribution is an older one. Hence, I have my custom Python > distribution in > > this /Python-2.5.2/ directory. The binary is /Python-2.5.2/python . > > Okay, don't do that. You will have to actually install Python to > another location. For example, make a directory ~/python2.5/. Now go > to the Python source directory; it would probably be best to start > with a clean one. Configure Python using ~/python2.5 as the prefix: > > $ ./configure --prefix=~/python2.5 > > Now "make" and "make install". Add ~/python2.5/bin to your $PATH, > preferably before /usr/bin or wherever the old python executable is. > Build and install numpy using the ~/python2.5/bin/python binary. You > should not need to set the --prefix. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ellisonbg.net at gmail.com Wed Apr 2 13:38:23 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Wed, 2 Apr 2008 11:38:23 -0600 Subject: [Numpy-discussion] Behavior of __array_wrap__? In-Reply-To: <47F3B246.5000006@enthought.com> References: <6ce0ac130804020832r776db1d4q2e49c145e63c890c@mail.gmail.com> <47F3B246.5000006@enthought.com> Message-ID: <6ce0ac130804021038l23dc1d64lba3f2ec511c2ee62@mail.gmail.com> Travis, Thanks. Here is the text from the numpybook that was confusing me: >From section 9.1.2 on ufuncs: The ufuncs can also all take output arguments. The output will be cast if necessary to the provided output array. If a class with an array method is used for the output, results will be written to the ob ject returned by array . Then, if the class also has an array wrap method, the returned ndarray result will be passed to that method just before passing control back to the caller. I can easily work around this for now though, so it is not a problem. Cheers, Brian On Wed, Apr 2, 2008 at 10:20 AM, Travis E. Oliphant wrote: > Brian Granger wrote: > > Hi, > > > > I am creating a custom array type (distributed memory arrays - > > DistArray) and I am using the __array__ and __array_wrap__ methods and > > __array_priority__ attribute to get these arrays to work with numpy's > > ufuncs. Things are working fine when I call a ufunc like this: > > > > # This works fine (c comes back as an DistArray) > > a = DistArray(10) > > b = DistArray(10) > > c = np.add(a, b) > > > > But, when you pass in an ndarray as the return array, the > > __array_wrap__ doesn't get called: > > > > That is true. Currently, the output argument must be an ndarray, > because the idea is to save memory. > > If you have an object that uses __array_wrap__ to channel the output > back into your object's memory, then there is no benefit to the output > value. > > It could be possible to allow additional output arguments that are not > ndarrays to be syntactic sugar for __array_wrap__, but this has not been > done and if the documentation led you to believe that it was possible, > then the docs need to be updated. > > Best regards, > > -Travis O. > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From stefan at sun.ac.za Wed Apr 2 17:25:39 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 2 Apr 2008 23:25:39 +0200 Subject: [Numpy-discussion] Applying PIL patch In-Reply-To: <818605.42949.qm@web50902.mail.re2.yahoo.com> References: <818605.42949.qm@web50902.mail.re2.yahoo.com> Message-ID: <9457e7c80804021425o7ea89873p539930c8d0cb3355@mail.gmail.com> Hi Izak On Tue, Apr 1, 2008 at 10:08 AM, izak marais wrote: > St?fan wrote: > "Unfortunately, RGBA images cannot be read this way. " > Apparently it does not work with 16bit greyscale tif images either. > For anyone else stumbling upon this thread, there is a work-about to get the > data into a numpy array. > > i = Image.open('16bitGreyscaleImage.tif') > a = numpy.array(i.getdata()) # a 1d numpy array > a = a.reshape(i.size) #2d numpy array > > Perhaps there is a better way of doing it (?), but this works for me. Would you do me a favour and see whether the patch from the Image-SIG mailing list fixes your problem as well? If not, please mail me an example 16-bit greyscale tiff off-list, so that I can rework the patch. Thanks St?fan From stefan at sun.ac.za Wed Apr 2 17:42:02 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 2 Apr 2008 23:42:02 +0200 Subject: [Numpy-discussion] confusion in eigenlib code In-Reply-To: References: Message-ID: <9457e7c80804021442t104b109di29694132a0e07c0c@mail.gmail.com> Hi Gordon On Tue, Apr 1, 2008 at 5:09 PM, gordon wrote: > hi > i came across some code for eigenface construction from some > images ,using the old Numeric . > http://www.owlnet.rice.edu/~elec301/Projects99/faces/code.html > In the eigenlib.py > http://www.owlnet.rice.edu/~elec301/Projects99/faces/code/eigenlib.py > i converted the calls to Numeric functions to their numpy equivalents > (to linalg.eigh() and numpy.dot())and ran the code.In this eigenlib.py > i am confused by some parts where they derrive eigenvectors and sort > them Sorry, I don't have an answer to your question, but I'd like to point you to some more Eigenfaces code. Have a look at the Py4Science examples in Matplotlib SVN: http://matplotlib.svn.sourceforge.net/viewvc/matplotlib/trunk/py4science/examples/faces/ Regards St?fan From charlesr.harris at gmail.com Wed Apr 2 18:23:10 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 2 Apr 2008 16:23:10 -0600 Subject: [Numpy-discussion] Thoughts for 1.1 Message-ID: Hi All, I think it would enhance broadcasting if functions like sum, mean, etc didn't change the number of dimensions. For example, suppose one wanted to subtract the mean along dimension 2 from the same axis of the original array, then something like In [44]: a = ones((2,3,4,5)) In [45]: a -= a.mean(2) would do the trick. Similar modifications might also suit functions of the argmax, argmin, argsort type and allow a common argtake function that would allow one to take along a specified axis, making easy something that is somewhat complicated at the moment. The main drawback that I see is that scalars would no longer be 0D, but that could be special cased as scalars will broadcast correctly no matter the ndim. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Apr 2 18:30:49 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 2 Apr 2008 17:30:49 -0500 Subject: [Numpy-discussion] Thoughts for 1.1 In-Reply-To: References: Message-ID: <3d375d730804021530r239f0732n21e56e7d50787ff7@mail.gmail.com> On undefined, Charles R Harris wrote: > Hi All, > > I think it would enhance broadcasting if functions like sum, mean, etc > didn't change the number of dimensions. For example, suppose one wanted to > subtract the mean along dimension 2 from the same axis of the original > array, then something like > > In [44]: a = ones((2,3,4,5)) > > In [45]: a -= a.mean(2) > > would do the trick. Similar modifications might also suit functions of the > argmax, argmin, argsort type and allow a common argtake function that would > allow one to take along a specified axis, making easy something that is > somewhat complicated at the moment. > > The main drawback that I see is that scalars would no longer be 0D, but that > could be special cased as scalars will broadcast correctly no matter the > ndim. -1 I really don't want to see this amount of code breakage, even in 1.1. Add another keyword argument if you wish, but don't break the current API. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at enthought.com Wed Apr 2 18:40:13 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Wed, 02 Apr 2008 17:40:13 -0500 Subject: [Numpy-discussion] Thoughts for 1.1 In-Reply-To: References: Message-ID: <47F40B4D.6030407@enthought.com> Charles R Harris wrote: > Hi All, > > I think it would enhance broadcasting if functions like sum, mean, etc > didn't change the number of dimensions. For example, suppose one > wanted to subtract the mean along dimension 2 from the same axis of > the original array, then something like > > In [44]: a = ones((2,3,4,5)) > > In [45]: a -= a.mean(2) > > would do the trick. Similar modifications might also suit functions of > the argmax, argmin, argsort type and allow a common argtake function > that would allow one to take along a specified axis, making easy > something that is somewhat complicated at the moment. I generally like the idea because I've seen this pattern many times myself and been annoyed at having to "add back" a dimension to make it work right. > > The main drawback that I see is that scalars would no longer be 0D, > but that could be special cased as scalars will broadcast correctly no > matter the ndim. Robert's point about code-breakage is relevant, however. I'd like to see some discussion on how gratuitous this actually is. What kind of code will actually break? Sure, the shape will be different, but will that matter greatly? Basically a.mean(2).squeeze() would restore current behavior. Alternatively, a.mean(axis=2, keepshape=1) could also be used, right? The advantage is that this could be added now. -Travis > Chuck > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Wed Apr 2 18:40:15 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 2 Apr 2008 16:40:15 -0600 Subject: [Numpy-discussion] Thoughts for 1.1 In-Reply-To: <3d375d730804021530r239f0732n21e56e7d50787ff7@mail.gmail.com> References: <3d375d730804021530r239f0732n21e56e7d50787ff7@mail.gmail.com> Message-ID: On Wed, Apr 2, 2008 at 4:30 PM, Robert Kern wrote: > On undefined, Charles R Harris wrote: > > Hi All, > > > > I think it would enhance broadcasting if functions like sum, mean, etc > > didn't change the number of dimensions. For example, suppose one wanted > to > > subtract the mean along dimension 2 from the same axis of the original > > array, then something like > > > > In [44]: a = ones((2,3,4,5)) > > > > In [45]: a -= a.mean(2) > > > > would do the trick. Similar modifications might also suit functions of > the > > argmax, argmin, argsort type and allow a common argtake function that > would > > allow one to take along a specified axis, making easy something that is > > somewhat complicated at the moment. > > > > The main drawback that I see is that scalars would no longer be 0D, but > that > > could be special cased as scalars will broadcast correctly no matter the > > ndim. > > -1 > > I really don't want to see this amount of code breakage, even in 1.1. > Add another keyword argument if you wish, but don't break the current > API. > Apart from that, what do you think of the idea? I currently spend more effort than I like doing newaxis magic. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Apr 2 18:45:56 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 2 Apr 2008 17:45:56 -0500 Subject: [Numpy-discussion] Thoughts for 1.1 In-Reply-To: References: <3d375d730804021530r239f0732n21e56e7d50787ff7@mail.gmail.com> Message-ID: <3d375d730804021545vced2c6dp95b2589b12291eaf@mail.gmail.com> On undefined, Charles R Harris wrote: > > On Wed, Apr 2, 2008 at 4:30 PM, Robert Kern wrote: > > > > On undefined, Charles R Harris wrote: > > > Hi All, > > > > > > I think it would enhance broadcasting if functions like sum, mean, etc > > > didn't change the number of dimensions. For example, suppose one wanted > to > > > subtract the mean along dimension 2 from the same axis of the original > > > array, then something like > > > > > > In [44]: a = ones((2,3,4,5)) > > > > > > In [45]: a -= a.mean(2) > > > > > > would do the trick. Similar modifications might also suit functions of > the > > > argmax, argmin, argsort type and allow a common argtake function that > would > > > allow one to take along a specified axis, making easy something that is > > > somewhat complicated at the moment. > > > > > > The main drawback that I see is that scalars would no longer be 0D, but > that > > > could be special cased as scalars will broadcast correctly no matter the > > > ndim. > > > > -1 > > > > I really don't want to see this amount of code breakage, even in 1.1. > > Add another keyword argument if you wish, but don't break the current > > API. > > Apart from that, what do you think of the idea? I currently spend more > effort than I like doing newaxis magic. It's fine. I don't encounter the use case all that frequently, so personally, I want it as new functionality rather than changing the old functionality. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From strang at nmr.mgh.harvard.edu Wed Apr 2 18:48:39 2008 From: strang at nmr.mgh.harvard.edu (Gary Strangman) Date: Wed, 2 Apr 2008 18:48:39 -0400 (EDT) Subject: [Numpy-discussion] Thoughts for 1.1 In-Reply-To: References: <3d375d730804021530r239f0732n21e56e7d50787ff7@mail.gmail.com> Message-ID: If you're looking for user input ... +1 on having a keepdims capability. I have myself implemented many such functions with a keepdims=1 keyword. No real preference on how it's impelemented, though the potential for breakage is a concern ... Gary On Wed, 2 Apr 2008, Charles R Harris wrote: > On Wed, Apr 2, 2008 at 4:30 PM, Robert Kern wrote: > >> On undefined, Charles R Harris wrote: >>> Hi All, >>> >>> I think it would enhance broadcasting if functions like sum, mean, etc >>> didn't change the number of dimensions. For example, suppose one wanted >> to >>> subtract the mean along dimension 2 from the same axis of the original >>> array, then something like >>> >>> In [44]: a = ones((2,3,4,5)) >>> >>> In [45]: a -= a.mean(2) >>> >>> would do the trick. Similar modifications might also suit functions of >> the >>> argmax, argmin, argsort type and allow a common argtake function that >> would >>> allow one to take along a specified axis, making easy something that is >>> somewhat complicated at the moment. >>> >>> The main drawback that I see is that scalars would no longer be 0D, but >> that >>> could be special cased as scalars will broadcast correctly no matter the >>> ndim. >> >> -1 >> >> I really don't want to see this amount of code breakage, even in 1.1. >> Add another keyword argument if you wish, but don't break the current >> API. >> > > Apart from that, what do you think of the idea? I currently spend more > effort than I like doing newaxis magic. > > Chuck > From charlesr.harris at gmail.com Wed Apr 2 18:50:14 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 2 Apr 2008 16:50:14 -0600 Subject: [Numpy-discussion] Thoughts for 1.1 In-Reply-To: <47F40B4D.6030407@enthought.com> References: <47F40B4D.6030407@enthought.com> Message-ID: On Wed, Apr 2, 2008 at 4:40 PM, Travis E. Oliphant wrote: > Charles R Harris wrote: > > Hi All, > > > > I think it would enhance broadcasting if functions like sum, mean, etc > > didn't change the number of dimensions. For example, suppose one > > wanted to subtract the mean along dimension 2 from the same axis of > > the original array, then something like > > > > In [44]: a = ones((2,3,4,5)) > > > > In [45]: a -= a.mean(2) > > > > would do the trick. Similar modifications might also suit functions of > > the argmax, argmin, argsort type and allow a common argtake function > > that would allow one to take along a specified axis, making easy > > something that is somewhat complicated at the moment. > > I generally like the idea because I've seen this pattern many times > myself and been annoyed at having to "add back" a dimension to make it > work right. > > > > The main drawback that I see is that scalars would no longer be 0D, > > but that could be special cased as scalars will broadcast correctly no > > matter the ndim. > > Robert's point about code-breakage is relevant, however. I'd like to > see some discussion on how gratuitous this actually is. What kind of > code will actually break? Sure, the shape will be different, but will > that matter greatly? > It would break all my current code where I add the missing axis back in. Apart from that and the scalar case I don't think it would make much difference. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at ieee.org Wed Apr 2 19:10:40 2008 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Wed, 2 Apr 2008 16:10:40 -0700 Subject: [Numpy-discussion] Thoughts for 1.1 In-Reply-To: <3d375d730804021545vced2c6dp95b2589b12291eaf@mail.gmail.com> References: <3d375d730804021530r239f0732n21e56e7d50787ff7@mail.gmail.com> <3d375d730804021545vced2c6dp95b2589b12291eaf@mail.gmail.com> Message-ID: [SNIP] The text is getting kind of broken up so I'm chopping it and starting from scratch. To the question of whether it's a good idea to change the default behavior of mean and friends to not reduce over the chosen axis, I have to agree with Robert: too much code breakage for to little gain, so I'd give it a -1 as well. As to it's general usefulness, I'm torn. I've definitely run into situations where I've had to add an axis that has been reduced away. On the other hand, if the default behavior was reversed one might well end up with a comparable number of cases where you had to manually reduce the result. And I don't think that the squeeze result will work in general, since when working with arrays of higher dimensions you sometimes want to keep a specific dimension of length-1 so everything broadcasts correctly. That's admittedly fairly rare though. As to what to name it, if it did come to pass. I'm not happy 'keepshape' since we'd not actually be keeping the shape, just the number of dimensions. 'keepdims' is better, but still seem awkard. I'd prefer something like 'reduce', so the signature would be a: mean(axis=None, dtype=None, out=None, reduce=True). -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Apr 2 19:17:59 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 2 Apr 2008 18:17:59 -0500 Subject: [Numpy-discussion] Thoughts for 1.1 In-Reply-To: References: <3d375d730804021530r239f0732n21e56e7d50787ff7@mail.gmail.com> <3d375d730804021545vced2c6dp95b2589b12291eaf@mail.gmail.com> Message-ID: <3d375d730804021617p449f1dafne6f06aade13e5700@mail.gmail.com> On undefined, Timothy Hochberg wrote: > As to what to name it, if it did come to pass. I'm not happy 'keepshape' > since we'd not actually be keeping the shape, just the number of dimensions. > 'keepdims' is better, but still seem awkard. I'd prefer something like > 'reduce', so the signature would be a: > mean(axis=None, dtype=None, out=None, reduce=True). I imagine we would end up adding this capability to the .reduce() method of all binary ufuncs, so 'reduce' seems ... odd. 'keeprank'? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From jh at physics.ucf.edu Wed Apr 2 19:28:02 2008 From: jh at physics.ucf.edu (Joe Harrington) Date: Wed, 02 Apr 2008 19:28:02 -0400 Subject: [Numpy-discussion] Thoughts for 1.1 In-Reply-To: (numpy-discussion-request@scipy.org) References: Message-ID: > I think it would enhance broadcasting if functions like sum, mean, etc > didn't change the number of dimensions. I strongly favor doing it, but with keepshape (or just "keep", to make it short) and not by default. It's at least as common to take a mean down an axis of a 2D array and plot it (requiring 1D) as to subtract it in a broadcasting way. But, the latter case is common. How about generalizing it to take a mean of more than one axis at a time, say to reduce 4 dimensions to 1 for plotting? That could be done by accepting a linear list of axis numbers to average over. a = ones((2,3,4,5)) b = a.mean([1,2,3]) I'd add an option to handle NaN data as missing, as well. That's standard in other array languages. The option is called "NAN" in IDL and I'd propose that name as well. Also, from the docstring... to compute the standard deviation of the flattened array. should say to compute the mean of the flattened array. Oops. But, it emphasizes the point that these behaviors should be universal to similar routines. Are there other routines for which these behaviors would not work well? --jh-- From efiring at hawaii.edu Wed Apr 2 19:34:19 2008 From: efiring at hawaii.edu (Eric Firing) Date: Wed, 02 Apr 2008 13:34:19 -1000 Subject: [Numpy-discussion] Thoughts for 1.1 In-Reply-To: <3d375d730804021617p449f1dafne6f06aade13e5700@mail.gmail.com> References: <3d375d730804021530r239f0732n21e56e7d50787ff7@mail.gmail.com> <3d375d730804021545vced2c6dp95b2589b12291eaf@mail.gmail.com> <3d375d730804021617p449f1dafne6f06aade13e5700@mail.gmail.com> Message-ID: <47F417FB.7060500@hawaii.edu> Robert Kern wrote: > On undefined, Timothy Hochberg wrote: > >> As to what to name it, if it did come to pass. I'm not happy 'keepshape' >> since we'd not actually be keeping the shape, just the number of dimensions. >> 'keepdims' is better, but still seem awkard. I'd prefer something like >> 'reduce', so the signature would be a: >> mean(axis=None, dtype=None, out=None, reduce=True). > > I imagine we would end up adding this capability to the .reduce() > method of all binary ufuncs, so 'reduce' seems ... odd. 'keeprank'? > What is maintained is the ndim attribute, so how about "keepndim"? Or, how about "squeeze=True" for the present behavior, "squeeze=False" for the new option? The only ambiguity would be whether "squeeze=True" is equivalent to calling the squeeze method, which should not be the case. I don't think this ambiguity in the terminology would cause much trouble in practice. Eric From sjtu_yh at yahoo.com Wed Apr 2 22:21:16 2008 From: sjtu_yh at yahoo.com (yunzhi cheng) Date: Wed, 2 Apr 2008 19:21:16 -0700 (PDT) Subject: [Numpy-discussion] Compile Numpy in VC++8 In-Reply-To: Message-ID: <540427.82339.qm@web36207.mail.mud.yahoo.com> Hi Matthieu, I have to call a commercial software APIs from Python. If I import Numpy and that software package in one script, they are not compatible. Sometimes error occurs. But either one of them works well in Python. Just they cannot exist together. The supporters of the commercial software told me that their software is complied in VC++8. It is not compatible with Numpy since Numpy is compiled not in VC++8. So they told me to try to compile Numpy in VC++8. It is a hard task for me to do that. But I have to use Numpy in my python script since I want to use some complicated algorithms. Any help is appreciated. Yunzhi Cheng Matthieu Brucher wrote: Hi, You can do this if you have Python compiled with VC++8 as well. If not, you can't. Usually, numpy must be compiled with Visual Studio 7.1 or Mingw. Matthieu 2008/4/2, yunzhi cheng : Hi Everybody, I am a new user of Python. I have to re-compile Numpy in VC++8. I download the resource files ( numpy-1.0.4.tar.gz ) of Numpy at http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 I am not good at software. I wonder if anybody give me some instructions to compile it in VC++8. Best Regards, Cheng --------------------------------- You rock. That's why Blockbuster's offering you one month of Blockbuster Total Access, No Cost. _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion --------------------------------- You rock. That's why Blockbuster's offering you one month of Blockbuster Total Access, No Cost. -------------- next part -------------- An HTML attachment was scrubbed... URL: From millman at berkeley.edu Thu Apr 3 00:36:03 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 2 Apr 2008 21:36:03 -0700 Subject: [Numpy-discussion] Final push for NumPy 1.0.5 (I need your help!) Message-ID: Hello, I know that I am beginning to sound like a broken record, but I think we are finally ready to roll out NumPy 1.0.5. Since my last email about 60 bug tickets have been closed. As of tonight I believe that there is no know regressions to justify further delaying this release. Unless something comes up between now and next week, I plan to release NumPy 1.0.5 by the end of next week (4/11). There are only 28 remaining open tickets. I would love to see as many of these tickets closed before next week. So if you have anytime at all, please take a quick look over the open tickets and see if there is anything you can quickly close: http://scipy.org/scipy/numpy/query?status=new&status=assigned&status=reopened&milestone=1.0.5 Also take a look at the current release notes for 1.0.5 and let me know if you see anything missing or incorrect: http://scipy.org/scipy/numpy/milestone/1.0.5 Since this may well be the last NumPy 1.0.x release (get ready to start working on the 1.1.0!), I hope that everyone will take time over the next week to build and test the trunk. Again, if you have any time to help close tickets or improve documentation, please take the time over the next few days to do so. And thank you to everyone who has been working to get this release ready! Cheers, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From matthieu.brucher at gmail.com Thu Apr 3 01:51:25 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 3 Apr 2008 07:51:25 +0200 Subject: [Numpy-discussion] Compile Numpy in VC++8 In-Reply-To: <540427.82339.qm@web36207.mail.mud.yahoo.com> References: <540427.82339.qm@web36207.mail.mud.yahoo.com> Message-ID: Hi, As I've said, you must start by compiling Python with VC++ 8, that means using the 2.6 alpha. Matthieu 2008/4/3, yunzhi cheng : > > Hi Matthieu, > > I have to call a commercial software APIs from Python. If I import Numpy > and that software package in one script, they are not compatible. Sometimes > error occurs. But either one of them works well in Python. Just they cannot > exist together. > The supporters of the commercial software told me that their software is > complied in VC++8. It is not compatible with Numpy since Numpy is compiled > not in VC++8. So they told me to try to compile Numpy in VC++8. > It is a hard task for me to do that. > But I have to use Numpy in my python script since I want to use some > complicated algorithms. > Any help is appreciated. > > Yunzhi Cheng > > > *Matthieu Brucher * wrote: > > Hi, > > You can do this if you have Python compiled with VC++8 as well. If not, > you can't. Usually, numpy must be compiled with Visual Studio 7.1 or Mingw. > > Matthieu > > 2008/4/2, yunzhi cheng : > > > > Hi Everybody, > > > > I am a new user of Python. > > I have to re-compile Numpy in VC++8. > > I download the resource files ( numpy-1.0.4.tar.gz) of Numpy at > > http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 > > > > I am not good at software. > > I wonder if anybody give me some instructions to compile it in VC++8. > > > > Best Regards, > > > > Cheng > > ------------------------------ > > You rock. That's why Blockbuster's offering you one month of Blockbuster > > Total Access, > > No Cost. > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > -- > French PhD student > Website : http://matthieu-brucher.developpez.com/ > Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > LinkedIn : http://www.linkedin.com/in/matthieubrucher_______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > ------------------------------ > You rock. That's why Blockbuster's offering you one month of Blockbuster > Total Access, > No Cost. > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at cheimes.de Thu Apr 3 06:24:32 2008 From: lists at cheimes.de (Christian Heimes) Date: Thu, 03 Apr 2008 12:24:32 +0200 Subject: [Numpy-discussion] Compile Numpy in VC++8 In-Reply-To: References: <540427.82339.qm@web36207.mail.mud.yahoo.com> Message-ID: Matthieu Brucher schrieb: > Hi, > > As I've said, you must start by compiling Python with VC++ 8, that means > using the 2.6 alpha. Negative Houston Python 2.6 and 3.0 are using VS 2008 aka VC 9.0 Christian From matthieu.brucher at gmail.com Thu Apr 3 06:36:25 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 3 Apr 2008 12:36:25 +0200 Subject: [Numpy-discussion] Compile Numpy in VC++8 In-Reply-To: References: <540427.82339.qm@web36207.mail.mud.yahoo.com> Message-ID: Sorry, I mixed up the version numbers :| This means that you are on your own with compiling Python with VC++8 (2005). Matthieu 2008/4/3, Christian Heimes : > > Matthieu Brucher schrieb: > > > Hi, > > > > As I've said, you must start by compiling Python with VC++ 8, that means > > using the 2.6 alpha. > > > Negative Houston > > Python 2.6 and 3.0 are using VS 2008 aka VC 9.0 > > > Christian > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Thu Apr 3 12:24:05 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 03 Apr 2008 09:24:05 -0700 Subject: [Numpy-discussion] Compile Numpy in VC++8 In-Reply-To: <540427.82339.qm@web36207.mail.mud.yahoo.com> References: <540427.82339.qm@web36207.mail.mud.yahoo.com> Message-ID: <47F504A5.1090607@noaa.gov> yunzhi cheng wrote: > I have to call a commercial software APIs from Python. If I import Numpy > and that software package in one script, they are not compatible. > Sometimes error occurs. But either one of them works well in Python. > Just they cannot exist together. > The supporters of the commercial software told me that their software is > complied in VC++8. So they have apparently compiled a python extension with VC++8 that works? Are you sure that it works with the "standard" Windows Python Build (Which version of Python, by the way?). From everything I've read, that can't be done, at least not easily. I've never understood that, as I think the issue is with the libraries, not the compiler, and you'd think that there'd be a way to use the old libs with the new compiler, but I know folks a lot smarter than me haven't figure out how. However, it looks like either your commercial software developer has figured out a way, or you misunderstood what they are doing -- maybe they have compiled their own Python? I'd push them for more of an explanation. One other option is to use try MingGW for building numpy -- you can build Python extensions with MingGW without conflict, so it may just work for you. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From matthieu.brucher at gmail.com Thu Apr 3 12:44:28 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 3 Apr 2008 18:44:28 +0200 Subject: [Numpy-discussion] Compile Numpy in VC++8 In-Reply-To: <47F504A5.1090607@noaa.gov> References: <540427.82339.qm@web36207.mail.mud.yahoo.com> <47F504A5.1090607@noaa.gov> Message-ID: > > > The supporters of the commercial software told me that their software is > > complied in VC++8. > > > So they have apparently compiled a python extension with VC++8 that > works? Are you sure that it works with the "standard" Windows Python > Build (Which version of Python, by the way?). From everything I've read, > that can't be done, at least not easily. I've never understood that, as > I think the issue is with the libraries, not the compiler, and you'd > think that there'd be a way to use the old libs with the new compiler, > but I know folks a lot smarter than me haven't figure out how. > I use Scons to build my Python modules (everything is in my blog if someone wonders) and I use Visual C++ 2005 with the official Python 2.5 binary, and everything works just fine. For the moment. If I start using some complicated stuff like FILE*, it won't work anymore, but for "simple" modules, it works really fine. Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From doutriaux1 at llnl.gov Thu Apr 3 15:24:37 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Thu, 03 Apr 2008 12:24:37 -0700 Subject: [Numpy-discussion] missing function in numpy.ma? In-Reply-To: <47F2B0A9.80800@enthought.com> References: <200803260948.02742.pgmdevlist@gmail.com> <200803311509.23010.pgmdevlist@gmail.com> <47F13FCA.8040003@llnl.gov> <200803311615.19175.pgmdevlist@gmail.com> <47F24FBD.7070707@llnl.gov> <47F25CC6.4070002@enthought.com> <47F26483.3010002@llnl.gov> <47F26AF1.3070000@enthought.com> <47F2713E.8040403@llnl.gov> <47F2B0A9.80800@enthought.com> Message-ID: <47F52EF5.9090709@llnl.gov> Travis, Pierre, Goood news what's in trunk right now seems to work great with our stuff, I only had to replace numpy.core.ma.take with numpy.ma.take (numpy.oldnumeric.ma.take didn't seem to work) So as soon as 1.0.5 is out, I'll start working toward using numpy.ma only. Also, I have one little request, i remember you said oldnumeric would be gone in the next version. But our users will have hundreds (thousands?) of script convert with "import numpy.oldnumeric.ma as MA" or "import numpy.oldnumeric as Numeric" And although we're insisting and using numpy and numpy.ma right away, i'm sure 99% of them will ignore this. So would be be horrible to leave as shortcut (with a warning while loading probably, actually we could have the warning already in numpy 1.0.5 ?) from numpy.oldnumeric back to numpy ? same for oldnumeric.ma pointing back to numpy.ma. I realize it's not really clean... At least it would be great to have the warning raised in 1.0.5 that it will disappear in 1.1. Like that users might be more inclined to start converting "cleaninly" Thanks for considering this and also thanks a LOT for all your help on this issue! C. Travis E. Oliphant wrote: > Charles Doutriaux wrote: > >> Hi Travis, >> Ok we're almost there, in my test suite i get: >> maresult = numpy.core.ma.take(ta, indices, axis=axis) >> AttributeError: 'module' object has no attribute 'ma' >> data = numpy.core.ma.take(ax[:], indices) >> AttributeError: 'module' object has no attribute 'ma' >> >> > > I think the problem here is that numpy.core.ma is no longer the correct > place. This should be > > numpy.oldnumeric.ma.take because numpy.oldnumeric.ma is the correct > location. > > In my mind you shouldn't really have been using "numpy.core.ma", but > instead numpy.ma because whether things are in core or lib could change > ;-) > > -Travis > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From lee.will at gmail.com Thu Apr 3 15:31:29 2008 From: lee.will at gmail.com (Will Lee) Date: Thu, 3 Apr 2008 14:31:29 -0500 Subject: [Numpy-discussion] problem with float64's str() Message-ID: <7f03db650804031231t39aa3000o719db8359a7769c7@mail.gmail.com> Hi, I seem to have problem with floating point printing with the latest numpy, python 2.5.2, gcc 4.1.4, and 64-bit linux: In [24]: print str(0.0012) 0.0012 In [25]: a = numpy.array([0.0012]) In [26]: print str(a[0]) 0.0011999999999999999 In [27]: print numpy.__version__ 1.0.5.dev4950 It seems like the str() behavior for float64 in the latest numpy's behavior is different than Python's default behavior (and previous numpy's behavior). As I have numerous doc tests, this seems to make many of them failed. Should the float64's str() behavior be consistent with what's described in http://docs.python.org/tut/node16.html? Is there any way to get around it so I can get to the default Python behavior? Thanks! Will -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Thu Apr 3 15:41:00 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 03 Apr 2008 14:41:00 -0500 Subject: [Numpy-discussion] missing function in numpy.ma? In-Reply-To: <47F52EF5.9090709@llnl.gov> References: <200803260948.02742.pgmdevlist@gmail.com> <200803311509.23010.pgmdevlist@gmail.com> <47F13FCA.8040003@llnl.gov> <200803311615.19175.pgmdevlist@gmail.com> <47F24FBD.7070707@llnl.gov> <47F25CC6.4070002@enthought.com> <47F26483.3010002@llnl.gov> <47F26AF1.3070000@enthought.com> <47F2713E.8040403@llnl.gov> <47F2B0A9.80800@enthought.com> <47F52EF5.9090709@llnl.gov> Message-ID: <47F532CC.5000303@enthought.com> Charles Doutriaux wrote: > Travis, Pierre, > > Goood news what's in trunk right now seems to work great with our stuff, > I only had to replace numpy.core.ma.take with numpy.ma.take > (numpy.oldnumeric.ma.take didn't seem to work) > > So as soon as 1.0.5 is out, I'll start working toward using numpy.ma only. > > Also, I have one little request, i remember you said oldnumeric would be > gone in the next version. But our users will have hundreds (thousands?) > of script convert with "import numpy.oldnumeric.ma as MA" or "import > numpy.oldnumeric as Numeric" > And although we're insisting and using numpy and numpy.ma right away, > i'm sure 99% of them will ignore this. So would be be horrible to leave > as shortcut (with a warning while loading probably, actually we could > have the warning already in numpy 1.0.5 ?) from numpy.oldnumeric back to > numpy ? same for oldnumeric.ma pointing back to numpy.ma. I realize it's > not really clean... At least it would be great to have the warning > raised in 1.0.5 that it will disappear in 1.1. Like that users might be > more inclined to start converting "cleaninly" > I'm pretty sure we will make a 1.0.X release that issues warnings prior to making a 1.1 release. -Travis From oliphant at enthought.com Thu Apr 3 15:44:43 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 03 Apr 2008 14:44:43 -0500 Subject: [Numpy-discussion] [SciPy-user] conforming to Python GIL... In-Reply-To: <47F52DE4.0@gmail.com> References: <47F52DE4.0@gmail.com> Message-ID: <47F533AB.6080807@enthought.com> fred wrote: > Hi, > > I use a lot of ConVeX OPTimsation and fortran (via f2py) routines in my > Traits app. > > As I want to compute the data and want to display them, I use threads. > > The issue I get is that data displayed (using Chaco2) are not updated > (app is frozen) while computing the input data. > > From D. Morrill's answer (the Traits guru ;-)), it appears that cvxopt > (and solve() from scipy, in fact) and fortran modules does not release > the "Python GIL (Global Intepreter Lock)". > This requires a bit of effort to solve. We need to in multiple places... 1) release the GIL 2) put semaphores into code that is not thread-safe. f2py should be modified to handle this. Other could should be modified to handle it as well. I suspect NumPy should grow an API to handle the semaphore thing easily (I think Python already has one), so these may be able to be wrapped into the MACROS already available for releasing the GIL. -Travis O. From efiring at hawaii.edu Thu Apr 3 15:47:41 2008 From: efiring at hawaii.edu (Eric Firing) Date: Thu, 03 Apr 2008 09:47:41 -1000 Subject: [Numpy-discussion] missing function in numpy.ma? In-Reply-To: <47F52EF5.9090709@llnl.gov> References: <200803260948.02742.pgmdevlist@gmail.com> <200803311509.23010.pgmdevlist@gmail.com> <47F13FCA.8040003@llnl.gov> <200803311615.19175.pgmdevlist@gmail.com> <47F24FBD.7070707@llnl.gov> <47F25CC6.4070002@enthought.com> <47F26483.3010002@llnl.gov> <47F26AF1.3070000@enthought.com> <47F2713E.8040403@llnl.gov> <47F2B0A9.80800@enthought.com> <47F52EF5.9090709@llnl.gov> Message-ID: <47F5345D.1050405@hawaii.edu> Charles Doutriaux wrote: > Travis, Pierre, > > Goood news what's in trunk right now seems to work great with our stuff, > I only had to replace numpy.core.ma.take with numpy.ma.take > (numpy.oldnumeric.ma.take didn't seem to work) I don't know if it fixes the problem you ran into, but there is a clear bug fixed by the following: efiring at manini:~/programs/py/numpy_svn/trunk/numpy/oldnumeric$ svn diff Index: ma.py =================================================================== --- ma.py (revision 4958) +++ ma.py (working copy) @@ -2264,6 +2264,6 @@ return new_average(a, axis, weights, returned) def take(a, indices, axis=0): - return new_take(a, indices, axis=0) + return new_take(a, indices, axis) Eric > > So as soon as 1.0.5 is out, I'll start working toward using numpy.ma only. > > Also, I have one little request, i remember you said oldnumeric would be > gone in the next version. But our users will have hundreds (thousands?) > of script convert with "import numpy.oldnumeric.ma as MA" or "import > numpy.oldnumeric as Numeric" > And although we're insisting and using numpy and numpy.ma right away, > i'm sure 99% of them will ignore this. So would be be horrible to leave > as shortcut (with a warning while loading probably, actually we could > have the warning already in numpy 1.0.5 ?) from numpy.oldnumeric back to > numpy ? same for oldnumeric.ma pointing back to numpy.ma. I realize it's > not really clean... At least it would be great to have the warning > raised in 1.0.5 that it will disappear in 1.1. Like that users might be > more inclined to start converting "cleaninly" > > Thanks for considering this and also thanks a LOT for all your help on > this issue! > > C. > > > Travis E. Oliphant wrote: >> Charles Doutriaux wrote: >> >>> Hi Travis, >>> Ok we're almost there, in my test suite i get: >>> maresult = numpy.core.ma.take(ta, indices, axis=axis) >>> AttributeError: 'module' object has no attribute 'ma' >>> data = numpy.core.ma.take(ax[:], indices) >>> AttributeError: 'module' object has no attribute 'ma' >>> >>> >> I think the problem here is that numpy.core.ma is no longer the correct >> place. This should be >> >> numpy.oldnumeric.ma.take because numpy.oldnumeric.ma is the correct >> location. >> >> In my mind you shouldn't really have been using "numpy.core.ma", but >> instead numpy.ma because whether things are in core or lib could change >> ;-) >> >> -Travis >> >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://projects.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From gael.varoquaux at normalesup.org Thu Apr 3 15:48:51 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 3 Apr 2008 21:48:51 +0200 Subject: [Numpy-discussion] [SciPy-user] conforming to Python GIL... In-Reply-To: <47F533AB.6080807@enthought.com> References: <47F52DE4.0@gmail.com> <47F533AB.6080807@enthought.com> Message-ID: <20080403194851.GB22709@phare.normalesup.org> On Thu, Apr 03, 2008 at 02:44:43PM -0500, Travis E. Oliphant wrote: > This requires a bit of effort to solve. We need to in multiple places... OK, then I suggest Fred looks into ipython1's parallel execution features. The problem is that I suspect it is not mature-enough for him. Go Fernando, go :->. Ga?l From robert.kern at gmail.com Thu Apr 3 15:56:29 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 3 Apr 2008 14:56:29 -0500 Subject: [Numpy-discussion] [SciPy-user] conforming to Python GIL... In-Reply-To: <20080403194851.GB22709@phare.normalesup.org> References: <47F52DE4.0@gmail.com> <47F533AB.6080807@enthought.com> <20080403194851.GB22709@phare.normalesup.org> Message-ID: <3d375d730804031256n1cca30f6t269a3806e34e1507@mail.gmail.com> On Thu, Apr 3, 2008 at 2:48 PM, Gael Varoquaux wrote: > On Thu, Apr 03, 2008 at 02:44:43PM -0500, Travis E. Oliphant wrote: > > This requires a bit of effort to solve. We need to in multiple places... > > OK, then I suggest Fred looks into ipython1's parallel execution > features. The problem is that I suspect it is not mature-enough for him. ipython1 is great, but it is somewhat overkill for a problem like this. The processing module is something worth looking at. http://pypi.python.org/pypi/processing -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From gael.varoquaux at normalesup.org Thu Apr 3 15:58:46 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 3 Apr 2008 21:58:46 +0200 Subject: [Numpy-discussion] [SciPy-user] conforming to Python GIL... In-Reply-To: <3d375d730804031256n1cca30f6t269a3806e34e1507@mail.gmail.com> References: <47F52DE4.0@gmail.com> <47F533AB.6080807@enthought.com> <20080403194851.GB22709@phare.normalesup.org> <3d375d730804031256n1cca30f6t269a3806e34e1507@mail.gmail.com> Message-ID: <20080403195846.GD22709@phare.normalesup.org> On Thu, Apr 03, 2008 at 02:56:29PM -0500, Robert Kern wrote: > ipython1 is great, but it is somewhat overkill for a problem like > this. The processing module is something worth looking at. > http://pypi.python.org/pypi/processing Wow, I didn't know this one. It seems very nice. Thanks for sharing the link. Ga?l From peridot.faceted at gmail.com Thu Apr 3 16:25:17 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 3 Apr 2008 22:25:17 +0200 Subject: [Numpy-discussion] [SciPy-user] conforming to Python GIL... In-Reply-To: <47F533AB.6080807@enthought.com> References: <47F52DE4.0@gmail.com> <47F533AB.6080807@enthought.com> Message-ID: On 03/04/2008, Travis E. Oliphant wrote: > fred wrote: > > Hi, > > > > I use a lot of ConVeX OPTimsation and fortran (via f2py) routines in my > > Traits app. > > > > As I want to compute the data and want to display them, I use threads. > > > > The issue I get is that data displayed (using Chaco2) are not updated > > (app is frozen) while computing the input data. > > > > From D. Morrill's answer (the Traits guru ;-)), it appears that cvxopt > > (and solve() from scipy, in fact) and fortran modules does not release > > the "Python GIL (Global Intepreter Lock)". > > > > This requires a bit of effort to solve. We need to in multiple places... > > 1) release the GIL > 2) put semaphores into code that is not thread-safe. > > f2py should be modified to handle this. Other could should be > modified to handle it as well. > > I suspect NumPy should grow an API to handle the semaphore thing easily > (I think Python already has one), so these may be able to be wrapped > into the MACROS already available for releasing the GIL. What is the story on f2py's releasing of the GIL? I had understood that wrapper code was able to declare the wrapped code to be thread-safe, so that the GIL was released while it executed. Is the problem here that cvxopt is not threadsafe? I guess the best solution then is to make a cvxopt- (and related routines)-specific lock? Anne From nbigaouette at gmail.com Thu Apr 3 16:30:40 2008 From: nbigaouette at gmail.com (Nicolas Bigaouette) Date: Thu, 3 Apr 2008 16:30:40 -0400 Subject: [Numpy-discussion] Efficient reading of binary data Message-ID: <1af523860804031330t173c7930rabbeb64f75d2042b@mail.gmail.com> Hi, I have a C program which outputs large (~GB) files. It is a simple binary dump of an array of structure containing 9 doubles. You can see this as a double 1D array of size 9*Stot (Stot being the allocated size of the array of structure). The 1D array represents a 3D array (Sx * Sy * Sz = Stot) containing 9 values per cell. I want to read these files in the most efficient way possible, and I would like to have your insight on this. Right now, the fastest way I found was: imzeros = zeros((Sy,Sz),dtype=float64,order='C') imex = imshow(imzeros) f = open(filename, 'rb') data = numpy.fromfile(file=f, dtype=numpy.float64, count=9*Stot) mask_Ex = numpy.arange(6,9*Stot,9) Ex = data[mask].reshape((Sz,Sy,Sx), order='C').transpose() imex.set_array(squeeze(Ex3D[:,:,z])) The arrays will be big, so everything should be well optimized. I have multiple questions: 1) Should I change this: Ex = data[mask].reshape((Sz,Sy,Sx), order='C').transpose() imex.set_array(squeeze(Ex3D[:,:,z])) to: imex.set_array(squeeze(data[mask].reshape((Sz,Sy,Sx), order='C').transpose()[:,:,z])) I mean, is I don't use a temporary variable, will it be faster or less memory hungry? 2) If not, is the operation "Ex = " update the variable data or create another one? Ideally I would like to only update it. Maybe this would be better: Ex[:,:,:] = data[mask].reshape((Sz,Sy,Sx), order='C').transpose() Would it? 3) The machine where the code will be run might be big-endian. Is there a way for python to read the big-endian file and "translate" it automatically to little-endian? Something like "numpy.fromfile(file=f, dtype=numpy.float64, count=9*Stot, endianness='big')"? Thanx a lot! ;) Nicolas -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Apr 3 16:47:28 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 3 Apr 2008 15:47:28 -0500 Subject: [Numpy-discussion] Efficient reading of binary data In-Reply-To: <1af523860804031330t173c7930rabbeb64f75d2042b@mail.gmail.com> References: <1af523860804031330t173c7930rabbeb64f75d2042b@mail.gmail.com> Message-ID: <3d375d730804031347q24591c52raceafe2524ec5c3a@mail.gmail.com> On Thu, Apr 3, 2008 at 3:30 PM, Nicolas Bigaouette wrote: > Hi, > > I have a C program which outputs large (~GB) files. It is a simple binary > dump of an array of structure containing 9 doubles. You can see this as a > double 1D array of size 9*Stot (Stot being the allocated size of the array > of structure). The 1D array represents a 3D array (Sx * Sy * Sz = Stot) > containing 9 values per cell. > > I want to read these files in the most efficient way possible, and I would > like to have your insight on this. > > Right now, the fastest way I found was: > imzeros = zeros((Sy,Sz),dtype=float64,order='C') > imex = imshow(imzeros) > f = open(filename, 'rb') > data = numpy.fromfile(file=f, dtype=numpy.float64, count=9*Stot) > mask_Ex = numpy.arange(6,9*Stot,9) This is something you can do much, much more efficiently by using a slice instead of indexing with an integer array. > Ex = data[mask].reshape((Sz,Sy,Sx), order='C').transpose() > imex.set_array(squeeze(Ex3D[:,:,z])) > > The arrays will be big, so everything should be well optimized. I have > multiple questions: > > 1) Should I change this: > Ex = data[mask].reshape((Sz,Sy,Sx), order='C').transpose() > imex.set_array(squeeze(Ex3D[:,:,z])) > to: > imex.set_array(squeeze(data[mask].reshape((Sz,Sy,Sx), > order='C').transpose()[:,:,z])) > I mean, is I don't use a temporary variable, will it be faster or less > memory hungry? No. The temporary exists whether you give it a name or not. If you use data[6::9] instead of data[mask], you won't be using any extra memory at all. The arrays will just be views into the original array. > 2) If not, is the operation "Ex = " update the variable data or create > another one? It just reassigns the name "Ex" to a different object specified on the right-hand side of the assignment. The relevant question is whether expression on the right-hand side takes up more memory. > Ideally I would like to only update it. Maybe this would be > better: > > Ex[:,:,:] = data[mask].reshape((Sz,Sy,Sx), order='C').transpose()Would it? If you use data[6::9] instead of data[mask], you should just use "Ex = " since no new memory will be used on the RHS. > 3) The machine where the code will be run might be big-endian. Is there a > way for python to read the big-endian file and "translate" it automatically > to little-endian? Something like "numpy.fromfile(file=f, > dtype=numpy.float64, count=9*Stot, endianness='big')"? dtype=numpy.dtype('>f8') -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Thu Apr 3 17:10:29 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 3 Apr 2008 16:10:29 -0500 Subject: [Numpy-discussion] Compile Numpy in VC++8 In-Reply-To: <540427.82339.qm@web36207.mail.mud.yahoo.com> References: <540427.82339.qm@web36207.mail.mud.yahoo.com> Message-ID: <3d375d730804031410j9575a9dy1b47393c3d36814f@mail.gmail.com> On Wed, Apr 2, 2008 at 9:21 PM, yunzhi cheng wrote: > Hi Matthieu, > > I have to call a commercial software APIs from Python. If I import Numpy and > that software package in one script, they are not compatible. Sometimes > error occurs. But either one of them works well in Python. Just they cannot > exist together. > The supporters of the commercial software told me that their software is > complied in VC++8. It is not compatible with Numpy since Numpy is compiled > not in VC++8. So they told me to try to compile Numpy in VC++8. > It is a hard task for me to do that. > But I have to use Numpy in my python script since I want to use some > complicated algorithms. > Any help is appreciated. Different versions of Microsoft's compiler use different libraries for the standard C library. Some simple Python extension modules compiled with a different compiler than the Python interpreter will usually work. The commercial library you are using appears to be one of them. numpy is not one of them. If the commercial module must be built with VC++8, then you will need to also get *both* the Python interpreter and numpy built with VC++8. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Chris.Barker at noaa.gov Thu Apr 3 17:52:24 2008 From: Chris.Barker at noaa.gov (Chris Barker) Date: Thu, 03 Apr 2008 14:52:24 -0700 Subject: [Numpy-discussion] Compile Numpy in VC++8 In-Reply-To: <3d375d730804031410j9575a9dy1b47393c3d36814f@mail.gmail.com> References: <540427.82339.qm@web36207.mail.mud.yahoo.com> <3d375d730804031410j9575a9dy1b47393c3d36814f@mail.gmail.com> Message-ID: <47F55198.9070904@noaa.gov> Robert Kern wrote: Just since that has been discussed a LOT, for years, I want to be clear: > Different versions of Microsoft's compiler use different libraries for > the standard C library. Some simple Python extension modules compiled > with a different compiler than the Python interpreter will usually > work. Do SOME modules USUALLY work just because of what routines in the standard library they happen to call? So it's just a matter of luck that a given module may not trigger a conflict between the two runtime libs? Thanks, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From robert.kern at gmail.com Thu Apr 3 17:59:58 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 3 Apr 2008 16:59:58 -0500 Subject: [Numpy-discussion] Compile Numpy in VC++8 In-Reply-To: <47F55198.9070904@noaa.gov> References: <540427.82339.qm@web36207.mail.mud.yahoo.com> <3d375d730804031410j9575a9dy1b47393c3d36814f@mail.gmail.com> <47F55198.9070904@noaa.gov> Message-ID: <3d375d730804031459o1d4d02aapb7a9ced1e1105179@mail.gmail.com> On Thu, Apr 3, 2008 at 4:52 PM, Chris Barker wrote: > Robert Kern wrote: > > Just since that has been discussed a LOT, for years, I want to be clear: > > > Different versions of Microsoft's compiler use different libraries for > > the standard C library. Some simple Python extension modules compiled > > with a different compiler than the Python interpreter will usually > > work. > > Do SOME modules USUALLY work just because of what routines in the > standard library they happen to call? So it's just a matter of luck that > a given module may not trigger a conflict between the two runtime libs? It is usually a matter of communication between the two systems. If an extension module with C runtime (CRT) A allocates some memory with its malloc() then passes it to the Python interpreter which tries to free it with CRT B's free() function, you will have problems. Similarly, FILE* pointers are frequent causes of problems in this regard. However, if nothing crosses the boundary between the two CRT's, there will (usually) be no problem. The failure will be deterministic, but possibly only a subset of the API of the extension module will exercise the failure mode. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From matthieu.brucher at gmail.com Thu Apr 3 18:05:53 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Fri, 4 Apr 2008 00:05:53 +0200 Subject: [Numpy-discussion] Compile Numpy in VC++8 In-Reply-To: <47F55198.9070904@noaa.gov> References: <540427.82339.qm@web36207.mail.mud.yahoo.com> <3d375d730804031410j9575a9dy1b47393c3d36814f@mail.gmail.com> <47F55198.9070904@noaa.gov> Message-ID: 2008/4/3, Chris Barker : > > Robert Kern wrote: > > Just since that has been discussed a LOT, for years, I want to be clear: > > > > Different versions of Microsoft's compiler use different libraries for > > the standard C library. Some simple Python extension modules compiled > > with a different compiler than the Python interpreter will usually > > work. > > > Do SOME modules USUALLY work just because of what routines in the > standard library they happen to call? So it's just a matter of luck that > a given module may not trigger a conflict between the two runtime libs? > Some parts of the C interface are fixed by the standard. Rely on them and you should be OK (safe perhaps for some memory functions, but I never saw a problem there). The other parts should never be trusted (as the I/O stuff). These are the rules I follow. Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From timmichelsen at gmx-topmail.de Thu Apr 3 18:44:09 2008 From: timmichelsen at gmx-topmail.de (Tim Michelsen) Date: Fri, 04 Apr 2008 00:44:09 +0200 Subject: [Numpy-discussion] loading data with gaps Message-ID: Hello! How can I load a data file (e.g. CSV, DAT) in ASCII which has some gaps? The file has been saved with from a spreadsheet program which leaves cells with not data empty: 1,23. 2,13. 3, 4,34. Would this code be correct: ### test_loadtxt.py ### import numpy import maskedarray # load data which has empty 'cells' as beeing saved from spreadsheet: # 1,23. # 2,13. # 3, # 4,34. data = numpy.loadtxt('./loadtxt_test.csv',dtype=str,delimiter=',') # create a masked array with all no data ('', empty cells from CSV) masked my_masked_array = maskedarray.masked_equal(data,'') ###### * How can I change the data type of my maskedarray (my_masked_array) to a type that allows me to perform calulations? * Would you do this task differently or more efficient? * What possibilities do I have to estimate/interpolate the masked values? A example would be nice. * How to I convert maskedarray (my_masked_array) to a array without masked values? Thanks in advance for your help, Tim Michelsenholg From strawman at astraw.com Thu Apr 3 18:46:24 2008 From: strawman at astraw.com (Andrew Straw) Date: Thu, 03 Apr 2008 15:46:24 -0700 Subject: [Numpy-discussion] Compile Numpy in VC++8 In-Reply-To: References: <540427.82339.qm@web36207.mail.mud.yahoo.com> <3d375d730804031410j9575a9dy1b47393c3d36814f@mail.gmail.com> <47F55198.9070904@noaa.gov> Message-ID: <47F55E40.9050300@astraw.com> Matthieu Brucher wrote: > > > 2008/4/3, Chris Barker >: > > Robert Kern wrote: > > Just since that has been discussed a LOT, for years, I want to be > clear: > > > > Different versions of Microsoft's compiler use different > libraries for > > the standard C library. Some simple Python extension modules > compiled > > with a different compiler than the Python interpreter will usually > > work. > > > Do SOME modules USUALLY work just because of what routines in the > standard library they happen to call? So it's just a matter of > luck that > a given module may not trigger a conflict between the two runtime > libs? > > > Some parts of the C interface are fixed by the standard. Rely on them > and you should be OK (safe perhaps for some memory functions, but I > never saw a problem there). The other parts should never be trusted > (as the I/O stuff). > These are the rules I follow. I have also had success compiling external code that required VS 2008 into a shared library (.dll) and then calling that using ctypes. I'm not sure if I was just lucky that this worked in my case, or if the windows linker can deal with the issues resulting from mixing C runtimes when using .dlls. -Andrew From nbigaouette at gmail.com Thu Apr 3 19:53:02 2008 From: nbigaouette at gmail.com (Nicolas Bigaouette) Date: Thu, 3 Apr 2008 19:53:02 -0400 Subject: [Numpy-discussion] Efficient reading of binary data In-Reply-To: <3d375d730804031347q24591c52raceafe2524ec5c3a@mail.gmail.com> References: <1af523860804031330t173c7930rabbeb64f75d2042b@mail.gmail.com> <3d375d730804031347q24591c52raceafe2524ec5c3a@mail.gmail.com> Message-ID: <1af523860804031653s1941608ehbd7147502f87f17b@mail.gmail.com> Thanx for the fast response Robert ;) I changed my code to use the slice: E = data[6::9] It is indeed faster and less eat less memory. Great. Thanx for the endiannes! I knew there was something like this ;) I suspect that, in '>f8', "f" means float and "8" means 8 bytes? >From some benchmarks, I see that the slowest thing is disk access. It can slow the displaying of data from around 1sec (when data is in os cache or buffer) to 8sec. So the next step would be to only read the needed data from the binary file... Is it possible to read from a file with a slice? So instead of: data = numpy.fromfile(file=f, dtype=float_dtype, count=9*Stot) E = data[6::9] maybe something like: E = numpy.fromfile(file=f, dtype=float_dtype, count=9*Stot, slice=6::9) Thank you! 2008/4/3, Robert Kern : > > On Thu, Apr 3, 2008 at 3:30 PM, Nicolas Bigaouette > wrote: > > Hi, > > > > I have a C program which outputs large (~GB) files. It is a simple > binary > > dump of an array of structure containing 9 doubles. You can see this as > a > > double 1D array of size 9*Stot (Stot being the allocated size of the > array > > of structure). The 1D array represents a 3D array (Sx * Sy * Sz = Stot) > > containing 9 values per cell. > > > > I want to read these files in the most efficient way possible, and I > would > > like to have your insight on this. > > > > Right now, the fastest way I found was: > > imzeros = zeros((Sy,Sz),dtype=float64,order='C') > > imex = imshow(imzeros) > > f = open(filename, 'rb') > > data = numpy.fromfile(file=f, dtype=numpy.float64, count=9*Stot) > > mask_Ex = numpy.arange(6,9*Stot,9) > > > This is something you can do much, much more efficiently by using a > slice instead of indexing with an integer array. > > > > Ex = data[mask].reshape((Sz,Sy,Sx), order='C').transpose() > > imex.set_array(squeeze(Ex3D[:,:,z])) > > > > The arrays will be big, so everything should be well optimized. I have > > multiple questions: > > > > 1) Should I change this: > > Ex = data[mask].reshape((Sz,Sy,Sx), order='C').transpose() > > imex.set_array(squeeze(Ex3D[:,:,z])) > > to: > > imex.set_array(squeeze(data[mask].reshape((Sz,Sy,Sx), > > order='C').transpose()[:,:,z])) > > I mean, is I don't use a temporary variable, will it be faster or less > > memory hungry? > > > No. The temporary exists whether you give it a name or not. If you use > data[6::9] instead of data[mask], you won't be using any extra memory > at all. The arrays will just be views into the original array. > > > > 2) If not, is the operation "Ex = " update the variable data or create > > another one? > > > It just reassigns the name "Ex" to a different object specified on the > right-hand side of the assignment. The relevant question is whether > expression on the right-hand side takes up more memory. > > > > Ideally I would like to only update it. Maybe this would be > > better: > > > > Ex[:,:,:] = data[mask].reshape((Sz,Sy,Sx), order='C').transpose()Would > it? > > > If you use data[6::9] instead of data[mask], you should just use "Ex = > " since no new memory will be used on the RHS. > > > > 3) The machine where the code will be run might be big-endian. Is there > a > > way for python to read the big-endian file and "translate" it > automatically > > to little-endian? Something like "numpy.fromfile(file=f, > > dtype=numpy.float64, count=9*Stot, endianness='big')"? > > > dtype=numpy.dtype('>f8') > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Apr 3 20:00:34 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 3 Apr 2008 19:00:34 -0500 Subject: [Numpy-discussion] Efficient reading of binary data In-Reply-To: <1af523860804031653s1941608ehbd7147502f87f17b@mail.gmail.com> References: <1af523860804031330t173c7930rabbeb64f75d2042b@mail.gmail.com> <3d375d730804031347q24591c52raceafe2524ec5c3a@mail.gmail.com> <1af523860804031653s1941608ehbd7147502f87f17b@mail.gmail.com> Message-ID: <3d375d730804031700t20d3c2fata823317b24bf3bc8@mail.gmail.com> On Thu, Apr 3, 2008 at 6:53 PM, Nicolas Bigaouette wrote: > Thanx for the fast response Robert ;) > > I changed my code to use the slice: > E = data[6::9]It is indeed faster and less eat less memory. Great. > > Thanx for the endiannes! I knew there was something like this ;) I suspect > that, in '>f8', "f" means float and "8" means 8 bytes? Yes, and the '>' means big-endian. '<' is little-endian, and '=' is native-endian. > From some benchmarks, I see that the slowest thing is disk access. It can > slow the displaying of data from around 1sec (when data is in os cache or > buffer) to 8sec. > > So the next step would be to only read the needed data from the binary > file... Is it possible to read from a file with a slice? So instead of: > > data = numpy.fromfile(file=f, dtype=float_dtype, count=9*Stot) > E = data[6::9] > maybe something like: > E = numpy.fromfile(file=f, dtype=float_dtype, count=9*Stot, slice=6::9) Instead of reading using fromfile(), you can try memory-mapping the array. from numpy import memmap E = memmap(f, dtype=float_dtype, mode='r')[6::9] That may or may not help. At least, it should decrease the latency before you start pulling out frames. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From nbigaouette at gmail.com Thu Apr 3 20:14:57 2008 From: nbigaouette at gmail.com (Nicolas Bigaouette) Date: Thu, 3 Apr 2008 20:14:57 -0400 Subject: [Numpy-discussion] Efficient reading of binary data In-Reply-To: <3d375d730804031700t20d3c2fata823317b24bf3bc8@mail.gmail.com> References: <1af523860804031330t173c7930rabbeb64f75d2042b@mail.gmail.com> <3d375d730804031347q24591c52raceafe2524ec5c3a@mail.gmail.com> <1af523860804031653s1941608ehbd7147502f87f17b@mail.gmail.com> <3d375d730804031700t20d3c2fata823317b24bf3bc8@mail.gmail.com> Message-ID: <1af523860804031714y35b69924rb57b4711dd3a69eb@mail.gmail.com> 2008/4/3, Robert Kern : > > On Thu, Apr 3, 2008 at 6:53 PM, Nicolas Bigaouette > wrote: > > Thanx for the fast response Robert ;) > > > > I changed my code to use the slice: > > E = data[6::9]It is indeed faster and less eat less memory. Great. > > > > Thanx for the endiannes! I knew there was something like this ;) I > suspect > > that, in '>f8', "f" means float and "8" means 8 bytes? > > > Yes, and the '>' means big-endian. '<' is little-endian, and '=' is > native-endian. I just tested it with a big-endian machine, it does work indeed great :) > From some benchmarks, I see that the slowest thing is disk access. It can > > slow the displaying of data from around 1sec (when data is in os cache > or > > buffer) to 8sec. > > > > So the next step would be to only read the needed data from the binary > > file... Is it possible to read from a file with a slice? So instead of: > > > > data = numpy.fromfile(file=f, dtype=float_dtype, count=9*Stot) > > E = data[6::9] > > maybe something like: > > E = numpy.fromfile(file=f, dtype=float_dtype, count=9*Stot, slice=6::9) > > > Instead of reading using fromfile(), you can try memory-mapping the array. > > from numpy import memmap > E = memmap(f, dtype=float_dtype, mode='r')[6::9] > > That may or may not help. At least, it should decrease the latency > before you start pulling out frames. > > It did not worked out of the box (memmap() takes the filename and not a file handler) but anyway, its getting late. Thanx -------------- next part -------------- An HTML attachment was scrubbed... URL: From oswald.harry at gmail.com Fri Apr 4 02:02:24 2008 From: oswald.harry at gmail.com (harryos) Date: Thu, 3 Apr 2008 23:02:24 -0700 (PDT) Subject: [Numpy-discussion] sorting ndarray Message-ID: <5a0e4665-183d-4cb3-bee3-7a8cb5284b65@s8g2000prg.googlegroups.com> i have a 1 dim numpy array D=array( [[ 3. , 2. , 1. , 4. , 5. , 1.5, 2.2]] ) i need to get this sorted in descending order and then access the elements . D.sort() will make D as [[ 1. 1.5 2. 2.2 3. 4. 5. ]] how will i reverse it? or is there a simpler way? harry From efiring at hawaii.edu Fri Apr 4 02:09:39 2008 From: efiring at hawaii.edu (Eric Firing) Date: Thu, 03 Apr 2008 20:09:39 -1000 Subject: [Numpy-discussion] sorting ndarray In-Reply-To: <5a0e4665-183d-4cb3-bee3-7a8cb5284b65@s8g2000prg.googlegroups.com> References: <5a0e4665-183d-4cb3-bee3-7a8cb5284b65@s8g2000prg.googlegroups.com> Message-ID: <47F5C623.6080108@hawaii.edu> harryos wrote: > i have a 1 dim numpy array > D=array( [[ 3. , 2. , 1. , 4. , 5. , 1.5, 2.2]] ) This is a 2-D array, not a 1-D array. > i need to get this sorted in descending order and then access the > elements . > D.sort() will make D as [[ 1. 1.5 2. 2.2 3. 4. 5. ]] > how will i reverse it? In [8]:D[:,::-1] Out[8]:array([[ 5. , 4. , 3. , 2.2, 2. , 1.5, 1. ]]) > or is there a simpler way? > harry > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From wilson.t.thompson at gmail.com Fri Apr 4 02:58:41 2008 From: wilson.t.thompson at gmail.com (wilson) Date: Thu, 3 Apr 2008 23:58:41 -0700 (PDT) Subject: [Numpy-discussion] multiply array Message-ID: hello i have two arrays #of shape (1,6) eval=array([[3.,3.2,1.,1.1,5.,0.5]]) #of shape (6,8) egimgs=array([ [3.,2.,1.,4.,5.,1.5,2.5,1.1], [1.1,3.,.5,.2,.1,4.3,3.2,1.2], [4.,3.,2.,6.,1.,4.,5.1,2.4], [3.2,1.3,2.2,4.4,1.1,2.1,3.3,2.4], [.4,.2,.1,.5,.3,.2,1.2,4.2], [5.2,2.1,4.3,5.5,2.2,1.1,1.4,3.2] ] ) i need to multiply the first row of egimgs with the first element of eval , second row of egimgs with second element of eval ..etc ...and so on..finally i should get a result array of same shape as egimgs. how should i proceed ? using loops seems inefficient.can someone suggest a better way? thanks W From millman at berkeley.edu Fri Apr 4 03:20:40 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Fri, 4 Apr 2008 00:20:40 -0700 Subject: [Numpy-discussion] Final push for NumPy 1.0.5 (I need your help!) In-Reply-To: References: Message-ID: On Wed, Apr 2, 2008 at 9:36 PM, Jarrod Millman wrote: > Again, if you have any time to help close tickets or improve > documentation, please take the time over the next few days to do so. > And thank you to everyone who has been working to get this release > ready! Since I sent my email last night another 5+ tickets have been closed. If we keep going at this rate, we should be able to release 1.0.5 next Friday (4/11) with every ticket closed. Specifically, thanks to Travis Oliphant, David Huard, and Stefan van der Walt for bug squashing. If anyone has some time, could you please check David's fix for the ticket "loadtxt fails with record arrays": http://projects.scipy.org/scipy/numpy/ticket/623 His fix looks correct to me (and even includes test!!); if someone else can confirm this, David or I will be able to close this ticket. There are now only 21 remaining open tickets. Please take a look over the open tickets and see if there is anything you can quickly close: http://scipy.org/scipy/numpy/query?status=new&status=assigned&status=reopened&milestone=1.0.5 Also please take a look at the current release notes for 1.0.5 and let me know if you see anything missing or incorrect: http://scipy.org/scipy/numpy/milestone/1.0.5 This release promises to be a very polished and stable release, which will allow us to quickly move on to developing the new 1.1 series. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From wilson.t.thompson at gmail.com Fri Apr 4 03:28:22 2008 From: wilson.t.thompson at gmail.com (wilson) Date: Fri, 4 Apr 2008 00:28:22 -0700 (PDT) Subject: [Numpy-discussion] multiply array In-Reply-To: References: Message-ID: > #of shape (1,6) > eval=array([[3.,3.2,1.,1.1,5.,0.5]]) > eval.shape=(-1,) please not the correction..i need to multiply first row of egimgs with 3.0 ,second row with 3.2,....last(sixth) row with 0.5 ..For that purpose i made the above into a 1 dimensional array. A for loop seems inefficient in the case of big arrays W From millman at berkeley.edu Fri Apr 4 03:35:41 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Fri, 4 Apr 2008 00:35:41 -0700 Subject: [Numpy-discussion] NumPy/SciPy sprint in Berkeley next week Message-ID: Hello, I have just organized a little NumPy/SciPy mini-sprint for next week. David Cournapeau is visiting me for a week and several other developers (Eric Jones, Robert Kern, Peter Wang, Jonathan Taylor, and Karl Young) will be stopping by during the week to work with the Berkeley team (Fernando Perez, Chris Burns, Tom Waite, and I). There may be a few others who will join us as well. I am still working on a preliminary list of topics that I hope for us to work on and I will send it out to the list before Monday. Among other things, I will be trying to push NumPy 1.0.5 out the door. I will send out another announcement as the date draws near, but I hope that some of you will be able to join us next week at irc.freenode.net (channel scipy). Please consider next week an official Bug/Doc week. Once we get NumPy 1.0.5 released, I will start focusing on getting SciPy 0.7.0 released--which means we need to start squashing SciPy bugs as relentlessly as we have been squashing NumPy's bugs during the last month. Thanks to everyone who is working so hard to make NumPy/SciPy the best foundation for scientific and numerical computing. Cheers, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From haase at msg.ucsf.edu Fri Apr 4 04:50:38 2008 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 4 Apr 2008 10:50:38 +0200 Subject: [Numpy-discussion] Efficient reading of binary data In-Reply-To: <1af523860804031714y35b69924rb57b4711dd3a69eb@mail.gmail.com> References: <1af523860804031330t173c7930rabbeb64f75d2042b@mail.gmail.com> <3d375d730804031347q24591c52raceafe2524ec5c3a@mail.gmail.com> <1af523860804031653s1941608ehbd7147502f87f17b@mail.gmail.com> <3d375d730804031700t20d3c2fata823317b24bf3bc8@mail.gmail.com> <1af523860804031714y35b69924rb57b4711dd3a69eb@mail.gmail.com> Message-ID: On Fri, Apr 4, 2008 at 2:14 AM, Nicolas Bigaouette wrote: > 2008/4/3, Robert Kern : > > > On Thu, Apr 3, 2008 at 6:53 PM, Nicolas Bigaouette > > wrote: > > > Thanx for the fast response Robert ;) > > > > > > I changed my code to use the slice: > > > E = data[6::9]It is indeed faster and less eat less memory. Great. > > > > > > Thanx for the endiannes! I knew there was something like this ;) I > suspect > > > that, in '>f8', "f" means float and "8" means 8 bytes? > > > > > > Yes, and the '>' means big-endian. '<' is little-endian, and '=' is > > native-endian. > > I just tested it with a big-endian machine, it does work indeed great :) > > > > > From some benchmarks, I see that the slowest thing is disk access. It > can > > > slow the displaying of data from around 1sec (when data is in os cache > or > > > buffer) to 8sec. > > > > > > So the next step would be to only read the needed data from the binary > > > file... Is it possible to read from a file with a slice? So instead of: > > > > > > data = numpy.fromfile(file=f, dtype=float_dtype, count=9*Stot) > > > E = data[6::9] > > > maybe something like: > > > E = numpy.fromfile(file=f, dtype=float_dtype, count=9*Stot, slice=6::9) > > > > > > Instead of reading using fromfile(), you can try memory-mapping the array. > > > > from numpy import memmap > > E = memmap(f, dtype=float_dtype, mode='r')[6::9] > > > > That may or may not help. At least, it should decrease the latency > > before you start pulling out frames. > > > > > It did not worked out of the box (memmap() takes the filename and not a file > handler) but anyway, its getting late. > Hi, Accidentally I'm exactly trying to do the same thing right now ..... What is the best way of memmapping into a file that is already open !? I have to read some text (header info) off the beginning of the file before I know where the data actually starts. I could of course get the position at that point ( f.tell() ) close the file, and reopen using memmap. However this doesn't sound optimal to me .... Any hints ? Could numpy's memmap be changed to also accept file-objects, or there a "rule" that memmap always has to have access to the entire file ? Thanks, Sebastian Haase From millman at berkeley.edu Fri Apr 4 05:33:38 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Fri, 4 Apr 2008 02:33:38 -0700 Subject: [Numpy-discussion] Efficient reading of binary data In-Reply-To: References: <1af523860804031330t173c7930rabbeb64f75d2042b@mail.gmail.com> <3d375d730804031347q24591c52raceafe2524ec5c3a@mail.gmail.com> <1af523860804031653s1941608ehbd7147502f87f17b@mail.gmail.com> <3d375d730804031700t20d3c2fata823317b24bf3bc8@mail.gmail.com> <1af523860804031714y35b69924rb57b4711dd3a69eb@mail.gmail.com> Message-ID: On Fri, Apr 4, 2008 at 1:50 AM, Sebastian Haase wrote: > Hi, > Accidentally I'm exactly trying to do the same thing right now ..... > > What is the best way of memmapping into a file that is already open !? > > I have to read some text (header info) off the beginning of the file > before I know where the data actually starts. > I could of course get the position at that point ( f.tell() ) close > the file, and reopen using memmap. > However this doesn't sound optimal to me .... > > Any hints ? > Could numpy's memmap be changed to also accept file-objects, or there > a "rule" that memmap always has to have access to the entire file ? I am getting a little tired, so this may be incorrect. But I believe Stefan modified memmaps to allow them to be created from file-like object: http://projects.scipy.org/scipy/numpy/changeset/4856 Are you running a released version of NumPy or the trunk? If you aren't using the trunk, could you give it a try? It would be good to have it tested before the 1.0.5 release. Cheers, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From nadavh at visionsense.com Fri Apr 4 06:07:52 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Fri, 4 Apr 2008 13:07:52 +0300 Subject: [Numpy-discussion] multiply array References: Message-ID: <710F2847B0018641891D9A21602763600B6F5A@ex3.envision.co.il> result = (egimgs.T * eval.flat).T or, in place E = egimgs.T E *= eval.flat (egimgs would updated) Nadav. -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? wilson ????: ? 04-?????-08 08:58 ??: numpy-discussion at scipy.org ????: [Numpy-discussion] multiply array hello i have two arrays #of shape (1,6) eval=array([[3.,3.2,1.,1.1,5.,0.5]]) #of shape (6,8) egimgs=array([ [3.,2.,1.,4.,5.,1.5,2.5,1.1], [1.1,3.,.5,.2,.1,4.3,3.2,1.2], [4.,3.,2.,6.,1.,4.,5.1,2.4], [3.2,1.3,2.2,4.4,1.1,2.1,3.3,2.4], [.4,.2,.1,.5,.3,.2,1.2,4.2], [5.2,2.1,4.3,5.5,2.2,1.1,1.4,3.2] ] ) i need to multiply the first row of egimgs with the first element of eval , second row of egimgs with second element of eval ..etc ...and so on..finally i should get a result array of same shape as egimgs. how should i proceed ? using loops seems inefficient.can someone suggest a better way? thanks W _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3166 bytes Desc: not available URL: From haase at msg.ucsf.edu Fri Apr 4 06:12:58 2008 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 4 Apr 2008 12:12:58 +0200 Subject: [Numpy-discussion] Efficient reading of binary data In-Reply-To: References: <1af523860804031330t173c7930rabbeb64f75d2042b@mail.gmail.com> <3d375d730804031347q24591c52raceafe2524ec5c3a@mail.gmail.com> <1af523860804031653s1941608ehbd7147502f87f17b@mail.gmail.com> <3d375d730804031700t20d3c2fata823317b24bf3bc8@mail.gmail.com> <1af523860804031714y35b69924rb57b4711dd3a69eb@mail.gmail.com> Message-ID: On Fri, Apr 4, 2008 at 11:33 AM, Jarrod Millman wrote: > On Fri, Apr 4, 2008 at 1:50 AM, Sebastian Haase wrote: > > Hi, > > Accidentally I'm exactly trying to do the same thing right now ..... > > > > What is the best way of memmapping into a file that is already open !? > > > > I have to read some text (header info) off the beginning of the file > > before I know where the data actually starts. > > I could of course get the position at that point ( f.tell() ) close > > the file, and reopen using memmap. > > However this doesn't sound optimal to me .... > > > > Any hints ? > > Could numpy's memmap be changed to also accept file-objects, or there > > a "rule" that memmap always has to have access to the entire file ? > > I am getting a little tired, so this may be incorrect. But I believe > Stefan modified memmaps to allow them to be created from file-like > object: http://projects.scipy.org/scipy/numpy/changeset/4856 > > Are you running a released version of NumPy or the trunk? If you > aren't using the trunk, could you give it a try? It would be good to > have it tested before the 1.0.5 release. > Hi Jarrod, Thanks for the reply. Indeed I'm running only N.__version__ '1.0.4.dev4312' I hope I find time to try the new feature. To clarify: if the file is already open, and the current position (f.tell() ) is somewhere in the middle, would the memmap "see" the file from there ? Could a "normal" file access and a concurrent memmap into that same file "step on each others feet" ? Thanks, Sebastian Haase From stefan at sun.ac.za Fri Apr 4 06:14:06 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 4 Apr 2008 12:14:06 +0200 Subject: [Numpy-discussion] problem with float64's str() In-Reply-To: <7f03db650804031231t39aa3000o719db8359a7769c7@mail.gmail.com> References: <7f03db650804031231t39aa3000o719db8359a7769c7@mail.gmail.com> Message-ID: <9457e7c80804040314t1db5e821h56ea782245cd4437@mail.gmail.com> Hi Will On 03/04/2008, Will Lee wrote: > I seem to have problem with floating point printing with the latest numpy, > python 2.5.2, gcc 4.1.4, and 64-bit linux: > > In [24]: print str(0.0012) > 0.0012 > > In [25]: a = numpy.array([0.0012]) > > In [26]: print str(a[0]) > 0.0011999999999999999 > > In [27]: print numpy.__version__ > 1.0.5.dev4950 > > It seems like the str() behavior for float64 in the latest numpy's behavior > is different than Python's default behavior (and previous numpy's behavior). > > > As I have numerous doc tests, this seems to make many of them failed. > Should the float64's str() behavior be consistent with what's described in > http://docs.python.org/tut/node16.html? Is there any way > to get around it so I can get to the default Python behavior? As a workaround, you can simply remove 'str' from your print statement: In [17]: print x[0] 0.0012 I am not sure why str([x[0]) is equivalent to repr(0.0012). Regards St?fan From peridot.faceted at gmail.com Fri Apr 4 09:33:31 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Fri, 4 Apr 2008 09:33:31 -0400 Subject: [Numpy-discussion] Final push for NumPy 1.0.5 (I need your help!) In-Reply-To: References: Message-ID: On 04/04/2008, Jarrod Millman wrote: > Since I sent my email last night another 5+ tickets have been closed. > If we keep going at this rate, we should be able to release 1.0.5 next > Friday (4/11) with every ticket closed. Specifically, thanks to > Travis Oliphant, David Huard, and Stefan van der Walt for bug > squashing. > > If anyone has some time, could you please check David's fix for the > ticket "loadtxt fails with record arrays": > http://projects.scipy.org/scipy/numpy/ticket/623 > His fix looks correct to me (and even includes test!!); if someone > else can confirm this, David or I will be able to close this ticket. > > There are now only 21 remaining open tickets. Please take a look over > > the open tickets and see if there is anything you can quickly close: > http://scipy.org/scipy/numpy/query?status=new&status=assigned&status=reopened&milestone=1.0.5 I've submitted patches for two (matrix_power and condition number functions) but I don't think I can actually manipulate the bug status in any way. Anne From bsouthey at gmail.com Fri Apr 4 09:42:52 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Fri, 04 Apr 2008 08:42:52 -0500 Subject: [Numpy-discussion] problem with float64's str() In-Reply-To: <9457e7c80804040314t1db5e821h56ea782245cd4437@mail.gmail.com> References: <7f03db650804031231t39aa3000o719db8359a7769c7@mail.gmail.com> <9457e7c80804040314t1db5e821h56ea782245cd4437@mail.gmail.com> Message-ID: <47F6305C.7060901@gmail.com> Hi, This topic has come up many times and the only problem is the lack of understanding how computers store numbers and computer numerical precision. The NumPy output is consistent with Python on my x86_64 linux system with Python 2.5.1: >>> a=0.0012 >>> a 0.0011999999999999999 Simply put, in generally, you can not just compare floating point numbers because of the variable precision by storage precision (see the different types supported by NumPy: such as 'float', 'float128', 'float32', 'float64'), OS and platform architecture. Thus, your tests are incorrectly implemented but hopefully something like NumPy's allclose() (http://www.scipy.org/Numpy_Example_List_With_Doc#allclose ) should be more appropriate. Bruce St?fan van der Walt wrote: > Hi Will > > On 03/04/2008, Will Lee wrote: > >> I seem to have problem with floating point printing with the latest numpy, >> python 2.5.2, gcc 4.1.4, and 64-bit linux: >> >> In [24]: print str(0.0012) >> 0.0012 >> >> In [25]: a = numpy.array([0.0012]) >> >> In [26]: print str(a[0]) >> 0.0011999999999999999 >> >> In [27]: print numpy.__version__ >> 1.0.5.dev4950 >> >> It seems like the str() behavior for float64 in the latest numpy's behavior >> is different than Python's default behavior (and previous numpy's behavior). >> >> >> As I have numerous doc tests, this seems to make many of them failed. >> Should the float64's str() behavior be consistent with what's described in >> http://docs.python.org/tut/node16.html? Is there any way >> to get around it so I can get to the default Python behavior? >> > > As a workaround, you can simply remove 'str' from your print statement: > > In [17]: print x[0] > 0.0012 > > I am not sure why str([x[0]) is equivalent to repr(0.0012). > > Regards > St?fan > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From oliphant at enthought.com Fri Apr 4 09:49:56 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 04 Apr 2008 08:49:56 -0500 Subject: [Numpy-discussion] Simple financial functions for NumPy Message-ID: <47F63204.5030000@enthought.com> Hi all, Last night I put together some simple financial functions based on the basic ones available in Excel (and on a financial calculator). It seems to me that NumPy ought to have these basic functions. There may be some disagreement about what to call them and what the interface should be. I've stuck with the Excel standard names and functionality because most of the people that will use these in the future, I expect will have seen them from Excel and it minimizes the impedance mismatch to have them be similarly named. The interface is also similar to the IMSL libraries. However, if clearly better interfaces can be discovered, then we could change it. For now, the functions are not imported into the numpy namespace but live in numpy.lib.financial I could see a future scipy module containing much, much more. Comments and improvement suggestions welcome. We are a week away from release of NumPy 1.0.5, and hopefully we can agree before then. Best regards, -Travis O. From dagss at student.matnat.uio.no Fri Apr 4 09:52:17 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 04 Apr 2008 15:52:17 +0200 Subject: [Numpy-discussion] problem with float64's str() In-Reply-To: <47F6305C.7060901@gmail.com> References: <7f03db650804031231t39aa3000o719db8359a7769c7@mail.gmail.com> <9457e7c80804040314t1db5e821h56ea782245cd4437@mail.gmail.com> <47F6305C.7060901@gmail.com> Message-ID: <47F63291.3030509@student.matnat.uio.no> Bruce Southey wrote: > Hi, > This topic has come up many times and the only problem is the lack of > understanding how computers store numbers and computer numerical precision. > > The NumPy output is consistent with Python on my x86_64 linux system > with Python 2.5.1: > >>> a=0.0012 > >>> a > 0.0011999999999999999 > Wasn't this discussion about the repr vs. str functions? >>> repr(0.0012) '0.0011999999999999999' >>> str(0.0012) '0.0012' Dag Sverre From oliphant at enthought.com Fri Apr 4 09:56:44 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 04 Apr 2008 08:56:44 -0500 Subject: [Numpy-discussion] Final push for NumPy 1.0.5 (I need your help!) In-Reply-To: References: Message-ID: <47F6339C.3050504@enthought.com> Anne Archibald wrote: > On 04/04/2008, Jarrod Millman wrote: > > >> Since I sent my email last night another 5+ tickets have been closed. >> If we keep going at this rate, we should be able to release 1.0.5 next >> Friday (4/11) with every ticket closed. Specifically, thanks to >> Travis Oliphant, David Huard, and Stefan van der Walt for bug >> squashing. >> >> If anyone has some time, could you please check David's fix for the >> ticket "loadtxt fails with record arrays": >> http://projects.scipy.org/scipy/numpy/ticket/623 >> His fix looks correct to me (and even includes test!!); if someone >> else can confirm this, David or I will be able to close this ticket. >> >> There are now only 21 remaining open tickets. Please take a look over >> >> the open tickets and see if there is anything you can quickly close: >> http://scipy.org/scipy/numpy/query?status=new&status=assigned&status=reopened&milestone=1.0.5 >> > > I've submitted patches for two (matrix_power and condition number > functions) but I don't think I can actually manipulate the bug status > in any way. > Hey Anne, Do you currently have SVN access? Would you like it? I think the SciPy/NumPy sprint would be a good time to clean-up the committers list and add new people interested in helping. -Travis From haase at msg.ucsf.edu Fri Apr 4 09:58:39 2008 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 4 Apr 2008 15:58:39 +0200 Subject: [Numpy-discussion] Simple financial functions for NumPy In-Reply-To: <47F63204.5030000@enthought.com> References: <47F63204.5030000@enthought.com> Message-ID: Hi Travis, This sounds of course very interesting, but could you elaborate on the reasoning why this should not rather be "only" in SciPy !? I thought many people think that numpy was already too crowded and should concentrate mostly on being a basic array handling facility. I'm sure you have a good reason for putting these into numpy. Do you have a list of the new functions ? Wiki page ? And once more, thanks for all your great work on numpy. I'm now even trying to make a career out of using numpy for microscopy image analysis. - Sebastian Haase On Fri, Apr 4, 2008 at 3:49 PM, Travis E. Oliphant wrote: > > Hi all, > > Last night I put together some simple financial functions based on the > basic ones available in Excel (and on a financial calculator). It seems > to me that NumPy ought to have these basic functions. > > There may be some disagreement about what to call them and what the > interface should be. I've stuck with the Excel standard names and > functionality because most of the people that will use these in the > future, I expect will have seen them from Excel and it minimizes the > impedance mismatch to have them be similarly named. The interface is > also similar to the IMSL libraries. > > However, if clearly better interfaces can be discovered, then we could > change it. For now, the functions are not imported into the numpy > namespace but live in > > numpy.lib.financial > > I could see a future scipy module containing much, much more. > > Comments and improvement suggestions welcome. We are a week away from > release of NumPy 1.0.5, and hopefully we can agree before then. > > Best regards, > > > -Travis O. From gael.varoquaux at normalesup.org Fri Apr 4 10:04:40 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 4 Apr 2008 16:04:40 +0200 Subject: [Numpy-discussion] Simple financial functions for NumPy In-Reply-To: References: <47F63204.5030000@enthought.com> Message-ID: <20080404140440.GA23808@phare.normalesup.org> On Fri, Apr 04, 2008 at 03:58:39PM +0200, Sebastian Haase wrote: > This sounds of course very interesting, but could you elaborate on the > reasoning why this should not rather be "only" in SciPy !? > I thought many people think that numpy was already too crowded and > should concentrate mostly on being a basic array handling facility. +1. Or in a separate package if you don't want to enforce a heavy C/fortran dependency like with scipy. Cheers, Ga?l From oliphant at enthought.com Fri Apr 4 10:11:37 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 04 Apr 2008 09:11:37 -0500 Subject: [Numpy-discussion] Simple financial functions for NumPy In-Reply-To: References: <47F63204.5030000@enthought.com> Message-ID: <47F63719.6030605@enthought.com> Sebastian Haase wrote: > Hi Travis, > This sounds of course very interesting, but could you elaborate on the > reasoning why this should not rather be "only" in SciPy !? > I thought many people think that numpy was already too crowded and > should concentrate mostly on being a basic array handling facility. > This is a valid concern and I'm interested in hearing feedback. There are only two reasons that I can think of right now to keep them in NumPy instead of moving them to SciPy. 1) These are "basic" functions and a scipy toolkit would contain much more. 2) These are widely used and would make NumPy attractive to a wider audience who don't want to install all of SciPy just to get these functions. NumPy already contains functions that make it equivalent to a basic scientific calculator, should it not also contain the functions that make it equivalent to the same calculator when placed in "financial" mode? On the other hand, package distribution is getting better, and having a more modular approach is useful. I could go both ways on this one. -Travis O. From oliphant at enthought.com Fri Apr 4 10:16:36 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 04 Apr 2008 09:16:36 -0500 Subject: [Numpy-discussion] Simple financial functions for NumPy In-Reply-To: References: <47F63204.5030000@enthought.com> Message-ID: <47F63844.40007@enthought.com> Sebastian Haase wrote: > Hi Travis, > This sounds of course very interesting, but could you elaborate on the > reasoning why this should not rather be "only" in SciPy !? > I thought many people think that numpy was already too crowded and > should concentrate mostly on being a basic array handling facility. > > I just thought of one more thing that is probably swaying that mysterious "gut feel" : NumPy is on the laptop in the OLPC project. I want to give those kids access to simple financial calculations for learning about time-preference for money and the value of saving. -Travis O. From bsouthey at gmail.com Fri Apr 4 10:55:13 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Fri, 04 Apr 2008 09:55:13 -0500 Subject: [Numpy-discussion] problem with float64's str() In-Reply-To: <47F63291.3030509@student.matnat.uio.no> References: <7f03db650804031231t39aa3000o719db8359a7769c7@mail.gmail.com> <9457e7c80804040314t1db5e821h56ea782245cd4437@mail.gmail.com> <47F6305C.7060901@gmail.com> <47F63291.3030509@student.matnat.uio.no> Message-ID: <47F64151.4090307@gmail.com> Hi, Note that at least under Python 2.5.1: >>> a=0.0012 >>> a 0.0011999999999999999 >>> str(a) '0.0012' >>> repr(a) '0.0011999999999999999' From Python docs, repr(): 'Return a string containing a printable representation of an object' and str(): 'Return a nice string representation of the object'. In this case the object is a Python float that approximates the base 10 number of 0.0012. This is not equivalent to convert the base 10 number 0.0012 to a string because computer numbers are base 2. Thus, repr() converts a Python object into a string but nothing about the numerical precision whereas str() tries to do something about maing a 'nice string'. The only consistent way to get 0.0012 (or any number that can not be represented exactly) is to use a Python object that stores 0.0012 exactly. For example, the Decimal module was introduced in Python 2.4: http://www.python.org/doc/2.5/lib/module-decimal.html >>> str(Decimal('0.0012')) '0.0012' >>> float(Decimal('0.0012')) 0.0011999999999999999 Also, see PEP 3141 'A Type Hierarchy for Numbers' (http://www.python.org/dev/peps/pep-3141/). Regards Bruce Dag Sverre Seljebotn wrote: > Bruce Southey wrote: > >> Hi, >> This topic has come up many times and the only problem is the lack of >> understanding how computers store numbers and computer numerical precision. >> >> The NumPy output is consistent with Python on my x86_64 linux system >> with Python 2.5.1: >> >>> a=0.0012 >> >>> a >> 0.0011999999999999999 >> >> > Wasn't this discussion about the repr vs. str functions? > > >>> repr(0.0012) > '0.0011999999999999999' > >>> str(0.0012) > '0.0012' > > > > Dag Sverre > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From lee.will at gmail.com Fri Apr 4 10:56:28 2008 From: lee.will at gmail.com (Will Lee) Date: Fri, 4 Apr 2008 09:56:28 -0500 Subject: [Numpy-discussion] problem with float64's str() In-Reply-To: <47F63291.3030509@student.matnat.uio.no> References: <7f03db650804031231t39aa3000o719db8359a7769c7@mail.gmail.com> <9457e7c80804040314t1db5e821h56ea782245cd4437@mail.gmail.com> <47F6305C.7060901@gmail.com> <47F63291.3030509@student.matnat.uio.no> Message-ID: <7f03db650804040756q20d0ad79x964b05fbad6fd8fb@mail.gmail.com> I understand the implication for the floating point comparison and the need for allclose. However, I think in a doctest context, this behavior makes the doc much harder to read. For example, if you have this in your doctest: def doSomething(a): ''' >>> print doSomething(0.0011)[0] >>> 0.0012 ''' return numpy.array([a]) + 0.0001 In the current numpy, you'll have to write: def doSomething(a): ''' >>> print doSomething(0.0011)[0] >>> 0.0011999999999999999 ''' return numpy.array([a]) + 0.0001 Using allclose, you'll need to write it like: def doSomething(a): ''' >>> print numpy.allclose(doSomething(0.0011)[0], 0.0012) >>> True ''' return numpy.array([a]) + 0.0001 I don't think either cases are ideal. You can also imagine if you're printing out a string with a float in it (like 'You got 0.0012 back'), then you can't really use allclose at all. Also, it's somewhat odd that the behavior for str is different for a numpy.float64 and python's float, since otherwise they're mostly the same. Will On Fri, Apr 4, 2008 at 8:52 AM, Dag Sverre Seljebotn < dagss at student.matnat.uio.no> wrote: > Bruce Southey wrote: > > Hi, > > This topic has come up many times and the only problem is the lack of > > understanding how computers store numbers and computer numerical > precision. > > > > The NumPy output is consistent with Python on my x86_64 linux system > > with Python 2.5.1: > > >>> a=0.0012 > > >>> a > > 0.0011999999999999999 > > > Wasn't this discussion about the repr vs. str functions? > > >>> repr(0.0012) > '0.0011999999999999999' > >>> str(0.0012) > '0.0012' > > > > Dag Sverre > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at carabos.com Fri Apr 4 11:05:33 2008 From: faltet at carabos.com (Francesc Altet) Date: Fri, 4 Apr 2008 17:05:33 +0200 Subject: [Numpy-discussion] Simple financial functions for NumPy In-Reply-To: <47F63844.40007@enthought.com> References: <47F63204.5030000@enthought.com> <47F63844.40007@enthought.com> Message-ID: <200804041705.33628.faltet@carabos.com> A Friday 04 April 2008, Travis E. Oliphant escrigu?: > Sebastian Haase wrote: > > Hi Travis, > > This sounds of course very interesting, but could you elaborate on > > the reasoning why this should not rather be "only" in SciPy !? I > > thought many people think that numpy was already too crowded and > > should concentrate mostly on being a basic array handling facility. > > I just thought of one more thing that is probably swaying that > mysterious "gut feel" : > > NumPy is on the laptop in the OLPC project. I want to give those > kids access to simple financial calculations for learning about > time-preference for money and the value of saving. Yeah :) +1 for including these in NumPy. -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From gael.varoquaux at normalesup.org Fri Apr 4 11:09:55 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 4 Apr 2008 17:09:55 +0200 Subject: [Numpy-discussion] Simple financial functions for NumPy In-Reply-To: <47F63719.6030605@enthought.com> References: <47F63204.5030000@enthought.com> <47F63719.6030605@enthought.com> Message-ID: <20080404150955.GB23808@phare.normalesup.org> On Fri, Apr 04, 2008 at 09:11:37AM -0500, Travis E. Oliphant wrote: > There are only two reasons that I can think of right now to keep them in > NumPy instead of moving them to SciPy. > 1) These are "basic" functions and a scipy toolkit would contain much more. > 2) These are widely used and would make NumPy attractive to a wider > audience who don't want to install all of SciPy just to get > these functions. > NumPy already contains functions that make it equivalent to a basic > scientific calculator, should it not also contain the functions that > make it equivalent to the same calculator when placed in "financial" mode? My concern is consistency. It is already pretty hard to define what goes in scipy and what goes in numpy, and I am not even mentioning code lying around in pylab. I really thing numpy should be as thin as possible, so that you can really say that it is only an array manipulation package. This will also make it easier to sell as a core package for developpers who do not care about "calculator" features. My 2 cents, Ga?l From jh at physics.ucf.edu Fri Apr 4 11:41:13 2008 From: jh at physics.ucf.edu (Joe Harrington) Date: Fri, 04 Apr 2008 11:41:13 -0400 Subject: [Numpy-discussion] Simple financial functions for NumPy In-Reply-To: (numpy-discussion-request@scipy.org) References: Message-ID: +1 for simple financial functions in numpy, and congrats that it's on OLPC! If we have an FFT in numpy, we should have an internal rate of return. Anyone with investments needs that, and that's more people than those needing an FFT. I agree that Excel will bring in the most familiarity, but their names are not always rational. Please don't propagate irrational names. Consider looking at what they're called in Matlab and IDL, as code conversion/familiarity from those communities counts as well. Maybe for each function take the most rational name and arg order from those three sources, with strong preference for Excel unless there is a clear better way to do it. --jh-- From aisaac at american.edu Fri Apr 4 12:13:01 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 4 Apr 2008 12:13:01 -0400 Subject: [Numpy-discussion] Simple financial functions for NumPy In-Reply-To: <20080404150955.GB23808@phare.normalesup.org> References: <47F63204.5030000@enthought.com><47F63719.6030605@enthought.com><20080404150955.GB23808@phare.normalesup.org> Message-ID: On Fri, 4 Apr 2008, Gael Varoquaux apparently wrote: > I really thing numpy should be as thin as possible, so > that you can really say that it is only an array > manipulation package. This will also make it easier to > sell as a core package for developpers who do not care > about "calculator" features. I'm a user rather than a developer, but I wonder: is this true? 1. Even as a user, I agree that what I really want from NumPy is a core array manipulation package (including matrices). BUT as long as this is the core of NumPy, will a developer care if other features are available? 2. Even if the answer to 1. is yes, could the build/installation process include an option not to build/install anything but the core array functionality? 3. It seems to me that pushing things out into SciPy remains a problem: a basic NumPy is easy to build on any platform, but SciPy still seems to generate many questions. 4. One reason this keeps coming up is that he NumPy/SciPy split is rather too crude. If the split were instead something like NumPy/SciPyBasic/SciPyFull/SciPyFull+Kits where SciPyBasic contained only pure Python code (no extensions), perhaps the desired location would be more obvious and some of this recurrent discussion would go away. fwiw, Alan Isaac From amcmorl at gmail.com Fri Apr 4 12:13:57 2008 From: amcmorl at gmail.com (Angus McMorland) Date: Fri, 4 Apr 2008 12:13:57 -0400 Subject: [Numpy-discussion] Simple financial functions for NumPy In-Reply-To: References: Message-ID: -1 for any functions added to numpy. As only an end-user, I realize I have little right to a say in these sorts of issues, but for whatever it may be worth, I strongly agree with Gael's viewpoint. We should be aiming towards modular systems for function distribution, and now that it seems that these are being gradually worked out (scikits?), I think we should avoid adding anything to numpy, which should rather be kept to a bare minimum: just the necessaries for array creation and manipulation. Everything else should go in the add-on modules which can be installed as required. This have the benefit that the numpy package stays well-defined and contained, meaning that end-users know exactly what to expect as available on a given system. Instead of wondering "Where do I find functions for x. I know numpy has some things. Maybe it's in there or maybe somewhere else." I would always know that in order to get functions for x I would install the correct, usefully named, module. This seems like the path of least surprise, and a cleanest interface. I agree it's great that numpy is on the OLPC, and would like to see it accompanied there by a "Basic Functions" module containing, for example, these financial functions, which certainly sound useful... but not for everyone. On 04/04/2008, Joe Harrington wrote: > +1 for simple financial functions in numpy, and congrats that it's on > OLPC! If we have an FFT in numpy, we should have an internal rate of > return. Anyone with investments needs that, and that's more people > than those needing an FFT. > > I agree that Excel will bring in the most familiarity, but their names > are not always rational. Please don't propagate irrational names. > Consider looking at what they're called in Matlab and IDL, as code > conversion/familiarity from those communities counts as well. Maybe > for each function take the most rational name and arg order from those > three sources, with strong preference for Excel unless there is a > clear better way to do it. > > > --jh-- > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- AJC McMorland, PhD candidate Physiology, University of Auckland Post-doctoral research fellow Neurobiology, University of Pittsburgh From Joris.DeRidder at ster.kuleuven.be Fri Apr 4 12:17:20 2008 From: Joris.DeRidder at ster.kuleuven.be (Joris De Ridder) Date: Fri, 4 Apr 2008 18:17:20 +0200 Subject: [Numpy-discussion] Simple financial functions for NumPy In-Reply-To: <47F63719.6030605@enthought.com> References: <47F63204.5030000@enthought.com> <47F63719.6030605@enthought.com> Message-ID: <4F02E529-980C-42C4-921B-4B69604A6B3B@ster.kuleuven.be> On 04 Apr 2008, at 16:11, Travis E. Oliphant wrote: > > There are only two reasons that I can think of right now to keep > them in > NumPy instead of moving them to SciPy. > > 1) These are "basic" functions and a scipy toolkit would contain > much more. Isn't this something you want to avoid? Functionality in two different packages: a small kit of functions in NumPy, and (eventually) another large toolkit in scipy. One package only, would be more convenient I think. I agree with Ga?l that it's not really consistent with the NumPy/SciPy philosophy either. :-). So, I would prefer to see this nice functionality in SciPy rather than in NumPy. Joris Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm From charlesr.harris at gmail.com Fri Apr 4 12:47:46 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 4 Apr 2008 10:47:46 -0600 Subject: [Numpy-discussion] Simple financial functions for NumPy In-Reply-To: <4F02E529-980C-42C4-921B-4B69604A6B3B@ster.kuleuven.be> References: <47F63204.5030000@enthought.com> <47F63719.6030605@enthought.com> <4F02E529-980C-42C4-921B-4B69604A6B3B@ster.kuleuven.be> Message-ID: On Fri, Apr 4, 2008 at 10:17 AM, Joris De Ridder < Joris.DeRidder at ster.kuleuven.be> wrote: > > On 04 Apr 2008, at 16:11, Travis E. Oliphant wrote: > > > > > There are only two reasons that I can think of right now to keep > > them in > > NumPy instead of moving them to SciPy. > > > > 1) These are "basic" functions and a scipy toolkit would contain > > much more. > > Isn't this something you want to avoid? Functionality in two different > packages: a small kit of functions in NumPy, and (eventually) another > large toolkit in scipy. One package only, would be more convenient I > think. > > I agree with Ga?l that it's not really consistent with the NumPy/SciPy > philosophy either. :-). > So, I would prefer to see this nice functionality in SciPy rather than > in NumPy. > Agree. I also think that the idea of basic, pure python extensions is a good one. There is a lot of useful functionality that can be made available without adding Fortran packages to the mix. These packages could even be included as part of numpy but they should remain in a separate namespace. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.huard at gmail.com Fri Apr 4 13:02:45 2008 From: david.huard at gmail.com (David Huard) Date: Fri, 4 Apr 2008 13:02:45 -0400 Subject: [Numpy-discussion] loading data with gaps In-Reply-To: References: Message-ID: <91cf711d0804041002g53e5f2c7ta394a48b712a40d7@mail.gmail.com> Hi Tim, Look at the thread posted a couple of weeks ago named: loadtxt and missing values I'm guessing you'll find answers to your questions, if not, don't hesitate to ask. David 2008/4/3, Tim Michelsen : > > Hello! > > How can I load a data file (e.g. CSV, DAT) in ASCII which has some gaps? > > The file has been saved with from a spreadsheet program which leaves > cells with not data empty: > > > 1,23. > 2,13. > 3, > 4,34. > > Would this code be correct: > ### test_loadtxt.py ### > import numpy > import maskedarray > > # load data which has empty 'cells' as beeing saved from spreadsheet: > # 1,23. > # 2,13. > # 3, > # 4,34. > data = numpy.loadtxt('./loadtxt_test.csv',dtype=str,delimiter=',') > > > # create a masked array with all no data ('', empty cells from CSV) masked > my_masked_array = maskedarray.masked_equal(data,'') > ###### > > * How can I change the data type of my maskedarray (my_masked_array) to > a type that allows me to perform calulations? > > * Would you do this task differently or more efficient? > > * What possibilities do I have to estimate/interpolate the masked values? > A example would be nice. > > * How to I convert maskedarray (my_masked_array) to a array without > masked values? > > Thanks in advance for your help, > Tim Michelsenholg > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Fri Apr 4 13:49:55 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Fri, 4 Apr 2008 19:49:55 +0200 Subject: [Numpy-discussion] Final push for NumPy 1.0.5 (I need your help!) In-Reply-To: <47F6339C.3050504@enthought.com> References: <47F6339C.3050504@enthought.com> Message-ID: On 04/04/2008, Travis E. Oliphant wrote: > Hey Anne, > > Do you currently have SVN access? Would you like it? > > I think the SciPy/NumPy sprint would be a good time to clean-up the > committers list and add new people interested in helping. I don't have SVN access. I'd be happy (and careful!) to have it, but I should warn you that I won't have time to do serious, regular development on scipy/numpy; I do hope to be able to write a little code here and there, though, and it would be handy to be able to add it directly instead of sending patches into the ether. Anne From chanley at stsci.edu Fri Apr 4 14:11:55 2008 From: chanley at stsci.edu (Christopher Hanley) Date: Fri, 04 Apr 2008 14:11:55 -0400 Subject: [Numpy-discussion] numpy unittest failure on Mac PPC and Solaris 10 Message-ID: <47F66F6B.4010506@stsci.edu> Hi, We are seeing the following error on both Solaris and Mac PPC when running the numpy unittests: .............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................F....................................................................................................................................................................................................................................................................... ====================================================================== FAIL: test_record (numpy.lib.tests.test_io.Testloadtxt) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/stsci/pyssgdev/2.5.1/numpy/lib/tests/test_io.py", line 42, in test_record assert_array_equal(x, a) File "/usr/stsci/pyssgdev/2.5.1/numpy/testing/utils.py", line 225, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "/usr/stsci/pyssgdev/2.5.1/numpy/testing/utils.py", line 217, in assert_array_compare assert cond, msg AssertionError: Arrays are not equal (mismatch 100.0%) x: array([(1, 2), (3, 4)], dtype=[('x', '>i4'), ('y', '>i4')]) y: array([(1, 2), (3, 4)], dtype=[('x', ' References: <5a0e4665-183d-4cb3-bee3-7a8cb5284b65@s8g2000prg.googlegroups.com> Message-ID: On Thu, 3 Apr 2008 23:02:24 -0700 (PDT) harryos wrote: > i have a 1 dim numpy array > D=array( [[ 3. , 2. , 1. , 4. , 5. , 1.5, 2.2]] ) > i need to get this sorted in descending order and then >access the > elements . > D.sort() will make D as [[ 1. 1.5 2. 2.2 3. 4. > 5. ]] > how will i reverse it? > or is there a simpler way? > harry > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > >>> D array([[ 3. , 2. , 1. , 4. , 5. , 1.5, 2.2]]) >>> fliplr(sort(D)) array([[ 5. , 4. , 3. , 2.2, 2. , 1.5, 1. ]]) Nils From jh at physics.ucf.edu Fri Apr 4 15:27:37 2008 From: jh at physics.ucf.edu (Joe Harrington) Date: Fri, 04 Apr 2008 15:27:37 -0400 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: (numpy-discussion-request@scipy.org) References: Message-ID: Every once in a while the issue of how to split things into packages comes up. In '04, I think, we had such a discussion regarding scipy (with Numeric as its base at the time). One idea was a core-plus-many-modules approach. We could then have metapackages that just consisted of dependencies and would draw in the packages that were needed for a given application. A user would install just one metapackage (one of which would install every package), or could roll their own. Then Travis took as much of scipy as was reasonable to maintain at once, took a chunk of numarray, and made numpy. Numpy is now a bit more than what I would have thought of as "core" functionality. One might consider trimming Travis's collection and moving many of the functions out to add-ons in scipy. The question is where to draw the line in the sand with regard to where a given function belongs, and the problem is that the line is very hard to draw in a way that pleases many people and mortally offends very few. Certainly lots of content areas would straddle the divide and require both a package in scipy and representation in the core. I would think of core functionality as what is now the ndarray class (including all slicing and broadcasting); things to make and index arrays like array, zeros, ones, where and some truth tests; the most elementary math functions, such as add, subtract, multiply, divide, power, sin, cos, tan, asin, acos, atan, log, log10; sum, mean, median, standard deviaton; the constants pi, e, NaN, Inf, -Inf; a few simple file I/O functions including loadtxt, and not a whole lot else. This is an incomplete list but you get the idea. Everything else would be in optional packages, possiby broken down by topic. Now ask yourself, what would you add to this to make your core? Would you take anything out? And the kicker: do you really think much of the community would agree with any one of our lists, including mine? Almost all such very-light collections would require users to load additional packages. For example, complex things like FFTs and random numbers would not belong in the core outlined above, nor would linear algebra, masked arrays, fitting, random numbers, most stats, or financial functions. There is one division that does make sense, and that would be to distribute ONLY ndarray as the core, and have NO basic math functions, etc., in the core. Then you have to load something, a lot of things, to make it useful, but you don't have the question of what should be in which package. But ndarray is already a stand-alone item. At that point you have to ask, if the core is so small that *everyone* has to load an add-on, what's the point of making the division? You can argue that it's easier maintenance-wise, but I'm not certain that having many packages to build, test, and distribute is easier. Travis already made a decision based on maintenance, and it seems to be working. That brings us to the motivation for dividing in the first place. I think people like the idea because we're all scientists and we like to categorize things. We like neat little packages. But we're not thinking about the implications for the code we'd actually write. Wouldn't you rather do: import numpy as N ... c = (N.sin(b) + N.exp(d)) / N.mean(g) rather than: import numpy as N import numpy.math as N.M import numpy.trig as N.T import numpy.stat as N.S ... c = (N.T.sin(b) + N.M.exp(d)) / N.S.mean(g) ? In the latter example, you start N.whatevering yourself to death, and it is harder to learn because you have to remember what little container each function is in and what you happened to name the container when you loaded it. Code is also harder to read as the N.whatevers distract from the sense of the equation. Lines start to lengthen. Sure, you can do: from whatever import functionlist to pull the functions into your top-level namespace, but do you really want to, in effect, declare every function you ever call? The whole point of array languages is to get rid of mechanics like function declarations so you can focus on programming, not housekeeping. As I've emphasized before, finding the function you want is a documentation problem, not a packaging problem. We're working on that. Anne's function index on the web site is an excellent start, and there will be much more done this summer. Though I didn't initially agree with it, I now think Travis's line in the sand is a pretty good one. Numpy is enough so many people don't have to go to scipy most of the time. It's being maintained well and released reasonably often. The problems of the rest of scipy are not holding back the core anymore. Installing today at a tiny 10 MB, numpy could easily stand to grow by adding small functions that are broadly used, without making it unwieldy for even the constrained space of OLPC. Let's think good and hard before we introduce more divisions into the namespace. --jh-- From robert.kern at gmail.com Fri Apr 4 15:47:46 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 4 Apr 2008 14:47:46 -0500 Subject: [Numpy-discussion] problem with float64's str() In-Reply-To: <7f03db650804040756q20d0ad79x964b05fbad6fd8fb@mail.gmail.com> References: <7f03db650804031231t39aa3000o719db8359a7769c7@mail.gmail.com> <9457e7c80804040314t1db5e821h56ea782245cd4437@mail.gmail.com> <47F6305C.7060901@gmail.com> <47F63291.3030509@student.matnat.uio.no> <7f03db650804040756q20d0ad79x964b05fbad6fd8fb@mail.gmail.com> Message-ID: <3d375d730804041247q25938d50h5083474a47f51a99@mail.gmail.com> On Fri, Apr 4, 2008 at 9:56 AM, Will Lee wrote: > I understand the implication for the floating point comparison and the need > for allclose. However, I think in a doctest context, this behavior makes > the doc much harder to read. Tabling the issue of the fact that we changed behavior for a moment, this is a fundamental problem with using doctests as unit tests for numerical code. The floating point results that you get *will* be different on different machines, but the code will still be correct. Using allclose() and similar techniques are the best tools available (although they still suck). Relying on visual representations of these results is simply an untenable strategy. Note that the string representation of NaNs and Infs are completely different across platforms. That said, str(float_numpy_scalar) really should have the same rules as str(some_python_float). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at enthought.com Fri Apr 4 16:12:40 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 04 Apr 2008 15:12:40 -0500 Subject: [Numpy-discussion] problem with float64's str() In-Reply-To: <3d375d730804041247q25938d50h5083474a47f51a99@mail.gmail.com> References: <7f03db650804031231t39aa3000o719db8359a7769c7@mail.gmail.com> <9457e7c80804040314t1db5e821h56ea782245cd4437@mail.gmail.com> <47F6305C.7060901@gmail.com> <47F63291.3030509@student.matnat.uio.no> <7f03db650804040756q20d0ad79x964b05fbad6fd8fb@mail.gmail.com> <3d375d730804041247q25938d50h5083474a47f51a99@mail.gmail.com> Message-ID: <47F68BB8.8010400@enthought.com> Robert Kern wrote: > On Fri, Apr 4, 2008 at 9:56 AM, Will Lee wrote: > >> I understand the implication for the floating point comparison and the need >> for allclose. However, I think in a doctest context, this behavior makes >> the doc much harder to read. >> > > Tabling the issue of the fact that we changed behavior for a moment, > this is a fundamental problem with using doctests as unit tests for > numerical code. The floating point results that you get *will* be > different on different machines, but the code will still be correct. > Using allclose() and similar techniques are the best tools available > (although they still suck). Relying on visual representations of these > results is simply an untenable strategy. Note that the string > representation of NaNs and Infs are completely different across > platforms. > > Well said, Robert. > That said, str(float_numpy_scalar) really should have the same rules > as str(some_python_float). > > +1 -Travis From aisaac at american.edu Fri Apr 4 16:29:03 2008 From: aisaac at american.edu (Alan Isaac) Date: Fri, 4 Apr 2008 16:29:03 -0400 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: References: Message-ID: On Fri, 04 Apr 2008, Joe Harrington wrote: > Wouldn't you rather do: > import numpy as N > ... > c = (N.sin(b) + N.exp(d)) / N.mean(g) > rather than: > import numpy as N > import numpy.math as N.M > import numpy.trig as N.T > import numpy.stat as N.S > ... > c = (N.T.sin(b) + N.M.exp(d)) / N.S.mean(g) I try to think of my students in such an environment. Frightening. Alan Isaac From gael.varoquaux at normalesup.org Fri Apr 4 16:31:33 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 4 Apr 2008 22:31:33 +0200 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: References: Message-ID: <20080404203133.GB27485@phare.normalesup.org> On Fri, Apr 04, 2008 at 04:29:03PM -0400, Alan Isaac wrote: > > import numpy as N > > import numpy.math as N.M > > import numpy.trig as N.T > > import numpy.stat as N.S > > ... > > c = (N.T.sin(b) + N.M.exp(d)) / N.S.mean(g) > I try to think of my students in such an environment. > Frightening. +1 (and s/students/colleagues). Ga?l From tim.hochberg at ieee.org Fri Apr 4 16:40:08 2008 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Fri, 4 Apr 2008 13:40:08 -0700 Subject: [Numpy-discussion] problem with float64's str() In-Reply-To: <3d375d730804041247q25938d50h5083474a47f51a99@mail.gmail.com> References: <7f03db650804031231t39aa3000o719db8359a7769c7@mail.gmail.com> <9457e7c80804040314t1db5e821h56ea782245cd4437@mail.gmail.com> <47F6305C.7060901@gmail.com> <47F63291.3030509@student.matnat.uio.no> <7f03db650804040756q20d0ad79x964b05fbad6fd8fb@mail.gmail.com> <3d375d730804041247q25938d50h5083474a47f51a99@mail.gmail.com> Message-ID: On Fri, Apr 4, 2008 at 12:47 PM, Robert Kern wrote: > On Fri, Apr 4, 2008 at 9:56 AM, Will Lee wrote: > > I understand the implication for the floating point comparison and the > need > > for allclose. However, I think in a doctest context, this behavior > makes > > the doc much harder to read. > > Tabling the issue of the fact that we changed behavior for a moment, > this is a fundamental problem with using doctests as unit tests for > numerical code. The floating point results that you get *will* be > different on different machines, but the code will still be correct. > Using allclose() and similar techniques are the best tools available > (although they still suck). Relying on visual representations of these > results is simply an untenable strategy. That is sometimes, but not always the case. Why? Because most of the time that one ends up with simple values, one is starting with arbitrary floating point values and doing at most simple operations on them. Thus a strategy that helps many of my unit tests look better and function reliably is to choose values that can be represented exactly in floating point. If the original value here had been 0.00125 rather than .0012, there would be no problem here. Well almost, you still are vulnerable to the rules for zero padding and what no getting changed and so forth, but in general it's more reliable and prettier. Of course this isn't always a solution. But I've found it's helpful for a lot cases. Note that the string > representation of NaNs and Infs are completely different across > platforms. > > That said, str(float_numpy_scalar) really should have the same rules > as str(some_python_float). +1 > > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From lxander.m at gmail.com Fri Apr 4 17:11:33 2008 From: lxander.m at gmail.com (Alexander Michael) Date: Fri, 4 Apr 2008 17:11:33 -0400 Subject: [Numpy-discussion] Simple financial functions for NumPy In-Reply-To: <47F63204.5030000@enthought.com> References: <47F63204.5030000@enthought.com> Message-ID: <525f23e80804041411o5eba83e4ja5f8178d7323e266@mail.gmail.com> On Fri, Apr 4, 2008 at 9:49 AM, Travis E. Oliphant wrote: > However, if clearly better interfaces can be discovered, then we could > change it. For now, the functions are not imported into the numpy > namespace but live in > > numpy.lib.financial > > I could see a future scipy module containing much, much more. > > Comments and improvement suggestions welcome. We are a week away from > release of NumPy 1.0.5, and hopefully we can agree before then. I'm generally in agreement with other opinions about keeping numpy lightweight even though I think these functions are useful and should be widely distributed with numpy. I've struggled with the various masked array implementations being worlds unto their own, falling down unexpectedly when mixed with other numpy functions, so keeping a narrow focus seems beneficial (as in its clear that I shouldn't expect A and B to work necessarily together). Nevertheless, I like getting a lot of utility from each package as it seems cognitive load is proportional to the number of packages required-- especially when the packages are compiled. Perhaps, as others have suggested, there should be some sort of pure-python numpy library package (a NumPyLib, if you will) that sits between numpy and scipy? I'm a numpy user but not a scipy user (I guess from an attempt to decrease the cognitive load of yet another compiled python package), so I'm speaking from that perspective. I also wouldn't be opposed to (for NumPy 4 :) breaking out the core ndarray class and basic linalg (solve, svg, eig, etc.) as NDArray and putting everything else into logically separated but independent NumKits. A blessed collection of which are together taken and distributed as "NumPy". Anything depending on one ore more NumKits would go into a SciKit, with a blessed collection distributed together as "SciPy". Has this basic distribution architecture already been proposed? I've heard hints of something along these lines. If so, then the new new financial functions should go into numpy.lib, where everything will later be broken out into a NumKit. Hmm. I've just argued myself in a circle... :O Regards, Alex From Chris.Barker at noaa.gov Fri Apr 4 17:20:26 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 04 Apr 2008 14:20:26 -0700 Subject: [Numpy-discussion] Efficient reading of binary data In-Reply-To: <1af523860804031653s1941608ehbd7147502f87f17b@mail.gmail.com> References: <1af523860804031330t173c7930rabbeb64f75d2042b@mail.gmail.com> <3d375d730804031347q24591c52raceafe2524ec5c3a@mail.gmail.com> <1af523860804031653s1941608ehbd7147502f87f17b@mail.gmail.com> Message-ID: <47F69B9A.4080007@noaa.gov> Nicolas Bigaouette wrote: > So the next step would be to only read the needed data from the binary > file... You've gotten some suggestions, but another option is to use file.seek(0 to get where your data is, and numpy.fromfile() from there. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From mforbes at physics.ubc.ca Fri Apr 4 18:18:06 2008 From: mforbes at physics.ubc.ca (Michael McNeil Forbes) Date: Fri, 4 Apr 2008 15:18:06 -0700 Subject: [Numpy-discussion] Problem with numpy.linalg.eig? In-Reply-To: References: <6be8b94a0711121010m4ee6a0e5w50e66c8388556d73@mail.gmail.com> <47392790.3040703@ar.media.kyoto-u.ac.jp> <47396237.60209@ar.media.kyoto-u.ac.jp> <473D1B31.2050207@ar.media.kyoto-u.ac.jp> Message-ID: On 16 Nov 2007, at 1:46 AM, Michael McNeil Forbes wrote: > On 15 Nov 2007, at 8:23 PM, David Cournapeau wrote: > >> Could you try without atlas ? Also, how did you configure atlas when >> building it ? It seems that atlas is definitely part of the problem >> (everybody having the problem does use atlas), and that it involves >> Core >> 2 duo. >> >> David > > It seems to work fine without ATLAS, but then again, it is a somewhat > "random" error. I will let some code run tonight and see if I detect > anything. Just an update. I am still having this problem, along with some additional problems where occasionally even dot returns nan's. I have confirmed that without ATLAS everything seems to be fine, and that the problem still remains with newer versions of ATLAS, Python, gcc etc. ATLAS was configured with ../configure --prefix=${BASE}/apps/${ATLAS}_${SUFFIX}\ --with-netlib-lapack=${BASE}/src/${LAPACK}_${SUFFIX}/ lapack_LINUX.a\ -A Core2Duo64SSE3\ --cflags=-fPIC\ -Fa alg -fPIC and it passed all the tests. The problem still exists with ATLAS version 3.8.1, gcc 4.3.0, and recent versions of numpy. >>> sys.version '2.5.2 (r252:60911, Mar 29 2008, 02:55:47) \n[GCC 4.3.0]' >>> numpy.version.version '1.0.5.dev4915' I have managed to extract a matrix that causes this failure repeatedly once every two or four times eigh is called, so hopefully I should be able to run gdb and track down the problem... From peridot.faceted at gmail.com Fri Apr 4 18:31:30 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Fri, 4 Apr 2008 18:31:30 -0400 Subject: [Numpy-discussion] Simple financial functions for NumPy In-Reply-To: References: <47F63204.5030000@enthought.com> <47F63719.6030605@enthought.com> <20080404150955.GB23808@phare.normalesup.org> Message-ID: On 04/04/2008, Alan G Isaac wrote: > On Fri, 4 Apr 2008, Gael Varoquaux apparently wrote: > > I really thing numpy should be as thin as possible, so > > that you can really say that it is only an array > > manipulation package. This will also make it easier to > > sell as a core package for developpers who do not care > > about "calculator" features. > > > I'm a user rather than a developer, but I wonder: > is this true? > > 1. Even as a user, I agree that what I really want from > NumPy is a core array manipulation package (including > matrices). BUT as long as this is the core of NumPy, > will a developer care if other features are available? > > 2. Even if the answer to 1. is yes, could the > build/installation process include an option not to > build/install anything but the core array functionality? > > 3. It seems to me that pushing things out into SciPy remains > a problem: a basic NumPy is easy to build on any platform, > but SciPy still seems to generate many questions. > > 4. One reason this keeps coming up is that he NumPy/SciPy > split is rather too crude. If the split were instead > something like NumPy/SciPyBasic/SciPyFull/SciPyFull+Kits > where SciPyBasic contained only pure Python code (no > extensions), perhaps the desired location would be more > obvious and some of this recurrent discussion would go away. It seems to me that there are two separate issues people are talking about when they talk about packaging: * How should functions be arranged in the namespaces? numpy.foo(), scipy.foo(), numpy.lib.financial.foo(), scikits.foo(), numkitfull.foo()? * Which code should be distributed together? Should scipy require separate downloading and compilation from numpy? The two questions are not completely independent - it would be horribly confusing to have the set of functions available in a given namespace depend on which packages you had installed - but for the most part it's not a problem to have several toplevel namespaces in one package (python's library is the biggest example of this I know of). For the first question, there's definitely a question about how much should be done with namespaces and how much with documentation. The second is a different story. Personally, I would prefer if numpy and scipy were distributed together, preferably with matplotlib. Then anybody who used numpy would have available all the scpy tools and all the plotting tools; I think it would cut down on wheel reinvention and make application development easier. Teachers would not need to restrict themselves to using only functions built into numpy for fear that their students might not have scipy installed - how many students have learned to save their arrays in unportable binary formats because their teacher didn't want them to have to install scipy? I realize that this poses technical problems. For me installing scipy is just a matter of clicking on a checkbox and installing a 30 MB package, but I realize that some platforms make this much more difficult. If we can't just bundle the two, fine. But I think it is mad to consider subdividing further if we don't have to. I think python's success is due in part to its "batteries included" library. The fact that you can just write a short python script with no extra dependencies that can download files from the Web, parse XML, manage subprocesses, and save persistent objects makes development much faster than if you had to forever decide between adding dependencies and reinventing the wheel. I think numpy and scipy should follow the same principle, of coming "batteries included". So in this specific case, yes, I think the financial functions should absolutely be included; whether they should be included in scipy or numpy is less important to me because I think everyone should install both packages. Anne From tim.hochberg at ieee.org Fri Apr 4 19:17:55 2008 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Fri, 4 Apr 2008 16:17:55 -0700 Subject: [Numpy-discussion] Simple financial functions for NumPy In-Reply-To: References: <47F63204.5030000@enthought.com> <47F63719.6030605@enthought.com> <20080404150955.GB23808@phare.normalesup.org> Message-ID: On Fri, Apr 4, 2008 at 3:31 PM, Anne Archibald wrote: > On 04/04/2008, Alan G Isaac wrote: > > On Fri, 4 Apr 2008, Gael Varoquaux apparently wrote: > > > I really thing numpy should be as thin as possible, so > > > that you can really say that it is only an array > > > manipulation package. This will also make it easier to > > > sell as a core package for developpers who do not care > > > about "calculator" features. > > > > > > I'm a user rather than a developer, but I wonder: > > is this true? > > > > 1. Even as a user, I agree that what I really want from > > NumPy is a core array manipulation package (including > > matrices). BUT as long as this is the core of NumPy, > > will a developer care if other features are available? > > > > 2. Even if the answer to 1. is yes, could the > > build/installation process include an option not to > > build/install anything but the core array functionality? > > > > 3. It seems to me that pushing things out into SciPy remains > > a problem: a basic NumPy is easy to build on any platform, > > but SciPy still seems to generate many questions. > > > > 4. One reason this keeps coming up is that he NumPy/SciPy > > split is rather too crude. If the split were instead > > something like NumPy/SciPyBasic/SciPyFull/SciPyFull+Kits > > where SciPyBasic contained only pure Python code (no > > extensions), perhaps the desired location would be more > > obvious and some of this recurrent discussion would go away. > > It seems to me that there are two separate issues people are talking > about when they talk about packaging: > > * How should functions be arranged in the namespaces? numpy.foo(), > scipy.foo(), numpy.lib.financial.foo(), scikits.foo(), > numkitfull.foo()? > > * Which code should be distributed together? Should scipy require > separate downloading and compilation from numpy? > > The two questions are not completely independent - it would be > horribly confusing to have the set of functions available in a given > namespace depend on which packages you had installed - but for the > most part it's not a problem to have several toplevel namespaces in > one package (python's library is the biggest example of this I know > of). > > For the first question, there's definitely a question about how much > should be done with namespaces and how much with documentation. The > second is a different story. > > Personally, I would prefer if numpy and scipy were distributed > together, preferably with matplotlib. Then anybody who used numpy > would have available all the scpy tools and all the plotting tools; I > think it would cut down on wheel reinvention and make application > development easier. Teachers would not need to restrict themselves to > using only functions built into numpy for fear that their students > might not have scipy installed - how many students have learned to > save their arrays in unportable binary formats because their teacher > didn't want them to have to install scipy? > > I realize that this poses technical problems. For me installing scipy > is just a matter of clicking on a checkbox and installing a 30 MB > package, but I realize that some platforms make this much more > difficult. If we can't just bundle the two, fine. But I think it is > mad to consider subdividing further if we don't have to. If these were tightly tied together, for instance in one big dll , this would be unpleasant for me. I still have people downloading stuff over 56k modems and adding an extra 30 MB to the already somewhat bloated numpy distribution would make there lives more tedious than they already are. I think python's success is due in part to its "batteries included" > library. The fact that you can just write a short python script with > no extra dependencies that can download files from the Web, parse XML, > manage subprocesses, and save persistent objects makes development > much faster than if you had to forever decide between adding > dependencies and reinventing the wheel. I think numpy and scipy should > follow the same principle, of coming "batteries included". One thing they try to do in Python proper is think a lot more before adding stuff to the standard library. Generally packages need to exist separately for some period of time to prove there general utility and to stabilize before they get accepted. Particularly in the core, but in the library as well, they make an effort to chose a compact set of primitive operations without a lot of duplication (the old "There should be one-- and preferably only one --obvious way to do it."). The numpy community has, particularly of late, been rather quick to add things that seem like they *might *be useful. One of the advantages of having multiple namespaces would have been to enforce a certain amount of discipline on what went into numpy, since it would've been easier to look at and evaluate a few dozen functions that might have comprised some subpackage rather than, let's say, five hundred or so. I suspect it's too late now; numpy has chosen the path of matlab and the other array packages and is slowly accumulating nearly everything in one big flat namespace. I don't like it, but it seems pointless to fight it at this point. So in this specific case, yes, I think the financial functions should > absolutely be included; whether they should be included in scipy or > numpy is less important to me because I think everyone should install > both packages. > Personally I think it's a bad idea to add stuff that, as far as I can tell, no has even asked for yet. Put them in the sandbox. Advertise them. If people use them, figure out what needs to be changed. Then add them to SciPy once they've stabilized, if they actually get used. -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From neilcrighton at gmail.com Sat Apr 5 06:12:08 2008 From: neilcrighton at gmail.com (Neil Crighton) Date: Sat, 5 Apr 2008 11:12:08 +0100 Subject: [Numpy-discussion] Simple financial functions for NumPy Message-ID: <63751c30804050312l52fd737ahc93d1c589cbe1837@mail.gmail.com> I'm just a numpy user, but for what it's worth, I would much prefer to have a single numpy namespace with a small as possible number of objects inside that namespace. To me, 'as small as possible' means that it only includes the array and associated array manipulation functions (searchsorted, where, and the record array functions), and the various functions that operate on arrays (exp, log, sin, cos, abs, any, etc). Having a small number of objects in a single namespace means that numpy is much easier to learn for beginners, as it's easier to find the appropriate thing for what you want to do (this is also helped by reducing duplication *shakes fist at .compress* and good documentation). It's also much easier to explore with dir() to jog your memory as to what function you need for a task. If I felt I contributed enough to this list to have a '1', I would be -1 on adding financial functions to numpy. > On Fri, Apr 4, 2008 at 3:31 PM, Anne Archibald > wrote: > > > On 04/04/2008, Alan G Isaac wrote: > > > > It seems to me that there are two separate issues people are talking > > about when they talk about packaging: > > > > * How should functions be arranged in the namespaces? numpy.foo(), > > scipy.foo(), numpy.lib.financial.foo(), scikits.foo(), > > numkitfull.foo()? > > > > * Which code should be distributed together? Should scipy require > > separate downloading and compilation from numpy? > > > > The two questions are not completely independent - it would be > > horribly confusing to have the set of functions available in a given > > namespace depend on which packages you had installed - but for the > > most part it's not a problem to have several toplevel namespaces in > > one package (python's library is the biggest example of this I know > > of). > > > > For the first question, there's definitely a question about how much > > should be done with namespaces and how much with documentation. The > > second is a different story. > > > > Personally, I would prefer if numpy and scipy were distributed > > together, preferably with matplotlib. Then anybody who used numpy > > would have available all the scpy tools and all the plotting tools; I > > think it would cut down on wheel reinvention and make application > > development easier. Teachers would not need to restrict themselves to > > using only functions built into numpy for fear that their students > > might not have scipy installed - how many students have learned to > > save their arrays in unportable binary formats because their teacher > > didn't want them to have to install scipy? > > > > I realize that this poses technical problems. For me installing scipy > > is just a matter of clicking on a checkbox and installing a 30 MB > > package, but I realize that some platforms make this much more > > difficult. If we can't just bundle the two, fine. But I think it is > > mad to consider subdividing further if we don't have to. > > > If these were tightly tied together, for instance in one big dll , this > would be unpleasant for me. I still have people downloading stuff over 56k > modems and adding an extra 30 MB to the already somewhat bloated numpy > distribution would make there lives more tedious than they already are. > > I think python's success is due in part to its "batteries included" > > > library. The fact that you can just write a short python script with > > no extra dependencies that can download files from the Web, parse XML, > > manage subprocesses, and save persistent objects makes development > > much faster than if you had to forever decide between adding > > dependencies and reinventing the wheel. I think numpy and scipy should > > follow the same principle, of coming "batteries included". > > > One thing they try to do in Python proper is think a lot more before adding > stuff to the standard library. Generally packages need to exist separately > for some period of time to prove there general utility and to stabilize > before they get accepted. Particularly in the core, but in the library as > well, they make an effort to chose a compact set of primitive operations > without a lot of duplication (the old "There should be one-- and preferably > only one --obvious way to do it."). The numpy community has, particularly of > late, been rather quick to add things that seem like they *might *be useful. > > One of the advantages of having multiple namespaces would have been to > enforce a certain amount of discipline on what went into numpy, since it > would've been easier to look at and evaluate a few dozen functions that > might have comprised some subpackage rather than, let's say, five hundred or > so. > > I suspect it's too late now; numpy has chosen the path of matlab and the > other array packages and is slowly accumulating nearly everything in one big > flat namespace. I don't like it, but it seems pointless to fight it at this > point. > > > So in this specific case, yes, I think the financial functions should > > absolutely be included; whether they should be included in scipy or > > numpy is less important to me because I think everyone should install > > both packages. > > > > Personally I think it's a bad idea to add stuff that, as far as I can tell, > no has even asked for yet. Put them in the sandbox. Advertise them. If > people use them, figure out what needs to be changed. Then add them to SciPy > once they've stabilized, if they actually get used. > > From matthew.brett at gmail.com Sat Apr 5 06:40:42 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 5 Apr 2008 10:40:42 +0000 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <20080404203133.GB27485@phare.normalesup.org> References: <20080404203133.GB27485@phare.normalesup.org> Message-ID: <1e2af89e0804050340q562c95a3pef8ede0e1b815140@mail.gmail.com> > +1 (and s/students/colleagues). Surely you mean: s.replace('students', colleagues') ! Matthew From bsouthey at gmail.com Sat Apr 5 14:01:45 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Sat, 5 Apr 2008 13:01:45 -0500 Subject: [Numpy-discussion] Ticket #605 Incorrect behavior of numpy.histogram Message-ID: Hi, I have been investigating Ticket #605 'Incorrect behavior of numpy.histogram' (http://scipy.org/scipy/numpy/ticket/605 ). The fix for this ticket really depends on what the expectations are for the bin limits and different applications have different behavior. Consequently, I think that feedback from the community is important. I have attached a modified histogram function where I use a very simple and obvious example: r= numpy.array([1,2,2,3,3,3,4,4,4,4,5,5,5,5,5]) dbin=[2,3,4] The current (Default) behavior provides the counts as array([2, 3, 9]). Here the values less than 2 are ignored and the last bin contains all values greater than or equal to 4. 1) Should the first bin contain all values less than or equal to the value of the first limit and the last bin contain all values greater than the value of the last limit? This produced the counts as: array([3, 3, 9]) (I termed this 'Accumulate' in the output). 2) Should any values outside than the range of the bins be excluded? That is remove any value that is smaller than the lowest value of the bin and higher than the highest value of the bin. This produced the counts as: array([2, 3, 4]) (I termed this 'Exclude' in the output) 3) Should there be extra bins for these values? While I did not implement this option, it would provide the counts as: array([1,2,3,4,5]) 4) Is there some other expectation? Thanks for any input, Bruce -------------- next part -------------- A non-text attachment was scrubbed... Name: histo.py Type: text/x-python Size: 2944 bytes Desc: not available URL: From philbinj at gmail.com Sat Apr 5 14:53:03 2008 From: philbinj at gmail.com (James Philbin) Date: Sat, 5 Apr 2008 19:53:03 +0100 Subject: [Numpy-discussion] Final push for NumPy 1.0.5 (I need your help!) In-Reply-To: References: <47F6339C.3050504@enthought.com> Message-ID: <2b1c8c4f0804051153p2a6dce5dy3fffe415d53e8313@mail.gmail.com> I've posted patches for: #630: If float('123.45') works, so should numpy.float32('123.45') #581: random.set_state does not reset state of random.standard_normal James On Fri, Apr 4, 2008 at 6:49 PM, Anne Archibald wrote: > On 04/04/2008, Travis E. Oliphant wrote: > > > Hey Anne, > > > > Do you currently have SVN access? Would you like it? > > > > I think the SciPy/NumPy sprint would be a good time to clean-up the > > committers list and add new people interested in helping. > > I don't have SVN access. I'd be happy (and careful!) to have it, but I > should warn you that I won't have time to do serious, regular > development on scipy/numpy; I do hope to be able to write a little > code here and there, though, and it would be handy to be able to add > it directly instead of sending patches into the ether. > > Anne > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- A non-text attachment was scrubbed... Name: random_setstate_gaussian.patch Type: text/x-patch Size: 617 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: f32_from_str.patch Type: text/x-patch Size: 687 bytes Desc: not available URL: From philbinj at gmail.com Sat Apr 5 15:17:23 2008 From: philbinj at gmail.com (James Philbin) Date: Sat, 5 Apr 2008 20:17:23 +0100 Subject: [Numpy-discussion] Ticket #605 Incorrect behavior of numpy.histogram In-Reply-To: References: Message-ID: <2b1c8c4f0804051217j4c9c05abs6a385f22311249c9@mail.gmail.com> The matlab behaviour is to extend the first bin to include all data down to -inf and extend the last bin to handle all data to inf. This is probably the behaviour with least suprise. Therefor, I would vote +1 for behaviour #1 by default, +1 for keeping the old behaviour #2 around as an option and -1 for any extending of the bins (#3). James From peridot.faceted at gmail.com Sat Apr 5 15:54:27 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sat, 5 Apr 2008 15:54:27 -0400 Subject: [Numpy-discussion] Ticket #605 Incorrect behavior of numpy.histogram In-Reply-To: References: Message-ID: On 05/04/2008, Bruce Southey wrote: > 1) Should the first bin contain all values less than or equal to the > value of the first limit and the last bin contain all values greater > than the value of the last limit? > This produced the counts as: array([3, 3, 9]) (I termed this > 'Accumulate' in the output). > > 2) Should any values outside than the range of the bins be excluded? > That is remove any value that is smaller than the lowest value of the > bin and higher than the highest value of the bin. > This produced the counts as: array([2, 3, 4]) (I termed this 'Exclude' > in the output) > > 3) Should there be extra bins for these values? > While I did not implement this option, it would provide the counts as: > array([1,2,3,4,5]) There's also a fourth option - raise an exception if any points are outside the range. I hope this is a question about defaults - really what I would most want is to have the choice, as a keyword option. For the default, I would be tempted to go with option 4, raising an exception. This seems pretty much guaranteed not to produce surprising results: if there's any question about what the results should be it produces an error, and the user can run it again with specific instructions. Plus in many contexts having any points that don't belong in any bin is the result of a programming error. Anne From peridot.faceted at gmail.com Sat Apr 5 18:10:11 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sat, 5 Apr 2008 18:10:11 -0400 Subject: [Numpy-discussion] Final push for NumPy 1.0.5 (I need your help!) In-Reply-To: <2b1c8c4f0804051153p2a6dce5dy3fffe415d53e8313@mail.gmail.com> References: <47F6339C.3050504@enthought.com> <2b1c8c4f0804051153p2a6dce5dy3fffe415d53e8313@mail.gmail.com> Message-ID: On 05/04/2008, James Philbin wrote: > I've posted patches for: > > #630: If float('123.45') works, so should numpy.float32('123.45') > #581: random.set_state does not reset state of random.standard_normal Patches for #601, #622, #692, #696, #717 now in trac; I'd like to do something about #720 (variance of complex arrays needs docs and tests), but any patch for it is necessarily going to tread on the toes of my patch for #696 (ddof parameter to var needs tests). I'm not quite sure what the best way to handle this sort of thing is. More generally, my local working copy is now rater divergent from the upstream. What's the recommended way to deal with this? Make sure I have all the patches submitted to trac, then just revert to raw SVN? So far I've just been editing the output of svn diff down to only the bit that's relevant to each patch, but it's getting pretty long. Anne From charlesr.harris at gmail.com Sat Apr 5 18:40:52 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 5 Apr 2008 16:40:52 -0600 Subject: [Numpy-discussion] Final push for NumPy 1.0.5 (I need your help!) In-Reply-To: References: <47F6339C.3050504@enthought.com> <2b1c8c4f0804051153p2a6dce5dy3fffe415d53e8313@mail.gmail.com> Message-ID: On Sat, Apr 5, 2008 at 4:10 PM, Anne Archibald wrote: > On 05/04/2008, James Philbin wrote: > > I've posted patches for: > > > > #630: If float('123.45') works, so should numpy.float32('123.45') > > #581: random.set_state does not reset state of random.standard_normal > > Patches for #601, #622, #692, #696, #717 now in trac; I'd like to do > something about #720 (variance of complex arrays needs docs and > tests), but any patch for it is necessarily going to tread on the toes > of my patch for #696 (ddof parameter to var needs tests). I'm not > quite sure what the best way to handle this sort of thing is. > > More generally, my local working copy is now rater divergent from the > upstream. What's the recommended way to deal with this? Make sure I > have all the patches submitted to trac, then just revert to raw SVN? > So far I've just been editing the output of svn diff down to only the > bit that's relevant to each patch, but it's getting pretty long. > Do you have commit access now? You could pull a fresh copy of svn, apply your patches one by one, and commit each time. That way you can avoid conflicts, which can be a pain when your local copy has diverged a lot from upstream. Lots of small commented changes are also preferable to one big patch. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Sat Apr 5 19:01:34 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sat, 5 Apr 2008 19:01:34 -0400 Subject: [Numpy-discussion] Final push for NumPy 1.0.5 (I need your help!) In-Reply-To: References: <47F6339C.3050504@enthought.com> <2b1c8c4f0804051153p2a6dce5dy3fffe415d53e8313@mail.gmail.com> Message-ID: On 05/04/2008, Charles R Harris wrote: > On Sat, Apr 5, 2008 at 4:10 PM, Anne Archibald > wrote: > > More generally, my local working copy is now rater divergent from the > > upstream. What's the recommended way to deal with this? Make sure I > > have all the patches submitted to trac, then just revert to raw SVN? > > So far I've just been editing the output of svn diff down to only the > > bit that's relevant to each patch, but it's getting pretty long. > > Do you have commit access now? You could pull a fresh copy of svn, apply > your patches one by one, and commit each time. That way you can avoid > conflicts, which can be a pain when your local copy has diverged a lot from > upstream. Lots of small commented changes are also preferable to one big > patch. No, or at least, I don't know a username/password to use. I also don't seem to be able to change the status of bugs on the trac. Anne From stefan at sun.ac.za Sat Apr 5 22:45:20 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 6 Apr 2008 04:45:20 +0200 Subject: [Numpy-discussion] matrix multiply Message-ID: <9457e7c80804051945u558edd93v12df417824e10877@mail.gmail.com> Hi all, Some discussion recently took place around raising a square matrices to integer powers. See ticket #601: http://scipy.org/scipy/numpy/ticket/601 Anne Archibald wrote a patch which factored 'matrix_multiply' out of defmatrix (the matrix power implemented for the Matrix class). After some further discussion on irc, and some careful footwork so that everything imports correctly, I checked in http://projects.scipy.org/scipy/numpy/changeset/4968 The matrix_multiply method is exposed as numpy.linalg.matrix_multiply, and is not in the top-level numpy namespace (it's a bit crowded there already). I'd be glad if you would review the changeset and comment. Thanks, St?fan From peridot.faceted at gmail.com Sat Apr 5 23:40:17 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sat, 5 Apr 2008 23:40:17 -0400 Subject: [Numpy-discussion] matrix power (was: matrix multiply) Message-ID: On 05/04/2008, St?fan van der Walt wrote: > Some discussion recently took place around raising a square matrices > to integer powers. See ticket #601: > > http://scipy.org/scipy/numpy/ticket/601 > > Anne Archibald wrote a patch which factored 'matrix_multiply' out of > defmatrix (the matrix power implemented for the Matrix class). After > some further discussion on irc, and some careful footwork so that > everything imports correctly, I checked in > > http://projects.scipy.org/scipy/numpy/changeset/4968 > > The matrix_multiply method is exposed as numpy.linalg.matrix_multiply, > and is not in the top-level numpy namespace (it's a bit crowded there > already). I'd be glad if you would review the changeset and comment. Just to be clear, the function is called matrix_power, not matrix_multiply. Anne From lists at informa.tiker.net Sun Apr 6 00:21:43 2008 From: lists at informa.tiker.net (Andreas =?utf-8?q?Kl=C3=B6ckner?=) Date: Sun, 6 Apr 2008 00:21:43 -0400 Subject: [Numpy-discussion] Forcing the use of -lgfortran Message-ID: <200804060021.45351.lists@informa.tiker.net> Hi all, I'm having trouble getting numpy to compile something usable on a cluster I'm using, in particular I see 8< ----------------------------------------------------- ImportError: /users/kloeckner/mach/x86_64/pool/lib/python2.5/site-packages/numpy/linalg/lapack_lite.so: undefined symbol: _gfortran_st_write 8< ----------------------------------------------------- Key point: I need -lgfortran on the link command line, or else I get unresolved symbols stemming from my LAPACK library. But even if I add this: 8< ----------------------------------------------------- [blas_opt] libraries = f77blas, cblas, atlas,gfortran library_dirs = /users/kloeckner/mach/x86_64/pool/lib include_dirs = /users/kloeckner/mach/x86_64/pool/include # [lapack_opt] libraries = lapack, f77blas, cblas, atlas,gfortran library_dirs = /users/kloeckner/mach/x86_64/pool/lib 8< ----------------------------------------------------- to site.cfg, numpy seemingly ignores this request and uses ATLAS's ptblas instead (which I positively do *not* want it to use). How can I fix this? This is what I get for __config__.py: 8< ----------------------------------------------------- blas_opt_info={'libraries': ['ptf77blas', 'ptcblas', 'atlas'], 'library_dirs': ['/users/kloeckner/mach/x86_64/pool/lib'], 'language': 'c', 'define_macros': [('ATLAS_INFO', '"\\"3.8.1\\""')], 'include_dirs': ['/users/kloeckner/mach/x86_64/pool/include']} atlas_blas_threads_info={'libraries': ['ptf77blas', 'ptcblas', 'atlas'], 'library_dirs': ['/users/kloeckner/mach/x86_64/pool/lib'], 'langu age': 'c', 'include_dirs': ['/users/kloeckner/mach/x86_64/pool/include']} lapack_opt_info={'libraries': ['lapack', 'ptf77blas', 'ptcblas', 'atlas'], 'library_dirs': ['/users/kloeckner/mach/x86_64/pool/lib'], 'lan guage': 'f77', 'define_macros': [('ATLAS_INFO', '"\\"3.8.1\\""')], 'include_dirs': ['/users/kloeckner/mach/x86_64/pool/include']} 8< ----------------------------------------------------- Thanks, Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From lists at informa.tiker.net Sun Apr 6 01:36:05 2008 From: lists at informa.tiker.net (Andreas =?iso-8859-1?q?Kl=F6ckner?=) Date: Sun, 6 Apr 2008 01:36:05 -0400 Subject: [Numpy-discussion] Forcing the use of -lgfortran In-Reply-To: <200804060021.45351.lists@informa.tiker.net> References: <200804060021.45351.lists@informa.tiker.net> Message-ID: <200804060136.12718.lists@informa.tiker.net> I can answer my own question now: 1) Option --fcompiler=gnu95 2) Add the following to site.cfg [atlas] library_dirs = /users/kloeckner/mach/x86_64/pool/lib,/usr/lib atlas_libs = lapack, f77blas, cblas, atlas Andreas On Sonntag 06 April 2008, Andreas Kl?ckner wrote: > Hi all, > > I'm having trouble getting numpy to compile something usable on a cluster > I'm using, in particular I see > > 8< ----------------------------------------------------- > ImportError: > /users/kloeckner/mach/x86_64/pool/lib/python2.5/site-packages/numpy/linalg/ >lapack_lite.so: undefined symbol: _gfortran_st_write > 8< ----------------------------------------------------- > > Key point: I need -lgfortran on the link command line, or else I get > unresolved symbols stemming from my LAPACK library. > > But even if I add this: > > 8< ----------------------------------------------------- > [blas_opt] > libraries = f77blas, cblas, atlas,gfortran > library_dirs = /users/kloeckner/mach/x86_64/pool/lib > include_dirs = /users/kloeckner/mach/x86_64/pool/include > # > [lapack_opt] > libraries = lapack, f77blas, cblas, atlas,gfortran > library_dirs = /users/kloeckner/mach/x86_64/pool/lib > 8< ----------------------------------------------------- > > to site.cfg, numpy seemingly ignores this request and uses ATLAS's ptblas > instead (which I positively do *not* want it to use). How can I fix this? > > This is what I get for __config__.py: > > 8< ----------------------------------------------------- > blas_opt_info={'libraries': ['ptf77blas', 'ptcblas', 'atlas'], > 'library_dirs': ['/users/kloeckner/mach/x86_64/pool/lib'], 'language': 'c', > 'define_macros': [('ATLAS_INFO', '"\\"3.8.1\\""')], 'include_dirs': > ['/users/kloeckner/mach/x86_64/pool/include']} > atlas_blas_threads_info={'libraries': > ['ptf77blas', 'ptcblas', 'atlas'], 'library_dirs': > ['/users/kloeckner/mach/x86_64/pool/lib'], 'langu > age': 'c', 'include_dirs': ['/users/kloeckner/mach/x86_64/pool/include']} > lapack_opt_info={'libraries': > ['lapack', 'ptf77blas', 'ptcblas', 'atlas'], 'library_dirs': > ['/users/kloeckner/mach/x86_64/pool/lib'], 'lan > guage': 'f77', 'define_macros': > [('ATLAS_INFO', '"\\"3.8.1\\""')], 'include_dirs': > ['/users/kloeckner/mach/x86_64/pool/include']} > 8< ----------------------------------------------------- > > Thanks, > Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From nwagner at iam.uni-stuttgart.de Sun Apr 6 04:55:35 2008 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Sun, 06 Apr 2008 10:55:35 +0200 Subject: [Numpy-discussion] Matrix powers Message-ID: Hi all, I tried to use the new function matrix_power, but I can't find it. >>> matrix_power(array([[0,1],[-1,0]]),10) Traceback (most recent call last): File "", line 1, in ? NameError: name 'matrix_power' is not defined >>> numpy.__version__ '1.0.5.dev4968' Am I missing something ? Nils From nwagner at iam.uni-stuttgart.de Sun Apr 6 05:01:06 2008 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Sun, 06 Apr 2008 11:01:06 +0200 Subject: [Numpy-discussion] Matrix powers In-Reply-To: References: Message-ID: On Sun, 06 Apr 2008 10:55:35 +0200 "Nils Wagner" wrote: > Hi all, > > I tried to use the new function matrix_power, but I >can't > find it. > >>>> matrix_power(array([[0,1],[-1,0]]),10) > Traceback (most recent call last): > File "", line 1, in ? > NameError: name 'matrix_power' is not defined >>>> numpy.__version__ > '1.0.5.dev4968' > > Am I missing something ? > > Nils Oops, I have missed from numpy.linalg import matrix_power Sorry for the noise. Nils From stefan at sun.ac.za Sun Apr 6 05:17:10 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 6 Apr 2008 11:17:10 +0200 Subject: [Numpy-discussion] matrix power (was: matrix multiply) In-Reply-To: References: Message-ID: <9457e7c80804060217q856556cud40595b7d81dcf12@mail.gmail.com> On 06/04/2008, Anne Archibald wrote: > On 05/04/2008, St?fan van der Walt wrote: > > > Some discussion recently took place around raising a square matrices > > to integer powers. See ticket #601: > > > > http://scipy.org/scipy/numpy/ticket/601 > > > > Anne Archibald wrote a patch which factored 'matrix_multiply' out of > > defmatrix (the matrix power implemented for the Matrix class). After > > some further discussion on irc, and some careful footwork so that > > everything imports correctly, I checked in > > > > http://projects.scipy.org/scipy/numpy/changeset/4968 > > > > The matrix_multiply method is exposed as numpy.linalg.matrix_multiply, > > and is not in the top-level numpy namespace (it's a bit crowded there > > already). I'd be glad if you would review the changeset and comment. > > Just to be clear, the function is called matrix_power, not matrix_multiply. Sorry, 05:00am is maybe not the time to write these posts. Now there is another good reason to review the changes :) Cheers St?fan From philbinj at gmail.com Sun Apr 6 08:16:42 2008 From: philbinj at gmail.com (James Philbin) Date: Sun, 6 Apr 2008 13:16:42 +0100 Subject: [Numpy-discussion] Final push for NumPy 1.0.5 (I need your help!) In-Reply-To: References: <47F6339C.3050504@enthought.com> <2b1c8c4f0804051153p2a6dce5dy3fffe415d53e8313@mail.gmail.com> Message-ID: <2b1c8c4f0804060516q69873084vb6a27f1abf2a72a3@mail.gmail.com> OK, here's a patch for: #718: Bug with numpy.float32.tolist Can someone commit it (I hope someone has committed the other patches i've sent)? James -------------- next part -------------- A non-text attachment was scrubbed... Name: scalar_tolist.patch Type: text/x-patch Size: 493 bytes Desc: not available URL: From aisaac at american.edu Sun Apr 6 12:04:00 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 6 Apr 2008 12:04:00 -0400 Subject: [Numpy-discussion] =?iso-8859-1?q?Final_push_for_NumPy_1=2E0=2E5_?= =?iso-8859-1?q?=28I_need_your=09help!=29?= In-Reply-To: <2b1c8c4f0804060516q69873084vb6a27f1abf2a72a3@mail.gmail.com> References: <47F6339C.3050504@enthought.com><2b1c8c4f0804051153p2a6dce5dy3fffe415d53e8313@mail.gmail.com><2b1c8c4f0804060516q69873084vb6a27f1abf2a72a3@mail.gmail.com> Message-ID: On Sun, 6 Apr 2008, James Philbin apparently wrote: > OK, here's a patch for: > #718: Bug with numpy.float32.tolist My impression has always been that to ensure a patch gets appropriate consideration it should be attached to a ticket... fwiw, Alan Isaac From oliphant at enthought.com Sun Apr 6 12:07:52 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Sun, 06 Apr 2008 11:07:52 -0500 Subject: [Numpy-discussion] Final push for NumPy 1.0.5 (I need your help!) In-Reply-To: <2b1c8c4f0804060516q69873084vb6a27f1abf2a72a3@mail.gmail.com> References: <47F6339C.3050504@enthought.com> <2b1c8c4f0804051153p2a6dce5dy3fffe415d53e8313@mail.gmail.com> <2b1c8c4f0804060516q69873084vb6a27f1abf2a72a3@mail.gmail.com> Message-ID: <47F8F558.2000603@enthought.com> James Philbin wrote: > OK, here's a patch for: > #718: Bug with numpy.float32.tolist > > Can someone commit it (I hope someone has committed the other patches > i've sent)? > I don't think this patch should be committed without more discussion. This changes behavior and it is intentional that tolist behaves as it does now. -Travis O. From philbinj at gmail.com Sun Apr 6 12:14:41 2008 From: philbinj at gmail.com (James Philbin) Date: Sun, 6 Apr 2008 17:14:41 +0100 Subject: [Numpy-discussion] Final push for NumPy 1.0.5 (I need your help!) In-Reply-To: <47F8F558.2000603@enthought.com> References: <47F6339C.3050504@enthought.com> <2b1c8c4f0804051153p2a6dce5dy3fffe415d53e8313@mail.gmail.com> <2b1c8c4f0804060516q69873084vb6a27f1abf2a72a3@mail.gmail.com> <47F8F558.2000603@enthought.com> Message-ID: <2b1c8c4f0804060914m4bc87e41ja48fc6ad5cce73d1@mail.gmail.com> > My impression has always been that to ensure > a patch gets appropriate consideration it > should be attached to a ticket... Point taken. > I don't think this patch should be committed without more discussion. > This changes behavior and it is intentional that tolist behaves as it > does now. I somewhat understand the argument about scalar tolist returning just the number (as it's 0-d), but it really does seem that a function called 'tolist' should do what it says on the packet (and the function doc) or be removed for scalars. James On Sun, Apr 6, 2008 at 5:07 PM, Travis E. Oliphant wrote: > > James Philbin wrote: > > OK, here's a patch for: > > #718: Bug with numpy.float32.tolist > > > > Can someone commit it (I hope someone has committed the other patches > > i've sent)? > > > > I don't think this patch should be committed without more discussion. > This changes behavior and it is intentional that tolist behaves as it > does now. > > -Travis O. > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From aisaac at american.edu Sun Apr 6 12:25:53 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 6 Apr 2008 12:25:53 -0400 Subject: [Numpy-discussion] matrix multiply In-Reply-To: <9457e7c80804051945u558edd93v12df417824e10877@mail.gmail.com> References: <9457e7c80804051945u558edd93v12df417824e10877@mail.gmail.com> Message-ID: On Sun, 6 Apr 2008, St?fan van der Walt apparently wrote: > I'd be glad if you would review the changeset and comment. Just checking: it's important to me that this won't change the behavior of boolean matrices, but I don't see a test for this. E.g., :: >>> import numpy as N >>> A = N.mat('1 0;1 1',dtype='bool') >>> A**2 matrix([[ True, False], [ True, True]], dtype=bool) Cheers, Alan Isaac From peridot.faceted at gmail.com Sun Apr 6 13:49:49 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sun, 6 Apr 2008 13:49:49 -0400 Subject: [Numpy-discussion] matrix multiply In-Reply-To: References: <9457e7c80804051945u558edd93v12df417824e10877@mail.gmail.com> Message-ID: On 06/04/2008, Alan G Isaac wrote: > Just checking: > it's important to me that this won't change > the behavior of boolean matrices, but I don't > see a test for this. E.g., :: > > >>> import numpy as N > >>> A = N.mat('1 0;1 1',dtype='bool') > >>> A**2 > matrix([[ True, False], > [ True, True]], dtype=bool) I have no desire to change the behaviour of boolean matrices, and I'll write a test, but what is it supposed to do with them? Just produce reduce(dot,[A]*n)? For zero it will give the identity, and for negative powers some sort of floating-point inverse. Currently for positive powers it should produce the right answer provided multiplication is associative (which I think it is). The inverse actually poses a problem: should it return (A**(-1))**n or (A**n)**(-1)? (No, they're not the same for boolean matrices.) Anne From aisaac at american.edu Sun Apr 6 14:59:54 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 6 Apr 2008 14:59:54 -0400 Subject: [Numpy-discussion] matrix multiply In-Reply-To: References: <9457e7c80804051945u558edd93v12df417824e10877@mail.gmail.com> Message-ID: > On 06/04/2008, Alan G Isaac wrote: >> Just checking: >> it's important to me that this won't change >> the behavior of boolean matrices, but I don't >> see a test for this. E.g., :: >> >>> import numpy as N >> >>> A = N.mat('1 0;1 1',dtype='bool') >> >>> A**2 >> matrix([[ True, False], >> [ True, True]], dtype=bool) On Sun, 6 Apr 2008, Anne Archibald apparently wrote: > I have no desire to change the behaviour of boolean matrices, and I'll > write a test, but what is it supposed to do with them? Just produce > reduce(dot,[A]*n)? That's the part I care about. > For zero it will give the identity, Yes. > and for negative powers some sort of floating-point > inverse. That deserves discussion. Not all "invertible" boolean matrices have an inverse in the algebra. Just the orthogonal ones do. I guess I would special case inverses for Boolean matrices. Just test if the matrix B is orthogonal (under Boolean multiplication) and if so return B's transpose. > Currently for positive powers it should produce the right > answer provided multiplication is associative (which > I think it is). Yes; N?N boolean matrices are I believe a semi-group under multiplication. > The inverse actually poses a problem: should it return > (A**(-1))**n or (A**n)**(-1)? (No, they're not the same > for boolean matrices.) I think it must be the latter ... ? By associativity, if B has an inverse A, then B**n must have inverse A**n. So you are observing that with boolean matrices we might find B**n is invertible even though B is not. Right? So the latter will work in more cases. So again: form B**n, test for orthogonality, and return the transpose if the test passes. Cheers, Alan Isaac From charlesr.harris at gmail.com Sun Apr 6 15:27:13 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 6 Apr 2008 13:27:13 -0600 Subject: [Numpy-discussion] matrix multiply In-Reply-To: References: <9457e7c80804051945u558edd93v12df417824e10877@mail.gmail.com> Message-ID: On Sun, Apr 6, 2008 at 12:59 PM, Alan G Isaac wrote: > > On 06/04/2008, Alan G Isaac wrote: > >> Just checking: > >> it's important to me that this won't change > >> the behavior of boolean matrices, but I don't > >> see a test for this. E.g., :: > > >> >>> import numpy as N > >> >>> A = N.mat('1 0;1 1',dtype='bool') > >> >>> A**2 > >> matrix([[ True, False], > >> [ True, True]], dtype=bool) > > On Sun, 6 Apr 2008, Anne Archibald apparently wrote: > > I have no desire to change the behaviour of boolean matrices, and I'll > > write a test, but what is it supposed to do with them? Just produce > > reduce(dot,[A]*n)? > > That's the part I care about. > > > For zero it will give the identity, > > Yes. > > > and for negative powers some sort of floating-point > > inverse. > > That deserves discussion. > Not all "invertible" boolean matrices have an inverse in the algebra. > Just the orthogonal ones do. > > I guess I would special case inverses for Boolean matrices. > Just test if the matrix B is orthogonal (under Boolean > multiplication) and if so return B's transpose. > > > Currently for positive powers it should produce the right > > answer provided multiplication is associative (which > > I think it is). > > Yes; N?N boolean matrices are I believe a semi-group under multiplication. The boolean algebra is a field and the correct addition is xor, which is the same as addition modulo 2. This makes all matrices with determinant 1 invertible. This isn't the current convention, however, as it was when Caratheodory was writing on measures and rings of sets were actually rings and the symmetric difference was used instead of union. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Sun Apr 6 15:55:44 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 6 Apr 2008 21:55:44 +0200 Subject: [Numpy-discussion] Final push for NumPy 1.0.5 (I need your help!) In-Reply-To: References: <47F6339C.3050504@enthought.com> <2b1c8c4f0804051153p2a6dce5dy3fffe415d53e8313@mail.gmail.com> <2b1c8c4f0804060516q69873084vb6a27f1abf2a72a3@mail.gmail.com> Message-ID: <9457e7c80804061255l1c29d435saa8200baf25fbf46@mail.gmail.com> On 06/04/2008, Alan G Isaac wrote: > On Sun, 6 Apr 2008, James Philbin apparently wrote: > > OK, here's a patch for: > > #718: Bug with numpy.float32.tolist > > > My impression has always been that to ensure > a patch gets appropriate consideration it > should be attached to a ticket... And, more importantly, it should be accompanied by a test (I'm guilty on the float to string ticket mentioned above). The test not only verifies the patch's correct behaviour, but also guides the reviewer in understanding possible use cases, as well as illustrating calling convention. Anything that saves the reviewer time improves your chances of having the patch accepted. Finally, it is also considerate of his/her time: you wrote the patch and recently reviewed the code, so writing a test takes you much less time than it would the reviewer. That being said, I am very happy with the quality of patches recently submitted by, amongst others, Pauli, Anne, David and Christoph. Your efforts are very much appreciated. Regards St?fan From aisaac at american.edu Sun Apr 6 16:34:10 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 6 Apr 2008 16:34:10 -0400 Subject: [Numpy-discussion] matrix multiply In-Reply-To: References: <9457e7c80804051945u558edd93v12df417824e10877@mail.gmail.com> Message-ID: On Sun, 6 Apr 2008, Charles R Harris wrote: > The boolean algebra is a field and the correct addition is xor, which is > the same as addition modulo 2. This makes all matrices with determinant 1 > invertible. This isn't the current convention, however, as it was when > Caratheodory was writing on measures and rings of sets were actually rings > and the symmetric difference was used instead of union. I am not sure what you are suggesting for matrix behavior, nor what "correct" means here. Comment: Standard *boolean algebra* axioms include distributivity, but 1 xor (0 and 0) = 1 xor 0 = 1 (1 xor 0) and (1 xor 0) = 1 and 1 = 1 So I guess (?) what you are saying is something like: if we have a boolen algebra with operators 'and' and 'or', we can generate a boolean ring with operations 'xor' and 'and'. When we do so, the '+' is traditionally used for the 'xor' operation. But where in the modern literature on boolean matrices is '+' given this interpretation? IANAM,* Alan Isaac * IANAM = I am not a mathematician. From charlesr.harris at gmail.com Sun Apr 6 18:10:28 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 6 Apr 2008 16:10:28 -0600 Subject: [Numpy-discussion] matrix multiply In-Reply-To: References: <9457e7c80804051945u558edd93v12df417824e10877@mail.gmail.com> Message-ID: On Sun, Apr 6, 2008 at 2:34 PM, Alan G Isaac wrote: > On Sun, 6 Apr 2008, Charles R Harris wrote: > > The boolean algebra is a field and the correct addition is xor, which > is > > the same as addition modulo 2. This makes all matrices with determinant > 1 > > invertible. This isn't the current convention, however, as it was when > > Caratheodory was writing on measures and rings of sets were actually > rings > > and the symmetric difference was used instead of union. > > I am not sure what you are suggesting for matrix behavior, > nor what "correct" means here. > > Comment: > Standard *boolean algebra* axioms include distributivity, but > 1 xor (0 and 0) = 1 xor 0 = 1 > (1 xor 0) and (1 xor 0) = 1 and 1 = 1 > > So I guess (?) what you are saying is something like: > if we have a boolen algebra with operators 'and' and 'or', > we can generate a boolean ring with operations 'xor' and 'and'. > When we do so, the '+' is traditionally used for the 'xor' operation. > > But where in the modern literature on boolean matrices is > '+' given this interpretation? > It's generally not. It used to be that \Sigma and + were used for set union, probably because that was what the printers had on hand and what the mathematicians were used to. Then there was the alternate desire to make boolean algebra conform to the standad ring structure which led to using the symmetric difference, \Delta. For instance, if + is 'or', then 1 has no additive inverse, whereas 1 xor 1 = 0. I'm just pointing to some of the history here that I've noticed in old papers. I prefer the modern usage myself as it is closer to the accepted logic operations, but applying algebraic manipulations like powers and matrix inverses in that context leads to strange results. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From garg1 at ualberta.ca Sun Apr 6 20:48:50 2008 From: garg1 at ualberta.ca (Rahul Garg) Date: Sun, 06 Apr 2008 18:48:50 -0600 Subject: [Numpy-discussion] New project : Spyke python-to-C compiler Message-ID: <20080406184850.xiqr9t0hwgco004s@webmail.ualberta.ca> Note this message has been posted to numpy-discussion and python-dev. Sorry for the multiple posting but I thought both python devs and numpy users will be interested. If you believe your list should not receive this email, let me know. Also I just wanted to introduce myself since I may ask doubts about Python and Numpy internals from time to time :) Hi. I am a student at Univ of Alberta doing my masters in computing science. I am writing a Python-to-C compiler as one part of my thesis. The compiler, named Spyke, will be made available in a couple of weeks and is geared towards scientific applications and will therefore focus mostly on needs of scientific app developers. What is Spyke? In many performance critical projects, it is often necessary to rewrite parts of the application in C. However writing C wrappers can be time consuming. Spyke offers an alternative approach. You add annotations to your Python code as strings. These strings are discarded by the Python interpreter but these are interpreted as types by Spyke compiler to convert to C. Example : "int -> int" def f(x): return 2*x In this case the Spyke compiler will consider the string "int -> int" as a decalration that the function accepts int as parameter and returns int. Spyke will then generate a C function and a wrapper function. This idea is directly copied from PLW (Python Language Wrapper) project. Once Python3k arrives, much of these declarations will be moved to function annotations and class decorators. This way you can do all your development and debugging interactively using the standard Python interpreter. When you need to compile to C, you just add type annotations to places that you want to convert and invoke spyke on the annotated module. This is different from Pyrex because Pyrex does not accept Python code. With Spyke, your code is 100% pure python. Spyke has basic support for functions and classes. Spyke can do very basic type inference for local variables in function bodies. Spyke also has partial support for homogenous lists and dictionaries and fixed length tuples. One big advantage of Spyke is that it understands at least part of numpy. Numpy arrays are treated as fundamental types and Spyke knows what C code to generate for slicing/indexing of numpy arrays etc. This should help a lot in scientific applications. Note that Spyke can handle only a subset of Python. Exceptions, iterators, generators, runtime code generation of any kind etc is not handled. Nested functions will be added soon. I will definitely add some of these missing features based on what is actually required for real world Python codes. Currently if Spyke does not understand a function, it just leaves it as Python code. Classes can be handled but special methods are not currently supported. The support of classes is a little brittle because I am trying to resolve some issues b/w old and new style of classes. Where is Spyke? Spyke will be available as a binary only release in a couple of weeks. I intend to make it open source after a few months. Spyke is written in Python and Java and should be platform independant. I do intend to make the source open in a few months. Right now its undergoing very rapid development and has negligible amounts of documentation so the source code right now is pretty useless to anyone else anyway. I need help: However I need a bit of help. I am having a couple of problems : a) I am finding it hard to get pure Python+NumPy testing codes. I need more codes to test the compiler. Developing a compiler without a test-suite is kind of useless. If you have some pure Python codes which need better performance, please contact me. I guarantee that your codes will not be released to public without your permission but might be referenced in academic publications. I can also make the compiler available to you hopefully after 10th of April. Its kind of unstable currently. I will also need your help in annotating the provided testing codes since I probably wont know what your application is doing. b) Libraries which interface with C/C++ : Many codes in SciPy for instance have mixed language codes. Part of the code is written in C/C++. Spyke only knows how to annotated Python codes. For C/C++ libraries wrapped into Python modules, Spyke will therefore need to know at least 2 things : i) The mapping of a C function name/struct etc to Python ii) The type information of the said C function. There are many many ways that people interact with C code. People either write wrappers manually, or use autogenerated wrappers using SWIG or SIP Boost.Python etc., use Pyrex or Cython while some people use ctypes. I dont have the time or resources to support these multitude of methods. I considered trying to parse the C code implementing wrappers but its "non-trivial" to put it mildly. Parsing only SWIG generated code is another possibility but its still hard. Another approach that I am seriously considering is to support a subset of ctypes (with additional restriction) instead. But my question is : Is ctypes good enough for most of you? Ctypes cannot interface with C++ code but its pure Python. However I have not seen too many instances of people using ctypes. c) Strings as type declarations : Do you think I should use decorators instead at least for function type declarations? thanks for patiently reading this, comments and inquiries sought. rahul From peridot.faceted at gmail.com Sun Apr 6 21:35:01 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sun, 6 Apr 2008 21:35:01 -0400 Subject: [Numpy-discussion] matrix multiply In-Reply-To: References: <9457e7c80804051945u558edd93v12df417824e10877@mail.gmail.com> Message-ID: > > and for negative powers some sort of floating-point > > inverse. > > That deserves discussion. > Not all "invertible" boolean matrices have an inverse in the algebra. > Just the orthogonal ones do. > > I guess I would special case inverses for Boolean matrices. > Just test if the matrix B is orthogonal (under Boolean > multiplication) and if so return B's transpose. This is actually a limitation of the whole linear algebra subsystem: we only support floating-point linear algebra (apart from dot()). There are algorithms for doing integer linear algebra, which might be interesting to implement. But that's a big job. Boolean matrices as you use them are another step again: because they are not a group under "+", almost all of linear algebra has to be reinterpreted. For example it's not obvious that matrix multiplication is associative; it turns out to be, because you can replace the matrices by integer matrices containing ones for True and zeros for False, then do the math, then interpret any nonzero integer as True, and zero as False. As an aside, if you use "+" to mean xor (which I am not suggesting!) all of linear algebra continues more or less unchanged; you're just working over the field of integers modulo 2. If you want eigenvalues you can pass to an algebraic closure. > > The inverse actually poses a problem: should it return > > (A**(-1))**n or (A**n)**(-1)? (No, they're not the same > > for boolean matrices.) > > I think it must be the latter ... ? > > By associativity, if B has an inverse A, > then B**n must have inverse A**n. > So you are observing that with boolean matrices > we might find B**n is invertible even though B is not. > Right? So the latter will work in more cases. > So again: form B**n, test for orthogonality, > and return the transpose if the test passes. Well, as it stands, since we have no integer linear algebra, inverses are always floating-point. When you do that, you find that, for example, ([[True,False],[True,True]]**(-1))**2 = [[1.,0.],[-2.,1.]] but ([[True,False],[True,True]]**2)**(-1) = [[1.,0.],[-1.,1.]] I am not aware of any algorithm for finding inverses, or even determining which matrices are invertible, in the peculiar Boolean arithmetic we use. Anne From tgrav at mac.com Sun Apr 6 22:36:05 2008 From: tgrav at mac.com (Tommy Grav) Date: Sun, 6 Apr 2008 22:36:05 -0400 Subject: [Numpy-discussion] Ticket #605 Incorrect behavior of numpy.histogram In-Reply-To: References: Message-ID: <00D24199-8FEF-48E2-8689-52961CD1B5DF@mac.com> On Apr 5, 2008, at 2:01 PM, Bruce Southey wrote: > Hi, > I have been investigating Ticket #605 'Incorrect behavior of > numpy.histogram' (http://scipy.org/scipy/numpy/ticket/605 ). I think that my preference depends on the definition of what the bin number means. If the bin numbers are the lower bounds of the bins (numpy default behavior) then it would make little sense to exclude anything above the largest bin. I don't have access to numpy on my laptop at the moment, so I can't remember wether numpy has a keyword for what the bins array is defining? Having this as a keyword of (lower,middle,upper) of the bin would be very helpful. Cheers Tommy From aisaac at american.edu Sun Apr 6 22:51:49 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 6 Apr 2008 22:51:49 -0400 Subject: [Numpy-discussion] matrix multiply In-Reply-To: References: <9457e7c80804051945u558edd93v12df417824e10877@mail.gmail.com> Message-ID: On Sun, 6 Apr 2008, Anne Archibald apparently wrote: > I am not aware of any algorithm for finding inverses, or > even determining which matrices are invertible, in the > peculiar Boolean arithmetic we use. Again, it is *not* peculiar, it is very standard for boolean matrices. And with this behavior, a nonnegative integer power has an obvious graph theoretic interpretation. Such boolean matrices are obviously invertible if they are orthogonal. It turn out this is a necessary condition as well. [1]_ Orthogonality is obviously easy to test. Cheers, Alan Isaac .. [1] Luce, D., 1952, "A Note on Boolean Matrix Theory", Proceeding of the Am. Math. Soc 3(3), p.382-8. From aisaac at american.edu Sun Apr 6 22:51:51 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 6 Apr 2008 22:51:51 -0400 Subject: [Numpy-discussion] matrix multiply In-Reply-To: References: <9457e7c80804051945u558edd93v12df417824e10877@mail.gmail.com> Message-ID: On Sun, 6 Apr 2008, Charles R Harris apparently wrote: > I prefer the modern usage myself as it is closer to the > accepted logic operations, but applying algebraic > manipulations like powers and matrix inverses in that > context leads to strange results. I have not really thought much about inverses, but nonnegative integer powers have a natural interpretation in graph theory (with the boolean algebra operations, not the boolean ring operations). This is exactly what I was requesting be preserved. Cheers, Alan Isaac From peridot.faceted at gmail.com Sun Apr 6 23:14:43 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sun, 6 Apr 2008 23:14:43 -0400 Subject: [Numpy-discussion] ma's stdu and varu Message-ID: Hi, I was just going through tidying up the documentation for all the many functions in numpy that compute standard deviations or variances (the functions, the methods, the methods on matrices, the methods on maskedarrays, all needed their docstrings updated in approximately the same way). I noticed that maskedarrays seem to have the functions "stdu" and "varu", for computing the unbiased standard deviation and variance (where one divides by N-1 rather than N). These are now available, I think, via std and var with ddof=1. More seriously, they still provide the peculiar older definition of var, where varu([1,1.j,-1,-1.j])==0. I don't use maskedarrays, and in fact I haven't been keeping track of what's going on with the two different implementations. So: what should be done about stdu and varu? Do the two implementations both need their definitions of std and var examined? Thanks, Anne From charlesr.harris at gmail.com Sun Apr 6 23:34:48 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 6 Apr 2008 21:34:48 -0600 Subject: [Numpy-discussion] matrix multiply In-Reply-To: References: <9457e7c80804051945u558edd93v12df417824e10877@mail.gmail.com> Message-ID: On Sun, Apr 6, 2008 at 8:51 PM, Alan G Isaac wrote: > On Sun, 6 Apr 2008, Charles R Harris apparently wrote: > > I prefer the modern usage myself as it is closer to the > > accepted logic operations, but applying algebraic > > manipulations like powers and matrix inverses in that > > context leads to strange results. > > I have not really thought much about inverses, > but nonnegative integer powers have a natural > interpretation in graph theory (with the boolean > algebra operations, not the boolean ring operations). > This is exactly what I was requesting be preserved. > You mean as edges in a directed graph? I suppose so, although graph algorithms tend to use different structures, lists of lists, trees, and such. I would think plain old integer matrices might actually carry more information; let me count the ways. And positive matrices have their own oddities. Hmm... I wonder what matrices over Z_2 mean in that context? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Mon Apr 7 00:02:57 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Sun, 06 Apr 2008 23:02:57 -0500 Subject: [Numpy-discussion] New project : Spyke python-to-C compiler In-Reply-To: <20080406184850.xiqr9t0hwgco004s@webmail.ualberta.ca> References: <20080406184850.xiqr9t0hwgco004s@webmail.ualberta.ca> Message-ID: <47F99CF1.1020602@enthought.com> Rahul Garg wrote: > Note this message has been posted to numpy-discussion and python-dev. > Sorry for the multiple posting but I thought both python devs and > numpy users will be interested. If you believe your list should not > receive this email, let me know. Also I just wanted to introduce > myself since I may ask doubts about Python and Numpy internals from > time to time :) > Hey Rahul, This is a very interesting project. I've been interested in something like this for a while. I'd love to see more about what you are doing. Best regards, -Travis Oliphant From oliphant at enthought.com Mon Apr 7 00:19:58 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Sun, 06 Apr 2008 23:19:58 -0500 Subject: [Numpy-discussion] New project : Spyke python-to-C compiler In-Reply-To: <20080406184850.xiqr9t0hwgco004s@webmail.ualberta.ca> References: <20080406184850.xiqr9t0hwgco004s@webmail.ualberta.ca> Message-ID: <47F9A0EE.7050405@enthought.com> What will be the licensing of this project? Do you know yet? I have a couple of comments because I've been thinking along these lines. > What is Spyke? > In many performance critical projects, it is often necessary to > rewrite parts of the application in C. However writing C wrappers can > be time consuming. Spyke offers an alternative approach. You add > annotations to your Python code as strings. These strings are > discarded by the Python > interpreter but these are interpreted as types by Spyke compiler to > convert to C. > > Example : > > "int -> int" > def f(x): return 2*x > > In this case the Spyke compiler will consider the string "int -> int" > as a decalration that the function accepts int as parameter and > returns int. Spyke will then generate a C function and a wrapper > function. What about the use of decorators in this case? Also, it would be great to be able to create ufuncs (and general numpy funcs) using this approach. A decorator would work well here as well. > Where is Spyke? > Spyke will be available as a binary only release in a couple of weeks. > I intend to make it open source after a few months. > I'd like to encourage you to make it available as open source as early as possible. I think you are likely to get help in ways you didn't expect. People are used to reading code, so even an alpha project can get help early. In fact given that you are looking for help. I think this may be the best way to get it. If you need help getting set up in terms of hosting it somewhere, I can help you do that. > Spyke is written in Python and Java and should be platform independant. > I do intend to make the source open in a few months. Right now its > undergoing very rapid development and has negligible amounts of > documentation so the source code right now is pretty useless to anyone > else anyway. > > c) Strings as type declarations : Do you think I should use decorators > instead at least for function type declarations? > I think you should use decorators. That way you can work towards having the compiler "embedded" in the decorator and happen seamlessly without invoking a separte "program" (it just happens when the module is loaded -- a.l.a weave). Best regards, -Travis Oliphant From oliphant at enthought.com Mon Apr 7 00:24:17 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Sun, 06 Apr 2008 23:24:17 -0500 Subject: [Numpy-discussion] ma's stdu and varu In-Reply-To: References: Message-ID: <47F9A1F1.4050302@enthought.com> Anne Archibald wrote: > Hi, > > I was just going through tidying up the documentation for all the many > functions in numpy that compute standard deviations or variances (the > functions, the methods, the methods on matrices, the methods on > maskedarrays, all needed their docstrings updated in approximately the > same way). I noticed that maskedarrays seem to have the functions > "stdu" and "varu", for computing the unbiased standard deviation and > variance (where one divides by N-1 rather than N). These are now > available, I think, via std and var with ddof=1. More seriously, they > still provide the peculiar older definition of var, where > varu([1,1.j,-1,-1.j])==0. I don't use maskedarrays, and in fact I > haven't been keeping track of what's going on with the two different > implementations. So: what should be done about stdu and varu? Do the > two implementations both need their definitions of std and var > examined? > Yes, the masked arrays should be changed and stdu and varu eliminated. -Travis From aisaac at american.edu Mon Apr 7 00:38:39 2008 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 7 Apr 2008 00:38:39 -0400 Subject: [Numpy-discussion] matrix multiply In-Reply-To: References: <9457e7c80804051945u558edd93v12df417824e10877@mail.gmail.com> Message-ID: On Sun, 6 Apr 2008, Charles R Harris apparently wrote: > You mean as edges in a directed graph? Yes. Naturally a boolean matrix is not the most compact representation of a directed graph, especially a sparse one. However it can be convenient. If B is a boolean matrix such that Bij=1 if there is and edge from i to j, then B**2 has unit entries where there is a path of length 2 from i to j. The transitive closure is similarly easy to represent (as a matrix power). Cheers, Alan Isaac From charlesr.harris at gmail.com Mon Apr 7 01:21:22 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 6 Apr 2008 23:21:22 -0600 Subject: [Numpy-discussion] matrix multiply In-Reply-To: References: <9457e7c80804051945u558edd93v12df417824e10877@mail.gmail.com> Message-ID: On Sun, Apr 6, 2008 at 10:38 PM, Alan G Isaac wrote: > On Sun, 6 Apr 2008, Charles R Harris apparently wrote: > > You mean as edges in a directed graph? > > Yes. > > Naturally a boolean matrix is not the most compact > representation of a directed graph, especially a > sparse one. However it can be convenient. > I've had occasional thoughts of adding a "computer science" kit to scipy with equivalence relations, trees, tries, graphs, and other such things that come in handy for some sorts of problems. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From garg1 at ualberta.ca Mon Apr 7 02:19:03 2008 From: garg1 at ualberta.ca (Rahul Garg) Date: Mon, 07 Apr 2008 00:19:03 -0600 Subject: [Numpy-discussion] Numpy-discussion Digest, Vol 19, Issue 24 In-Reply-To: References: Message-ID: <20080407001903.o1rv31y4g00s0884@webmail.ualberta.ca> Quoting numpy-discussion-request at scipy.org: > What will be the licensing of this project? Do you know yet? I am thinking GPL for the compiler and LGPL for any runtime components which should be similar to GCC. Which version : Version 2 or version 3 of the license is undecided. Will also check with uni to see if they have any problems (they shouldnt). Also need to check with uni for hosting. I believe I will need to host on uni servers. > I have a couple of comments because I've been thinking along these lines. >> What is Spyke? >> In many performance critical projects, it is often necessary to >> rewrite parts of the application in C. However writing C wrappers can >> be time consuming. Spyke offers an alternative approach. You add >> annotations to your Python code as strings. These strings are >> discarded by the Python >> interpreter but these are interpreted as types by Spyke compiler to >> convert to C. >> >> Example : >> >> "int -> int" >> def f(x): return 2*x >> >> In this case the Spyke compiler will consider the string "int -> int" >> as a decalration that the function accepts int as parameter and >> returns int. Spyke will then generate a C function and a wrapper >> function. > What about the use of decorators in this case? I can certainly use decorators. Will implement this change soon. > Also, it would be great to be able to create ufuncs (and general numpy > funcs) using this approach. A decorator would work well here as well. >> Where is Spyke? >> Spyke will be available as a binary only release in a couple of weeks. >> I intend to make it open source after a few months. >> > I'd like to encourage you to make it available as open source as early > as possible. I think you are likely to get help in ways you didn't > expect. People are used to reading code, so even an alpha project can > get help early. In fact given that you are looking for help. I > think this may be the best way to get it. Ok .. I will release the source along with the binary. Need to sort some stuff out so might take a couple of weeks. Note that much of the compiler is (for better or worse) written in Java. The codebase isnt very OOP (full of static methods and looks more like garbage collected C) but not too complex either. I use cpython's "compiler" module to dump the AST into an intermediate file which is then parsed by the compiler in java. The compiler is using AST representation throughout. The compiler also depends upon the antlr java runtime. For hosting, I will probably get some space at the univ servers. I will try to get trac installed. I will release when all the following "work": a) Basic support for functions and classes. b) Keyword parameters not supported. c) Special methods not supported except __init__. d) __init__ is treated as constructor. Custom __new__ not supported. e) Nested functions may be broken. f) Functions will be divided into 2 types : static and dynamic. Static functions should not be redefined at runtime while dynamic functions can be redefined at runtime but will be more costly to call since I need to lookup the binding at each time. Also even though dynamic functions can be redefined its type signature should not change. If a static function calls another static function, then the compiler will try to insert a call to the C function instead of wrapped function thus bypassing the interpreter if possible. g) Compiled classes should not redefine methods at runtime. Will have an option to annotate classes as "final" meaning user shouldnt subclass it. For such classes, its easier to generate efficient code for attribute access. Also compiled classes shouldnt dynamically add/delete attributes. h) Users shouldnt subclass numpy array. i) For method calls on objects, mostly the code generated will just end up making a call to interpreter thus the performance in this case will not be particularly good currently. For ints, floats etc the equivalent C code will be generated so for these types the code should be fast enough. j) For indexing of numpy arrays, unsafe code is generated. I directly access the array without any index checking. k) Loops : This is the weakest point currently. I only allow for-loops over range() or xrange() allowing easy conversion to C. Cannot loop over elements of other lists or numpy arrays etc. l) Exec, eval, metaclasses, dynamic class creation, dynamic adding/deleting attributes etc not allowed inside typed code. m) A module cannot currently mix typed and untyped code. A module has to be completely typed/annotated or it should be left alone and not compiled. Also a typed module cannot have arbitrary executable code and should only consist of single statement variable declarations, function and class definitions. Of course rest of your application can be left untyped. In the future I will try allow mixing typed and untyped code in a module. n) Importing of other typed modules also mostly supported. o) Builtin functions : range and len mostly work. But cannot guarantee anything else. p) Lists, tuples and dictionaries can be used but need to be homogeneous. Not all methods supported yet. Moreover the code generated for these types mostly just generates function calls to python interpreter so this doesnt speed things up (yet). Not very sure how to handle subclasses of these and other builtin types. q) For function parameters of user-defined-class types, you can declare the parameter as "final". Example type can be declared as "final SomeClass" meaning that you will only pass SomeClass and not subclass. This allows the compiler (in the future) to generate better code for attribute access. Expected release date : Soon. Hopefully by 15th to 20th april. > If you need help getting set up in terms of hosting it somewhere, I can > help you do that. >> Spyke is written in Python and Java and should be platform independant. >> I do intend to make the source open in a few months. Right now its >> undergoing very rapid development and has negligible amounts of >> documentation so the source code right now is pretty useless to anyone >> else anyway. >> > >> c) Strings as type declarations : Do you think I should use decorators >> instead at least for function type declarations? >> > I think you should use decorators. That way you can work towards > having the compiler "embedded" in the decorator and happen seamlessly > without invoking a separte "program" (it just happens when the module is > loaded -- a.l.a weave). Well that can be done provided certain restrictions are met. One major problem is that it will make user applications dependent upon presence of a JVM since the compiler is in Java. Secondly seeing as much code as possible at compile time helps the compiler. For example, if you have a function G called inside function F, then the compiler needs to know the type of G which may not have been encountered yet. Also I am trying to work my way towards a whole program compiler since some of the optimizations that I want to research for my thesis are dependent on seeing the whole program. Those havent been implemented yet but will be in the future. Basically, if we call the compiler on one function at a time, the applicability of the compiler is reduced somewhat. So I will try to provide an option to invoke it at runtime too but it will have less features and in most cases less performance. Also, any thoughts on interfacing with existing C code? thanks, rahul From charlesr.harris at gmail.com Mon Apr 7 02:49:19 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 7 Apr 2008 00:49:19 -0600 Subject: [Numpy-discussion] Numpy-discussion Digest, Vol 19, Issue 24 In-Reply-To: <20080407001903.o1rv31y4g00s0884@webmail.ualberta.ca> References: <20080407001903.o1rv31y4g00s0884@webmail.ualberta.ca> Message-ID: On Mon, Apr 7, 2008 at 12:19 AM, Rahul Garg wrote: > Quoting numpy-discussion-request at scipy.org: > > > What will be the licensing of this project? Do you know yet? > > I am thinking GPL for the compiler and LGPL for any runtime components > which should be similar to GCC. Which version : Version 2 or version 3 > of the license is undecided. Will also check with uni to see if they > have any problems (they shouldnt). Also need to check with uni for > hosting. I believe I will need to host on uni servers. > Scipy and Numpy are BSD and don't include GPL components, so you might want to consider a more liberal license. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at carabos.com Mon Apr 7 03:54:04 2008 From: faltet at carabos.com (Francesc Altet) Date: Mon, 7 Apr 2008 09:54:04 +0200 Subject: [Numpy-discussion] Spyke python-to-C compiler Was: Numpy-discussion Digest, Vol 19, Issue 24 In-Reply-To: References: <20080407001903.o1rv31y4g00s0884@webmail.ualberta.ca> Message-ID: <200804070954.07757.faltet@carabos.com> A Monday 07 April 2008, Charles R Harris escrigu?: > On Mon, Apr 7, 2008 at 12:19 AM, Rahul Garg wrote: > > Quoting numpy-discussion-request at scipy.org: > > > What will be the licensing of this project? Do you know yet? > > > > I am thinking GPL for the compiler and LGPL for any runtime > > components which should be similar to GCC. Which version : Version > > 2 or version 3 of the license is undecided. Will also check with > > uni to see if they have any problems (they shouldnt). Also need to > > check with uni for hosting. I believe I will need to host on uni > > servers. > > Scipy and Numpy are BSD and don't include GPL components, so you > might want to consider a more liberal license. As I see it, and provided that parts of Spyke are written in Java, it would be very unlikely that this will ever be included in NumPy/SciPy. It looks like a compiler, so having the same licensing than GCC shouldn't bother the NumPy community, IMO. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From gael.varoquaux at normalesup.org Mon Apr 7 04:03:47 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 7 Apr 2008 10:03:47 +0200 Subject: [Numpy-discussion] New project : Spyke python-to-C compiler In-Reply-To: <47F9A0EE.7050405@enthought.com> References: <20080406184850.xiqr9t0hwgco004s@webmail.ualberta.ca> <47F9A0EE.7050405@enthought.com> Message-ID: <20080407080347.GB18555@phare.normalesup.org> Hi Rahul, Nice project. I think you are taking the right direction with type annotations. I you get this working and reliable, you will be much loved by the community. On Sun, Apr 06, 2008 at 11:19:58PM -0500, Travis E. Oliphant wrote: > > c) Strings as type declarations : Do you think I should use decorators > > instead at least for function type declarations? > I think you should use decorators. That way you can work towards > having the compiler "embedded" in the decorator and happen seamlessly > without invoking a separte "program" (it just happens when the module is > loaded -- a.l.a weave). +1. This is a very promising route. You can then choose exactly what you want to compile and what you want to keep pure Python (something similar to Cython, without the intermediate file). I would even stick with some decoration after Python3K, say "@compiled()", so that you keep this compilation on the fly that Travis is mentionning. Cheers, Ga?l From dagss at student.matnat.uio.no Mon Apr 7 05:31:12 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 7 Apr 2008 11:31:12 +0200 (CEST) Subject: [Numpy-discussion] New project : Spyke python-to-C compiler In-Reply-To: <20080406184850.xiqr9t0hwgco004s@webmail.ualberta.ca> References: <20080406184850.xiqr9t0hwgco004s@webmail.ualberta.ca> Message-ID: <50807.193.157.243.12.1207560672.squirrel@webmail.uio.no> > What is Spyke? > In many performance critical projects, it is often necessary to > rewrite parts of the application in C. However writing C wrappers can > be time consuming. Spyke offers an alternative approach. You add > annotations to your Python code as strings. These strings are > discarded by the Python > interpreter but these are interpreted as types by Spyke compiler to > convert to C. Have you had a look at Cython? http://cython.org. >From seeing what you write, it looks like we have almost exactly the same long-term goals; one could almost say that the two pieces of software will be complete duplicates in functionality. (Cython isn't "there" just yet though.) (Though as the saying goes, little duplication is normal (and perhaps wanted) for open source software.) Dag Sverre From dagss at student.matnat.uio.no Mon Apr 7 05:33:05 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 7 Apr 2008 11:33:05 +0200 (CEST) Subject: [Numpy-discussion] New project : Spyke python-to-C compiler In-Reply-To: <50807.193.157.243.12.1207560672.squirrel@webmail.uio.no> References: <20080406184850.xiqr9t0hwgco004s@webmail.ualberta.ca> <50807.193.157.243.12.1207560672.squirrel@webmail.uio.no> Message-ID: <50810.193.157.243.12.1207560785.squirrel@webmail.uio.no> > (Though as the saying goes, little duplication is normal (and perhaps > wanted) for open source software.) Sorry! I meant "a little", completely reversing the meaning of my sentence. Dag Sverre From meine at informatik.uni-hamburg.de Mon Apr 7 05:40:24 2008 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Mon, 7 Apr 2008 11:40:24 +0200 Subject: [Numpy-discussion] PCA on set of face images In-Reply-To: References: Message-ID: <200804071140.24932.meine@informatik.uni-hamburg.de> Am Dienstag, 11. M?rz 2008 00:24:04 schrieb David Bolme: > The steps you describe here are correct. I am putting together an > open source computer vision library based on numpy/scipy. It will > include an automatic PCA algorithm with face detection, eye detection, > PCA dimensionally reduction, and distance measurement. If you are > interested let me know and I will redouble my efforts to release the > code soon. That's interesting, we're also working on a computer vision module using NumPy (actually, a VIGRA <-> NumPy binding sort of), and there's scipy.ndimage, too. Maybe (part of) your code could be integrated into the latter? I am looking forward to it anyway. -- Ciao, / / /--/ / / ANS From michael.j.mclay at gmail.com Mon Apr 7 07:57:26 2008 From: michael.j.mclay at gmail.com (Michael McLay) Date: Mon, 7 Apr 2008 07:57:26 -0400 Subject: [Numpy-discussion] New project : Spyke python-to-C compiler In-Reply-To: <20080406184850.xiqr9t0hwgco004s@webmail.ualberta.ca> References: <20080406184850.xiqr9t0hwgco004s@webmail.ualberta.ca> Message-ID: <683f83340804070457n3200de6an7217616568ff6af5@mail.gmail.com> On Sun, Apr 6, 2008 at 8:48 PM, Rahul Garg wrote: > function. This > idea is directly copied from PLW (Python Language Wrapper) project. > Once Python3k arrives, much of these declarations will be moved to > function annotations and class decorators. Python 3k alpha4 is available. Why not skip the string based version and go directly to using the function annotation capability of 3.0. Your project could be a good test case for the new feature of the language. From meine at informatik.uni-hamburg.de Mon Apr 7 08:34:08 2008 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Mon, 7 Apr 2008 14:34:08 +0200 Subject: [Numpy-discussion] Ticket #605 Incorrect behavior of numpy.histogram In-Reply-To: References: Message-ID: <200804071434.09201.meine@informatik.uni-hamburg.de> Am Samstag, 05. April 2008 21:54:27 schrieb Anne Archibald: > There's also a fourth option - raise an exception if any points are > outside the range. +1 I think this should be the default. Otherwise, I tend towards "exclude", in order to have comparable bin sizes (when plotting, I always find peaks at the ends annoying); this could also be called "clip" BTW. But really, an exception would follow the Zen: "In the face of ambiguity, refuse the temptation to guess." And with a kwarg: "Explicit is better than implicit." histogram(a, arange(10), outliers = "clip") histogram(a, arange(10), outliers = "include") # better names? "include"->"accumulate"/"map to border"/"map"/"boundary" -- Ciao, / / /--/ / / ANS From aisaac at american.edu Mon Apr 7 09:42:48 2008 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 7 Apr 2008 09:42:48 -0400 Subject: [Numpy-discussion] Numpy-discussion Digest, Vol 19, Issue 24 In-Reply-To: <20080407001903.o1rv31y4g00s0884@webmail.ualberta.ca> References: <20080407001903.o1rv31y4g00s0884@webmail.ualberta.ca> Message-ID: On Mon, 07 Apr 2008, Rahul Garg apparently wrote: > I am thinking GPL for the compiler and LGPL for any > runtime components which should be similar to GCC. Which > version : Version 2 or version 3 of the license is > undecided. The author determines the license, of course. But don't forget to consider this one. Cheers, Alan Isaac From pgmdevlist at gmail.com Mon Apr 7 09:41:34 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 7 Apr 2008 09:41:34 -0400 Subject: [Numpy-discussion] varu and stdu In-Reply-To: <47F9A4BD.10800@enthought.com> References: <47F9A4BD.10800@enthought.com> Message-ID: <200804070941.35028.pgmdevlist@gmail.com> Anne, Travis, I have no problem to get rid of varu and stdu in MaskedArray: they were introduced for my own convenience, and they're indeed outdated with the introduction of the ddof parameters. I'll get rid of them next time I post something on the SVN. From david.huard at gmail.com Mon Apr 7 09:55:39 2008 From: david.huard at gmail.com (David Huard) Date: Mon, 7 Apr 2008 09:55:39 -0400 Subject: [Numpy-discussion] Ticket #605 Incorrect behavior of numpy.histogram In-Reply-To: <200804071434.09201.meine@informatik.uni-hamburg.de> References: <200804071434.09201.meine@informatik.uni-hamburg.de> Message-ID: <91cf711d0804070655s42a3d12cp320254cb79d57ba2@mail.gmail.com> +1 for an outlier keyword. Note, that this implies that when bins are passed explicitly, the edges are given (nbins+1), not simply the left edges (nbins). While we are refactoring histogram, I'd suggest adding an axis keyword. This is pretty straightforward to implement using the np.apply_along_axis function. Also, I noticed that current normalization is buggy for non-uniform bin sizes. if normed: db = bins[1] - bins[0] return 1.0/(a.size*db) * n, bins Finally, whatever option is chosen in the end, we should make sure it is consistent across all histogram functions. This may mean that we will also break the behavior of histogramdd and histogram2d. Bruce: I did some work over the weekend on the histogram function, including tests. If you want, I'll send that to you in the evening. David 2008/4/7, Hans Meine : > > Am Samstag, 05. April 2008 21:54:27 schrieb Anne Archibald: > > > There's also a fourth option - raise an exception if any points are > > outside the range. > > > +1 > > I think this should be the default. Otherwise, I tend towards "exclude", > in > order to have comparable bin sizes (when plotting, I always find peaks at > the > ends annoying); this could also be called "clip" BTW. > > But really, an exception would follow the Zen: "In the face of ambiguity, > refuse the temptation to guess." And with a kwarg: "Explicit is better > than > implicit." > > histogram(a, arange(10), outliers = "clip") > histogram(a, arange(10), outliers = "include") > # better names? "include"->"accumulate"/"map to border"/"map"/"boundary" > > > -- > Ciao, / / > /--/ > / / ANS > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lanhaiping at gmail.com Mon Apr 7 10:14:55 2008 From: lanhaiping at gmail.com (lan haiping) Date: Mon, 7 Apr 2008 22:14:55 +0800 Subject: [Numpy-discussion] Any multigrid package to recommend ? Message-ID: Dear Guys @list : I wanna do some application with mulgrid method for electrostatic problems, is there some python package available for my purpose ? Best, -- Hai-Ping Lan Department of Electronics , Peking University , Bejing, 100871 lanhaiping at gmail.com, hplan at pku.edu.cn -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Mon Apr 7 10:44:10 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 7 Apr 2008 09:44:10 -0500 Subject: [Numpy-discussion] Ticket #605 Incorrect behavior of numpy.histogram In-Reply-To: <91cf711d0804070655s42a3d12cp320254cb79d57ba2@mail.gmail.com> References: <200804071434.09201.meine@informatik.uni-hamburg.de> <91cf711d0804070655s42a3d12cp320254cb79d57ba2@mail.gmail.com> Message-ID: Hi, Thanks David for pointing the piece of information I forgot to add in my original email. -1 for 'raise an exception' because, as Dan points out, the problem stems from user providing bins. +1 for the outliers keyword. Should 'exclude' distinguish points that are too low and those that are too high? +1 for axis. Really I was only looking at seeing what it would take to close this bug, but I am willing to test any code. Thanks Bruce On Mon, Apr 7, 2008 at 8:55 AM, David Huard wrote: > +1 for an outlier keyword. Note, that this implies that when bins are passed > explicitly, the edges are given (nbins+1), not simply the left edges > (nbins). > > While we are refactoring histogram, I'd suggest adding an axis keyword. This > is pretty straightforward to implement using the np.apply_along_axis > function. > > Also, I noticed that current normalization is buggy for non-uniform bin > sizes. > if normed: > db = bins[1] - bins[0] > return 1.0/(a.size*db) * n, bins > > Finally, whatever option is chosen in the end, we should make sure it is > consistent across all histogram functions. This may mean that we will also > break the behavior of histogramdd and histogram2d. > > Bruce: I did some work over the weekend on the histogram function, including > tests. If you want, I'll send that to you in the evening. > > David > > > > > 2008/4/7, Hans Meine : > > > Am Samstag, 05. April 2008 21:54:27 schrieb Anne Archibald: > > > > > There's also a fourth option - raise an exception if any points are > > > outside the range. > > > > > > +1 > > > > I think this should be the default. Otherwise, I tend towards "exclude", > in > > order to have comparable bin sizes (when plotting, I always find peaks at > the > > ends annoying); this could also be called "clip" BTW. > > > > But really, an exception would follow the Zen: "In the face of ambiguity, > > refuse the temptation to guess." And with a kwarg: "Explicit is better > than > > implicit." > > > > histogram(a, arange(10), outliers = "clip") > > histogram(a, arange(10), outliers = "include") > > # better names? "include"->"accumulate"/"map to border"/"map"/"boundary" > > > > > > -- > > Ciao, / / > > /--/ > > / / ANS > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From david.huard at gmail.com Mon Apr 7 10:51:22 2008 From: david.huard at gmail.com (David Huard) Date: Mon, 7 Apr 2008 10:51:22 -0400 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: References: Message-ID: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> 2008/4/4, Joe Harrington : > > import numpy as N > import numpy.math as N.M > import numpy.trig as N.T > import numpy.stat as N.S I don't think the issue is whether to put everything in the base namespace // everything in individual namespace, but rather to find an optimal and intuitive mix between the two. For instance, the io functions would be easier to find by typing np.io.loadtxt than by sifting through the 500+ items of the base namespace. The stats functions could equally well be in a separate namespace, given that the most used are implemented as array methods. I think this would allow numpy to grow more gracefully. As for the financial functions, being specific to a discipline, I think they rather belongs with scipy. The numpy namespace will quickly become a mess if we add np.geology, np.biology, np.material, etc. Of course, this raises the problem of distributing scipy, and here is a suggestion: Change the structure of scipy so that it looks like the scikits: scipy/ sparse/ cluster/ financial/ ... fftpack/ setup.py scipy/ __init__.py fftpack/ The advantage is that each subpackage can be installed independently of the others. For distribution, we could lump all the pure python or easy to compile packages into scipy.common, and distribute the other packages such as sparse and fftpack independently. My feeling is that such a lighter structure would encourage projects with large code base to join the scipy community. It would also allow folks with 56k modems to download only what they need. David -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Mon Apr 7 11:20:47 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 7 Apr 2008 17:20:47 +0200 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> Message-ID: <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> On 07/04/2008, David Huard wrote: > > 2008/4/4, Joe Harrington : > > import numpy as N > > import numpy.math as N.M > > import numpy.trig as N.T > > import numpy.stat as N.S > > > I don't think the issue is whether to put everything in the base namespace > // everything in individual namespace, but rather to find an optimal and > intuitive mix between the two. For instance, the io functions would be > easier to find by typing np.io.loadtxt than by sifting through the 500+ > items of the base namespace. The stats functions could equally well be in a > separate namespace, given that the most used are implemented as array > methods. I think this would allow numpy to grow more gracefully. I agree, and I think we can come to some compromise -- maybe a numpy.all namespace, that simply imports all the other subpackages. Regards St?fan From bioinformed at gmail.com Mon Apr 7 11:43:30 2008 From: bioinformed at gmail.com (Kevin Jacobs ) Date: Mon, 7 Apr 2008 11:43:30 -0400 Subject: [Numpy-discussion] varu and stdu In-Reply-To: <200804070941.35028.pgmdevlist@gmail.com> References: <47F9A4BD.10800@enthought.com> <200804070941.35028.pgmdevlist@gmail.com> Message-ID: <2e1434c10804070843s290b1a01o954005e8c3a9a3f2@mail.gmail.com> I know I'm off topic and maybe a day late, but I'm pained by the naming of ddof. It is simply not intuitive for anyone other than the person who thought it up (and from my recollection, maybe not even for him). For one, most stats folk use 'df' as the abbreviation for 'degrees of freedom'. Secondly, the we tend to think of the constant bias adjustment as an ~adjustment~ of the sample size or df. So 'df_adjust=0' or 'sample_adjust=0' will resonate much more. The other issue is to clearly describe if 'N-1' is obtained by setting the adjustment (whatever it is called) to +1 or -1. There is a reason why most stats packages have different functions or take a parameter to indicate 'sample' versus 'population' variance calculation. Though don't take this as a recommendation to use var and varu -- rather I'm merely pointing out that var(X, vardef='sample') is an option (using SAS's PROC MEANS parameter name as an arbitrary example). In the extremely rare cases I need any other denominator, I'm fine with multiplying by var(x)*n/(n-adjust). -Kevin On Mon, Apr 7, 2008 at 9:41 AM, Pierre GM wrote: > Anne, Travis, > I have no problem to get rid of varu and stdu in MaskedArray: they were > introduced for my own convenience, and they're indeed outdated with the > introduction of the ddof parameters. I'll get rid of them next time I post > something on the SVN. > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Mon Apr 7 11:49:05 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 7 Apr 2008 17:49:05 +0200 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> Message-ID: <20080407154905.GI9851@phare.normalesup.org> On Mon, Apr 07, 2008 at 05:20:47PM +0200, St?fan van der Walt wrote: > I agree, and I think we can come to some compromise -- maybe a > numpy.all namespace, that simply imports all the other subpackages. For the beginner, "from numpy.all import *" is more confusing than "from numpy import *" (which is already confusing). I know namespace are good things, but the beginner struggles with them. This is why I used the "import *" in my above examples. My 2 cents, Ga?l From wfspotz at sandia.gov Mon Apr 7 11:55:47 2008 From: wfspotz at sandia.gov (Bill Spotz) Date: Mon, 7 Apr 2008 09:55:47 -0600 Subject: [Numpy-discussion] Any multigrid package to recommend ? In-Reply-To: References: Message-ID: I am the lead developer of PyTrilinos, a python interface to the Trilinos project: http://trilinos.sandia.gov. Trilinos has many packages related to solvers, including ML, the multilevel preconditioning package. There may be a little bit of a learning curve, but there are example scripts to look at. I also built in quite a bit of compatibility with numpy. On Apr 7, 2008, at 8:14 AM, lan haiping wrote: > Dear Guys @list : > I wanna do some application with mulgrid method for electrostatic > problems, > is there some python package available for my purpose ? ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From matthieu.brucher at gmail.com Mon Apr 7 12:04:09 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 7 Apr 2008 18:04:09 +0200 Subject: [Numpy-discussion] Interaction between Numpy and the nose framework (was : packaging scipy) Message-ID: BTW, I stumbled on something strange with the nose framework. If you from numpy.testing import * in a test file, the nose framework will try to test the testing module by calling every test* method. I just mention it there because I think I'm not the only one to do this for set_package_path, assert_equal, ... Matthieu 2008/4/7, Gael Varoquaux : > > On Mon, Apr 07, 2008 at 05:20:47PM +0200, St?fan van der Walt wrote: > > I agree, and I think we can come to some compromise -- maybe a > > numpy.all namespace, that simply imports all the other subpackages. > > > For the beginner, "from numpy.all import *" is more confusing than "from > numpy import *" (which is already confusing). > > I know namespace are good things, but the beginner struggles with them. > This is why I used the "import *" in my above examples. > > My 2 cents, > > Ga?l > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From lanhaiping at gmail.com Mon Apr 7 12:08:05 2008 From: lanhaiping at gmail.com (lan haiping) Date: Tue, 8 Apr 2008 00:08:05 +0800 Subject: [Numpy-discussion] Any multigrid package to recommend ? In-Reply-To: References: Message-ID: Thank you, Bill. I will try to fill that curve . On Mon, Apr 7, 2008 at 11:55 PM, Bill Spotz wrote: > I am the lead developer of PyTrilinos, a python interface to the > Trilinos project: http://trilinos.sandia.gov. > > Trilinos has many packages related to solvers, including ML, the > multilevel preconditioning package. There may be a little bit of a > learning curve, but there are example scripts to look at. I also > built in quite a bit of compatibility with numpy. > > On Apr 7, 2008, at 8:14 AM, lan haiping wrote: > > > Dear Guys @list : > > I wanna do some application with mulgrid method for electrostatic > > problems, > > is there some python package available for my purpose ? > > ** Bill Spotz ** > ** Sandia National Laboratories Voice: (505)845-0170 ** > ** P.O. Box 5800 Fax: (505)284-0154 ** > ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** > > > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- Hai-Ping Lan Department of Electronics , Peking University , Bejing, 100871 lanhaiping at gmail.com, hplan at pku.edu.cn -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Mon Apr 7 12:13:50 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 7 Apr 2008 18:13:50 +0200 Subject: [Numpy-discussion] Any multigrid package to recommend ? In-Reply-To: References: Message-ID: <9457e7c80804070913wad84021wd67345e5215b659a@mail.gmail.com> On 07/04/2008, lan haiping wrote: > Dear Guys @list : > I wanna do some application with mulgrid method for electrostatic > problems, > is there some python package available for my purpose ? Nathan bell has a mesmerisingly beautiful webpage at http://graphics.cs.uiuc.edu/~wnbell/ At the bottom, he mentions PyAMG. Regards St?fan From stefan at sun.ac.za Mon Apr 7 12:22:28 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 7 Apr 2008 18:22:28 +0200 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <20080407154905.GI9851@phare.normalesup.org> References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> Message-ID: <9457e7c80804070922r67ae453cy4bf3473545c05692@mail.gmail.com> On 07/04/2008, Gael Varoquaux wrote: > On Mon, Apr 07, 2008 at 05:20:47PM +0200, St?fan van der Walt wrote: > > I agree, and I think we can come to some compromise -- maybe a > > numpy.all namespace, that simply imports all the other subpackages. > > > For the beginner, "from numpy.all import *" is more confusing than "from > numpy import *" (which is already confusing). > > I know namespace are good things, but the beginner struggles with them. > This is why I used the "import *" in my above examples. You're only a beginner for a short while, and after that the lack of namespaces really start to bite. I am all in favour of catering for those who are busy learning numpy, but should we do that at the cost of our advanced users? There must be a way to make both groups happy -- any suggestions? Regards St?fan From stefan at sun.ac.za Mon Apr 7 12:25:09 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 7 Apr 2008 18:25:09 +0200 Subject: [Numpy-discussion] Interaction between Numpy and the nose framework (was : packaging scipy) In-Reply-To: References: Message-ID: <9457e7c80804070925h22c30887q59558fe673b9289f@mail.gmail.com> On 07/04/2008, Matthieu Brucher wrote: > BTW, I stumbled on something strange with the nose framework. If you from > numpy.testing import * in a test file, the nose framework will try to test > the testing module by calling every test* method. > > I just mention it there because I think I'm not the only one to do this for > set_package_path, assert_equal, ... I've noticed that behaviour, too. Note, however, that you do not need to use set_package_path and friends with nose; you can instead do a fully qualified import: from numpy.submod.mod import foo Regards St?fan From matthew.brett at gmail.com Mon Apr 7 12:38:25 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 7 Apr 2008 16:38:25 +0000 Subject: [Numpy-discussion] Interaction between Numpy and the nose framework (was : packaging scipy) In-Reply-To: <9457e7c80804070925h22c30887q59558fe673b9289f@mail.gmail.com> References: <9457e7c80804070925h22c30887q59558fe673b9289f@mail.gmail.com> Message-ID: <1e2af89e0804070938k6e9f6dc7qe6b1ef048016be88@mail.gmail.com> Hi, On Mon, Apr 7, 2008 at 4:25 PM, St?fan van der Walt wrote: > On 07/04/2008, Matthieu Brucher wrote: > > BTW, I stumbled on something strange with the nose framework. If you from > > numpy.testing import * in a test file, the nose framework will try to test > > the testing module by calling every test* method. > > > > I just mention it there because I think I'm not the only one to do this for > > set_package_path, assert_equal, ... > > I've noticed that behaviour, too. Note, however, that you do not need > to use set_package_path and friends with nose; you can instead do a > fully qualified import: > > from numpy.submod.mod import foo Actually, it was intentional to make the scipy.testing * space a more limited version of the numpy testing space - in particular, set_package_path was often being used unnecessarily (by me among others) and was clearly leading to confusion. You do however have assert_equal and friends with from scipy.testing import *, so I'd strongly recommend you use that in preference to the numpy.testing import within scipy. Best, Matthew From gael.varoquaux at normalesup.org Mon Apr 7 12:57:40 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 7 Apr 2008 18:57:40 +0200 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <9457e7c80804070922r67ae453cy4bf3473545c05692@mail.gmail.com> References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <9457e7c80804070922r67ae453cy4bf3473545c05692@mail.gmail.com> Message-ID: <20080407165740.GK9851@phare.normalesup.org> On Mon, Apr 07, 2008 at 06:22:28PM +0200, St?fan van der Walt wrote: > You're only a beginner for a short while, and after that the lack of > namespaces really start to bite. I am all in favour of catering for > those who are busy learning numpy, but should we do that at the cost > of our advanced users? I agree with you. However lowering the bar is a good thing. > There must be a way to make both groups happy -- any suggestions? Hum, I am still trying to find ideas. If only "from foo.bar import baz" didn't import all what is in foo.__init__ !!! By the way, the standard solution to this problem is to use a module called "api" and not "all". That's what people have been doing to solve the problem we are faced with. If we are going to go this way, I suggest we stick to the "api" convention (eventhought it sucks). Cheers, Ga?l From tim.hochberg at ieee.org Mon Apr 7 13:16:22 2008 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Mon, 7 Apr 2008 10:16:22 -0700 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <20080407165740.GK9851@phare.normalesup.org> References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <9457e7c80804070922r67ae453cy4bf3473545c05692@mail.gmail.com> <20080407165740.GK9851@phare.normalesup.org> Message-ID: On Mon, Apr 7, 2008 at 9:57 AM, Gael Varoquaux < gael.varoquaux at normalesup.org> wrote: > On Mon, Apr 07, 2008 at 06:22:28PM +0200, St?fan van der Walt wrote: > > You're only a beginner for a short while, and after that the lack of > > namespaces really start to bite. I am all in favour of catering for > > those who are busy learning numpy, but should we do that at the cost > > of our advanced users? > > I agree with you. However lowering the bar is a good thing. > > > There must be a way to make both groups happy -- any suggestions? > > Hum, I am still trying to find ideas. If only "from foo.bar import baz" > didn't import all what is in foo.__init__ !!! By the way, the standard > solution to this problem is to use a module called "api" and not "all". > That's what people have been doing to solve the problem we are faced > with. If we are going to go this way, I suggest we stick to the "api" > convention (eventhought it sucks). I prefer 'all' for this since it has the correct meaning. 'api' assuming that one can remember what it means doesn't fit. The 'all' module would not contain the api, at least not the preferred api (in my book at least), but it would contain everything. If "from numpy.all import *" is really too complicated, which although possible, seems a little disheartening, I suspect it would be easy enough to have a separate module that pulled everything in so that you could use "from big_numpy import *". Or, to preserve backward compatibility, make numpy the unsplit namespace and expose the split namespace under a different name, let's say 'np' because that's what I already use as a numpy abbreviation. Then "import np" would get you just the core np functions (which I imagine we could argue about endlessly) and the various subpackages would be imported separately. 'np' is 'numpy' with some stuff removed: get it? OK, so that's a weak joke, sorry. -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From timmichelsen at gmx-topmail.de Mon Apr 7 13:22:58 2008 From: timmichelsen at gmx-topmail.de (Tim Michelsen) Date: Mon, 07 Apr 2008 19:22:58 +0200 Subject: [Numpy-discussion] Scipy and Numpy at Nabble Forums Message-ID: Hello, I registered the Scipy and Numpy mailing lists at the Nabble Web Forums: Scipy http://www.nabble.com/Scipy-User-f33045.html Numpy http://www.nabble.com/Numpy-discussion-f33046.html I still have to import the old emails from the archives. Kind regards and happy communicating, Tim Michelsen From steve at shrogers.com Mon Apr 7 13:26:44 2008 From: steve at shrogers.com (Steven H. Rogers) Date: Mon, 7 Apr 2008 11:26:44 -0600 (MDT) Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <9457e7c80804070922r67ae453cy4bf3473545c05692@mail.gmail.com> <20080407165740.GK9851@phare.normalesup.org> Message-ID: <36510.192.55.12.36.1207589204.squirrel@mail2.webfaction.com> On Mon, April 7, 2008 11:16, Timothy Hochberg wrote: > > If "from numpy.all import *" is really too complicated, which although > possible, seems a little disheartening, I suspect it would be easy enough > to > have a separate module that pulled everything in so that you could use > "from > big_numpy import *". Or, to preserve backward compatibility, make numpy > the > unsplit namespace and expose the split namespace under a different name, > let's say 'np' because that's what I already use as a numpy abbreviation. > Then "import np" would get you just the core np functions (which I imagine > we could argue about endlessly) and the various subpackages would be > imported separately. 'np' is 'numpy' with some stuff removed: get it? OK, > so > that's a weak joke, sorry. > May not be the epitome of wit, but not bad. +1 for np package being a minimalist numpy and numpy being bigger. # Steve From gael.varoquaux at normalesup.org Mon Apr 7 13:30:48 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 7 Apr 2008 19:30:48 +0200 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <9457e7c80804070922r67ae453cy4bf3473545c05692@mail.gmail.com> <20080407165740.GK9851@phare.normalesup.org> Message-ID: <20080407173048.GM9851@phare.normalesup.org> On Mon, Apr 07, 2008 at 10:16:22AM -0700, Timothy Hochberg wrote: > I prefer 'all' for this since it has the correct meaning. 'api' assuming > that one can remember what it means doesn't fit. The 'all' module would > not contain the api, at least not the preferred api (in my book at least), > but it would contain everything. Sure, but everybody does it different. Convention are important, especially in coding. See http://ivory.idyll.org/blog/sep-07/not-sucking for a good argumentation about the point. I agree 100% with the author. Especially the conclusion. > If "from numpy.all import *" is really too complicated, which although > possible, seems a little disheartening, How much have you tried forcing Python on people who don't care at all about computers. In my work we spend maybe 2% of our time dealing with computers, and the rest struggling with electronics, optics, lasers, mechanical design... People don't want to have to learn _anything_ about computers. I am not saying they are right, I am however saying that we need to provide easy entry point, from where they can evolve and learn. > I suspect it would be easy enough to have a separate module that > pulled everything in so that you could use "from big_numpy import > *". Or, to preserve backward compatibility, make numpy the unsplit > namespace and expose the split namespace under a different name, > let's say 'np' because that's what I already use as a numpy > abbreviation. That's the only solution I see wich would make everybody happy. IMHO the pylab option is quite nice: matplotlib is nice and modular, but pylab has it all. Use whichever you want. Now the difficulty is to find a good name for the latter module/namespace. Cheers, Ga?l From tim.hochberg at ieee.org Mon Apr 7 14:02:12 2008 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Mon, 7 Apr 2008 11:02:12 -0700 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <20080407173048.GM9851@phare.normalesup.org> References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <9457e7c80804070922r67ae453cy4bf3473545c05692@mail.gmail.com> <20080407165740.GK9851@phare.normalesup.org> <20080407173048.GM9851@phare.normalesup.org> Message-ID: On Mon, Apr 7, 2008 at 10:30 AM, Gael Varoquaux < gael.varoquaux at normalesup.org> wrote: > On Mon, Apr 07, 2008 at 10:16:22AM -0700, Timothy Hochberg wrote: > > I prefer 'all' for this since it has the correct meaning. 'api' > assuming > > that one can remember what it means doesn't fit. The 'all' module > would > > not contain the api, at least not the preferred api (in my book at > least), > > but it would contain everything. > > Sure, but everybody does it different. Convention are important, > especially in coding. See http://ivory.idyll.org/blog/sep-07/not-sucking > for a good argumentation about the point. I agree 100% with the author. > Especially the conclusion. This is all moot since we agree below, but I don't see that the page your reference, which seem uncontroversial on a casual reading, is all that relevant. It's not that I disagree, that following convention is important where reasonable, I just don't see that this is a case where there is a convention to follow. I'm at a bit of a disadvantage since the convention in question hasn't penetrated the parts of of Python land that I inhabit (which could either imply something about my experience or about the universality of the 'api' convention, take your pick). However, I think that I vaguely recall it from back in my C-programming days, and as I recall/infer/guess the 'api' namespace is how you are supposed to use the functions in question, while the actual modules are split out for implementation purposes only. However, in numpy, that is not the case. Any splitting of the namespace would be to support a better, more organized interface, not as an implementation details. So. referring to the collected, flat namespace as 'api' would be confusing at best since it would imply that was the official approved way to access the python functions rather than one of two equivalent apis, where the flat namespace is provided primarily for beginners. > > > > If "from numpy.all import *" is really too complicated, which > although > > possible, seems a little disheartening, > > How much have you tried forcing Python on people who don't care at all > about computers. Almost none, thankfully. > In my work we spend maybe 2% of our time dealing with > computers, and the rest struggling with electronics, optics, lasers, > mechanical design... People don't want to have to learn _anything_ about > computers. I am not saying they are right, I am however saying that we > need to provide easy entry point, from where they can evolve and learn. > > > > I suspect it would be easy enough to have a separate module that > > pulled everything in so that you could use "from big_numpy import > > *". Or, to preserve backward compatibility, make numpy the unsplit > > namespace and expose the split namespace under a different name, > > let's say 'np' because that's what I already use as a numpy > > abbreviation. > > That's the only solution I see wich would make everybody happy. IMHO the > pylab option is quite nice: matplotlib is nice and modular, but pylab has > it all. Use whichever you want. > > Now the difficulty is to find a good name for the latter > module/namespace. > > Cheers, > > Ga?l > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Mon Apr 7 14:19:31 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 07 Apr 2008 13:19:31 -0500 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <36510.192.55.12.36.1207589204.squirrel@mail2.webfaction.com> References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <9457e7c80804070922r67ae453cy4bf3473545c05692@mail.gmail.com> <20080407165740.GK9851@phare.normalesup.org> <36510.192.55.12.36.1207589204.squirrel@mail2.webfaction.com> Message-ID: <47FA65B3.4000307@gmail.com> Steven H. Rogers wrote: > On Mon, April 7, 2008 11:16, Timothy Hochberg wrote: > >> If "from numpy.all import *" is really too complicated, which although >> possible, seems a little disheartening, I suspect it would be easy enough >> to >> have a separate module that pulled everything in so that you could use >> "from >> big_numpy import *". Or, to preserve backward compatibility, make numpy >> the >> unsplit namespace and expose the split namespace under a different name, >> let's say 'np' because that's what I already use as a numpy abbreviation. >> Then "import np" would get you just the core np functions (which I imagine >> we could argue about endlessly) and the various subpackages would be >> imported separately. 'np' is 'numpy' with some stuff removed: get it? OK, >> so >> that's a weak joke, sorry. >> >> > May not be the epitome of wit, but not bad. > > +1 for np package being a minimalist numpy and numpy being bigger. > > # Steve > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > Hi, I think that splitting the NumPy namespace should not happen within a major release series because it would cause too many breakages. Rather it should be in a forthcoming release like the 2.0 series where it may be very feasible to have a true core functionality (NumPy), extended functionality (SciPy) and specific applications (Scikits). At the same time, Python 3K would be fully supported because it will break lots of code. It is really nice to think about having NumPy Core, NumPy Full, NumPyKits, SciPy Core, SciPy Full and SciPyKits. But splitting namespaces like core and complete brings into the problem of conflicts and how to resolve them. Regardless of the content of each, I have the suspicion that most people would just take the full versions of each eventhough most of them only use a very small fraction of NumPy (just probably different amongst users). In the past, the real distinction between Numpy and SciPy for me was the requirement of having a full Lapack installation and a Fortran compiler for SciPy. This made the current scheme very usable especially the frustrations of getting SciPy to install. Fortunately Linux and GCC Fortran has really developed over the years that these are not as big as they were although these still cause issues (especially if packages are broken). However, it remains a big concern if everything has to be built from scratch (perhaps with different compilers) or if developers continue to tell users to get the latest version from svn (problem if you used a precompiled version). Regards Bruce From robert.kern at gmail.com Mon Apr 7 14:26:52 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 7 Apr 2008 11:26:52 -0700 Subject: [Numpy-discussion] Spyke python-to-C compiler Was: Numpy-discussion Digest, Vol 19, Issue 24 In-Reply-To: <200804070954.07757.faltet@carabos.com> References: <20080407001903.o1rv31y4g00s0884@webmail.ualberta.ca> <200804070954.07757.faltet@carabos.com> Message-ID: <3d375d730804071126v31b7cd41s684ed205b33b87d8@mail.gmail.com> On Mon, Apr 7, 2008 at 12:54 AM, Francesc Altet wrote: > A Monday 07 April 2008, Charles R Harris escrigu?: > > On Mon, Apr 7, 2008 at 12:19 AM, Rahul Garg wrote: > > > Quoting numpy-discussion-request at scipy.org: > > > > What will be the licensing of this project? Do you know yet? > > > > > > I am thinking GPL for the compiler and LGPL for any runtime > > > components which should be similar to GCC. Which version : Version > > > 2 or version 3 of the license is undecided. Will also check with > > > uni to see if they have any problems (they shouldnt). Also need to > > > check with uni for hosting. I believe I will need to host on uni > > > servers. > > > > Scipy and Numpy are BSD and don't include GPL components, so you > > might want to consider a more liberal license. > > As I see it, and provided that parts of Spyke are written in Java, it > would be very unlikely that this will ever be included in NumPy/SciPy. > It looks like a compiler, so having the same licensing than GCC > shouldn't bother the NumPy community, IMO. If the runtime isn't BSD, I won't bother using it. I can deal with an LGPLed glibc because I never have to bother distributing it. I would have to distribute the Spyke runtime. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Mon Apr 7 14:29:41 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 7 Apr 2008 11:29:41 -0700 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <47FA65B3.4000307@gmail.com> References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <9457e7c80804070922r67ae453cy4bf3473545c05692@mail.gmail.com> <20080407165740.GK9851@phare.normalesup.org> <36510.192.55.12.36.1207589204.squirrel@mail2.webfaction.com> <47FA65B3.4000307@gmail.com> Message-ID: <3d375d730804071129k19cab8efn5a22a6127d6aa424@mail.gmail.com> On Mon, Apr 7, 2008 at 11:19 AM, Bruce Southey wrote: > Hi, > I think that splitting the NumPy namespace should not happen within a > major release series because it would cause too many breakages. Rather > it should be in a forthcoming release like the 2.0 series where it may > be very feasible to have a true core functionality (NumPy), extended > functionality (SciPy) and specific applications (Scikits). At the same > time, Python 3K would be fully supported because it will break lots of > code. I would prefer not to do it at all. We've just gotten people moved over from Numeric; I'd hate to break their trust again. > It is really nice to think about having NumPy Core, NumPy Full, > NumPyKits, SciPy Core, SciPy Full and SciPyKits. Really? It gives me the shivers, frankly. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From gael.varoquaux at normalesup.org Mon Apr 7 14:29:54 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 7 Apr 2008 20:29:54 +0200 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <3d375d730804071129k19cab8efn5a22a6127d6aa424@mail.gmail.com> References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <9457e7c80804070922r67ae453cy4bf3473545c05692@mail.gmail.com> <20080407165740.GK9851@phare.normalesup.org> <36510.192.55.12.36.1207589204.squirrel@mail2.webfaction.com> <47FA65B3.4000307@gmail.com> <3d375d730804071129k19cab8efn5a22a6127d6aa424@mail.gmail.com> Message-ID: <20080407182954.GO9851@phare.normalesup.org> On Mon, Apr 07, 2008 at 11:29:41AM -0700, Robert Kern wrote: > I would prefer not to do it at all. We've just gotten people moved > over from Numeric; I'd hate to break their trust again. +1. Ga?l From millman at berkeley.edu Mon Apr 7 14:46:34 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 7 Apr 2008 11:46:34 -0700 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <20080407182954.GO9851@phare.normalesup.org> References: <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <9457e7c80804070922r67ae453cy4bf3473545c05692@mail.gmail.com> <20080407165740.GK9851@phare.normalesup.org> <36510.192.55.12.36.1207589204.squirrel@mail2.webfaction.com> <47FA65B3.4000307@gmail.com> <3d375d730804071129k19cab8efn5a22a6127d6aa424@mail.gmail.com> <20080407182954.GO9851@phare.normalesup.org> Message-ID: On Mon, Apr 7, 2008 at 11:29 AM, Gael Varoquaux wrote: > On Mon, Apr 07, 2008 at 11:29:41AM -0700, Robert Kern wrote: > > I would prefer not to do it at all. We've just gotten people moved > > over from Numeric; I'd hate to break their trust again. +1 I also think we have a big enough proliferation of namespaces (with numpy, scipy, and scikits). -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From lists at informa.tiker.net Mon Apr 7 14:47:29 2008 From: lists at informa.tiker.net (Andreas =?iso-8859-1?q?Kl=F6ckner?=) Date: Mon, 7 Apr 2008 14:47:29 -0400 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <3d375d730804071129k19cab8efn5a22a6127d6aa424@mail.gmail.com> References: <47FA65B3.4000307@gmail.com> <3d375d730804071129k19cab8efn5a22a6127d6aa424@mail.gmail.com> Message-ID: <200804071447.30956.lists@informa.tiker.net> On Montag 07 April 2008, Robert Kern wrote: > I would prefer not to do it at all. We've just gotten people moved > over from Numeric; I'd hate to break their trust again. +1. IMO, numpy has arrived at a state where there's just enough namespace clutter to allow most use cases to get by without importing much sub-namespace junk, and I think that's a good place to be (and to stay). For now, I'd be very careful about adding more. > > It is really nice to think about having NumPy Core, NumPy Full, > > NumPyKits, SciPy Core, SciPy Full and SciPyKits. > > Really? It gives me the shivers, frankly. Couldn't agree more. Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From millman at berkeley.edu Mon Apr 7 14:50:02 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 7 Apr 2008 11:50:02 -0700 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <9457e7c80804070922r67ae453cy4bf3473545c05692@mail.gmail.com> <20080407165740.GK9851@phare.normalesup.org> <20080407173048.GM9851@phare.normalesup.org> Message-ID: On Mon, Apr 7, 2008 at 11:02 AM, Timothy Hochberg wrote: > I'm at a bit of a disadvantage since the convention in question hasn't > penetrated the parts of of Python land that I inhabit (which could either > imply something about my experience or about the universality of the 'api' > convention, take your pick). However, I think that I vaguely recall it from > back in my C-programming days, and as I recall/infer/guess the 'api' > namespace is how you are supposed to use the functions in question, while > the actual modules are split out for implementation purposes only. I haven't been following how many projects have been using the api.py convention, but when I last looked about a year ago there was enthought, peak, zope, trac, etc. See this note for a bit more information: http://neuroimaging.scipy.org/neuroimaging/ni/ticket/86 Hope this helps, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From nadavh at visionsense.com Mon Apr 7 15:21:14 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Mon, 7 Apr 2008 22:21:14 +0300 Subject: [Numpy-discussion] site.cfg doesnt function? Message-ID: <710F2847B0018641891D9A21602763600B6F61@ex3.envision.co.il> I checked out numpy from svn few hours ago, and created a site.cfg following site.cfg.example. During the build process I am getting an warning that unoptimized lapack in being used. Machine: dual core amd64 running gentoo linux. Relevant packages: python 2.5.1, blas-atlas-3.8.0, lapack-atlas-3.8.0 # site.cfg [ALL] library_dirs = /usr/lib64/lapack/atlas:/usr/lib64/blas/threaded-atlas:/usr/lib include_dirs = /usr/include/atlas:/usr/include [blas_opt] library_dirs = /usr/lib64/blas/threaded-atlas:/usr/lib64 libraries = blas, cblas, atlas [lapack_opt] library_dirs = /usr/lib64/lapack/atlas:/usr/lib64 libraries = lapack, blas, cblas, atlas [fftw] libraries = fftw3 I added the following print lines in system_info class: def __init__ (self, default_lib_dirs=default_lib_dirs, default_include_dirs=default_include_dirs, verbosity = 1, ): print '\n\n=====================================' print ' class: ',self.__class__ print ' libs: ', default_lib_dirs print ' includes: ', default_include_dirs print '=====================================\n\n' A partial dump out of "python setup.py build": Running from numpy source directory. F2PY Version 2_4971 ===================================== class: numpy.distutils.system_info.blas_opt_info libs: ['/usr/local/lib', '/usr/lib'] includes: ['/usr/local/include', '/usr/include'] ===================================== blas_opt_info: ===================================== class: numpy.distutils.system_info.blas_mkl_info libs: ['/usr/local/lib', '/usr/lib'] includes: ['/usr/local/include', '/usr/include'] ===================================== blas_mkl_info: libraries mkl,vml,guide not found in /usr/lib libraries mkl,vml,guide not found in /usr/local/lib NOT AVAILABLE ===================================== class: numpy.distutils.system_info.atlas_blas_threads_info libs: ['/usr/local/lib', '/usr/lib'] includes: ['/usr/local/include', '/usr/include'] ===================================== atlas_blas_threads_info: Setting PTATLAS=ATLAS NOT AVAILABLE ===================================== class: numpy.distutils.system_info.atlas_blas_info libs: ['/usr/local/lib', '/usr/lib'] includes: ['/usr/local/include', '/usr/include'] ===================================== atlas_blas_info: NOT AVAILABLE /home/nadav/numpy/numpy/distutils/system_info.py:1345: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) ===================================== class: numpy.distutils.system_info.blas_info libs: ['/usr/local/lib', '/usr/lib'] includes: ['/usr/local/include', '/usr/include'] ===================================== blas_info: NOT AVAILABLE /home/nadav/numpy/numpy/distutils/system_info.py:1354: UserWarning: Blas (http://www.netlib.org/blas/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [blas]) or by setting the BLAS environment variable. warnings.warn(BlasNotFoundError.__doc__) ===================================== class: numpy.distutils.system_info.blas_src_info libs: ['/usr/local/lib', '/usr/lib'] includes: ['/usr/local/include', '/usr/include'] ===================================== blas_src_info: NOT AVAILABLE /home/nadav/numpy/numpy/distutils/system_info.py:1357: UserWarning: Blas (http://www.netlib.org/blas/) sources not found. Directories to search for the sources can be specified in the numpy/distutils/site.cfg file (section [blas_src]) or by setting the BLAS_SRC environment variable. warnings.warn(BlasSrcNotFoundError.__doc__) NOT AVAILABLE ===================================== class: numpy.distutils.system_info.lapack_opt_info libs: ['/usr/local/lib', '/usr/lib'] includes: ['/usr/local/include', '/usr/include'] ===================================== lapack_opt_info: ===================================== class: numpy.distutils.system_info.lapack_mkl_info libs: ['/usr/local/lib', '/usr/lib'] includes: ['/usr/local/include', '/usr/include'] ===================================== lapack_mkl_info: ===================================== class: numpy.distutils.system_info.mkl_info libs: ['/usr/local/lib', '/usr/lib'] includes: ['/usr/local/include', '/usr/include'] ===================================== mkl_info: libraries mkl,vml,guide not found in /usr/lib libraries mkl,vml,guide not found in /usr/local/lib NOT AVAILABLE NOT AVAILABLE ===================================== class: numpy.distutils.system_info.atlas_threads_info libs: ['/usr/local/lib', '/usr/lib'] includes: ['/usr/local/include', '/usr/include'] ===================================== atlas_threads_info: Setting PTATLAS=ATLAS numpy.distutils.system_info.atlas_threads_info NOT AVAILABLE ===================================== class: numpy.distutils.system_info.atlas_info libs: ['/usr/local/lib', '/usr/lib'] includes: ['/usr/local/include', '/usr/include'] ===================================== atlas_info: numpy.distutils.system_info.atlas_info NOT AVAILABLE /home/nadav/numpy/numpy/distutils/system_info.py:1252: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) ===================================== class: numpy.distutils.system_info.lapack_info libs: ['/usr/local/lib', '/usr/lib'] includes: ['/usr/local/include', '/usr/include'] ===================================== lapack_info: NOT AVAILABLE /home/nadav/numpy/numpy/distutils/system_info.py:1263: UserWarning: Lapack (http://www.netlib.org/lapack/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [lapack]) or by setting the LAPACK environment variable. warnings.warn(LapackNotFoundError.__doc__) ===================================== class: numpy.distutils.system_info.lapack_src_info libs: ['/usr/local/lib', '/usr/lib'] includes: ['/usr/local/include', '/usr/include'] ===================================== lapack_src_info: NOT AVAILABLE /home/nadav/numpy/numpy/distutils/system_info.py:1266: UserWarning: Lapack (http://www.netlib.org/lapack/) sources not found. Directories to search for the sources can be specified in the numpy/distutils/site.cfg file (section [lapack_src]) or by setting the LAPACK_SRC environment variable. warnings.warn(LapackSrcNotFoundError.__doc__) NOT AVAILABLE running build running scons . . . building extension "numpy.core._dotblas" sources building extension "numpy.lib._compiled_base" sources building extension "numpy.numarray._capi" sources building extension "numpy.fft.fftpack_lite" sources building extension "numpy.linalg.lapack_lite" sources creating build/src.linux-x86_64-2.5/numpy/linalg ### Warning: Using unoptimized lapack ### adding 'numpy/linalg/lapack_litemodule.c' to sources. adding 'numpy/linalg/zlapack_lite.c' to sources. adding 'numpy/linalg/dlapack_lite.c' to sources. adding 'numpy/linalg/blas_lite.c' to sources. adding 'numpy/linalg/dlamch.c' to sources. adding 'numpy/linalg/f2c_lite.c' to sources. ------------------------------------------- * Why /usr/local/lib and /usr/local/include are there although they are not in site.cfg? * Why lapack library was not found? * site.cfg.example is located in ~/numpy but the UserWarning indicates numpy/distutils/site.cfg. I copied site.cfg also to ~/numpy/numpy/distutils but it did not help. Nadav From lists at informa.tiker.net Mon Apr 7 15:56:23 2008 From: lists at informa.tiker.net (Andreas =?iso-8859-1?q?Kl=F6ckner?=) Date: Mon, 7 Apr 2008 15:56:23 -0400 Subject: [Numpy-discussion] site.cfg doesnt function? In-Reply-To: <710F2847B0018641891D9A21602763600B6F61@ex3.envision.co.il> References: <710F2847B0018641891D9A21602763600B6F61@ex3.envision.co.il> Message-ID: <200804071556.25483.lists@informa.tiker.net> Hi Nadav, On Montag 07 April 2008, Nadav Horesh wrote: > [snip] Try something like this: [atlas] library_dirs = /users/kloeckner/mach/x86_64/pool/lib,/usr/lib atlas_libs = lapack, f77blas, cblas, atlas Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From tjreedy at udel.edu Mon Apr 7 12:35:00 2008 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 7 Apr 2008 12:35:00 -0400 Subject: [Numpy-discussion] New project : Spyke python-to-C compiler References: <20080406184850.xiqr9t0hwgco004s@webmail.ualberta.ca> Message-ID: "Rahul Garg" wrote in message news:20080406184850.xiqr9t0hwgco004s at webmail.ualberta.ca... | Note this message has been posted to numpy-discussion and python-dev. | Sorry for the multiple posting but I thought both python devs and | numpy users will be interested. If you believe your list should not | receive this email, let me know. Also I just wanted to introduce | myself since I may ask doubts about Python and Numpy internals from | time to time :) Pydev is for discussion about development of future versions of Python (ie, 2.6 and some of 3.0). I think this would have been better posted on comp.lang.python (or gmane.comp.python.general or the corresponding mailing list). You can get answers about current Python internals there. From berthe.loic at gmail.com Mon Apr 7 16:14:20 2008 From: berthe.loic at gmail.com (LB) Date: Mon, 7 Apr 2008 13:14:20 -0700 (PDT) Subject: [Numpy-discussion] Ticket #605 Incorrect behavior of numpy.histogram In-Reply-To: References: <200804071434.09201.meine@informatik.uni-hamburg.de> <91cf711d0804070655s42a3d12cp320254cb79d57ba2@mail.gmail.com> Message-ID: <2cd9d43b-9c9a-4539-8e2c-3e1a7485a1b7@m3g2000hsc.googlegroups.com> +1 for axis and +1 for a keyword to define what to do with values outside the range. For the keyword, ather than 'outliers', I would propose 'discard' or 'exclude', because it could be used to describe the four possibilities : - discard='low' => values lower than the range are discarded, values higher are added to the last bin - discard='up' => values higher than the range are discarded, values lower are added to the first bin - discard='out' => values out of the range are discarded - discard=None => values outside of this range are allocated to the closest bin For the default behavior, most of the case, the sum of the bins 's population should be equal to the size of the original one for me, so I would prefer discard=None. But I'm also okay with discard='low' in order not to break older code, if this is clearly stated. -- LB From stefan at sun.ac.za Mon Apr 7 16:16:57 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 7 Apr 2008 22:16:57 +0200 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <200804071447.30956.lists@informa.tiker.net> References: <47FA65B3.4000307@gmail.com> <3d375d730804071129k19cab8efn5a22a6127d6aa424@mail.gmail.com> <200804071447.30956.lists@informa.tiker.net> Message-ID: <9457e7c80804071316v6a79685ft6ab5f866f014fa09@mail.gmail.com> On 07/04/2008, Andreas Kl?ckner wrote: > On Montag 07 April 2008, Robert Kern wrote: > > I would prefer not to do it at all. We've just gotten people moved > > over from Numeric; I'd hate to break their trust again. > > > +1. > > IMO, numpy has arrived at a state where there's just enough namespace clutter > to allow most use cases to get by without importing much sub-namespace junk, > and I think that's a good place to be (and to stay). I wouldn't exactly call 494 functions "just enough namespace clutter"; I'd much prefer to have a clean api to work with. I certainly don't propose forcing such an api upon all users, but it should be available as an option, at least. Tim's suggestion for a separate package that pulls in a "structured" numpy would suit my needs. As Gael mentioned, __init__'s are cursed, otherwise we'd be able to provide numpy.* for the flat earth society (all in friendly jest ;) and numpy.api to expose a somewhat organised underlying structure. As it is, importing numpy.api would trigger the __init__ of the flat namespace as well; but I'd still be amenable to this solution since the import doesn't take long, and the organisation of the api is more important to me. Would it therefore make sense to a) Reorganise numpy to expose functionality as numpy.api.* b) Do a series of imports in numpy.__init__ which pulls in from numpy.api. This way, numpy.* would look exactly as it does now, bar the added member 'api'. Regards St?fan From gael.varoquaux at normalesup.org Mon Apr 7 16:24:58 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 7 Apr 2008 22:24:58 +0200 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <9457e7c80804071316v6a79685ft6ab5f866f014fa09@mail.gmail.com> References: <47FA65B3.4000307@gmail.com> <3d375d730804071129k19cab8efn5a22a6127d6aa424@mail.gmail.com> <200804071447.30956.lists@informa.tiker.net> <9457e7c80804071316v6a79685ft6ab5f866f014fa09@mail.gmail.com> Message-ID: <20080407202458.GC21709@phare.normalesup.org> On Mon, Apr 07, 2008 at 10:16:57PM +0200, St?fan van der Walt wrote: > Would it therefore make sense to > a) Reorganise numpy to expose functionality as numpy.api.* > b) Do a series of imports in numpy.__init__ which pulls in from numpy.api. > This way, numpy.* would look exactly as it does now, bar the added member 'api'. +1. That way you don't break compatibility, but you provide nested namespace for people interested in them. You still get the import overhead. That's too bad. With some very good engineering, you might even make it possible to ship only part of numpy for custom installations. Cheers, Ga?l From tgrav at mac.com Mon Apr 7 16:26:29 2008 From: tgrav at mac.com (Tommy Grav) Date: Mon, 7 Apr 2008 16:26:29 -0400 Subject: [Numpy-discussion] Ticket #605 Incorrect behavior of numpy.histogram In-Reply-To: <2cd9d43b-9c9a-4539-8e2c-3e1a7485a1b7@m3g2000hsc.googlegroups.com> References: <200804071434.09201.meine@informatik.uni-hamburg.de> <91cf711d0804070655s42a3d12cp320254cb79d57ba2@mail.gmail.com> <2cd9d43b-9c9a-4539-8e2c-3e1a7485a1b7@m3g2000hsc.googlegroups.com> Message-ID: On Apr 7, 2008, at 4:14 PM, LB wrote: > +1 for axis and +1 for a keyword to define what to do with values > outside the range. > > For the keyword, ather than 'outliers', I would propose 'discard' or > 'exclude', because it could be used to describe the four > possibilities : > - discard='low' => values lower than the range are discarded, > values higher are added to the last bin > - discard='up' => values higher than the range are discarded, > values lower are added to the first bin > - discard='out' => values out of the range are discarded > - discard=None => values outside of this range are allocated to > the closest bin > > For the default behavior, most of the case, the sum of the bins 's > population should be equal to the size of the original one for me, so > I would prefer discard=None. But I'm also okay with discard='low' in > order not to break older code, if this is clearly stated. It seems that people in this discussion are forgetting that the bins are actually defined by the lower boundaries supplied, such that bins = [1,3,5] actually currently means bin1 -> 1 to 2.99999... bin2 -> 3 to 4.99999... bin3 -> 5 to inf (of course in version 1.0.1 the documentation is inconsistent with the behavior as described by the original poster). This definition of bins makes it hard to exclude values as it forces the user to give an extra value in the bin definition, i.e. the bins statement above only give two bins, while supplying three values. That seems confusing to me. I am not sure what the right approach is, but currently using range will clip the values outside the number the user wants. Cheers Tommy From oliphant at enthought.com Mon Apr 7 16:52:45 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 07 Apr 2008 15:52:45 -0500 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <9457e7c80804071316v6a79685ft6ab5f866f014fa09@mail.gmail.com> References: <47FA65B3.4000307@gmail.com> <3d375d730804071129k19cab8efn5a22a6127d6aa424@mail.gmail.com> <200804071447.30956.lists@informa.tiker.net> <9457e7c80804071316v6a79685ft6ab5f866f014fa09@mail.gmail.com> Message-ID: <47FA899D.7010404@enthought.com> > I wouldn't exactly call 494 functions "just enough namespace clutter"; > I'd much prefer to have a clean api to work with I don't know. The 494 functions do not seem like many to me. Apparently, I tend to come down in the "flat earth society" although I do like some structure (after all that's why numpy has numpy.linalg and numpy.fft). I don't think this is the most pressing issue we are facing. > Would it therefore make sense to > > a) Reorganise numpy to expose functionality as numpy.api.* > b) Do a series of imports in numpy.__init__ which pulls in from numpy.api. > > This way, numpy.* would look exactly as it does now, bar the added member 'api'. > This discussion is interesting, but it is really better suited for 1.1, isn't it? -0 on adding the .api name in 1.0.5 -Travis From gael.varoquaux at normalesup.org Mon Apr 7 16:56:15 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 7 Apr 2008 22:56:15 +0200 Subject: [Numpy-discussion] Back to Simple financial functions for NumPy (was Re: packaging scipy) In-Reply-To: <47FA899D.7010404@enthought.com> References: <47FA65B3.4000307@gmail.com> <3d375d730804071129k19cab8efn5a22a6127d6aa424@mail.gmail.com> <200804071447.30956.lists@informa.tiker.net> <9457e7c80804071316v6a79685ft6ab5f866f014fa09@mail.gmail.com> <47FA899D.7010404@enthought.com> Message-ID: <20080407205615.GD21709@phare.normalesup.org> On Mon, Apr 07, 2008 at 03:52:45PM -0500, Travis E. Oliphant wrote: > This discussion is interesting, but it is really better suited for 1.1, > isn't it? Yes. The original dicussion was about adding simple financial functions for Numpy. I think the functions you wrote should land somewhere. We may disagree on where, and you should choose based on feedback here and your personnal feeling, but I hope they will indeed be published somewhere. Cheers, Ga?l From lists at informa.tiker.net Mon Apr 7 16:55:59 2008 From: lists at informa.tiker.net (Andreas =?iso-8859-1?q?Kl=F6ckner?=) Date: Mon, 7 Apr 2008 16:55:59 -0400 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <9457e7c80804071316v6a79685ft6ab5f866f014fa09@mail.gmail.com> References: <200804071447.30956.lists@informa.tiker.net> <9457e7c80804071316v6a79685ft6ab5f866f014fa09@mail.gmail.com> Message-ID: <200804071656.00342.lists@informa.tiker.net> On Montag 07 April 2008, St?fan van der Walt wrote: > I wouldn't exactly call 494 functions "just enough namespace clutter"; > I'd much prefer to have a clean api to work with. Not to bicker, but... >>> import numpy >>> len(dir(numpy)) 494 >>> numpy.__version__ '1.0.4' >>> funcs = [s for s in dir(numpy) if type(getattr(numpy, s)) in [type(numpy.array), type(numpy.who)]] >>> len(funcs) 251 >>> classes = [s for s in dir(numpy) if type(getattr(numpy, s)) == type(numpy.ndarray)] >>> len(classes) 88 >>> ufuncs = [s for s in dir(numpy) if type(getattr(numpy, s)) == type(numpy.sin)] >>> len(ufuncs) 69 >>> (and, therefore, another 69 names of "fluff") I honestly don't see much of a problem. The only things that maybe should not have been added to numpy.* are the polynomial functions and the convolution windows, conceptually. But in my book that's not big enough to even think of breaking people's code for. Andreas Proud Member of the Flat Earth Society -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From stefan at sun.ac.za Mon Apr 7 17:09:06 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 7 Apr 2008 23:09:06 +0200 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <47FA899D.7010404@enthought.com> References: <47FA65B3.4000307@gmail.com> <3d375d730804071129k19cab8efn5a22a6127d6aa424@mail.gmail.com> <200804071447.30956.lists@informa.tiker.net> <9457e7c80804071316v6a79685ft6ab5f866f014fa09@mail.gmail.com> <47FA899D.7010404@enthought.com> Message-ID: <9457e7c80804071409j77b5e12bo5044203633198364@mail.gmail.com> On 07/04/2008, Travis E. Oliphant wrote: > > Would it therefore make sense to > > > > a) Reorganise numpy to expose functionality as numpy.api.* > > b) Do a series of imports in numpy.__init__ which pulls in from numpy.api. > > > > This way, numpy.* would look exactly as it does now, bar the added member 'api'. > > > > This discussion is interesting, but it is really better suited for 1.1, > isn't it? Certainly -- this suggestion is aimed at 1.1 (along with the discussion we had with Anne on API refactoring). > -0 on adding the .api name in 1.0.5 Aargh, the +- zeros again! I thought we closed that ticket :) Cheers St?fan From stefan at sun.ac.za Mon Apr 7 17:21:41 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 7 Apr 2008 23:21:41 +0200 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <200804071656.00342.lists@informa.tiker.net> References: <200804071447.30956.lists@informa.tiker.net> <9457e7c80804071316v6a79685ft6ab5f866f014fa09@mail.gmail.com> <200804071656.00342.lists@informa.tiker.net> Message-ID: <9457e7c80804071421i74276059ud6b68cf50e93c29@mail.gmail.com> On 07/04/2008, Andreas Kl?ckner wrote: > On Montag 07 April 2008, St?fan van der Walt wrote: > > I wouldn't exactly call 494 functions "just enough namespace clutter"; > > I'd much prefer to have a clean api to work with. > > > Not to bicker, but... > > >>> import numpy > >>> len(dir(numpy)) > 494 You'd be glad to know that that your investment increased as of r4964: 494 -> 504 251 -> 258 88 -> 89 69 -> 71 > I honestly don't see much of a problem. I see at least two: a) These numbers are growing (albeit slowly) and b) numpy.TAB under IPython shows 516 completions It doesn't matter *what* these completions are -- they're still there. Sifting through 500 options isn't fun -- not for a newbie, nor a salty old sailor. I agree with Joe that the problem can be ameliorated by documentation, but I do think that an (optional) fundamental restructuring is ultimately useful. > Proud Member of the Flat Earth Society :) Cheers St?fan From gael.varoquaux at normalesup.org Mon Apr 7 17:26:04 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 7 Apr 2008 23:26:04 +0200 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <9457e7c80804071421i74276059ud6b68cf50e93c29@mail.gmail.com> References: <200804071447.30956.lists@informa.tiker.net> <9457e7c80804071316v6a79685ft6ab5f866f014fa09@mail.gmail.com> <200804071656.00342.lists@informa.tiker.net> <9457e7c80804071421i74276059ud6b68cf50e93c29@mail.gmail.com> Message-ID: <20080407212604.GF21709@phare.normalesup.org> On Mon, Apr 07, 2008 at 11:21:41PM +0200, St?fan van der Walt wrote: > It doesn't matter *what* these completions are -- they're still there. > Sifting through 500 options isn't fun I get more than that when I tab in an empty shell on my box. :-} Why do you expect to be able to inspect a module of the size of numpy with tab-completion. I agree that for this usecase (which is more a question of discovering/exploring the API than using it) a nested namespace is better. And as I think this usecase is valid, this is why I like the proposition of having an "api" submodule, with a nested namespace. My two cents, Ga?l From charlesr.harris at gmail.com Mon Apr 7 17:28:35 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 7 Apr 2008 15:28:35 -0600 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <9457e7c80804071421i74276059ud6b68cf50e93c29@mail.gmail.com> References: <200804071447.30956.lists@informa.tiker.net> <9457e7c80804071316v6a79685ft6ab5f866f014fa09@mail.gmail.com> <200804071656.00342.lists@informa.tiker.net> <9457e7c80804071421i74276059ud6b68cf50e93c29@mail.gmail.com> Message-ID: On Mon, Apr 7, 2008 at 3:21 PM, St?fan van der Walt wrote: > On 07/04/2008, Andreas Kl?ckner wrote: > > On Montag 07 April 2008, St?fan van der Walt wrote: > > > I wouldn't exactly call 494 functions "just enough namespace > clutter"; > > > I'd much prefer to have a clean api to work with. > > > > > > Not to bicker, but... > > > > >>> import numpy > > >>> len(dir(numpy)) > > 494 > > You'd be glad to know that that your investment increased as of r4964: > > 494 -> 504 > 251 -> 258 > 88 -> 89 > 69 -> 71 > > > I honestly don't see much of a problem. > > I see at least two: > > a) These numbers are growing (albeit slowly) and > b) numpy.TAB under IPython shows 516 completions > > It doesn't matter *what* these completions are -- they're still there. > Sifting through 500 options isn't fun -- not for a newbie, nor a > salty old sailor. I agree with Joe that the problem can be > ameliorated by documentation, but I do think that an (optional) > fundamental restructuring is ultimately useful. > Yeah, dir needs to print in two columns ;) I think we could use an apropos sort of function that indexes the documentation, making it easier to find relevant functions. Apart from that, I think we should stop adding to the numpy namespace. Polynomials, financial functions, image processing, all that is nice to have around, but I don't think it belongs in the top level. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From perry at stsci.edu Mon Apr 7 17:27:10 2008 From: perry at stsci.edu (Perry Greenfield) Date: Mon, 7 Apr 2008 17:27:10 -0400 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <3d375d730804071129k19cab8efn5a22a6127d6aa424@mail.gmail.com> References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <9457e7c80804070922r67ae453cy4bf3473545c05692@mail.gmail.com> <20080407165740.GK9851@phare.normalesup.org> <36510.192.55.12.36.1207589204.squirrel@mail2.webfaction.com> <47FA65B3.4000307@gmail.com> <3d375d730804071129k19cab8efn5a22a6127d6aa424@mail.gmail.com> Message-ID: On Apr 7, 2008, at 2:29 PM, Robert Kern wrote: > On Mon, Apr 7, 2008 at 11:19 AM, Bruce Southey > wrote: >> Hi, >> I think that splitting the NumPy namespace should not happen >> within a >> major release series because it would cause too many breakages. >> Rather >> it should be in a forthcoming release like the 2.0 series where >> it may >> be very feasible to have a true core functionality (NumPy), extended >> functionality (SciPy) and specific applications (Scikits). At the >> same >> time, Python 3K would be fully supported because it will break >> lots of >> code. > > I would prefer not to do it at all. We've just gotten people moved > over from Numeric; I'd hate to break their trust again. > Amen. >> It is really nice to think about having NumPy Core, NumPy Full, >> NumPyKits, SciPy Core, SciPy Full and SciPyKits. > > Really? It gives me the shivers, frankly. > Me too. Some random comments: 1) It seems to me that the primary problem people have with a big flat namespace is that it makes the output of dir() long and unusable, and by implication, that a nice hierarchical organization would make it easier for people to find stuff. As to the latter, less so than you might think. From what I've seen, there is no generally agreed upon organization that all will agree to or find intuitive. There will always be functions that logically belong to more than one category. Ultimately, this is why flatter is better as far as that goes. If one wants to find things by category, we would be much better off tagging functions with categories and then building some dir-like tool that helps display things in that category. Some have already alluded to that (Joe Harrington I believe). The only thing namespaces solve is name collisions imho. I don't believe that the current numpy has too many names in its basic namespace, and it already has split out some things into subpackages (fft, random, linear algebra) that have such a potential. 2) Some may feel that the combination of "from numpy import *" with a lot of names makes it hard to see other things in your namespace. True enough. But no amount of winnowing is going to keep the namespace at a point that isn't going to overwhelm everything else. The answer is simple in that case. If that's a problem for you, don't use "from numpy import *". Or perhaps another dir-like tool that filters out all numpy/scipy/pylab... items. 3) Some don't like the bloat (in disk space or download sizes) of adding things to numpy. In my case, as long as the addition doesn't make installations any more difficult I don't care. For the great majority, the current size or anything within an order of magnitude is not an important issue. For the 56Kb modem people, perhaps we can construct a numpy-lite, but it shouldn't be the standard distribution. I don't mind the financial functions going into numpy. I think it's a good idea since a lot of people may find that very handy to be part of the core distribution, probably many more than worry about more exotic packages, and likely many more than care about fft, random and linear algebra. 4) The api interface is perhaps a good idea, but as Travis mentions can be deferred. But I don't think I would need it. (see 1) Perry From ellisonbg.net at gmail.com Mon Apr 7 17:54:15 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Mon, 7 Apr 2008 15:54:15 -0600 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: References: <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <9457e7c80804070922r67ae453cy4bf3473545c05692@mail.gmail.com> <20080407165740.GK9851@phare.normalesup.org> <36510.192.55.12.36.1207589204.squirrel@mail2.webfaction.com> <47FA65B3.4000307@gmail.com> <3d375d730804071129k19cab8efn5a22a6127d6aa424@mail.gmail.com> Message-ID: <6ce0ac130804071454p4b2d9a77p5d12dbf6a872a1e2@mail.gmail.com> > 3) Some don't like the bloat (in disk space or download sizes) of > adding things to numpy. In my case, as long as the addition doesn't > make installations any more difficult I don't care. For the great > majority, the current size or anything within an order of magnitude > is not an important issue. For the 56Kb modem people, perhaps we can > construct a numpy-lite, but it shouldn't be the standard > distribution. I don't mind the financial functions going into numpy. > I think it's a good idea since a lot of people may find that very > handy to be part of the core distribution, probably many more than > worry about more exotic packages, and likely many more than care > about fft, random and linear algebra. The only problem is that if we keep adding things to numpy that could be in scipy, it will _never_ be clear to users where they can expect to find things. It is already bad enough. How do I explain to a user/student/scientist that ffts and linear algebra are in numpy, but that integration and interpolation are in scipy. That doesn't make any sense to them. Oh but wait, linear algebra and ffts are also in scipy! Random numbers - take a guess - wrong, they are in numpy. As far as I am concerned, financial fucntions are completely outside the conceptual scope that numpy has established = arrays, fft, linalg, random. In fact, they are far outside it. Simply putting things into numpy because of convenience (numpy is easier to install) only encourages people to never install or use scipy. If scipy that much of a pain to install and use - we should spend our time improving scipy. Cheers, Brian From perry at stsci.edu Mon Apr 7 18:03:56 2008 From: perry at stsci.edu (Perry Greenfield) Date: Mon, 7 Apr 2008 18:03:56 -0400 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <6ce0ac130804071454p4b2d9a77p5d12dbf6a872a1e2@mail.gmail.com> References: <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <9457e7c80804070922r67ae453cy4bf3473545c05692@mail.gmail.com> <20080407165740.GK9851@phare.normalesup.org> <36510.192.55.12.36.1207589204.squirrel@mail2.webfaction.com> <47FA65B3.4000307@gmail.com> <3d375d730804071129k19cab8efn5a22a6127d6aa424@mail.gmail.com> <6ce0ac130804071454p4b2d9a77p5d12dbf6a872a1e2@mail.gmail.com> Message-ID: On Apr 7, 2008, at 5:54 PM, Brian Granger wrote: >> > The only problem is that if we keep adding things to numpy that could > be in scipy, it will _never_ be clear to users where they can expect > to find things. It is already bad enough. How do I explain to a > user/student/scientist that ffts and linear algebra are in numpy, but > that integration and interpolation are in scipy. That doesn't make > any sense to them. Oh but wait, linear algebra and ffts are also in > scipy! Random numbers - take a guess - wrong, they are in numpy. > > As far as I am concerned, financial fucntions are completely outside > the conceptual scope that numpy has established = arrays, fft, linalg, > random. In fact, they are far outside it. Simply putting things into > numpy because of convenience (numpy is easier to install) only > encourages people to never install or use scipy. If scipy that much > of a pain to install and use - we should spend our time improving > scipy. > > Cheers, > > Brian > To me, the biggest characteristic difference between the two is the ease of installation. If installation weren't an issue, I would tell everyone to use scipy and then the confusion would be ended. But the installation issue is not trivial one to solve (if it were, we'd already be there). But a nice ideal is that the numpy namespace should map directly into scipy's so that if I expected numpy.xxx to work, the scipy.xxx should also work. That would lessen the confusion of finding things. If it isn't in numpy, it's in scipy. But otherwise one is a subset of the other. (I say this from complete ignorance, I'm not sure what prevents this, and there may be very good reasons why this can't be done). Perry From ellisonbg.net at gmail.com Mon Apr 7 18:49:35 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Mon, 7 Apr 2008 16:49:35 -0600 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: References: <9457e7c80804070922r67ae453cy4bf3473545c05692@mail.gmail.com> <20080407165740.GK9851@phare.normalesup.org> <36510.192.55.12.36.1207589204.squirrel@mail2.webfaction.com> <47FA65B3.4000307@gmail.com> <3d375d730804071129k19cab8efn5a22a6127d6aa424@mail.gmail.com> <6ce0ac130804071454p4b2d9a77p5d12dbf6a872a1e2@mail.gmail.com> Message-ID: <6ce0ac130804071549x797da86fgc15e76f94e599e93@mail.gmail.com> On Mon, Apr 7, 2008 at 4:03 PM, Perry Greenfield wrote: > > On Apr 7, 2008, at 5:54 PM, Brian Granger wrote: > >> > > The only problem is that if we keep adding things to numpy that could > > be in scipy, it will _never_ be clear to users where they can expect > > to find things. It is already bad enough. How do I explain to a > > user/student/scientist that ffts and linear algebra are in numpy, but > > that integration and interpolation are in scipy. That doesn't make > > any sense to them. Oh but wait, linear algebra and ffts are also in > > scipy! Random numbers - take a guess - wrong, they are in numpy. > > > > As far as I am concerned, financial fucntions are completely outside > > the conceptual scope that numpy has established = arrays, fft, linalg, > > random. In fact, they are far outside it. Simply putting things into > > numpy because of convenience (numpy is easier to install) only > > encourages people to never install or use scipy. If scipy that much > > of a pain to install and use - we should spend our time improving > > scipy. > > > > Cheers, > > > > Brian > > > > To me, the biggest characteristic difference between the two is the > ease of installation. If installation weren't an issue, I would tell > everyone to use scipy and then the confusion would be ended. But the > installation issue is not trivial one to solve (if it were, we'd > already be there). I definitely understand the installation issue. Is the main thing people run into the fortran compiler, BLAS and LAPACK? To me it seems like the scipy install is pretty simple these days. Do we need better installation documentation? Deep down I so wish we could ditch fortran! In almost all cases I know of projects that have fortran involved are the worse for it. > But a nice ideal is that the numpy namespace should map directly into > scipy's so that if I expected numpy.xxx to work, the scipy.xxx should > also work. That would lessen the confusion of finding things. If it > isn't in numpy, it's in scipy. But otherwise one is a subset of the > other. (I say this from complete ignorance, I'm not sure what > prevents this, and there may be very good reasons why this can't be > done). I think this is a very good idea to follow, at least for things that do happen to in both places. But, I still don't think it we should have many things that are in both places. Brian > > > Perry > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From fperez.net at gmail.com Mon Apr 7 18:56:53 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 7 Apr 2008 15:56:53 -0700 Subject: [Numpy-discussion] Tests 32/64 bits Message-ID: Hi all, here's the output difference between 32/64 bit tests run: bic128[scipy]$ diff -u tests_32.txt tests_64.txt --- tests_32.txt 2008-04-07 15:54:29.000000000 -0700 +++ tests_64.txt 2008-04-07 15:53:58.000000000 -0700 @@ -611,12 +611,6 @@ testip_types (numpy.core.tests.test_multiarray.TestPutmask) ... ok testip_types (numpy.core.tests.test_multiarray.TestPutmask) ... ok testip_types (numpy.core.tests.test_multiarray.TestPutmask) ... ok -testip_types (numpy.core.tests.test_multiarray.TestPutmask) ... ok -testip_types (numpy.core.tests.test_multiarray.TestPutmask) ... ok -testip_types (numpy.core.tests.test_multiarray.TestPutmask) ... ok -testip_types (numpy.core.tests.test_multiarray.TestPutmask) ... ok -testip_types (numpy.core.tests.test_multiarray.TestPutmask) ... ok -testip_types (numpy.core.tests.test_multiarray.TestPutmask) ... ok testip_zero (numpy.tests.test_linalg.TestMatrixPower) ... ok testip_zero (numpy.tests.test_linalg.TestMatrixPower) ... ok testip_zero (numpy.tests.test_linalg.TestMatrixPower) ... ok On 32 bits, 869 are found and on Fedora x86_64, it's 863. Above is the difference (requested by rkern). Gotta run... f From robert.kern at gmail.com Mon Apr 7 19:04:34 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 7 Apr 2008 16:04:34 -0700 Subject: [Numpy-discussion] Tests 32/64 bits In-Reply-To: References: Message-ID: <3d375d730804071604h1e646d20x855cec5acabda4c@mail.gmail.com> On Mon, Apr 7, 2008 at 3:56 PM, Fernando Perez wrote: > Hi all, > > here's the output difference between 32/64 bit tests run: > > bic128[scipy]$ diff -u tests_32.txt tests_64.txt > --- tests_32.txt 2008-04-07 15:54:29.000000000 -0700 > +++ tests_64.txt 2008-04-07 15:53:58.000000000 -0700 > @@ -611,12 +611,6 @@ > testip_types (numpy.core.tests.test_multiarray.TestPutmask) ... ok > testip_types (numpy.core.tests.test_multiarray.TestPutmask) ... ok > testip_types (numpy.core.tests.test_multiarray.TestPutmask) ... ok > -testip_types (numpy.core.tests.test_multiarray.TestPutmask) ... ok > -testip_types (numpy.core.tests.test_multiarray.TestPutmask) ... ok > -testip_types (numpy.core.tests.test_multiarray.TestPutmask) ... ok > -testip_types (numpy.core.tests.test_multiarray.TestPutmask) ... ok > -testip_types (numpy.core.tests.test_multiarray.TestPutmask) ... ok > -testip_types (numpy.core.tests.test_multiarray.TestPutmask) ... ok > testip_zero (numpy.tests.test_linalg.TestMatrixPower) ... ok > testip_zero (numpy.tests.test_linalg.TestMatrixPower) ... ok > testip_zero (numpy.tests.test_linalg.TestMatrixPower) ... ok > > > On 32 bits, 869 are found and on Fedora x86_64, it's 863. Above is > the difference (requested by rkern). I think this is fine. The different arises because of extra scalar types on the 32 bit system that don't show up on the AMD system, presumably float96 and complex192. Check numpy.sctypes for the difference. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From garg1 at ualberta.ca Mon Apr 7 20:28:26 2008 From: garg1 at ualberta.ca (Rahul Garg) Date: Mon, 07 Apr 2008 18:28:26 -0600 Subject: [Numpy-discussion] Spyke : moving forward Message-ID: <20080407182826.uwn4u02pco88080g@webmail.ualberta.ca> Thanks everyone for suggestions and enthusiasm. 1. For type declarations moving from string based annotations to decorators for functions. (Function annotations in 3k: Well it should be easy to add them once that feature comes out.) 2. License : Keeping it as GPLv3 for compiler. The "runtime" is just a python script that invokes the real compiler binary and can be licensed under LGPL or BSD if thats what people prefer. 3. Release : Will release source+binary in 2-3 weeks. Need to get some stuff sorted at uni. Please be patient :) 4. Will establish a testsuite at google code in a couple of days. Everyone is encouraged to contribute test cases whether big or small. Having a proper test suite will mean we can better track the bugs and features in the compiler on a daily basis. Are there any suggestions on how the test suite should be organized? I want a test suite where we have lets say N tests and a script runs all N tests and compares the expected output and says "pass/fail" for each one of them. Also what license is suitable for testsuite? The code will remain 100% pure Python so I believe any license can be chosen. 5. Interop with C : I am thinking of a module which wraps the functionality of ctypes with some added type declarations. But this wont work anytime soon. 6 Invoking compiler at runtime instead of as a standalone compiler : I had not thought of invoking Spyke at runtime through the decorator. So currently Spyke is invoked standalone from the commandline. But now that we are adding decorators, as suggested by Travis and others I will look into how to invoke compiler directly from the decorator. 7. Support for classes : Basic support for classes but the code generated currently for classes is pretty slow. No support for exceptions currently. 8. Long term plans : I intend to use Spyke as a platform for some research into compiler optimizations. From time to time I may play with some things but those will be kept out of the trunk. thanks, rahul From robert.kern at gmail.com Mon Apr 7 20:42:05 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 7 Apr 2008 17:42:05 -0700 Subject: [Numpy-discussion] Spyke : moving forward In-Reply-To: <20080407182826.uwn4u02pco88080g@webmail.ualberta.ca> References: <20080407182826.uwn4u02pco88080g@webmail.ualberta.ca> Message-ID: <3d375d730804071742ob7675c7wede6f52c95ee999d@mail.gmail.com> On Mon, Apr 7, 2008 at 5:28 PM, Rahul Garg wrote: > 2. License : Keeping it as GPLv3 for compiler. The "runtime" is just a > python script that invokes the real compiler binary and can be > licensed under LGPL or BSD if thats what people prefer. Can you clarify this? That is not what I would have called a "runtime". -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From fperez.net at gmail.com Mon Apr 7 21:29:43 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 7 Apr 2008 18:29:43 -0700 Subject: [Numpy-discussion] Tests 32/64 bits In-Reply-To: <3d375d730804071604h1e646d20x855cec5acabda4c@mail.gmail.com> References: <3d375d730804071604h1e646d20x855cec5acabda4c@mail.gmail.com> Message-ID: On Mon, Apr 7, 2008 at 4:04 PM, Robert Kern wrote: > > On 32 bits, 869 are found and on Fedora x86_64, it's 863. Above is > > the difference (requested by rkern). > > I think this is fine. The different arises because of extra scalar > types on the 32 bit system that don't show up on the AMD system, > presumably float96 and complex192. Check numpy.sctypes for the > difference. - complex192 isn't on the 64bit box, but complex256 is, so that keeps the number of tests for that type equal. - float96 -> float128, again no change in test count - for some reason, the 32-bit box gives 'int': [, , , , ], so there's a repeated int32 type listed there. I don't know what that means, but obviously it produces extra tests (possibly redundant?) - Same for uint: 'uint': [, , , , ]} In any case, other than this minor difference, current numpy SVN passes all tests on Fedora8/64bit and Ubuntu Gutsy/32bit. Cheers, f From robert.kern at gmail.com Mon Apr 7 21:36:39 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 7 Apr 2008 18:36:39 -0700 Subject: [Numpy-discussion] Tests 32/64 bits In-Reply-To: References: <3d375d730804071604h1e646d20x855cec5acabda4c@mail.gmail.com> Message-ID: <3d375d730804071836k76b10cc0lf90e79924998d3b@mail.gmail.com> On Mon, Apr 7, 2008 at 6:29 PM, Fernando Perez wrote: > On Mon, Apr 7, 2008 at 4:04 PM, Robert Kern wrote: > > > > On 32 bits, 869 are found and on Fedora x86_64, it's 863. Above is > > > the difference (requested by rkern). > > > > I think this is fine. The different arises because of extra scalar > > types on the 32 bit system that don't show up on the AMD system, > > presumably float96 and complex192. Check numpy.sctypes for the > > difference. > > - complex192 isn't on the 64bit box, but complex256 is, so that keeps > the number of tests for that type equal. > > - float96 -> float128, again no change in test count > > - for some reason, the 32-bit box gives > 'int': [, > , > , > , > ], > > so there's a repeated int32 type listed there. I don't know what that > means, but obviously it produces extra tests (possibly redundant?) Definitely redundant, but harmless. The code that generates these lists is numpy/core/numerictypes.py:_set_array_types(). The redundant ones are dtype('p').type and dtype('P').type. For some reason these do not compare equal to numpy.{u}int32. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david.huard at gmail.com Mon Apr 7 22:12:05 2008 From: david.huard at gmail.com (David Huard) Date: Mon, 7 Apr 2008 22:12:05 -0400 Subject: [Numpy-discussion] Ticket #605 Incorrect behavior of numpy.histogram In-Reply-To: References: <200804071434.09201.meine@informatik.uni-hamburg.de> <91cf711d0804070655s42a3d12cp320254cb79d57ba2@mail.gmail.com> <2cd9d43b-9c9a-4539-8e2c-3e1a7485a1b7@m3g2000hsc.googlegroups.com> Message-ID: <91cf711d0804071912g5aa3ff30i73591360b3ef99a@mail.gmail.com> > On Apr 7, 2008, at 4:14 PM, LB wrote: > > +1 for axis and +1 for a keyword to define what to do with values > > outside the range. > > > > For the keyword, ather than 'outliers', I would propose 'discard' or > > 'exclude', because it could be used to describe the four > > possibilities : > > - discard='low' => values lower than the range are discarded, > > values higher are added to the last bin > > - discard='up' => values higher than the range are discarded, > > values lower are added to the first bin > > - discard='out' => values out of the range are discarded > > - discard=None => values outside of this range are allocated to > > the closest bin > > Suppose you set bins=5, range=[0,10], discard=None, should the returned bins be [0,2,4,6,810] or [-inf, 2, 4, 6, 8, inf] ? Now suppose normed=True, what should be the density for the first and last bin ? It seems to me it should be zero since we are assuming that the bins extend to -infinity and infinity, but then, taking the outliers into account seems pretty useless. Overall, I think "discard" is a confusing option with little added value. Getting the outliers is simply a matter of defining the bin edges explictly, ie [-inf, x0, x1, ..., xn, inf]. In any case, attached is a version of histogram implementing the axis and discard keywords. I'd really prefer though if we dumped the discard option. David > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: histo2.py Type: text/x-python Size: 4952 bytes Desc: not available URL: From nadavh at visionsense.com Tue Apr 8 01:28:34 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Tue, 8 Apr 2008 08:28:34 +0300 Subject: [Numpy-discussion] site.cfg doesnt function? References: <710F2847B0018641891D9A21602763600B6F61@ex3.envision.co.il> <200804071556.25483.lists@informa.tiker.net> Message-ID: <710F2847B0018641891D9A21602763600B6F63@ex3.envision.co.il> I tried: [ALL] library_dirs = /usr/lib64/lapack/atlas:/usr/lib64/blas/threaded-atlas:/usr/lib include_dirs = /usr/include/atlas:/usr/include [blas_opt] library_dirs = /usr/lib64/blas/threaded-atlas:/usr/lib64 libraries = blas, cblas, atlas [lapack_opt] library_dirs = /usr/lib64/lapack/atlas:/usr/lib64 libraries = lapack, blas, cblas, atlas [fftw] libraries = fftw3 [atlas] library_dirs = /usr/lib64/lapack/atlas:/usr/lib64/blas/threaded-atlas:/usr/lib include_dirs = /usr/include/atlas:/usr/include libraries = lapack, blas, cblas, atlas but it did not change anything. any ideas? Nadav. -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? Andreas Kl?ckner ????: ? 07-?????-08 21:56 ??: Discussion of Numerical Python ????: Re: [Numpy-discussion] site.cfg doesnt function? Hi Nadav, On Montag 07 April 2008, Nadav Horesh wrote: > [snip] Try something like this: [atlas] library_dirs = /users/kloeckner/mach/x86_64/pool/lib,/usr/lib atlas_libs = lapack, f77blas, cblas, atlas Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3109 bytes Desc: not available URL: From hep.sebastien.binet at gmail.com Tue Apr 8 01:31:19 2008 From: hep.sebastien.binet at gmail.com (Sebastien Binet) Date: Mon, 7 Apr 2008 22:31:19 -0700 Subject: [Numpy-discussion] site.cfg doesnt function? In-Reply-To: <710F2847B0018641891D9A21602763600B6F63@ex3.envision.co.il> References: <710F2847B0018641891D9A21602763600B6F61@ex3.envision.co.il> <200804071556.25483.lists@informa.tiker.net> <710F2847B0018641891D9A21602763600B6F63@ex3.envision.co.il> Message-ID: <200804072231.19546.binet@cern.ch> Nadav, > [ALL] > library_dirs = > /usr/lib64/lapack/atlas:/usr/lib64/blas/threaded-atlas:/usr/lib > include_dirs = /usr/include/atlas:/usr/include I believe (contrary to my 'unix' intuition) that you should replace the colons by commas. ie: include_dirs = /usr/include/atlas,/usr/include library_dirs = /foo/path,/foo/path2 Cheers, Sebastien. -- ################################### # Dr. Sebastien Binet # # Lawrence Berkeley National Lab. # # 1 Cyclotron Road # # Berkeley, CA 94720 # ################################### -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part. URL: From charlesr.harris at gmail.com Tue Apr 8 03:09:04 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 8 Apr 2008 01:09:04 -0600 Subject: [Numpy-discussion] Back to Simple financial functions for NumPy (was Re: packaging scipy) In-Reply-To: <20080407205615.GD21709@phare.normalesup.org> References: <47FA65B3.4000307@gmail.com> <3d375d730804071129k19cab8efn5a22a6127d6aa424@mail.gmail.com> <200804071447.30956.lists@informa.tiker.net> <9457e7c80804071316v6a79685ft6ab5f866f014fa09@mail.gmail.com> <47FA899D.7010404@enthought.com> <20080407205615.GD21709@phare.normalesup.org> Message-ID: On Mon, Apr 7, 2008 at 2:56 PM, Gael Varoquaux < gael.varoquaux at normalesup.org> wrote: > On Mon, Apr 07, 2008 at 03:52:45PM -0500, Travis E. Oliphant wrote: > > This discussion is interesting, but it is really better suited for 1.1, > > isn't it? > > Yes. The original dicussion was about adding simple financial functions > for Numpy. > > I think the functions you wrote should land somewhere. We may disagree on > where, and you should choose based on feedback here and your personnal > feeling, but I hope they will indeed be published somewhere. > These functions are of immediate and pressing concern because US tax returns are due April 15 ;) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From meine at informatik.uni-hamburg.de Tue Apr 8 04:00:19 2008 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Tue, 8 Apr 2008 10:00:19 +0200 Subject: [Numpy-discussion] Ticket #605 Incorrect behavior of numpy.histogram In-Reply-To: <200804071434.09201.meine@informatik.uni-hamburg.de> References: <200804071434.09201.meine@informatik.uni-hamburg.de> Message-ID: <200804081000.20749.meine@informatik.uni-hamburg.de> Am Montag, 07. April 2008 14:34:08 schrieb Hans Meine: > Am Samstag, 05. April 2008 21:54:27 schrieb Anne Archibald: > > There's also a fourth option - raise an exception if any points are > > outside the range. > > +1 > > I think this should be the default. Otherwise, I tend towards "exclude", > in order to have comparable bin sizes (when plotting, I always find peaks > at the ends annoying); this could also be called "clip" BTW. > > But really, an exception would follow the Zen: "In the face of ambiguity, > refuse the temptation to guess." And with a kwarg: "Explicit is better > than implicit." When posting this, I did indeed not think this through fully; as David (and Tommy) pointed out, this API does not fit well with the existing `bins` option, especially when a sequence of bin bounds is given. (I guess I was mostly thinking about the special case of discrete values and 1:1 bins, as typical for uint8 data.) Thus, I would like to withdraw my above opinion from and instead state that I find the current API as clear as it gets. If you want to exclude values, simply pass an additional right bound, and for including outliers, passing -inf as additional left bound seems to do the trick. This could be possibly added to the documentation though. The only critical aspect I see is the `normed` arg. As it is now, the rightmost bin has always infinite size, but it is not treated like that: In [1]: from numpy import * In [2]: histogram(arange(10), [2,3,4], normed = True) Out[2]: (array([ 0.1, 0.1, 0.6]), array([2, 3, 4])) Even worse, if you try to add an infinite bin to the left, this pulls all values to zero (technically, I understand that, but it looks really undesirable to me): In [3]: histogram(arange(10), [-inf, 2,3,4], normed = True) Out[3]: (array([ 0., 0., 0., 0.]), array([-Inf, 2., 3., 4.])) -- Ciao, / / /--/ / / ANS From meine at informatik.uni-hamburg.de Tue Apr 8 04:21:41 2008 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Tue, 8 Apr 2008 10:21:41 +0200 Subject: [Numpy-discussion] Generically Creating Intermediate Data Compatible with Either ndarray or MasledArray Types In-Reply-To: <525f23e80803111257j1af05b84yaec145ad4d1a90a4@mail.gmail.com> References: <525f23e80803111210t14a4e334waf127107b4240baa@mail.gmail.com> <47D6E0BF.6050208@hawaii.edu> <525f23e80803111257j1af05b84yaec145ad4d1a90a4@mail.gmail.com> Message-ID: <200804081021.42453.meine@informatik.uni-hamburg.de> Am Dienstag, 11. M?rz 2008 20:57:30 schrieb Alexander Michael: > Incidentally, while ones_like appears to play nice with derived > classes, empty_like and zeros_like do not seem to do the same. Shouldn't this be fixed? (Obviously, this stems from the fact that ones_like is implemented in C, while the two others are helpers by fperez copied over from IPython.) Maybe using Robert's suggestion? (Patch attached.) Or implement them all the same? -- Ciao, / / /--/ / / ANS -------------- next part -------------- A non-text attachment was scrubbed... Name: numpy_xxx_like_preserve_type.diff Type: text/x-diff Size: 722 bytes Desc: not available URL: From meine at informatik.uni-hamburg.de Tue Apr 8 04:47:44 2008 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Tue, 8 Apr 2008 10:47:44 +0200 Subject: [Numpy-discussion] bug with with fill_values in masked arrays? In-Reply-To: <47E90D56.9020903@simplistix.co.uk> References: <47E0F2AC.7040200@simplistix.co.uk> <200803212024.40630.pgmdevlist@gmail.com> <47E90D56.9020903@simplistix.co.uk> Message-ID: <200804081047.45192.meine@informatik.uni-hamburg.de> Am Dienstag, 25. M?rz 2008 15:33:58 schrieb Chris Withers: > > Because in your particular case, you're inspecting elements one by one, > > and then, your masked data becomes the masked singleton which is a > > special value. > > I'd argue that the masked singleton having a different fill value to the > ma it comes from is a bug. Note that there's no "ma it comes from". It's a singleton. A special value. And your suggestion with isinstance would surely be less efficient than the current solution, since using the "is" operator for identity checking is as efficient as it gets. Just ignore the fill_value, which is only there for technical reason; it's unused in any case. Thanks to this discussion, I finally got an impression of ma. -- Ciao, / / /--/ / / ANS From steve at shrogers.com Tue Apr 8 08:15:25 2008 From: steve at shrogers.com (Steven H. Rogers) Date: Tue, 08 Apr 2008 06:15:25 -0600 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <9457e7c80804070922r67ae453cy4bf3473545c05692@mail.gmail.com> <20080407165740.GK9851@phare.normalesup.org> <36510.192.55.12.36.1207589204.squirrel@mail2.webfaction.com> <47FA65B3.4000307@gmail.com> <3d375d730804071129k19cab8efn5a22a6127d6aa424@mail.gmail.com> Message-ID: <47FB61DD.2070809@shrogers.com> Perry Greenfield wrote: > ... > Some random comments: > > 1) It seems to me that the primary problem people have with a big > flat namespace is that it makes the output of dir() long and > unusable, and by implication, that a nice hierarchical organization > would make it easier for people to find stuff. As to the latter, less > so than you might think. From what I've seen, there is no generally > agreed upon organization that all will agree to or find intuitive. > There will always be functions that logically belong to more than one > category. Ultimately, this is why flatter is better as far as that > goes. If one wants to find things by category, we would be much > better off tagging functions with categories and then building some > dir-like tool that helps display things in that category. Some have > already alluded to that (Joe Harrington I believe). The only thing > namespaces solve is name collisions imho. I don't believe that the > current numpy has too many names in its basic namespace, and it > already has split out some things into subpackages (fft, random, > linear algebra) that have such a potential. > At the IPython Sprint in Boulder last year Fernando suggested that someone look at this issue. I've given it some thought and started a wiki page for it. Inputs would be welcome and might motivate me to find the time to implement something. http://ipython.scipy.org/moin/Developer_Zone/SearchDocs # Steve From david.huard at gmail.com Tue Apr 8 09:12:06 2008 From: david.huard at gmail.com (David Huard) Date: Tue, 8 Apr 2008 09:12:06 -0400 Subject: [Numpy-discussion] Ticket #605 Incorrect behavior of numpy.histogram In-Reply-To: <200804081000.20749.meine@informatik.uni-hamburg.de> References: <200804071434.09201.meine@informatik.uni-hamburg.de> <200804081000.20749.meine@informatik.uni-hamburg.de> Message-ID: <91cf711d0804080612l4b8abat7c5c8117ed41d21b@mail.gmail.com> Hans, Note that the current histogram is buggy, in the sense that it assumes that all bins have the same width and computes db = bins[1]-bin[0]. This is why you get zeros everywhere. The current behavior has been heavily criticized and I think we should change it. My proposal is to have for histogram the same behavior as for histogramdd and histogram2d: bins are the bin edges, including the rightmost bin, and values outside of the bins are not tallied. The problem with this is that it breaks code, and I'm not sure it's such a good idea to do this in a point release. My short term proposal would be to fix the normalization bug and document the current behavior of histogram for the 1.0.5 release. Once it's done, we can modify histogram and maybe print a warning the first time it's used to notice users of the change. I'd like to hear the voice of experienced devs on this. This issue has been raised a number of times since I follow this ML. It's not the first time I've proposed patches, and I've already documented the weird behavior only to see the comments disappear after a while. I hope this time some kind of agreement will be reached. Regards, David 2008/4/8, Hans Meine : > > Am Montag, 07. April 2008 14:34:08 schrieb Hans Meine: > > > Am Samstag, 05. April 2008 21:54:27 schrieb Anne Archibald: > > > There's also a fourth option - raise an exception if any points are > > > outside the range. > > > > +1 > > > > I think this should be the default. Otherwise, I tend towards > "exclude", > > in order to have comparable bin sizes (when plotting, I always find > peaks > > at the ends annoying); this could also be called "clip" BTW. > > > > But really, an exception would follow the Zen: "In the face of > ambiguity, > > refuse the temptation to guess." And with a kwarg: "Explicit is better > > than implicit." > > > When posting this, I did indeed not think this through fully; as David > (and > Tommy) pointed out, this API does not fit well with the existing `bins` > option, especially when a sequence of bin bounds is given. (I guess I was > mostly thinking about the special case of discrete values and 1:1 bins, as > typical for uint8 data.) > > Thus, I would like to withdraw my above opinion from and instead state > that I > find the current API as clear as it gets. If you want to exclude values, > simply pass an additional right bound, and for including outliers, > passing -inf as additional left bound seems to do the trick. This could > be > possibly added to the documentation though. > > The only critical aspect I see is the `normed` arg. As it is now, the > rightmost bin has always infinite size, but it is not treated like that: > > In [1]: from numpy import * > > In [2]: histogram(arange(10), [2,3,4], normed = True) > Out[2]: (array([ 0.1, 0.1, 0.6]), array([2, 3, 4])) > > Even worse, if you try to add an infinite bin to the left, this pulls all > values to zero (technically, I understand that, but it looks really > undesirable to me): > > In [3]: histogram(arange(10), [-inf, 2,3,4], normed = True) > Out[3]: (array([ 0., 0., 0., 0.]), array([-Inf, 2., 3., 4.])) > > > -- > Ciao, / / > /--/ > / / ANS > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Tue Apr 8 09:28:54 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 8 Apr 2008 15:28:54 +0200 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <47FB61DD.2070809@shrogers.com> References: <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <9457e7c80804070922r67ae453cy4bf3473545c05692@mail.gmail.com> <20080407165740.GK9851@phare.normalesup.org> <36510.192.55.12.36.1207589204.squirrel@mail2.webfaction.com> <47FA65B3.4000307@gmail.com> <3d375d730804071129k19cab8efn5a22a6127d6aa424@mail.gmail.com> <47FB61DD.2070809@shrogers.com> Message-ID: <20080408132854.GA28183@phare.normalesup.org> On Tue, Apr 08, 2008 at 06:15:25AM -0600, Steven H. Rogers wrote: > At the IPython Sprint in Boulder last year Fernando suggested that > someone look at this issue. I've given it some thought and started a > wiki page for it. Inputs would be welcome and might motivate me to find > the time to implement something. > http://ipython.scipy.org/moin/Developer_Zone/SearchDocs A huge progress has been made in the direction of solving this problem with the release of sphinx (http://sphinx.pocoo.org/). This is a documentation-generation tool that creates html docs from restructured text. The docs are really nice, but a very interesting feature is that they have a "search" page, which works with javascript code on the client: no need for a server. For documenting Mayavi, I generated a function reference from the docstrings. It is very easy using the inspect module. Then I used my homebrewed rst comiler to build the docs. I plan to replace this with sphinx very soon. This will give us ability to search the docstrings and the full docs (dostrings cannot replace a user manual). Ipython and sympy have also decided to go the sphinx way. In fact Ondrej has already converted the ipython1 docs to sphinx, you can check them out on: http://ipython.scipy.org/doc/ipython1/html/ http://ipython.scipy.org/doc/ipython1/ipython1.pdf Sphinx seems a very promising to the everlasting problem of documentation. Cheers, Ga?l From kbasye1 at jhu.edu Tue Apr 8 10:06:15 2008 From: kbasye1 at jhu.edu (Ken Basye) Date: Tue, 08 Apr 2008 10:06:15 -0400 Subject: [Numpy-discussion] Array printing and another question Message-ID: <47FB7BD7.1020506@jhu.edu> Hi Folks, I'm pretty new to Numpy, and I hope someone can help me out with a couple newbie questions. First, I'd like, for diagnostic purposes, to print arrays of floating point numbers in a format of my choosing. Specifically, I have a formatting function that generates strings which represent all the bits of an FP number exactly, but it formats the exponent, exponent sign, and sign separately from the mantissa so I can get a rough idea of the magnitude at a glance. We use this to verify that certain operations are yielding exactly the same FP values either across platforms or across algorithmic changes. I'd like to be able to print numpy arrays in this format. At this point I've thought of three ways to try this. First, there's using set_string_function() and then writing my own version of array2string() that uses my function for FP arrays. OK, but I'd kind of like to avoid the work of duplicating the nice way that array2string() formats arrays of different ranks. Enticingly, I see code in core/arrayprint.py that's looking for an _format attribute and using it if it exists. So the second idea is to hook into this somehow, but I don't see how (indeed, I'm not sure when an array ever has an _format attribute?). Finally, I imagine I could make my own dtype so that array2string will format with %s, then convert FP arrays into arrays of that dtype for printing. Any ideas which of these would be better, or other ideas, would be most appreciated. Second, I wanted to find out how numpy developers and experts feel about cross-platform differences in FP behavior. Is this considered unavoidable and expected, a bug, or somewhere in between? Until recently I was about to get exact matches everywhere, but I now have at least one example of a difference. I should clarify that I'm not talking about processor-family dependency; all my machines are x86 at this point. Indeed, the differences appear across virtual machines running different OSs on the same physical machine. Thanks in advance, Ken Basye kbasye1 at jhu.edu From aisaac at american.edu Tue Apr 8 10:55:56 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 8 Apr 2008 10:55:56 -0400 Subject: [Numpy-discussion] Array printing and another question In-Reply-To: <47FB7BD7.1020506@jhu.edu> References: <47FB7BD7.1020506@jhu.edu> Message-ID: Will set_printoptions not work for you? hth, Alan Isaac From kbasye1 at jhu.edu Tue Apr 8 11:22:33 2008 From: kbasye1 at jhu.edu (Ken Basye) Date: Tue, 08 Apr 2008 11:22:33 -0400 Subject: [Numpy-discussion] Array printing and another question In-Reply-To: References: <47FB7BD7.1020506@jhu.edu> Message-ID: <47FB8DB9.3030700@jhu.edu> Hi, Thanks, but it's been my experience that formatting FP numbers into decimal causes a lot of false alarms all by itself; that is, you get different decimal representations of the same FP memory value. I've had this happen often enough that I found the first thing I did when an output difference arose was to print the FP in hex to see if the difference was "real" or just a formatting artifact. This formatting function is designed to do that all the time, while providing enough clarity to examine values. Here are a couple examples: >>> pi = 3.14159 >>> float_to_readable_string(-100*pi) '-(+0008)0x3a28b43958106' # Note that the exponent is a decimal value with a base of 2, so "8" is 256. >>> float_to_readable_string(0.0) '+(-1023)0x0000000000000' These strings can also be converted back to FP with absolute accuracy, which is almost impossible to guarantee with decimal representations, in my experience. Ken Alan G Isaac wrote: > Will set_printoptions not work for you? > > > hth, > Alan Isaac > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From bsouthey at gmail.com Tue Apr 8 13:27:08 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Tue, 08 Apr 2008 12:27:08 -0500 Subject: [Numpy-discussion] Ticket #605 Incorrect behavior of numpy.histogram In-Reply-To: <91cf711d0804080612l4b8abat7c5c8117ed41d21b@mail.gmail.com> References: <200804071434.09201.meine@informatik.uni-hamburg.de> <200804081000.20749.meine@informatik.uni-hamburg.de> <91cf711d0804080612l4b8abat7c5c8117ed41d21b@mail.gmail.com> Message-ID: <47FBAAEC.7020306@gmail.com> Hi, I agree that the current histogram should be changed. However, I am not sure 1.0.5 is the correct release for that. David, this doesn't work for your code: r= np.array([1,2,2,3,3,3,4,4,4,4,5,5,5,5,5]) dbin=[2,3,4] rc, rb=histogram(r, bins=dbin, discard=None) Returns: rc=[3 3] # Really should be [3, 3, 9] rb=[-9223372036854775808 3 -9223372036854775808] But I have not had time to find the error. Regards Bruce David Huard wrote: > Hans, > > Note that the current histogram is buggy, in the sense that it assumes > that all bins have the same width and computes db = bins[1]-bin[0]. > This is why you get zeros everywhere. > > The current behavior has been heavily criticized and I think we should > change it. My proposal is to have for histogram the same behavior as > for histogramdd and histogram2d: bins are the bin edges, including the > rightmost bin, and values outside of the bins are not tallied. The > problem with this is that it breaks code, and I'm not sure it's such a > good idea to do this in a point release. > > My short term proposal would be to fix the normalization bug and > document the current behavior of histogram for the 1.0.5 release. Once > it's done, we can modify histogram and maybe print a warning the first > time it's used to notice users of the change. > > I'd like to hear the voice of experienced devs on this. This issue has > been raised a number of times since I follow this ML. It's not the > first time I've proposed patches, and I've already documented the > weird behavior only to see the comments disappear after a while. I > hope this time some kind of agreement will be reached. > > Regards, > > David > > > > > 2008/4/8, Hans Meine >: > > Am Montag, 07. April 2008 14:34:08 schrieb Hans Meine: > > > Am Samstag, 05. April 2008 21:54:27 schrieb Anne Archibald: > > > There's also a fourth option - raise an exception if any > points are > > > outside the range. > > > > +1 > > > > I think this should be the default. Otherwise, I tend towards > "exclude", > > in order to have comparable bin sizes (when plotting, I always > find peaks > > at the ends annoying); this could also be called "clip" BTW. > > > > But really, an exception would follow the Zen: "In the face of > ambiguity, > > refuse the temptation to guess." And with a kwarg: "Explicit is > better > > than implicit." > > > When posting this, I did indeed not think this through fully; as > David (and > Tommy) pointed out, this API does not fit well with the existing > `bins` > option, especially when a sequence of bin bounds is given. (I > guess I was > mostly thinking about the special case of discrete values and 1:1 > bins, as > typical for uint8 data.) > > Thus, I would like to withdraw my above opinion from and instead > state that I > find the current API as clear as it gets. If you want to exclude > values, > simply pass an additional right bound, and for including outliers, > passing -inf as additional left bound seems to do the trick. This > could be > possibly added to the documentation though. > > The only critical aspect I see is the `normed` arg. As it is now, the > rightmost bin has always infinite size, but it is not treated like > that: > > In [1]: from numpy import * > > In [2]: histogram(arange(10), [2,3,4], normed = True) > Out[2]: (array([ 0.1, 0.1, 0.6]), array([2, 3, 4])) > > Even worse, if you try to add an infinite bin to the left, this > pulls all > values to zero (technically, I understand that, but it looks really > undesirable to me): > > In [3]: histogram(arange(10), [-inf, 2,3,4], normed = True) > Out[3]: (array([ 0., 0., 0., 0.]), array([-Inf, 2., 3., 4.])) > > > -- > Ciao, / / > /--/ > / / ANS > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From nwagner at iam.uni-stuttgart.de Tue Apr 8 15:03:45 2008 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Tue, 08 Apr 2008 21:03:45 +0200 Subject: [Numpy-discussion] test_financial.py Message-ID: Hi all, several test failures are present in svn File "/usr/lib/python2.4/site-packages/numpy/lib/tests/test_financial.py", line 4, in test_financial Failed example: rate(10,0,-3500,10000) Exception raised: Traceback (most recent call last): File "/usr/lib/python2.4/doctest.py", line 1243, in __run compileflags, 1) in test.globs File "", line 1, in ? rate(10,0,-3500,10000) NameError: name 'rate' is not defined ********************************************************************** File "/usr/lib/python2.4/site-packages/numpy/lib/tests/test_financial.py", line 7, in test_financial Failed example: irr([-150000, 15000, 25000, 35000, 45000, 60000]) Exception raised: Traceback (most recent call last): File "/usr/lib/python2.4/doctest.py", line 1243, in __run compileflags, 1) in test.globs File "", line 1, in ? irr([-150000, 15000, 25000, 35000, 45000, 60000]) NameError: name 'irr' is not defined ********************************************************************** File "/usr/lib/python2.4/site-packages/numpy/lib/tests/test_financial.py", line 10, in test_financial Failed example: pv(0.07,20,12000,0) Exception raised: Traceback (most recent call last): File "/usr/lib/python2.4/doctest.py", line 1243, in __run compileflags, 1) in test.globs File "", line 1, in ? pv(0.07,20,12000,0) NameError: name 'pv' is not defined ********************************************************************** File "/usr/lib/python2.4/site-packages/numpy/lib/tests/test_financial.py", line 13, in test_financial Failed example: fv(0.075, 20, -2000,0,0) Exception raised: Traceback (most recent call last): File "/usr/lib/python2.4/doctest.py", line 1243, in __run compileflags, 1) in test.globs File "", line 1, in ? fv(0.075, 20, -2000,0,0) NameError: name 'fv' is not defined ********************************************************************** File "/usr/lib/python2.4/site-packages/numpy/lib/tests/test_financial.py", line 16, in test_financial Failed example: pmt(0.08/12,5*12,15000) Exception raised: Traceback (most recent call last): File "/usr/lib/python2.4/doctest.py", line 1243, in __run compileflags, 1) in test.globs File "", line 1, in ? pmt(0.08/12,5*12,15000) NameError: name 'pmt' is not defined ********************************************************************** File "/usr/lib/python2.4/site-packages/numpy/lib/tests/test_financial.py", line 19, in test_financial Failed example: nper(0.075,-2000,0,100000.) Exception raised: Traceback (most recent call last): File "/usr/lib/python2.4/doctest.py", line 1243, in __run compileflags, 1) in test.globs File "", line 1, in ? nper(0.075,-2000,0,100000.) NameError: name 'nper' is not defined ********************************************************************** File "/usr/lib/python2.4/site-packages/numpy/lib/tests/test_financial.py", line 22, in test_financial Failed example: npv(0.05,[-15000,1500,2500,3500,4500,6000]) Exception raised: Traceback (most recent call last): File "/usr/lib/python2.4/doctest.py", line 1243, in __run compileflags, 1) in test.globs File "", line 1, in ? npv(0.05,[-15000,1500,2500,3500,4500,6000]) NameError: name 'npv' is not defined ********************************************************************** File "/usr/lib/python2.4/site-packages/numpy/lib/tests/test_financial.py", line 25, in test_financial Failed example: mirr([-4500,-800,800,800,600,600,800,800,700,3000],0.08,0.055) Exception raised: Traceback (most recent call last): File "/usr/lib/python2.4/doctest.py", line 1243, in __run compileflags, 1) in test.globs File "", line 1, in ? mirr([-4500,-800,800,800,600,600,800,800,700,3000],0.08,0.055) NameError: name 'mirr' is not defined ********************************************************************** File "/usr/lib/python2.4/site-packages/numpy/lib/tests/test_financial.py", line 28, in test_financial Failed example: mirr([-120000,39000,30000,21000,37000,46000],0.10,0.12) Exception raised: Traceback (most recent call last): File "/usr/lib/python2.4/doctest.py", line 1243, in __run compileflags, 1) in test.globs File "", line 1, in ? mirr([-120000,39000,30000,21000,37000,46000],0.10,0.12) NameError: name 'mirr' is not defined Nils From david.huard at gmail.com Tue Apr 8 15:25:00 2008 From: david.huard at gmail.com (David Huard) Date: Tue, 8 Apr 2008 15:25:00 -0400 Subject: [Numpy-discussion] Ticket #605 Incorrect behavior of numpy.histogram In-Reply-To: <47FBAAEC.7020306@gmail.com> References: <200804071434.09201.meine@informatik.uni-hamburg.de> <200804081000.20749.meine@informatik.uni-hamburg.de> <91cf711d0804080612l4b8abat7c5c8117ed41d21b@mail.gmail.com> <47FBAAEC.7020306@gmail.com> Message-ID: <91cf711d0804081225t196a9730v460d8955b60a2686@mail.gmail.com> 2008/4/8, Bruce Southey : > > Hi, > I agree that the current histogram should be changed. However, I am not > sure 1.0.5 is the correct release for that. We both agree. David, this doesn't work for your code: > r= np.array([1,2,2,3,3,3,4,4,4,4,5,5,5,5,5]) > dbin=[2,3,4] > rc, rb=histogram(r, bins=dbin, discard=None) Returns: > rc=[3 3] # Really should be [3, 3, 9] > rb=[-9223372036854775808 3 -9223372036854775808] I used the convention that bins are the bin edges, including the right most edge, this is why len(rc) =2 and len(rb)=3. Now there clearly is a bug, and I traced it to the use of np.r_. Check this out: In [26]: dbin = [1,2,3] In [27]: np.r_[-np.inf, dbin, np.inf] Out[27]: array([-Inf, 1., 2., 3., Inf]) In [28]: np.r_[-np.inf, asarray(dbin), np.inf] Out[28]: array([-9223372036854775808, 1, 2, 3, -9223372036854775808]) In [29]: np.r_[-np.inf, asarray(dbin).astype(float), np.inf] Out[29]: array([-Inf, 1., 2., 3., Inf]) Is this a misuse of r_ or a bug ? David But I have not had time to find the error. > > Regards > Bruce > > > > David Huard wrote: > > Hans, > > > > Note that the current histogram is buggy, in the sense that it assumes > > that all bins have the same width and computes db = bins[1]-bin[0]. > > This is why you get zeros everywhere. > > > > The current behavior has been heavily criticized and I think we should > > change it. My proposal is to have for histogram the same behavior as > > for histogramdd and histogram2d: bins are the bin edges, including the > > rightmost bin, and values outside of the bins are not tallied. The > > problem with this is that it breaks code, and I'm not sure it's such a > > good idea to do this in a point release. > > > > My short term proposal would be to fix the normalization bug and > > document the current behavior of histogram for the 1.0.5 release. Once > > it's done, we can modify histogram and maybe print a warning the first > > time it's used to notice users of the change. > > > > I'd like to hear the voice of experienced devs on this. This issue has > > been raised a number of times since I follow this ML. It's not the > > first time I've proposed patches, and I've already documented the > > weird behavior only to see the comments disappear after a while. I > > hope this time some kind of agreement will be reached. > > > > Regards, > > > > David > > > > > > > > > > 2008/4/8, Hans Meine > > >: > > > > > Am Montag, 07. April 2008 14:34:08 schrieb Hans Meine: > > > > > Am Samstag, 05. April 2008 21:54:27 schrieb Anne Archibald: > > > > There's also a fourth option - raise an exception if any > > points are > > > > outside the range. > > > > > > +1 > > > > > > I think this should be the default. Otherwise, I tend towards > > "exclude", > > > in order to have comparable bin sizes (when plotting, I always > > find peaks > > > at the ends annoying); this could also be called "clip" BTW. > > > > > > But really, an exception would follow the Zen: "In the face of > > ambiguity, > > > refuse the temptation to guess." And with a kwarg: "Explicit is > > better > > > than implicit." > > > > > > When posting this, I did indeed not think this through fully; as > > David (and > > Tommy) pointed out, this API does not fit well with the existing > > `bins` > > option, especially when a sequence of bin bounds is given. (I > > guess I was > > mostly thinking about the special case of discrete values and 1:1 > > bins, as > > typical for uint8 data.) > > > > Thus, I would like to withdraw my above opinion from and instead > > state that I > > find the current API as clear as it gets. If you want to exclude > > values, > > simply pass an additional right bound, and for including outliers, > > passing -inf as additional left bound seems to do the trick. This > > could be > > possibly added to the documentation though. > > > > The only critical aspect I see is the `normed` arg. As it is now, > the > > rightmost bin has always infinite size, but it is not treated like > > that: > > > > In [1]: from numpy import * > > > > In [2]: histogram(arange(10), [2,3,4], normed = True) > > Out[2]: (array([ 0.1, 0.1, 0.6]), array([2, 3, 4])) > > > > Even worse, if you try to add an infinite bin to the left, this > > pulls all > > values to zero (technically, I understand that, but it looks really > > undesirable to me): > > > > In [3]: histogram(arange(10), [-inf, 2,3,4], normed = True) > > Out[3]: (array([ 0., 0., 0., 0.]), array([-Inf, 2., 3., > 4.])) > > > > > > -- > > Ciao, / / > > /--/ > > / / ANS > > _______________________________________________ > > Numpy-discussion mailing list > > > Numpy-discussion at scipy.org > > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > ------------------------------------------------------------------------ > > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at shrogers.com Tue Apr 8 17:23:21 2008 From: steve at shrogers.com (Steven H. Rogers) Date: Tue, 8 Apr 2008 15:23:21 -0600 (MDT) Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <20080408132854.GA28183@phare.normalesup.org> References: <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <9457e7c80804070922r67ae453cy4bf3473545c05692@mail.gmail.com> <20080407165740.GK9851@phare.normalesup.org> <36510.192.55.12.36.1207589204.squirrel@mail2.webfaction.com> <47FA65B3.4000307@gmail.com> <3d375d730804071129k19cab8efn5a22a6127d6aa424@mail.gmail.com> <47FB61DD.2070809@shrogers.com> <20080408132854.GA28183@phare.normalesup.org> Message-ID: <58783.192.55.12.36.1207689801.squirrel@mail2.webfaction.com> On Tue, April 8, 2008 07:28, Gael Varoquaux wrote: > ... > > http://ipython.scipy.org/doc/ipython1/html/ > http://ipython.scipy.org/doc/ipython1/ipython1.pdf > > Sphinx seems a very promising to the everlasting problem of documentation. > Thanks for bringing this up. I need to look at Sphinx. Regards, Steve From fperez.net at gmail.com Tue Apr 8 17:32:23 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 8 Apr 2008 14:32:23 -0700 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <58783.192.55.12.36.1207689801.squirrel@mail2.webfaction.com> References: <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407165740.GK9851@phare.normalesup.org> <36510.192.55.12.36.1207589204.squirrel@mail2.webfaction.com> <47FA65B3.4000307@gmail.com> <3d375d730804071129k19cab8efn5a22a6127d6aa424@mail.gmail.com> <47FB61DD.2070809@shrogers.com> <20080408132854.GA28183@phare.normalesup.org> <58783.192.55.12.36.1207689801.squirrel@mail2.webfaction.com> Message-ID: On Tue, Apr 8, 2008 at 2:23 PM, Steven H. Rogers wrote: > On Tue, April 8, 2008 07:28, Gael Varoquaux wrote: > > ... > > > > > http://ipython.scipy.org/doc/ipython1/html/ > > http://ipython.scipy.org/doc/ipython1/ipython1.pdf > > > > Sphinx seems a very promising to the everlasting problem of documentation. > > > Thanks for bringing this up. I need to look at Sphinx. Just to mention that this doesn't necessarily override the need for a faster/more roubst database-backed doc index, but it certainly provides a lot of funcitonality out of the box for end users, which is one of the reasons I'm so excited (and grateful) about it. If more of our combined/related projects use sphinx, users will have a reasonably uniform doc-searching experience with high-quality (at least visually :) docs, and PDF to boot. Cheers, f From fperez.net at gmail.com Tue Apr 8 19:00:17 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 8 Apr 2008 16:00:17 -0700 Subject: [Numpy-discussion] numpyx.pyx (recent svn) works? In-Reply-To: <3d375d730802280817h6271fcf6gcd027b1c774af855@mail.gmail.com> References: <3d375d730802280817h6271fcf6gcd027b1c774af855@mail.gmail.com> Message-ID: Hey, On Thu, Feb 28, 2008 at 9:17 AM, Robert Kern wrote: > On Thu, Feb 28, 2008 at 7:35 AM, Neal Becker wrote: > > I tried numpyx.pyx with cython-0.9.6.12. > > These were written for and still work with Pyrex. If it doesn't work > with Cython then that is either a bug in Cython or an intentional > incompatibility of Cython. I just updated all this code to run with cython here: https://code.launchpad.net/~fdo.perez/+junk/numpy-cython Do we want to just switch over to cython and use this instead? If people think that the cython switch is OK for 1.0.5 I can do it, but I'm not about to touch this before others approve... Cheers, f From stefan at sun.ac.za Tue Apr 8 19:15:23 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 9 Apr 2008 01:15:23 +0200 Subject: [Numpy-discussion] numpyx.pyx (recent svn) works? In-Reply-To: References: <3d375d730802280817h6271fcf6gcd027b1c774af855@mail.gmail.com> Message-ID: <9457e7c80804081615y341f574ai20cae30d6ee0905e@mail.gmail.com> On 09/04/2008, Fernando Perez wrote: > Do we want to just switch over to cython and use this instead? If > people think that the cython switch is OK for 1.0.5 I can do it, but > I'm not about to touch this before others approve... I'm in favour of switching -- pyrex is as good as unmaintained, and in contrast cython is improving by the day. Do the changes cause problems with pyrex, or can we even have both working at the same time? Either way, we should add a note to the docstring. Regards St?fan From gael.varoquaux at normalesup.org Tue Apr 8 19:17:14 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 9 Apr 2008 01:17:14 +0200 Subject: [Numpy-discussion] numpyx.pyx (recent svn) works? In-Reply-To: <9457e7c80804081615y341f574ai20cae30d6ee0905e@mail.gmail.com> References: <3d375d730802280817h6271fcf6gcd027b1c774af855@mail.gmail.com> <9457e7c80804081615y341f574ai20cae30d6ee0905e@mail.gmail.com> Message-ID: <20080408231714.GA16696@phare.normalesup.org> On Wed, Apr 09, 2008 at 01:15:23AM +0200, St?fan van der Walt wrote: > I'm in favour of switching -- pyrex is as good as unmaintained, and in > contrast cython is improving by the day. Do the changes cause > problems with pyrex, or can we even have both working at the same > time? Either way, we should add a note to the docstring. How about waiting for after 1.0.5 release. It should be anytime soon now, and let us not risk breaking the workflow of potential contributors for little gain. I have little to say on the subject, as I am not helping to get the release out, but here are my 2 cents. Ga?l From robert.kern at gmail.com Tue Apr 8 19:20:28 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 8 Apr 2008 16:20:28 -0700 Subject: [Numpy-discussion] numpyx.pyx (recent svn) works? In-Reply-To: <20080408231714.GA16696@phare.normalesup.org> References: <3d375d730802280817h6271fcf6gcd027b1c774af855@mail.gmail.com> <9457e7c80804081615y341f574ai20cae30d6ee0905e@mail.gmail.com> <20080408231714.GA16696@phare.normalesup.org> Message-ID: <3d375d730804081620g74f7a513pa858b7c5ba4fde34@mail.gmail.com> On Tue, Apr 8, 2008 at 4:17 PM, Gael Varoquaux wrote: > On Wed, Apr 09, 2008 at 01:15:23AM +0200, St?fan van der Walt wrote: > > I'm in favour of switching -- pyrex is as good as unmaintained, and in > > contrast cython is improving by the day. Do the changes cause > > problems with pyrex, or can we even have both working at the same > > time? Either way, we should add a note to the docstring. > > How about waiting for after 1.0.5 release. It should be anytime soon now, > and let us not risk breaking the workflow of potential contributors for > little gain. I'm not sure how it could. It's example code, not part of numpy itself. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From gael.varoquaux at normalesup.org Tue Apr 8 19:21:10 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 9 Apr 2008 01:21:10 +0200 Subject: [Numpy-discussion] numpyx.pyx (recent svn) works? In-Reply-To: <3d375d730804081620g74f7a513pa858b7c5ba4fde34@mail.gmail.com> References: <3d375d730802280817h6271fcf6gcd027b1c774af855@mail.gmail.com> <9457e7c80804081615y341f574ai20cae30d6ee0905e@mail.gmail.com> <20080408231714.GA16696@phare.normalesup.org> <3d375d730804081620g74f7a513pa858b7c5ba4fde34@mail.gmail.com> Message-ID: <20080408232110.GB16696@phare.normalesup.org> On Tue, Apr 08, 2008 at 04:20:28PM -0700, Robert Kern wrote: > I'm not sure how it could. It's example code, not part of numpy itself. OK, maybe I should keep my 2 cents. They appear to be forged money, and worth nothing. :-). Ga?l From fperez.net at gmail.com Tue Apr 8 19:28:11 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 8 Apr 2008 16:28:11 -0700 Subject: [Numpy-discussion] numpyx.pyx (recent svn) works? In-Reply-To: <20080408232110.GB16696@phare.normalesup.org> References: <3d375d730802280817h6271fcf6gcd027b1c774af855@mail.gmail.com> <9457e7c80804081615y341f574ai20cae30d6ee0905e@mail.gmail.com> <20080408231714.GA16696@phare.normalesup.org> <3d375d730804081620g74f7a513pa858b7c5ba4fde34@mail.gmail.com> <20080408232110.GB16696@phare.normalesup.org> Message-ID: On Tue, Apr 8, 2008 at 4:21 PM, Gael Varoquaux wrote: > On Tue, Apr 08, 2008 at 04:20:28PM -0700, Robert Kern wrote: > > I'm not sure how it could. It's example code, not part of numpy itself. > > OK, maybe I should keep my 2 cents. They appear to be forged money, and > worth nothing. > > :-). I *could* make it pyrex/cython-valid, it's trivial but just adds noise IMHO... As Stefan said, pyrex is essentially unmaintained as far as publicly-visible development goes, while cython is very actively moving ahead and likely picking up better numpy support soon (thanks to Dag and other GSoC work), so why not just follow that? Cheers, f From gael.varoquaux at normalesup.org Tue Apr 8 19:31:47 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 9 Apr 2008 01:31:47 +0200 Subject: [Numpy-discussion] numpyx.pyx (recent svn) works? In-Reply-To: References: <3d375d730802280817h6271fcf6gcd027b1c774af855@mail.gmail.com> <9457e7c80804081615y341f574ai20cae30d6ee0905e@mail.gmail.com> <20080408231714.GA16696@phare.normalesup.org> <3d375d730804081620g74f7a513pa858b7c5ba4fde34@mail.gmail.com> <20080408232110.GB16696@phare.normalesup.org> Message-ID: <20080408233147.GC16696@phare.normalesup.org> On Tue, Apr 08, 2008 at 04:28:11PM -0700, Fernando Perez wrote: > I *could* make it pyrex/cython-valid, it's trivial but just adds noise > IMHO... As Stefan said, pyrex is essentially unmaintained as far as > publicly-visible development goes, while cython is very actively > moving ahead and likely picking up better numpy support soon (thanks > to Dag and other GSoC work), so why not just follow that? Because Cython is not yet shipped be all the major distros (including not Enthought's EPD current Beta, AFAIK), so it rules out a fair amount of people who cannot/do not want to install something not shipped. Note that I don't care that much amount this issue. I am just explaining why I think it should be after 1.0.5 ships. Cheers, Ga?l From fperez.net at gmail.com Tue Apr 8 19:36:29 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 8 Apr 2008 16:36:29 -0700 Subject: [Numpy-discussion] numpyx.pyx (recent svn) works? In-Reply-To: <20080408233147.GC16696@phare.normalesup.org> References: <3d375d730802280817h6271fcf6gcd027b1c774af855@mail.gmail.com> <9457e7c80804081615y341f574ai20cae30d6ee0905e@mail.gmail.com> <20080408231714.GA16696@phare.normalesup.org> <3d375d730804081620g74f7a513pa858b7c5ba4fde34@mail.gmail.com> <20080408232110.GB16696@phare.normalesup.org> <20080408233147.GC16696@phare.normalesup.org> Message-ID: On Tue, Apr 8, 2008 at 4:31 PM, Gael Varoquaux wrote: > On Tue, Apr 08, 2008 at 04:28:11PM -0700, Fernando Perez wrote: > > I *could* make it pyrex/cython-valid, it's trivial but just adds noise > > IMHO... As Stefan said, pyrex is essentially unmaintained as far as > > publicly-visible development goes, while cython is very actively > > moving ahead and likely picking up better numpy support soon (thanks > > to Dag and other GSoC work), so why not just follow that? > > Because Cython is not yet shipped be all the major distros (including not > Enthought's EPD current Beta, AFAIK), so it rules out a fair amount of > people who cannot/do not want to install something not shipped. Unfortunately this is a case where the tool that is distributed (pyrex) has serious limitations for our purposes, and will hopefully soon (as Cython grows numpy support) be simply unusable: anyone wanting numpy/.pyx use will HAVE to use cython. So in this case, the above argument doesn't really apply: if the shipped tool simply doesn't do what you need, having it available by default does you zero good. Cheers, f From oliphant at enthought.com Tue Apr 8 19:38:09 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 08 Apr 2008 18:38:09 -0500 Subject: [Numpy-discussion] numpyx.pyx (recent svn) works? In-Reply-To: References: <3d375d730802280817h6271fcf6gcd027b1c774af855@mail.gmail.com> <9457e7c80804081615y341f574ai20cae30d6ee0905e@mail.gmail.com> <20080408231714.GA16696@phare.normalesup.org> <3d375d730804081620g74f7a513pa858b7c5ba4fde34@mail.gmail.com> <20080408232110.GB16696@phare.normalesup.org> Message-ID: <47FC01E1.3080400@enthought.com> Fernando Perez wrote: > On Tue, Apr 8, 2008 at 4:21 PM, Gael Varoquaux > wrote: > >> On Tue, Apr 08, 2008 at 04:20:28PM -0700, Robert Kern wrote: >> > I'm not sure how it could. It's example code, not part of numpy itself. >> >> OK, maybe I should keep my 2 cents. They appear to be forged money, and >> worth nothing. >> >> :-). >> > > I *could* make it pyrex/cython-valid, it's trivial but just adds noise > IMHO... As Stefan said, pyrex is essentially unmaintained as far as > publicly-visible development goes, while cython is very actively > moving ahead and likely picking up better numpy support soon (thanks > to Dag and other GSoC work), so why not just follow that? > I say just add it. We should move forward with Cython. More important is to see if random actually builds with Cython right now. There was an issue that I recall from a few weeks ago that Cython could not build the pyrex extension in NumPy. -Travis From fperez.net at gmail.com Tue Apr 8 19:55:57 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 8 Apr 2008 16:55:57 -0700 Subject: [Numpy-discussion] numpyx.pyx (recent svn) works? In-Reply-To: <47FC01E1.3080400@enthought.com> References: <3d375d730802280817h6271fcf6gcd027b1c774af855@mail.gmail.com> <9457e7c80804081615y341f574ai20cae30d6ee0905e@mail.gmail.com> <20080408231714.GA16696@phare.normalesup.org> <3d375d730804081620g74f7a513pa858b7c5ba4fde34@mail.gmail.com> <20080408232110.GB16696@phare.normalesup.org> <47FC01E1.3080400@enthought.com> Message-ID: On Tue, Apr 8, 2008 at 4:38 PM, Travis E. Oliphant wrote: > I say just add it. We should move forward with Cython. More important > is to see if random actually builds with Cython right now. There was an > issue that I recall from a few weeks ago that Cython could not build the > pyrex extension in NumPy. OK, I'll play with random for a bit. BTW, the original 'bug' that started this thread is due to a change in Cython's casting behavior explained here: http://wiki.cython.org/DifferencesFromPyrex it's fixed with a simple extra (void *) cast as shown here: print 'Printing array info for ndarray at 0x%0lx'% \ (arr,) I just committed that code to the bzr branch. In summary: 1. Do you want me to obliterate the old numpy/doc/pyrex and replace it with numpy/doc/cython? That would make it clear what the tool to use is... 2. I'll work on random and report shortly. Cheers, f From strawman at astraw.com Tue Apr 8 21:52:46 2008 From: strawman at astraw.com (Andrew Straw) Date: Tue, 08 Apr 2008 18:52:46 -0700 Subject: [Numpy-discussion] numpyx.pyx (recent svn) works? In-Reply-To: References: <3d375d730802280817h6271fcf6gcd027b1c774af855@mail.gmail.com> <9457e7c80804081615y341f574ai20cae30d6ee0905e@mail.gmail.com> <20080408231714.GA16696@phare.normalesup.org> <3d375d730804081620g74f7a513pa858b7c5ba4fde34@mail.gmail.com> <20080408232110.GB16696@phare.normalesup.org> <47FC01E1.3080400@enthought.com> Message-ID: <47FC216E.7040609@astraw.com> This is off-topic and should be directed to the pyrex/cython list, but since we're on the subject: I suppose the following is true, but let me ask, since I have not used Cython. Please correct me if I'm wrong. I have a bunch of pyrex compiled .pyx code. If I start adding some Cython compiled code, all the extensions will live happily without colliding, even if I don't re-compile my old .pyx files with Cython. (I can't imagine how they'd conflict, since they both translate to .c extensions without anything to collide. Or are there linker-level symbols that collide or something difficult?) -Andrew Fernando Perez wrote: > On Tue, Apr 8, 2008 at 4:38 PM, Travis E. Oliphant > wrote: > > >> I say just add it. We should move forward with Cython. More important >> is to see if random actually builds with Cython right now. There was an >> issue that I recall from a few weeks ago that Cython could not build the >> pyrex extension in NumPy. >> > > OK, I'll play with random for a bit. > > BTW, the original 'bug' that started this thread is due to a change in > Cython's casting behavior explained here: > > http://wiki.cython.org/DifferencesFromPyrex > > it's fixed with a simple extra (void *) cast as shown here: > > print 'Printing array info for ndarray at 0x%0lx'% \ > (arr,) > > > I just committed that code to the bzr branch. > > In summary: > > 1. Do you want me to obliterate the old numpy/doc/pyrex and replace it > with numpy/doc/cython? That would make it clear what the tool to use > is... > > 2. I'll work on random and report shortly. > > Cheers, > > f > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From fperez.net at gmail.com Tue Apr 8 21:56:39 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 8 Apr 2008 18:56:39 -0700 Subject: [Numpy-discussion] numpyx.pyx (recent svn) works? In-Reply-To: References: <3d375d730802280817h6271fcf6gcd027b1c774af855@mail.gmail.com> <9457e7c80804081615y341f574ai20cae30d6ee0905e@mail.gmail.com> <20080408231714.GA16696@phare.normalesup.org> <3d375d730804081620g74f7a513pa858b7c5ba4fde34@mail.gmail.com> <20080408232110.GB16696@phare.normalesup.org> <47FC01E1.3080400@enthought.com> Message-ID: Howdy, On Tue, Apr 8, 2008 at 4:55 PM, Fernando Perez wrote: > 1. Do you want me to obliterate the old numpy/doc/pyrex and replace it > with numpy/doc/cython? That would make it clear what the tool to use > is... OK, the code is ready and I just updated the docs/makefile a little to make it more new user-friendly. I went ahead and made the commit here: http://projects.scipy.org/scipy/numpy/changeset/4994 Instead of deleting the old pyrex dir, I just put deprecation markers in there. I can remove it if everyone agrees, but I figured it would be best to be careful (this is not code I work on regularly...) Cheers, f From fperez.net at gmail.com Tue Apr 8 21:57:46 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 8 Apr 2008 18:57:46 -0700 Subject: [Numpy-discussion] numpyx.pyx (recent svn) works? In-Reply-To: <47FC216E.7040609@astraw.com> References: <9457e7c80804081615y341f574ai20cae30d6ee0905e@mail.gmail.com> <20080408231714.GA16696@phare.normalesup.org> <3d375d730804081620g74f7a513pa858b7c5ba4fde34@mail.gmail.com> <20080408232110.GB16696@phare.normalesup.org> <47FC01E1.3080400@enthought.com> <47FC216E.7040609@astraw.com> Message-ID: On Tue, Apr 8, 2008 at 6:52 PM, Andrew Straw wrote: > This is off-topic and should be directed to the pyrex/cython list, but > since we're on the subject: > > I suppose the following is true, but let me ask, since I have not used > Cython. Please correct me if I'm wrong. > > I have a bunch of pyrex compiled .pyx code. If I start adding some > Cython compiled code, all the extensions will live happily without > colliding, even if I don't re-compile my old .pyx files with Cython. > > (I can't imagine how they'd conflict, since they both translate to .c > extensions without anything to collide. Or are there linker-level > symbols that collide or something difficult?) I think you're correct, I fail to see how it could be any different. But I'm far from a cython expert, so I may be missing something. cheers, f From steve at shrogers.com Tue Apr 8 22:19:09 2008 From: steve at shrogers.com (Steven H. Rogers) Date: Tue, 08 Apr 2008 20:19:09 -0600 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: References: <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407165740.GK9851@phare.normalesup.org> <36510.192.55.12.36.1207589204.squirrel@mail2.webfaction.com> <47FA65B3.4000307@gmail.com> <3d375d730804071129k19cab8efn5a22a6127d6aa424@mail.gmail.com> <47FB61DD.2070809@shrogers.com> <20080408132854.GA28183@phare.normalesup.org> <58783.192.55.12.36.1207689801.squirrel@mail2.webfaction.com> Message-ID: <47FC279D.5060206@shrogers.com> Fernando Perez wrote: > On Tue, Apr 8, 2008 at 2:23 PM, Steven H. Rogers wrote: > >> On Tue, April 8, 2008 07:28, Gael Varoquaux wrote: >> > ... >> >> >> > http://ipython.scipy.org/doc/ipython1/html/ >> > http://ipython.scipy.org/doc/ipython1/ipython1.pdf >> > >> > Sphinx seems a very promising to the everlasting problem of documentation. >> > >> Thanks for bringing this up. I need to look at Sphinx. >> > > Just to mention that this doesn't necessarily override the need for a > faster/more roubst database-backed doc index, but it certainly > provides a lot of funcitonality out of the box for end users, which is > one of the reasons I'm so excited (and grateful) about it. If more of > our combined/related projects use sphinx, users will have a reasonably > uniform doc-searching experience with high-quality (at least visually > :) docs, and PDF to boot. > > Cheers, > > f There's definitely a place for docsearch from the shell. There's opportunity for context sensitive search that Sphinx generated docs couldn't easily duplicate. # Steve From fperez.net at gmail.com Wed Apr 9 02:25:13 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 8 Apr 2008 23:25:13 -0700 Subject: [Numpy-discussion] numpyx.pyx (recent svn) works? In-Reply-To: References: <3d375d730802280817h6271fcf6gcd027b1c774af855@mail.gmail.com> <9457e7c80804081615y341f574ai20cae30d6ee0905e@mail.gmail.com> <20080408231714.GA16696@phare.normalesup.org> <3d375d730804081620g74f7a513pa858b7c5ba4fde34@mail.gmail.com> <20080408232110.GB16696@phare.normalesup.org> <47FC01E1.3080400@enthought.com> Message-ID: On Tue, Apr 8, 2008 at 4:55 PM, Fernando Perez wrote: > 2. I'll work on random and report shortly. As far as I can tell, there are no problems at all. I'm running bic128[scipy]$ cython --version Cython version 0.9.6.12 This is both on 64-bit Fedora 8 and 32-bit Ubuntu 7.10. I made sure I was actually running off the newly generated C code written out by cython instead of the old pyrex one. If all agree, we can push the C sources and the generate_mtrand script as a single commit where I don't touch *anything* else. That way the buildbots can run the tests, and if that C commit changes anything, then it's easy to back off. I scanned the sources for a while, and the only other file that has significant amounts of pyrex mention is build_src.py: http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/distutils/command/build_src.py I'd rather not touch that file yet, since distutils isn't exactly my favorite code to mess with :) So I'd like to know whether people would want to see this one isolated commit to happen, for the buildbots to test things out on other platforms or not. I can also make the patch available somewhere (it's huge, since it involves the auto-generated mtrand.c file) if you all prefer... Cheers, f From faltet at carabos.com Wed Apr 9 03:25:54 2008 From: faltet at carabos.com (Francesc Altet) Date: Wed, 9 Apr 2008 09:25:54 +0200 Subject: [Numpy-discussion] numpyx.pyx (recent svn) works? In-Reply-To: <47FC216E.7040609@astraw.com> References: <47FC216E.7040609@astraw.com> Message-ID: <200804090925.54672.faltet@carabos.com> A Wednesday 09 April 2008, Andrew Straw escrigu?: > This is off-topic and should be directed to the pyrex/cython list, > but since we're on the subject: > > I suppose the following is true, but let me ask, since I have not > used Cython. Please correct me if I'm wrong. > > I have a bunch of pyrex compiled .pyx code. If I start adding some > Cython compiled code, all the extensions will live happily without > colliding, even if I don't re-compile my old .pyx files with Cython. > > (I can't imagine how they'd conflict, since they both translate to .c > extensions without anything to collide. Or are there linker-level > symbols that collide or something difficult?) I don't expect you having problems in that regard either. However, I've been having problems compiling perfectly valid Pyrex code with the Cython compiler. I just haven't had time to locate where the problem is and report it, but people using Pyrex and planning to migrate to Cython must be aware that these sort of things happen. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From faltet at carabos.com Wed Apr 9 03:44:40 2008 From: faltet at carabos.com (Francesc Altet) Date: Wed, 9 Apr 2008 09:44:40 +0200 Subject: [Numpy-discussion] numpyx.pyx (recent svn) works? In-Reply-To: <200804090925.54672.faltet@carabos.com> References: <47FC216E.7040609@astraw.com> <200804090925.54672.faltet@carabos.com> Message-ID: <200804090944.40335.faltet@carabos.com> A Wednesday 09 April 2008, Francesc Altet escrigu?: > A Wednesday 09 April 2008, Andrew Straw escrigu?: > > This is off-topic and should be directed to the pyrex/cython list, > > but since we're on the subject: > > > > I suppose the following is true, but let me ask, since I have not > > used Cython. Please correct me if I'm wrong. > > > > I have a bunch of pyrex compiled .pyx code. If I start adding some > > Cython compiled code, all the extensions will live happily without > > colliding, even if I don't re-compile my old .pyx files with > > Cython. > > > > (I can't imagine how they'd conflict, since they both translate to > > .c extensions without anything to collide. Or are there > > linker-level symbols that collide or something difficult?) > > I don't expect you having problems in that regard either. However, > I've been having problems compiling perfectly valid Pyrex code with > the Cython compiler. I just haven't had time to locate where the > problem is and report it, but people using Pyrex and planning to > migrate to Cython must be aware that these sort of things happen. Just to clarify, my problems were more due to file arrangement to create extensions (.pxd, .pxi, etc...) than the issue in the relatively simple Pyrex example distributed with NumPy. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From fperez.net at gmail.com Wed Apr 9 03:47:25 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 9 Apr 2008 00:47:25 -0700 Subject: [Numpy-discussion] numpyx.pyx (recent svn) works? In-Reply-To: <200804090925.54672.faltet@carabos.com> References: <47FC216E.7040609@astraw.com> <200804090925.54672.faltet@carabos.com> Message-ID: On Wed, Apr 9, 2008 at 12:25 AM, Francesc Altet wrote: > I don't expect you having problems in that regard either. However, I've > been having problems compiling perfectly valid Pyrex code with the > Cython compiler. I just haven't had time to locate where the problem > is and report it, but people using Pyrex and planning to migrate to > Cython must be aware that these sort of things happen. It would be great to have some examples of this, so we can better understand whether they are bugs in cython or deliberate changes in behavior. There are some of the latter, described here: http://wiki.cython.org/DifferencesFromPyrex and in fact one of those already bit us in the tiny numpy/pyrex example, as reported originally in this thread. If something more serious arises we certainly need to know about it, but I know the cython devs are a pretty responsive bunch, so I hope we'll have a good chance of clean resolutions for those. Considering how pyrex has essentially no public development process to speak of, I find it a worrying tool to commit to for the long run, while cython is very actively developed, which is always a bonus. But I'm not trying to ram anything down people's throats, which is why I didn't even want to put in the mtrand.c changes before there was more feedback (even as a self-contained single commit). Here's the patch that commit would contain, against current SVN, so others can test it without putting anything into SVN itself: http://amath.colorado.edu/faculty/fperez/tmp/mtrand-cython.patch.gz I'll be happy to leave it as a new trac ticket with an attachment for it if it will make the testing/review process easier in the long run. Cheers, f From robert.kern at gmail.com Wed Apr 9 03:49:35 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 9 Apr 2008 02:49:35 -0500 Subject: [Numpy-discussion] numpyx.pyx (recent svn) works? In-Reply-To: References: <47FC216E.7040609@astraw.com> <200804090925.54672.faltet@carabos.com> Message-ID: <3d375d730804090049n684d4003qb96440737158f660@mail.gmail.com> On Wed, Apr 9, 2008 at 2:47 AM, Fernando Perez wrote: > I'll be happy to leave it as a new trac ticket with an attachment for > it if it will make the testing/review process easier in the long run. Please do. I don't want to change mtrand over until after 1.0.5 is released. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From fperez.net at gmail.com Wed Apr 9 04:00:34 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 9 Apr 2008 01:00:34 -0700 Subject: [Numpy-discussion] numpyx.pyx (recent svn) works? In-Reply-To: <3d375d730804090049n684d4003qb96440737158f660@mail.gmail.com> References: <47FC216E.7040609@astraw.com> <200804090925.54672.faltet@carabos.com> <3d375d730804090049n684d4003qb96440737158f660@mail.gmail.com> Message-ID: On Wed, Apr 9, 2008 at 12:49 AM, Robert Kern wrote: > On Wed, Apr 9, 2008 at 2:47 AM, Fernando Perez wrote: > > I'll be happy to leave it as a new trac ticket with an attachment for > > it if it will make the testing/review process easier in the long run. > > Please do. I don't want to change mtrand over until after 1.0.5 is released. http://scipy.org/scipy/numpy/ticket/727 I assigned it to you for now, since I figured you'd have the most to say on the fate of mtrand. Cheers, f From faltet at carabos.com Wed Apr 9 04:22:48 2008 From: faltet at carabos.com (Francesc Altet) Date: Wed, 9 Apr 2008 10:22:48 +0200 Subject: [Numpy-discussion] numpyx.pyx (recent svn) works? In-Reply-To: References: <200804090925.54672.faltet@carabos.com> Message-ID: <200804091022.49195.faltet@carabos.com> A Wednesday 09 April 2008, Fernando Perez escrigu?: > On Wed, Apr 9, 2008 at 12:25 AM, Francesc Altet wrote: > > I don't expect you having problems in that regard either. > > However, I've been having problems compiling perfectly valid Pyrex > > code with the Cython compiler. I just haven't had time to locate > > where the problem is and report it, but people using Pyrex and > > planning to migrate to Cython must be aware that these sort of > > things happen. > > It would be great to have some examples of this, so we can better > understand whether they are bugs in cython or deliberate changes in > behavior. This is in my TODO list, yes. > There are some of the latter, described here: > > http://wiki.cython.org/DifferencesFromPyrex > > and in fact one of those already bit us in the tiny numpy/pyrex > example, as reported originally in this thread. If something more > serious arises we certainly need to know about it, but I know the > cython devs are a pretty responsive bunch, so I hope we'll have a > good chance of clean resolutions for those. Considering how pyrex > has essentially no public development process to speak of, I find it > a worrying tool to commit to for the long run, while cython is very > actively developed, which is always a bonus. Well, I agree that Greg Ewing (the Pyrex creator) has possibly not been very speedy in adding the suggested patches (Greg has his own thoughts on what should be added to Pyrex and what not), which ultimately brought to the need of the Cython fork, but let me stress out that he has always been *very* responsive to the questions on the Pyrex list, and quick enough in fixing real problems in Pyrex. I'm personally very satisfied with Pyrex as does its job extremely well. And I'm specially grateful to Greg for his *huge* contribution to make the job of doing Python extensions a pretty simple job. So, I don't really think that Pyrex should be considered a "worrying tool" at all (even in the "long run"), rather the contrary, it is a extremely useful tool. Having said that, I'm all for pushing Cython forward (specially if the integration with NumPy becomes a reality :). Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From meine at informatik.uni-hamburg.de Wed Apr 9 04:26:57 2008 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Wed, 9 Apr 2008 10:26:57 +0200 Subject: [Numpy-discussion] Array printing and another question In-Reply-To: <47FB8DB9.3030700@jhu.edu> References: <47FB7BD7.1020506@jhu.edu> <47FB8DB9.3030700@jhu.edu> Message-ID: <200804091026.57767.meine@informatik.uni-hamburg.de> Am Dienstag, 08. April 2008 17:22:33 schrieb Ken Basye: > I've had this happen > often enough that I found the first thing I did when an output > difference arose was to print the FP in hex to see if the > difference was "real" or just a formatting artifact. Nice idea - is that code available somewhere / could you post it? As a side note, there is a second big source of bug-like experiences with the x86 architecture: floating point values are represented with a higher precision inside the FPU (e.g. 80 bits instead of 64), but that probably only matters for compiled programs where the compiler may (or may not) optimize intermediate, truncating FPU->memory operations away (which leads to differing results and is one of the most often reported "bugs" of the GCC optimizer). -- Ciao, / / /--/ / / ANS From stefan at sun.ac.za Wed Apr 9 04:47:37 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 9 Apr 2008 10:47:37 +0200 Subject: [Numpy-discussion] numpyx.pyx (recent svn) works? In-Reply-To: <200804091022.49195.faltet@carabos.com> References: <200804090925.54672.faltet@carabos.com> <200804091022.49195.faltet@carabos.com> Message-ID: <9457e7c80804090147y2336114ft7358d19d8cba5d7@mail.gmail.com> On 09/04/2008, Francesc Altet wrote: > Well, I agree that Greg Ewing (the Pyrex creator) has possibly not been > very speedy in adding the suggested patches (Greg has his own thoughts > on what should be added to Pyrex and what not), which ultimately > brought to the need of the Cython fork, but let me stress out that he > has always been *very* responsive to the questions on the Pyrex list, > and quick enough in fixing real problems in Pyrex. I'm personally very > satisfied with Pyrex as does its job extremely well. And I'm specially > grateful to Greg for his *huge* contribution to make the job of doing > Python extensions a pretty simple job. > So, I don't really think that Pyrex should be considered a "worrying > tool" at all (even in the "long run"), rather the contrary, it is a > extremely useful tool. Your first paragraph above served to convince me that this is indeed a worrying tool. a) The author is not quick (or willing?) to add patches b) The author wants to decide what goes in and what not (contrary to the community) (Sounds a bit like g77 vs gfortran.) Pyrex is therefore pretty much in maintenance mode. Cython has added some vast improvements; amongs other things - List comprehension - For i in range (instead of the hacked for i from ...) - Introspection of source into .pyx file I do agree with you that Pyrex is very useful, but Cython does exactly the same thing, and more, with the additional bonuses that it is being actively developed, and that the SAGE guys like NumPy :) Cheers St?fan From fperez.net at gmail.com Wed Apr 9 04:51:47 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 9 Apr 2008 01:51:47 -0700 Subject: [Numpy-discussion] numpyx.pyx (recent svn) works? In-Reply-To: <200804091022.49195.faltet@carabos.com> References: <200804090925.54672.faltet@carabos.com> <200804091022.49195.faltet@carabos.com> Message-ID: On Wed, Apr 9, 2008 at 1:22 AM, Francesc Altet wrote: > Well, I agree that Greg Ewing (the Pyrex creator) has possibly not been > very speedy in adding the suggested patches (Greg has his own thoughts > on what should be added to Pyrex and what not), which ultimately > brought to the need of the Cython fork, but let me stress out that he > has always been *very* responsive to the questions on the Pyrex list, > and quick enough in fixing real problems in Pyrex. I'm personally very > satisfied with Pyrex as does its job extremely well. And I'm specially > grateful to Greg for his *huge* contribution to make the job of doing > Python extensions a pretty simple job. > > So, I don't really think that Pyrex should be considered a "worrying > tool" at all (even in the "long run"), rather the contrary, it is a > extremely useful tool. Having said that, I'm all for pushing Cython > forward (specially if the integration with NumPy becomes a reality :). I certainly didn't mean to bash the developer, nor to diminish in any way his contribution. Obviously cython wouldn't exist in the first place if it weren't thanks to pyrex :) But for an open source project, in this day and age, to have this as its only public source of code history: http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/hg/ isn't really great IMO: public updates from 4 months ago? That's why I feel better about a project with an open development model as a core tool to depend on. Ultimately each developer can choose to run his personal project in any way he chooses to, but other projects are also free to choose what bases they build upon :) Cheers, f From faltet at carabos.com Wed Apr 9 05:15:28 2008 From: faltet at carabos.com (Francesc Altet) Date: Wed, 9 Apr 2008 11:15:28 +0200 Subject: [Numpy-discussion] numpyx.pyx (recent svn) works? In-Reply-To: <9457e7c80804090147y2336114ft7358d19d8cba5d7@mail.gmail.com> References: <200804091022.49195.faltet@carabos.com> <9457e7c80804090147y2336114ft7358d19d8cba5d7@mail.gmail.com> Message-ID: <200804091115.29248.faltet@carabos.com> A Wednesday 09 April 2008, St?fan van der Walt escrigu?: > On 09/04/2008, Francesc Altet wrote: > > Well, I agree that Greg Ewing (the Pyrex creator) has possibly not > > been very speedy in adding the suggested patches (Greg has his own > > thoughts on what should be added to Pyrex and what not), which > > ultimately brought to the need of the Cython fork, but let me > > stress out that he has always been *very* responsive to the > > questions on the Pyrex list, and quick enough in fixing real > > problems in Pyrex. I'm personally very satisfied with Pyrex as > > does its job extremely well. And I'm specially grateful to Greg > > for his *huge* contribution to make the job of doing Python > > extensions a pretty simple job. > > > > So, I don't really think that Pyrex should be considered a > > "worrying tool" at all (even in the "long run"), rather the > > contrary, it is a extremely useful tool. > > Your first paragraph above served to convince me that this is indeed > a worrying tool. Well, now that I read it better, maybe there is effectively something to worry about the Pyrex future ;) I just wanted to point out that Pyrex is still a good piece of software and much kudos should go to Greg as the originator of the idea. > a) The author is not quick (or willing?) to add patches > b) The author wants to decide what goes in and what not (contrary to > the community) Yes, this is my perception from reading the Pyrex list. However, it is important to point out that he has been always fast in fixing real problems (with his own patches or contributed ones). > Pyrex is therefore pretty much in maintenance mode. Perhaps, although it seems to me that Greg is still planning to enhance and pushing Pyrex forward. And in fact, the Cython guys seems to be commited to maintain compatibility with future Pyrex, which is really nice. > Cython has added > some vast improvements; amongs other things > > - List comprehension > - For i in range (instead of the hacked for i from ...) > - Introspection of source into .pyx file > > I do agree with you that Pyrex is very useful, but Cython does > exactly the same thing, and more, with the additional bonuses that it > is being actively developed, and that the SAGE guys like NumPy :) I'm not saying the contrary, indeed! Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From millman at berkeley.edu Wed Apr 9 07:04:33 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 9 Apr 2008 04:04:33 -0700 Subject: [Numpy-discussion] NumPy 1.0.5 final blockers Message-ID: Hello, The NumPy 1.0.5 release is fast approaching! There are currently 109 closed tickets and just 19 open ones: http://projects.scipy.org/scipy/numpy/milestone/1.0.5 The amount of code cleanup, fixes, documentation, and new tests added over the last month is just phenomenal. With just a few days left until I tag the release, now is the time to put in that last push to make sure that this is a rock solid release. Since this may be the last release in the 1.0.x series, please help make it the best one. I am going to go through all the open tickets and create an email thread for tickets I need more information on. So please read and possibly respond to the next few emails I send. I would greatly appreciate it if we could close a few more of these tickets before Friday. Finally if anyone knows of any issue that should delay the release (i.e., a regression), please speak up now. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From millman at berkeley.edu Wed Apr 9 07:08:47 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 9 Apr 2008 04:08:47 -0700 Subject: [Numpy-discussion] ticket #539 Message-ID: This could possibly be a regression: http://scipy.org/scipy/numpy/ticket/539 Can someone verify that these now work? -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From millman at berkeley.edu Wed Apr 9 07:11:04 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 9 Apr 2008 04:11:04 -0700 Subject: [Numpy-discussion] ticket #539 Message-ID: Hello Robert Kern (or some else who wants to take this). This one has a patch: http://projects.scipy.org/scipy/numpy/ticket/581 Can you verify that it is safe/correct and commit it? -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From millman at berkeley.edu Wed Apr 9 07:13:47 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 9 Apr 2008 04:13:47 -0700 Subject: [Numpy-discussion] ticket #587 Message-ID: Hey Pearu, Could you take a quick look at this: http://projects.scipy.org/scipy/numpy/ticket/587 -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From millman at berkeley.edu Wed Apr 9 07:29:36 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 9 Apr 2008 04:29:36 -0700 Subject: [Numpy-discussion] ticket #605 Message-ID: Hello, I just turned this one into a blocker for now. There has been a very long and good discussion about this ticket: http://projects.scipy.org/scipy/numpy/ticket/605 Could someone (David?, Bruce?) briefly summarize the problem and the current proposed solution for us again? Let's agree on the problem and the solution. I want to have something similiar to what is written about median for this release: http://projects.scipy.org/scipy/numpy/milestone/1.0.5 I agree with David's sentiment: "This issue has been raised a number of times since I follow this ML. It's not the first time I've proposed patches, and I've already documented the weird behavior only to see the comments disappear after a while. I hope this time some kind of agreement will be reached." If you give me the short summary I will make sure Travis or Eric respond (and I will put it in the release notes). Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From millman at berkeley.edu Wed Apr 9 07:31:56 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 9 Apr 2008 04:31:56 -0700 Subject: [Numpy-discussion] ticket #663 Message-ID: This has a patch: http://projects.scipy.org/scipy/numpy/ticket/663 Could someone review and commit it? Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From millman at berkeley.edu Wed Apr 9 07:34:03 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 9 Apr 2008 04:34:03 -0700 Subject: [Numpy-discussion] 673 Message-ID: This has a patch: http://projects.scipy.org/scipy/numpy/ticket/673 Could someone review, comment or commit? Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From millman at berkeley.edu Wed Apr 9 07:39:56 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 9 Apr 2008 04:39:56 -0700 Subject: [Numpy-discussion] ticket #693 Message-ID: http://projects.scipy.org/scipy/numpy/ticket/693 Could someone add a test for this changeset: http://projects.scipy.org/scipy/numpy/changeset/4826 Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From millman at berkeley.edu Wed Apr 9 07:49:58 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 9 Apr 2008 04:49:58 -0700 Subject: [Numpy-discussion] ticket #719 Message-ID: Have sufficient tests been added to close this: http://projects.scipy.org/scipy/numpy/ticket/719 Travis -- Are you planning to add more? -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From pearu at cens.ioc.ee Wed Apr 9 07:59:41 2008 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Wed, 9 Apr 2008 14:59:41 +0300 (EEST) Subject: [Numpy-discussion] ticket #587 In-Reply-To: References: Message-ID: <40498.129.240.228.53.1207742381.squirrel@cens.ioc.ee> On Wed, April 9, 2008 2:13 pm, Jarrod Millman wrote: > Hey Pearu, > > Could you take a quick look at this: > http://projects.scipy.org/scipy/numpy/ticket/587 I have fixed it in r4996. However, when trying to change the ticked status, I get forbidden error: TICKET_APPEND privileges are required to perform this operation Could track admins add required privileges to me? Regards, Pearu From millman at berkeley.edu Wed Apr 9 08:04:20 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 9 Apr 2008 05:04:20 -0700 Subject: [Numpy-discussion] ticket #655 Message-ID: This ticket has a patch: http://projects.scipy.org/scipy/numpy/ticket/655 -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From fullung at gmail.com Wed Apr 9 08:22:52 2008 From: fullung at gmail.com (Albert Strasheim) Date: Wed, 9 Apr 2008 14:22:52 +0200 Subject: [Numpy-discussion] ticket #655 In-Reply-To: References: Message-ID: <5eec5f300804090522o1699ce69lb485ce2f04b8751c@mail.gmail.com> This code can probably be incorporated into NumPy in the 1.1 timeframe. I don't think anyone is going to miss it before then. On Wed, Apr 9, 2008 at 2:04 PM, Jarrod Millman wrote: > This ticket has a patch: > http://projects.scipy.org/scipy/numpy/ticket/655 > > -- > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From millman at berkeley.edu Wed Apr 9 08:25:47 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 9 Apr 2008 05:25:47 -0700 Subject: [Numpy-discussion] ticket #587 In-Reply-To: <40498.129.240.228.53.1207742381.squirrel@cens.ioc.ee> References: <40498.129.240.228.53.1207742381.squirrel@cens.ioc.ee> Message-ID: On Wed, Apr 9, 2008 at 4:59 AM, Pearu Peterson wrote: > I have fixed it in r4996. Thanks, > However, when trying to change the ticked status, I get forbidden error: > > TICKET_APPEND privileges are required to perform this operation > > Could track admins add required privileges to me? Hmm... Your permissions look right (i.e., you have TICKET_ADMIN, which is a superset of TICKET_APPEND) and it looks like you were able to change the status OK. Are you still having troubles with the Trac interface? -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From millman at berkeley.edu Wed Apr 9 08:28:19 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 9 Apr 2008 05:28:19 -0700 Subject: [Numpy-discussion] NumPy 1.0.5 final blockers In-Reply-To: References: Message-ID: On Wed, Apr 9, 2008 at 4:04 AM, Jarrod Millman wrote: > The NumPy 1.0.5 release is fast approaching! There are currently 109 > closed tickets and just 19 open ones: > http://projects.scipy.org/scipy/numpy/milestone/1.0.5 We are now at: 111 closed tickets and 17 open! Keep up the good work. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From stefan at sun.ac.za Wed Apr 9 08:30:43 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 9 Apr 2008 14:30:43 +0200 Subject: [Numpy-discussion] 673 In-Reply-To: References: Message-ID: <9457e7c80804090530q7c448971wd26b8acf51bcb0ca@mail.gmail.com> I'm busy working on this one. Can't get the patch working as advertised, yet, so I'm wondering if lapack_lite is preferring some other xerbla over the one provided. On 09/04/2008, Jarrod Millman wrote: > This has a patch: > http://projects.scipy.org/scipy/numpy/ticket/673 > > Could someone review, comment or commit? > > Thanks, From stefan at sun.ac.za Wed Apr 9 08:34:09 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 9 Apr 2008 14:34:09 +0200 Subject: [Numpy-discussion] ticket #719 In-Reply-To: References: Message-ID: <9457e7c80804090534l5e0f71e8o74d9eee8f2ee45ed@mail.gmail.com> Thus far, none have been added. On 09/04/2008, Jarrod Millman wrote: > Have sufficient tests been added to close this: > http://projects.scipy.org/scipy/numpy/ticket/719 > > Travis -- Are you planning to add more? From pearu at cens.ioc.ee Wed Apr 9 08:34:34 2008 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Wed, 9 Apr 2008 15:34:34 +0300 (EEST) Subject: [Numpy-discussion] ticket #587 In-Reply-To: References: <40498.129.240.228.53.1207742381.squirrel@cens.ioc.ee> Message-ID: <59199.129.240.228.53.1207744474.squirrel@cens.ioc.ee> On Wed, April 9, 2008 3:25 pm, Jarrod Millman wrote: > On Wed, Apr 9, 2008 at 4:59 AM, Pearu Peterson wrote: >> I have fixed it in r4996. > > Thanks, > >> However, when trying to change the ticked status, I get forbidden >> error: >> >> TICKET_APPEND privileges are required to perform this operation >> >> Could track admins add required privileges to me? > > Hmm... Your permissions look right (i.e., you have TICKET_ADMIN, > which is a superset of TICKET_APPEND) and it looks like you were able > to change the status OK. Are you still having troubles with the Trac > interface? Yes, now I can change the tickets again. I thought that somebody fixed the permissions, if not, then I guess my browser was in some weird state.. But all good now. Thanks, Pearu From stefan at sun.ac.za Wed Apr 9 08:37:32 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 9 Apr 2008 14:37:32 +0200 Subject: [Numpy-discussion] ticket #587 In-Reply-To: <40498.129.240.228.53.1207742381.squirrel@cens.ioc.ee> References: <40498.129.240.228.53.1207742381.squirrel@cens.ioc.ee> Message-ID: <9457e7c80804090537p338eaf8dpd95ff2a126a3dce6@mail.gmail.com> On 09/04/2008, Pearu Peterson wrote: > On Wed, April 9, 2008 2:13 pm, Jarrod Millman wrote: > > Hey Pearu, > > > > Could you take a quick look at this: > > http://projects.scipy.org/scipy/numpy/ticket/587 > > > I have fixed it in r4996. > > However, when trying to change the ticked status, I get forbidden error: > > TICKET_APPEND privileges are required to perform this operation > > Could track admins add required privileges to me? Trac might be wonky again. Earlier on, I had to login 3 times to get this to work (same error message). Cheers St?fan From stefan at sun.ac.za Wed Apr 9 09:34:12 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 9 Apr 2008 15:34:12 +0200 Subject: [Numpy-discussion] 673 In-Reply-To: <9457e7c80804090530q7c448971wd26b8acf51bcb0ca@mail.gmail.com> References: <9457e7c80804090530q7c448971wd26b8acf51bcb0ca@mail.gmail.com> Message-ID: <9457e7c80804090634t53c29d62h6a662e3b3d1309c@mail.gmail.com> Unfortunately, I couldn't get this patch to work, and my time has run out. Maybe someone with more knowledge the precedences/order of functions during linking can take a look. I don't know how to tell the system BLAS to call our custom xerbla, instead of the one provided. The patch addresses an important issue, though, so it warrants some more attention. Regards St?fan On 09/04/2008, St?fan van der Walt wrote: > I'm busy working on this one. Can't get the patch working as > advertised, yet, so I'm wondering if lapack_lite is preferring some > other xerbla over the one provided. > > > On 09/04/2008, Jarrod Millman wrote: > > This has a patch: > > http://projects.scipy.org/scipy/numpy/ticket/673 > > > > Could someone review, comment or commit? > > > > Thanks, > From bsouthey at gmail.com Wed Apr 9 09:56:15 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Wed, 09 Apr 2008 08:56:15 -0500 Subject: [Numpy-discussion] ticket #605 In-Reply-To: References: Message-ID: <47FCCAFF.6020405@gmail.com> Jarrod Millman wrote: > Hello, > > I just turned this one into a blocker for now. There has been a very > long and good discussion about this ticket: > http://projects.scipy.org/scipy/numpy/ticket/605 > > Could someone (David?, Bruce?) briefly summarize the problem and the > current proposed solution for us again? Let's agree on the problem > and the solution. I want to have something similiar to what is > written about median for this release: > http://projects.scipy.org/scipy/numpy/milestone/1.0.5 > > I agree with David's sentiment: "This issue has been raised a number > of times since I follow this ML. It's not the first time I've proposed > patches, and I've already documented the weird behavior only to see > the comments disappear after a while. I hope this time some kind of > agreement will be reached." > > If you give me the short summary I will make sure Travis or Eric > respond (and I will put it in the release notes). > > Thanks, > > Hi, Simply put, there are actually multiple problems with the histogram function for certain cases. 1) The initial problem was that points below the first bin are ignored: From Tommy Grav's email: bin1 -> 1 to 2.99999... bin2 -> 3 to 4.99999... bin3 -> 5 to inf This means there is no bin for -inf to 1 and, thus, the cause of the initial bug report. 2) The second problem is to address how to account for any 'outliers'. Based on the responses, David included the keyword 'discard' to handle these. 3) The 'norm' option may be wrong but I do not have any current understanding of this one. Solution: David has provided a new version of the histogram function that was provided to the list. It also had some enhancements like an axis keyword. However, there is a potential bug associated with the use of the numpy.r_ function. Once that is overcome, I think that his code is an excellent replacement for the current version. But I can understand if this is applied to the next release. Regards Bruce From david.huard at gmail.com Wed Apr 9 10:01:27 2008 From: david.huard at gmail.com (David Huard) Date: Wed, 9 Apr 2008 10:01:27 -0400 Subject: [Numpy-discussion] ticket #605 In-Reply-To: References: Message-ID: <91cf711d0804090701m2e8567bfye9c0515bd1b26bf9@mail.gmail.com> Hello Jarrod and co., here is my personal version of the histogram saga. The current version of histogram puts in the rightmost bin all values larger than range, but does not put in the leftmost bin all values smaller than bin, eg. In [6]: histogram([1,2,3,4,5,6], bins=3, range=[2,5]) Out[6]: (array([1, 1, 3]), array([ 2., 3., 4.])) It discards 1, but puts 2 in the first bin, 3 in the second bin, and 4,5,6 in the third bin. Also, the docstring says that outliers are put in the closest bin, which is false. Another point to consider is normalization. Currently, the normalization factor is db=bin[1]-bin[0]. Of course, if the bins are not equally spaced, this will yield a spurious density. Also, I'd argue that since the rightmost bin covers the space from bin[-1] to infinity, it's density should always be zero. Now if someone wants to explain all that in the docstring, that's fine by me. I fully understand the need to avoid breaking people's code. I simply hope that in the next big release, this behavior can be changed to something that is simpler: bins are the bin edges (instead of the left edges), and everything outside the edges is ignored. This would be a nice occasion to add an axis keyword and possibly weights, and would make histogram consistent with histogramdd. I'm willing to implement those changes, but I don't know how to do so without breaking histogram's behavior. I just got Bruce reply, so sorry for the overlap. David 2008/4/9, Jarrod Millman : > > Hello, > > I just turned this one into a blocker for now. There has been a very > long and good discussion about this ticket: > http://projects.scipy.org/scipy/numpy/ticket/605 > > Could someone (David?, Bruce?) briefly summarize the problem and the > current proposed solution for us again? Let's agree on the problem > and the solution. I want to have something similiar to what is > written about median for this release: > http://projects.scipy.org/scipy/numpy/milestone/1.0.5 > > I agree with David's sentiment: "This issue has been raised a number > of times since I follow this ML. It's not the first time I've proposed > patches, and I've already documented the weird behavior only to see > the comments disappear after a while. I hope this time some kind of > agreement will be reached." > > If you give me the short summary I will make sure Travis or Eric > respond (and I will put it in the release notes). > > Thanks, > > > -- > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Wed Apr 9 10:50:35 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Wed, 09 Apr 2008 09:50:35 -0500 Subject: [Numpy-discussion] Ticket #605 Incorrect behavior of numpy.histogram In-Reply-To: <91cf711d0804081225t196a9730v460d8955b60a2686@mail.gmail.com> References: <200804071434.09201.meine@informatik.uni-hamburg.de> <200804081000.20749.meine@informatik.uni-hamburg.de> <91cf711d0804080612l4b8abat7c5c8117ed41d21b@mail.gmail.com> <47FBAAEC.7020306@gmail.com> <91cf711d0804081225t196a9730v460d8955b60a2686@mail.gmail.com> Message-ID: <47FCD7BB.5030400@gmail.com> Hi, I should have asked first (I hope that you don't mind), but I created a ticket Ticket #728 (http://scipy.org/scipy/numpy/ticket/728 ) for numpy.r_ because this incorrectly casts based on the array types. The bug is that -inf and inf are numpy floats but dbin is an array of ints. Unfortunately, numpy.r_ returns the type of the array with the highest precision (well at least 'float' wins over 'int') and thus there is lost precision. The fix is as you indicated below. Regards Bruce David Huard wrote: > > > 2008/4/8, Bruce Southey >: > > Hi, > I agree that the current histogram should be changed. However, I > am not > sure 1.0.5 is the correct release for that. > > > We both agree. > > David, this doesn't work for your code: > r= np.array([1,2,2,3,3,3,4,4,4,4,5,5,5,5,5]) > dbin=[2,3,4] > rc, rb=histogram(r, bins=dbin, discard=None) > > Returns: > rc=[3 3] # Really should be [3, 3, 9] > rb=[-9223372036854775808 3 -9223372036854775808] > > > I used the convention that bins are the bin edges, including the right > most edge, this is why len(rc) =2 and len(rb)=3. > > Now there clearly is a bug, and I traced it to the use of np.r_. Check > this out: > > In [26]: dbin = [1,2,3] > > In [27]: np.r_[-np.inf, dbin, np.inf] > Out[27]: array([-Inf, 1., 2., 3., Inf]) > > In [28]: np.r_[-np.inf, asarray(dbin), np.inf] > Out[28]: > array([-9223372036854775808, 1, > 2, 3, -9223372036854775808]) > > In [29]: np.r_[-np.inf, asarray(dbin).astype(float), np.inf] > Out[29]: array([-Inf, 1., 2., 3., Inf]) > > Is this a misuse of r_ or a bug ? > > > David > > > > > > > > > But I have not had time to find the error. > > Regards > Bruce > > > > David Huard wrote: > > Hans, > > > > Note that the current histogram is buggy, in the sense that it > assumes > > that all bins have the same width and computes db = bins[1]-bin[0]. > > This is why you get zeros everywhere. > > > > The current behavior has been heavily criticized and I think we > should > > change it. My proposal is to have for histogram the same behavior as > > for histogramdd and histogram2d: bins are the bin edges, > including the > > rightmost bin, and values outside of the bins are not tallied. The > > problem with this is that it breaks code, and I'm not sure it's > such a > > good idea to do this in a point release. > > > > My short term proposal would be to fix the normalization bug and > > document the current behavior of histogram for the 1.0.5 > release. Once > > it's done, we can modify histogram and maybe print a warning the > first > > time it's used to notice users of the change. > > > > I'd like to hear the voice of experienced devs on this. This > issue has > > been raised a number of times since I follow this ML. It's not the > > first time I've proposed patches, and I've already documented the > > weird behavior only to see the comments disappear after a while. I > > hope this time some kind of agreement will be reached. > > > > Regards, > > > > David > > > > > > > > > > 2008/4/8, Hans Meine > > > >>: > > > > > Am Montag, 07. April 2008 14:34:08 schrieb Hans Meine: > > > > > Am Samstag, 05. April 2008 21:54:27 schrieb Anne Archibald: > > > > There's also a fourth option - raise an exception if any > > points are > > > > outside the range. > > > > > > +1 > > > > > > I think this should be the default. Otherwise, I tend towards > > "exclude", > > > in order to have comparable bin sizes (when plotting, I always > > find peaks > > > at the ends annoying); this could also be called "clip" BTW. > > > > > > But really, an exception would follow the Zen: "In the face of > > ambiguity, > > > refuse the temptation to guess." And with a kwarg: > "Explicit is > > better > > > than implicit." > > > > > > When posting this, I did indeed not think this through fully; as > > David (and > > Tommy) pointed out, this API does not fit well with the existing > > `bins` > > option, especially when a sequence of bin bounds is given. (I > > guess I was > > mostly thinking about the special case of discrete values > and 1:1 > > bins, as > > typical for uint8 data.) > > > > Thus, I would like to withdraw my above opinion from and instead > > state that I > > find the current API as clear as it gets. If you want to > exclude > > values, > > simply pass an additional right bound, and for including > outliers, > > passing -inf as additional left bound seems to do the > trick. This > > could be > > possibly added to the documentation though. > > > > The only critical aspect I see is the `normed` arg. As it > is now, the > > rightmost bin has always infinite size, but it is not > treated like > > that: > > > > In [1]: from numpy import * > > > > In [2]: histogram(arange(10), [2,3,4], normed = True) > > Out[2]: (array([ 0.1, 0.1, 0.6]), array([2, 3, 4])) > > > > Even worse, if you try to add an infinite bin to the left, this > > pulls all > > values to zero (technically, I understand that, but it looks > really > > undesirable to me): > > > > In [3]: histogram(arange(10), [-inf, 2,3,4], normed = True) > > Out[3]: (array([ 0., 0., 0., 0.]), array([-Inf, 2., > 3., 4.])) > > > > > > -- > > Ciao, / / > > /--/ > > / / ANS > > _______________________________________________ > > Numpy-discussion mailing list > > > Numpy-discussion at scipy.org > > > > > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > ------------------------------------------------------------------------ > > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From lists at informa.tiker.net Wed Apr 9 11:28:32 2008 From: lists at informa.tiker.net (Andreas =?iso-8859-1?q?Kl=F6ckner?=) Date: Wed, 9 Apr 2008 11:28:32 -0400 Subject: [Numpy-discussion] vander() docstring In-Reply-To: References: <200803261922.58425.lists@informa.tiker.net> Message-ID: <200804091128.33116.lists@informa.tiker.net> On Mittwoch 26 M?rz 2008, Charles R Harris wrote: > The docstring is incorrect. The Vandermonde matrix produced is compatible > with numpy polynomials that also go from high to low powers. I would have > done it the other way round, so index matched power, but that isn't how it > is. Patch attached. Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: numpy-vander-docstring.patch Type: text/x-diff Size: 497 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From doutriaux1 at llnl.gov Wed Apr 9 12:50:31 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Wed, 09 Apr 2008 09:50:31 -0700 Subject: [Numpy-discussion] numpy.ma dtype.char set to "?" In-Reply-To: <200804091128.33116.lists@informa.tiker.net> References: <200803261922.58425.lists@informa.tiker.net> <200804091128.33116.lists@informa.tiker.net> Message-ID: <47FCF3D7.8090404@llnl.gov> Hi I'm tracked down a bug in our code back to the numpy.ma s2=numpy.array([[10,60],[65,45]]) s3=numpy.ma.masked_greater(s2,50.) s3.mask.dtype.char returns: '?' In the Numeric -> numpy.ma (page 35) the "?" is not listed. Is it an omission in numpy.ma ? Or is it a valid char type ? s3.dtype does say it is of type "bool" which I guess should be 'B' or 'b' no ? Thanks, C. From doutriaux1 at llnl.gov Wed Apr 9 12:57:27 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Wed, 09 Apr 2008 09:57:27 -0700 Subject: [Numpy-discussion] numpy.ma dtype.char set to "?" In-Reply-To: <47FCF3D7.8090404@llnl.gov> References: <200803261922.58425.lists@informa.tiker.net> <200804091128.33116.lists@informa.tiker.net> <47FCF3D7.8090404@llnl.gov> Message-ID: <47FCF577.8000203@llnl.gov> Sorry, I can answer my own question (page 22 of numpy book) booleans are supposed to be "?" my mistake. C. Charles Doutriaux wrote: > Hi I'm tracked down a bug in our code back to the numpy.ma > > s2=numpy.array([[10,60],[65,45]]) > s3=numpy.ma.masked_greater(s2,50.) > s3.mask.dtype.char > returns: '?' > > In the Numeric -> numpy.ma (page 35) the "?" is not listed. Is it an > omission in numpy.ma ? Or is it a valid char type ? > s3.dtype does say it is of type "bool" which I guess should be 'B' or > 'b' no ? > > Thanks, > > C. > > From rshepard at appl-ecosys.com Wed Apr 9 13:00:25 2008 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Wed, 9 Apr 2008 10:00:25 -0700 (PDT) Subject: [Numpy-discussion] Formatting Results from linalg.eig() Message-ID: I'm using linalg.eig() and it works exactly as intended. The values of the principal eigenvector are presented as real numbers (e.g., 0.159317312615085), and I'm wondering if there's a way within NumPy to multiply these values by 100 and limit the precision to 4 significant digits (e.g., 15.93). If I was printing the results I could use format specifiers, but these numbers are being inserted into a database table as well as being displayed within wxPython widgets. Rich From stefan at sun.ac.za Wed Apr 9 13:07:24 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 9 Apr 2008 19:07:24 +0200 Subject: [Numpy-discussion] vander() docstring In-Reply-To: <200804091128.33116.lists@informa.tiker.net> References: <200803261922.58425.lists@informa.tiker.net> <200804091128.33116.lists@informa.tiker.net> Message-ID: <9457e7c80804091007p4009f9adlcc424cf2f8349f5e@mail.gmail.com> On 09/04/2008, Andreas Kl?ckner wrote: > On Mittwoch 26 M?rz 2008, Charles R Harris wrote: > > The docstring is incorrect. The Vandermonde matrix produced is compatible > > with numpy polynomials that also go from high to low powers. I would have > > done it the other way round, so index matched power, but that isn't how it > > is. I've never seen the vector powers that way around in literature. Shouldn't we fix it to correspond to the (vast) majority of publications? I'm also somewhat skeptical about including 2-liners like these in the main numpy namespace, but we've been there this week already, so I'll keep quiet :) Regards St?fan From tim.hochberg at ieee.org Wed Apr 9 13:24:39 2008 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Wed, 9 Apr 2008 10:24:39 -0700 Subject: [Numpy-discussion] ticket #605 In-Reply-To: <91cf711d0804090701m2e8567bfye9c0515bd1b26bf9@mail.gmail.com> References: <91cf711d0804090701m2e8567bfye9c0515bd1b26bf9@mail.gmail.com> Message-ID: On Wed, Apr 9, 2008 at 7:01 AM, David Huard wrote: > Hello Jarrod and co., > > here is my personal version of the histogram saga. > > The current version of histogram puts in the rightmost bin all values > larger than range, but does not put in the leftmost bin all values smaller > than bin, eg. > > In [6]: histogram([1,2,3,4,5,6], bins=3, range=[2,5]) > Out[6]: (array([1, 1, 3]), array([ 2., 3., 4.])) > > It discards 1, but puts 2 in the first bin, 3 in the second bin, and 4,5,6 > in the third bin. Also, the docstring says that outliers are put in the > closest bin, which is false. Another point to consider is normalization. > Currently, the normalization factor is db=bin[1]-bin[0]. Of course, if the > bins are not equally spaced, this will yield a spurious density. Also, I'd > argue that since the rightmost bin covers the space from bin[-1] to > infinity, it's density should always be zero. > > Now if someone wants to explain all that in the docstring, that's fine by > me. I fully understand the need to avoid breaking people's code. I simply > hope that in the next big release, this behavior can be changed to something > that is simpler: bins are the bin edges (instead of the left edges), and > everything outside the edges is ignored. This would be a nice occasion to > add an axis keyword and possibly weights, and would make histogram > consistent with histogramdd. I'm willing to implement those changes, but I > don't know how to do so without breaking histogram's behavior. Here's one way which is more or less what they tend to do in the core Python to avoid breaking things. 1. Choose a new name for histogram with the desired behavior. 'histogram1D' for example. 2. Add the function with the new behavior to major release X and modify the old 'histogram' to produce a PendingDeprecationWarning (which by default does nothing, you need to change the warning filter to see anything). 3. In major release X+1, change the PendingDeprecationWarning to a DeprecationWarning. Now people will start to see warnings when they use histogram. 4. In major release X+2, rip out histogram. So, if you got the new version into 1.1, in 1.2 it would start complaining when you used histogram and in 1.3 histogram would be gone, but the new version would be in it's place. In this way, there's no point where the behavior of histogram just changes subtly; since it disappears one is forced to figure out where it went and implement appropriate changes in ones code. > > > I just got Bruce reply, so sorry for the overlap. > > David > > 2008/4/9, Jarrod Millman : > > > > Hello, > > > > I just turned this one into a blocker for now. There has been a very > > long and good discussion about this ticket: > > http://projects.scipy.org/scipy/numpy/ticket/605 > > > > Could someone (David?, Bruce?) briefly summarize the problem and the > > current proposed solution for us again? Let's agree on the problem > > and the solution. I want to have something similiar to what is > > written about median for this release: > > http://projects.scipy.org/scipy/numpy/milestone/1.0.5 > > > > I agree with David's sentiment: "This issue has been raised a number > > of times since I follow this ML. It's not the first time I've proposed > > patches, and I've already documented the weird behavior only to see > > the comments disappear after a while. I hope this time some kind of > > agreement will be reached." > > > > If you give me the short summary I will make sure Travis or Eric > > respond (and I will put it in the release notes). > > > > Thanks, > > > > > > -- > > Jarrod Millman > > Computational Infrastructure for Research Labs > > 10 Giannini Hall, UC Berkeley > > phone: 510.643.4014 > > http://cirl.berkeley.edu/ > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Apr 9 13:26:19 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 9 Apr 2008 11:26:19 -0600 Subject: [Numpy-discussion] vander() docstring In-Reply-To: <9457e7c80804091007p4009f9adlcc424cf2f8349f5e@mail.gmail.com> References: <200803261922.58425.lists@informa.tiker.net> <200804091128.33116.lists@informa.tiker.net> <9457e7c80804091007p4009f9adlcc424cf2f8349f5e@mail.gmail.com> Message-ID: On Wed, Apr 9, 2008 at 11:07 AM, St?fan van der Walt wrote: > On 09/04/2008, Andreas Kl?ckner wrote: > > On Mittwoch 26 M?rz 2008, Charles R Harris wrote: > > > The docstring is incorrect. The Vandermonde matrix produced is > compatible > > > with numpy polynomials that also go from high to low powers. I would > have > > > done it the other way round, so index matched power, but that isn't > how it > > > is. > > I've never seen the vector powers that way around in literature. > Shouldn't we fix it to correspond to the (vast) majority of > publications? > It would affect polyfit, where the powers correspond to the numpy polynomial coefficients. That can be fixed, and as far as I can determine that is the only numpy function that uses vander, but it might break some software out there in the wild. Maybe we can put it on the list for 1.1. I'd like to change numpy polynomials also, but that is probably a mod too far. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From rshepard at appl-ecosys.com Wed Apr 9 13:43:51 2008 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Wed, 9 Apr 2008 10:43:51 -0700 (PDT) Subject: [Numpy-discussion] Formatting Results from linalg.eig() -- RESOLVED In-Reply-To: References: Message-ID: On Wed, 9 Apr 2008, Rich Shepard wrote: > ... and I'm wondering if there's a way within NumPy to multiply these > values by 100 and limit the precision to 4 significant digits Got it all worked out. All resolved. Rich From aisaac at american.edu Wed Apr 9 13:45:40 2008 From: aisaac at american.edu (Alan Isaac) Date: Wed, 9 Apr 2008 13:45:40 -0400 Subject: [Numpy-discussion] vander() docstring In-Reply-To: References: <200803261922.58425.lists@informa.tiker.net><200804091128.33116.lists@informa.tiker.net><9457e7c80804091007p4009f9adlcc424cf2f8349f5e@mail.gmail.com> Message-ID: On Wed, 9 Apr 2008, Charles R Harris wrote: > It would affect polyfit, where the powers correspond to > the numpy polynomial coefficients. That can be fixed, and > as far as I can determine that is the only numpy function > that uses vander, but it might break some software out > there in the wild. Maybe we can put it on the list for > 1.1. I'd like to change numpy polynomials also, but that > is probably a mod too far. How about tagging the coefficient order, so everyone can use these the way they prefer. Alan Isaac From lists at informa.tiker.net Wed Apr 9 13:50:23 2008 From: lists at informa.tiker.net (Andreas =?iso-8859-1?q?Kl=F6ckner?=) Date: Wed, 9 Apr 2008 13:50:23 -0400 Subject: [Numpy-discussion] vander() docstring In-Reply-To: References: <200803261922.58425.lists@informa.tiker.net> <9457e7c80804091007p4009f9adlcc424cf2f8349f5e@mail.gmail.com> Message-ID: <200804091350.25032.lists@informa.tiker.net> Hi Chuck, all, On Mittwoch 09 April 2008, Charles R Harris wrote: > It would affect polyfit, where the powers correspond to the numpy > polynomial coefficients. That can be fixed, and as far as I can determine > that is the only numpy function that uses vander, but it might break some > software out there in the wild. Maybe we can put it on the list for 1.1. > I'd like to change numpy polynomials also, but that is probably a mod too > far. IMHO, a) 1.0.5 should ship with the docstring fixed, and nothing else. b) maybe we should deprecate the numpy.*-contained polynomial functions and move this stuff to numpy.poly for 1.1, and use this opportunity to fix the ordering in the moved functions. ? Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From Chris.Barker at noaa.gov Wed Apr 9 14:27:04 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 09 Apr 2008 11:27:04 -0700 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <20080407154905.GI9851@phare.normalesup.org> References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> Message-ID: <47FD0A78.90705@noaa.gov> Sorry to be late on this thread, but I was out of town, and I do feel strongly about this issue. Gael Varoquaux wrote: > For the beginner, "from numpy.all import *" is more confusing than "from > numpy import *" (which is already confusing). except that the beginner, nor anyone else, should ever use "import *" anyway! > I know namespace are good things, but the beginner struggles with them. > This is why I used the "import *" in my above examples. You're better off with a good foundation -- really. And particularly for a beginner, knowing what comes from numpy, and what from python (or other packages) is a "good thing". It's a mixed bag, but I like namespaces a lot -- there's a lot to be said for thinking: I need some stats, an doing something like: from numpy.stats import stats That being said, having a numpy.all namespace has its uses, particularly for interactive use, but let's not make it the default. Look at history here: Everyone used to do "from Numeric import *", now many (most?) folks use the numpy namespace, with something like "import numpy as N". Matplotlib started out with a Matlab like: "from pylab import *", now there is a separate namespace for plotting, etc, and a movement towards using separate namespaces. "Namespaces are one honking great idea -- let's do more of those!" That's "more", not "fewer" Gael Varoquaux wrote: > Convention are important, especially in coding. This is really, really important. What if we all used a different name for "self?" -- just as correct, but it would be a lot harder to understand other's code. I really don't get the reluctance -- EVERY major package I've worked with has moved AWAY from "import *" (numpy, wxPython, matplotlib, ...). We should never, never, recommend it to beginners. Period. And it would be very nice to use a standard. I use "import numpy as N", but would be quite happy to use "np" or "nx", or anything else short that becomes a standard. > IMHO the > pylab option is quite nice: matplotlib is nice and modular, but pylab has > it all. Use whichever you want. I disagree -- with pylab and numpy, there is constant confusion from folks as they move past the beginner stage -- "where did that function come from?", "What do I have to change now that I'm embedding my MPL code in a GUI?" "There should be one-- and preferably only one --obvious way to do it." Maybe there is a large population of folks that never do move past the beginner stage -- but I say -- let then use Octave! I use Python specifically because it's a more sophisticated language than Matlab. > The only thing > namespaces solve is name collisions imho. The other one is readability -- I like knowing where things come from, and what that have to do with. This is really an augment against "import *", but it applied to hierarchical namespaces too -- you can see the structure in the code, -- I like that. > I don't believe that the > current numpy has too many names in its basic namespace, It's a little too big, rather than a lot too small though -- remember this thread started from "where do we put financial functions". > and it > already has split out some things into subpackages (fft, random, > linear algebra) that have such a potential. exactly -- so numpy.finance fits right in... > 3) Some don't like the bloat (in disk space or download sizes) of > adding things to numpy. In my case, as long as the addition doesn't > make installations any more difficult I don't care. +1 Easy of installation is far more important than download size. Brian Granger wrote: > Simply putting things into > numpy because of convenience (numpy is easier to install) only > encourages people to never install or use scipy. Actually it's worse -- it discourages us from making scipy easier to install. I still don't use it. But we can save most of this for 1.1 (or 2.0, or...) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From gael.varoquaux at normalesup.org Wed Apr 9 15:25:59 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 9 Apr 2008 21:25:59 +0200 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <47FD0A78.90705@noaa.gov> References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <47FD0A78.90705@noaa.gov> Message-ID: <20080409192559.GB11856@phare.normalesup.org> On Wed, Apr 09, 2008 at 11:27:04AM -0700, Christopher Barker wrote: > Gael Varoquaux wrote: > > For the beginner, "from numpy.all import *" is more confusing than "from > > numpy import *" (which is already confusing). > except that the beginner, nor anyone else, should ever use "import *" > anyway! Right! Sure. Tell this to a medical doctor who just wants to learn as little things as possible about a computer in order to process his MRI data and finish his PhD to never have to worry anymore with stupid things like programming language. "from foo import *" does have a usecase. It is suited only for small scripts, but for people who want to learn as little as possible, it is nice. """ As you design something, ask ?is this relevant to what people are trying to do?? rather than ?is this confusing?? [?] It doesn?t matter whether people could figure something out. It matters whether they?re interested in figuring it out - is it part of what they?re trying to do, or an annoying sidetrack? """ (taken from http://log.ometer.com/2008-03.html#5) Some people do not want their scripts to scale or to last more than a day. Ga?l From david.huard at gmail.com Wed Apr 9 16:18:41 2008 From: david.huard at gmail.com (David Huard) Date: Wed, 9 Apr 2008 16:18:41 -0400 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <20080409192559.GB11856@phare.normalesup.org> References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <47FD0A78.90705@noaa.gov> <20080409192559.GB11856@phare.normalesup.org> Message-ID: <91cf711d0804091318u192b624fp85a1286a2fc2fd07@mail.gmail.com> 2008/4/9, Gael Varoquaux : > > [snip] > > Some people do not want their scripts to scale or to last more than a day. And that's what Matlab is especially good at ! ; ) And I'll say the thing I'm dying to say since this started: If anybody other than Travis had suggested we put financial functions in numpy the response would have been: make it a scikit, let the functions mature and evolve, get some feedback from users and then we'll see where they fit in. The fact that we are still discussing this shows the huge amount of respect Travis has in this community, but also the lack of guidelines for NumPy's growth. Maybe it's time for us to decide on a procedure for NEPs (Numpy Enhancement Proposals) ! Regards, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Wed Apr 9 16:29:27 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 9 Apr 2008 22:29:27 +0200 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <91cf711d0804091318u192b624fp85a1286a2fc2fd07@mail.gmail.com> References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <47FD0A78.90705@noaa.gov> <20080409192559.GB11856@phare.normalesup.org> <91cf711d0804091318u192b624fp85a1286a2fc2fd07@mail.gmail.com> Message-ID: <20080409202927.GF28763@phare.normalesup.org> On Wed, Apr 09, 2008 at 04:18:41PM -0400, David Huard wrote: > And that's what Matlab is especially good at ! ; ) Exactly. I would like to have the same ease of use for beginners than Matlab. The reason being that _I_ would be able develop my own module using the powerful feature of Python, but I could then give it to non technical users. Cheers, Ga?l From rkern at enthought.com Wed Apr 9 16:33:49 2008 From: rkern at enthought.com (Robert Kern) Date: Wed, 9 Apr 2008 13:33:49 -0700 Subject: [Numpy-discussion] ticket #539 In-Reply-To: References: Message-ID: <3d375d730804091333g6aa5f469oac898223514bc927@mail.gmail.com> On Wed, Apr 9, 2008 at 4:11 AM, Jarrod Millman wrote: > Hello Robert Kern (or some else who wants to take this). > > This one has a patch: > http://projects.scipy.org/scipy/numpy/ticket/581 > > Can you verify that it is safe/correct and commit it? Fixed, along with a couple of ancillary issues in r5007 and r5008. The state tuple needed to be updated with the cached Gaussian value. Since this is the actual piece of data that gets pickled to reconstruct a RandomState object, set_state() still accepts an older tuple that does not have this value. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Chris.Barker at noaa.gov Wed Apr 9 17:03:48 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 09 Apr 2008 14:03:48 -0700 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <20080409192559.GB11856@phare.normalesup.org> References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <47FD0A78.90705@noaa.gov> <20080409192559.GB11856@phare.normalesup.org> Message-ID: <47FD2F34.3030509@noaa.gov> Gael Varoquaux wrote: >> the beginner, nor anyone else, should ever use "import *" > > Right! Sure. Tell this to a medical doctor who just wants to learn as > little things as possible about a computer in order to process his MRI > data and finish his PhD to never have to worry anymore with stupid things > like programming language. > > "from foo import *" does have a usecase. I still think that if someone is writing more than ten lines of code, "import *" is a bad idea. > It is suited only for small scripts, There is a difference between scripting and programming, and you can do either with Python. Indeed, that is one of its strengths. > Some people do not want their scripts to scale or to last more than a day. True, but is Python the best tool for that? I don't know. I I know I use it, rather than Matlab, or bash, because I do want that scalability. I also think it's better to be able to scale with it feeling like you're learning a new language when you do so. > The reason being that _I_ would be able develop my own module using the > powerful feature of Python, but I could then give it to non technical > users. Then I suppose you can give them a ready to use environment, with the relevant stuff to their problem already imported for them -- that could be very useful. Anyway, yes, there are use cases, but I think we should generally advocate a more scalable style. I have yet to advocate that the Matlab users in my group (The Scientists that happen to need a bit of programming, but have no interest in it) start using Python, but, frankly, "import *", and minor syntax like that has nothing to do with it. Aside from the fact that they are already familiar with Matlab, the big issues are the fact that there is no one thing they can install to get an IDE, complete and well integrated plotting, and extensive library of assorted functions. Numpy+Scipy+MPL+Who-knows-what-editor are close, but the ease of use is not there yet, and it's NOT because of too many namespaces. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From millman at berkeley.edu Wed Apr 9 18:57:41 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 9 Apr 2008 15:57:41 -0700 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <47FD0A78.90705@noaa.gov> References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <47FD0A78.90705@noaa.gov> Message-ID: On Wed, Apr 9, 2008 at 11:27 AM, Christopher Barker wrote: > except that the beginner, nor anyone else, should ever use "import *" > anyway! +1 > "Namespaces are one honking great idea -- let's do more of those!" > > That's "more", not "fewer" Agreed. > I really don't get the reluctance -- EVERY major package I've worked > with has moved AWAY from "import *" (numpy, wxPython, matplotlib, ...). > We should never, never, recommend it to beginners. Period. And it would > be very nice to use a standard. I use "import numpy as N", but would be > quite happy to use "np" or "nx", or anything else short that becomes a > standard. Absolutely. Let's please standardize on: import numpy as np import scipy as sp See: http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines http://svn.scipy.org/svn/numpy/trunk/numpy/doc/example.py It is slowly becoming a standard and I would strongly encourage everyone to follow it. Thanks for bringing this up! -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From fperez.net at gmail.com Wed Apr 9 19:01:25 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 9 Apr 2008 16:01:25 -0700 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <47FD0A78.90705@noaa.gov> Message-ID: On Wed, Apr 9, 2008 at 3:57 PM, Jarrod Millman wrote: > Absolutely. Let's please standardize on: > import numpy as np > import scipy as sp > > See: > http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines > http://svn.scipy.org/svn/numpy/trunk/numpy/doc/example.py > > It is slowly becoming a standard and I would strongly encourage > everyone to follow it. > > Thanks for bringing this up! If I recall correctly, at the December sprint there was an IRC/gchat discussion on this and the above was agreed upon, so I've been trying to retrain my finger memory already since then. Cheers, f From charlesr.harris at gmail.com Wed Apr 9 19:11:58 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 9 Apr 2008 17:11:58 -0600 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <47FD0A78.90705@noaa.gov> Message-ID: On Wed, Apr 9, 2008 at 4:57 PM, Jarrod Millman wrote: > On Wed, Apr 9, 2008 at 11:27 AM, Christopher Barker > wrote: > > except that the beginner, nor anyone else, should ever use "import *" > > anyway! > > +1 > > > "Namespaces are one honking great idea -- let's do more of those!" > > > > That's "more", not "fewer" > > Agreed. > > > I really don't get the reluctance -- EVERY major package I've worked > > with has moved AWAY from "import *" (numpy, wxPython, matplotlib, ...). > > We should never, never, recommend it to beginners. Period. And it would > > be very nice to use a standard. I use "import numpy as N", but would be > > quite happy to use "np" or "nx", or anything else short that becomes a > > standard. > > Absolutely. Let's please standardize on: > import numpy as np > import scipy as sp import numpy.linalg as la ? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at informa.tiker.net Wed Apr 9 20:01:02 2008 From: lists at informa.tiker.net (Andreas =?iso-8859-1?q?Kl=F6ckner?=) Date: Wed, 9 Apr 2008 20:01:02 -0400 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: References: Message-ID: <200804092001.04253.lists@informa.tiker.net> On Mittwoch 09 April 2008, Charles R Harris wrote: > import numpy.linalg as la ? Yes! :) Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From oliphant at enthought.com Wed Apr 9 23:21:16 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Wed, 09 Apr 2008 22:21:16 -0500 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <91cf711d0804091318u192b624fp85a1286a2fc2fd07@mail.gmail.com> References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <47FD0A78.90705@noaa.gov> <20080409192559.GB11856@phare.normalesup.org> <91cf711d0804091318u192b624fp85a1286a2fc2fd07@mail.gmail.com> Message-ID: <47FD87AC.5000601@enthought.com> > And I'll say the thing I'm dying to say since this started: If anybody > other than Travis had suggested we put financial functions in numpy > the response would have been: make it a scikit, let the functions > mature and evolve, get some feedback from users and then we'll see > where they fit in. The fact that we are still discussing this shows > the huge amount of respect Travis has in this community, but also the > lack of guidelines for NumPy's growth. Maybe it's time for us to > decide on a procedure for NEPs (Numpy Enhancement Proposals) ! I appreciate that. It is rewarding to have time invested be regarded usefully by others. But, I've always seen the growth of NumPy as a community effort and there have been many voices with more wisdom than mine who have guided it. So, I'm not sure if it is accurate that some are not expressing their true attitudes toward the addition of these functions, but if it is then please don't hold back. I really do want accurate and sincere feedback. NumPy owes a great intellectual debt to all the mailing list discussions over the years. Right now it looks like there is a mix of attitudes, about the financial functions. They are a small enough addition, that I don't think it matters terribly much what we do with them. So, it seems to me that keeping them in numpy.lib and following the rule for that namespace for 1.0.5 will be viewed as anywhere from tolerable to a good idea depending on your point of view. The discussion does demonstrate that there are a lot of opinions. This to me is the sign of a healthy community. Fantastic! Only by hearing all the points of view can NumPy continue to improve. -Travis From stefan at sun.ac.za Thu Apr 10 01:38:51 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 10 Apr 2008 07:38:51 +0200 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <47FD2F34.3030509@noaa.gov> References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <47FD0A78.90705@noaa.gov> <20080409192559.GB11856@phare.normalesup.org> <47FD2F34.3030509@noaa.gov> Message-ID: <9457e7c80804092238m25f4bc0cybfc822a407290d7d@mail.gmail.com> Hi Chris On 09/04/2008, Christopher Barker wrote: > I have yet to advocate that the Matlab users in my group (The Scientists > that happen to need a bit of programming, but have no interest in it) > start using Python, but, frankly, "import *", and minor syntax like that > has nothing to do with it. Aside from the fact that they are already > familiar with Matlab, the big issues are the fact that there is no one > thing they can install to get an IDE, complete and well integrated > plotting, and extensive library of assorted functions. > > Numpy+Scipy+MPL+Who-knows-what-editor are close, but the ease of use is > not there yet, and it's NOT because of too many namespaces. I'm toying with the idea of having our students install SAGE this year. This would give them a great framework for interactive experimentation. I am worried, however, that it would keep them from learning how to write standalone Python code, with proper structure in terms of classes etc. -- SAGE is not an IDE after all. On the other hand it does address the packaging issue. I'm placing my hope in IPython (IPython1), with its (in-development) notebook interface. Regards St?fan From jh at physics.ucf.edu Thu Apr 10 02:21:05 2008 From: jh at physics.ucf.edu (Joe Harrington) Date: Thu, 10 Apr 2008 02:21:05 -0400 Subject: [Numpy-discussion] Numpy-discussion Digest, Vol 19, Issue 44 In-Reply-To: (numpy-discussion-request@scipy.org) References: Message-ID: > Absolutely. Let's please standardize on: > import numpy as np > import scipy as sp I hope we do NOT standardize on these abbreviations. While a few may have discussed it at a sprint, it hasn't seen broad discussion and there are reasons to prefer the other practice (numpy as N, scipy as S, pylab as P). My reasons for saying this go back to my reasons for disliking lots of heirarchical namespaces at all: if we must have namespaces, let's minimize the visual and typing impact by making them short and visually distinct from the function names (by capitalizing them). What concerns me about the discussion is that we are still not thinking like communications and thought-process experts, we are thinking like categorizers and accountants. The arguments we are raising don't have to do, positively or negatively, with the difficult acts of communicating with a computer and with other readers of our code. Those are the sole purposes of computer languages. Namespaces add characters to code that have a high redundancy factor. This means they pollute code, make it slow and inaccurate to read, and making learning harder. Lines get longer and may wrap if they contain several calls. It is harder while visually scanning code to distinguish the function name if it's adjacent to a bunch of other text, particularly if that text appears commonly in the nearby code. It therefore becomes harder to spot bugs. Mathematical code becomes less and less like the math expressions we write on paper when doing derivations, making it harder to interpret and verify. You have to memorize which subpackage each function is in, which is hard to do for those functions that could naturally go in two subpackages. While many math function names are obvious, subpackage names are not. Is it .stat or .stats or .statistics? .rand or .random? .fin or .financial? Some functions have this problem, but *every* namespace name has it in spades. The arguments people are raising are arguments related to how emotionally satisfying it is to have a place for everything and everything in its place, and to know you know everything there is to know. While we like both those things, as scientists, engineers, and mathematicians, they are almost irrelevant to coding. There is simply no reduction in readability, writeability, or debugability if you don't have namespace prefixes on everything, and knowing you know everything is easily accomplished now with the online categorized function list. We can incorporate that functionality into the doc reading apparatus ("help", currently) by using keywords in ReST comments in the docstrings and providing a way for "help" and its friends to list the keywords and what functions are connected to them. What nobody has said is "if we have lots of namespaces, my code will look prettier" or "if we have lots of namespaces, normal people will learn faster" or "if we have lots of namespaces, my code will be easier to verify and debug". I don't believe any of these statements to be true. Do you? Similarly, nobody has said, "if we have lots of namespaces, I'll be a faster coder". There is a *very* high obnoxiousness factor in typing redundant stuff at an interpreter. It's already annoying to type N.sin instead of sin, but N.T.sin? Or worse, np.tg.sin? Now the prefix has twice the characters of the function itself! Most IDL users *hate* that you have to type "print, " in order to inspect the contents of a variable. Yet, with multiple layers of namespaces we'd have lots more than seven extra characters on most lines of code, and unlike the IDL mess you'd have to *think* to recall what the right extra characters were for each function call, unlike just telling your hands to run the "print, " finger macro once again. The reasons we all like Python relate to how quick and easy it is to emit code from our fingertips that is similar to what we are thinking in our brains, compared to other languages. The brain doesn't declare variables, nor run loops over arrays. Neither does Python. When we average up the rows of a 2D array and subtract that average from the image, we don't first imagine making a new 2D array by repeating the averaged row, and neither does Python, it just broadcasts behind the scenes. I could go on, and so could all of you. Python feels more like thought than other languages. But now we are talking about breaking this commitment to lightness of code text, learnability, readability, and debugability by adding layer upon layer of prefixes to all the functions we write. There is a vital place for namespaces. Using import *, or not having namespaces at all, has unpredictable consequences, especially in the future when someone may add a function with a name identical to one you are using to one of the packages you import, breaking existing code. Namespaces make it possible for two developers who are not in communication to produce different packages that contain the same names, and not worry. This is critical in open source, so we live with it or we go back to declaring our functions, as in C. We can reduce the impact by sticking with short, distinctive abbreviations (capital N rather than lowercase np) and by not going heirarchical. Where we need multiple packages, we should have them at the top level, and not heirarchical. I'll go so far as to suggest that if scipy must have multiple packages within it, we could have them each be their own top-level package, and drop the "scipy." (or "S.", or "sp.") prefix entirely. They can still be tested as a unit and released together if we want that. There is no problem with doing it this way that good documentation does not fix. I'd rather flatten scipy, however, because the main reason to have namespaces is still satisfied that way. Of course, we should break the docs down as it's currently packaged, for easier learning and management. We just don't have to instantiate that into the language itself. What worries me is that the EXPERIENCE of reading and writing code in Python is not much being raised in this discussion, when it should be the *key* topic of any argument about the direction of the language. So, in closing, I'd like to exhort everyone to try harder to think like a sociologist, psychologist, and linguist in addition to thinking like a computer scientist, physicist, or mathematician. A computer language is a means for communicating with a computer, and with others who may use the code later. We use languages like Python over the much-faster assembly for a single reason: We spend too much time coding, and it is faster and more accurate for the author and reader to produce and consume code in Python than in assembly - or any other language. Let our guiding principle be to make this ever more true. --jh-- From robert.kern at gmail.com Thu Apr 10 02:40:56 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 10 Apr 2008 01:40:56 -0500 Subject: [Numpy-discussion] Numpy-discussion Digest, Vol 19, Issue 44 In-Reply-To: References: Message-ID: <3d375d730804092340m37f83bc6p82553e51000202e0@mail.gmail.com> Please do not respond to digest messages. If you want to respond to messages, subscribe to receive messages individually. Respond to the just messages you are interested in and keep the Subject lines intact. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From millman at berkeley.edu Thu Apr 10 03:55:40 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Thu, 10 Apr 2008 00:55:40 -0700 Subject: [Numpy-discussion] Numpy-discussion Digest, Vol 19, Issue 44 In-Reply-To: References: Message-ID: On Wed, Apr 9, 2008 at 11:21 PM, Joe Harrington wrote: > > Absolutely. Let's please standardize on: > > import numpy as np > > import scipy as sp > > I hope we do NOT standardize on these abbreviations. While a few may > have discussed it at a sprint, it hasn't seen broad discussion and > there are reasons to prefer the other practice (numpy as N, scipy as > S, pylab as P). My reasons for saying this go back to my reasons for > disliking lots of heirarchical namespaces at all: if we must have > namespaces, let's minimize the visual and typing impact by making them > short and visually distinct from the function names (by capitalizing > them). Using single capital letters was discussed and dismissed. The standard abbreviations should be at least two letters and they should follow the Python naming convention for packages (i.e., all lowercase). The single upper case letter actually uses two keys anyway. Following the convention used by the NumPy C-API and as suggested by the camelcase spelling, it was agreed to abbreviate numpy as np. After that we agreed to follow this pattern and name scipy sp. We also spoke with John Hunter and some of the matplotlib developers and agreed that pylab would be abbreviated as plt. So in summary: import numpy as np import scipy as sp import pylab as plt Why I don't want to shut anyone out of the discussion, I hope we can just agree to use these standards. There is quite a bit of work to do and too few hands and too little time. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From gael.varoquaux at normalesup.org Thu Apr 10 03:58:42 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 10 Apr 2008 09:58:42 +0200 Subject: [Numpy-discussion] Numpy-discussion Digest, Vol 19, Issue 44 In-Reply-To: References: Message-ID: <20080410075842.GD16157@phare.normalesup.org> On Thu, Apr 10, 2008 at 02:21:05AM -0400, Joe Harrington wrote: > I'll go so far as to suggest that if scipy must > have multiple packages within it, we could have them each be their own > top-level package, and drop the "scipy." (or "S.", or "sp.") prefix > entirely. Sound like C-type namespaces done with reserved prefixes. History has shown this is a dead-end, don't waist your time on it. Ga?l From matthew.brett at gmail.com Thu Apr 10 05:05:35 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 10 Apr 2008 09:05:35 +0000 Subject: [Numpy-discussion] Numpy-discussion Digest, Vol 19, Issue 44 In-Reply-To: <20080410075842.GD16157@phare.normalesup.org> References: <20080410075842.GD16157@phare.normalesup.org> Message-ID: <1e2af89e0804100205s59bb3992vbba394e27719d647@mail.gmail.com> Hi Joe, I do see your point - and agree that typing and cruft make code harder to write and read, to some extent. But I think the point is - and I'm increasingly finding this - that a judicious use of namespaces and 'from' statements makes the code much easier to read, as in import numpy as np from numpy import dot, any, arange, reshape and so on. And, personally, I tend to prefer to use lower case for modules, and the occasional upper case for loop variables and the like: for L in lines: print L type thing. Upper case also draws the eye to the capital letter, so print N.sin(a) pulls the eye to the N, so you have to disengage and remind yourself that it's the sin(a) that is important, whereas: print np.sin(a) less so - in my view. So, as a previous user of 'import numpy as N', I prefer 'import numpy as np' Best, Matthew From haase at msg.ucsf.edu Thu Apr 10 05:31:23 2008 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu, 10 Apr 2008 11:31:23 +0200 Subject: [Numpy-discussion] Numpy-discussion Digest, Vol 19, Issue 44 In-Reply-To: <1e2af89e0804100205s59bb3992vbba394e27719d647@mail.gmail.com> References: <20080410075842.GD16157@phare.normalesup.org> <1e2af89e0804100205s59bb3992vbba394e27719d647@mail.gmail.com> Message-ID: On Thu, Apr 10, 2008 at 11:05 AM, Matthew Brett wrote: > Hi Joe, > > I do see your point - and agree that typing and cruft make code harder > to write and read, to some extent. But I think the point is - and I'm > increasingly finding this - that a judicious use of namespaces and > 'from' statements makes the code much easier to read, as in > > import numpy as np > from numpy import dot, any, arange, reshape > > and so on. And, personally, I tend to prefer to use lower case for > modules, and the occasional upper case for loop variables and the > like: > > for L in lines: > print L > > type thing. Upper case also draws the eye to the capital letter, so > > print N.sin(a) > > pulls the eye to the N, so you have to disengage and remind yourself > that it's the sin(a) that is important, whereas: > > print np.sin(a) > > less so - in my view. So, as a previous user of 'import numpy as N', > I prefer 'import numpy as np' > > Best, > > Matthew > I hope I won't we excluded from further discussions if I would prefer to stick with my "single capital" approach for my "every day modules". I already have a default, "import numpy as N" (and some others of my own modules) for more then 6 years. All over my code. I've just gotten really used to it ..... At least Chris Barker seems to like the capital "N" also ;-) Most important: please don't use "from numpy import *" I like being reminded of the (495 or so) names coming from there .... Cheers, Sebastian Haase From meine at informatik.uni-hamburg.de Thu Apr 10 05:34:24 2008 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Thu, 10 Apr 2008 11:34:24 +0200 Subject: [Numpy-discussion] Numpy-discussion Digest, Vol 19, Issue 44 In-Reply-To: <1e2af89e0804100205s59bb3992vbba394e27719d647@mail.gmail.com> References: <20080410075842.GD16157@phare.normalesup.org> <1e2af89e0804100205s59bb3992vbba394e27719d647@mail.gmail.com> Message-ID: <200804101134.24996.meine@informatik.uni-hamburg.de> Am Donnerstag, 10. April 2008 11:05:35 schrieb Matthew Brett: > type thing. Upper case also draws the eye to the capital letter, so > > print N.sin(a) > > pulls the eye to the N, so you have to disengage and remind yourself > that it's the sin(a) that is important, whereas: > > print np.sin(a) > > less so - in my view. So, as a previous user of 'import numpy as N', > I prefer 'import numpy as np' +1 -- Ciao, / / /--/ / / ANS From millman at berkeley.edu Thu Apr 10 06:10:43 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Thu, 10 Apr 2008 03:10:43 -0700 Subject: [Numpy-discussion] NumPy 1.0.5 final blockers In-Reply-To: References: Message-ID: On Wed, Apr 9, 2008 at 5:28 AM, Jarrod Millman wrote: > On Wed, Apr 9, 2008 at 4:04 AM, Jarrod Millman wrote: > > The NumPy 1.0.5 release is fast approaching! There are currently 109 > > closed tickets and just 19 open ones: > > http://projects.scipy.org/scipy/numpy/milestone/1.0.5 > > We are now at: 111 closed tickets and 17 open! Keep up the good work. We are now at 115 closed tickets and 17 open. Thanks to everyone who is working on this release! We are in the final stretch so *please* make one last attempt to rid of us bugs and trouble tickets! -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From silva at lma.cnrs-mrs.fr Fri Apr 4 06:29:29 2008 From: silva at lma.cnrs-mrs.fr (Fabrice Silva) Date: Fri, 04 Apr 2008 12:29:29 +0200 Subject: [Numpy-discussion] multiply array In-Reply-To: References: Message-ID: <1207304970.2881.4.camel@Portable-s2m.cnrs-mrs.fr> ? Le vendredi 04 avril 2008 ? 00:28 -0700, wilson a ?crit : > > #of shape (1,6) > > eval=array([[3.,3.2,1.,1.1,5.,0.5]]) > > > > eval.shape=(-1,) > > please not the correction..i need to multiply first row of egimgs with > 3.0 ,second row with 3.2,....last(sixth) row with 0.5 ..For that > purpose i made the above into a 1 dimensional array. > A for loop seems inefficient in the case of big arrays > W Hi I suggest you three ways: * using matrices : mat(eval*identity(eval.shape[1]))*mat(egimgs) creates first a diagonal matrix with eval coefficients on main diagonal, then multiply the matrices diagonal and egimgs * using a temporary matrix with repeated eval coefficients eval.T.repeat(egimgs.shape[1],axis=1)*egimgs repeats array eval as many times as required to have two arrays of the same size that can be multiplied. and maybe the more efficient that does not require you to create a temp array of size egimgs.shape : eval.T*egimgs or for inplace update of egimgs egimgs *= eval.T provided the len of 1darray and one of the dimension of the 2darray match, you can multiply them directly. -- Fabrice Silva, PhD student LMA UPR CNRS 7051 - ?quipe S2M From millman at berkeley.edu Thu Apr 10 06:21:02 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Thu, 10 Apr 2008 03:21:02 -0700 Subject: [Numpy-discussion] Numpy-discussion Digest, Vol 19, Issue 44 In-Reply-To: References: <20080410075842.GD16157@phare.normalesup.org> <1e2af89e0804100205s59bb3992vbba394e27719d647@mail.gmail.com> Message-ID: On Thu, Apr 10, 2008 at 2:31 AM, Sebastian Haase wrote: > I hope I won't we excluded from further discussions if I would prefer > to stick with my "single capital" approach for my "every day modules". > I already have a default, "import numpy as N" (and some others of my > own modules) for more then 6 years. All over my code. > I've just gotten really used to it ..... > At least Chris Barker seems to like the capital "N" also ;-) I would really like to see everyone starting to follow the same conventions for naming, documentation, testing, writing extension code, etc. as much as possible. When writing new code please use the agreed upon standard. No need to spend your time changing your code now, but maybe you could start developing the habit of using the standards. I am sorry that we can't all agree on these standards, but some of it will obviously just come down to personal tastes--which means that someone will have to compromise. > Most important: > please don't use "from numpy import *" > I like being reminded of the (495 or so) names coming from there .... +1, Absolutely!! -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From stefan at sun.ac.za Thu Apr 10 06:55:18 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 10 Apr 2008 12:55:18 +0200 Subject: [Numpy-discussion] Making NumPy accessible to everyone (or no-one) (was Numpy-discussion Digest, Vol 19, Issue 44) Message-ID: <9457e7c80804100355q48cb107as9d5ccc3f3761ef0f@mail.gmail.com> Hi Joe, all On 10/04/2008, Joe Harrington wrote: > > Absolutely. Let's please standardize on: > > import numpy as np > > import scipy as sp > > I hope we do NOT standardize on these abbreviations. While a few may > have discussed it at a sprint, it hasn't seen broad discussion and > there are reasons to prefer the other practice (numpy as N, scipy as > S, pylab as P). "N" is a very unfortunate choice of abbreviation, given that so many algorithms use it to indicate the number of elements in things. "np" is much safer and, like Jarrod mentioned, also only takes two keys to type. Sebastian, a simple regexp replace should fix your problem (investment in hundreds of lines of N.* usage). > My reasons for saying this go back to my reasons for > disliking lots of heirarchical namespaces at all: if we must have > namespaces, let's minimize the visual and typing impact by making them > short and visually distinct from the function names (by capitalizing > them). The Python Style Guide (PEP08) recommends that we stick to lowercase, underscore-separated names. We'd do our users a real disservice by not following community defined standards. Namespaces throttle the amount of information with which the user is presented, and well thought through design leads to logical, intuitive segmentation of functionality. Searchable documentation and indices then become essential in guiding the user to the right place. On the other hand, when I was a freshman, we had a course on MATLAB; I remember spending countless hours using "lookfor" (I think that's what it is called?). That was one of the effects of a flat namespace. For interactive work, a flat namespace may be ideal (and I have no problem with us providing that as well), but otherwise, for file based code, I'd much prefer to have a (relatively shallow) namespace structure. > What concerns me about the discussion is that we are still not > thinking like communications and thought-process experts, we are > thinking like categorizers and accountants. The arguments we are > raising don't have to do, positively or negatively, with the difficult > acts of communicating with a computer and with other readers of our > code. Those are the sole purposes of computer languages. Isn't it easier to explain how to use a well-structured, organised library, rather than some functions-all-over-the-floor mess? If an accountant can import numpy.finance and do his work, how is that more difficult than importing every possible function included, and then sifting through them? > Namespaces add characters to code that have a high redundancy factor. > This means they pollute code, make it slow and inaccurate to read, and > making learning harder. Lines get longer and may wrap if they contain > several calls. It is harder while visually scanning code to > distinguish the function name if it's adjacent to a bunch of other > text, particularly if that text appears commonly in the nearby code. Python provides very good machinery with dealing with this verbosity: import very_foo_module as f from very.deeply.nested.namespace import func and even def foo(args): c = commonly_used_func result = c(3) + c(4) + 2*c(5) At the moment, everyone warns against using '*' with numpy, but with proper namespace, the * can be quite handy: from numpy.math import * a = sin(theta) + 3*cos(theta**2) (the example above already works in current numpy) > It therefore becomes harder to spot bugs. Mathematical code becomes > less and less like the math expressions we write on paper when doing > derivations, making it harder to interpret and verify. You have to > memorize which subpackage each function is in, which is hard to do for > those functions that could naturally go in two subpackages. If you have need to use a subset of functions defined over different namespaces, it is very easy to create a custom module, say my_field_of_study.py: from numpy.math import cosh, sinh from numpy.linalg import inv etc. Then, a simple "from my_field_of_study import *" provides you with everything you need. This needs to be done once in your life, and can be advocated as a Cookbook recipe. Memorisation be gone (but who needs to memorise with TAB-completion anyway). > While > many math function names are obvious, subpackage names are not. Is it > .stat or .stats or .statistics? .rand or .random? .fin or > .financial? Some functions have this problem, but *every* namespace > name has it in spades. Introspection is such a joy with IPython, or with the SAGE notebook, and many editors even provide similar functionality. Stuffing domain-specific functions into a flat namespace sounds like the ideal way of confusion a new user. > There is simply > no reduction in readability, writeability, or debugability if you > don't have namespace prefixes on everything, and knowing you know > everything is easily accomplished now with the online categorized > function list. You're right, we read code much more often than we write it. Even so, we seldom read thousands of lines of code, and often focus on a narrow part of the module, where the namespace provides context about the origins of the function being used. The penalty on writability isn't severe, in my experience. As for debugging, I'm reminded of the speed issues with numpy.sum and python's math.sum. > We can incorporate that functionality into the doc > reading apparatus ("help", currently) by using keywords in ReST > comments in the docstrings and providing a way for "help" and its > friends to list the keywords and what functions are connected to them. Good idea. > What nobody has said is "if we have lots of namespaces, my code will > look prettier" or "if we have lots of namespaces, normal people will > learn faster" or "if we have lots of namespaces, my code will be > easier to verify and debug". I don't believe any of these statements > to be true. Do you? Given that "lots" mean "the necessary amount", I'd answer "yes" to 1, 2 and 3. 1) More organised, 2) therefore easier to fit into my mind and 3) therefore easier to verify and debug. > Similarly, nobody has said, "if we have lots of namespaces, I'll be a > faster coder". We're not pushing for lots of namespaces, but for a fitting number (probably very few in the case of numpy). The impact on coding speed should be minimal. > There is a *very* high obnoxiousness factor in typing > redundant stuff at an interpreter. For the interpreted I'd suggest a different structure (i.e. an IPython profile, importing my_field_of_study.py :). > users *hate* that you have to type "print, " in order to inspect the > contents of a variable. Again, IPython -- we have the right tool for the job. > The reasons we all like Python relate to how quick and easy it is to > emit code from our fingertips that is similar to what we are thinking > in our brains, compared to other languages. And your brain has a very small cache, so use it with care. > documentation does not fix. I'd rather flatten scipy, however, > because the main reason to have namespaces is still satisfied that > way. I shudder to think of a flattened SciPy. > What worries me is that the EXPERIENCE of reading and writing code in > Python is not much being raised in this discussion, when it should be > the *key* topic of any argument about the direction of the language. Agreed. Fortunately, we already have many talented developers working on tools to make that part easier. > So, in closing, I'd like to exhort everyone to try harder to think > like a sociologist, psychologist, and linguist in addition to thinking > like a computer scientist, physicist, or mathematician. You're asking a lot -- and I don't know if that is fair. Surely, we should expect users of numpy to think a little bit like computer scientists, physicists, engineers and methematicians as well? Regards St?fan From Joris.DeRidder at ster.kuleuven.be Thu Apr 10 07:24:26 2008 From: Joris.DeRidder at ster.kuleuven.be (Joris De Ridder) Date: Thu, 10 Apr 2008 13:24:26 +0200 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <47FD87AC.5000601@enthought.com> References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <47FD0A78.90705@noaa.gov> <20080409192559.GB11856@phare.normalesup.org> <91cf711d0804091318u192b624fp85a1286a2fc2fd07@mail.gmail.com> <47FD87AC.5000601@enthought.com> Message-ID: <0B32C908-3E58-4FA5-A1EB-992ED8562D46@ster.kuleuven.be> On 10 Apr 2008, at 05:21, Travis E. Oliphant wrote: > Right now it looks like there is a mix of attitudes, about the > financial > functions. They are a small enough addition, that I don't think it > matters terribly much what we do with them. So, it seems to me that > keeping them in numpy.lib and following the rule for that namespace > for > 1.0.5 will be viewed as anywhere from tolerable to a good idea > depending > on your point of view. Just to be sure, you are talking about functions to compute interest rates, cumulative interests, asset depreciation, annuity periodic payments, security yields, and stuff like this? Joris Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm From lxander.m at gmail.com Thu Apr 10 08:37:29 2008 From: lxander.m at gmail.com (Alexander Michael) Date: Thu, 10 Apr 2008 08:37:29 -0400 Subject: [Numpy-discussion] Making NumPy accessible to everyone (or no-one) (was Numpy-discussion Digest, Vol 19, Issue 44) In-Reply-To: <9457e7c80804100355q48cb107as9d5ccc3f3761ef0f@mail.gmail.com> References: <9457e7c80804100355q48cb107as9d5ccc3f3761ef0f@mail.gmail.com> Message-ID: <525f23e80804100537k1298f89eg29368b691ec7126c@mail.gmail.com> On Thu, Apr 10, 2008 at 6:55 AM, St?fan van der Walt wrote: > Hi Joe, all > > On 10/04/2008, Joe Harrington wrote: > > > Absolutely. Let's please standardize on: > > > import numpy as np > > > import scipy as sp > > > > I hope we do NOT standardize on these abbreviations. While a few may > > have discussed it at a sprint, it hasn't seen broad discussion and > > there are reasons to prefer the other practice (numpy as N, scipy as > > S, pylab as P). > > "N" is a very unfortunate choice of abbreviation, given that so many > algorithms use it to indicate the number of elements in things. "np" > is much safer and, like Jarrod mentioned, also only takes two keys to > type. Sebastian, a simple regexp replace should fix your problem > (investment in hundreds of lines of N.* usage). Hey! I use np *all the time* as an abbreviation for "number of points". I don't really see what the problem is with using numpy.whatever in library code and published scripts and whatever you want in one-off throw-away scripts. It's easy to setup a shortcut key in almost any editor to alleviate the typing burden, if that is the main objection. If you have a section of an algorithm that you are trying to make look as much like text-book pseudocode as possible, than you can't do better than "from numpy import whatever" both for clarity and python coding convention. You can also say "d = numpy.dot" in the local scope at the top of your algorithm so you can write "d(x,y)" in the algorithm itself for very pithy code that doesn't require a FAQ to understand. From bsouthey at gmail.com Thu Apr 10 09:26:36 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 10 Apr 2008 08:26:36 -0500 Subject: [Numpy-discussion] Making NumPy accessible to everyone (or no-one) (was Numpy-discussion Digest, Vol 19, Issue 44) In-Reply-To: <525f23e80804100537k1298f89eg29368b691ec7126c@mail.gmail.com> References: <9457e7c80804100355q48cb107as9d5ccc3f3761ef0f@mail.gmail.com> <525f23e80804100537k1298f89eg29368b691ec7126c@mail.gmail.com> Message-ID: <47FE158C.4020208@gmail.com> Alexander Michael wrote: > On Thu, Apr 10, 2008 at 6:55 AM, St?fan van der Walt wrote: > >> Hi Joe, all >> >> On 10/04/2008, Joe Harrington wrote: >> > > Absolutely. Let's please standardize on: >> > > import numpy as np >> > > import scipy as sp >> > >> > I hope we do NOT standardize on these abbreviations. While a few may >> > have discussed it at a sprint, it hasn't seen broad discussion and >> > there are reasons to prefer the other practice (numpy as N, scipy as >> > S, pylab as P). >> >> "N" is a very unfortunate choice of abbreviation, given that so many >> algorithms use it to indicate the number of elements in things. "np" >> is much safer and, like Jarrod mentioned, also only takes two keys to >> type. Sebastian, a simple regexp replace should fix your problem >> (investment in hundreds of lines of N.* usage). >> > > Hey! I use np *all the time* as an abbreviation for "number of points". I don't > really see what the problem is with using numpy.whatever in library code and > published scripts and whatever you want in one-off throw-away scripts. It's easy > to setup a shortcut key in almost any editor to alleviate the typing burden, if > that is the main objection. If you have a section of an algorithm that you are > trying to make look as much like text-book pseudocode as possible, than you > can't do better than "from numpy import whatever" both for clarity and python > coding convention. You can also say "d = numpy.dot" in the local scope at the > top of your algorithm so you can write "d(x,y)" in the algorithm itself for very > pithy code that doesn't require a FAQ to understand. > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > Hi, I would prefer that the user has the choice and we have to remember that Python is dynamic typed. It is one thing to address experienced users but it takes time to get experienced. Also, some people may be just using some other code without knowing numpy is being used. That means users can 'import numpy as np' either directly or indirectly and somewhere further in their code entry 'np=1'. Python will be quite happy until the code wants a numpy function - a harsh but important lesson to learn. Readability (and debugging) is another situation where numpy is more informative than np (which is better than N) especially if someone is not familiar with numpy. Regards Bruce From stefan at sun.ac.za Thu Apr 10 09:37:51 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 10 Apr 2008 15:37:51 +0200 Subject: [Numpy-discussion] Making NumPy accessible to everyone (or no-one) (was Numpy-discussion Digest, Vol 19, Issue 44) In-Reply-To: <525f23e80804100537k1298f89eg29368b691ec7126c@mail.gmail.com> References: <9457e7c80804100355q48cb107as9d5ccc3f3761ef0f@mail.gmail.com> <525f23e80804100537k1298f89eg29368b691ec7126c@mail.gmail.com> Message-ID: <9457e7c80804100637p4f3ae180l619be1019a42823a@mail.gmail.com> On 10/04/2008, Alexander Michael wrote: > On Thu, Apr 10, 2008 at 6:55 AM, St?fan van der Walt wrote: > > Hi Joe, all > > > > On 10/04/2008, Joe Harrington wrote: > > > > Absolutely. Let's please standardize on: > > > > import numpy as np > > > > import scipy as sp > > > > > > I hope we do NOT standardize on these abbreviations. While a few may > > > have discussed it at a sprint, it hasn't seen broad discussion and > > > there are reasons to prefer the other practice (numpy as N, scipy as > > > S, pylab as P). > > > > "N" is a very unfortunate choice of abbreviation, given that so many > > algorithms use it to indicate the number of elements in things. "np" > > is much safer and, like Jarrod mentioned, also only takes two keys to > > type. Sebastian, a simple regexp replace should fix your problem > > (investment in hundreds of lines of N.* usage). > > > Hey! I use np *all the time* as an abbreviation for "number of points". Is that the de-facto abbreviation used for number of pts in your field (i.e. in literature)? If so, I'd recommend you stick to N :) Cheers St?fan From neilcrighton at gmail.com Thu Apr 10 10:07:13 2008 From: neilcrighton at gmail.com (Neil Crighton) Date: Thu, 10 Apr 2008 15:07:13 +0100 Subject: [Numpy-discussion] Making NumPy accessible to everyone (or no-one) (was Numpy-discussion Digest, Vol 19, Issue 44) Message-ID: <63751c30804100707ga68b443m30ee03f3231b0810@mail.gmail.com> Thanks Joe for the excellent post. It mirrors my experience with Python and Numpy very eloquently, and I think it presents a good argument against the excessive use of namespaces. I'm not so worried about N. vs np. though - I use the same method Matthew Brett suggests. If I'm going to use, say, sin and cos a lot in a script such that all the np. prefixes would make the code hard to read, I'll use: import numpy as np from numpy import sin,cos To those people who have invoked 'Namespaces are a honking great idea - let's do more of those', I'll cancel that with 'Flat is better than nested' :) I certainly wouldn't argue that using namespaces to separate categories of functions is always a bad thing, but I think it should only be done as a last resort. Neil On 10/04/2008, Joe Harrington wrote: > > Absolutely. Let's please standardize on: > > import numpy as np > > import scipy as sp > > I hope we do NOT standardize on these abbreviations. While a few may > have discussed it at a sprint, it hasn't seen broad discussion and > there are reasons to prefer the other practice (numpy as N, scipy as > S, pylab as P). My reasons for saying this go back to my reasons for > disliking lots of heirarchical namespaces at all: if we must have > namespaces, let's minimize the visual and typing impact by making them > short and visually distinct from the function names (by capitalizing > them). > > What concerns me about the discussion is that we are still not > thinking like communications and thought-process experts, we are > thinking like categorizers and accountants. The arguments we are > raising don't have to do, positively or negatively, with the difficult > acts of communicating with a computer and with other readers of our > code. Those are the sole purposes of computer languages. > > Namespaces add characters to code that have a high redundancy factor. > This means they pollute code, make it slow and inaccurate to read, and > making learning harder. Lines get longer and may wrap if they contain > several calls. It is harder while visually scanning code to > distinguish the function name if it's adjacent to a bunch of other > text, particularly if that text appears commonly in the nearby code. > It therefore becomes harder to spot bugs. Mathematical code becomes > less and less like the math expressions we write on paper when doing > derivations, making it harder to interpret and verify. You have to > memorize which subpackage each function is in, which is hard to do for > those functions that could naturally go in two subpackages. While > many math function names are obvious, subpackage names are not. Is it > .stat or .stats or .statistics? .rand or .random? .fin or > .financial? Some functions have this problem, but *every* namespace > name has it in spades. > > The arguments people are raising are arguments related to how > emotionally satisfying it is to have a place for everything and > everything in its place, and to know you know everything there is to > know. While we like both those things, as scientists, engineers, and > mathematicians, they are almost irrelevant to coding. There is simply > no reduction in readability, writeability, or debugability if you > don't have namespace prefixes on everything, and knowing you know > everything is easily accomplished now with the online categorized > function list. We can incorporate that functionality into the doc > reading apparatus ("help", currently) by using keywords in ReST > comments in the docstrings and providing a way for "help" and its > friends to list the keywords and what functions are connected to them. > > What nobody has said is "if we have lots of namespaces, my code will > look prettier" or "if we have lots of namespaces, normal people will > learn faster" or "if we have lots of namespaces, my code will be > easier to verify and debug". I don't believe any of these statements > to be true. Do you? > > Similarly, nobody has said, "if we have lots of namespaces, I'll be a > faster coder". There is a *very* high obnoxiousness factor in typing > redundant stuff at an interpreter. It's already annoying to type > N.sin instead of sin, but N.T.sin? Or worse, np.tg.sin? Now the > prefix has twice the characters of the function itself! Most IDL > users *hate* that you have to type "print, " in order to inspect the > contents of a variable. Yet, with multiple layers of namespaces we'd > have lots more than seven extra characters on most lines of code, and > unlike the IDL mess you'd have to *think* to recall what the right > extra characters were for each function call, unlike just telling your > hands to run the "print, " finger macro once again. > > The reasons we all like Python relate to how quick and easy it is to > emit code from our fingertips that is similar to what we are thinking > in our brains, compared to other languages. The brain doesn't declare > variables, nor run loops over arrays. Neither does Python. When we > average up the rows of a 2D array and subtract that average from the > image, we don't first imagine making a new 2D array by repeating the > averaged row, and neither does Python, it just broadcasts behind the > scenes. I could go on, and so could all of you. Python feels more > like thought than other languages. > > But now we are talking about breaking this commitment to lightness of > code text, learnability, readability, and debugability by adding layer > upon layer of prefixes to all the functions we write. > > There is a vital place for namespaces. Using import *, or not having > namespaces at all, has unpredictable consequences, especially in the > future when someone may add a function with a name identical to one > you are using to one of the packages you import, breaking existing > code. Namespaces make it possible for two developers who are not in > communication to produce different packages that contain the same > names, and not worry. This is critical in open source, so we live > with it or we go back to declaring our functions, as in C. We can > reduce the impact by sticking with short, distinctive abbreviations > (capital N rather than lowercase np) and by not going heirarchical. > Where we need multiple packages, we should have them at the top level, > and not heirarchical. I'll go so far as to suggest that if scipy must > have multiple packages within it, we could have them each be their own > top-level package, and drop the "scipy." (or "S.", or "sp.") prefix > entirely. They can still be tested as a unit and released together if > we want that. There is no problem with doing it this way that good > documentation does not fix. I'd rather flatten scipy, however, > because the main reason to have namespaces is still satisfied that > way. Of course, we should break the docs down as it's currently > packaged, for easier learning and management. We just don't have to > instantiate that into the language itself. > > What worries me is that the EXPERIENCE of reading and writing code in > Python is not much being raised in this discussion, when it should be > the *key* topic of any argument about the direction of the language. > So, in closing, I'd like to exhort everyone to try harder to think > like a sociologist, psychologist, and linguist in addition to thinking > like a computer scientist, physicist, or mathematician. A computer > language is a means for communicating with a computer, and with others > who may use the code later. We use languages like Python over the > much-faster assembly for a single reason: We spend too much time > coding, and it is faster and more accurate for the author and reader > to produce and consume code in Python than in assembly - or any other > language. > > Let our guiding principle be to make this ever more true. > > --jh-- > > From bsouthey at gmail.com Thu Apr 10 10:29:52 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 10 Apr 2008 09:29:52 -0500 Subject: [Numpy-discussion] Infinity definitions Message-ID: <47FE2460.6050107@gmail.com> Hi, Since we are discussing namespace and standardization, I am curious in why there are multiple definitions for defining infinity in numpy when perhaps there should be two (one for positive infinity and one for negative infinity). I really do understand that other people have use of these definitions and that it is easier to leave them in than take them out. Also, it is minor reduction in namespace because I do know that much of the namespace is either defining variables (like different floats and complex numbers) or mathematical functions (like logs and trig functions). Currently we have: numpy.Inf numpy.Infinity numpy.inf numpy.infty numpy.NINF numpy.PINF Most of these are defined in numeric.py: 'Inf = inf = infty = Infinity = PINF' In the f2py/tests subdirectories, the files return_real.py and return_complex.py uses both 'inf','Infinity'. The only occurrence of NINF and PINF are in core/src/umathmodule.c but I don't see any other usage. There does not seem to be any use of 'infty'. A similar case could be made for 'Not a Number' (nan = NaN = NAN). Regards Bruce From p.e.creasey.00 at googlemail.com Thu Apr 10 10:32:19 2008 From: p.e.creasey.00 at googlemail.com (Peter Creasey) Date: Thu, 10 Apr 2008 15:32:19 +0100 Subject: [Numpy-discussion] Simple financial functions for NumPy Message-ID: <6be8b94a0804100732y26ae4503jdb8b1090f3a06d90@mail.gmail.com> > > Right now it looks like there is a mix of attitudes, about the > > financial > > functions. They are a small enough addition, that I don't think it > > matters terribly much what we do with them. So, it seems to me that > > keeping them in numpy.lib and following the rule for that namespace > > for > > 1.0.5 will be viewed as anywhere from tolerable to a good idea > > depending > > on your point of view. > > Just to be sure, you are talking about functions to compute interest > rates, cumulative interests, asset depreciation, annuity periodic > payments, security yields, and stuff like this? > > Joris Actually, I was wondering about this, I wasn't sure if you might mean option pricing, stochastic calculus and black-scholes analytic formulae. I use these things fairly heavily, calling NumPy and SciPy functions. My instinct is that these are probably more appropriate for SciPy since they are quite niche (in comparison to something like fourier transforms). Peter From oliphant at enthought.com Thu Apr 10 11:15:31 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 10 Apr 2008 10:15:31 -0500 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <9457e7c80804092238m25f4bc0cybfc822a407290d7d@mail.gmail.com> References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <47FD0A78.90705@noaa.gov> <20080409192559.GB11856@phare.normalesup.org> <47FD2F34.3030509@noaa.gov> <9457e7c80804092238m25f4bc0cybfc822a407290d7d@mail.gmail.com> Message-ID: <47FE2F13.6070005@enthought.com> St?fan van der Walt wrote: > Hi Chris > > On 09/04/2008, Christopher Barker wrote: > >> I have yet to advocate that the Matlab users in my group (The Scientists >> that happen to need a bit of programming, but have no interest in it) >> start using Python, but, frankly, "import *", and minor syntax like that >> has nothing to do with it. Aside from the fact that they are already >> familiar with Matlab, the big issues are the fact that there is no one >> thing they can install to get an IDE, complete and well integrated >> plotting, and extensive library of assorted functions. >> >> Numpy+Scipy+MPL+Who-knows-what-editor are close, but the ease of use is >> not there yet, and it's NOT because of too many namespaces. >> > > I'm toying with the idea of having our students install SAGE this > year. This would give them a great framework for interactive > experimentation. I am worried, however, that it would keep them from > learning how to write standalone Python code, with proper structure in > terms of classes etc. -- SAGE is not an IDE after all. On the other > hand it does address the packaging issue. > So does the Enthought Python Distribution. -Travis From gael.varoquaux at normalesup.org Thu Apr 10 11:38:36 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 10 Apr 2008 17:38:36 +0200 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <47FE2F13.6070005@enthought.com> References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <47FD0A78.90705@noaa.gov> <20080409192559.GB11856@phare.normalesup.org> <47FD2F34.3030509@noaa.gov> <9457e7c80804092238m25f4bc0cybfc822a407290d7d@mail.gmail.com> <47FE2F13.6070005@enthought.com> Message-ID: <20080410153835.GB5605@phare.normalesup.org> On Thu, Apr 10, 2008 at 10:15:31AM -0500, Travis E. Oliphant wrote: > > I'm toying with the idea of having our students install SAGE this > > year. This would give them a great framework for interactive > > experimentation. I am worried, however, that it would keep them from > > learning how to write standalone Python code, with proper structure in > > terms of classes etc. -- SAGE is not an IDE after all. On the other > > hand it does address the packaging issue. > So does the Enthought Python Distribution. Well, EPD targets packaging, but not the IDE (want to have a go at this problem?). Cheers, Ga?l From Chris.Barker at noaa.gov Thu Apr 10 12:18:13 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 10 Apr 2008 09:18:13 -0700 Subject: [Numpy-discussion] Where to put financial functions... In-Reply-To: <47FD87AC.5000601@enthought.com> References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <47FD0A78.90705@noaa.gov> <20080409192559.GB11856@phare.normalesup.org> <91cf711d0804091318u192b624fp85a1286a2fc2fd07@mail.gmail.com> <47FD87AC.5000601@enthought.com> Message-ID: <47FE3DC5.4030901@noaa.gov> renamed as this was a Digest response.. Travis E. Oliphant wrote: > I appreciate that. It is rewarding to have time invested be regarded > usefully by others. Very well regarded! > They are a small enough addition, that I don't think it > matters terribly much what we do with them. I think there is some key president establishing going on here though... > So, it seems to me that keeping them in numpy.lib and following the rule for that namespace for > 1.0.5 I'm confused as to what this means. Clearly there is no consensus here, but I vote for something like numpy.finance or scipy.finance. Whether it's numpy or scipy seems to be a factor of how it's maintained and distributed, which I'll leave up to the folks that maintain and distribute... Yes, it's only a few functions now, but don't we hope it will grow? And that doesn't preclude those folks that want a flat namespace from importing them into a uber-namespace (more on that in another note). -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Thu Apr 10 12:54:49 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 10 Apr 2008 09:54:49 -0700 Subject: [Numpy-discussion] numpy namespaces In-Reply-To: References: Message-ID: <47FE4659.7090804@noaa.gov> [renaming thread due to digest reply...] Joe Harrington wrote: lots of good points... > What concerns me about the discussion is that we are still not > thinking like communications and thought-process experts, we are > thinking like categorizers and accountants. Well, yes and no. I'm thinking like myself -- measuring usability by me experience, which, I acknowledge doesn't necessarily represent anyone else, but I am very much in the target of numpy users. > Namespaces add characters to code that have a high redundancy factor. I don't think it's high redundancy -- where a name comes from is relevant, particularly when reading code. I don't do it, but let's face it, lot's of folks using code without namespaces preface names with prefix for the package it came from, and may even prefix it with a code for the type of the variable (Hungarian notation, isn't it?). Anyway, the point is, this is done a lot because people find it useful to have extra info in the names. > Lines get longer and may wrap if they contain several calls. Which is why we want "N" or "np" rather than, say, "Numeric". > Mathematical code becomes > less and less like the math expressions we write on paper when doing > derivations, making it harder to interpret and verify. This is true, and I suppose why a number of folks do: from numpy import sin, cos, ...... However, I find there really is a small fraction of my code that's a math expression. > Is it .stat or .stats or .statistics? .rand or .random? .fin or > .financial? Some functions have this problem, but *every* namespace > name has it in spades. I suppose so, but it doesn't feel onerous to me -- I d remember spending lots of time looking up names in Matlab -- only one namespace, but lots of names -- as long as we have a way to search across namespaces, I'll be happy. > There is simply > no reduction in readability, writeability, or debugability if you > don't have namespace prefixes on everything, I don't think it's that simple. I know for a fact that when _I_ read old flat namespace code, I spend a lot of time trying to figure out where the heck a given function came from. > We can incorporate that functionality into the doc > reading apparatus The doc reading apparatus can make up more many of the limitations you defined too. At least we all seem to agree that powerful doc reading apparatus is key to usability. > What nobody has said is "if we have lots of namespaces, my code will > look prettier" I'll grant you that, but pretty isn't really my most important criteria or "if we have lots of namespaces, normal people will > learn faster" or "if we have lots of namespaces, my code will be > easier to verify and debug". OK: If we have more (which is not necessarily lots) of namespaces, but code will be easier to write, debug and verify. If we have more namespaces, it will be easier to teach newbies, and, critically, easier for newbies to become programmers. > Similarly, nobody has said, "if we have lots of namespaces, I'll be a > faster coder". There is a *very* high obnoxiousness factor in typing > redundant stuff at an interpreter. It's already annoying to type > N.sin instead of sin, but N.T.sin? Maybe we need to be more specific -- are you arguing for "from pylab import *", or are you arguing for a small number of namespaces: import numpy as np import pylab as plt 'cause there can certainly be too much of a good thing! Anyway, Here is my personal experience: when I started using python some years ago, I used both: from Numeric import * and from wxPython import * I (and both communities, for the most part) does neither. I am definitely much happier reading and writing code with the namespaces. > Most IDL users *hate* that you have to type "print, "... This is a key issue -- for interactive use, the amount of typing does become an issue, and flat namespaces do have a real advantage there. However, I was a major Matlab user for years, and Python user for years now, and while I do play and test things out at the interactive prompt, I almost always regret it when I write more than 3 lines at the prompt, rather than putting them in a file, so it's hard for me to see interactive use as a priority. And there is still a place for something like pylab, that dumps a whole bunch of stuff into a single namespace for interactive use -- I just won't use it. > What worries me is that the EXPERIENCE of reading and writing code in > Python Frankly, while we aren't, on the whole linguists (come to think of it, isn't Larry Wall a linguist? -- and look where that lead!), neither are we primarily computer scientists -- I think most of us are basing our opinions on little else than OUR EXPERIENCE. A lot of software usability suffers from the fact that the folks writing the program aren't the the target users -- that's not that case here -- we are the target users! Yes, newbies and less-interested-in-programming-than-us scientists are another target, but they are not so different. We're not trying to write AppleScript here. Sebastian Haase wrote: > I've just gotten really used to it ..... > At least Chris Barker seems to like the capital "N" also ;-) For what it's worth, yes, that's what I settled on, but I think standardization is far more important than my minor preference -- "np" it is for me from now on. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From lou_boog2000 at yahoo.com Thu Apr 10 13:17:29 2008 From: lou_boog2000 at yahoo.com (Lou Pecora) Date: Thu, 10 Apr 2008 10:17:29 -0700 (PDT) Subject: [Numpy-discussion] Making NumPy accessible to everyone (or no-one) (was Numpy-discussion Digest, Vol 19, Issue 44) In-Reply-To: <525f23e80804100537k1298f89eg29368b691ec7126c@mail.gmail.com> Message-ID: <504669.8295.qm@web34401.mail.mud.yahoo.com> --- Alexander Michael wrote: > Hey! I use np *all the time* as an abbreviation for > "number of points". I don't > really see what the problem is with using > numpy.whatever in library code and > published scripts and whatever you want in one-off > throw-away scripts. It's easy > to setup a shortcut key in almost any editor to > alleviate the typing burden, if > that is the main objection. If you have a section of > an algorithm that you are > trying to make look as much like text-book > pseudocode as possible, than you > can't do better than "from numpy import whatever" > both for clarity and python > coding convention. You can also say "d = numpy.dot" > in the local scope at the > top of your algorithm so you can write "d(x,y)" in > the algorithm itself for very > pithy code that doesn't require a FAQ to understand. Yes, I use np= number of points, too. But you all might want to use something else. That's the point of the flexibility of import ... as Trying to lock in namespaces as np or N or whatever is a BAD idea. Allow the flexibility. You can admonish against from ... import * for newbies and then tell them to use from ... import actual function names (as mentioned above). But locking people into a standard, even an informal one is, as someone else said, acting a bit too much like accountants. Stop, please! -- Lou Pecora, my views are my own. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From millman at berkeley.edu Thu Apr 10 13:18:55 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Thu, 10 Apr 2008 10:18:55 -0700 Subject: [Numpy-discussion] Win32 installer: please test it Message-ID: Hello, David Cournapeau has prepared a new win32 installer, which is aimed at solving the recurring problem of non working atlas on different sets of CPU. This installer simply checks which cpu you have, and installs the appropriate numpy accordingly (without atlas if your cpu is not supported). Windows users, please test the installer, and report problems with it; we hope to use this scheme for all numpy and scipy installers in the future. Download from here: https://cirl.berkeley.edu/numpy/numpy-superpack-python2.5.exe Technical details: - ATLAS is 3.8.0, and built with cygwin. Built on Core 2 duo for SSE3, pentium 4 xeon for SSE2. - BLAS/LAPACK are built with mingw. - only SSE3 and SSE2 are supported. If you don't have at least sse2, it will not use ATLAS, but netlib blas/lapack instead. - the numpy installers are prepared with build_wininst for each different supported architecture - the numpy installers are then bundled in another installer, based on nsis. - Nsis is an open source installer, based on a small scripting language, and is extensible by plug-ins. I wrote a plug-in to detect the cpu type, and the installer will simply execute a different numpy installer depending on the cpu. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From stefan at sun.ac.za Thu Apr 10 13:58:25 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 10 Apr 2008 19:58:25 +0200 Subject: [Numpy-discussion] Making NumPy accessible to everyone (or no-one) (was Numpy-discussion Digest, Vol 19, Issue 44) In-Reply-To: <504669.8295.qm@web34401.mail.mud.yahoo.com> References: <525f23e80804100537k1298f89eg29368b691ec7126c@mail.gmail.com> <504669.8295.qm@web34401.mail.mud.yahoo.com> Message-ID: <9457e7c80804101058m6890c59m2b1362165a55bed8@mail.gmail.com> Lou, On 10/04/2008, Lou Pecora wrote: > But locking people into a standard, > even an informal one is, as someone else said, acting > a bit too much like accountants. Stop, please! This is just the recommended standard for code inside of numpy and scipy. You are, of course, free to do in your own code whatever you want. Regards St?fan From oliphant at enthought.com Thu Apr 10 14:01:03 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 10 Apr 2008 13:01:03 -0500 Subject: [Numpy-discussion] Infinity definitions In-Reply-To: <47FE2460.6050107@gmail.com> References: <47FE2460.6050107@gmail.com> Message-ID: <47FE55DF.3040201@enthought.com> Bruce Southey wrote: > Hi, > Since we are discussing namespace and standardization, I am curious in > why there are multiple definitions for defining infinity in numpy when > perhaps there should be two (one for positive infinity and one for > negative infinity). I really do understand that other people have use of > these definitions and that it is easier to leave them in than take them > out. Also, it is minor reduction in namespace because I do know that > much of the namespace is either defining variables (like different > floats and complex numbers) or mathematical functions (like logs and > trig functions). > > Currently we have: > numpy.Inf > numpy.Infinity > numpy.inf > numpy.infty > numpy.NINF > numpy.PINF > > Most of these are defined in numeric.py: 'Inf = inf = infty = Infinity = > PINF' > In the f2py/tests subdirectories, the files return_real.py and > return_complex.py uses both 'inf','Infinity'. > The only occurrence of NINF and PINF are in core/src/umathmodule.c but I > don't see any other usage. > There does not seem to be any use of 'infty'. > I think this is a product of bringing together a few definitions into one and not forcing a standard. numpy.inf numpy.nan should be used except for backward compatibility. -Travis From jturner at gemini.edu Thu Apr 10 14:09:23 2008 From: jturner at gemini.edu (James Turner) Date: Thu, 10 Apr 2008 14:09:23 -0400 Subject: [Numpy-discussion] Numpy-discussion Digest, Vol 19, Issue, 44 In-Reply-To: References: Message-ID: <47FE57D3.5080507@gemini.edu> Hi Robert et al., > Please do not respond to digest messages. If you want to respond to > messages, subscribe to receive messages individually. Respond to the > just messages you are interested in and keep the Subject lines intact. Just a suggestion, which I hope doesn't annoy anyone :-). I receive the NumPy and SciPy digests because the amount of traffic on the lists is so high and my time to keep up with them is so limited (there is probably a way to filter emails in Thunderbird that would work, but I still have to figure that out). I found a way to reply to posts from the digest that is fiddly but avoids the problem that Robert points out, for occasional posters... 1. Press reply 2. Copy the subject line from the original email you want to reply to and paste it over the subject line in the new email window. 3. Copy the "Message-ID" string from the original email you are replying to and paste it into both a "References:" line and a "In-Reply-To:" line in the message header (where you specify the recipients etc.). I hope I got that right, since it's been a while... In order to get "References:" and "In-Reply-To:" in the pull-down boxes in Thunderbird, I had to add an entry in the prefs.js file of my profile as follows: user_pref("mail.compose.other.header", "References,In-Reply-To"); Sorry to perpetuate the subject line, but I did keep it intact! Hope that helps someone (and keeps everyone happy). Cheers, James. From oliphant at enthought.com Thu Apr 10 14:20:24 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 10 Apr 2008 13:20:24 -0500 Subject: [Numpy-discussion] packaging scipy (was Re: Simple financial functions for NumPy) In-Reply-To: <20080410153835.GB5605@phare.normalesup.org> References: <91cf711d0804070751va65602hb8ebbf2db693efc7@mail.gmail.com> <9457e7c80804070820j56c8f4e6k9f13945fce218d58@mail.gmail.com> <20080407154905.GI9851@phare.normalesup.org> <47FD0A78.90705@noaa.gov> <20080409192559.GB11856@phare.normalesup.org> <47FD2F34.3030509@noaa.gov> <9457e7c80804092238m25f4bc0cybfc822a407290d7d@mail.gmail.com> <47FE2F13.6070005@enthought.com> <20080410153835.GB5605@phare.normalesup.org> Message-ID: <47FE5A68.5020903@enthought.com> Gael Varoquaux wrote: > On Thu, Apr 10, 2008 at 10:15:31AM -0500, Travis E. Oliphant wrote: > > >>> I'm toying with the idea of having our students install SAGE this >>> year. This would give them a great framework for interactive >>> experimentation. I am worried, however, that it would keep them from >>> learning how to write standalone Python code, with proper structure in >>> terms of classes etc. -- SAGE is not an IDE after all. On the other >>> hand it does address the packaging issue. >>> > > > >> So does the Enthought Python Distribution. >> > > Well, EPD targets packaging, but not the IDE (want to have a go at this > problem?). > Already in the works... -teo From peridot.faceted at gmail.com Thu Apr 10 14:26:02 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 10 Apr 2008 14:26:02 -0400 Subject: [Numpy-discussion] boolean indexing of rank-0 arrays and scalars - ticket #721 Message-ID: Hi, Ticket #721 is titled "0-dimensional boolean arrays should work as masks for array scalars". It is not quite clear to me what the right behaviour is. We are generally trying to make zero-dimensional arrays behave the same as array scalars, but I'm not sure how that should work in this case: In [2]: scalar = np.array([1,2])[0] In [3]: rank0 = np.array(1) In [4]: scalar[np.array(False)] --------------------------------------------------------------------------- Traceback (most recent call last) /home/peridot/software/numpy/test/lib/python2.5/site-packages/ in () : invalid index to scalar variable. In [5]: rank0[np.array(False)] Out[5]: array([], dtype=int32) In [6]: rank0[np.array(False)].shape Out[6]: (0,) In [7]: rank0[np.array(True)].shape Out[7]: () Here indexing a zero-dimensional array produces either a zero-dimensional array if the argument is True or a one-dimensional array of length zero if the argument is False. This is necessary because there's not really a way to represent "no value" with a zero-dimensional array - they always have exactly one value. Should a boolean index into a scalar always produce an array as a result? Only if that boolean is False? I should also point out that we have another difference between array scalars and zero-dimensional arrays: In [7]: rank0[np.array(True)].shape Out[7]: () In [8]: rank0[np.array([True,False])[0]].shape --------------------------------------------------------------------------- Traceback (most recent call last) /home/peridot/software/numpy/test/lib/python2.5/site-packages/ in () : 0-d arrays can't be indexed Zero-dimensional arrays can be used as indices, but apparently-equivalent scalars cannot. I am totally unclear on what the distinction between rank-zero arrays and scalars should be and what indexing into them should look like. Is there a document describing this (or are we just throwing an undocumented mass of confusing corner cases at users)? Thanks, Anne From Chris.Barker at noaa.gov Thu Apr 10 14:35:22 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 10 Apr 2008 11:35:22 -0700 Subject: [Numpy-discussion] Making NumPy accessible to everyone (or no-one) (was Numpy-discussion Digest, Vol 19, Issue 44) In-Reply-To: <504669.8295.qm@web34401.mail.mud.yahoo.com> References: <504669.8295.qm@web34401.mail.mud.yahoo.com> Message-ID: <47FE5DEA.20203@noaa.gov> Lou Pecora wrote: > That's the point of the flexibility of import ... as Yes, it is. > Trying to lock in namespaces as np or N or whatever is > a BAD idea. Allow the flexibility. I don't think anyone is trying to lock anyone into anything. > But locking people into a standard, > even an informal one is, as someone else said, acting > a bit too much like accountants. Stop, please! Locking in is bad, but establishing a standard is good. Particularly for newbies, it's really very, very helpful if example code in mailing list, in wikis, in the docs, etc, is all similar. Most folks learn by copying and pasting code form examples, having a proliferation different aliases for numpy is just asking for a lot of confusion. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From doutriaux1 at llnl.gov Thu Apr 10 14:38:07 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Thu, 10 Apr 2008 11:38:07 -0700 Subject: [Numpy-discussion] float32 is not a float ? In-Reply-To: <9457e7c80804101058m6890c59m2b1362165a55bed8@mail.gmail.com> References: <525f23e80804100537k1298f89eg29368b691ec7126c@mail.gmail.com> <504669.8295.qm@web34401.mail.mud.yahoo.com> <9457e7c80804101058m6890c59m2b1362165a55bed8@mail.gmail.com> Message-ID: <47FE5E8F.1040400@llnl.gov> Hello, I guess this maybe "normal" but it breaks a lot of thing when conterting from Numeric >>> a=numpy.ones(5,dtype=numpy.float32) >>> isinstance(a[0],float) False >>> float64 works... I can see why one could argue for returning False, but then the converter might be too zealous things that used to work like: if type(item) in [types.IntType, types.FloatType]: or: isinstance(item, types.FloatType) now fail, should we be concerned? should we consider returning True ? Thanks, C. From oliphant at enthought.com Thu Apr 10 14:42:46 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 10 Apr 2008 13:42:46 -0500 Subject: [Numpy-discussion] Numpy-discussion Digest, Vol 19, Issue, 44 In-Reply-To: <47FE57D3.5080507@gemini.edu> References: <47FE57D3.5080507@gemini.edu> Message-ID: <47FE5FA6.9020004@enthought.com> James, Thank you for that very helpful set of suggestions. I think the main thing is to keep the subject heading so that we can parse which conversation the response is targeted to. Also, not including the entire digest in the response is useful as well. We don't want to push occasional posters away (indeed often the most useful comments will come from them because they may not be caught up in the history of any particular decision and provide a fresh perspective). Please post :-) just use the relevant subject line and filter out non relevant content in the response. Best regards, -Travis O. > Just a suggestion, which I hope doesn't annoy anyone :-). > > I receive the NumPy and SciPy digests because the amount of traffic > on the lists is so high and my time to keep up with them is so limited > (there is probably a way to filter emails in Thunderbird that would > work, but I still have to figure that out). I found a way to reply to > posts from the digest that is fiddly but avoids the problem that > Robert points out, for occasional posters... > > 1. Press reply > 2. Copy the subject line from the original email you want to reply > to and paste it over the subject line in the new email window. > 3. Copy the "Message-ID" string from the original email you are > replying to and paste it into both a "References:" line and a > "In-Reply-To:" line in the message header (where you specify > the recipients etc.). > > I hope I got that right, since it's been a while... In order to get > "References:" and "In-Reply-To:" in the pull-down boxes in Thunderbird, > I had to add an entry in the prefs.js file of my profile as follows: > > user_pref("mail.compose.other.header", "References,In-Reply-To"); > > Sorry to perpetuate the subject line, but I did keep it intact! Hope > that helps someone (and keeps everyone happy). > > Cheers, > > James. > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From mailinglist.honeypot at gmail.com Thu Apr 10 14:46:47 2008 From: mailinglist.honeypot at gmail.com (Steve Lianoglou) Date: Thu, 10 Apr 2008 14:46:47 -0400 Subject: [Numpy-discussion] Numpy-discussion Digest, Vol 19, Issue, 44 In-Reply-To: <47FE57D3.5080507@gemini.edu> References: <47FE57D3.5080507@gemini.edu> Message-ID: <2EF4D54A-58CF-4B6B-881F-B2C81FF76D00@gmail.com> Hi, > I receive the NumPy and SciPy digests because the amount of traffic > on the lists is so high and my time to keep up with them is so limited > (there is probably a way to filter emails in Thunderbird that would > work, but I still have to figure that out). Why not just create a filter that (i) marks the message as read, and (ii) files it into a numpy folder. You can then just look in there whenever you feel like you have the time/inclination and the high traffic should go pretty much unnoticed, no? It's what I do, anyway ... oh, and also having the mails sent to a list-only email address helps, too. -steve From peridot.faceted at gmail.com Thu Apr 10 14:57:57 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 10 Apr 2008 14:57:57 -0400 Subject: [Numpy-discussion] Infinity definitions In-Reply-To: <47FE55DF.3040201@enthought.com> References: <47FE2460.6050107@gmail.com> <47FE55DF.3040201@enthought.com> Message-ID: On 10/04/2008, Travis E. Oliphant wrote: > Bruce Southey wrote: > > Hi, > > Since we are discussing namespace and standardization, I am curious in > > why there are multiple definitions for defining infinity in numpy when > > perhaps there should be two (one for positive infinity and one for > > negative infinity). I really do understand that other people have use of > > these definitions and that it is easier to leave them in than take them > > out. Also, it is minor reduction in namespace because I do know that > > much of the namespace is either defining variables (like different > > floats and complex numbers) or mathematical functions (like logs and > > trig functions). > > > > Currently we have: > > numpy.Inf > > numpy.Infinity > > numpy.inf > > numpy.infty > > numpy.NINF > > numpy.PINF > > > > Most of these are defined in numeric.py: 'Inf = inf = infty = Infinity = > > PINF' > > In the f2py/tests subdirectories, the files return_real.py and > > return_complex.py uses both 'inf','Infinity'. > > The only occurrence of NINF and PINF are in core/src/umathmodule.c but I > > don't see any other usage. > > There does not seem to be any use of 'infty'. > > > > I think this is a product of bringing together a few definitions into > one and not forcing a standard. > > numpy.inf > numpy.nan > > should be used except for backward compatibility. The others have some use if you want to be able to use the results of repr() as literals - as I understand it the output representation of a NaN depends on the C library, and users seeing, say, "NaN" might well expect to be able to type NaN (after from numpy import NaN) and get a NaN. Anne From stefan at sun.ac.za Thu Apr 10 15:17:01 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 10 Apr 2008 21:17:01 +0200 Subject: [Numpy-discussion] float32 is not a float ? In-Reply-To: <47FE5E8F.1040400@llnl.gov> References: <525f23e80804100537k1298f89eg29368b691ec7126c@mail.gmail.com> <504669.8295.qm@web34401.mail.mud.yahoo.com> <9457e7c80804101058m6890c59m2b1362165a55bed8@mail.gmail.com> <47FE5E8F.1040400@llnl.gov> Message-ID: <9457e7c80804101217n61f04322sef7ecd41e2f6d868@mail.gmail.com> Hi Charles On 10/04/2008, Charles Doutriaux wrote: > >>> a=numpy.ones(5,dtype=numpy.float32) > >>> isinstance(a[0],float) I'd recommend using issubdtype instead: np.issubdtype(a[0],float) Regards St?fan From bsouthey at gmail.com Thu Apr 10 15:19:08 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 10 Apr 2008 14:19:08 -0500 Subject: [Numpy-discussion] Infinity definitions In-Reply-To: <47FE55DF.3040201@enthought.com> References: <47FE2460.6050107@gmail.com> <47FE55DF.3040201@enthought.com> Message-ID: <47FE682C.2070606@gmail.com> Travis E. Oliphant wrote: > Bruce Southey wrote: > >> Hi, >> Since we are discussing namespace and standardization, I am curious in >> why there are multiple definitions for defining infinity in numpy when >> perhaps there should be two (one for positive infinity and one for >> negative infinity). I really do understand that other people have use of >> these definitions and that it is easier to leave them in than take them >> out. Also, it is minor reduction in namespace because I do know that >> much of the namespace is either defining variables (like different >> floats and complex numbers) or mathematical functions (like logs and >> trig functions). >> >> Currently we have: >> numpy.Inf >> numpy.Infinity >> numpy.inf >> numpy.infty >> numpy.NINF >> numpy.PINF >> >> Most of these are defined in numeric.py: 'Inf = inf = infty = Infinity = >> PINF' >> In the f2py/tests subdirectories, the files return_real.py and >> return_complex.py uses both 'inf','Infinity'. >> The only occurrence of NINF and PINF are in core/src/umathmodule.c but I >> don't see any other usage. >> There does not seem to be any use of 'infty'. >> >> > I think this is a product of bringing together a few definitions into > one and not forcing a standard. > > numpy.inf > numpy.nan > > should be used except for backward compatibility. > > -Travis > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > Hi, Thanks, Do you want me to file a ticket or something since it be nice to remove the extra definitions? Bruce From wfspotz at sandia.gov Thu Apr 10 15:22:48 2008 From: wfspotz at sandia.gov (Bill Spotz) Date: Thu, 10 Apr 2008 13:22:48 -0600 Subject: [Numpy-discussion] float32 is not a float ? In-Reply-To: <47FE5E8F.1040400@llnl.gov> References: <525f23e80804100537k1298f89eg29368b691ec7126c@mail.gmail.com> <504669.8295.qm@web34401.mail.mud.yahoo.com> <9457e7c80804101058m6890c59m2b1362165a55bed8@mail.gmail.com> <47FE5E8F.1040400@llnl.gov> Message-ID: isinstance() can take a tuple of classes/types as its second argument: In [1]: isinstance? Type: builtin_function_or_method Base Class: String Form: Namespace: Python builtin Docstring: isinstance(object, class-or-type-or-tuple) -> bool Return whether an object is an instance of a class or of a subclass thereof. With a type as second argument, return whether that is the object's type. The form using a tuple, isinstance(x, (A, B, ...)), is a shortcut for isinstance(x, A) or isinstance(x, B) or ... (etc.). On Apr 10, 2008, at 12:38 PM, Charles Doutriaux wrote: > Hello, > > I guess this maybe "normal" but it breaks a lot of thing when > conterting > from Numeric > >>>> a=numpy.ones(5,dtype=numpy.float32) >>>> isinstance(a[0],float) > False >>>> > > float64 works... > > I can see why one could argue for returning False, but then the > converter might be too zealous > things that used to work like: > if type(item) in [types.IntType, types.FloatType]: > > or: > > isinstance(item, types.FloatType) > > now fail, > > should we be concerned? should we consider returning True ? > > Thanks, > > C. ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From charlesr.harris at gmail.com Thu Apr 10 15:32:35 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 10 Apr 2008 13:32:35 -0600 Subject: [Numpy-discussion] float32 is not a float ? In-Reply-To: <47FE5E8F.1040400@llnl.gov> References: <525f23e80804100537k1298f89eg29368b691ec7126c@mail.gmail.com> <504669.8295.qm@web34401.mail.mud.yahoo.com> <9457e7c80804101058m6890c59m2b1362165a55bed8@mail.gmail.com> <47FE5E8F.1040400@llnl.gov> Message-ID: On Thu, Apr 10, 2008 at 12:38 PM, Charles Doutriaux wrote: > Hello, > > I guess this maybe "normal" but it breaks a lot of thing when conterting > from Numeric > > >>> a=numpy.ones(5,dtype=numpy.float32) > >>> isinstance(a[0],float) > False > >>> It looks like float in this case is a python float, not a numpy float. > > float64 works... It has the same underlying c type as the python float. Maybe it should fail? > > I can see why one could argue for returning False, but then the > converter might be too zealous > things that used to work like: > if type(item) in [types.IntType, types.FloatType]: > > or: > > isinstance(item, types.FloatType) > > now fail, > > > should we be concerned? should we consider returning True ? > I think you want the isreal function, but it will also return true for complex with 0 imaginary part. Hmm... the various iswhatever functions seem to be lacking in coverage. Maybe we should fix that. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at enthought.com Thu Apr 10 16:13:08 2008 From: travis at enthought.com (Travis Vaught) Date: Thu, 10 Apr 2008 15:13:08 -0500 Subject: [Numpy-discussion] [ANN] EuroSciPy Registration now open Message-ID: <238028CC-F3FA-4716-9027-A0C40EC5083D@enthought.com> Greetings, I'm pleased to announce that the registration for the first-annual EuroSciPy Conference is now open. http://scipy.org/EuroSciPy2008 Please take advantage of the early-bird rate and register soon. We'd love to have an early idea of attendance so that we can scale the venue appropriately (the available room is flexible in this regard). The EuroSciPy Conference will be held July 26-27, 2008 in Leipzig, Germany. About EuroSciPy --------------- EuroSciPy is designed to complement the popular SciPy Conferences which have been held for the last 7 years at Caltech (the 2008 SciPy Conference in the U.S. will be held the week of August 19-24). Similarly, the EuroSciPy Conference provides a unique opportunity to learn and affect what is happening in the realm of scientific computing with Python. Attendees will have the opportunity to review the available tools and how they apply to specific problems. By providing a forum for developers to share their Python expertise with the wider commercial, academic, and research communities, this conference fosters collaboration and facilitates the sharing of software components, techniques and a vision for high level language use in scientific computing. Typical presentations include general python use in the sciences, as well as NumPy and SciPy usage for general problem solving. Beyond the excellent talks, there are inter- session discussions that prove stimulating and helpful. Registration ------------ The direct link to the registration site is here: http://www.python-academy.com/euroscipy/index.html The registration fee will be 100.00? for early registrants and will increase to 150.00? for late registration (after June 15). Registration will include breakfast, snacks and lunch for Saturday and Sunday. Call for Participation ---------------------- If you are interested in presenting at the EuroSciPy Conference you may submit an abstract in Plain Text, PDF or MS Word formats to euroabstracts at scipy.org . The deadline for abstract submission is April 30,2008. Papers and/ or presentation slides are acceptable and are due by June 15, 2008. Presentations will be allotted 30 minutes. Please pass this announcement along to any other relevant contacts. Many Thanks, Travis N. Vaught From oliphant at enthought.com Thu Apr 10 16:34:09 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 10 Apr 2008 15:34:09 -0500 Subject: [Numpy-discussion] float32 is not a float ? In-Reply-To: References: <525f23e80804100537k1298f89eg29368b691ec7126c@mail.gmail.com> <504669.8295.qm@web34401.mail.mud.yahoo.com> <9457e7c80804101058m6890c59m2b1362165a55bed8@mail.gmail.com> <47FE5E8F.1040400@llnl.gov> Message-ID: <47FE79C1.60307@enthought.com> Charles R Harris wrote: > > > On Thu, Apr 10, 2008 at 12:38 PM, Charles Doutriaux > > wrote: > > Hello, > > I guess this maybe "normal" but it breaks a lot of thing when > conterting > from Numeric > > >>> a=numpy.ones(5,dtype=numpy.float32) > >>> isinstance(a[0],float) > False > >>> > > > It looks like float in this case is a python float, not a numpy float. > > > > float64 works... > > > It has the same underlying c type as the python float. Maybe it should > fail? float64 does inherit from float so isinstance succeeds. > > > > I can see why one could argue for returning False, but then the > converter might be too zealous > things that used to work like: > if type(item) in [types.IntType, types.FloatType]: > > or: > > isinstance(item, types.FloatType) > > now fail, > > > > > > should we be concerned? should we consider returning True ? > > > I think you want the isreal function, but it will also return true for > complex with 0 imaginary part. Hmm... the various iswhatever functions > seem to be lacking in coverage. Maybe we should fix that. We definitely could try and fix it. From tjhnson at gmail.com Thu Apr 10 15:56:44 2008 From: tjhnson at gmail.com (Tom Johnson) Date: Thu, 10 Apr 2008 12:56:44 -0700 Subject: [Numpy-discussion] commutative allclose Message-ID: Should allclose() be commutative, so as to prevent the following: >>> x = 1.00001001 >>> allclose(x,1), allclose(1,x) (False, True) There is some discussion here which provides two possible solutions: http://www.boost.org/doc/libs/1_35_0/libs/test/doc/components/test_tools/floating_point_comparison.html Notice, the discussion states that their solutions are not transitive---nevertheless, I think commutativity is a worthwhile improvement. Also, they mention that rtol * abs(y) can cause underflow issues. Thus, they implement, |x-y|/|y| <= rtol AND(OR) |x-y|/|x| <= rtol without an atol option, rather than |x-y| <= atol + rtol* |y| Naively, it seems like atol is attempting to correct this very issue. So do we really need atol? From pav at iki.fi Thu Apr 10 17:00:29 2008 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 11 Apr 2008 00:00:29 +0300 Subject: [Numpy-discussion] #734: interactive docstring search (lookfor) Message-ID: <47FE7FED.2060701@iki.fi> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Steven H. Rogers wrote: > There's definitely a place for docsearch from the shell. There's > opportunity for context sensitive search that Sphinx generated docs > couldn't easily duplicate. I think this is a good idea: full-namespace docstring search ? la Matlab lookfor. I wrote a quick implementation for numpy here: http://scipy.org/scipy/numpy/ticket/734 Example of use: >>> import numpy as np >>> np.lookfor("eigenvalue hermitean") Search results for 'eigenvalue hermitean' - ----------------------------------------- numpy.linalg.eigvalsh Compute the eigenvalues of a Hermitean or real symmetric matrix. numpy.linalg.eigh Compute eigenvalues for a Hermitian or real symmetric matrix. numpy.linalg.eigvals Compute the eigenvalues of a general matrix. >>> np.lookfor("least squares", "scipy") Search results for 'least squares' - ---------------------------------- scipy.polyfit Least squares polynomial fit. scipy.linalg.lstsq Compute least-squares solution to equation :m:`a x = b` scipy.linalg.pinv Compute the (Moore-Penrose) pseudo-inverse of a matrix. scipy.optimize.leastsq Minimize the sum of squares of a set of equations. scipy.optimize.fmin_slsqp Minimize a function using Sequential Least SQuares Programming ... more results, shown in a pager ... - -- Pauli Virtanen -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFH/n/t6BQxb7O0pWARAgQeAJ9K5RJB3G2hbJW7tuVtAHMbAMzM3QCfQrEp VLnmxqtEwP2jRPpqdsWDugk= =7t1z -----END PGP SIGNATURE----- From oliphant at enthought.com Thu Apr 10 17:23:57 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 10 Apr 2008 16:23:57 -0500 Subject: [Numpy-discussion] #734: interactive docstring search (lookfor) In-Reply-To: <47FE7FED.2060701@iki.fi> References: <47FE7FED.2060701@iki.fi> Message-ID: <47FE856D.5050300@enthought.com> Pauli Virtanen wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Steven H. Rogers wrote: > >> There's definitely a place for docsearch from the shell. There's >> opportunity for context sensitive search that Sphinx generated docs >> couldn't easily duplicate. >> > > I think this is a good idea: full-namespace docstring search ? la Matlab > lookfor. I wrote a quick implementation for numpy here: > http://scipy.org/scipy/numpy/ticket/734 > Cool. I started scipy.misc.info a long time ago to try and do this. I didn't advertise it well enough ;-) scipy.misc.info('eigvals') I'm happy to move this functionality into numpy (but it might be better in IPython). -Travis From pav at iki.fi Thu Apr 10 17:42:05 2008 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 11 Apr 2008 00:42:05 +0300 Subject: [Numpy-discussion] #734: interactive docstring search (lookfor) In-Reply-To: <47FE856D.5050300@enthought.com> References: <47FE7FED.2060701@iki.fi> <47FE856D.5050300@enthought.com> Message-ID: <47FE89AD.8010708@iki.fi> Travis E. Oliphant kirjoitti: > Pauli Virtanen wrote: [clip] >> I think this is a good idea: full-namespace docstring search ? la Matlab >> lookfor. I wrote a quick implementation for numpy here: >> http://scipy.org/scipy/numpy/ticket/734 >> > > Cool. I started scipy.misc.info a long time ago to try and do this. I > didn't advertise it well enough ;-) > > scipy.misc.info('eigvals') > > I'm happy to move this functionality into numpy (but it might be better > in IPython). Aha! I was wondering if something this simple was already present, but I didn't notice that numpy.lib.utils.info also accepted strings as parameters. Btw, the info function is already in numpy and the code is duplicated in scipy.misc. However, "info" is a bit different from "lookfor" in that it finds documentation by the exact function name, not by looking into the docstrings. So, should info and lookfor be combined, or kept separate? -- Pauli Virtanen From charlesr.harris at gmail.com Thu Apr 10 17:49:46 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 10 Apr 2008 15:49:46 -0600 Subject: [Numpy-discussion] commutative allclose In-Reply-To: References: Message-ID: On Thu, Apr 10, 2008 at 1:56 PM, Tom Johnson wrote: > Should allclose() be commutative, so as to prevent the following: > > >>> x = 1.00001001 > >>> allclose(x,1), allclose(1,x) > (False, True) > > There is some discussion here which provides two possible solutions: > > > http://www.boost.org/doc/libs/1_35_0/libs/test/doc/components/test_tools/floating_point_comparison.html > > Notice, the discussion states that their solutions are not > transitive---nevertheless, I think commutativity is a worthwhile > improvement. Also, they mention that rtol * abs(y) can cause > underflow issues. Thus, they implement, > > |x-y|/|y| <= rtol AND(OR) |x-y|/|x| <= rtol > > without an atol option, rather than > > |x-y| <= atol + rtol* |y| > > Naively, it seems like atol is attempting to correct this very issue. > So do we really need atol? I think it useful, especially when working with values that may be zero. However, it might not be necessary in the allclose context. On the other point, it is also possible to use the max, avg, or sum of |x| and |y|, which will avoid the case when on of them gets too small. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From hoytak at gmail.com Thu Apr 10 17:54:48 2008 From: hoytak at gmail.com (Hoyt Koepke) Date: Thu, 10 Apr 2008 14:54:48 -0700 Subject: [Numpy-discussion] #734: interactive docstring search (lookfor) In-Reply-To: <47FE89AD.8010708@iki.fi> References: <47FE7FED.2060701@iki.fi> <47FE856D.5050300@enthought.com> <47FE89AD.8010708@iki.fi> Message-ID: <4db580fd0804101454rdce6d36q811085d8ac9f4b55@mail.gmail.com> My personal 2 cents is that we should put it in the most easy-to-find place. I wish I had the propsed lookfor function at my fingertips when I started to learn numpy/scipy a year ago, coming from matlab. Like many others (I assume), I just jumped in and learned what numpy could do by trial and error and I never noticed numpy.lib.utils.info. Thus my personal vote (though I understand this may not be the best and there are arguments against it) is to put a lookfor or info function in the top level namespace, with an obvious name, so newbies could easily run across it and have a more enjoyable numpy learning experience. That's just my humble opinion ... --Hoyt On Thu, Apr 10, 2008 at 2:42 PM, Pauli Virtanen wrote: > Travis E. Oliphant kirjoitti: > > Pauli Virtanen wrote: > [clip] > > >> I think this is a good idea: full-namespace docstring search ? la Matlab > >> lookfor. I wrote a quick implementation for numpy here: > >> http://scipy.org/scipy/numpy/ticket/734 > >> > > > > Cool. I started scipy.misc.info a long time ago to try and do this. I > > didn't advertise it well enough ;-) > > > > scipy.misc.info('eigvals') > > > > I'm happy to move this functionality into numpy (but it might be better > > in IPython). > > Aha! I was wondering if something this simple was already present, but I > didn't notice that numpy.lib.utils.info also accepted strings as > parameters. Btw, the info function is already in numpy and the code is > duplicated in scipy.misc. > > However, "info" is a bit different from "lookfor" in that it finds > documentation by the exact function name, not by looking into the > docstrings. > > So, should info and lookfor be combined, or kept separate? > > -- > Pauli Virtanen > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- +++++++++++++++++++++++++++++++++++ Hoyt Koepke UBC Department of Computer Science http://www.cs.ubc.ca/~hoytak/ hoytak at gmail.com +++++++++++++++++++++++++++++++++++ From robert.kern at gmail.com Thu Apr 10 17:59:17 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 10 Apr 2008 16:59:17 -0500 Subject: [Numpy-discussion] #734: interactive docstring search (lookfor) In-Reply-To: <4db580fd0804101454rdce6d36q811085d8ac9f4b55@mail.gmail.com> References: <47FE7FED.2060701@iki.fi> <47FE856D.5050300@enthought.com> <47FE89AD.8010708@iki.fi> <4db580fd0804101454rdce6d36q811085d8ac9f4b55@mail.gmail.com> Message-ID: <3d375d730804101459p384dee22sbb20a8afd00c9015@mail.gmail.com> On Thu, Apr 10, 2008 at 4:54 PM, Hoyt Koepke wrote: > My personal 2 cents is that we should put it in the most easy-to-find > place. I wish I had the propsed lookfor function at my fingertips > when I started to learn numpy/scipy a year ago, coming from matlab. > Like many others (I assume), I just jumped in and learned what numpy > could do by trial and error and I never noticed numpy.lib.utils.info. > Thus my personal vote (though I understand this may not be the best > and there are arguments against it) is to put a lookfor or info > function in the top level namespace, with an obvious name, so newbies > could easily run across it and have a more enjoyable numpy learning > experience. In [1]: import numpy In [2]: numpy.info('eigvals') *** Found in numpy.linalg *** eigvals(a) ... -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From gael.varoquaux at normalesup.org Thu Apr 10 17:58:43 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 10 Apr 2008 23:58:43 +0200 Subject: [Numpy-discussion] #734: interactive docstring search (lookfor) In-Reply-To: <4db580fd0804101454rdce6d36q811085d8ac9f4b55@mail.gmail.com> References: <47FE7FED.2060701@iki.fi> <47FE856D.5050300@enthought.com> <47FE89AD.8010708@iki.fi> <4db580fd0804101454rdce6d36q811085d8ac9f4b55@mail.gmail.com> Message-ID: <20080410215843.GG30722@phare.normalesup.org> On Thu, Apr 10, 2008 at 02:54:48PM -0700, Hoyt Koepke wrote: > My personal 2 cents is that we should put it in the most easy-to-find > place. I wish I had the propsed lookfor function at my fingertips > when I started to learn numpy/scipy a year ago, coming from matlab. > Like many others (I assume), I just jumped in and learned what numpy > could do by trial and error and I never noticed numpy.lib.utils.info. > Thus my personal vote (though I understand this may not be the best > and there are arguments against it) is to put a lookfor or info > function in the top level namespace, with an obvious name, so newbies > could easily run across it and have a more enjoyable numpy learning > experience. Numpy is not the place where this should go. This should go in eg ipython, which provides an interactiv frontend. One of the reasons is that this is certainly not numpy-dependent. We could add it for now because we really, really want it, and depreciate it in the future. This would be very bad because users would start relying on it. My suggestion is that you write a self-contained file that provides this feature. You put it on the wiki for people to download, and you post to the ipython-dev suggesting it for inclusion. These guys are quite reactiv and they will adapt your work to fit into ipython. Worst case, if nobody does it, I commit to doing it (I have lots of over things, FOSS-related to do, so I'd rather avoid, but if I have to...). My 2 cents, Ga?l From hoytak at gmail.com Thu Apr 10 18:08:33 2008 From: hoytak at gmail.com (Hoyt Koepke) Date: Thu, 10 Apr 2008 15:08:33 -0700 Subject: [Numpy-discussion] #734: interactive docstring search (lookfor) In-Reply-To: <3d375d730804101459p384dee22sbb20a8afd00c9015@mail.gmail.com> References: <47FE7FED.2060701@iki.fi> <47FE856D.5050300@enthought.com> <47FE89AD.8010708@iki.fi> <4db580fd0804101454rdce6d36q811085d8ac9f4b55@mail.gmail.com> <3d375d730804101459p384dee22sbb20a8afd00c9015@mail.gmail.com> Message-ID: <4db580fd0804101508v5b1df13ere9ee5e731ed33642@mail.gmail.com> > In [1]: import numpy > > In [2]: numpy.info('eigvals') > *** Found in numpy.linalg *** > eigvals(a) Fair enough.... Don't know why I missed that, prob relied too much on Google search :-) Having it as part of iPython does make sense. --Hoyt From fperez.net at gmail.com Thu Apr 10 18:12:08 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 10 Apr 2008 15:12:08 -0700 Subject: [Numpy-discussion] #734: interactive docstring search (lookfor) In-Reply-To: <4db580fd0804101508v5b1df13ere9ee5e731ed33642@mail.gmail.com> References: <47FE7FED.2060701@iki.fi> <47FE856D.5050300@enthought.com> <47FE89AD.8010708@iki.fi> <4db580fd0804101454rdce6d36q811085d8ac9f4b55@mail.gmail.com> <3d375d730804101459p384dee22sbb20a8afd00c9015@mail.gmail.com> <4db580fd0804101508v5b1df13ere9ee5e731ed33642@mail.gmail.com> Message-ID: On Thu, Apr 10, 2008 at 3:08 PM, Hoyt Koepke wrote: > > In [1]: import numpy > > > > In [2]: numpy.info('eigvals') > > *** Found in numpy.linalg *** > > eigvals(a) > > Fair enough.... Don't know why I missed that, prob relied too much on > Google search :-) > > Having it as part of iPython does make sense. bit swamped ritght now, but just to say that we'll be happy to put it in... f From wilson.t.thompson at gmail.com Thu Apr 10 18:18:17 2008 From: wilson.t.thompson at gmail.com (wilson) Date: Thu, 10 Apr 2008 15:18:17 -0700 (PDT) Subject: [Numpy-discussion] meaning of accumulation/normalisation Message-ID: <8fe56208-917c-4b6d-83e1-74ce7117081f@u36g2000prf.googlegroups.com> hi i came across some image processing code in java and tried to duplicate the operation on an ndarray.the array is supposed to contain pixel values of a gryscale image generated by the program. however the code does some accumulation operation as below to obtain a value 'norm' ul=array(([....],[....]....,[...])) #containing float values def accumOperation(ul): nr,nc=ul.shape print "nr:",nr,"nc:",nc val=0.0 for r in range(nr): for c in range(nc): val+=ul[r][c]*ul[r][c] return val norm=accumOperation(ul) newul=ul/norm the java doc mentions that by the above steps ul is normalised to unit length (vector length) can someone explain this?it is not very clear to me and (i am trying to understand the code by reading and tying out in debugger.. the newul array content is then used to generate a pgm image in the code) thanks W From Joris.DeRidder at ster.kuleuven.be Thu Apr 10 18:29:55 2008 From: Joris.DeRidder at ster.kuleuven.be (Joris De Ridder) Date: Fri, 11 Apr 2008 00:29:55 +0200 Subject: [Numpy-discussion] #734: interactive docstring search (lookfor) In-Reply-To: <47FE856D.5050300@enthought.com> References: <47FE7FED.2060701@iki.fi> <47FE856D.5050300@enthought.com> Message-ID: <29B13911-1C9E-41FA-B0AB-7CF2FC0677AF@ster.kuleuven.be> On 10 Apr 2008, at 23:23, Travis E. Oliphant wrote: > Cool. I started scipy.misc.info a long time ago to try and do > this. I > didn't advertise it well enough ;-) Yep, I also started to write my own docsearch tool but neglected to advertise it. On 10 Apr 2008, at 23:58, Gael Varoquaux wrote: > Numpy is not the place where this should go. This should go in eg > ipython, which provides an interactiv frontend. One of the reasons is > that this is certainly not numpy-dependent. I kind of disagree here. Such a tool can be made much more intelligent and thus useful if it really hooks into NumPy/SciPy. Provide "See also" functionality, give info on categories of functions, use magic words, etc. None of these things is too difficult to implement. However, such capabilities are beyond the more general docstring search tool that would be suitable for IPython. Of course, no problem if IPython is willing to special-case NumPy/SciPy. Joris Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm From pav at iki.fi Thu Apr 10 18:34:46 2008 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 11 Apr 2008 01:34:46 +0300 Subject: [Numpy-discussion] #734: interactive docstring search (lookfor) In-Reply-To: References: <47FE7FED.2060701@iki.fi> <47FE856D.5050300@enthought.com> <47FE89AD.8010708@iki.fi> <4db580fd0804101454rdce6d36q811085d8ac9f4b55@mail.gmail.com> <3d375d730804101459p384dee22sbb20a8afd00c9015@mail.gmail.com> <4db580fd0804101508v5b1df13ere9ee5e731ed33642@mail.gmail.com> Message-ID: <47FE9606.10500@iki.fi> Fernando Perez kirjoitti: > On Thu, Apr 10, 2008 at 3:08 PM, Hoyt Koepke wrote: >>> In [1]: import numpy >> > >> > In [2]: numpy.info('eigvals') >> > *** Found in numpy.linalg *** >> > eigvals(a) >> >> Fair enough.... Don't know why I missed that, prob relied too much on >> Google search :-) >> >> Having it as part of iPython does make sense. > > bit swamped ritght now, but just to say that we'll be happy to put it in... I wrote an Ipython0 extension implementing %lookfor, and filed it in the ticket http://ipython.scipy.org/ipython/ipython/ticket/245 so that I don't forget about it. (Download the file, and import ipy_lookfor.py to get a magic %lookfor command.) It's a bit rough right now, but I don't have time to finish it today. -- Pauli Virtanen From fperez.net at gmail.com Thu Apr 10 18:47:56 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 10 Apr 2008 15:47:56 -0700 Subject: [Numpy-discussion] #734: interactive docstring search (lookfor) In-Reply-To: <29B13911-1C9E-41FA-B0AB-7CF2FC0677AF@ster.kuleuven.be> References: <47FE7FED.2060701@iki.fi> <47FE856D.5050300@enthought.com> <29B13911-1C9E-41FA-B0AB-7CF2FC0677AF@ster.kuleuven.be> Message-ID: On Thu, Apr 10, 2008 at 3:29 PM, Joris De Ridder wrote: > > On 10 Apr 2008, at 23:23, Travis E. Oliphant wrote: > > > Cool. I started scipy.misc.info a long time ago to try and do > > this. I > > didn't advertise it well enough ;-) > > Yep, I also started to write my own docsearch tool but neglected to > advertise it. > > > > > On 10 Apr 2008, at 23:58, Gael Varoquaux wrote: > > > Numpy is not the place where this should go. This should go in eg > > ipython, which provides an interactiv frontend. One of the reasons is > > that this is certainly not numpy-dependent. > > I kind of disagree here. Such a tool can be made much more intelligent > and thus useful if it really hooks into NumPy/SciPy. Provide "See > also" functionality, give info on categories of functions, use magic > words, etc. None of these things is too difficult to implement. > However, such capabilities are beyond the more general docstring > search tool that would be suitable for IPython. Of course, no problem > if IPython is willing to special-case NumPy/SciPy. Given how I happen to make a living with numpy/scipy, the answer would be yes ;) A slightly more proper answer: we can have this special-casing easily available without 'polluting' the ipython core code, since ipython can load fairly self-contained features, even automatically. So this extension (which Paul just provided as an ipython ticket, thanks!!) can be loaded only if numpy is around for use. Cheers, f From charlesr.harris at gmail.com Thu Apr 10 19:20:44 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 10 Apr 2008 17:20:44 -0600 Subject: [Numpy-discussion] meaning of accumulation/normalisation In-Reply-To: <8fe56208-917c-4b6d-83e1-74ce7117081f@u36g2000prf.googlegroups.com> References: <8fe56208-917c-4b6d-83e1-74ce7117081f@u36g2000prf.googlegroups.com> Message-ID: On Thu, Apr 10, 2008 at 4:18 PM, wilson wrote: > hi > i came across some image processing code in java and tried to > duplicate the operation on an ndarray.the array is supposed to contain > pixel values of a gryscale image generated by the program. however the > code does some accumulation operation as below to obtain a value > 'norm' > > ul=array(([....],[....]....,[...])) #containing float values > > def accumOperation(ul): > nr,nc=ul.shape > print "nr:",nr,"nc:",nc > val=0.0 > for r in range(nr): > for c in range(nc): > val+=ul[r][c]*ul[r][c] > return val > norm=accumOperation(ul) > newul=ul/norm > > the java doc mentions that by the above steps ul is normalised to unit > length (vector length) Umm, not quite, it is missing a square root. You can get the same result by using the Frobenius norm NORM(X,'fro') is the Frobenius norm, sqrt(sum(diag(X'*X))) Which is matlab speak for sqrt(trace(dot(X.T, X))) or sqrt(inner(X.flat,X.flat)), but you can get it easier using norm(X,'fro'). I don't know why your code is missing a sqrt unless it drops out somewhere else. Basically, just treat an nxn array as a vector of length n**2. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Thu Apr 10 19:43:25 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 10 Apr 2008 16:43:25 -0700 Subject: [Numpy-discussion] #734: interactive docstring search (lookfor) In-Reply-To: <47FE856D.5050300@enthought.com> References: <47FE7FED.2060701@iki.fi> <47FE856D.5050300@enthought.com> Message-ID: <47FEA61D.4040004@noaa.gov> Travis E. Oliphant wrote: >> I think this is a good idea: full-namespace docstring search ? la Matlab >> lookfor. I wrote a quick implementation for numpy here: >> http://scipy.org/scipy/numpy/ticket/734 cool! > I'm happy to move this functionality into numpy (but it might be better > in IPython). Is there anything numpy/scypi specific about it? I d like to use this kind of thing with other packages, in which case it definitely does not belong in numpy or scipy. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From robert.kern at gmail.com Thu Apr 10 19:44:00 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 10 Apr 2008 18:44:00 -0500 Subject: [Numpy-discussion] #734: interactive docstring search (lookfor) In-Reply-To: <47FEA61D.4040004@noaa.gov> References: <47FE7FED.2060701@iki.fi> <47FE856D.5050300@enthought.com> <47FEA61D.4040004@noaa.gov> Message-ID: <3d375d730804101644g48b60292r8256f74013709ec2@mail.gmail.com> On Thu, Apr 10, 2008 at 6:43 PM, Christopher Barker wrote: > Is there anything numpy/scypi specific about it? I > d like to use this kind of thing with other packages, in which case it > definitely does not belong in numpy or scipy. Basically, it's a configuration issue, I think. The tool needs to know which packages to search in because it can't search every installed package (it needs to load them all!). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From lee.will at gmail.com Thu Apr 10 19:46:40 2008 From: lee.will at gmail.com (Will Lee) Date: Thu, 10 Apr 2008 18:46:40 -0500 Subject: [Numpy-discussion] problem with float64's str() In-Reply-To: References: <7f03db650804031231t39aa3000o719db8359a7769c7@mail.gmail.com> <9457e7c80804040314t1db5e821h56ea782245cd4437@mail.gmail.com> <47F6305C.7060901@gmail.com> <47F63291.3030509@student.matnat.uio.no> <7f03db650804040756q20d0ad79x964b05fbad6fd8fb@mail.gmail.com> <3d375d730804041247q25938d50h5083474a47f51a99@mail.gmail.com> Message-ID: <7f03db650804101646m4066910w95425d96bb555f87@mail.gmail.com> Thanks for all the comments about this issue. Do you know if there's a ticket that's open for this? Is this an easy fix before the 1.0.5 release? Thanks, Will On Fri, Apr 4, 2008 at 3:40 PM, Timothy Hochberg wrote: > > > On Fri, Apr 4, 2008 at 12:47 PM, Robert Kern > wrote: > > > On Fri, Apr 4, 2008 at 9:56 AM, Will Lee wrote: > > > I understand the implication for the floating point comparison and the > > need > > > for allclose. However, I think in a doctest context, this behavior > > makes > > > the doc much harder to read. > > > > Tabling the issue of the fact that we changed behavior for a moment, > > this is a fundamental problem with using doctests as unit tests for > > numerical code. The floating point results that you get *will* be > > different on different machines, but the code will still be correct. > > Using allclose() and similar techniques are the best tools available > > (although they still suck). Relying on visual representations of these > > results is simply an untenable strategy. > > > That is sometimes, but not always the case. Why? Because most of the time > that one ends up with simple values, one is starting with arbitrary floating > point values and doing at most simple operations on them. Thus a strategy > that helps many of my unit tests look better and function reliably is to > choose values that can be represented exactly in floating point. If the > original value here had been 0.00125 rather than .0012, there would be no > problem here. Well almost, you still are vulnerable to the rules for zero > padding and what no getting changed and so forth, but in general it's more > reliable and prettier. > > Of course this isn't always a solution. But I've found it's helpful for a > lot cases. > > Note that the string > > representation of NaNs and Infs are completely different across > > platforms. > > > > That said, str(float_numpy_scalar) really should have the same rules > > as str(some_python_float). > > > +1 > > > > > > > > > > -- > > Robert Kern > > > > "I have come to believe that the whole world is an enigma, a harmless > > enigma that is made terrible by our own mad attempt to interpret it as > > though it had an underlying truth." > > -- Umberto Eco > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > -- > . __ > . |-\ > . > . tim.hochberg at ieee.org > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Apr 10 20:31:47 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 10 Apr 2008 18:31:47 -0600 Subject: [Numpy-discussion] problem with float64's str() In-Reply-To: <3d375d730804041247q25938d50h5083474a47f51a99@mail.gmail.com> References: <7f03db650804031231t39aa3000o719db8359a7769c7@mail.gmail.com> <9457e7c80804040314t1db5e821h56ea782245cd4437@mail.gmail.com> <47F6305C.7060901@gmail.com> <47F63291.3030509@student.matnat.uio.no> <7f03db650804040756q20d0ad79x964b05fbad6fd8fb@mail.gmail.com> <3d375d730804041247q25938d50h5083474a47f51a99@mail.gmail.com> Message-ID: On Fri, Apr 4, 2008 at 1:47 PM, Robert Kern wrote: > On Fri, Apr 4, 2008 at 9:56 AM, Will Lee wrote: > > I understand the implication for the floating point comparison and the > need > > for allclose. However, I think in a doctest context, this behavior > makes > > the doc much harder to read. > > Tabling the issue of the fact that we changed behavior for a moment, > this is a fundamental problem with using doctests as unit tests for > numerical code. The floating point results that you get *will* be > different on different machines, but the code will still be correct. > Using allclose() and similar techniques are the best tools available > (although they still suck). Relying on visual representations of these > results is simply an untenable strategy. Note that the string > representation of NaNs and Infs are completely different across > platforms. > > That said, str(float_numpy_scalar) really should have the same rules > as str(some_python_float). > For all different precisions? And what should the rules be. I note that numpy doesn't distinguish between repr and str, maybe we could specify different behavior for the two. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Apr 10 20:38:45 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 10 Apr 2008 19:38:45 -0500 Subject: [Numpy-discussion] problem with float64's str() In-Reply-To: References: <7f03db650804031231t39aa3000o719db8359a7769c7@mail.gmail.com> <9457e7c80804040314t1db5e821h56ea782245cd4437@mail.gmail.com> <47F6305C.7060901@gmail.com> <47F63291.3030509@student.matnat.uio.no> <7f03db650804040756q20d0ad79x964b05fbad6fd8fb@mail.gmail.com> <3d375d730804041247q25938d50h5083474a47f51a99@mail.gmail.com> Message-ID: <3d375d730804101738t676872acnf3ef99c954b7e504@mail.gmail.com> On Thu, Apr 10, 2008 at 7:31 PM, Charles R Harris wrote: > > That said, str(float_numpy_scalar) really should have the same rules > > as str(some_python_float). > > For all different precisions? No. I should have said str(float64_numpy_scalar). I am content to leave the other types alone. > And what should the rules be. All Python does is use a lower decimal precision for __str__ than __repr__. > I note that > numpy doesn't distinguish between repr and str, maybe we could specify > different behavior for the two. Yes, precisely. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Thu Apr 10 20:57:26 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 10 Apr 2008 18:57:26 -0600 Subject: [Numpy-discussion] problem with float64's str() In-Reply-To: <3d375d730804101738t676872acnf3ef99c954b7e504@mail.gmail.com> References: <7f03db650804031231t39aa3000o719db8359a7769c7@mail.gmail.com> <9457e7c80804040314t1db5e821h56ea782245cd4437@mail.gmail.com> <47F6305C.7060901@gmail.com> <47F63291.3030509@student.matnat.uio.no> <7f03db650804040756q20d0ad79x964b05fbad6fd8fb@mail.gmail.com> <3d375d730804041247q25938d50h5083474a47f51a99@mail.gmail.com> <3d375d730804101738t676872acnf3ef99c954b7e504@mail.gmail.com> Message-ID: On Thu, Apr 10, 2008 at 6:38 PM, Robert Kern wrote: > On Thu, Apr 10, 2008 at 7:31 PM, Charles R Harris > wrote: > > > That said, str(float_numpy_scalar) really should have the same rules > > > as str(some_python_float). > > > > For all different precisions? > > No. I should have said str(float64_numpy_scalar). I am content to > leave the other types alone. > > > And what should the rules be. > > All Python does is use a lower decimal precision for __str__ than > __repr__. > > > I note that > > numpy doesn't distinguish between repr and str, maybe we could specify > > different behavior for the two. > > Yes, precisely. > Well, I know where to do that and have a ticket for it. What I would also like to do is use float.h for setting the repr precision, but I am not sure I can count on its presence as it only became part of the spec in 1999. Then again, that's almost ten years ago. Anyway, python on my machine generates 12 significant digits. Is that common to everyone? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Apr 10 21:06:39 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 10 Apr 2008 20:06:39 -0500 Subject: [Numpy-discussion] problem with float64's str() In-Reply-To: References: <7f03db650804031231t39aa3000o719db8359a7769c7@mail.gmail.com> <9457e7c80804040314t1db5e821h56ea782245cd4437@mail.gmail.com> <47F6305C.7060901@gmail.com> <47F63291.3030509@student.matnat.uio.no> <7f03db650804040756q20d0ad79x964b05fbad6fd8fb@mail.gmail.com> <3d375d730804041247q25938d50h5083474a47f51a99@mail.gmail.com> <3d375d730804101738t676872acnf3ef99c954b7e504@mail.gmail.com> Message-ID: <3d375d730804101806w5fced280g95365a55b950ab64@mail.gmail.com> On Thu, Apr 10, 2008 at 7:57 PM, Charles R Harris wrote: > > On Thu, Apr 10, 2008 at 6:38 PM, Robert Kern wrote: > > > > On Thu, Apr 10, 2008 at 7:31 PM, Charles R Harris > > wrote: > > > > That said, str(float_numpy_scalar) really should have the same rules > > > > as str(some_python_float). > > > > > > For all different precisions? > > > > No. I should have said str(float64_numpy_scalar). I am content to > > leave the other types alone. > > > > > And what should the rules be. > > > > All Python does is use a lower decimal precision for __str__ than > __repr__. > > > > > > > I note that > > > numpy doesn't distinguish between repr and str, maybe we could specify > > > different behavior for the two. > > > > Yes, precisely. > > Well, I know where to do that and have a ticket for it. What I would also > like to do is use float.h for setting the repr precision, but I am not sure > I can count on its presence as it only became part of the spec in 1999. Then > again, that's almost ten years ago. Anyway, python on my machine generates > 12 significant digits. Is that common to everyone? Here is the relevant portion of Objects/floatobject.c: /* Precisions used by repr() and str(), respectively. The repr() precision (17 significant decimal digits) is the minimal number that is guaranteed to have enough precision so that if the number is read back in the exact same binary value is recreated. This is true for IEEE floating point by design, and also happens to work for all other modern hardware. The str() precision is chosen so that in most cases, the rounding noise created by various operations is suppressed, while giving plenty of precision for practical use. */ #define PREC_REPR 17 #define PREC_STR 12 svn blame tells me that those have been there unchanged since 1999. You may want to steal the function format_float() that is defined in that file, too. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at enthought.com Thu Apr 10 21:09:03 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 10 Apr 2008 20:09:03 -0500 Subject: [Numpy-discussion] #734: interactive docstring search (lookfor) In-Reply-To: <47FE89AD.8010708@iki.fi> References: <47FE7FED.2060701@iki.fi> <47FE856D.5050300@enthought.com> <47FE89AD.8010708@iki.fi> Message-ID: <47FEBA2F.2050608@enthought.com> Pauli Virtanen wrote: > Travis E. Oliphant kirjoitti: > >> Pauli Virtanen wrote: >> > [clip] > >>> I think this is a good idea: full-namespace docstring search ? la Matlab >>> lookfor. I wrote a quick implementation for numpy here: >>> http://scipy.org/scipy/numpy/ticket/734 >>> >>> >> Cool. I started scipy.misc.info a long time ago to try and do this. I >> didn't advertise it well enough ;-) >> >> scipy.misc.info('eigvals') >> >> I'm happy to move this functionality into numpy (but it might be better >> in IPython). >> > > Aha! I was wondering if something this simple was already present, but I > didn't notice that numpy.lib.utils.info also accepted strings as > parameters. Btw, the info function is already in numpy and the code is > duplicated in scipy.misc. > > However, "info" is a bit different from "lookfor" in that it finds > documentation by the exact function name, not by looking into the > docstrings. > > So, should info and lookfor be combined, or kept separate? > > combined --- i.e. info should be replace by lookfor -Travis From charlesr.harris at gmail.com Thu Apr 10 21:25:23 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 10 Apr 2008 19:25:23 -0600 Subject: [Numpy-discussion] problem with float64's str() In-Reply-To: <3d375d730804101806w5fced280g95365a55b950ab64@mail.gmail.com> References: <7f03db650804031231t39aa3000o719db8359a7769c7@mail.gmail.com> <9457e7c80804040314t1db5e821h56ea782245cd4437@mail.gmail.com> <47F6305C.7060901@gmail.com> <47F63291.3030509@student.matnat.uio.no> <7f03db650804040756q20d0ad79x964b05fbad6fd8fb@mail.gmail.com> <3d375d730804041247q25938d50h5083474a47f51a99@mail.gmail.com> <3d375d730804101738t676872acnf3ef99c954b7e504@mail.gmail.com> <3d375d730804101806w5fced280g95365a55b950ab64@mail.gmail.com> Message-ID: On Thu, Apr 10, 2008 at 7:06 PM, Robert Kern wrote: > On Thu, Apr 10, 2008 at 7:57 PM, Charles R Harris > wrote: > > > > On Thu, Apr 10, 2008 at 6:38 PM, Robert Kern > wrote: > > > > > > On Thu, Apr 10, 2008 at 7:31 PM, Charles R Harris > > > wrote: > > > > > That said, str(float_numpy_scalar) really should have the same > rules > > > > > as str(some_python_float). > > > > > > > > For all different precisions? > > > > > > No. I should have said str(float64_numpy_scalar). I am content to > > > leave the other types alone. > > > > > > > And what should the rules be. > > > > > > All Python does is use a lower decimal precision for __str__ than > > __repr__. > > > > > > > > > > I note that > > > > numpy doesn't distinguish between repr and str, maybe we could > specify > > > > different behavior for the two. > > > > > > Yes, precisely. > > > > Well, I know where to do that and have a ticket for it. What I would > also > > like to do is use float.h for setting the repr precision, but I am not > sure > > I can count on its presence as it only became part of the spec in 1999. > Then > > again, that's almost ten years ago. Anyway, python on my machine > generates > > 12 significant digits. Is that common to everyone? > > Here is the relevant portion of Objects/floatobject.c: > > /* Precisions used by repr() and str(), respectively. > > The repr() precision (17 significant decimal digits) is the minimal > number > that is guaranteed to have enough precision so that if the number is > read > back in the exact same binary value is recreated. This is true for IEEE > floating point by design, and also happens to work for all other modern > hardware. > > The str() precision is chosen so that in most cases, the rounding noise > created by various operations is suppressed, while giving plenty of > precision for practical use. > > */ > > #define PREC_REPR 17 > #define PREC_STR 12 > > > > svn blame tells me that those have been there unchanged since 1999. > I left this note on my ticket. These values should really be determined at compile time, not hardwired in at lines 611-621 of scalartypes.inc.src. Maybe use the values in float.h, which on my machine give single digits 6 double digits 15 long double digits 18 The current values we are using are 8, 17, and 22 whereas the values above are supposed to guarantee reversible conversion to and from decimal. Of course, that doesn't seem to be the case in practice, they seem to need at least one more digit. The other question is if all the common compilers support float.h The numbers above were generated by #include #include int main(int argc, char** argv) { printf("single digits %d\n",FLT_DIG); printf("double digits %d\n",DBL_DIG); printf("long double digits %d\n",LDBL_DIG); return 1; } The reason I wanted to use float.h for the repr precisions is that at some point long double is bound to be quad precision, I think it already is on some machines and it used to be on vaxen. So I wanted it to carry over naturally when the change came. I note that arrays and scalars print differently and I'm not sure where the array print is implemented. Chuck > > You may want to steal the function format_float() that is defined in > that file, too. > I'll look at it. At the moment we use %.*g Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Apr 10 21:58:29 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 10 Apr 2008 19:58:29 -0600 Subject: [Numpy-discussion] problem with float64's str() In-Reply-To: <3d375d730804101806w5fced280g95365a55b950ab64@mail.gmail.com> References: <7f03db650804031231t39aa3000o719db8359a7769c7@mail.gmail.com> <9457e7c80804040314t1db5e821h56ea782245cd4437@mail.gmail.com> <47F6305C.7060901@gmail.com> <47F63291.3030509@student.matnat.uio.no> <7f03db650804040756q20d0ad79x964b05fbad6fd8fb@mail.gmail.com> <3d375d730804041247q25938d50h5083474a47f51a99@mail.gmail.com> <3d375d730804101738t676872acnf3ef99c954b7e504@mail.gmail.com> <3d375d730804101806w5fced280g95365a55b950ab64@mail.gmail.com> Message-ID: On Thu, Apr 10, 2008 at 7:06 PM, Robert Kern wrote: > On Thu, Apr 10, 2008 at 7:57 PM, Charles R Harris > wrote: > > > > On Thu, Apr 10, 2008 at 6:38 PM, Robert Kern > wrote: > > > > > > On Thu, Apr 10, 2008 at 7:31 PM, Charles R Harris > > > wrote: > > > > > That said, str(float_numpy_scalar) really should have the same > rules > > > > > as str(some_python_float). > > > > > > > > For all different precisions? > > > > > > No. I should have said str(float64_numpy_scalar). I am content to > > > leave the other types alone. > > > > > > > And what should the rules be. > > > > > > All Python does is use a lower decimal precision for __str__ than > > __repr__. > > > > > > > > > > I note that > > > > numpy doesn't distinguish between repr and str, maybe we could > specify > > > > different behavior for the two. > > > > > > Yes, precisely. > > > > Well, I know where to do that and have a ticket for it. What I would > also > > like to do is use float.h for setting the repr precision, but I am not > sure > > I can count on its presence as it only became part of the spec in 1999. > Then > > again, that's almost ten years ago. Anyway, python on my machine > generates > > 12 significant digits. Is that common to everyone? > > Here is the relevant portion of Objects/floatobject.c: > > /* Precisions used by repr() and str(), respectively. > > The repr() precision (17 significant decimal digits) is the minimal > number > that is guaranteed to have enough precision so that if the number is > read > back in the exact same binary value is recreated. This is true for IEEE > floating point by design, and also happens to work for all other modern > hardware. > > The str() precision is chosen so that in most cases, the rounding noise > created by various operations is suppressed, while giving plenty of > precision for practical use. > > */ > > #define PREC_REPR 17 > #define PREC_STR 12 > > OK, I've committed a change that fixes the problem that started this thread, but I'm going to leave the ticket open for a while until I decide what to do about longdouble. The precisions are now #define FLOATPREC_REPR 8 #define FLOATPREC_STR 6 #define DOUBLEPREC_REPR 17 #define DOUBLEPREC_STR 12 #if SIZEOF_LONGDOUBLE == SIZEOF_DOUBLE #define LONGDOUBLEPREC_REPR DOUBLEPREC_REPR #define LONGDOUBLEPREC_STR DOUBLEPREC_STR #else /* More than probably needed on Intel FP */ #define LONGDOUBLEPREC_REPR 20 #define LONGDOUBLEPREC_STR 12 #endif I'm open to suggestions. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Apr 10 22:01:20 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 10 Apr 2008 21:01:20 -0500 Subject: [Numpy-discussion] problem with float64's str() In-Reply-To: References: <7f03db650804031231t39aa3000o719db8359a7769c7@mail.gmail.com> <47F6305C.7060901@gmail.com> <47F63291.3030509@student.matnat.uio.no> <7f03db650804040756q20d0ad79x964b05fbad6fd8fb@mail.gmail.com> <3d375d730804041247q25938d50h5083474a47f51a99@mail.gmail.com> <3d375d730804101738t676872acnf3ef99c954b7e504@mail.gmail.com> <3d375d730804101806w5fced280g95365a55b950ab64@mail.gmail.com> Message-ID: <3d375d730804101901x4eebc4d3r6c0d190d3e2002c5@mail.gmail.com> On Thu, Apr 10, 2008 at 8:58 PM, Charles R Harris wrote: > I'm open to suggestions. I have nothing better to offer than what you've done. Thank you! -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Thu Apr 10 22:23:37 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 10 Apr 2008 20:23:37 -0600 Subject: [Numpy-discussion] problem with float64's str() In-Reply-To: <3d375d730804101901x4eebc4d3r6c0d190d3e2002c5@mail.gmail.com> References: <7f03db650804031231t39aa3000o719db8359a7769c7@mail.gmail.com> <47F63291.3030509@student.matnat.uio.no> <7f03db650804040756q20d0ad79x964b05fbad6fd8fb@mail.gmail.com> <3d375d730804041247q25938d50h5083474a47f51a99@mail.gmail.com> <3d375d730804101738t676872acnf3ef99c954b7e504@mail.gmail.com> <3d375d730804101806w5fced280g95365a55b950ab64@mail.gmail.com> <3d375d730804101901x4eebc4d3r6c0d190d3e2002c5@mail.gmail.com> Message-ID: On Thu, Apr 10, 2008 at 8:01 PM, Robert Kern wrote: > On Thu, Apr 10, 2008 at 8:58 PM, Charles R Harris > wrote: > > I'm open to suggestions. > > I have nothing better to offer than what you've done. Thank you! > OK, but it looks like I need to implement our own conversion to strings functions to correctly display longdouble. PyOS_snprintf is what we are currently using and it only takes doubles. Grrrr. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Thu Apr 10 23:26:28 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 10 Apr 2008 20:26:28 -0700 Subject: [Numpy-discussion] #734: interactive docstring search (lookfor) In-Reply-To: <3d375d730804101644g48b60292r8256f74013709ec2@mail.gmail.com> References: <47FE7FED.2060701@iki.fi> <47FE856D.5050300@enthought.com> <47FEA61D.4040004@noaa.gov> <3d375d730804101644g48b60292r8256f74013709ec2@mail.gmail.com> Message-ID: On Thu, Apr 10, 2008 at 4:44 PM, Robert Kern wrote: > On Thu, Apr 10, 2008 at 6:43 PM, Christopher Barker > wrote: > > Is there anything numpy/scypi specific about it? I > > d like to use this kind of thing with other packages, in which case it > > definitely does not belong in numpy or scipy. > > Basically, it's a configuration issue, I think. The tool needs to know > which packages to search in because it can't search every installed > package (it needs to load them all!). And it's worth keeping in mind that there may be packages out there who can't all happily coexist at the same time in memory, or that have very expensive imports... I'm thinking of having an API for this so the user can register a set of default packages to always have loaded, as well as an easy way to load others at runtime into the available index. That way, users can easily load what they need in any given session (one day it could be Qt and the next WX). Cheers, f From charlesr.harris at gmail.com Thu Apr 10 23:57:42 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 10 Apr 2008 21:57:42 -0600 Subject: [Numpy-discussion] vander() docstring In-Reply-To: <9457e7c80804091007p4009f9adlcc424cf2f8349f5e@mail.gmail.com> References: <200803261922.58425.lists@informa.tiker.net> <200804091128.33116.lists@informa.tiker.net> <9457e7c80804091007p4009f9adlcc424cf2f8349f5e@mail.gmail.com> Message-ID: On Wed, Apr 9, 2008 at 11:07 AM, St?fan van der Walt wrote: > On 09/04/2008, Andreas Kl?ckner wrote: > > On Mittwoch 26 M?rz 2008, Charles R Harris wrote: > > > The docstring is incorrect. The Vandermonde matrix produced is > compatible > > > with numpy polynomials that also go from high to low powers. I would > have > > > done it the other way round, so index matched power, but that isn't > how it > > > is. > > I've never seen the vector powers that way around in literature. > Shouldn't we fix it to correspond to the (vast) majority of > publications? > Turns out it matches the matlab definition. Maybe we just need another function: vandermonde Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Apr 11 00:02:47 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 10 Apr 2008 23:02:47 -0500 Subject: [Numpy-discussion] vander() docstring In-Reply-To: References: <200803261922.58425.lists@informa.tiker.net> <200804091128.33116.lists@informa.tiker.net> <9457e7c80804091007p4009f9adlcc424cf2f8349f5e@mail.gmail.com> Message-ID: <3d375d730804102102h79b8bcf2g4b644dc40e896367@mail.gmail.com> On Thu, Apr 10, 2008 at 10:57 PM, Charles R Harris wrote: > Turns out it matches the matlab definition. Maybe we just need another > function: vandermonde -1 It's needless duplication. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From peridot.faceted at gmail.com Fri Apr 11 02:58:18 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Fri, 11 Apr 2008 02:58:18 -0400 Subject: [Numpy-discussion] float32 is not a float ? In-Reply-To: References: <525f23e80804100537k1298f89eg29368b691ec7126c@mail.gmail.com> <504669.8295.qm@web34401.mail.mud.yahoo.com> <9457e7c80804101058m6890c59m2b1362165a55bed8@mail.gmail.com> <47FE5E8F.1040400@llnl.gov> Message-ID: On 10/04/2008, Charles R Harris wrote: > I think you want the isreal function, but it will also return true for > complex with 0 imaginary part. Hmm... the various iswhatever functions seem > to be lacking in coverage. Maybe we should fix that. icomplexobj is designed to solve that problem. Anne From wilson.t.thompson at gmail.com Fri Apr 11 03:52:52 2008 From: wilson.t.thompson at gmail.com (wilson) Date: Fri, 11 Apr 2008 00:52:52 -0700 (PDT) Subject: [Numpy-discussion] meaning of accumulation/normalisation In-Reply-To: References: <8fe56208-917c-4b6d-83e1-74ce7117081f@u36g2000prf.googlegroups.com> Message-ID: <9974a4f3-7188-4dbb-9f17-cecd24b771ef@w8g2000prd.googlegroups.com> > > newul=ul/norm >>the java doc mentions that by the above steps ul is normalised to unit length (vector length) > Umm, not quite, it is missing a square root. You can get the same result by > using the Frobenius norm thanks Chuck.. i found the norm as you advised and then found newul=ul/norm ,is there some test condition the newul should satisfy? the java code hints at // u_l normalised to unit length (vector length) // i.e. u_l (dot) u_l = 1 # i think he means the accumOperation() i tried accumOperation(newul) and it returns 1 can someone pls explain the maths in this thanks W From millman at berkeley.edu Fri Apr 11 04:04:05 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Fri, 11 Apr 2008 01:04:05 -0700 Subject: [Numpy-discussion] Making NumPy accessible to everyone (or no-one) (was Numpy-discussion Digest, Vol 19, Issue 44) In-Reply-To: <504669.8295.qm@web34401.mail.mud.yahoo.com> References: <525f23e80804100537k1298f89eg29368b691ec7126c@mail.gmail.com> <504669.8295.qm@web34401.mail.mud.yahoo.com> Message-ID: On Thu, Apr 10, 2008 at 10:17 AM, Lou Pecora wrote: > Yes, I use np= number of points, too. But you all > might want to use something else. That's the point of > the flexibility of import ... as I would recommend against using np as a variable name. Variable names should be short and informative. I would much rather see something like "num_points". > Trying to lock in namespaces as np or N or whatever is > a BAD idea. Allow the flexibility. You can admonish > against from ... import * for newbies and then tell > them to use from ... import actual function names (as > mentioned above). But locking people into a standard, > even an informal one is, as someone else said, acting > a bit too much like accountants. Stop, please! Coding standards and conventions in a large, collaborative codebase are essential. If you want to contribute to NumPy or SciPy you will have to conform to these conventions. In your own private code or private project do whatever you want. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From cburns at berkeley.edu Fri Apr 11 04:57:31 2008 From: cburns at berkeley.edu (Christopher Burns) Date: Fri, 11 Apr 2008 01:57:31 -0700 Subject: [Numpy-discussion] Making NumPy accessible to everyone (or no-one) (was Numpy-discussion Digest, Vol 19, Issue 44) In-Reply-To: <9457e7c80804100355q48cb107as9d5ccc3f3761ef0f@mail.gmail.com> References: <9457e7c80804100355q48cb107as9d5ccc3f3761ef0f@mail.gmail.com> Message-ID: <764e38540804110157x6daa9999ufe8d2b95f2979f0f@mail.gmail.com> On Thu, Apr 10, 2008 at 3:55 AM, St?fan van der Walt wrote: > Hi Joe, all > > On 10/04/2008, Joe Harrington wrote: > > > Absolutely. Let's please standardize on: > > > import numpy as np > > > import scipy as sp > > > > I hope we do NOT standardize on these abbreviations. While a few may > > have discussed it at a sprint, it hasn't seen broad discussion Valid point... Travis did a wonderful job of summarizing that sprint and posting to the list. However, the N vs. np discussion was missed. http://projects.scipy.org/pipermail/scipy-dev/2007-December/008078.html > Namespaces throttle the amount of information with which the user is > presented, and well thought through design leads to logical, intuitive > segmentation of functionality. > > > Namespaces add characters to code that have a high redundancy factor. > > This means they pollute code, make it slow and inaccurate to read, and > > making learning harder. Lines get longer and may wrap if they contain > > several calls. It is harder while visually scanning code to > > distinguish the function name if it's adjacent to a bunch of other > > text, particularly if that text appears commonly in the nearby code. I think namespaces are one of the crown-jewels that make python more attractive to scientists (not programmers) over Matlab. Even if they don't realize it yet. :) I think a lot of researchers would spend less time debugging their code if they were using python with namespaces instead of adding this: addpath(genpath('mydirectory')) in all of their Matlab code! Or other path manipulation. We certainly need a better discover mechanism for users to find functions however. -- Christopher Burns Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From pearu at cens.ioc.ee Fri Apr 11 07:07:52 2008 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Fri, 11 Apr 2008 13:07:52 +0200 Subject: [Numpy-discussion] F2PY future Message-ID: <47FF4688.8090005@cens.ioc.ee> Hi, I am in a process of writing a scientific paper about F2PY that will provide an automatic solution to the Python and Fortran connection problem. While writing it, I also need to decide what will be the future of F2PY. In particulary, I have the following main questions to which I am looking for suggestions: 1) where the future users of F2PY should find it, 2) how the users can get support (documentation, mailing lists, etc). 3) where to continue the development of F2PY. Currently, F2PY has three "home pages": 1) http://cens.ioc.ee/projects/f2py2e/ - this has old f2py. The old f2py is unique in that it covers Numeric and numarray support, but is not being developed anymore. 2) http://www.scipy.org/F2py - this covers the current f2py included in NumPy. f2py in numpy is rather stable and is being maintained. There is no plans to add new functionalities (like F90 derived type support) to the numpy f2py. 3) http://projects.scipy.org/scipy/numpy/wiki/G3F2PY - this is a wiki page for the third generation of f2py. It aims at adding full Fortran 90/../2003 support to the f2py tool, including F90 derived types as well as POINTER arguments. It should replace numpy f2py in future. Obviosly, the three "home pages" for f2py is too much, even when they cover three different code sets. So, now I am looking for to unify these places to one site that will cover all three code sets with software, documentation, and support. Currently I can think of the following options: Use Google Code. Pros: it provides necessary infrastructure to develop software projects and I am used to it. Cons: in my experience Google Code has been too many times broken (at least three times in half a year), though this may improve in future. Also, Google Code provides only SVN, no hg. Since f2py will be an important tool for numpy/scipy users, it would be natural to continue developing f2py under these projects. However, there are rumours of cleaning up scipy/numpy from extension generation tools and so in long term, f2py may need to look for another home. So, I wonder if a hosting could be provided for f2py? Say, in a form of f2py.scipy.org or www.f2py.org? I am rather ignorant about these matters, so any help will be appreciated. Thanks, Pearu From wbaxter at gmail.com Fri Apr 11 08:44:55 2008 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 11 Apr 2008 21:44:55 +0900 Subject: [Numpy-discussion] F2PY future In-Reply-To: <47FF4688.8090005@cens.ioc.ee> References: <47FF4688.8090005@cens.ioc.ee> Message-ID: I'm afraid I'm not much help answering your questions. But one thing I've wondered about f2py is if it could be generalized into an f2*** tool. How intertwined is the analysis of the fortran with the synthesis of the python? There are lots of languages that could benefit from a fortran wrapper generator tool. --bb On Fri, Apr 11, 2008 at 8:07 PM, Pearu Peterson wrote: > Hi, > > I am in a process of writing a scientific paper about F2PY that will > provide an automatic solution to the Python and Fortran connection > problem. While writing it, I also need to decide what will be the future > of F2PY. In particulary, I have the following main questions to which I > am looking for suggestions: > 1) where the future users of F2PY should find it, > 2) how the users can get support (documentation, mailing lists, etc). > 3) where to continue the development of F2PY. > > Currently, F2PY has three "home pages": > 1) http://cens.ioc.ee/projects/f2py2e/ - this has old f2py. The old f2py > is unique in that it covers Numeric and numarray support, but is > not being developed anymore. > 2) http://www.scipy.org/F2py - this covers the current f2py included > in NumPy. f2py in numpy is rather stable and is being maintained. There > is no plans to add new functionalities (like F90 derived type support) > to the numpy f2py. > 3) http://projects.scipy.org/scipy/numpy/wiki/G3F2PY - this is a > wiki page for the third generation of f2py. It aims at adding full > Fortran 90/../2003 support to the f2py tool, including F90 derived types > as well as POINTER arguments. It should replace numpy f2py in future. > > Obviosly, the three "home pages" for f2py is too much, even when they > cover three different code sets. So, now I am looking for to unify these > places to one site that will cover all three code sets with software, > documentation, and support. > > Currently I can think of the following options: > > Use Google Code. Pros: it provides necessary infrastructure to develop > software projects and I am used to it. Cons: in my experience Google > Code has been too many times broken (at least three times in half a > year), though this may improve in future. Also, Google Code provides > only SVN, no hg. > > Since f2py will be an important tool for numpy/scipy users, > it would be natural to continue developing f2py under these projects. > However, there are rumours of cleaning up scipy/numpy from extension > generation tools and so in long term, f2py may need to look for another > home. So, I wonder if a hosting could be provided for f2py? Say, in a > form of f2py.scipy.org or www.f2py.org? I am rather ignorant about these > matters, so any help will be appreciated. > > Thanks, > Pearu > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From pearu at cens.ioc.ee Fri Apr 11 08:54:37 2008 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Fri, 11 Apr 2008 14:54:37 +0200 Subject: [Numpy-discussion] F2PY future In-Reply-To: References: <47FF4688.8090005@cens.ioc.ee> Message-ID: <47FF5F8D.6000605@cens.ioc.ee> Bill Baxter wrote: > I'm afraid I'm not much help answering your questions. But one thing > I've wondered about f2py is if it could be generalized into an f2*** > tool. How intertwined is the analysis of the fortran with the > synthesis of the python? There are lots of languages that could > benefit from a fortran wrapper generator tool. This is a very good question. In fact, the g3 f2py contains a full parser of Fortran 77/../2003 codes that is independent of how the parsed tree will be used. It could be used as a starting point to many tools like Fortran to C/C++ converters, as well as generating wrappers for other scripting languages. So, I think this is something worth of keeping in mind indeed when developing g3 f2py. Although, I would need more support to take this task of generalizing f2py to f2**. Regards, Pearu From lists_ravi at lavabit.com Fri Apr 11 09:08:18 2008 From: lists_ravi at lavabit.com (Ravi) Date: Fri, 11 Apr 2008 09:08:18 -0400 Subject: [Numpy-discussion] Making NumPy accessible to everyone (or no-one) In-Reply-To: <764e38540804110157x6daa9999ufe8d2b95f2979f0f@mail.gmail.com> References: <9457e7c80804100355q48cb107as9d5ccc3f3761ef0f@mail.gmail.com> <764e38540804110157x6daa9999ufe8d2b95f2979f0f@mail.gmail.com> Message-ID: <200804110908.18853.lists_ravi@lavabit.com> On Friday 11 April 2008 04:57:31 am Christopher Burns wrote: > I think namespaces are one of the crown-jewels that make python more > attractive to scientists (not programmers) over Matlab. ?Even if they don't > realize it yet. ?:) As a humble user who has neither the python-fu nor extension-fu to contribute to the numpy codebase, I second this. I have spent hours tracking down problems related to names coming from pylab vs from scipy. Now, my rule is very simple: all names in any source file come from one of 3 cases: 1. fully qualified (scipy.linalg.norm) 2. locally defined in the file 3. explicitly imported using "from xxx import ..." The only time I break this rule is when playing round in the ipython console. Rule 1 above is almost never used. With explicit name imports (practically never above 25 names), external dependencies are very clear and it is easy to figure out the extent of any effort to convert that module to C++ (usually for speed). An added bonus is an ability to automatically reload a dependent module when the code for it changes simply using a trivial script that scans dependencies in "from xxx import ..." lines -- very useful for Matlab-style code development when you have a system whose innards are under modification. Of course, this is only the view of a _user_ coming from Matlab-land. The combination of Python and C++ (especially boost.python) beats every other system that I have seen for engineering (telecommunications and statistical signal processing). Namespaces, forced indentation, and object orientation are the main reasons for me to use numpy/scipy/python. Perhaps professional programmers and software engineers know better, but the features listed above are a big part of the reason I crossed over from Matlab+C to Python+C++. Regards, Ravi From kbasye1 at jhu.edu Fri Apr 11 09:35:26 2008 From: kbasye1 at jhu.edu (Ken Basye) Date: Fri, 11 Apr 2008 09:35:26 -0400 Subject: [Numpy-discussion] Array printing and another question In-Reply-To: <200804091026.57767.meine@informatik.uni-hamburg.de> References: <47FB7BD7.1020506@jhu.edu> <47FB8DB9.3030700@jhu.edu> <200804091026.57767.meine@informatik.uni-hamburg.de> Message-ID: <47FF691E.40003@jhu.edu> Hi, Thanks for the reply. The formatting code, such as it is, is below. It uses Martin Jansche's double.py (http://symptotic.com/mj/double/double.py) and then does some simple bit twiddling. I'm still hoping someone can help me find a way to use this format for arrays of float64s. Ken ================================ import double as _double def float_to_readable_string(f): ftype = _double.fpclassify(f) assert ftype == 'NORMAL' or ftype == 'ZERO' bits = _double.doubleToRawLongBits(f) exponent = (bits >> 52) & 0x7ff mantissa = bits & 0x000fFfffFfffFfffL sign = (bits >> 63) exponent -= 1023 pm = '+' if sign == 0 else '-' s = '%s(%+05d)0x%013x' % (pm, exponent, mantissa) return s def readable_string_to_float(s): assert len(s) == 23 pm = s[0] assert pm == '+' or pm == '-' sign = 1 if pm == '-' else 0 assert s[1] == '(' and s[7] == ')' e = s[2:7] exponent = int(e, 10) + 1023 assert 0 <= exponent < 0x7ff m = s[8:] assert m[:2] == '0x' mantissa = int(m, 16) bits = (sign << 63) + (exponent << 52) + mantissa return _double.longBitsToDouble(bits) Hans Meine wrote: > Am Dienstag, 08. April 2008 17:22:33 schrieb Ken Basye: > >> I've had this happen >> often enough that I found the first thing I did when an output >> difference arose was to print the FP in hex to see if the >> difference was "real" or just a formatting artifact. >> > > Nice idea - is that code available somewhere / could you post it? > > As a side note, there is a second big source of bug-like experiences with the > x86 architecture: floating point values are represented with a higher > precision inside the FPU (e.g. 80 bits instead of 64), but that probably only > matters for compiled programs where the compiler may (or may not) optimize > intermediate, truncating FPU->memory operations away (which leads to > differing results and is one of the most often reported "bugs" of the GCC > optimizer). > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at informa.tiker.net Fri Apr 11 10:20:47 2008 From: lists at informa.tiker.net (Andreas =?iso-8859-1?q?Kl=F6ckner?=) Date: Fri, 11 Apr 2008 10:20:47 -0400 Subject: [Numpy-discussion] vander() docstring In-Reply-To: <3d375d730804102102h79b8bcf2g4b644dc40e896367@mail.gmail.com> References: <200803261922.58425.lists@informa.tiker.net> <3d375d730804102102h79b8bcf2g4b644dc40e896367@mail.gmail.com> Message-ID: <200804111020.49365.lists@informa.tiker.net> On Freitag 11 April 2008, Robert Kern wrote: > On Thu, Apr 10, 2008 at 10:57 PM, Charles R Harris > > wrote: > > Turns out it matches the matlab definition. Maybe we just need another > > function: vandermonde > > -1 It's needless duplication. Agree. Let's just live with Matlab's definition. Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From rshepard at appl-ecosys.com Fri Apr 11 12:05:06 2008 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Fri, 11 Apr 2008 09:05:06 -0700 (PDT) Subject: [Numpy-discussion] Normal Distribution With NumPy? Message-ID: I see in the NumPy Book that there are functions to allow generation of beta, binomial, and poisson curves, but I don't see one for normal curves. Is there such a function? Currently I'm using code (I forget from where) that creates a Gaussian distribution, but the tails do not reach zero (within the range of the x axis) unless the inflection point is less than 0.5 on the y axis. I'm using PyX to plot the resulting curves and would like to avoid re-inventing the wheel. If there's python code to calculate a normal distribution given the center, width, and inflection point I would appreciate a pointer to the source. Rich -- Richard B. Shepard, Ph.D. | Integrity Credibility Applied Ecosystem Services, Inc. | Innovation Voice: 503-667-4517 Fax: 503-667-8863 From kwgoodman at gmail.com Fri Apr 11 12:12:29 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 11 Apr 2008 09:12:29 -0700 Subject: [Numpy-discussion] Normal Distribution With NumPy? In-Reply-To: References: Message-ID: On Fri, Apr 11, 2008 at 9:05 AM, Rich Shepard wrote: > I see in the NumPy Book that there are functions to allow generation of > beta, binomial, and poisson curves, but I don't see one for normal curves. > Is there such a function? > > Currently I'm using code (I forget from where) that creates a Gaussian > distribution, but the tails do not reach zero (within the range of the x > axis) unless the inflection point is less than 0.5 on the y axis. > > I'm using PyX to plot the resulting curves and would like to avoid > re-inventing the wheel. If there's python code to calculate a normal > distribution given the center, width, and inflection point I would > appreciate a pointer to the source. Here's the formula: http://en.wikipedia.org/wiki/Normal_distribution If you want the tails to be zero you could do y - y[0]. From rshepard at appl-ecosys.com Fri Apr 11 12:40:51 2008 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Fri, 11 Apr 2008 09:40:51 -0700 (PDT) Subject: [Numpy-discussion] Normal Distribution With NumPy? In-Reply-To: References: Message-ID: On Fri, 11 Apr 2008, Keith Goodman wrote: > Here's the formula: Thanks, Keith. While I had read that page before I had not followed the link to the FWMH page, and that provided the insight I needed. It's all working now. Rich -- Richard B. Shepard, Ph.D. | Integrity Credibility Applied Ecosystem Services, Inc. | Innovation Voice: 503-667-4517 Fax: 503-667-8863 From strawman at astraw.com Fri Apr 11 12:48:21 2008 From: strawman at astraw.com (Andrew Straw) Date: Fri, 11 Apr 2008 09:48:21 -0700 Subject: [Numpy-discussion] F2PY future In-Reply-To: <47FF4688.8090005@cens.ioc.ee> References: <47FF4688.8090005@cens.ioc.ee> Message-ID: <47FF9655.1090407@astraw.com> Pearu Peterson wrote: > Use Google Code. Pros: it provides necessary infrastructure to develop > software projects and I am used to it. Cons: in my experience Google > Code has been too many times broken (at least three times in half a > year), though this may improve in future. Also, Google Code provides > only SVN, no hg. > Another option: the IPython people have been using launchpad.net ( https://launchpad.net/ipython ) -- it supports bzr. I'm not sure how happy they are with it, but I think happy enough to stick with it rather than attempt to get a server with hg set up. IIRC, they did initially marginally prefer hg over bzr but the immediate availability of bzr hosting swung them over. -Andrew From stefan at sun.ac.za Fri Apr 11 13:01:35 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 11 Apr 2008 19:01:35 +0200 Subject: [Numpy-discussion] vander() docstring In-Reply-To: <200804111020.49365.lists@informa.tiker.net> References: <200803261922.58425.lists@informa.tiker.net> <3d375d730804102102h79b8bcf2g4b644dc40e896367@mail.gmail.com> <200804111020.49365.lists@informa.tiker.net> Message-ID: <9457e7c80804111001k6ab64ff2v9cc9344a6d2cf4e5@mail.gmail.com> On 11/04/2008, Andreas Kl?ckner wrote: > On Freitag 11 April 2008, Robert Kern wrote: > > On Thu, Apr 10, 2008 at 10:57 PM, Charles R Harris > > > > wrote: > > > Turns out it matches the matlab definition. Maybe we just need another > > > function: vandermonde > > > > -1 It's needless duplication. > > > Agree. Let's just live with Matlab's definition. I thought that's why I wasn't working on Octave any more! :) Have a good weekend, St?fan From guillaume.desjardins at gmail.com Fri Apr 11 13:06:48 2008 From: guillaume.desjardins at gmail.com (Guillaume Desjardins) Date: Fri, 11 Apr 2008 13:06:48 -0400 Subject: [Numpy-discussion] OWNDATA flag and reshape() -- views vs. copies Message-ID: I'm pretty new to Python and numpy (longtime c / matlab programmer), but after a read through some of the past threads and Travis' "Guide to Numpy", I think I have a fairly good understanding of how the reshape() function / methods work, with regards to views and copies. For what its worth (and to make things clearer), my understanding is that: * Function reshape: create a view, or if it can't (i.e. resulting array cannot be expressed with constant stride parameters), a copy * Method obj.reshape(): creates a view, or throws an exception if it can't Now assuming that is the case, I do not understand the behaviour of the following code. a = array([[1,2,3,4],[5,6,7,8],[9,10,11,12]]); a.flags C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False b = a[1:,1:3] # creates view of a >> array([[ 6, 7], [10, 11]]) b.flags C_CONTIGUOUS : False F_CONTIGUOUS : False OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False c = reshape(b,(4,1)) # RESHAPE FUNCTION: should copy, as view of this is impossible (? isn't it ?) c >> array([[ 6], [ 7], [10], [11]]) c.flags C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False First question: if c is a copy of b, then why isn't the OWNDATA set to true ? To prove that it is indeed a copy, >> c[0] = 600 modifies the values of c but not of b and a (while modifying b - which is clearly a view - also modifies a) d = b.reshape((4,1)) # RESHAPE METHOD: same thing but with method... shouldn't this give an error ? d.flags C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False Second question: shouldn't b.reshape((4,1)) raise a ValueError since it can't (as far as I see it) create a view ? Once again, modifying d does not affect b or a (hence it is a copy). Thanks for clearing up this issue. It is imperative that I know exactly when numpy creates views vs. copies (and that I understand all the intricacies), as I'm planning on doing A LOT of reshape() with inplace modifications. Hope I'm not polluting the mailing list with a newbie question. Thanks in advance, -- Guillaume ps: sorry to the moderators for having submitted this twice (once as a non-member, and after signing up) From oliphant at enthought.com Fri Apr 11 13:23:34 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 11 Apr 2008 12:23:34 -0500 Subject: [Numpy-discussion] SciPy to be down intermittently today Message-ID: <47FF9E96.4030202@enthought.com> Hey everyone, The scipy.org site will be down intermittently today. We are trying to upgrade its memory to improve performance. Thank you, -Travis O. From ellisonbg.net at gmail.com Fri Apr 11 13:28:19 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Fri, 11 Apr 2008 11:28:19 -0600 Subject: [Numpy-discussion] [SciPy-dev] F2PY future In-Reply-To: <47FF9655.1090407@astraw.com> References: <47FF4688.8090005@cens.ioc.ee> <47FF9655.1090407@astraw.com> Message-ID: <6ce0ac130804111028p64b56a0fyd49d3cb036641eb0@mail.gmail.com> > Another option: the IPython people have been using launchpad.net ( > https://launchpad.net/ipython ) -- it supports bzr. I'm not sure how > happy they are with it, but I think happy enough to stick with it rather > than attempt to get a server with hg set up. IIRC, they did initially > marginally prefer hg over bzr but the immediate availability of bzr > hosting swung them over. I can't speak for the rest of the ipython dev team, but overall, we have really liked the bzr+launchpad combo. We haven't made a final decision to stick with it, but I imagine we will (everyone has been very happy so far with it). Bottom line: we have been better able to collaborate and the admin side of things has been essentially 0. We all like hg as well, but having the supernice hosting on launchpad really makes a difference. Brian > -Andrew > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > From fperez.net at gmail.com Fri Apr 11 13:34:57 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 11 Apr 2008 10:34:57 -0700 Subject: [Numpy-discussion] F2PY future In-Reply-To: <47FF9655.1090407@astraw.com> References: <47FF4688.8090005@cens.ioc.ee> <47FF9655.1090407@astraw.com> Message-ID: On Fri, Apr 11, 2008 at 9:48 AM, Andrew Straw wrote: > Pearu Peterson wrote: > > Use Google Code. Pros: it provides necessary infrastructure to develop > > software projects and I am used to it. Cons: in my experience Google > > Code has been too many times broken (at least three times in half a > > year), though this may improve in future. Also, Google Code provides > > only SVN, no hg. > > > Another option: the IPython people have been using launchpad.net ( > https://launchpad.net/ipython ) -- it supports bzr. I'm not sure how > happy they are with it, but I think happy enough to stick with it rather > than attempt to get a server with hg set up. IIRC, they did initially > marginally prefer hg over bzr but the immediate availability of bzr > hosting swung them over. Pretty happy, for the most part. The launchpad site is a bit messy to navigate, some things aren't totally obvious (which should be) and we've all made a few mistakes with the transition to a bzr workflow, but we'll likely stick to it. Over the next couple of weeks, in between grant deadlines, I hope to push the process of moving 'for real' our code hosting over to lp, which will require doing a real import of the SVN history and something with our Trac bugs. It's important to note that we are NOT abandoning ipython.scipy.org: we'll keep the Moin site, mailing lists, etc as it is. But unless something weird comes up in a final "let's do this" discussion with ipython-dev, I think the decision to make a permanent transition is more or less done. I just don't see anything better out there that exists *right now*. >From a technical perspective I think the hg/bzr discussion is nearly a toss-up, and I'm not interested in spending weeks on the fine points that may lie there. Bzr does the job, more than well enough for us. The real kicker was the vast amount of infrastructure provided by launchpad, as you point out above. While it's not perfect, it does a lot of very good things, and it's widely used by many more projects, thus likely to continue improving. As a note, we've also just switched neuroimaging.scipy.org to launchpad (again, just the code hosting part): https://launchpad.net/nipy If others here are interested, I can give more details on the benefits we've already reaped from the launchpad transition on both ipython and nipy. Regards, f From gael.varoquaux at normalesup.org Fri Apr 11 14:01:09 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 11 Apr 2008 20:01:09 +0200 Subject: [Numpy-discussion] F2PY future In-Reply-To: <47FF9655.1090407@astraw.com> References: <47FF4688.8090005@cens.ioc.ee> <47FF9655.1090407@astraw.com> Message-ID: <20080411180109.GD11288@phare.normalesup.org> On Fri, Apr 11, 2008 at 09:48:21AM -0700, Andrew Straw wrote: > Another option: the IPython people have been using launchpad.net ( > https://launchpad.net/ipython ) -- it supports bzr. I'm not sure how > happy they are with it, but I think happy enough to stick with it rather > than attempt to get a server with hg set up. IIRC, they did initially > marginally prefer hg over bzr but the immediate availability of bzr > hosting swung them over. To add my view to the comments of Fernando and Brian, I am _very_ happy with the workflow, and a bit disappointed by the usability of launchpad. Hopefully it will get better. All in all I don't regret the shift at all. However, I am a bit worried we might loose developpers, and I am not encouraging big projects like numpy or scipy to do the shift yet. My two cents, Ga?l From efiring at hawaii.edu Sat Apr 12 19:26:41 2008 From: efiring at hawaii.edu (Eric Firing) Date: Sat, 12 Apr 2008 13:26:41 -1000 Subject: [Numpy-discussion] standard pyplot, numpy, scipy import syntax Message-ID: <48014531.60006@hawaii.edu> In reference to: http://news.gmane.org/find-root.php?message_id=%3cc7009a550804100055g6388b20ej520e85d8e679a55%40mail.gmail.com%3e A point was brought up that deserves wider dissemination and a correction, hence my posting to two lists. To reduce confusion among new users and to improve code readability, participants in a numpy sprint, including John D. Hunter (remotely), agreed to promote the following standard: import numpy as np import scipy as sp import matplotlib.pyplot as plt This differs from the message referenced above in that the standard entrance point for the plotting functionality of pylab is the pyplot module of matplotlib, not the pylab module. Pyplot is a fairly recent addition that provides the state-machine plotting interface; pylab is essentially the result of dumping pyplot and numpy into a single namespace, which is sometimes convenient for interactive use but certainly is not encouraged for programming. The recommended standard is intended to promote consistency and readability in the numpy, scipy, and matplotlib families of modules, and in their documentation; it is not intended to imply any restriction on the user's freedom. The recommendation may be helpful in your own code, or it may not; use it where it helps. This message is intended to be informational; it is not a trial balloon, and it is not intended to stimulate discussion. I hope the related discussions that have already occurred on the numpy and matplotlib lists will suffice. Eric From pav at iki.fi Sat Apr 12 21:01:23 2008 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 13 Apr 2008 04:01:23 +0300 Subject: [Numpy-discussion] ticket #551 Message-ID: <48015B63.1080308@iki.fi> http://projects.scipy.org/scipy/numpy/ticket/551 This ticket (milestone 1.0.5 critical) seems to occur because cblas_DGEMV in SSE2-enabled Atlas, at least the one shipped with Debian/Ubuntu, apparently requires that doubles are aligned on 8-byte boundaries in memory. If not, a segmentation fault ensues. When unpickling arrays, numpy uses data directly from a Python string, which is consistently not aligned on a 8-byte boundary => crash. [I'm quite sure this is the reason. There is a minimal testcase in comment 22 in the ticket is someone wants to confirm this.] *** So should numpy try to work around this by reallocating the memory when unpickling, and forcibly aligning it as required by SSE2-Atlas? Or, should dotblas_matrixproduct in _dotblas.c check check the alignment? Or, should the ticket be closed as WONTFIX, and the bug forwarded to upstream + Debian? In principle there's the possibility that this is a compiler bug. -- Pauli Virtanen From charlesr.harris at gmail.com Sat Apr 12 21:19:03 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 12 Apr 2008 19:19:03 -0600 Subject: [Numpy-discussion] ticket #551 In-Reply-To: <48015B63.1080308@iki.fi> References: <48015B63.1080308@iki.fi> Message-ID: On Sat, Apr 12, 2008 at 7:01 PM, Pauli Virtanen wrote: > http://projects.scipy.org/scipy/numpy/ticket/551 > > This ticket (milestone 1.0.5 critical) seems to occur because > cblas_DGEMV in SSE2-enabled Atlas, at least the one shipped with > Debian/Ubuntu, apparently requires that doubles are aligned on 8-byte > boundaries in memory. If not, a segmentation fault ensues. When > unpickling arrays, numpy uses data directly from a Python string, which > is consistently not aligned on a 8-byte boundary => crash. > > [I'm quite sure this is the reason. There is a minimal testcase in > comment 22 in the ticket is someone wants to confirm this.] > > *** > > So should numpy try to work around this by reallocating the memory when > unpickling, and forcibly aligning it as required by SSE2-Atlas? > My vote is for checking the alignment and reallocating if needed and sending a ticket upstream. Pickling is probably not the preferred way to store large arrays so a workaround is acceptable. I wonder if Python itself shouldn't be allocating strings on 8 byte boundaries? Or maybe it is the unpickling routine that could use fixing, maybe its reading into a large string and using an offset not divisible by 8. Have there been any problems with Atlas in other distributions? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Apr 13 02:12:15 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 13 Apr 2008 00:12:15 -0600 Subject: [Numpy-discussion] Setting scalar values from strings, what to do? Message-ID: Reposting the comment I left on ticket #736 because I think we need to decide how conversion of strings to scalars should work. There is more to this problem: > > > {{{ > In [5]: int8('1') > ------------------------------ > --------------------------------------------- > ValueError Traceback (most recent call > last) > > /home/charris/ in () > > ValueError: setting an array element with a sequence. > > In [6]: int16('1') > > --------------------------------------------------------------------------- > ValueError Traceback (most recent call > last) > > /home/charris/ in () > > ValueError: setting an array element with a sequence. > > In [7]: int32('1') > Out[7]: 1 > > In [8]: int64('1') > Out[8]: array([1], dtype=int64) > > In [9]: uint8('1') > > --------------------------------------------------------------------------- > ValueError Traceback (most recent call > last) > > /home/charris/ in () > > ValueError: setting an array element with a sequence. > > In [10]: uint16('1') > > --------------------------------------------------------------------------- > ValueError Traceback (most recent call > last) > > /home/charris/ in () > > ValueError: setting an array element with a sequence. > > In [11]: uint32('1') > Out[11]: array([1], dtype=uint32) > > In [12]: uint64('1') > Out[12]: array([1], dtype=uint64) > > > }}} > > > The ones that return arrays go though numpy routines that call > PyNumber_Long on the string. I suspect int32 is a subtype of the python > int so that it returns a number. We also have > > > {{{ > In [10]: array('11',dtype=uint64) > Out[10]: array([1, 1], dtype=uint64) > > In [11]: array('11',dtype=int8) > > --------------------------------------------------------------------------- > ValueError Traceback (most recent call > last) > > /home/charris/ in () > > ValueError: setting an array element with a sequence. > > In [12]: array('11').astype(int8) > Out[12]: array(11, dtype=int8) > }}} > > > > In other words, we have an inconsistent mess on our hands. I think we > need > to decide what the behavior for all these functions should be, along with > floats and complex, when presented with strings. And what to do when the > strings are out of range. These problems arise in the setitem functions > in > arraytypes.inc.src and, I suspect, in the array function itself which > treats the astype method and dtype keyword differently when strings are > passed in. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilson.t.thompson at gmail.com Sun Apr 13 04:30:22 2008 From: wilson.t.thompson at gmail.com (wilson) Date: Sun, 13 Apr 2008 01:30:22 -0700 (PDT) Subject: [Numpy-discussion] meaning of accumulation/normalisation In-Reply-To: References: <8fe56208-917c-4b6d-83e1-74ce7117081f@u36g2000prf.googlegroups.com> Message-ID: <9d08eb28-a2d8-4e4d-9312-e8495d54dbec@s39g2000prd.googlegroups.com> the Frobenius norm, > Which is matlab speak for sqrt(trace(dot(X.T, X))) thanks for that one Chuck.. 1. i have seen similar normailzations in most image processing code..what exactly is the purpose of such normalization?before making images these values will have to be reprocessed to get the pixel values.being a beginner this question has been bugging me for a while.if anyone can explain it wd be great help 2.in this case,the java doc mentions that by the above steps ul is normalised to unit length (vector length) ,and that accumOperation(newul) should return 1, i tested it norm=sqrt(trace(dot(ul.T, ul))) newul=ul/norm testnorm=sqrt(trace(dot(newul.T, newul))) ====>1 can someone make clear the maths in it thanks W From millman at berkeley.edu Sun Apr 13 06:00:19 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Sun, 13 Apr 2008 03:00:19 -0700 Subject: [Numpy-discussion] Fwd: [matplotlib-devel] standard abbreviation for pyplot or pylab? In-Reply-To: References: <48010B4B.9070205@hawaii.edu> <88e473830804121524q5665af2bn4ddb307d2c1e7ec5@mail.gmail.com> Message-ID: I just realized that this discussion was only on the matplotlib mailing list. Basically, they were clarifying that the import statement for pylab should be: import matplotlib.pyplot as plt Sorry for the confusion, ---------- Forwarded message ---------- From: Jarrod Millman Date: Sun, Apr 13, 2008 at 2:58 AM Subject: Re: [matplotlib-devel] standard abbreviation for pyplot or pylab? To: John Hunter Cc: Eric Firing , matplotlib development list On Sat, Apr 12, 2008 at 3:24 PM, John Hunter wrote: > On Sat, Apr 12, 2008 at 2:19 PM, Eric Firing wrote: > > and then this was added, as something else that had been agreed at the > > same sprint: > > import pylab as plt > > I think this is a mistake, and it should have been > > > > import matplotlib.pyplot as plt > > This is what we agreed to (import matplotlib.pyplot as plt) during the > numpy sprint (I was on google chat remotely but participated in the > discussion). I agree that pylab as plt would just add to the > confusion. We agreed on promoting (and I currently use) > > > import numpy as np > import scipy as sp > > import matplotlib.pyplot as plt Sorry, that was my mistake. Thanks for clearing it up. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From wilson.t.thompson at gmail.com Sun Apr 13 09:40:17 2008 From: wilson.t.thompson at gmail.com (wilson) Date: Sun, 13 Apr 2008 06:40:17 -0700 (PDT) Subject: [Numpy-discussion] diagonalising a 2d array Message-ID: hi what exactly does diagonalising a matrix mean? how do you do it on a symmetric numpy array? W From pav at iki.fi Sun Apr 13 09:56:06 2008 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 13 Apr 2008 16:56:06 +0300 Subject: [Numpy-discussion] diagonalising a 2d array In-Reply-To: References: Message-ID: <480210F6.6070703@iki.fi> wilson kirjoitti: > hi > what exactly does diagonalising a matrix mean? http://en.wikipedia.org/wiki/Matrix_diagonalization > how do you do it on a symmetric numpy array? Compute the eigenvalues and eigenvectors using numpy.linalg.eigh. -- Pauli Virtanen From david at ar.media.kyoto-u.ac.jp Sun Apr 13 20:58:36 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 14 Apr 2008 09:58:36 +0900 Subject: [Numpy-discussion] F2PY future In-Reply-To: <20080411180109.GD11288@phare.normalesup.org> References: <47FF4688.8090005@cens.ioc.ee> <47FF9655.1090407@astraw.com> <20080411180109.GD11288@phare.normalesup.org> Message-ID: <4802AC3C.9010206@ar.media.kyoto-u.ac.jp> Gael Varoquaux wrote: > > To add my view to the comments of Fernando and Brian, I am _very_ happy > with the workflow, and a bit disappointed by the usability of launchpad. > Hopefully it will get better. Well, it got better. At first, it was really unusable, I could not even find how to fill bugs :) Still far from being really good at this point, unfortunately, but launchpad being official one of the cashcow of cannonical, I guess it will be improved. On the bzr side of things, one of the bzr developer (Aaron Bentley) was hired by Canonical to work officialy on better integration between launchpad and bzr a few months ago. I would really like to be able to check most of the bug-related informations from bzr directly (for now, only marking bug as solved in one branch is possible, AFAIK). cheers, David From david at ar.media.kyoto-u.ac.jp Sun Apr 13 21:07:12 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 14 Apr 2008 10:07:12 +0900 Subject: [Numpy-discussion] Win32 installer: please test it In-Reply-To: References: Message-ID: <4802AE40.5090108@ar.media.kyoto-u.ac.jp> Jarrod Millman wrote: > Hello, > > David Cournapeau has prepared a new win32 installer, which is aimed at > solving the recurring problem of non working atlas on different sets > of CPU. This installer simply checks which cpu you have, and installs > the appropriate numpy accordingly (without atlas if your cpu is not > supported). Windows users, please test the installer, and report > problems with it; we hope to use this scheme for all numpy and scipy > installers in the future. Sorry for being insistent, but I would really like to get rid of those windows/sse* problems once for all for numpy/scipy, and this means having people to test the installer. cheers, David From wbaxter at gmail.com Sun Apr 13 21:37:03 2008 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 14 Apr 2008 10:37:03 +0900 Subject: [Numpy-discussion] Win32 installer: please test it In-Reply-To: <4802AE40.5090108@ar.media.kyoto-u.ac.jp> References: <4802AE40.5090108@ar.media.kyoto-u.ac.jp> Message-ID: On Mon, Apr 14, 2008 at 10:07 AM, David Cournapeau wrote: > Jarrod Millman wrote: > > Hello, > > > > David Cournapeau has prepared a new win32 installer, which is aimed at > > solving the recurring problem of non working atlas on different sets > > of CPU. This installer simply checks which cpu you have, and installs > > the appropriate numpy accordingly (without atlas if your cpu is not > > supported). Windows users, please test the installer, and report > > problems with it; we hope to use this scheme for all numpy and scipy > > installers in the future. > > Sorry for being insistent, but I would really like to get rid of those > windows/sse* problems once for all for numpy/scipy, and this means > having people to test the installer. Maybe your insistence might be more effective if it included the link part of the message. :-) """ Download from here: https://cirl.berkeley.edu/numpy/numpy-superpack-python2.5.exe """ I'll give it a try. I've been meaning to upgrade my numpy for a while now. --bb From wbaxter at gmail.com Sun Apr 13 21:51:41 2008 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 14 Apr 2008 10:51:41 +0900 Subject: [Numpy-discussion] Win32 installer: please test it In-Reply-To: References: <4802AE40.5090108@ar.media.kyoto-u.ac.jp> Message-ID: What's the installer supposed to do? It appears to have decided I have SSE3 extracted an sse3 exe to somewhere, and then quit without doing anything else. I can't seem to find where it put the numpy-1.0.5-sse3.exe it extracted, either. The details, say "Execute: numpy-1.0.5-sse3.exe" but no executing seems to have happened. --bb On Mon, Apr 14, 2008 at 10:37 AM, Bill Baxter wrote: > On Mon, Apr 14, 2008 at 10:07 AM, David Cournapeau > wrote: > > Jarrod Millman wrote: > > > Hello, > > > > > > David Cournapeau has prepared a new win32 installer, which is aimed at > > > solving the recurring problem of non working atlas on different sets > > > of CPU. This installer simply checks which cpu you have, and installs > > > the appropriate numpy accordingly (without atlas if your cpu is not > > > supported). Windows users, please test the installer, and report > > > problems with it; we hope to use this scheme for all numpy and scipy > > > installers in the future. > > > > Sorry for being insistent, but I would really like to get rid of those > > windows/sse* problems once for all for numpy/scipy, and this means > > having people to test the installer. > > Maybe your insistence might be more effective if it included the link > part of the message. :-) > > """ > Download from here: > https://cirl.berkeley.edu/numpy/numpy-superpack-python2.5.exe > """ > > I'll give it a try. I've been meaning to upgrade my numpy for a while now. > > --bb > From eads at soe.ucsc.edu Sun Apr 13 21:51:39 2008 From: eads at soe.ucsc.edu (Damian Eads) Date: Sun, 13 Apr 2008 18:51:39 -0700 Subject: [Numpy-discussion] Win32 installer: please test it In-Reply-To: <4802AE40.5090108@ar.media.kyoto-u.ac.jp> References: <4802AE40.5090108@ar.media.kyoto-u.ac.jp> Message-ID: <4802B8AB.3050706@soe.ucsc.edu> David Cournapeau wrote: > Jarrod Millman wrote: >> Hello, >> >> David Cournapeau has prepared a new win32 installer, which is aimed at >> solving the recurring problem of non working atlas on different sets >> of CPU. This installer simply checks which cpu you have, and installs >> the appropriate numpy accordingly (without atlas if your cpu is not >> supported). Windows users, please test the installer, and report >> problems with it; we hope to use this scheme for all numpy and scipy >> installers in the future. > > Sorry for being insistent, but I would really like to get rid of those > windows/sse* problems once for all for numpy/scipy, and this means > having people to test the installer. > > cheers, > > David Hi David, I'd be happy to help you test the installer (and let you know what my processor supports). Is it just the installer you want tested or do you also want us to try some computation with numpy? Nice work btw! Cheers, Damian From david at ar.media.kyoto-u.ac.jp Sun Apr 13 21:47:26 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 14 Apr 2008 10:47:26 +0900 Subject: [Numpy-discussion] Win32 installer: please test it In-Reply-To: <4802B8AB.3050706@soe.ucsc.edu> References: <4802AE40.5090108@ar.media.kyoto-u.ac.jp> <4802B8AB.3050706@soe.ucsc.edu> Message-ID: <4802B7AE.9030702@ar.media.kyoto-u.ac.jp> Damian Eads wrote: > Hi David, > > I'd be happy to help you test the installer (and let you know what my > processor supports). Is it just the installer you want tested or do you > also want us to try some computation with numpy? > Doing numpy.test() should be fine. The actual numpy installers are built the exact same way than before, so if no crash happens, it should mean it is fine, hopefully. cheers, David From david at ar.media.kyoto-u.ac.jp Sun Apr 13 21:57:36 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 14 Apr 2008 10:57:36 +0900 Subject: [Numpy-discussion] Win32 installer: please test it In-Reply-To: References: <4802AE40.5090108@ar.media.kyoto-u.ac.jp> Message-ID: <4802BA10.1080807@ar.media.kyoto-u.ac.jp> Bill Baxter wrote: > What's the installer supposed to do? It is supposed to install numpy :) The fact that it is not clear is not good, obviously. Suggestions to make it more obvious are welcomed. > It appears to have decided I have SSE3 extracted an sse3 exe to > somewhere, and then quit without doing anything else. I can't seem to > find where it put the numpy-1.0.5-sse3.exe it extracted, either. You did not have the usual serie of screens for numpy installation (python version detection, etc...) ? If not, this means the installer does not work at all. If working, you are not supposed to find the numpy-1.0.5-sse3.exe, the whole idea of the installer is to choose the right numpy installer, and run it. This is meant to simplify windows' users life, the ones who do not know nor care about what SSE or ATLAS is and whether their computer support it. > The details, say "Execute: numpy-1.0.5-sse3.exe" but no executing > seems to have happened. No executing and no error ? That's the worse thing which could happen, and the exact thing I was fearing (this and vista support). thanks for testing this out, cheers, David From wbaxter at gmail.com Sun Apr 13 22:16:37 2008 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 14 Apr 2008 11:16:37 +0900 Subject: [Numpy-discussion] Win32 installer: please test it In-Reply-To: <4802BA10.1080807@ar.media.kyoto-u.ac.jp> References: <4802AE40.5090108@ar.media.kyoto-u.ac.jp> <4802BA10.1080807@ar.media.kyoto-u.ac.jp> Message-ID: On Mon, Apr 14, 2008 at 10:57 AM, David Cournapeau wrote: > Bill Baxter wrote: > > What's the installer supposed to do? > > It is supposed to install numpy :) The fact that it is not clear is not > good, obviously. Suggestions to make it more obvious are welcomed. > > > > It appears to have decided I have SSE3 extracted an sse3 exe to > > somewhere, and then quit without doing anything else. I can't seem to > > find where it put the numpy-1.0.5-sse3.exe it extracted, either. > > You did not have the usual serie of screens for numpy installation > (python version detection, etc...) ? If not, this means the installer > does not work at all. > > If working, you are not supposed to find the numpy-1.0.5-sse3.exe, the > whole idea of the installer is to choose the right numpy installer, and > run it. This is meant to simplify windows' users life, the ones who do > not know nor care about what SSE or ATLAS is and whether their computer > support it. > > > > The details, say "Execute: numpy-1.0.5-sse3.exe" but no executing > > seems to have happened. > > No executing and no error ? That's the worse thing which could happen, > and the exact thing I was fearing (this and vista support). That's right. No execution and no error. The installer finishes (quite quickly!) as if it did everything according to plan. Anything I can do to help you track down the problem? It's a Win XP system. Some of my environment vars that might be relevant: PROCESSOR_ARCHITECTURE=x86 PROCESSOR_IDENTIFIER=x86 Family 15 Model 4 Stepping 1, GenuineIntel PROCESSOR_LEVEL=15 PROCESSOR_REVISION=0401 NUMBER_OF_PROCESSORS=2 HOME=c:\bax HOMEDRIVE=C: HOMEPATH=\Documents and Settings\baxter PYTHONPATH=c:\bax\code\python;f:\bax\code\python PYTHONSTARTUP=f:\bax\code\python\mystartup.py The default encoding of the machine is set to Japanese (CP931), but the langauge is English. That's about all I can think of that might be unusual. --bb From david at ar.media.kyoto-u.ac.jp Sun Apr 13 22:08:27 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 14 Apr 2008 11:08:27 +0900 Subject: [Numpy-discussion] Win32 installer: please test it In-Reply-To: References: <4802AE40.5090108@ar.media.kyoto-u.ac.jp> Message-ID: <4802BC9B.6050806@ar.media.kyoto-u.ac.jp> Bill Baxter wrote: > What's the installer supposed to do? > It appears to have decided I have SSE3 extracted an sse3 exe to > somewhere, and then quit without doing anything else. I can't seem to > find where it put the numpy-1.0.5-sse3.exe it extracted, either. > > The details, say "Execute: numpy-1.0.5-sse3.exe" but no executing > seems to have happened. > Ok, I've just tested it on my vmware installed windows, and got the exact same behaviour. Maybe I screwed up and uploaded the wrong file, I am looking at it now. cheers, David From david at ar.media.kyoto-u.ac.jp Sun Apr 13 23:24:04 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 14 Apr 2008 12:24:04 +0900 Subject: [Numpy-discussion] Win32 installer: please test it In-Reply-To: References: <4802AE40.5090108@ar.media.kyoto-u.ac.jp> <4802BA10.1080807@ar.media.kyoto-u.ac.jp> Message-ID: <4802CE54.6080001@ar.media.kyoto-u.ac.jp> Bill Baxter wrote: > > That's right. No execution and no error. The installer finishes > (quite quickly!) Doing nothing is quick, even on windows :) > as if it did everything according to plan. > > Anything I can do to help you track down the problem? It's a Win XP system. I think I got the problem. Now, I got it working on my two vmware windows at hand. Could you try this: http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy-superpack-python2.5.exe The problem was that I thought I did set the install directory to a windows temp directory, but I actually only set up its default value, and as such the actual numpy installer was put in C:\ (which is a really crappy default ...). Because C:\ is not in the PATH, it does not work. I set up the directory correctly, and there should be a message if the execution of the actual numpy installer fails (there is also more details in the installation log). I still do not understand why it worked on one of my windows and not the other, though. > The default encoding of the machine is set to Japanese (CP931), but > the langauge is English. Well, I happend to have the same thing: my university's laptop windows is in Japanese, and it works on this one too. cheers, David From wbaxter at gmail.com Sun Apr 13 23:46:48 2008 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 14 Apr 2008 12:46:48 +0900 Subject: [Numpy-discussion] Win32 installer: please test it In-Reply-To: <4802CE54.6080001@ar.media.kyoto-u.ac.jp> References: <4802AE40.5090108@ar.media.kyoto-u.ac.jp> <4802BA10.1080807@ar.media.kyoto-u.ac.jp> <4802CE54.6080001@ar.media.kyoto-u.ac.jp> Message-ID: Seems to work here now, too! It doesn't tell you in an easy to see place what version of SSE it decides to use. Do you think that's ok? (You can tell by looking at the "details" at then end of installation, though. Is there some way to tell this info from inside NumPy itself? If so then maybe it doesn't matter. --bb -------------------- Numpy is installed in c:\python25\lib\site-packages\numpy Numpy version 1.0.5.dev5008 Python version 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)] Found 10/10 tests for numpy.core.defmatrix Found 3/3 tests for numpy.core.memmap Found 266/266 tests for numpy.core.multiarray Found 69/69 tests for numpy.core.numeric Found 31/31 tests for numpy.core.numerictypes Found 12/12 tests for numpy.core.records Found 7/7 tests for numpy.core.scalarmath Found 16/16 tests for numpy.core.umath Found 5/5 tests for numpy.ctypeslib Found 5/5 tests for numpy.distutils.misc_util Found 2/2 tests for numpy.fft.fftpack Found 3/3 tests for numpy.fft.helper Found 20/20 tests for numpy.lib._datasource Found 10/10 tests for numpy.lib.arraysetops Found 1/1 tests for numpy.lib.financial Found 0/0 tests for numpy.lib.format Found 48/48 tests for numpy.lib.function_base Found 5/5 tests for numpy.lib.getlimits Found 4/4 tests for numpy.lib.index_tricks Found 7/7 tests for numpy.lib.io Found 4/4 tests for numpy.lib.polynomial Found 49/49 tests for numpy.lib.shape_base Found 15/15 tests for numpy.lib.twodim_base Found 43/43 tests for numpy.lib.type_check Found 1/1 tests for numpy.lib.ufunclike Found 59/59 tests for numpy.linalg Found 92/92 tests for numpy.ma.core Found 14/14 tests for numpy.ma.extras Found 7/7 tests for numpy.random Found 0/0 tests for numpy.testing.utils Found 0/0 tests for __main__ ....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... ---------------------------------------------------------------------- Ran 887 tests in 1.750s OK On Mon, Apr 14, 2008 at 12:24 PM, David Cournapeau wrote: > Bill Baxter wrote: > > > > That's right. No execution and no error. The installer finishes > > (quite quickly!) > > Doing nothing is quick, even on windows :) > > > > as if it did everything according to plan. > > > > Anything I can do to help you track down the problem? It's a Win XP system. > > I think I got the problem. Now, I got it working on my two vmware > windows at hand. Could you try this: > > http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy-superpack-python2.5.exe > > The problem was that I thought I did set the install directory to a > windows temp directory, but I actually only set up its default value, > and as such the actual numpy installer was put in C:\ (which is a really > crappy default ...). Because C:\ is not in the PATH, it does not work. > > I set up the directory correctly, and there should be a message if the > execution of the actual numpy installer fails (there is also more > details in the installation log). I still do not understand why it > worked on one of my windows and not the other, though. > > > > The default encoding of the machine is set to Japanese (CP931), but > > the langauge is English. > > Well, I happend to have the same thing: my university's laptop windows > is in Japanese, and it works on this one too. > > > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From bolme1234 at comcast.net Sun Apr 13 23:56:05 2008 From: bolme1234 at comcast.net (David Bolme) Date: Sun, 13 Apr 2008 21:56:05 -0600 Subject: [Numpy-discussion] PCA on set of face images In-Reply-To: <200804071140.24932.meine@informatik.uni-hamburg.de> References: <200804071140.24932.meine@informatik.uni-hamburg.de> Message-ID: <4E00F37B-F706-4C1B-9294-F2C5532AF726@comcast.net> If you are interested I now have the code on source forge. It still needs some critical documentation. I am planing to get this documented and a beta release sometime this summer. Currently it ties together PIL, OpenCV, numpy/scipy, LibSVM, and some of own code with an emphases on face recognition since that is my research area. http://pyvision.sourceforge.net On Apr 7, 2008, at 3:40 AM, Hans Meine wrote: > Am Dienstag, 11. M?rz 2008 00:24:04 schrieb David Bolme: >> The steps you describe here are correct. I am putting together an >> open source computer vision library based on numpy/scipy. It will >> include an automatic PCA algorithm with face detection, eye >> detection, >> PCA dimensionally reduction, and distance measurement. If you are >> interested let me know and I will redouble my efforts to release the >> code soon. > > That's interesting, we're also working on a computer vision module > using NumPy > (actually, a VIGRA <-> NumPy binding sort of), and there's > scipy.ndimage, > too. Maybe (part of) your code could be integrated into the > latter? I am > looking forward to it anyway. > > -- > Ciao, / / > /--/ > / / ANS > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From david at ar.media.kyoto-u.ac.jp Mon Apr 14 00:08:56 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 14 Apr 2008 13:08:56 +0900 Subject: [Numpy-discussion] Win32 installer: please test it In-Reply-To: References: <4802AE40.5090108@ar.media.kyoto-u.ac.jp> <4802BA10.1080807@ar.media.kyoto-u.ac.jp> <4802CE54.6080001@ar.media.kyoto-u.ac.jp> Message-ID: <4802D8D8.2040602@ar.media.kyoto-u.ac.jp> Bill Baxter wrote: > Seems to work here now, too! > > It doesn't tell you in an easy to see place what version of SSE it > decides to use. Do you think that's ok? I simply did not think it was useful information. How would you like to get the information ? To be honest, I do not want to spend too much time on this (I do not use windows at all): I was just really fed up with all the messages about numpy crashes on windows, and wanted a quick, working solution. I intend to put everything necessary to create those installers in some place on the scipy.org website of somewhere else in a near future, so that other people can contribute and enhance (the big part is actually another installer, which I annouced a few weeks ago, and which contain blas/lapack and ATLAS for both mingw and vs-based builds, as well as a bunch of scripts to regenerate all of them automatically). cheers, David From wbaxter at gmail.com Mon Apr 14 00:40:08 2008 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 14 Apr 2008 13:40:08 +0900 Subject: [Numpy-discussion] Win32 installer: please test it In-Reply-To: <4802D8D8.2040602@ar.media.kyoto-u.ac.jp> References: <4802AE40.5090108@ar.media.kyoto-u.ac.jp> <4802BA10.1080807@ar.media.kyoto-u.ac.jp> <4802CE54.6080001@ar.media.kyoto-u.ac.jp> <4802D8D8.2040602@ar.media.kyoto-u.ac.jp> Message-ID: On Mon, Apr 14, 2008 at 1:08 PM, David Cournapeau wrote: > Bill Baxter wrote: > > Seems to work here now, too! > > > > It doesn't tell you in an easy to see place what version of SSE it > > decides to use. Do you think that's ok? > > I simply did not think it was useful information. How would you like to > get the information ? Well, reason for wanting it were two-fold 1) I think my computer has SSE3, so I would be surprised if it came up saying it was going to install the SSE2 version. 2) If there are still reports about Numpy crashing even using this, then the first thing you will want to ask people is "which version did the installer install for you". They're more likely to know the answer if the installer tells them. I could imagine a couple of ways to show the information: 1) after determining the version to use pop up a message box saying something like "This will install the SSE3 enabled version of Numpy 1.0.5. OK?" 2) Put the version used in some text on the final panel (the panel with the "Details button"). There's a big blank space there now. It could say "successfully installed Numpy version Blah.Blah-SSE3" > To be honest, I do not want to spend too much time > on this (I do not use windows at all): I was just really fed up with all > the messages about numpy crashes on windows, and wanted a quick, working > solution. Ok. Didn't realize that. > I intend to put everything necessary to create those installers in some > place on the scipy.org website of somewhere else in a near future, so > that other people can contribute and enhance (the big part is actually > another installer, which I annouced a few weeks ago, and which contain > blas/lapack and ATLAS for both mingw and vs-based builds, as well as a > bunch of scripts to regenerate all of them automatically). That sounds good. From there it probably wouldn't take someone else very long to make the suggested change. I've used NSIS before, so I could probably figure out how. --bb From ondrej at certik.cz Mon Apr 14 06:06:08 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 14 Apr 2008 12:06:08 +0200 Subject: [Numpy-discussion] numpy release Message-ID: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> Hi Jarrod, any news with the 1.0.5? If you have same prerelease, I'd like to test it. Debian has just moved from python2.4 to python2.5 yesterday, so I'd like to test numpy in advance, I am sure there will be some issues to fix. Ondrej From gruben at bigpond.net.au Mon Apr 14 07:47:19 2008 From: gruben at bigpond.net.au (Gary Ruben) Date: Mon, 14 Apr 2008 21:47:19 +1000 Subject: [Numpy-discussion] Win32 installer: please test it In-Reply-To: References: Message-ID: <48034447.2080706@bigpond.net.au> Tested fine on my old Classic Athlon 500 (no SSE) under Win98. It correctly reported installing the non-SSE version when I clicked on the details button on the last page of the install wizard. Whereas previously numpy.test() would bring up an illegal operation dialog box, now all tests pass. Nice!. Gary R. >>> N.test() Numpy is installed in C:\PYTHON25\lib\site-packages\numpy Numpy version 1.0.5.dev5008 Python version 2.5.2a0 (release25-maint:59715M, Jan 3 2008, 22:45:43) [MSC v.13 10 32 bit (Intel)] Found 10/10 tests for numpy.core.defmatrix Found 3/3 tests for numpy.core.memmap Found 266/266 tests for numpy.core.multiarray Found 69/69 tests for numpy.core.numeric Found 31/31 tests for numpy.core.numerictypes Found 12/12 tests for numpy.core.records Found 7/7 tests for numpy.core.scalarmath Found 16/16 tests for numpy.core.umath Found 5/5 tests for numpy.ctypeslib Found 5/5 tests for numpy.distutils.misc_util Found 2/2 tests for numpy.fft.fftpack Found 3/3 tests for numpy.fft.helper Found 20/20 tests for numpy.lib._datasource Found 10/10 tests for numpy.lib.arraysetops Found 1/1 tests for numpy.lib.financial Found 0/0 tests for numpy.lib.format Found 48/48 tests for numpy.lib.function_base Found 5/5 tests for numpy.lib.getlimits Found 4/4 tests for numpy.lib.index_tricks Found 7/7 tests for numpy.lib.io Found 4/4 tests for numpy.lib.polynomial Found 49/49 tests for numpy.lib.shape_base Found 15/15 tests for numpy.lib.twodim_base Found 43/43 tests for numpy.lib.type_check Found 1/1 tests for numpy.lib.ufunclike Found 59/59 tests for numpy.linalg Found 92/92 tests for numpy.ma.core Found 14/14 tests for numpy.ma.extras Found 7/7 tests for numpy.random Found 0/0 tests for numpy.testing.utils Found 0/0 tests for __main__ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ....... ---------------------------------------------------------------------- Ran 887 tests in 7.580s OK Jarrod Millman wrote: > Hello, > > David Cournapeau has prepared a new win32 installer, which is aimed at > solving the recurring problem of non working atlas on different sets > of CPU. This installer simply checks which cpu you have, and installs > the appropriate numpy accordingly (without atlas if your cpu is not > supported). Windows users, please test the installer, and report From josef.pktd at gmail.com Mon Apr 14 11:32:10 2008 From: josef.pktd at gmail.com (joep) Date: Mon, 14 Apr 2008 08:32:10 -0700 (PDT) Subject: [Numpy-discussion] Win32 installer: please test it In-Reply-To: <48034447.2080706@bigpond.net.au> References: <48034447.2080706@bigpond.net.au> Message-ID: <25724e4e-1516-451f-8215-63799e14878f@b9g2000prh.googlegroups.com> Everything installed without problem on Intel Pentium M on my notebook recognized as SSE2 capable. Installer found Python 2.5. immediately, which I just installed and all my windows environment settings are still setup for python 2.4 Thanks, Josef >>> numpy.test() Numpy is installed in C:\Programs\Python25\lib\site-packages\numpy Numpy version 1.0.5.dev5008 Python version 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] Found 10/10 tests for numpy.core.defmatrix Found <...> ---------------------------------------------------------------------- Ran 887 tests in 5.922s OK >>> numpy.show_config() atlas_threads_info: NOT AVAILABLE blas_opt_info: libraries = ['f77blas', 'cblas', 'atlas'] library_dirs = ['C:\\local\\lib\\sse2'] define_macros = [('NO_ATLAS_INFO', 2)] language = c atlas_blas_threads_info: NOT AVAILABLE lapack_opt_info: libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] library_dirs = ['C:\\local\\lib\\sse2'] define_macros = [('NO_ATLAS_INFO', 2)] language = f77 atlas_info: libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] library_dirs = ['C:\\local\\lib\\sse2'] language = f77 lapack_mkl_info: NOT AVAILABLE blas_mkl_info: NOT AVAILABLE atlas_blas_info: libraries = ['f77blas', 'cblas', 'atlas'] library_dirs = ['C:\\local\\lib\\sse2'] language = c mkl_info: NOT AVAILABLE From millman at berkeley.edu Mon Apr 14 14:43:23 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 14 Apr 2008 11:43:23 -0700 Subject: [Numpy-discussion] Win32 installer: please test it In-Reply-To: <4802CE54.6080001@ar.media.kyoto-u.ac.jp> References: <4802AE40.5090108@ar.media.kyoto-u.ac.jp> <4802BA10.1080807@ar.media.kyoto-u.ac.jp> <4802CE54.6080001@ar.media.kyoto-u.ac.jp> Message-ID: On Sun, Apr 13, 2008 at 8:24 PM, David Cournapeau wrote: > I think I got the problem. Now, I got it working on my two vmware > windows at hand. Could you try this: > > http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy-superpack-python2.5.exe Did you have a problem putting it on the cirl site? -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From efiring at hawaii.edu Mon Apr 14 14:59:30 2008 From: efiring at hawaii.edu (Eric Firing) Date: Mon, 14 Apr 2008 08:59:30 -1000 Subject: [Numpy-discussion] ones with non-native dtype byteorder Message-ID: <4803A992.4090504@hawaii.edu> This (on little-endian machine) surprised me: In [23]:np.ones((1,), dtype=' References: <4803A992.4090504@hawaii.edu> Message-ID: <3d375d730804141304m366d59dpdd9a6a7d137c50fc@mail.gmail.com> On Mon, Apr 14, 2008 at 1:59 PM, Eric Firing wrote: > This (on little-endian machine) surprised me: > > In [23]:np.ones((1,), dtype=' Out[23]:array([1], dtype=int16) > > In [24]:np.ones((1,), dtype='>i2') > Out[24]:array([256], dtype=int16) > > I expected the value to be [1] in either case. Am I wrong? You are correct. It is a bug in the @NAME at _fillwithscalar() template in arraytypes.inc.src; it does not take endianness into consideration. I mentioned this to Travis, and he might have time to hop on this, but if anyone else can fix it faster, please go for it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From neilcrighton at gmail.com Mon Apr 14 16:47:48 2008 From: neilcrighton at gmail.com (Neil Crighton) Date: Mon, 14 Apr 2008 21:47:48 +0100 Subject: [Numpy-discussion] Win32 installer: please test it (Bill Baxter) Message-ID: <63751c30804141347u699e0c24r8243cabb5a17ff53@mail.gmail.com> The Win32 installer works on my Vista machine. There is one failed test, but I think that's just because it tries to write somewhere it doesn't have permission - I installed Python in /Program Files/Python25/, and you need to be an administrator to write to Program Files/. Here's the error message: ERROR: test_ValidHTTP (numpy.lib.tests.test__datasource.TestDataSourceOpen) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Program Files\Python25\lib\site-packages\numpy\lib\tests\test__datasource.py", line 79, in test_ValidHTTP assert self.ds.open(valid_httpurl()) File "C:\Program Files\Python25\lib\site-packages\numpy\lib\_datasource.py", line 366, in open found = self._findfile(path) File "C:\Program Files\Python25\lib\site-packages\numpy\lib\_datasource.py", line 243, in _findfile name = self._cache(name) File "C:\Program Files\Python25\lib\site-packages\numpy\lib\_datasource.py", line 203, in _cache file(upath, 'w').write(openedurl.read()) IOError: [Errno 13] Permission denied: '/index.html' ---------------------------------------------------------------------- Ran 887 tests in 4.914s FAILED (errors=1) Neil From oliphant at enthought.com Mon Apr 14 16:47:49 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 14 Apr 2008 15:47:49 -0500 Subject: [Numpy-discussion] ones with non-native dtype byteorder In-Reply-To: <3d375d730804141304m366d59dpdd9a6a7d137c50fc@mail.gmail.com> References: <4803A992.4090504@hawaii.edu> <3d375d730804141304m366d59dpdd9a6a7d137c50fc@mail.gmail.com> Message-ID: <4803C2F5.2090200@enthought.com> Robert Kern wrote: > On Mon, Apr 14, 2008 at 1:59 PM, Eric Firing wrote: > >> This (on little-endian machine) surprised me: >> >> In [23]:np.ones((1,), dtype='> Out[23]:array([1], dtype=int16) >> >> In [24]:np.ones((1,), dtype='>i2') >> Out[24]:array([256], dtype=int16) >> >> I expected the value to be [1] in either case. Am I wrong? >> > > You are correct. It is a bug in the @NAME at _fillwithscalar() template > in arraytypes.inc.src; it does not take endianness into consideration. > I mentioned this to Travis, and he might have time to hop on this, but > if anyone else can fix it faster, please go for it. > > Could you check latest SVN. This should be fixed now. -Travis O. From bsouthey at gmail.com Mon Apr 14 17:37:22 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 14 Apr 2008 16:37:22 -0500 Subject: [Numpy-discussion] Win32 installer: please test it In-Reply-To: References: Message-ID: Hi, The installer requires 2.5 but that is not explicitly stated. Given that numpy and scipy technically require Python 2.3 as a minimum, the installer should support Python 2.3 and Python 2.4. Otherwise, it needs to clearly state that Python 2.5 is a must. I did install Python 2.5.2 on my WIndows XP AMD Athlon XP 2100+ system and all the tests passed. Regards Bruce On Thu, Apr 10, 2008 at 12:18 PM, Jarrod Millman wrote: > Hello, > > David Cournapeau has prepared a new win32 installer, which is aimed at > solving the recurring problem of non working atlas on different sets > of CPU. This installer simply checks which cpu you have, and installs > the appropriate numpy accordingly (without atlas if your cpu is not > supported). Windows users, please test the installer, and report > problems with it; we hope to use this scheme for all numpy and scipy > installers in the future. > > Download from here: > https://cirl.berkeley.edu/numpy/numpy-superpack-python2.5.exe > > Technical details: > > - ATLAS is 3.8.0, and built with cygwin. Built on Core 2 duo for > SSE3, pentium 4 xeon for SSE2. > - BLAS/LAPACK are built with mingw. > - only SSE3 and SSE2 are supported. If you don't have at least sse2, > it will not use ATLAS, but netlib blas/lapack instead. > - the numpy installers are prepared with build_wininst for each > different supported architecture > - the numpy installers are then bundled in another installer, based on nsis. > - Nsis is an open source installer, based on a small scripting > language, and is extensible by plug-ins. I wrote a plug-in to detect > the cpu type, and the installer will simply execute a different numpy > installer depending on the cpu. > > Thanks, > > -- > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From efiring at hawaii.edu Mon Apr 14 19:45:34 2008 From: efiring at hawaii.edu (Eric Firing) Date: Mon, 14 Apr 2008 13:45:34 -1000 Subject: [Numpy-discussion] ones with non-native dtype byteorder In-Reply-To: <4803C2F5.2090200@enthought.com> References: <4803A992.4090504@hawaii.edu> <3d375d730804141304m366d59dpdd9a6a7d137c50fc@mail.gmail.com> <4803C2F5.2090200@enthought.com> Message-ID: <4803EC9E.6060700@hawaii.edu> Travis E. Oliphant wrote: > Robert Kern wrote: >> On Mon, Apr 14, 2008 at 1:59 PM, Eric Firing wrote: >> >>> This (on little-endian machine) surprised me: >>> >>> In [23]:np.ones((1,), dtype='>> Out[23]:array([1], dtype=int16) >>> >>> In [24]:np.ones((1,), dtype='>i2') >>> Out[24]:array([256], dtype=int16) >>> >>> I expected the value to be [1] in either case. Am I wrong? >>> >> You are correct. It is a bug in the @NAME at _fillwithscalar() template >> in arraytypes.inc.src; it does not take endianness into consideration. >> I mentioned this to Travis, and he might have time to hop on this, but >> if anyone else can fix it faster, please go for it. >> >> > Could you check latest SVN. This should be fixed now. Indeed it is. Thank you. Eric > > -Travis O. > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From david at ar.media.kyoto-u.ac.jp Mon Apr 14 22:09:30 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 15 Apr 2008 11:09:30 +0900 Subject: [Numpy-discussion] Win32 installer: please test it In-Reply-To: References: Message-ID: <48040E5A.8060300@ar.media.kyoto-u.ac.jp> Bruce Southey wrote: > Hi, > The installer requires 2.5 but that is not explicitly stated. Given > that numpy and scipy technically require Python 2.3 as a minimum, the > installer should support Python 2.3 and Python 2.4. Otherwise, it > needs to clearly state that Python 2.5 is a must. > Stricly speaking, when numpy says it supports all python starting at 2.3, this does not mean we have to provide binaries for all versions. It means that the code itself is compatible. That being said, the posted installer was just a test version. We will do just as before, that is provide a separate installer for python 2.3, 2.4 and 2.5. cheers, David From david at ar.media.kyoto-u.ac.jp Mon Apr 14 22:12:10 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 15 Apr 2008 11:12:10 +0900 Subject: [Numpy-discussion] Win32 installer: please test it (Bill Baxter) In-Reply-To: <63751c30804141347u699e0c24r8243cabb5a17ff53@mail.gmail.com> References: <63751c30804141347u699e0c24r8243cabb5a17ff53@mail.gmail.com> Message-ID: <48040EFA.5010304@ar.media.kyoto-u.ac.jp> Neil Crighton wrote: > The Win32 installer works on my Vista machine. There is one failed > test, but I think that's just because it tries to write somewhere it > doesn't have permission - I installed Python in /Program > Files/Python25/, and you need to be an administrator to write to > Program Files/. > > Here's the error message: > > ERROR: test_ValidHTTP (numpy.lib.tests.test__datasource.TestDataSourceOpen) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Program > Files\Python25\lib\site-packages\numpy\lib\tests\test__datasource.py", > line 79, in test_ValidHTTP > assert self.ds.open(valid_httpurl()) > File "C:\Program > Files\Python25\lib\site-packages\numpy\lib\_datasource.py", line 366, > in open > found = self._findfile(path) > File "C:\Program > Files\Python25\lib\site-packages\numpy\lib\_datasource.py", line 243, > in _findfile > name = self._cache(name) > File "C:\Program > Files\Python25\lib\site-packages\numpy\lib\_datasource.py", line 203, > in _cache > file(upath, 'w').write(openedurl.read()) > IOError: [Errno 13] Permission denied: '/index.html' > > ---------------------------------------------------------------------- > Ran 887 tests in 4.914s > Yes, I noticed that too. I opened a bug, but this has nothing to do with this installer, this would have happened anyway. David From oliphant at enthought.com Tue Apr 15 00:59:14 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 14 Apr 2008 23:59:14 -0500 Subject: [Numpy-discussion] Release of NumPy Message-ID: <48043622.1090606@enthought.com> After the SciPy sprints some useful discussions took place that helped us all realize that we have made enough changes to the code base that we will need to call any release from the trunk 1.1 I don't think that is a big problem. However, there have also been a lot of substantial bug fixes that really deserve to be applied against 1.0.4 and released as 1.0.5 So, the question is. Do we have enough energy in the community to release the current trunk as 1.1 as well as release 1.0.5 which does not contain any of the code/api changes, but just the bug fixes? The answer will be yes if somebody volunteers to back-port just the bug-fixes to 1.0.4. I would really like there to be a bug-fix 1.0.5 release if at all possible. Best regards, -Travis From millman at berkeley.edu Tue Apr 15 01:05:56 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 14 Apr 2008 22:05:56 -0700 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <48043622.1090606@enthought.com> References: <48043622.1090606@enthought.com> Message-ID: On Mon, Apr 14, 2008 at 9:59 PM, Travis E. Oliphant wrote: > So, the question is. Do we have enough energy in the community to > release the current trunk as 1.1 as well as release 1.0.5 which does not > contain any of the code/api changes, but just the bug fixes? The answer > will be yes if somebody volunteers to back-port just the bug-fixes to > 1.0.4. I would really like there to be a bug-fix 1.0.5 release if at > all possible. Personally, I would rather see everyone's effort focused on 1.1 and the 1.2 (previously referred to as 1.1). But if anyone is interested in backporting some of the bug-fixes, I will help coordinate a 1.0.5 release. For now, I hope to keep everyone focused on the trunk, which will become 1.1.0 very soon. I will send out an email fairly soon listing the last things that need to be done before releasing 1.1.0. Basically, I would like to have some testing of the new Windows and Mac binaries. David C. has all ready sent out an email asking for help testing the new Window's binaries. Chris Burns or I will be sending out an email asking for help testing the new Universal Mac binaries within the next day. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From charlesr.harris at gmail.com Tue Apr 15 01:10:21 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 14 Apr 2008 23:10:21 -0600 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <48043622.1090606@enthought.com> References: <48043622.1090606@enthought.com> Message-ID: On Mon, Apr 14, 2008 at 10:59 PM, Travis E. Oliphant wrote: > After the SciPy sprints some useful discussions took place that helped > us all realize that we have made enough changes to the code base that we > will need to call any release from the trunk 1.1 > > I don't think that is a big problem. However, there have also been a > lot of substantial bug fixes that really deserve to be applied against > 1.0.4 and released as 1.0.5 > > So, the question is. Do we have enough energy in the community to > release the current trunk as 1.1 as well as release 1.0.5 which does not > contain any of the code/api changes, but just the bug fixes? The answer > will be yes if somebody volunteers to back-port just the bug-fixes to > 1.0.4. I would really like there to be a bug-fix 1.0.5 release if at > all possible. > I think we have a long way to go to get to 1.1. What API changes have we made that are so great that we can't release 1.0.5? The only area where I have seen real problems is in masked arrays. My preference would be to release the trunk as 1.0.5, and if not, simply deprecate 1.0.4 and move to 1.1. I don't think it's worth the time and effort to backport anything. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From millman at berkeley.edu Tue Apr 15 01:20:52 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 14 Apr 2008 22:20:52 -0700 Subject: [Numpy-discussion] NumPy release numbering Message-ID: Hey, This is just a short note to explain the plan for future release numbering. Basically, we are just going to be much more rigorous about following everyone's expectations about release numbers. 1.1.0 will be released ASAP and will include some minor API breakage and new features 1.1.x will be a purely bug-fix release 1.2.0 will be released by the end of the summer and will include some minor API breakage and new features 1.2.x will be a purely bug-fix release At this point we will probably have a longer period of code stabilization in the 1.2.x release series before starting to work on a 1.3 release. I also hope to get a SciPy 0.7 release out soon based on NumPy 1.1 and then we will release a SciPy 0.8 based on NumPy 1.2. When/if we release NumPy 2.0, we will allow much more radically API breakage. The idea being that a major release might require people 'rewriting' code where a minor release would require a small amount of refactoring. Bugfixes to a release shouldn't require any changes to code depending on our software. Just for reference: major.minor.bugfix I.e., we are in the 1.x major release cycle and we are making two new minor releases this summer (1.1 ASAP and 1.2 by the end of summer). We will release bug-fixes as necessary. For example, we could release 1.0.5 if anyone feels there are some important bug-fixes they wish to backport from the trunk to the 1.0 branch (which I will be happy to make if anyone so wishes). Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From david at ar.media.kyoto-u.ac.jp Tue Apr 15 01:12:06 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 15 Apr 2008 14:12:06 +0900 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com> Message-ID: <48043926.4080305@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > I think we have a long way to go to get to 1.1. What API changes have > we made that are so great that we can't release 1.0.5? The only area > where I have seen real problems is in masked arrays. My preference > would be to release the trunk as 1.0.5, and if not, simply deprecate > 1.0.4 and move to 1.1. I don't think it's worth the time and effort > to backport anything. Normally, api versioning does not depend on how big your changes are: if you break even one function api, you should signal an API break. It really boils down to conventions: what do you do when you break API ? For major C/C++ libraries which care about API/ABI stability, this means a new major version. For python, this means new minor version. For numpy, it could be new micro version... cheers, David From millman at berkeley.edu Tue Apr 15 01:28:05 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 14 Apr 2008 22:28:05 -0700 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com> Message-ID: On Mon, Apr 14, 2008 at 10:10 PM, Charles R Harris wrote: > I think we have a long way to go to get to 1.1. What API changes have we > made that are so great that we can't release 1.0.5? The only area where I > have seen real problems is in masked arrays. My preference would be to > release the trunk as 1.0.5, and if not, simply deprecate 1.0.4 and move to > 1.1. I don't think it's worth the time and effort to backport anything. I was originally expecting to release the trunk as 1.0.5, but I just started worrying that we had been a little too liberal (mea culpa) with allowing some code changes that were pushing in API breakage and new features. Since I think the trunk is fairly stable and we should release ASAP, I propose to have a very short-lived 1.1 and move to developing 1.2 as soon as 1.1 is released. This will also allow us to "officially" deprecate or warn users about upcoming changes in 1.2 in the 1.1 release series. Sorry that I wasn't more careful in watching over the 1.0.5 release and that we are having to go to a 1.1 release on such short notice. Again, just to be clear: This doesn't signify a change in the scope of the next release, we are just being more honest about the status of this release. We could have added more API breakage and features to this release, if I had noticed what was happening sooner. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From millman at berkeley.edu Tue Apr 15 01:36:19 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 14 Apr 2008 22:36:19 -0700 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com> Message-ID: On Mon, Apr 14, 2008 at 10:28 PM, Jarrod Millman wrote: > On Mon, Apr 14, 2008 at 10:10 PM, Charles R Harris > wrote: > > I think we have a long way to go to get to 1.1. What API changes have we > > made that are so great that we can't release 1.0.5? The only area where I > > have seen real problems is in masked arrays. My preference would be to > > release the trunk as 1.0.5, and if not, simply deprecate 1.0.4 and move to > > 1.1. I don't think it's worth the time and effort to backport anything. Also as I was maintaining the release notes for "1.0.5", I started noticing that we have a fairly extensive list of features/changes for a microrelease, particularly for such a late microrelease: http://projects.scipy.org/scipy/numpy/milestone/1.0.5 I would expect most users to expect substantially less changes between 1.0.4 and 1.0.5 than between 1.0.0 and 1.0.2 (or 1.0.3). This numbering thing is really all about making sure we don't violate our user's expectations. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From charlesr.harris at gmail.com Tue Apr 15 01:38:44 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 14 Apr 2008 23:38:44 -0600 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com> Message-ID: On Mon, Apr 14, 2008 at 11:28 PM, Jarrod Millman wrote: > On Mon, Apr 14, 2008 at 10:10 PM, Charles R Harris > wrote: > > I think we have a long way to go to get to 1.1. What API changes have we > > made that are so great that we can't release 1.0.5? The only area where > I > > have seen real problems is in masked arrays. My preference would be to > > release the trunk as 1.0.5, and if not, simply deprecate 1.0.4 and move > to > > 1.1. I don't think it's worth the time and effort to backport anything. > > I was originally expecting to release the trunk as 1.0.5, but I just > started worrying that we had been a little too liberal (mea culpa) > with allowing some code changes that were pushing in API breakage and > new features. Since I think the trunk is fairly stable and we should > release ASAP, I propose to have a very short-lived 1.1 and move to > developing 1.2 as soon as 1.1 is released. This will also allow us to > "officially" deprecate or warn users about upcoming changes in 1.2 in > the 1.1 release series. > Let's not rush 1.1, then. I think with another week or two we should be able to settle most of the outstanding bugs and it would be good to do so before getting caught up in planning 1.2. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From millman at berkeley.edu Tue Apr 15 01:43:14 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 14 Apr 2008 22:43:14 -0700 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com> Message-ID: On Mon, Apr 14, 2008 at 10:38 PM, Charles R Harris wrote: > Let's not rush 1.1, then. I think with another week or two we should be able > to settle most of the outstanding bugs and it would be good to do so before > getting caught up in planning 1.2. +1. I think that we are so close to closing every reported bug that spending another week or two would be wise. I also want to have very stable and easy-to-use binaries for both Windows and Mac for this release, which we are very close to doing. Chris has all ready made MacOS X binaries; he is just cleaning up a few things before asking for more testing. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From aisaac at american.edu Tue Apr 15 01:54:14 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 15 Apr 2008 01:54:14 -0400 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com> Message-ID: On Mon, 14 Apr 2008, Charles R Harris apparently wrote: > Let's not rush 1.1, then. Will matrix behavior change in 1.1, as discussed from time to time? Perhaps it just takes a very small change in __getitem__: Cheers, Alan Isaac From millman at berkeley.edu Tue Apr 15 02:33:16 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 14 Apr 2008 23:33:16 -0700 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com> Message-ID: On Mon, Apr 14, 2008 at 10:54 PM, Alan G Isaac wrote: > On Mon, 14 Apr 2008, Charles R Harris apparently wrote: > > Let's not rush 1.1, then. > > Will matrix behavior change in 1.1, as discussed from time > to time? Perhaps it just takes a very small change in __getitem__: > Given that the next release will be 1.1, I think it is reasonable to include a few additional API breaks. But I would prefer that we focus on fixing bugs and not introducing new ones, so I would strongly encourage everyone to be *very* conservative in adding new features or API breaks at this point. Those are just general comments, I don't have any particular opinion on your specific question. I would be interested in what other people think. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From robert.kern at gmail.com Tue Apr 15 02:38:45 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 15 Apr 2008 01:38:45 -0500 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com> Message-ID: <3d375d730804142338g2833ef75w84b01e7a99611f2@mail.gmail.com> On Tue, Apr 15, 2008 at 1:33 AM, Jarrod Millman wrote: > On Mon, Apr 14, 2008 at 10:54 PM, Alan G Isaac wrote: > > On Mon, 14 Apr 2008, Charles R Harris apparently wrote: > > > Let's not rush 1.1, then. > > > > Will matrix behavior change in 1.1, as discussed from time > > to time? Perhaps it just takes a very small change in __getitem__: > > > > Given that the next release will be 1.1, I think it is reasonable to > include a few additional API breaks. -lots. I don't want to break API compatibility again no matter what version number we bump to, 1.1, 2.0, or 24. It is simply not okay to do this again and again. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From millman at berkeley.edu Tue Apr 15 02:44:41 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 14 Apr 2008 23:44:41 -0700 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <3d375d730804142338g2833ef75w84b01e7a99611f2@mail.gmail.com> References: <48043622.1090606@enthought.com> <3d375d730804142338g2833ef75w84b01e7a99611f2@mail.gmail.com> Message-ID: On Mon, Apr 14, 2008 at 11:38 PM, Robert Kern wrote: > > Given that the next release will be 1.1, I think it is reasonable to > > include a few additional API breaks. > > -lots. I don't want to break API compatibility again no matter what > version number we bump to, 1.1, 2.0, or 24. It is simply not okay to > do this again and again. I didn't mean to sound like I was encouraging API breakage. I agree with Robert we need to be extremely careful about breaking the API. NumPy is a fairly mature project that is depended on by lots and lots of code. If we break the API, we better have an extremely good reason and, even then, we need to minimize it as much as possible. My main concern is that when we do break APIs we can't do it in a micro release like 1.0.5. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From david at ar.media.kyoto-u.ac.jp Tue Apr 15 02:41:29 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 15 Apr 2008 15:41:29 +0900 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <3d375d730804142338g2833ef75w84b01e7a99611f2@mail.gmail.com> References: <48043622.1090606@enthought.com> <3d375d730804142338g2833ef75w84b01e7a99611f2@mail.gmail.com> Message-ID: <48044E19.2050906@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > > -lots. I don't want to break API compatibility again no matter what > version number we bump to, 1.1, 2.0, or 24. It is simply not okay to > do this again and again. > > I didn't dare saying anything, but now that I see I am not the only one, I can sheepishly say I agree more than 100 %. At some point, we will really have to stop breaking api every few weeks. That was ok when numpy was not 1.*, but as numpy will - hopefully - get some bigger traction, it will just become unmanageable to keep breaking things, users will hate it. This is so much more important than import conventions and other details when you have a large user-base :) cheers, David From wright at esrf.fr Tue Apr 15 03:31:57 2008 From: wright at esrf.fr (Jon Wright) Date: Tue, 15 Apr 2008 09:31:57 +0200 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com> Message-ID: <480459ED.4090308@esrf.fr> Alan G Isaac wrote: > > Will matrix behavior change in 1.1, as discussed from time > to time? Perhaps it just takes a very small change in __getitem__: > Quoting from: http://mail.python.org/pipermail/python-dev/2008-March/077723.html """ Executive summary: Python 2.6 and 3.0 finals are planned for September 3, 2008. """ In the event that there really is a valid reason to change the API (much like serial killers have their own valid reasons for their crimes...). Why not finalise that perfect API for the python3.0 build, when everyone is expecting code to change for 3K? Name them numpy3.x.x to match the python major version. I bring this up because I'm actually wondering which bits of numpy will make it into the 3k standard library? Must've already been decided by now, seeing as 3K is so close to release already. Has anyone tried numpy on the alpha releases already? Jon From robert.kern at gmail.com Tue Apr 15 03:41:18 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 15 Apr 2008 02:41:18 -0500 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <480459ED.4090308@esrf.fr> References: <48043622.1090606@enthought.com> <480459ED.4090308@esrf.fr> Message-ID: <3d375d730804150041j67356737ic17452c9f41cecf5@mail.gmail.com> On Tue, Apr 15, 2008 at 2:31 AM, Jon Wright wrote: > Alan G Isaac wrote: > > > > Will matrix behavior change in 1.1, as discussed from time > > to time? Perhaps it just takes a very small change in __getitem__: > > > > Quoting from: > http://mail.python.org/pipermail/python-dev/2008-March/077723.html > """ > Executive summary: Python 2.6 and 3.0 finals are planned for September > 3, 2008. > """ > > In the event that there really is a valid reason to change the API (much > like serial killers have their own valid reasons for their crimes...). > Why not finalise that perfect API for the python3.0 build, when everyone > is expecting code to change for 3K? Name them numpy3.x.x to match the > python major version. The Python community has been begging package owners *not* to do this. That would impose yet another hurdle to the adoption of Python 3.0. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From haase at msg.ucsf.edu Tue Apr 15 03:58:07 2008 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 15 Apr 2008 09:58:07 +0200 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <3d375d730804150041j67356737ic17452c9f41cecf5@mail.gmail.com> References: <48043622.1090606@enthought.com> <480459ED.4090308@esrf.fr> <3d375d730804150041j67356737ic17452c9f41cecf5@mail.gmail.com> Message-ID: On Tue, Apr 15, 2008 at 9:41 AM, Robert Kern wrote: > On Tue, Apr 15, 2008 at 2:31 AM, Jon Wright wrote: > > Alan G Isaac wrote: > > > > > > Will matrix behavior change in 1.1, as discussed from time > > > to time? Perhaps it just takes a very small change in __getitem__: > > > > > > > Quoting from: > > http://mail.python.org/pipermail/python-dev/2008-March/077723.html > > """ > > Executive summary: Python 2.6 and 3.0 finals are planned for September > > 3, 2008. > > """ > > > > In the event that there really is a valid reason to change the API (much > > like serial killers have their own valid reasons for their crimes...). > > Why not finalise that perfect API for the python3.0 build, when everyone > > is expecting code to change for 3K? Name them numpy3.x.x to match the > > python major version. > > The Python community has been begging package owners *not* to do this. > That would impose yet another hurdle to the adoption of Python 3.0. > > So, to summarize one more time: """ What used to be talked about as "1.1" will now become "1.2" --- and "1.0.5" will be "1.1" """ Did I get this right !? I'm curious to know _which_ were the changes that break the API. I thought all additions like "axis=0" were made without breaking the backwards compatibility. Otherwise it would (should !!!) have been added as "axis=None". Please keep in mind that there are a number of these "awkward" "wrong-default" arguments. I was hoping these would be unified [e.g. always same default, axis=None] very soon -- that is in 1.1. Also on my list is N.resize vs. arr.resize, were one fills with zeros, the other repeats. {{{ >>> N.resize([1,2,3], 6) [1 2 3 1 2 3] >>> a=N.array([1,2,3]);a.resize(6);a [1 2 3 0 0 0] }}} very confusing !! Please speak out ! Regards and thanks you all for the good/hard work ! -Sebastian Haase From david at ar.media.kyoto-u.ac.jp Tue Apr 15 03:56:51 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 15 Apr 2008 16:56:51 +0900 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com> <480459ED.4090308@esrf.fr> <3d375d730804150041j67356737ic17452c9f41cecf5@mail.gmail.com> Message-ID: <48045FC3.5040402@ar.media.kyoto-u.ac.jp> Sebastian Haase wrote: > > So, to summarize one more time: > """ > What used to be talked about as "1.1" will now become "1.2" --- > and "1.0.5" will be "1.1" > """ > > Did I get this right !? > > I'm curious to know _which_ were the changes that break the API. I > thought all additions like "axis=0" were made without breaking the > backwards compatibility. Otherwise it would (should !!!) have been > added as "axis=None". The new masked array has significant different behaviour than the old one, for once. This may be the biggest change for the formally known as 1.0.5. Personally, I think focusing on version number is wrong: what matters is API changes/compatibility, and how we handle it. Everything else is just nitpicking. > > Please keep in mind that there are a number of these "awkward" > "wrong-default" arguments. I was hoping these would be unified [e.g. > always same default, axis=None] very soon -- that is in 1.1. > Is this noted somewhere (e.g. is there a ticket for it ?). I think unifying API is really important, and the only way to do it is to keep track of those inconsistencies, so that we can decide what to do with them. cheers, David From millman at berkeley.edu Tue Apr 15 04:32:06 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 15 Apr 2008 01:32:06 -0700 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <48045FC3.5040402@ar.media.kyoto-u.ac.jp> References: <48043622.1090606@enthought.com> <480459ED.4090308@esrf.fr> <3d375d730804150041j67356737ic17452c9f41cecf5@mail.gmail.com> <48045FC3.5040402@ar.media.kyoto-u.ac.jp> Message-ID: On Tue, Apr 15, 2008 at 12:56 AM, David Cournapeau wrote: > > Please keep in mind that there are a number of these "awkward" > > "wrong-default" arguments. I was hoping these would be unified [e.g. > > always same default, axis=None] very soon -- that is in 1.1. > > Is this noted somewhere (e.g. is there a ticket for it ?). I think > unifying API is really important, and the only way to do it is to keep > track of those inconsistencies, so that we can decide what to do with them. I agree that unifying the API is really important. So I agree with Robert that we need to be extremely conservative in breaking the API. However, unifying the API is one area that I think could arguably (depending on the specific case) justify breaking the existing API. For instance, the 1.1.0 release, which will come out ASAP, introduces 'axis' support for numpy.median(). In 1.1, by default "axis=0" and then in the 1.2.0 release, which will come out at the end of summer, "axis=None" by default: http://scipy.org/scipy/numpy/ticket/558 Another example of a potential API break has been proposed for histogram: http://projects.scipy.org/pipermail/numpy-discussion/2008-April/032450.html http://scipy.org/scipy/numpy/ticket/605 If someone could put together a list of instances like these that we should discuss, it would be extremely useful. We could potentially try and put in warnings for functions that we want to modify. Again we need to be extremely careful here. We should avoid breaking our user's code as much as possible. However, when our current API is "broken" we should be open to the possibility that in may need to be fixed. Functions with anomalous interfaces seem like a good example of candidates that would be worth modifying. However, we need to consider whether the code is so widely used that modifying it would unacceptable. Furthermore, I would be less concerned about modifying the interface to functions, if we can also include an upgrade script that will automatically fix old code. So, for example, we could include an upgrade script with 1.2, which modifies calls to numpy.median() so that 'axis=0'. That way our users could upgrade their code to work with NumPy 1.2 by simply running the upgrade script. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From haase at msg.ucsf.edu Tue Apr 15 04:40:35 2008 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 15 Apr 2008 10:40:35 +0200 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <48045FC3.5040402@ar.media.kyoto-u.ac.jp> References: <48043622.1090606@enthought.com> <480459ED.4090308@esrf.fr> <3d375d730804150041j67356737ic17452c9f41cecf5@mail.gmail.com> <48045FC3.5040402@ar.media.kyoto-u.ac.jp> Message-ID: On Tue, Apr 15, 2008 at 9:56 AM, David Cournapeau wrote: > Sebastian Haase wrote: > > > > So, to summarize one more time: > > """ > > What used to be talked about as "1.1" will now become "1.2" --- > > and "1.0.5" will be "1.1" > > """ > > > > Did I get this right !? > > > > I'm curious to know _which_ were the changes that break the API. I > > thought all additions like "axis=0" were made without breaking the > > backwards compatibility. Otherwise it would (should !!!) have been > > added as "axis=None". > > The new masked array has significant different behaviour than the old > one, for once. This may be the biggest change for the formally known as > 1.0.5. Personally, I think focusing on version number is wrong: what > matters is API changes/compatibility, and how we handle it. Everything > else is just nitpicking. > How about releasing 1.0.5 without the new masked array, i.e. 1. put back the old masked array and release as 1.0.5 2. then release 1.1.0 with the new masked array 3. start working on 1.2 by unifying the arguments, "np.resize", and other things alike. --Sebastian From david at ar.media.kyoto-u.ac.jp Tue Apr 15 04:35:42 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 15 Apr 2008 17:35:42 +0900 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com> <480459ED.4090308@esrf.fr> <3d375d730804150041j67356737ic17452c9f41cecf5@mail.gmail.com> <48045FC3.5040402@ar.media.kyoto-u.ac.jp> Message-ID: <480468DE.5000802@ar.media.kyoto-u.ac.jp> Sebastian Haase wrote: > > > How about releasing 1.0.5 without the new masked array, > i.e. > 1. put back the old masked array and release as 1.0.5 Now that it is done, going back will be a lot of work. It would be great if someone were willing to do it, but who ? cheers, David From millman at berkeley.edu Tue Apr 15 04:59:22 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 15 Apr 2008 01:59:22 -0700 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <480468DE.5000802@ar.media.kyoto-u.ac.jp> References: <48043622.1090606@enthought.com> <480459ED.4090308@esrf.fr> <3d375d730804150041j67356737ic17452c9f41cecf5@mail.gmail.com> <48045FC3.5040402@ar.media.kyoto-u.ac.jp> <480468DE.5000802@ar.media.kyoto-u.ac.jp> Message-ID: On Tue, Apr 15, 2008 at 1:35 AM, David Cournapeau wrote: > Sebastian Haase wrote: > > How about releasing 1.0.5 without the new masked array, > > i.e. > > 1. put back the old masked array and release as 1.0.5 > > Now that it is done, going back will be a lot of work. It would be great > if someone were willing to do it, but who ? I am not convinced that having a 1.0.5 (sans the new masked array implementation) is very important. So I am personally not inclined to put any effort into making this happen. Even if someone else were to make the branch without the new ma, it would still require work on my part. I would need to do some testing, ask everyone else to try out the releases, get the binaries built, put together release notes, upload everything to sourceforge (if you haven't used sourceforge, trust me it is no fun), update the scipy wiki, update the PyPI site, and then send out the announcement. This process will divert my effort as well as whoever actually does the majority of the work. It will also divert the community as I ask several people to test and help me prepare the release. If there is some unknown regression in the current code, then we will need to make a 1.0.6 and a 1.1.1 release. Having to support the 1.0.x and 1.1.x series as we attempt to start developing 1.2 may be to onerous. We also need to start concentrating on getting SciPy 0.7 released. Moreover, I intend to put in more effort on this year's conference and I intend to ask others to help. And, of course, we all have our day jobs. So given the added effort that a 1.0.5 release would require, we should think very carefully about what having a 1.0.5 release would gain us before starting on that path. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From haase at msg.ucsf.edu Tue Apr 15 05:15:53 2008 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 15 Apr 2008 11:15:53 +0200 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com> <480459ED.4090308@esrf.fr> <3d375d730804150041j67356737ic17452c9f41cecf5@mail.gmail.com> <48045FC3.5040402@ar.media.kyoto-u.ac.jp> <480468DE.5000802@ar.media.kyoto-u.ac.jp> Message-ID: On Tue, Apr 15, 2008 at 10:59 AM, Jarrod Millman wrote: > On Tue, Apr 15, 2008 at 1:35 AM, David Cournapeau > > wrote: > > > Sebastian Haase wrote: > > > How about releasing 1.0.5 without the new masked array, > > > i.e. > > > 1. put back the old masked array and release as 1.0.5 > > > > Now that it is done, going back will be a lot of work. It would be great > > if someone were willing to do it, but who ? > > I am not convinced that having a 1.0.5 (sans the new masked array > implementation) is very important. So I am personally not inclined to > put any effort into making this happen. Even if someone else were to > make the branch without the new ma, it would still require work on my > part. I would need to do some testing, ask everyone else to try out > the releases, get the binaries built, put together release notes, > upload everything to sourceforge (if you haven't used sourceforge, > trust me it is no fun), update the scipy wiki, update the PyPI site, > and then send out the announcement. This process will divert my > effort as well as whoever actually does the majority of the work. It > will also divert the community as I ask several people to test and > help me prepare the release. > > If there is some unknown regression in the current code, then we will > need to make a 1.0.6 and a 1.1.1 release. Having to support the 1.0.x > and 1.1.x series as we attempt to start developing 1.2 may be to > onerous. We also need to start concentrating on getting SciPy 0.7 > released. Moreover, I intend to put in more effort on this year's > conference and I intend to ask others to help. And, of course, we all > have our day jobs. > > So given the added effort that a 1.0.5 release would require, we > should think very carefully about what having a 1.0.5 release would > gain us before starting on that path. > Also *I* would never install 1.0.5 if there was already a 1.1 release. I presume that the masked array user base is a sub-group anyway -- furthermore I think that those people would *benefit* from the new ma. So, let's just call the numpy 1.0.5 release the one that was never released, because we (you guys) were too enthusiastic getting numpy into what will now be known as numpy 1.1 ((It's getting late in California -- even though I'm actually in Berlin,Germany these days .... )) Cheers, -Sebastian Haase From gael.varoquaux at normalesup.org Tue Apr 15 05:41:43 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 15 Apr 2008 11:41:43 +0200 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <3d375d730804142338g2833ef75w84b01e7a99611f2@mail.gmail.com> References: <48043622.1090606@enthought.com> <3d375d730804142338g2833ef75w84b01e7a99611f2@mail.gmail.com> Message-ID: <20080415094143.GG9692@phare.normalesup.org> On Tue, Apr 15, 2008 at 01:38:45AM -0500, Robert Kern wrote: > > Given that the next release will be 1.1, I think it is reasonable to > > include a few additional API breaks. > -lots. I don't want to break API compatibility again no matter what > version number we bump to, 1.1, 2.0, or 24. It is simply not okay to > do this again and again. +1 with Robert. Ga?l From chanley at stsci.edu Tue Apr 15 09:01:22 2008 From: chanley at stsci.edu (Christopher Hanley) Date: Tue, 15 Apr 2008 09:01:22 -0400 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com> <480459ED.4090308@esrf.fr> <3d375d730804150041j67356737ic17452c9f41cecf5@mail.gmail.com> <48045FC3.5040402@ar.media.kyoto-u.ac.jp> Message-ID: <4804A722.9070904@stsci.edu> Sebastian Haase wrote: > How about releasing 1.0.5 without the new masked array, > i.e. > 1. put back the old masked array and release as 1.0.5 > 2. then release 1.1.0 with the new masked array > 3. start working on 1.2 by unifying the arguments, "np.resize", and > other things alike. > > --Sebastian We at STScI have already begun modifying our code to use the new masked array. It is a very expensive proposition for us to attempt to support two different APIs for numpy for any length of time. It is likely that we would force our customer base to 1.1 to avoid the added support costs. I also hope that there aren't any additional API breaks in the works. A stable API keeps our customers happy. API changes tend to make them cranky. Just my two cents. Chris -- Christopher Hanley Systems Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338 From p.e.creasey.00 at googlemail.com Tue Apr 15 09:35:04 2008 From: p.e.creasey.00 at googlemail.com (Peter Creasey) Date: Tue, 15 Apr 2008 14:35:04 +0100 Subject: [Numpy-discussion] Win32 installer: please test it Message-ID: <6be8b94a0804150635p666b4a46q488b012e6a152bb9@mail.gmail.com> Yes, I am one of those users with crashes in numpy 1.4. Your build seems to pass for me. For reference my machine is Windows XP, Intel Xeon 5140 ----------------- Numpy is installed in C:\Python25\lib\site-packages\numpy Numpy version 1.0.5.dev5008 Python version 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Int el)] Found 10/10 tests for numpy.core.defmatrix Found 3/3 tests for numpy.core.memmap Found 266/266 tests for numpy.core.multiarray Found 69/69 tests for numpy.core.numeric Found 31/31 tests for numpy.core.numerictypes Found 12/12 tests for numpy.core.records Found 7/7 tests for numpy.core.scalarmath Found 16/16 tests for numpy.core.umath Found 5/5 tests for numpy.ctypeslib Found 5/5 tests for numpy.distutils.misc_util Found 2/2 tests for numpy.fft.fftpack Found 3/3 tests for numpy.fft.helper Found 20/20 tests for numpy.lib._datasource Found 10/10 tests for numpy.lib.arraysetops Found 1/1 tests for numpy.lib.financial Found 0/0 tests for numpy.lib.format Found 48/48 tests for numpy.lib.function_base Found 5/5 tests for numpy.lib.getlimits Found 4/4 tests for numpy.lib.index_tricks Found 7/7 tests for numpy.lib.io Found 4/4 tests for numpy.lib.polynomial Found 49/49 tests for numpy.lib.shape_base Found 15/15 tests for numpy.lib.twodim_base Found 43/43 tests for numpy.lib.type_check Found 1/1 tests for numpy.lib.ufunclike Found 59/59 tests for numpy.linalg Found 92/92 tests for numpy.ma.core Found 14/14 tests for numpy.ma.extras Found 7/7 tests for numpy.random Found 0/0 tests for numpy.testing.utils Found 0/0 tests for __main__ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ....... ---------------------------------------------------------------------- Ran 887 tests in 0.890s OK On 14/04/2008, Bill Baxter wrote: > Seems to work here now, too! > > It doesn't tell you in an easy to see place what version of SSE it > decides to use. Do you think that's ok? (You can tell by looking at > the "details" at then end of installation, though. > > Is there some way to tell this info from inside NumPy itself? If so > then maybe it doesn't matter. > From aisaac at american.edu Tue Apr 15 09:42:06 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 15 Apr 2008 09:42:06 -0400 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com><480459ED.4090308@esrf.fr><3d375d730804150041j67356737ic17452c9f41cecf5@mail.gmail.com><48045FC3.5040402@ar.media.kyoto-u.ac.jp> Message-ID: On Tue, 15 Apr 2008, Jarrod Millman apparently wrote: > However, when our current API is "broken" we should be > open to the possibility that in may need to be fixed. > Functions with anomalous interfaces seem like a good > example of candidates that would be worth modifying. > However, we need to consider whether the code is so widely > used that modifying it would unacceptable. What about objects with anomalous interfaces? E.g.,:: >>> import numpy as np >>> x = np.mat('1 2;3 4') >>> x[0][0] matrix([[1, 2]]) >>> Fixing this has been under discussion for the 1.1 release for some time now. To my recollection, no users of matrices objected, and the abstract objections from non-users subsided. It would be very disappointing for the possibility of fixing this to go away. The fix appears to be a very small change in __getitem__: Cheers, Alan Isaac From doutriaux1 at llnl.gov Tue Apr 15 09:55:45 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Tue, 15 Apr 2008 06:55:45 -0700 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com><480459ED.4090308@esrf.fr><3d375d730804150041j67356737ic17452c9f41cecf5@mail.gmail.com><48045FC3.5040402@ar.media.kyoto-u.ac.jp> Message-ID: <4804B3E1.9050803@llnl.gov> As a developper for a community that's using massively the old numeric maksed arrays (MA), I'll just add my 2 cents worth... I think what's in the trunk right now (at least last week) is ok, the new ma, with the oldnumeric.ma This allow to still be backward compatible for a while and change our code slowly and even more slowly prepare our users... I'm strongly pushing to keep the numpy.oldnumeric.ma in the next release! Thanks, C. Alan G Isaac wrote: > On Tue, 15 Apr 2008, Jarrod Millman apparently wrote: > >> However, when our current API is "broken" we should be >> open to the possibility that in may need to be fixed. >> Functions with anomalous interfaces seem like a good >> example of candidates that would be worth modifying. >> However, we need to consider whether the code is so widely >> used that modifying it would unacceptable. >> > > What about objects with anomalous interfaces? E.g.,:: > > >>> import numpy as np > >>> x = np.mat('1 2;3 4') > >>> x[0][0] > matrix([[1, 2]]) > >>> > > Fixing this has been under discussion for the 1.1 release > for some time now. To my recollection, no users of matrices > objected, and the abstract objections from non-users > subsided. It would be very disappointing for the > possibility of fixing this to go away. The fix appears to > be a very small change in __getitem__: > > > Cheers, > Alan Isaac > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From stefan at sun.ac.za Tue Apr 15 10:01:38 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 15 Apr 2008 16:01:38 +0200 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com> <480459ED.4090308@esrf.fr> <3d375d730804150041j67356737ic17452c9f41cecf5@mail.gmail.com> <48045FC3.5040402@ar.media.kyoto-u.ac.jp> Message-ID: <9457e7c80804150701k5e643893he29a73ff2df0e975@mail.gmail.com> On 15/04/2008, Alan G Isaac wrote: > On Tue, 15 Apr 2008, Jarrod Millman apparently wrote: > Fixing this has been under discussion for the 1.1 release > for some time now. To my recollection, no users of matrices > objected, and the abstract objections from non-users > subsided. I'm still -1 on this. I think we should follow Tim's suggestion and implement RowVectors and ColumnVectors. Regards St?fan From aisaac at american.edu Tue Apr 15 10:33:27 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 15 Apr 2008 10:33:27 -0400 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <9457e7c80804150701k5e643893he29a73ff2df0e975@mail.gmail.com> References: <48043622.1090606@enthought.com><480459ED.4090308@esrf.fr><3d375d730804150041j67356737ic17452c9f41cecf5@mail.gmail.com><48045FC3.5040402@ar.media.kyoto-u.ac.jp> <9457e7c80804150701k5e643893he29a73ff2df0e975@mail.gmail.com> Message-ID: > On 15/04/2008, Alan G Isaac wrote: >> Fixing this has been under discussion for the 1.1 >> release >> for some time now. To my recollection, no users of matrices >> objected, and the abstract objections from non-users >> subsided. On Tue, 15 Apr 2008, St?fan van der Walt apparently wrote: > I'm still -1 on this. I think we should follow Tim's suggestion and > implement RowVectors and ColumnVectors. I thought the context of the discussion had become something like this: there is no reason for the matrix interface to deviate from the array interface except as needed to provide specific desired functionality. Essentially, - matrix multiplication - powers of square matrices - submatrix creation The proposal on the table is to remove an unneeded (and unwanted) deviation of the matrix API from the ndarray API. That is all. Adding row and column vectors is really orthogonal to the change under discussion. (And let me add, as a user of matrices, I am never feeling the need for these. Are you?) Cheers, Alan Isaac From lxander.m at gmail.com Tue Apr 15 10:54:05 2008 From: lxander.m at gmail.com (Alexander Michael) Date: Tue, 15 Apr 2008 10:54:05 -0400 Subject: [Numpy-discussion] Generically Creating Views of Equal Dimensions Message-ID: <525f23e80804150754p3e3d982awcdc29695a7417cb0@mail.gmail.com> Is there an already existing method to create views that add as many dimensions as required to bring a collection of arrays to the same dimensionality by adding the appropriate number of numpy.newaxis's to the ends? For example: In [1]: a = numpy.array([1, 2, 3, 4]) In [2]: b = numpy.array([[1,10], [1,10], [1,10], [1,10]]) In [3]: a_,b_ = dimensionalize(a,b) # returns a[:,numpy.newaxis],b In [4]: a_*b_ array([[ 1, 10], [ 2, 20], [ 3, 30], [ 4, 40]]) Or perhaps there is better way to do the same thing. Thanks, Alex From stefan at sun.ac.za Tue Apr 15 12:12:14 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 15 Apr 2008 18:12:14 +0200 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com> <480459ED.4090308@esrf.fr> <3d375d730804150041j67356737ic17452c9f41cecf5@mail.gmail.com> <48045FC3.5040402@ar.media.kyoto-u.ac.jp> <9457e7c80804150701k5e643893he29a73ff2df0e975@mail.gmail.com> Message-ID: <9457e7c80804150912s538b67aataeff9f2312e2d8fc@mail.gmail.com> On 15/04/2008, Alan G Isaac wrote: > I thought the context of the discussion had become something > like this: there is no reason for the matrix interface to > deviate from the array interface except as needed to provide > specific desired functionality. Essentially, > > - matrix multiplication > - powers of square matrices > - submatrix creation > > The proposal on the table is to remove an unneeded (and > unwanted) deviation of the matrix API from the ndarray API. > That is all. Unless my memory fails me (again), this would result in another deviation from numpy: that x[0] no longer returns the first row of any 2D array (matrix), especially when dealing with matrices of shape of (1,N). The whole issue occurs because a Matrix is not a proper container. An N-d array is a container of N-1-d arrays. This does not currently hold for matrices, but it *could* hold for matrices if we introduced RowVector and ColumnVector. Then we'd have, for arrays x is a (3,3) array -> x[0] is a (3,) array -> x[0][0] is a scalar and for matrices x is a (3,3) matrix -> x[0] is a (3,) ColumnVector or RowVector -> x[0][0] is a scalar > Adding row and column vectors is really orthogonal to the > change under discussion. (And let me add, as a user of > matrices, I am never feeling the need for these. Are you?) Orthogonal? -- so far, this is the only suggested solution that behaves in an entirely consistent way. That said, any solution is better than nothing; the current behaviour is confusing. My favorite quote in these threads so far has to be by Charles Harris: "All this for want of an operator ;)" Regards St?fan From wright at esrf.fr Tue Apr 15 12:31:35 2008 From: wright at esrf.fr (Jon Wright) Date: Tue, 15 Apr 2008 18:31:35 +0200 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <9457e7c80804150912s538b67aataeff9f2312e2d8fc@mail.gmail.com> References: <48043622.1090606@enthought.com> <480459ED.4090308@esrf.fr> <3d375d730804150041j67356737ic17452c9f41cecf5@mail.gmail.com> <48045FC3.5040402@ar.media.kyoto-u.ac.jp> <9457e7c80804150701k5e643893he29a73ff2df0e975@mail.gmail.com> <9457e7c80804150912s538b67aataeff9f2312e2d8fc@mail.gmail.com> Message-ID: <4804D867.7000006@esrf.fr> > On 15/04/2008, Alan G Isaac wrote: ... > The proposal on the table is to remove an unneeded (and > unwanted) deviation of the matrix API from the ndarray API. ... How about writing up the changes needed PEP style on the wiki? > The fix appears to be a very small change in __getitem__: ... together with an implementation and the results of running the current numpy testsuite. Or do you normally just use your own matrix object which inherits from the numpy one and then does what you prefer? Jon From wright at esrf.fr Tue Apr 15 14:22:35 2008 From: wright at esrf.fr (Jon Wright) Date: Tue, 15 Apr 2008 20:22:35 +0200 Subject: [Numpy-discussion] Py3K In-Reply-To: <480459ED.4090308@esrf.fr> References: <48043622.1090606@enthought.com> <480459ED.4090308@esrf.fr> Message-ID: <4804F26B.3060402@esrf.fr> I tried to build numpy from svn using python-3.0a4 from python.org and noticed there are some issues (print, except syntax, distutils magic). Has anyone been through this already? I actually want to start going through my own code, but I need numpy before I get started... Thanks, Jon From charlesr.harris at gmail.com Tue Apr 15 14:31:14 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 15 Apr 2008 12:31:14 -0600 Subject: [Numpy-discussion] Py3K In-Reply-To: <4804F26B.3060402@esrf.fr> References: <48043622.1090606@enthought.com> <480459ED.4090308@esrf.fr> <4804F26B.3060402@esrf.fr> Message-ID: On Tue, Apr 15, 2008 at 12:22 PM, Jon Wright wrote: > I tried to build numpy from svn using python-3.0a4 from python.org and > noticed there are some issues (print, except syntax, distutils magic). > Has anyone been through this already? I actually want to start going > through my own code, but I need numpy before I get started... > This looks like something that need to go on a list for numpy 1.2. Can you post a list of the specific problems that need to be addressed? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wright at esrf.fr Tue Apr 15 15:16:20 2008 From: wright at esrf.fr (Jon Wright) Date: Tue, 15 Apr 2008 21:16:20 +0200 Subject: [Numpy-discussion] Py3K In-Reply-To: References: <48043622.1090606@enthought.com> <480459ED.4090308@esrf.fr> <4804F26B.3060402@esrf.fr> Message-ID: <4804FF04.3040604@esrf.fr> Charles R Harris wrote: > Jon Wright wrote: > > I tried to build numpy from svn using python-3.0a4 from python.org > and > noticed there are some issues (print, except syntax, distutils magic). > Has anyone been through this already? I actually want to start going > through my own code, but I need numpy before I get started... > > > This looks like something that need to go on a list for numpy 1.2. Can > you post a list of the specific problems that need to be addressed? OK, will try to actually follow the instructions about going via 2.6 (alpha is out) and 3k warnings and then get back to you. Please save me now if you know the answers already... Jon From bsouthey at gmail.com Tue Apr 15 15:21:07 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Tue, 15 Apr 2008 14:21:07 -0500 Subject: [Numpy-discussion] Py3K In-Reply-To: <4804FF04.3040604@esrf.fr> References: <48043622.1090606@enthought.com> <480459ED.4090308@esrf.fr> <4804F26B.3060402@esrf.fr> <4804FF04.3040604@esrf.fr> Message-ID: Hi, You probably should start with the 2to3.py tool (see link at http://www.python.org/download/releases/3.0/). Bruce On Tue, Apr 15, 2008 at 2:16 PM, Jon Wright wrote: > Charles R Harris wrote: > > > Jon Wright wrote: > > > > I tried to build numpy from svn using python-3.0a4 from python.org > > and > > > noticed there are some issues (print, except syntax, distutils magic). > > Has anyone been through this already? I actually want to start going > > through my own code, but I need numpy before I get started... > > > > > > This looks like something that need to go on a list for numpy 1.2. Can > > you post a list of the specific problems that need to be addressed? > > OK, will try to actually follow the instructions about going via 2.6 > (alpha is out) and 3k warnings and then get back to you. Please save me > now if you know the answers already... > > > > Jon > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From peridot.faceted at gmail.com Tue Apr 15 15:29:16 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Tue, 15 Apr 2008 15:29:16 -0400 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <4804D867.7000006@esrf.fr> References: <48043622.1090606@enthought.com> <3d375d730804150041j67356737ic17452c9f41cecf5@mail.gmail.com> <48045FC3.5040402@ar.media.kyoto-u.ac.jp> <9457e7c80804150701k5e643893he29a73ff2df0e975@mail.gmail.com> <9457e7c80804150912s538b67aataeff9f2312e2d8fc@mail.gmail.com> <4804D867.7000006@esrf.fr> Message-ID: On 15/04/2008, Jon Wright wrote: > > On 15/04/2008, Alan G Isaac wrote: > > ... > > > The proposal on the table is to remove an unneeded (and > > unwanted) deviation of the matrix API from the ndarray API. > > ... > > How about writing up the changes needed PEP style on the wiki? +1 This discussion risks going around in circles. Write up your proposed solutions, with example code, in PEP style, here: http://www.scipy.org/ProposedEnhancements Anne From robert.kern at gmail.com Tue Apr 15 15:38:42 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 15 Apr 2008 14:38:42 -0500 Subject: [Numpy-discussion] Generically Creating Views of Equal Dimensions In-Reply-To: <525f23e80804150754p3e3d982awcdc29695a7417cb0@mail.gmail.com> References: <525f23e80804150754p3e3d982awcdc29695a7417cb0@mail.gmail.com> Message-ID: <3d375d730804151238m5ecd2991gcda2f40c93916070@mail.gmail.com> On Tue, Apr 15, 2008 at 9:54 AM, Alexander Michael wrote: > Is there an already existing method to create views that add as many > dimensions as required to bring a collection of arrays to the same > dimensionality by adding the appropriate number of numpy.newaxis's to > the ends? For example: The usual broadcasting rule goes the other way; newaxis's are *prepended* to the beginning of the shape. I wouldn't put a function into numpy to do something the opposite of that convention and risk confusing people. However, if you would like a utility function for your own code: from numpy import newaxis def dimensionalize(a, b): """ Try to make the ranks of two arrays compatible for broadcasting by *appending* new axes. This is the opposite of the usual broadcasting convention which *prepends* new axes. """ ranka = len(a.shape) rankb = len(b.shape) if ranka > rankb: b = b[(Ellipsis,)+(newaxis,)*(ranka-rankb)] elif rankb > ranka: a = a[(Ellipsis,)+(newaxis,)*(rankb-ranka)] return a, b -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Tue Apr 15 16:23:12 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 15 Apr 2008 14:23:12 -0600 Subject: [Numpy-discussion] Py3K In-Reply-To: References: <48043622.1090606@enthought.com> <480459ED.4090308@esrf.fr> <4804F26B.3060402@esrf.fr> <4804FF04.3040604@esrf.fr> Message-ID: On Tue, Apr 15, 2008 at 1:21 PM, Bruce Southey wrote: > Hi, > You probably should start with the 2to3.py tool (see link at > http://www.python.org/download/releases/3.0/). > Looks like a lot of the changes aren't at all backward compatible. The new style print, for instance, doesn't work with the older versions of python. So it looks like like we will have to fork the trunk and also decide on when to release a Python3.0 version and how long to maintain support for the earlier versions. I think the best thing to do is release a numpy version, say 1.2, as bug free as we can and with few API changes, put it in maintenance mode, and move on to 3.0 with a release that is a simple adaption without API changes. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Apr 15 16:27:39 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 15 Apr 2008 14:27:39 -0600 Subject: [Numpy-discussion] Py3K In-Reply-To: References: <48043622.1090606@enthought.com> <480459ED.4090308@esrf.fr> <4804F26B.3060402@esrf.fr> <4804FF04.3040604@esrf.fr> Message-ID: On Tue, Apr 15, 2008 at 2:23 PM, Charles R Harris wrote: > > > On Tue, Apr 15, 2008 at 1:21 PM, Bruce Southey wrote: > > > Hi, > > You probably should start with the 2to3.py tool (see link at > > http://www.python.org/download/releases/3.0/). > > > > Looks like a lot of the changes aren't at all backward compatible. The new > style print, for instance, doesn't work with the older versions of python. > So it looks like like we will have to fork the trunk and also decide on when > to release a Python3.0 version and how long to maintain support for the > earlier versions. I think the best thing to do is release a numpy version, > say 1.2, as bug free as we can and with few API changes, put it in > maintenance mode, and move on to 3.0 with a release that is a simple > adaption without API changes. > Oh, and making the transition will be made a lot easier by having a complete set of tests. Getting the tests and documentation into good state might be the best focus for our 1.2 effort. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Tue Apr 15 16:35:37 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 15 Apr 2008 15:35:37 -0500 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com> <480459ED.4090308@esrf.fr> <3d375d730804150041j67356737ic17452c9f41cecf5@mail.gmail.com> Message-ID: <48051199.4020202@enthought.com> Sebastian Haase wrote: > On Tue, Apr 15, 2008 at 9:41 AM, Robert Kern wrote: > >> On Tue, Apr 15, 2008 at 2:31 AM, Jon Wright wrote: >> > Alan G Isaac wrote: >> > > >> > > Will matrix behavior change in 1.1, as discussed from time >> > > to time? Perhaps it just takes a very small change in __getitem__: >> > > >> > >> > Quoting from: >> > http://mail.python.org/pipermail/python-dev/2008-March/077723.html >> > """ >> > Executive summary: Python 2.6 and 3.0 finals are planned for September >> > 3, 2008. >> > """ >> > >> > In the event that there really is a valid reason to change the API (much >> > like serial killers have their own valid reasons for their crimes...). >> > Why not finalise that perfect API for the python3.0 build, when everyone >> > is expecting code to change for 3K? Name them numpy3.x.x to match the >> > python major version. >> >> The Python community has been begging package owners *not* to do this. >> That would impose yet another hurdle to the adoption of Python 3.0. >> >> >> > > So, to summarize one more time: > """ > What used to be talked about as "1.1" will now become "1.2" --- > and "1.0.5" will be "1.1" > """ > > Did I get this right !? > > I'm curious to know _which_ were the changes that break the API. I > thought all additions like "axis=0" were made without breaking the > backwards compatibility. Otherwise it would (should !!!) have been > added as "axis=None". > > Please keep in mind that there are a number of these "awkward" > "wrong-default" arguments. I was hoping these would be unified [e.g. > always same default, axis=None] very soon -- that is in 1.1. > Also on my list is N.resize vs. arr.resize, were one fills with > zeros, the other repeats. > This is a carry over from Numeric. I should have been more proactive here and forced the change, but I think one of the internal functions depended on the old behavior and I punted. So, we should allow both behaviors but I agree have them do the same thing by default. -Travis From lxander.m at gmail.com Tue Apr 15 16:45:01 2008 From: lxander.m at gmail.com (Alexander Michael) Date: Tue, 15 Apr 2008 16:45:01 -0400 Subject: [Numpy-discussion] Generically Creating Views of Equal Dimensions In-Reply-To: <3d375d730804151238m5ecd2991gcda2f40c93916070@mail.gmail.com> References: <525f23e80804150754p3e3d982awcdc29695a7417cb0@mail.gmail.com> <3d375d730804151238m5ecd2991gcda2f40c93916070@mail.gmail.com> Message-ID: <525f23e80804151345s5f79cb37kcdfd32af0fd002b4@mail.gmail.com> On Tue, Apr 15, 2008 at 3:38 PM, Robert Kern wrote: > On Tue, Apr 15, 2008 at 9:54 AM, Alexander Michael wrote: > > Is there an already existing method to create views that add as many > > dimensions as required to bring a collection of arrays to the same > > dimensionality by adding the appropriate number of numpy.newaxis's to > > the ends? For example: > > The usual broadcasting rule goes the other way; newaxis's are > *prepended* to the beginning of the shape. I wouldn't put a function > into numpy to do something the opposite of that convention and risk > confusing people. Ah, I see. If my convention was the transpose of what it currently is, I wouldn't need to do anything. Thanks! Alex From fperez.net at gmail.com Tue Apr 15 18:53:07 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 15 Apr 2008 15:53:07 -0700 Subject: [Numpy-discussion] Py3K In-Reply-To: References: <48043622.1090606@enthought.com> <480459ED.4090308@esrf.fr> <4804F26B.3060402@esrf.fr> <4804FF04.3040604@esrf.fr> Message-ID: On Tue, Apr 15, 2008 at 1:27 PM, Charles R Harris wrote: > Oh, and making the transition will be made a lot easier by having a complete > set of tests. Getting the tests and documentation into good state might be > the best focus for our 1.2 effort. +1 Bonus points for whoever adds coverage reporting to our test suite, so that we can get at least basic metrics of coverage by: - docstrings - doctests - plain tests Even if imperfect, having some kind of metric to trac progress on this front would be useful, I think. Nose does have a coverage plugin, but unfortunately it seems that coverage and doctests don't totally play nice in nose: http://www.somethingaboutorange.com/mrl/projects/nose/doc/plugin_doctests.html (see comment in code). I don't know enough about the details of this to say much more... Cheers, f From eads at soe.ucsc.edu Tue Apr 15 20:01:40 2008 From: eads at soe.ucsc.edu (Damian Eads) Date: Tue, 15 Apr 2008 17:01:40 -0700 Subject: [Numpy-discussion] Win32 installer: please test it In-Reply-To: <4802AE40.5090108@ar.media.kyoto-u.ac.jp> References: <4802AE40.5090108@ar.media.kyoto-u.ac.jp> Message-ID: <480541E4.303@soe.ucsc.edu> David Cournapeau wrote: > Jarrod Millman wrote: >> Hello, >> >> David Cournapeau has prepared a new win32 installer, which is aimed at >> solving the recurring problem of non working atlas on different sets >> of CPU. This installer simply checks which cpu you have, and installs >> the appropriate numpy accordingly (without atlas if your cpu is not >> supported). Windows users, please test the installer, and report >> problems with it; we hope to use this scheme for all numpy and scipy >> installers in the future. > > Sorry for being insistent, but I would really like to get rid of those > windows/sse* problems once for all for numpy/scipy, and this means > having people to test the installer. > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion Sorry for getting back to you so late. I was out of range of wireless networks for a day and a half. Shown below are my /proc/cpu-info and result of running numpy.test(). SSE and SSE2 are supported but SSE3 is not. Nice work David! Damian ------------------------------------------- processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 13 model name : Intel(R) Pentium(R) M processor 1.60GHz stepping : 6 cpu MHz : 1600.000 cache size : 2048 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr mce cx8 mtrr pge mca cmov pat clflush dts acpi mmx fxsr sse sse2 ss tm pbe up bts est tm2 bogomips : 3198.55 clflush size : 64 Numpy version 1.0.4 Python version 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Int el)] Found 10/10 tests for numpy.core.defmatrix Found 36/36 tests for numpy.core.ma Found 223/223 tests for numpy.core.multiarray Found 65/65 tests for numpy.core.numeric Found 31/31 tests for numpy.core.numerictypes Found 12/12 tests for numpy.core.records Found 6/6 tests for numpy.core.scalarmath Found 14/14 tests for numpy.core.umath Found 4/4 tests for numpy.ctypeslib Found 5/5 tests for numpy.distutils.misc_util Found 1/1 tests for numpy.fft.fftpack Found 3/3 tests for numpy.fft.helper Found 9/9 tests for numpy.lib.arraysetops Found 46/46 tests for numpy.lib.function_base Found 5/5 tests for numpy.lib.getlimits Found 4/4 tests for numpy.lib.index_tricks Found 3/3 tests for numpy.lib.polynomial Found 49/49 tests for numpy.lib.shape_base Found 15/15 tests for numpy.lib.twodim_base Found 43/43 tests for numpy.lib.type_check Found 1/1 tests for numpy.lib.ufunclike Found 40/40 tests for numpy.linalg Found 2/2 tests for numpy.random Found 0/0 tests for __main__ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ....................................................................Warning: inv alid value encountered in absolute Warning: invalid value encountered in absolute Warning: invalid value encountered in less_equal ................................................................................ ................................................................................ ................................................................................ .......................................................... ---------------------------------------------------------------------- Ran 686 tests in 1.250s OK ------------------------ From aisaac at american.edu Wed Apr 16 02:08:29 2008 From: aisaac at american.edu (Alan G Isaac) Date: Wed, 16 Apr 2008 02:08:29 -0400 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com><3d375d730804150041j67356737ic17452c9f41cecf5@mail.gmail.com><48045FC3.5040402@ar.media.kyoto-u.ac.jp><9457e7c80804150701k5e643893he29a73ff2df0e975@mail.gmail.com><9457e7c80804150912s538b67aataeff9f2312e2d8fc@mail.gmail.com><4804D867.7000006@esrf.fr> Message-ID: On Tue, 15 Apr 2008, Anne Archibald apparently wrote: > This discussion risks going around in circles. Write up > your proposed solutions, with example code, in PEP style, > here: > http://www.scipy.org/ProposedEnhancements Done: http://www.scipy.org/MatrixIndexing I am a NumPy user, not a NumPy developer, but I believe the example code appropriately produces the desired behavior change. I believe this is really the last opportunity for this change. It is clear that it is already almost impossible to make API changes, even those that like this one are unlikely to affect much real code. So if it is a good idea, as I argue, its time has come. Cheers, Alan Isaac From aisaac at american.edu Wed Apr 16 02:08:34 2008 From: aisaac at american.edu (Alan G Isaac) Date: Wed, 16 Apr 2008 02:08:34 -0400 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <9457e7c80804150912s538b67aataeff9f2312e2d8fc@mail.gmail.com> References: <48043622.1090606@enthought.com><480459ED.4090308@esrf.fr><3d375d730804150041j67356737ic17452c9f41cecf5@mail.gmail.com><48045FC3.5040402@ar.media.kyoto-u.ac.jp><9457e7c80804150701k5e643893he29a73ff2df0e975@mail.gmail.com> <9457e7c80804150912s538b67aataeff9f2312e2d8fc@mail.gmail.com> Message-ID: > On 15/04/2008, Alan G Isaac wrote: >> I thought the context of the discussion had become something >> like this: there is no reason for the matrix interface to >> deviate from the array interface except as needed to provide >> specific desired functionality. Essentially, >> - matrix multiplication >> - powers of square matrices >> - submatrix creation >> The proposal on the table is to remove an unneeded (and >> unwanted) deviation of the matrix API from the ndarray API. >> That is all. On Tue, 15 Apr 2008, St?fan van der Walt apparently wrote: > Unless my memory fails me (again), this would result in > another deviation from numpy: that x[0] no longer returns > the first row of any 2D array (matrix), especially when > dealing with matrices of shape of (1,N). Either I do not understand you, or this is not correct, or... Anyway, for matrix x, x[0] should be the same as the current x.A[0] > The whole issue occurs because a Matrix is not a proper > container. Right. And *that* is the case because of the attempt to treat matrices as containers of matrices instead of as containers of 1d arrays. I can see no real advantage to having matrices be containers of row vectors instead. A row vector would just be a matrix (i.e., essentially 2d) that allowed 1d style indexing. Again I ask, have you ever needed such a thing? > My favorite quote in these threads so far has to be by Charles Harris: > "All this for want of an operator ;)" I agree this is a good perspective. (But *two* operators: * and **.) It implies: there are deviations from array behavior that are buying us nothing. This is my point. Cheers, Alan From stefan at sun.ac.za Wed Apr 16 04:35:02 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 16 Apr 2008 10:35:02 +0200 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com> <3d375d730804150041j67356737ic17452c9f41cecf5@mail.gmail.com> <48045FC3.5040402@ar.media.kyoto-u.ac.jp> <9457e7c80804150701k5e643893he29a73ff2df0e975@mail.gmail.com> <9457e7c80804150912s538b67aataeff9f2312e2d8fc@mail.gmail.com> Message-ID: <9457e7c80804160135v660ab494gc30e251dfedc2eec@mail.gmail.com> On 16/04/2008, Alan G Isaac wrote: > > The whole issue occurs because a Matrix is not a proper > > container. > > > Right. And *that* is the case because of the attempt to > treat matrices as containers of matrices instead of as > containers of 1d arrays. > > I can see no real advantage to having matrices be containers > of row vectors instead. A row vector would just be a matrix > (i.e., essentially 2d) that allowed 1d style indexing. > Again I ask, have you ever needed such a thing? In the Matrix world, there is no such thing as an (N,) array -- which is why the current Matrix object behaves so strangely, and always returns a result with ndim > 1. Your proposal suggests that a Matrix be a container of arrays, but it does not address the slicing of column vectors, i.e. x[0] x[0,:] x[:,0] would then behave in completely different ways, whereas with ndarrays they don't. If you modified the proposal to say "any slicing operation that returns a (1,N) or (N,1) matrix should now return an (N,) ndarray", it would be more consistent, but then, wouldn't that break some other expected behaviour? I thought one of the powerful features of a matrix is that it preserves the orientation of a vector slice? The only way I've seen so far to have consistent slicing, is to have an N-dim Matrix object be a container of N-1 dim Matrices, up to the 2-D case, in which it becomes a container of Vectors which, in turn, contain scalars. > I agree this is a good perspective. (But *two* operators: * and **.) > It implies: there are deviations from array behavior that > are buying us nothing. This is my point. Yup, and fortunately we now have both those implemented for arrays as well :) Regards St?fan From peridot.faceted at gmail.com Wed Apr 16 05:14:42 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Wed, 16 Apr 2008 05:14:42 -0400 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <9457e7c80804160135v660ab494gc30e251dfedc2eec@mail.gmail.com> References: <48043622.1090606@enthought.com> <48045FC3.5040402@ar.media.kyoto-u.ac.jp> <9457e7c80804150701k5e643893he29a73ff2df0e975@mail.gmail.com> <9457e7c80804150912s538b67aataeff9f2312e2d8fc@mail.gmail.com> <9457e7c80804160135v660ab494gc30e251dfedc2eec@mail.gmail.com> Message-ID: On 16/04/2008, St?fan van der Walt wrote: > On 16/04/2008, Alan G Isaac wrote: > > > The whole issue occurs because a Matrix is not a proper > > > container. > > > > > > Right. And *that* is the case because of the attempt to > > treat matrices as containers of matrices instead of as > > containers of 1d arrays. > > > > I can see no real advantage to having matrices be containers > > of row vectors instead. A row vector would just be a matrix > > (i.e., essentially 2d) that allowed 1d style indexing. > > Again I ask, have you ever needed such a thing? > > > In the Matrix world, there is no such thing as an (N,) array -- which > is why the current Matrix object behaves so strangely, and always > returns a result with ndim > 1. > > Your proposal suggests that a Matrix be a container of arrays, but it > does not address the slicing of column vectors, i.e. > > x[0] > x[0,:] > x[:,0] > > would then behave in completely different ways, whereas with ndarrays > they don't. > > If you modified the proposal to say "any slicing operation that > returns a (1,N) or (N,1) matrix should now return an (N,) ndarray", it > would be more consistent, but then, wouldn't that break some other > expected behaviour? I thought one of the powerful features of a > matrix is that it preserves the orientation of a vector slice? > > The only way I've seen so far to have consistent slicing, is to have > an N-dim Matrix object be a container of N-1 dim Matrices, up to the > 2-D case, in which it becomes a container of Vectors which, in turn, > contain scalars. I don't think of arrays as containers of anything but scalars, so I find this whole argument from intuition extremely strange. My (draconian) suggestion would be to simply raise an exception when a matrix is indexed with a scalar. They're inherently two-dimensional; if you want a submatrix you should provide both indices (possibly including a ":"). If you actually want a subarray, as with an array, use ".A". That said, I don't actually use matrices, so I don't get a vote. Anne From stefan at sun.ac.za Wed Apr 16 05:45:02 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 16 Apr 2008 11:45:02 +0200 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com> <48045FC3.5040402@ar.media.kyoto-u.ac.jp> <9457e7c80804150701k5e643893he29a73ff2df0e975@mail.gmail.com> <9457e7c80804150912s538b67aataeff9f2312e2d8fc@mail.gmail.com> <9457e7c80804160135v660ab494gc30e251dfedc2eec@mail.gmail.com> Message-ID: <9457e7c80804160245v1f2fb06dla9cece7fa0afddd3@mail.gmail.com> On 16/04/2008, Anne Archibald wrote: > I don't think of arrays as containers of anything but scalars, so I > find this whole argument from intuition extremely strange. I see now for the first time that Matrices can't have dims > 2. Grim. I do think that ColumnVector and RowVector could be useful in general; some statements read more clearly, e.g. (for x an (N,)-array) ColumnVector(x) instead of np.c_[x] # turn x into a column vector And, while np.dot(x,x) is valid, RowVector(x) * ColumnVector(x) is clearer than x = np.dot(np.r_[x], np.c_[x]) (which is a pattern I'd expect to find amongst linear algebra users) The last expression also yields array([14]) instead of 14! > My (draconian) suggestion would be to simply raise an exception when a > matrix is indexed with a scalar. They're inherently two-dimensional; > if you want a submatrix you should provide both indices (possibly > including a ":"). If you actually want a subarray, as with an array, > use ".A". Your idea isn't that far out -- but I think we can provide the expected (ndarray-like) behaviour in this case. > That said, I don't actually use matrices, so I don't get a vote. Apparently, neither do I :) But I do get to modify http://www.scipy.org/MatrixIndexing I also changed ProposedEnhancements into a Category page, that automatically accumulates all WikiPages tagged with ProposedEnhancements. Regards St?fan From timmichelsen at gmx-topmail.de Wed Apr 16 07:04:57 2008 From: timmichelsen at gmx-topmail.de (Tim Michelsen) Date: Wed, 16 Apr 2008 11:04:57 +0000 (UTC) Subject: [Numpy-discussion] Py3K References: <48043622.1090606@enthought.com> <480459ED.4090308@esrf.fr> <4804F26B.3060402@esrf.fr> Message-ID: > This looks like something that need to go on a list for numpy 1.2. Can you post a list of the specific problems that need to be addressed?Chuck Maybe this post can give some hints: All Things Pythonic Python 3000 and You by Guido van Rossum March 17, 2008 Summary I've posted the slides from my PyCon 2008 keynote on python.org. Here are the URLs, plus a warning against the temptation of changing your APIs at the same time as porting to Py3k. This is really important! http://www.artima.com/weblogs/viewpost.jsp?thread=227041 From gnurser at googlemail.com Wed Apr 16 08:44:29 2008 From: gnurser at googlemail.com (George Nurser) Date: Wed, 16 Apr 2008 13:44:29 +0100 Subject: [Numpy-discussion] numpy setup.py too restrictive, prevents use of fblas with cblas Message-ID: <1d1e6ea70804160544v7f0a42d2k20c39f4da5bd0332@mail.gmail.com> Apologies for coming out of the woodwork so late here.. For blas/atlas etc in numpy & scipy on an opteron I use the AMD libraries (which only have fblas) together with cblas. This works very well. Current-ish SVN (v4779) in line 295-296 of numpy/core/setup.py has: if ('NO_ATLAS_INFO',1) in blas_info.get('define_macros',[]): return None # dotblas needs ATLAS, Fortran compiled blas will not be sufficient To get my AMD fblas/cblas approach to work, I have to comment out these two lines. Can this restriction preventing use of AMD fblas be removed for v1.1? Should I file a bug here? Regards, George Nurser. From stefan at sun.ac.za Wed Apr 16 08:56:04 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 16 Apr 2008 14:56:04 +0200 Subject: [Numpy-discussion] Py3K In-Reply-To: References: <48043622.1090606@enthought.com> <480459ED.4090308@esrf.fr> <4804F26B.3060402@esrf.fr> <4804FF04.3040604@esrf.fr> Message-ID: <9457e7c80804160556q2003e8b6k2e3c3049a4d2e62d@mail.gmail.com> On 16/04/2008, Fernando Perez wrote: > On Tue, Apr 15, 2008 at 1:27 PM, Charles R Harris > > Oh, and making the transition will be made a lot easier by having a complete > > set of tests. Getting the tests and documentation into good state might be > > the best focus for our 1.2 effort. > > > +1 > > Bonus points for whoever adds coverage reporting to our test suite, so > that we can get at least basic metrics of coverage by: > > - docstrings > - doctests > - plain tests I'd love to see that. In the meantime, here is a figleaf coverage report: http://mentat.za.net/numpy/coverage/ Regards St?fan From bsouthey at gmail.com Wed Apr 16 09:47:41 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Wed, 16 Apr 2008 08:47:41 -0500 Subject: [Numpy-discussion] Py3K In-Reply-To: References: <48043622.1090606@enthought.com> <480459ED.4090308@esrf.fr> <4804F26B.3060402@esrf.fr> Message-ID: <4806037D.6040502@gmail.com> Tim Michelsen wrote: >> This looks like something that need to go on a list for numpy 1.2. Can you >> > post a list of the specific problems that need to be addressed?Chuck > Maybe this post can give some hints: > All Things Pythonic > Python 3000 and You > by Guido van Rossum > March 17, 2008 > > Summary > I've posted the slides from my PyCon 2008 keynote on python.org. Here are > the URLs, plus a warning against the temptation of changing your APIs at the > same time as porting to Py3k. This is really important! > http://www.artima.com/weblogs/viewpost.jsp?thread=227041 > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > Hi, As Guido highlights in his sides (linked from his blog), the Python 2.6 series has a flag (-3) to help porting. So I would suggest that Python 2.6 compatibility would be an first intermediate step . I have the suspicion that some of the 'small' changes are going to be very important and hard to track down without checking that is works in Python 2.6. Bruce From aisaac at american.edu Wed Apr 16 10:06:03 2008 From: aisaac at american.edu (Alan G Isaac) Date: Wed, 16 Apr 2008 10:06:03 -0400 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <9457e7c80804160135v660ab494gc30e251dfedc2eec@mail.gmail.com> References: <48043622.1090606@enthought.com><3d375d730804150041j67356737ic17452c9f41cecf5@mail.gmail.com><48045FC3.5040402@ar.media.kyoto-u.ac.jp><9457e7c80804150701k5e643893he29a73ff2df0e975@mail.gmail.com><9457e7c80804150912s538b67aataeff9f2312e2d8fc@mail.gmail.com> <9457e7c80804160135v660ab494gc30e251dfedc2eec@mail.gmail.com> Message-ID: On Wed, 16 Apr 2008, St?fan van der Walt apparently wrote: > Your proposal suggests that a Matrix be a container of arrays, but it > does not address the slicing of column vectors, i.e. > x[0] > x[0,:] > x[:,0] The only thing that changes is the handling of scalar indices (and thus of iteration). The other two cases remain unchanged. This should be clear from the proposal: http://www.scipy.org/MatrixIndexing The rule is: to get a submatrix, use multiple indices. As Anne has argued, this is natural. Cheers, Alan Isaac From aisaac at american.edu Wed Apr 16 10:06:05 2008 From: aisaac at american.edu (Alan G Isaac) Date: Wed, 16 Apr 2008 10:06:05 -0400 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com><48045FC3.5040402@ar.media.kyoto-u.ac.jp><9457e7c80804150701k5e643893he29a73ff2df0e975@mail.gmail.com><9457e7c80804150912s538b67aataeff9f2312e2d8fc@mail.gmail.com><9457e7c80804160135v660ab494gc30e251dfedc2eec@mail.gmail.com> Message-ID: On Wed, 16 Apr 2008, Anne Archibald apparently wrote: > My (draconian) suggestion would be to simply raise an > exception when a matrix is indexed with a scalar. This has been suggested before. But then, why? Again, this imposes a deviation from the behavior of arrays that provides no gain in functionality. My proposal is: deviate from array behavior only when it gives a clear gain in functionality. > They're inherently two-dimensional; Of course. But the question is what follows from this as an implication for the behavior of 1d indexing and iteration. The answer is: nothing! We get to do whatever is most useful! > if you want a submatrix you should provide both indices > (possibly including a ":"). Yes. We exactly agree on this. Please persuade Stefan. > If you actually want a subarray, as with an array, use > ".A". Again we agree on this. But none of this answers the question: what should you get when you iterate over a matrix. Stefan seems to agree with me to this extent: we should get its "rows". This disagreement is over what that means: are these submatrices, a new vector object, or 1d arrays? I claim that if you use matrices, you will find it natural for these to be 1d arrays. > That said, I don't actually use matrices, so I don't get a vote. As far as I know, no objections have been raised by users of matrices. In this sense, the objections have all been "abstract": they do not reflect experience with the implied convenience or inconvenience or particular behaviors. Cheers, Alan Isaac From ondrej at certik.cz Wed Apr 16 10:06:29 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 16 Apr 2008 16:06:29 +0200 Subject: [Numpy-discussion] Py3K In-Reply-To: <9457e7c80804160556q2003e8b6k2e3c3049a4d2e62d@mail.gmail.com> References: <48043622.1090606@enthought.com> <480459ED.4090308@esrf.fr> <4804F26B.3060402@esrf.fr> <4804FF04.3040604@esrf.fr> <9457e7c80804160556q2003e8b6k2e3c3049a4d2e62d@mail.gmail.com> Message-ID: <85b5c3130804160706k33794d8du6b98a798dbfe275@mail.gmail.com> On Wed, Apr 16, 2008 at 2:56 PM, St?fan van der Walt wrote: > On 16/04/2008, Fernando Perez wrote: > > On Tue, Apr 15, 2008 at 1:27 PM, Charles R Harris > > > > Oh, and making the transition will be made a lot easier by having a complete > > > set of tests. Getting the tests and documentation into good state might be > > > the best focus for our 1.2 effort. > > > > > > +1 > > > > Bonus points for whoever adds coverage reporting to our test suite, so > > that we can get at least basic metrics of coverage by: > > > > - docstrings > > - doctests > > - plain tests > > I'd love to see that. In the meantime, here is a figleaf coverage report: > > http://mentat.za.net/numpy/coverage/ Wow, figleaf is really cool. I didn't know about this tool. Ondrej From oliphant at enthought.com Wed Apr 16 10:11:02 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Wed, 16 Apr 2008 09:11:02 -0500 Subject: [Numpy-discussion] numpy setup.py too restrictive, prevents use of fblas with cblas In-Reply-To: <1d1e6ea70804160544v7f0a42d2k20c39f4da5bd0332@mail.gmail.com> References: <1d1e6ea70804160544v7f0a42d2k20c39f4da5bd0332@mail.gmail.com> Message-ID: <480608F6.80904@enthought.com> George Nurser wrote: > Apologies for coming out of the woodwork so late here.. > > For blas/atlas etc in numpy & scipy on an opteron I use the AMD > libraries (which only have fblas) together with cblas. This works very > well. > > Current-ish SVN (v4779) in line 295-296 of numpy/core/setup.py has: > > if ('NO_ATLAS_INFO',1) in blas_info.get('define_macros',[]): > return None # dotblas needs ATLAS, Fortran compiled > blas will not be sufficient > > To get my AMD fblas/cblas approach to work, I have to comment out > these two lines. > Can this restriction preventing use of AMD fblas be removed for v1.1? > > Should I file a bug here? > Yes, you should put a ticket together. The more information you can provide, the more likely someone will act on the ticket. Some kind of info on why you had to comment out those two lines would be helpful. -Travis From bsouthey at gmail.com Wed Apr 16 10:28:40 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Wed, 16 Apr 2008 09:28:40 -0500 Subject: [Numpy-discussion] Win32 installer: please test it In-Reply-To: <480541E4.303@soe.ucsc.edu> References: <4802AE40.5090108@ar.media.kyoto-u.ac.jp> <480541E4.303@soe.ucsc.edu> Message-ID: Hi, Excellent job! I got NO errors (including Vista), all tests passed . I did see different 'sse' packages installed for the different processors (given below) that mainly cover AMD Athlon processors and different favors of SSE. Everything was done using Python 2.5.2 installer from the official Python download site. I used CPU-Z 1.44 to find the processor instructions. The version of your numpy installer provided numpy version 1.0.5.dev5008. Windows XP: Athlon (Thunderbird) 1050MHz: MMX(+), 3DNow(+) Athlon XP 2100+: MMX(+), 3DNow!(+), SSE Athlon 64 3400: MMX(+), 3DNow!(+), SSE, SSE2, X86_64 Opteron 270: MMX(+), 3DNow!(+), SSE, SSE2,SSE3, X86_64 Vista Basic Intel Celeron M530: MMX, SSE, SSE2, SSE3, SSSE3,EM6AT Regards Bruce On Tue, Apr 15, 2008 at 7:01 PM, Damian Eads wrote: > > David Cournapeau wrote: > > Jarrod Millman wrote: > >> Hello, > >> > >> David Cournapeau has prepared a new win32 installer, which is aimed at > >> solving the recurring problem of non working atlas on different sets > >> of CPU. This installer simply checks which cpu you have, and installs > >> the appropriate numpy accordingly (without atlas if your cpu is not > >> supported). Windows users, please test the installer, and report > >> problems with it; we hope to use this scheme for all numpy and scipy > >> installers in the future. > > > > Sorry for being insistent, but I would really like to get rid of those > > windows/sse* problems once for all for numpy/scipy, and this means > > having people to test the installer. > > > > cheers, > > > > David > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > Sorry for getting back to you so late. I was out of range of wireless > networks for a day and a half. Shown below are my /proc/cpu-info and > result of running numpy.test(). SSE and SSE2 are supported but SSE3 is not. > > Nice work David! > > Damian > > ------------------------------------------- > processor : 0 > vendor_id : GenuineIntel > cpu family : 6 > model : 13 > model name : Intel(R) Pentium(R) M processor 1.60GHz > stepping : 6 > cpu MHz : 1600.000 > cache size : 2048 KB > fdiv_bug : no > hlt_bug : no > f00f_bug : no > coma_bug : no > fpu : yes > fpu_exception : yes > cpuid level : 2 > wp : yes > flags : fpu vme de pse tsc msr mce cx8 mtrr pge mca cmov pat > clflush dts acpi mmx fxsr sse sse2 ss tm pbe up bts est tm2 > bogomips : 3198.55 > clflush size : 64 > > > Numpy version 1.0.4 > Python version 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 > > bit (Int > el)] > Found 10/10 tests for numpy.core.defmatrix > Found 36/36 tests for numpy.core.ma > Found 223/223 tests for numpy.core.multiarray > Found 65/65 tests for numpy.core.numeric > > Found 31/31 tests for numpy.core.numerictypes > Found 12/12 tests for numpy.core.records > Found 6/6 tests for numpy.core.scalarmath > Found 14/14 tests for numpy.core.umath > Found 4/4 tests for numpy.ctypeslib > > Found 5/5 tests for numpy.distutils.misc_util > Found 1/1 tests for numpy.fft.fftpack > > Found 3/3 tests for numpy.fft.helper > Found 9/9 tests for numpy.lib.arraysetops > Found 46/46 tests for numpy.lib.function_base > > Found 5/5 tests for numpy.lib.getlimits > Found 4/4 tests for numpy.lib.index_tricks > Found 3/3 tests for numpy.lib.polynomial > > Found 49/49 tests for numpy.lib.shape_base > Found 15/15 tests for numpy.lib.twodim_base > Found 43/43 tests for numpy.lib.type_check > Found 1/1 tests for numpy.lib.ufunclike > Found 40/40 tests for numpy.linalg > Found 2/2 tests for numpy.random > Found 0/0 tests for __main__ > ................................................................................ > ................................................................................ > ................................................................................ > ................................................................................ > ....................................................................Warning: > inv > alid value encountered in absolute > Warning: invalid value encountered in absolute > Warning: invalid value encountered in less_equal > ................................................................................ > ................................................................................ > ................................................................................ > .......................................................... > ---------------------------------------------------------------------- > Ran 686 tests in 1.250s > > OK > > ------------------------ > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From konrad.hinsen at laposte.net Wed Apr 16 11:26:34 2008 From: konrad.hinsen at laposte.net (Konrad Hinsen) Date: Wed, 16 Apr 2008 17:26:34 +0200 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <20080415094143.GG9692@phare.normalesup.org> References: <48043622.1090606@enthought.com> <3d375d730804142338g2833ef75w84b01e7a99611f2@mail.gmail.com> <20080415094143.GG9692@phare.normalesup.org> Message-ID: <9262B6BB-F3A1-427F-A1BD-15DD02F772A7@laposte.net> On Apr 15, 2008, at 11:41, Gael Varoquaux wrote: > On Tue, Apr 15, 2008 at 01:38:45AM -0500, Robert Kern wrote: >>> Given that the next release will be 1.1, I think it is >>> reasonable to >>> include a few additional API breaks. > >> -lots. I don't want to break API compatibility again no matter what >> version number we bump to, 1.1, 2.0, or 24. It is simply not okay to >> do this again and again. > > +1 with Robert. +1 with Robert as well. My code still uses the oldnumeric API, so I am not directly concerned by changes to the new one, but I'd like to be able to move to the new API one day but most definitely only ONCE. Konrad. From charlesr.harris at gmail.com Wed Apr 16 11:24:52 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 16 Apr 2008 09:24:52 -0600 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com> <9457e7c80804150701k5e643893he29a73ff2df0e975@mail.gmail.com> <9457e7c80804150912s538b67aataeff9f2312e2d8fc@mail.gmail.com> <9457e7c80804160135v660ab494gc30e251dfedc2eec@mail.gmail.com> Message-ID: On Wed, Apr 16, 2008 at 8:06 AM, Alan G Isaac wrote: > On Wed, 16 Apr 2008, Anne Archibald apparently wrote: > > My (draconian) suggestion would be to simply raise an > > exception when a matrix is indexed with a scalar. > > This has been suggested before. But then, why? > Again, this imposes a deviation from the behavior of > arrays that provides no gain in functionality. > My proposal is: deviate from array behavior only when > it gives a clear gain in functionality. > > > > They're inherently two-dimensional; > > Of course. But the question is what follows from this > as an implication for the behavior of 1d indexing and > iteration. The answer is: nothing! > We get to do whatever is most useful! > > > > if you want a submatrix you should provide both indices > > (possibly including a ":"). > > Yes. We exactly agree on this. > Please persuade Stefan. > > > > If you actually want a subarray, as with an array, use > > ".A". > > Again we agree on this. > > But none of this answers the question: > what should you get when you iterate over a matrix. > Stefan seems to agree with me to this extent: > we should get its "rows". This disagreement is > over what that means: Just to make things interesting, if one uses rows (covariant) and column (contravariant) vectors, then there are lots of ways of looking at matrices. A covariance matrix, for instance, is a sum of tensor products of two column vectors, its inverse of row vectors, and a normal linear transformation is the sum of tensor products of column and row vectors. This all makes good mathematical sense and is like tracking units. That is, it isn't done in most numerical software ;) And remember, covariant and contravariant are reversed from their usual meaning due to historical accident. Next up, tensors as a subtype of ndarrays! Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Apr 16 11:28:31 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 16 Apr 2008 09:28:31 -0600 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <9262B6BB-F3A1-427F-A1BD-15DD02F772A7@laposte.net> References: <48043622.1090606@enthought.com> <3d375d730804142338g2833ef75w84b01e7a99611f2@mail.gmail.com> <20080415094143.GG9692@phare.normalesup.org> <9262B6BB-F3A1-427F-A1BD-15DD02F772A7@laposte.net> Message-ID: On Wed, Apr 16, 2008 at 9:26 AM, Konrad Hinsen wrote: > On Apr 15, 2008, at 11:41, Gael Varoquaux wrote: > > > On Tue, Apr 15, 2008 at 01:38:45AM -0500, Robert Kern wrote: > >>> Given that the next release will be 1.1, I think it is > >>> reasonable to > >>> include a few additional API breaks. > > > >> -lots. I don't want to break API compatibility again no matter what > >> version number we bump to, 1.1, 2.0, or 24. It is simply not okay to > >> do this again and again. > > > > +1 with Robert. > > +1 with Robert as well. My code still uses the oldnumeric API, so I > am not directly concerned by changes to the new one, but I'd like to > be able to move to the new API one day but most definitely only ONCE. > Konrad, if I recall you had some concerns about the behavior of numpy scalars. Do you want to make some suggestions while we are considering how to finalize the API and behavior? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Wed Apr 16 12:42:31 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 16 Apr 2008 18:42:31 +0200 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com> <48045FC3.5040402@ar.media.kyoto-u.ac.jp> <9457e7c80804150701k5e643893he29a73ff2df0e975@mail.gmail.com> <9457e7c80804150912s538b67aataeff9f2312e2d8fc@mail.gmail.com> <9457e7c80804160135v660ab494gc30e251dfedc2eec@mail.gmail.com> Message-ID: <9457e7c80804160942x58b03d1cibb9d41218767f7d1@mail.gmail.com> On 16/04/2008, Alan G Isaac wrote: > The rule is: > to get a submatrix, > use multiple indices. > As Anne has argued, > this is natural. That is *not* the rule for arrays; you argued the compatibility point yourself. > As far as I know, no objections have been raised by users of > matrices. I haven't seen many comments either way. Are you opposed to fixing the problem the way Tim suggested? > I claim that if you use matrices, you will find it natural for these to be 1d arrays. Why would linear algebra users prefer 1d arrays instead of vectors? Attached, please find a first attempt at implementing a Vector class. It passes all tests on my machine. Regards St?fan -------------- next part -------------- A non-text attachment was scrubbed... Name: numpy_mat_vector.patch Type: application/octet-stream Size: 3500 bytes Desc: not available URL: From gael.varoquaux at normalesup.org Wed Apr 16 13:01:45 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 16 Apr 2008 19:01:45 +0200 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: Message-ID: <20080416170145.GA15879@phare.normalesup.org> On Wed, Apr 16, 2008 at 10:06:05AM -0400, Alan G Isaac wrote: > > if you want a submatrix you should provide both indices > > (possibly including a ":"). > Yes. We exactly agree on this. > Please persuade Stefan. Alan, instead of trying blindly to persuade Stefan, please listen to his arguments. Or code an example implementation and show it to us. Stefan, and incidently myself, believe your position cannot be described in a consistent way, and that the day you try to code it, it will fall down. This is why the RowVector and ColumnVector idea was proposed. For me it is the only consistent proposal I have heard so far. I don't care that much about all this, but I do care that you are trying to get developpers lose their time on something that cannot work. > But none of this answers the question: > what should you get when you iterate over a matrix. > Stefan seems to agree with me to this extent: > we should get its "rows". This disagreement is > over what that means: > are these submatrices, a new vector object, or 1d arrays? > I claim that if you use matrices, you will find it > natural for these to be 1d arrays. OK, let us pretend A[:, 1] returns a 1D array, as you seem to be wanting, (A and B are matrices), What is A[1, :] * B then? You are multiply an array with a matrix. That already is dodgy, but numpy deals with it, and you get what you want. What about B*A[:, 1]? You don't get what you want here. So you have broken the user's expectation once again. The question is should we break this expectation (1), the expectation that A[x][y] = A[x, y] (2), or add RowVector and ColumnVector (3)? I believe that 3 is better than 2, which is better than 1. By the way, I think this has already been discussed. Let us not have the discussion go in circles, and try to keep in mind the different arguments. Maybe I am wrong, and you can make this work in a consistent way. In this case prove me wrong, and write some demo code that does this. The numpy developpers can than work with you to adapt it to a patch for numpy, and it can go in. > > That said, I don't actually use matrices, so I don't get a vote. > As far as I know, no objections have been raised by users of > matrices. In this sense, the objections have all been "abstract": > they do not reflect experience with the implied convenience > or inconvenience or particular behaviors. I have seen many times users ask for langage features that break its consistency, just because they do not realize it. The fact that you can use a language doesn't mean that you are skilled-enough to invent one. Cheers, Ga?l From charlesr.harris at gmail.com Wed Apr 16 13:41:56 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 16 Apr 2008 11:41:56 -0600 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <20080416170145.GA15879@phare.normalesup.org> References: <20080416170145.GA15879@phare.normalesup.org> Message-ID: On Wed, Apr 16, 2008 at 11:01 AM, Gael Varoquaux < gael.varoquaux at normalesup.org> wrote: > On Wed, Apr 16, 2008 at 10:06:05AM -0400, Alan G Isaac wrote: > > > if you want a submatrix you should provide both indices > > > (possibly including a ":"). > > > Yes. We exactly agree on this. > > Please persuade Stefan. > > Alan, instead of trying blindly to persuade Stefan, please listen to his > arguments. Or code an example implementation and show it to us. Stefan, > and incidently myself, believe your position cannot be described in a > consistent way, and that the day you try to code it, it will fall down. > This is why the RowVector and ColumnVector idea was proposed. For me it > is the only consistent proposal I have heard so far. I don't care that > much about all this, but I do care that you are trying to get developpers > lose their time on something that cannot work. > > > But none of this answers the question: > > what should you get when you iterate over a matrix. > > Stefan seems to agree with me to this extent: > > we should get its "rows". This disagreement is > > over what that means: > > are these submatrices, a new vector object, or 1d arrays? > > I claim that if you use matrices, you will find it > > natural for these to be 1d arrays. > > OK, let us pretend A[:, 1] returns a 1D array, as you seem to be wanting, > (A and B are matrices), > > What is A[1, :] * B then? You are multiply an array with a matrix. That > already is dodgy, but numpy deals with it, and you get what you want. > > What about B*A[:, 1]? You don't get what you want here. So you have > broken the user's expectation once again. > > The question is should we break this expectation (1), the expectation that > A[x][y] = A[x, y] (2), or add RowVector and ColumnVector (3)? > Dirac called them bra and ket vectors; put them together and you get bracket (inner product). Those might be alternative, shorter names. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Wed Apr 16 13:57:49 2008 From: aisaac at american.edu (Alan Isaac) Date: Wed, 16 Apr 2008 13:57:49 -0400 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <20080416170145.GA15879@phare.normalesup.org> References: <20080416170145.GA15879@phare.normalesup.org> Message-ID: On Wed, 16 Apr 2008, Gael Varoquaux wrote: > let us pretend A[:, 1] returns a 1D array, as you seem to > be wanting Where did I say anything like that?? Please look at the proposal. It affects **only** scalar indexing (and thereby iteration). Recall how emphatically I agreed with you: Multiple indexes should always return submatrices, and the right way to get submatrices is with multiple indexes. I have proposed no change in the behavior of non-scalar indexes. None. I specify an actual code change on the page you asked me to create. It is a trivial change in __getitem__. It does not lead to any of the breakage you suggest, because it does not change the behavior you consider. Please look: it changes *only* the processing of a scalar index. Since I'm not familiar with NumPy internals, I offer this only to pin down what I am talking about, not as a proposed code change. But it should make it clear that I am not asking what you suggest. AND (!!) my proposed change means that A[x][y] == A[x, y] which is not currently true. One good thing has come out of this discussion: as far as I can tell, everyone agrees that there should be a change such that A[x][y] == A[x, y] I believe I am proposing the simplest change, which leaves matrices as much like ndarrays as possible. My main objection to Stefan's proposal is that it adds substantial complexity for no apparent gain. Charles may have been suggesting that he can see a substantial gain: his comment was too cryptic for me. If so, I would be interested. Cheers, Alan From aisaac at american.edu Wed Apr 16 14:30:19 2008 From: aisaac at american.edu (Alan Isaac) Date: Wed, 16 Apr 2008 14:30:19 -0400 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <9457e7c80804160942x58b03d1cibb9d41218767f7d1@mail.gmail.com> References: <48043622.1090606@enthought.com><48045FC3.5040402@ar.media.kyoto-u.ac.jp><9457e7c80804150701k5e643893he29a73ff2df0e975@mail.gmail.com><9457e7c80804150912s538b67aataeff9f2312e2d8fc@mail.gmail.com><9457e7c80804160135v660ab494gc30e251dfedc2eec@mail.gmail.com><9457e7c80804160942x58b03d1cibb9d41218767f7d1@mail.gmail.com> Message-ID: > On 16/04/2008, Alan G Isaac wrote: >> The rule is: >> to get a submatrix, >> use multiple indices. On Wed, 16 Apr 2008, St?fan van der Walt wrote: > That is not the rule for arrays; you argued the compatibility point yourself. Sorry, I do not understand. I am saying only: I propose no change in current matrix behavior in response to nonscalar indexes. On Wed, 16 Apr 2008, St?fan van der Walt wrote: > Are you opposed to fixing the problem the way Tim > suggested? It seems lots of complexity for a payoff I do not see. My proposal is simpler and stays closer to ndarray behavior. But having a matrix be a container of such "RowVectors" is better than the current behavior. I do not see that it gets in the way of anything desirable, as long as the array attribute of your "vectors" will return a 1d array. > Why would linear algebra users prefer 1d arrays instead of vectors? What is a "vector"? To me a vector is an element of a vector space. As far as I can tell, a 1d array *is* a vector. If you mean by "vector" what people mean when they say "row vector" or "column vector", this is just short hand for special matrices. If I want these special matrices, I can have them right now. Of course I cannot index the elements with a scalar... Cheers, Alan Isaac From robert.kern at gmail.com Wed Apr 16 16:00:12 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 16 Apr 2008 15:00:12 -0500 Subject: [Numpy-discussion] numpy setup.py too restrictive, prevents use of fblas with cblas In-Reply-To: <480608F6.80904@enthought.com> References: <1d1e6ea70804160544v7f0a42d2k20c39f4da5bd0332@mail.gmail.com> <480608F6.80904@enthought.com> Message-ID: <3d375d730804161300lf0f7a97od3fd889165f3fb11@mail.gmail.com> On Wed, Apr 16, 2008 at 9:11 AM, Travis E. Oliphant wrote: > George Nurser wrote: > > Apologies for coming out of the woodwork so late here.. > > > > For blas/atlas etc in numpy & scipy on an opteron I use the AMD > > libraries (which only have fblas) together with cblas. This works very > > well. > > > > Current-ish SVN (v4779) in line 295-296 of numpy/core/setup.py has: > > > > if ('NO_ATLAS_INFO',1) in blas_info.get('define_macros',[]): > > return None # dotblas needs ATLAS, Fortran compiled > > blas will not be sufficient > > > > To get my AMD fblas/cblas approach to work, I have to comment out > > these two lines. > > Can this restriction preventing use of AMD fblas be removed for v1.1? > > > > Should I file a bug here? > > > Yes, you should put a ticket together. The more information you can > provide, the more likely someone will act on the ticket. Some kind of > info on why you had to comment out those two lines would be helpful. I am guessing that he built the cblas interface from here: http://www.netlib.org/blas/blast-forum/cblas.tgz Note that this is same C interface provided by ATLAS, the MKL, and GOTO and is *not* the version simply converted from the FORTRAN using f2c. It calls the (hopefully accelerated) FORTRAN subroutine where possible after manipulating the arguments to tell the subroutine that the matrices are transposed if row-major. The correct fix needs to be more sophisticated than removing those two lines. We need to recognize the MKL and the GOTO BLAS and allow them, too. It might also be worth including the appropriate subset of the cblas code provided in the tarball such that we can use any accelerated FORTRAN BLAS without the standard cblas interface. Then George wouldn't have the build the cblas library himself, either. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From stefan at sun.ac.za Wed Apr 16 16:22:11 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 16 Apr 2008 22:22:11 +0200 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com> <9457e7c80804150701k5e643893he29a73ff2df0e975@mail.gmail.com> <9457e7c80804150912s538b67aataeff9f2312e2d8fc@mail.gmail.com> <9457e7c80804160135v660ab494gc30e251dfedc2eec@mail.gmail.com> <9457e7c80804160942x58b03d1cibb9d41218767f7d1@mail.gmail.com> Message-ID: <9457e7c80804161322m4ed4bb76le5bce0a94b8cdc54@mail.gmail.com> On 16/04/2008, Alan Isaac wrote: > It seems lots of complexity for a payoff I do not see. > My proposal is simpler and stays closer to ndarray behavior. I showed you exactly where your proposal breaks down -- numerous times: x[0] is no longer the same as x[0,:] > But having a matrix be a container of such "RowVectors" is > better than the current behavior. I do not see that it > gets in the way of anything desirable, as long as the > array attribute of your "vectors" will return a 1d array. Did you try the patch? Do you think that a (column) vector should convert to a 1d array? St?fan From stefan at sun.ac.za Wed Apr 16 16:37:16 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 16 Apr 2008 22:37:16 +0200 Subject: [Numpy-discussion] numpy setup.py too restrictive, prevents use of fblas with cblas In-Reply-To: <3d375d730804161300lf0f7a97od3fd889165f3fb11@mail.gmail.com> References: <1d1e6ea70804160544v7f0a42d2k20c39f4da5bd0332@mail.gmail.com> <480608F6.80904@enthought.com> <3d375d730804161300lf0f7a97od3fd889165f3fb11@mail.gmail.com> Message-ID: <9457e7c80804161337i4d266126y8c745747fd7281e9@mail.gmail.com> Hi Robert On 16/04/2008, Robert Kern wrote: > The correct fix needs to be more sophisticated than removing those two > lines. We need to recognize the MKL and the GOTO BLAS and allow them, > too. It might also be worth including the appropriate subset of the > cblas code provided in the tarball such that we can use any > accelerated FORTRAN BLAS without the standard cblas interface. Then > George wouldn't have the build the cblas library himself, either. The inclusion of those cblas routines sounds like a good idea. Could you describe which we need, and what would be required to get this done? Cheers St?fan From cburns at berkeley.edu Wed Apr 16 17:36:37 2008 From: cburns at berkeley.edu (Christopher Burns) Date: Wed, 16 Apr 2008 14:36:37 -0700 Subject: [Numpy-discussion] OSX installer: please test Message-ID: <764e38540804161436j5ce7a468w8bea6a3c3598d69e@mail.gmail.com> I've built a Universal Mac binary for numpy 1.1.0. If Mac people would kindly test it, I'd appreciate any feedback. Download here: https://cirl.berkeley.edu/numpy/numpy-1.1.0rc1-py2.5-macosx10.5.dmg Technical details: - Built on OSX 10.5.2, Intel Core 2 Duo - Using XCode 3.0 with gcc 4.0.1 and the Accelerate.framework 1.4.2, gfortran 4.2.1 and Python 2.5.2 from python.org (MacPython) - Used bdist_mpkg 0.4.3 to make the distribution Since it uses the MacPython, it will install numpy under here: /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages Thanks -- Christopher Burns Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Apr 16 17:46:57 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 16 Apr 2008 16:46:57 -0500 Subject: [Numpy-discussion] numpy setup.py too restrictive, prevents use of fblas with cblas In-Reply-To: <9457e7c80804161337i4d266126y8c745747fd7281e9@mail.gmail.com> References: <1d1e6ea70804160544v7f0a42d2k20c39f4da5bd0332@mail.gmail.com> <480608F6.80904@enthought.com> <3d375d730804161300lf0f7a97od3fd889165f3fb11@mail.gmail.com> <9457e7c80804161337i4d266126y8c745747fd7281e9@mail.gmail.com> Message-ID: <3d375d730804161446u1ce83c3ft7ab253e7a9b2ad3e@mail.gmail.com> On Wed, Apr 16, 2008 at 3:37 PM, St?fan van der Walt wrote: > Hi Robert > > On 16/04/2008, Robert Kern wrote: > > The correct fix needs to be more sophisticated than removing those two > > lines. We need to recognize the MKL and the GOTO BLAS and allow them, > > too. It might also be worth including the appropriate subset of the > > cblas code provided in the tarball such that we can use any > > accelerated FORTRAN BLAS without the standard cblas interface. Then > > George wouldn't have the build the cblas library himself, either. > > The inclusion of those cblas routines sounds like a good idea. Could > you describe which we need, and what would be required to get this > done? Basically all of the files defining functions used in _dotblas.c and their dependencies; grepping for "cblas_" should find them all. The tricky part, naturally, is the build configuration. We need to recognize when we have an accelerated fblas and cblas (do not use our own cblas routines), when we have an accelerated fblas only (do use our own cblas routines), and when we don't have anything accelerated (do not build dotblas at all). I don't have a particularly clear idea on how to do that cleanly. One potential problem that I've just noticed is that there are some Fortran files in the cblas. For the most part, these seem to be used to convert Fortran function calls which return complex scalars to Fortran subroutines which can be easily interfaced to C. For example cblas_cdotu_sub.c requires cdotc.f. dotblas uses four such functions: cblas_[cz]dot[cu]_sub(). We would have to find another way to do these; possibly, we can just apply f2c to the Fortran files. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From lists at informa.tiker.net Wed Apr 16 17:56:03 2008 From: lists at informa.tiker.net (Andreas =?iso-8859-1?q?Kl=F6ckner?=) Date: Wed, 16 Apr 2008 17:56:03 -0400 Subject: [Numpy-discussion] numpy setup.py too restrictive, prevents use of fblas with cblas In-Reply-To: <9457e7c80804161337i4d266126y8c745747fd7281e9@mail.gmail.com> References: <1d1e6ea70804160544v7f0a42d2k20c39f4da5bd0332@mail.gmail.com> <3d375d730804161300lf0f7a97od3fd889165f3fb11@mail.gmail.com> <9457e7c80804161337i4d266126y8c745747fd7281e9@mail.gmail.com> Message-ID: <200804161756.05830.lists@informa.tiker.net> On Mittwoch 16 April 2008, St?fan van der Walt wrote: > The inclusion of those cblas routines sounds like a good idea. Could > you describe which we need, and what would be required to get this > done? Suppose cblas gets included in numpy, but for some reason someone decides to link another copy of cblas with their (separate) extension. Can we be certain that this does not lead to crashes on any platform supported by numpy? Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From aisaac at american.edu Wed Apr 16 18:02:50 2008 From: aisaac at american.edu (Alan G Isaac) Date: Wed, 16 Apr 2008 18:02:50 -0400 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <9457e7c80804161322m4ed4bb76le5bce0a94b8cdc54@mail.gmail.com> References: <48043622.1090606@enthought.com><9457e7c80804150701k5e643893he29a73ff2df0e975@mail.gmail.com><9457e7c80804150912s538b67aataeff9f2312e2d8fc@mail.gmail.com><9457e7c80804160135v660ab494gc30e251dfedc2eec@mail.gmail.com><9457e7c80804160942x58b03d1cibb9d41218767f7d1@mail.gmail.com> <9457e7c80804161322m4ed4bb76le5bce0a94b8cdc54@mail.gmail.com> Message-ID: On Wed, 16 Apr 2008, St?fan van der Walt apparently wrote: > Do you think that a (column) vector should convert to a 1d > array? Yes: for consistency with row vector conversion, and for indexing consistency. Again, I understand what you have done, and it addresses my core issue. I do not "object" to it. I just think there is a better way. (Better = simpler and just as functional.) I strongly support your proposal over no change. I just do not think it is a better proposal. But frankly I am not even sure of that: I see some pedagogical advantages in your approach, and I think Charles suggested that it may enable some new functionality. (But what is that?) In sum: if you put this in NumPy, I will happy. Cheers, Alan From aisaac at american.edu Wed Apr 16 18:02:53 2008 From: aisaac at american.edu (Alan G Isaac) Date: Wed, 16 Apr 2008 18:02:53 -0400 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <9457e7c80804161322m4ed4bb76le5bce0a94b8cdc54@mail.gmail.com> References: <48043622.1090606@enthought.com><9457e7c80804150701k5e643893he29a73ff2df0e975@mail.gmail.com><9457e7c80804150912s538b67aataeff9f2312e2d8fc@mail.gmail.com><9457e7c80804160135v660ab494gc30e251dfedc2eec@mail.gmail.com><9457e7c80804160942x58b03d1cibb9d41218767f7d1@mail.gmail.com> <9457e7c80804161322m4ed4bb76le5bce0a94b8cdc54@mail.gmail.com> Message-ID: On Wed, 16 Apr 2008, St?fan van der Walt apparently wrote: > I showed you exactly where your proposal breaks down -- > numerous times: x[0] is no longer the same as x[0,:] And as I explained back: this is a good thing (TM). There is no need for these to be the same. I also gave you the simple rule for users to rely on: nonscalar indexes are used for submatrix extraction. It is not a breakdown. It is the proposal: restore the proper behavior of x[0], but keep submatrix extraction **exactly** the same as it is now (for nonscalar indexes). What it gains is that x[i][j] == x[i,j]. Alan From robert.kern at gmail.com Wed Apr 16 18:13:09 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 16 Apr 2008 17:13:09 -0500 Subject: [Numpy-discussion] numpy setup.py too restrictive, prevents use of fblas with cblas In-Reply-To: <200804161756.05830.lists@informa.tiker.net> References: <1d1e6ea70804160544v7f0a42d2k20c39f4da5bd0332@mail.gmail.com> <3d375d730804161300lf0f7a97od3fd889165f3fb11@mail.gmail.com> <9457e7c80804161337i4d266126y8c745747fd7281e9@mail.gmail.com> <200804161756.05830.lists@informa.tiker.net> Message-ID: <3d375d730804161513v336a9e4anf07fb8c6e28b05e2@mail.gmail.com> On Wed, Apr 16, 2008 at 4:56 PM, Andreas Kl?ckner wrote: > On Mittwoch 16 April 2008, St?fan van der Walt wrote: > > The inclusion of those cblas routines sounds like a good idea. Could > > you describe which we need, and what would be required to get this > > done? > > Suppose cblas gets included in numpy, but for some reason someone decides to > link another copy of cblas with their (separate) extension. Can we be certain > that this does not lead to crashes on any platform supported by numpy? I imagine we would have the same problem with the f2c'ed default LAPACK routines we include to support numpy.linalg. I haven't heard of this ever causing a problem. However, if there is a conflict, we can get around it with clever #defines such that the actual symbols in the extension module would be, say, numpy_cblas_dgemm, etc. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From stefan at sun.ac.za Wed Apr 16 18:20:25 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 17 Apr 2008 00:20:25 +0200 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com> <9457e7c80804150912s538b67aataeff9f2312e2d8fc@mail.gmail.com> <9457e7c80804160135v660ab494gc30e251dfedc2eec@mail.gmail.com> <9457e7c80804160942x58b03d1cibb9d41218767f7d1@mail.gmail.com> <9457e7c80804161322m4ed4bb76le5bce0a94b8cdc54@mail.gmail.com> Message-ID: <9457e7c80804161520n7141f1c3p5ebc8018784fde38@mail.gmail.com> On 17/04/2008, Alan G Isaac wrote: > It is not a breakdown. > It is the proposal: > restore the proper behavior of x[0], > but keep submatrix extraction **exactly** > the same as it is now (for nonscalar indexes). > > What it gains is that x[i][j] == x[i,j]. As the patch I sent earlier shows, we can gain that without compromising on x[0] == x[0,:]. The Vector class can be tweaked further in how it is converted to an array. It would be easy to produce a 1d-array -- and there's some argument for consistency there. I don't want you to feel that I steam-rollered your proposal, though, so before I commit I'd like for someone to objectively take a look at both approaches/patches and comment. Goodnight, St?fan From gael.varoquaux at normalesup.org Wed Apr 16 18:34:19 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 17 Apr 2008 00:34:19 +0200 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <9457e7c80804161322m4ed4bb76le5bce0a94b8cdc54@mail.gmail.com> Message-ID: <20080416223419.GF10057@phare.normalesup.org> On Wed, Apr 16, 2008 at 06:02:53PM -0400, Alan G Isaac wrote: > On Wed, 16 Apr 2008, St?fan van der Walt apparently wrote: > > I showed you exactly where your proposal breaks down -- > > numerous times: x[0] is no longer the same as x[0,:] > And as I explained back: this is a good thing (TM). > There is no need for these to be the same. [...] > What it gains is that x[i][j] == x[i,j]. I am sorry, I don't see why you prioritize "x[i][j] == x[i,j]" (1) more than "x[0] == x[0,:]" (2). For me it is the other way around, because we should be encouraging users not to use "x[i][j]", which is a very poor way of indexing compared to "x[i,j]". The first requires two __getitem__ calls and an object creation, whereas the first requires only one, and no object creation. Ga?l From stefan at sun.ac.za Wed Apr 16 19:09:39 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 17 Apr 2008 01:09:39 +0200 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <9457e7c80804161520n7141f1c3p5ebc8018784fde38@mail.gmail.com> References: <48043622.1090606@enthought.com> <9457e7c80804150912s538b67aataeff9f2312e2d8fc@mail.gmail.com> <9457e7c80804160135v660ab494gc30e251dfedc2eec@mail.gmail.com> <9457e7c80804160942x58b03d1cibb9d41218767f7d1@mail.gmail.com> <9457e7c80804161322m4ed4bb76le5bce0a94b8cdc54@mail.gmail.com> <9457e7c80804161520n7141f1c3p5ebc8018784fde38@mail.gmail.com> Message-ID: <9457e7c80804161609x28047124uc2b3c8abf405698d@mail.gmail.com> On 17/04/2008, St?fan van der Walt wrote: > On 17/04/2008, Alan G Isaac wrote: > > It is not a breakdown. > > It is the proposal: > > restore the proper behavior of x[0], > > but keep submatrix extraction **exactly** > > the same as it is now (for nonscalar indexes). > > > > What it gains is that x[i][j] == x[i,j]. > > > As the patch I sent earlier shows, we can gain that without > compromising on x[0] == x[0,:]. > > The Vector class can be tweaked further in how it is converted to an > array. It would be easy to produce a 1d-array -- and there's some > argument for consistency there. > > I don't want you to feel that I steam-rollered your proposal, though, > so before I commit I'd like for someone to objectively take a look at > both approaches/patches and comment. Split infinitive -- I'd get in trouble for that. Please use the latest patch (attached), which fixes a bug with assignment. I experimented with returning an (N,) array when converting using vector.A, but I'm not convinced that that is consistent behaviour, so I let that issue remain for now. Cheers St?fan -------------- next part -------------- A non-text attachment was scrubbed... Name: numpy_mat_vector_01.patch Type: application/octet-stream Size: 4256 bytes Desc: not available URL: From aisaac at american.edu Wed Apr 16 19:52:47 2008 From: aisaac at american.edu (Alan G Isaac) Date: Wed, 16 Apr 2008 19:52:47 -0400 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <20080416223419.GF10057@phare.normalesup.org> References: <9457e7c80804161322m4ed4bb76le5bce0a94b8cdc54@mail.gmail.com> <20080416223419.GF10057@phare.normalesup.org> Message-ID: On Thu, 17 Apr 2008, Gael Varoquaux apparently wrote: > I am sorry, I don't see why you prioritize "x[i][j] == x[i,j]" (1) more than > "x[0] == x[0,:]" (2). Well the quick answer is: use matrices for awhile, and I expect you will see why, and teach them for awhile, and I am quite sure you will see why. But to provide some explanation. The underlying issue is not quite whether ``x[i][j] == x[i,j]``, but whether ``x[i][j]`` accesses the i,j-th element. The is a *fundamental* expectation for Python 2d containers. Naturally we can eventually train people to expect anything, but since I teach people to use Python, I can tell you the current matrix behavior is a pain in the butt. *Everyone* is surprised by the outcome of ``x[i][j]`` when x is a matrix. (Ask yourself: weren't you the first time?) Surprises are bad. In contrast, expectations about how nonscalar indexing works are much less of a given. And this is especially so if one teaches matrices before arrays, which I generally do. (More people are familiar with matrices.) You can just teach people how nonscalar indexing produces submatrices. No sweat. Related to this, it is a *fairly* fundamental expectation that iteration over a 2d Python object will yield 1d objects. This very natural expectation is met by ndarrays but not currently by matrices. My proposal allows all these natural expectations to be met. Stefan's scores well too, so why do I prefer mine? As I said, I am not sure mine is better, but I think it is, an I can offer some reasons to prefer it. Two are: - it is simpler - it retains more ndarray behavior Simplicity is always good, for any given level of functionality. Keeping close to ndarray behavior is good, since it means more transferable skills across the two types. A third more personal reason is essentially aesthetic. I like it when containers of a single type implement containment as a weak order in the type hierarchy. So I like that the less primitive matrix object would be a container of more primitive ndarrays (when iterating). Similarly, I like that ndarrays are containers of ndarrays. In contrast Stefan's implementation will treat matrices as containing a matrix subclass---a *less* primitive object, in this sense. I certainly do not expect others to share this preference, and I offer it up in expectation that it will garner some harsh criticism. But there it is. So that is my explanation. But again, if Stefan implements his proposal (tweaked as discussed), I will be quite pleased. Cheers, Alan From oliphant at enthought.com Thu Apr 17 01:30:33 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 17 Apr 2008 00:30:33 -0500 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <9457e7c80804161322m4ed4bb76le5bce0a94b8cdc54@mail.gmail.com> <20080416223419.GF10057@phare.normalesup.org> Message-ID: <4806E079.7070904@enthought.com> Alan G Isaac wrote: > On Thu, 17 Apr 2008, Gael Varoquaux apparently wrote: > >> I am sorry, I don't see why you prioritize "x[i][j] == x[i,j]" (1) more than >> "x[0] == x[0,:]" (2). >> > > Well the quick answer is: > use matrices for awhile, > and I expect you will see why, > and teach them for awhile, and > I am quite sure you will see why. > I have mixed feelings on all of this discussion. I do want to make sure, however, that two ideas are considered: 1) How will sparse matrices behave? It seems to me that the matrix object and sparse matrices should behave identically if at all possible. 2) How much code will break if we change the current behavior. Regarding (1), What does x[0] return when x is a sparse matrix? -Travis From matthieu.brucher at gmail.com Thu Apr 17 03:33:48 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 17 Apr 2008 09:33:48 +0200 Subject: [Numpy-discussion] Strange behaviour of linalg.svd() and linalg.eigh() Message-ID: Hi, Ive implemented the classical MultiDimensional Scaling for the scikit learn using both functions. Their behavior surprised me for "big" arrays (10000 by 10000, symmetric as it is a similarity matrix). linalg.svd() raises a memory error because it tries to allocate a (7000000,) array (in fact bigger than that !). This is strange because the test was made on a 64bits Linux, so memory should not have been a problem. linalg.eigh() fails to diagonalize the matrix, it gives me NaN as a result, and this is not very useful. A direct optimization of the underlying cost function can give me an adequate solution. I cannot attach the matrix file (more than 700MB when pickled), but if anyone has a clue, I'll be glad. Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From bakker at itc.nl Thu Apr 17 04:10:51 2008 From: bakker at itc.nl (Wim Bakker) Date: Thu, 17 Apr 2008 10:10:51 +0200 Subject: [Numpy-discussion] Windows/numpy1.0.4 memmap & astype produce loads of warnings on delete Message-ID: <5AF149DBB6DFE24AA3F4F53201E539AD020A5741@itcnt24.itc.nl> The last version of numpy gives me headaches. I've been able to trace the problem to the use of astype(). When a memmap is deleted I get the following warning: Exception exceptions.ValueError: 'mmap closed or invalid' in ignored The memmap still seems to work but these error messages slow don't the processing considerably. When reverting back to numpy 1.0.2 the problem disappears. Below is a short program that reproduces the warnings: == import numpy a = numpy.memmap(r'C:\Temp\test.dat', mode='w+', shape=(10,), dtype='b') for i in range(10): a[i] = i del a a = numpy.memmap(r'C:\Temp\test.dat', mode='r', shape=(10,), dtype='b') a = a.astype('d') del a == The last delete produces the warnings. Am I doing something wrong or should this be fixed in numpy? Regards, Wim Bakker International Institute for Geo-Information Science and Earth Observation (ITC) Chamber of Commerce: 410 27 560 E-mail disclaimer The information in this e-mail, including any attachments, is intended for the addressee only. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or action in relation to the content of this information is strictly prohibited. If you have received this e-mail by mistake, please delete the message and any attachment and inform the sender by return e-mail. ITC accepts no liability for any error or omission in the message content or for damage of any kind that may arise as a result of e-mail transmission. From robert.kern at gmail.com Thu Apr 17 04:21:12 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 17 Apr 2008 03:21:12 -0500 Subject: [Numpy-discussion] Windows/numpy1.0.4 memmap & astype produce loads of warnings on delete In-Reply-To: <5AF149DBB6DFE24AA3F4F53201E539AD020A5741@itcnt24.itc.nl> References: <5AF149DBB6DFE24AA3F4F53201E539AD020A5741@itcnt24.itc.nl> Message-ID: <3d375d730804170121l39486153i4f8591abd5ba83e8@mail.gmail.com> On Thu, Apr 17, 2008 at 3:10 AM, Wim Bakker wrote: > The last version of numpy gives me headaches. I've been able to trace > the problem to the use of astype(). When a memmap is deleted I get the > following warning: > > Exception exceptions.ValueError: 'mmap closed or invalid' in method memmap.__del__ of memmap([ 0., 1., 2., 3., 4., 5., 6., 7., > 8., 9.])> ignored > > The memmap still seems to work but these error messages slow don't the > processing considerably. When reverting back to numpy 1.0.2 the problem > disappears. > > Below is a short program that reproduces the warnings: > > == > import numpy > > a = numpy.memmap(r'C:\Temp\test.dat', mode='w+', shape=(10,), dtype='b') > > for i in range(10): > a[i] = i > > del a > > a = numpy.memmap(r'C:\Temp\test.dat', mode='r', shape=(10,), dtype='b') > > a = a.astype('d') > > del a > == > > The last delete produces the warnings. > > Am I doing something wrong or should this be fixed in numpy? I think I or someone else fixed most of these issues in SVN. Can you try out SVN numpy and see if the warnings are gone in your code? I don't see them on OS X. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From bakker at itc.nl Thu Apr 17 05:22:34 2008 From: bakker at itc.nl (Wim Bakker) Date: Thu, 17 Apr 2008 11:22:34 +0200 Subject: [Numpy-discussion] Windows/numpy1.0.4 memmap & astype produce loads of warnings on delete Message-ID: <5AF149DBB6DFE24AA3F4F53201E539AD020A5742@itcnt24.itc.nl> Robert, I've replaced my C:\Python24\Lib\site-packages\numpy\core\memmap.py with the one from http://svn.scipy.org/svn/numpy/trunk/numpy/core/ and that fixes the problem. Thanks for your quick help! When is the new version of numpy due? Wim Bakker Any garbage below this line is not mine. -- International Institute for Geo-Information Science and Earth Observation (ITC) Chamber of Commerce: 410 27 560 E-mail disclaimer The information in this e-mail, including any attachments, is intended for the addressee only. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or action in relation to the content of this information is strictly prohibited. If you have received this e-mail by mistake, please delete the message and any attachment and inform the sender by return e-mail. ITC accepts no liability for any error or omission in the message content or for damage of any kind that may arise as a result of e-mail transmission. From sertier at biomserv.univ-lyon1.fr Thu Apr 17 05:51:04 2008 From: sertier at biomserv.univ-lyon1.fr (Anne-Sophie Sertier) Date: Thu, 17 Apr 2008 11:51:04 +0200 Subject: [Numpy-discussion] Segmentation fault Message-ID: <48071D88.6010604@biomserv.univ-lyon1.fr> Hello ! I'm a new user of Numpy. I want to use it to work with matrices (very huge matrices), select column, make product, etc ... Configuration : Debian, python 2.4.4 and numpy 1.0.1 So here is my problem: I initialize a matrix which will contain only 0 and 1 >>> from numpy import * >>> matCons = zeros((194844,267595)),dtype=int8) then I put the 1s according to my conditions >>> for i in range(194844): try: clusters = consensus[namesP[i]] except KeyError: continue for clus in clusters: ind = argmax(namesC.find(clus)) # search of index matrix[i,ind] = 1 finaly I want to select some columns in this matrix. So I did for example: >>> matCons[:,(1,2)] and I obtain the segmentation fault. I try to select one column by one, which is possible : >>> a = matCons[:,1] but if I try to put it in a new matrix, I have again te segmentation fault >>> tmp = zeros((194844,2),int8) >>> tmp[:,0] = a I make some tests like this one : ---------------------------------------------------------- >>> matrix = zeros((194844,267595),int8) >>> a = matrix[:,0] >>> for i in range(len(a)): print i, a[i] ... ... 2221 0 2222 0 2223 0 2224 0 2225 0 2226 0 2227 0 2228 0 2229 0 2230 0 2231 0 2232 0 2233 0 2234 0 2235 0 2236 0 2237 0 2238 0 2239 0 2240 0 2241 95 2242 0 2243 0 2244 101 2245 0 2246 15 2247 -112 2248 -115 2249 -1 2250 84 2251 -99 2252 -1 2253 60 2254 -49 2255 -1 2256 17 2257 -35 2258 -119 2259 -35 2260 -1 2261 52 2262 -1 2263 -1 2264 -35 2265 -1 2266 121 2267 1 2268 0 2269 0 2270 0 2271 0 2272 28 2273 50 2274 4 2275 -1 2276 -24 2277 0 2278 0 2279 0 2280 -58 2281 0 2282 0 2283 0 2284 0 2285 -1 2286 -69 2287 12 2288 -36 2289 8 2290 66 Erreur de segmentation ------------------------------------------ But if I do the same things on a matrix with dimension 194844*300, it works well. So if someone can explain me why it does not work in my case... Thanks in advance. Anne-Sophie -- Anne-Sophie Sertier, Doctorante monitrice UMR 5558, Laboratoire de Biom?trie et Biologie Evolutive Universit? Lyon1, b?t G. Mendel 2? ?tage 43 bd du 11 novembre 1918 69622 Villeurbanne cedex France Tel: 04 26 23 44 76 From gnurser at googlemail.com Thu Apr 17 06:09:09 2008 From: gnurser at googlemail.com (George Nurser) Date: Thu, 17 Apr 2008 11:09:09 +0100 Subject: [Numpy-discussion] numpy setup.py too restrictive, prevents use of fblas with cblas In-Reply-To: <3d375d730804161300lf0f7a97od3fd889165f3fb11@mail.gmail.com> References: <1d1e6ea70804160544v7f0a42d2k20c39f4da5bd0332@mail.gmail.com> <480608F6.80904@enthought.com> <3d375d730804161300lf0f7a97od3fd889165f3fb11@mail.gmail.com> Message-ID: <1d1e6ea70804170309w451efd5eof11bd5b3084cc3b9@mail.gmail.com> > I am guessing that he built the cblas interface from here: > > http://www.netlib.org/blas/blast-forum/cblas.tgz > > Note that this is same C interface provided by ATLAS, the MKL, and > GOTO and is *not* the version simply converted from the FORTRAN using > f2c. It calls the (hopefully accelerated) FORTRAN subroutine where > possible after manipulating the arguments to tell the subroutine that > the matrices are transposed if row-major. > > The correct fix needs to be more sophisticated than removing those two > lines. We need to recognize the MKL and the GOTO BLAS and allow them, > too. It might also be worth including the appropriate subset of the > cblas code provided in the tarball such that we can use any > accelerated FORTRAN BLAS without the standard cblas interface. Then > George wouldn't have the build the cblas library himself, either. I have filed a ticket on this. Yes, it is the cblas interface from http://www.netlib.org/blas/blast-forum/cblas.tgz. Not having to build the interface would indeed make it a whole lot easier. George Nurser. From Joris.DeRidder at ster.kuleuven.be Thu Apr 17 06:10:02 2008 From: Joris.DeRidder at ster.kuleuven.be (Joris De Ridder) Date: Thu, 17 Apr 2008 12:10:02 +0200 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <9457e7c80804161609x28047124uc2b3c8abf405698d@mail.gmail.com> References: <48043622.1090606@enthought.com> <9457e7c80804150912s538b67aataeff9f2312e2d8fc@mail.gmail.com> <9457e7c80804160135v660ab494gc30e251dfedc2eec@mail.gmail.com> <9457e7c80804160942x58b03d1cibb9d41218767f7d1@mail.gmail.com> <9457e7c80804161322m4ed4bb76le5bce0a94b8cdc54@mail.gmail.com> <9457e7c80804161520n7141f1c3p5ebc8018784fde38@mail.gmail.com> <9457e7c80804161609x28047124uc2b3c8abf405698d@mail.gmail.com> Message-ID: On 17 Apr 2008, at 01:09, St?fan van der Walt wrote: > Split infinitive -- I'd get in trouble for that. > > Please use the latest patch (attached), which fixes a bug with > assignment. > > I experimented with returning an (N,) array when converting using > vector.A, but I'm not convinced that that is consistent behaviour, so > I let that issue remain for now. The patch introduces "Vector", but I assume it's your intend to change this into "RowVector" or "ColumnVector" in the end, correct? With "Vector" I expect tons of emails of (newbie) people confused whether and when they should use array and when Vector. Especially when they come from a language where "vector" is used for what we call an array. Just my 2 cents, Joris Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm From david at ar.media.kyoto-u.ac.jp Thu Apr 17 05:59:24 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 17 Apr 2008 18:59:24 +0900 Subject: [Numpy-discussion] Segmentation fault In-Reply-To: <48071D88.6010604@biomserv.univ-lyon1.fr> References: <48071D88.6010604@biomserv.univ-lyon1.fr> Message-ID: <48071F7C.7020101@ar.media.kyoto-u.ac.jp> Anne-Sophie Sertier wrote: > Hello ! > > I'm a new user of Numpy. I want to use it to work with matrices (very > huge matrices), select column, make product, etc ... > Configuration : Debian, python 2.4.4 and numpy 1.0.1 > > So here is my problem: > I initialize a matrix which will contain only 0 and 1 > > >>> from numpy import * > >>>> matCons = zeros((194844,267595)),dtype=int8) >>>> Unless you are executing this on a gigantic computer, this won't work very well: you are asking to create an array which has ~ 2e5^2 elements, that is around 40 Gb. There is a bug, but the bug happens at the above line: the zeros call did not fail whereas it should have. It is likely caused because the number of elements cannot fit into a 32 bits integers, which means it overflows: import numpy as np n , m = 1e5,1e5 a = np.zeros((n, m), np.int8) assert a.size == n * m Will raise an assertion error (n * m is just big enough to not fit into a 32 bits integer in this case). cheers, David From david at ar.media.kyoto-u.ac.jp Thu Apr 17 06:01:39 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 17 Apr 2008 19:01:39 +0900 Subject: [Numpy-discussion] Segmentation fault In-Reply-To: <48071F7C.7020101@ar.media.kyoto-u.ac.jp> References: <48071D88.6010604@biomserv.univ-lyon1.fr> <48071F7C.7020101@ar.media.kyoto-u.ac.jp> Message-ID: <48072003.80708@ar.media.kyoto-u.ac.jp> David Cournapeau wrote: > > Unless you are executing this on a gigantic computer, this won't work > very well: you are asking to create an array which has ~ 2e5^2 elements, > that is around 40 Gb. > > There is a bug, but the bug happens at the above line: the zeros call > did not fail whereas it should have. It is likely caused because the > number of elements cannot fit into a 32 bits integers, which means it > overflows: > > import numpy as np > n , m = 1e5,1e5 > a = np.zeros((n, m), np.int8) > assert a.size == n * m > > Will raise an assertion error (n * m is just big enough to not fit into > a 32 bits integer in this case). > I forgot to add: this is a bug in numpy, you should not get a segmentation fault, but it would not work anyway (because you are asking for a too big array) :) cheers, David From wilson.t.thompson at gmail.com Thu Apr 17 06:18:45 2008 From: wilson.t.thompson at gmail.com (wilson) Date: Thu, 17 Apr 2008 03:18:45 -0700 (PDT) Subject: [Numpy-discussion] sort Message-ID: <5e01b4af-a7fd-4c32-8786-4b60b5a6e574@8g2000hse.googlegroups.com> i have a 1 dimensional array of floats that i want to sort in descending order.i did as follows from numpy import tolist,array,sort y=array([......]) #may be 1000 items z=sort(y) l=z.tolist() l.reverse() rl=array(l) now rl is a 1 dim array sorted in descending order. but i am looking for a better,compact way to do this.can someone help? From robert.kern at gmail.com Thu Apr 17 06:22:55 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 17 Apr 2008 05:22:55 -0500 Subject: [Numpy-discussion] sort In-Reply-To: <5e01b4af-a7fd-4c32-8786-4b60b5a6e574@8g2000hse.googlegroups.com> References: <5e01b4af-a7fd-4c32-8786-4b60b5a6e574@8g2000hse.googlegroups.com> Message-ID: <3d375d730804170322u3c06e94dvd5606e4ef4f790b7@mail.gmail.com> On Thu, Apr 17, 2008 at 5:18 AM, wilson wrote: > i have a 1 dimensional array of floats that i want to sort in > descending order.i did as follows > > from numpy import tolist,array,sort > y=array([......]) #may be 1000 items > z=sort(y) > l=z.tolist() > l.reverse() > rl=array(l) > > now rl is a 1 dim array sorted in descending order. > but i am looking for a better,compact way to do this.can someone help? rl = sort(y)[::-1] -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From gael.varoquaux at normalesup.org Thu Apr 17 06:23:38 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 17 Apr 2008 12:23:38 +0200 Subject: [Numpy-discussion] sort In-Reply-To: <5e01b4af-a7fd-4c32-8786-4b60b5a6e574@8g2000hse.googlegroups.com> References: <5e01b4af-a7fd-4c32-8786-4b60b5a6e574@8g2000hse.googlegroups.com> Message-ID: <20080417102338.GB16672@phare.normalesup.org> On Thu, Apr 17, 2008 at 03:18:45AM -0700, wilson wrote: > i have a 1 dimensional array of floats that i want to sort in > descending order.i did as follows > from numpy import tolist,array,sort > y=array([......]) #may be 1000 items > z=sort(y) > l=z.tolist() > l.reverse() > rl=array(l) > now rl is a 1 dim array sorted in descending order. > but i am looking for a better,compact way to do this.can someone help? Not tested, but something like y = array([......]) #may be 1000 items y.sort() y = y[::-1] Ga?l From sertier at biomserv.univ-lyon1.fr Thu Apr 17 07:04:24 2008 From: sertier at biomserv.univ-lyon1.fr (Anne-Sophie Sertier) Date: Thu, 17 Apr 2008 13:04:24 +0200 Subject: [Numpy-discussion] Segmentation fault In-Reply-To: <48072003.80708@ar.media.kyoto-u.ac.jp> References: <48071D88.6010604@biomserv.univ-lyon1.fr> <48071F7C.7020101@ar.media.kyoto-u.ac.jp> <48072003.80708@ar.media.kyoto-u.ac.jp> Message-ID: <48072EB8.7040701@biomserv.univ-lyon1.fr> Thanks a lot for your answer ! I will try another way ! Anne-Sophie David Cournapeau a ?crit : > David Cournapeau wrote: >> Unless you are executing this on a gigantic computer, this won't work >> very well: you are asking to create an array which has ~ 2e5^2 elements, >> that is around 40 Gb. >> >> There is a bug, but the bug happens at the above line: the zeros call >> did not fail whereas it should have. It is likely caused because the >> number of elements cannot fit into a 32 bits integers, which means it >> overflows: >> >> import numpy as np >> n , m = 1e5,1e5 >> a = np.zeros((n, m), np.int8) >> assert a.size == n * m >> >> Will raise an assertion error (n * m is just big enough to not fit into >> a 32 bits integer in this case). >> > > I forgot to add: this is a bug in numpy, you should not get a > segmentation fault, but it would not work anyway (because you are asking > for a too big array) :) > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -- Anne-Sophie Sertier, Doctorante monitrice UMR 5558, Laboratoire de Biom?trie et Biologie Evolutive Universit? Lyon1, b?t G. Mendel 2? ?tage 43 bd du 11 novembre 1918 69622 Villeurbanne cedex France Tel: 04 26 23 44 76 From stefan at sun.ac.za Thu Apr 17 07:10:51 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 17 Apr 2008 13:10:51 +0200 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <9457e7c80804161322m4ed4bb76le5bce0a94b8cdc54@mail.gmail.com> <20080416223419.GF10057@phare.normalesup.org> Message-ID: <9457e7c80804170410m66b30078l29dddd75625ad5e1@mail.gmail.com> On 17/04/2008, Alan G Isaac wrote: > Stefan's scores well too, so why do I prefer mine? > As I said, I am not sure mine is better, but I think it is, > an I can offer some reasons to prefer it. Two are: > > - it is simpler > - it retains more ndarray behavior That is simply not true. The patch I sent provides everything you asked for *without* breaking *any* ndarray behaviour, whereas your suggestion breaks x[0] == x[0,:]. St?fan From stefan at sun.ac.za Thu Apr 17 07:16:02 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 17 Apr 2008 13:16:02 +0200 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <4806E079.7070904@enthought.com> References: <9457e7c80804161322m4ed4bb76le5bce0a94b8cdc54@mail.gmail.com> <20080416223419.GF10057@phare.normalesup.org> <4806E079.7070904@enthought.com> Message-ID: <9457e7c80804170416y2e9b009dka8cc6af664bdcb0c@mail.gmail.com> On 17/04/2008, Travis E. Oliphant wrote: > I have mixed feelings on all of this discussion. I do want to make > sure, however, that two ideas are considered: > > 1) How will sparse matrices behave? It seems to me that the matrix > object and sparse matrices should behave identically if at all possible. That's a good question. Sparse matrices are not containers -- they are always 2-dimensional, so it brings us back to what a "row" slice of a sparse matrix should be: probably a sparse matrix. That's the way matrices behave currently, but that behaviour differs from the ndarray behaviour -- where we have N-d arrays containing (N-1)-d arrays. > 2) How much code will break if we change the current behavior. None (for dense matrices), if we introduce vectors. The only difference in behaviour would be for x[0][0] which then becomes equivalent to the ndarray syntax. St?fan From stefan at sun.ac.za Thu Apr 17 07:25:30 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 17 Apr 2008 13:25:30 +0200 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <48043622.1090606@enthought.com> <9457e7c80804160135v660ab494gc30e251dfedc2eec@mail.gmail.com> <9457e7c80804160942x58b03d1cibb9d41218767f7d1@mail.gmail.com> <9457e7c80804161322m4ed4bb76le5bce0a94b8cdc54@mail.gmail.com> <9457e7c80804161520n7141f1c3p5ebc8018784fde38@mail.gmail.com> <9457e7c80804161609x28047124uc2b3c8abf405698d@mail.gmail.com> Message-ID: <9457e7c80804170425n7e1bbe81m1f2180d1f1096ddd@mail.gmail.com> Hi Joris On 17/04/2008, Joris De Ridder wrote: > The patch introduces "Vector", but I assume it's your intend to change > this into "RowVector" or "ColumnVector" in the end, correct? This was just of a proof of concept. It depends very much on what you guys want. Vector currently remains 2-dimensional, but allows scalar indexing, i.e. my_vector[0] returns the zero'th element. The other option is to make Vectors 1-dimensional, and to introduce both RowVector and ColumnVector. That's easy enough to do -- a person just needs to take care when it comes to multiplication. > With > "Vector" I expect tons of emails of (newbie) people confused whether > and when they should use array and when Vector. I'd think you should stick to either using (Matrix,ColumnVector,RowVector) or using the full ndarray capability. > Especially when they > come from a language where "vector" is used for what we call an array. But they also come from languages where the word "matrix" is used to describe what we call an nd-array, so that's neither here nor there. Regards St?fan From david at ar.media.kyoto-u.ac.jp Thu Apr 17 07:34:48 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 17 Apr 2008 20:34:48 +0900 Subject: [Numpy-discussion] Distutils: using different linker options for c++ and c code Message-ID: <480735D8.9020102@ar.media.kyoto-u.ac.jp> Hi, I cannot seem to make scipy.sparse works on windows with mingw32, and it seems related to the msvc runtime. I always get segfaults in the sparsetools module (c++); if I build the module manually to control the options precisely and remove the -lmsvc71 from the build command, I cannot reproduce the segfault anymore (but the segfault does not happen 100 % of the time, so it is hard to be sure) So is there a way in distutils to make the difference between C++ and C: for extension with C, use the msvc, for extensions in C++, do not use it ? (The obvious hack consisting in remove msvc alltogether does not work: it crahses in scipy.io, which is linked to incompatible FILE structures between mingw and VS, I guess). thanks, David From matthieu.brucher at gmail.com Thu Apr 17 10:07:38 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 17 Apr 2008 16:07:38 +0200 Subject: [Numpy-discussion] numpy.testing.raises lacks the definition of make_decorator Message-ID: Hi, I'm trying to use raises with numpy, but when I call nosetests, it tells me that make_decorator is not defined. And indeed, it is not defined or imported in this file, but in nose/tools.py Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Thu Apr 17 10:49:55 2008 From: aisaac at american.edu (Alan G Isaac) Date: Thu, 17 Apr 2008 10:49:55 -0400 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <9457e7c80804170410m66b30078l29dddd75625ad5e1@mail.gmail.com> References: <9457e7c80804161322m4ed4bb76le5bce0a94b8cdc54@mail.gmail.com><20080416223419.GF10057@phare.normalesup.org> <9457e7c80804170410m66b30078l29dddd75625ad5e1@mail.gmail.com> Message-ID: > On 17/04/2008, Alan G Isaac wrote: >> - it retains more ndarray behavior On Thu, 17 Apr 2008, St?fan van der Walt apparently wrote: > That is simply not true. The patch I sent provides everything you > asked for without breaking any ndarray behaviour, whereas your > suggestion breaks x[0] == x[0,:]. The behavior I was referring to is: iteration through the matrix would yield 1d arrays (and all their associated behavior). Sorry that was not clear. Again, please note that I have said multiple times that if iteration yields instead 1 by N matrices that can be indexed with with a single index ("vectors") I will be happy. What is more, Travis's question seems to kill my proposal, since it seems to imply that - the behavior of matrices and sparse matrices should match - that x[0] should yield a sparse matrix if x is a sparse matrix I do not use sparse matrices, so I have no opinion on these point. Cheers, Alan PS I think that it would be good behavior for x.A[0][0] to match x[0].A[0] From oliphant at enthought.com Thu Apr 17 11:11:17 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 17 Apr 2008 10:11:17 -0500 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: References: <9457e7c80804161322m4ed4bb76le5bce0a94b8cdc54@mail.gmail.com><20080416223419.GF10057@phare.normalesup.org> <9457e7c80804170410m66b30078l29dddd75625ad5e1@mail.gmail.com> Message-ID: <48076895.9020403@enthought.com> Alan G Isaac wrote: >> On 17/04/2008, Alan G Isaac wrote: >> >>> - it retains more ndarray behavior >>> > > > On Thu, 17 Apr 2008, St?fan van der Walt apparently wrote: > >> That is simply not true. The patch I sent provides everything you >> asked for without breaking any ndarray behaviour, whereas your >> suggestion breaks x[0] == x[0,:]. >> > > > The behavior I was referring to is: > iteration through the matrix would > yield 1d arrays (and all their > associated behavior). > > Sorry that was not clear. > > Again, please note that I have said > multiple times that if iteration > yields instead 1 by N matrices that > can be indexed with with a single index > ("vectors") I will be happy. > > What is more, Travis's question seems > to kill my proposal, since it seems to > imply that > I didn't intentionally mean to do that. > - the behavior of matrices and sparse matrices > should match > This is what I would like. > - that x[0] should yield a sparse matrix if x is > a sparse matrix > This is what is debatable. -Travis From wfspotz at sandia.gov Thu Apr 17 11:45:06 2008 From: wfspotz at sandia.gov (Bill Spotz) Date: Thu, 17 Apr 2008 09:45:06 -0600 Subject: [Numpy-discussion] Release of NumPy In-Reply-To: <48076895.9020403@enthought.com> References: <9457e7c80804161322m4ed4bb76le5bce0a94b8cdc54@mail.gmail.com><20080416223419.GF10057@phare.normalesup.org> <9457e7c80804170410m66b30078l29dddd75625ad5e1@mail.gmail.com> <48076895.9020403@enthought.com> Message-ID: Since I am the one who initially proposed the RowVector/ColumnVector idea, I have followed the discussion here with interest. However, since I typically deal with sparse matrices, I didn't feel like I had much to contribute to the indexing issues. But now that we are getting into sparse matrices, that has changed a little.... On Apr 17, 2008, at 9:11 AM, Travis E. Oliphant wrote: > Alan G Isaac wrote: >> >> What is more, Travis's question seems >> to kill my proposal, since it seems to >> imply that >> > I didn't intentionally mean to do that. > >> - the behavior of matrices and sparse matrices >> should match >> > This is what I would like. >> - that x[0] should yield a sparse matrix if x is >> a sparse matrix >> > This is what is debatable. So, I would expect to be able to access S[i,j], where S is a sparse matrix, and get 0 if (i,j) is not stored, and the correct value if (i,j) is stored. This would be consistent behavior with dense matrices. But what should S[i] return, (or S[:,j])? Along the lines of RowVector/ColumnVector, we could define SparseRowVector/SparseColumnVector classes, which would provide the expected behavior. Internally, they would store non-zero elements, and their corresponding indexes. Externally, v[i] (where v is one of the SparseVector classes) would just return the appropriate (zero or non-zero) value. Then S[i] -> SparseRowVector S[:,j] -> SparseColumnVector This would be consistent behavior between dense and sparse matrices and support proper M[i][j] access. I don't know what iterating over a SparseVector would mean/imply. And it provides an example of the additional capabilities Alan has been asking about. (I am assuming here a compressed-row or compressed-column storage format, but I think it could apply to other sparse matrix formats as well.) ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From zachary.pincus at yale.edu Thu Apr 17 12:02:55 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Thu, 17 Apr 2008 12:02:55 -0400 Subject: [Numpy-discussion] Fast histogram Message-ID: <4556817B-6C7C-4DBA-8E81-CE9343B6A483@yale.edu> Hi folks, I'm working on a live-video display for some microscope control tools I'm building. For this, I need a fast histogram function to work on large-ish images (1000x2000 or so) at video rate, with cycles left over for more interesting calculations (like autofocus). Now, numpy.histogram is a bit slower than I'd like, probably because it's pretty general (and of course cf. the recent discussion about its speed). I just need even bins within a set range. This is easy enough to do with a C-extension, or perhaps even cython, but before I go there, I was wondering if there's a numpy function that can help. Here's what I have in mind: def histogram(arr, bins, range): min, max = range indices = numpy.clip(((arr.astype(float) - min) * bins / (max - min)).astype(int), 0, bins-1) histogram = numpy.zeros(bins, numpy.uint32) for i in indices: histogram[i] += 1 Now, clearly, the last loop is what needs speeding up. Are there any numpy functions that can do this kind of operation? Also perhaps unnecessarily slow is the conversion of 'arr' to a float -- I do this to avoid overflow issues with integer arrays. Any advice? Should I go ahead and write this up in C (easy enough), or can I do this in numpy? Probably the indices-computation line I'll speed up with numexpr, if I use a pure-numpy approach. Thanks, Zach From charlesr.harris at gmail.com Thu Apr 17 12:18:44 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 17 Apr 2008 10:18:44 -0600 Subject: [Numpy-discussion] Fast histogram In-Reply-To: <4556817B-6C7C-4DBA-8E81-CE9343B6A483@yale.edu> References: <4556817B-6C7C-4DBA-8E81-CE9343B6A483@yale.edu> Message-ID: On Thu, Apr 17, 2008 at 10:02 AM, Zachary Pincus wrote: > Hi folks, > > I'm working on a live-video display for some microscope control tools > I'm building. For this, I need a fast histogram function to work on > large-ish images (1000x2000 or so) at video rate, with cycles left > over for more interesting calculations (like autofocus). > > Now, numpy.histogram is a bit slower than I'd like, probably because > it's pretty general (and of course cf. the recent discussion about its > speed). I just need even bins within a set range. This is easy enough > to do with a C-extension, or perhaps even cython, but before I go > there, I was wondering if there's a numpy function that can help. > > Here's what I have in mind: > > def histogram(arr, bins, range): > min, max = range > indices = numpy.clip(((arr.astype(float) - min) * bins / (max - > min)).astype(int), 0, bins-1) > histogram = numpy.zeros(bins, numpy.uint32) > for i in indices: > histogram[i] += 1 > > Now, clearly, the last loop is what needs speeding up. Are there any > numpy functions that can do this kind of operation? Also perhaps > unnecessarily slow is the conversion of 'arr' to a float -- I do this > to avoid overflow issues with integer arrays. > How about a combination of sort, followed by searchsorted right/left using the bin boundaries as keys? The difference of the two resulting vectors is the bin value. Something like: In [1]: data = arange(100) In [2]: bins = [0,10,50,70,100] In [3]: lind = data.searchsorted(bins) In [4]: print lind[1:] - lind[:-1] [10 40 20 30] This won't be as fast as a c implementation, but at least avoids the loop. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From haase at msg.ucsf.edu Thu Apr 17 12:38:22 2008 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu, 17 Apr 2008 18:38:22 +0200 Subject: [Numpy-discussion] Fast histogram In-Reply-To: References: <4556817B-6C7C-4DBA-8E81-CE9343B6A483@yale.edu> Message-ID: On Thu, Apr 17, 2008 at 6:18 PM, Charles R Harris wrote: > > > On Thu, Apr 17, 2008 at 10:02 AM, Zachary Pincus > wrote: > > Hi folks, > > > > I'm working on a live-video display for some microscope control tools > > I'm building. For this, I need a fast histogram function to work on > > large-ish images (1000x2000 or so) at video rate, with cycles left > > over for more interesting calculations (like autofocus). > > > > Now, numpy.histogram is a bit slower than I'd like, probably because > > it's pretty general (and of course cf. the recent discussion about its > > speed). I just need even bins within a set range. This is easy enough > > to do with a C-extension, or perhaps even cython, but before I go > > there, I was wondering if there's a numpy function that can help. > > > > Here's what I have in mind: > > > > def histogram(arr, bins, range): > > min, max = range > > indices = numpy.clip(((arr.astype(float) - min) * bins / (max - > > min)).astype(int), 0, bins-1) > > histogram = numpy.zeros(bins, numpy.uint32) > > for i in indices: > > histogram[i] += 1 > > > > Now, clearly, the last loop is what needs speeding up. Are there any > > numpy functions that can do this kind of operation? Also perhaps > > unnecessarily slow is the conversion of 'arr' to a float -- I do this > > to avoid overflow issues with integer arrays. > > > > How about a combination of sort, followed by searchsorted right/left using > the bin boundaries as keys? The difference of the two resulting vectors is > the bin value. Something like: > > In [1]: data = arange(100) > > In [2]: bins = [0,10,50,70,100] > > In [3]: lind = data.searchsorted(bins) > > In [4]: print lind[1:] - lind[:-1] > [10 40 20 30] > > This won't be as fast as a c implementation, but at least avoids the loop. > > Chuck How many bits per pixel does your camera actually generate !? If its for example a 12 bit camera, you could just fill in directly into 4096 preallocated bins. You would not need any sorting !! That's what I did for a 16 bit camera -- but I wrote it in C and I had 4 cameras at 30 Hz. -Sebastian Haase From zachary.pincus at yale.edu Thu Apr 17 12:46:27 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Thu, 17 Apr 2008 12:46:27 -0400 Subject: [Numpy-discussion] Fast histogram In-Reply-To: References: <4556817B-6C7C-4DBA-8E81-CE9343B6A483@yale.edu> Message-ID: Hi, > How about a combination of sort, followed by searchsorted right/left > using the bin boundaries as keys? The difference of the two > resulting vectors is the bin value. Something like: > > In [1]: data = arange(100) > > In [2]: bins = [0,10,50,70,100] > > In [3]: lind = data.searchsorted(bins) > > In [4]: print lind[1:] - lind[:-1] > [10 40 20 30] > > This won't be as fast as a c implementation, but at least avoids the > loop. This is, more or less, what the current numpy.histogram does, no? I was hoping to avoid the O(n log n) sorting, because the image arrays are pretty big, and numpy.histogram doesn't get close to video rate for me... Perhaps, though, some of the slow-down from numpy.histogram is from other overhead, and not the sorting. I'll try this, but I think I'll probably just have to write the c loop... Zach From zachary.pincus at yale.edu Thu Apr 17 12:54:18 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Thu, 17 Apr 2008 12:54:18 -0400 Subject: [Numpy-discussion] Fast histogram In-Reply-To: References: <4556817B-6C7C-4DBA-8E81-CE9343B6A483@yale.edu> Message-ID: <293D57C0-44AD-4099-A428-F08CA3E04C60@yale.edu> Hi, and thanks for the suggestion! > How many bits per pixel does your camera actually generate !? > If its for example a 12 bit camera, you could just fill in directly > into 4096 preallocated bins. > You would not need any sorting !! > That's what I did for a 16 bit camera -- but I wrote it in C and I had > 4 cameras at 30 Hz. That approach avoids the bin-index calculation line: indices = numpy.clip(((array.astype(float) - min) * bins / (max - min)).astype(int), 0, bins-1) But even if indices = array, one still needs to do something like: for index in indices: histogram[index] += 1 Which is slow in python and fast in C. I'm guessing that there's no utility function in numpy that does a loop like this? If so, that would be handy, but if now, I think I need to dig out the numpy book and write a little extension... Zach From haase at msg.ucsf.edu Thu Apr 17 13:00:42 2008 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu, 17 Apr 2008 19:00:42 +0200 Subject: [Numpy-discussion] Fast histogram In-Reply-To: <293D57C0-44AD-4099-A428-F08CA3E04C60@yale.edu> References: <4556817B-6C7C-4DBA-8E81-CE9343B6A483@yale.edu> <293D57C0-44AD-4099-A428-F08CA3E04C60@yale.edu> Message-ID: On Thu, Apr 17, 2008 at 6:54 PM, Zachary Pincus wrote: > Hi, and thanks for the suggestion! > > > > How many bits per pixel does your camera actually generate !? > > If its for example a 12 bit camera, you could just fill in directly > > into 4096 preallocated bins. > > You would not need any sorting !! > > That's what I did for a 16 bit camera -- but I wrote it in C and I had > > 4 cameras at 30 Hz. > > That approach avoids the bin-index calculation line: > indices = numpy.clip(((array.astype(float) - min) * bins / (max - > > min)).astype(int), 0, bins-1) > > But even if indices = array, one still needs to do something like: > for index in indices: histogram[index] += 1 > > Which is slow in python and fast in C. > > I'm guessing that there's no utility function in numpy that does a > loop like this? If so, that would be handy, but if now, I think I need > to dig out the numpy book and write a little extension... > > I thought of a broadcasting approach... what are the chances that a simple bins[:] = 0 bins[ img.flat ] += 1 works !? Or something along those lines .... -Sebastian From zachary.pincus at yale.edu Thu Apr 17 13:09:13 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Thu, 17 Apr 2008 13:09:13 -0400 Subject: [Numpy-discussion] Fast histogram In-Reply-To: References: <4556817B-6C7C-4DBA-8E81-CE9343B6A483@yale.edu> <293D57C0-44AD-4099-A428-F08CA3E04C60@yale.edu> Message-ID: >> But even if indices = array, one still needs to do something like: >> for index in indices: histogram[index] += 1 >> >> Which is slow in python and fast in C. >> >> > I thought of a broadcasting approach... what are the chances that a > simple > > bins[:] = 0 > bins[ img.flat ] += 1 That doesn't work cumulatively, it seems. Some bins get a value of '1', but it never gets incremented past that. It was worth a try, though... in some cases in-place updating seems to work this way (which is usually a source of confusion on this list, not the desired result!) Zach From emsellem at obs.univ-lyon1.fr Thu Apr 17 13:11:47 2008 From: emsellem at obs.univ-lyon1.fr (Eric Emsellem) Date: Thu, 17 Apr 2008 19:11:47 +0200 Subject: [Numpy-discussion] /usr/bin/python: double free or corruption Message-ID: <480784D3.40405@obs.univ-lyon1.fr> Hi, I just installed a new openSuse 10.3, python, numpy, etc, on a 32 bit PC (using the rpm provide on the science Suse repository) When using numpy.fromfile, I get a glibc error when it tries to read something which is not there (end of the file). So for example with a file "tmp" which has nothing in it: > file = open("tmp","rb") > import numpy as num > num.fromfile(file,dtype=num.int16,count=1) 1 items requested but only 0 read *** glibc detected *** /usr/bin/python: double free or corruption (fasttop): 0x08a5ac78 *** but on my other machine it does it well and just answers: 1 items requested but only 0 read Out[102]:array([], dtype=int16) Any hint why this fails? thanks! Eric extra INFO : ============ The machine where it does not work is 32bit, and: matplotlib version 0.91.2 platform is linux2 numerix numpy 1.0.4 backend GTKAgg version 2.10.6 Python 2.5.1 (r251:54863, Jan 10 2008, 18:01:57 IPython 0.8.1 -- An enhanced Interactive Python. From lee.j.joon at gmail.com Thu Apr 17 13:12:36 2008 From: lee.j.joon at gmail.com (Jae-Joon Lee) Date: Thu, 17 Apr 2008 13:12:36 -0400 Subject: [Numpy-discussion] Fast histogram In-Reply-To: <293D57C0-44AD-4099-A428-F08CA3E04C60@yale.edu> References: <4556817B-6C7C-4DBA-8E81-CE9343B6A483@yale.edu> <293D57C0-44AD-4099-A428-F08CA3E04C60@yale.edu> Message-ID: <6e8d907b0804171012q69fb588do35335110f0ddd21d@mail.gmail.com> > But even if indices = array, one still needs to do something like: > for index in indices: histogram[index] += 1 > > Which is slow in python and fast in C. > > I'm guessing that there's no utility function in numpy that does a > loop like this? If so, that would be handy, but if now, I think I need > to dig out the numpy book and write a little extension... > numpy.bincount? Docstring: bincount(x,weights=None) Return the number of occurrences of each value in x. x must be a list of non-negative integers. The output, b[i], represents the number of times that i is found in x. If weights is specified, every occurrence of i at a position p contributes weights[p] instead of 1. See also: histogram, digitize, unique. -JJ From p.e.creasey.00 at cantab.net Thu Apr 17 13:33:00 2008 From: p.e.creasey.00 at cantab.net (Peter Creasey) Date: Thu, 17 Apr 2008 18:33:00 +0100 Subject: [Numpy-discussion] Fast histogram Message-ID: <6be8b94a0804171033l64e92450s5e1c22cb858a19d@mail.gmail.com> On 17/04/2008, Zachary Pincus wrote: > But even if indices = array, one still needs to do something like: > for index in indices: histogram[index] += 1 > > Which is slow in python and fast in C. > I haven't tried this, but if you want the sum in C you could do for x in unique(indices): histogram[x] = (indices==x).sum() Of course, this just replaces an O(N log N) algorithm by an O(N * M) (M is the number of bins), which is only going to be worth for very small M. Peter From oliphant at enthought.com Thu Apr 17 13:40:34 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 17 Apr 2008 12:40:34 -0500 Subject: [Numpy-discussion] /usr/bin/python: double free or corruption In-Reply-To: <480784D3.40405@obs.univ-lyon1.fr> References: <480784D3.40405@obs.univ-lyon1.fr> Message-ID: <48078B92.6000107@enthought.com> Eric Emsellem wrote: > Hi, > > I just installed a new openSuse 10.3, python, numpy, etc, on a 32 bit PC (using > the rpm provide on the science Suse repository) > > When using numpy.fromfile, I get a glibc error when it tries to read something > which is not there (end of the file). > So for example with a file "tmp" which has nothing in it: > > >> file = open("tmp","rb") >> import numpy as num >> num.fromfile(file,dtype=num.int16,count=1) >> > > 1 items requested but only 0 read > *** glibc detected *** /usr/bin/python: double free or corruption (fasttop): > 0x08a5ac78 *** > > > but on my other machine it does it well and just answers: > > 1 items requested but only 0 read > Out[102]:array([], dtype=int16) > > Any hint why this fails? > > This is fixed in SVN. The problem is the behavior of realloc when size is 0 is not consistent across platforms. In the one case it does not free the original memory, but in the other it frees the memory. -Travis O. From p.e.creasey.00 at googlemail.com Thu Apr 17 13:45:57 2008 From: p.e.creasey.00 at googlemail.com (Peter Creasey) Date: Thu, 17 Apr 2008 18:45:57 +0100 Subject: [Numpy-discussion] Fwd: Fast histogram In-Reply-To: <6be8b94a0804171033l64e92450s5e1c22cb858a19d@mail.gmail.com> References: <6be8b94a0804171033l64e92450s5e1c22cb858a19d@mail.gmail.com> Message-ID: <6be8b94a0804171045w464bf3efh895c3d66934a61b@mail.gmail.com> On 17/04/2008, Zachary Pincus wrote: > But even if indices = array, one still needs to do something like: > for index in indices: histogram[index] += 1 > > Which is slow in python and fast in C. > I haven't tried this, but if you want the sum in C you could do for x in unique(indices): histogram[x] = (indices==x).sum() Of course, this just replaces an O(N log N) algorithm by an O(N * M) (M is the number of bins), which is only going to be worth it for very small M. Peter From strawman at astraw.com Thu Apr 17 13:47:32 2008 From: strawman at astraw.com (Andrew Straw) Date: Thu, 17 Apr 2008 10:47:32 -0700 Subject: [Numpy-discussion] Fast histogram In-Reply-To: References: <4556817B-6C7C-4DBA-8E81-CE9343B6A483@yale.edu> Message-ID: <48078D34.1030705@astraw.com> Hi Zach, I have a similar loop which I wrote using scipy.weave. This was my first foray into weave, and I had to dig through the intermediate C sources to find the macros that did the indexing in the way I make use of here, but this snipped may get you started. There are 2 functions, which each do the same thing, but one is in python, the other is in C. Note that this is for a 3D histogram -- presumably you could remove B and C from this example. I'm sure there are better (more documented) ways to do this using weave -- but I had this code written, it works, and it appears it may be useful to you... (Sorry it's not documented, however.) -Andrew Zachary Pincus wrote: > Hi, > > >> How about a combination of sort, followed by searchsorted right/left >> using the bin boundaries as keys? The difference of the two >> resulting vectors is the bin value. Something like: >> >> In [1]: data = arange(100) >> >> In [2]: bins = [0,10,50,70,100] >> >> In [3]: lind = data.searchsorted(bins) >> >> In [4]: print lind[1:] - lind[:-1] >> [10 40 20 30] >> >> This won't be as fast as a c implementation, but at least avoids the >> loop. >> > > This is, more or less, what the current numpy.histogram does, no? I > was hoping to avoid the O(n log n) sorting, because the image arrays > are pretty big, and numpy.histogram doesn't get close to video rate > for me... > > Perhaps, though, some of the slow-down from numpy.histogram is from > other overhead, and not the sorting. I'll try this, but I think I'll > probably just have to write the c loop... > > Zach > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- A non-text attachment was scrubbed... Name: for_zach.py Type: text/x-python Size: 719 bytes Desc: not available URL: From zachary.pincus at yale.edu Thu Apr 17 14:11:21 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Thu, 17 Apr 2008 14:11:21 -0400 Subject: [Numpy-discussion] Fast histogram In-Reply-To: <6e8d907b0804171012q69fb588do35335110f0ddd21d@mail.gmail.com> References: <4556817B-6C7C-4DBA-8E81-CE9343B6A483@yale.edu> <293D57C0-44AD-4099-A428-F08CA3E04C60@yale.edu> <6e8d907b0804171012q69fb588do35335110f0ddd21d@mail.gmail.com> Message-ID: Hello, >> But even if indices = array, one still needs to do something like: >> for index in indices: histogram[index] += 1 > numpy.bincount? That is indeed what I was looking for! I knew I'd seen such a function. However, the speed is a bit disappointing. I guess the sorting isn't too much of a penalty: def histogram(array, bins, range): min, max = range indices = numpy.clip(((array.astype(float) - min) * bins / (max - min)).astype(int), 0, bins-1).flat return numpy.bincount(indices) import numexpr def histogram_numexpr(array, bins, range): min, max = range min = float(min) max = float(max) indices = numexpr.evaluate('(array - min) * bins / (max - min)') indices = numpy.clip(indices.astype(int), 0, bins-1).flat return numpy.bincount(indices) >>> arr.shape (1300, 1030) >>> timeit numpy.histogram(arr, 12, [0, 5000]) 10 loops, best of 3: 99.9 ms per loop >>> timeit histogram(arr, 12, [0, 5000]) 10 loops, best of 3: 127 ms per loop >>> timeit histogram_numexpr(arr, 12, [0, 5000]) 10 loops, best of 3: 109 ms per loop >>> timeit numpy.histogram(arr, 5000, [0, 5000]) 10 loops, best of 3: 111 ms per loop >>> timeit histogram(arr, 5000, [0, 5000]) 10 loops, best of 3: 127 ms per loop >>> timeit histogram_numexpr(arr, 5000, [0, 5000]) 10 loops, best of 3: 108 ms per loop So, they're all quite close, and it seems that numpy.histogram is the definite winner. Huh. I guess I will have to go to C or maybe weave to get up to video-rate, unless folks can suggest some further optimizations... Zach From efiring at hawaii.edu Thu Apr 17 14:16:15 2008 From: efiring at hawaii.edu (Eric Firing) Date: Thu, 17 Apr 2008 08:16:15 -1000 Subject: [Numpy-discussion] broken numpy.core.tests.test_multiarray.TestScalarIndexing.SetUp Message-ID: <480793EF.1000006@hawaii.edu> While trying to figure out how to write tests for "take", I stumbled across this in numpy.core.tests.test_multiarray.py: class TestScalarIndexing(NumpyTestCase): def setUp(self): self.d = array([0,1])[0], def check_ellipsis_subscript(self): a, = self.d self.failUnlessEqual(a[...], 0) self.failUnlessEqual(a[...].shape,()) setUp is clearly broken, but the numpy test suite still runs happily. This seems odd. Eric From zachary.pincus at yale.edu Thu Apr 17 14:17:03 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Thu, 17 Apr 2008 14:17:03 -0400 Subject: [Numpy-discussion] Fast histogram In-Reply-To: References: <4556817B-6C7C-4DBA-8E81-CE9343B6A483@yale.edu> <293D57C0-44AD-4099-A428-F08CA3E04C60@yale.edu> <6e8d907b0804171012q69fb588do35335110f0ddd21d@mail.gmail.com> Message-ID: <51471265-F785-4DC4-836F-86E83B49C5C9@yale.edu> Combining Sebastian and Jae-Joon's suggestions, I have something that might work: >>> timeit numpy.bincount(array.flat) 10 loops, best of 3: 28.2 ms per loop This is close enough to video-rate... And I can then combine bins as needed to get a particular bin count/range after the fact. Thanks, everyone, Zach From efiring at hawaii.edu Thu Apr 17 14:21:04 2008 From: efiring at hawaii.edu (Eric Firing) Date: Thu, 17 Apr 2008 08:21:04 -1000 Subject: [Numpy-discussion] broken numpy.core.tests.test_multiarray.TestScalarIndexing.SetUp In-Reply-To: <480793EF.1000006@hawaii.edu> References: <480793EF.1000006@hawaii.edu> Message-ID: <48079510.6080704@hawaii.edu> Arg! Cancel that! I didn't look carefully enough. How embarrassing! Sorry for the noise. Eric Eric Firing wrote: > While trying to figure out how to write tests for "take", I stumbled > across this in numpy.core.tests.test_multiarray.py: > > class TestScalarIndexing(NumpyTestCase): > def setUp(self): > self.d = array([0,1])[0], > > def check_ellipsis_subscript(self): > a, = self.d > self.failUnlessEqual(a[...], 0) > self.failUnlessEqual(a[...].shape,()) > > > setUp is clearly broken, but the numpy test suite still runs happily. > This seems odd. > > Eric > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From santanu.chatter at gmail.com Thu Apr 17 15:02:36 2008 From: santanu.chatter at gmail.com (Santanu Chatterjee) Date: Thu, 17 Apr 2008 15:02:36 -0400 Subject: [Numpy-discussion] Matrix vs ndarray Message-ID: Hi Numpy users, I used MATLAB to do numerical calculations for a long time. Recently I am digging into python and numpy. I am wondering about the following question : 1) What is the difference between ndarray and matrix in numpy? My idea is that having N-dimensional array is sufficient (of course a MATLAB users point of view). If anyone can provide some idea, I will appreciate it. Thanks & regards, Santanu -------------- next part -------------- An HTML attachment was scrubbed... URL: From doutriaux1 at llnl.gov Thu Apr 17 15:12:06 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Thu, 17 Apr 2008 12:12:06 -0700 Subject: [Numpy-discussion] bz2 Message-ID: <4807A106.6060406@llnl.gov> Hi, Just so you know, yesterday i was doing some test on a fresh ubuntu installation, trying to figure out the external minimum dependencies for our system. I built (from sources) python (2.5.2) then numpy. All seemed ok, but when importing numpy i had an error, trying to import module bz2 Turns out Python didn't build with bz2 because the bz2 devel package wasn't in. I'm just reporting this to you guys, has you may want to add a test for bz2 in your setup.py before going ahead and build the whole thing. Hope this helps, C. From aisaac at american.edu Thu Apr 17 15:16:43 2008 From: aisaac at american.edu (Alan G Isaac) Date: Thu, 17 Apr 2008 15:16:43 -0400 Subject: [Numpy-discussion] Matrix vs ndarray In-Reply-To: References: Message-ID: On Thu, 17 Apr 2008, Santanu Chatterjee apparently wrote: > 1) What is the difference between ndarray and matrix in > numpy? My idea is that having N-dimensional array is > sufficient (of course a MATLAB users point of view). If > anyone can provide some idea, I will appreciate it. Matrices are 2d only. The * are ** operators have matrix definitions instead of element-by-element definitions. Matrix subarrays are always 2d ... Matrices have an A attribute to return an array view of the data. (Useful for element by element operations.) hth, Alan Isaac From wbaxter at gmail.com Thu Apr 17 15:14:03 2008 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 18 Apr 2008 04:14:03 +0900 Subject: [Numpy-discussion] Matrix vs ndarray In-Reply-To: References: Message-ID: You might find out a lot from reading through this page: http://www.scipy.org/NumPy_for_Matlab_Users What I think that doesn't say is why the two classes are needed in NumPy. Basically, the reason for that is that Matlab has .* and * which mean different things, but Python only has the one * operator. Some people think that should mean .*, some want it to mean *. So NumPy offers two different classes with different behavior instead of two different operators. --bb On Fri, Apr 18, 2008 at 4:02 AM, Santanu Chatterjee wrote: > Hi Numpy users, > I used MATLAB to do numerical calculations for a long time. Recently I > am digging into python and numpy. I am wondering about the following > question : > > 1) What is the difference between ndarray and matrix in numpy? My idea is > that having N-dimensional array is sufficient (of course a MATLAB users > point of view). If anyone can provide some idea, I will appreciate it. > > Thanks & regards, > Santanu > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From peridot.faceted at gmail.com Thu Apr 17 15:14:56 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 17 Apr 2008 21:14:56 +0200 Subject: [Numpy-discussion] Matrix vs ndarray In-Reply-To: References: Message-ID: On 17/04/2008, Santanu Chatterjee wrote: > Hi Numpy users, > I used MATLAB to do numerical calculations for a long time. Recently I > am digging into python and numpy. I am wondering about the following > question : > > 1) What is the difference between ndarray and matrix in numpy? My idea is > that having N-dimensional array is sufficient (of course a MATLAB users > point of view). If anyone can provide some idea, I will appreciate it. The difference comes about because python has a paucity of operators. In particular, there is only one multiplication operator. Thus for ndarrays "*" does elementwise multiplication, and to get matrix multiplication you must use the function "dot". Similarly "**" does elementwise exponentiation, and to get matrix exponentiation you must use the function "matrix_power". Since some people find this inconvenient, the "matrix" class exists. It represents arrays that are always two-dimensional, and for them "*" does matrix multiplication. The only difference is how the operators (and certain indexing operations) are interpreted. If you want to interpret a matrix M as an array, just use "M.A"; it you want to interpret an array X as a matrix, do "matrix(X)". In neither case is data copied. Personally, I do not ever use matrices; I find "dot" is quite convenient enough for me. Moreover the numpy standard library has an undetermined number of bugs where standard functions acting on matrices return the correct values but in the form of arrays instead; thus matrix users occasionally find that they have inadvertently (and silently) transformed their matrices back into arrays. Anne From robert.kern at gmail.com Thu Apr 17 16:02:33 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 17 Apr 2008 15:02:33 -0500 Subject: [Numpy-discussion] Distutils: using different linker options for c++ and c code In-Reply-To: <480735D8.9020102@ar.media.kyoto-u.ac.jp> References: <480735D8.9020102@ar.media.kyoto-u.ac.jp> Message-ID: <3d375d730804171302v288e526ep3061af7e59a89ce3@mail.gmail.com> On Thu, Apr 17, 2008 at 6:34 AM, David Cournapeau wrote: > Hi, > > I cannot seem to make scipy.sparse works on windows with mingw32, > and it seems related to the msvc runtime. I always get segfaults in the > sparsetools module (c++); if I build the module manually to control the > options precisely and remove the -lmsvc71 from the build command, I > cannot reproduce the segfault anymore (but the segfault does not happen > 100 % of the time, so it is hard to be sure) > > So is there a way in distutils to make the difference between C++ > and C: for extension with C, use the msvc, for extensions in C++, do not > use it ? (The obvious hack consisting in remove msvc alltogether does > not work: it crahses in scipy.io, which is linked to incompatible FILE > structures between mingw and VS, I guess). Ah, this problem again. The build of mingw that you are using were written with msvcrt in mind. For the most part they are compatible with msvcr71, but there are a few places where they reference a table that is different between the two runtimes, namely iostream in C++ and ischar() in C. Consequently, there are some extension modules built with mingw which work with msvcrt because they need iostream, some with msvcr71 because they need to pass FILE pointers, and probably some which won't work with either. The core problem won't be fixed until mingw writes their headers for msvcr71. They may have; it looks like they just released some new builds this month. It would be worth checking these out. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Thu Apr 17 16:16:44 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 17 Apr 2008 15:16:44 -0500 Subject: [Numpy-discussion] bz2 In-Reply-To: <4807A106.6060406@llnl.gov> References: <4807A106.6060406@llnl.gov> Message-ID: <3d375d730804171316p1777bbfbwa2c80c8901bac10d@mail.gmail.com> On Thu, Apr 17, 2008 at 2:12 PM, Charles Doutriaux wrote: > Hi, > > Just so you know, yesterday i was doing some test on a fresh ubuntu > installation, trying to figure out the external minimum dependencies for > our system. > > I built (from sources) python (2.5.2) then numpy. > > All seemed ok, but when importing numpy i had an error, trying to import > module bz2 Fixed in r5040. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Thu Apr 17 16:23:57 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 17 Apr 2008 15:23:57 -0500 Subject: [Numpy-discussion] broken numpy.core.tests.test_multiarray.TestScalarIndexing.SetUp In-Reply-To: <48079510.6080704@hawaii.edu> References: <480793EF.1000006@hawaii.edu> <48079510.6080704@hawaii.edu> Message-ID: <3d375d730804171323pad3806an8c0dab1afbddc2a7@mail.gmail.com> On Thu, Apr 17, 2008 at 1:21 PM, Eric Firing wrote: > Arg! Cancel that! I didn't look carefully enough. How embarrassing! > Sorry for the noise. Don't apologize. That is very odd code. Stefan, is there a reason to form a 1-item tuple then do 1-item tuple unpacking everywhere? The test works the same after removing the extraneous commas. Anyways, I've checked that in. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From peridot.faceted at gmail.com Thu Apr 17 16:34:03 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 17 Apr 2008 22:34:03 +0200 Subject: [Numpy-discussion] broken numpy.core.tests.test_multiarray.TestScalarIndexing.SetUp In-Reply-To: <3d375d730804171323pad3806an8c0dab1afbddc2a7@mail.gmail.com> References: <480793EF.1000006@hawaii.edu> <48079510.6080704@hawaii.edu> <3d375d730804171323pad3806an8c0dab1afbddc2a7@mail.gmail.com> Message-ID: On 17/04/2008, Robert Kern wrote: > On Thu, Apr 17, 2008 at 1:21 PM, Eric Firing wrote: > > Arg! Cancel that! I didn't look carefully enough. How embarrassing! > > Sorry for the noise. > > > Don't apologize. That is very odd code. Stefan, is there a reason to > form a 1-item tuple then do 1-item tuple unpacking everywhere? The > test works the same after removing the extraneous commas. Anyways, > I've checked that in. This is not necessarily a justification, but many tests construct tuples of test objects which are then unpacked at the beginning of every function. This is not unreasonable when multiple objects are present: class TestSomething(NumpyTestCase): def setUp(self): A = array([1,2,3]) B = array([4,5,6]) self.d = A, B def test_something(self): A, B = self.d assert_not_equal(A,B) It's a little less cumbersome than using self.A and self.B inside each test case. Does it make sense to use a length-1 tuple when there's only one piece of test data, just for consistency? Anne From robert.kern at gmail.com Thu Apr 17 16:51:17 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 17 Apr 2008 15:51:17 -0500 Subject: [Numpy-discussion] numpy.testing.raises lacks the definition of make_decorator In-Reply-To: References: Message-ID: <3d375d730804171351r1345eda2l903efff2c875d89d@mail.gmail.com> On Thu, Apr 17, 2008 at 9:07 AM, Matthieu Brucher wrote: > Hi, > > I'm trying to use raises with numpy, but when I call nosetests, it tells me > that make_decorator is not defined. And indeed, it is not defined or > imported in this file, but in nose/tools.py Fixed in r5042. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Thu Apr 17 16:53:04 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 17 Apr 2008 15:53:04 -0500 Subject: [Numpy-discussion] broken numpy.core.tests.test_multiarray.TestScalarIndexing.SetUp In-Reply-To: References: <480793EF.1000006@hawaii.edu> <48079510.6080704@hawaii.edu> <3d375d730804171323pad3806an8c0dab1afbddc2a7@mail.gmail.com> Message-ID: <3d375d730804171353v3665c9b2hdafe27b7d22de6ac@mail.gmail.com> On Thu, Apr 17, 2008 at 3:34 PM, Anne Archibald wrote: > On 17/04/2008, Robert Kern wrote: > > On Thu, Apr 17, 2008 at 1:21 PM, Eric Firing wrote: > > > Arg! Cancel that! I didn't look carefully enough. How embarrassing! > > > Sorry for the noise. > > > > Don't apologize. That is very odd code. Stefan, is there a reason to > > form a 1-item tuple then do 1-item tuple unpacking everywhere? The > > test works the same after removing the extraneous commas. Anyways, > > I've checked that in. > > This is not necessarily a justification, but many tests construct > tuples of test objects which are then unpacked at the beginning of > every function. This is not unreasonable when multiple objects are > present: > > class TestSomething(NumpyTestCase): > def setUp(self): > A = array([1,2,3]) > B = array([4,5,6]) > self.d = A, B > > def test_something(self): > A, B = self.d > assert_not_equal(A,B) > > It's a little less cumbersome than using self.A and self.B inside each > test case. > > Does it make sense to use a length-1 tuple when there's only one piece > of test data, just for consistency? I don't think so. A trailing comma is too easy to miss and looks like an error when it isn't missed. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From matthieu.brucher at gmail.com Thu Apr 17 16:58:35 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Thu, 17 Apr 2008 22:58:35 +0200 Subject: [Numpy-discussion] numpy.testing.raises lacks the definition of make_decorator In-Reply-To: <3d375d730804171351r1345eda2l903efff2c875d89d@mail.gmail.com> References: <3d375d730804171351r1345eda2l903efff2c875d89d@mail.gmail.com> Message-ID: Thank you ! Matthieu 2008/4/17, Robert Kern : > > On Thu, Apr 17, 2008 at 9:07 AM, Matthieu Brucher > wrote: > > Hi, > > > > I'm trying to use raises with numpy, but when I call nosetests, it tells > me > > that make_decorator is not defined. And indeed, it is not defined or > > imported in this file, but in nose/tools.py > > > Fixed in r5042. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Thu Apr 17 18:40:04 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 18 Apr 2008 00:40:04 +0200 Subject: [Numpy-discussion] broken numpy.core.tests.test_multiarray.TestScalarIndexing.SetUp In-Reply-To: <3d375d730804171323pad3806an8c0dab1afbddc2a7@mail.gmail.com> References: <480793EF.1000006@hawaii.edu> <48079510.6080704@hawaii.edu> <3d375d730804171323pad3806an8c0dab1afbddc2a7@mail.gmail.com> Message-ID: <9457e7c80804171540j6e9b1567ye3d3eff1d5c6f49@mail.gmail.com> On 17/04/2008, Robert Kern wrote: > On Thu, Apr 17, 2008 at 1:21 PM, Eric Firing wrote: > > Arg! Cancel that! I didn't look carefully enough. How embarrassing! > > Sorry for the noise. > > > Don't apologize. That is very odd code. Stefan, is there a reason to > form a 1-item tuple then do 1-item tuple unpacking everywhere? Did I write that code? I agree that formatting is odd. Cheers St?fan From robert.kern at gmail.com Thu Apr 17 18:43:10 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 17 Apr 2008 17:43:10 -0500 Subject: [Numpy-discussion] broken numpy.core.tests.test_multiarray.TestScalarIndexing.SetUp In-Reply-To: <9457e7c80804171540j6e9b1567ye3d3eff1d5c6f49@mail.gmail.com> References: <480793EF.1000006@hawaii.edu> <48079510.6080704@hawaii.edu> <3d375d730804171323pad3806an8c0dab1afbddc2a7@mail.gmail.com> <9457e7c80804171540j6e9b1567ye3d3eff1d5c6f49@mail.gmail.com> Message-ID: <3d375d730804171543l173a3e70v3f2715c7aab94b58@mail.gmail.com> On Thu, Apr 17, 2008 at 5:40 PM, St?fan van der Walt wrote: > On 17/04/2008, Robert Kern wrote: > > > On Thu, Apr 17, 2008 at 1:21 PM, Eric Firing wrote: > > > Arg! Cancel that! I didn't look carefully enough. How embarrassing! > > > Sorry for the noise. > > > > > > Don't apologize. That is very odd code. Stefan, is there a reason to > > form a 1-item tuple then do 1-item tuple unpacking everywhere? > > Did I write that code? I agree that formatting is odd. Sorry, I just looked at svn blame rather than the log message which states that you just checked in Anne's patch. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From peridot.faceted at gmail.com Thu Apr 17 19:26:41 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 17 Apr 2008 19:26:41 -0400 Subject: [Numpy-discussion] broken numpy.core.tests.test_multiarray.TestScalarIndexing.SetUp In-Reply-To: <9457e7c80804171540j6e9b1567ye3d3eff1d5c6f49@mail.gmail.com> References: <480793EF.1000006@hawaii.edu> <48079510.6080704@hawaii.edu> <3d375d730804171323pad3806an8c0dab1afbddc2a7@mail.gmail.com> <9457e7c80804171540j6e9b1567ye3d3eff1d5c6f49@mail.gmail.com> Message-ID: On 17/04/2008, St?fan van der Walt wrote: > On 17/04/2008, Robert Kern wrote: > > > On Thu, Apr 17, 2008 at 1:21 PM, Eric Firing wrote: > > > Arg! Cancel that! I didn't look carefully enough. How embarrassing! > > > Sorry for the noise. > > > > > > Don't apologize. That is very odd code. Stefan, is there a reason to > > form a 1-item tuple then do 1-item tuple unpacking everywhere? > > > Did I write that code? I agree that formatting is odd. Oh! Actually I think that one's my fault. Comes of being too minimal in modifying one of your test cases. Anne From robert.kern at gmail.com Thu Apr 17 22:22:52 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 17 Apr 2008 21:22:52 -0500 Subject: [Numpy-discussion] Windows/numpy1.0.4 memmap & astype produce loads of warnings on delete In-Reply-To: <5AF149DBB6DFE24AA3F4F53201E539AD020A5742@itcnt24.itc.nl> References: <5AF149DBB6DFE24AA3F4F53201E539AD020A5742@itcnt24.itc.nl> Message-ID: <3d375d730804171922l3f3c8137o89657ff6daf32f8d@mail.gmail.com> On Thu, Apr 17, 2008 at 4:22 AM, Wim Bakker wrote: > Robert, > > I've replaced my C:\Python24\Lib\site-packages\numpy\core\memmap.py > with the one from http://svn.scipy.org/svn/numpy/trunk/numpy/core/ > and that fixes the problem. > > Thanks for your quick help! > > When is the new version of numpy due? Not too long from now. A couple weeks if not sooner. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Thu Apr 17 22:36:04 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 18 Apr 2008 11:36:04 +0900 Subject: [Numpy-discussion] Distutils: using different linker options for c++ and c code In-Reply-To: <3d375d730804171302v288e526ep3061af7e59a89ce3@mail.gmail.com> References: <480735D8.9020102@ar.media.kyoto-u.ac.jp> <3d375d730804171302v288e526ep3061af7e59a89ce3@mail.gmail.com> Message-ID: <48080914.9080004@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > > Ah, this problem again. The build of mingw that you are using were > written with msvcrt in mind. For the most part they are compatible > with msvcr71, but there are a few places where they reference a table > that is different between the two runtimes, namely iostream in C++ and > ischar() in C. Do you know by any chance any document/email which talks about that ? The only things I found were very general (I know file related do not work, but that's it). > Consequently, there are some extension modules built > with mingw which work with msvcrt because they need iostream, some > with msvcr71 because they need to pass FILE pointers, and probably > some which won't work with either. The core problem won't be fixed > until mingw writes their headers for msvcr71. They may have; it looks > like they just released some new builds this month. It would be worth > checking these out. > Are you talking about the 3.* or the 4.* releases ? The new 3.* release seems to only contain a new release note (all the files except the release note have 20060117-2 in their names). I also found an unofficial installer for gcc 4.*, which at least claim to care about python: http://www.develer.com/oss/GccWinBinaries The problems with the new official (but beta) 4.* builds is the lack of fortran support. For some reasons, building gfortran on windows does not sound really appealing :) Also, I noticed when developing numscons that numpy (implicetely ?) relied on buggy mingw headers, bugs which were fixed in 4.* and hence did not work forr numpy (nothing serious, though; it should be easily fixable in distutils). Thank you for those information, I will dig a bit deeper. Should we consider this as a major blocker for numpy as well (since it would not be possible to build a working scipy with numpy 1.1.0 ?) ? David From robert.kern at gmail.com Thu Apr 17 22:56:10 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 17 Apr 2008 21:56:10 -0500 Subject: [Numpy-discussion] Distutils: using different linker options for c++ and c code In-Reply-To: <48080914.9080004@ar.media.kyoto-u.ac.jp> References: <480735D8.9020102@ar.media.kyoto-u.ac.jp> <3d375d730804171302v288e526ep3061af7e59a89ce3@mail.gmail.com> <48080914.9080004@ar.media.kyoto-u.ac.jp> Message-ID: <3d375d730804171956h2e2482e4w35e9b30b534377ee@mail.gmail.com> On Thu, Apr 17, 2008 at 9:36 PM, David Cournapeau wrote: > Robert Kern wrote: > > > > Ah, this problem again. The build of mingw that you are using were > > written with msvcrt in mind. For the most part they are compatible > > with msvcr71, but there are a few places where they reference a table > > that is different between the two runtimes, namely iostream in C++ and > > ischar() in C. > > Do you know by any chance any document/email which talks about that ? > The only things I found were very general (I know file related do not > work, but that's it). Only my own emails. I found this out by largely experimentation and reading the header files. > > Consequently, there are some extension modules built > > with mingw which work with msvcrt because they need iostream, some > > with msvcr71 because they need to pass FILE pointers, and probably > > some which won't work with either. The core problem won't be fixed > > until mingw writes their headers for msvcr71. They may have; it looks > > like they just released some new builds this month. It would be worth > > checking these out. > > Are you talking about the 3.* or the 4.* releases ? > The new 3.* release seems to only contain a new release note (all the > files except the release note have 20060117-2 in their names). Ah, I missed that. It looks like it's just a re-release of the 2006 build with a bugfix for Vista. > I also found an unofficial installer for gcc 4.*, which at least claim > to care about python: > > http://www.develer.com/oss/GccWinBinaries > > The problems with the new official (but beta) 4.* builds is the lack of > fortran support. For some reasons, building gfortran on windows does not > sound really appealing :) Also, I noticed when developing numscons that > numpy (implicetely ?) relied on buggy mingw headers, bugs which were > fixed in 4.* and hence did not work forr numpy (nothing serious, though; > it should be easily fixable in distutils). What were these problems? > Thank you for those information, I will dig a bit deeper. Should we > consider this as a major blocker for numpy as well (since it would not > be possible to build a working scipy with numpy 1.1.0 ?) ? It depends entirely on what the ultimate solution is. I don't think that numpy can be "fixed" in order to solve this problem. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Thu Apr 17 23:00:57 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 18 Apr 2008 12:00:57 +0900 Subject: [Numpy-discussion] Distutils: using different linker options for c++ and c code In-Reply-To: <3d375d730804171956h2e2482e4w35e9b30b534377ee@mail.gmail.com> References: <480735D8.9020102@ar.media.kyoto-u.ac.jp> <3d375d730804171302v288e526ep3061af7e59a89ce3@mail.gmail.com> <48080914.9080004@ar.media.kyoto-u.ac.jp> <3d375d730804171956h2e2482e4w35e9b30b534377ee@mail.gmail.com> Message-ID: <48080EE9.2030203@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > On Thu, Apr 17, 2008 at 9:36 PM, David Cournapeau > wrote: > >> Robert Kern wrote: >> > >> > Ah, this problem again. The build of mingw that you are using were >> > written with msvcrt in mind. For the most part they are compatible >> > with msvcr71, but there are a few places where they reference a table >> > that is different between the two runtimes, namely iostream in C++ and >> > ischar() in C. >> >> Do you know by any chance any document/email which talks about that ? >> The only things I found were very general (I know file related do not >> work, but that's it). >> > > Only my own emails. I found this out by largely experimentation and > reading the header files. > > >> > Consequently, there are some extension modules built >> > with mingw which work with msvcrt because they need iostream, some >> > with msvcr71 because they need to pass FILE pointers, and probably >> > some which won't work with either. The core problem won't be fixed >> > until mingw writes their headers for msvcr71. They may have; it looks >> > like they just released some new builds this month. It would be worth >> > checking these out. >> >> Are you talking about the 3.* or the 4.* releases ? >> The new 3.* release seems to only contain a new release note (all the >> files except the release note have 20060117-2 in their names). >> > > Ah, I missed that. It looks like it's just a re-release of the 2006 > build with a bugfix for Vista. > > >> I also found an unofficial installer for gcc 4.*, which at least claim >> to care about python: >> >> http://www.develer.com/oss/GccWinBinaries >> >> The problems with the new official (but beta) 4.* builds is the lack of >> fortran support. For some reasons, building gfortran on windows does not >> sound really appealing :) Also, I noticed when developing numscons that >> numpy (implicetely ?) relied on buggy mingw headers, bugs which were >> fixed in 4.* and hence did not work forr numpy (nothing serious, though; >> it should be easily fixable in distutils). >> > > What were these problems? > Distutils uses some tests function to see wether a particular set of float functions is available (FUNCTIONS_TO_CHECK dict in numpy/core/setup.py), detects a particular set is not available on mingw (say 'HAVE_INVERSE_HYPERBOLIC'), and define all of them. The problem is that they are available in the library, just not declared (I remember because at first, when I built the core with numscons, I only detected functions by linking to them, and I had discrepancies between numscons config.h and distutils config.h: that's why I now use both declaration and presence detection in the m library). Concretely, some redefinition from static to non static; but that's just a configuration problem I think. > >> Thank you for those information, I will dig a bit deeper. Should we >> consider this as a major blocker for numpy as well (since it would not >> be possible to build a working scipy with numpy 1.1.0 ?) ? >> > > It depends entirely on what the ultimate solution is. Yes. I still hope this can be solved using a different build chain than the 3.4.5 mingw. If not, there is nothing we can do about it. But then, should we build official binaries with MS compilers ? cheers, David From david at ar.media.kyoto-u.ac.jp Thu Apr 17 23:16:39 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 18 Apr 2008 12:16:39 +0900 Subject: [Numpy-discussion] Distutils: using different linker options for c++ and c code In-Reply-To: <48080EE9.2030203@ar.media.kyoto-u.ac.jp> References: <480735D8.9020102@ar.media.kyoto-u.ac.jp> <3d375d730804171302v288e526ep3061af7e59a89ce3@mail.gmail.com> <48080914.9080004@ar.media.kyoto-u.ac.jp> <3d375d730804171956h2e2482e4w35e9b30b534377ee@mail.gmail.com> <48080EE9.2030203@ar.media.kyoto-u.ac.jp> Message-ID: <48081297.1050901@ar.media.kyoto-u.ac.jp> David Cournapeau wrote: > > Distutils uses some tests function to see wether a particular set of > float functions is available (FUNCTIONS_TO_CHECK dict in > numpy/core/setup.py), detects a particular set is not available on mingw > (say 'HAVE_INVERSE_HYPERBOLIC'), and define all of them. The problem is > that they are available in the library, just not declared (I remember > because at first, when I built the core with numscons, I only detected > functions by linking to them, and I had discrepancies between numscons > config.h and distutils config.h: that's why I now use both declaration > and presence detection in the m library). > > Concretely, some redefinition from static to non static; but that's just > a configuration problem I think. To be more precise, here is what I get with gcc 4.1.2 on windows (sorry for the buggy copy-and-paste, but I am surprised I can copy from windows in linux at all): gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes -Ibuild\src.win32-2.5\numpy\core\src -Inumpy\core\include -Ibuild\src.win32-2.5\numpy\core -Inumpy\core\src -Inumpy\cor e\include -IC:\Python25\include -IC:\Python25\PC -IC:\Python25\include -IC:\Python25\PC -c build\src.win32-2.5\numpy\core\src\umathmodule.c -o build\temp.win32-2.5\R elease\build\src.win32-2.5\numpy\core\src\umathmodule.o numpy\core\src\umathmodule.c.src:128: error: static declaration of 'acoshf' follows non-static declaration numpy\core\src\umathmodule.c.src:136: error: static declaration of 'asinhf' follows non-static declaration error: Command "gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes -Ibuild\src.win32-2.5\numpy\core\src -Inumpy\core\include -Ibuild\src.win32-2.5\numpy\core -Inumpy\core \src -Inumpy\core\include -IC:\Python25\include -IC:\Python25\PC -IC:\Python25\include -IC:\Python25\PC -c build\src.win32-2.5\numpy\core\src\umathmodule.c -o build\ temp.win32-2.5\Release\build\src.win32-2.5\numpy\core\src\umathmodule.o" failed with exit status 1 cheers, David From robert.kern at gmail.com Thu Apr 17 23:55:18 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 17 Apr 2008 22:55:18 -0500 Subject: [Numpy-discussion] Distutils: using different linker options for c++ and c code In-Reply-To: <48080914.9080004@ar.media.kyoto-u.ac.jp> References: <480735D8.9020102@ar.media.kyoto-u.ac.jp> <3d375d730804171302v288e526ep3061af7e59a89ce3@mail.gmail.com> <48080914.9080004@ar.media.kyoto-u.ac.jp> Message-ID: <3d375d730804172055s21be84bdp5e52e6d5b4df7cf@mail.gmail.com> On Thu, Apr 17, 2008 at 9:36 PM, David Cournapeau wrote: > I also found an unofficial installer for gcc 4.*, which at least claim > to care about python: > > http://www.develer.com/oss/GccWinBinaries > > The problems with the new official (but beta) 4.* builds is the lack of > fortran support. For some reasons, building gfortran on windows does not > sound really appealing :) It seems that Giovanni is getting his gcc from here: http://www.tdragon.net/recentgcc/ which also appears to have gfortran binaries, too. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Thu Apr 17 23:56:17 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 18 Apr 2008 12:56:17 +0900 Subject: [Numpy-discussion] Distutils: using different linker options for c++ and c code In-Reply-To: <3d375d730804172055s21be84bdp5e52e6d5b4df7cf@mail.gmail.com> References: <480735D8.9020102@ar.media.kyoto-u.ac.jp> <3d375d730804171302v288e526ep3061af7e59a89ce3@mail.gmail.com> <48080914.9080004@ar.media.kyoto-u.ac.jp> <3d375d730804172055s21be84bdp5e52e6d5b4df7cf@mail.gmail.com> Message-ID: <48081BE1.1040606@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > > It seems that Giovanni is getting his gcc from here: > > http://www.tdragon.net/recentgcc/ > > which also appears to have gfortran binaries, too. > > Would that be acceptable to use for numpy/scipy ? I mean, is it enough if we say to people: you need to install mingw from there instead of the official ones if you want to build both numpy and scipy ? cheers, David From robert.kern at gmail.com Fri Apr 18 00:15:54 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 17 Apr 2008 23:15:54 -0500 Subject: [Numpy-discussion] Distutils: using different linker options for c++ and c code In-Reply-To: <48081BE1.1040606@ar.media.kyoto-u.ac.jp> References: <480735D8.9020102@ar.media.kyoto-u.ac.jp> <3d375d730804171302v288e526ep3061af7e59a89ce3@mail.gmail.com> <48080914.9080004@ar.media.kyoto-u.ac.jp> <3d375d730804172055s21be84bdp5e52e6d5b4df7cf@mail.gmail.com> <48081BE1.1040606@ar.media.kyoto-u.ac.jp> Message-ID: <3d375d730804172115l2b6b8dd4u364c95b33ddebaa7@mail.gmail.com> On Thu, Apr 17, 2008 at 10:56 PM, David Cournapeau wrote: > Robert Kern wrote: > > > > It seems that Giovanni is getting his gcc from here: > > > > http://www.tdragon.net/recentgcc/ > > > > which also appears to have gfortran binaries, too. > > Would that be acceptable to use for numpy/scipy ? I mean, is it enough > if we say to people: you need to install mingw from there instead of the > official ones if you want to build both numpy and scipy ? Well, if the official mingw team is committed to not supporting the features that we need, then yes, absolutely. This appears to be the case. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Fri Apr 18 00:33:10 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 17 Apr 2008 22:33:10 -0600 Subject: [Numpy-discussion] Distutils: using different linker options for c++ and c code In-Reply-To: <3d375d730804172115l2b6b8dd4u364c95b33ddebaa7@mail.gmail.com> References: <480735D8.9020102@ar.media.kyoto-u.ac.jp> <3d375d730804171302v288e526ep3061af7e59a89ce3@mail.gmail.com> <48080914.9080004@ar.media.kyoto-u.ac.jp> <3d375d730804172055s21be84bdp5e52e6d5b4df7cf@mail.gmail.com> <48081BE1.1040606@ar.media.kyoto-u.ac.jp> <3d375d730804172115l2b6b8dd4u364c95b33ddebaa7@mail.gmail.com> Message-ID: Question. How does numpy link against the python library when there is only a static version present? I compiled/installed python2.6.2a and the only library it generated was /usr/local/lib/python2.6/config/libpython2.6.a. Furthermore, I didn't see the python library specifically linked by any of the numpy code. Compare with the python2.5 build: $[charris at f8 numpy]$ grep /numpy/fft/fftpack.o log2.6 gcc -pthread -shared -Wall -march=nocona -mtune=generic -O2 -pipe -fomit-frame-pointer -fno-strict-aliasing -finline-functions build/temp.linux-i686-2.6/numpy/fft/fftpack_litemodule.o build/temp.linux-i686-2.6/numpy/fft/fftpack.o -o build/lib.linux-i686-2.6/numpy/fft/fftpack_lite.so $[charris at f8 numpy]$ grep /numpy/fft/fftpack.o log2.5 gcc -pthread -shared -Wall -march=nocona -mtune=generic -O2 -pipe -fomit-frame-pointer -fno-strict-aliasing -finline-functions build/temp.linux-i686-2.5/numpy/fft/fftpack_litemodule.o build/temp.linux-i686-2.5/numpy/fft/fftpack.o -L/usr/lib -lpython2.5 -o build/lib.linux-i686-2.5/numpy/fft/fftpack_lite.so I'm trying to track down the failed check_object_casting failure and I am finding some very strange things. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Apr 18 00:37:00 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 17 Apr 2008 23:37:00 -0500 Subject: [Numpy-discussion] Distutils: using different linker options for c++ and c code In-Reply-To: References: <480735D8.9020102@ar.media.kyoto-u.ac.jp> <3d375d730804171302v288e526ep3061af7e59a89ce3@mail.gmail.com> <48080914.9080004@ar.media.kyoto-u.ac.jp> <3d375d730804172055s21be84bdp5e52e6d5b4df7cf@mail.gmail.com> <48081BE1.1040606@ar.media.kyoto-u.ac.jp> <3d375d730804172115l2b6b8dd4u364c95b33ddebaa7@mail.gmail.com> Message-ID: <3d375d730804172137r32c91ac5m6b5e1c3c1d1edc32@mail.gmail.com> On Thu, Apr 17, 2008 at 11:33 PM, Charles R Harris wrote: > Question. How does numpy link against the python library when there is only > a static version present? However distutils tells it to. The information is contained in /usr/local/lib/python2.6/config/Makefile, mostly in the LDFLAGS and LDSHARED variables. > I compiled/installed python2.6.2a and the only > library it generated was /usr/local/lib/python2.6/config/libpython2.6.a. > Furthermore, I didn't see the python library specifically linked by any of > the numpy code. This looks like a problem with your installation of Python. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Fri Apr 18 00:49:57 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 18 Apr 2008 13:49:57 +0900 Subject: [Numpy-discussion] Distutils: using different linker options for c++ and c code In-Reply-To: <3d375d730804172115l2b6b8dd4u364c95b33ddebaa7@mail.gmail.com> References: <480735D8.9020102@ar.media.kyoto-u.ac.jp> <3d375d730804171302v288e526ep3061af7e59a89ce3@mail.gmail.com> <48080914.9080004@ar.media.kyoto-u.ac.jp> <3d375d730804172055s21be84bdp5e52e6d5b4df7cf@mail.gmail.com> <48081BE1.1040606@ar.media.kyoto-u.ac.jp> <3d375d730804172115l2b6b8dd4u364c95b33ddebaa7@mail.gmail.com> Message-ID: <48082875.2090800@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > > Well, if the official mingw team is committed to not supporting the > features that we need, then yes, absolutely. This appears to be the > case. > Well, this does not seem to work either with gcc 4.3* releases. I got segfaults too. So should we use VS built instead for numpy and scipy in the future ? cheers, David From robert.kern at gmail.com Fri Apr 18 01:04:22 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 18 Apr 2008 00:04:22 -0500 Subject: [Numpy-discussion] Distutils: using different linker options for c++ and c code In-Reply-To: <48082875.2090800@ar.media.kyoto-u.ac.jp> References: <480735D8.9020102@ar.media.kyoto-u.ac.jp> <3d375d730804171302v288e526ep3061af7e59a89ce3@mail.gmail.com> <48080914.9080004@ar.media.kyoto-u.ac.jp> <3d375d730804172055s21be84bdp5e52e6d5b4df7cf@mail.gmail.com> <48081BE1.1040606@ar.media.kyoto-u.ac.jp> <3d375d730804172115l2b6b8dd4u364c95b33ddebaa7@mail.gmail.com> <48082875.2090800@ar.media.kyoto-u.ac.jp> Message-ID: <3d375d730804172204j2d5ca309w84c342f4ee60e7b5@mail.gmail.com> On Thu, Apr 17, 2008 at 11:49 PM, David Cournapeau wrote: > Robert Kern wrote: > > > > Well, if the official mingw team is committed to not supporting the > > features that we need, then yes, absolutely. This appears to be the > > case. > > Well, this does not seem to work either with gcc 4.3* releases. I got > segfaults too. So should we use VS built instead for numpy and scipy in > the future ? Quite possibly. Can you run the segfaulting code in a debugger so we can try to isolate the actual cause? It is possible that we can patch it up to work with msvcr71. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Fri Apr 18 01:05:42 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 18 Apr 2008 14:05:42 +0900 Subject: [Numpy-discussion] Distutils: using different linker options for c++ and c code In-Reply-To: <3d375d730804172204j2d5ca309w84c342f4ee60e7b5@mail.gmail.com> References: <480735D8.9020102@ar.media.kyoto-u.ac.jp> <3d375d730804171302v288e526ep3061af7e59a89ce3@mail.gmail.com> <48080914.9080004@ar.media.kyoto-u.ac.jp> <3d375d730804172055s21be84bdp5e52e6d5b4df7cf@mail.gmail.com> <48081BE1.1040606@ar.media.kyoto-u.ac.jp> <3d375d730804172115l2b6b8dd4u364c95b33ddebaa7@mail.gmail.com> <48082875.2090800@ar.media.kyoto-u.ac.jp> <3d375d730804172204j2d5ca309w84c342f4ee60e7b5@mail.gmail.com> Message-ID: <48082C26.1050806@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > On Thu, Apr 17, 2008 at 11:49 PM, David Cournapeau > wrote: > >> Robert Kern wrote: >> > >> > Well, if the official mingw team is committed to not supporting the >> > features that we need, then yes, absolutely. This appears to be the >> > case. >> >> Well, this does not seem to work either with gcc 4.3* releases. I got >> segfaults too. So should we use VS built instead for numpy and scipy in >> the future ? >> > > Quite possibly. Can you run the segfaulting code in a debugger so we > can try to isolate the actual cause? It is possible that we can patch > it up to work with msvcr71. > Ok, I will have to take a look at how to do that, then, I don't really have experience with debugger, much less on windows. Will get back one I understand how this works. David From david at ar.media.kyoto-u.ac.jp Fri Apr 18 01:27:14 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 18 Apr 2008 14:27:14 +0900 Subject: [Numpy-discussion] Distutils: using different linker options for c++ and c code In-Reply-To: <48082C26.1050806@ar.media.kyoto-u.ac.jp> References: <480735D8.9020102@ar.media.kyoto-u.ac.jp> <3d375d730804171302v288e526ep3061af7e59a89ce3@mail.gmail.com> <48080914.9080004@ar.media.kyoto-u.ac.jp> <3d375d730804172055s21be84bdp5e52e6d5b4df7cf@mail.gmail.com> <48081BE1.1040606@ar.media.kyoto-u.ac.jp> <3d375d730804172115l2b6b8dd4u364c95b33ddebaa7@mail.gmail.com> <48082875.2090800@ar.media.kyoto-u.ac.jp> <3d375d730804172204j2d5ca309w84c342f4ee60e7b5@mail.gmail.com> <48082C26.1050806@ar.media.kyoto-u.ac.jp> Message-ID: <48083132.8070202@ar.media.kyoto-u.ac.jp> David Cournapeau wrote: > > > Ok, I will have to take a look at how to do that, then, I don't really > have experience with debugger, much less on windows. Will get back one I > understand how this works. > Ok, this is great: when run in gdb, no crash. But when run in cmd.exe, it always crash. Is there anything else I can do to get the crash under a debugger ? David From matthieu.brucher at gmail.com Fri Apr 18 01:57:58 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Fri, 18 Apr 2008 07:57:58 +0200 Subject: [Numpy-discussion] Distutils: using different linker options for c++ and c code In-Reply-To: <48082875.2090800@ar.media.kyoto-u.ac.jp> References: <480735D8.9020102@ar.media.kyoto-u.ac.jp> <3d375d730804171302v288e526ep3061af7e59a89ce3@mail.gmail.com> <48080914.9080004@ar.media.kyoto-u.ac.jp> <3d375d730804172055s21be84bdp5e52e6d5b4df7cf@mail.gmail.com> <48081BE1.1040606@ar.media.kyoto-u.ac.jp> <3d375d730804172115l2b6b8dd4u364c95b33ddebaa7@mail.gmail.com> <48082875.2090800@ar.media.kyoto-u.ac.jp> Message-ID: 2008/4/18, David Cournapeau : > > Robert Kern wrote: > > > > Well, if the official mingw team is committed to not supporting the > > features that we need, then yes, absolutely. This appears to be the > > case. > > > > > Well, this does not seem to work either with gcc 4.3* releases. I got > segfaults too. So should we use VS built instead for numpy and scipy in > the future ? I can help with packaging at least numpy with Visual Studio 2003 (well, I have to check the EULA if I'm allowed to do that !). For scipy, it is a matter of Fortran compiler :| Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Apr 18 02:03:05 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 18 Apr 2008 01:03:05 -0500 Subject: [Numpy-discussion] Distutils: using different linker options for c++ and c code In-Reply-To: References: <480735D8.9020102@ar.media.kyoto-u.ac.jp> <3d375d730804171302v288e526ep3061af7e59a89ce3@mail.gmail.com> <48080914.9080004@ar.media.kyoto-u.ac.jp> <3d375d730804172055s21be84bdp5e52e6d5b4df7cf@mail.gmail.com> <48081BE1.1040606@ar.media.kyoto-u.ac.jp> <3d375d730804172115l2b6b8dd4u364c95b33ddebaa7@mail.gmail.com> <48082875.2090800@ar.media.kyoto-u.ac.jp> Message-ID: <3d375d730804172303k2dcb00b2j4d6cb8800ce072ff@mail.gmail.com> On Fri, Apr 18, 2008 at 12:57 AM, Matthieu Brucher wrote: > I can help with packaging at least numpy with Visual Studio 2003 (well, I > have to check the EULA if I'm allowed to do that !). For scipy, it is a > matter of Fortran compiler :| That probably won't work. I believe that the debugging symbol formats are different between mingw and MSVC. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From matthieu.brucher at gmail.com Fri Apr 18 02:15:16 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Fri, 18 Apr 2008 08:15:16 +0200 Subject: [Numpy-discussion] Distutils: using different linker options for c++ and c code In-Reply-To: <3d375d730804172303k2dcb00b2j4d6cb8800ce072ff@mail.gmail.com> References: <480735D8.9020102@ar.media.kyoto-u.ac.jp> <3d375d730804171302v288e526ep3061af7e59a89ce3@mail.gmail.com> <48080914.9080004@ar.media.kyoto-u.ac.jp> <3d375d730804172055s21be84bdp5e52e6d5b4df7cf@mail.gmail.com> <48081BE1.1040606@ar.media.kyoto-u.ac.jp> <3d375d730804172115l2b6b8dd4u364c95b33ddebaa7@mail.gmail.com> <48082875.2090800@ar.media.kyoto-u.ac.jp> <3d375d730804172303k2dcb00b2j4d6cb8800ce072ff@mail.gmail.com> Message-ID: 2008/4/18, Robert Kern : > > On Fri, Apr 18, 2008 at 12:57 AM, Matthieu Brucher > wrote: > > I can help with packaging at least numpy with Visual Studio 2003 (well, > I > > have to check the EULA if I'm allowed to do that !). For scipy, it is a > > matter of Fortran compiler :| > > > That probably won't work. I believe that the debugging symbol formats > are different between mingw and MSVC. > That was in case we publish binaries made with Visual Studio, as David asked about this matter. The debugging symbols are indeed different, but if it is possible to use Visual Studio, why shouldn't we ? Visual Studio debugger is far better than gdb when debugging optimized code (IMHO). Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Apr 18 02:18:30 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 18 Apr 2008 01:18:30 -0500 Subject: [Numpy-discussion] Distutils: using different linker options for c++ and c code In-Reply-To: References: <480735D8.9020102@ar.media.kyoto-u.ac.jp> <3d375d730804171302v288e526ep3061af7e59a89ce3@mail.gmail.com> <48080914.9080004@ar.media.kyoto-u.ac.jp> <3d375d730804172055s21be84bdp5e52e6d5b4df7cf@mail.gmail.com> <48081BE1.1040606@ar.media.kyoto-u.ac.jp> <3d375d730804172115l2b6b8dd4u364c95b33ddebaa7@mail.gmail.com> <48082875.2090800@ar.media.kyoto-u.ac.jp> <3d375d730804172303k2dcb00b2j4d6cb8800ce072ff@mail.gmail.com> Message-ID: <3d375d730804172318y29bbf4afq9451af40c34a986f@mail.gmail.com> On Fri, Apr 18, 2008 at 1:15 AM, Matthieu Brucher wrote: > > > 2008/4/18, Robert Kern : > > On Fri, Apr 18, 2008 at 12:57 AM, Matthieu Brucher > > wrote: > > > I can help with packaging at least numpy with Visual Studio 2003 (well, > I > > > have to check the EULA if I'm allowed to do that !). For scipy, it is a > > > matter of Fortran compiler :| > > > > That probably won't work. I believe that the debugging symbol formats > > are different between mingw and MSVC. > > That was in case we publish binaries made with Visual Studio, as David asked > about this matter. The debugging symbols are indeed different, but if it is > possible to use Visual Studio, why shouldn't we ? Visual Studio debugger is > far better than gdb when debugging optimized code (IMHO). Sorry, I completely misread what you wrote. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Fri Apr 18 02:12:25 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 18 Apr 2008 15:12:25 +0900 Subject: [Numpy-discussion] Distutils: using different linker options for c++ and c code In-Reply-To: References: <480735D8.9020102@ar.media.kyoto-u.ac.jp> <3d375d730804171302v288e526ep3061af7e59a89ce3@mail.gmail.com> <48080914.9080004@ar.media.kyoto-u.ac.jp> <3d375d730804172055s21be84bdp5e52e6d5b4df7cf@mail.gmail.com> <48081BE1.1040606@ar.media.kyoto-u.ac.jp> <3d375d730804172115l2b6b8dd4u364c95b33ddebaa7@mail.gmail.com> <48082875.2090800@ar.media.kyoto-u.ac.jp> <3d375d730804172303k2dcb00b2j4d6cb8800ce072ff@mail.gmail.com> Message-ID: <48083BC9.40503@ar.media.kyoto-u.ac.jp> Matthieu Brucher wrote: > > > 2008/4/18, Robert Kern >: > > On Fri, Apr 18, 2008 at 12:57 AM, Matthieu Brucher > > > wrote: > > I can help with packaging at least numpy with Visual Studio 2003 > (well, I > > have to check the EULA if I'm allowed to do that !). For scipy, > it is a > > matter of Fortran compiler :| > > > That probably won't work. I believe that the debugging symbol formats > are different between mingw and MSVC. > > > That was in case we publish binaries made with Visual Studio, as David > asked about this matter. The debugging symbols are indeed different, > but if it is possible to use Visual Studio, why shouldn't we ? Well, because it is not available freely and is not open source. I am really reluctant to do this, personally, but if we don't have a choice... It will mean that someone will have to do all the work for windows, though. I personally won't waste any time using visual studio. cheers, David From david at ar.media.kyoto-u.ac.jp Fri Apr 18 02:21:22 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 18 Apr 2008 15:21:22 +0900 Subject: [Numpy-discussion] Distutils: using different linker options for c++ and c code In-Reply-To: <3d375d730804172204j2d5ca309w84c342f4ee60e7b5@mail.gmail.com> References: <480735D8.9020102@ar.media.kyoto-u.ac.jp> <3d375d730804171302v288e526ep3061af7e59a89ce3@mail.gmail.com> <48080914.9080004@ar.media.kyoto-u.ac.jp> <3d375d730804172055s21be84bdp5e52e6d5b4df7cf@mail.gmail.com> <48081BE1.1040606@ar.media.kyoto-u.ac.jp> <3d375d730804172115l2b6b8dd4u364c95b33ddebaa7@mail.gmail.com> <48082875.2090800@ar.media.kyoto-u.ac.jp> <3d375d730804172204j2d5ca309w84c342f4ee60e7b5@mail.gmail.com> Message-ID: <48083DE2.2070307@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > > Quite possibly. Can you run the segfaulting code in a debugger so we > can try to isolate the actual cause? It is possible that we can patch > it up to work with msvcr71. I finally managed to do something, for reference: - I hacked distutils to put -g everywhere (using the --debug in distutils lead to nowhere for me: the debugging library is not ABI compatible so just copying python25 to python25_d does not work, and I can't rebuild python in debug mode since I don't have VS 2003) - I rebuilt everything (scipy only) - I used drmingw as the so called JIT debugger (registering it with drmingw -i): now, when scipy.sparse crashes, I get into the code. python.exe caused an Access Violation at location 66d571c1 in module _bsr.pyd Reading from location 015d6000. Registers: eax=015d5fe8 ebx=00000006 ecx=00000002 edx=00000012 esi=00000006 edi=0000000a eip=66d571c1 esp=0021c488 ebp=0021c4bc iopl=0 nv up ei ng nz ac po cy cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000297 Call stack: 66D571C1 _bsr.pyd:66D571C1 void csr_binop_csr >(int, int, int const*, int const*, short const*, int const*, int const*, short const*, int*, int*, short*, std::plus const&) csr.h:663 ... nnz++; } > B_j = Bj[++B_pos]; } } ... 66D472B0 _bsr.pyd:66D472B0 void bsr_binop_bsr >(int, int, int, int, int const*, int const*, short const*, int const*, int const*, short const*, int*, int*, short*, std::plus const&) bsr.h:550 ... } > Cp[i+1] = nnz; } } ... 66D3934C _bsr.pyd:66D3934C void bsr_plus_bsr(int, int, int, int, int const*, int const*, short const*, int const*, int const*, short const*, int*, int*, short*) bsr.h:580 ... I Cp[], I Cj[], T Cx[]) { > bsr_binop_bsr(n_row,n_col,R,C,Ap,Aj,Ax,Bp,Bj,Bx,Cp,Cj,Cx,std::plus()); } ... 66D01087 _bsr.pyd:66D01087 _wrap_bsr_plus_bsr bsr_wrap.cxx:1226 ... SWIG_Py_Void(void) { > PyObject *none = Py_None; Py_INCREF(none); return none; ... 1E08E137 python25.dll:1E08E137 PyCFunction_Call 1E1E7A10 python25.dll:1E1E7A10 PyTuple_Type So it *may* be unrelated to msvc runtime; maybe it just caused a problem which has always been there. I will try to see if valgrind says anything useful for sparsetools under linux. David From matthieu.brucher at gmail.com Fri Apr 18 02:37:23 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Fri, 18 Apr 2008 08:37:23 +0200 Subject: [Numpy-discussion] Distutils: using different linker options for c++ and c code In-Reply-To: <48083BC9.40503@ar.media.kyoto-u.ac.jp> References: <480735D8.9020102@ar.media.kyoto-u.ac.jp> <48080914.9080004@ar.media.kyoto-u.ac.jp> <3d375d730804172055s21be84bdp5e52e6d5b4df7cf@mail.gmail.com> <48081BE1.1040606@ar.media.kyoto-u.ac.jp> <3d375d730804172115l2b6b8dd4u364c95b33ddebaa7@mail.gmail.com> <48082875.2090800@ar.media.kyoto-u.ac.jp> <3d375d730804172303k2dcb00b2j4d6cb8800ce072ff@mail.gmail.com> <48083BC9.40503@ar.media.kyoto-u.ac.jp> Message-ID: > > > That was in case we publish binaries made with Visual Studio, as David > > asked about this matter. The debugging symbols are indeed different, > > but if it is possible to use Visual Studio, why shouldn't we ? > > > Well, because it is not available freely and is not open source. I am > really reluctant to do this, personally, but if we don't have a > choice... It will mean that someone will have to do all the work for > windows, though. I personally won't waste any time using visual studio. I remember our chat about this some months ago, and I agree, you shouldn't take time for Visual Studio :) Hopefully with Python 2.6, everyone will be able to use Visual Studio 2008. Meanwhile, I can only spend time building numpy (and scipy if there is a free Fortran compiler available that is compatible with VS 2003) :( Perhaps in some months I'll be freer. Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Fri Apr 18 02:43:02 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 18 Apr 2008 15:43:02 +0900 Subject: [Numpy-discussion] Distutils: using different linker options for c++ and c code In-Reply-To: <48083DE2.2070307@ar.media.kyoto-u.ac.jp> References: <480735D8.9020102@ar.media.kyoto-u.ac.jp> <3d375d730804171302v288e526ep3061af7e59a89ce3@mail.gmail.com> <48080914.9080004@ar.media.kyoto-u.ac.jp> <3d375d730804172055s21be84bdp5e52e6d5b4df7cf@mail.gmail.com> <48081BE1.1040606@ar.media.kyoto-u.ac.jp> <3d375d730804172115l2b6b8dd4u364c95b33ddebaa7@mail.gmail.com> <48082875.2090800@ar.media.kyoto-u.ac.jp> <3d375d730804172204j2d5ca309w84c342f4ee60e7b5@mail.gmail.com> <48083DE2.2070307@ar.media.kyoto-u.ac.jp> Message-ID: <480842F6.8030603@ar.media.kyoto-u.ac.jp> David Cournapeau wrote: > So it *may* be unrelated to msvc runtime; maybe it just caused a problem > which has always been there. I will try to see if valgrind says anything > useful for sparsetools under linux. > Ok, I think it is unlikely to be caused by msvc runtime problems. Valgrind reports a huge number of invalid read (see scipy ticket #616). cheers, David From robert.kern at gmail.com Fri Apr 18 02:55:28 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 18 Apr 2008 01:55:28 -0500 Subject: [Numpy-discussion] Distutils: using different linker options for c++ and c code In-Reply-To: <480842F6.8030603@ar.media.kyoto-u.ac.jp> References: <480735D8.9020102@ar.media.kyoto-u.ac.jp> <3d375d730804171302v288e526ep3061af7e59a89ce3@mail.gmail.com> <48080914.9080004@ar.media.kyoto-u.ac.jp> <3d375d730804172055s21be84bdp5e52e6d5b4df7cf@mail.gmail.com> <48081BE1.1040606@ar.media.kyoto-u.ac.jp> <3d375d730804172115l2b6b8dd4u364c95b33ddebaa7@mail.gmail.com> <48082875.2090800@ar.media.kyoto-u.ac.jp> <3d375d730804172204j2d5ca309w84c342f4ee60e7b5@mail.gmail.com> <48083DE2.2070307@ar.media.kyoto-u.ac.jp> <480842F6.8030603@ar.media.kyoto-u.ac.jp> Message-ID: <3d375d730804172355k42f08b6eo481cd4183e8f536f@mail.gmail.com> On Fri, Apr 18, 2008 at 1:43 AM, David Cournapeau wrote: > David Cournapeau wrote: > > So it *may* be unrelated to msvc runtime; maybe it just caused a problem > > which has always been there. I will try to see if valgrind says anything > > useful for sparsetools under linux. > > Ok, I think it is unlikely to be caused by msvc runtime problems. > Valgrind reports a huge number of invalid read (see scipy ticket #616). Whew! Dodged that bullet. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From efiring at hawaii.edu Fri Apr 18 04:17:13 2008 From: efiring at hawaii.edu (Eric Firing) Date: Thu, 17 Apr 2008 22:17:13 -1000 Subject: [Numpy-discussion] fast take patch Message-ID: <48085909.9050209@hawaii.edu> Stefan, (or anyone else who might be interested) Since you committed my fast putmask patch many months ago, I thought you might like to deal with my fast take patch. Attached is the diff relative to 5043, ignoring whitespace. (Yes, those pesky whitespace anomalies are still cropping up.) The motivation for the patch is that in matplotlib color mapping, such as when displaying an image or pcolor plot, the time-limiting factor can be a single numpy operation of indexing into the color map. (http://www.mail-archive.com/matplotlib-users at lists.sourceforge.net/msg06482.html) It was being done with fancy indexing. I changed that to use the take method, which sped it up by a factor of two, but I found that take itself is slower than it needs to be by a factor of 2-3. As with putmask, the culprit is memmove, so I used the same strategy to substitute direct variable assignment when possible. The patch includes tests for the take method. I did not find any such pre-existing tests. Thanks for taking a look. Let me know what needs to be changed. Eric -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: fasttake.diff_5043uw URL: From zelbier at gmail.com Fri Apr 18 07:11:37 2008 From: zelbier at gmail.com (Olivier Verdier) Date: Fri, 18 Apr 2008 13:11:37 +0200 Subject: [Numpy-discussion] Truth value of an array Message-ID: In mathematics, if I compare two function, it means that I compare on all its "coordinates". If I say "f < g" I mean "f(x) < g(x) for all x". The same holds for a vector, if I write "v == w" I mean "v[i] == w[i] for all i". How come this doesn't work in numpy? And why the message about the truth value of an array being ambiguous? What is ambiguous? In my opinion any boolean array should be cast with all() automatically so that people can simply write: if v == w: ... Or is there a good reason why this is not possible? Thank you! From david.douard at logilab.fr Fri Apr 18 07:55:17 2008 From: david.douard at logilab.fr (David Douard) Date: Fri, 18 Apr 2008 13:55:17 +0200 Subject: [Numpy-discussion] Truth value of an array In-Reply-To: References: Message-ID: <20080418115517.GB30358@logilab.fr> On Fri, Apr 18, 2008 at 01:11:37PM +0200, Olivier Verdier wrote: > In mathematics, if I compare two function, it means that I compare on > all its "coordinates". If I say "f < g" I mean "f(x) < g(x) for all > x". > > The same holds for a vector, if I write "v == w" I mean "v[i] == w[i] > for all i". > > How come this doesn't work in numpy? And why the message about the > truth value of an array being ambiguous? What is ambiguous? > > In my opinion any boolean array should be cast with all() > automatically so that people can simply write: > if v == w: > ... No. A comparison operator on ndarray produce an ndarray of booleans. So you have to write if numpy.alltrue(u == v): blah This is a very good idea, since it allows you to also write things like: u[v<0] = 0 Which you could definitely not do if the comparison returns a scalar bool instead of a ndarray. More, you may also want to write things like: if numpy.sometrue(u<0): blah Same remark. > > Or is there a good reason why this is not possible? > > Thank you! > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- David Douard LOGILAB, Paris (France), +33 1 45 32 03 12 Formations Python, Zope, Debian : http://www.logilab.fr/formations D?veloppement logiciel sur mesure : http://www.logilab.fr/services Informatique scientifique : http://www.logilab.fr/science -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From zelbier at gmail.com Fri Apr 18 08:33:28 2008 From: zelbier at gmail.com (Olivier Verdier) Date: Fri, 18 Apr 2008 14:33:28 +0200 Subject: [Numpy-discussion] Truth value of an array In-Reply-To: <20080418115517.GB30358@logilab.fr> References: <20080418115517.GB30358@logilab.fr> Message-ID: I certainly didn't mean that "A==B" should return a boolean!! "A==B" should return an array of boolean as it does now. This is all right. *However* "bool(A==B)" should return a boolean, *not* raise an exception. Why raise an exception? What is ambiguous about "bool(A==B)"?? This is what happens when you write "if A==B" because then "bool(A==B)" is somehow triggered and bang, an exception is raised. As I tried to explain, the default behaviour should be that "bool(A)" return "A.all()" but not an exception. Why is there an exception raising at all? I hope that my question is clearer now... Thanks. On 18/04/2008, David Douard wrote: > On Fri, Apr 18, 2008 at 01:11:37PM +0200, Olivier Verdier wrote: > > In mathematics, if I compare two function, it means that I compare on > > all its "coordinates". If I say "f < g" I mean "f(x) < g(x) for all > > x". > > > > The same holds for a vector, if I write "v == w" I mean "v[i] == w[i] > > for all i". > > > > How come this doesn't work in numpy? And why the message about the > > truth value of an array being ambiguous? What is ambiguous? > > > > In my opinion any boolean array should be cast with all() > > automatically so that people can simply write: > > if v == w: > > ... > > > No. A comparison operator on ndarray produce an ndarray of booleans. So > you have to write > > if numpy.alltrue(u == v): > blah > > This is a very good idea, since it allows you to also write things > like: > > u[v<0] = 0 > > Which you could definitely not do if the comparison returns a scalar > bool instead of a ndarray. > > More, you may also want to write things like: > > if numpy.sometrue(u<0): > blah > > Same remark. > > > > > > > > Or is there a good reason why this is not possible? > > > > Thank you! > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > -- > David Douard LOGILAB, Paris (France), +33 1 45 32 03 12 > Formations Python, Zope, Debian : http://www.logilab.fr/formations > D?veloppement logiciel sur mesure : http://www.logilab.fr/services > Informatique scientifique : http://www.logilab.fr/science > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.6 (GNU/Linux) > > iD8DBQFICIwl15pXfQ0A2uMRAm/3AJ9nqNHSH7vY5NpIULCXjgXdqkCUUgCfdxyi > NIYA8A4grCPDp5lShTKhcoY= > =S6VB > -----END PGP SIGNATURE----- > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > From matthieu.brucher at gmail.com Fri Apr 18 08:43:40 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Fri, 18 Apr 2008 14:43:40 +0200 Subject: [Numpy-discussion] Truth value of an array In-Reply-To: References: <20080418115517.GB30358@logilab.fr> Message-ID: 2008/4/18, Olivier Verdier : > > I certainly didn't mean that "A==B" should return a boolean!! > > "A==B" should return an array of boolean as it does now. This is all > right. > > *However* "bool(A==B)" should return a boolean, *not* raise an > exception. Why raise an exception? What is ambiguous about > "bool(A==B)"?? > > This is what happens when you write "if A==B" because then > "bool(A==B)" is somehow triggered and bang, an exception is raised. > > As I tried to explain, the default behaviour should be that "bool(A)" > return "A.all()" but not an exception. Why is there an exception > raising at all? Sometimes, you want .all(), sometimes .any(). It really depends on the question you are asking. Even for bool(A), there is no easy answer. In some case I want True if some elements are true, in other cases only if all elements are true. I would agree with you some years ago, but after using bool(A) = A.all(), I started noticing that it brakes my coding instead of speeding it and does not give me any benefit at all. Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Fri Apr 18 08:46:36 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 18 Apr 2008 14:46:36 +0200 Subject: [Numpy-discussion] fast take patch In-Reply-To: <48085909.9050209@hawaii.edu> References: <48085909.9050209@hawaii.edu> Message-ID: <9457e7c80804180546r25424fa6n14f2107aad61e1df@mail.gmail.com> Hi Eric On 18/04/2008, Eric Firing wrote: > The motivation for the patch is that in matplotlib color mapping, such as [...] Beautiful patch, good motivation. Thank you! Applied in r5044. Regards St?fan From lxander.m at gmail.com Fri Apr 18 09:03:41 2008 From: lxander.m at gmail.com (Alexander Michael) Date: Fri, 18 Apr 2008 09:03:41 -0400 Subject: [Numpy-discussion] Truth value of an array In-Reply-To: References: <20080418115517.GB30358@logilab.fr> Message-ID: <525f23e80804180603v6cca4506u9b9d20a009544fe2@mail.gmail.com> On Fri, Apr 18, 2008 at 8:43 AM, Matthieu Brucher wrote: > > > 2008/4/18, Olivier Verdier : > > I certainly didn't mean that "A==B" should return a boolean!! > > > > "A==B" should return an array of boolean as it does now. This is all > right. > > > > *However* "bool(A==B)" should return a boolean, *not* raise an > > exception. Why raise an exception? What is ambiguous about > > "bool(A==B)"?? > > > > This is what happens when you write "if A==B" because then > > "bool(A==B)" is somehow triggered and bang, an exception is raised. > > > > As I tried to explain, the default behaviour should be that "bool(A)" > > return "A.all()" but not an exception. Why is there an exception > > raising at all? > > Sometimes, you want .all(), sometimes .any(). It really depends on the > question you are asking. Even for bool(A), there is no easy answer. In some > case I want True if some elements are true, in other cases only if all > elements are true. > I would agree with you some years ago, but after using bool(A) = A.all(), I > started noticing that it brakes my coding instead of speeding it and does > not give me any benefit at all. Not only might reasonable people differ on want any or all semantics, it is also quite reasonable to want __nonzero__ to merely indicate whether or not the ndarray has any elements in the first dimension (i.e. len >0) like the built-in list and array. At least I think this is reasonable as this is the semantic that I often want! But seeing how there are multiple prior conceptions, perhaps the current behavior is best because it forces us to make sure what we think is being returned is in fact being returned? Regards, Alex From aisaac at american.edu Fri Apr 18 09:20:01 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 18 Apr 2008 09:20:01 -0400 Subject: [Numpy-discussion] Truth value of an array In-Reply-To: References: <20080418115517.GB30358@logilab.fr> Message-ID: On Fri, 18 Apr 2008, Olivier Verdier apparently wrote: > What is ambiguous about "bool(A==B)"? A==B is an array. Compare: >>> bool([]) False >>> bool([0]) True Even if you decide the second should be false, what about [0,1]? (I.e., all or any?) Cheers, Alan Isaac From zelbier at gmail.com Fri Apr 18 09:36:23 2008 From: zelbier at gmail.com (Olivier) Date: Fri, 18 Apr 2008 06:36:23 -0700 (PDT) Subject: [Numpy-discussion] Truth value of an array In-Reply-To: References: <20080418115517.GB30358@logilab.fr> Message-ID: <6573665e-c09f-4a24-82bb-93e481cb781f@a22g2000hsc.googlegroups.com> Let's restrict the discussion the case to boolean arrays (dtype bool), since all the comparisons (A==B, A!=B, A wrote: > On Fri, 18 Apr 2008, Olivier Verdier apparently wrote: > > What is ambiguous about "bool(A==B)"? > > A==B is an array. Compare: > > ? ? >>> bool([]) > ? ? False > ? ? >>> bool([0]) > ? ? True > > Even if you decide the second should be false, > what about [0,1]? ?(I.e., all or any?) > > Cheers, > Alan Isaac > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion From matthew.brett at gmail.com Fri Apr 18 09:45:56 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 18 Apr 2008 14:45:56 +0100 Subject: [Numpy-discussion] Truth value of an array In-Reply-To: <6573665e-c09f-4a24-82bb-93e481cb781f@a22g2000hsc.googlegroups.com> References: <20080418115517.GB30358@logilab.fr> <6573665e-c09f-4a24-82bb-93e481cb781f@a22g2000hsc.googlegroups.com> Message-ID: <1e2af89e0804180645y62ea6ff5q268d629012b765de@mail.gmail.com> Hi, I must say, I agree with the other posters here, that it is not completely obvious to me that: a = np.array([True, False]) bool(a) should return False. Especially given: L = [True, False] bool(L) returns True. Given that it's not completely obvious, and a.all() is completely obvious, and readable, the current behavior seems right to me... Otherwise someone somewhere is going to assume that bool(a) does the same as a.any(), and run into problems. Explicit is better than implicit... Best, Matthew From peridot.faceted at gmail.com Fri Apr 18 14:10:55 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Fri, 18 Apr 2008 20:10:55 +0200 Subject: [Numpy-discussion] Truth value of an array In-Reply-To: <6573665e-c09f-4a24-82bb-93e481cb781f@a22g2000hsc.googlegroups.com> References: <20080418115517.GB30358@logilab.fr> <6573665e-c09f-4a24-82bb-93e481cb781f@a22g2000hsc.googlegroups.com> Message-ID: On 18/04/2008, Olivier wrote: > Let's restrict the discussion the case to boolean arrays (dtype bool), > since all the comparisons (A==B, A!=B, A arrays). > > So I have an array filled with booleans. Is there a reason not to map > "bool(A)" to "A.all()" but instead raise an exception? > > As far as I can see, "if A==B" is clear enough; one always means > "(A==B).all()". Isn't that so? > > I would like to see examples where there is clearly an ambiguity so > that I understand the benefit of having an exception raised for > boolean matrices instead of simply using all(). > > If there is no such clear example, then why not map "bool(A)" to > "A.all()" in numpy? In [1]: import numpy as np In [2]: A = np.array([0,1]) In [3]: B = np.array([0,0]) In [4]: (A==B).all() Out[4]: False In [5]: (A!=B).all() Out[5]: False One would expect A!=B to be the logical opposite of A==B, but with your proposed suggestion it is not. In math, when comparing functions, one can compare the functions as a whole, or one can compare them pointwise. numpy's implementation does pointwise comparison, for a variety of good reasons. As for why converting an array to a boolean doesn't automatically do all(), if you don't know you are dealing with an array and using all(), you will surely shoot yourself in the foot sooner rather than later (as the above example shows). If you do, simply wrapping it in all() is easy and clear. Anne From jh at physics.ucf.edu Fri Apr 18 16:33:34 2008 From: jh at physics.ucf.edu (Joe Harrington) Date: Fri, 18 Apr 2008 16:33:34 -0400 Subject: [Numpy-discussion] Truth value of an array In-Reply-To: (numpy-discussion-request@scipy.org) References: Message-ID: For that matter, is there a reason logical operations don't work on arrays other than booleans? What about: import numpy x = numpy.ones((10), dtype='Bool') y = numpy.ones((10), dtype='Bool') y[6] = False z = x and y # logical AND: this one fails with an error about arrays --------------------------------------------------------------------------- Traceback (most recent call last) /home/jh/ in () : The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() --------------------------------------------------------------------------- z = x & y # bitwise AND: this one succeeds print z [ True True True True True True False True True True] But, both operators should return the same result, since booleans have one bit. Where this really matters is ANDing ints: x = numpy.ones((10), dtype='uint8') y = numpy.ones((10), dtype='uint8') y[6] = 2 # still True this time: http://docs.python.org/ref/Booleans.html z = x and y # logical AND: this one fails with an error about arrays --------------------------------------------------------------------------- Traceback (most recent call last) /home/jh/ in () : The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() --------------------------------------------------------------------------- z = x & y # bitwise AND: this one succeeds...but it isn't the AND I want! print z [1 1 1 1 1 1 0 1 1 1] # the AND I want makes this array all True Here, both x and y are completely True, but you can't trivially do a logical AND, and the bitwise & isn't the one you want. You have to cast to boolean manually and do the logical instead. Can this be fixed or is there a reason for the exception? --jh-- From robert.kern at gmail.com Fri Apr 18 16:39:04 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 18 Apr 2008 15:39:04 -0500 Subject: [Numpy-discussion] Truth value of an array In-Reply-To: References: Message-ID: <3d375d730804181339l34661f7bq69392feb8d72c00e@mail.gmail.com> On Fri, Apr 18, 2008 at 3:33 PM, Joe Harrington wrote: > For that matter, is there a reason logical operations don't work on > arrays other than booleans? What about: The keywords "and", "or", and "not" only work on bool objects; Python tries to convert the operands using bool(). bool(some_array) raises the exception just as we've been discussing. > Here, both x and y are completely True, but you can't trivially do a > logical AND, and the bitwise & isn't the one you want. You have to > cast to boolean manually and do the logical instead. > > Can this be fixed or is there a reason for the exception? No, it cannot be fixed. Python does not give us a way to overload the behavior of these keywords. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From aisaac at american.edu Fri Apr 18 16:42:39 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 18 Apr 2008 16:42:39 -0400 Subject: [Numpy-discussion] Truth value of an array In-Reply-To: References: Message-ID: On Fri, 18 Apr 2008, Joe Harrington apparently wrote: > For that matter, is there a reason logical operations don't work on > arrays other than booleans? What about: > import numpy > x = numpy.ones((10), dtype='Bool') > y = numpy.ones((10), dtype='Bool') > y[6] = False > z = x and y # logical AND: this one fails with an error about arrays It's the same reason. (See the other comments in this thread.) Do this:: x = numpy.ones((10), dtype='Bool') y = numpy.ones((10), dtype='Bool') y[6] = False z= x&y hth, Alan Isaac From peridot.faceted at gmail.com Fri Apr 18 17:09:50 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Fri, 18 Apr 2008 23:09:50 +0200 Subject: [Numpy-discussion] Truth value of an array In-Reply-To: <3d375d730804181339l34661f7bq69392feb8d72c00e@mail.gmail.com> References: <3d375d730804181339l34661f7bq69392feb8d72c00e@mail.gmail.com> Message-ID: On 18/04/2008, Robert Kern wrote: > On Fri, Apr 18, 2008 at 3:33 PM, Joe Harrington wrote: > > For that matter, is there a reason logical operations don't work on > > arrays other than booleans? What about: > > The keywords "and", "or", and "not" only work on bool objects; Python > tries to convert the operands using bool(). bool(some_array) raises > the exception just as we've been discussing. This is a bit misleading, as the actual value is returned: In [167]: "" or "A" Out[167]: 'A' In [168]: "A" and "B" Out[168]: 'B' In fact the problem is that "and" and "or" are short-circuiting, which means that "return X or Y" is equivalent to something like: if X: return X elif Y: return Y else: return False These are handled specially by syntax so that, for example, you can do "False and 1/0" and not raise a ZeroDivisionError. So "and" and "or" aren't even really operators, and it's not just impossible to change them but unlikely to ever become possible. Just cast your arrays to booleans if you want to do boolean operations on them. Anne From fperez.net at gmail.com Fri Apr 18 18:14:02 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 18 Apr 2008 15:14:02 -0700 Subject: [Numpy-discussion] [OT - IPython] Old 'broken terminal' bug finally fixed Message-ID: [ Sorry for the cross-post, but I know this is something that has hit quite a few people on this list. If you have any questions on it, please ask on the ipython list, this is just an FYI ] Hi all, there's a very old, *extremely* annoying bug that multiple people have asked about (on list and in person) that is finally just fixed. The behavior was that at some point during a normal session, after a call to 'foo?', your terminal would be totally messed up, with no displayed input. You could (blindly) type !reset to issue the system terminal reset command, but that would only help until the next time foo? was called, and the problem would then return. Most of us would end up just quitting ipython and restarting, often losing useful session state. The problem with this is that we never knew how to reliably reproduce it... Anyway, it's fixed now in current bzr: http://bazaar.launchpad.net/~ipython/ipython/stable-dev/revision/79 I don't actually know how to trigger it, but it hit me during an important session where I really couldn't afford to lose what I was working on, and I managed to track it down to what I'm pretty sure is a curses bug. Basically curses.initscr() fails to correctly initialize the terminal sometimes (I still don't have a clue why), and after that it's all lost. But it turns out that via termios one can in fact reset the terminal state reliably , so now we unconditionally do that. Anyway, I figured this would be worth mentioning here, since I know the problem is one that quite often bites people in the middle of work sessions an it can be very, very annoying. Cheers, f back to what I was busy doing, with my terminal now fully functional again... :) From broman at spawar.navy.mil Fri Apr 18 21:19:09 2008 From: broman at spawar.navy.mil (Vincent Broman) Date: Fri, 18 Apr 2008 18:19:09 -0700 Subject: [Numpy-discussion] powerpc yellow dog linux port of numpy In-Reply-To: References: Message-ID: <200804181819.09930@b00d61a8cecf8b2266f81358fd170621.navy.mil> I reported back on August 30 to this list, with some discussion following on September 4 and 5, about my attempt to build numpy on an ancient powerpc setup. I'm running yellow dog linux 2.1, gcc 2.95.3.20010111, on processors from Curtiss-Wright Controls. Don't tell me to just upgrade; this configuration will be fighting the good fight for several more years. I just retried with the latest numpy (svn yesterday) and gotten further than I did before. umathmodule.c gets many compiler errors from gcc, of two kinds. The simpler were like warning: conflicting types for built-in function `sinl' repeated for `cosl', `fabsl', and `sqrtl'. These seem to be caused by npy_longdouble being typedef'ed as double not long double, due to the latter two types having the same size. umathmodule.c defines its own sinl, sqrtl, etc. with npy_longdouble arguments and results, which then conflict with the builtin sinl, sqrtl provided by gcc that expect long double. I worked around that by adding the "-fno-builtin" argument to the extra_compiler_args in setup.py. The other compiler complaints from the same file were: inconsistent operand constraints in an `asm' which came from every line that raised a division by zero exception, the code in each case being "feraiseexcept( FE_DIVBYZERO)" after preprocessing. That function is defined in fenv.h with a "__THROW" attribute, but I saw no sign of it being an inline asm or anything. I don't understand "__THROW". I'm afraid I would need to find the asm code involved, before I could see what "operand constraints" are "inconsistent". Any hints where to look? Any way to make the call go to a nice simple library instead? Vincent Broman From robert.kern at gmail.com Fri Apr 18 21:35:39 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 18 Apr 2008 20:35:39 -0500 Subject: [Numpy-discussion] powerpc yellow dog linux port of numpy In-Reply-To: <200804181819.09930@b00d61a8cecf8b2266f81358fd170621.navy.mil> References: <200804181819.09930@b00d61a8cecf8b2266f81358fd170621.navy.mil> Message-ID: <3d375d730804181835y11dac2au909780b549a5f8d@mail.gmail.com> On Fri, Apr 18, 2008 at 8:19 PM, Vincent Broman wrote: > I reported back on August 30 to this list, > with some discussion following on September 4 and 5, > about my attempt to build numpy on an ancient powerpc setup. > I'm running yellow dog linux 2.1, gcc 2.95.3.20010111, on processors from Curtiss-Wright Controls. > Don't tell me to just upgrade; this configuration will be > fighting the good fight for several more years. > I just retried with the latest numpy (svn yesterday) and gotten further than I did before. > > umathmodule.c gets many compiler errors from gcc, of two kinds. > > The simpler were like > > warning: conflicting types for built-in function `sinl' > > repeated for `cosl', `fabsl', and `sqrtl'. > These seem to be caused by npy_longdouble being typedef'ed as double not long double, > due to the latter two types having the same size. > umathmodule.c defines its own sinl, sqrtl, etc. with npy_longdouble arguments and results, > which then conflict with the builtin sinl, sqrtl provided by gcc that expect long double. We check for the presence of expl() to determine if all of the rest are provided and set the HAVE_LONGDOUBLE_FUNCS flag. It is possible that you don't have expl() but do have these other functions. > I worked around that by adding the "-fno-builtin" argument to the extra_compiler_args in setup.py. This is not unreasonable. > The other compiler complaints from the same file were: > > inconsistent operand constraints in an `asm' > > which came from every line that raised a division by zero exception, > the code in each case being "feraiseexcept( FE_DIVBYZERO)" after preprocessing. > That function is defined in fenv.h with a "__THROW" attribute, > but I saw no sign of it being an inline asm or anything. > I don't understand "__THROW". > > I'm afraid I would need to find the asm code involved, before I could > see what "operand constraints" are "inconsistent". > Any hints where to look? > Any way to make the call go to a nice simple library instead? In the file numpy/core/include/numpy/ufuncobject.h, there is a stanza that looks like this: #if defined(__GLIBC__) || defined(__APPLE__) || defined(__MINGW32__) || defined(__FreeBSD__) #include #elif defined(__CYGWIN__) #include "fenv/fenv.c" #endif I assume that you have __GLIBC__ defined. You will have to find your platform's fenv.[ch] file from your libc sources. You may want to comment out all of that and use our included #include "fenv/fenv.c" Also, edit numpy/core/setup.py to include these files for your platform in addition to Cygwin. # Don't install fenv unless we need them. if sys.platform == 'cygwin': config.add_data_dir('include/numpy/fenv') -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Sat Apr 19 00:47:21 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 18 Apr 2008 22:47:21 -0600 Subject: [Numpy-discussion] numpy1.2 : make sorts unary ufuncs Message-ID: The signature for a ufunc is something like @TYPE at _@kind@(char **args, intp *dimensions, intp *steps, void *func) Which contains all the info necessary to do a sort. Means and other such functions could also be implemented that way. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Apr 19 00:53:28 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 18 Apr 2008 23:53:28 -0500 Subject: [Numpy-discussion] numpy1.2 : make sorts unary ufuncs In-Reply-To: References: Message-ID: <3d375d730804182153w7afbdfbejccf9ea1b44584db@mail.gmail.com> On Fri, Apr 18, 2008 at 11:47 PM, Charles R Harris wrote: > The signature for a ufunc is something like > > @TYPE at _@kind@(char **args, intp *dimensions, intp *steps, void *func) > > Which contains all the info necessary to do a sort. Means and other such > functions could also be implemented that way. axis? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Sat Apr 19 00:58:44 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 18 Apr 2008 22:58:44 -0600 Subject: [Numpy-discussion] numpy1.2 : make sorts unary ufuncs In-Reply-To: <3d375d730804182153w7afbdfbejccf9ea1b44584db@mail.gmail.com> References: <3d375d730804182153w7afbdfbejccf9ea1b44584db@mail.gmail.com> Message-ID: On Fri, Apr 18, 2008 at 10:53 PM, Robert Kern wrote: > On Fri, Apr 18, 2008 at 11:47 PM, Charles R Harris > wrote: > > The signature for a ufunc is something like > > > > @TYPE at _@kind@(char **args, intp *dimensions, intp *steps, void *func) > > > > Which contains all the info necessary to do a sort. Means and other such > > functions could also be implemented that way. > > axis? > I believe Travis is already selecting an axis to get the best cache usage, so it just needs a tweak. Axis = None is the special case. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Apr 19 01:02:00 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 19 Apr 2008 00:02:00 -0500 Subject: [Numpy-discussion] numpy1.2 : make sorts unary ufuncs In-Reply-To: References: <3d375d730804182153w7afbdfbejccf9ea1b44584db@mail.gmail.com> Message-ID: <3d375d730804182202u5851535as70e60ce46b1b5f74@mail.gmail.com> On Fri, Apr 18, 2008 at 11:58 PM, Charles R Harris wrote: > > On Fri, Apr 18, 2008 at 10:53 PM, Robert Kern wrote: > > > > On Fri, Apr 18, 2008 at 11:47 PM, Charles R Harris > > wrote: > > > The signature for a ufunc is something like > > > > > > @TYPE at _@kind@(char **args, intp *dimensions, intp *steps, void *func) > > > > > > Which contains all the info necessary to do a sort. Means and other such > > > functions could also be implemented that way. > > > > axis? > > I believe Travis is already selecting an axis to get the best cache usage, > so it just needs a tweak. Axis = None is the special case. So what is the benefit here? Why bother? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Sat Apr 19 01:18:05 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 18 Apr 2008 23:18:05 -0600 Subject: [Numpy-discussion] numpy1.2 : make sorts unary ufuncs In-Reply-To: <3d375d730804182202u5851535as70e60ce46b1b5f74@mail.gmail.com> References: <3d375d730804182153w7afbdfbejccf9ea1b44584db@mail.gmail.com> <3d375d730804182202u5851535as70e60ce46b1b5f74@mail.gmail.com> Message-ID: On Fri, Apr 18, 2008 at 11:02 PM, Robert Kern wrote: > On Fri, Apr 18, 2008 at 11:58 PM, Charles R Harris > wrote: > > > > On Fri, Apr 18, 2008 at 10:53 PM, Robert Kern > wrote: > > > > > > On Fri, Apr 18, 2008 at 11:47 PM, Charles R Harris > > > wrote: > > > > The signature for a ufunc is something like > > > > > > > > @TYPE at _@kind@(char **args, intp *dimensions, intp *steps, void > *func) > > > > > > > > Which contains all the info necessary to do a sort. Means and other > such > > > > functions could also be implemented that way. > > > > > > axis? > > > > I believe Travis is already selecting an axis to get the best cache > usage, > > so it just needs a tweak. Axis = None is the special case. > > So what is the benefit here? Why bother? > Code reuse and a unified concept. Why duplicate all the complication as we do now? Argsorts become binary ufuncs, etc. Besides, I'm trying to track down a ufunc call bug, feeling annoyed, and it would be nice to have all the crap in one location so we can work on cleaning it up. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Apr 19 01:23:33 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 19 Apr 2008 00:23:33 -0500 Subject: [Numpy-discussion] numpy1.2 : make sorts unary ufuncs In-Reply-To: References: <3d375d730804182153w7afbdfbejccf9ea1b44584db@mail.gmail.com> <3d375d730804182202u5851535as70e60ce46b1b5f74@mail.gmail.com> Message-ID: <3d375d730804182223q604d6f8bg23e1a15f4192038a@mail.gmail.com> On Sat, Apr 19, 2008 at 12:18 AM, Charles R Harris wrote: > On Fri, Apr 18, 2008 at 11:02 PM, Robert Kern wrote: > > On Fri, Apr 18, 2008 at 11:58 PM, Charles R Harris > > > > wrote: > > > > > > On Fri, Apr 18, 2008 at 10:53 PM, Robert Kern > wrote: > > > > > > > > On Fri, Apr 18, 2008 at 11:47 PM, Charles R Harris > > > > wrote: > > > > > The signature for a ufunc is something like > > > > > > > > > > @TYPE at _@kind@(char **args, intp *dimensions, intp *steps, void > *func) > > > > > > > > > > Which contains all the info necessary to do a sort. Means and other > such > > > > > functions could also be implemented that way. > > > > > > > > axis? > > > > > > I believe Travis is already selecting an axis to get the best cache > usage, > > > so it just needs a tweak. Axis = None is the special case. > > > > So what is the benefit here? Why bother? > > Code reuse and a unified concept. Why duplicate all the complication as we > do now? Argsorts become binary ufuncs, etc. Besides, I'm trying to track > down a ufunc call bug, feeling annoyed, and it would be nice to have all the > crap in one location so we can work on cleaning it up. I have a suspicion that tweaking ufuncs to allow the special case of sorting will negate the benefits. It smells like an abuse of the machinery rather than a natural merging of similar functionality. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Sat Apr 19 01:29:17 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 18 Apr 2008 23:29:17 -0600 Subject: [Numpy-discussion] numpy1.2 : make sorts unary ufuncs In-Reply-To: <3d375d730804182223q604d6f8bg23e1a15f4192038a@mail.gmail.com> References: <3d375d730804182153w7afbdfbejccf9ea1b44584db@mail.gmail.com> <3d375d730804182202u5851535as70e60ce46b1b5f74@mail.gmail.com> <3d375d730804182223q604d6f8bg23e1a15f4192038a@mail.gmail.com> Message-ID: On Fri, Apr 18, 2008 at 11:23 PM, Robert Kern wrote: > On Sat, Apr 19, 2008 at 12:18 AM, Charles R Harris > wrote: > > On Fri, Apr 18, 2008 at 11:02 PM, Robert Kern > wrote: > > > On Fri, Apr 18, 2008 at 11:58 PM, Charles R Harris > > > > > > wrote: > > > > > > > > On Fri, Apr 18, 2008 at 10:53 PM, Robert Kern > > > wrote: > > > > > > > > > > On Fri, Apr 18, 2008 at 11:47 PM, Charles R Harris > > > > > wrote: > > > > > > The signature for a ufunc is something like > > > > > > > > > > > > @TYPE at _@kind@(char **args, intp *dimensions, intp *steps, void > > *func) > > > > > > > > > > > > Which contains all the info necessary to do a sort. Means and > other > > such > > > > > > functions could also be implemented that way. > > > > > > > > > > axis? > > > > > > > > I believe Travis is already selecting an axis to get the best cache > > usage, > > > > so it just needs a tweak. Axis = None is the special case. > > > > > > So what is the benefit here? Why bother? > > > > Code reuse and a unified concept. Why duplicate all the complication as > we > > do now? Argsorts become binary ufuncs, etc. Besides, I'm trying to track > > down a ufunc call bug, feeling annoyed, and it would be nice to have all > the > > crap in one location so we can work on cleaning it up. > > I have a suspicion that tweaking ufuncs to allow the special case of > sorting will negate the benefits. It smells like an abuse of the > machinery rather than a natural merging of similar functionality. Don't think so, the ufunc inner loops are all over a given axis with the other indices varying. I think anything that processes arrays along an axis is a natural for the framework. And if we add bells and whistles, like threads, it might be nice to keep everything in one place. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Apr 19 01:41:47 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 18 Apr 2008 23:41:47 -0600 Subject: [Numpy-discussion] numpy1.2 : make sorts unary ufuncs In-Reply-To: References: <3d375d730804182153w7afbdfbejccf9ea1b44584db@mail.gmail.com> <3d375d730804182202u5851535as70e60ce46b1b5f74@mail.gmail.com> <3d375d730804182223q604d6f8bg23e1a15f4192038a@mail.gmail.com> Message-ID: On Fri, Apr 18, 2008 at 11:29 PM, Charles R Harris < charlesr.harris at gmail.com> wrote: > > > On Fri, Apr 18, 2008 at 11:23 PM, Robert Kern > wrote: > > > On Sat, Apr 19, 2008 at 12:18 AM, Charles R Harris > > wrote: > > > On Fri, Apr 18, 2008 at 11:02 PM, Robert Kern > > wrote: > > > > On Fri, Apr 18, 2008 at 11:58 PM, Charles R Harris > > > > > > > > wrote: > > > > > > > > > > On Fri, Apr 18, 2008 at 10:53 PM, Robert Kern < > > robert.kern at gmail.com> > > > wrote: > > > > > > > > > > > > On Fri, Apr 18, 2008 at 11:47 PM, Charles R Harris > > > > > > wrote: > > > > > > > The signature for a ufunc is something like > > > > > > > > > > > > > > @TYPE at _@kind@(char **args, intp *dimensions, intp *steps, void > > > *func) > > > > > > > > > > > > > > Which contains all the info necessary to do a sort. Means and > > other > > > such > > > > > > > functions could also be implemented that way. > > > > > > > > > > > > axis? > > > > > > > > > > I believe Travis is already selecting an axis to get the best > > cache > > > usage, > > > > > so it just needs a tweak. Axis = None is the special case. > > > > > > > > So what is the benefit here? Why bother? > > > > > > Code reuse and a unified concept. Why duplicate all the complication > > as we > > > do now? Argsorts become binary ufuncs, etc. Besides, I'm trying to > > track > > > down a ufunc call bug, feeling annoyed, and it would be nice to have > > all the > > > crap in one location so we can work on cleaning it up. > > > > I have a suspicion that tweaking ufuncs to allow the special case of > > sorting will negate the benefits. It smells like an abuse of the > > machinery rather than a natural merging of similar functionality. > > > Don't think so, the ufunc inner loops are all over a given axis with the > other indices varying. I think anything that processes arrays along an axis > is a natural for the framework. And if we add bells and whistles, like > threads, it might be nice to keep everything in one place. > You can also do matrix multiplication in a ufunc. It's got two inputs and an output. Needs two axis specs, but other than that goes in quite nicely. IIRC, BLAS already has the strides built in. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Sat Apr 19 02:20:38 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Sat, 19 Apr 2008 01:20:38 -0500 Subject: [Numpy-discussion] numpy1.2 : make sorts unary ufuncs In-Reply-To: References: Message-ID: <48098F36.5050505@enthought.com> Charles R Harris wrote: > The signature for a ufunc is something like > > @TYPE at _@kind@(char **args, intp *dimensions, intp *steps, void *func) > > Which contains all the info necessary to do a sort. Means and other > such functions could also be implemented that way. I dont' think the ufunc is the right place for this. Perhaps sorting can be viewed however as a "general function" which I'd like to add. Maybe, in that case the ufunc could be a special case of the "general function" concept. But, that is the direction I would expect the discussion to go. The ufunc object is a very particular kind of thing. The 1-d inner loop signature is a key part of it, but there is more to it than that. I don't see how the sorting function can be pushed into this concept. But, there is a case to be made for perhaps figuring out how to merge the data-type functions and the ufuncs into one idea that allows for better code re-use. -Travis From jh at physics.ucf.edu Sat Apr 19 02:32:31 2008 From: jh at physics.ucf.edu (Joe Harrington) Date: Sat, 19 Apr 2008 02:32:31 -0400 Subject: [Numpy-discussion] Truth value of an array In-Reply-To: (numpy-discussion-request@scipy.org) References: Message-ID: > Just cast your arrays to booleans if you want to do boolean operations > on them. It turns out there's an even better way: logical_and() and its friends do boolean operations on arrays. IDL solves the problem exactly as numpy does, erroring on arrays in conditionals and short-circuiting boolean operators. They, too, provide logical_and() and friends to do guarranteed-to-execute-on-all- elements logical array operations, which was in fact why I thought to look for numpy's. I'm curious what Matlab does (I don't have access). What wasn't obvious was how to cast an array to bool. Doing that the obvious way doesn't work, as is the subject of this thread: numpy.bool(x) Traceback (most recent call last): File "", line 1, in ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() My next thought was: numpy.array(x, dtype=bool) array([ True, True, True, True, True, True, True, True, True, True], dtype=bool) This works, but is a lot to type, and it's pretty opaque to a new user that you'd have to do it this way. Then I discovered the numpy version of bool() (took 45 minutes): numpy.bool_(x) # note trailing underscore array([ True, True, True, True, True, True, True, True, True, True], dtype=bool) I just added bool_ and the other casting functions to the categorized list of functions on the web site. Each of these links points to the Numpy_Example_List_With_Doc page, where there is no example, however. I hope that's ok, for now. --jh-- From robert.kern at gmail.com Sat Apr 19 02:44:39 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 19 Apr 2008 01:44:39 -0500 Subject: [Numpy-discussion] Truth value of an array In-Reply-To: References: Message-ID: <3d375d730804182344p7cee7d7dn44a893c0cc79cd4@mail.gmail.com> On Sat, Apr 19, 2008 at 1:32 AM, Joe Harrington wrote: > What wasn't obvious was how to cast an array to bool. Just like any other type: x.astype(bool) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Sat Apr 19 02:55:49 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 19 Apr 2008 00:55:49 -0600 Subject: [Numpy-discussion] numpy1.2 : make sorts unary ufuncs In-Reply-To: <48098F36.5050505@enthought.com> References: <48098F36.5050505@enthought.com> Message-ID: On Sat, Apr 19, 2008 at 12:20 AM, Travis E. Oliphant wrote: > Charles R Harris wrote: > > The signature for a ufunc is something like > > > > @TYPE at _@kind@(char **args, intp *dimensions, intp *steps, void *func) > > > > Which contains all the info necessary to do a sort. Means and other > > such functions could also be implemented that way. > I dont' think the ufunc is the right place for this. Perhaps sorting > can be viewed however as a "general function" which I'd like to add. > Maybe, in that case the ufunc could be a special case of the "general > function" concept. But, that is the direction I would expect the > discussion to go. > > The ufunc object is a very particular kind of thing. The 1-d inner > loop signature is a key part of it, but there is more to it than that. > I don't see how the sorting function can be pushed into this concept. > Yes, but the inner loop is just something that uses the array values along that axis to produce another set of values, i.e., it is a vector valued function of vectors. So is a sort, so is argsort, so is the inner product, so on and so forth. That's what we have here: typedef void (*PyUFuncGenericFunction) (char **, npy_intp *, npy_intp *, void *); No difference that I can see. It is the call function in PyUFuncObject that matters. > But, there is a case to be made for perhaps figuring out how to merge > the data-type functions and the ufuncs into one idea that allows for > better code re-use. > I don't think we can get all of them, but I think we can get most. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Apr 19 03:12:57 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 19 Apr 2008 02:12:57 -0500 Subject: [Numpy-discussion] numpy1.2 : make sorts unary ufuncs In-Reply-To: References: <48098F36.5050505@enthought.com> Message-ID: <3d375d730804190012v30de7a1ft5dd5155317d247fa@mail.gmail.com> On Sat, Apr 19, 2008 at 1:55 AM, Charles R Harris wrote: > Yes, but the inner loop is just something that uses the array values along > that axis to produce another set of values, i.e., it is a vector valued > function of vectors. So is a sort, so is argsort, so is the inner product, > so on and so forth. That's what we have here: > > typedef void (*PyUFuncGenericFunction) (char **, npy_intp *, npy_intp *, > void *); > > No difference that I can see. It is the call function in PyUFuncObject that > matters. I believe this is the disconnect. From my perspective, the fact that the inner kernel function of a ufunc has a sufficient argument list to do a sort isn't important. The signature of that kernel function isn't what makes a ufunc; it's all of the code around it that does broadcasting, type matching and manipulation, etc. If we're changing that code to accommodate sorting, we haven't gained anything. We've just moved some code around; possibly we've reduced the line count, but I fear that we will muddy ufunc implementation with non-ufunc functionality and special cases. If you want to go down this road, I think you need to do what Travis suggests: factor out some of the common code between ufuncs and sorts into a "superclass" (not really, but you get the idea), and then implement ufuncs and sorts based on that. I think trying to shove sorts into ufuncs-qua-ufuncs is a bad idea. There is more than one path to code reuse. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at enthought.com Sat Apr 19 03:19:33 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Sat, 19 Apr 2008 02:19:33 -0500 Subject: [Numpy-discussion] numpy1.2 : make sorts unary ufuncs In-Reply-To: References: <48098F36.5050505@enthought.com> Message-ID: <48099D05.1030609@enthought.com> Charles R Harris wrote: > > > On Sat, Apr 19, 2008 at 12:20 AM, Travis E. Oliphant > > wrote: > > Charles R Harris wrote: > > The signature for a ufunc is something like > > > > @TYPE at _@kind@(char **args, intp *dimensions, intp *steps, void > *func) > > > > Which contains all the info necessary to do a sort. Means and other > > such functions could also be implemented that way. > I dont' think the ufunc is the right place for this. Perhaps > sorting > can be viewed however as a "general function" which I'd like to add. > Maybe, in that case the ufunc could be a special case of the "general > function" concept. But, that is the direction I would expect the > discussion to go. > > The ufunc object is a very particular kind of thing. The 1-d inner > loop signature is a key part of it, but there is more to it than that. > I don't see how the sorting function can be pushed into this concept. > > > Yes, but the inner loop is just something that uses the array values > along that axis to produce another set of values, i.e., it is a vector > valued function of vectors. So is a sort, so is argsort, so is the > inner product, so on and so forth. That's what we have here: > > typedef void (*PyUFuncGenericFunction) (char **, npy_intp *, npy_intp > *, void *); > > No difference that I can see. It is the call function in > PyUFuncObject that matters. Yes, you are right that the low-level 1-d loop signature is similar (or can be made similar) to other signatures. I'm not sure what is gained by that, though. Can you explain exactly what you want to do? In my mind, the call function is exactly what makes a ufunc a ufunc, so I'm not sure why discussing sort in the same breath is meaningful. Perhaps we are just running around each others definitions. -Travis From zelbier at gmail.com Sat Apr 19 03:21:40 2008 From: zelbier at gmail.com (Olivier Verdier) Date: Sat, 19 Apr 2008 09:21:40 +0200 Subject: [Numpy-discussion] Truth value of an array In-Reply-To: References: <20080418115517.GB30358@logilab.fr> <6573665e-c09f-4a24-82bb-93e481cb781f@a22g2000hsc.googlegroups.com> Message-ID: Anne, thank you, this was the example I was looking for. Indeed A!=B would not work as expected if the bool(A) always returned A.all(). Now I can teach my student why there is no automatic conversion from boolean arrays to booleans. == Olivier On 18/04/2008, Anne Archibald wrote: > On 18/04/2008, Olivier wrote: > > Let's restrict the discussion the case to boolean arrays (dtype bool), > > since all the comparisons (A==B, A!=B, A > arrays). > > > > So I have an array filled with booleans. Is there a reason not to map > > "bool(A)" to "A.all()" but instead raise an exception? > > > > As far as I can see, "if A==B" is clear enough; one always means > > "(A==B).all()". Isn't that so? > > > > I would like to see examples where there is clearly an ambiguity so > > that I understand the benefit of having an exception raised for > > boolean matrices instead of simply using all(). > > > > If there is no such clear example, then why not map "bool(A)" to > > "A.all()" in numpy? > > > In [1]: import numpy as np > > In [2]: A = np.array([0,1]) > > In [3]: B = np.array([0,0]) > > In [4]: (A==B).all() > Out[4]: False > > In [5]: (A!=B).all() > Out[5]: False > > One would expect A!=B to be the logical opposite of A==B, but with > your proposed suggestion it is not. > > In math, when comparing functions, one can compare the functions as a > whole, or one can compare them pointwise. numpy's implementation does > pointwise comparison, for a variety of good reasons. As for why > converting an array to a boolean doesn't automatically do all(), if > you don't know you are dealing with an array and using all(), you will > surely shoot yourself in the foot sooner rather than later (as the > above example shows). If you do, simply wrapping it in all() is easy > and clear. > > > Anne > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Sat Apr 19 03:29:13 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 19 Apr 2008 01:29:13 -0600 Subject: [Numpy-discussion] numpy1.2 : make sorts unary ufuncs In-Reply-To: <3d375d730804190012v30de7a1ft5dd5155317d247fa@mail.gmail.com> References: <48098F36.5050505@enthought.com> <3d375d730804190012v30de7a1ft5dd5155317d247fa@mail.gmail.com> Message-ID: On Sat, Apr 19, 2008 at 1:12 AM, Robert Kern wrote: > On Sat, Apr 19, 2008 at 1:55 AM, Charles R Harris > wrote: > > > Yes, but the inner loop is just something that uses the array values > along > > that axis to produce another set of values, i.e., it is a vector valued > > function of vectors. So is a sort, so is argsort, so is the inner > product, > > so on and so forth. That's what we have here: > > > > typedef void (*PyUFuncGenericFunction) (char **, npy_intp *, npy_intp *, > > void *); > > > > No difference that I can see. It is the call function in PyUFuncObject > that > > matters. > > I believe this is the disconnect. From my perspective, the fact that > the inner kernel function of a ufunc has a sufficient argument list to > do a sort isn't important. The signature of that kernel function isn't > what makes a ufunc; it's all of the code around it that does > broadcasting, type matching and manipulation, etc. If we're changing > that code to accommodate sorting, we haven't gained anything. We've > just moved some code around; possibly we've reduced the line count, > but I fear that we will muddy ufunc implementation with non-ufunc > functionality and special cases. > > If you want to go down this road, I think you need to do what Travis > suggests: factor out some of the common code between ufuncs and sorts > into a "superclass" (not really, but you get the idea), and then > implement ufuncs and sorts based on that. I think trying to shove > sorts into ufuncs-qua-ufuncs is a bad idea. There is more than one > path to code reuse. > Right now we have: typedef struct { PyObject_HEAD int nin, nout, nargs; int identity; PyUFuncGenericFunction *functions; void **data; int ntypes; int check_return; char *name, *types; char *doc; void *ptr; PyObject *obj; PyObject *userloops; } PyUFuncObject; Which could be derived from something slightly more general. We could also leave out reduce, accumulate, etc., which are special cases. We then have common code for registration, etc. The call function still has to check types, dispatch the calls for the axis, maybe create output arrays, as for maximum.reduce, and so on. Broadcasting isn't applicable to unary type things and many functions, say in argsort, look unary from the top, so that doesn't enter in. Chuck > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Apr 19 03:46:22 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 19 Apr 2008 01:46:22 -0600 Subject: [Numpy-discussion] numpy1.2 : make sorts unary ufuncs In-Reply-To: References: <48098F36.5050505@enthought.com> <3d375d730804190012v30de7a1ft5dd5155317d247fa@mail.gmail.com> Message-ID: On Sat, Apr 19, 2008 at 1:29 AM, Charles R Harris wrote: > > On Sat, Apr 19, 2008 at 1:12 AM, Robert Kern > wrote: > > > On Sat, Apr 19, 2008 at 1:55 AM, Charles R Harris > > wrote: > > > > > Yes, but the inner loop is just something that uses the array values > > along > > > that axis to produce another set of values, i.e., it is a vector > > valued > > > function of vectors. So is a sort, so is argsort, so is the inner > > product, > > > so on and so forth. That's what we have here: > > > > > > typedef void (*PyUFuncGenericFunction) (char **, npy_intp *, npy_intp > > *, > > > void *); > > > > > > No difference that I can see. It is the call function in > > PyUFuncObject that > > > matters. > > > > I believe this is the disconnect. From my perspective, the fact that > > the inner kernel function of a ufunc has a sufficient argument list to > > do a sort isn't important. The signature of that kernel function isn't > > what makes a ufunc; it's all of the code around it that does > > broadcasting, type matching and manipulation, etc. If we're changing > > that code to accommodate sorting, we haven't gained anything. We've > > just moved some code around; possibly we've reduced the line count, > > but I fear that we will muddy ufunc implementation with non-ufunc > > functionality and special cases. > > > > If you want to go down this road, I think you need to do what Travis > > suggests: factor out some of the common code between ufuncs and sorts > > into a "superclass" (not really, but you get the idea), and then > > implement ufuncs and sorts based on that. I think trying to shove > > sorts into ufuncs-qua-ufuncs is a bad idea. There is more than one > > path to code reuse. > > > > Right now we have: > > typedef struct { > PyObject_HEAD > int nin, nout, nargs; > int identity; > PyUFuncGenericFunction *functions; > void **data; > int ntypes; > int check_return; > char *name, *types; > char *doc; > void *ptr; > PyObject *obj; > PyObject *userloops; > } PyUFuncObject; > > Which could be derived from something slightly more general. We could also > leave out reduce, accumulate, etc., which are special cases. We then have > common code for registration, etc. The call function still has to check > types, dispatch the calls for the axis, maybe create output arrays, as for > maximum.reduce, and so on. Broadcasting isn't applicable to unary type > things and many functions, say in argsort, look unary from the top, so that > doesn't enter in. > For instance static void BOOL_ at kind@(char **args, intp *dimensions, intp *steps, void *func) { register intp i; intp is1=steps[0],is2=steps[1],os=steps[2], n=dimensions[0]; char *i1=args[0], *i2=args[1], *op=args[2]; Bool in1, in2; for(i=0; i From millman at berkeley.edu Sat Apr 19 03:46:32 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Sat, 19 Apr 2008 00:46:32 -0700 Subject: [Numpy-discussion] OSX installer: please test In-Reply-To: <764e38540804161436j5ce7a468w8bea6a3c3598d69e@mail.gmail.com> References: <764e38540804161436j5ce7a468w8bea6a3c3598d69e@mail.gmail.com> Message-ID: On Wed, Apr 16, 2008 at 2:36 PM, Christopher Burns wrote: > I've built a Universal Mac binary for numpy 1.1.0. If Mac people would > kindly test it, I'd appreciate any feedback. > > Download here: > https://cirl.berkeley.edu/numpy/numpy-1.1.0rc1-py2.5-macosx10.5.dmg > > Technical details: > - Built on OSX 10.5.2, Intel Core 2 Duo > - Using XCode 3.0 with gcc 4.0.1 and the Accelerate.framework 1.4.2, > gfortran 4.2.1 and Python 2.5.2 from python.org (MacPython) > - Used bdist_mpkg 0.4.3 to make the distribution > > Since it uses the MacPython, it will install numpy under here: > /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages Hello, If anyone has tested this out, please let us know. If you have a Mac and haven't tested this, please do so. This is one of the very last things that I need feedback on before finally scheduling the next NumPy release. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From stefan at sun.ac.za Sat Apr 19 04:45:54 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sat, 19 Apr 2008 10:45:54 +0200 Subject: [Numpy-discussion] OSX installer: please test In-Reply-To: References: <764e38540804161436j5ce7a468w8bea6a3c3598d69e@mail.gmail.com> Message-ID: <9457e7c80804190145j60249bbcv9e73a891cbdbdb6f@mail.gmail.com> Hi, On 19/04/2008, Jarrod Millman wrote: > On Wed, Apr 16, 2008 at 2:36 PM, Christopher Burns wrote: > > I've built a Universal Mac binary for numpy 1.1.0. If Mac people would > > kindly test it, I'd appreciate any feedback. > > > > Download here: > > https://cirl.berkeley.edu/numpy/numpy-1.1.0rc1-py2.5-macosx10.5.dmg > > > > Technical details: > > - Built on OSX 10.5.2, Intel Core 2 Duo > > - Using XCode 3.0 with gcc 4.0.1 and the Accelerate.framework 1.4.2, > > gfortran 4.2.1 and Python 2.5.2 from python.org (MacPython) > > - Used bdist_mpkg 0.4.3 to make the distribution > > > > Since it uses the MacPython, it will install numpy under here: > > /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages > > > Hello, > > If anyone has tested this out, please let us know. If you have a Mac > and haven't tested this, please do so. This is one of the very last > things that I need feedback on before finally scheduling the next > NumPy release. Is there any reason why "System Python 2.5" (specifically) is a requirement? I run Python 2.5 from macports, so the installer prohibits me from installing NumPy. Cheers St?fan From robert.kern at gmail.com Sat Apr 19 04:54:32 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 19 Apr 2008 03:54:32 -0500 Subject: [Numpy-discussion] OSX installer: please test In-Reply-To: <9457e7c80804190145j60249bbcv9e73a891cbdbdb6f@mail.gmail.com> References: <764e38540804161436j5ce7a468w8bea6a3c3598d69e@mail.gmail.com> <9457e7c80804190145j60249bbcv9e73a891cbdbdb6f@mail.gmail.com> Message-ID: <3d375d730804190154n2abb8b92le8a855733756c1fe@mail.gmail.com> On Sat, Apr 19, 2008 at 3:45 AM, St?fan van der Walt wrote: > Is there any reason why "System Python 2.5" (specifically) is a > requirement? Actually, it's not the System Python 2.5 but the python.org binary. Installer.app isn't particularly flexible, so we can really only install to a given location. Consequently, we install a binary for the Python that we built the package with. In any case, MacPorts' Python is not a framework build, so the binary wouldn't work anyways. > I run Python 2.5 from macports, so the installer > prohibits me from installing NumPy. MacPorts users' needs are (or should be) served by the MacPorts numpy package. The installer doesn't prohibit you from anything; it's just not serving you. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Sat Apr 19 04:55:30 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 19 Apr 2008 03:55:30 -0500 Subject: [Numpy-discussion] OSX installer: please test In-Reply-To: References: <764e38540804161436j5ce7a468w8bea6a3c3598d69e@mail.gmail.com> Message-ID: <3d375d730804190155i32e1f008w6aa0cc23970fc7d@mail.gmail.com> On Sat, Apr 19, 2008 at 2:46 AM, Jarrod Millman > If anyone has tested this out, please let us know. If you have a Mac > and haven't tested this, please do so. This is one of the very last > things that I need feedback on before finally scheduling the next > NumPy release. Works for me on OS X 10.5.2 (Intel Core 2 Duo). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From skip at pobox.com Sat Apr 19 05:56:03 2008 From: skip at pobox.com (Skip Montanaro) Date: Sat, 19 Apr 2008 09:56:03 +0000 (UTC) Subject: [Numpy-discussion] Suppress colored output during numpy build? Message-ID: Is it possible to suppress the colored output that's emitted during the build process? I find it next-to-impossible to read yellow text on my white terminal background. I saw nothing in the --help-commands output which suggested this was possible. Thanks, Skip Montanaro From robert.kern at gmail.com Sat Apr 19 06:23:32 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 19 Apr 2008 05:23:32 -0500 Subject: [Numpy-discussion] Suppress colored output during numpy build? In-Reply-To: References: Message-ID: <3d375d730804190323o153cd60dk2d4085e7957873c4@mail.gmail.com> On Sat, Apr 19, 2008 at 4:56 AM, Skip Montanaro wrote: > Is it possible to suppress the colored output that's emitted > during the build process? I find it next-to-impossible to > read yellow text on my white terminal background. I saw nothing > in the --help-commands output which suggested this was possible. The yellow, at least, has been fixed in r5039. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From lxander.m at gmail.com Sat Apr 19 06:50:13 2008 From: lxander.m at gmail.com (Alexander Michael) Date: Sat, 19 Apr 2008 06:50:13 -0400 Subject: [Numpy-discussion] OSX installer: please test In-Reply-To: <764e38540804161436j5ce7a468w8bea6a3c3598d69e@mail.gmail.com> References: <764e38540804161436j5ce7a468w8bea6a3c3598d69e@mail.gmail.com> Message-ID: <525f23e80804190350q5cee8b70vd4d910e2f518fb8f@mail.gmail.com> On Wed, Apr 16, 2008 at 5:36 PM, Christopher Burns wrote: > I've built a Universal Mac binary for numpy 1.1.0. If Mac people would > kindly test it, I'd appreciate any feedback. > > > Download here: > https://cirl.berkeley.edu/numpy/numpy-1.1.0rc1-py2.5-macosx10.5.dmg > > Technical details: > - Built on OSX 10.5.2, Intel Core 2 Duo > - Using XCode 3.0 with gcc 4.0.1 and the Accelerate.framework 1.4.2, > gfortran 4.2.1 and Python 2.5.2 from python.org (MacPython) > - Used bdist_mpkg 0.4.3 to make the distribution > > > Since it uses the MacPython, it will install numpy under here: > /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages Successfully installed and passed all tests on: Intel Core 2 Duo, Mac OS X 10.5.2 PowerPC G4, Mac OS X 10.4.11 Thanks! Alex From rpyle at post.harvard.edu Sat Apr 19 10:34:25 2008 From: rpyle at post.harvard.edu (Robert Pyle) Date: Sat, 19 Apr 2008 10:34:25 -0400 Subject: [Numpy-discussion] OSX installer: please test In-Reply-To: References: <764e38540804161436j5ce7a468w8bea6a3c3598d69e@mail.gmail.com> Message-ID: Hi, I have installed it with no problems on a dual G5 (i.e., PPC). It works fine, but I haven't beat on it very hard. (Array manipulations, fft, math functions, mostly). Bob Pyle On Apr 19, 2008, at 3:46 AM, Jarrod Millman wrote: > On Wed, Apr 16, 2008 at 2:36 PM, Christopher Burns > wrote: >> I've built a Universal Mac binary for numpy 1.1.0. If Mac people >> would >> kindly test it, I'd appreciate any feedback. >> >> Download here: >> https://cirl.berkeley.edu/numpy/numpy-1.1.0rc1-py2.5-macosx10.5.dmg >> >> Technical details: >> - Built on OSX 10.5.2, Intel Core 2 Duo >> - Using XCode 3.0 with gcc 4.0.1 and the Accelerate.framework 1.4.2, >> gfortran 4.2.1 and Python 2.5.2 from python.org (MacPython) >> - Used bdist_mpkg 0.4.3 to make the distribution >> >> Since it uses the MacPython, it will install numpy under here: >> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/ >> site-packages > > Hello, > > If anyone has tested this out, please let us know. If you have a Mac > and haven't tested this, please do so. This is one of the very last > things that I need feedback on before finally scheduling the next > NumPy release. From stefan at sun.ac.za Sat Apr 19 12:58:11 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sat, 19 Apr 2008 18:58:11 +0200 Subject: [Numpy-discussion] OSX installer: please test In-Reply-To: <3d375d730804190154n2abb8b92le8a855733756c1fe@mail.gmail.com> References: <764e38540804161436j5ce7a468w8bea6a3c3598d69e@mail.gmail.com> <9457e7c80804190145j60249bbcv9e73a891cbdbdb6f@mail.gmail.com> <3d375d730804190154n2abb8b92le8a855733756c1fe@mail.gmail.com> Message-ID: <9457e7c80804190958l7ba37535p4626def636c0062e@mail.gmail.com> On 19/04/2008, Robert Kern wrote: > In any case, MacPorts' Python is not a framework build, so the binary > wouldn't work anyways. Could you explain what you mean by that? Mine was compiled with the +framework flag, so it should be a "framework" build? I understand the reasons for requiring the standard Python.org installation though; thanks for clearing that up. Cheers St?fan From oliphant at enthought.com Sat Apr 19 13:40:44 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Sat, 19 Apr 2008 12:40:44 -0500 Subject: [Numpy-discussion] numpy1.2 : make sorts unary ufuncs In-Reply-To: References: <48098F36.5050505@enthought.com> <3d375d730804190012v30de7a1ft5dd5155317d247fa@mail.gmail.com> Message-ID: <480A2E9C.4030101@enthought.com> Charles R Harris wrote: > > > On Sat, Apr 19, 2008 at 1:29 AM, Charles R Harris > > wrote: > > > > On Sat, Apr 19, 2008 at 1:12 AM, Robert Kern > > wrote: > > On Sat, Apr 19, 2008 at 1:55 AM, Charles R Harris > > > wrote: > > > Yes, but the inner loop is just something that uses the > array values along > > that axis to produce another set of values, i.e., it is a > vector valued > > function of vectors. So is a sort, so is argsort, so is the > inner product, > > so on and so forth. That's what we have here: > > > > typedef void (*PyUFuncGenericFunction) (char **, npy_intp *, > npy_intp *, > > void *); > > > > No difference that I can see. It is the call function in > PyUFuncObject that > > matters. > > I believe this is the disconnect. From my perspective, the > fact that > the inner kernel function of a ufunc has a sufficient argument > list to > do a sort isn't important. The signature of that kernel > function isn't > what makes a ufunc; it's all of the code around it that does > broadcasting, type matching and manipulation, etc. If we're > changing > that code to accommodate sorting, we haven't gained anything. > We've > just moved some code around; possibly we've reduced the line > count, > but I fear that we will muddy ufunc implementation with non-ufunc > functionality and special cases. > > If you want to go down this road, I think you need to do what > Travis > suggests: factor out some of the common code between ufuncs > and sorts > into a "superclass" (not really, but you get the idea), and then > implement ufuncs and sorts based on that. I think trying to shove > sorts into ufuncs-qua-ufuncs is a bad idea. There is more than one > path to code reuse. > > > Right now we have: > > typedef struct { > PyObject_HEAD > int nin, nout, nargs; > int identity; > PyUFuncGenericFunction *functions; > void **data; > int ntypes; > int check_return; > char *name, *types; > char *doc; > void *ptr; > PyObject *obj; > PyObject *userloops; > } PyUFuncObject; > > Which could be derived from something slightly more general. We > could also leave out reduce, accumulate, etc., which are special > cases. We then have common code for registration, etc. The call > function still has to check types, dispatch the calls for the > axis, maybe create output arrays, as for maximum.reduce, and so > on. Broadcasting isn't applicable to unary type things and many > functions, say in argsort, look unary from the top, so that > doesn't enter in. > > > For instance > > static void > BOOL_ at kind@(char **args, intp *dimensions, intp *steps, void *func) > { > register intp i; > intp is1=steps[0],is2=steps[1],os=steps[2], n=dimensions[0]; > char *i1=args[0], *i2=args[1], *op=args[2]; > Bool in1, in2; > for(i=0; i in1 = (*((Bool *)i1) != 0); > in2 = (*((Bool *)i2) != 0); > *((Bool *)op)= in1 @OP@ in2; > } > } > > It looks to me like broadcasting is achieved by adjusting the step > size. The only bothersome detail here is getting the count from the > first dimension, that looks a bit fragile. It shouldn't be fragile. It's a historical accident that the signature looks like that. This is the signature inherited from Numeric. All of scipy-special would have to be changed in order to change it. Perhaps the thinking was that there would be multiple "counts" to keep track of at some time. But, I'm not sure. I've only seen the "first" entry used so dimensions is really just ptr_to_int rather than any kind of "shape". -Travis O. From charlesr.harris at gmail.com Sat Apr 19 16:11:56 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 19 Apr 2008 14:11:56 -0600 Subject: [Numpy-discussion] numpy1.2 : make sorts unary ufuncs In-Reply-To: <480A2E9C.4030101@enthought.com> References: <48098F36.5050505@enthought.com> <3d375d730804190012v30de7a1ft5dd5155317d247fa@mail.gmail.com> <480A2E9C.4030101@enthought.com> Message-ID: On Sat, Apr 19, 2008 at 11:40 AM, Travis E. Oliphant wrote: > Charles R Harris wrote: > > > > > > On Sat, Apr 19, 2008 at 1:29 AM, Charles R Harris > > > wrote: > > > > > > > > On Sat, Apr 19, 2008 at 1:12 AM, Robert Kern > > > wrote: > > > > On Sat, Apr 19, 2008 at 1:55 AM, Charles R Harris > > > > > wrote: > > > > > Yes, but the inner loop is just something that uses the > > array values along > > > that axis to produce another set of values, i.e., it is a > > vector valued > > > function of vectors. So is a sort, so is argsort, so is the > > inner product, > > > so on and so forth. That's what we have here: > > > > > > typedef void (*PyUFuncGenericFunction) (char **, npy_intp *, > > npy_intp *, > > > void *); > > > > > > No difference that I can see. It is the call function in > > PyUFuncObject that > > > matters. > > > > I believe this is the disconnect. From my perspective, the > > fact that > > the inner kernel function of a ufunc has a sufficient argument > > list to > > do a sort isn't important. The signature of that kernel > > function isn't > > what makes a ufunc; it's all of the code around it that does > > broadcasting, type matching and manipulation, etc. If we're > > changing > > that code to accommodate sorting, we haven't gained anything. > > We've > > just moved some code around; possibly we've reduced the line > > count, > > but I fear that we will muddy ufunc implementation with > non-ufunc > > functionality and special cases. > > > > If you want to go down this road, I think you need to do what > > Travis > > suggests: factor out some of the common code between ufuncs > > and sorts > > into a "superclass" (not really, but you get the idea), and then > > implement ufuncs and sorts based on that. I think trying to > shove > > sorts into ufuncs-qua-ufuncs is a bad idea. There is more than > one > > path to code reuse. > > > > > > Right now we have: > > > > typedef struct { > > PyObject_HEAD > > int nin, nout, nargs; > > int identity; > > PyUFuncGenericFunction *functions; > > void **data; > > int ntypes; > > int check_return; > > char *name, *types; > > char *doc; > > void *ptr; > > PyObject *obj; > > PyObject *userloops; > > } PyUFuncObject; > > > > Which could be derived from something slightly more general. We > > could also leave out reduce, accumulate, etc., which are special > > cases. We then have common code for registration, etc. The call > > function still has to check types, dispatch the calls for the > > axis, maybe create output arrays, as for maximum.reduce, and so > > on. Broadcasting isn't applicable to unary type things and many > > functions, say in argsort, look unary from the top, so that > > doesn't enter in. > > > > > > For instance > > > > static void > > BOOL_ at kind@(char **args, intp *dimensions, intp *steps, void *func) > > { > > register intp i; > > intp is1=steps[0],is2=steps[1],os=steps[2], n=dimensions[0]; > > char *i1=args[0], *i2=args[1], *op=args[2]; > > Bool in1, in2; > > for(i=0; i > in1 = (*((Bool *)i1) != 0); > > in2 = (*((Bool *)i2) != 0); > > *((Bool *)op)= in1 @OP@ in2; > > } > > } > > > > It looks to me like broadcasting is achieved by adjusting the step > > size. The only bothersome detail here is getting the count from the > > first dimension, that looks a bit fragile. > It shouldn't be fragile. It's a historical accident that the signature > looks like that. This is the signature inherited from Numeric. All of > scipy-special would have to be changed in order to change it. > > Perhaps the thinking was that there would be multiple "counts" to keep > track of at some time. But, I'm not sure. I've only seen the "first" > entry used so dimensions is really just ptr_to_int rather than any kind > of "shape". Ah, that was my mistake, then. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From malkarouri at yahoo.co.uk Sat Apr 19 19:23:45 2008 From: malkarouri at yahoo.co.uk (Muhammad Alkarouri) Date: Sun, 20 Apr 2008 00:23:45 +0100 (BST) Subject: [Numpy-discussion] OSX installer: please test Message-ID: <479600.38566.qm@web27903.mail.ukl.yahoo.com> Hi, Testing on Intel Core Duo Mac OS X 10.4.11: In [1]: import numpy as np In [2]: np.test(all=True) Numpy is installed in /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy Numpy version 1.1.0rc1 Python version 2.5.2 (r252:60911, Feb 22 2008, 07:57:53) [GCC 4.0.1 (Apple Computer, Inc. build 5363)] Found 10/10 tests for numpy.core.tests.test_defmatrix [... lots of other tests ] Found 14/14 tests for numpy.ma.tests.test_extras Failed importing /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/ma/tests/test_morestats.py: No module named scipy.stats.distributions Found 17/17 tests for numpy.ma.tests.test_mrecords [... some other tests ] ... [testing] .......................................E................... ====================================================================== ERROR: check_testUfuncRegression (numpy.ma.tests.test_old_ma.TestUfuncs) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/ma/tests/test_old_ma.py", line 657, in check_testUfuncRegression uf = getattr(umath, f) NameError: global name 'umath' is not defined ---------------------------------------------------------------------- Ran 1179 tests in 2.347s FAILED (errors=1) Actually another couple of tests failed before when scipy 0.6.0 was installed, but I removed that as I have no current work using it. So another question is whether the new numpy will work with the old scipy, or should one wait for a new scipy release/use svn? Keep up the good work. Regards, Muhammad Alkarouri __________________________________________________________ Sent from Yahoo! Mail. A Smarter Email http://uk.docs.yahoo.com/nowyoucan.html From efiring at hawaii.edu Sat Apr 19 22:33:47 2008 From: efiring at hawaii.edu (Eric Firing) Date: Sat, 19 Apr 2008 16:33:47 -1000 Subject: [Numpy-discussion] OWNDATA flag and reshape() -- views vs. copies In-Reply-To: References: Message-ID: <480AAB8B.6070705@hawaii.edu> Guillaume Desjardins wrote: > I'm pretty new to Python and numpy (longtime c / matlab programmer), > but after a read through some of the past threads and Travis' "Guide > to Numpy", I think I have a fairly good understanding of how the > reshape() function / methods work, with regards to views and copies. > For what its worth (and to make things clearer), my understanding is > that: > > * Function reshape: create a view, or if it can't (i.e. resulting > array cannot be expressed with constant stride parameters), a copy > * Method obj.reshape(): creates a view, or throws an exception if it can't > > Now assuming that is the case, I do not understand the behaviour of > the following code. > > a = array([[1,2,3,4],[5,6,7,8],[9,10,11,12]]); > a.flags > C_CONTIGUOUS : True > F_CONTIGUOUS : False > OWNDATA : True > WRITEABLE : True > ALIGNED : True > UPDATEIFCOPY : False > > b = a[1:,1:3] # creates view of a >>> array([[ 6, 7], > [10, 11]]) > b.flags > C_CONTIGUOUS : False > F_CONTIGUOUS : False > OWNDATA : False > WRITEABLE : True > ALIGNED : True > UPDATEIFCOPY : False > > c = reshape(b,(4,1)) # RESHAPE FUNCTION: should copy, as view of this > is impossible (? isn't it ?) > c >>> array([[ 6], > [ 7], > [10], > [11]]) > c.flags > C_CONTIGUOUS : True > F_CONTIGUOUS : False > OWNDATA : False > WRITEABLE : True > ALIGNED : True > UPDATEIFCOPY : False > > First question: if c is a copy of b, then why isn't the OWNDATA set to true ? It looks like creation of c is done in two steps: first a copy is made of b, and then c is returned as a view of that copy. I don't know if this is intentional; it sees a bit odd. I agree with you that I would have expected c to own its data rather than refer to a base hidden in the shadows. In [62]:b.base Out[62]: array([[ 1, 2, 3, 4], [ 5, 6, 7, 8], [ 9, 10, 11, 12]]) In [63]:b Out[63]: array([[ 6, 7], [10, 11]]) In [64]:c.base Out[64]: array([[ 6, 7], [10, 11]]) In [65]:c Out[65]: array([[ 6], [ 7], [10], [11]]) Eric From robert.kern at gmail.com Sun Apr 20 03:21:58 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 20 Apr 2008 02:21:58 -0500 Subject: [Numpy-discussion] OSX installer: please test In-Reply-To: <9457e7c80804190958l7ba37535p4626def636c0062e@mail.gmail.com> References: <764e38540804161436j5ce7a468w8bea6a3c3598d69e@mail.gmail.com> <9457e7c80804190145j60249bbcv9e73a891cbdbdb6f@mail.gmail.com> <3d375d730804190154n2abb8b92le8a855733756c1fe@mail.gmail.com> <9457e7c80804190958l7ba37535p4626def636c0062e@mail.gmail.com> Message-ID: <3d375d730804200021n4e51790dr5f882e01515c19ab@mail.gmail.com> On Sat, Apr 19, 2008 at 11:58 AM, St?fan van der Walt wrote: > On 19/04/2008, Robert Kern wrote: > > In any case, MacPorts' Python is not a framework build, so the binary > > wouldn't work anyways. > > Could you explain what you mean by that? Mine was compiled with the > +framework flag, so it should be a "framework" build? Hmm, I was looking at my python25/Portfile which does not have this variant and uses ./configure --disable-framework. The $Id$ crud at the top says it is from 2008-02-22. Perhaps you have an older or newer version of the MacPorts tree. I thought mine was up to date. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From matthias.baas at gmail.com Sun Apr 20 08:33:24 2008 From: matthias.baas at gmail.com (Matthias Baas) Date: Sun, 20 Apr 2008 13:33:24 +0100 Subject: [Numpy-discussion] __array_interface__ questions Message-ID: Hi, I would like to make use of the __array_interface__ to be able to get large data sets into and out of C/C++ functions without having to touch each item individually by Python code. I was reading through the __array_interface__ description and now I have a couple of questions to clarify things: - There are two attributes, __array_interface__ and __array_struct__, and the spec says that an array has to implement only one of the two and an object processing an array also has only to check for one those attributes. But this means both sides can be compliant to the spec but they still can't be used with each other. Or is this meant to be two separate specs that just serve the same purpose? - There is a version attribute specifying the version of the interface. If I support version 3, what am I supposed to do when I encounter a higher version number? Is it always guaranteed that higher versions are backwards compatible? (the last sentence in the description seems to suggest that but can this really be guaranteed?). Shouldn't there be something like a major and minor version where different major versions are incompatible to each other? Or maybe a current version number and a compatibility version number or something like that? - There are several variants how the "data" attribute can look like and in some cases it is necessary to use the Python buffer interface. Is there an example how to use that interface (as I haven't done this before)? Or even better, is there a reference implementation for module authors that they could just use and that provides a simple to use API for obtaining the internal buffer pointer and layout information given an arbitrary Python object? I realized that writing a correct implementation of this myself is not really a trivial task (mainly because of all the variations that are allowed by the spec). - Is there anything particular I have to know to make sure my module also works on 64bit systems? What types should I use/assume when reading the pointer from the "data" tuple? - I had a look at PIL 1.1.6 and the __array_interface__ dict that the image object provides. The first thing I noticed is that it doesn't have a "version" attribute even though the spec says this one is required. The second thing is that the "data" attribute is a string, so I would have to use the buffer interface to access the data. Is there a way to get at the data pointer from Python so that I can pass it to a C function via ctypes? - Using the array interface for reading data is fine, but what is the recommended way of writing data into a buffer when the number of items is not known in advance? (so what would be the equivalent of the C++ vector::push_back() call?) - What's the status on getting this interface into the Python core? Thanks, - Matthias - From millman at berkeley.edu Sun Apr 20 07:36:09 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Sun, 20 Apr 2008 04:36:09 -0700 Subject: [Numpy-discussion] f2py on windows cant find gfortran because split_quoted() removed from ccompiler.py Message-ID: Hey Pearu, Sorry about messing up distutils; I have been warned that it can be hairy. Anyway for those who haven't been following it, last November I emailed the list a couple of times (obviously I should have checked with Pearu and could have avoided this) asking if I could remove some code from numpy/distutils/ccompiler.py. Basically, we had our own version of distutils.util.split_quoted() in numpy/distutils/ccompiler.py, which made a minor change to distutils default behavior (the fix keeps quotes when a quoted word doesn't contain whitespace). I created a ticket and ended up removing the code: http://scipy.org/scipy/numpy/ticket/619 Now it appears that this actually is breaking f2py on Windows: http://scipy.org/scipy/numpy/ticket/723 Since my change may have broken something, I went ahead and reverted it in r5054: http://projects.scipy.org/scipy/numpy/changeset/5054 Hopefully, this will fix the problem with gfortran not being found and won't introduce new problems. Now we have come full circle and are back where we were when I originally brought this issue to the list. Our version of split_quoted() was added several years ago and should I think be submitted for inclusion upstream. I know that distutils isn't exactly being maintained, but--Pearu--is it possible that we could get this upstream? Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From stefan at sun.ac.za Sun Apr 20 09:32:12 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 20 Apr 2008 15:32:12 +0200 Subject: [Numpy-discussion] OSX installer: please test In-Reply-To: <3d375d730804200021n4e51790dr5f882e01515c19ab@mail.gmail.com> References: <764e38540804161436j5ce7a468w8bea6a3c3598d69e@mail.gmail.com> <9457e7c80804190145j60249bbcv9e73a891cbdbdb6f@mail.gmail.com> <3d375d730804190154n2abb8b92le8a855733756c1fe@mail.gmail.com> <9457e7c80804190958l7ba37535p4626def636c0062e@mail.gmail.com> <3d375d730804200021n4e51790dr5f882e01515c19ab@mail.gmail.com> Message-ID: <9457e7c80804200632i5de08547v7d3354d4fbb0c0ea@mail.gmail.com> Hi Robert On 20/04/2008, Robert Kern wrote: > On Sat, Apr 19, 2008 at 11:58 AM, St?fan van der Walt wrote: > > On 19/04/2008, Robert Kern wrote: > > > In any case, MacPorts' Python is not a framework build, so the binary > > > wouldn't work anyways. > > > > Could you explain what you mean by that? Mine was compiled with the > > +framework flag, so it should be a "framework" build? > > > Hmm, I was looking at my python25/Portfile which does not have this > variant and uses ./configure --disable-framework. The $Id$ crud at the > top says it is from 2008-02-22. Perhaps you have an older or newer > version of the MacPorts tree. I thought mine was up to date. This is the most up-to-date information I could find: http://trac.macosforge.org/projects/macports/ticket/11267 I have # $Id: Portfile 34729 2008-03-03 22:07:37Z raimue at macports.org $ Regards St?fan From kwgoodman at gmail.com Sun Apr 20 15:03:39 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Sun, 20 Apr 2008 12:03:39 -0700 Subject: [Numpy-discussion] Speed of matrix multiplication Message-ID: Why is a.T*b slower than M.dot(a.T, b)? Does it take longer to parse or something? >> a = M.randn(500,1) >> b = M.randn(500,1) >> timeit a.T*b 100000 loops, best of 3: 11 ?s per loop >> timeit M.dot(a.T, b) 100000 loops, best of 3: 6.72 ?s per loop >> a = M.randn(5,1) >> b = M.randn(5,1) >> timeit a.T*b 100000 loops, best of 3: 10.1 ?s per loop >> timeit M.dot(a.T, b) 100000 loops, best of 3: 6.16 ?s per loop I use matrices, but for inner loops I'll convert to arrays: >> a = a.A >> b = b.A >> dot = M.dot >> timeit dot(a.T, b) 1000000 loops, best of 3: 1.82 ?s per loop >> a = a.T >> timeit dot(a, b) 1000000 loops, best of 3: 1.44 ?s per loop From stefan at sun.ac.za Sun Apr 20 15:41:33 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 20 Apr 2008 21:41:33 +0200 Subject: [Numpy-discussion] Speed of matrix multiplication In-Reply-To: References: Message-ID: <9457e7c80804201241x7e786b52u15c56de55f36d94e@mail.gmail.com> On 20/04/2008, Keith Goodman wrote: > Why is a.T*b slower than M.dot(a.T, b)? Does it take longer to parse > or something? Looks like a bit of overhead. If I change the number of elements from 500 to 500000, the difference disappears. Regards St?fan From python-ml at nn7.de Sun Apr 20 16:58:34 2008 From: python-ml at nn7.de (Soeren Sonnenburg) Date: Sun, 20 Apr 2008 22:58:34 +0200 Subject: [Numpy-discussion] Shogun - A Large Scale Machine Learning Toolbox Message-ID: <1208725114.23244.6.camel@localhost> SHOGUN ( http://www.shogun-toolbox.org ) is a large scale machine learning toolbox with focus on especially Support Vector Machines (SVM). * It provides a generic SVM object interfacing to several different SVM implementations, among them the state of the art LibSVM, SVMLight, SVMLin and GPDT. Each of the SVMs can be combined with a variety of kernels. The toolbox not only provides efficient implementations of the most common kernels, like the Linear, Polynomial, Gaussian and Sigmoid Kernel but also comes with a number of recent string kernels as e.g. the Locality Improved, Fischer, TOP, Spectrum, Weighted Degree Kernel (with shifts). For the latter the efficient LINADD optimizations are implemented. * Also SHOGUN offers the freedom of working with custom pre-computed kernels. One of its key features is the combined kernel which can be constructed by a weighted linear combination of a number of sub-kernels, each of which not necessarily working on the same domain. An optimal sub-kernel weighting can be learned using Multiple Kernel Learning. Currently SVM 2-class classification and regression problems can be dealt with. * SHOGUN also implements a number of linear methods like Linear Discriminant Analysis (LDA), Linear Programming Machine (LPM), (Kernel) Perceptrons and features algorithms to train hidden markov models. * The input feature-objects can be dense, sparse or strings and of type int/short/double/char and can be converted into different feature types. Chains of preprocessors (e.g. substracting the mean) can be attached to each feature object allowing for on-the-fly pre-processing. * SHOGUN is implemented in C++ and interfaces to Matlab(tm), R, Octave and Python and is proudly released as Machine Learning Open Source Software ( http://mloss.org/ ) under the GPLv3+. From python-ml at nn7.de Sun Apr 20 16:59:55 2008 From: python-ml at nn7.de (Soeren Sonnenburg) Date: Sun, 20 Apr 2008 22:59:55 +0200 Subject: [Numpy-discussion] sparse matrices -> numpy ? Message-ID: <1208725195.23244.9.camel@localhost> Dear all, is there any schedule when sparse matrices will arrive in the numpy core? Best, Soeren From robert.kern at gmail.com Sun Apr 20 20:24:54 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 20 Apr 2008 19:24:54 -0500 Subject: [Numpy-discussion] OSX installer: please test In-Reply-To: <9457e7c80804200632i5de08547v7d3354d4fbb0c0ea@mail.gmail.com> References: <764e38540804161436j5ce7a468w8bea6a3c3598d69e@mail.gmail.com> <9457e7c80804190145j60249bbcv9e73a891cbdbdb6f@mail.gmail.com> <3d375d730804190154n2abb8b92le8a855733756c1fe@mail.gmail.com> <9457e7c80804190958l7ba37535p4626def636c0062e@mail.gmail.com> <3d375d730804200021n4e51790dr5f882e01515c19ab@mail.gmail.com> <9457e7c80804200632i5de08547v7d3354d4fbb0c0ea@mail.gmail.com> Message-ID: <3d375d730804201724k6cdf5315g967cd8ac8a8bdef2@mail.gmail.com> On Sun, Apr 20, 2008 at 8:32 AM, St?fan van der Walt wrote: > Hi Robert > > On 20/04/2008, Robert Kern wrote: > > On Sat, Apr 19, 2008 at 11:58 AM, St?fan van der Walt wrote: > > > On 19/04/2008, Robert Kern wrote: > > > > In any case, MacPorts' Python is not a framework build, so the binary > > > > wouldn't work anyways. > > > > > > Could you explain what you mean by that? Mine was compiled with the > > > +framework flag, so it should be a "framework" build? > > > > Hmm, I was looking at my python25/Portfile which does not have this > > variant and uses ./configure --disable-framework. The $Id$ crud at the > > top says it is from 2008-02-22. Perhaps you have an older or newer > > version of the MacPorts tree. I thought mine was up to date. > > This is the most up-to-date information I could find: > > http://trac.macosforge.org/projects/macports/ticket/11267 > > I have > > # $Id: Portfile 34729 2008-03-03 22:07:37Z raimue at macports.org $ Ah, good. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Sun Apr 20 20:34:23 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 20 Apr 2008 18:34:23 -0600 Subject: [Numpy-discussion] API scan Message-ID: Hi All, I would like to scan generated files for API functions, but it seems that the API scan runs on files in the numpy/core/src directory, not in the build directory where the generated files end up. Is there an easy way to fix this? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Sun Apr 20 21:08:40 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 20 Apr 2008 21:08:40 -0400 Subject: [Numpy-discussion] sparse matrices -> numpy ? In-Reply-To: <1208725195.23244.9.camel@localhost> References: <1208725195.23244.9.camel@localhost> Message-ID: On Sun, 20 Apr 2008, Soeren Sonnenburg apparently wrote: > is there any schedule when sparse matrices will arrive in > the numpy core? And related: was there any resolution on the revision of the matrix object. (A primary concern was maintaining compatibility of behavior between matrices and sparse matrices.) Thank you, Alan Isaac From charlesr.harris at gmail.com Sun Apr 20 22:04:19 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 20 Apr 2008 20:04:19 -0600 Subject: [Numpy-discussion] Buildbot is down Message-ID: The buildbot seems to be off line these days. Is that planned? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Mon Apr 21 01:51:13 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 21 Apr 2008 07:51:13 +0200 Subject: [Numpy-discussion] API scan In-Reply-To: References: Message-ID: <9457e7c80804202251g358b6bd9qcaa77127eb79907b@mail.gmail.com> Hi Charles On 21/04/2008, Charles R Harris wrote: > I would like to scan generated files for API functions, but it seems that > the API scan runs on files in the numpy/core/src directory, not in the build > directory where the generated files end up. Is there an easy way to fix > this? Could you expand a bit? Which tool are you using? Regards St?fan From stefan at sun.ac.za Mon Apr 21 02:07:07 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 21 Apr 2008 08:07:07 +0200 Subject: [Numpy-discussion] Buildbot is down In-Reply-To: References: Message-ID: <9457e7c80804202307p55a87eedyc4420a488cce5435@mail.gmail.com> On 21/04/2008, Charles R Harris wrote: > The buildbot seems to be off line these days. Is that planned? No, that's not supposed to be. I'll get on it. Thanks St?fan From david at ar.media.kyoto-u.ac.jp Mon Apr 21 05:38:43 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 21 Apr 2008 18:38:43 +0900 Subject: [Numpy-discussion] [numscons] 0.6.1 release: it build scipy, and on windows ! Message-ID: <480C60A3.9030201@ar.media.kyoto-u.ac.jp> Hi, I am pleased to announce the 0.6.1 release of numscons. You can get tarballs, eggs for python 2.4/2.5 and windows installers on launchpad: https://launchpad.net/numpy.scons.support/0.6/0.6.1 I did not announce any release of numscons for some time, but this one is a significant milestone: it can build both numpy and scipy on linux, mac os X and windows (mingw only for now), and all tests pass on those three platforms (that is, all tests which pass on distutils build). Here is a full changelog since the last announced release: - Update the local copy of scons to 0.98. All internal modifications to scons needed for numscons are now integrated upstream, which means it is theoretically possible to use an external scons - f2py scons tool is now executed in its own process to avoid race issues with parallel builds. Concretely, this means building scipy with the --jobs options to use multi core should work. - f2py has been almost entirely rewritten: it can now scan the module name automatically, and should be much more reliable. - Msvc runtime issues are now fixed: the PythonExtension builder does not link the msvc runtime if its sources contain some fortran files - Added initial support for C++ flags customization cheers, David From matthieu.brucher at gmail.com Mon Apr 21 06:52:09 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 21 Apr 2008 12:52:09 +0200 Subject: [Numpy-discussion] [numscons] 0.6.1 release: it build scipy, and on windows ! In-Reply-To: <480C60A3.9030201@ar.media.kyoto-u.ac.jp> References: <480C60A3.9030201@ar.media.kyoto-u.ac.jp> Message-ID: Congratulations for this :) Matthieu 2008/4/21, David Cournapeau : > > Hi, > > I am pleased to announce the 0.6.1 release of numscons. You can get > tarballs, eggs for python 2.4/2.5 and windows installers on launchpad: > > https://launchpad.net/numpy.scons.support/0.6/0.6.1 > > I did not announce any release of numscons for some time, but this one > is a significant milestone: it can build both numpy and scipy on linux, > mac os X and windows (mingw only for now), and all tests pass on those > three platforms (that is, all tests which pass on distutils build). > > Here is a full changelog since the last announced release: > - Update the local copy of scons to 0.98. All internal modifications > to scons needed for numscons are now integrated upstream, which means it > is theoretically possible to use an external scons > - f2py scons tool is now executed in its own process to avoid race > issues with parallel builds. Concretely, this means building scipy with > the --jobs options to use multi core should work. > - f2py has been almost entirely rewritten: it can now scan the > module name automatically, and should be much more reliable. > - Msvc runtime issues are now fixed: the PythonExtension builder > does not link the msvc runtime if its sources contain some fortran files > - Added initial support for C++ flags customization > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From pearu at cens.ioc.ee Mon Apr 21 07:00:58 2008 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Mon, 21 Apr 2008 13:00:58 +0200 Subject: [Numpy-discussion] f2py on windows cant find gfortran because split_quoted() removed from ccompiler.py In-Reply-To: References: Message-ID: <480C73EA.1000705@cens.ioc.ee> Jarrod Millman wrote: > Our version of split_quoted() was added several years ago and should > I think be submitted for inclusion upstream. I know that distutils > isn't exactly being maintained, but--Pearu--is it possible that we > could get this upstream? I looked into the distutils code again and remembered the following circumstances: The split_quoted hack that we have in numpy.distutils, was the "minimal fix" of the path name handling issue - it basically adds only two extra lines to the function body and works well. However, the split_quoted hack is not the "right fix". The "right fix" would require more substantial changes to distutils (note that distutils Windows support was developed on a unix platform and it looks like many issues were fixed for specific usage cases). Since the "minimal fix" did its job (consider it as a workaround to distutils misbehavior) and the "right fix" would require much more work as well as lots of effort to prove upstream about the importance of required changes (consider the current distutils development status and the fact that the numpy.distutils expands distutils in a direction that upstream might not be interested in because the current issue is irrelevant for std distutils tasks), I was satisfied with the "minimal fix" (as at the time there were more important issues to be resolved). I suggest to leave this issue for practical reasons: the workaround works and the correct fix is to painful to work out. Let practicality beat purity in this case. I hope distutils will retire some day:) Regards, Pearu From pearu at cens.ioc.ee Mon Apr 21 07:03:19 2008 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Mon, 21 Apr 2008 13:03:19 +0200 Subject: [Numpy-discussion] [numscons] 0.6.1 release: it build scipy, and on windows ! In-Reply-To: <480C60A3.9030201@ar.media.kyoto-u.ac.jp> References: <480C60A3.9030201@ar.media.kyoto-u.ac.jp> Message-ID: <480C7477.1040909@cens.ioc.ee> David Cournapeau wrote: > - f2py has been almost entirely rewritten: it can now scan the ^^^^^^^^^^^^^^^^^^^^^^^^ > module name automatically, and should be much more reliable. What do you mean by ^^^? ;) Pearu From david at ar.media.kyoto-u.ac.jp Mon Apr 21 06:59:38 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 21 Apr 2008 19:59:38 +0900 Subject: [Numpy-discussion] [numscons] 0.6.1 release: it build scipy, and on windows ! In-Reply-To: <480C7477.1040909@cens.ioc.ee> References: <480C60A3.9030201@ar.media.kyoto-u.ac.jp> <480C7477.1040909@cens.ioc.ee> Message-ID: <480C739A.8060105@ar.media.kyoto-u.ac.jp> Pearu Peterson wrote: > David Cournapeau wrote: > > >> - f2py has been almost entirely rewritten: it can now scan the >> > ^^^^^^^^^^^^^^^^^^^^^^^^ > >> module name automatically, and should be much more reliable. >> > > What do you mean by ^^^? ;) > I did not touch f2py, don't worry :) I meant the scons tool f2py: tools are basically the interface between general tools (compilers, linkers, interface generetors like swig, etc...) and scons. Because f2py itself is meant to be used in distutils, which is command based, it was not trivial to adapt it to a tool like scons which is target based (that is handle dependencies through a DAG, like make). cheers, David From tonyt at logyst.com Mon Apr 21 07:48:46 2008 From: tonyt at logyst.com (Tony Theodore) Date: Mon, 21 Apr 2008 11:48:46 +0000 (UTC) Subject: [Numpy-discussion] OSX installer: please test References: <764e38540804161436j5ce7a468w8bea6a3c3598d69e@mail.gmail.com> Message-ID: Christopher Burns berkeley.edu> writes: > > I've built a Universal Mac binary for numpy 1.1.0.? Installed correctly on OSX 10.5.2, Intel Core 2 Duo and all tests passed. Thanks, Tony From oliphant at enthought.com Mon Apr 21 10:02:16 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 21 Apr 2008 09:02:16 -0500 Subject: [Numpy-discussion] Speed of matrix multiplication In-Reply-To: References: Message-ID: <480C9E68.5090605@enthought.com> Keith Goodman wrote: > Why is a.T*b slower than M.dot(a.T, b)? Does it take longer to parse > or something? > The issue I think is that a is a Python class and so takes a bit longer to do all the things being done which includes: 1) running the Python __mul__ function 2) creating a new array (including running the __array_finalize__ Python function). -Travis From charlesr.harris at gmail.com Mon Apr 21 13:05:18 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 21 Apr 2008 11:05:18 -0600 Subject: [Numpy-discussion] API scan In-Reply-To: <9457e7c80804202251g358b6bd9qcaa77127eb79907b@mail.gmail.com> References: <9457e7c80804202251g358b6bd9qcaa77127eb79907b@mail.gmail.com> Message-ID: On Sun, Apr 20, 2008 at 11:51 PM, St?fan van der Walt wrote: > Hi Charles > > On 21/04/2008, Charles R Harris wrote: > > I would like to scan generated files for API functions, but it seems > that > > the API scan runs on files in the numpy/core/src directory, not in the > build > > directory where the generated files end up. Is there an easy way to fix > > this? > > Could you expand a bit? Which tool are you using? > Perhaps I should describe what I want to do. In ufuncobject.c there are a lot of generic loops that go into the ufunc API and I would like to generate those loops using a template, so what I want is something like: generate code from template, scan for UFUNC_API tags. At the moment these actions occur in that order, but ufuncobject is not on the list of template files and generated code is placed in the build directories and the code that is scanned for tags is in the original source directory. I think the whole process would be streamlined if *all* source files were treated as templates with the output going into the build directory, and then all files in the build directory were scanned for API tags. Alternatively, we could generate the API files from a separate list. The latter actually makes sense to me because the way it is now there is no central place to track the API except by looking at the result of a build. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From b3i4old02 at sneakemail.com Mon Apr 21 17:28:14 2008 From: b3i4old02 at sneakemail.com (Michael Hoffman) Date: Mon, 21 Apr 2008 22:28:14 +0100 Subject: [Numpy-discussion] Summing indices of heterogeneous shapes Message-ID: In the following example I can sum up lists of column indices: >>> x = numpy.arange(30) >>> x.shape = (6, 5) >>> x array([[ 0, 1, 2, 3, 4], [ 5, 6, 7, 8, 9], [10, 11, 12, 13, 14], [15, 16, 17, 18, 19], [20, 21, 22, 23, 24], [25, 26, 27, 28, 29]]) >>> cols = [[0, 1], [2, 3]] >>> x[:, cols].sum(2) array([[ 1, 5], [11, 15], [21, 25], [31, 35], [41, 45], [51, 55]]) Is there a way to do this if the shapes of the lists are heterogeneous? For example, what if I wanted to set cols = [[0, 1], [2, 3], [1, 2, 3]]? What's the best way to get a sensible result? Thanks! -- Michael Hoffman From broman at spawar.navy.mil Mon Apr 21 19:31:34 2008 From: broman at spawar.navy.mil (Vincent Broman) Date: Mon, 21 Apr 2008 16:31:34 -0700 Subject: [Numpy-discussion] powerpc yellow dog linux port of numpy In-Reply-To: References: Message-ID: <200804211631.34130@b00d61a8cecf8b2266f81358fd170621.navy.mil> I succeeded in working around the other Yellow Dog Linux porting problem connected with the floating point exception calls. It turns out that a problematic #include was protected by a "#ifdef __OPTIMIZE__" so my preprocessing with "gcc -E" never saw its effect. So, by avoiding optimization of umath, I was able to get a slower, but working copy of numpy compiled, which passed the numpy.test() suite. The one-line patch avoiding my two compile problems adds to numpy/core/setup.py after line 270 the following line. extra_compile_args = ["-fno-builtin -O0"] I don't know enough to say whether the quoted args should be split(). I do want to find real fixes, though, not just bandaids. In numpy/core/include/numpy/ndarrayobject.h is code which causes me one problem. #if NPY_SIZEOF_LONGDOUBLE == NPY_SIZEOF_DOUBLE typedef double npy_longdouble; #define NPY_LONGDOUBLE_FMT "g" #else typedef long double npy_longdouble; #define NPY_LONGDOUBLE_FMT "Lg" #endif I do not see why the size is critical. Couldn't we just use long double for any compiler that supports long double? If the double and long double have the same format, why do we prefer double? My other assembler code problem is caused, when optimization is enabled, by code using the following definition in /usr/include/bits/fenvinline.h, which I do not understand, never having done ppc assembler. ------------------quote------------------------------------------------- /* The weird 'i#*X' constraints on the following suppress a gcc warning when __excepts is not a constant. Otherwise, they mean the same as just plain 'i'. */ /* Inline definition for feraiseexcept. */ # define feraiseexcept(__excepts) \ ((__builtin_constant_p (__excepts) \ && ((__excepts) & ((__excepts)-1) == 0 \ && (__excepts) != FE_INVALID) \ ? ((__excepts) != 0 \ ? (__extension__ ({ __asm__ __volatile__ \ ("mtfsb1 %s0" \ : : "i#*X"(__builtin_ffs (__excepts))); \ 0; })) \ : 0) \ : (feraiseexcept) (__excepts)) ----------------endquote--------------------------------------------------- This definition causes all the occurences in umathmodule.c of feraiseexcept( FE_DIVBYZERO) to make gcc complain of "inconsistent operand constraints in an ?`asm'". Is there anyone who can parse that hornet's nest of a definition? Vincent Broman From travis at enthought.com Mon Apr 21 19:59:24 2008 From: travis at enthought.com (Travis Vaught) Date: Mon, 21 Apr 2008 18:59:24 -0500 Subject: [Numpy-discussion] ANN: EPD - Enthought Python Distribution released Message-ID: Greetings, Enthought is pleased to announce the release of the Enthought Python Distribution (EPD) version 2.5.2001. http://www.enthought.com/epd This release makes available both the RedHat 3.x (amd64) and Windows XP (x86) installers. OS X, Ubuntu and more (modern) RHEL versions are coming soon(!). About EPD --------- The Enthought Python Distribution is a "kitchen-sink-included" distribution of the Python Programming Language as well as over 60 additional tools and libraries. It includes NumPy, SciPy, IPython, 2D and 3D visualization, database adapters and a lot of other tools right out of the box. Enthought is offering access to this bundle as a free service to academic and other non-profit organizations. We also offer an annual fee-based subscription service for Commercial and Governmental users to download and update the software bundle. (Everyone may try it out for free. Please see the License Information below.) Included Software ----------------- A short list includes: Python 2.5.2, NumPy, SciPy, Traits, Mayavi, Chaco, Kiva, Enable, Matplotlib, wxPython and VTK. The complete list of software with version numbers is available here: http://www.enthought.com/products/epdlibraries.php License Information ------------------- EPD is a bundle of software, every piece of which is available separately for free under various open-source licenses. Not-for- profit, private-sector access to the bundle and its updates is, and will remain, free under the terms of the Subscription Agreement (see http://www.enthought.com/products/epdlicense.php ). Commercial and Governmental users may try the bundle for free for 30 days. After the trial period, users may purchase a one-year subscription to download and update the bundle. Downloaded software obtained under the subscription agreement may be used by the subscriber in perpetuity. This model should sound familiar, as our commercial offering is quite similar to the business model of a certain linux distributor. More information is also available in the FAQ ( http://www.enthought.com/products/epdfaq.php ). For larger deployments, or those with special build or distribution needs, an Enterprise Subscription is also available. Thanks ------ EPD is compelling because it solves a thorny packaging and distribution problem, but also because of the libraries which it includes. The folks here at Enthought would like to thank the Python developer community and the wider community that authors and contributes to these included libraries. We put these things to work every day and would be much less productive without them. So, thanks! From charlesr.harris at gmail.com Mon Apr 21 22:21:39 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 21 Apr 2008 20:21:39 -0600 Subject: [Numpy-discussion] Code generator bug and fix? Message-ID: I've gotten my own python class with a logical_not method to work correctly if I goto numpy/core/code_generators/generate_umath.py and change 'logical_not' : Ufunc(1, 1, None, 'returns not x elementwise.', TD(noobj, out='?'), TD(M, f='logical_not', out='?'), ), to 'logical_not' : Ufunc(1, 1, None, 'returns not x elementwise.', TD(noobj, out='?'), TD(M, f='logical_not'), ), There are a whole bunch of functions with this same error and before I start changing them, I would like someone more familiar with this code to tell me if I'm doing the right thing. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Mon Apr 21 22:28:29 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 21 Apr 2008 21:28:29 -0500 Subject: [Numpy-discussion] Code generator bug and fix? In-Reply-To: References: Message-ID: <480D4D4D.5020708@enthought.com> Charles R Harris wrote: > I've gotten my own python class with a logical_not method to work > correctly if I goto numpy/core/code_generators/generate_umath.py and > change I need more context for this. Why does the umath generator matter for your python class? > > 'logical_not' : > Ufunc(1, 1, None, > 'returns not x elementwise.', > TD(noobj, out='?'), > TD(M, f='logical_not', out='?'), > ), > > to > > 'logical_not' : > Ufunc(1, 1, None, > 'returns not x elementwise.', > TD(noobj, out='?'), > TD(M, f='logical_not'), > ), > Why is this an error? Is the difference only removing the out variable? It's been a while since I reviewed this code, so what does removing the out variable do functionally (What is the difference in the ufunc that is generated)? -Travis From charlesr.harris at gmail.com Mon Apr 21 22:53:20 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 21 Apr 2008 20:53:20 -0600 Subject: [Numpy-discussion] Code generator bug and fix? In-Reply-To: <480D4D4D.5020708@enthought.com> References: <480D4D4D.5020708@enthought.com> Message-ID: On Mon, Apr 21, 2008 at 8:28 PM, Travis E. Oliphant wrote: > Charles R Harris wrote: > > I've gotten my own python class with a logical_not method to work > > correctly if I goto numpy/core/code_generators/generate_umath.py and > > change > I need more context for this. Why does the umath generator matter for > your python class? > > > > 'logical_not' : > > Ufunc(1, 1, None, > > 'returns not x elementwise.', > > TD(noobj, out='?'), > > TD(M, f='logical_not', out='?'), > > ), > > > > to > > > > 'logical_not' : > > Ufunc(1, 1, None, > > 'returns not x elementwise.', > > TD(noobj, out='?'), > > TD(M, f='logical_not'), > > ), > > > Why is this an error? Is the difference only removing the out > variable? It's been a while since I reviewed this code, so what does > removing the out variable do functionally (What is the difference in the > ufunc that is generated)? > The way it is, it passes a boolean array to the PyUFunc_O_O_method loop where the loop is expecting an object array. I checked the step size to see this, and that's also how the generated signature reads. Consequently, when the reference count is decremented bad things happen. I suspect this hasn't been seen before because it hadn't been tried. I wrote loop tests before cleaning up the loop code and the bug turned up then. My guess is that here M means object called through non-Python method (logical_not), and omitting out means the output type is the same as the input. I suspect that '?' should do the same thing and that there might be a bug in the function dispatcher or the signature generator, but I'm not clear on that yet. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Apr 21 23:06:16 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 21 Apr 2008 21:06:16 -0600 Subject: [Numpy-discussion] Code generator bug and fix? In-Reply-To: References: <480D4D4D.5020708@enthought.com> Message-ID: On Mon, Apr 21, 2008 at 8:53 PM, Charles R Harris wrote: > > > On Mon, Apr 21, 2008 at 8:28 PM, Travis E. Oliphant < > oliphant at enthought.com> wrote: > > > Charles R Harris wrote: > > > I've gotten my own python class with a logical_not method to work > > > correctly if I goto numpy/core/code_generators/generate_umath.py and > > > change > > I need more context for this. Why does the umath generator matter for > > your python class? > > > > > > 'logical_not' : > > > Ufunc(1, 1, None, > > > 'returns not x elementwise.', > > > TD(noobj, out='?'), > > > TD(M, f='logical_not', out='?'), > > > ), > > > > > > to > > > > > > 'logical_not' : > > > Ufunc(1, 1, None, > > > 'returns not x elementwise.', > > > TD(noobj, out='?'), > > > TD(M, f='logical_not'), > > > ), > > > > > Why is this an error? Is the difference only removing the out > > variable? It's been a while since I reviewed this code, so what does > > removing the out variable do functionally (What is the difference in the > > ufunc that is generated)? > > > > The way it is, it passes a boolean array to the PyUFunc_O_O_method loop > where the loop is expecting an object array. I checked the step size to see > this, and that's also how the generated signature reads. Consequently, when > the reference count is decremented bad things happen. I suspect this hasn't > been seen before because it hadn't been tried. I wrote loop tests before > cleaning up the loop code and the bug turned up then. > > My guess is that here M means object called through non-Python method > (logical_not), and omitting out means the output type is the same as the > input. I suspect that '?' should do the same thing and that there might be a > bug in the function dispatcher or the signature generator, but I'm not clear > on that yet. > If out is omitted, the output types matches the input types. Here's where the output types are generated: def finish_signature(self, nin, nout): if self.in_ is None: self.in_ = self.type * nin assert len(self.in_) == nin if self.out is None: self.out = self.type * nout assert len(self.out) == nout So that seems like the right thing to do. I still don't know what '?' means, though. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Mon Apr 21 23:09:39 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 21 Apr 2008 22:09:39 -0500 Subject: [Numpy-discussion] Code generator bug and fix? In-Reply-To: References: <480D4D4D.5020708@enthought.com> Message-ID: <480D56F3.5060509@enthought.com> Charles R Harris wrote: > > > On Mon, Apr 21, 2008 at 8:53 PM, Charles R Harris > > wrote: > > > > On Mon, Apr 21, 2008 at 8:28 PM, Travis E. Oliphant > > wrote: > > Charles R Harris wrote: > > I've gotten my own python class with a logical_not method to > work > > correctly if I goto > numpy/core/code_generators/generate_umath.py and > > change > I need more context for this. Why does the umath generator > matter for > your python class? > > > > 'logical_not' : > > Ufunc(1, 1, None, > > 'returns not x elementwise.', > > TD(noobj, out='?'), > > TD(M, f='logical_not', out='?'), > > ), > > > > to > > > > 'logical_not' : > > Ufunc(1, 1, None, > > 'returns not x elementwise.', > > TD(noobj, out='?'), > > TD(M, f='logical_not'), > > ), > > > Why is this an error? Is the difference only removing the out > variable? It's been a while since I reviewed this code, so > what does > removing the out variable do functionally (What is the > difference in the > ufunc that is generated)? > > > The way it is, it passes a boolean array to the PyUFunc_O_O_method > loop where the loop is expecting an object array. I checked the > step size to see this, and that's also how the generated signature > reads. Consequently, when the reference count is decremented bad > things happen. I suspect this hasn't been seen before because it > hadn't been tried. I wrote loop tests before cleaning up the loop > code and the bug turned up then. > > My guess is that here M means object called through non-Python > method (logical_not), and omitting out means the output type is > the same as the input. I suspect that '?' should do the same thing > and that there might be a bug in the function dispatcher or the > signature generator, but I'm not clear on that yet. > > > If out is omitted, the output types matches the input types. Here's > where the output types are generated: > > def finish_signature(self, nin, nout): > if self.in_ is None: > self.in_ = self.type * nin > assert len(self.in_) == nin > if self.out is None: > self.out = self.type * nout > assert len(self.out) == nout > > So that seems like the right thing to do. I still don't know what '?' > means, though. Thanks Chuck, I get it now (I should have spent a minute looking at the code). The '?' is the character code for "boolean" So, I think you are right about the bug. -Travis From charlesr.harris at gmail.com Mon Apr 21 23:20:10 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 21 Apr 2008 21:20:10 -0600 Subject: [Numpy-discussion] Code generator bug and fix? In-Reply-To: <480D56F3.5060509@enthought.com> References: <480D4D4D.5020708@enthought.com> <480D56F3.5060509@enthought.com> Message-ID: On Mon, Apr 21, 2008 at 9:09 PM, Travis E. Oliphant wrote: > Charles R Harris wrote: > > > > > > On Mon, Apr 21, 2008 at 8:53 PM, Charles R Harris > > > wrote: > > > > > > > > On Mon, Apr 21, 2008 at 8:28 PM, Travis E. Oliphant > > > wrote: > > > > Charles R Harris wrote: > > > I've gotten my own python class with a logical_not method to > > work > > > correctly if I goto > > numpy/core/code_generators/generate_umath.py and > > > change > > I need more context for this. Why does the umath generator > > matter for > > your python class? > > > > > > 'logical_not' : > > > Ufunc(1, 1, None, > > > 'returns not x elementwise.', > > > TD(noobj, out='?'), > > > TD(M, f='logical_not', out='?'), > > > ), > > > > > > to > > > > > > 'logical_not' : > > > Ufunc(1, 1, None, > > > 'returns not x elementwise.', > > > TD(noobj, out='?'), > > > TD(M, f='logical_not'), > > > ), > > > > > Why is this an error? Is the difference only removing the out > > variable? It's been a while since I reviewed this code, so > > what does > > removing the out variable do functionally (What is the > > difference in the > > ufunc that is generated)? > > > > > > The way it is, it passes a boolean array to the PyUFunc_O_O_method > > loop where the loop is expecting an object array. I checked the > > step size to see this, and that's also how the generated signature > > reads. Consequently, when the reference count is decremented bad > > things happen. I suspect this hasn't been seen before because it > > hadn't been tried. I wrote loop tests before cleaning up the loop > > code and the bug turned up then. > > > > My guess is that here M means object called through non-Python > > method (logical_not), and omitting out means the output type is > > the same as the input. I suspect that '?' should do the same thing > > and that there might be a bug in the function dispatcher or the > > signature generator, but I'm not clear on that yet. > > > > > > If out is omitted, the output types matches the input types. Here's > > where the output types are generated: > > > > def finish_signature(self, nin, nout): > > if self.in_ is None: > > self.in_ = self.type * nin > > assert len(self.in_) == nin > > if self.out is None: > > self.out = self.type * nout > > assert len(self.out) == nout > > > > So that seems like the right thing to do. I still don't know what '?' > > means, though. > > Thanks Chuck, > > I get it now (I should have spent a minute looking at the code). The > '?' is the character code for "boolean" > > So, I think you are right about the bug. > OK, So I quess that everywhere I see TD(M... I should remove the "out='?' ". Will do. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Apr 22 00:01:38 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 21 Apr 2008 22:01:38 -0600 Subject: [Numpy-discussion] Code generator bug and fix? In-Reply-To: <480D56F3.5060509@enthought.com> References: <480D4D4D.5020708@enthought.com> <480D56F3.5060509@enthought.com> Message-ID: On Mon, Apr 21, 2008 at 9:09 PM, Travis E. Oliphant wrote: > Charles R Harris wrote: > > > > > > On Mon, Apr 21, 2008 at 8:53 PM, Charles R Harris > > > wrote: > > > > > > > > On Mon, Apr 21, 2008 at 8:28 PM, Travis E. Oliphant > > > wrote: > > > > Charles R Harris wrote: > > > I've gotten my own python class with a logical_not method to > > work > > > correctly if I goto > > numpy/core/code_generators/generate_umath.py and > > > change > > I need more context for this. Why does the umath generator > > matter for > > your python class? > > > > > > 'logical_not' : > > > Ufunc(1, 1, None, > > > 'returns not x elementwise.', > > > TD(noobj, out='?'), > > > TD(M, f='logical_not', out='?'), > > > ), > > > > > > to > > > > > > 'logical_not' : > > > Ufunc(1, 1, None, > > > 'returns not x elementwise.', > > > TD(noobj, out='?'), > > > TD(M, f='logical_not'), > > > ), > > > > > Why is this an error? Is the difference only removing the out > > variable? It's been a while since I reviewed this code, so > > what does > > removing the out variable do functionally (What is the > > difference in the > > ufunc that is generated)? > > > > > > The way it is, it passes a boolean array to the PyUFunc_O_O_method > > loop where the loop is expecting an object array. I checked the > > step size to see this, and that's also how the generated signature > > reads. Consequently, when the reference count is decremented bad > > things happen. I suspect this hasn't been seen before because it > > hadn't been tried. I wrote loop tests before cleaning up the loop > > code and the bug turned up then. > > > > My guess is that here M means object called through non-Python > > method (logical_not), and omitting out means the output type is > > the same as the input. I suspect that '?' should do the same thing > > and that there might be a bug in the function dispatcher or the > > signature generator, but I'm not clear on that yet. > > > > > > If out is omitted, the output types matches the input types. Here's > > where the output types are generated: > > > > def finish_signature(self, nin, nout): > > if self.in_ is None: > > self.in_ = self.type * nin > > assert len(self.in_) == nin > > if self.out is None: > > self.out = self.type * nout > > assert len(self.out) == nout > > > > So that seems like the right thing to do. I still don't know what '?' > > means, though. > > Thanks Chuck, > > I get it now (I should have spent a minute looking at the code). The > '?' is the character code for "boolean" > > So, I think you are right about the bug. > I note that other logical operators, <, ==, do in fact return booleans when operating on objects. So another fix is to write special case loops for logical_{not, or, and, xor} that do the same. Perhaps a ticket for an enhancement should be opened. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Tue Apr 22 00:04:12 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 21 Apr 2008 23:04:12 -0500 Subject: [Numpy-discussion] Boolean output for Object arrays with M Message-ID: <480D63BC.5090208@enthought.com> Hi Chuck, O.K. I've finally caught up with you, I think (at least as far as the umath generators go). All TD(M,.....) lines should *not* have out='?' The reasoning is that this specifies a ufunc type with object inputs and boolean array outputs where the function is a method of the object input. However, the output of a Python method is not going to be a C-boolean. It will be an Object. Thus, I think that you are right and removing the out='?' from all TD(M,...) lines is the right thing to do. -Travis From b3i4old02 at sneakemail.com Tue Apr 22 04:31:22 2008 From: b3i4old02 at sneakemail.com (Michael Hoffman) Date: Tue, 22 Apr 2008 09:31:22 +0100 Subject: [Numpy-discussion] Summing indices of heterogeneous shapes In-Reply-To: References: Message-ID: Michael Hoffman wrote: > In the following example I can sum up lists of column indices: > > >>> x = numpy.arange(30) > >>> x.shape = (6, 5) > >>> x > array([[ 0, 1, 2, 3, 4], > [ 5, 6, 7, 8, 9], > [10, 11, 12, 13, 14], > [15, 16, 17, 18, 19], > [20, 21, 22, 23, 24], > [25, 26, 27, 28, 29]]) > >>> cols = [[0, 1], [2, 3]] > >>> x[:, cols].sum(2) > array([[ 1, 5], > [11, 15], > [21, 25], > [31, 35], > [41, 45], > [51, 55]]) > > Is there a way to do this if the shapes of the lists are heterogeneous? > For example, what if I wanted to set cols = [[0, 1], [2, 3], [1, 2, 3]]? > What's the best way to get a sensible result? One way I thought of is to add an extra column to x that is full of zeros, and fill in any smaller lists in cols with that. But this will be pretty clunky and slow. From david at ar.media.kyoto-u.ac.jp Tue Apr 22 05:13:19 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 22 Apr 2008 18:13:19 +0900 Subject: [Numpy-discussion] [numscons] 0.6.3 release: building scipy with MS compilers works Message-ID: <480DAC2F.9010602@ar.media.kyoto-u.ac.jp> Hi, Sorry for announcing one more numscons release in such a short time: this releases can finally handle the last common platform, Visual Studio on win32. Win32 installers and source tarballs can be found on launchpad, as usual: https://code.launchpad.net/numpy.scons.support/0.6/0.6.3 Python eggs on pypi should follow soon (I cannot upload to pypi from my lab, unfortunately). Except the various fixes necessary to make the build work with MS compilers, there were a few improvements: - some more improvements for f2py scons tool, which simplified a bit some scipy scons scripts - the addition of a silent mode, to get more terse command lines output (not perfect yet), ala kbuild for people familiar with compiling linux (e.g. you get CC foo.c instead of gcc -W -Wall blas blas blas). The goal is to make warning more obvious (and to fix them at some point, of course :) ). Just use python setupscons.py scons --silent=N with N between 1 and 3 to see the result. As this was the last major platform I wanted to support, I can now move on polishing the API as well as starting a real documentation for package developers. cheers, David From lbolla at gmail.com Tue Apr 22 05:45:05 2008 From: lbolla at gmail.com (lorenzo bolla) Date: Tue, 22 Apr 2008 11:45:05 +0200 Subject: [Numpy-discussion] Numpy 1.1.0 and matplotlib 0.90.1 Message-ID: <80c99e790804220245u3e89b250h65e6bff4b359bd31@mail.gmail.com> Hello, the latest svn numpy version 1.1.0.dev5061 does not work with matplotlib 0.90.1 (version shipped with enthought distribution), unless a change in Python25/Lib/site-packages/matplotlib-0.90.1.0003-py2.5-win32.egg/matplotlib/numerix/ma/__init__.py is done: $ diff __init__.py.orig __init__.py 12c12 < from numpy.core.ma import * --- > from numpy.ma import * Maybe this should be forwarded to the pylab mailing list, but I'm not subscribed there... L. -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Tue Apr 22 10:15:02 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 22 Apr 2008 09:15:02 -0500 Subject: [Numpy-discussion] Code generator bug and fix? In-Reply-To: References: <480D4D4D.5020708@enthought.com> <480D56F3.5060509@enthought.com> Message-ID: <480DF2E6.8020308@enthought.com> Charles R Harris wrote: > > I note that other logical operators, <, ==, do in fact return booleans > when operating on objects. So another fix is to write special case > loops for logical_{not, or, and, xor} that do the same. Perhaps a > ticket for an enhancement should be opened. There should already be special-case loops for those cases, right? Only the general purpose loops O_O and OO_O which were being used for the "method-call" on object arrays. Are you saying that we should actually return "boolean" arrays instead of object arrays for the logical operations on Object arrays? That is a relevant suggestion but is a change in code. I'm inclined not to do it, because it limits the ways that logical_and can be used. Perhaps a single boolean is not what is desired on output, but a whole "object" of them. -Travis From mdroe at stsci.edu Tue Apr 22 10:25:05 2008 From: mdroe at stsci.edu (Michael Droettboom) Date: Tue, 22 Apr 2008 10:25:05 -0400 Subject: [Numpy-discussion] Numpy 1.1.0 and matplotlib 0.90.1 In-Reply-To: <80c99e790804220245u3e89b250h65e6bff4b359bd31@mail.gmail.com> References: <80c99e790804220245u3e89b250h65e6bff4b359bd31@mail.gmail.com> Message-ID: <480DF541.2030402@stsci.edu> I will forward it to the matplotlib-devel mailing list on your behalf. Cheers, Mike lorenzo bolla wrote: > Hello, > > the latest svn numpy version 1.1.0.dev5061 does not work with > matplotlib 0.90.1 (version shipped with enthought distribution), > unless a change in > Python25/Lib/site-packages/matplotlib-0.90.1.0003-py2.5-win32.egg/matplotlib/numerix/ma/__init__.py > is done: > > $ diff __init__.py.orig __init__.py > 12c12 > < from numpy.core.ma import * > --- > > from numpy.ma import * > > Maybe this should be forwarded to the pylab mailing list, but I'm not > subscribed there... > > L. > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA From charlesr.harris at gmail.com Tue Apr 22 10:32:13 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 22 Apr 2008 08:32:13 -0600 Subject: [Numpy-discussion] Code generator bug and fix? In-Reply-To: <480DF2E6.8020308@enthought.com> References: <480D4D4D.5020708@enthought.com> <480D56F3.5060509@enthought.com> <480DF2E6.8020308@enthought.com> Message-ID: On Tue, Apr 22, 2008 at 8:15 AM, Travis E. Oliphant wrote: > Charles R Harris wrote: > > > > I note that other logical operators, <, ==, do in fact return booleans > > when operating on objects. So another fix is to write special case > > loops for logical_{not, or, and, xor} that do the same. Perhaps a > > ticket for an enhancement should be opened. > There should already be special-case loops for those cases, right? > Only the general purpose loops O_O and OO_O which were being used for > the "method-call" on object arrays. > > Are you saying that we should actually return "boolean" arrays instead > of object arrays for the logical operations on Object arrays? That is > a relevant suggestion but is a change in code. I'm inclined not to do > it, because it limits the ways that logical_and can be used. Perhaps a > single boolean is not what is desired on output, but a whole "object" of > them. > I'm saying we already return boolean arrays for some object arrays. For instance In [1]: x = ones(5, dtype=object) In [2]: x Out[2]: array([1, 1, 1, 1, 1], dtype=object) In [3]: x == x Out[3]: array([ True, True, True, True, True], dtype=bool) Because there are special case loops for the comparison operators. We don't do the same for the logical operators. I'm agnostic about this, although objects in objects out is easy to remember. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From lbolla at gmail.com Tue Apr 22 11:07:36 2008 From: lbolla at gmail.com (lorenzo bolla) Date: Tue, 22 Apr 2008 17:07:36 +0200 Subject: [Numpy-discussion] different behaviour in asfarray(None) Message-ID: <80c99e790804220807x5d9fa3as891f5f53ba6c937f@mail.gmail.com> I noticed a change in the behaviour of numpy.asfarray between numpy version 1.0.5 and 1.1.0: 1.0.5 ==== In [3]: numpy.asfarray(None) Out[3]: array(nan) In [4]: numpy.__version__ Out[4]: '1.0.5.dev4455' 1.1.0 ==== In [16]: numpy.asfarray(None) --------------------------------------------------------------------------- Traceback (most recent call last) c:\Documents and Settings\bollalo001\My Documents\archive\python\openopt\ in () c:\Python25\Lib\site-packages\numpy\lib\type_check.py in asfarray(a, dtype) 47 if not issubclass(dtype, _nx.inexact): 48 dtype = _nx.float_ ---> 49 return asarray(a,dtype=dtype) 50 51 def real(val): c:\Python25\Lib\site-packages\numpy\core\numeric.py in asarray(a, dtype, order) 133 are converted to base class ndarray. 134 """ --> 135 return array(a, dtype, copy=False, order=order) 136 137 def asanyarray(a, dtype=None, order=None): : float() argument must be a string or a number In [17]: numpy.__version__ Out[17]: '1.1.0.dev5061' Is this intended? why? L. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Tue Apr 22 11:29:21 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 22 Apr 2008 17:29:21 +0200 Subject: [Numpy-discussion] different behaviour in asfarray(None) In-Reply-To: <80c99e790804220807x5d9fa3as891f5f53ba6c937f@mail.gmail.com> References: <80c99e790804220807x5d9fa3as891f5f53ba6c937f@mail.gmail.com> Message-ID: <9457e7c80804220829u4f824068i11eddd832a85b6d1@mail.gmail.com> 2008/4/22 lorenzo bolla : > I noticed a change in the behaviour of numpy.asfarray between numpy version > 1.0.5 and 1.1.0: > > 1.0.5 > ==== > > In [3]: numpy.asfarray(None) > Out[3]: array(nan) > In [4]: numpy.__version__ > Out[4]: '1.0.5.dev4455' > > 1.1.0 > ==== > > In [16]: numpy.asfarray(None) > --------------------------------------------------------------------------- > : float() argument must be a string or a number > > Is this intended? why? Yes, 'asfarray' is equivalent to array(input).astype(dtype) I think it would be wrong to assume that None means NaN. Cheers St?fan From millman at berkeley.edu Tue Apr 22 14:13:53 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 22 Apr 2008 13:13:53 -0500 Subject: [Numpy-discussion] Numpy 1.1.0 and matplotlib 0.90.1 In-Reply-To: <80c99e790804220245u3e89b250h65e6bff4b359bd31@mail.gmail.com> References: <80c99e790804220245u3e89b250h65e6bff4b359bd31@mail.gmail.com> Message-ID: On Tue, Apr 22, 2008 at 4:45 AM, lorenzo bolla wrote: > the latest svn numpy version 1.1.0.dev5061 does not work with matplotlib > 0.90.1 (version shipped with enthought distribution), ... I believe this is fixed with 0.91.2. Please see their release notes: http://matplotlib.sourceforge.net/whats_new.html Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From efiring at hawaii.edu Tue Apr 22 14:29:09 2008 From: efiring at hawaii.edu (Eric Firing) Date: Tue, 22 Apr 2008 08:29:09 -1000 Subject: [Numpy-discussion] [matplotlib-devel] Numpy 1.1.0 and matplotlib 0.90.1 In-Reply-To: <480DF541.2030402@stsci.edu> References: <80c99e790804220245u3e89b250h65e6bff4b359bd31@mail.gmail.com> <480DF541.2030402@stsci.edu> Message-ID: <480E2E75.3010103@hawaii.edu> This is inherent; mpl 0.90.1 is permanently incompatible with numpy 1.1, short of each user making the change suggested below. The earliest mpl that should work with numpy 1.1 is 0.91.2. The change in masked array module is a major reason why numpy is getting a version bump to 1.1.0 instead of 1.0.5. Eric Michael Droettboom wrote: > I will forward it to the matplotlib-devel mailing list on your behalf. > > Cheers, > Mike > > lorenzo bolla wrote: >> Hello, >> >> the latest svn numpy version 1.1.0.dev5061 does not work with >> matplotlib 0.90.1 (version shipped with enthought distribution), >> unless a change in >> Python25/Lib/site-packages/matplotlib-0.90.1.0003-py2.5-win32.egg/matplotlib/numerix/ma/__init__.py >> is done: >> >> $ diff __init__.py.orig __init__.py >> 12c12 >> < from numpy.core.ma import * >> --- >>> from numpy.ma import * >> Maybe this should be forwarded to the pylab mailing list, but I'm not >> subscribed there... >> >> L. >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://projects.scipy.org/mailman/listinfo/numpy-discussion >> > From xavier.gnata at gmail.com Tue Apr 22 14:42:21 2008 From: xavier.gnata at gmail.com (Gnata Xavier) Date: Tue, 22 Apr 2008 20:42:21 +0200 Subject: [Numpy-discussion] Fromfunction generalization Message-ID: <480E318D.5030308@gmail.com> Hi, fromfunction is fine but I have like to be able to create 2Darrays using a function of i,j but also of one (or more) parameters. Something like that : def f(i,j,a): return (i+j)*a #replace that by another non trivial computation M=fromfunction(f( hum well something like i,j,a),(1000,1000)) this only way I know is to use global : a=5 def f(i,j): global a return (i+j)*a M=fromfunction(f,(1000,1000)) a=7 P=fromfunction(f,(1000,1000)) but it is quite ugly :( Is there a clean and *fast* way to do that ? Xavier From lbolla at gmail.com Tue Apr 22 14:50:53 2008 From: lbolla at gmail.com (lorenzo bolla) Date: Tue, 22 Apr 2008 20:50:53 +0200 Subject: [Numpy-discussion] Fromfunction generalization In-Reply-To: <480E318D.5030308@gmail.com> References: <480E318D.5030308@gmail.com> Message-ID: <80c99e790804221150m3552decfxf7d7a89848fb9134@mail.gmail.com> what about using lambda functions? M=fromfunction(lambda i,j:f(i,j,5),(1000,1000)) P=fromfunction(lambda i,j:f(i,j,7),(1000,1000)) L. On Tue, Apr 22, 2008 at 8:42 PM, Gnata Xavier wrote: > Hi, > > fromfunction is fine but I have like to be able to create 2Darrays using > a function of i,j but also of one (or more) parameters. > Something like that : > > def f(i,j,a): > return (i+j)*a #replace that by another non trivial computation > > M=fromfunction(f( hum well something like i,j,a),(1000,1000)) > > this only way I know is to use global : > > a=5 > > def f(i,j): > global a > return (i+j)*a > > M=fromfunction(f,(1000,1000)) > a=7 > P=fromfunction(f,(1000,1000)) > > but it is quite ugly :( > > Is there a clean and *fast* way to do that ? > > Xavier > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- Lorenzo Bolla lbolla at gmail.com http://lorenzobolla.emurse.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Apr 22 15:08:49 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 22 Apr 2008 13:08:49 -0600 Subject: [Numpy-discussion] Adding nested loops to template processor. Message-ID: Hi All, I want to use nested loops in our c template processor. The least intrusive way I see to do this is to change the repeat header slightly, i.e. /**begin repeat #var1= a, b, c# #var2= d, e, f# #nest=1# #var3= w, x# #var4= y, z# */ Where "nest" becomes a keyword. The variables then take values in the order var1=a, var2=d, var3=w, var4=y var1=a, var2=d, var3=x, var4=z .... Does anyone object to this change or have a better suggestion? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From xavier.gnata at gmail.com Tue Apr 22 15:10:07 2008 From: xavier.gnata at gmail.com (Gnata Xavier) Date: Tue, 22 Apr 2008 21:10:07 +0200 Subject: [Numpy-discussion] Fromfunction generalization In-Reply-To: <80c99e790804221150m3552decfxf7d7a89848fb9134@mail.gmail.com> References: <480E318D.5030308@gmail.com> <80c99e790804221150m3552decfxf7d7a89848fb9134@mail.gmail.com> Message-ID: <480E380F.5060303@gmail.com> Well done in python :) Is there a "numpy way" to do that (only asking for the fun because this solution is nice)? Xavier, who do like lamba functions :) > what about using lambda functions? > > M=fromfunction(lambda i,j:f(i,j,5),(1000,1000)) > P=fromfunction(lambda i,j:f(i,j,7),(1000,1000)) > > L. > > On Tue, Apr 22, 2008 at 8:42 PM, Gnata Xavier > wrote: > > Hi, > > fromfunction is fine but I have like to be able to create 2Darrays > using > a function of i,j but also of one (or more) parameters. > Something like that : > > def f(i,j,a): > return (i+j)*a #replace that by another non trivial computation > > M=fromfunction(f( hum well something like i,j,a),(1000,1000)) > > this only way I know is to use global : > > a=5 > > def f(i,j): > global a > return (i+j)*a > > M=fromfunction(f,(1000,1000)) > a=7 > P=fromfunction(f,(1000,1000)) > > but it is quite ugly :( > > Is there a clean and *fast* way to do that ? > > Xavier > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > -- > Lorenzo Bolla > lbolla at gmail.com > http://lorenzobolla.emurse.com/ > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From aisaac at american.edu Tue Apr 22 15:31:33 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 22 Apr 2008 15:31:33 -0400 Subject: [Numpy-discussion] Fromfunction generalization In-Reply-To: <80c99e790804221150m3552decfxf7d7a89848fb9134@mail.gmail.com> References: <480E318D.5030308@gmail.com> <80c99e790804221150m3552decfxf7d7a89848fb9134@mail.gmail.com> Message-ID: On Tue, 22 Apr 2008, lorenzo bolla apparently wrote: > M=fromfunction(lambda i,j:f(i,j,5),(1000,1000)) > P=fromfunction(lambda i,j:f(i,j,7),(1000,1000)) Maybe partial function application is nicer: fwiw, Alan Isaac From thrabe at burnham.org Tue Apr 22 17:38:16 2008 From: thrabe at burnham.org (Thomas Hrabe) Date: Tue, 22 Apr 2008 14:38:16 -0700 Subject: [Numpy-discussion] access ndarray in C++ Message-ID: Hi all! I am currently developing a python module in C/C++ which is supposed to access nd arrays as defined by the following command in python a = numpy.array([1,1,1]) I want to access the array the following way and use the nd array data for further processing in C. mymod.doSthg(a) The example code on http://numpy.sourceforge.net/numdoc/HTML/numdoc.htm#pgfId-36721 (!PyArg_ParseTuple(args, "O!",&PyArray_Type, &array)) does not work for nd arrays. I always get TypeError: argument 1 must be array, not numpy.ndarray I assume the error is the constant provided as the third parameter, saying that the imput is of PyArray_Type and no nd array. So here are my questions: 1. is there any good tutorial / example code for acessing nd arrays in C? 2. what is the difference between both (arrays and nd arrays? - I am new to python and heard there were different approaches to arrays and that nd arrays work better for multi dimensional applications. Is this right?) 3. which one of both will be used in the future? Thank you in advance for your help, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Tue Apr 22 18:00:04 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 22 Apr 2008 17:00:04 -0500 Subject: [Numpy-discussion] different behaviour in asfarray(None) In-Reply-To: <9457e7c80804220829u4f824068i11eddd832a85b6d1@mail.gmail.com> References: <80c99e790804220807x5d9fa3as891f5f53ba6c937f@mail.gmail.com> <9457e7c80804220829u4f824068i11eddd832a85b6d1@mail.gmail.com> Message-ID: <480E5FE4.5040504@enthought.com> St?fan van der Walt wrote: > 2008/4/22 lorenzo bolla : > >> I noticed a change in the behaviour of numpy.asfarray between numpy version >> 1.0.5 and 1.1.0: >> >> 1.0.5 >> ==== >> >> In [3]: numpy.asfarray(None) >> Out[3]: array(nan) >> In [4]: numpy.__version__ >> Out[4]: '1.0.5.dev4455' >> >> 1.1.0 >> ==== >> >> In [16]: numpy.asfarray(None) >> --------------------------------------------------------------------------- >> : float() argument must be a string or a number >> >> Is this intended? why? >> > > Yes, 'asfarray' is equivalent to > > array(input).astype(dtype) > > I think it would be wrong to assume that None means NaN. > I'm curious who made the change and why. There was code intentionally there to interpret None as nan for float arrays. So, I don't understand why and/or when it changed. -Travis From charlesr.harris at gmail.com Tue Apr 22 19:51:48 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 22 Apr 2008 17:51:48 -0600 Subject: [Numpy-discussion] different behaviour in asfarray(None) In-Reply-To: <480E5FE4.5040504@enthought.com> References: <80c99e790804220807x5d9fa3as891f5f53ba6c937f@mail.gmail.com> <9457e7c80804220829u4f824068i11eddd832a85b6d1@mail.gmail.com> <480E5FE4.5040504@enthought.com> Message-ID: On Tue, Apr 22, 2008 at 4:00 PM, Travis E. Oliphant wrote: > St?fan van der Walt wrote: > > 2008/4/22 lorenzo bolla : > > > >> I noticed a change in the behaviour of numpy.asfarray between numpy > version > >> 1.0.5 and 1.1.0: > >> > >> 1.0.5 > >> ==== > >> > >> In [3]: numpy.asfarray(None) > >> Out[3]: array(nan) > >> In [4]: numpy.__version__ > >> Out[4]: '1.0.5.dev4455' > >> > >> 1.1.0 > >> ==== > >> > >> In [16]: numpy.asfarray(None) > >> > --------------------------------------------------------------------------- > >> : float() argument must be a string or a > number > >> > >> Is this intended? why? > >> > > > > Yes, 'asfarray' is equivalent to > > > > array(input).astype(dtype) > > > > I think it would be wrong to assume that None means NaN. > > > I'm curious who made the change and why. There was code intentionally > there to interpret None as nan for float arrays. So, I don't > understand why and/or when it changed. > Could be here: static double MyPyFloat_AsDouble(PyObject *obj) { double ret = 0; PyObject *num = PyNumber_Float(obj); if (num == NULL) { return _getNAN(); } ret = PyFloat_AsDouble(num); Py_DECREF(num); return ret; } I don't know what PyNumber_Float does with None. I suspect all we need to do is call PyErr_Clear() in the if. The change was made so floats could be passed as strings. Chuck > > -Travis > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Apr 22 20:18:51 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 22 Apr 2008 18:18:51 -0600 Subject: [Numpy-discussion] different behaviour in asfarray(None) In-Reply-To: References: <80c99e790804220807x5d9fa3as891f5f53ba6c937f@mail.gmail.com> <9457e7c80804220829u4f824068i11eddd832a85b6d1@mail.gmail.com> <480E5FE4.5040504@enthought.com> Message-ID: On Tue, Apr 22, 2008 at 5:51 PM, Charles R Harris wrote: > > > On Tue, Apr 22, 2008 at 4:00 PM, Travis E. Oliphant < > oliphant at enthought.com> wrote: > > > St?fan van der Walt wrote: > > > 2008/4/22 lorenzo bolla : > > > > > >> I noticed a change in the behaviour of numpy.asfarray between numpy > > version > > >> 1.0.5 and 1.1.0: > > >> > > >> 1.0.5 > > >> ==== > > >> > > >> In [3]: numpy.asfarray(None) > > >> Out[3]: array(nan) > > >> In [4]: numpy.__version__ > > >> Out[4]: '1.0.5.dev4455' > > >> > > >> 1.1.0 > > >> ==== > > >> > > >> In [16]: numpy.asfarray(None) > > >> > > --------------------------------------------------------------------------- > > >> : float() argument must be a string or a > > number > > >> > > >> Is this intended? why? > > >> > > > > > > Yes, 'asfarray' is equivalent to > > > > > > array(input).astype(dtype) > > > > > > I think it would be wrong to assume that None means NaN. > > > > > I'm curious who made the change and why. There was code intentionally > > there to interpret None as nan for float arrays. So, I don't > > understand why and/or when it changed. > > > > Could be here: > > static double > MyPyFloat_AsDouble(PyObject *obj) > { > double ret = 0; > PyObject *num = PyNumber_Float(obj); > > if (num == NULL) { > return _getNAN(); > } > ret = PyFloat_AsDouble(num); > Py_DECREF(num); > return ret; > } > > I don't know what PyNumber_Float does with None. I suspect all we need to > do is call PyErr_Clear() in the if. The change was made so floats could be > passed as strings. > And that does fix that problem. However, the fix need to be a bit more complicated because evidently the array routine depends on an error return to find strings, i.e, the simple approach of removing the error gives float32('1.2') as array([1, nan, 2], dtype=float32), which is almost as annoying as In [4]: int8('123') Out[4]: array([1, 2, 3], dtype=int8) which really needs to be fixed also. Can we please check for strings *before* calling the conversion routines? Chuck > > > -Travis > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Joris.DeRidder at ster.kuleuven.be Tue Apr 22 21:56:21 2008 From: Joris.DeRidder at ster.kuleuven.be (Joris De Ridder) Date: Wed, 23 Apr 2008 03:56:21 +0200 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: References: Message-ID: On http://www.scipy.org/JorisDeRidder I've just put an example how I passed multidimensional Numpy arrays to C++ using ctypes. Perhaps it's helpful for your application. I didn't put it in the cookbook yet, because I would first like to test it a bit more. Up to now I didn't experience any bugs though. Joris On 22 Apr 2008, at 23:38, Thomas Hrabe wrote: > Hi all! > > I am currently developing a python module in C/C++ which is supposed > to access nd arrays as defined by the following command in python > > a = numpy.array([1,1,1]) > > I want to access the array the following way and use the nd array > data for further processing in C. > > mymod.doSthg(a) > > The example code on > http://numpy.sourceforge.net/numdoc/HTML/numdoc.htm#pgfId-36721 > > (!PyArg_ParseTuple(args, "O!",&PyArray_Type, &array)) > > does not work for nd arrays. I always get > TypeError: argument 1 must be array, not numpy.ndarray > > I assume the error is the constant provided as the third parameter, > saying that the imput is of PyArray_Type and no nd array. > > So here are my questions: > 1. is there any good tutorial / example code for acessing nd arrays > in C? > 2. what is the difference between both (arrays and nd arrays? - I am > new to python and heard there were different approaches to arrays > and that nd arrays work better for multi dimensional applications. > Is this right?) > 3. which one of both will be used in the future? > > Thank you in advance for your help, > Thomas > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Apr 22 22:28:04 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 22 Apr 2008 21:28:04 -0500 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: References: Message-ID: <3d375d730804221928k3db24581nbe45f0a6046d68c6@mail.gmail.com> On Tue, Apr 22, 2008 at 4:38 PM, Thomas Hrabe wrote: > > Hi all! > > I am currently developing a python module in C/C++ which is supposed to > access nd arrays as defined by the following command in python > > a = numpy.array([1,1,1]) > > I want to access the array the following way and use the nd array data for > further processing in C. > > mymod.doSthg(a) > > The example code on > http://numpy.sourceforge.net/numdoc/HTML/numdoc.htm#pgfId-36721 > > (!PyArg_ParseTuple(args, "O!",&PyArray_Type, &array)) > > does not work for nd arrays. I always get > TypeError: argument 1 must be array, not numpy.ndarray That page refers to Numeric, numpy's old predecessor. If you #included "Numeric/arrayobject.h" at the top of that C file, then the C extension is expecting a Numeric array rather than a numpy one. Here is more up-to-date documentation: http://projects.scipy.org/scipy/numpy/wiki/NumPyCAPI > I assume the error is the constant provided as the third parameter, saying > that the imput is of PyArray_Type and no nd array. > > So here are my questions: > 1. is there any good tutorial / example code for acessing nd arrays in C? > 2. what is the difference between both (arrays and nd arrays? - I am new to > python and heard there were different approaches to arrays and that nd > arrays work better for multi dimensional applications. Is this right?) There is a bit of terminological confusion. The Python standard library has a module named "array" which provides homogeneous, typed 1D arrays. numpy provides a much richer array object (technically, numpy.ndarray, but you usually construct one using the numpy.array() function). If you have numpy, the standard library's array module is of very little use; you will almost always want to use numpy. This has nothing to do with the error you got above. > 3. which one of both will be used in the future? Well, the standard library's array module will continue to exist as will numpy. Numeric, the likely cause of your problem above, is no longer maintained, so please use numpy. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From wilson.t.thompson at gmail.com Wed Apr 23 02:00:56 2008 From: wilson.t.thompson at gmail.com (wilson) Date: Tue, 22 Apr 2008 23:00:56 -0700 (PDT) Subject: [Numpy-discussion] finding minimum distance using arrays Message-ID: <2c5cd0ed-642e-4437-975f-67e29251e9d3@f63g2000hsf.googlegroups.com> hi i wrote a function to find euclidian distance between two vectors and applied it to the rows of a 2d array of floats as below from math import sqrt from numpy import array,sum def distance(vec1, vec2): return sqrt(sum([(x-y)**2 for x,y in zip(vec1, vec2)])) def findmatch(wts,inputwt): mindist=99.0 index=0 for i in range(len(wts)): d=distance(wts[i],inputwt) if d == 0.0: mindist=d index=i break elif d < mindist: mindist=d index=i return index,mindist inputwt is a 1 dim array,wts is a 2d array .i want to find the distance between inputwt and each row of wts...I used mindist=99.0 as an improbable value for distance for the sample float arrays which i used to test it... i am not sure if this is the proper way to write this function..can someone suggest how i can make it better? From robert.kern at gmail.com Wed Apr 23 02:10:15 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 23 Apr 2008 01:10:15 -0500 Subject: [Numpy-discussion] finding minimum distance using arrays In-Reply-To: <2c5cd0ed-642e-4437-975f-67e29251e9d3@f63g2000hsf.googlegroups.com> References: <2c5cd0ed-642e-4437-975f-67e29251e9d3@f63g2000hsf.googlegroups.com> Message-ID: <3d375d730804222310q2177dee1xa04865dea7552bc0@mail.gmail.com> On Wed, Apr 23, 2008 at 1:00 AM, wilson wrote: > hi > i wrote a function to find euclidian distance between two vectors and > applied it to the rows of a 2d array of floats as below > > > from math import sqrt > from numpy import array,sum > > def distance(vec1, vec2): > return sqrt(sum([(x-y)**2 for x,y in zip(vec1, vec2)])) > > def findmatch(wts,inputwt): > mindist=99.0 > index=0 > for i in range(len(wts)): > d=distance(wts[i],inputwt) > if d == 0.0: > mindist=d > index=i > break > elif d < mindist: > mindist=d > index=i > return index,mindist > > inputwt is a 1 dim array,wts is a 2d array .i want to find the > distance between inputwt and each row of wts...I used mindist=99.0 as > an improbable value for distance for the sample float arrays which i > used to test it... > i am not sure if this is the proper way to write this function..can > someone suggest how i can make it better? import numpy def findmatch(wts, inputwts): wts = numpy.asarray(wts) inputwts = numpy.asarray(wts) dx = wts - inputwts distances = (dx * dx).sum(axis=1) index = distances.argmin() mindist = distances[index] return index, mindist -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From nadavh at visionsense.com Wed Apr 23 02:14:17 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Wed, 23 Apr 2008 09:14:17 +0300 Subject: [Numpy-discussion] finding minimum distance using arrays References: <2c5cd0ed-642e-4437-975f-67e29251e9d3@f63g2000hsf.googlegroups.com> Message-ID: <710F2847B0018641891D9A216027636029C10F@ex3.envision.co.il> from numpy import * def distance(v1,v2): return sqrt(((v2-v1)**2).sum()) def findmatch(wts,inputwt): dist2 = ((wts-inputwt)**2).sum(axis=1) idx = argmin(dist2) return idx, sqrt(dist2[idx]) Nadav -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? wilson ????: ? 23-?????-08 08:00 ??: numpy-discussion at scipy.org ????: [Numpy-discussion] finding minimum distance using arrays hi i wrote a function to find euclidian distance between two vectors and applied it to the rows of a 2d array of floats as below from math import sqrt from numpy import array,sum def distance(vec1, vec2): return sqrt(sum([(x-y)**2 for x,y in zip(vec1, vec2)])) def findmatch(wts,inputwt): mindist=99.0 index=0 for i in range(len(wts)): d=distance(wts[i],inputwt) if d == 0.0: mindist=d index=i break elif d < mindist: mindist=d index=i return index,mindist inputwt is a 1 dim array,wts is a 2d array .i want to find the distance between inputwt and each row of wts...I used mindist=99.0 as an improbable value for distance for the sample float arrays which i used to test it... i am not sure if this is the proper way to write this function..can someone suggest how i can make it better? _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3453 bytes Desc: not available URL: From schut at sarvision.nl Wed Apr 23 02:59:45 2008 From: schut at sarvision.nl (Vincent Schut) Date: Wed, 23 Apr 2008 08:59:45 +0200 Subject: [Numpy-discussion] Warning: converting a masked element to nan Message-ID: Hi, Using maskedarrays (from svn numpy trunk), I sometimes get this warning: "Warning: converting a masked element to nan". It is not entirely clear to me what it means, and why it happens. Does it mean that numpy.ma is converting an element of the data part of the MA to NaN, on it's own? If so, why, and when? Or does it mean that I deliberatly have changed a data element to NaN, and if so, why on earth should I be warned for that? I think it is pretty normal to be able to create NaN's in my array... Or does it mean yet something else? And, also important: how can I avoid this (not suppress, but what should I stop doing that currently triggers this warning)? I hope someone could eleborate a bit on this... Cheers, Vincent. From stefan at sun.ac.za Wed Apr 23 03:30:18 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 23 Apr 2008 09:30:18 +0200 Subject: [Numpy-discussion] different behaviour in asfarray(None) In-Reply-To: <480E5FE4.5040504@enthought.com> References: <80c99e790804220807x5d9fa3as891f5f53ba6c937f@mail.gmail.com> <9457e7c80804220829u4f824068i11eddd832a85b6d1@mail.gmail.com> <480E5FE4.5040504@enthought.com> Message-ID: <9457e7c80804230030r511d02c8m42dd82f7eb1b5868@mail.gmail.com> 2008/4/23 Travis E. Oliphant : > I'm curious who made the change and why. There was code intentionally > there to interpret None as nan for float arrays. So, I don't > understand why and/or when it changed. I find that behaviour counterintuitive (I'm not the one who changed it, though). Why shouldn't asfarray behave the same way as np.array(x).astype(float)? Regards St?fan From haase at msg.ucsf.edu Wed Apr 23 04:06:00 2008 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 23 Apr 2008 10:06:00 +0200 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: References: Message-ID: On Wed, Apr 23, 2008 at 3:56 AM, Joris De Ridder wrote: > > > On http://www.scipy.org/JorisDeRidder I've just put an example how I passed > multidimensional Numpy arrays to C++ using ctypes. Perhaps it's helpful for > your application. I didn't put it in the cookbook yet, because I would first > like to test it a bit more. Up to now I didn't experience any bugs though. > > Joris > Hi Joris, this is a great ( short ) recipe ! Could you elaborate on the line "You need to compile myextension.cpp and make a shared library from it. The easiest way is to use Scons with the constructor file:" !? How do you call Scons in your example !? On Windows ( = cygwin !? ) , Linux and on OS-X ? Scons is now part of numpy (svn), right ? (at least the scons version you mean) Thanks again, Sebastian Haase From pgmdevlist at gmail.com Wed Apr 23 03:59:12 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 23 Apr 2008 03:59:12 -0400 Subject: [Numpy-discussion] Warning: converting a masked element to nan In-Reply-To: References: Message-ID: <200804230359.13465.pgmdevlist@gmail.com> Vincent, As a generic rule, this warning is output when the masked value (numpy.ma.masked) has to be converted to a float: in that case, a NaN is returned. Now, the exact reason why the masked value has to be converted is specific to your problem. A simple example of when this warning occurs would help to give you a detailed answer. Note that in the previous version (numpy.oldnumeric.ma), an exception was raised instead of a warning. I thought it was a bit too drastic. > Does > it mean that numpy.ma is converting an element of the data part of the > MA to NaN, on it's own? No, it shouldn't affect the data part. > Or does it mean that I > deliberatly have changed a data element to NaN, and if so, why on earth > should I be warned for that? I think it is pretty normal to be able to > create NaN's in my array... Or does it mean yet something else? Take it as a note-to-self: somewhere down the line, a masked value had to be converted to a float, not a 0d array > And, also important: how can I avoid this (not suppress, but what should > I stop doing that currently triggers this warning)? Once again, that depends, and I;m sorry I can be more specific than that: I don't have a particular example in mind... From david at ar.media.kyoto-u.ac.jp Wed Apr 23 04:14:59 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 23 Apr 2008 17:14:59 +0900 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: References: Message-ID: <480EF003.2040106@ar.media.kyoto-u.ac.jp> Sebastian Haase wrote: > > Hi Joris, > this is a great ( short ) recipe ! Could you elaborate on the line > "You need to compile myextension.cpp and make a shared library from > it. The easiest way is to use Scons with the constructor file:" > !? > How do you call Scons in your example !? On Windows ( = cygwin !? ) , > Linux and on OS-X ? > I think Joris mentioned scons because that's the easiest way to build a shared library in a cross platform way (taking care of -fPIC on unix, linker option, etc...). Scons is like make, and SConstruct files are like makefiles: you just call scons instead of make when you have a SConstruct file. > Scons is now part of numpy (svn), right ? (at least the scons version you mean) > I think this had nothing to do with using scons to build numpy/scipy per se (numpy svn does not include scons, BTW, only hooks to call scons from distutils). cheers, David From matthieu.brucher at gmail.com Wed Apr 23 04:40:52 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 23 Apr 2008 10:40:52 +0200 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: <480EF003.2040106@ar.media.kyoto-u.ac.jp> References: <480EF003.2040106@ar.media.kyoto-u.ac.jp> Message-ID: Hi David ! Is it possible to construct a Python module with Scons without tampering with different flags? I think you built your own builder (just like I did), but did you manage to put it in Scons 0.98 ? Matthieu 2008/4/23, David Cournapeau : > > Sebastian Haase wrote: > > > > Hi Joris, > > this is a great ( short ) recipe ! Could you elaborate on the line > > "You need to compile myextension.cpp and make a shared library from > > it. The easiest way is to use Scons with the constructor file:" > > !? > > How do you call Scons in your example !? On Windows ( = cygwin !? ) , > > Linux and on OS-X ? > > > > > I think Joris mentioned scons because that's the easiest way to build a > shared library in a cross platform way (taking care of -fPIC on unix, > linker option, etc...). > > Scons is like make, and SConstruct files are like makefiles: you just > call scons instead of make when you have a SConstruct file. > > > > Scons is now part of numpy (svn), right ? (at least the scons version > you mean) > > > > > I think this had nothing to do with using scons to build numpy/scipy per > se (numpy svn does not include scons, BTW, only hooks to call scons from > distutils). > > cheers, > > > David > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Wed Apr 23 04:40:50 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 23 Apr 2008 17:40:50 +0900 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: References: <480EF003.2040106@ar.media.kyoto-u.ac.jp> Message-ID: <480EF612.1060601@ar.media.kyoto-u.ac.jp> Matthieu Brucher wrote: > Hi David ! > > Is it possible to construct a Python module with Scons without > tampering with different flags? I think you built your own builder > (just like I did), but did you manage to put it in Scons 0.98 ? Unfortunately, no. I had to make a choice on which features to push to a good enough quality for 0.98 timeframe, and better fortran support was more important than python extension builder for me (I can incorporate a python extension builder without touching scons sources, but fixing fortran support meant that my own scons was different than official scons, which is not good). Having a good python extension builder is also more difficult than I first expected, too (you can't retrieve options for MS compilers from distutils, for example and I would like to support building with and without distutils' help). cheers, David From efiring at hawaii.edu Wed Apr 23 04:00:13 2008 From: efiring at hawaii.edu (Eric Firing) Date: Tue, 22 Apr 2008 22:00:13 -1000 Subject: [Numpy-discussion] Warning: converting a masked element to nan In-Reply-To: References: Message-ID: <480EEC8D.9000003@hawaii.edu> Vincent Schut wrote: > Hi, > > Using maskedarrays (from svn numpy trunk), I sometimes get this warning: > > "Warning: converting a masked element to nan". > > It is not entirely clear to me what it means, and why it happens. Does > it mean that numpy.ma is converting an element of the data part of the > MA to NaN, on it's own? If so, why, and when? Or does it mean that I > deliberatly have changed a data element to NaN, and if so, why on earth > should I be warned for that? I think it is pretty normal to be able to > create NaN's in my array... Or does it mean yet something else? > > And, also important: how can I avoid this (not suppress, but what should > I stop doing that currently triggers this warning)? > > I hope someone could eleborate a bit on this... Maybe you are doing something equivalent to this: In [1]:xx = ma.array([1.1,2.2], mask=[True,False]) In [2]:float(xx) /usr/local/lib/python2.5/site-packages/numpy/ma/core.py:1744: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") Out[2]:nan It is the __float__ method of the masked array that gives this warning. I think there is a bug in that method; it always returns a nan, sometimes with the warning, and sometimes without. By analogy with the ndarray method, it should raise an exception if the array has more than one element, and it there is only one element, it should try to convert it to a python float. Eric > > Cheers, > Vincent. > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From garry.willgoose at newcastle.edu.au Wed Apr 23 05:11:00 2008 From: garry.willgoose at newcastle.edu.au (Garry Willgoose) Date: Wed, 23 Apr 2008 19:11:00 +1000 Subject: [Numpy-discussion] f2py: optional parameters and present() bug? Message-ID: <0CA390B9-3B8C-47FA-9180-B1BF9A82B29A@newcastle.edu.au> in a F90 routine called from python the present() test (for optional parameters) always returns true whether the parameter is present in the function call or not. My F90 test code looks like this subroutine test(test1,test2,test3) integer,intent(in) :: test1 integer,intent(in),optional :: test2 integer,intent(inout),optional :: test3 write (*,*) test1,present(test2),present(test3) if (present(test2)) write (*,*) 'test2',test2 if (present(test3)) then write (*,*) 'test3',test3 test3=5 end if end subroutine test The output from python calls to this routine (its inside a module tsdtm.tsg ... just to explain why the function looks like it does). I have just created this with f2py automatically generating the interface with no hand modification of the interfaces. What this test shows is that present() always returns true no matter (1) the intent (2) whether the parameter is present in the call, or (3) the type of the parameter (real, integer ... though I haven't tested arrays etc) as well. Secondly I can't see how to get the returned value of 'test3'. It doesn't seem to be returned as a result of the function? The docs say f2py handles optional parameters and other than present () my testing suggests that optional parameters seem otherwise OK. I'm on Mac OSX and using numpy 1.0.3. Any thoughts? >>> print tsimdtm.tsg.test.__doc__ test - Function signature: test(test1,[test2,test3]) Required arguments: test1 : input int Optional arguments: test2 : input int test3 : in/output rank-0 array(int,'i') >>> tsimdtm.tsg.test(1) 1 T T test2 0 test3 0 >>> tsimdtm.tsg.test(1,2) 1 T T test2 2 test3 0 >>> tsimdtm.tsg.test(1,2,3) 1 T T test2 2 test3 3 >>> tsimdtm.tsg.test(1,test2=2) 1 T T test2 2 test3 0 >>> tsimdtm.tsg.test(1,test3=3) 1 T T test2 0 test3 3 >>> tsimdtm.tsg.test(1,test2=2,test3=3) 1 T T test2 2 test3 3 ==================================================================== Prof Garry Willgoose, Australian Professorial Fellow in Environmental Engineering, Director, Centre for Climate Impact Management (C2IM), School of Engineering, The University of Newcastle, Callaghan, 2308 Australia. Centre webpage: www.c2im.org.au Phone: (International) +61 2 4921 6050 (Tues-Fri AM); +61 2 6545 9574 (Fri PM-Mon) FAX: (International) +61 2 4921 6991 (Uni); +61 2 6545 9574 (personal and Telluric) Env. Engg. Secretary: (International) +61 2 4921 6042 email: garry.willgoose at newcastle.edu.au; g.willgoose at telluricresearch.com email-for-life: garry.willgoose at alum.mit.edu personal webpage: www.telluricresearch.com/garry ==================================================================== "Do not go where the path may lead, go instead where there is no path and leave a trail" Ralph Waldo Emerson ==================================================================== From matthieu.brucher at gmail.com Wed Apr 23 05:13:16 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 23 Apr 2008 11:13:16 +0200 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: <480EF612.1060601@ar.media.kyoto-u.ac.jp> References: <480EF003.2040106@ar.media.kyoto-u.ac.jp> <480EF612.1060601@ar.media.kyoto-u.ac.jp> Message-ID: > > Having a good python extension builder is also more difficult than I > first expected, too (you can't retrieve options for MS compilers from > distutils, for example and I would like to support building with and > without distutils' help). What are these options that must be retrieved? With a simple extension builder, we didn't encountered any bug although we have a fair number of extensions. Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Wed Apr 23 04:57:33 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 23 Apr 2008 04:57:33 -0400 Subject: [Numpy-discussion] Warning: converting a masked element to nan In-Reply-To: <480EEC8D.9000003@hawaii.edu> References: <480EEC8D.9000003@hawaii.edu> Message-ID: <200804230457.34395.pgmdevlist@gmail.com> On Wednesday 23 April 2008 04:00:13 Eric Firing wrote: > I think there is a bug in that method; it always returns a nan, > sometimes with the warning, and sometimes without. Well, the first time you get a warning, you don't the following times. > By analogy with the > ndarray method, it should raise an exception if the array has more than > one element, and it there is only one element, it should try to convert > it to a python float. Sounds like a plan. I'll take care of that tomorrow. From david at ar.media.kyoto-u.ac.jp Wed Apr 23 05:15:00 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 23 Apr 2008 18:15:00 +0900 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: References: <480EF003.2040106@ar.media.kyoto-u.ac.jp> <480EF612.1060601@ar.media.kyoto-u.ac.jp> Message-ID: <480EFE14.9070402@ar.media.kyoto-u.ac.jp> Matthieu Brucher wrote: > > Having a good python extension builder is also more difficult than I > first expected, too (you can't retrieve options for MS compilers from > distutils, for example and I would like to support building with and > without distutils' help). > > > What are these options that must be retrieved? the compiler, the compiler flags, the linker, the linker flags, the location of Python.h, etc... > With a simple extension builder, we didn't encountered any bug > although we have a fair number of extensions. Building a helloworld like extension or say numpy.core is kind of the same from a build point of view. The problem really is to work in many different configurations (for example supporting building python extensions for a different python than the one running scons), on different platforms. Waf (a project which started as a fork of scons and is now a totally different project for all purpose) has a python tool, and it is 300 lines of code: http://code.google.com/p/waf/source/browse/trunk/wafadmin/Tools/python.py And it has more support for this kind of things than scons. My current work for a python extension for scons is ~ 200 lines, not including unit tests. cheers, David From matthieu.brucher at gmail.com Wed Apr 23 05:43:01 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 23 Apr 2008 11:43:01 +0200 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: <480EFE14.9070402@ar.media.kyoto-u.ac.jp> References: <480EF003.2040106@ar.media.kyoto-u.ac.jp> <480EF612.1060601@ar.media.kyoto-u.ac.jp> <480EFE14.9070402@ar.media.kyoto-u.ac.jp> Message-ID: > > the compiler, the compiler flags, the linker, the linker flags, the > location of Python.h, etc... > > > > With a simple extension builder, we didn't encountered any bug > > although we have a fair number of extensions. > > > Building a helloworld like extension or say numpy.core is kind of the > same from a build point of view. The problem really is to work in many > different configurations (for example supporting building python > extensions for a different python than the one running scons), on > different platforms. Indeed, if you want a different python, a more complex builder has to be written, I agree ;) Matthieu Waf (a project which started as a fork of scons and is now a totally > different project for all purpose) has a python tool, and it is 300 > lines of code: > > http://code.google.com/p/waf/source/browse/trunk/wafadmin/Tools/python.py > > And it has more support for this kind of things than scons. My current > work for a python extension for scons is ~ 200 lines, not including unit > tests. > > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From Joris.DeRidder at ster.kuleuven.be Wed Apr 23 07:11:53 2008 From: Joris.DeRidder at ster.kuleuven.be (Joris De Ridder) Date: Wed, 23 Apr 2008 13:11:53 +0200 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: References: Message-ID: <14863182-A8B8-428A-9AE4-0B44F4831E24@ster.kuleuven.be> Hi Sebastian, > this is a great ( short ) recipe ! Thanks! > Could you elaborate on the line > "You need to compile myextension.cpp and make a shared library from > it. The easiest way is to use Scons with the constructor file: !? David already gave the answer. Scons allows you to make shared libraries of your C++ code without having to worry about whether it should be a .dll, a .so, or a .dylab library. Plus, it's very easy to install. Cheers, Joris Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm From pearu at cens.ioc.ee Wed Apr 23 07:45:22 2008 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Wed, 23 Apr 2008 13:45:22 +0200 Subject: [Numpy-discussion] f2py: optional parameters and present() bug? In-Reply-To: <0CA390B9-3B8C-47FA-9180-B1BF9A82B29A@newcastle.edu.au> References: <0CA390B9-3B8C-47FA-9180-B1BF9A82B29A@newcastle.edu.au> Message-ID: <480F2152.8020602@cens.ioc.ee> Garry Willgoose wrote: > in a F90 routine called from python the present() test (for optional > parameters) always returns true whether the parameter is present in > the function call or not. My F90 test code looks like this > > subroutine test(test1,test2,test3) > integer,intent(in) :: test1 > integer,intent(in),optional :: test2 > integer,intent(inout),optional :: test3 > write (*,*) test1,present(test2),present(test3) > if (present(test2)) write (*,*) 'test2',test2 > if (present(test3)) then > write (*,*) 'test3',test3 > test3=5 > end if > end subroutine test > > The output from python calls to this routine (its inside a module > tsdtm.tsg ... just to explain why the function looks like it does). I > have just created this with f2py automatically generating the > interface with no hand modification of the interfaces. What this test > shows is that present() always returns true no matter (1) the intent > (2) whether the parameter is present in the call, or (3) the type of > the parameter (real, integer ... though I haven't tested arrays etc) > as well. f2py generated wrappers always call Fortran functions with the full list of arguments. This is the reason why present always returns True. With the current f2py I can only suggest the following workaround: !f2py integer intent(in), optional :: test2 = -1 ! assuming that -1 is the magic value that indicates that option was not specified. if (present(test2).and.(.not.(test2.eq.-1))) ... > Secondly I can't see how to get the returned value of 'test3'. It > doesn't seem to be returned as a result of the function? Yes, because f2py treats `intent(inout)` arguments as input arguments that can be changed in place. Add the following comment to F90 code: !f2py intent(out) test3 that will make the wrapper to return the value of test3. > The docs say f2py handles optional parameters and other than present > () my testing suggests that optional parameters seem otherwise OK. Yes, f2py is not aware of the present function. Adding present awarness support is not trivial as it may be compiler dependent (f2py would need to know what is the value of an optional arguments such that present would return False). However, there exists a technique to circumvent this problem that I plan to implement for g3 f2py. HTH, Pearu From ondrej at certik.cz Wed Apr 23 08:32:16 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 23 Apr 2008 14:32:16 +0200 Subject: [Numpy-discussion] numpy release In-Reply-To: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> Message-ID: <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> On Mon, Apr 14, 2008 at 12:06 PM, Ondrej Certik wrote: > Hi Jarrod, > > any news with the 1.0.5? If you have same prerelease, I'd like to test > it. Debian has just moved from python2.4 to python2.5 yesterday, so > I'd like to test numpy in advance, I am sure there will be some issues > to fix. What is the plan with the release? There are some minor problems in the Debian package, some of which are fixed by the new release. I didn't fix those in Debian as I thought the new release is coming out. But if it's going to take let's say month or two, I'll fix the current Debian package now. Thanks, Ondrej From david at ar.media.kyoto-u.ac.jp Wed Apr 23 08:28:52 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 23 Apr 2008 21:28:52 +0900 Subject: [Numpy-discussion] About sse, automatic vectorization and deployment issues Message-ID: <480F2B84.6040002@ar.media.kyoto-u.ac.jp> Hi, I just came across a long thread on sse, automatic vectorization by gcc and deployment issues (linux specific, but still relevant for numpy I think). I thought it was interesting, and tackles most issues we would face once we go to that route: http://lalists.stanford.edu/lad/2008/04/0023.html cheers, David From stefan at sun.ac.za Wed Apr 23 09:21:57 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 23 Apr 2008 15:21:57 +0200 Subject: [Numpy-discussion] numpy release In-Reply-To: <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> Message-ID: <9457e7c80804230621i5b53b0aaj9882a5d790939a3b@mail.gmail.com> 2008/4/23 Ondrej Certik : > What is the plan with the release? There are some minor problems in > the Debian package, some of which are fixed by the new release. > I didn't fix those in Debian as I thought the new release is coming > out. But if it's going to take let's say month or two, I'll fix the > current Debian package now. The question is: what blocks the release of 1.1? The following tickets deserve attention: http://projects.scipy.org/scipy/numpy/ticket/750 http://projects.scipy.org/scipy/numpy/ticket/605 http://projects.scipy.org/scipy/numpy/ticket/551 http://projects.scipy.org/scipy/numpy/ticket/738 http://projects.scipy.org/scipy/numpy/ticket/743 http://projects.scipy.org/scipy/numpy/ticket/748 http://projects.scipy.org/scipy/numpy/ticket/751 http://projects.scipy.org/scipy/numpy/ticket/736 Further issues: - Alan's matrix indexing: x[0][0] for matrices currently yield x[0] Regards St?fan From fullung at gmail.com Wed Apr 23 10:15:06 2008 From: fullung at gmail.com (Albert Strasheim) Date: Wed, 23 Apr 2008 16:15:06 +0200 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: References: Message-ID: <5eec5f300804230715u4c944e7etd3b0fb72eaf38bee@mail.gmail.com> Hello, On Tue, Apr 22, 2008 at 11:38 PM, Thomas Hrabe wrote: > I am currently developing a python module in C/C++ which is supposed to > access nd arrays as defined by the following command in python You might also be interested in: http://mathema.tician.de/software/pyublas Regards, Albert From rpyle at post.harvard.edu Wed Apr 23 10:51:17 2008 From: rpyle at post.harvard.edu (Robert Pyle) Date: Wed, 23 Apr 2008 10:51:17 -0400 Subject: [Numpy-discussion] OSX installer: please test In-Reply-To: References: <764e38540804161436j5ce7a468w8bea6a3c3598d69e@mail.gmail.com> Message-ID: <397D0115-86AC-454E-9871-EA287F2FC3C2@post.harvard.edu> Here are some more complete tests on my assorted Macs. Note that on the dual G5 tower, /usr/local/lib/libgfortran.2.dylib seems to be missing, under both Tiger (10.4.11) and Leopard (10.5.2). However, under both operating systems, /usr/local/gfortran/lib/ libgfortran.2.dylib exists, as does /usr/local/lib/libgfortran.3.dylib. There is a soft link from /usr/local/lib/libgfortran.dylib to /usr/ local/lib/libgfortran.3.dylib . Should numpy be looking for libgfortran.dylib rather than libgfortran.2.dylib? (I'm just guessing; I have no idea why the file is needed.) Bob Pyle ################################# 700 MHz G3 iBook, OSX 10.4.11: Python 2.5.2 (r252:60911, Feb 22 2008, 07:57:53) [GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> np.test(all=True) Numpy is installed in /Library/Frameworks/Python.framework/Versions/ 2.5/lib/python2.5/site-packages/numpy Numpy version 1.1.0rc1 Python version 2.5.2 (r252:60911, Feb 22 2008, 07:57:53) [GCC 4.0.1 (Apple Computer, Inc. build 5363)] [ ... many successful tests ... ] Failed importing /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/ma/tests/test_morestats.py: No module named scipy.stats.distributions [ ... more successful tests ... ] ............................................./Library/Frameworks/ Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/core/ ma.py:609: UserWarning: Cannot automatically convert masked array to numeric because data is masked in one or more locations. warnings.warn("Cannot automatically convert masked array to "\ E....................................................................... ........................................................................ ........................................................................ ......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................E................... ====================================================================== ERROR: check_testUfuncRegression (numpy.core.tests.test_ma.TestUfuncs) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/core/tests/test_ma.py", line 691, in check_testUfuncRegression self.failUnless(eq(ur.filled(0), mr.filled(0), f)) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/ma/core.py", line 1556, in filled result = self._data.copy() File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/ma/core.py", line 1478, in _get_data return self.view(self._baseclass) TypeError: Cannot change data-type for object array. ====================================================================== ERROR: check_testUfuncRegression (numpy.ma.tests.test_old_ma.TestUfuncs) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/ma/tests/test_old_ma.py", line 657, in check_testUfuncRegression uf = getattr(umath, f) NameError: global name 'umath' is not defined ---------------------------------------------------------------------- Ran 1215 tests in 14.448s FAILED (errors=2) ################################# Dual G5 tower, OSX 10.4.11: ~ $ python Python 2.5.2 (r252:60911, Feb 22 2008, 07:57:53) [GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> np.test(all=True) Numpy is installed in /Library/Frameworks/Python.framework/Versions/ 2.5/lib/python2.5/site-packages/numpy Numpy version 1.1.0rc1 Python version 2.5.2 (r252:60911, Feb 22 2008, 07:57:53) [GCC 4.0.1 (Apple Computer, Inc. build 5363)] [ ... many successful tests ... ] Failed importing /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/ma/tests/test_morestats.py: dlopen(/ Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- packages/scipy/special/_cephes.so, 2): Library not loaded: /usr/local/ lib/libgfortran.2.dylib Referenced from: /Library/Frameworks/Python.framework/Versions/2.5/ lib/python2.5/site-packages/scipy/special/_cephes.so Reason: image not found [ ... more successful tests ... ] ............................................./Library/Frameworks/ Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/core/ ma.py:609: UserWarning: Cannot automatically convert masked array to numeric because data is masked in one or more locations. warnings.warn("Cannot automatically convert masked array to "\ E....................................................................... ........................................................................ ........................................................................ ........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................E................... ====================================================================== ERROR: check_testUfuncRegression (numpy.core.tests.test_ma.TestUfuncs) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/core/tests/test_ma.py", line 691, in check_testUfuncRegression self.failUnless(eq(ur.filled(0), mr.filled(0), f)) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/ma/core.py", line 1556, in filled result = self._data.copy() File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/ma/core.py", line 1478, in _get_data return self.view(self._baseclass) TypeError: Cannot change data-type for object array. ====================================================================== ERROR: check_testUfuncRegression (numpy.ma.tests.test_old_ma.TestUfuncs) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/ma/tests/test_old_ma.py", line 657, in check_testUfuncRegression uf = getattr(umath, f) NameError: global name 'umath' is not defined ---------------------------------------------------------------------- Ran 1217 tests in 3.004s FAILED (errors=2) >>> ################################# Dual G5 tower, OSX 10.5.2: ~ $ python Python 2.5.2 (r252:60911, Feb 22 2008, 07:57:53) [GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> np.test(all=True) Numpy is installed in /Library/Frameworks/Python.framework/Versions/ 2.5/lib/python2.5/site-packages/numpy Numpy version 1.1.0rc1 Python version 2.5.2 (r252:60911, Feb 22 2008, 07:57:53) [GCC 4.0.1 (Apple Computer, Inc. build 5363)] [ ... many successful tests ... ] Failed importing /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/ma/tests/test_morestats.py: dlopen(/ Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- packages/scipy/special/_cephes.so, 2): Library not loaded: /usr/local/ lib/libgfortran.2.dylib Referenced from: /Library/Frameworks/Python.framework/Versions/2.5/ lib/python2.5/site-packages/scipy/special/_cephes.so Reason: image not found [ ... more successful tests ... ] ........................................................................ ........................................................................ ........................................................................ ...............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................E................... ====================================================================== ERROR: check_testUfuncRegression (numpy.ma.tests.test_old_ma.TestUfuncs) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/ma/tests/test_old_ma.py", line 657, in check_testUfuncRegression uf = getattr(umath, f) NameError: global name 'umath' is not defined ---------------------------------------------------------------------- Ran 1179 tests in 4.035s FAILED (errors=1) >>> ################################# MacBook Pro, Intel Core Duo, OSX 10.5.2: Last login: Sun Apr 20 13:44:38 on console ~ $ python Python 2.5.2 (r252:60911, Feb 22 2008, 07:57:53) [GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> np.test(all=True) Numpy is installed in /Library/Frameworks/Python.framework/Versions/ 2.5/lib/python2.5/site-packages/numpy Numpy version 1.1.0rc1 Python version 2.5.2 (r252:60911, Feb 22 2008, 07:57:53) [GCC 4.0.1 (Apple Computer, Inc. build 5363)] [ ... many successful tests ... ] ............................................./Library/Frameworks/ Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/core/ ma.py:608: UserWarning: Cannot automatically convert masked array to numeric because data is masked in one or more locations. warnings.warn("Cannot automatically convert masked array to "\ E....................................................................... ........................................................................ ...........................................................E............ .....................................................................................................................................................................................................................................................................................................................................................E......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................E................... ====================================================================== ERROR: check_testUfuncRegression (numpy.core.tests.test_ma.test_ufuncs) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/core/tests/test_ma.py", line 691, in check_testUfuncRegression self.failUnless(eq(ur.filled(0), mr.filled(0), f)) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/ma/core.py", line 1556, in filled result = self._data.copy() File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/ma/core.py", line 1478, in _get_data return self.view(self._baseclass) TypeError: Cannot change data-type for object array. ====================================================================== ERROR: Ticket #396 ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/core/tests/test_regression.py", line 600, in check_poly1d_nan_roots self.failUnlessRaises(np.linalg.LinAlgError,getattr,p,"r") File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/unittest.py", line 320, in failUnlessRaises callableObj(*args, **kwargs) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/lib/polynomial.py", line 661, in __getattr__ return roots(self.coeffs) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/lib/polynomial.py", line 124, in roots roots = _eigvals(A) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/lib/polynomial.py", line 40, in _eigvals return eigvals(arg) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/scipy/linalg/decomp.py", line 378, in eigvals return eig(a,b=b,left=0,right=0,overwrite_a=overwrite_a) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/scipy/linalg/decomp.py", line 128, in eig a1 = asarray_chkfinite(a) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/lib/function_base.py", line 415, in asarray_chkfinite raise ValueError, "array must not contain infs or NaNs" ValueError: array must not contain infs or NaNs ====================================================================== ERROR: Ticket #396 ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/core/tests/test_regression.py", line 600, in check_poly1d_nan_roots self.failUnlessRaises(np.linalg.LinAlgError,getattr,p,"r") File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/unittest.py", line 320, in failUnlessRaises callableObj(*args, **kwargs) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/lib/polynomial.py", line 661, in __getattr__ return roots(self.coeffs) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/lib/polynomial.py", line 124, in roots roots = _eigvals(A) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/lib/polynomial.py", line 37, in _eigvals return eigvals(arg) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/scipy/linalg/decomp.py", line 378, in eigvals return eig(a,b=b,left=0,right=0,overwrite_a=overwrite_a) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/scipy/linalg/decomp.py", line 128, in eig a1 = asarray_chkfinite(a) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/lib/function_base.py", line 415, in asarray_chkfinite raise ValueError, "array must not contain infs or NaNs" ValueError: array must not contain infs or NaNs ====================================================================== ERROR: check_testUfuncRegression (numpy.ma.tests.test_old_ma.TestUfuncs) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/numpy/ma/tests/test_old_ma.py", line 657, in check_testUfuncRegression uf = getattr(umath, f) NameError: global name 'umath' is not defined ---------------------------------------------------------------------- Ran 1221 tests in 3.267s FAILED (errors=4) From haase at msg.ucsf.edu Wed Apr 23 10:56:24 2008 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 23 Apr 2008 16:56:24 +0200 Subject: [Numpy-discussion] numpy release In-Reply-To: <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> Message-ID: On Wed, Apr 23, 2008 at 2:32 PM, Ondrej Certik wrote: > On Mon, Apr 14, 2008 at 12:06 PM, Ondrej Certik wrote: > > Hi Jarrod, > > > > any news with the 1.0.5? If you have same prerelease, I'd like to test > > it. Debian has just moved from python2.4 to python2.5 yesterday, so > > I'd like to test numpy in advance, I am sure there will be some issues > > to fix. > > What is the plan with the release? There are some minor problems in > the Debian package, some of which are fixed by the new release. > I didn't fix those in Debian as I thought the new release is coming > out. But if it's going to take let's say month or two, I'll fix the > current Debian package now. > The problem is, that it was noticed that non-backwards compatible changes were committed into svn. Most noticeably a complete rewrite of the "ma" (Masked Array) package was done. As a consequence it was decided, they no one would be interested in "temporarily taking those changes out" "just to release 1.0.5". Instead the next release will be called 1.1 -- which is (only; rather still) "mostly" compatible with 1.0.x What used to be referred to a the 1.1 version, that can break more stuff, to allow for a cleaner design, will now be 1.2 HTH, Sebastian Haase From tgrav at mac.com Wed Apr 23 11:19:02 2008 From: tgrav at mac.com (Tommy Grav) Date: Wed, 23 Apr 2008 11:19:02 -0400 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: References: Message-ID: <726EE7D5-1FEE-485F-8B82-6F5E84BCDFBA@mac.com> On Apr 22, 2008, at 9:56 PM, Joris De Ridder wrote: > > On http://www.scipy.org/JorisDeRidder I've just put an example how I > passed multidimensional Numpy arrays to C++ using ctypes. Perhaps > it's helpful for your application. I didn't put it in the cookbook > yet, because I would first like to test it a bit more. Up to now I > didn't experience any bugs though. > > Joris As a total newbie in the field of extensions, where do I find ndarray.h and numpyctypes? Cheers Tommy From Joris.DeRidder at ster.kuleuven.be Wed Apr 23 11:26:23 2008 From: Joris.DeRidder at ster.kuleuven.be (Joris De Ridder) Date: Wed, 23 Apr 2008 17:26:23 +0200 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: <726EE7D5-1FEE-485F-8B82-6F5E84BCDFBA@mac.com> References: <726EE7D5-1FEE-485F-8B82-6F5E84BCDFBA@mac.com> Message-ID: <29D2E7D0-4323-4384-BEC0-95A7F3EDD07D@ster.kuleuven.be> They are attached to the wiki page. Click on "Attachments" in the menu on the left. Joris On 23 Apr 2008, at 17:19, Tommy Grav wrote: > > On Apr 22, 2008, at 9:56 PM, Joris De Ridder wrote: > >> >> On http://www.scipy.org/JorisDeRidder I've just put an example how I >> passed multidimensional Numpy arrays to C++ using ctypes. Perhaps >> it's helpful for your application. I didn't put it in the cookbook >> yet, because I would first like to test it a bit more. Up to now I >> didn't experience any bugs though. >> >> Joris > > As a total newbie in the field of extensions, where do I find > ndarray.h and numpyctypes? > > Cheers > Tommy > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm From stefan at sun.ac.za Wed Apr 23 11:34:31 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 23 Apr 2008 17:34:31 +0200 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: <726EE7D5-1FEE-485F-8B82-6F5E84BCDFBA@mac.com> References: <726EE7D5-1FEE-485F-8B82-6F5E84BCDFBA@mac.com> Message-ID: <9457e7c80804230834m2a25502bjf45b2eca7013a0b1@mail.gmail.com> 2008/4/23 Tommy Grav : > > On Apr 22, 2008, at 9:56 PM, Joris De Ridder wrote: > > > > > On http://www.scipy.org/JorisDeRidder I've just put an example how I > > passed multidimensional Numpy arrays to C++ using ctypes. Perhaps > > it's helpful for your application. I didn't put it in the cookbook > > yet, because I would first like to test it a bit more. Up to now I > > didn't experience any bugs though. > > > > Joris > > As a total newbie in the field of extensions, where do I find > ndarray.h and numpyctypes? import numpy as np print np.get_include() numpyctypes is np.ctypeslib, I assume. Regards St?fan From lists at informa.tiker.net Wed Apr 23 11:45:50 2008 From: lists at informa.tiker.net (Andreas =?iso-8859-1?q?Kl=F6ckner?=) Date: Wed, 23 Apr 2008 11:45:50 -0400 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: <5eec5f300804230715u4c944e7etd3b0fb72eaf38bee@mail.gmail.com> References: <5eec5f300804230715u4c944e7etd3b0fb72eaf38bee@mail.gmail.com> Message-ID: <200804231145.54733.lists@informa.tiker.net> On Mittwoch 23 April 2008, Albert Strasheim wrote: > Hello, > > On Tue, Apr 22, 2008 at 11:38 PM, Thomas Hrabe wrote: > > I am currently developing a python module in C/C++ which is supposed to > > access nd arrays as defined by the following command in python > > You might also be interested in: > > http://mathema.tician.de/software/pyublas Argh-- you scooped me! :) I'm preparing version 0.92.1 right now, due out later today. It should iron out most of the wrinkles that are still present in the current version, 0.92. --Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From ondrej at certik.cz Wed Apr 23 11:47:13 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 23 Apr 2008 17:47:13 +0200 Subject: [Numpy-discussion] numpy release In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> Message-ID: <85b5c3130804230847s766400cape66204950b3f54db@mail.gmail.com> On Wed, Apr 23, 2008 at 4:56 PM, Sebastian Haase wrote: > > On Wed, Apr 23, 2008 at 2:32 PM, Ondrej Certik wrote: > > On Mon, Apr 14, 2008 at 12:06 PM, Ondrej Certik wrote: > > > Hi Jarrod, > > > > > > any news with the 1.0.5? If you have same prerelease, I'd like to test > > > it. Debian has just moved from python2.4 to python2.5 yesterday, so > > > I'd like to test numpy in advance, I am sure there will be some issues > > > to fix. > > > > What is the plan with the release? There are some minor problems in > > the Debian package, some of which are fixed by the new release. > > I didn't fix those in Debian as I thought the new release is coming > > out. But if it's going to take let's say month or two, I'll fix the > > current Debian package now. > > > > The problem is, that it was noticed that non-backwards compatible > changes were committed into svn. > Most noticeably a complete rewrite of the "ma" (Masked Array) package was done. > As a consequence it was decided, they no one would be interested in > "temporarily taking those changes out" "just to release 1.0.5". > > Instead the next release will be called 1.1 -- which is (only; rather > still) "mostly" compatible with 1.0.x > What used to be referred to a the 1.1 version, that can break more > stuff, to allow for a cleaner design, will now be 1.2 OK, so I think we are going to try to fix the package in Debian right now and not wait for the release. I think this uncertainty about releases and about backward compatibility is not good. The same about uncertainty where f2py is going to end up. But you know the drill "talk is cheap, show me the code", so I know I could step up and help Jarrod with the release, but unfortunately, I don't have time for it. :) Ondrej From tgrav at mac.com Wed Apr 23 11:50:45 2008 From: tgrav at mac.com (Tommy Grav) Date: Wed, 23 Apr 2008 11:50:45 -0400 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: <29D2E7D0-4323-4384-BEC0-95A7F3EDD07D@ster.kuleuven.be> References: <726EE7D5-1FEE-485F-8B82-6F5E84BCDFBA@mac.com> <29D2E7D0-4323-4384-BEC0-95A7F3EDD07D@ster.kuleuven.be> Message-ID: <8B57C0AF-BE62-4185-8820-BDA6C306B9A6@mac.com> On Apr 23, 2008, at 11:26 AM, Joris De Ridder wrote: > > They are attached to the wiki page. Click on "Attachments" in the menu > on the left. > > Joris Thanks. Didn't know that wiki's had that :) I tried you example on a Mac OS X 10.5.2 (I am not using Scons) and got [skathi:~/Work/myCode/pyCextensions] tgrav% gcc -dynamiclib myextension.cpp -o libmyextension.dylibUndefined symbols: "___gxx_personality_v0", referenced from: ___gxx_personality_v0$non_lazy_ptr in cct0CmDB.o ld: symbol(s) not found collect2: ld returned 1 exit status Anyone have any idea what I am doing wrong? Cheers Tommy From wfspotz at sandia.gov Wed Apr 23 11:59:44 2008 From: wfspotz at sandia.gov (Bill Spotz) Date: Wed, 23 Apr 2008 09:59:44 -0600 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: <8B57C0AF-BE62-4185-8820-BDA6C306B9A6@mac.com> References: <726EE7D5-1FEE-485F-8B82-6F5E84BCDFBA@mac.com> <29D2E7D0-4323-4384-BEC0-95A7F3EDD07D@ster.kuleuven.be> <8B57C0AF-BE62-4185-8820-BDA6C306B9A6@mac.com> Message-ID: <832093D4-1576-490B-A6CA-33D891750685@sandia.gov> On Apr 23, 2008, at 9:50 AM, Tommy Grav wrote: > > On Apr 23, 2008, at 11:26 AM, Joris De Ridder wrote: > >> >> They are attached to the wiki page. Click on "Attachments" in the >> menu >> on the left. >> >> Joris > > Thanks. Didn't know that wiki's had that :) > > I tried you example on a Mac OS X 10.5.2 (I am not using Scons) and > got > > [skathi:~/Work/myCode/pyCextensions] tgrav% gcc -dynamiclib > myextension.cpp -o libmyextension.dylibUndefined symbols: > "___gxx_personality_v0", referenced from: > ___gxx_personality_v0$non_lazy_ptr in cct0CmDB.o > ld: symbol(s) not found > collect2: ld returned 1 exit status > > Anyone have any idea what I am doing wrong? These "personality" errors indicate a version mismatch with the gcc compiilers. ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From pgmdevlist at gmail.com Wed Apr 23 12:05:50 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 23 Apr 2008 12:05:50 -0400 Subject: [Numpy-discussion] Warning: converting a masked element to nan In-Reply-To: <480EEC8D.9000003@hawaii.edu> References: <480EEC8D.9000003@hawaii.edu> Message-ID: <200804231205.50459.pgmdevlist@gmail.com> On Wednesday 23 April 2008 04:00:13 Eric Firing wrote: > I think there is a bug in that method; it always returns a nan, > sometimes with the warning, and sometimes without. By analogy with the > ndarray method, it should raise an exception if the array has more than > one element, and it there is only one element, it should try to convert > it to a python float. OK, fixed in SVN5071, along with __int__ From Joris.DeRidder at ster.kuleuven.be Wed Apr 23 12:09:51 2008 From: Joris.DeRidder at ster.kuleuven.be (Joris De Ridder) Date: Wed, 23 Apr 2008 18:09:51 +0200 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: <8B57C0AF-BE62-4185-8820-BDA6C306B9A6@mac.com> References: <726EE7D5-1FEE-485F-8B82-6F5E84BCDFBA@mac.com> <29D2E7D0-4323-4384-BEC0-95A7F3EDD07D@ster.kuleuven.be> <8B57C0AF-BE62-4185-8820-BDA6C306B9A6@mac.com> Message-ID: On 23 Apr 2008, at 17:50, Tommy Grav wrote: > > On Apr 23, 2008, at 11:26 AM, Joris De Ridder wrote: > >> >> They are attached to the wiki page. Click on "Attachments" in the >> menu >> on the left. >> >> Joris > > Thanks. Didn't know that wiki's had that :) > > I tried you example on a Mac OS X 10.5.2 (I am not using Scons) and > got > > [skathi:~/Work/myCode/pyCextensions] tgrav% gcc -dynamiclib > myextension.cpp -o libmyextension.dylibUndefined symbols: > "___gxx_personality_v0", referenced from: > ___gxx_personality_v0$non_lazy_ptr in cct0CmDB.o > ld: symbol(s) not found > collect2: ld returned 1 exit status > > Anyone have any idea what I am doing wrong? Try g++ -o myextension.os -c -fPIC myextension.cpp g++ -o libmyextension.dylib -dynamiclib myextension.os on the bash prompt. (Or use Scons :-). Joris Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm From Chris.Barker at noaa.gov Wed Apr 23 13:15:54 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 23 Apr 2008 10:15:54 -0700 Subject: [Numpy-discussion] finding minimum distance using arrays In-Reply-To: <710F2847B0018641891D9A216027636029C10F@ex3.envision.co.il> References: <2c5cd0ed-642e-4437-975f-67e29251e9d3@f63g2000hsf.googlegroups.com> <710F2847B0018641891D9A216027636029C10F@ex3.envision.co.il> Message-ID: <480F6ECA.5020909@noaa.gov> Nadav Horesh wrote: > def distance(v1,v2): > return sqrt(((v2-v1)**2).sum()) note that Roberts version didn't compute the sqrt, as to find the minimum distance, you can just used the squared value. If you do want the actual distance, then you might want to use numpy.hypot, something like (untested): import numpy as np def distance(v1, v2): diff = v1-v2 return np.hypot(diff[:,0], diff[:,1]) It should be faster -- less data copying, one loop. By the way, it would be nice to have a version that would take a NX2 array, so you could write: np.hypot(v1-v2) or even np.EuclidDistance(v1, v2) But maybe it's not worth bloating numpy for... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From millman at berkeley.edu Wed Apr 23 13:23:23 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 23 Apr 2008 10:23:23 -0700 Subject: [Numpy-discussion] numpy release In-Reply-To: <9457e7c80804230621i5b53b0aaj9882a5d790939a3b@mail.gmail.com> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> <9457e7c80804230621i5b53b0aaj9882a5d790939a3b@mail.gmail.com> Message-ID: On Wed, Apr 23, 2008 at 6:21 AM, St?fan van der Walt wrote: > The question is: what blocks the release of 1.1? The following > tickets deserve attention: > > http://projects.scipy.org/scipy/numpy/ticket/750 > http://projects.scipy.org/scipy/numpy/ticket/605 > http://projects.scipy.org/scipy/numpy/ticket/551 > http://projects.scipy.org/scipy/numpy/ticket/738 > http://projects.scipy.org/scipy/numpy/ticket/743 > http://projects.scipy.org/scipy/numpy/ticket/748 > http://projects.scipy.org/scipy/numpy/ticket/751 > http://projects.scipy.org/scipy/numpy/ticket/736 > > Further issues: > > - Alan's matrix indexing: x[0][0] for matrices currently yield x[0] I was still hoping to get 1.1.0 out ASAP. Unfortunately, I have been too busy for the last week to pay attention to the release blockers. The last time I looked the only issues I felt absolutely needed to be resolved before the release were: 1. Testing the Windows binary, which is now done. 2. Testing the Mac binary, which is now done. 3. Testing or removing fromregex, which is now done: http://projects.scipy.org/scipy/numpy/ticket/719 I haven't looked carefully at the above tickets, but at a cursory glance it didn't seem like there were any new regressions. Stefan could you confirm that? If so, I would like to branch 1.1.x and tag 1.1.0 on Friday night. Unless there is some concrete reason that we need to delay this release any longer, I will send out an email later today with an 'official' timeline for the release, which will probably be on Monday if I tag on Friday night. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From stefan at sun.ac.za Wed Apr 23 13:46:40 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 23 Apr 2008 19:46:40 +0200 Subject: [Numpy-discussion] numpy release In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> <9457e7c80804230621i5b53b0aaj9882a5d790939a3b@mail.gmail.com> Message-ID: <9457e7c80804231046w3a5a1b4dm41b546403f1c6e6f@mail.gmail.com> Hi Jarrod Of those tickets, the following are serious: http://projects.scipy.org/scipy/numpy/ticket/605 (a patch is available?, David Huard) Fixing of histogram. http://projects.scipy.org/scipy/numpy/ticket/551 (old regression, Chuck and Travis) Unpickled arrays don't work as expected, because of memory alignment issues. http://projects.scipy.org/scipy/numpy/ticket/748 (old regression, Stefan) ifft pads incorrectly. I can fix this tonight, if someone would verify that this is, indeed, a bug. http://projects.scipy.org/scipy/numpy/ticket/751 (old/new regression, Stefan, David C*) ALL section not recognised in site.cfg. In Python 2.6, the DEFAULT section no longer works. I replaced it with ALL, but apparently not everything works as expected. David has a way to reproduce the problem, so I hope he can have a look, otherwise I'll try and fix it tomorrow. Regards St?fan 2008/4/23 Jarrod Millman : > On Wed, Apr 23, 2008 at 6:21 AM, St?fan van der Walt wrote: > > The question is: what blocks the release of 1.1? The following > > tickets deserve attention: > > > > http://projects.scipy.org/scipy/numpy/ticket/750 > > http://projects.scipy.org/scipy/numpy/ticket/605 > > http://projects.scipy.org/scipy/numpy/ticket/551 > > http://projects.scipy.org/scipy/numpy/ticket/738 > > http://projects.scipy.org/scipy/numpy/ticket/743 > > http://projects.scipy.org/scipy/numpy/ticket/748 > > http://projects.scipy.org/scipy/numpy/ticket/751 > > http://projects.scipy.org/scipy/numpy/ticket/736 > > > > Further issues: > > > > - Alan's matrix indexing: x[0][0] for matrices currently yield x[0] > > I was still hoping to get 1.1.0 out ASAP. Unfortunately, I have been > too busy for the last week to pay attention to the release blockers. > The last time I looked the only issues I felt absolutely needed to be > resolved before the release were: > 1. Testing the Windows binary, which is now done. > 2. Testing the Mac binary, which is now done. > 3. Testing or removing fromregex, which is now done: > http://projects.scipy.org/scipy/numpy/ticket/719 > > I haven't looked carefully at the above tickets, but at a cursory > glance it didn't seem like there were any new regressions. Stefan > could you confirm that? If so, I would like to branch 1.1.x and tag > 1.1.0 on Friday night. Unless there is some concrete reason that we > need to delay this release any longer, I will send out an email later > today with an 'official' timeline for the release, which will probably > be on Monday if I tag on Friday night. > > Thanks, > > -- > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > From thrabe at burnham.org Wed Apr 23 14:03:15 2008 From: thrabe at burnham.org (Thomas Hrabe) Date: Wed, 23 Apr 2008 11:03:15 -0700 Subject: [Numpy-discussion] access ndarray in C++ References: Message-ID: Hi all, first of all, thank you for the rich spectrum of answers, especially to Joris' sample code. What I still do not understand is the following: One can find multiple approaches to arrays, the old numeric array and the new ndarray. The new stuff seems to be backward compatible. However, where is a free documentation of the ndarray? Which headers do I have to use in order to access the ndarray object that, in my eyes, should be compatible wit PyObject in some way? I find definitions of the old array types, but none for the ndarray. I tend to give Joris code a try; but, however, the whole situation confueses me. Joris code (the ndarray.h) does not include any numpy object definition for instance. All I want to do is the following: >a = numpy.array([1,1,1]) >a.__class__ mymod.run(a ) run in C should look like this: PyObject run{ PyObject* ndArray; PyArg_ParseTuple( argv, ???? ,ndArray); for(x axis) for(y axis) do something with ndArray.data[x][y]; return something. } Is it possible to access the ndarray object like this? I do not want to create new array objects right now, later maybe. There must be a predefined way to access the data in C without using a custom hack. Finally, my primary aim is to access 3dimensional arrays of floats in C. Best, Thomas -----Original Message----- From: numpy-discussion-bounces at scipy.org on behalf of Thomas Hrabe Sent: Tue 4/22/2008 2:38 PM To: numpy-discussion at scipy.org Subject: [Numpy-discussion] access ndarray in C++ Hi all! I am currently developing a python module in C/C++ which is supposed to access nd arrays as defined by the following command in python a = numpy.array([1,1,1]) I want to access the array the following way and use the nd array data for further processing in C. mymod.doSthg(a) The example code on http://numpy.sourceforge.net/numdoc/HTML/numdoc.htm#pgfId-36721 (!PyArg_ParseTuple(args, "O!",&PyArray_Type, &array)) does not work for nd arrays. I always get TypeError: argument 1 must be array, not numpy.ndarray I assume the error is the constant provided as the third parameter, saying that the imput is of PyArray_Type and no nd array. So here are my questions: 1. is there any good tutorial / example code for acessing nd arrays in C? 2. what is the difference between both (arrays and nd arrays? - I am new to python and heard there were different approaches to arrays and that nd arrays work better for multi dimensional applications. Is this right?) 3. which one of both will be used in the future? Thank you in advance for your help, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Wed Apr 23 14:21:06 2008 From: aisaac at american.edu (Alan Isaac) Date: Wed, 23 Apr 2008 14:21:06 -0400 Subject: [Numpy-discussion] numpy release In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com><85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> Message-ID: On Wed, 23 Apr 2008, Sebastian Haase wrote: > What used to be referred to a the 1.1 version, that can > break more stuff, to allow for a cleaner design, will now > be 1.2 So ... fixing x[0][0] for matrices should wait until 1.2. Is that correct? Thank you, Alan Isaac From peridot.faceted at gmail.com Wed Apr 23 14:32:21 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Wed, 23 Apr 2008 20:32:21 +0200 Subject: [Numpy-discussion] numpy release In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> Message-ID: On 23/04/2008, Alan Isaac wrote: > On Wed, 23 Apr 2008, Sebastian Haase wrote: > > What used to be referred to a the 1.1 version, that can > > break more stuff, to allow for a cleaner design, will now > > be 1.2 > > So ... fixing x[0][0] for matrices should wait > until 1.2. Is that correct? It seems to me that everyone agrees that the current situation is broken, but there is considerable disagreement on what the correct fix is. My inclination is to apply the minimal fix you suggest, with tests and documentation, as a stopgap, and let the different factions sort it out by discussion on the way to 1.2. Objections? Anne From Chris.Barker at noaa.gov Wed Apr 23 15:08:40 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 23 Apr 2008 12:08:40 -0700 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: References: Message-ID: <480F8938.6070705@noaa.gov> Thomas Hrabe wrote: > One can find multiple approaches to arrays, the old numeric array and > the new ndarray. The new stuff seems to be backward compatible. mostly, yes. numpy (ndarray) is the only option going forward, as it's the only one being maintained. > However, where is a free documentation of the ndarray? You are right, the most complete set is Travis' book. It's a pretty good deal when you consider that he wrote much of numpy -- if it saves you an hour, it's worth the money! That being said, I think you were pointed to: http://projects.scipy.org/scipy/numpy/wiki/NumPyCAPI Which should get you going, though it does seem to be missing your key question: > Which headers do I have to use in order to access the ndarray object > that, in my eyes, should be compatible wit PyObject in some way? #include You should use distutils to compile your extension, and in your setup.py, you can put: include_dirs=[numpy.get_include()] and the header should be found. > run in C should look like this: > PyObject run{ > PyObject* ndArray; > PyArg_ParseTuple( argv, ???? ,ndArray); I think you had that right before -- you were just using the wrong headers: PyArg_ParseTuple(args, "O!",&PyArray_Type, &array) > Is it possible to access the ndarray object like this? yes, something like that certainly should work. When you've got it worked out, please add it to the Wiki. I'd poke around the wiki, and this mailing list archives, for more examples. NOTE: Most folks now think that the pain of writing extensions completely by hand is not worth it -- it's just too easy to make reference counting mistakes, etc. Most folks are now using one of: Cython (or Pyrex) SWIG ctypes For example, the SWIG typemaps distributed with numpy make it very easy to pass a numpy array into a C function such as: void MyFunc(int i, int j, int k, double *arr) where *arr is a pointer to a block of doubles that represent an (i,j,k) 3-d array. Equally easy methods are available with Cython and ctypes. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From millman at berkeley.edu Wed Apr 23 15:06:33 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 23 Apr 2008 12:06:33 -0700 Subject: [Numpy-discussion] numpy release In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> Message-ID: On Wed, Apr 23, 2008 at 11:32 AM, Anne Archibald wrote: > > So ... fixing x[0][0] for matrices should wait > > until 1.2. Is that correct? > > It seems to me that everyone agrees that the current situation is > broken, but there is considerable disagreement on what the correct fix > is. My inclination is to apply the minimal fix you suggest, with tests > and documentation, as a stopgap, and let the different factions sort > it out by discussion on the way to 1.2. +1 That sound reasonable to me. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From zachary.pincus at yale.edu Wed Apr 23 15:10:43 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Wed, 23 Apr 2008 15:10:43 -0400 Subject: [Numpy-discussion] "aligned" matrix / ctypes Message-ID: Hello all, I need to allocate a numpy array that I will then pass to a camera driver (via ctypes) so that the driver can fill the array with pixels. The catch is that the driver requires that rows of pixels start at 4- byte boundaries. The sample C++ code given for allocating memory for this is (pixels are unsigned shorts): // Two bytes for each pixel, then round // up to the next multiple of four. long width_bytes = ( ( 2 * width_pixels ) + 3 ) & -4; long allocated_size = width_bytes * height; unsigned char* image_data = new unsigned char[allocated_size]; I could make an empty uint8 numpy array of the required shape (allocated_size,) and adjust its dtype, shape, and strides to get the desired result. I'd then feed the array's ctypes data attribute to the driver to get filled in. Alternately I could allocate the data buffer via ctypes and then construct an array around it. Is either option better? How does one construct a numpy array around a ctypes memory object? Can the array "take over" the memory management for the buffer? Thanks for any suggestions, Zach From robert.kern at gmail.com Wed Apr 23 15:24:01 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 23 Apr 2008 14:24:01 -0500 Subject: [Numpy-discussion] OSX installer: please test In-Reply-To: <397D0115-86AC-454E-9871-EA287F2FC3C2@post.harvard.edu> References: <764e38540804161436j5ce7a468w8bea6a3c3598d69e@mail.gmail.com> <397D0115-86AC-454E-9871-EA287F2FC3C2@post.harvard.edu> Message-ID: <3d375d730804231224h2d266e6eva59c3f47966a11b3@mail.gmail.com> On Wed, Apr 23, 2008 at 9:51 AM, Robert Pyle wrote: > Here are some more complete tests on my assorted Macs. > > Note that on the dual G5 tower, /usr/local/lib/libgfortran.2.dylib > seems to be missing, under both Tiger (10.4.11) and Leopard (10.5.2). > However, under both operating systems, /usr/local/gfortran/lib/ > libgfortran.2.dylib exists, as does /usr/local/lib/libgfortran.3.dylib. > > There is a soft link from /usr/local/lib/libgfortran.dylib to /usr/ > local/lib/libgfortran.3.dylib . Should numpy be looking for > libgfortran.dylib rather than libgfortran.2.dylib? (I'm just > guessing; I have no idea why the file is needed.) numpy doesn't. scipy does. Unfortunately, some code snuck in that requires scipy. The code itself is optional, but the test suite loads it to test. It will be removed. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From lists at informa.tiker.net Wed Apr 23 15:30:22 2008 From: lists at informa.tiker.net (Andreas =?iso-8859-1?q?Kl=F6ckner?=) Date: Wed, 23 Apr 2008 15:30:22 -0400 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: <480F8938.6070705@noaa.gov> References: <480F8938.6070705@noaa.gov> Message-ID: <200804231530.25141.lists@informa.tiker.net> On Mittwoch 23 April 2008, Christopher Barker wrote: > NOTE: > Most folks now think that the pain of writing extensions completely by > hand is not worth it -- it's just too easy to make reference counting > mistakes, etc. Most folks are now using one of: > > Cython (or Pyrex) > SWIG > ctypes IMO, all of these deal better with C than they do with C++. There is also a number of more C++-affine solutions: - Boost Python [1]. Especially if you want usable C++ integration. (ie. more than basic templates, etc.) - sip [2]. Used for PyQt. Andreas [1] http://www.boost.org/doc/libs/1_35_0/libs/python/doc/index.html [2] http://www.riverbankcomputing.co.uk/sip/index.php -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From Chris.Barker at noaa.gov Wed Apr 23 15:46:08 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 23 Apr 2008 12:46:08 -0700 Subject: [Numpy-discussion] OSX installer: please test In-Reply-To: <764e38540804161436j5ce7a468w8bea6a3c3598d69e@mail.gmail.com> References: <764e38540804161436j5ce7a468w8bea6a3c3598d69e@mail.gmail.com> Message-ID: <480F9200.9060803@noaa.gov> Christopher Burns wrote: > I've built a Universal Mac binary for numpy 1.1.0. If > Mac people would kindly test it, I'd appreciate any feedback. > > > Download here: > https://cirl.berkeley.edu/numpy/numpy-1.1.0rc1-py2.5-macosx10.5.dmg Note that it's called *osx10.5.dmg, but it seems to work just fine on 10.4 -- same python. All tests pass: Dual PPC G5 OS-X 10.4.11 python.org version 2.5.1 Thanks! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Wed Apr 23 15:48:23 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 23 Apr 2008 12:48:23 -0700 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: <200804231530.25141.lists@informa.tiker.net> References: <480F8938.6070705@noaa.gov> <200804231530.25141.lists@informa.tiker.net> Message-ID: <480F9287.7050405@noaa.gov> Andreas Kl?ckner wrote: > IMO, all of these deal better with C than they do with C++. True, though SWIG works OK with C++. > There is also a > number of more C++-affine solutions: > > - Boost Python [1]. Especially if you want usable C++ integration. (ie. more > than basic templates, etc.) What's the status of the Boost array object? maintained? updated for recent numpy? > - sip [2]. Used for PyQt. Any numpy-specific stuff for sip? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From hoytak at gmail.com Wed Apr 23 15:51:33 2008 From: hoytak at gmail.com (Hoyt Koepke) Date: Wed, 23 Apr 2008 12:51:33 -0700 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: <200804231530.25141.lists@informa.tiker.net> References: <480F8938.6070705@noaa.gov> <200804231530.25141.lists@informa.tiker.net> Message-ID: <4db580fd0804231251k1c910410k8638c0ddc40b9e69@mail.gmail.com> Just to add something from personal experience in case it's useful... I write a lot of code that works on numpy arrays that goes between python and c++ (too used to OO to stick with pure c). I've been using scipy.weave to interface to blitz++ arrays in my c++ code. The blitz++ library has been wonderful in my experience -- it supports arrays up to 11 dimensions, slicing, etc. -- essentially a very useful subset of numpy's operations. It's also nice because it can interface without copying any data; it just provides a wrapper class that sits on a numpy array. The library is included as part of scipy. I recently wrote a function that automatically generates wrappers for a c++ function that accepts blitz++ arrays which I recently posted to the scipy list. You just specify function names and the list of argument types (nparrays included) and it generates the wrapper code. Quite easy. It's pertinent to the discussion here, so I'll send it again. Included is a full working example with setup.py. let me know if it helps/is confusing/doesn't work/etc. --Hoyt On Wed, Apr 23, 2008 at 12:30 PM, Andreas Kl?ckner wrote: > On Mittwoch 23 April 2008, Christopher Barker wrote: > > NOTE: > > Most folks now think that the pain of writing extensions completely by > > hand is not worth it -- it's just too easy to make reference counting > > mistakes, etc. Most folks are now using one of: > > > > Cython (or Pyrex) > > SWIG > > ctypes > > IMO, all of these deal better with C than they do with C++. There is also a > number of more C++-affine solutions: > > - Boost Python [1]. Especially if you want usable C++ integration. (ie. more > than basic templates, etc.) > > - sip [2]. Used for PyQt. > > Andreas > > [1] http://www.boost.org/doc/libs/1_35_0/libs/python/doc/index.html > [2] http://www.riverbankcomputing.co.uk/sip/index.php > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -- +++++++++++++++++++++++++++++++++++ Hoyt Koepke UBC Department of Computer Science http://www.cs.ubc.ca/~hoytak/ hoytak at gmail.com +++++++++++++++++++++++++++++++++++ -------------- next part -------------- A non-text attachment was scrubbed... Name: weave_wrapper.tar.gz Type: application/x-gzip Size: 3688 bytes Desc: not available URL: From Chris.Barker at noaa.gov Wed Apr 23 16:15:10 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 23 Apr 2008 13:15:10 -0700 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: <480F8938.6070705@noaa.gov> References: <480F8938.6070705@noaa.gov> Message-ID: <480F98CE.9020209@noaa.gov> Christopher Barker wrote: > Thomas Hrabe wrote: > I'd poke around the wiki, and this mailing list archives, for more examples. This may help: http://scipy.org/Cookbook/C_Extensions -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From david.huard at gmail.com Wed Apr 23 16:20:41 2008 From: david.huard at gmail.com (David Huard) Date: Wed, 23 Apr 2008 16:20:41 -0400 Subject: [Numpy-discussion] numpy release In-Reply-To: <9457e7c80804231046w3a5a1b4dm41b546403f1c6e6f@mail.gmail.com> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> <9457e7c80804230621i5b53b0aaj9882a5d790939a3b@mail.gmail.com> <9457e7c80804231046w3a5a1b4dm41b546403f1c6e6f@mail.gmail.com> Message-ID: <91cf711d0804231320i64401ab3jdf0f16ec6dc66606@mail.gmail.com> 2008/4/23, St?fan van der Walt : > > Hi Jarrod > > Of those tickets, the following are serious: > > http://projects.scipy.org/scipy/numpy/ticket/605 (a patch is > available?, David Huard) > Fixing of histogram. > I haven't found a way to fix histogram reliably without breaking the current behavior. There is a patch attached to the ticket, if the decision is to break histogram. David -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Wed Apr 23 16:34:36 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 23 Apr 2008 22:34:36 +0200 Subject: [Numpy-discussion] numpy release In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> Message-ID: <9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com> 2008/4/23 Jarrod Millman : > On Wed, Apr 23, 2008 at 11:32 AM, Anne Archibald > wrote: > > > So ... fixing x[0][0] for matrices should wait > > > until 1.2. Is that correct? > > > > It seems to me that everyone agrees that the current situation is > > broken, but there is considerable disagreement on what the correct fix > > is. My inclination is to apply the minimal fix you suggest, with tests > > and documentation, as a stopgap, and let the different factions sort > > it out by discussion on the way to 1.2. > > +1 > That sound reasonable to me. Done in r5072. Cheers St?fan From thrabe at burnham.org Wed Apr 23 16:46:34 2008 From: thrabe at burnham.org (Thomas Hrabe) Date: Wed, 23 Apr 2008 13:46:34 -0700 Subject: [Numpy-discussion] access ndarray in C++ References: <480F8938.6070705@noaa.gov> <480F98CE.9020209@noaa.gov> Message-ID: Finally! http://scipy.org/Cookbook/C_Extensions is a nice example. I have discovered a bug in ndarrayobject.h by the way. I do not know which numpy version we have installed here, but when compiling my c code with -pendantic I got this error: /home/global/python32/lib/python2.4/site-packages/numpy/core/include/numpy/ndarrayobject.h:192: error: comma at end of enumerator list /home/global/python32/lib/python2.4/site-packages/numpy/core/include/numpy/ndarrayobject.h:199: error: comma at end of enumerator list /home/global/python32/lib/python2.4/site-packages/numpy/core/include/numpy/ndarrayobject.h:211: error: comma at end of enumerator list I have no access to the newest numpy version, so if anybody has, please fix it... (in case its not on full purpose for something...) Thank you and I bet I will post here again soon, Thomas -----Original Message----- From: numpy-discussion-bounces at scipy.org on behalf of Christopher Barker Sent: Wed 4/23/2008 1:15 PM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] access ndarray in C++ Christopher Barker wrote: > Thomas Hrabe wrote: > I'd poke around the wiki, and this mailing list archives, for more examples. This may help: http://scipy.org/Cookbook/C_Extensions -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Apr 23 16:50:15 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 23 Apr 2008 15:50:15 -0500 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: References: <480F8938.6070705@noaa.gov> <480F98CE.9020209@noaa.gov> Message-ID: <3d375d730804231350r7167a52dnd50f5f4d6f3a1b6f@mail.gmail.com> On Wed, Apr 23, 2008 at 3:46 PM, Thomas Hrabe wrote: > I have discovered a bug in ndarrayobject.h by the way. I do not know which > numpy version we have installed here, but when compiling my c code with > -pendantic I got this error: > > > /home/global/python32/lib/python2.4/site-packages/numpy/core/include/numpy/ndarrayobject.h:192: > error: comma at end of enumerator list > > /home/global/python32/lib/python2.4/site-packages/numpy/core/include/numpy/ndarrayobject.h:199: > error: comma at end of enumerator list > > /home/global/python32/lib/python2.4/site-packages/numpy/core/include/numpy/ndarrayobject.h:211: > error: comma at end of enumerator list > > I have no access to the newest numpy version, so if anybody has, please fix > it... (in case its not on full purpose for something...) Fixed in SVN, thank you. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From aisaac at american.edu Wed Apr 23 16:54:42 2008 From: aisaac at american.edu (Alan Isaac) Date: Wed, 23 Apr 2008 16:54:42 -0400 Subject: [Numpy-discussion] numpy release In-Reply-To: <9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com><85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com><9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com> Message-ID: On Wed, 23 Apr 2008, St?fan van der Walt wrote: > Done in r5072. Much appreciated. I have updated to reflect this change (and its provisional status). Alan From robert.kern at gmail.com Wed Apr 23 17:02:44 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 23 Apr 2008 16:02:44 -0500 Subject: [Numpy-discussion] "aligned" matrix / ctypes In-Reply-To: References: Message-ID: <3d375d730804231402g7e4bc135hdba82c82d8057055@mail.gmail.com> On Wed, Apr 23, 2008 at 2:10 PM, Zachary Pincus wrote: > Hello all, > > I need to allocate a numpy array that I will then pass to a camera > driver (via ctypes) so that the driver can fill the array with pixels. > The catch is that the driver requires that rows of pixels start at 4- > byte boundaries. > > The sample C++ code given for allocating memory for this is (pixels > are unsigned shorts): > > // Two bytes for each pixel, then round > // up to the next multiple of four. > long width_bytes = ( ( 2 * width_pixels ) + 3 ) & -4; > long allocated_size = width_bytes * height; > unsigned char* image_data = new unsigned char[allocated_size]; > I could make an empty uint8 numpy array of the required shape > (allocated_size,) and adjust its dtype, shape, and strides to get the > desired result. I'd then feed the array's ctypes data attribute to the > driver to get filled in. Alternately I could allocate the data buffer > via ctypes and then construct an array around it. Note that the approach above doesn't ensure that the first row is correctly aligned. It just assumes that the allocator will always start a new block aligned at 4 bytes (which may be reasonable for the platforms you are targetting). Ignoring that issue for a moment, the way to make sure that the rows are aligned is very similar to how you do it in C. Round up the row length, make an array with the larger dimensions (height,width_pixels+width_pixels%2), then slice out [:,:width_pixels]. To solve the initial alignment, you overallocate a 1D array by 3 bytes and find the offset from the allocated initial address which is correctly aligned. Slice out [:allocated_size] portion of this, .view() it as uint16, reshape it to (height,width_pixels+width_pixels%2), then slice out [:,:width_pixels]. > Is either option better? How does one construct a numpy array around a > ctypes memory object? Can the array "take over" the memory management > for the buffer? Whenever possible, I try to get numpy to do the allocating. That saves many headaches. It is possible to do otherwise, but let's cover that only if you need it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From zachary.pincus at yale.edu Wed Apr 23 17:26:51 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Wed, 23 Apr 2008 17:26:51 -0400 Subject: [Numpy-discussion] "aligned" matrix / ctypes In-Reply-To: <3d375d730804231402g7e4bc135hdba82c82d8057055@mail.gmail.com> References: <3d375d730804231402g7e4bc135hdba82c82d8057055@mail.gmail.com> Message-ID: Hi, Thanks a ton for the advice, Robert! Taking an array slice (instead of trying to set up the strides, etc. myself) is a slick way of getting this result indeed. >> I need to allocate a numpy array that I will then pass to a camera >> driver (via ctypes) so that the driver can fill the array with >> pixels. >> The catch is that the driver requires that rows of pixels start at 4- >> byte boundaries. >> >> The sample C++ code given for allocating memory for this is (pixels >> are unsigned shorts): >> >> // Two bytes for each pixel, then round >> // up to the next multiple of four. >> long width_bytes = ( ( 2 * width_pixels ) + 3 ) & -4; >> long allocated_size = width_bytes * height; >> unsigned char* image_data = new unsigned char[allocated_size]; > Note that the approach above doesn't ensure that the first row is > correctly aligned. It just assumes that the allocator will always > start a new block aligned at 4 bytes (which may be reasonable for the > platforms you are targetting). Hmm, good point. The SDK/example code is pretty flakey, so I bet they just make this assumption. > To solve the initial alignment, you overallocate a 1D array by 3 bytes > and find the offset from the allocated initial address which is > correctly aligned. Sorry -- I haven't had to ever concern myself with alignment before this, so I'm not sure how to go about this step. Do I just look at the raw pointer address and see what offset I need to give it to get it to be divisible by four? > Slice out [:allocated_size] portion of this, > .view() it as uint16, reshape it to > (height,width_pixels+width_pixels%2), then slice out > [:,:width_pixels]. Everything else makes good sense. Thanks again! Zach From robert.kern at gmail.com Wed Apr 23 17:45:05 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 23 Apr 2008 16:45:05 -0500 Subject: [Numpy-discussion] "aligned" matrix / ctypes In-Reply-To: References: <3d375d730804231402g7e4bc135hdba82c82d8057055@mail.gmail.com> Message-ID: <3d375d730804231445k5379b9f9m516c54ad25265178@mail.gmail.com> On Wed, Apr 23, 2008 at 4:26 PM, Zachary Pincus wrote: [ I wrote] > > To solve the initial alignment, you overallocate a 1D array by 3 bytes > > and find the offset from the allocated initial address which is > > correctly aligned. > > Sorry -- I haven't had to ever concern myself with alignment before > this, so I'm not sure how to go about this step. Do I just look at the > raw pointer address and see what offset I need to give it to get it to > be divisible by four? Yes. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From peridot.faceted at gmail.com Wed Apr 23 17:49:53 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Wed, 23 Apr 2008 23:49:53 +0200 Subject: [Numpy-discussion] "aligned" matrix / ctypes In-Reply-To: References: <3d375d730804231402g7e4bc135hdba82c82d8057055@mail.gmail.com> Message-ID: On 23/04/2008, Zachary Pincus wrote: > Hi, > > Thanks a ton for the advice, Robert! Taking an array slice (instead of > trying to set up the strides, etc. myself) is a slick way of getting > this result indeed. It's worth mentioning that there was some discussion of adding an allocator for aligned arrays. It got sidetracked into a discussion of SSE support for numpy, but there are all sorts of reasons to want aligned arrays in numpy, as this post demonstrates. Seeing as it's just a few lines worth of pure python, is anyone interested in writing an aligned_empty() for numpy? Anne From Chris.Barker at noaa.gov Wed Apr 23 17:55:27 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 23 Apr 2008 14:55:27 -0700 Subject: [Numpy-discussion] numpy release In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> <9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com> Message-ID: <480FB04F.80006@noaa.gov> Alan Isaac wrote: > I have updated > to reflect this change (and its provisional status). Thanks for writing this up -- it really clarifies what's being proposed. A few comments on that write up: > For matrix x, should x[0].A[0] == x.A[0][0]? That is, should the array > attribute of a Row Vector or Column Vector be 1d? Absolutely -- a vector is a 1-d array. The only difference here is that we can preserve the concept of a row vs. a column vector -- key to linear algebra, which is the whole point of the Matrix class. I'd be just as happy if a RowVector an ColumnVector were a more unique object -- a 1-d array with the operation overloaded, rather than a (1,n) or (n,1) matrix, but I understand that would be more work. > Deviations from the behavior of ndarrays. Specifically, iteration over > a matrix will not yield 1d NumPy arrays. Why should they? indexing a nested list gives a different result than indexing an ndarray, etc, etc, etc. The POINT of matrixes is that they are different! This proposal makes them a little more similar to ndarrays than the old behavior (always returning a matrix). > Inconsistency in the indexing style for producing row vectors and > column vectors not really: row vector: M[i,:] column vector: M[:,i] The fact that there is another way to get a row vector: M[i] is a bonus. I once proposed making M[i] raise an exception for the sake of enforcing consistency, but no one liked that! > Loss of a standard way to extract a row or a column as a submatrix > (rather than a vector) What's wrong with: M[i:i+1, :] and M[:, i:i+1] This is totally consistent with arrays (and other python sequences). Yes, it's a bit more awkward, but with the new vectors, it's also going to be needed far less. > No clear gain in functionality over the simpler proposal 2. Well, I won't judge what's "clear" but I think this adds functionality. The concept of row and column vectors is a key part of linear algebra. Sure, you can model them as matrixes that happen to have only one row or one column, but that's what we had, but there seem to be real issues with use of that. Option (2), is "Let a Matrix be a container of 1d arrays.". Sounds simple, but do those arrays represent row or column vectors? Either would make sense. The proposal is for them to be row vectors, which is fine, but then if you want them to act like a row vectors, do you need to convert them back into matrices? Even if not, how do you get a column vector -- by indexing the current way, and getting a (n,1) matrix, so that now we have inconsistency between how row and column vectors are treated -- not a clean API. And this is about a clean API for linear algebra -- otherwise, we'd just all use arrays (which many do!) Aside from the fact that someone needs to write the code -- why don't people like the row/column vector idea? It just feels so natural to me: A matrix is a 2-d array with some functionality overloaded to make it convenient to do linear algebra. A vector is a 1-d array with some functionality overloaded to make it convenient to do linear algebra. A scalar is the same in both contexts, so no need to change anything there. And yes, maybe this will lead to tensors some day! Another couple questions: -- would there be constructors for vectors? np.RowVector((1,2,3,4,5)) np.ColumnVector((1,2,3,4,5,6)) and would: A_RowVector.T == A_ColumnVector I would think so. One more note: All the example code I've seen during this discussion have been examples of iterating and indexing -- not one has actually been linear algebra -- perhaps we should see some of those before making any decisions. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From wfspotz at sandia.gov Wed Apr 23 19:08:58 2008 From: wfspotz at sandia.gov (Bill Spotz) Date: Wed, 23 Apr 2008 17:08:58 -0600 Subject: [Numpy-discussion] numpy release In-Reply-To: <480FB04F.80006@noaa.gov> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> <9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com> <480FB04F.80006@noaa.gov> Message-ID: On Apr 23, 2008, at 3:55 PM, Christopher Barker wrote: >> For matrix x, should x[0].A[0] == x.A[0][0]? That is, should the >> array >> attribute of a Row Vector or Column Vector be 1d? > > Absolutely -- a vector is a 1-d array. The only difference here is > that > we can preserve the concept of a row vs. a column vector -- key to > linear algebra, which is the whole point of the Matrix class. I'd be > just as happy if a RowVector an ColumnVector were a more unique object > -- a 1-d array with the operation overloaded, rather than a (1,n) or > (n,1) matrix, but I understand that would be more work. The way I envisioned this was that RowVector and ColumnVector would inherit from Matrix, so that you would inherit all of the Matrix functionality. They would just be special cases where ncol=0 or nrow=0, allowing single indexing. If Matrix-Matrix multiplication is implemented properly, then the Row/ ColumnVector multiplication operators should automatically work. >> Inconsistency in the indexing style for producing row vectors and >> column vectors > > not really: I agree with Alan here. M[i] returns a RowVector, but there is no corresponding single index way to retrieve a ColumnVector (unless we want to use __call__() to make M(j) return a ColumnVector . . . but this is highly unintuitive.) Thinking out loud: we could support M[i=#] return a RowVector M[j=#] return a ColumnVector M[#] equivalent to M[i=#] > The fact that there is another way to get a row vector: M[i] is a > bonus. Except that the behavior of M[i] is one of the driving issues of the conversation. >> Loss of a standard way to extract a row or a column as a submatrix >> (rather than a vector) If Row/ColumnVector inherit from Matrix, then this is not strictly true. There is no reason that these classes could not support [i,j] indexing, which means they would BE Matrices (inheritence) and they would BEHAVE like Matrices (except for V[i][j] . . . argh). >> No clear gain in functionality over the simpler proposal 2. Gains: (1) non-scalar extractions from linear algebra objects ARE and BEHAVE like linear algebra objects; (2) a clear path for dense and sparse matrices to have the same (or at least analogous) interfaces. > Aside from the fact that someone needs to write the code -- why don't > people like the row/column vector idea? It just feels so natural to > me: Unless I'm misremembering, Alan is the only one who has expressed concerns and he is willing to concede to the design if others agree. > A matrix is a 2-d array with some functionality overloaded to make it > convenient to do linear algebra. > > A vector is a 1-d array with some functionality overloaded to make it > convenient to do linear algebra. Again, I would argue for Vectors inheriting from Matrix. I would make the case based on code reuse and elegance, although these might be overridden by some other concern. > And yes, maybe this will lead to tensors some day! Don't see why not. > Another couple questions: -- would there be constructors for vectors? > > np.RowVector((1,2,3,4,5)) > np.ColumnVector((1,2,3,4,5,6)) Yes. They should be full-fledged classes, although with inheriting from Matrix, the implementation should be relatively small. What about np.Matrix() np.Matrix() ? > and would: > A_RowVector.T == A_ColumnVector Yes. > All the example code I've seen during this discussion have been > examples > of iterating and indexing -- not one has actually been linear > algebra -- > perhaps we should see some of those before making any decisions. Right. I have generally thought about this in the context of, say, a Krylov-space iterative method, and what that type of interface would lead to the most readable code. ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From millman at berkeley.edu Wed Apr 23 19:28:51 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 23 Apr 2008 18:28:51 -0500 Subject: [Numpy-discussion] Fixing/breaking histogram for numpy 1.1 Message-ID: On Wed, Apr 23, 2008 at 3:20 PM, David Huard wrote: > I haven't found a way to fix histogram reliably without breaking the current > behavior. There is a patch attached to the ticket, if the decision is to > break histogram. +1 As far as I am concerned it seems like histogram is all ready broken, so fixing it should be more important than retaining backward compatibility. Is anyone opposed to David's patch: http://projects.scipy.org/scipy/numpy/ticket/605 -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From stefan at sun.ac.za Wed Apr 23 20:20:46 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 24 Apr 2008 02:20:46 +0200 Subject: [Numpy-discussion] numpy release In-Reply-To: <480FB04F.80006@noaa.gov> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> <9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com> <480FB04F.80006@noaa.gov> Message-ID: <9457e7c80804231720y75f7ccf9wbcba4d3e48eebad6@mail.gmail.com> 2008/4/23 Christopher Barker : > Aside from the fact that someone needs to write the code -- why don't > people like the row/column vector idea? It just feels so natural to me: I wrote most of the code last week (see the previous thread on the mailing list for a patch). It currently only implements Vector (not RowVector and ColumnVector), but those two would be easy to add. Cheers St?fan From lists at informa.tiker.net Wed Apr 23 21:47:46 2008 From: lists at informa.tiker.net (Andreas =?iso-8859-1?q?Kl=F6ckner?=) Date: Wed, 23 Apr 2008 21:47:46 -0400 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: <480F9287.7050405@noaa.gov> References: <200804231530.25141.lists@informa.tiker.net> <480F9287.7050405@noaa.gov> Message-ID: <200804232147.50022.lists@informa.tiker.net> On Mittwoch 23 April 2008, Christopher Barker wrote: > What's the status of the Boost array object? maintained? updated for > recent numpy? The numeric.hpp included in Boost.Python is a joke. It does not use the native API. PyUblas [1] fills this gap, by allowing you to use Boost.Ublas on the C++ side and Numpy on the Python side. It is somewhat like what Hoyt describes, except for a different environment. Here's a table: | Hoyt | Andreas -------------------+------------+-------- C++ Matrix Library | Blitz++ | Boost.Ublas Wrapper Generator | Weave | Boost.Python Wrapper | w_wrap.tgz | PyUblas :) Sorry, that was too much fun to pass up. [1] http://tiker.net/doc/pyublas/index.html > > - sip [2]. Used for PyQt. > > Any numpy-specific stuff for sip? Not as far as I'm aware. In fact, I don't know of any uses of sip outside of Qt/KDE-related things. Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From tim.hochberg at ieee.org Wed Apr 23 22:16:35 2008 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Wed, 23 Apr 2008 19:16:35 -0700 Subject: [Numpy-discussion] numpy release In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> <9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com> <480FB04F.80006@noaa.gov> Message-ID: [CHOP] The proposals thus far don't address two of the major issues I have with the matrix class: 1. The matrices and arrays should become more alike if possible and should share more of the same code base. From what I've seen, the people who write the code (for numpy) don't actually use matrices, they use the arrays. This is one of the main reasons matrices still have so many problems IMO; if one of the main developers were using them, they'd never have put up with it. This may be changing to some extent now, but the more we can make matrices and arrays share code and interface, the happier we'll be. None of the new rules strike me as helping in this arena and may in fact hurt. Again, IMO. 2. This doesn't support the concept of arrays of matrices/vectors, which is one thing I've always wanted out of the matrix class. For example it would be nice to be able to spell: 'A' is a 1D array of matrices (3D) overall, 'B' is a 1D array of vectors (3D) overall, matrix multiply them together, yielding a 1D array of matrices (3D) overall. I have frequently run into various permutations of this problem. You can fake it with loops over dot, but the efficiency is very bad for small arrays, plus it's ugly. I have lot's of vague ideas on this subject that have been floating around my head for the past couple of years. Basically, I think the way to go is to probably add some meta information to arrays that make them look a little bit more like tensors. Then, if one is careful (and things work out as I hope) matrices could be implemented as a thin layer on top of arrays. Or perhaps, if we are very lucky, done away with altogether. Anyway, that's all very vague and hand-wavey; if I can spring free some time, I'll try to write something up in the not too distant future. I'm glad to see that any major changes are being put off for a while: I think we should be able to do better than just patch up the old matrix class. -tim -------------- next part -------------- An HTML attachment was scrubbed... URL: From millman at berkeley.edu Thu Apr 24 00:23:59 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 23 Apr 2008 21:23:59 -0700 Subject: [Numpy-discussion] [SciPy-dev] stats.mstats & stats.mmorestats In-Reply-To: <200804141455.10414.pgmdevlist@gmail.com> References: <200804141455.10414.pgmdevlist@gmail.com> Message-ID: On Mon, Apr 14, 2008 at 11:55 AM, Pierre GM wrote: > I just committed two modules (and the corresponding unittests) to the > scipy.stats package. Feel free to test and send some feedback. These packages > are meant to replace numpy.ma.mstats and numpy.ma.morestats: may I go and > erase those packages from the numpy SVN ? Yes, please go ahead and remove numpy.ma.mstats and numpy.ma.morestats. Also feel free to close: http://projects.scipy.org/scipy/numpy/ticket/753 Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From aisaac at american.edu Thu Apr 24 02:31:03 2008 From: aisaac at american.edu (Alan G Isaac) Date: Thu, 24 Apr 2008 02:31:03 -0400 Subject: [Numpy-discussion] numpy release In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com><85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com><9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com><480FB04F.80006@noaa.gov> Message-ID: On Wed, 23 Apr 2008, Bill Spotz apparently wrote: > Gains: (1) non-scalar extractions from linear algebra > objects ARE and BEHAVE like linear algebra objects; I believe this is not a gain, since there is a standard way to do this now, which would remain under the second proposal. > (2) a clear path for dense and sparse matrices to have the > same (or at least analogous) interfaces. This may prove true. I'm agnostic. What actually turns on this? I believe it is the following: if iterating over a sparse matrix yields sparse vectors that behave like a sparse matrix, then iterating over a matrix should return a vector that behaves like a matrix. But I have not seen any detailed discussion of this design issue. E.g., iterating over a sparse matrix could yield instead a sparse object that behaves like a 1d array. I should add that if the first approach is taken, I agree that it is natural for these "vectors" to inherit from matrix. With respect to the matrix object, my only "objection" has been that I'd like to hear someone state clearly what functionality is gained by following the more complex first proposal instead of the second proposal. The reason: the cost of complexity should be justified by a gain in functionality. It *may* be that the link between the behavior of matrices and that of sparse matrices is where the gain manifests, but that is not yet clear to me. > Unless I'm misremembering, Alan is the only one who has > expressed concerns and he is willing to concede to the > design if others agree. I'm in no position to concede or not concede anything, since I am not writing code, but my core concerns will be met by either proposal. Thanks! Alan From klemm at phys.ethz.ch Thu Apr 24 05:35:41 2008 From: klemm at phys.ethz.ch (Hanno Klemm) Date: Thu, 24 Apr 2008 11:35:41 +0200 Subject: [Numpy-discussion] Unexpected behaviour by in place addition of array and masked array Message-ID: Hi, I tried adding a masked array and a usual array in place and got back the usual array. Is this expected behaviour? I would have expected to get a masked array back. The below example illustrates what I mean. >>> import numpy as N >>> a = N.zeros((3,3), dtype=float) >>> b = N.ma.MaskedArray([[1,2,3],[2,3,4],[4,5,6]], mask = [[0,0,0],[1,0,1], [0,0,0]]) >>> b array(data = [[ 1 2 3] [999999 3 999999] [ 4 5 6]], mask = [[False False False] [ True False True] [False False False]], fill_value=999999) >>> a+b array(data = [[ 1.00000000e+00 2.00000000e+00 3.00000000e+00] [ 1.00000000e+20 3.00000000e+00 1.00000000e+20] [ 4.00000000e+00 5.00000000e+00 6.00000000e+00]], mask = [[False False False] [ True False True] [False False False]], fill_value=1e+20) >>> a+=b >>> a array([[ 1., 2., 3.], [ 0., 3., 0.], [ 4., 5., 6.]]) >>> N.__version__ '1.0.4' >>> Best regards, Hanno -- Hanno Klemm klemm at phys.ethz.ch From nadavh at visionsense.com Thu Apr 24 06:34:28 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Thu, 24 Apr 2008 13:34:28 +0300 Subject: [Numpy-discussion] numpy release References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com><85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com><9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com><480FB04F.80006@noaa.gov> Message-ID: <710F2847B0018641891D9A216027636029C11A@ex3.envision.co.il> +1 Nadav -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? Timothy Hochberg ????: ? 24-?????-08 04:16 ??: Discussion of Numerical Python ????: Re: [Numpy-discussion] numpy release [CHOP] The proposals thus far don't address two of the major issues I have with the matrix class: 1. The matrices and arrays should become more alike if possible and should share more of the same code base. From what I've seen, the people who write the code (for numpy) don't actually use matrices, they use the arrays. This is one of the main reasons matrices still have so many problems IMO; if one of the main developers were using them, they'd never have put up with it. This may be changing to some extent now, but the more we can make matrices and arrays share code and interface, the happier we'll be. None of the new rules strike me as helping in this arena and may in fact hurt. Again, IMO. 2. This doesn't support the concept of arrays of matrices/vectors, which is one thing I've always wanted out of the matrix class. For example it would be nice to be able to spell: 'A' is a 1D array of matrices (3D) overall, 'B' is a 1D array of vectors (3D) overall, matrix multiply them together, yielding a 1D array of matrices (3D) overall. I have frequently run into various permutations of this problem. You can fake it with loops over dot, but the efficiency is very bad for small arrays, plus it's ugly. I have lot's of vague ideas on this subject that have been floating around my head for the past couple of years. Basically, I think the way to go is to probably add some meta information to arrays that make them look a little bit more like tensors. Then, if one is careful (and things work out as I hope) matrices could be implemented as a thin layer on top of arrays. Or perhaps, if we are very lucky, done away with altogether. Anyway, that's all very vague and hand-wavey; if I can spring free some time, I'll try to write something up in the not too distant future. I'm glad to see that any major changes are being put off for a while: I think we should be able to do better than just patch up the old matrix class. -tim -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 4529 bytes Desc: not available URL: From stefan at sun.ac.za Thu Apr 24 07:13:00 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 24 Apr 2008 13:13:00 +0200 Subject: [Numpy-discussion] numpy release In-Reply-To: <710F2847B0018641891D9A216027636029C11A@ex3.envision.co.il> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com> <480FB04F.80006@noaa.gov> <710F2847B0018641891D9A216027636029C11A@ex3.envision.co.il> Message-ID: <9457e7c80804240413k6cfbaf03i93cb67b12e0b1c0f@mail.gmail.com> 2008/4/24 Nadav Horesh : > +1 +1 to what? I'm glad that Tim chimed in -- his opinion is always valued -- but I didn't see any concrete suggestions on how to improve the situation. I'll gladly wait for his code, though :) Regards St?fan From pearu at cens.ioc.ee Thu Apr 24 08:50:05 2008 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Thu, 24 Apr 2008 14:50:05 +0200 Subject: [Numpy-discussion] New web site for the F2PY. tool Message-ID: <481081FD.3020902@cens.ioc.ee> Hi, I have created a new web site for the F2PY tool: http://www.f2py.org that will be used to collect F2PY related information. At the moment, the site contains minimal information but hopefully this will improve in future. One can add content to the f2py.org site after registration (see Register in the Login page). Best regards, Pearu From nadavh at visionsense.com Thu Apr 24 08:53:07 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Thu, 24 Apr 2008 15:53:07 +0300 Subject: [Numpy-discussion] numpy release References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com><9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com><480FB04F.80006@noaa.gov><710F2847B0018641891D9A216027636029C11A@ex3.envision.co.il> <9457e7c80804240413k6cfbaf03i93cb67b12e0b1c0f@mail.gmail.com> Message-ID: <710F2847B0018641891D9A216027636029C11C@ex3.envision.co.il> Some, including me, stated that the matrix class suffer some deficiencies, so, after playing a bit with matrices, we always go back to the good old ndarray class. There were some arguments on this list what/if should be dome to correct it. I welcome any attempt to build a code that demonstrates an alternative. Nadav. -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? St?fan van der Walt ????: ? 24-?????-08 13:13 ??: Discussion of Numerical Python ????: Re: [Numpy-discussion] numpy release 2008/4/24 Nadav Horesh : > +1 +1 to what? I'm glad that Tim chimed in -- his opinion is always valued -- but I didn't see any concrete suggestions on how to improve the situation. I'll gladly wait for his code, though :) Regards St?fan _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3709 bytes Desc: not available URL: From bsouthey at gmail.com Thu Apr 24 09:46:19 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 24 Apr 2008 08:46:19 -0500 Subject: [Numpy-discussion] numpy release In-Reply-To: <710F2847B0018641891D9A216027636029C11C@ex3.envision.co.il> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com><9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com><480FB04F.80006@noaa.gov><710F2847B0018641891D9A216027636029C11A@ex3.envision.co.il> <9457e7c80804240413k6cfbaf03i93cb67b12e0b1c0f@mail.gmail.com> <710F2847B0018641891D9A216027636029C11C@ex3.envision.co.il> Message-ID: <48108F2B.60500@gmail.com> Hi, I would like to use the matrix functionality but I, like others, get by without it. +1 to Tim's and Nadav's comments. As Tim said, there should be seamless integration between concepts of vectors and matrices - after all there really no distinction between them in linear algebra. -2 for having rowvector and columnvector - both numpy and I should know the orientation, if not, I deserve garbage or an error. 0 for vector - I don't see the need for it as a unique entity but understand the simplicity of saying vector. Yes, I agree that talk is cheap but it avoids wasting time on code that will not be used and allows time to understand what people really want. Bruce Nadav Horesh wrote: > Some, including me, stated that the matrix class suffer some deficiencies, so, after playing a bit with matrices, we always go back to the good old ndarray class. There were some arguments on this list what/if should be dome to correct it. I welcome any attempt to build a code that demonstrates an alternative. > > Nadav. > > -----????? ??????----- > ???: numpy-discussion-bounces at scipy.org ??? St?fan van der Walt > ????: ? 24-?????-08 13:13 > ??: Discussion of Numerical Python > ????: Re: [Numpy-discussion] numpy release > > 2008/4/24 Nadav Horesh : > >> +1 >> > > +1 to what? I'm glad that Tim chimed in -- his opinion is always > valued -- but I didn't see any concrete suggestions on how to improve > the situation. I'll gladly wait for his code, though :) > > Regards > St?fan > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From pgmdevlist at gmail.com Thu Apr 24 09:55:18 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 24 Apr 2008 09:55:18 -0400 Subject: [Numpy-discussion] Unexpected behaviour by in place addition of array and masked array In-Reply-To: References: Message-ID: <200804240955.18166.pgmdevlist@gmail.com> Hanno, On Thursday 24 April 2008 05:35:41 Hanno Klemm wrote: > >>> N.__version__ > '1.0.4' OK, you're using the "old" version of masked array. > I tried adding a masked array and a usual array in place and got back > the usual array. Is this expected behaviour? I would have expected to > get a masked array back. Yes, it is expected: as you're adding in place, you can't change the nature of the array, so a ndarray will stay a ndarray. Following the same line, adding a float array in place to a int array will give you a int array. From pav at iki.fi Thu Apr 24 10:28:39 2008 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 24 Apr 2008 14:28:39 +0000 (UTC) Subject: [Numpy-discussion] numpy release References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> <9457e7c80804230621i5b53b0aaj9882a5d790939a3b@mail.gmail.com> <9457e7c80804231046w3a5a1b4dm41b546403f1c6e6f@mail.gmail.com> <91cf711d0804231320i64401ab3jdf0f16ec6dc66606@mail.gmail.com> Message-ID: Wed, 23 Apr 2008 16:20:41 -0400, David Huard wrote: > 2008/4/23, St?fan van der Walt : > > Of those tickets, the following are serious: > > > > http://projects.scipy.org/scipy/numpy/ticket/605 (a patch is > > available?, David Huard) > > Fixing of histogram. > > I haven't found a way to fix histogram reliably without breaking the > current behavior. There is a patch attached to the ticket, if the > decision is to break histogram. Summary of the facts (again...): a) histogram's docstring does not match its behavior wrt discarding data b) given variable-width bins, histogram(..., normed=True) the results are wrong c) it might make more sense to handle discarding data in some other way than what histogram does now I think there are now a couple of choices what to do with this: A) Change the semantics of histogram function. Old code using histogram will just simply break, maybe in mysterious ways. B) Rename the bins parameter to bin_edges or something else, so that any old code using histogram immediately raises an exception that is easily understood. C) Create a new parameter with more sensible behavior and a name different from "bins", and deprecate (at least giving sequences to) the "bins" parameter: put up a DeprecationWarning if the user does this, but still produce the same results as the old histogram. This way the user can forward-port her code at leisure. D) Or, retain the old behavior (values below lowest bin ignored) and just fix the docstring and the normed=True bug? (I have a patch doing this.) So which one (or something else) do we choose for 1.1.0? -- Pauli Virtanen From tgrav at mac.com Thu Apr 24 10:58:08 2008 From: tgrav at mac.com (Tommy Grav) Date: Thu, 24 Apr 2008 10:58:08 -0400 Subject: [Numpy-discussion] numpy release In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> <9457e7c80804230621i5b53b0aaj9882a5d790939a3b@mail.gmail.com> <9457e7c80804231046w3a5a1b4dm41b546403f1c6e6f@mail.gmail.com> <91cf711d0804231320i64401ab3jdf0f16ec6dc66606@mail.gmail.com> Message-ID: <0A5D85E4-5774-4878-BD5C-690150EE7B93@mac.com> I think a long term strategy needs to be adopted for histogram. Right now there is a great confusion in what the "bins" keyword does. Right now it is defined as the lower edge of each bin, meaning that the last bin is open ended and [inf,bin0> does not exist. While this may not be the right thing to fix in 1.1.0, I would really like to see it fixed somewhere down the line. On Apr 24, 2008, at 10:28 AM, Pauli Virtanen wrote: > Wed, 23 Apr 2008 16:20:41 -0400, David Huard wrote: >> I haven't found a way to fix histogram reliably without breaking the >> current behavior. There is a patch attached to the ticket, if the >> decision is to break histogram. > > Summary of the facts (again...): > > a) histogram's docstring does not match its behavior wrt > discarding data This is an easy fix and should definitively go into 1.1.0 :) > b) given variable-width bins, histogram(..., normed=True) > the results are wrong Also a quick fix that should be part of 1.1.0 > c) it might make more sense to handle discarding data in some > other way than what histogram does now I would like to see this, but it does not have to happen in 1.1.0 :) > I think there are now a couple of choices what to do with this: > > A) Change the semantics of histogram function. Old code using > histogram > will just simply break, maybe in mysterious ways Not really a satisfactory approach. I really don't mind, even though it would break some code of mine. I would rather see a better function and have to do some code changes, than the current confusion. Other people will likely disagree. > B) Rename the bins parameter to bin_edges or something else, so that > any old code using histogram immediately raises an exception that is > easily understood. Given this approach bin_edges would contain one more value than bins given that the right edge of the last bin has to be defined. > C) Create a new parameter with more sensible behavior and a name > different from "bins", and deprecate (at least giving sequences to) > the > "bins" parameter: put up a DeprecationWarning if the user does this, > but > still produce the same results as the old histogram. This way the user > can forward-port her code at leisure. I think this is probably the best approach to accommodate everyone. > So which one (or something else) do we choose for 1.1.0? > > -- > Pauli Virtanen Cheers Tommy From zachary.pincus at yale.edu Thu Apr 24 11:41:21 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Thu, 24 Apr 2008 11:41:21 -0400 Subject: [Numpy-discussion] problem with view() and strided arrays? Message-ID: Hello all, I'm writing up a general function to allocate aligned numpy arrays (I'll post it shortly, as Anne suggested that such a function would be useful). However, I've run into trouble with using ndarray.view() in odd corner- cases: In : numpy.__version__ Out: '1.1.0.dev5077' In : a = numpy.ones((3,8),dtype=numpy.uint8) In : a.view(numpy.uint16) Out: array([[257, 257, 257, 257], [257, 257, 257, 257], [257, 257, 257, 257]], dtype=uint16) In : a = numpy.ones((3,9),dtype=numpy.uint8) In : a[:,1:].view(numpy.uint16) ValueError: new type not compatible with array. In : a[:,:-1].view(numpy.uint16) ValueError: new type not compatible with array. In : numpy.array(a[:,:-1]).view(numpy.uint16) Out: array([[257, 257, 257, 257], [257, 257, 257, 257], [257, 257, 257, 257]], dtype=uint16) It seems like 'view' won't create a view where the stride length is not a whole multiple of the dtype's itemsize. However, since strides are measured in bytes (right?), this shouldn't be a problem. Is this a minor bug? Or am I woefully misunderstanding the issue involved here? Thanks, Zach From zachary.pincus at yale.edu Thu Apr 24 12:00:24 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Thu, 24 Apr 2008 12:00:24 -0400 Subject: [Numpy-discussion] problem with view() and strided arrays? In-Reply-To: References: Message-ID: <0E96FCDA-DF09-40AE-9C1E-A2EE1FD3F40F@yale.edu> > It seems like 'view' won't create a view where the stride length is > not a whole multiple of the dtype's itemsize. However, since strides > are measured in bytes (right?), this shouldn't be a problem. Actually -- it seems like view() doesn't work with strided arrays at all. (?) In : a = numpy.ones((4,32), dtype=numpy.uint8) In : a.view(numpy.uint16).shape Out: (4, 16) In : a[:,:16].view(numpy.uint16) ValueError: new type not compatible with array. I think this might be a recent regression, because before I updated my numpy installation to the latest SVN version (to check if the bug was fixed!), I'm pretty sure this kind of operation worked. Zach From lists at informa.tiker.net Thu Apr 24 12:04:17 2008 From: lists at informa.tiker.net (Andreas =?utf-8?q?Kl=C3=B6ckner?=) Date: Thu, 24 Apr 2008 12:04:17 -0400 Subject: [Numpy-discussion] numpy as a setuptools dependency Message-ID: <200804241204.18863.lists@informa.tiker.net> Hi all, numpy (1.0.4 anyway) doesn't seem to install right when it's pulled in as a setuptools depdency from a package. If you need an example: http://pypi.python.org/pypi/PyUblas/0.92.3 This is the error message: error: /tmp/easy_install-BhSDYE/numpy-1.0.4/temp/tmpUk0utg/LLgCtQ: No such file or directory Also, numpy doesn't use setuptools and therefore doesn't install an egg-info on Python 2.4, so that often setuptools will wrongly conclude that it's not installed, even if it is. Ideas? Is this something worth fixing for 1.1.0? If so, I'll open a ticket. Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From zachary.pincus at yale.edu Thu Apr 24 12:29:14 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Thu, 24 Apr 2008 12:29:14 -0400 Subject: [Numpy-discussion] "aligned" matrix / ctypes In-Reply-To: References: <3d375d730804231402g7e4bc135hdba82c82d8057055@mail.gmail.com> Message-ID: <3E5D8350-AF60-40C0-9E0F-74A3DBE6935B@yale.edu> Hi all, > It's worth mentioning that there was some discussion of adding an > allocator for aligned arrays. It got sidetracked into a discussion of > SSE support for numpy, but there are all sorts of reasons to want > aligned arrays in numpy, as this post demonstrates. Seeing as it's > just a few lines worth of pure python, is anyone interested in writing > an aligned_empty() for numpy? Here's my attempt at aligned_empty(). It only accepts alignments that are an integer multiple (or divisor) of the dtype's itemsize, because of limitations in ndarray.view(). I've also attached aligned_empty_arbitrary(), which is similar, but allows any bizarro alignment requested. Or at least it would, if ndarray.view() were able to handle strided arrays. It could probably be tightened a bit -- I tried to make the code err on the "demonstrative" side rather than "terse". Here are some basic tests: import aligned_allocator as aa a = aa.aligned_empty((9,7), dtype=numpy.uint8, byte_alignment=64, order='C') for i in range(a.shape[0]): assert(a[i,:].ctypes.data % 64 == 0) a = aa.aligned_empty((9,7), dtype=numpy.float32, byte_alignment=16, order='C') for i in range(a.shape[0]): assert(a[i,:].ctypes.data % 32 == 0) a = aa.aligned_empty((9,7), dtype=numpy.uint16, byte_alignment=32, order='F') for i in range(a.shape[1]): assert(a[:,i].ctypes.data % 32 == 0) Best, Zach -------------- next part -------------- A non-text attachment was scrubbed... Name: aligned_allocator.py Type: text/x-python-script Size: 2554 bytes Desc: not available URL: From rshepard at appl-ecosys.com Thu Apr 24 13:01:47 2008 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Thu, 24 Apr 2008 10:01:47 -0700 (PDT) Subject: [Numpy-discussion] Using normal() Message-ID: Rereading "Guide to NumPy" once again, I saw what I had missed all the previous times: the normal() distribution function (Chapter 10, page 173). I have several questions on using it in my application. The syntax is normal(loc=0.0, scale=1.0, size=None), but I've not seen what those represent, nor how to properly invoke this function. A clue will be much appreciated. I want to call normal() passing at least the width of the curve(at the end points where y=0.0), and the center (where y=1.0). Being able to specify the y value of the inflection point (by default 0.5) would also be helpful. In the application's function I now have: from numpy import * x = nx.arange(0, 100, 0.1) y = nx.normal(center,width) # passed to the function when called and I then pass x,y to PyX for plotting. But python responds with: y = nx.normal(center,width) AttributeError: 'module' object has no attribute 'normal' Pointers to what I should be doing are needed. TIA, Rich -- Richard B. Shepard, Ph.D. | Integrity Credibility Applied Ecosystem Services, Inc. | Innovation Voice: 503-667-4517 Fax: 503-667-8863 From oliphant at enthought.com Thu Apr 24 13:07:24 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 24 Apr 2008 12:07:24 -0500 Subject: [Numpy-discussion] numpy release In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> <9457e7c80804230621i5b53b0aaj9882a5d790939a3b@mail.gmail.com> <9457e7c80804231046w3a5a1b4dm41b546403f1c6e6f@mail.gmail.com> <91cf711d0804231320i64401ab3jdf0f16ec6dc66606@mail.gmail.com> Message-ID: <4810BE4C.8080508@enthought.com> Pauli Virtanen wrote: > > C) Create a new parameter with more sensible behavior and a name > different from "bins", and deprecate (at least giving sequences to) the > "bins" parameter: put up a DeprecationWarning if the user does this, but > still produce the same results as the old histogram. This way the user > can forward-port her code at leisure. > > D) Or, retain the old behavior (values below lowest bin ignored) and > just fix the docstring and the normed=True bug? (I have a patch doing > this.) > > > So which one (or something else) do we choose for 1.1.0? > > I like either C or D, but prefer C if it can be done before 1.1.0. -Travis From kwgoodman at gmail.com Thu Apr 24 13:16:58 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 24 Apr 2008 10:16:58 -0700 Subject: [Numpy-discussion] Using normal() In-Reply-To: References: Message-ID: On Thu, Apr 24, 2008 at 10:01 AM, Rich Shepard wrote: > In the application's function I now have: > > from numpy import * > > x = nx.arange(0, 100, 0.1) > y = nx.normal(center,width) # passed to the function when called > > and I then pass x,y to PyX for plotting. But python responds with: > > y = nx.normal(center,width) > AttributeError: 'module' object has no attribute 'normal' It's random.normal(loc, scale). But that will give you random x values drawn from the normal distribution, not y values at your x. So you'll have to code your own normal, which will probably look something like norm = 1 / (scale * sqrt(2 * pi)) y = norm * exp(-power((x - loc), 2) / (2 * scale**2)) From rshepard at appl-ecosys.com Thu Apr 24 13:26:12 2008 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Thu, 24 Apr 2008 10:26:12 -0700 (PDT) Subject: [Numpy-discussion] Using normal() In-Reply-To: References: Message-ID: On Thu, 24 Apr 2008, Keith Goodman wrote: > It's random.normal(loc, scale). But that will give you random x values > drawn from the normal distribution, not y values at your x. So you'll have > to code your own normal, which will probably look something like Keith, I overlooked that, thanks. > norm = 1 / (scale * sqrt(2 * pi)) > y = norm * exp(-power((x - loc), 2) / (2 * scale**2)) Can do. So, scale would equate to width and loc to center, yes? It's been so many years since I've dealt intimately with statistics and mathematics that I've gotten quite rusty. (However, I still retain what I need to accomplish a given objective.) Much appreciated, Rich -- Richard B. Shepard, Ph.D. | Integrity Credibility Applied Ecosystem Services, Inc. | Innovation Voice: 503-667-4517 Fax: 503-667-8863 From zachary.pincus at yale.edu Thu Apr 24 13:28:38 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Thu, 24 Apr 2008 13:28:38 -0400 Subject: [Numpy-discussion] Using normal() In-Reply-To: References: Message-ID: > The syntax is normal(loc=0.0, scale=1.0, size=None), but I've not > seen > what those represent, nor how to properly invoke this function. A > clue will > be much appreciated. > > I want to call normal() passing at least the width of the curve(at > the end > points where y=0.0), and the center (where y=1.0). Being able to > specify the > y value of the inflection point (by default 0.5) would also be > helpful. The 'normal dstribution' is the Gaussian function: N(x) = 1/(std*sqrt(2*pi))*exp(-(x-mean)**2/(2*std**2)), where N(x) is the probability density at position x, given a normal distribution characterized by 'mean' and 'std' (standard deviation). http://en.wikipedia.org/wiki/Normal_distribution Now, numpy.random.normal gives random samples distributed according to the above probability density function. The only remaining mystery is how 'loc' and 'scale' -- the parameters of numpy.random.normal -- map to 'mean' and 'standard deviation', which is how a normal distribution is usually parameterized. Fortunately, the documentation reveals this: >>> print numpy.random.normal.__doc__ Normal distribution (mean=loc, stdev=scale). normal(loc=0.0, scale=1.0, size=None) -> random values If you need an alternate parameterization of the normal (e.g. in terms of the y value of the inflection point), just solve that out analytically from the definition of the normal in terms of mean and std. However, it looks like you're trying to plot the normal function, not get random samples. Just evaluate the function (as above) at the x positions: mean, std = (0, 1) x = numpy.linspace(-10, 10, 200) # 200 evenly-spaced points from -10 to 10 y = 1/(std*numpy.sqrt(2*numpy.pi))*numpy.exp(-(x-mean)**2/(2*std**2)) Zach From ellisonbg.net at gmail.com Thu Apr 24 13:30:42 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Thu, 24 Apr 2008 11:30:42 -0600 Subject: [Numpy-discussion] numpy as a setuptools dependency In-Reply-To: <200804241204.18863.lists@informa.tiker.net> References: <200804241204.18863.lists@informa.tiker.net> Message-ID: <6ce0ac130804241030p4f5b42ecq4f1f87ae7dfe39f8@mail.gmail.com> Ooooooh. I am glad someone else is seeing this. I see the same thing when I try to install Twisted 8 pulled in as a setuptools dependency. My guess is that this is a setuptools problem, not a problem with numpy or Twisted. Brian On Thu, Apr 24, 2008 at 10:04 AM, Andreas Kl?ckner wrote: > Hi all, > > numpy (1.0.4 anyway) doesn't seem to install right when it's pulled in as a > setuptools depdency from a package. If you need an example: > > http://pypi.python.org/pypi/PyUblas/0.92.3 > > This is the error message: > > error: /tmp/easy_install-BhSDYE/numpy-1.0.4/temp/tmpUk0utg/LLgCtQ: No such > file or directory > > Also, numpy doesn't use setuptools and therefore doesn't install an egg-info > on Python 2.4, so that often setuptools will wrongly conclude that it's not > installed, even if it is. > > Ideas? Is this something worth fixing for 1.1.0? If so, I'll open a ticket. > > Andreas > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From rshepard at appl-ecosys.com Thu Apr 24 13:38:57 2008 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Thu, 24 Apr 2008 10:38:57 -0700 (PDT) Subject: [Numpy-discussion] Using normal() In-Reply-To: References: Message-ID: On Thu, 24 Apr 2008, Zachary Pincus wrote: > The only remaining mystery is how 'loc' and 'scale' -- the parameters of > numpy.random.normal -- map to 'mean' and 'standard deviation', which is > how a normal distribution is usually parameterized. Fortunately, the > documentation reveals this: > > >>> print numpy.random.normal.__doc__ > Normal distribution (mean=loc, stdev=scale). > > normal(loc=0.0, scale=1.0, size=None) -> random values Zachary, Thank you. I looked in the printed manual, not the built-in docs. > If you need an alternate parameterization of the normal (e.g. in terms > of the y value of the inflection point), just solve that out > analytically from the definition of the normal in terms of mean and std. > > However, it looks like you're trying to plot the normal function, not > get random samples. Just evaluate the function (as above) at the x > positions: > > mean, std = (0, 1) > x = numpy.linspace(-10, 10, 200) # 200 evenly-spaced points from -10 > to 10 > y = 1/(std*numpy.sqrt(2*numpy.pi))*numpy.exp(-(x-mean)**2/(2*std**2)) OK. I'll work with this as well as the arange. Rich -- Richard B. Shepard, Ph.D. | Integrity Credibility Applied Ecosystem Services, Inc. | Innovation Voice: 503-667-4517 Fax: 503-667-8863 From rshepard at appl-ecosys.com Thu Apr 24 13:51:15 2008 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Thu, 24 Apr 2008 10:51:15 -0700 (PDT) Subject: [Numpy-discussion] Using normal() In-Reply-To: References: Message-ID: On Thu, 24 Apr 2008, Keith Goodman wrote: > norm = 1 / (scale * sqrt(2 * pi)) > y = norm * exp(-power((x - loc), 2) / (2 * scale**2)) Hmm-m-m. I don't understand the source of the error: y = norm * exp(-pow((x - loc), 2) / (2 * scale**2)) TypeError: only length-1 arrays can be converted to Python scalars Rich -- Richard B. Shepard, Ph.D. | Integrity Credibility Applied Ecosystem Services, Inc. | Innovation Voice: 503-667-4517 Fax: 503-667-8863 From zachary.pincus at yale.edu Thu Apr 24 13:57:49 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Thu, 24 Apr 2008 13:57:49 -0400 Subject: [Numpy-discussion] Using normal() In-Reply-To: References: Message-ID: >> norm = 1 / (scale * sqrt(2 * pi)) >> y = norm * exp(-power((x - loc), 2) / (2 * scale**2)) > > Hmm-m-m. I don't understand the source of the error: > > y = norm * exp(-pow((x - loc), 2) / (2 * scale**2)) > TypeError: only length-1 arrays can be converted to Python scalars Python's built in pow() and exp() functions can't handle numpy arrays, and thus try (and fail) to convert arrays to scalar values. You want to use numpy.exp and numpy.power (or just the ** operator), to do these operations to numpy arrays elementwise. From kwgoodman at gmail.com Thu Apr 24 13:58:34 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 24 Apr 2008 10:58:34 -0700 Subject: [Numpy-discussion] Using normal() In-Reply-To: References: Message-ID: On Thu, Apr 24, 2008 at 10:51 AM, Rich Shepard wrote: > On Thu, 24 Apr 2008, Keith Goodman wrote: > > > norm = 1 / (scale * sqrt(2 * pi)) > > y = norm * exp(-power((x - loc), 2) / (2 * scale**2)) > > Hmm-m-m. I don't understand the source of the error: > > y = norm * exp(-pow((x - loc), 2) / (2 * scale**2)) > TypeError: only length-1 arrays can be converted to Python scalars I don't understand the error (I don't use arrays). Maybe try evaluating pieces of the formula, like pow(x-loc, 2) etc to see where the error is. It works for me: >> x = arange(0,10) >> scale=1 >> loc=1 >> norm = 1 / (scale * sqrt(2 * pi)) >> y = norm * exp(-power((x - loc), 2) / (2 * scale**2)) >> y array([ 1.46762663e-01, 3.98942280e-01, 1.46762663e-01, 5.39909665e-02, 2.68805194e-03, 1.33830226e-04, 9.01740968e-07, 6.07588285e-09, 5.54048800e-12, 5.05227108e-15]) From lists at informa.tiker.net Thu Apr 24 14:04:38 2008 From: lists at informa.tiker.net (Andreas =?utf-8?q?Kl=C3=B6ckner?=) Date: Thu, 24 Apr 2008 14:04:38 -0400 Subject: [Numpy-discussion] Fwd: Re: numpy as a setuptools dependency Message-ID: <200804241404.40485.lists@informa.tiker.net> Ok, forwarding this to distutils-sig. :) Andreas -------------- next part -------------- An embedded message was scrubbed... From: "Brian Granger" Subject: Re: [Numpy-discussion] numpy as a setuptools dependency Date: Thu, 24 Apr 2008 11:30:42 -0600 Size: 3881 URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From zachary.pincus at yale.edu Thu Apr 24 14:12:54 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Thu, 24 Apr 2008 14:12:54 -0400 Subject: [Numpy-discussion] Using normal() In-Reply-To: References: Message-ID: <234BB53F-D437-4489-830A-BFCA5DD2BA58@yale.edu> > It works for me: > >>> x = arange(0,10) >>> scale=1 >>> loc=1 >>> norm = 1 / (scale * sqrt(2 * pi)) >>> y = norm * exp(-power((x - loc), 2) / (2 * scale**2)) >>> y > > array([ 1.46762663e-01, 3.98942280e-01, 1.46762663e-01, > 5.39909665e-02, 2.68805194e-03, 1.33830226e-04, > 9.01740968e-07, 6.07588285e-09, 5.54048800e-12, > 5.05227108e-15]) I assume you 'from numpy import *'? This is why it works -- because that import causes the python built-in exp() to be replaced (in the current namespace) by numpy.exp(). From Joris.DeRidder at ster.kuleuven.be Thu Apr 24 14:16:48 2008 From: Joris.DeRidder at ster.kuleuven.be (Joris De Ridder) Date: Thu, 24 Apr 2008 20:16:48 +0200 Subject: [Numpy-discussion] Using normal() In-Reply-To: References: Message-ID: <982D2093-1295-42A2-8527-E3B172DA25C6@ster.kuleuven.be> On 24 Apr 2008, at 19:26, Rich Shepard wrote: >> norm = 1 / (scale * sqrt(2 * pi)) >> y = norm * exp(-power((x - loc), 2) / (2 * scale**2)) > > Can do. So, scale would equate to width and loc to center, yes? Scale is half the width between the inflection points, mind the factor of 2. J. Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm From david.huard at gmail.com Thu Apr 24 14:22:41 2008 From: david.huard at gmail.com (David Huard) Date: Thu, 24 Apr 2008 14:22:41 -0400 Subject: [Numpy-discussion] numpy release In-Reply-To: <4810BE4C.8080508@enthought.com> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> <9457e7c80804230621i5b53b0aaj9882a5d790939a3b@mail.gmail.com> <9457e7c80804231046w3a5a1b4dm41b546403f1c6e6f@mail.gmail.com> <91cf711d0804231320i64401ab3jdf0f16ec6dc66606@mail.gmail.com> <4810BE4C.8080508@enthought.com> Message-ID: <91cf711d0804241122t4c2e39f1q957c4c1f691ea4b8@mail.gmail.com> The problem I see with C is that it will break compatibility with the other histogram functions, which also use bins. So here is suggestion E: The most common use case ( I think) is the following: h, b = histogram(r, number_of_bins, normed=True/False) for which the function behaves correctly. Assuming we want the next version to : ignore values outside of range and accept and return the bin edges instead of the left edges, here could be the new signature for 1.1: h, edges = histogram(a, bins=10, normed=False, range=None, normed=False, new=False) If new=False, return the histogram and the left edges, with a warning that in the next version, the edges will be returned. If new=True, return the histogram and the edges. If range is given explicitly , raise a warning saying that in the next version, the outliers will be ignored. To ignore outliers, use new=True. If bins is a sequence, raise an error saying that bins should be an integer. To use explicit edges, use new=True. In 1.2, set new=True as the default, and in 2.3, remove new altogether. David 2008/4/24 Travis E. Oliphant : > Pauli Virtanen wrote: > > > > C) Create a new parameter with more sensible behavior and a name > > different from "bins", and deprecate (at least giving sequences to) the > > "bins" parameter: put up a DeprecationWarning if the user does this, but > > still produce the same results as the old histogram. This way the user > > can forward-port her code at leisure. > > > > > > D) Or, retain the old behavior (values below lowest bin ignored) and > > just fix the docstring and the normed=True bug? (I have a patch doing > > this.) > > > > > > So which one (or something else) do we choose for 1.1.0? > > > > > I like either C or D, but prefer C if it can be done before 1.1.0. > > -Travis > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Apr 24 14:23:01 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 24 Apr 2008 13:23:01 -0500 Subject: [Numpy-discussion] numpy as a setuptools dependency In-Reply-To: <200804241204.18863.lists@informa.tiker.net> References: <200804241204.18863.lists@informa.tiker.net> Message-ID: <3d375d730804241123y1bee92d6tca9ecfaa793df08e@mail.gmail.com> On Thu, Apr 24, 2008 at 11:04 AM, Andreas Kl?ckner wrote: > Hi all, > > numpy (1.0.4 anyway) doesn't seem to install right when it's pulled in as a > setuptools depdency from a package. If you need an example: > > http://pypi.python.org/pypi/PyUblas/0.92.3 > > This is the error message: > > error: /tmp/easy_install-BhSDYE/numpy-1.0.4/temp/tmpUk0utg/LLgCtQ: No such > file or directory The actual error is above this. Please find it and post it. > Also, numpy doesn't use setuptools and therefore doesn't install an egg-info > on Python 2.4, so that often setuptools will wrongly conclude that it's not > installed, even if it is. Use setupegg.py. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From rshepard at appl-ecosys.com Thu Apr 24 14:39:23 2008 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Thu, 24 Apr 2008 11:39:23 -0700 (PDT) Subject: [Numpy-discussion] Using normal() In-Reply-To: <982D2093-1295-42A2-8527-E3B172DA25C6@ster.kuleuven.be> References: <982D2093-1295-42A2-8527-E3B172DA25C6@ster.kuleuven.be> Message-ID: On Thu, 24 Apr 2008, Joris De Ridder wrote: > Scale is half the width between the inflection points, mind the factor of > 2. Joris, So, half of the full width at half the maximum height. Thank you. Rich -- Richard B. Shepard, Ph.D. | Integrity Credibility Applied Ecosystem Services, Inc. | Innovation Voice: 503-667-4517 Fax: 503-667-8863 From rshepard at appl-ecosys.com Thu Apr 24 14:40:11 2008 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Thu, 24 Apr 2008 11:40:11 -0700 (PDT) Subject: [Numpy-discussion] Using normal() In-Reply-To: References: Message-ID: On Thu, 24 Apr 2008, Zachary Pincus wrote: > Python's built in pow() and exp() functions can't handle numpy arrays, and > thus try (and fail) to convert arrays to scalar values. You want to use > numpy.exp and numpy.power (or just the ** operator), to do these > operations to numpy arrays elementwise. Zachary, Ah, now I understand! Thank you for the lessons, Rich -- Richard B. Shepard, Ph.D. | Integrity Credibility Applied Ecosystem Services, Inc. | Innovation Voice: 503-667-4517 Fax: 503-667-8863 From robert.kern at gmail.com Thu Apr 24 14:45:58 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 24 Apr 2008 13:45:58 -0500 Subject: [Numpy-discussion] "aligned" matrix / ctypes In-Reply-To: <3E5D8350-AF60-40C0-9E0F-74A3DBE6935B@yale.edu> References: <3d375d730804231402g7e4bc135hdba82c82d8057055@mail.gmail.com> <3E5D8350-AF60-40C0-9E0F-74A3DBE6935B@yale.edu> Message-ID: <3d375d730804241145w6c55fed5mfc032a0b0a9f3d0f@mail.gmail.com> On Thu, Apr 24, 2008 at 11:29 AM, Zachary Pincus wrote: > Here's my attempt at aligned_empty(). It only accepts alignments that are > an integer multiple (or divisor) of the dtype's itemsize, because of > limitations in ndarray.view(). aligned_empty() works for me (after removing the if: suite that guards this case) because it correctly .view()s before it slices. aligned_empty_arbitrary() doesn't because it slices before it .view()s. In [14]: aligned_allocator.aligned_empty((5,5), float, 3) Out[14]: array([[ -2.00773895e+002, 2.12196342e-314, 3.41914975e-307, 5.47724529e-306, 9.16023223e-310], [ 7.82990492e-317, 1.36379628e-316, 0.00000000e+000, 0.00000000e+000, 1.27361195e-313], [ 4.94065646e-324, 7.00258612e-313, 0.00000000e+000, 0.00000000e+000, 0.00000000e+000], [ 0.00000000e+000, 0.00000000e+000, 0.00000000e+000, 0.00000000e+000, 0.00000000e+000], [ 0.00000000e+000, 0.00000000e+000, 0.00000000e+000, 0.00000000e+000, 0.00000000e+000]]) In [15]: aligned_allocator.aligned_empty_arbitrary((5,5), float, 3) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) /Users/rkern/today/ in () /Users/rkern/today/aligned_allocator.py in aligned_empty_arbitrary(shape, dtype, byte_alignment, order) 53 byte_shape = tuple(other_shape) + (aligned_byte_width,) 54 aligned_view = aligned_buffer.reshape(byte_shape, order='C') ---> 55 requested_view = aligned_view[..., :byte_width].view(dtype) 56 if order == 'F': 57 return requested_view.T ValueError: new type not compatible with array. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From peridot.faceted at gmail.com Thu Apr 24 14:49:12 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 24 Apr 2008 20:49:12 +0200 Subject: [Numpy-discussion] Using normal() In-Reply-To: References: Message-ID: On 24/04/2008, Keith Goodman wrote: > On Thu, Apr 24, 2008 at 10:01 AM, Rich Shepard wrote: > > In the application's function I now have: > > > > from numpy import * > > > > x = nx.arange(0, 100, 0.1) > > y = nx.normal(center,width) # passed to the function when called > > > > and I then pass x,y to PyX for plotting. But python responds with: > > > > y = nx.normal(center,width) > > AttributeError: 'module' object has no attribute 'normal' > > It's random.normal(loc, scale). But that will give you random x values > drawn from the normal distribution, not y values at your x. So you'll > have to code your own normal, which will probably look something like > > norm = 1 / (scale * sqrt(2 * pi)) > y = norm * exp(-power((x - loc), 2) / (2 * scale**2)) I really have to recommend you instead install scipy, which contains this as well as many other probability distributions: D = scipy.stats.norm(0, 1) Then D.pdf(x) gives you the probability density function at x, D.cdf(x) gives you the probability that the function value is less than x, D.icdf(y) gives you the x value for which the probability is y that the value will be less than x, and so on. This is also available for more probability distributions than I had ever heard of. Anne From rshepard at appl-ecosys.com Thu Apr 24 14:49:09 2008 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Thu, 24 Apr 2008 11:49:09 -0700 (PDT) Subject: [Numpy-discussion] Using normal() In-Reply-To: <234BB53F-D437-4489-830A-BFCA5DD2BA58@yale.edu> References: <234BB53F-D437-4489-830A-BFCA5DD2BA58@yale.edu> Message-ID: On Thu, 24 Apr 2008, Zachary Pincus wrote: > I assume you 'from numpy import *'? This is why it works -- because that > import causes the python built-in exp() to be replaced (in the current > namespace) by numpy.exp(). The equivalent (I think): import numpy as nx. Rich -- Richard B. Shepard, Ph.D. | Integrity Credibility Applied Ecosystem Services, Inc. | Innovation Voice: 503-667-4517 Fax: 503-667-8863 From robert.kern at gmail.com Thu Apr 24 14:51:49 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 24 Apr 2008 13:51:49 -0500 Subject: [Numpy-discussion] Using normal() In-Reply-To: References: <982D2093-1295-42A2-8527-E3B172DA25C6@ster.kuleuven.be> Message-ID: <3d375d730804241151q5091d72ewdf24689080be3190@mail.gmail.com> On Thu, Apr 24, 2008 at 1:39 PM, Rich Shepard wrote: > On Thu, 24 Apr 2008, Joris De Ridder wrote: > > > Scale is half the width between the inflection points, mind the factor of > > 2. > > Joris, > > So, half of the full width at half the maximum height. Thank you. No, it's not! scale is the standard deviation of the Normal distribution, not half the FWHM! -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From rshepard at appl-ecosys.com Thu Apr 24 14:57:40 2008 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Thu, 24 Apr 2008 11:57:40 -0700 (PDT) Subject: [Numpy-discussion] Using normal() In-Reply-To: <3d375d730804241151q5091d72ewdf24689080be3190@mail.gmail.com> References: <982D2093-1295-42A2-8527-E3B172DA25C6@ster.kuleuven.be> <3d375d730804241151q5091d72ewdf24689080be3190@mail.gmail.com> Message-ID: On Thu, 24 Apr 2008, Robert Kern wrote: > No, it's not! scale is the standard deviation of the Normal distribution, > not half the FWHM! OK. Thanks for clearing up my mis-understanding. Rich -- Richard B. Shepard, Ph.D. | Integrity Credibility Applied Ecosystem Services, Inc. | Innovation Voice: 503-667-4517 Fax: 503-667-8863 From millman at berkeley.edu Thu Apr 24 15:09:54 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Thu, 24 Apr 2008 14:09:54 -0500 Subject: [Numpy-discussion] numpy release In-Reply-To: <91cf711d0804241122t4c2e39f1q957c4c1f691ea4b8@mail.gmail.com> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> <9457e7c80804230621i5b53b0aaj9882a5d790939a3b@mail.gmail.com> <9457e7c80804231046w3a5a1b4dm41b546403f1c6e6f@mail.gmail.com> <91cf711d0804231320i64401ab3jdf0f16ec6dc66606@mail.gmail.com> <4810BE4C.8080508@enthought.com> <91cf711d0804241122t4c2e39f1q957c4c1f691ea4b8@mail.gmail.com> Message-ID: On Thu, Apr 24, 2008 at 1:22 PM, David Huard wrote: > Assuming we want the next version to : ignore values outside of range and > accept and return the bin edges instead of the left edges, here could be the > new signature for 1.1: > h, edges = histogram(a, bins=10, normed=False, range=None, normed=False, > new=False) > > If new=False, return the histogram and the left edges, with a warning that > in the next version, the edges will be returned. If new=True, return the > histogram and the edges. > If range is given explicitly , raise a warning saying that in the next > version, the outliers will be ignored. To ignore outliers, use new=True. > If bins is a sequence, raise an error saying that bins should be an > integer. To use explicit edges, use new=True. > > In 1.2, set new=True as the default, and in 2.3, remove new altogether. +1 That sounds fine to me assuming 2.3 is 1.3. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From shao at msg.ucsf.edu Thu Apr 24 15:24:35 2008 From: shao at msg.ucsf.edu (Lin Shao) Date: Thu, 24 Apr 2008 12:24:35 -0700 Subject: [Numpy-discussion] EPD PIL _imaging C module Message-ID: Running scipy.test() in the latest EPD windows distribution, I got the following exception: ====================================================================== ERROR: test_imresize (scipy.misc.tests.test_pilutil.test_pilutil) ---------------------------------------------------------------------- Traceback (most recent call last): File "E:\Python25\lib\site-packages\scipy-0.6.0.0003-py2.5-win32.egg\scipy\misc\tests\test_pilutil.py", line 17, in test_imresize im1 = pilutil.imresize(im,T(1.1)) File "E:\Python25\lib\site-packages\scipy-0.6.0.0003-py2.5-win32.egg\scipy\misc\pilutil.py", line 225, in imresize im = toimage(arr) File "E:\Python25\lib\site-packages\scipy-0.6.0.0003-py2.5-win32.egg\scipy\misc\pilutil.py", line 104, in toimage image = Image.fromstring('L',shape,bytedata.tostring()) File "E:\Python25\lib\site-packages\PIL-1.1.6.0003_s-py2.5-win32.egg\PIL\Image.py", line 1743, in fromstring im = new(mode, size) File "E:\Python25\lib\site-packages\PIL-1.1.6.0003_s-py2.5-win32.egg\PIL\Image.py", line 1710, in new return Image()._new(core.fill(mode, size, color)) File "E:\Python25\lib\site-packages\PIL-1.1.6.0003_s-py2.5-win32.egg\PIL\Image.py", line 36, in __getattr__ raise ImportError("The _imaging C module is not installed") ImportError: The _imaging C module is not installed It's a bit strange because I do see _imging.pyd in the PIL folder. I'm using XP 32-bit. -lin -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Thu Apr 24 15:25:22 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 24 Apr 2008 12:25:22 -0700 Subject: [Numpy-discussion] numpy release In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> <9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com> <480FB04F.80006@noaa.gov> Message-ID: <4810DEA2.60501@noaa.gov> Alan G Isaac wrote: > the > cost of complexity should be justified by a gain in > functionality. I don't think functionality is the right word here. the Matrix class(es) is all about clean, convenient API, i.e. style, not functionality -- we have all the functionality already, indeed we have it with plain old arrays, so I think that's really beside the point. I just like the "feel" of Row and Column Vectors. But this brings me back to: We need examples of doing linear algebra operations, not just iterating operations. Frankly, if you have a matrix, and you want to iterate through all the rows, and you want to index into them as 1-d arrays, why they heck don't you just: for r in MyMatrix.A: .... The answer (I think), is that folks have simply been confused and annoyed by the fact that they need that extra ".A", which brings us back to style. One extra functionally: you can index a ColumnVector with a scalar: col = M[:,i] col[i] You can't do that now. This brings me to: > the people who write the code (for numpy) don't actually use matrices, > they use the arrays I don't really write numpy code, but I don't really use arrays either. Which, I suppose, means I should just shut up. But maybe I would if the API was better. So we're back to: who does want matrices? is it only newbies coming from Matlab, where they were used to them? For once, we do have someone that is writing the code that wants improvements, so I guess that's why this discussion has been revived. Timothy Hochberg wrote: > 1. The matrices and arrays should become more alike if possible I'm not sure I agree -- the more alike they are, the less point there is to having them. > should share more of the same code base. Don't they share almost all the same code already? My understanding is that they are arrays with a couple operators overloaded -- how much more code could they share? Nevertheless, I'm interested to see what Tim comes up with. > 2. This doesn't support the concept of arrays of matrices/vectors, > which is one thing I've always wanted out of the matrix class. For > example it would be nice to be able to spell: 'A' is a 1D array of > matrices (3D) overall, 'B' is a 1D array of vectors (3D) overall, You could do this now with a 1-d object array, with a bunch of matrices in it. > matrix multiply them together, yielding a 1D array of matrices > (3D) overall. But this, or course, would fail. However, if we expand this scenario to A and B being 1-d arrays of other arrays that may not all be the same size (and thus you couldn't just use a 3-d array), and be able to run various ufuncs on them, that would be cool! Bill Spotz wrote: >> The fact that there is another way to get a row vector: M[i] is a >> bonus. > > Except that the behavior of M[i] is one of the driving issues of the > conversation. well, yes, but a bit of a red herring, I think -- it's syntactic sugar. The issue, I think, is that even if you do: r = M[:,i] you can't then index that with a single index -- it's a matrix, NOT a 1-d array. Again, I proposed disallowing M[i] altogether, as it is ambiguous, but that really is gratuitous consistency. >> A vector is a 1-d array with some functionality overloaded to make it >> convenient to do linear algebra. > > Again, I would argue for Vectors inheriting from Matrix. I would make > the case based on code reuse and elegance, although these might be > overridden by some other concern. I think that's a bit of an implementation detail -- we need to define behavior that we want, and decide how best to implement that. i.e.: RowVector[i] -> scalar RowVector.A -> 1-d array (same for column) How do we want it pretty-printed? We need to focus on column vectors here -- It's easy to say: a row vector is a 1-d array -- that's easy and clean. It's column vectors that are tricky -- indeed they are a pain with regular old arrays. What about > np.Matrix() > np.Matrix() I'd say you get a matrix from each of those, but they would be transposes of each-other -- another good reason for the Vector classes. though maybe those should be spelled: np.row_stack and np.column_stack > I have generally thought about this in the context of, say, a > Krylov-space iterative method, and what that type of interface would > lead to the most readable code. Can you whip up a small example, starting with the current implementation? Bruce Southey wrote: > Hi, > I would like to use the matrix functionality but I, like others, get by > without it. What does it lack that keeps you from using it? That's the key question? > +1 to Tim's and Nadav's comments. As Tim said, there should be seamless > integration between concepts of vectors and matrices - after all there > really no distinction between them in linear algebra. This is all about being able to index a "vector" with a single index -- that's it, I think. Matlab handles this by special casing matrices that have a single dimension of one. It also has an easy way to spell a literal for column vector, though I suppose: np.Matrix((1,2,3,4,5)).T isn't so bad. > -2 for having rowvector and columnvector - both numpy and I should know > the orientation, if not, I deserve garbage or an error. But HOW can you know the orientation? a 1-d array has no orientation -- which is why the current version always returns a matrix when indexing. > 0 for vector - I don't see the need for it as a unique entity but > understand the simplicity of saying vector. I don't think we can have vector without ColumnVector, though I suppose a "Vector" can be either a (n,1) or a (1,n) matrix that allows single indexing. By the way, maybe a column vector could be handy for doing array broadcasting, as an cleaner way to do: x = np.arange(10) y = np.arange(20).reshape((-1,1)) z = FunctionOfXandY(x,y) Final note: If no one has an linear algebra examples, then we're wasting out time with even this cheap talk. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From aisaac at american.edu Thu Apr 24 16:24:26 2008 From: aisaac at american.edu (Alan G Isaac) Date: Thu, 24 Apr 2008 16:24:26 -0400 Subject: [Numpy-discussion] numpy release In-Reply-To: <4810DEA2.60501@noaa.gov> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com><85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com><9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com><480FB04F.80006@noaa.gov> <4810DEA2.60501@noaa.gov> Message-ID: On Thu, 24 Apr 2008, Christopher Barker apparently wrote: > for r in MyMatrix.A: Of course: this possibility is mentioned explicitly in several of my posts. There are a few different answers to why asking users to do this annoying: 1. It is a deviation from the behavior of 2d arrays that does not seem to "buy" anything, so it pointlessly breaks a natural expectation. (But maybe some will show that it does buy a benefit.) So it breaks the rule: stick with the behavior of 2d arrays except for *, **, and nonscalar indexing, which is a pretty understandable rule. To this extent I agree with Tim that matrices and arrays should be as alike as it is feasible to make them, while still getting the convenience of *, **, and submatrices via nonscalar indexing. 2. Whenever you do not know ahead of time whether you will pass an array or a matrix, you must bear the (small but annoying) cost of working around your ignorance. 3. In my own experience, whenever I want to iterate over a matrix, I want the 1d arrays. Others may have different experiences, but they have not shared them. Unless we find users who actually want the vectors you want to provide them with, you seem to be promoting this approach based on fairly abstract design considerations. (Which is not necessarily bad of course.) 4. If users do want to iterate over rows and columns, it is more natural (i.e., symmetrical) to provide these as attributes of the matrix. I do not mean for any of this to be determinative. Indeed, against this are two possible arguments. 1. For matrices and sparse matrices to nicely behave alike, your proposal will be necessary. (This case has not been made yet, however.) 2. I think Stefan might argue something like, to leave the "linear algebra world", you should have to be quite explicit, as x.A is. I find it explict enough if we use a matrix as an iterator---to me that means indeed I am leaving the matrices in favor of arrays. Cheers, Alan Isaac From zachary.pincus at yale.edu Thu Apr 24 17:57:22 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Thu, 24 Apr 2008 17:57:22 -0400 Subject: [Numpy-discussion] "aligned" matrix / ctypes In-Reply-To: <3d375d730804241145w6c55fed5mfc032a0b0a9f3d0f@mail.gmail.com> References: <3d375d730804231402g7e4bc135hdba82c82d8057055@mail.gmail.com> <3E5D8350-AF60-40C0-9E0F-74A3DBE6935B@yale.edu> <3d375d730804241145w6c55fed5mfc032a0b0a9f3d0f@mail.gmail.com> Message-ID: Zach: >> Here's my attempt at aligned_empty(). It only accepts alignments >> that are >> an integer multiple (or divisor) of the dtype's itemsize, because of >> limitations in ndarray.view(). Robert: > aligned_empty() works for me (after removing the if: suite that guards > this case) because it correctly .view()s before it slices. > aligned_empty_arbitrary() doesn't because it slices before it > .view()s. As I noted, aligned_empty_arbitrary() does not currently work because ndarray.view() cannot deal with strided arrays, which (as per another email thread) I think may be a bug. If it could, then aligned_empty_arbitrary() would work, and not raise an exception. However, the truth is that aligned_empty does not work with arbitrary alignments! If you remove the 'if' guard, various assumptions break, due to the floor division here: element_alignment = byte_alignment / dtype.itemsize which only works right when byte_alignment is an integral multiple of itemsize. Otherwise you get: In [1]: import aligned_allocator In [2]: a = aligned_allocator.aligned_empty((5,5), float, 3) In [3]: a[0,:].ctypes.data % 3 Out[3]: 0 In [4]: a[1,:].ctypes.data % 3 Out[4]: 1 In [5]: a[2,:].ctypes.data % 3 Out[5]: 2 Which is to say, the data isn't aligned right. The reason that one must slice before .view()ing to allow arbitrary alignment is as follows. Imagine that we want rows of four 2-byte shorts aligned to 3-byte boundaries. (Assume that we already have a buffer that starts on a 3-byte boundary.) So we need an array that's 9 bytes wide by however many rows, and then we just want to use the first eight bytes of row. If we slice first, we can get a strided array that is eight bytes wide, and thus something that we can interpret as four shorts. (That is, if .view() could handle strided arrays.) On the other hand, there's absolutely no way that we can .view() before slicing, because our underlying array is 9 bytes wide, and you can't look at 9 bytes as any integral number of 2-byte shorts. So .view() should properly fail, and thus we can't get to the slicing. Fortunately this is all an obtuse corner case, since who ever needs to align to values that aren't either an integral multiple or divisor of the itemsize? Nevertheless, anyone who wants to use the aligned_empty() code should keep the 'if' guard there. In the mean time, is there any way to (from within python) construct an array from another array by specifying the dtype and strides? That is, some other way to do the view/slice business directly? Thanks, Zach From robert.kern at gmail.com Thu Apr 24 18:08:14 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 24 Apr 2008 17:08:14 -0500 Subject: [Numpy-discussion] "aligned" matrix / ctypes In-Reply-To: References: <3d375d730804231402g7e4bc135hdba82c82d8057055@mail.gmail.com> <3E5D8350-AF60-40C0-9E0F-74A3DBE6935B@yale.edu> <3d375d730804241145w6c55fed5mfc032a0b0a9f3d0f@mail.gmail.com> Message-ID: <3d375d730804241508v15b53f03gf44a1e6c20fba081@mail.gmail.com> On Thu, Apr 24, 2008 at 4:57 PM, Zachary Pincus wrote: > The reason that one must slice before .view()ing to allow arbitrary > alignment is as follows. Imagine that we want rows of four 2-byte > shorts aligned to 3-byte boundaries. (Assume that we already have a > buffer that starts on a 3-byte boundary.) So we need an array that's 9 > bytes wide by however many rows, and then we just want to use the > first eight bytes of row. If we slice first, we can get a strided > array that is eight bytes wide, and thus something that we can > interpret as four shorts. (That is, if .view() could handle strided > arrays.) > > On the other hand, there's absolutely no way that we can .view() > before slicing, because our underlying array is 9 bytes wide, and you > can't look at 9 bytes as any integral number of 2-byte shorts. > So .view() should properly fail, and thus we can't get to the slicing. Yes, you are right, sorry. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From peridot.faceted at gmail.com Thu Apr 24 18:09:08 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 24 Apr 2008 18:09:08 -0400 Subject: [Numpy-discussion] "aligned" matrix / ctypes In-Reply-To: References: <3d375d730804231402g7e4bc135hdba82c82d8057055@mail.gmail.com> <3E5D8350-AF60-40C0-9E0F-74A3DBE6935B@yale.edu> <3d375d730804241145w6c55fed5mfc032a0b0a9f3d0f@mail.gmail.com> Message-ID: On 24/04/2008, Zachary Pincus wrote: > In the mean time, is there any way to (from within python) construct > an array from another array by specifying the dtype and strides? That > is, some other way to do the view/slice business directly? There's the ndarray constructor, which lets you do some highly dodgy things like create arrays whose rows overlap in memory. I would think it would be possible to do this with that. You might also want to think about allowing arrays to have their rows aligned (for example, a 100 by 100 array of float32s which wants 64-byte alignment for every row). Anne From aisaac at american.edu Thu Apr 24 18:18:43 2008 From: aisaac at american.edu (Alan G Isaac) Date: Thu, 24 Apr 2008 18:18:43 -0400 Subject: [Numpy-discussion] numpy release In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com><85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com><9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com><480FB04F.80006@noaa.gov> Message-ID: On Wed, 23 Apr 2008, Timothy Hochberg apparently wrote: > I think the way to go is to probably add some meta > information I have added this as Proposal 4 at Forgive anything said the misrepresents your intent. Cheers, Alan From eike.welk at gmx.net Thu Apr 24 18:21:37 2008 From: eike.welk at gmx.net (Eike Welk) Date: Fri, 25 Apr 2008 00:21:37 +0200 Subject: [Numpy-discussion] ndarray.__mul__ is too gready? In-Reply-To: <54727.85.166.31.187.1201086688.squirrel@cens.ioc.ee> References: <54727.85.166.31.187.1201086688.squirrel@cens.ioc.ee> Message-ID: <200804250021.39692.eike.welk@gmx.net> Just for the record, because I had a very similar problem: On Wednesday 23 January 2008 12:11, Pearu Peterson wrote: > Users may accidentally hit vector * matrix and this should raise > an exception. But when vector is numpy array, then ndarray.__mul__ > calls element * A for each array element and here A.__rmul__ > gets only the types of array elements (that are floats, for > instance) as rhs and hence is not able to decide that it should > return NotImplemented. Instead, it computes scalar * matrix and > the final result will be > array([element1*matrix, element2*matrix, ..]) > which is not desired already because of the memory usage and > should have raised an exception instead. > > Here is a small example illustrating the problem: > > class A: > def __rmul__(self, other): > if isinstance(other, ndarray): > return NotImplemented > if isinstance(other, (int, float)): > return self # pretend other==1 for an example > return NotImplemented > > def __repr__(self): return 'A()' > > >>> array([1,2,3]) * A() > > array([A(), A(), A()], dtype=object) > # expected TypeError > > Is there any workaround to this problem? > In other situations/applications the above behavior of > ndarray.__mul__ is a feature but is there any way to > avoid this feature? > The class attribute __array_priority__ seems to control what exactly happens when those special functions are called. It is said to be documented in Travis Oliphant's Numpy book. Writing __array_priority__ = 10.0 should do what you want. So, the class definition should look like this: class A(object): __array_priority__ = 10.0 def __rmul__(self, other): ... ... > Thanks, > Pearu Kind regards, Eike. From aisaac at american.edu Thu Apr 24 18:29:44 2008 From: aisaac at american.edu (Alan G Isaac) Date: Thu, 24 Apr 2008 18:29:44 -0400 Subject: [Numpy-discussion] numpy release In-Reply-To: <4810DEA2.60501@noaa.gov> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com><85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com><9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com><480FB04F.80006@noaa.gov> <4810DEA2.60501@noaa.gov> Message-ID: On Thu, 24 Apr 2008, Christopher Barker apparently wrote: > I suppose a "Vector" can be either a (n,1) or a (1,n) > matrix that allows single indexing. This bothers me. So, if X is 2 by 2, then X[0] will be a row vector. But if X is 1 by 2, then X[0] will be a scalar? Ouch! Bye bye generic code. Or are you still having separate classes and not just changing ``__getitem__``? OK, I see that is most likely what you meant. Cheers, Alan From charlesr.harris at gmail.com Thu Apr 24 18:37:37 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 24 Apr 2008 16:37:37 -0600 Subject: [Numpy-discussion] Fixing #736 and possible memory leak Message-ID: Hi, I've been looking into ticket #736 and playing with some things. In arrayobject.c starting at line 8534 I added a check for strings. if (PyString_Check(op)) { r = Array_FromPyScalar(op, newtype); } if (PySequence_Check(op)) { PyObject *thiserr = NULL; /* necessary but not sufficient */ Py_INCREF(newtype); r = Array_FromSequence(op, newtype, flags & FORTRAN, min_depth, max_depth); if (r == NULL && (thiserr=PyErr_Occurred())) { if (PyErr_GivenExceptionMatches(thiserr, PyExc_MemoryError)) { return NULL; } I think there may be a failure to decrement the reference to newtype unless Array_FromSequence does that (nasty side effect); Anyway, the added check for a string fixes the conversion problem for such things as int32('123'). There remains a problem with array('123', dtype=int32) and with array(['123','123'], dtype=int32), but I think I can track those down. The question is, will changing the current behavior so that strings get converted to numbers cause problems with other programs out there. I suspect I also need to check that strings are converted this way only when the type is explicitly given, not detected. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From rshepard at appl-ecosys.com Thu Apr 24 19:15:58 2008 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Thu, 24 Apr 2008 16:15:58 -0700 (PDT) Subject: [Numpy-discussion] Generating Bell Curves (was: Using normal() ) Message-ID: Thanks to several of you I produced test code using the normal density function, and it does not do what we need. Neither does the Gaussian function using fwhm that I've tried. The latter comes closer, but the ends do not reach y=0 when the inflection point is y=0.5. So, let me ask the collective expertise here how to generate the curves that we need. We need to generate bell-shaped curves given a midpoint, width (where y=0) and inflection point (by default, y=0.5) where y is [0.0, 1.0], and x is usually [0, 100], but can vary. Using the NumPy arange() function to produce the x values (e.g, arange(0, 100, 0.1)), I need a function that will produce the associated y values for a bell-shaped curve. These curves represent the membership functions for fuzzy term sets, and generally adjacent curves overlap where y=0.5. It would be a bonus to be able to adjust the skew and kurtosis of the curves, but the controlling data would be the center/midpoint and width, with defaults for inflection point, and other parameters. I've been searching for quite some time without finding a solution that works as we need it to work. TIA, Rich -- Richard B. Shepard, Ph.D. | Integrity Credibility Applied Ecosystem Services, Inc. | Innovation Voice: 503-667-4517 Fax: 503-667-8863 From stefan at sun.ac.za Thu Apr 24 19:23:11 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 25 Apr 2008 01:23:11 +0200 Subject: [Numpy-discussion] numpy release In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com> <480FB04F.80006@noaa.gov> <4810DEA2.60501@noaa.gov> Message-ID: <9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com> 2008/4/25 Alan G Isaac : > So, if X is 2 by 2, then X[0] will be a row vector. > But if X is 1 by 2, then X[0] will be a scalar? > Ouch! > Bye bye generic code. Yup. That's the current state of things. St?fan From tim.hochberg at ieee.org Thu Apr 24 19:45:21 2008 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Thu, 24 Apr 2008 16:45:21 -0700 Subject: [Numpy-discussion] Matrix Class [was numpy release] Message-ID: Chris.Barker wrote: > Alan G Isaac wrote: > > the cost of complexity should be justified by a gain in functionality. > > I don't think functionality is the right word here. the Matrix class(es) > is all about clean, convenient API, i.e. style, not functionality -- we > have all the functionality already, indeed we have it with plain old > arrays, so I think that's really beside the point. Not entirely, there's no good way do deal with arrays of matrices at present. This could be fixed by tweaking dot, but it could also be part of a reform of the matrix class. [CHOP] > > Timothy Hochberg wrote: > > 1. The matrices and arrays should become more alike if possible > > I'm not sure I agree -- the more alike they are, the less point there is > to having them. That's the best possible outcome. If some solution can be reached that naturally supports enough matrix operations on array, without significantly complexifying array, to satisfy the matrix users then that would be great. Less stuff to maintain, less stuff to learn, etc, etc. With that in mind, what is minimum amount of stuff that matrix should support: 1. Indexing along a single axis should produce a row or column vector as appropriate. 2. '*' should mean matrix multiplication. 3. '**' should mean matrix exponentiation. I suspect that this is less crucial than 2, but no more difficult. There's some other stuff that's less critical IMO (.H and .I for example) but feel free to yell at me if you think I'm mistaken. There's some other properties that a fix should have as well, in my opinion and in some cases in others opinions as well. 1. A single index into an array should yield a 1D object just as indexing an array does. This does not have to inconsistent with #1 above; Chris Barker proposed one solution. I'm not sold on the details of that solution, but I approve of the direction that it points to. [In general A[i][j] should equal A[i,j]. I know that fancy indexing breaks this; that was a mistake IMO, but that's a discussion for another day]. 2. It should be possible to embed both vectors and matrices naturally into arrays. This would allow natural and efficient implementation of, for example, rotating a 1000 3D vectors. One could spell that R * V, where R is a 2x2 matrix and V is a 1000x3 array, where the last axis is understood to be a vector. 3. I'm pretty certain there's a third thing I wanted to mention but it escapes me. It'll resurface at the most inopportune time.... Let's play with Chris Barker's RowVector and ColVector proposal for a moment. Let's suppose that we have four four classes: Scalar, RowVector, ColVector and Matrix. They're all pretty much the same as today's array class except that: 1. They are displayed a little differently so that you can tell them apart. 2. __mul__ and __pow__ are treated differently Let's consider __mul__: when a RowVector multiplied with ColVector, the dot product is computed. If, for example, the arrays are both 2D, the they are treated as arrays of vectors and the dot product of each pair of vectors is computed in turn: I think broadcasting should work correctly, but I haven't thought that all the way through yet. Ignoring broadcasting for a moment, the rules are: 1. (n)D Scalar * (n)D Scalar => (n)D Scalar (elementwise) 2. (n)D RowVector * (n)D ColVector => (n-1)D Scalar (dot product) 3. (n+1)D Matrix * (n)D ColVector => (n)D ColVector (matrix-vector product) 4. (n)D Matrix * n(D) Matrix => (n)D Matrix (matrix) product Other combinations would be an error. In principal you could do dyadic products, but I suspect we'd be better off using a separate function for that since most of the times that would just indicate a mistake. Note that in this example Scalar is playing the role of the present day array, which I've assumed has magically vanished from the scene somehow. This looks like it gets of most the way towards where we want to be. There are some issues though. For example, all of the sudden you want to different transpose operators; one that transposes matrices and flips colVectors to RowVectors and leaves Scalars alone, and another that manipulates the rest of the array structure. There is probably other stuff like that too, it doesn't look insurmountable to me right now, however. Still, I'm not sold on the whole RowVector/ColVector/Matrix approach. I have a gut feeling that we'd be better off with a single array class that was somehow tagged to indicate it's type. Also, I keep hoping to come up with an elegant generalization to this that turns arrays into quasi-tensor objects where all the matrix behavior falls out naturally. Sadly, attempts in this direction keep collapsing under there own weight but I'm still thinking about it. However, I do think that the RowVector/ColVector/Matrix approach is a step in the right direction and is certainly worth discussing to see where it leads. -tim > > > should share more of the same code base. > > Don't they share almost all the same code already? My understanding is > that they are arrays with a couple operators overloaded -- how much more > code could they share? > > Nevertheless, I'm interested to see what Tim comes up with. > > > 2. This doesn't support the concept of arrays of matrices/vectors, > > which is one thing I've always wanted out of the matrix class. For > > example it would be nice to be able to spell: 'A' is a 1D array of > > matrices (3D) overall, 'B' is a 1D array of vectors (3D) overall, > > You could do this now with a 1-d object array, with a bunch of matrices > in it. > > > matrix multiply them together, yielding a 1D array of matrices > > (3D) overall. > > But this, or course, would fail. However, if we expand this scenario to > A and B being 1-d arrays of other arrays that may not all be the same > size (and thus you couldn't just use a 3-d array), and be able to run > various ufuncs on them, that would be cool! > > Bill Spotz wrote: > >> The fact that there is another way to get a row vector: M[i] is a > >> bonus. > > > > Except that the behavior of M[i] is one of the driving issues of the > > conversation. > > well, yes, but a bit of a red herring, I think -- it's syntactic sugar. > The issue, I think, is that even if you do: > > r = M[:,i] > > you can't then index that with a single index -- it's a matrix, NOT a > 1-d array. Again, I proposed disallowing M[i] altogether, as it is > ambiguous, but that really is gratuitous consistency. > > >> A vector is a 1-d array with some functionality overloaded to make it > >> convenient to do linear algebra. > > > > Again, I would argue for Vectors inheriting from Matrix. I would make > > the case based on code reuse and elegance, although these might be > > overridden by some other concern. > > I think that's a bit of an implementation detail -- we need to define > behavior that we want, and decide how best to implement that. i.e.: > > RowVector[i] -> scalar > RowVector.A -> 1-d array > (same for column) > > How do we want it pretty-printed? > > We need to focus on column vectors here -- It's easy to say: a row > vector is a 1-d array -- that's easy and clean. It's column vectors that > are tricky -- indeed they are a pain with regular old arrays. > > What about > > np.Matrix() > > np.Matrix() > > I'd say you get a matrix from each of those, but they would be > transposes of each-other -- another good reason for the Vector classes. > though maybe those should be spelled: > > np.row_stack and np.column_stack > > > I have generally thought about this in the context of, say, a > > Krylov-space iterative method, and what that type of interface would > > lead to the most readable code. > > Can you whip up a small example, starting with the current implementation? > > Bruce Southey wrote: > > Hi, > > I would like to use the matrix functionality but I, like others, get by > > without it. > > What does it lack that keeps you from using it? That's the key question? > > > +1 to Tim's and Nadav's comments. As Tim said, there should be seamless > > integration between concepts of vectors and matrices - after all there > > really no distinction between them in linear algebra. > > This is all about being able to index a "vector" with a single index -- > that's it, I think. Matlab handles this by special casing matrices that > have a single dimension of one. It also has an easy way to spell a > literal for column vector, though I suppose: > > np.Matrix((1,2,3,4,5)).T > > isn't so bad. > > > -2 for having rowvector and columnvector - both numpy and I should know > > the orientation, if not, I deserve garbage or an error. > > But HOW can you know the orientation? a 1-d array has no orientation -- > which is why the current version always returns a matrix when indexing. > > > 0 for vector - I don't see the need for it as a unique entity but > > understand the simplicity of saying vector. > > I don't think we can have vector without ColumnVector, though I suppose > a "Vector" can be either a (n,1) or a (1,n) matrix that allows single > indexing. > > By the way, maybe a column vector could be handy for doing array > broadcasting, as an cleaner way to do: > > x = np.arange(10) > y = np.arange(20).reshape((-1,1)) > > z = FunctionOfXandY(x,y) > > > Final note: > > If no one has an linear algebra examples, then we're wasting out time > with even this cheap talk. > > -Chris > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Apr 24 19:58:44 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 24 Apr 2008 18:58:44 -0500 Subject: [Numpy-discussion] Fixing #736 and possible memory leak In-Reply-To: References: Message-ID: <3d375d730804241658m12ce72eqc9c3e0f6a2e19a24@mail.gmail.com> On Thu, Apr 24, 2008 at 5:37 PM, Charles R Harris wrote: > Hi, > > I've been looking into ticket #736 and playing with some things. In > arrayobject.c starting at line 8534 I added a check for strings. > > if (PyString_Check(op)) { > r = Array_FromPyScalar(op, newtype); > } > if (PySequence_Check(op)) { > PyObject *thiserr = NULL; > > /* necessary but not sufficient */ > Py_INCREF(newtype); > r = Array_FromSequence(op, newtype, flags & FORTRAN, > min_depth, max_depth); > if (r == NULL && (thiserr=PyErr_Occurred())) { > if (PyErr_GivenExceptionMatches(thiserr, > PyExc_MemoryError)) { > return NULL; > } > > I think there may be a failure to decrement the reference to newtype unless > Array_FromSequence does that (nasty side effect); > > Anyway, the added check for a string fixes the conversion problem for such > things as int32('123'). There remains a problem with array('123', > dtype=int32) and with array(['123','123'], dtype=int32), but I think I can > track those down. The question is, will changing the current behavior so > that strings get converted to numbers cause problems with other programs out > there. I suspect I also need to check that strings are converted this way > only when the type is explicitly given, not detected. Seems to work for me. In [5]: array([124, '123', '123']) Out[5]: array(['124', '123', '123'], dtype='|S4') -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From wbaxter at gmail.com Thu Apr 24 20:13:52 2008 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 25 Apr 2008 09:13:52 +0900 Subject: [Numpy-discussion] numpy release In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com> <480FB04F.80006@noaa.gov> Message-ID: On Thu, Apr 24, 2008 at 11:16 AM, Timothy Hochberg wrote: > > [CHOP] > > The proposals thus far don't address two of the major issues I have with the > matrix class: The thing that seems missing to me is support for LAPACK's banded and packed (triangular) storage formats. I don't have an urgent need for it, but it seems like a serious front end for doing linear algebra should at least provide a convenient way to access all of what LAPACK offers. Perhaps these other formats could/should be supported by different classes, though, the way it is done for sparse matrices. But the fact that more matrix types could exist some day is all the more reason to make sure the interface makes sense for more than just strided memory. My hunch is if the API is workable for both ndarray matrices and sparse matrices, then it should be ok for these other formats too. --bb From charlesr.harris at gmail.com Thu Apr 24 21:11:27 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 24 Apr 2008 19:11:27 -0600 Subject: [Numpy-discussion] Fixing #736 and possible memory leak In-Reply-To: <3d375d730804241658m12ce72eqc9c3e0f6a2e19a24@mail.gmail.com> References: <3d375d730804241658m12ce72eqc9c3e0f6a2e19a24@mail.gmail.com> Message-ID: On Thu, Apr 24, 2008 at 5:58 PM, Robert Kern wrote: > On Thu, Apr 24, 2008 at 5:37 PM, Charles R Harris > wrote: > > Hi, > > > > I've been looking into ticket #736 and playing with some things. In > > arrayobject.c starting at line 8534 I added a check for strings. > > > > if (PyString_Check(op)) { > > r = Array_FromPyScalar(op, newtype); > > } > > if (PySequence_Check(op)) { > > PyObject *thiserr = NULL; > > > > /* necessary but not sufficient */ > > Py_INCREF(newtype); > > r = Array_FromSequence(op, newtype, flags & FORTRAN, > > min_depth, max_depth); > > if (r == NULL && (thiserr=PyErr_Occurred())) { > > if (PyErr_GivenExceptionMatches(thiserr, > > PyExc_MemoryError)) { > > return NULL; > > } > > > > I think there may be a failure to decrement the reference to newtype > unless > > Array_FromSequence does that (nasty side effect); > > > > Anyway, the added check for a string fixes the conversion problem for > such > > things as int32('123'). There remains a problem with array('123', > > dtype=int32) and with array(['123','123'], dtype=int32), but I think I > can > > track those down. The question is, will changing the current behavior so > > that strings get converted to numbers cause problems with other programs > out > > there. I suspect I also need to check that strings are converted this way > > only when the type is explicitly given, not detected. > > Seems to work for me. > > In [5]: array([124, '123', '123']) > Out[5]: > array(['124', '123', '123'], > dtype='|S4') Sure, but you didn't specify the type, so numpy determined that it was numpy string type. Wrong test. Try In [1]: array(['123'], dtype=int32) Out[1]: array([[1, 2, 3]]) In [2]: a = ones(3, dtype=int32) In [3]: a[...] = '123' In [4]: a Out[4]: array([1, 2, 3]) In [5]: a[...] = int32('123') In [6]: a Out[6]: array([123, 123, 123]) So on and so forth. The problem is this bit of code (among others) stop_at_string = ((type == PyArray_OBJECT) || (type == PyArray_STRING && typecode->type == PyArray_STRINGLTR) || (type == PyArray_UNICODE) || (type == PyArray_VOID)); The question is, how do we interpret a string when the type is specified? I think in that case we should try to convert the string to the relevant type, just as we cast numbers to the relevant type. So we should always stop at string. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Thu Apr 24 21:23:29 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 24 Apr 2008 18:23:29 -0700 Subject: [Numpy-discussion] numpy release In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> <9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com> <480FB04F.80006@noaa.gov> <4810DEA2.60501@noaa.gov> Message-ID: <48113291.5060009@noaa.gov> Alan G Isaac wrote: > On Thu, 24 Apr 2008, Christopher Barker apparently wrote: >> I suppose a "Vector" can be either a (n,1) or a (1,n) >> matrix that allows single indexing. > > This bothers me. > > So, if X is 2 by 2, then X[0] will be a row vector. > But if X is 1 by 2, then X[0] will be a scalar? > Ouch! > Bye bye generic code. exactly one of the things that really bugged me with MATLAB! > Or are you still having separate classes and > not just changing ``__getitem__``? OK, I see > that is most likely what you meant. yes, it would still be a vector class, so a 1 by 2 matrix (or array) would still act like a 2-d object. The only issue here is if you need a separate "row vector" and "column vector" class -- I think it's a matter of what whether these vectors are special versions of 1-d arrays or 2-d matrices. I kind of like them as special versions of 1-d arrays, as a vector is a 1-d object, but I see that the implementation is a bit cleaner if you just make them 2-d matrices with a different scalar indexing. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From wbaxter at gmail.com Thu Apr 24 21:27:57 2008 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 25 Apr 2008 10:27:57 +0900 Subject: [Numpy-discussion] numpy release In-Reply-To: <48113291.5060009@noaa.gov> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com> <480FB04F.80006@noaa.gov> <4810DEA2.60501@noaa.gov> <48113291.5060009@noaa.gov> Message-ID: On Fri, Apr 25, 2008 at 10:23 AM, Christopher Barker wrote: > Alan G Isaac wrote: > > On Thu, 24 Apr 2008, Christopher Barker apparently wrote: > >> I suppose a "Vector" can be either a (n,1) or a (1,n) > >> matrix that allows single indexing. > > > > This bothers me. > > > > So, if X is 2 by 2, then X[0] will be a row vector. > > But if X is 1 by 2, then X[0] will be a scalar? > > Ouch! > > Bye bye generic code. > > exactly one of the things that really bugged me with MATLAB! I thought MATLAB treated a single index as a flat index. So it yields a scalar no matter what (actually a 1x1 matrix no matter what, since it's all matrices in matlab). --bb From kwgoodman at gmail.com Thu Apr 24 21:30:42 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 24 Apr 2008 18:30:42 -0700 Subject: [Numpy-discussion] Generating Bell Curves (was: Using normal() ) In-Reply-To: References: Message-ID: On Thu, Apr 24, 2008 at 4:15 PM, Rich Shepard wrote: > We need to generate bell-shaped curves given a midpoint, width (where y=0) A Gaussian never reaches zero. But I suppose your code will return a zero for large x when y is less than the smallest number your computer can handle. But that is a poor way to define the width of a Gaussian. The slope in the tails of the Gaussian is near zero so a small change in y will results in a large change in your width. From charlesr.harris at gmail.com Thu Apr 24 21:51:21 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 24 Apr 2008 19:51:21 -0600 Subject: [Numpy-discussion] Fixing #736 and possible memory leak In-Reply-To: References: <3d375d730804241658m12ce72eqc9c3e0f6a2e19a24@mail.gmail.com> Message-ID: On Thu, Apr 24, 2008 at 7:11 PM, Charles R Harris wrote: > > > On Thu, Apr 24, 2008 at 5:58 PM, Robert Kern > wrote: > >> On Thu, Apr 24, 2008 at 5:37 PM, Charles R Harris >> wrote: >> > Hi, >> > >> > I've been looking into ticket #736 and playing with some things. In >> > arrayobject.c starting at line 8534 I added a check for strings. >> > >> > if (PyString_Check(op)) { >> > r = Array_FromPyScalar(op, newtype); >> > } >> > if (PySequence_Check(op)) { >> > PyObject *thiserr = NULL; >> > >> > /* necessary but not sufficient */ >> > Py_INCREF(newtype); >> > r = Array_FromSequence(op, newtype, flags & FORTRAN, >> > min_depth, max_depth); >> > if (r == NULL && (thiserr=PyErr_Occurred())) { >> > if (PyErr_GivenExceptionMatches(thiserr, >> > PyExc_MemoryError)) { >> > return NULL; >> > } >> > >> > I think there may be a failure to decrement the reference to newtype >> unless >> > Array_FromSequence does that (nasty side effect); >> > >> > Anyway, the added check for a string fixes the conversion problem for >> such >> > things as int32('123'). There remains a problem with array('123', >> > dtype=int32) and with array(['123','123'], dtype=int32), but I think I >> can >> > track those down. The question is, will changing the current behavior so >> > that strings get converted to numbers cause problems with other programs >> out >> > there. I suspect I also need to check that strings are converted this >> way >> > only when the type is explicitly given, not detected. >> >> Seems to work for me. >> >> In [5]: array([124, '123', '123']) >> Out[5]: >> array(['124', '123', '123'], >> dtype='|S4') > > > Sure, but you didn't specify the type, so numpy determined that it was > numpy string type. Wrong test. Try > > In [1]: array(['123'], dtype=int32) > Out[1]: array([[1, 2, 3]]) > > In [2]: a = ones(3, dtype=int32) > > In [3]: a[...] = '123' > > In [4]: a > Out[4]: array([1, 2, 3]) > > In [5]: a[...] = int32('123') > > In [6]: a > Out[6]: array([123, 123, 123]) > > So on and so forth. The problem is this bit of code (among others) > > stop_at_string = ((type == PyArray_OBJECT) || > (type == PyArray_STRING && > typecode->type == PyArray_STRINGLTR) || > (type == PyArray_UNICODE) || > (type == PyArray_VOID)); > > > The question is, how do we interpret a string when the type is specified? I > think in that case we should try to convert the string to the relevant type, > just as we cast numbers to the relevant type. So we should always stop at > string. > Setting stop_at_string = 1 fixes all the problems I can see, but introduces one new one: Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/numpy/core/tests/test_regression.py", line 537, in check_numeric_carray_compare assert_equal(np.array([ 'X' ], 'c'),'X') ValueError: setting an array element with a sequence on the other hand array(['X']) == 'X' works fine, so I don't know what's going on with the 'c' type. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Apr 24 21:56:02 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 24 Apr 2008 19:56:02 -0600 Subject: [Numpy-discussion] Fixing #736 and possible memory leak In-Reply-To: <3d375d730804241658m12ce72eqc9c3e0f6a2e19a24@mail.gmail.com> References: <3d375d730804241658m12ce72eqc9c3e0f6a2e19a24@mail.gmail.com> Message-ID: On Thu, Apr 24, 2008 at 5:58 PM, Robert Kern wrote: > On Thu, Apr 24, 2008 at 5:37 PM, Charles R Harris > wrote: > > Hi, > > > > I've been looking into ticket #736 and playing with some things. In > > arrayobject.c starting at line 8534 I added a check for strings. > > > > if (PyString_Check(op)) { > > r = Array_FromPyScalar(op, newtype); > > } > > if (PySequence_Check(op)) { > > PyObject *thiserr = NULL; > > > > /* necessary but not sufficient */ > > Py_INCREF(newtype); > > r = Array_FromSequence(op, newtype, flags & FORTRAN, > > min_depth, max_depth); > > if (r == NULL && (thiserr=PyErr_Occurred())) { > > if (PyErr_GivenExceptionMatches(thiserr, > > PyExc_MemoryError)) { > > return NULL; > > } > > > > I think there may be a failure to decrement the reference to newtype > unless > > Array_FromSequence does that (nasty side effect); > > > > Anyway, the added check for a string fixes the conversion problem for > such > > things as int32('123'). There remains a problem with array('123', > > dtype=int32) and with array(['123','123'], dtype=int32), but I think I > can > > track those down. The question is, will changing the current behavior so > > that strings get converted to numbers cause problems with other programs > out > > there. I suspect I also need to check that strings are converted this way > > only when the type is explicitly given, not detected. > > Seems to work for me. > > In [5]: array([124, '123', '123']) > Out[5]: > array(['124', '123', '123'], > dtype='|S4') > BTW, why is it '|S4', shouldn't it be '|S3' ? That's what I get when everything is quoted. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Thu Apr 24 22:15:33 2008 From: sturla at molden.no (Sturla Molden) Date: Fri, 25 Apr 2008 04:15:33 +0200 (CEST) Subject: [Numpy-discussion] "aligned" matrix / ctypes In-Reply-To: <3d375d730804241508v15b53f03gf44a1e6c20fba081@mail.gmail.com> References: <3d375d730804231402g7e4bc135hdba82c82d8057055@mail.gmail.com> <3E5D8350-AF60-40C0-9E0F-74A3DBE6935B@yale.edu> <3d375d730804241145w6c55fed5mfc032a0b0a9f3d0f@mail.gmail.com> <3d375d730804241508v15b53f03gf44a1e6c20fba081@mail.gmail.com> Message-ID: <39340.85.167.148.49.1209089733.squirrel@webmail.uio.no> The problem with alignment on 3 byte boundaries, is that 3 is a prime and not a factor of the size of any common data type. (The only exception I can think of is 24 bit RGB values.) So in general, the elements in an array for which the first element is aligned on a 3 byte boundary, may or may not not be 3-byte aligned. Byte boundary alignment should thus be a bit intelligent. If the size of the dtype is not divisable by the byte boundary, an exception should be raised. In practice, only alignment on 2-, 4- and perhaps 8-byte boundaries are really required. Alignment on 2 byte boundaries should perhaps be NumPy's default (over no alignment), as MMX and SSE extensions depend on it. nVidia's CUDA also require alignment on 2 byte boundaries. Sturla Molden > On Thu, Apr 24, 2008 at 4:57 PM, Zachary Pincus > wrote: > >> The reason that one must slice before .view()ing to allow arbitrary >> alignment is as follows. Imagine that we want rows of four 2-byte >> shorts aligned to 3-byte boundaries. (Assume that we already have a >> buffer that starts on a 3-byte boundary.) So we need an array that's 9 >> bytes wide by however many rows, and then we just want to use the >> first eight bytes of row. If we slice first, we can get a strided >> array that is eight bytes wide, and thus something that we can >> interpret as four shorts. (That is, if .view() could handle strided >> arrays.) >> >> On the other hand, there's absolutely no way that we can .view() >> before slicing, because our underlying array is 9 bytes wide, and you >> can't look at 9 bytes as any integral number of 2-byte shorts. >> So .view() should properly fail, and thus we can't get to the slicing. > > Yes, you are right, sorry. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From cournapeau at cslab.kecl.ntt.co.jp Thu Apr 24 22:34:59 2008 From: cournapeau at cslab.kecl.ntt.co.jp (David Cournapeau) Date: Fri, 25 Apr 2008 11:34:59 +0900 Subject: [Numpy-discussion] "aligned" matrix / ctypes In-Reply-To: <39340.85.167.148.49.1209089733.squirrel@webmail.uio.no> References: <3d375d730804231402g7e4bc135hdba82c82d8057055@mail.gmail.com> <3E5D8350-AF60-40C0-9E0F-74A3DBE6935B@yale.edu> <3d375d730804241145w6c55fed5mfc032a0b0a9f3d0f@mail.gmail.com> <3d375d730804241508v15b53f03gf44a1e6c20fba081@mail.gmail.com> <39340.85.167.148.49.1209089733.squirrel@webmail.uio.no> Message-ID: <1209090899.17497.9.camel@bbc8> On Fri, 2008-04-25 at 04:15 +0200, Sturla Molden wrote: > The problem with alignment on 3 byte boundaries, is that 3 is a prime and > not a factor of the size of any common data type. (The only exception I > can think of is 24 bit RGB values.) So in general, the elements in an > array for which the first element is aligned on a 3 byte boundary, may or > may not not be 3-byte aligned. > Byte boundary alignment should thus be a bit intelligent. If the size of > the dtype is not divisable by the byte boundary, an exception should be > raised. > > In practice, only alignment on 2-, 4- and perhaps 8-byte boundaries are > really required. There are other useful alignements. I don't know for mmx, but for SSE, 16 bytes alignement is almost required for useful speedup (to be able to use movps instead of movups, which is extremely slow, when loading data from memory into sse registers). I saw once mention that the mkl also sometimes requires 64 byte alignement. > Alignment on 2 byte boundaries should perhaps be NumPy's > default (over no alignment), as MMX and SSE extensions depend on it. malloc on glibc alloc on 8 bytes boundaries by default, and malloc on mac os X on 16 bytes. I guess, but should check whether the same is true on solaris, since sparc does not like unusual alignement (bus errors if float are not 4 byte aligned, for example). I have somewhere the code for portable aligned allocators (mostly given by Steve Johnson from fftw fame) + a C api to access them in C extensions + plus default alignement to 16 bytes for PyDataMem_NEW (which is just a wrapper around those aligned allocators). cheers, David From wfspotz at sandia.gov Thu Apr 24 22:52:10 2008 From: wfspotz at sandia.gov (Bill Spotz) Date: Thu, 24 Apr 2008 20:52:10 -0600 Subject: [Numpy-discussion] Matrix Class [was numpy release] In-Reply-To: References: Message-ID: <5DF32B21-7C5A-4A2B-ADC0-5AB603FF2C46@sandia.gov> On Apr 24, 2008, at 5:45 PM, Timothy Hochberg wrote: > Bill Spotz wrote: > > I have generally thought about this in the context of, say, a > > Krylov-space iterative method, and what that type of interface would > > lead to the most readable code. > > Can you whip up a small example, starting with the current > implementation? This sounds like work. ;-) I'll shoot for putting something together this weekend. I think Tim is right; we've been talking around in circles, and we need something concrete. I also think that a conjugate gradient algorithm, say, is high-level and doesn't require iterating over rows (or columns). We should also look at (at least) one other algorithm, one that iterates over rows. I would suggest Gaussian elimination, unless this is too obviously a high-level functionality that should be part of the extension module. ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Apr 24 23:32:24 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 24 Apr 2008 21:32:24 -0600 Subject: [Numpy-discussion] Matrix Class [was numpy release] In-Reply-To: References: Message-ID: On Thu, Apr 24, 2008 at 5:45 PM, Timothy Hochberg wrote: > > > Chris.Barker wrote: > >> Alan G Isaac wrote: >> > the cost of complexity should be justified by a gain in functionality. >> >> I don't think functionality is the right word here. the Matrix class(es) >> is all about clean, convenient API, i.e. style, not functionality -- we >> have all the functionality already, indeed we have it with plain old >> arrays, so I think that's really beside the point. > > > Not entirely, there's no good way do deal with arrays of matrices at > present. This could be fixed by tweaking dot, but it could also be part of a > reform of the matrix class. > > [CHOP] > >> >> Timothy Hochberg wrote: >> > 1. The matrices and arrays should become more alike if possible >> >> I'm not sure I agree -- the more alike they are, the less point there is >> to having them. > > > That's the best possible outcome. If some solution can be reached that > naturally supports enough matrix operations on array, without significantly > complexifying array, to satisfy the matrix users then that would be great. > Less stuff to maintain, less stuff to learn, etc, etc. > > With that in mind, what is minimum amount of stuff that matrix should > support: > > 1. Indexing along a single axis should produce a row or column vector > as appropriate. > 2. '*' should mean matrix multiplication. > 3. '**' should mean matrix exponentiation. I suspect that this is less > crucial than 2, but no more difficult. > > There's some other stuff that's less critical IMO (.H and .I for example) > but feel free to yell at me if you think I'm mistaken. > > There's some other properties that a fix should have as well, in my opinion > and in some cases in others opinions as well. > > 1. A single index into an array should yield a 1D object just as > indexing an array does. This does not have to inconsistent with #1 above; > Chris Barker proposed one solution. I'm not sold on the details of that > solution, but I approve of the direction that it points to. [In general > A[i][j] should equal A[i,j]. I know that fancy indexing breaks this; that > was a mistake IMO, but that's a discussion for another day]. > 2. It should be possible to embed both vectors and matrices naturally > into arrays. This would allow natural and efficient implementation of, for > example, rotating a 1000 3D vectors. One could spell that R * V, where R is > a 2x2 matrix and V is a 1000x3 array, where the last axis is understood to > be a vector. > > R is 3x3? This works already, whats tough is an array of rotation matrices applies "element wise" to an array of vectors. > 1. > 2. I'm pretty certain there's a third thing I wanted to mention but it > escapes me. It'll resurface at the most inopportune time.... > > Let's play with Chris Barker's RowVector and ColVector proposal for a > moment. Let's suppose that we have four four classes: Scalar, RowVector, > ColVector and Matrix. They're all pretty much the same as today's array > class except that: > > 1. They are displayed a little differently so that you can tell them > apart. > 2. __mul__ and __pow__ are treated differently > > Let's consider __mul__: when a RowVector multiplied with ColVector, the dot > product is computed. If, for example, the arrays are both 2D, the they are > treated as arrays of vectors and the dot product of each pair of vectors is > computed in turn: I think broadcasting should work correctly, but I haven't > thought that all the way through yet. Ignoring broadcasting for a moment, > the rules are: > > 1. (n)D Scalar * (n)D Scalar => (n)D Scalar (elementwise) > 2. (n)D RowVector * (n)D ColVector => (n-1)D Scalar (dot product) > 3. (n+1)D Matrix * (n)D ColVector => (n)D ColVector (matrix-vector > product) > 4. (n)D Matrix * n(D) Matrix => (n)D Matrix (matrix) product > > And ColVector * RowVector is a matrix? Or illegal. The latter, I suppose, if * is contraction on indices. > Other combinations would be an error. In principal you could do dyadic > products, but I suspect we'd be better off using a separate function for > that since most of the times that would just indicate a mistake. Note that > in this example Scalar is playing the role of the present day array, which > I've assumed has magically vanished from the scene somehow. > > This looks like it gets of most the way towards where we want to be. There > are some issues though. For example, all of the sudden you want to different > transpose operators; one that transposes matrices and flips colVectors to > RowVectors and leaves Scalars alone, and another that manipulates the rest > of the array structure. There is probably other stuff like that too, it > doesn't look insurmountable to me right now, however. > > Still, I'm not sold on the whole RowVector/ColVector/Matrix approach. I > have a gut feeling that we'd be better off with a single array class that > was somehow tagged to indicate it's type. Also, I keep hoping to come up > with an elegant generalization to this that turns arrays into quasi-tensor > objects where all the matrix behavior falls out naturally. Sadly, attempts > in this direction keep collapsing under there own weight but I'm still > thinking about it. > I think a start could be made by adding a signature to arrays, indicating whether the corresponding indices are up or down. Up is column, down is row. The in tensor parlanc, matrices have signature (u,d). The matrix product sums an up with a down, contraction, so you get the usual rules. Note that row*col == col*row == inner product in this scheme, which may seem a bit odd, but transpose(row) := col as usual. This doesn't solve the problem of containers of matrices, however. I think that matrices could be treated as objects with special, efficient storage in arrays, and that way the element-wise operators will do what you need using broadcasting. Essentially the last two/one dimensions are reserved. This is a bit like matlab cell arrays, except the arrays must be homogeneous in the matrix type. Note that I'm making a distinction between 1xn matrices and row vectors. The former has signature (u,d), the latter (u,). Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Apr 24 23:53:42 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 24 Apr 2008 21:53:42 -0600 Subject: [Numpy-discussion] problem with view() and strided arrays? In-Reply-To: <0E96FCDA-DF09-40AE-9C1E-A2EE1FD3F40F@yale.edu> References: <0E96FCDA-DF09-40AE-9C1E-A2EE1FD3F40F@yale.edu> Message-ID: On Thu, Apr 24, 2008 at 10:00 AM, Zachary Pincus wrote: > > It seems like 'view' won't create a view where the stride length is > > not a whole multiple of the dtype's itemsize. However, since strides > > are measured in bytes (right?), this shouldn't be a problem. > > > Actually -- it seems like view() doesn't work with strided arrays at > all. (?) > > In : a = numpy.ones((4,32), dtype=numpy.uint8) > > In : a.view(numpy.uint16).shape > Out: (4, 16) > > In : a[:,:16].view(numpy.uint16) > ValueError: new type not compatible with array. > > I think this might be a recent regression, because before I updated my > numpy installation to the latest SVN version (to check if the bug was > fixed!), I'm pretty sure this kind of operation worked. > Views only work for items of the same size. You are trying to view bytes as two byte values. View is like strict pointer aliasing in C. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Apr 25 00:25:10 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 24 Apr 2008 23:25:10 -0500 Subject: [Numpy-discussion] problem with view() and strided arrays? In-Reply-To: References: <0E96FCDA-DF09-40AE-9C1E-A2EE1FD3F40F@yale.edu> Message-ID: <3d375d730804242125ib85dfd7q3f2a4f0dbaca5ef4@mail.gmail.com> On Thu, Apr 24, 2008 at 10:53 PM, Charles R Harris wrote: > Views only work for items of the same size. You are trying to view bytes as > two byte values. View is like strict pointer aliasing in C. No, it works just fine if you have a contiguous array. In [5]: from numpy import * In [6]: a = ones(10, dtype=uint8) In [7]: a Out[7]: array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1], dtype=uint8) In [8]: b = a.view(uint16) In [9]: b Out[9]: array([257, 257, 257, 257, 257], dtype=uint16) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From haase at msg.ucsf.edu Fri Apr 25 05:17:14 2008 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 25 Apr 2008 11:17:14 +0200 Subject: [Numpy-discussion] OSX installer: please test In-Reply-To: <480F9200.9060803@noaa.gov> References: <764e38540804161436j5ce7a468w8bea6a3c3598d69e@mail.gmail.com> <480F9200.9060803@noaa.gov> Message-ID: OT: How do you make a dmg ? Is there a (simple) command line tool for this ? Thanks, Sebastian Haase On Wed, Apr 23, 2008 at 9:46 PM, Christopher Barker wrote: > Christopher Burns wrote: > > I've built a Universal Mac binary for numpy 1.1.0. If > > > Mac people would kindly test it, I'd appreciate any feedback. > > > > > > Download here: > > https://cirl.berkeley.edu/numpy/numpy-1.1.0rc1-py2.5-macosx10.5.dmg > > Note that it's called *osx10.5.dmg, but it seems to work just fine on > 10.4 -- same python. > > All tests pass: > Dual PPC G5 > > OS-X 10.4.11 > python.org version 2.5.1 > > Thanks! > > -Chris From cournapeau at cslab.kecl.ntt.co.jp Fri Apr 25 05:24:22 2008 From: cournapeau at cslab.kecl.ntt.co.jp (David Cournapeau) Date: Fri, 25 Apr 2008 18:24:22 +0900 Subject: [Numpy-discussion] OSX installer: please test In-Reply-To: References: <764e38540804161436j5ce7a468w8bea6a3c3598d69e@mail.gmail.com> <480F9200.9060803@noaa.gov> Message-ID: <1209115462.17497.14.camel@bbc8> On Fri, 2008-04-25 at 11:17 +0200, Sebastian Haase wrote: > OT: How do you make a dmg ? Is there a (simple) command line tool for this ? .dmg is just an iso 9660 file (e.g. a CD "fs"), which is recognized by Mac OS X as such. It really is not different than mounting an iso on any unix (you can mount dmg on linux, normally). To get to your point: hdiutil is the command you are looking for. I don't have my macbook available right now, and I have not used it for quite a while, but I guess man hdiutil should give you all the information you are looking for. http://www.kernelthread.com/mac/apme/archive/ cheers, David From tournesol33 at gmail.com Fri Apr 25 05:55:23 2008 From: tournesol33 at gmail.com (tournesol) Date: Fri, 25 Apr 2008 18:55:23 +0900 Subject: [Numpy-discussion] insert 1D to a 2D array and change it to 3D Message-ID: <4811AA8B.5070604@gmail.com> Hi All. I just want to conver Fortran 77 source to Python. Here is my F77 source. DIMENSION A(25,60,13),B(25,60,13) open(15,file='data.dat') DO 60 K=1,2 READ(15,1602) ((B(I,J),J=1,60),I=1,25) 60 CONTINUE 1602 FORMAT(15I4) DO 63 K=1,10 DO 62 I=1,25 DO 62 J=1,60 A(I,J,K)=B(I,J) 62 CONTINUE 63 CONTINUE END Q1: Fortran-contiguous is ARRAY(row,colum,depth). How about the Python-contiguous ? array(depth,row,colum) ? Q2: How can I insert 1D to a 2D array and make it to 3D array. ex:) B:25x60 ==> A: 10X25X60 Any advice please. From matthieu.brucher at gmail.com Fri Apr 25 06:09:42 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Fri, 25 Apr 2008 12:09:42 +0200 Subject: [Numpy-discussion] insert 1D to a 2D array and change it to 3D In-Reply-To: <4811AA8B.5070604@gmail.com> References: <4811AA8B.5070604@gmail.com> Message-ID: 2008/4/25, tournesol : > > Hi All. > > > I just want to conver Fortran 77 source to > Python. > > Here is my F77 source. > > DIMENSION A(25,60,13),B(25,60,13) > > open(15,file='data.dat') > DO 60 K=1,2 > READ(15,1602) ((B(I,J),J=1,60),I=1,25) > 60 CONTINUE > 1602 FORMAT(15I4) > > DO 63 K=1,10 > DO 62 I=1,25 > DO 62 J=1,60 > A(I,J,K)=B(I,J) > 62 CONTINUE > 63 CONTINUE > END > > Q1: Fortran-contiguous is ARRAY(row,colum,depth). > How about the Python-contiguous ? array(depth,row,colum) ? Default is C-contiguous, but you can you Fortran contiguous arrays. Q2: How can I insert 1D to a 2D array and make it to > 3D array. ex:) B:25x60 ==> A: 10X25X60 > I don't understand what you want to do, but broadcasting allows copying several instances of an array into another one. Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From lbolla at gmail.com Fri Apr 25 06:52:30 2008 From: lbolla at gmail.com (lorenzo bolla) Date: Fri, 25 Apr 2008 12:52:30 +0200 Subject: [Numpy-discussion] insert 1D to a 2D array and change it to 3D In-Reply-To: References: <4811AA8B.5070604@gmail.com> Message-ID: <80c99e790804250352n7951fa12h7322893325a28c76@mail.gmail.com> why not using something like numpy.repeat? In [18]: B = numpy.random.rand(4,3) In [19]: A = numpy.repeat(B[:,:,numpy.newaxis],2,axis=2) In [20]: B.shape Out[20]: (4, 3) In [21]: A.shape Out[21]: (4, 3, 2) In [22]: numpy.all(A[:,:,0] == A[:,:,1]) Out[22]: True hth, L. On Fri, Apr 25, 2008 at 12:09 PM, Matthieu Brucher < matthieu.brucher at gmail.com> wrote: > > > 2008/4/25, tournesol : >> >> Hi All. >> >> >> I just want to conver Fortran 77 source to >> Python. >> >> Here is my F77 source. >> >> DIMENSION A(25,60,13),B(25,60,13) >> >> open(15,file='data.dat') >> DO 60 K=1,2 >> READ(15,1602) ((B(I,J),J=1,60),I=1,25) >> 60 CONTINUE >> 1602 FORMAT(15I4) >> >> DO 63 K=1,10 >> DO 62 I=1,25 >> DO 62 J=1,60 >> A(I,J,K)=B(I,J) >> 62 CONTINUE >> 63 CONTINUE >> END >> >> Q1: Fortran-contiguous is ARRAY(row,colum,depth). >> How about the Python-contiguous ? array(depth,row,colum) ? > > > > Default is C-contiguous, but you can you Fortran contiguous arrays. > > > Q2: How can I insert 1D to a 2D array and make it to >> 3D array. ex:) B:25x60 ==> A: 10X25X60 >> > > I don't understand what you want to do, but broadcasting allows copying > several instances of an array into another one. > > Matthieu > -- > French PhD student > Website : http://matthieu-brucher.developpez.com/ > Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > LinkedIn : http://www.linkedin.com/in/matthieubrucher > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -- Lorenzo Bolla lbolla at gmail.com http://lorenzobolla.emurse.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rshepard at appl-ecosys.com Fri Apr 25 09:32:19 2008 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Fri, 25 Apr 2008 06:32:19 -0700 (PDT) Subject: [Numpy-discussion] Generating Bell Curves (was: Using normal() ) In-Reply-To: References: Message-ID: On Thu, 24 Apr 2008, Keith Goodman wrote: > A Gaussian never reaches zero. Keith, I know, and that's why I need to find another way to draw these curves. While mathematically any 'y' value < 0.2 (the default) is equivalent to zero, the curves must reach zero in the figures. Briefly, this model is used in regulatory compliance that may also end up in legal proceedings. While the results are robust and not affected by the curve not reaching zero, the appearance can cause problems in the regulatory and legal arenas. They would be a distraction, a red herring, and consume resources to explain. All this can be avoided by bell curves that reach y=0.0 at the ends. Rich -- Richard B. Shepard, Ph.D. | Integrity Credibility Applied Ecosystem Services, Inc. | Innovation Voice: 503-667-4517 Fax: 503-667-8863 From zachary.pincus at yale.edu Fri Apr 25 09:36:39 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Fri, 25 Apr 2008 09:36:39 -0400 Subject: [Numpy-discussion] problem with view() and strided arrays? In-Reply-To: <0E96FCDA-DF09-40AE-9C1E-A2EE1FD3F40F@yale.edu> References: <0E96FCDA-DF09-40AE-9C1E-A2EE1FD3F40F@yale.edu> Message-ID: Hi all, > Actually -- it seems like view() doesn't work with strided arrays at > all. (?) > > In : a = numpy.ones((4,32), dtype=numpy.uint8) > > In : a.view(numpy.uint16).shape > Out: (4, 16) > > In : a[:,:16].view(numpy.uint16) > ValueError: new type not compatible with array. > > I think this might be a recent regression, because before I updated my > numpy installation to the latest SVN version (to check if the bug was > fixed!), I'm pretty sure this kind of operation worked. The problem starts out in arrayobject.c:6392, in array_descr_set(), where the following test is performed: if ((newtype->elsize != self->descr->elsize) && \ (self->nd == 0 || !PyArray_ISONESEGMENT(self) || \ newtype->subarray)) goto fail; I *think* I could fix it by relaxing the restrictions to only require that the array have at least one dimension where self->strides[dim] == self->descr->elsize, and then adjust the size of that dimension. Here's my perhaps-fix. The old code is: if ((newtype->elsize != self->descr->elsize) && \ (self->nd == 0 || !PyArray_ISONESEGMENT(self) || \ newtype->subarray)) goto fail; if (PyArray_ISCONTIGUOUS(self)) index = self->nd - 1; else index = 0; My suggested fix is: if ((newtype->elsize != self->descr->elsize) && \ (self->nd == 0 || newtype->subarray)) goto fail; if (PyArray_ISCONTIGUOUS(self)) index = self->nd - 1; else if (PyArray_ISFORTRAN(self)) index = 0; else { int index_found = FALSE; for (index = 0; index < self->nd; index++) { if (self->strides[index] == self->descr->elsize) { index_found = TRUE; break; } } if (!index_found) goto fail; } Could someone look this over? If it looks basically right, I'll make this a proper patch and post it to trac. Thanks, Zach From aisaac at american.edu Fri Apr 25 09:40:51 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 25 Apr 2008 09:40:51 -0400 Subject: [Numpy-discussion] numpy release In-Reply-To: <9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com><9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com><480FB04F.80006@noaa.gov><4810DEA2.60501@noaa.gov> <9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com> Message-ID: > 2008/4/25 Alan G Isaac : >> So, if X is 2 by 2, then X[0] will be a row vector. >> But if X is 1 by 2, then X[0] will be a scalar? >> Ouch! >> Bye bye generic code. On Fri, 25 Apr 2008, Stefan van der Walt apparently wrote: > Yup. That's the current state of things. I do not understand. The released "state of things" is that for matrix ``x`` we have that ``x[0]`` is a **matrix**. I'm not working with SVN NumPy, but my understanding was that as of revision r5072 ``x[0]`` always returns a 1d array. The core requirement was that ``x[0][0]`` produce the first element of the matrix. I do not have time to look at the revision right now, but if a matrix ``x`` we have that ``x[0]`` can return a scalar, that is very undesirable. Cheers, Alan From aisaac at american.edu Fri Apr 25 09:50:20 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 25 Apr 2008 09:50:20 -0400 Subject: [Numpy-discussion] Generating Bell Curves (was: Using normal() ) In-Reply-To: References: Message-ID: On Fri, 25 Apr 2008, Rich Shepard apparently wrote: > bell curves that reach y=0.0 at the ends. Just change the function you are graphing. def bell(x, xmin, xmax): if xmin < x < xmax: return density(x) return 0 Here ``density`` is whatever function you are currently using to produce the dependent variable. hth, Alan From bsouthey at gmail.com Fri Apr 25 09:51:36 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Fri, 25 Apr 2008 08:51:36 -0500 Subject: [Numpy-discussion] Generating Bell Curves (was: Using normal() ) In-Reply-To: References: Message-ID: <4811E1E8.8010204@gmail.com> Rich Shepard wrote: > Thanks to several of you I produced test code using the normal density > function, and it does not do what we need. Neither does the Gaussian > function using fwhm that I've tried. The latter comes closer, but the ends > do not reach y=0 when the inflection point is y=0.5. > > So, let me ask the collective expertise here how to generate the curves > that we need. > > We need to generate bell-shaped curves given a midpoint, width (where y=0) > and inflection point (by default, y=0.5) where y is [0.0, 1.0], and x is > usually [0, 100], but can vary. Using the NumPy arange() function to produce > the x values (e.g, arange(0, 100, 0.1)), I need a function that will produce > the associated y values for a bell-shaped curve. These curves represent the > membership functions for fuzzy term sets, and generally adjacent curves > overlap where y=0.5. It would be a bonus to be able to adjust the skew and > kurtosis of the curves, but the controlling data would be the > center/midpoint and width, with defaults for inflection point, and other > parameters. > > I've been searching for quite some time without finding a solution that > works as we need it to work. > > TIA, > > Rich > > Hi, You could use a Gamma distribution to get a skewed distribution. But to extend Keith's comment, continuous distributions typically go from minus infinity or zero to positive infinity and, furthermore, the probability of a single point in a continuous distribution is always zero. The only way you are going to get this from a single continuous distribution is via some truncated distribution - essentially Keith's reply. Alternatively, you may get away with a discrete distribution like the Poisson since it very quickly approaches normality but is skewed. A multinomial distribution may also work but that is more assumptions. In either case, you have map the points into the valid space because it is the distribution within the set that is used not the distribution of the data. I do not see the requirement for overlapping curves because the expected distribution of each set should be independent of the data and of the other sets. In that case, you just find the mean and variance of each set to get the degree of overlap you require. The inflection point requirement is very hard to understand as it different meanings such as just crossing or same area under the curve. I don't see any simple solution to that - two normals with the same variance but different means probably would. If the sets are dependent then you need a multivariate solution. Really you probably need a mixture of distributions and/or generate your own function to get something that meets you full requirements. Regards Bruce From oliphant at enthought.com Fri Apr 25 10:27:43 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 25 Apr 2008 09:27:43 -0500 Subject: [Numpy-discussion] Fixing #736 and possible memory leak In-Reply-To: References: <3d375d730804241658m12ce72eqc9c3e0f6a2e19a24@mail.gmail.com> Message-ID: <4811EA5F.9060909@enthought.com> > > So on and so forth. The problem is this bit of code (among others) > > stop_at_string = ((type == PyArray_OBJECT) || > (type == PyArray_STRING && > typecode->type == PyArray_STRINGLTR) || > (type == PyArray_UNICODE) || > (type == PyArray_VOID)); > > > The question is, how do we interpret a string when the type is > specified? I think in that case we should try to convert the string to > the relevant type, just as we cast numbers to the relevant type. So we > should always stop at string. > The extra test there is to not stop at the string when the typecode is 'c' (this is to ensure the behavior of 'c' that was in Numeric). So, this test should not be taken away. I'm not sure why it should be causing the kind of problems you are describing. But, if so, the fix is elsewhere. -Travis From bsouthey at gmail.com Fri Apr 25 10:40:09 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Fri, 25 Apr 2008 09:40:09 -0500 Subject: [Numpy-discussion] numpy release In-Reply-To: <4810DEA2.60501@noaa.gov> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> <9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com> <480FB04F.80006@noaa.gov> <4810DEA2.60501@noaa.gov> Message-ID: <4811ED49.9040900@gmail.com> Christopher Barker wrote: [cjop] > > Bruce Southey wrote: > >> Hi, >> I would like to use the matrix functionality but I, like others, get by >> without it. >> > > What does it lack that keeps you from using it? That's the key question? > I mainly do matrix-matrix and matrix-vector multiplications like in linear models and predictions. The main reason I have already learnt how to do to what I want using Numpy via mainly Numarray. So at the moment I am not interested in doing something else. >> +1 to Tim's and Nadav's comments. As Tim said, there should be seamless >> integration between concepts of vectors and matrices - after all there >> really no distinction between them in linear algebra. >> > > This is all about being able to index a "vector" with a single index -- > that's it, I think. Matlab handles this by special casing matrices that > have a single dimension of one. It also has an easy way to spell a > literal for column vector, though I suppose: > > np.Matrix((1,2,3,4,5)).T > > isn't so bad. > > >> -2 for having rowvector and columnvector - both numpy and I should know >> the orientation, if not, I deserve garbage or an error. >> > > But HOW can you know the orientation? a 1-d array has no orientation -- > I fully agree and which is why there is absolutely no sense in saying row or column vector/array! But vectors always have an orientation in the context of a matrix (as you also indicate below) or 2-d array so in that situation vectors are matrices. Numpy and other options have an implicit orientation because they usually always check that an operation is feasible first so errors are returned if the dimensions are not appropriate. So, as I understand it, Numpy's array do have an implicit orientation because Numpy's 1-d arrays act like 1 by n arrays when used with m by n arrays: import numpy a=numpy.ones((4)) b=numpy.ones((4,3)) numpy.dot(a,b) # gives array([ 4., 4., 4.]) numpy.dot(b,a) # gives error numpy.dot(b.T,a) #gives array([ 4., 4., 4.]) Thus, Numpy's 1-d arrays do have an orientation otherwise all three multiplications should return an error! > which is why the current version always returns a matrix when indexing. > As you point out, that is inconsistent with 1-d arrays having NO orientation. But this is fully consistent with the view that a vector is a special case of a matrix when we commonly refer to an n by 1 or 1 by n vector (as you also indicate below). >> 0 for vector - I don't see the need for it as a unique entity but >> understand the simplicity of saying vector. >> > > I don't think we can have vector without ColumnVector, though I suppose > a "Vector" can be either a (n,1) or a (1,n) matrix that allows single > indexing. > But didn't you say that '1-d array has no orientation'... :-) This why I am -2 on row and column vector because one should just be the transpose of the other. > By the way, maybe a column vector could be handy for doing array > broadcasting, as an cleaner way to do: > > x = np.arange(10) > y = np.arange(20).reshape((-1,1)) > > z = FunctionOfXandY(x,y) > > > Final note: > > If no one has an linear algebra examples, then we're wasting out time > with even this cheap talk. > > -Chris > > > Bruce From stefan at sun.ac.za Fri Apr 25 10:54:24 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 25 Apr 2008 16:54:24 +0200 Subject: [Numpy-discussion] numpy release In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com> <480FB04F.80006@noaa.gov> <4810DEA2.60501@noaa.gov> <9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com> Message-ID: <9457e7c80804250754n4967da64o1440164439407990@mail.gmail.com> 2008/4/25 Alan G Isaac : > > 2008/4/25 Alan G Isaac : > > >> So, if X is 2 by 2, then X[0] will be a row vector. > >> But if X is 1 by 2, then X[0] will be a scalar? > >> Ouch! > >> Bye bye generic code. > > > On Fri, 25 Apr 2008, Stefan van der Walt apparently wrote: > > Yup. That's the current state of things. > > I do not understand. > The released "state of things" is that for matrix ``x`` > we have that ``x[0]`` is a **matrix**. > > I'm not working with SVN NumPy, but my understanding > was that as of revision r5072 ``x[0]`` always > returns a 1d array. The core requirement was that > ``x[0][0]`` produce the first element of the matrix. > > I do not have time to look at the revision right now, > but if a matrix ``x`` we have that ``x[0]`` > can return a scalar, that is very undesirable. In current SVN: In [3]: x = np.matrix(np.arange(9).reshape((3,3))) In [5]: x Out[5]: matrix([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) In [6]: x[0] Out[6]: matrix([[0, 1, 2]]) In [7]: x[0,:] Out[7]: matrix([[0, 1, 2]]) In [8]: x[0][0] Out[8]: 0 In [9]: x[0,:][0] Out[9]: 0 In [10]: x[:,0] Out[10]: matrix([[0], [3], [6]]) In [11]: x[:,0][0] Out[11]: 0 In [12]: x[:2,:2][0] Out[12]: matrix([[0, 1]]) Regards St?fan From david.huard at gmail.com Fri Apr 25 11:33:17 2008 From: david.huard at gmail.com (David Huard) Date: Fri, 25 Apr 2008 11:33:17 -0400 Subject: [Numpy-discussion] Generating Bell Curves (was: Using normal() ) In-Reply-To: <4811E1E8.8010204@gmail.com> References: <4811E1E8.8010204@gmail.com> Message-ID: <91cf711d0804250833jab26c80qb50d2a8b92b064a7@mail.gmail.com> Other suggestions for bounded bell-shaped functions that reach zero on a finite interval: - Beta distribution: http://en.wikipedia.org/wiki/Beta_distribution - Cubic B-splines:http://www.ibiblio.org/e-notes/Splines/Basis.htm 2008/4/25 Bruce Southey : > Rich Shepard wrote: > > Thanks to several of you I produced test code using the normal > density > > function, and it does not do what we need. Neither does the Gaussian > > function using fwhm that I've tried. The latter comes closer, but the > ends > > do not reach y=0 when the inflection point is y=0.5. > > > > So, let me ask the collective expertise here how to generate the > curves > > that we need. > > > > We need to generate bell-shaped curves given a midpoint, width (where > y=0) > > and inflection point (by default, y=0.5) where y is [0.0, 1.0], and x is > > usually [0, 100], but can vary. Using the NumPy arange() function to > produce > > the x values (e.g, arange(0, 100, 0.1)), I need a function that will > produce > > the associated y values for a bell-shaped curve. These curves represent > the > > membership functions for fuzzy term sets, and generally adjacent curves > > overlap where y=0.5. It would be a bonus to be able to adjust the skew > and > > kurtosis of the curves, but the controlling data would be the > > center/midpoint and width, with defaults for inflection point, and other > > parameters. > > > > I've been searching for quite some time without finding a solution > that > > works as we need it to work. > > > > TIA, > > > > Rich > > > > > Hi, > You could use a Gamma distribution to get a skewed distribution. But to > extend Keith's comment, continuous distributions typically go from > minus infinity or zero to positive infinity and, furthermore, the > probability of a single point in a continuous distribution is always > zero. The only way you are going to get this from a single continuous > distribution is via some truncated distribution - essentially Keith's > reply. > > Alternatively, you may get away with a discrete distribution like the > Poisson since it very quickly approaches normality but is skewed. A > multinomial distribution may also work but that is more assumptions. In > either case, you have map the points into the valid space because it is > the distribution within the set that is used not the distribution of the > data. > > I do not see the requirement for overlapping curves because the expected > distribution of each set should be independent of the data and of the > other sets. In that case, you just find the mean and variance of each > set to get the degree of overlap you require. The inflection point > requirement is very hard to understand as it different meanings such as > just crossing or same area under the curve. I don't see any simple > solution to that - two normals with the same variance but different > means probably would. If the sets are dependent then you need a > multivariate solution. Really you probably need a mixture of > distributions and/or generate your own function to get something that > meets you full requirements. > > Regards > Bruce > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Apr 25 11:39:41 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 25 Apr 2008 09:39:41 -0600 Subject: [Numpy-discussion] Fixing #736 and possible memory leak In-Reply-To: <4811EA5F.9060909@enthought.com> References: <3d375d730804241658m12ce72eqc9c3e0f6a2e19a24@mail.gmail.com> <4811EA5F.9060909@enthought.com> Message-ID: On Fri, Apr 25, 2008 at 8:27 AM, Travis E. Oliphant wrote: > > > > > So on and so forth. The problem is this bit of code (among others) > > > > stop_at_string = ((type == PyArray_OBJECT) || > > (type == PyArray_STRING && > > typecode->type == PyArray_STRINGLTR) || > > (type == PyArray_UNICODE) || > > (type == PyArray_VOID)); > > > > > > The question is, how do we interpret a string when the type is > > specified? I think in that case we should try to convert the string to > > the relevant type, just as we cast numbers to the relevant type. So we > > should always stop at string. > > > > The extra test there is to not stop at the string when the typecode is > 'c' (this is to ensure the behavior of 'c' that was in Numeric). So, > this test should not be taken away. I'm not sure why it should be > causing the kind of problems you are describing. But, if so, the fix is > elsewhere. > I set it to stop at strings except for 'c' types and it seems to work now. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From rshepard at appl-ecosys.com Fri Apr 25 11:46:15 2008 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Fri, 25 Apr 2008 08:46:15 -0700 (PDT) Subject: [Numpy-discussion] Generating Bell Curves (was: Using normal() ) In-Reply-To: <91cf711d0804250833jab26c80qb50d2a8b92b064a7@mail.gmail.com> References: <4811E1E8.8010204@gmail.com> <91cf711d0804250833jab26c80qb50d2a8b92b064a7@mail.gmail.com> Message-ID: On Fri, 25 Apr 2008, David Huard wrote: > Other suggestions for bounded bell-shaped functions that reach zero on a > finite interval: > > - Beta distribution: http://en.wikipedia.org/wiki/Beta_distribution > - Cubic B-splines:http://www.ibiblio.org/e-notes/Splines/Basis.htm Thanks, David. I'm aware of the Beta distribution, but the cubic B-splines are completely new to me. Certainly something to look into. Rich -- Richard B. Shepard, Ph.D. | Integrity Credibility Applied Ecosystem Services, Inc. | Innovation Voice: 503-667-4517 Fax: 503-667-8863 From aisaac at american.edu Fri Apr 25 12:04:27 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 25 Apr 2008 12:04:27 -0400 Subject: [Numpy-discussion] numpy release In-Reply-To: <4811ED49.9040900@gmail.com> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> <9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com> <480FB04F.80006@noaa.gov> <4810DEA2.60501@noaa.gov> <4811ED49.9040900@gmail.com> Message-ID: I think the use of the term 'vector' in this thread is becoming a bit confusing. An M by N matrix is a vector. (I.e., it is an element of a vector space.) Many people use the terms "row vector" and "column vector" to refer to special matrices. What is special is *not* that they are vectors (since all matrices are) but that they are a single row or a single column. Perhaps better terminology would have been "row matrix" and "column matrix". Using this terminology, we clearly expect a row matrix and a column matrix to have transposes. The question is, can their elements be indexed by scalars. The answer in all the proposals, I believe, is "yes". I believe we are finding that this answer has some problems. x is a 2 by 2 matrix x[0] is a 1 by 2 matrix x[0][0] is x[0,0] GOOD!! y = x[0] -> y is a 1 by 2 matrix y[0][0] is a TypeError BAD!! So I am becoming more and more persuaded that for a matrix ``x``: x[0] should be a 1d array (always!) x[0,:] should be a matrix The rule: use non-scalar indexing to extract submatrices fwiw, Alan Isaac From aisaac at american.edu Fri Apr 25 12:04:30 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 25 Apr 2008 12:04:30 -0400 Subject: [Numpy-discussion] numpy release In-Reply-To: <9457e7c80804250754n4967da64o1440164439407990@mail.gmail.com> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com><9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com><480FB04F.80006@noaa.gov><4810DEA2.60501@noaa.gov><9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com> <9457e7c80804250754n4967da64o1440164439407990@mail.gmail.com> Message-ID: On Fri, 25 Apr 2008, St?fan van der Walt apparently wrote: > In current SVN: > In [6]: x[0] > Out[6]: matrix([[0, 1, 2]]) I must have misunderstood: I thought the agreement was to provisionally return a 1d array for x[0], while we hashed through the other proposals. Cheers, Alan From zachary.pincus at yale.edu Fri Apr 25 12:09:55 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Fri, 25 Apr 2008 12:09:55 -0400 Subject: [Numpy-discussion] "aligned" matrix / ctypes In-Reply-To: <1209090899.17497.9.camel@bbc8> References: <3d375d730804231402g7e4bc135hdba82c82d8057055@mail.gmail.com> <3E5D8350-AF60-40C0-9E0F-74A3DBE6935B@yale.edu> <3d375d730804241145w6c55fed5mfc032a0b0a9f3d0f@mail.gmail.com> <3d375d730804241508v15b53f03gf44a1e6c20fba081@mail.gmail.com> <39340.85.167.148.49.1209089733.squirrel@webmail.uio.no> <1209090899.17497.9.camel@bbc8> Message-ID: Hello all, Attached is code (plus tests) for allocating aligned arrays -- I think this addresses all the requests in this thread, with regard to allowing for different kinds of alignment. Thanks Robert and Anne for your help and suggestions. Hopefully this will be useful. The core is a function for allocating arrays with totally arbitrary alignment along each dimension (e.g. you could allocate an 10x20 array of uint16's where each uint16 is aligned to 4-byte boundaries and each row of 20 uint16's is aligned to 32-byte boundaries, and the entire buffer is aligned to a 128-byte boundary.) I've also included helper functions for two common cases: when you want everything aligned to a particular multiple (every element, row, etc. as well as the whole buffer), and when you want an array where the rows (second-fastest moving index) are so aligned (this was my original use case, for fast image-blitting). Zach def aligned_empty(shape, dtype, dim_alignments, array_alignment): '''Allocate an empty array with the given shape and dtype, where the array buffer starts at a memory address evenly-divisible by array_alignment, and where items along each dimension are offset from the first item on that dimension by a byte offset that is an integer multiple of the corresponding value in the dim_alignments tuple. Example: To allocate a 20x30 array of floats32s, where individual data elements are aligned to 16-bute boundaries, each row is aligned to a 64-byte boundary, and the array's buffer starts on a 128-byte boundary, call: aligned_empty((20,30), numpy.float32, (64, 16), 128) ''' def aligned_rows_empty(shape, dtype, alignment, order='C'): '''Return an array where the rows (second-fastest-moving index) are aligned to byte boundaries evenly-divisible by 'alignment'. If 'order' is 'C', then the indexing is such that the fastest-moving index is the last one; if the order is 'F', then the fastest-moving index is the first.''' def aligned_elements_empty(shape, dtype, alignment, order='C'): '''Return an array where each element is aligned to byte boundaries evenly- divisible by 'alignment'.''' -------------- next part -------------- A non-text attachment was scrubbed... Name: aligned_allocator.py Type: text/x-python-script Size: 4449 bytes Desc: not available URL: -------------- next part -------------- From stefan at sun.ac.za Fri Apr 25 12:33:00 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 25 Apr 2008 18:33:00 +0200 Subject: [Numpy-discussion] numpy release In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <480FB04F.80006@noaa.gov> <4810DEA2.60501@noaa.gov> <9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com> <9457e7c80804250754n4967da64o1440164439407990@mail.gmail.com> Message-ID: <9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com> 2008/4/25 Alan G Isaac : > I must have misunderstood: > I thought the agreement was to > provisionally return a 1d array for x[0], > while we hashed through the other proposals. The agreement was: a) That x[0][0] should be equal to x[0,0] and b) That x[0,:] should be equal to x[0] (as for ndarrays) This breaks as little functionality as possible, at the cost of one (instead of two) API changes. We should now discuss the proposals on the table, choose a good one, and implement all the API changes necessary for 1.2 or 2. It's a pity we have to change the API again, but the current situation is not tenable. Cheers St?fan From peridot.faceted at gmail.com Fri Apr 25 12:52:55 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Fri, 25 Apr 2008 12:52:55 -0400 Subject: [Numpy-discussion] numpy release In-Reply-To: <9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <4810DEA2.60501@noaa.gov> <9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com> <9457e7c80804250754n4967da64o1440164439407990@mail.gmail.com> <9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com> Message-ID: On 25/04/2008, St?fan van der Walt wrote: > 2008/4/25 Alan G Isaac : > > > I must have misunderstood: > > I thought the agreement was to > > provisionally return a 1d array for x[0], > > while we hashed through the other proposals. > > The agreement was: > > a) That x[0][0] should be equal to x[0,0] and > b) That x[0,:] should be equal to x[0] (as for ndarrays) > > This breaks as little functionality as possible, at the cost of one > (instead of two) API changes. Hold on. There has definitely been some confusion here. This is not what I thought I was suggesting, or what Alan thought he was suggesting. I do not think special-casing matrices for which one dimension happens to be one is a good idea at all, even temporarily. This is the kind of thing that drives users crazy. My suggested stopgap fix was to make x[0] return a 1D *array*; I feel that this will result in less special-casing. In fact I wasn't aware that anyone had proposed the fix you implemented. Can we change the stopgap solution? > We should now discuss the proposals on the table, choose a good one, > and implement all the API changes necessary for 1.2 or 2. It's a pity > we have to change the API again, but the current situation is not > tenable. Yes, well, it really looks unlikely we will be able to agree on what the correct solution is before 1.1, so I would like to have something non-broken for that release. Anne From charlesr.harris at gmail.com Fri Apr 25 13:01:08 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 25 Apr 2008 11:01:08 -0600 Subject: [Numpy-discussion] numpy release In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <9457e7c80804231334g48d568dbif76c885c1637e14b@mail.gmail.com> <480FB04F.80006@noaa.gov> <4810DEA2.60501@noaa.gov> <4811ED49.9040900@gmail.com> Message-ID: On Fri, Apr 25, 2008 at 10:04 AM, Alan G Isaac wrote: > I think the use of the term 'vector' in this > thread is becoming a bit confusing. > > An M by N matrix is a vector. (I.e., it is > an element of a vector space.) > Sure, but the important thing is the multiplication. If it wasn't we could just flatten them all and be done with this discussion. Super bonus, no need for the '*' operator. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Fri Apr 25 13:04:53 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 25 Apr 2008 13:04:53 -0400 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com><480FB04F.80006@noaa.gov><4810DEA2.60501@noaa.gov><9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com><9457e7c80804250754n4967da64o1440164439407990@mail.gmail.com> <9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com> Message-ID: > 2008/4/25 Alan G Isaac : >> I must have misunderstood: >> I thought the agreement was to >> provisionally return a 1d array for x[0], >> while we hashed through the other proposals. On Fri, 25 Apr 2008, St?fan van der Walt apparently wrote: > The agreement was: > a) That x[0][0] should be equal to x[0,0] and > b) That x[0,:] should be equal to x[0] (as for ndarrays) 1. This is **not** what I understood as the agreement (and I think the current solution is bad). I certainly did not mean to support the change that as implemented, and it is not clear to me that others did either. 2. These two goals are substantially in conflict for matrices, as we are seeing. 3. The important goal, (goal a., which everyone agrees on), has NOT been accomplished by the current change: x[0][0] raises a TypeError when x is a 1 by N matrix. > This breaks as little functionality as possible, at the > cost of one (instead of two) API changes. I cannot agree with this assessment. > the current situation is not tenable. I agree with this! Better than the SVN behavior would be to raise an error in response to scalar indexing of matrices. I strongly hope that this behavior will not show up in a release! Please return 1d arrays in response to scalar indexing as the provisional arrangement. This is certainly better than the new behavior. Alan Isaac From peridot.faceted at gmail.com Fri Apr 25 13:03:37 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Fri, 25 Apr 2008 13:03:37 -0400 Subject: [Numpy-discussion] Generating Bell Curves (was: Using normal() ) In-Reply-To: References: Message-ID: On 24/04/2008, Rich Shepard wrote: > Thanks to several of you I produced test code using the normal density > function, and it does not do what we need. Neither does the Gaussian > function using fwhm that I've tried. The latter comes closer, but the ends > do not reach y=0 when the inflection point is y=0.5. > > So, let me ask the collective expertise here how to generate the curves > that we need. > > We need to generate bell-shaped curves given a midpoint, width (where y=0) > and inflection point (by default, y=0.5) where y is [0.0, 1.0], and x is > usually [0, 100], but can vary. Using the NumPy arange() function to produce > the x values (e.g, arange(0, 100, 0.1)), I need a function that will produce > the associated y values for a bell-shaped curve. These curves represent the > membership functions for fuzzy term sets, and generally adjacent curves > overlap where y=0.5. It would be a bonus to be able to adjust the skew and > kurtosis of the curves, but the controlling data would be the > center/midpoint and width, with defaults for inflection point, and other > parameters. > > I've been searching for quite some time without finding a solution that > works as we need it to work. First I should say, please don't call these "bell curves"! It is confusing people, since that usually means specifically a Gaussian. In fact it seems that you want something more usually called a "sigmoid", or just a curve with a particular shape. I would look at http://en.wikipedia.org/wiki/Sigmoid_function In particular, they point out that the integral of any smooth, positive, "bump-shaped" function will be a sigmoid. So dreaming up an appropriate bump-shaped function is one way to go. Alternatively, tou can look at polynomial fitting - if you want, say, a function that is 1 with derivative zero at 0, 0.5 with derivative -1 at x, and 0 with derivative 0 at 1, you can construct a unique degree-6 polynomial that does exactly that; there's a new tool, KroghInterpolator, in scipy svn that can do that for you. Anne From doutriaux1 at llnl.gov Fri Apr 25 13:04:10 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Fri, 25 Apr 2008 10:04:10 -0700 Subject: [Numpy-discussion] numpy release In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <4810DEA2.60501@noaa.gov> <9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com> <9457e7c80804250754n4967da64o1440164439407990@mail.gmail.com> <9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com> Message-ID: <48120F0A.6040704@llnl.gov> Anne Archibald wrote: > Yes, well, it really looks unlikely we will be able to agree on what > the correct solution is before 1.1, so I would like to have something > non-broken for that release. > +1 on that! From stefan at sun.ac.za Fri Apr 25 13:07:25 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 25 Apr 2008 19:07:25 +0200 Subject: [Numpy-discussion] "aligned" matrix / ctypes In-Reply-To: References: <3E5D8350-AF60-40C0-9E0F-74A3DBE6935B@yale.edu> <3d375d730804241145w6c55fed5mfc032a0b0a9f3d0f@mail.gmail.com> <3d375d730804241508v15b53f03gf44a1e6c20fba081@mail.gmail.com> <39340.85.167.148.49.1209089733.squirrel@webmail.uio.no> <1209090899.17497.9.camel@bbc8> Message-ID: <9457e7c80804251007k6529765du54a5e658d5e0bb75@mail.gmail.com> Robert, Can we check this in somewhere under numpy.core? It seems very useful. St?fan 2008/4/25 Zachary Pincus : > Hello all, > > Attached is code (plus tests) for allocating aligned arrays -- I think this > addresses all the requests in this thread, with regard to allowing for > different kinds of alignment. Thanks Robert and Anne for your help and > suggestions. Hopefully this will be useful. > > The core is a function for allocating arrays with totally arbitrary > alignment along each dimension (e.g. you could allocate an 10x20 array of > uint16's where each uint16 is aligned to 4-byte boundaries and each row of > 20 uint16's is aligned to 32-byte boundaries, and the entire buffer is > aligned to a 128-byte boundary.) I've also included helper functions for two > common cases: when you want everything aligned to a particular multiple > (every element, row, etc. as well as the whole buffer), and when you want an > array where the rows (second-fastest moving index) are so aligned (this was > my original use case, for fast image-blitting). > > Zach > > > def aligned_empty(shape, dtype, dim_alignments, array_alignment): > '''Allocate an empty array with the given shape and dtype, where the array > buffer starts at a memory address evenly-divisible by array_alignment, and > where items along each dimension are offset from the first item on that > dimension by a byte offset that is an integer multiple of the > corresponding > value in the dim_alignments tuple. > > Example: To allocate a 20x30 array of floats32s, where individual data > elements are aligned to 16-bute boundaries, each row is aligned to a > 64-byte > boundary, and the array's buffer starts on a 128-byte boundary, call: > aligned_empty((20,30), numpy.float32, (64, 16), 128) > ''' > > def aligned_rows_empty(shape, dtype, alignment, order='C'): > '''Return an array where the rows (second-fastest-moving index) are > aligned > to byte boundaries evenly-divisible by 'alignment'. If 'order' is 'C', > then > the indexing is such that the fastest-moving index is the last one; if the > order is 'F', then the fastest-moving index is the first.''' > > def aligned_elements_empty(shape, dtype, alignment, order='C'): > '''Return an array where each element is aligned to byte boundaries > evenly- > divisible by 'alignment'.''' > > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From millman at berkeley.edu Fri Apr 25 13:09:22 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Fri, 25 Apr 2008 12:09:22 -0500 Subject: [Numpy-discussion] numpy release In-Reply-To: <9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <4810DEA2.60501@noaa.gov> <9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com> <9457e7c80804250754n4967da64o1440164439407990@mail.gmail.com> <9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com> Message-ID: I was hoping to get NumPy 1.1 tagged today, but it seems very unlikely at this point. Unfortunately, I haven't followed the matrix discussion as closely as I would like, so I can't tell if there is anything so uncontroversial that it would make sense to change for the 1.1.0 release. If there is something that can be unanimously agreed on within the next 24 hours or so, I would be happy to have it included in 1.1. If not, I would rather see us move on to working on 1.2 and releasing 1.1 ASAP. If there is unanimous agreement on a minor fix, we will need document the changes on the release notes. So once an agreement is reached, could someone either send me some text to insert into the release notes or update the document themselves: http://projects.scipy.org/scipy/numpy/milestone/1.1.0 This also might be a good opportunity to start a new thread with a subject line like "1.2 matrices". The discussion about matrices in this thread makes it difficult for me to quickly see what still needs to done before the 1.1 release. And there may also be some members of the community who would be very interested in the matrices discussion, but aren't reading this thread. -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From aisaac at american.edu Fri Apr 25 13:23:15 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 25 Apr 2008 13:23:15 -0400 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com><480FB04F.80006@noaa.gov><4810DEA2.60501@noaa.gov><9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com><9457e7c80804250754n4967da64o1440164439407990@mail.gmail.com><9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com> Message-ID: On Fri, 25 Apr 2008, Jarrod Millman apparently wrote: > I would like, so I can't tell if there is anything so > uncontroversial that it would make sense to change for the > 1.1.0 release. I think it is clear from reactions that the revision r5072 to matrix behavior should NOT go into any release. > If there is something that can be unanimously agreed on > within the next 24 hours or so, I would be happy to have > it included in 1.1. I see 3 possibilities (in rank order of preference): 1. return 1d arrays in response to scalar indexing of matrices 2. raise an error on scalar indexing of matrices 3. simply revert the r5072 change. Since there is universal accord that x[0][0] == x[0,0] is desirable, and we will get that with option 1, I think it is the right provisional behavior. I believe that option 1 provides a step in the direction of the ultimate behavior, no matter what that will be. Cheers, Alan Isaac From stefan at sun.ac.za Fri Apr 25 13:27:41 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 25 Apr 2008 19:27:41 +0200 Subject: [Numpy-discussion] numpy release In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <4810DEA2.60501@noaa.gov> <9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com> <9457e7c80804250754n4967da64o1440164439407990@mail.gmail.com> <9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com> Message-ID: <9457e7c80804251027y5e71b45fueb0598da40690306@mail.gmail.com> 2008/4/25 Anne Archibald : > > The agreement was: Clearly that was only the agreement in my mind. I apologise if I jumped the gun. > > a) That x[0][0] should be equal to x[0,0] and > > b) That x[0,:] should be equal to x[0] (as for ndarrays) > > > > This breaks as little functionality as possible, at the cost of one > > (instead of two) API changes. > > Hold on. There has definitely been some confusion here. This is not > what I thought I was suggesting, or what Alan thought he was > suggesting. I do not think special-casing matrices for which one > dimension happens to be one is a good idea at all, even temporarily. > This is the kind of thing that drives users crazy. Apparently, not having x[0][0] == x[0,0] drives them even crazier :) I was trying to satisfy Alan's request without breaking the API in more than one place. The whole idea of matrices was that you always work with 2-dimensional arrays, even when you just extract a row. Unless you have a proper hierarchical container (as illustrated in the other patch I sent), returning a 1D array breaks at least some expectation. If that's what the matrix users want, though, then we should change it. > My suggested stopgap fix was to make x[0] return a 1D *array*; I feel > that this will result in less special-casing. In fact I wasn't aware > that anyone had proposed the fix you implemented. Can we change the > stopgap solution? Yes, of course. NumPy is a community project, after all. > > We should now discuss the proposals on the table, choose a good one, > > and implement all the API changes necessary for 1.2 or 2. It's a pity > > we have to change the API again, but the current situation is not > > tenable. > > Yes, well, it really looks unlikely we will be able to agree on what > the correct solution is before 1.1, so I would like to have something > non-broken for that release. Unless we agree on something, and soon, that won't happen. Your workaround would break x[0] == x[0,:], so we're just swapping one set of broken functionality for another. I'm starting to see Chris Barker's point; allowing x[0] is causing more problems than it is worth. On the other hand, how would you index into a vector (as in http://en.wikipedia.org/wiki/Vector_(spatial)) without it? Regards St?fan From stefan at sun.ac.za Fri Apr 25 13:35:47 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 25 Apr 2008 19:35:47 +0200 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <4810DEA2.60501@noaa.gov> <9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com> <9457e7c80804250754n4967da64o1440164439407990@mail.gmail.com> <9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com> Message-ID: <9457e7c80804251035l5439fee1x7724728a8d9ccf4b@mail.gmail.com> 2008/4/25 Alan G Isaac : > 1. This is **not** what I understood as the agreement > (and I think the current solution is bad). Reverted in r5084. Cheers St?fan From dineshbvadhia at hotmail.com Fri Apr 25 13:37:24 2008 From: dineshbvadhia at hotmail.com (Dinesh B Vadhia) Date: Fri, 25 Apr 2008 10:37:24 -0700 Subject: [Numpy-discussion] Does Unreasonable Matrix Behavior affect Scipy Sparse Message-ID: I had b = Ax working where A is sparse using scipy.sparse. I'm now using the latest svn and b = Ax is not working and returns garbage results. Nothing has changed except the latest svn. Any thoughts? Dinesh -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Fri Apr 25 13:40:29 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 25 Apr 2008 13:40:29 -0400 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com><480FB04F.80006@noaa.gov><4810DEA2.60501@noaa.gov><9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com><9457e7c80804250754n4967da64o1440164439407990@mail.gmail.com><9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com> Message-ID: On Fri, 25 Apr 2008, St?fan van der Walt wrote: > workaround would break x[0] == x[0,:] But there is not universal agreement that x[0] == x[0,:] is desirable. In contrast, there *is* universal agreement that x[0][0]==x[0,0] is desirable. Or so I've understood the discussion. As you know, my view is that nonscalar indexing should return submatrices, while scalar indexing should return 1d arrays. Thus I do not see x[0] != x[0,:] as breakage: it is desirable. Indeed, without it, we will have an untenable situation, under all the different proposals. So it is a goner one way or another. Cheers, Alan Isaac From stefan at sun.ac.za Fri Apr 25 13:40:54 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 25 Apr 2008 19:40:54 +0200 Subject: [Numpy-discussion] numpy release In-Reply-To: <9457e7c80804251027y5e71b45fueb0598da40690306@mail.gmail.com> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <4810DEA2.60501@noaa.gov> <9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com> <9457e7c80804250754n4967da64o1440164439407990@mail.gmail.com> <9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com> <9457e7c80804251027y5e71b45fueb0598da40690306@mail.gmail.com> Message-ID: <9457e7c80804251040g22facf5dp569dbd32fdb03466@mail.gmail.com> 2008/4/25 St?fan van der Walt : > I'm starting to see Chris Barker's point; allowing x[0] is causing > more problems than it is worth. On the other hand, how would you > index into a vector (as in > http://en.wikipedia.org/wiki/Vector_(spatial)) without it? To answer my own question: you wouldn't. Vectors won't exist, everything will be 2D, always. I could go with that. Cheers St?fan From david.huard at gmail.com Fri Apr 25 13:47:46 2008 From: david.huard at gmail.com (David Huard) Date: Fri, 25 Apr 2008 13:47:46 -0400 Subject: [Numpy-discussion] numpy release In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <85b5c3130804230532nf8e46d4qc7603ae8c0bb3137@mail.gmail.com> <9457e7c80804230621i5b53b0aaj9882a5d790939a3b@mail.gmail.com> <9457e7c80804231046w3a5a1b4dm41b546403f1c6e6f@mail.gmail.com> <91cf711d0804231320i64401ab3jdf0f16ec6dc66606@mail.gmail.com> <4810BE4C.8080508@enthought.com> <91cf711d0804241122t4c2e39f1q957c4c1f691ea4b8@mail.gmail.com> Message-ID: <91cf711d0804251047u6baf2ab8t894d8918819c119f@mail.gmail.com> 2008/4/24 Jarrod Millman : > On Thu, Apr 24, 2008 at 1:22 PM, David Huard wrote: > > Assuming we want the next version to : ignore values outside of range > and > > accept and return the bin edges instead of the left edges, here could be > the > > new signature for 1.1: > > h, edges = histogram(a, bins=10, normed=False, range=None, > normed=False, > > new=False) > > > > If new=False, return the histogram and the left edges, with a warning > that > > in the next version, the edges will be returned. If new=True, return the > > histogram and the edges. > > If range is given explicitly , raise a warning saying that in the next > > version, the outliers will be ignored. To ignore outliers, use > new=True. > > If bins is a sequence, raise an error saying that bins should be an > > integer. To use explicit edges, use new=True. > > > > In 1.2, set new=True as the default, and in 2.3, remove new altogether. > > +1 > That sounds fine to me assuming 2.3 is 1.3. > Indeed. Done in r5085. I added a bunch of tests, but I'd appreciate if someone could double check before the release. This is not the time to introduce new bugs. Hopefully this is the end of the histogram saga. David > -- > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.huard at gmail.com Fri Apr 25 13:55:01 2008 From: david.huard at gmail.com (David Huard) Date: Fri, 25 Apr 2008 13:55:01 -0400 Subject: [Numpy-discussion] numpy release In-Reply-To: <91cf711d0804251047u6baf2ab8t894d8918819c119f@mail.gmail.com> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <9457e7c80804230621i5b53b0aaj9882a5d790939a3b@mail.gmail.com> <9457e7c80804231046w3a5a1b4dm41b546403f1c6e6f@mail.gmail.com> <91cf711d0804231320i64401ab3jdf0f16ec6dc66606@mail.gmail.com> <4810BE4C.8080508@enthought.com> <91cf711d0804241122t4c2e39f1q957c4c1f691ea4b8@mail.gmail.com> <91cf711d0804251047u6baf2ab8t894d8918819c119f@mail.gmail.com> Message-ID: <91cf711d0804251055g28c7bfcbw8a9e2ee71f7a94@mail.gmail.com> 2008/4/25 David Huard : > 2008/4/24 Jarrod Millman : > > > On Thu, Apr 24, 2008 at 1:22 PM, David Huard wrote: > > > Assuming we want the next version to : ignore values outside of range > > and > > > accept and return the bin edges instead of the left edges, here could > > be the > > > new signature for 1.1: > > > h, edges = histogram(a, bins=10, normed=False, range=None, > > normed=False, > > > new=False) > > > > > > If new=False, return the histogram and the left edges, with a warning > > that > > > in the next version, the edges will be returned. If new=True, return > > the > > > histogram and the edges. > > > If range is given explicitly , raise a warning saying that in the > > next > > > version, the outliers will be ignored. To ignore outliers, use > > new=True. > > > If bins is a sequence, raise an error saying that bins should be an > > > integer. To use explicit edges, use new=True. > > > > > > In 1.2, set new=True as the default, and in 2.3, remove new > > altogether. > > > > +1 > > That sounds fine to me assuming 2.3 is 1.3. > > > > Indeed. > > Done in r5085. I added a bunch of tests, but I'd appreciate if someone > could double check before the release. This is not the time to introduce new > bugs. > > Hopefully this is the end of the histogram saga. > Well, it's not... there is still an issue... give me a couple of minutes to fix it. > > David > > > > -- > > Jarrod Millman > > Computational Infrastructure for Research Labs > > 10 Giannini Hall, UC Berkeley > > phone: 510.643.4014 > > http://cirl.berkeley.edu/ > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Fri Apr 25 14:02:02 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 25 Apr 2008 14:02:02 -0400 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <9457e7c80804251035l5439fee1x7724728a8d9ccf4b@mail.gmail.com> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com><4810DEA2.60501@noaa.gov><9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com><9457e7c80804250754n4967da64o1440164439407990@mail.gmail.com><9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com> <9457e7c80804251035l5439fee1x7724728a8d9ccf4b@mail.gmail.com> Message-ID: On Fri, 25 Apr 2008, St?fan van der Walt apparently wrote: > Reverted in r5084. Thank you. I think we have discovered that there is a basic conflict between two behaviors: x[0] == x[0,:] vs. x[0][0] == x[0,0] To my recollection, everyone has agree that the second behavior is desirable as a *basic expectation* about the behavior of 2d objects. This implies that *eventually* we will have ``x[0][0] == x[0,0]``. But then eventually we MUST eventually have x[0] != x[0,:]. Since a current behavior must disappear eventually, we should make it disappear as soon as possible: before the release. The question is how. I see two simple ways to move forward immediately: 1. scalar indexing of matrices returns a 1d array 2. scalar indexing of matrices raises a TypeError If we raise a TypeError, we lose both behaviors. If we return a 1d array, we retain the behavior that has been seen as desirable (as a *basic expectation* about the behavior of 2d objects). Stefan, do you agree, and if so, would you be willing to have a vote on these two options? Alan From aisaac at american.edu Fri Apr 25 14:03:37 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 25 Apr 2008 14:03:37 -0400 Subject: [Numpy-discussion] Does Unreasonable Matrix Behavior affect Scipy Sparse In-Reply-To: References: Message-ID: On Fri, 25 Apr 2008, Dinesh B Vadhia apparently wrote: > where A is sparse using scipy.sparse. ... I'm now using > the latest svn and b = Ax 1. Please post a small example. 2. Do you have the *very* latest SVN (post r5084)? Cheers, Alan Isaac From robert.kern at gmail.com Fri Apr 25 14:41:14 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 25 Apr 2008 13:41:14 -0500 Subject: [Numpy-discussion] Generating Bell Curves (was: Using normal() ) In-Reply-To: References: Message-ID: <3d375d730804251141s550d1d06o82c5097ac3fc2139@mail.gmail.com> On Fri, Apr 25, 2008 at 8:32 AM, Rich Shepard wrote: > On Thu, 24 Apr 2008, Keith Goodman wrote: > > > A Gaussian never reaches zero. > > Keith, > > I know, and that's why I need to find another way to draw these curves. > While mathematically any 'y' value < 0.2 (the default) is equivalent to > zero, the curves must reach zero in the figures. > > Briefly, this model is used in regulatory compliance that may also end up > in legal proceedings. While the results are robust and not affected by the > curve not reaching zero, the appearance can cause problems in the regulatory > and legal arenas. They would be a distraction, a red herring, and consume > resources to explain. All this can be avoided by bell curves that reach > y=0.0 at the ends. In that case, you need to search the literature of your field for precise details on how to construct the curve that you want. As Anne notes, "bell-shaped curve," while seemingly generic, usually specifies Gaussians, and Gaussians do not have the properties you need. There are any number of curves which we could (and have) suggested as "looking bell-shaped," but if your field is as finicky as you say it is, there must be literature giving these details. I don't think any of us are familiar with that literature. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From rshepard at appl-ecosys.com Fri Apr 25 14:48:59 2008 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Fri, 25 Apr 2008 11:48:59 -0700 (PDT) Subject: [Numpy-discussion] Generating Bell Curves (was: Using normal() ) In-Reply-To: <3d375d730804251141s550d1d06o82c5097ac3fc2139@mail.gmail.com> References: <3d375d730804251141s550d1d06o82c5097ac3fc2139@mail.gmail.com> Message-ID: On Fri, 25 Apr 2008, Robert Kern wrote: > In that case, you need to search the literature of your field for precise > details on how to construct the curve that you want. Robert, Considering how few of us work in this subject area there's not much in the way of resources. Regardless, for now the working Gaussian code will do quite well. As time permits I'll see what I can find (or create) to produce a curve that reaches zero at the bounds. Many thanks, all, Rich -- Richard B. Shepard, Ph.D. | Integrity Credibility Applied Ecosystem Services, Inc. | Innovation Voice: 503-667-4517 Fax: 503-667-8863 From robert.kern at gmail.com Fri Apr 25 14:51:37 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 25 Apr 2008 13:51:37 -0500 Subject: [Numpy-discussion] "aligned" matrix / ctypes In-Reply-To: References: <3E5D8350-AF60-40C0-9E0F-74A3DBE6935B@yale.edu> <3d375d730804241145w6c55fed5mfc032a0b0a9f3d0f@mail.gmail.com> <3d375d730804241508v15b53f03gf44a1e6c20fba081@mail.gmail.com> <39340.85.167.148.49.1209089733.squirrel@webmail.uio.no> <1209090899.17497.9.camel@bbc8> Message-ID: <3d375d730804251151r474149efl781211f0f55762b1@mail.gmail.com> On Fri, Apr 25, 2008 at 11:09 AM, Zachary Pincus wrote: > Hello all, > > Attached is code (plus tests) for allocating aligned arrays -- I think this > addresses all the requests in this thread, with regard to allowing for > different kinds of alignment. Thanks Robert and Anne for your help and > suggestions. Hopefully this will be useful. > > The core is a function for allocating arrays with totally arbitrary > alignment along each dimension (e.g. you could allocate an 10x20 array of > uint16's where each uint16 is aligned to 4-byte boundaries and each row of > 20 uint16's is aligned to 32-byte boundaries, and the entire buffer is > aligned to a 128-byte boundary.) I've also included helper functions for two > common cases: when you want everything aligned to a particular multiple > (every element, row, etc. as well as the whole buffer), and when you want an > array where the rows (second-fastest moving index) are so aligned (this was > my original use case, for fast image-blitting). There is one more important case, which is aligning just the beginning of the array. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From millman at berkeley.edu Fri Apr 25 14:55:21 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Fri, 25 Apr 2008 13:55:21 -0500 Subject: [Numpy-discussion] numpy release In-Reply-To: <91cf711d0804251055g28c7bfcbw8a9e2ee71f7a94@mail.gmail.com> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <9457e7c80804231046w3a5a1b4dm41b546403f1c6e6f@mail.gmail.com> <91cf711d0804231320i64401ab3jdf0f16ec6dc66606@mail.gmail.com> <4810BE4C.8080508@enthought.com> <91cf711d0804241122t4c2e39f1q957c4c1f691ea4b8@mail.gmail.com> <91cf711d0804251047u6baf2ab8t894d8918819c119f@mail.gmail.com> <91cf711d0804251055g28c7bfcbw8a9e2ee71f7a94@mail.gmail.com> Message-ID: On Fri, Apr 25, 2008 at 12:55 PM, David Huard wrote: > > Done in r5085. I added a bunch of tests, but I'd appreciate if someone > could double check before the release. This is not the time to introduce new > bugs. > > > > Hopefully this is the end of the histogram saga. > > > > Well, it's not... there is still an issue... give me a couple of minutes to > fix it. Hey David, Excellent! I have one more request: Could you either send me a little blurb for the release notes or go ahead and edit it yourself: http://projects.scipy.org/scipy/numpy/milestone/1.1.0 Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From robert.kern at gmail.com Fri Apr 25 14:59:05 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 25 Apr 2008 13:59:05 -0500 Subject: [Numpy-discussion] "aligned" matrix / ctypes In-Reply-To: <9457e7c80804251007k6529765du54a5e658d5e0bb75@mail.gmail.com> References: <3E5D8350-AF60-40C0-9E0F-74A3DBE6935B@yale.edu> <3d375d730804241145w6c55fed5mfc032a0b0a9f3d0f@mail.gmail.com> <3d375d730804241508v15b53f03gf44a1e6c20fba081@mail.gmail.com> <39340.85.167.148.49.1209089733.squirrel@webmail.uio.no> <1209090899.17497.9.camel@bbc8> <9457e7c80804251007k6529765du54a5e658d5e0bb75@mail.gmail.com> Message-ID: <3d375d730804251159u682c835esd79d19316db11770@mail.gmail.com> On Fri, Apr 25, 2008 at 12:07 PM, St?fan van der Walt wrote: > Robert, > > Can we check this in somewhere under numpy.core? It seems very useful. Yes, but I want 1.1.0 out first before adding new features. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From zachary.pincus at yale.edu Fri Apr 25 15:03:46 2008 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Fri, 25 Apr 2008 15:03:46 -0400 Subject: [Numpy-discussion] "aligned" matrix / ctypes In-Reply-To: <3d375d730804251151r474149efl781211f0f55762b1@mail.gmail.com> References: <3E5D8350-AF60-40C0-9E0F-74A3DBE6935B@yale.edu> <3d375d730804241145w6c55fed5mfc032a0b0a9f3d0f@mail.gmail.com> <3d375d730804241508v15b53f03gf44a1e6c20fba081@mail.gmail.com> <39340.85.167.148.49.1209089733.squirrel@webmail.uio.no> <1209090899.17497.9.camel@bbc8> <3d375d730804251151r474149efl781211f0f55762b1@mail.gmail.com> Message-ID: <7976AC14-09D1-4F87-945C-71A4382D1990@yale.edu> > There is one more important case, which is aligning just the beginning > of the array. Indeed -- thanks! The following should take care of that case (it's all fluff around calling aligned_empty() with a dim_alignments tuple of all 1's, and the array_alignment parameter as required): def aligned_start_empty(shape, dtype, alignment, order='C'): '''Return an array with the first element aligned to a byte that is evenly-divisible by the specified alignment.''' order = order.upper() if order not in ('C', 'F'): raise ValueError("Order must be 'C' or 'F'.") dim_alignments = [1 for dim in shape] if order == 'F': shape = shape[::-1] return aligned_empty(shape, dtype, dim_alignments, alignment).T else: return aligned_empty(shape, dtype, dim_alignments, alignment) Zach From bsouthey at gmail.com Fri Apr 25 15:06:30 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Fri, 25 Apr 2008 14:06:30 -0500 Subject: [Numpy-discussion] Generating Bell Curves (was: Using normal() ) In-Reply-To: References: <3d375d730804251141s550d1d06o82c5097ac3fc2139@mail.gmail.com> Message-ID: <48122BB6.9050607@gmail.com> Rich Shepard wrote: > On Fri, 25 Apr 2008, Robert Kern wrote: > > >> In that case, you need to search the literature of your field for precise >> details on how to construct the curve that you want. >> > > Robert, > > Considering how few of us work in this subject area there's not much in > the way of resources. > > Regardless, for now the working Gaussian code will do quite well. As time > permits I'll see what I can find (or create) to produce a curve that reaches > zero at the bounds. > > Many thanks, all, > > Rich > > Hi, Just use a truncated distribution as these are well known: http://en.wikipedia.org/wiki/Truncated_distribution http://en.wikipedia.org/wiki/Truncated_normal_distribution Bruce From charlesr.harris at gmail.com Fri Apr 25 15:07:17 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 25 Apr 2008 13:07:17 -0600 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com> <9457e7c80804250754n4967da64o1440164439407990@mail.gmail.com> <9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com> <9457e7c80804251035l5439fee1x7724728a8d9ccf4b@mail.gmail.com> Message-ID: On Fri, Apr 25, 2008 at 12:02 PM, Alan G Isaac wrote: > On Fri, 25 Apr 2008, St?fan van der Walt apparently wrote: > > Reverted in r5084. > > Thank you. > > I think we have discovered that there is a basic conflict > between two behaviors: > > x[0] == x[0,:] > vs. > x[0][0] == x[0,0] > > To my recollection, everyone has agree that the second > behavior is desirable as a *basic expectation* about the > behavior of 2d objects. This implies that *eventually* we > will have ``x[0][0] == x[0,0]``. But then eventually > we MUST eventually have x[0] != x[0,:]. > Choices: 1) x[0] equiv x[0:1,:] -- current behavior 2) x[0] equiv x[0,:] -- array behavior, gives x[0][0] == x[0,0] These are incompatible. I think 1) is more convenient in the matrix domain and should be kept, while 2) should be given up. What is the reason for wanting 2)? Maybe we could overload the () operator so that x(0) gives 2)? Or vice versa. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Apr 25 15:13:01 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 25 Apr 2008 13:13:01 -0600 Subject: [Numpy-discussion] Generating Bell Curves (was: Using normal() ) In-Reply-To: References: <3d375d730804251141s550d1d06o82c5097ac3fc2139@mail.gmail.com> Message-ID: On Fri, Apr 25, 2008 at 12:48 PM, Rich Shepard wrote: > On Fri, 25 Apr 2008, Robert Kern wrote: > > > In that case, you need to search the literature of your field for precise > > details on how to construct the curve that you want. > > Robert, > > Considering how few of us work in this subject area there's not much in > the way of resources. > > Regardless, for now the working Gaussian code will do quite well. As time > permits I'll see what I can find (or create) to produce a curve that > reaches > zero at the bounds. > You can use something like f(x) = (1-x**2)**2 , which has inflection points and vanishes at +/- 1. Any of the B-splines will also do the trick. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Fri Apr 25 15:19:43 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Fri, 25 Apr 2008 21:19:43 +0200 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com> <9457e7c80804250754n4967da64o1440164439407990@mail.gmail.com> <9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com> <9457e7c80804251035l5439fee1x7724728a8d9ccf4b@mail.gmail.com> Message-ID: On 25/04/2008, Charles R Harris wrote: > > > On Fri, Apr 25, 2008 at 12:02 PM, Alan G Isaac wrote: > > I think we have discovered that there is a basic conflict > > between two behaviors: > > > > x[0] == x[0,:] > > vs. > > > > x[0][0] == x[0,0] > > > > To my recollection, everyone has agree that the second > > behavior is desirable as a *basic expectation* about the > > behavior of 2d objects. This implies that *eventually* we > > will have ``x[0][0] == x[0,0]``. But then eventually > > we MUST eventually have x[0] != x[0,:]. > > > Choices: > > 1) x[0] equiv x[0:1,:] -- current behavior > 2) x[0] equiv x[0,:] -- array behavior, gives x[0][0] == x[0,0] > > These are incompatible. I think 1) is more convenient in the matrix domain > and should be kept, while 2) should be given up. What is the reason for > wanting 2)? Maybe we could overload the () operator so that x(0) gives 2)? > Or vice versa. We are currently operating in a fog of abstraction, trying to decide which operations are more "natural" or more in correspondence with our imagined idea of a user. We're not getting anywhere. Let's start writing up (perhaps on the matrix indexing wiki page) plausible use cases for scalar indexing, and how they would look under each proposal. For example: * Iterating over the rows of a matrix: # array-return proposal; x is a scalar for row in M: if np.all(row==0): raise ValueError # exception-raising proposal for i in xrange(M.shape[0]): if np.all(M[i,:]==0): raise ValueError Other applications? Anne From rshepard at appl-ecosys.com Fri Apr 25 15:22:20 2008 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Fri, 25 Apr 2008 12:22:20 -0700 (PDT) Subject: [Numpy-discussion] Generating Bell Curves (was: Using normal() ) In-Reply-To: <48122BB6.9050607@gmail.com> References: <3d375d730804251141s550d1d06o82c5097ac3fc2139@mail.gmail.com> <48122BB6.9050607@gmail.com> Message-ID: On Fri, 25 Apr 2008, Bruce Southey wrote: > Just use a truncated distribution as these are well known: > http://en.wikipedia.org/wiki/Truncated_distribution > http://en.wikipedia.org/wiki/Truncated_normal_distribution Bruce, I considered the truncated normal distribution, but having the tails of the Gaussian distribution above zero is more acceptable. Thanks very much for the suggestion, Rich -- Richard B. Shepard, Ph.D. | Integrity Credibility Applied Ecosystem Services, Inc. | Innovation Voice: 503-667-4517 Fax: 503-667-8863 From rshepard at appl-ecosys.com Fri Apr 25 15:25:57 2008 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Fri, 25 Apr 2008 12:25:57 -0700 (PDT) Subject: [Numpy-discussion] Generating Bell Curves (was: Using normal() ) In-Reply-To: References: <3d375d730804251141s550d1d06o82c5097ac3fc2139@mail.gmail.com> Message-ID: On Fri, 25 Apr 2008, Charles R Harris wrote: > You can use something like f(x) = (1-x**2)**2 , which has inflection > points and vanishes at +/- 1. Any of the B-splines will also do the trick. Chuck, Thank you. I need to make some time to understand the B-splines to use them appropriately. Unfortunately, my mathematical statistics learning was many years in the past ... but we had moved ahead of writing on clay tablets by that time. Not needing to retain that knowledge for many years means it was replaced by more pressing current knowledge. The B-splines do look promising, though. Rich -- Richard B. Shepard, Ph.D. | Integrity Credibility Applied Ecosystem Services, Inc. | Innovation Voice: 503-667-4517 Fax: 503-667-8863 From Chris.Barker at noaa.gov Fri Apr 25 15:32:44 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 25 Apr 2008 12:32:44 -0700 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <480FB04F.80006@noaa.gov> <4810DEA2.60501@noaa.gov> <9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com> <9457e7c80804250754n4967da64o1440164439407990@mail.gmail.com> <9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com> Message-ID: <481231DC.2010900@noaa.gov> Alan G Isaac wrote: > Please return 1d arrays in response to scalar > indexing as the provisional arrangement. +1 (for the provisional solution) This has clarified it a bit for me. The current situation allows one to create "row vectors" and "column vectors" by generating matrices (in various ways) that happen to be shape (1,n) or (n,1). These objects behave as we all want in arithmetic operations (*, **). So what's missing: ***** The ability to index one of these "vector" objects by a single index, and get a scalar as a result. ***** If I'm not mistaken (and I probably am) -- that is the crux of this entire conversation. Do we want that? I would say yes, it's a pretty natural and common thing to want. Note that with the current version of matrix (numpy 1.1.0rc1), you can index a (n,1) matrix with a scalar, and get a (1,1) matrix, but you get an exception if you try to index a (1,n) matrix with a scalar: >>> rv matrix([[3, 4, 5]]) >>> rv[1] Traceback (most recent call last): File "", line 1, in File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/core/defmatrix.py", line 228, in __getitem__ out = N.ndarray.__getitem__(self, index) IndexError: index out of bounds I know why that's happening, but I think it's a demonstration of why we might want some sort of "vector" object, for a cleaner API. How do we achieve this? One possibility is to special case (n,1) and (1,n) matrices, but I think there is consensus that that is a BAD IDEA (tm) Which leaves us with needing a "vector" object -- which acts a lot like a 1-d array, except that it acts like a (n,1) or (1,n) matrix with * and **, and may get printed differently. I think there are two ways to accomplish this: (1) RowVector or ColumnVector objects that are essentially 1-d ndarrays, with a few methods overridden. (2) A Vector object that is a matrix of either (n,1) or (1,n) in shape, with a couple methods overridden. I think the difference is an implementation detail. They should behave about the same either way. In either case, we should probably have factory functions for these objects: RowVector() and ColumnVector() By the way, I think this brings us closer to the goal of matrices "acting as much like arrays as possible" -- when you index into a 2-d array: A[:,i], you get a 1-d array. These "vectors" would act a lot like 1-d arrays, so that would be more similar than the current situation. It also might be nice to have iterators over rows and columns, like: for row in M.rows: ... and for col in M.columns: ... We can get the rows as the default iterator, but it feels nice and symmetric to be able to iterate over columns too (though I have no idea how useful that would really be) St?fan van der Walt wrote: > The whole idea of matrices was that you always > work with 2-dimensional arrays, even when you just extract a row. > Unless you have a proper hierarchical container (as illustrated in the > other patch I sent), returning a 1D array breaks at least some > expectation. I agree -- stopgap is OK (but having people do M.A[i] is a fine stopgap too), but if we really want this, we need some sort of "vector" object. > If that's what the matrix users want, though, then we > should change it. I still want to see some linear algebra examples -- that's what matrices are for. We can iterate over 2-d arrays and get 1-d arrays just fine now, if that's what you want. > Unless we agree on something, and soon, that won't happen. Your > workaround would break x[0] == x[0,:], so we're just swapping one set > of broken functionality for another. Alan G Isaac wrote: > But then we MUST eventually have x[0] != x[0,:]. I don't think so -- I think a Vector object would allow both of: M[i,j] == M[i][j] and M[i] == M[i,:] however, M[i] != M[:,i] -- M[i] can only mean one thing, and this doesn't hold for arrays, either. and (if we do the iterators): M.rows[i] == M[i,:] M.columns[i] == M[:,i] You can get rows a bit more conveniently than columns, which is really just an accident of how C arranges memory, but why not? Again, to make this "pure" we'd disallow M[i], but practicality beats purity, after all. > how would you > index into a vector (as in http://en.wikipedia.org/wiki/Vector_(spatial)) without it? Right -- which is why a 1-d "vector" object has its uses. Alan G Isaac wrote: > Since a current behavior must disappear eventually, we > should make it disappear as soon as possible: before the > release. The question is how. I see two simple ways to > move forward immediately: > > 1. scalar indexing of matrices returns a 1d array > 2. scalar indexing of matrices raises a TypeError I don't agree -- what we have is what we have, not changing it will not break any existing code, so we should only make a change if it moves us closer to what we want in the future -- it does seem that most folks do want M[i] to return some sort of 1-d object, so I'm fine with (1). I'm also fine with (2), but I'm pretty sure that one was laughed out of the discussion a good while back! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From charlesr.harris at gmail.com Fri Apr 25 16:15:08 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 25 Apr 2008 14:15:08 -0600 Subject: [Numpy-discussion] numpy release In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <9457e7c80804231046w3a5a1b4dm41b546403f1c6e6f@mail.gmail.com> <91cf711d0804231320i64401ab3jdf0f16ec6dc66606@mail.gmail.com> <4810BE4C.8080508@enthought.com> <91cf711d0804241122t4c2e39f1q957c4c1f691ea4b8@mail.gmail.com> <91cf711d0804251047u6baf2ab8t894d8918819c119f@mail.gmail.com> <91cf711d0804251055g28c7bfcbw8a9e2ee71f7a94@mail.gmail.com> Message-ID: On Fri, Apr 25, 2008 at 12:55 PM, Jarrod Millman wrote: > On Fri, Apr 25, 2008 at 12:55 PM, David Huard > wrote: > > > Done in r5085. I added a bunch of tests, but I'd appreciate if someone > > could double check before the release. This is not the time to introduce > new > > bugs. > > > > > > Hopefully this is the end of the histogram saga. > > > > > This one? ERROR: Ticket #632 ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/numpy/core/tests/test_regression.py", line 812, in check_hist_bins_as_list hist,edges = np.histogram([1,2,3,4],[1,2]) File "/usr/lib/python2.5/site-packages/numpy/lib/function_base.py", line 184, in histogram raise ValueError, 'Use new=True to pass bin edges explicitly.' ValueError: Use new=True to pass bin edges explicitly. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Fri Apr 25 16:35:09 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 25 Apr 2008 13:35:09 -0700 Subject: [Numpy-discussion] OSX installer: please test In-Reply-To: <1209115462.17497.14.camel@bbc8> References: <764e38540804161436j5ce7a468w8bea6a3c3598d69e@mail.gmail.com> <480F9200.9060803@noaa.gov> <1209115462.17497.14.camel@bbc8> Message-ID: <4812407D.60003@noaa.gov> David Cournapeau wrote: > To get to your point: hdiutil is the command you are looking for. yup. Here's an example: hdiutil create -srcfolder YourDir -volname "A Name" -ov "Something.dmg" It will build a disk image from the directory: YourDir -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From dineshbvadhia at hotmail.com Fri Apr 25 16:46:52 2008 From: dineshbvadhia at hotmail.com (Dinesh B Vadhia) Date: Fri, 25 Apr 2008 13:46:52 -0700 Subject: [Numpy-discussion] Does Unreasonable Matrix Behavior affectScipy Sparse References: Message-ID: Alan I posted this on the scipy list: I have a working program with b=Ax, where A is a large sparse matrix. However, I need the int8 support in the sparse library to utilize much larger matrices. I managed to get hold of a numpy svn 5066 and scipy svn 4167 build, and b=Ax now returns garbage results. Nothing was changed in the program except replacing getrow with .todense() and I checked these to make sure the right rows were being picked up. Wish I could point out where exactly the problem lies but there was no Traceback just the wrong results but it can safely be assumed it is with the numpy/scipy matrix support. Any ideas? Dinesh ----- Original Message ----- From: Alan G Isaac To: Discussion of Numerical Python Sent: Friday, April 25, 2008 11:03 AM Subject: Re: [Numpy-discussion] Does Unreasonable Matrix Behavior affectScipy Sparse On Fri, 25 Apr 2008, Dinesh B Vadhia apparently wrote: > where A is sparse using scipy.sparse. ... I'm now using > the latest svn and b = Ax 1. Please post a small example. 2. Do you have the *very* latest SVN (post r5084)? Cheers, Alan Isaac -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Fri Apr 25 16:57:30 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 25 Apr 2008 16:57:30 -0400 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <481231DC.2010900@noaa.gov> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com><480FB04F.80006@noaa.gov><4810DEA2.60501@noaa.gov><9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com><9457e7c80804250754n4967da64o1440164439407990@mail.gmail.com><9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com> <481231DC.2010900@noaa.gov> Message-ID: On Fri, 25 Apr 2008, Christopher Barker apparently wrote: > I think a Vector object would allow both of: > M[i,j] == M[i][j] > and > M[i] == M[i,:] The problem is that it would be a crime to give up the natural production of submatrices. The NATURAL RULE is: to get a submatrix, use nonscalar indices. We should *not* give up that x[0,:] is a sub*matrix* whose first element is x[0,0] and equivalently x[0][0]. *This* is why we must have x[0]!=x[0,:] if we want, as we do, that x[0][0]==x[0,0]. Note that the idea for attributes ``rows`` and ``columns`` is contained on the discussion page: I claim that it is natural for these attributes to yield properly shaped *matrices*. Once we have these attributes, it is difficult to see what we gain by introducing the complication of a separate vector class. I still see no real gain from a separate vector class (or row and column vector classes). Everything that has been proposed to do with these is achievable with the corresponding matrices with one exception: scalar indexing -> element so that x[0][0]==x[0,0]. But this outcome is achieved MUCH more simply by letting x[0] be a 1d array. Anyway, I am glad you agree that letting ``x[0]`` be a 1d array is the proper provisional solution. Cheers, Alan From david.huard at gmail.com Fri Apr 25 17:04:26 2008 From: david.huard at gmail.com (David Huard) Date: Fri, 25 Apr 2008 17:04:26 -0400 Subject: [Numpy-discussion] numpy release In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <91cf711d0804231320i64401ab3jdf0f16ec6dc66606@mail.gmail.com> <4810BE4C.8080508@enthought.com> <91cf711d0804241122t4c2e39f1q957c4c1f691ea4b8@mail.gmail.com> <91cf711d0804251047u6baf2ab8t894d8918819c119f@mail.gmail.com> <91cf711d0804251055g28c7bfcbw8a9e2ee71f7a94@mail.gmail.com> Message-ID: <91cf711d0804251404wa558c44vff8ad490c2d33f6f@mail.gmail.com> Thanks Chuck, I didn't know there were other tests for histogram outside of test_function_base. The error is now raised only if bins are passed explicitly and normed=True. David 2008/4/25 Charles R Harris : > > > On Fri, Apr 25, 2008 at 12:55 PM, Jarrod Millman > wrote: > > > On Fri, Apr 25, 2008 at 12:55 PM, David Huard > > wrote: > > > > Done in r5085. I added a bunch of tests, but I'd appreciate if > > someone > > > could double check before the release. This is not the time to > > introduce new > > > bugs. > > > > > > > > Hopefully this is the end of the histogram saga. > > > > > > > > > > This one? > > ERROR: Ticket #632 > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/usr/lib/python2.5/site-packages/numpy/core/tests/test_regression.py", line > 812, in check_hist_bins_as_list > hist,edges = np.histogram([1,2,3,4],[1,2]) > File "/usr/lib/python2.5/site-packages/numpy/lib/function_base.py", line > 184, in histogram > raise ValueError, 'Use new=True to pass bin edges explicitly.' > ValueError: Use new=True to pass bin edges explicitly. > > Chuck > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Apr 25 17:43:29 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 25 Apr 2008 15:43:29 -0600 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com> <9457e7c80804250754n4967da64o1440164439407990@mail.gmail.com> <9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com> <481231DC.2010900@noaa.gov> Message-ID: On Fri, Apr 25, 2008 at 2:57 PM, Alan G Isaac wrote: > On Fri, 25 Apr 2008, Christopher Barker apparently wrote: > > I think a Vector object would allow both of: > > M[i,j] == M[i][j] > > and > > M[i] == M[i,:] > > The problem is that it would be a crime to give up > the natural production of submatrices. The NATURAL RULE > is: to get a submatrix, use nonscalar indices. > We should *not* give up that x[0,:] is a sub*matrix* > whose first element is x[0,0] and equivalently x[0][0]. > *This* is why we must have x[0]!=x[0,:] if we want, > as we do, that x[0][0]==x[0,0]. > I would give it up and let x(0) be a submatrix, the normal indexing is then exactly like an array. True, the result of x(0) can't be used as an lvalue (only getitem equivalence), but otherwise it should work fine. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Apr 25 17:46:10 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 25 Apr 2008 15:46:10 -0600 Subject: [Numpy-discussion] Does Unreasonable Matrix Behavior affectScipy Sparse In-Reply-To: References: Message-ID: On Fri, Apr 25, 2008 at 2:46 PM, Dinesh B Vadhia wrote: > Alan > > I posted this on the scipy list: > > I have a working program with b=Ax, where A is a large sparse matrix. > However, I need the int8 support in the sparse library to utilize much > larger matrices. I managed to get hold of a numpy svn 5066 and scipy svn > 4167 build, and b=Ax now returns garbage results. Nothing was changed in > the program except replacing getrow with todense() and I checked these to > make sure the right rows were being picked up. > Wish I could point out where exactly the problem lies but there was no > Traceback just the wrong results but it can safely be assumed it is with the > numpy/scipy matrix support. > > Any ideas? > > Dinesh, what is the svn revision you are using? 'Latest' doen't tell us much because it changes all the time. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Fri Apr 25 19:17:17 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 25 Apr 2008 19:17:17 -0400 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com><9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com><9457e7c80804250754n4967da64o1440164439407990@mail.gmail.com><9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com><481231DC.2010900@noaa.gov> Message-ID: >> On Fri, 25 Apr 2008, Christopher Barker apparently wrote: >>> I think a Vector object would allow both of: >>> M[i,j] == M[i][j] >>> and >>> M[i] == M[i,:] > On Fri, Apr 25, 2008 at 2:57 PM, Alan G Isaac > wrote: >> The problem is that it would be a crime to give up >> the natural production of submatrices. The NATURAL RULE >> is: to get a submatrix, use nonscalar indices. >> We should not give up that x[0,:] is a sub*matrix* >> whose first element is x[0,0] and equivalently x[0][0]. >> This is why we must have x[0]!=x[0,:] if we want, >> as we do, that x[0][0]==x[0,0]. On Fri, 25 Apr 2008, Charles R Harris apparently wrote: > I would give it up and let x(0) be a submatrix, the normal > indexing is then exactly like an array. True, the result > of x(0) can't be used as an lvalue (only getitem > equivalence), but otherwise it should work fine. The "it" here is ambiguous. Which do you want to give up? - x[0][0]==x[0,0] ? the argument has been that this is a fundamental expectations for 2d array-like objects, and I think that argument is right - x[0]==x[0,:] ? I believe you mean we should give this up, and if so, I strongly agree. As you know. Enforcing this is proving untenable and costly. Cheers, Alan Isaac From aisaac at american.edu Fri Apr 25 19:17:19 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 25 Apr 2008 19:17:19 -0400 Subject: [Numpy-discussion] Does Unreasonable Matrix Behavior affectScipy Sparse In-Reply-To: References: Message-ID: >> 1. Please post a small example. 2. Do you have the very >> latest SVN (post r5084)? On Fri, 25 Apr 2008, Dinesh B Vadhia apparently wrote: > I have a working program with b=Ax, where A is a large > sparse matrix. However, I need the int8 support in the > sparse library to utilize much larger matrices. I managed > to get hold of a numpy svn 5066 and scipy svn 4167 build, > and b=Ax now returns garbage results. Nothing was changed > in the program except replacing getrow with .todense() and > I checked these to make sure the right rows were being > picked up. > Wish I could point out where exactly the problem lies but > there was no Traceback just the wrong results but it can > safely be assumed it is with the numpy/scipy matrix > support. You need to post a small example (actual code) that generates the error. But first update NumPy from SVN and confirm that th problem still exists. Cheers, Alan Isaac From gael.varoquaux at normalesup.org Fri Apr 25 19:25:30 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sat, 26 Apr 2008 01:25:30 +0200 Subject: [Numpy-discussion] Generating Bell Curves (was: Using normal() ) In-Reply-To: <3d375d730804251141s550d1d06o82c5097ac3fc2139@mail.gmail.com> References: <3d375d730804251141s550d1d06o82c5097ac3fc2139@mail.gmail.com> Message-ID: <20080425232530.GB7685@phare.normalesup.org> On Fri, Apr 25, 2008 at 01:41:14PM -0500, Robert Kern wrote: > As Anne notes, "bell-shaped curve," while seemingly generic, usually > specifies Gaussians, and Gaussians do not have the properties you need. > There are any number of curves which we could (and have) suggested as > "looking bell-shaped," That's strange. In my field bell-shaped curves means any curve that is centered on one specific point, stricly monotonous for x<0 and x>0, positiv, and with a finite maximum. For instance a Lorentzian is a bell-shaped curve. This shows that you can easily get strong miss-comprehension between fields. Cheers, Ga?l From charlesr.harris at gmail.com Fri Apr 25 19:39:30 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 25 Apr 2008 17:39:30 -0600 Subject: [Numpy-discussion] Generating Bell Curves (was: Using normal() ) In-Reply-To: References: <3d375d730804251141s550d1d06o82c5097ac3fc2139@mail.gmail.com> Message-ID: On Fri, Apr 25, 2008 at 1:25 PM, Rich Shepard wrote: > On Fri, 25 Apr 2008, Charles R Harris wrote: > > > You can use something like f(x) = (1-x**2)**2 , which has inflection > > points and vanishes at +/- 1. Any of the B-splines will also do the > trick. > > Chuck, > > Thank you. I need to make some time to understand the B-splines to use > them appropriately. Unfortunately, my mathematical statistics learning was > many years in the past ... but we had moved ahead of writing on clay > tablets > by that time. Not needing to retain that knowledge for many years means it > was replaced by more pressing current knowledge. The B-splines do look > promising, though. > Here's a B-spline approximation to a Gaussian: http://www.doc.ic.ac.uk/~dfg/AndysSplineTutorial/BSplines.html Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From hoytak at gmail.com Fri Apr 25 19:51:49 2008 From: hoytak at gmail.com (Hoyt Koepke) Date: Fri, 25 Apr 2008 16:51:49 -0700 Subject: [Numpy-discussion] Generating Bell Curves (was: Using normal() ) In-Reply-To: References: <3d375d730804251141s550d1d06o82c5097ac3fc2139@mail.gmail.com> Message-ID: <4db580fd0804251651g27a265fcv1ea3524c2f214fc6@mail.gmail.com> Another suggestion from machine learning stuff to throw into the mix: A soft step function that we use often is y = e^(ax) / ( 1 + e^(ax)). It has the nice property that the result y is always in (0,1). If you invert this, you get x = -(1/a)*log(y - 1); this maps (0,1) to the whole real line, and the a parameter controls how sharp that mapping is. Now use that as the input to a Gaussian, and you can get a soft truncation at (0,1). Additionally, you can do this twice to have more control over the shape of the resulting distribution. I suspect the resulting kurtosis/skew factors would be calculable. This has the advantage of giving you a calculable pdf (just normalize the resulting distribution using the inverse det-of-Jacobian factor) without too much hassle. Furthermore, it should be easy to fit the parameters to data without too much difficulty (though I haven't tried). Just a thought, though I've never worked with all this much so I can't say for sure how well it would work. --Hoyt On Fri, Apr 25, 2008 at 4:39 PM, Charles R Harris wrote: > > > > On Fri, Apr 25, 2008 at 1:25 PM, Rich Shepard > wrote: > > > > On Fri, 25 Apr 2008, Charles R Harris wrote: > > > > > You can use something like f(x) = (1-x**2)**2 , which has inflection > > > points and vanishes at +/- 1. Any of the B-splines will also do the > trick. > > > > Chuck, > > > > Thank you. I need to make some time to understand the B-splines to use > > them appropriately. Unfortunately, my mathematical statistics learning was > > many years in the past ... but we had moved ahead of writing on clay > tablets > > by that time. Not needing to retain that knowledge for many years means it > > was replaced by more pressing current knowledge. The B-splines do look > > promising, though. > > > > > > > > > > Here's a B-spline approximation to a Gaussian: > http://www.doc.ic.ac.uk/~dfg/AndysSplineTutorial/BSplines.html > > Chuck > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -- +++++++++++++++++++++++++++++++++++ Hoyt Koepke UBC Department of Computer Science http://www.cs.ubc.ca/~hoytak/ hoytak at gmail.com +++++++++++++++++++++++++++++++++++ From aisaac at american.edu Fri Apr 25 20:38:19 2008 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 25 Apr 2008 20:38:19 -0400 Subject: [Numpy-discussion] prerelease proposal for matrix behavior In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com><9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com><9457e7c80804250754n4967da64o1440164439407990@mail.gmail.com><9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com><481231DC.2010900@noaa.gov> Message-ID: OK, we are not converging in time for the release. So can we at least raise a TypeError on scalar indexing of matrices, so that we remain free to choose the ultimate behavior? Those who have spoke up have generally favored letting x[0] return a 1d array, if I count correctly. And I think that is the right "provisional" behavior. (As well as ultimate.) But many have not spoken up. So let's at least signal that this is an area of change. Right? Thank you, Alan Isaac From tournesol33 at gmail.com Fri Apr 25 21:16:39 2008 From: tournesol33 at gmail.com (tournesol) Date: Sat, 26 Apr 2008 10:16:39 +0900 Subject: [Numpy-discussion] 2D array to 3D Message-ID: <48128277.90104@gmail.com> Hi All. Is there a easy way to insert 1D(j) array into another 2D array(B:jxk) and conver B to B:ixjxk ? ex:) >>> from numpy import * >>> a=arange(4) >>> a array([0, 1, 2, 3]) >>> b=arange(9) >>> b.shape=3,3 >>> b array([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) I just wanna insert A into B B:1x3x3, [[[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]] B:2x3x3, [[[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]] ?? [[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]]] B:3x3x3, [[[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]] [[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]] [[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]]] Thanks for your help From charlesr.harris at gmail.com Fri Apr 25 21:27:56 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 25 Apr 2008 19:27:56 -0600 Subject: [Numpy-discussion] prerelease proposal for matrix behavior In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <9457e7c80804250754n4967da64o1440164439407990@mail.gmail.com> <9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com> <481231DC.2010900@noaa.gov> Message-ID: On Fri, Apr 25, 2008 at 6:38 PM, Alan G Isaac wrote: > OK, we are not converging in time for the release. > So can we at least raise a TypeError on scalar > indexing of matrices, so that we remain free to choose > the ultimate behavior? > > Those who have spoke up have generally favored > letting x[0] return a 1d array, if I count correctly. > And I think that is the right "provisional" behavior. > (As well as ultimate.) But many have not spoken up. > That would be the most compatible with arrays, but won't it break a lot of code? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Fri Apr 25 21:41:24 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Fri, 25 Apr 2008 20:41:24 -0500 Subject: [Numpy-discussion] prerelease proposal for matrix behavior In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com><9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com><9457e7c80804250754n4967da64o1440164439407990@mail.gmail.com><9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com><481231DC.2010900@noaa.gov> Message-ID: <48128844.7040509@enthought.com> Alan G Isaac wrote: > OK, we are not converging in time for the release. > So can we at least raise a TypeError on scalar > indexing of matrices, so that we remain free to choose > the ultimate behavior? > I think this is wise for the time being. At this point, I'm leaning in the direction of the RowVector / ColumnVector approach (even if these are not really advertised and just used during indexing). -Travis From wbaxter at gmail.com Fri Apr 25 21:42:11 2008 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 26 Apr 2008 10:42:11 +0900 Subject: [Numpy-discussion] prerelease proposal for matrix behavior In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com> <481231DC.2010900@noaa.gov> Message-ID: On Sat, Apr 26, 2008 at 10:27 AM, Charles R Harris wrote: > > > On Fri, Apr 25, 2008 at 6:38 PM, Alan G Isaac wrote: > > OK, we are not converging in time for the release. > > So can we at least raise a TypeError on scalar > > indexing of matrices, so that we remain free to choose > > the ultimate behavior? > > > > Those who have spoke up have generally favored > > letting x[0] return a 1d array, if I count correctly. > > And I think that is the right "provisional" behavior. > > (As well as ultimate.) But many have not spoken up. > > > > That would be the most compatible with arrays, but won't it break a lot of > code? Any change to the behavior of x[0] for matrices is going to break a lot of code. It actually seems like a good idea to me to make it an error for a while -- or maybe some kind of settable option -- to help people figure out where they need to fix their code. Better to get an error immediately. --bb From cburns at berkeley.edu Sat Apr 26 00:08:37 2008 From: cburns at berkeley.edu (Christopher Burns) Date: Fri, 25 Apr 2008 21:08:37 -0700 Subject: [Numpy-discussion] OSX installer: please test In-Reply-To: <4812407D.60003@noaa.gov> References: <764e38540804161436j5ce7a468w8bea6a3c3598d69e@mail.gmail.com> <480F9200.9060803@noaa.gov> <1209115462.17497.14.camel@bbc8> <4812407D.60003@noaa.gov> Message-ID: <764e38540804252108q40bf93b1sbde7b9e9d9b9a6e8@mail.gmail.com> There is also a gui, Disk Utility.app which is what I used for the installer. Chris On Fri, Apr 25, 2008 at 2:17 AM, Sebastian Haase wrote: OT: How do you make a dmg ? Is there a (simple) command line tool for this ? Thanks, Sebastian Haase On Fri, Apr 25, 2008 at 1:35 PM, Christopher Barker wrote: > David Cournapeau wrote: > > To get to your point: hdiutil is the command you are looking for. > > yup. Here's an example: > > hdiutil create -srcfolder YourDir -volname "A Name" -ov "Something.dmg" > > It will build a disk image from the directory: YourDir > > -Chris > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- Christopher Burns Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Sat Apr 26 01:19:50 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sat, 26 Apr 2008 01:19:50 -0400 Subject: [Numpy-discussion] Movement of ma breaks matplotlib Message-ID: Hi, In the upcoming release of numpy. numpy.core.ma ceases to exist. One must use numpy.ma (for the new interface) or numpy.oldnumeric.ma (for the old interface). This has the unfortunate effect of breaking matplotlib(which does "from numpy.core.ma import *") - I cannot even "import pylab" with a stock matplotlib (0.90.1) and an SVN numpy. Is this an intended effect? Anne From charlesr.harris at gmail.com Sat Apr 26 01:41:50 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 25 Apr 2008 23:41:50 -0600 Subject: [Numpy-discussion] Movement of ma breaks matplotlib In-Reply-To: References: Message-ID: On Fri, Apr 25, 2008 at 11:19 PM, Anne Archibald wrote: > Hi, > > In the upcoming release of numpy. numpy.core.ma ceases to exist. One > must use numpy.ma (for the new interface) or numpy.oldnumeric.ma (for > the old interface). This has the unfortunate effect of breaking > matplotlib(which does "from numpy.core.ma import *") - I cannot even > "import pylab" with a stock matplotlib (0.90.1) and an SVN numpy. Is > this an intended effect? IIRC, there was something about this on the MPL mailing list. I think you need to go to a later version, or something like that. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Apr 26 05:24:58 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 26 Apr 2008 04:24:58 -0500 Subject: [Numpy-discussion] 2D array to 3D In-Reply-To: <48128277.90104@gmail.com> References: <48128277.90104@gmail.com> Message-ID: <3d375d730804260224u63fc5f75m1b1aa9dd4db43caa@mail.gmail.com> 2008/4/25 tournesol : > Hi All. > > Is there a easy way to insert 1D(j) array into another 2D array(B:jxk) > and conver B to B:ixjxk ? > > ex:) > > >>> from numpy import * > >>> a=arange(4) > >>> a > array([0, 1, 2, 3]) > >>> b=arange(9) > >>> b.shape=3,3 > >>> b > array([[0, 1, 2], > [3, 4, 5], > [6, 7, 8]]) > > I just wanna insert A into B > B:1x3x3, > > [[[ 0, 1, 2, 3], > [ 4, 5, 6, 7], > [ 8, 9, 10, 11]] > > > B:2x3x3, > > [[[ 0, 1, 2, 3], > [ 4, 5, 6, 7], > [ 8, 9, 10, 11]] > > [[ 0, 1, 2, 3], > [ 4, 5, 6, 7], > [ 8, 9, 10, 11]]] > > B:3x3x3, > [[[ 0, 1, 2, 3], > [ 4, 5, 6, 7], > [ 8, 9, 10, 11]] > > [[ 0, 1, 2, 3], > [ 4, 5, 6, 7], > [ 8, 9, 10, 11]] > > [[ 0, 1, 2, 3], > [ 4, 5, 6, 7], > [ 8, 9, 10, 11]]] I'm sorry, but I cannot understand what you are asking for. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From gael.varoquaux at normalesup.org Sat Apr 26 06:03:04 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sat, 26 Apr 2008 12:03:04 +0200 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com> Message-ID: <20080426100304.GC24606@phare.normalesup.org> On Fri, Apr 25, 2008 at 01:04:53PM -0400, Alan G Isaac wrote: > On Fri, 25 Apr 2008, St?fan van der Walt apparently wrote: > > The agreement was: > > a) That x[0][0] should be equal to x[0,0] and > > b) That x[0,:] should be equal to x[0] (as for ndarrays) > 1. This is **not** what I understood as the agreement > (and I think the current solution is bad). > I certainly did not mean to support the change that > as implemented, and it is not clear to me that others > did either. > 2. These two goals are substantially in conflict for > matrices, as we are seeing. > 3. The important goal, (goal a., which everyone agrees on), > has NOT been accomplished by the current change: > x[0][0] raises a TypeError when x is a 1 by N matrix. I claim b is more important than a. IMHO, a is plain wrong: you should't be indexing x with x[0][0]. I am OK making a work, as long as it doesn't break b. In addition breaking be is backward incompatible changes for no good reason (replacing one missfeature with another). I would like this issue addressed in next release, not this one. I have the feeling the discussion is not sane enough, right now. Cheers, Ga?l From gael.varoquaux at normalesup.org Sat Apr 26 06:09:38 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sat, 26 Apr 2008 12:09:38 +0200 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: Message-ID: <20080426100938.GD24606@phare.normalesup.org> On Fri, Apr 25, 2008 at 01:40:29PM -0400, Alan G Isaac wrote: > On Fri, 25 Apr 2008, St?fan van der Walt wrote: > > workaround would break x[0] == x[0,:] > But there is not universal agreement that x[0] == x[0,:] is > desirable. In contrast, there *is* universal agreement that > x[0][0]==x[0,0] is desirable. Or so I've understood the > discussion. I DON'T AGREE! Sorry for yelling, but I don't see where you get your universality. You want to break backward incompatibility to accomodated wrongful indexing, and break rightful indexing. I don't know why people are indexing matrices with A[x][y], but they shouldn't. We have multidimensional objects, and a lot of effort has been put in that. Indexing with one dimension only is poor amongst other things for performance reasons. We also see that it breaks indexing semantics and fobrid doing clever indexing. That cannot be worked around, because by indexing with one dimension you are putting less information into the indexing routine than by indexing with two dimension. > Thus I do not see x[0] != x[0,:] as breakage: > it is desirable. You may think what you want. We may desagree on what is Right, but the breaks backward compatibility, and thus is a breakage and should be given a lot of thought. Ga?l From charlesr.harris at gmail.com Sat Apr 26 06:47:59 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 26 Apr 2008 04:47:59 -0600 Subject: [Numpy-discussion] Threading question for Travis Message-ID: Travis, Is this correct? NPY_LOOP_BEGIN_THREADS; switch(loop->meth) { case ONE_UFUNCLOOP: /* * Everything is contiguous, notswapped, aligned, * and of the right type. -- Fastest. * Or if not contiguous, then a single-stride * increment moves through the entire array. */ /*fprintf(stderr, "ONE...%d\n", loop->size);*/ loop->function((char **)loop->bufptr, &(loop->size), loop->steps, loop->funcdata); UFUNC_CHECK_ERROR(loop); break; Note that UFUNC_CHECK_ERROR calls PyErr_Occurred. That doesn't seem thread safe to me. Or maybe there is something special about that function I'm missing. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Sat Apr 26 06:54:54 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sat, 26 Apr 2008 12:54:54 +0200 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: <200804232147.50022.lists@informa.tiker.net> References: <200804231530.25141.lists@informa.tiker.net> <480F9287.7050405@noaa.gov> <200804232147.50022.lists@informa.tiker.net> Message-ID: <20080426105454.GG24606@phare.normalesup.org> On Wed, Apr 23, 2008 at 09:47:46PM -0400, Andreas Kl?ckner wrote: > > Any numpy-specific stuff for sip? > Not as far as I'm aware. In fact, I don't know of any uses of sip outside of > Qt/KDE-related things. Airbus uses it for heavy numerical work. They claim they have benchmarked all the tools and SIP was the fastest. If you want more information on that, you should contact the sip developer, Phil Thompson, he does some contracting job for Airbus. Cheers, Ga?l From aisaac at american.edu Sat Apr 26 09:46:57 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 26 Apr 2008 09:46:57 -0400 Subject: [Numpy-discussion] prerelease proposal for matrix behavior In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com><9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com><481231DC.2010900@noaa.gov> Message-ID: On Sat, 26 Apr 2008, Bill Baxter apparently wrote: > Any change to the behavior of x[0] for matrices is going to break a lot of code. OK, how about a warning then? E.g., see attached patch. Cheers, Alan -------------- next part -------------- --- defmatrix.old Sat Apr 26 09:34:11 2008 +++ defmatrix.py Sat Apr 26 09:37:13 2008 @@ -1,6 +1,7 @@ __all__ = ['matrix', 'bmat', 'mat', 'asmatrix'] import sys +import warnings import numeric as N from numeric import concatenate, isscalar, binary_repr @@ -119,6 +120,9 @@ return def __getitem__(self, index): + if isscalar(index): + warnings.warn("scalar indexing of matrices is deprecated, use x[i,:] instead", + DeprecationWarning, stacklevel=2) self._getitem = True try: out = N.ndarray.__getitem__(self, index) From aisaac at american.edu Sat Apr 26 10:12:46 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 26 Apr 2008 10:12:46 -0400 Subject: [Numpy-discussion] prerelease proposal for matrix behavior In-Reply-To: <48128844.7040509@enthought.com> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com><9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com><9457e7c80804250754n4967da64o1440164439407990@mail.gmail.com><9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com><481231DC.2010900@noaa.gov> <48128844.7040509@enthought.com> Message-ID: On Fri, 25 Apr 2008, "Travis E. Oliphant" apparently wrote: > At this point, I'm leaning in the direction of the > RowVector / ColumnVector approach (even if these are not > really advertised and just used during indexing). I believe that this conflicts with submatrix extraction. Details follow. Suppose ``x`` is a matrix, and I say ``v=A[0]``. What is ``v``? 1. If ``v`` is a 1d array, its behavior is known. E.g., ``v[0]`` is the first element and ``v[0,0]`` is an IndexError. If we want to keep submatrix extraction (as we should), we give up ``x[0] == x[0,:]``. 2. If ``v`` is a matrix, its behavior can be learned by the experienced but breaks basic expectations (the x[0][0] problem). 3. If ``v`` is a "RowVector" what behavior do we get? Suppose the idea is that this will rescue ``x[0][0]==x[0,0]`` **and** ``x[0]=x[0,:]``. But then we must give up that ``x[0,:]`` is a submatrix. Must ``v`` deviate from submatrix behavior in an important way? Yes: ``v[0][0]`` is an IndexError. (Note that the same problem arises if we just override ``__getitem__`` for the matrix class to special case row and column matrices.) Since submatrix extraction is fundamental, I think it is a *very* bad idea to give it up. If so, then under the RowVector proposal we must give up ``x[0]==x[0,:]``. But this is just what we give up with the much simpler proposal that ``v`` be a 1d array. Cheers, Alan From aisaac at american.edu Sat Apr 26 11:13:12 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 26 Apr 2008 11:13:12 -0400 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <20080426100304.GC24606@phare.normalesup.org> References: <9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com> <20080426100304.GC24606@phare.normalesup.org> Message-ID: On Sat, 26 Apr 2008, Gael Varoquaux apparently wrote: > I claim b is more important than a. IMHO, a is plain > wrong: you should't be indexing x with x[0][0]. Why?? Would you say this about a 2d array? Why the difference? The core argument has been that it is a **basic expectation** of the behavior of 2d array-like objects that you will be able to get, e.g., the first element with x[0][0]. (E.g., lists, tuples, arrays ... is there an exception?? It should not be broken!) I teach with matrices and I thought you might too: if so, you surely have run into this expectation (which is **natural**). In fact I truly doubt anyone has not been puzzled on first encounter by this:: >>> x matrix([[1, 2], [3, 4]]) >>> x[0] matrix([[1, 2]]) >>> x[0][0] matrix([[1, 2]]) Another argument, which I agree with, has been that matrix behavior should match 2d array behavior except where there is a clear payoff from deviation. What I do agree with, which I take to be implicit in your statement, is that if users of matrices should generally use multiple indexes. But that does not answer the question about what we should offer when x[0] is requested. (Or, more crucially, what you get when you iterate over a matrix.) Cheers, Alan Isaac PS If you have good arguments, please add them to From aisaac at american.edu Sat Apr 26 11:15:14 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 26 Apr 2008 11:15:14 -0400 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <20080426100938.GD24606@phare.normalesup.org> References: <20080426100938.GD24606@phare.normalesup.org> Message-ID: On Sat, 26 Apr 2008, Gael Varoquaux apparently wrote: > I don't see where you get your universality. Lists, tuples, arrays ... Where is the exception? Only matrices are the only 2d container I can think of that are broken this way. Cheers, Alan Isaac From aisaac at american.edu Sat Apr 26 11:29:01 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 26 Apr 2008 11:29:01 -0400 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <20080426100938.GD24606@phare.normalesup.org> References: <20080426100938.GD24606@phare.normalesup.org> Message-ID: On Sat, 26 Apr 2008, Gael Varoquaux apparently wrote: > We may desagree on what is Right, but the breaks backward > compatibility, and thus is a breakage and should be given > a lot of thought. I agree with this. I am a matrix user, and I have given it a lot of thought. I have been making this case for a *long* time. It has come to a head because of the announced desire to severely constrain API changes moving forward. As I am best able to understand you abstract view, the right thing to do would be for matrices to raise an error in response to a scalar index. However, I deduce, you would just leave things alone to avoid backward incompatibility. I weight the future more heavily. We are approaching a last chance to do things better, and we should seize it. The right questions looking forward: - what behavior allows the most generic code? - what behavior breaks fewest expectations? - what behavior is most useful? Cheers, Alan Isaac PS I cannot recall: do you use matrices? From gael.varoquaux at normalesup.org Sat Apr 26 11:23:32 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sat, 26 Apr 2008 17:23:32 +0200 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100304.GC24606@phare.normalesup.org> Message-ID: <20080426152332.GB17624@phare.normalesup.org> On Sat, Apr 26, 2008 at 11:13:12AM -0400, Alan G Isaac wrote: > On Sat, 26 Apr 2008, Gael Varoquaux apparently wrote: > > I claim b is more important than a. IMHO, a is plain > > wrong: you should't be indexing x with x[0][0]. > Why?? Because a 2D object is not a list of list. It is more than that. The numpy array actually exposes this interface, but I think using it systematicaly when it is not required is abuse. Moreover you are creating temporary objects that are useless and harmful. They are harmful for performance: you have to create useless objects and call their __getitem__ functions. > Would you say this about a 2d array? Of course. Even more. Suppose you have an nd array, with n=10, to index object using 10 different indexing, ie A[a][b]... you have to create 9 intermediate objects that get indexed. This is ridiculous and potentially very harmful for performance. > The core argument has been that it is a **basic > expectation** of the behavior of 2d array-like objects that > you will be able to get, e.g., the first element with x[0][0]. > (E.g., lists, tuples, arrays ... is there an exception?? For me this is wrong. list and tuples are not 2D. Numpy arrays happen to offer this feature, but you should not use it do to multiple dimension indexing. > I teach with matrices and I thought you might too: if so, > you surely have run into this expectation (which is **natural**). Well, I don't find it that natural. I would naturally expect A[0][0] to be invalid. I have not heavily run into that expectation. People around me are used to indexing with several indexes. This probably comes from their fortran/Mathematica/Maple/Matlab background. I can see that someone coming from C would expect this multiple consecutive indexing. This is because C has no notion of multiple dimension objects. The C model is limited and not as powerful as modern array programming languages, let us just move along. > In fact I truly doubt anyone has not been puzzled on first > encounter by this:: > >>> x > matrix([[1, 2], > [3, 4]]) > >>> x[0] > matrix([[1, 2]]) > >>> x[0][0] > matrix([[1, 2]]) Well, actually, I can say that this behavior is surprising. When I teach numpy and somebody encounters this behavior (which doesn't happen often, because people use multiple dimension indexing), I have to point out that x[0] is a 2D object also. I sympathize with your crusade to fix this inconvenience. This is why I support the RowVector/ColumnVector proposal. However your proposal breaks what I consider as normal and important for something I consider should be avoided. Ga?l From gael.varoquaux at normalesup.org Sat Apr 26 11:25:09 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sat, 26 Apr 2008 17:25:09 +0200 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org> Message-ID: <20080426152509.GC17624@phare.normalesup.org> On Sat, Apr 26, 2008 at 11:15:14AM -0400, Alan G Isaac wrote: > On Sat, 26 Apr 2008, Gael Varoquaux apparently wrote: > > I don't see where you get your universality. > Lists, tuples, arrays ... > Where is the exception? > Only matrices are the only 2d container I can > think of that are broken this way. List and Tuples are not 2D containers. Ga?l From millman at berkeley.edu Sat Apr 26 11:41:58 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Sat, 26 Apr 2008 08:41:58 -0700 Subject: [Numpy-discussion] Movement of ma breaks matplotlib In-Reply-To: References: Message-ID: On Fri, Apr 25, 2008 at 10:19 PM, Anne Archibald wrote: > In the upcoming release of numpy. numpy.core.ma ceases to exist. One > must use numpy.ma (for the new interface) or numpy.oldnumeric.ma (for > the old interface). This has the unfortunate effect of breaking > matplotlib(which does "from numpy.core.ma import *") - I cannot even > "import pylab" with a stock matplotlib (0.90.1) and an SVN numpy. Is > this an intended effect? You need matplotlib 0.91.2 or later. 0.91.2 was released in January: http://matplotlib.sourceforge.net/whats_new.html Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From zbyszek at in.waw.pl Sat Apr 26 12:53:30 2008 From: zbyszek at in.waw.pl (Zbyszek Szmek) Date: Sat, 26 Apr 2008 18:53:30 +0200 Subject: [Numpy-discussion] tests in distutils/exec_command.py Message-ID: <20080426165330.GB16762@szyszka.in.waw.pl> Hi, while looking at test coverage statistics published St?fan van der Walt at http://mentat.za.net/numpy/coverage/, I noticed that the least-covered file, numpy/distutils/exec_command.py has it's own test routines, e.g.: def test_svn(**kws): s,o = exec_command(['svn','status'],**kws) assert s,(s,o) print 'svn ok' called as if __name__ == "__main__": test_svn(use_tee=1) The sense of this test seems reversed (svn status runs just fine): Traceback (most recent call last): File "numpy/distutils/exec_command.py", line 591, in test_svn(use_tee=1) File "numpy/distutils/exec_command.py", line 567, in test_svn assert s,(s,o) AssertionError: (0, '') Should the test be cleaned up and moved into a seperate file in numpy/distutils/tests/ ? Regard, Zbyszek From pearu at cens.ioc.ee Sat Apr 26 13:03:22 2008 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Sat, 26 Apr 2008 20:03:22 +0300 (EEST) Subject: [Numpy-discussion] tests in distutils/exec_command.py In-Reply-To: <20080426165330.GB16762@szyszka.in.waw.pl> References: <20080426165330.GB16762@szyszka.in.waw.pl> Message-ID: <64476.88.89.194.145.1209229402.squirrel@cens.ioc.ee> On Sat, April 26, 2008 7:53 pm, Zbyszek Szmek wrote: > Hi, > while looking at test coverage statistics published St?fan van der Walt > at http://mentat.za.net/numpy/coverage/, I noticed that the > least-covered file, numpy/distutils/exec_command.py has it's own > test routines, e.g.: > def test_svn(**kws): > s,o = exec_command(['svn','status'],**kws) > assert s,(s,o) > print 'svn ok' > > called as > if __name__ == "__main__": > test_svn(use_tee=1) > > The sense of this test seems reversed (svn status runs just fine): > Traceback (most recent call last): > File "numpy/distutils/exec_command.py", line 591, in > test_svn(use_tee=1) > File "numpy/distutils/exec_command.py", line 567, in test_svn > assert s,(s,o) > AssertionError: (0, '') > > Should the test be cleaned up and moved into a seperate file in > numpy/distutils/tests/ ? Note that not all tests in exec_command.py are platform independent (as it is the case with most distutils tools in general). So, be careful when copying the tests to numpy/distutils/tests. Pearu From oliphant at enthought.com Sat Apr 26 13:21:01 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Sat, 26 Apr 2008 12:21:01 -0500 Subject: [Numpy-discussion] Threading question for Travis In-Reply-To: References: Message-ID: <4813647D.1060305@enthought.com> Charles R Harris wrote: > Travis, > > Is this correct? > Yes. > NPY_LOOP_BEGIN_THREADS; > switch(loop->meth) { > case ONE_UFUNCLOOP: > /* > * Everything is contiguous, notswapped, aligned, > * and of the right type. -- Fastest. > * Or if not contiguous, then a single-stride > * increment moves through the entire array. > */ > /*fprintf(stderr, "ONE...%d\n", loop->size);*/ > loop->function((char **)loop->bufptr, &(loop->size), > loop->steps, loop->funcdata); > UFUNC_CHECK_ERROR(loop); > break; > > Note that UFUNC_CHECK_ERROR calls PyErr_Occurred. That doesn't seem > thread safe to me. Or maybe there is something special about that > function I'm missing. Check the definition of the macro. It only calls PyErr_Occurred if the obj variable is non-zero on the loop structure, indicating an OBJECT-array loop. In that case, the NPY_LOOP_BEGIN_THREADS does not actually release the GIL either. -Travis From pav at iki.fi Sat Apr 26 13:29:22 2008 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 26 Apr 2008 20:29:22 +0300 Subject: [Numpy-discussion] reshape docstrings conflicting Message-ID: <48136672.8070608@iki.fi> Hi all, The ndarray.reshape docstring claims: "Also always returns a view or raises a ValueError if that is impossible." whereas fromnumeric.reshape claims: "This will be a new view object if possible; otherwise, it will be a copy." while the code paths for both functions are the same. So, which one of these is correct? Or, does ndarray.reshape always return a view? This is not immediately obvious looking at the code... I'll fix up the docstrings once I know. -- Pauli Virtanen From aisaac at american.edu Sat Apr 26 14:01:45 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 26 Apr 2008 14:01:45 -0400 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <20080426152332.GB17624@phare.normalesup.org> References: <20080426100304.GC24606@phare.normalesup.org> <20080426152332.GB17624@phare.normalesup.org> Message-ID: On Sat, 26 Apr 2008, Gael Varoquaux apparently wrote: > For me this is wrong. list and tuples are not 2D. Numpy > arrays happen to offer this feature, but you should not > use it do to multiple dimension indexing. But there is no proposal that people should index like this. The underlying issue is iteration. Currently:: >>> A matrix([[1, 2], [ 3, 4]]) >>> A = np.mat(x) >>> for row in A: ... for col in row: ... print col ... [[1 2]] [[3 4]] So are you saying that one should not be able to iterate through to the elements of a matrix? So many natural expectations are broken by the current behavior! Cheers, Alan From gael.varoquaux at normalesup.org Sat Apr 26 14:01:43 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sat, 26 Apr 2008 20:01:43 +0200 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426152332.GB17624@phare.normalesup.org> Message-ID: <20080426180143.GH17624@phare.normalesup.org> On Sat, Apr 26, 2008 at 02:01:45PM -0400, Alan G Isaac wrote: > >>> A > matrix([[1, 2], > [ 3, 4]]) > >>> A = np.mat(x) > >>> for row in A: > ... for col in row: > ... print col > ... > [[1 2]] > [[3 4]] > So are you saying that one should not be able to iterate > through to the elements of a matrix? I kinda expect this was the real issue. I think the only way to get consistent behavior for such code is either * to use a syntax similar to dictionnaries: for row in A.rows(): for col in row.cols() I actually think this is much better than the code you currently use, * or implement row and column objects. The problem in your code is that you do not distinguish iterating over rows and columns, and in linear algebra these are different objects. The two solutions I propose do this distinction. My favorite one is the first one (the rows and cols methods). Cheers, Ga?l From charlesr.harris at gmail.com Sat Apr 26 14:24:28 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 26 Apr 2008 12:24:28 -0600 Subject: [Numpy-discussion] reshape docstrings conflicting In-Reply-To: <48136672.8070608@iki.fi> References: <48136672.8070608@iki.fi> Message-ID: On Sat, Apr 26, 2008 at 11:29 AM, Pauli Virtanen wrote: > Hi all, > > The ndarray.reshape docstring claims: > > "Also always returns a view or raises a ValueError if that is > impossible." > > whereas fromnumeric.reshape claims: > > "This will be a new view object if possible; otherwise, it will > be a copy." > > while the code paths for both functions are the same. > > So, which one of these is correct? Or, does ndarray.reshape always > return a view? This is not immediately obvious looking at the code... > > I'll fix up the docstrings once I know. > Hi Pauli, I noticed that you removed the ReST markup of the tables in sort documentation. I think you should leave it in. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Sat Apr 26 14:32:01 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 26 Apr 2008 14:32:01 -0400 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <20080426152509.GC17624@phare.normalesup.org> References: <20080426100938.GD24606@phare.normalesup.org> <20080426152509.GC17624@phare.normalesup.org> Message-ID: >> On Sat, 26 Apr 2008, Gael Varoquaux apparently wrote: >>> I don't see where you get your universality. > On Sat, Apr 26, 2008 at 11:15:14AM -0400, Alan G Isaac > wrote: >> Lists, tuples, arrays ... >> Where is the exception? >> Only matrices are the only 2d container I can >> think of that are broken this way. On Sat, 26 Apr 2008, Gael Varoquaux apparently wrote: > List and Tuples are not 2D containers. A list of lists is naturally conceived as 2d, which is why you can initialize a 2d matrix with a list of lists. Try a Google search on "python two dimensional list" to confirm that I am not promoting an anomalous view. So I do not buy the way you are trying to parse language here. (Although I'm open to a fuller argument.) Cheers, Alan From gael.varoquaux at normalesup.org Sat Apr 26 14:32:40 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sat, 26 Apr 2008 20:32:40 +0200 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426152509.GC17624@phare.normalesup.org> Message-ID: <20080426183240.GK17624@phare.normalesup.org> On Sat, Apr 26, 2008 at 02:32:01PM -0400, Alan G Isaac wrote: > A list of lists is naturally conceived as 2d, > which is why you can initialize a 2d matrix > with a list of lists. > Try a Google search on "python two dimensional list" > to confirm that I am not promoting an anomalous view. > So I do not buy the way you are trying to parse > language here. (Although I'm open to a fuller > argument.) The object is not 2D. You can assemble several objects into something that looks 2D, but it is not 2D. For exemple: [[1, 2, 3], [1, 2]] This is not 2D in an nd array sens. It how exposes a double successiv indexing, but doubly 1D, and no 2D. For nD objects, I think people should rather use either nD indexing, or explicite 1D indexing of an nD object. By this is mean "A[1, ...]" rather than "A[1]". If the later breaks, I, for one, won't cry. Ga?l From aisaac at american.edu Sat Apr 26 14:42:04 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 26 Apr 2008 14:42:04 -0400 Subject: [Numpy-discussion] adding ``rows`` and ``columns`` attributes to matrices In-Reply-To: <20080426180143.GH17624@phare.normalesup.org> References: <20080426152332.GB17624@phare.normalesup.org> <20080426180143.GH17624@phare.normalesup.org> Message-ID: On Sat, 26 Apr 2008, Gael Varoquaux apparently wrote: > for row in A.rows(): > for col in row.cols() Actually I am in favor of adding ``rows`` and ``cols`` attributes to allow such iteration. The only thing is, I say these should be matrices (i.e., 2d). I.e., this should provide a symmetric syntax (for rows and columns) for iteration across submatrices. But once we have these, it is all the more reason not to object to letting iteration on the matrix yield 1d arrays. As others have observed: why break array behavior for no purpose? Cheers, Alan Isaac PS The possibility of adding these attributes is mentioned on the discussion page: as are most of the issues that have been coming up again. From gael.varoquaux at normalesup.org Sat Apr 26 14:38:35 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sat, 26 Apr 2008 20:38:35 +0200 Subject: [Numpy-discussion] adding ``rows`` and ``columns`` attributes to matrices In-Reply-To: References: <20080426180143.GH17624@phare.normalesup.org> Message-ID: <20080426183835.GL17624@phare.normalesup.org> On Sat, Apr 26, 2008 at 02:42:04PM -0400, Alan G Isaac wrote: > As others have observed: why break array > behavior for no purpose? Because the behavior you are relying on is IMHO a coincidence. Are other people on the mailing list relying on this? I strongly think it is wrong, but maybe I am the lone man. Ga?l From oliphant at enthought.com Sat Apr 26 14:52:25 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Sat, 26 Apr 2008 13:52:25 -0500 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100304.GC24606@phare.normalesup.org> <20080426152332.GB17624@phare.normalesup.org> Message-ID: <481379E9.5000907@enthought.com> Alan G Isaac wrote: > On Sat, 26 Apr 2008, Gael Varoquaux apparently wrote: > >> For me this is wrong. list and tuples are not 2D. Numpy >> arrays happen to offer this feature, but you should not >> use it do to multiple dimension indexing. >> > > But there is no proposal that people should index like this. > The underlying issue is iteration. Currently:: > > >>> A > matrix([[1, 2], > [ 3, 4]]) > >>> A = np.mat(x) > >>> for row in A: > ... for col in row: > ... print col > ... > [[1 2]] > [[3 4]] > > So are you saying that one should not be able to iterate > through to the elements of a matrix? > Iteration can be handled separately with __iter__ method that returns the actual iterator object which may be different. Or, as others have proposed .rows and .cols iterators. I'm not personally persuaded by the iteration argument, because we can change iteration independently of mapping (__getitem__) access. -Travis From millman at berkeley.edu Sat Apr 26 14:53:53 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Sat, 26 Apr 2008 11:53:53 -0700 Subject: [Numpy-discussion] reshape docstrings conflicting In-Reply-To: References: <48136672.8070608@iki.fi> Message-ID: On Sat, Apr 26, 2008 at 11:24 AM, Charles R Harris wrote: > I noticed that you removed the ReST markup of the tables in sort > documentation. I think you should leave it in. I haven't tried to render it, but I think he just changed the tables from ReST's grid table to ReST's simple table format: http://docutils.sourceforge.net/docs/user/rst/quickref.html#tables -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From aisaac at american.edu Sat Apr 26 15:04:08 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 26 Apr 2008 15:04:08 -0400 Subject: [Numpy-discussion] prerelease proposal for matrix behavior In-Reply-To: <48128844.7040509@enthought.com> References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com><9457e7c80804241623n350895c5r17aebce28b0a9cf0@mail.gmail.com><9457e7c80804250754n4967da64o1440164439407990@mail.gmail.com><9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com><481231DC.2010900@noaa.gov> <48128844.7040509@enthought.com> Message-ID: > Alan G Isaac wrote: >> OK, we are not converging in time for the release. >> So can we at least raise a TypeError on scalar >> indexing of matrices, so that we remain free to choose >> the ultimate behavior? On Fri, 25 Apr 2008, "Travis E. Oliphant" apparently wrote: > I think this is wise for the time being. In response to Bill's concern, I chose a warning instead: fwiw, Alan From pav at iki.fi Sat Apr 26 15:01:09 2008 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 26 Apr 2008 22:01:09 +0300 Subject: [Numpy-discussion] reshape docstrings conflicting In-Reply-To: References: <48136672.8070608@iki.fi> Message-ID: <48137BF5.8020109@iki.fi> Hi Chuck, Charles R Harris wrote: [clip] > I noticed that you removed the ReST markup of the tables in sort > documentation. I think you should leave it in. Also the simplified table syntax ======== ======== ======== header 1 header 2 header 3 ======== ======== ======== line 1a line 1b line 1c line 2a line 2b line 2c ======== ======== ======== is valid ReST [1], and easier to type and IMHO to read than the gridded tables: +----------+----------+----------+ | header 1 | header 2 | header 3 | +==========+==========+==========+ | line 1a | line 1b | line 1c | | line 2a | line 2b | line 2c | +----------+----------+----------+ If you strongly prefer the gridded syntax, I can revert it back. Pauli .. [1] http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#simple-tables From aisaac at american.edu Sat Apr 26 15:13:22 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 26 Apr 2008 15:13:22 -0400 Subject: [Numpy-discussion] leaving matrix indexing discussion Message-ID: At this point, I am starting to feel that I have abused my interlocutors. So I will withdraw from the conversation. Apologies to those who feel I should have done so earlier. The core issues are, I think, on the discussion page. I hope people will read it before presenting arguments. And improve it too. Cheers, Alan Isaac From aisaac at american.edu Sat Apr 26 15:13:24 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 26 Apr 2008 15:13:24 -0400 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <481379E9.5000907@enthought.com> References: <20080426100304.GC24606@phare.normalesup.org> <20080426152332.GB17624@phare.normalesup.org> <481379E9.5000907@enthought.com> Message-ID: On Sat, 26 Apr 2008, "Travis E. Oliphant" apparently wrote: > I'm not personally persuaded by the iteration argument, because we can > change iteration independently of mapping (__getitem__) access. Would that not impose another deviation from array behavior? I just do not see the purpose. I support .row and .cols iterators (returning submatrices), and have included the idea for some time on the discussion page . Cheers, Alan From gael.varoquaux at normalesup.org Sat Apr 26 15:09:39 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sat, 26 Apr 2008 21:09:39 +0200 Subject: [Numpy-discussion] leaving matrix indexing discussion In-Reply-To: References: Message-ID: <20080426190939.GM17624@phare.normalesup.org> On Sat, Apr 26, 2008 at 03:13:22PM -0400, Alan G Isaac wrote: > At this point, I am starting to feel that I have abused my > interlocutors. So I will withdraw from the conversation. Let us continue this discussion after numpy 1.1 is release, with no feeling of urgency. Ga?l From hoytak at gmail.com Sat Apr 26 16:52:38 2008 From: hoytak at gmail.com (Hoyt Koepke) Date: Sat, 26 Apr 2008 13:52:38 -0700 Subject: [Numpy-discussion] access ndarray in C++ In-Reply-To: <20080426105454.GG24606@phare.normalesup.org> References: <200804231530.25141.lists@informa.tiker.net> <480F9287.7050405@noaa.gov> <200804232147.50022.lists@informa.tiker.net> <20080426105454.GG24606@phare.normalesup.org> Message-ID: <4db580fd0804261352i374211a3v8110f4a4f73ced20@mail.gmail.com> Let me say also that I tried using boost.python a while back to interface numpy with c++, and, while I got some things working, I found the distribution and packaging end of things an order of magnitude more complicated than what I found with weave. Since weave is built into scipy, as well as blitz itself (now recently under the BSD license), it fits really well with numpy's distutils. The only dependencies on the user end are numpy and scipy, which they would need anyway from the python end of the code. Also, I find the syntax of blitz++ to be really simple and numpy-like. E.g. Array A; A.resize(20,20); for(int i = 0; i < 20; ++i) A(i, Range::all() ) = i; Array B = A(Range(2,18), 0); B *= 2; A(2, Range(5,10) ) += 2; Array C = B*A(2, Range(1,17) ); and so on... (I just typed this in here, prob some typos). Anyway, after bouncing around for a bit I think I've found this to be the solution that most closely fits my needs. --Hoyt On Sat, Apr 26, 2008 at 3:54 AM, Gael Varoquaux wrote: > On Wed, Apr 23, 2008 at 09:47:46PM -0400, Andreas Kl?ckner wrote: > > > Any numpy-specific stuff for sip? > > > Not as far as I'm aware. In fact, I don't know of any uses of sip outside of > > Qt/KDE-related things. > > Airbus uses it for heavy numerical work. They claim they have benchmarked > all the tools and SIP was the fastest. > If you want more information on that, you should contact the sip > developer, Phil Thompson, he does some contracting job for Airbus. > > Cheers, > > Ga?l > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- +++++++++++++++++++++++++++++++++++ Hoyt Koepke UBC Department of Computer Science http://www.cs.ubc.ca/~hoytak/ hoytak at gmail.com +++++++++++++++++++++++++++++++++++ From dcuccia at gmail.com Sat Apr 26 18:27:36 2008 From: dcuccia at gmail.com (David) Date: Sat, 26 Apr 2008 22:27:36 +0000 (UTC) Subject: [Numpy-discussion] [python] Re: Request for advice: project to get NumPy working in IronPython References: <4713AFAF.10201@resolversystems.com> <4713C334.2000407@enthought.com> <4722325F.7070302@scipy.org> <47224310.1080109@enthought.com> <4722530F.603@voidspace.org.uk> Message-ID: Michael Foord voidspace.org.uk> writes: > > are you really going to run matrix inversion on the CLR in a browser?) Yes! From charlesr.harris at gmail.com Sat Apr 26 20:25:28 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 26 Apr 2008 18:25:28 -0600 Subject: [Numpy-discussion] Threading question for Travis In-Reply-To: <4813647D.1060305@enthought.com> References: <4813647D.1060305@enthought.com> Message-ID: On Sat, Apr 26, 2008 at 11:21 AM, Travis E. Oliphant wrote: > Charles R Harris wrote: > > Travis, > > > > Is this correct? > > > Yes. > > NPY_LOOP_BEGIN_THREADS; > > switch(loop->meth) { > > case ONE_UFUNCLOOP: > > /* > > * Everything is contiguous, notswapped, aligned, > > * and of the right type. -- Fastest. > > * Or if not contiguous, then a single-stride > > * increment moves through the entire array. > > */ > > /*fprintf(stderr, "ONE...%d\n", loop->size);*/ > > loop->function((char **)loop->bufptr, &(loop->size), > > loop->steps, loop->funcdata); > > UFUNC_CHECK_ERROR(loop); > > break; > > > > Note that UFUNC_CHECK_ERROR calls PyErr_Occurred. That doesn't seem > > thread safe to me. Or maybe there is something special about that > > function I'm missing. > Check the definition of the macro. It only calls PyErr_Occurred if the > obj variable is non-zero on the loop structure, indicating an > OBJECT-array loop. In that case, the NPY_LOOP_BEGIN_THREADS does not > actually release the GIL either. > I'm still bothered by the fact that moving the check fixed ticket #733 on python2.6, whereas in python2.5 the error was caught, just too late to be useful. I'm also not sure why the x |= x loop goes through a buffer loop. With a nice contiguous array it should have gone straight through the fast loop. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Sat Apr 26 20:54:47 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Sat, 26 Apr 2008 19:54:47 -0500 Subject: [Numpy-discussion] Threading question for Travis In-Reply-To: References: <4813647D.1060305@enthought.com> Message-ID: <4813CED7.7060504@enthought.com> Charles R Harris wrote: > > > On Sat, Apr 26, 2008 at 11:21 AM, Travis E. Oliphant > > wrote: > > Charles R Harris wrote: > > Travis, > > > > Is this correct? > > > Yes. > > NPY_LOOP_BEGIN_THREADS; > > switch(loop->meth) { > > case ONE_UFUNCLOOP: > > /* > > * Everything is contiguous, notswapped, aligned, > > * and of the right type. -- Fastest. > > * Or if not contiguous, then a single-stride > > * increment moves through the entire array. > > */ > > /*fprintf(stderr, "ONE...%d\n", loop->size);*/ > > loop->function((char **)loop->bufptr, &(loop->size), > > loop->steps, loop->funcdata); > > UFUNC_CHECK_ERROR(loop); > > break; > > > > Note that UFUNC_CHECK_ERROR calls PyErr_Occurred. That doesn't seem > > thread safe to me. Or maybe there is something special about that > > function I'm missing. > Check the definition of the macro. It only calls PyErr_Occurred > if the > obj variable is non-zero on the loop structure, indicating an > OBJECT-array loop. In that case, the NPY_LOOP_BEGIN_THREADS > does not > actually release the GIL either. > > > I'm still bothered by the fact that moving the check fixed ticket #733 > on python2.6, whereas in python2.5 the error was caught, just too late > to be useful. I'm also not sure why the x |= x loop goes through a > buffer loop. With a nice contiguous array it should have gone straight > through the fast loop. It could be the need for data-type conversion as well. I need to catch up with the ticket to understand what is going on, though. -Travis From charlesr.harris at gmail.com Sat Apr 26 22:27:52 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 26 Apr 2008 20:27:52 -0600 Subject: [Numpy-discussion] What is __array_wrap__ supposed to do? Message-ID: Because it doesn't do what it says it does. In [31]: b.__array_wrap__(a).dtype Out[31]: dtype('float64') In [32]: a.__array_wrap__(b).dtype Out[32]: dtype('int8') In [33]: a Out[33]: array([[ 1., 1., 1.], [ 1., 1., 1.], [ 1., 1., 1.]]) In [34]: b Out[34]: array([[1, 1, 1], [1, 1, 1], [1, 1, 1]], dtype=int8) Here is what the docstring says: Help on method_descriptor: __array_wrap__(...) a.__array_wrap__(obj) -> Object of same type as a from ndarray obj. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Apr 26 22:59:48 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 26 Apr 2008 21:59:48 -0500 Subject: [Numpy-discussion] What is __array_wrap__ supposed to do? In-Reply-To: References: Message-ID: <3d375d730804261959o1e0e7b19q750f755624aad009@mail.gmail.com> On Sat, Apr 26, 2008 at 9:27 PM, Charles R Harris wrote: > Because it doesn't do what it says it does. > Here is what the docstring says: > > Help on method_descriptor: > > __array_wrap__(...) > a.__array_wrap__(obj) -> Object of same type as a from ndarray obj. Python type, not numpy dtype. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Sat Apr 26 23:04:07 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 26 Apr 2008 21:04:07 -0600 Subject: [Numpy-discussion] What is __array_wrap__ supposed to do? In-Reply-To: <3d375d730804261959o1e0e7b19q750f755624aad009@mail.gmail.com> References: <3d375d730804261959o1e0e7b19q750f755624aad009@mail.gmail.com> Message-ID: On Sat, Apr 26, 2008 at 8:59 PM, Robert Kern wrote: > On Sat, Apr 26, 2008 at 9:27 PM, Charles R Harris > wrote: > > Because it doesn't do what it says it does. > > > Here is what the docstring says: > > > > Help on method_descriptor: > > > > __array_wrap__(...) > > a.__array_wrap__(obj) -> Object of same type as a from ndarray obj. > > Python type, not numpy dtype. > So basically it's good for keeping matrices matrices? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Apr 26 23:29:51 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 26 Apr 2008 22:29:51 -0500 Subject: [Numpy-discussion] What is __array_wrap__ supposed to do? In-Reply-To: References: <3d375d730804261959o1e0e7b19q750f755624aad009@mail.gmail.com> Message-ID: <3d375d730804262029j25729cffl903968025dcd90dd@mail.gmail.com> On Sat, Apr 26, 2008 at 10:04 PM, Charles R Harris wrote: > > On Sat, Apr 26, 2008 at 8:59 PM, Robert Kern wrote: > > > > On Sat, Apr 26, 2008 at 9:27 PM, Charles R Harris > > wrote: > > > Because it doesn't do what it says it does. > > > > > > > Here is what the docstring says: > > > > > > Help on method_descriptor: > > > > > > __array_wrap__(...) > > > a.__array_wrap__(obj) -> Object of same type as a from ndarray obj. > > > > Python type, not numpy dtype. > > So basically it's good for keeping matrices matrices? Yes, that is what it is used for. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Sat Apr 26 23:56:39 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 26 Apr 2008 21:56:39 -0600 Subject: [Numpy-discussion] What is __array_wrap__ supposed to do? In-Reply-To: <3d375d730804262029j25729cffl903968025dcd90dd@mail.gmail.com> References: <3d375d730804261959o1e0e7b19q750f755624aad009@mail.gmail.com> <3d375d730804262029j25729cffl903968025dcd90dd@mail.gmail.com> Message-ID: On Sat, Apr 26, 2008 at 9:29 PM, Robert Kern wrote: > On Sat, Apr 26, 2008 at 10:04 PM, Charles R Harris > wrote: > > > > On Sat, Apr 26, 2008 at 8:59 PM, Robert Kern > wrote: > > > > > > On Sat, Apr 26, 2008 at 9:27 PM, Charles R Harris > > > wrote: > > > > Because it doesn't do what it says it does. > > > > > > > > > > Here is what the docstring says: > > > > > > > > Help on method_descriptor: > > > > > > > > __array_wrap__(...) > > > > a.__array_wrap__(obj) -> Object of same type as a from ndarray > obj. > > > > > > Python type, not numpy dtype. > > > > So basically it's good for keeping matrices matrices? > > Yes, that is what it is used for. > I'm working through the linalg module. There is currently variation as to whether or not eigenvalues/singular_values/residuals, are returned as arrays or as matrices. I'm leaning towards making them all matrices. What do you think? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Apr 27 00:23:23 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 26 Apr 2008 22:23:23 -0600 Subject: [Numpy-discussion] To wrap or not to wrap. Message-ID: Hi All, I'm working through the linalg module. There is currently variation as to whether or not eigenvalues/singular_values/residuals, are returned as arrays or as matrices. I'm leaning towards making them all matrices. But I expect masked arrays will also be preserved and I don't know what that will mean to people. What should be the standard here? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sun Apr 27 00:31:17 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 26 Apr 2008 23:31:17 -0500 Subject: [Numpy-discussion] To wrap or not to wrap. In-Reply-To: References: Message-ID: <3d375d730804262131v8ae46fbq67476fe275d813f3@mail.gmail.com> On Sat, Apr 26, 2008 at 11:23 PM, Charles R Harris wrote: > Hi All, > > I'm working through the linalg module. There is currently variation as to > whether or not eigenvalues/singular_values/residuals, are returned as > arrays or as matrices. I'm leaning towards making them all matrices. Only when the inputs are matrices, right? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From lists at informa.tiker.net Sun Apr 27 00:31:44 2008 From: lists at informa.tiker.net (Andreas =?utf-8?q?Kl=C3=B6ckner?=) Date: Sun, 27 Apr 2008 00:31:44 -0400 Subject: [Numpy-discussion] What is __array_wrap__ supposed to do? In-Reply-To: References: <3d375d730804262029j25729cffl903968025dcd90dd@mail.gmail.com> Message-ID: <200804270031.52047.lists@informa.tiker.net> On Samstag 26 April 2008, Charles R Harris wrote: > I'm working through the linalg module. There is currently variation as to > whether or not eigenvalues/singular_values/residuals, are returned as > arrays or as matrices. I'm leaning towards making them all matrices. What > do you think? IMO, the user should only get matrices if he used them to start with. I'm somewhat scared by all the disagreement about matrix semantics, and so if I don't touch matrices myself, I don't want them forced on me, thank you very much. Or, if you prefer a briefer reply: NOoooooooooooooooooooooooooooooooooo! :) Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From charlesr.harris at gmail.com Sun Apr 27 00:39:51 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 26 Apr 2008 22:39:51 -0600 Subject: [Numpy-discussion] What is __array_wrap__ supposed to do? In-Reply-To: <200804270031.52047.lists@informa.tiker.net> References: <3d375d730804262029j25729cffl903968025dcd90dd@mail.gmail.com> <200804270031.52047.lists@informa.tiker.net> Message-ID: On Sat, Apr 26, 2008 at 10:31 PM, Andreas Kl?ckner wrote: > On Samstag 26 April 2008, Charles R Harris wrote: > > I'm working through the linalg module. There is currently variation as > to > > whether or not eigenvalues/singular_values/residuals, are returned as > > arrays or as matrices. I'm leaning towards making them all matrices. > What > > do you think? > > IMO, the user should only get matrices if he used them to start with. I'm > somewhat scared by all the disagreement about matrix semantics, and so if > I > don't touch matrices myself, I don't want them forced on me, thank you > very > much. > Well, that's what I meant, if not quite what I said.. The question is what they should be when the input is a matrix. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Apr 27 00:41:18 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 26 Apr 2008 22:41:18 -0600 Subject: [Numpy-discussion] To wrap or not to wrap. In-Reply-To: <3d375d730804262131v8ae46fbq67476fe275d813f3@mail.gmail.com> References: <3d375d730804262131v8ae46fbq67476fe275d813f3@mail.gmail.com> Message-ID: On Sat, Apr 26, 2008 at 10:31 PM, Robert Kern wrote: > On Sat, Apr 26, 2008 at 11:23 PM, Charles R Harris > wrote: > > Hi All, > > > > I'm working through the linalg module. There is currently variation as > to > > whether or not eigenvalues/singular_values/residuals, are returned as > > arrays or as matrices. I'm leaning towards making them all matrices. > > Only when the inputs are matrices, right? > Right. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilson.t.thompson at gmail.com Sun Apr 27 13:15:31 2008 From: wilson.t.thompson at gmail.com (wilson) Date: Sun, 27 Apr 2008 10:15:31 -0700 (PDT) Subject: [Numpy-discussion] setting rows of a numpy array Message-ID: hi all, i have a numpy array of floats whose rows i need to set. i am setting the each row using pixel values of each image in some folder.I wrote a class MyImage that has a field pixelarray which stores the pixels of that image as a numpy array.I made several MyImage instances and stored them in a list called myimagefileslist. i can get the pixels of each image as a numpy array using myimagefileslist[i]._pixelarray where _pixelarray is a field of MyImage class . now i defined a function that sets a row of a given array as below def putrow(myarray,inrow ,rownum): myarray[rownum,:]=inrow in my code i initially make an empty array myimagesdata=zeros((numofimgs,numofpixels)) then i set the rows using for i in range(numofimgs): pixarray=asfarray(myimagefileslist[i]._pixelarray) putrow(myimagesdata,pixarray,i) this gets me the myimagesdata array updated with the pixeldata of images from myimagefileslist.But i am wondering if there is a better way to do this.If anyone can suggest improvements it wd be helpful W From charlesr.harris at gmail.com Sun Apr 27 22:53:48 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 27 Apr 2008 20:53:48 -0600 Subject: [Numpy-discussion] To wrap or not to wrap. In-Reply-To: References: <3d375d730804262131v8ae46fbq67476fe275d813f3@mail.gmail.com> Message-ID: On Sat, Apr 26, 2008 at 10:41 PM, Charles R Harris < charlesr.harris at gmail.com> wrote: > > > On Sat, Apr 26, 2008 at 10:31 PM, Robert Kern > wrote: > > > On Sat, Apr 26, 2008 at 11:23 PM, Charles R Harris > > wrote: > > > Hi All, > > > > > > I'm working through the linalg module. There is currently variation as > > to > > > whether or not eigenvalues/singular_values/residuals, are returned as > > > arrays or as matrices. I'm leaning towards making them all matrices. > > > > Only when the inputs are matrices, right? > > > > Right. > And now it is so. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From millman at berkeley.edu Mon Apr 28 03:14:20 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 28 Apr 2008 00:14:20 -0700 Subject: [Numpy-discussion] 673 In-Reply-To: <9457e7c80804090634t53c29d62h6a662e3b3d1309c@mail.gmail.com> References: <9457e7c80804090530q7c448971wd26b8acf51bcb0ca@mail.gmail.com> <9457e7c80804090634t53c29d62h6a662e3b3d1309c@mail.gmail.com> Message-ID: On Wed, Apr 9, 2008 at 6:34 AM, St?fan van der Walt wrote: > Unfortunately, I couldn't get this patch to work, and my time has run > out. Maybe someone with more knowledge the precedences/order of > functions during linking can take a look. I don't know how to tell > the system BLAS to call our custom xerbla, instead of the one > provided. > > The patch addresses an important issue, though, so it warrants some > more attention. Hey, I was wondering what the status of this ticket is? Is this something that should be fixed before 1.1.0? Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From pearu at cens.ioc.ee Mon Apr 28 03:45:26 2008 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Mon, 28 Apr 2008 09:45:26 +0200 Subject: [Numpy-discussion] 673 In-Reply-To: References: <9457e7c80804090530q7c448971wd26b8acf51bcb0ca@mail.gmail.com> <9457e7c80804090634t53c29d62h6a662e3b3d1309c@mail.gmail.com> Message-ID: <48158096.8020208@cens.ioc.ee> Hi, As far as I am concerned, the issue needs a cosmetic fix of renaming pythonxerbla to python_xerbla and the rest of the issue can be postponed to 1.2. Note that this isn't purely a numpy issue. To fix the issue, system or user provided blas/lapack libraries need to be changed, we can only give instructions how to do that. Doing the change automatically requires testing the corresponding support code for many different platforms and setups - this requires some effort and time.. and most importantly, some volunteers. Pearu Jarrod Millman wrote: > On Wed, Apr 9, 2008 at 6:34 AM, St?fan van der Walt wrote: >> Unfortunately, I couldn't get this patch to work, and my time has run >> out. Maybe someone with more knowledge the precedences/order of >> functions during linking can take a look. I don't know how to tell >> the system BLAS to call our custom xerbla, instead of the one >> provided. >> >> The patch addresses an important issue, though, so it warrants some >> more attention. > > Hey, > > I was wondering what the status of this ticket is? Is this something > that should be fixed before 1.1.0? > > Thanks, > From stefan at sun.ac.za Mon Apr 28 04:14:35 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 28 Apr 2008 10:14:35 +0200 Subject: [Numpy-discussion] 673 In-Reply-To: <48158096.8020208@cens.ioc.ee> References: <9457e7c80804090530q7c448971wd26b8acf51bcb0ca@mail.gmail.com> <9457e7c80804090634t53c29d62h6a662e3b3d1309c@mail.gmail.com> <48158096.8020208@cens.ioc.ee> Message-ID: <9457e7c80804280114n7d1926eco1c8dad299573b41f@mail.gmail.com> 2008/4/28 Pearu Peterson : > Hi, > > As far as I am concerned, the issue needs a cosmetic fix of renaming > pythonxerbla to python_xerbla Done. > and the rest of the issue can be postponed to 1.2. Cheers St?fan From andrea.gavana at gmail.com Mon Apr 28 07:41:16 2008 From: andrea.gavana at gmail.com (Andrea Gavana) Date: Mon, 28 Apr 2008 12:41:16 +0100 Subject: [Numpy-discussion] Linear Interpolation Question Message-ID: Hi All, I have 2 matrices coming from 2 different simulations: the first column of the matrices is a date (time) at which all the other results in the matrix have been reported (simulation step). In these 2 matrices, very often the simulation steps do not coincide, so I just want to interpolate the results in the second matrix using the dates in the first matrix. The problem is, I have close to 13,000 columns in every matrices, and repeating interp1d all over the columns is quite expensive. An example of what I am doing is as follows: # Loop over all the columns for indx in indices: # Set up a linear interpolation with: # x = dates in the second simulation # y = single column in the second matrix simulation function = interp1d(secondaryMatrixDates, secondaryMatrixResults[:, indx], kind='linear') # Interpolate the second matrix results using the first simulation dates interpolationResults = function(mainMatrixDates) # I need the difference between the first simulation and the second newMatrix[:, indx] = mainMatrixResults[:, indx] - interpolationResults This is somehow a costly step, as it's taking up a lot of CPU (increasing at every iteration) and quite a long time (every column has about 350 data). Is there anything I can do to speed up this loop? Or may someone suggest a better approach? Thank you very much for your suggestions. Andrea. "Imagination Is The Only Weapon In The War Against Reality." http://xoomer.alice.it/infinity77/ From rshepard at appl-ecosys.com Mon Apr 28 09:13:19 2008 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Mon, 28 Apr 2008 06:13:19 -0700 (PDT) Subject: [Numpy-discussion] Distribution Functions Change Behavior Message-ID: AFter the extensive input from folks here last week, and my examination of alternatives, I accepted the visual appearance of Gaussian curves in our model. As I check the plots for data errors I find a behavior change when the x-axis length is 14 rather than 100, and I do not understand why. I would appreciate the ideas of the mathematically adroit here to help resolve the problem. However, if this is not the appropriate forum for such a discussion, please point me to the more appropriate one. Most of the plots cover a range of independent values of 0-100. Some are greater (e.g., 0-1,000 or 0-10,000). However, for pH the range is 0-14. The plots for pH do not consist of sigmoid curves (a 'Z-curve' from 0-7 and an 'S-curve' from 7-14); they are straight lines. The Gaussian curve does not approach zero on either end (0 or 14). I can provide the functions for these curves. I need to understand why the results change when the range of the independent variable is short and learn how to produce correct results. Rich -- Richard B. Shepard, Ph.D. | Integrity Credibility Applied Ecosystem Services, Inc. | Innovation Voice: 503-667-4517 Fax: 503-667-8863 From andrea.gavana at gmail.com Mon Apr 28 09:32:25 2008 From: andrea.gavana at gmail.com (Andrea Gavana) Date: Mon, 28 Apr 2008 14:32:25 +0100 Subject: [Numpy-discussion] Linear Interpolation Question In-Reply-To: References: Message-ID: Hi All, On Mon, Apr 28, 2008 at 12:41 PM, Andrea Gavana wrote: > Hi All, > > I have 2 matrices coming from 2 different simulations: the first > column of the matrices is a date (time) at which all the other results > in the matrix have been reported (simulation step). In these 2 > matrices, very often the simulation steps do not coincide, so I just > want to interpolate the results in the second matrix using the dates > in the first matrix. The problem is, I have close to 13,000 columns in > every matrices, and repeating interp1d all over the columns is quite > expensive. An example of what I am doing is as follows: > > # Loop over all the columns > for indx in indices: > > # Set up a linear interpolation with: > # x = dates in the second simulation > # y = single column in the second matrix simulation > function = interp1d(secondaryMatrixDates, > secondaryMatrixResults[:, indx], kind='linear') > > # Interpolate the second matrix results using the first simulation dates > interpolationResults = function(mainMatrixDates) > > # I need the difference between the first simulation and the second > newMatrix[:, indx] = mainMatrixResults[:, indx] - interpolationResults > > This is somehow a costly step, as it's taking up a lot of CPU > (increasing at every iteration) and quite a long time (every column > has about 350 data). Is there anything I can do to speed up this loop? > Or may someone suggest a better approach? > > Thank you very much for your suggestions. Ok, I have tried to be smart and use interp2d, but interp2d gives me a strange error message which I can't understand: D:\MyProjects>Interp2DSample.py Traceback (most recent call last): File "D:\MyProjects\Interp2DSample.py", line 25, in function = interp2d(xx, yy, z, kind="linear", copy=False) File "C:\Python25\lib\site-packages\scipy\interpolate\interpolate.py", line 91, in __init__ self.tck = fitpack.bisplrep(self.x, self.y, self.z, kx=kx, ky=ky, s=0.) File "C:\Python25\lib\site-packages\scipy\interpolate\fitpack.py", line 677, in bisplrep tx,ty,nxest,nyest,wrk,lwrk1,lwrk2) OverflowError: long int too large to convert to int I am able to get this error message using this simple script: import datetime import numpy from scipy.interpolate import interp2d date1, date2 = [], [] numColumns = 13000 for year in xrange(2007, 2038): for month in xrange(1, 13): date1.append(datetime.date(year, month, 1).toordinal()) date2.append(datetime.date(year, month, 5).toordinal()) timeSteps = len(date2) x = [date1[0] for i in xrange(numColumns)] y = date1 z = numpy.random.rand(timeSteps, numColumns) xx, yy = numpy.meshgrid(x, y) newX = [date2[0] for i in xrange(numColumns)] newY = date2 function = interp2d(xx, yy, z, kind="linear", copy=False) newZ = function(newX, newY) Does anyone know what I am doing wrong? I am on Windows XP, Python 2.5, scipy 0.5.2.1, numpy 1.0.3.1. Thank you very much for your suggestions. Andrea. "Imagination Is The Only Weapon In The War Against Reality." http://xoomer.alice.it/infinity77/ From aisaac at american.edu Mon Apr 28 10:18:06 2008 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 28 Apr 2008 10:18:06 -0400 Subject: [Numpy-discussion] Distribution Functions Change Behavior In-Reply-To: References: Message-ID: Hi Rich, If your data is truncated at zero, it is not Gaussian (drawn from a normal). You will notice this when you shrink the range of values (unless the variance is tiny). Cheers, Alan Isaac From david at ar.media.kyoto-u.ac.jp Mon Apr 28 10:44:57 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 28 Apr 2008 23:44:57 +0900 Subject: [Numpy-discussion] Starting to work on runtime plugin system for plugin (automatic sse optimization, etc...) Message-ID: <4815E2E9.5020506@ar.media.kyoto-u.ac.jp> Hi, I've just started working on a prototype for a plugin system for numpy. The plugin aims at providing a framework for the following user cases: - runtime selection of blas/lapack/etc...: instead of harcoding in the binary one blas/lapack implementation, numpy could choose the SSE optimized if the CPU supports SSE, etc... - this could also be used for core numpy, for example ufuncs: if we want to start implementing some tight loop with aggressively optimized code (SSE, etc...), we could again ship with a default pure C implementation, and choose the best one at runtime. - we could even have a system to choose a different implementation (for example, right now, scipy is shipped with a slow fft for licensing issues mainly, and people installing fftw could then tell scipy to use fftw instead of the included one). Right now, the prototype does not do much, and only works for linux; I mainly focused on automatic generation of the plugin from a list of functions, and transparent use from numpy point of view. It provides the plugin api through pure function pointers, without the need for the user to be aware of it. For example, if you have an api with the following functions: void foo1(); int foo2(); int foo3(int); int foo4(double* , double*); int foo5(double* , double*, int); The current implementation would build the boilerplate to load those functions, etc... and you would just use those functions in numpy like the following: init_foo(); /* all functions are prefixed with npyw, for numpy wrapper */ npyw_foo1(); npyw_foo2(n); etc... The code can be found there: https://code.launchpad.net/~david-ar/+junk/numplug And some thinking (pretty low content for now): http://www.scipy.org/RuntimeOptimization cheers, David From hoytak at gmail.com Mon Apr 28 11:29:45 2008 From: hoytak at gmail.com (Hoyt Koepke) Date: Mon, 28 Apr 2008 08:29:45 -0700 Subject: [Numpy-discussion] Distribution Functions Change Behavior In-Reply-To: References: Message-ID: <4db580fd0804280829w3c2ba57fs7b79ede3a15037d7@mail.gmail.com> I may not understand what you are asking, Rich, but I'm not sure I agree with Alan. A Gaussian fit to data x should fit exactly as well as data fit to ax, a > 0, just with a variance a^2 times the original. The only way this would not be true is if: 1. You are not fitting the variance, but only the mean 2. There's some numerical issue (like some of the data are represented as integers, etc.) Don't know if it could be one of those issues... --Hoyt On Mon, Apr 28, 2008 at 7:18 AM, Alan G Isaac wrote: > Hi Rich, > > If your data is truncated at zero, it is not Gaussian (drawn > from a normal). You will notice this when you shrink the > range of values (unless the variance is tiny). > > Cheers, > Alan Isaac > > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- +++++++++++++++++++++++++++++++++++ Hoyt Koepke UBC Department of Computer Science http://www.cs.ubc.ca/~hoytak/ hoytak at gmail.com +++++++++++++++++++++++++++++++++++ From hoytak at gmail.com Mon Apr 28 11:34:00 2008 From: hoytak at gmail.com (Hoyt Koepke) Date: Mon, 28 Apr 2008 08:34:00 -0700 Subject: [Numpy-discussion] Distribution Functions Change Behavior In-Reply-To: <4db580fd0804280829w3c2ba57fs7b79ede3a15037d7@mail.gmail.com> References: <4db580fd0804280829w3c2ba57fs7b79ede3a15037d7@mail.gmail.com> Message-ID: <4db580fd0804280834q23c3e3dehae7ea081343d3558@mail.gmail.com> Wait, I think I see what Alan is saying. When you use a gaussian approximation on truncated data, the accuracy of the truncation is very dependent on where in the interval the mean is. If it's near the edges, the results will be worse. The width of the interval, though, is a separate factor. --Hoyt On Mon, Apr 28, 2008 at 8:29 AM, Hoyt Koepke wrote: > I may not understand what you are asking, Rich, but I'm not sure I > agree with Alan. A Gaussian fit to data x should fit exactly as well > as data fit to ax, a > 0, just with a variance a^2 times the original. > The only way this would not be true is if: > > 1. You are not fitting the variance, but only the mean > 2. There's some numerical issue (like some of the data are represented > as integers, etc.) > > Don't know if it could be one of those issues... > > --Hoyt > > > > On Mon, Apr 28, 2008 at 7:18 AM, Alan G Isaac wrote: > > Hi Rich, > > > > If your data is truncated at zero, it is not Gaussian (drawn > > from a normal). You will notice this when you shrink the > > range of values (unless the variance is tiny). > > > > Cheers, > > Alan Isaac > > > > > > > > > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > > -- > +++++++++++++++++++++++++++++++++++ > Hoyt Koepke > UBC Department of Computer Science > http://www.cs.ubc.ca/~hoytak/ > hoytak at gmail.com > +++++++++++++++++++++++++++++++++++ > -- +++++++++++++++++++++++++++++++++++ Hoyt Koepke UBC Department of Computer Science http://www.cs.ubc.ca/~hoytak/ hoytak at gmail.com +++++++++++++++++++++++++++++++++++ From chris at percious.com Mon Apr 28 11:39:17 2008 From: chris at percious.com (Chris Perkins) Date: Mon, 28 Apr 2008 09:39:17 -0600 Subject: [Numpy-discussion] Python Student Internship - National Renewable Energy Laboratory Message-ID: <3f37c7c10804280839w1163695fp66fb61a9e64e5e22@mail.gmail.com> Student Intern ? Scientific Computing Group 5900 5900-7259 A student internship is available in the National Renewable Energy Laboratory's (NREL) Scientific Computing Group. NREL is the nation's primary laboratory for research, development and deployment of renewable energy and energy efficiency technologies. The intern will be supporting work concerning management of scientific and technical data. Our data group is cutting-edge with respect to capturing rapidly changing scientific metadata and allowing the scientists to relate different kinds of data in a meaningful way. We have an immediate opening for a summer student internship with possible extension to one year in our Golden, Colorado office. The position would be part-time (15 - 25 hours per week) during the school year and/or full time during the summer. DUTIES: Will include working with researchers on techniques to enable the capture and storage of technical data in a scientific setting. Your role in our development team would be to support data harvesting using existing software, and develop new visualization techniques for existing data sets. DESIRED QUALIFICATIONS: Undergraduate or graduate student in computer science or related field, with demonstrated experience in programming, databases and software development. Experience using agile techniques and test-driven development. Demonstrated of Unit Testing. Experience with major dynamic languages like Python, Ruby, or C#. PREFERRED: Demonstrated good writing skills and computer skills, specifically including programming in python and database use. Experience with systems related to management of scientific data. Candidate must be a US citizen. Qualified candidates should e-mail their resume to: Laura Davis NREL, Human Resources Office Reference: Req. #5900-7259 E-Mail: laura_da... @nrel.gov ========================================= feel free to email me with any questions. cheers. -chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From rshepard at appl-ecosys.com Mon Apr 28 11:40:24 2008 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Mon, 28 Apr 2008 08:40:24 -0700 (PDT) Subject: [Numpy-discussion] Distribution Functions Change Behavior In-Reply-To: <4db580fd0804280829w3c2ba57fs7b79ede3a15037d7@mail.gmail.com> References: <4db580fd0804280829w3c2ba57fs7b79ede3a15037d7@mail.gmail.com> Message-ID: On Mon, 28 Apr 2008, Hoyt Koepke wrote: > A Gaussian fit to data x should fit exactly as well as data fit to ax, a > > 0, just with a variance a^2 times the original. The only way this would > not be true is if: Hoyt, This is what I expected, too. > 1. You are not fitting the variance, but only the mean That's probably it; I'm using the midpoint and the width as a surrogate for FWHM. However, by adjusting the FWHM for the Gaussian curve, and tau for the sigmoid curves when pH is to be plotted, I can achieve the results I need. Just why I don't really understand, but since this is a very small part of a large, complex, approximate reasoning model I'll happily remain ignorant as I move on to testing the rest of the code. Thanks to both you and Alan, Rich -- Richard B. Shepard, Ph.D. | Integrity Credibility Applied Ecosystem Services, Inc. | Innovation Voice: 503-667-4517 Fax: 503-667-8863 From stefan at sun.ac.za Mon Apr 28 12:00:16 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 28 Apr 2008 18:00:16 +0200 Subject: [Numpy-discussion] Starting to work on runtime plugin system for plugin (automatic sse optimization, etc...) In-Reply-To: <4815E2E9.5020506@ar.media.kyoto-u.ac.jp> References: <4815E2E9.5020506@ar.media.kyoto-u.ac.jp> Message-ID: <9457e7c80804280900r302fbfach788b51d7109faf82@mail.gmail.com> 2008/4/28 David Cournapeau : > - this could also be used for core numpy, for example ufuncs: if we > want to start implementing some tight loop with aggressively optimized > code (SSE, etc...), we could again ship with a default pure C > implementation, and choose the best one at runtime. This would be a *fantastic* addition, especially if a user can add his own ufuncs written in, say Cython. > Right now, the prototype does not do much, and only works for linux; I > mainly focused on automatic generation of the plugin from a list of > functions, and transparent use from numpy point of view. It provides the > plugin api through pure function pointers, without the need for the user > to be aware of it. For example, if you have an api with the following > functions: > > void foo1(); > int foo2(); > int foo3(int); > int foo4(double* , double*); > int foo5(double* , double*, int); > > The current implementation would build the boilerplate to load those > functions, etc... and you would just use those functions in numpy like > the following: I assume that, since you call it a plugin system, it can be done at runtime a-la ctypes? Cheers St?fan From aisaac at american.edu Mon Apr 28 12:02:32 2008 From: aisaac at american.edu (Alan Isaac) Date: Mon, 28 Apr 2008 12:02:32 -0400 Subject: [Numpy-discussion] Distribution Functions Change Behavior In-Reply-To: <4db580fd0804280829w3c2ba57fs7b79ede3a15037d7@mail.gmail.com> References: <4db580fd0804280829w3c2ba57fs7b79ede3a15037d7@mail.gmail.com> Message-ID: On Mon, 28 Apr 2008, Hoyt Koepke wrote: > I may not understand what you are asking, Rich, but I'm > not sure I agree with Alan. A Gaussian fit to data > x should fit exactly as well as data fit to ax, a > 0, > just with a variance a^2 times the original My point was different. If you truncate at zero but your mean is at 50 and your standard deviation is 10, you will hardly notice the truncation. If your mean is at 14 and the standard deviation is 10, you will definitely see the truncation. That's what I understood Rich to be seeing. Cheers, Alan Isaac From cournape at gmail.com Mon Apr 28 12:31:49 2008 From: cournape at gmail.com (David Cournapeau) Date: Tue, 29 Apr 2008 01:31:49 +0900 Subject: [Numpy-discussion] Starting to work on runtime plugin system for plugin (automatic sse optimization, etc...) In-Reply-To: <9457e7c80804280900r302fbfach788b51d7109faf82@mail.gmail.com> References: <4815E2E9.5020506@ar.media.kyoto-u.ac.jp> <9457e7c80804280900r302fbfach788b51d7109faf82@mail.gmail.com> Message-ID: <5b8d13220804280931l1296eb2fj5c268f69a7475484@mail.gmail.com> On Tue, Apr 29, 2008 at 1:00 AM, St?fan van der Walt wrote: > I assume that, since you call it a plugin system, it can be done at > runtime a-la ctypes? I am not sure to understand what you mean exactly by a-la ctypes, but yes, the actual implementation of the npyw_* functions would be decided at runtime, that's the whole point. For example, instead of using directly cblas_*dot* functions in blasdot, we would use npyw_cblas*dot* functions, which would point to something in SSE3 optimized atlas if run on SSE3 cpu, etc... For the actual code change in numpy and scipy to be minimal, it should only involve renaming functions, which is what this first prototype focus on. Once the sytem is ready to be integrated (not anytime soon; for once, we would need support for the build system to build dynamically loaded libraries), this would mean numpy and scipy would work like matlab, which ship with different blas/lapack (atlas_sse, atlas_sse2, mkl), without the user to have to deal with this kind of low level details. It would also help to make (if only by making the incentive) a cleaner difference between pure C implementation and python C api boilerplate code. I think that optimizing some ufuncs with SSE and co without a runtime optimization would be a nightmare to deploy otherwise. It is basically linked to the directions I am the most interested in for future python numpy releases cheers, David From matthew.brett at gmail.com Mon Apr 28 13:44:27 2008 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 28 Apr 2008 17:44:27 +0000 Subject: [Numpy-discussion] Starting to work on runtime plugin system for plugin (automatic sse optimization, etc...) In-Reply-To: <9457e7c80804280900r302fbfach788b51d7109faf82@mail.gmail.com> References: <4815E2E9.5020506@ar.media.kyoto-u.ac.jp> <9457e7c80804280900r302fbfach788b51d7109faf82@mail.gmail.com> Message-ID: <1e2af89e0804281044i626a7191l65e687e8ecb47668@mail.gmail.com> > This would be a *fantastic* addition, especially if a user can add his > own ufuncs written in, say Cython. I'd like to add some large number to whatever *fantastic* means in terms of +N! Best, Matthew From tim.hochberg at ieee.org Mon Apr 28 14:54:37 2008 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Mon, 28 Apr 2008 11:54:37 -0700 Subject: [Numpy-discussion] prerelease proposal for matrix behavior In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com> <9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com> <481231DC.2010900@noaa.gov> <48128844.7040509@enthought.com> Message-ID: On Sat, Apr 26, 2008 at 7:12 AM, Alan G Isaac wrote: > On Fri, 25 Apr 2008, "Travis E. Oliphant" apparently wrote: > > At this point, I'm leaning in the direction of the > > RowVector / ColumnVector approach (even if these are not > > really advertised and just used during indexing). > > I believe that this conflicts with submatrix extraction. > Details follow. > > Suppose ``x`` is a matrix, and I say ``v=A[0]``. > What is ``v``? > > 1. If ``v`` is a 1d array, its behavior is known. > E.g., ``v[0]`` is the first element > and ``v[0,0]`` is an IndexError. > If we want to keep submatrix extraction (as we should), > we give up ``x[0] == x[0,:]``. > > 2. If ``v`` is a matrix, its behavior can be learned > by the experienced but breaks basic expectations > (the x[0][0] problem). > > 3. If ``v`` is a "RowVector" what behavior do we get? > Suppose the idea is that this will rescue > ``x[0][0]==x[0,0]`` **and** ``x[0]=x[0,:]``. > But then we must give up that ``x[0,:]`` is a submatrix. > Must ``v`` deviate from submatrix behavior in an important way? > Yes: ``v[0][0]`` is an IndexError. > (Note that the same problem arises if we just override > ``__getitem__`` for the matrix class to special case > row and column matrices.) > > Since submatrix extraction is fundamental, I think it is > a *very* bad idea to give it up. If so, then under the > RowVector proposal we must give up ``x[0]==x[0,:]``. > But this is just what we give up with > the much simpler proposal that ``v`` be a 1d array. > I may have missed something since I've been getting these emails all out of order for some reason and it's been hard to follow. Can you clarify what you mean by submatrix extraction? It sounds like you want to be able index into an MxN array and get out a 1xN or Mx1 matrix. If that's the case, wouldn't the natural way to spell that under the RowVector/ColumnVector approach (and most others that I can think of) be: m2 = m1[:1] or m2 = m1[:,:1] Here we're extracting the first row and column respectively as a matrix. This doesn't appear to have any of the conflicts with row/vector extraction and in general meshes much better with arrays than having some magic associated with appending an extra ':', which seems ill advised to me. -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Mon Apr 28 15:02:57 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 28 Apr 2008 12:02:57 -0700 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <20080426100938.GD24606@phare.normalesup.org> References: <20080426100938.GD24606@phare.normalesup.org> Message-ID: <48161F61.5090204@noaa.gov> Gael Varoquaux wrote: > On Fri, Apr 25, 2008 at 01:40:29PM -0400, Alan G Isaac wrote: >> In contrast, there *is* universal agreement that >> x[0][0]==x[0,0] is desirable. Or so I've understood the >> discussion. > I don't know why people are indexing matrices with A[x][y], but they > shouldn't. I think there has been a misunderstanding here. I don't think anyone is suggesting that if a coder wants an element of a matrix, that s/he should write that: element = M[i,j] Rather, that one might want to extract a row from a Matrix, and then index THAT object with a scalar: row = M[i] now do something with each element in that row: something = row[j] This is a rational, normal thing to do, and I think the desire for this is the core of this entire discussion, coming from the fact that in the current version of matrix both of M[i] and M[i,:] yield a matrix, which is 2-d, and cannot be indexed with a scalar. This is odd, as it is pretty natural to expect a single row (or column) to behave like a 1-d object. Alan G Isaac wrote: > I believe that this conflicts with submatrix extraction. > If we want to keep submatrix extraction (as we should), why? with 2-d arrays, you get a subarray with: A[i:j,:] and a 1-d array with A[i,:] -- why does that need to be different for Matrices? > 3. If ``v`` is a "RowVector" what behavior do we get? > Suppose the idea is that this will rescue > ``x[0][0]==x[0,0]`` **and** ``x[0]=x[0,:]``. > But then we must give up that ``x[0,:]`` is a submatrix. Correct, but why should it be -- we can get a submatrix with slicing, as above. Indeed, I was about to post this comment the other day, as someone was concerned that there needs to be a distinction between a ColumnVector and a Matrix that happens to have a second dimension of one. I think that the Vector proposal satisfies that: if you have code like: SubMatrix = M[i:j, k] Then you will always get a SubMatrix, even if j == i+1. If you index like so: Vector = M[i,k] you will always get a vector (a 1-d object) > Must ``v`` deviate from submatrix behavior in an important way? > Yes: ``v[0][0]`` is an IndexError. Correct, but if you're writing the code that generated that vector, you'd know it was a 1-d object. > Since submatrix extraction is fundamental, I think it is > a *very* bad idea to give it up. I agree -- but we're not giving it up, what we are doing is making a distinction between extracting a single row or column, and extracting a submatrix (that may or may not be a single row or column) -- just like that distinction is make for regular old ndarrays. > RowVector proposal we must give up ``x[0]==x[0,:]``. > But this is just what we give up with > the much simpler proposal that ``v`` be a 1d array. no, we also give up that v acts like a row or column (particularly column) vector in computation (*, **) We still need real linear algebra computation examples.... Alan G Isaac wrote: > I weight the future more heavily. We are approaching a last > chance to do things better, and we should seize it. Yes, but it seems that while a consensus will not be reached in time for 1.1, there is one that a change will probably occur in the next version, so there is a lot to be said for waiting until a proposal is settled on, then make the whole change at once. > The right questions looking forward: > > - what behavior allows the most generic code? > - what behavior breaks fewest expectations? > - what behavior is most useful? Yes, but I'm going to try to put down what I think are the key, very simple, questions: 1) Do we want to be able to extract a single row or column from a Matrix, and have it be indexable like a 1-d object? 2) Do we want to be able to do that without having to explicitly call the Array attribute: M.A[i]? 3) Do we want to have those 1-d objects act like Row or Column vectors in linear algebra operations (and broadcasting)? 4) Do we want to require slice notation to get a submatrix that happens to have a single row or column? If we want (1) and (2), then we need to make a change. If we want (3) and (4), then something like the Row/ColumnVector proposal is needed. I really think it's that simple. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Mon Apr 28 15:03:25 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 28 Apr 2008 12:03:25 -0700 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <20080426180143.GH17624@phare.normalesup.org> References: <20080426152332.GB17624@phare.normalesup.org> <20080426180143.GH17624@phare.normalesup.org> Message-ID: <48161F7D.3090805@noaa.gov> Gael Varoquaux wrote: > * to use a syntax similar to dictionnaries: > > for row in A.rows(): > for col in row.cols() > > I actually think this is much better than the code you currently use, > > * or implement row and column objects. > > The problem in your code is that you do not distinguish iterating over > rows and columns, and in linear algebra these are different objects. The > two solutions I propose do this distinction. My favorite one is the first > one (the rows and cols methods). What would rows() and columns() return? I'd like them to return a vector object. If they return matrices, then you can't index them with a single value, if they return 1-d arrays, then you can't distinguish between rows and columns (which may be OK). Travis E. Oliphant wrote: > I'm not personally persuaded by the iteration argument, because we can > change iteration independently of mapping (__getitem__) access. or just use: for row in M.A: ... How hard is that? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From stefan at sun.ac.za Mon Apr 28 15:03:11 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 28 Apr 2008 21:03:11 +0200 Subject: [Numpy-discussion] Starting to work on runtime plugin system for plugin (automatic sse optimization, etc...) In-Reply-To: <5b8d13220804280931l1296eb2fj5c268f69a7475484@mail.gmail.com> References: <4815E2E9.5020506@ar.media.kyoto-u.ac.jp> <9457e7c80804280900r302fbfach788b51d7109faf82@mail.gmail.com> <5b8d13220804280931l1296eb2fj5c268f69a7475484@mail.gmail.com> Message-ID: <9457e7c80804281203oda32fc2x3e9b2d6a4afec38d@mail.gmail.com> 2008/4/28 David Cournapeau : > On Tue, Apr 29, 2008 at 1:00 AM, St?fan van der Walt wrote: > > > I assume that, since you call it a plugin system, it can be done at > > runtime a-la ctypes? What I meant was: would the plugin "slots" be decided beforehand, or could we manipulate them at runtime? I.e. what I would really enjoy doing is define arbitrary ufuncs and plug them in (not only the blas funcs and a select few others). Either way, this is good news for distributors -- it would make it so much easier to provide an optimised version of scipy to windows users, who can't easily compile it themselves. Cheers St?fan From Chris.Barker at noaa.gov Mon Apr 28 15:10:35 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 28 Apr 2008 12:10:35 -0700 Subject: [Numpy-discussion] setting rows of a numpy array In-Reply-To: References: Message-ID: <4816212B.6080400@noaa.gov> A couple suggestions: wilson wrote: > I am setting > the each row using pixel values of each image in some folder. As an image is generally a 2-d array of pixels, you may want to use a N X Width X Height 3-d array, rather than flattening each image to a single row. > def putrow(myarray,inrow ,rownum): > myarray[rownum,:]=inrow I'm not sure I'd both writing a function for a one liner like that -- why not just put it in the code and avoid the function overhead? > for i in range(numofimgs): > pixarray=asfarray(myimagefileslist[i]._pixelarray) > putrow(myimagesdata,pixarray,i) no need to make an array out of _pixelarray first -- you can just do: for i in range(numofimgs): myimagesdata[i,:] = myimagefileslist[i]._pixelarray Can you load the data right into myimagesdata up front, rather than creating your MyImage instances first? That'll save you a bunch of data copying. In fact, if you build the big array, you can then have the MyImage objects reference a view intot he big array, so no data copying: myimagefileslist[i]._pixelarray = myimagesdata[i,:] will give you a 1-d array that shares data with the main 2-d array. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Mon Apr 28 15:29:06 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 28 Apr 2008 12:29:06 -0700 Subject: [Numpy-discussion] leaving matrix indexing discussion In-Reply-To: References: Message-ID: <48162582.7000702@noaa.gov> Alan G Isaac wrote: > I am starting to feel that I have abused my interlocutors. Not at all -- you have contributed a great deal to the conversation, and your maintenance of: > Is particularly valuable. Thank you, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Mon Apr 28 15:37:17 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 28 Apr 2008 12:37:17 -0700 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <48161F61.5090204@noaa.gov> References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> Message-ID: <4816276D.9030200@noaa.gov> oops, typo! Christopher Barker wrote: > Gael Varoquaux wrote: >> I don't know why people are indexing matrices with A[x][y], but they >> shouldn't. > > I think there has been a misunderstanding here. I don't think anyone is > suggesting that if a coder wants an element of a matrix, that s/he > should write that: > > element = M[i,j] I meant -- no one is suggesting: element = M[i][j] because, of course, M[i,j] is exactly what you should do do get an element. Which, by the way, brings up another inconsistency in the current behavior: >>> M matrix([[1, 2, 3], [4, 5, 6], [7, 8, 9]]) >>> M[1,1] 5 >>> M[1,:] matrix([[4, 5, 6]]) so if you pull out a single element, you get a scalar(0-d), but if you pull out a row, you get a matrix(2-d). It makes me want vectors more... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From aisaac at american.edu Mon Apr 28 17:25:18 2008 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 28 Apr 2008 17:25:18 -0400 Subject: [Numpy-discussion] prerelease proposal for matrix behavior In-Reply-To: References: <85b5c3130804140306w1400d654ic407ca956d6bf645@mail.gmail.com><9457e7c80804250933ldd6dbe4t5b2fcffc0a027bbf@mail.gmail.com><481231DC.2010900@noaa.gov><48128844.7040509@enthought.com> Message-ID: On Mon, 28 Apr 2008, Timothy Hochberg apparently wrote: > Can you clarify what you > mean by submatrix extraction? It sounds like you want to be able index into > an MxN array and get out a 1xN or Mx1 matrix. If that's the case, wouldn't > the natural way to spell that under the RowVector/ColumnVector approach (and > most others that I can think of) be: > m2 = m1[:1] > or > m2 = m1[:,:1] I have tried to capture your view in proposal #6 at Cheers, Alan PS I would **love** to see things go this way! More consistency with ndarray behavior is better!! (Note: this conflicts with current behavior, where ``x[0]`` and ``x[0,:]`` return matrices.) From wfspotz at sandia.gov Mon Apr 28 17:44:11 2008 From: wfspotz at sandia.gov (Bill Spotz) Date: Mon, 28 Apr 2008 15:44:11 -0600 Subject: [Numpy-discussion] Matrix Class [was numpy release] In-Reply-To: <5DF32B21-7C5A-4A2B-ADC0-5AB603FF2C46@sandia.gov> References: <5DF32B21-7C5A-4A2B-ADC0-5AB603FF2C46@sandia.gov> Message-ID: On Apr 24, 2008, at 8:52 PM, Bill Spotz wrote: > On Apr 24, 2008, at 5:45 PM, Timothy Hochberg wrote: >> Bill Spotz wrote: >> > I have generally thought about this in the context of, say, a >> > Krylov-space iterative method, and what that type of interface >> would >> > lead to the most readable code. >> >> Can you whip up a small example, starting with the current >> implementation? > > This sounds like work. ;-) > > I'll shoot for putting something together this weekend. I think Tim > is right; we've been talking around in circles, and we need > something concrete. I have posted my example (using the current interface) at http://www.scipy.org/ConjugateGradientExample . I have also added a link to it from Alan's http://www.scipy.org/MatrixIndexing so that they will be somewhat connected. It is just one example, and provides one small area where the vector classes would be useful. I hope others will post similar examples, so that the design discussion can address concrete, agreed-upon situations. Note that I started using "row_vector" and "col_vector" as the class names in my discussion. No reason to re-introduce camel-case to numpy, and I think "col" is a universally recognized abbreviation for "column" and I like the symmetry of the three-letter "row_" and "col_" prefixes. ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Apr 28 21:03:07 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 28 Apr 2008 19:03:07 -0600 Subject: [Numpy-discussion] Indexing a matrix with a scalar and ticket #707 Message-ID: Yes, indeed. Ticket #707: numpy.array fails if the input is a list of matrixes (with more then one column). The subroutine discover_dimensions in arrayobject.c indexes a matrix with a scalar. It is a recursive routine and expects to find the next lower dimension as it recurses down into the matrix. This works fine with matrices with one column because the matrix row also has first dimension 1. Things don't do so well if the matrices have more than one column. I expect this is just the tip of a big pile of, um, stuff. The easy fix is to make indexing by scalars the same for matrices and arrays. So I think the best thing to do is provide that and figure out how to get the subarrays some other way. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Apr 28 21:41:22 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 28 Apr 2008 19:41:22 -0600 Subject: [Numpy-discussion] Indexing a matrix with a scalar and ticket #707 In-Reply-To: References: Message-ID: On Mon, Apr 28, 2008 at 7:03 PM, Charles R Harris wrote: > Yes, indeed. > > Ticket #707: numpy.array fails if the input is a list of matrixes (with > more then one column). > > The subroutine discover_dimensions in arrayobject.c indexes a matrix with > a scalar. It is a recursive routine and expects to find the next lower > dimension as it recurses down into the matrix. This works fine with matrices > with one column because the matrix row also has first dimension 1. Things > don't do so well if the matrices have more than one column. I expect this is > just the tip of a big pile of, um, stuff. The easy fix is to make indexing > by scalars the same for matrices and arrays. So I think the best thing to do > is provide that and figure out how to get the subarrays some other way. > Let me add that this settles the case of the matrix iterator for me: it has to return a 1D array. That behavior is part of the numpy contract that the core code is built on. The behavior of scalar indexing may also be part of that contract, but I am not sure of that. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Mon Apr 28 21:48:47 2008 From: cournape at gmail.com (David Cournapeau) Date: Tue, 29 Apr 2008 10:48:47 +0900 Subject: [Numpy-discussion] Starting to work on runtime plugin system for plugin (automatic sse optimization, etc...) In-Reply-To: <9457e7c80804281203oda32fc2x3e9b2d6a4afec38d@mail.gmail.com> References: <4815E2E9.5020506@ar.media.kyoto-u.ac.jp> <9457e7c80804280900r302fbfach788b51d7109faf82@mail.gmail.com> <5b8d13220804280931l1296eb2fj5c268f69a7475484@mail.gmail.com> <9457e7c80804281203oda32fc2x3e9b2d6a4afec38d@mail.gmail.com> Message-ID: <5b8d13220804281848w5ba570aek4091cb01e3adf04@mail.gmail.com> On Tue, Apr 29, 2008 at 4:03 AM, St?fan van der Walt wrote: > > What I meant was: would the plugin "slots" be decided beforehand, or > could we manipulate them at runtime? I.e. what I would really enjoy > doing is define arbitrary ufuncs and plug them in (not only the blas > funcs and a select few others). For each set of functions, there is a init function (just like python extensions) you must call before calling any function, and you can do what you want there: this could be controlled by environment variables and the likes. I still don't have a clear idea on the API for handling the different ways you would like to load the plugin (multi thread vs mono-thread, SSE1/2/3/4 vs no SSE, etc...), though. I would prefer every plugin to have exactly the same signature for the init function, and would like to avoid as much as possible global state. The thing I wanted to do but quickly gave up because of the complexity is unloading/reloading a plugin. For example, at the beginning, you may want to load atlas for the blas, and then use mkl instead. cheers, David From charlesr.harris at gmail.com Mon Apr 28 22:02:52 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 28 Apr 2008 20:02:52 -0600 Subject: [Numpy-discussion] Indexing a matrix with a scalar and ticket #707 In-Reply-To: References: Message-ID: On Mon, Apr 28, 2008 at 7:41 PM, Charles R Harris wrote: > > > On Mon, Apr 28, 2008 at 7:03 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > > > Yes, indeed. > > > > Ticket #707: numpy.array fails if the input is a list of matrixes (with > > more then one column). > > > > The subroutine discover_dimensions in arrayobject.c indexes a matrix > > with a scalar. It is a recursive routine and expects to find the next lower > > dimension as it recurses down into the matrix. This works fine with matrices > > with one column because the matrix row also has first dimension 1. Things > > don't do so well if the matrices have more than one column. I expect this is > > just the tip of a big pile of, um, stuff. The easy fix is to make indexing > > by scalars the same for matrices and arrays. So I think the best thing to do > > is provide that and figure out how to get the subarrays some other way. > > > > Let me add that this settles the case of the matrix iterator for me: it > has to return a 1D array. That behavior is part of the numpy contract that > the core code is built on. The behavior of scalar indexing may also be part > of that contract, but I am not sure of that. > In particular, if a is a matrix, then for row in matrix: for elem in row : pass has to iterate through the elements of a. Chuck > > Chuck > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Tue Apr 29 00:14:58 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 29 Apr 2008 00:14:58 -0400 Subject: [Numpy-discussion] should outer take an output argument? Message-ID: As I was looking at Bill's conjugate gradient posting, I found myself wondering if there would be a payoff to an output argument for ``numpy.outer``. (It is fairly natural to repeatedly recreate the outer product of the adjusted residuals, which is only needed during a single iteration.) Cheers, Alan Isaac From aisaac at american.edu Tue Apr 29 00:15:02 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 29 Apr 2008 00:15:02 -0400 Subject: [Numpy-discussion] Matrix Class [was numpy release] In-Reply-To: References: <5DF32B21-7C5A-4A2B-ADC0-5AB603FF2C46@sandia.gov> Message-ID: On Mon, 28 Apr 2008, Bill Spotz apparently wrote: > http://www.scipy.org/ConjugateGradientExample ... provides > one small area where the vector classes would be useful. Maybe not. I posted an alternate version of your algorithm, just below yours, sticking very close to your example. Cheers, Alan From wfspotz at sandia.gov Tue Apr 29 00:47:51 2008 From: wfspotz at sandia.gov (Bill Spotz) Date: Mon, 28 Apr 2008 22:47:51 -0600 Subject: [Numpy-discussion] Matrix Class [was numpy release] In-Reply-To: References: <5DF32B21-7C5A-4A2B-ADC0-5AB603FF2C46@sandia.gov> Message-ID: <2E084D31-B568-42D6-B2F8-3583F8E9B287@sandia.gov> On Apr 28, 2008, at 10:15 PM, Alan G Isaac wrote: > On Mon, 28 Apr 2008, Bill Spotz apparently wrote: >> http://www.scipy.org/ConjugateGradientExample ... provides >> one small area where the vector classes would be useful. > > Maybe not. > I posted an alternate version of your algorithm, > just below yours, > sticking very close to your example. Alan, Nice. To quote: "The second version of the algorithm provided above suggests that little if anything was gained by the use of matrices." If matrix multiplication in my example is replaced with np.dot() in yours, then when IS anything gained by using matrices? As for this example, my version should work with a properly implemented sparse_matrix A, but the array approach precludes that. That is to say, I could convert A to a matrix if it is provided as an array, but you could not convert a sparse_matrix to an array. ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From wfspotz at sandia.gov Tue Apr 29 00:52:24 2008 From: wfspotz at sandia.gov (Bill Spotz) Date: Mon, 28 Apr 2008 22:52:24 -0600 Subject: [Numpy-discussion] Matrix Class [was numpy release] In-Reply-To: <2E084D31-B568-42D6-B2F8-3583F8E9B287@sandia.gov> References: <5DF32B21-7C5A-4A2B-ADC0-5AB603FF2C46@sandia.gov> <2E084D31-B568-42D6-B2F8-3583F8E9B287@sandia.gov> Message-ID: On Apr 28, 2008, at 10:47 PM, Bill Spotz wrote: > As for this example, my version should work with a properly > implemented sparse_matrix A, but the array approach precludes that. > That is to say, I could convert A to a matrix if it is provided as an > array, but you could not convert a sparse_matrix to an array. I may be jumping to conclusions here. We could conceivably implement a sparse_array class upon which the sparse_matrix class is based. ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From aisaac at american.edu Tue Apr 29 01:04:27 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 29 Apr 2008 01:04:27 -0400 Subject: [Numpy-discussion] Matrix Class [was numpy release] In-Reply-To: <2E084D31-B568-42D6-B2F8-3583F8E9B287@sandia.gov> References: <5DF32B21-7C5A-4A2B-ADC0-5AB603FF2C46@sandia.gov> <2E084D31-B568-42D6-B2F8-3583F8E9B287@sandia.gov> Message-ID: On Mon, 28 Apr 2008, Bill Spotz apparently wrote: > If matrix multiplication in my example is replaced with > np.dot() in yours, then when IS anything gained by using > matrices? When matrix algebra is clearer than array algebra. But that is not the case for this algorithm. (Just the opposite, I would say.) Simple example: (X.T * X).I * (X.T * Y) (Not that this is computationally nice ...) > As for this example, my version should work with a properly > implemented sparse_matrix A, but the array approach precludes that. > That is to say, I could convert A to a matrix if it is provided as an > array, but you could not convert a sparse_matrix to an array. A.todense().A? (I don't think you can save on memory use without substantial changes...) In an earlier discussion, I suggested that if iteration over matrices were to yield 1d arrays, then iteration over sparse matrices should yield 1d "sparse arrays". Cheers, Alan From david at ar.media.kyoto-u.ac.jp Tue Apr 29 01:40:49 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 29 Apr 2008 14:40:49 +0900 Subject: [Numpy-discussion] Starting to work on runtime plugin system for plugin (automatic sse optimization, etc...) In-Reply-To: <9457e7c80804281203oda32fc2x3e9b2d6a4afec38d@mail.gmail.com> References: <4815E2E9.5020506@ar.media.kyoto-u.ac.jp> <9457e7c80804280900r302fbfach788b51d7109faf82@mail.gmail.com> <5b8d13220804280931l1296eb2fj5c268f69a7475484@mail.gmail.com> <9457e7c80804281203oda32fc2x3e9b2d6a4afec38d@mail.gmail.com> Message-ID: <4816B4E1.1040404@ar.media.kyoto-u.ac.jp> St?fan van der Walt wrote: > > What I meant was: would the plugin "slots" be decided beforehand, or > could we manipulate them at runtime? I.e. what I would really enjoy > doing is define arbitrary ufuncs and plug them in (not only the blas > funcs and a select few others). > Do you want to define the ufuncs at runtime ? Or do you want to be able to select the ufuncs at runtime ? cheers, David From aisaac at american.edu Tue Apr 29 02:20:06 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 29 Apr 2008 02:20:06 -0400 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <48161F61.5090204@noaa.gov> References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> Message-ID: On Mon, 28 Apr 2008, Christopher Barker apparently wrote: > I'm going to try to put down what I think are the key, > very simple, questions: For useful reference, these are now here: Cheers, Alan From stefan at sun.ac.za Tue Apr 29 02:24:22 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 29 Apr 2008 08:24:22 +0200 Subject: [Numpy-discussion] Starting to work on runtime plugin system for plugin (automatic sse optimization, etc...) In-Reply-To: <4816B4E1.1040404@ar.media.kyoto-u.ac.jp> References: <4815E2E9.5020506@ar.media.kyoto-u.ac.jp> <9457e7c80804280900r302fbfach788b51d7109faf82@mail.gmail.com> <5b8d13220804280931l1296eb2fj5c268f69a7475484@mail.gmail.com> <9457e7c80804281203oda32fc2x3e9b2d6a4afec38d@mail.gmail.com> <4816B4E1.1040404@ar.media.kyoto-u.ac.jp> Message-ID: <9457e7c80804282324x1b4d8a50qb2493ccf377c209b@mail.gmail.com> 2008/4/29 David Cournapeau : > St?fan van der Walt wrote: > > > > What I meant was: would the plugin "slots" be decided beforehand, or > > could we manipulate them at runtime? I.e. what I would really enjoy > > doing is define arbitrary ufuncs and plug them in (not only the blas > > funcs and a select few others). > > > > Do you want to define the ufuncs at runtime ? Or do you want to be able > to select the ufuncs at runtime ? Both, eventually, but I realise that this is somewhat out of the scope of your original suggestion. Don't let me distract you, I'm very keen to see what you come up with! Cheers St?fan From hoytak at gmail.com Tue Apr 29 02:28:14 2008 From: hoytak at gmail.com (Hoyt Koepke) Date: Mon, 28 Apr 2008 23:28:14 -0700 Subject: [Numpy-discussion] types as functions convert 1 elm arrays to scalars Message-ID: <4db580fd0804282328t167bda30v4a220a92c1cd32a0@mail.gmail.com> Hello, I have a quick question that I'm hoping will improve my numpy understanding. I noticed some behavior when using float64 to convert a matrix type that I didn't expect: In [35]: b1 = array([1.0]) In [36]: float64(b1) Out[36]: 1.0 In [37]: b2 = array([1.0, 2.0]) In [38]: float64(b2) Out[38]: array([ 1., 2.]) I didn't expect calling float64 would convert b1 to a scalar. Seems like an inconsistency. I assume this is intentional, as someone would have noticed it a long time ago if not, so could someone explain the reasoning behind it? (or point me to a source that will help?) Thanks! --Hoyt -- +++++++++++++++++++++++++++++++++++ Hoyt Koepke UBC Department of Computer Science http://www.cs.ubc.ca/~hoytak/ hoytak at gmail.com +++++++++++++++++++++++++++++++++++ From charlesr.harris at gmail.com Tue Apr 29 02:39:29 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 29 Apr 2008 00:39:29 -0600 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> Message-ID: On Tue, Apr 29, 2008 at 12:20 AM, Alan G Isaac wrote: > On Mon, 28 Apr 2008, Christopher Barker apparently wrote: > > I'm going to try to put down what I think are the key, > > very simple, questions: > > For useful reference, these are now here: > > > Cheers, > Alan > > May I add that if I edit defmatrix.py to act like an array for scalar indexing, then the following works. In [1]: a = matrix(eye(2)) In [2]: array([a,a]) Out[2]: array([[[ 1., 0.], [ 0., 1.]], [[ 1., 0.], [ 0., 1.]]]) This generates an error with the current version of matrix and, frankly, I am not going to be bothered going all through the numpy c sources to special case matrices to fix that. Someone else can do it if they wish. There are recursive routines that expect the dimensions to decrease on each call. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Apr 29 02:51:14 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 29 Apr 2008 00:51:14 -0600 Subject: [Numpy-discussion] types as functions convert 1 elm arrays to scalars In-Reply-To: <4db580fd0804282328t167bda30v4a220a92c1cd32a0@mail.gmail.com> References: <4db580fd0804282328t167bda30v4a220a92c1cd32a0@mail.gmail.com> Message-ID: On Tue, Apr 29, 2008 at 12:28 AM, Hoyt Koepke wrote: > Hello, > > I have a quick question that I'm hoping will improve my numpy > understanding. I noticed some behavior when using float64 to convert > a matrix type that I didn't expect: > > > In [35]: b1 = array([1.0]) > > In [36]: float64(b1) > Out[36]: 1.0 > > In [37]: b2 = array([1.0, 2.0]) > > In [38]: float64(b2) > Out[38]: array([ 1., 2.]) > > > I didn't expect calling float64 would convert b1 to a scalar. Seems > like an inconsistency. I assume this is intentional, as someone would > have noticed it a long time ago if not, so could someone explain the > reasoning behind it? (or point me to a source that will help?) > It's inconsistent and looks like a bug: In [4]: float32(array([[[1]]])) Out[4]: array([[[ 1.]]], dtype=float32) In [5]: float64(array([[[1]]])) Out[5]: 1.0 Float64 is a bit special because it starts as the python float. Maybe Travis can say what the differences are. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Tue Apr 29 04:01:29 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 29 Apr 2008 17:01:29 +0900 Subject: [Numpy-discussion] Starting to work on runtime plugin system for plugin (automatic sse optimization, etc...) In-Reply-To: <9457e7c80804282324x1b4d8a50qb2493ccf377c209b@mail.gmail.com> References: <4815E2E9.5020506@ar.media.kyoto-u.ac.jp> <9457e7c80804280900r302fbfach788b51d7109faf82@mail.gmail.com> <5b8d13220804280931l1296eb2fj5c268f69a7475484@mail.gmail.com> <9457e7c80804281203oda32fc2x3e9b2d6a4afec38d@mail.gmail.com> <4816B4E1.1040404@ar.media.kyoto-u.ac.jp> <9457e7c80804282324x1b4d8a50qb2493ccf377c209b@mail.gmail.com> Message-ID: <4816D5D9.6000608@ar.media.kyoto-u.ac.jp> St?fan van der Walt wrote: > Don't let me distract you, I'm very keen to see what you come up with! > Well, the point of making my preliminary work public is to get "distracted", as you put it :) It is really easy to come up with something not that useful without feedback or people remarks. Typically, I would have never thought about the usefulness of defining ufunc functions on the fly; I have my own ideas on how/why I do things, but I would consider it a failure if it was limited to that, cheers, David From cimrman3 at ntc.zcu.cz Tue Apr 29 04:50:59 2008 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Tue, 29 Apr 2008 10:50:59 +0200 Subject: [Numpy-discussion] Matrix Class [was numpy release] In-Reply-To: References: <5DF32B21-7C5A-4A2B-ADC0-5AB603FF2C46@sandia.gov> <2E084D31-B568-42D6-B2F8-3583F8E9B287@sandia.gov> Message-ID: <4816E173.9020201@ntc.zcu.cz> Bill Spotz wrote: > On Apr 28, 2008, at 10:47 PM, Bill Spotz wrote: > >> As for this example, my version should work with a properly >> implemented sparse_matrix A, but the array approach precludes that. >> That is to say, I could convert A to a matrix if it is provided as an >> array, but you could not convert a sparse_matrix to an array. > > > I may be jumping to conclusions here. We could conceivably implement > a sparse_array class upon which the sparse_matrix class is based. FYI: In scipy, the iterative solvers (cg and like) should work with LinearOperator (a class with matvec, rmatvec, and eventually matmat methods) passed instead of the A matrix argument. There is a function aslinearoperator() is scipy.sparse.linalg, that constructs a LinearOperator instance from a numpy array, matrix and scipy sparse matrix. It could be generalized to accept a function for a matrix-free computation, too. Also LinearOperator.__mul__, __rmul__ could be easily defined via matvec, rmatvec. IMHO the all iterative solvers could use such a concept, to shield them from what the system matrix A or the preconditioner actually are and to be able writing them in a visually appealing way (i.e. close to what one writes on a paper). regards, r. From david at ar.media.kyoto-u.ac.jp Tue Apr 29 05:11:12 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 29 Apr 2008 18:11:12 +0900 Subject: [Numpy-discussion] Can anyone look at #759 Message-ID: <4816E630.2040407@ar.media.kyoto-u.ac.jp> Hi, I have a patch for an issue which had bugged me for some time, but since I am not familiar with that part of numpy code, I would prefer someone checking I am not doing anything stupid before checking in, http://www.scipy.org/scipy/numpy/ticket/759 cheers, David From wfspotz at sandia.gov Tue Apr 29 08:50:22 2008 From: wfspotz at sandia.gov (Bill Spotz) Date: Tue, 29 Apr 2008 06:50:22 -0600 Subject: [Numpy-discussion] Matrix Class [was numpy release] In-Reply-To: <4816E173.9020201@ntc.zcu.cz> References: <5DF32B21-7C5A-4A2B-ADC0-5AB603FF2C46@sandia.gov> <2E084D31-B568-42D6-B2F8-3583F8E9B287@sandia.gov> <4816E173.9020201@ntc.zcu.cz> Message-ID: <44C188F5-C5CD-4191-8987-19F9FB7018AA@sandia.gov> On Apr 29, 2008, at 2:50 AM, Robert Cimrman wrote: > FYI: > In scipy, the iterative solvers (cg and like) should work with > LinearOperator (a class with matvec, rmatvec, and eventually matmat > methods) passed instead of the A matrix argument. There is a function > aslinearoperator() is scipy.sparse.linalg, that constructs a > LinearOperator instance from a numpy array, matrix and scipy sparse > matrix. It could be generalized to accept a function for a matrix-free > computation, too. Also LinearOperator.__mul__, __rmul__ could be > easily > defined via matvec, rmatvec. > > IMHO the all iterative solvers could use such a concept, to shield > them > from what the system matrix A or the preconditioner actually are and > to > be able writing them in a visually appealing way (i.e. close to what > one > writes on a paper). I agree. This is very close to what we use in Sandia's C++ Trilinos project: http://trilinos.sandia.gov. The linear algebra services package (Epetra: http://trilinos.sandia.gov/packages/epetra) supports the following class hierarchies: Epetra_Operator <- Epetra_RowMatrix <- Epetra_CrsMatrix Epetra_MultiVector <- Epetra_Vector (Where "Crs" stands for "compressed row storage"). A typical solver package (e.g. AztecOO: http://trilinos.sandia.gov/packages/aztecoo) defines solver routines that take an Epetra_Operator for A (and preconditioner P) and an Epetra_MultiVector for b and x. The most common way to call them is with an Epetra_CrsMatrix and an Epetra_Vector, but the flexibility is there. ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From stefan at sun.ac.za Tue Apr 29 09:24:58 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 29 Apr 2008 15:24:58 +0200 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> Message-ID: <9457e7c80804290624n754e74a6w6a1bb0295ae4a79b@mail.gmail.com> Hi Charles 2008/4/29 Charles R Harris : > May I add that if I edit defmatrix.py to act like an array for scalar > indexing, then the following works. > > In [1]: a = matrix(eye(2)) > > In [2]: array([a,a]) > Out[2]: > array([[[ 1., 0.], > [ 0., 1.]], > > [[ 1., 0.], > [ 0., 1.]]]) > > This generates an error with the current version of matrix and, frankly, I > am not going to be bothered going all through the numpy c sources to special > case matrices to fix that. Someone else can do it if they wish. There are > recursive routines that expect the dimensions to decrease on each call. I'd also like to see matrices become proper hierarchical containers -- the question is just how to do that. Thus far, I'm most convinced by the arguments for RowVectors/Columns, which leaves us with a sane model for doing linear algebra, while providing the enhancements you mentioned here and in comments to another ticket. We were thinking of raising a warning on scalar indexing for 1.1, but given the above, would that be sensical? Regards St?fan From aisaac at american.edu Tue Apr 29 09:54:52 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 29 Apr 2008 09:54:52 -0400 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org><48161F61.5090204@noaa.gov> Message-ID: On Tue, 29 Apr 2008, Charles R Harris apparently wrote: > There are recursive routines that expect the dimensions to > decrease on each call. I have added your example as anomaly #2 at the very top of the discussion page: Cheers, Alan Isaac From aisaac at american.edu Tue Apr 29 09:54:56 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 29 Apr 2008 09:54:56 -0400 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <9457e7c80804290624n754e74a6w6a1bb0295ae4a79b@mail.gmail.com> References: <20080426100938.GD24606@phare.normalesup.org><48161F61.5090204@noaa.gov> <9457e7c80804290624n754e74a6w6a1bb0295ae4a79b@mail.gmail.com> Message-ID: On Tue, 29 Apr 2008, St?fan van der Walt apparently wrote: > We were thinking of raising a warning on scalar indexing > for 1.1, but given the above, would that be sensical? There seem to be three basic proposals for scalar indexing: - raise an error - return a 1d array - return a new type of 1d container (apparently to be called a "vector" for some reason ...) I do not think anyone is still arguing in favor of the keeping the current behavior (where scalar indexing returns a matrix)? So the question seems to be: how best to warn users that the behavior of scalar indexing is very likely to change. Cheers, Alan Isaac From charlesr.harris at gmail.com Tue Apr 29 10:01:13 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 29 Apr 2008 08:01:13 -0600 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <9457e7c80804290624n754e74a6w6a1bb0295ae4a79b@mail.gmail.com> References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <9457e7c80804290624n754e74a6w6a1bb0295ae4a79b@mail.gmail.com> Message-ID: On Tue, Apr 29, 2008 at 7:24 AM, St?fan van der Walt wrote: > Hi Charles > > 2008/4/29 Charles R Harris : > > May I add that if I edit defmatrix.py to act like an array for scalar > > indexing, then the following works. > > > > In [1]: a = matrix(eye(2)) > > > > In [2]: array([a,a]) > > Out[2]: > > array([[[ 1., 0.], > > [ 0., 1.]], > > > > [[ 1., 0.], > > [ 0., 1.]]]) > > > > This generates an error with the current version of matrix and, frankly, > I > > am not going to be bothered going all through the numpy c sources to > special > > case matrices to fix that. Someone else can do it if they wish. There > are > > recursive routines that expect the dimensions to decrease on each call. > > I'd also like to see matrices become proper hierarchical containers -- > the question is just how to do that. Thus far, I'm most convinced by > the arguments for RowVectors/Columns, which leaves us with a sane > model for doing linear algebra, while providing the enhancements you > mentioned here and in comments to another ticket. > > We were thinking of raising a warning on scalar indexing for 1.1, but > given the above, would that be sensical? > Hi Stefan, The numpy c routines call PySequence_GetItem(s,i) as well as ask for the length (first index), so I think these should be left as they are for arrays in order to guarantee that matrices are compatible with all the normal array operations. This means either returning special row objects that index as expected, or returning 1D arrays. I don't think the '*' operator has these problems, but in any case that is a well documented feature of matrices. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Tue Apr 29 11:01:35 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 29 Apr 2008 10:01:35 -0500 Subject: [Numpy-discussion] Can anyone look at #759 In-Reply-To: <4816E630.2040407@ar.media.kyoto-u.ac.jp> References: <4816E630.2040407@ar.media.kyoto-u.ac.jp> Message-ID: <4817384F.9000406@enthought.com> David Cournapeau wrote: > Hi, > > I have a patch for an issue which had bugged me for some time, but > since I am not familiar with that part of numpy code, I would prefer > someone checking I am not doing anything stupid before checking in, > It looks good to me. -teo From Glen.Mabey at swri.org Tue Apr 29 11:21:51 2008 From: Glen.Mabey at swri.org (Glen W. Mabey) Date: Tue, 29 Apr 2008 10:21:51 -0500 Subject: [Numpy-discussion] failure building numpy using icc In-Reply-To: <3d375d730803010017r1505e28fwd68bc554060b5ba3@mail.gmail.com> References: <20080228192138.GA21482@swri.org> <3d375d730803010017r1505e28fwd68bc554060b5ba3@mail.gmail.com> Message-ID: <20080429152151.GI23710@swri.org> Robert, I do appreciate your response to this question (oh way back when) and am just now getting back to this problem. On Sat, Mar 01, 2008 at 02:17:46AM -0600, Robert Kern wrote: > On Thu, Feb 28, 2008 at 1:21 PM, Glen W. Mabey wrote: > > Hello, > > > > I'm using svn numpy and get the following error upon executing > > > > /usr/local/bin/python2.5 setup.py config --noisy --cc=/opt/intel/cce/10.0.025/bin/icc --compiler=intel --fcompiler=intel build_clib build_ext > > > > I see: > > > > conv_template:> build/src.linux-x86_64-2.5/numpy/core/src/scalartypes.inc > > Traceback (most recent call last): > > File "setup.py", line 96, in > > setup_package() > > File "setup.py", line 89, in setup_package > > configuration=configuration ) > > File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080228_svn/numpy/distutils/core.py", line 184, in setup > > return old_setup(**new_attr) > > File "/usr/local/lib/python2.5/distutils/core.py", line 151, in setup > > dist.run_commands() > > File "/usr/local/lib/python2.5/distutils/dist.py", line 974, in run_commands > > self.run_command(cmd) > > File "/usr/local/lib/python2.5/distutils/dist.py", line 994, in run_command > > cmd_obj.run() > > File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080228_svn/numpy/distutils/command/build_ext.py", line 56, in run > > self.run_command('build_src') > > File "/usr/local/lib/python2.5/distutils/cmd.py", line 333, in run_command > > self.distribution.run_command(command) > > File "/usr/local/lib/python2.5/distutils/dist.py", line 994, in run_command > > cmd_obj.run() > > File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080228_svn/numpy/distutils/command/build_src.py", line 130, in run > > self.build_sources() > > File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080228_svn/numpy/distutils/command/build_src.py", line 147, in build_sources > > self.build_extension_sources(ext) > > File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080228_svn/numpy/distutils/command/build_src.py", line 252, in build_extension_sources > > sources = self.template_sources(sources, ext) > > File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080228_svn/numpy/distutils/command/build_src.py", line 359, in template_sources > > outstr = process_c_file(source) > > File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080228_svn/numpy/distutils/conv_template.py", line 185, in process_file > > % (sourcefile, process_str(''.join(lines)))) > > File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080228_svn/numpy/distutils/conv_template.py", line 150, in process_str > > newstr[sub[0]:sub[1]], sub[4]) > > File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080228_svn/numpy/distutils/conv_template.py", line 117, in expand_sub > > % (line, template_re.sub(namerepl, substr))) > > File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080228_svn/numpy/distutils/conv_template.py", line 113, in namerepl > > return names[name][thissub[0]] > > KeyError: 'PREFIX' > > > > > > And I do not see any errors when building the same svn version with gcc (on > > a different machine). > > > > I've unsuccessfully tried to follow that backtrace of functions to > > figure out exactly what is going on. > > > > Any hints/suggestions? > > Off-hand, no, sorry. I'm not sure why the compiler would matter in > this part of the code, though. Can you try using gcc on the same > machine? I did successfully use gcc on the same machine without any problems. However, I still encounter a similar error with today's svn. Now, I don't know anything about how code is produced within numpy, but within pdb I was able to find this string that some sort of pattern substitution is acting on. It suspiciously looks like the first function belongs after the last one ... static PyObject * gentype_ at name@(PyObject *m1) { PyObject *arr, *ret; arr = PyArray_FromScalar(m1, NULL); if (arr == NULL) return NULL; ret = arr->ob_type->tp_as_number->nb_ at name@(arr); Py_DECREF(arr); return ret; } /**end repeat**/ static int gentype_nonzero_number(PyObject *m1) { PyObject *arr; int ret; arr = PyArray_FromScalar(m1, NULL); if (arr == NULL) return -1; ret = arr->ob_type->tp_as_number->nb_nonzero(arr); Py_DECREF(arr); return ret; } static PyObject * gentype_str(PyObject *self) { PyArrayObject *arr; PyObject *ret; arr = (PyArrayObject *)PyArray_FromScalar(self, NULL); if (arr==NULL) return NULL; ret = PyObject_Str((PyObject *)arr); Py_DECREF(arr); return ret; } static PyObject * gentype_repr(PyObject *self) { PyArrayObject *arr; PyObject *ret; arr = (PyArrayObject *)PyArray_FromScalar(self, NULL); if (arr==NULL) return NULL; ret = PyObject_Str((PyObject *)arr); Py_DECREF(arr); return ret; } /**begin repeat #name=float, double, longdouble# #NAME=FLOAT, DOUBLE, LONGDOUBLE# #PREFIX=NPY_,NPY_,NPY_# */ static void format_ at name@(char *buf, size_t buflen, @name@ val, unsigned int precision) { char *cp; PyOS_snprintf(buf, buflen, "%.*" @PREFIX@@NAME at _FMT, precision, val); cp = buf; if (*cp == '-') cp++; for (; *cp != '\0'; cp++) { if (!isdigit(Py_CHARMASK(*cp))) break; } if (*cp == '\0') { *cp++ = '.'; *cp++ = '0'; *cp++ = '\0'; } } And the command I ran to get to this point was: python setup.py config --noisy --cc=/opt/intel/cce/10.0.025/bin/icc --compiler=intel --fcompiler=intel build_clib build_ext Thank you, Glen Mabey From charlesr.harris at gmail.com Tue Apr 29 11:41:25 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 29 Apr 2008 09:41:25 -0600 Subject: [Numpy-discussion] failure building numpy using icc In-Reply-To: <20080429152151.GI23710@swri.org> References: <20080228192138.GA21482@swri.org> <3d375d730803010017r1505e28fwd68bc554060b5ba3@mail.gmail.com> <20080429152151.GI23710@swri.org> Message-ID: On Tue, Apr 29, 2008 at 9:21 AM, Glen W. Mabey wrote: > Robert, > Could you post the new output? The conv_template routine has changed a bit, you should make sure you have the latest version. Also, it is pure python and the use of icc shouldn't make a difference in substituting the keys in the template to generate the source files. Also try deleting the build directory before the new build. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Tue Apr 29 11:49:50 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 29 Apr 2008 10:49:50 -0500 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <9457e7c80804290624n754e74a6w6a1bb0295ae4a79b@mail.gmail.com> Message-ID: <4817439E.9020002@enthought.com> Charles R Harris wrote: > > > On Tue, Apr 29, 2008 at 7:24 AM, St?fan van der Walt > wrote: > > Hi Charles > > 2008/4/29 Charles R Harris >: > > May I add that if I edit defmatrix.py to act like an array for > scalar > > indexing, then the following works. > > > > In [1]: a = matrix(eye(2)) > > > > In [2]: array([a,a]) > > Out[2]: > > array([[[ 1., 0.], > > [ 0., 1.]], > > > > [[ 1., 0.], > > [ 0., 1.]]]) > > > > This generates an error with the current version of matrix and, > frankly, I > > am not going to be bothered going all through the numpy c > sources to special > > case matrices to fix that. Someone else can do it if they wish. > There are > > recursive routines that expect the dimensions to decrease on > each call. > > I'd also like to see matrices become proper hierarchical containers -- > the question is just how to do that. Thus far, I'm most convinced by > the arguments for RowVectors/Columns, which leaves us with a sane > model for doing linear algebra, while providing the enhancements you > mentioned here and in comments to another ticket. > > We were thinking of raising a warning on scalar indexing for 1.1, but > given the above, would that be sensical? > > > Hi Stefan, > > The numpy c routines call PySequence_GetItem(s,i) as well as ask for > the length (first index), so I think these should be left as they are > for arrays in order to guarantee that matrices are compatible with all > the normal array operations. This means either returning special row > objects that index as expected > or returning 1D arrays. I don't think the '*' operator has these > problems, but in any case that is a well documented feature of matrices. Thanks for looking in to this Chuck, I'm quite persuaded now that a[i] should return a 1-d object for arrays. In addition to the places Chuck identified, there are at least 2 other places where special code was written to work-around the expectation that item selection returns an object with reduced dimensionality (a special-cased .tolist for matrices and a special-cased getitem in the fancy indexing code). As the number of special-case work-arounds grows the more I'm convinced the conceptualization is wrong. So, I now believe we should change the a[i] for matrices to return a 1-d array. The only down-side I see is that a[i] != a[i,:] for matrices. However, matrix(a[i]) == a[i,:], and so I'm not sure there is really a problem, there. I also don't think that whatever problem may actually be buried in the fact that type(a[i]) != type(a[i,:]) is worse than the problem that several pieces of NumPy code actually expect hierarchical container behavior of multi-dimensional sequences. I don't think making the small change to have a[i] return 1-d arrays precludes us from that 1-d array being a bit more formalized in 1.2 as a RowVector should we choose that direction. It does, however, fix several real bugs right now. So, I'm +1 on making the following change: def __getitem__(self, index): + if isscalar(index): + return self.__array__()[index] -Travis From Glen.Mabey at swri.org Tue Apr 29 11:50:32 2008 From: Glen.Mabey at swri.org (Glen W. Mabey) Date: Tue, 29 Apr 2008 10:50:32 -0500 Subject: [Numpy-discussion] failure building numpy using icc In-Reply-To: References: <20080228192138.GA21482@swri.org> <3d375d730803010017r1505e28fwd68bc554060b5ba3@mail.gmail.com> <20080429152151.GI23710@swri.org> Message-ID: <20080429155032.GA4718@swri.org> On Tue, Apr 29, 2008 at 10:41:25AM -0500, Charles R Harris wrote: > Could you post the new output? The conv_template routine has changed a bit, you should make sure you have the latest version. Also, it is pure python and the use of icc shouldn't make a difference in substituting the keys in the template to generate the source files. Also try deleting the build directory before the new build. Okay, I deleted the build directory, and here is the full output: executing numpy/core/code_generators/generate_array_api.py adding 'build/src.linux-x86_64-2.5/numpy/core/__multiarray_api.h' to sources. creating build/src.linux-x86_64-2.5/numpy/core/src conv_template:> build/src.linux-x86_64-2.5/numpy/core/src/scalartypes.inc Traceback (most recent call last): File "setup.py", line 96, in setup_package() File "setup.py", line 89, in setup_package configuration=configuration ) File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/core.py", line 184, in setup return old_setup(**new_attr) File "/usr/local/lib/python2.5/distutils/core.py", line 151, in setup dist.run_commands() File "/usr/local/lib/python2.5/distutils/dist.py", line 974, in run_commands self.run_command(cmd) File "/usr/local/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/command/build_ext.py", line 56, in run self.run_command('build_src') File "/usr/local/lib/python2.5/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/local/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/command/build_src.py", line 130, in run self.build_sources() File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/command/build_src.py", line 147, in build_sources self.build_extension_sources(ext) File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/command/build_src.py", line 252, in build_extension_sources sources = self.template_sources(sources, ext) File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/command/build_src.py", line 359, in template_sources outstr = process_c_file(source) File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/conv_template.py", line 252, in process_file code = process_str(''.join(lines)) File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/conv_template.py", line 222, in process_str code.extend(parse_string(astr, global_names, 0, 1)) File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/conv_template.py", line 210, in parse_string newcode = parse_string(text, newenv, newlevel, newline) File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/conv_template.py", line 216, in parse_string code.append(replace_re.sub(replace, astr)) File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/conv_template.py", line 187, in replace raise KeyError, msg KeyError: "#line 497\n: 'PREFIX'" Thank you, Glen Mabey From charlesr.harris at gmail.com Tue Apr 29 12:24:56 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 29 Apr 2008 10:24:56 -0600 Subject: [Numpy-discussion] failure building numpy using icc In-Reply-To: <20080429155032.GA4718@swri.org> References: <20080228192138.GA21482@swri.org> <3d375d730803010017r1505e28fwd68bc554060b5ba3@mail.gmail.com> <20080429152151.GI23710@swri.org> <20080429155032.GA4718@swri.org> Message-ID: On Tue, Apr 29, 2008 at 9:50 AM, Glen W. Mabey wrote: > On Tue, Apr 29, 2008 at 10:41:25AM -0500, Charles R Harris wrote: > > Could you post the new output? The conv_template routine has changed a > bit, you should make sure you have the latest version. Also, it is pure > python and the use of icc shouldn't make a difference in substituting the > keys in the template to generate the source files. Also try deleting the > build directory before the new build. > > Okay, I deleted the build directory, and here is the full output: > > > executing numpy/core/code_generators/generate_array_api.py > adding 'build/src.linux-x86_64-2.5/numpy/core/__multiarray_api.h' to > sources. > creating build/src.linux-x86_64-2.5/numpy/core/src > conv_template:> build/src.linux-x86_64-2.5/numpy/core/src/scalartypes.inc > Traceback (most recent call last): > File "setup.py", line 96, in > setup_package() > File "setup.py", line 89, in setup_package > configuration=configuration ) > File > "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/core.py", > line 184, in setup > return old_setup(**new_attr) > File "/usr/local/lib/python2.5/distutils/core.py", line 151, in setup > dist.run_commands() > File "/usr/local/lib/python2.5/distutils/dist.py", line 974, in > run_commands > self.run_command(cmd) > File "/usr/local/lib/python2.5/distutils/dist.py", line 994, in > run_command > cmd_obj.run() > File > "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/command/build_ext.py", > line 56, in run > self.run_command('build_src') > File "/usr/local/lib/python2.5/distutils/cmd.py", line 333, in > run_command > self.distribution.run_command(command) > File "/usr/local/lib/python2.5/distutils/dist.py", line 994, in > run_command > cmd_obj.run() > File > "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/command/build_src.py", > line 130, in run > self.build_sources() > File > "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/command/build_src.py", > line 147, in build_sources > self.build_extension_sources(ext) > File > "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/command/build_src.py", > line 252, in build_extension_sources > sources = self.template_sources(sources, ext) > File > "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/command/build_src.py", > line 359, in template_sources > outstr = process_c_file(source) > File > "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/conv_template.py", > line 252, in process_file > code = process_str(''.join(lines)) > File > "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/conv_template.py", > line 222, in process_str > code.extend(parse_string(astr, global_names, 0, 1)) > File > "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/conv_template.py", > line 210, in parse_string > newcode = parse_string(text, newenv, newlevel, newline) > File > "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/conv_template.py", > line 216, in parse_string > code.append(replace_re.sub(replace, astr)) > File > "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/conv_template.py", > line 187, in replace > raise KeyError, msg > KeyError: "#line 497\n: 'PREFIX'" > > Hmm, something must be mucking with the sources. You can run conv_template on the pure file from the src directory with python ../../distutils/conv_template.py scalartypes.inc.src However, when called from within python the routine also processes the include files and I suspect that may be the problem. Try going into distutils/conv_template and find the line include_src_re = re.compile(r"(\n|\A)#include\s*['\"]" r"(?P[\w\d./\\]+[.]src)['\"]", re.I) and replace it with include_src_re = re.compile(r"(\n|\A)\s*#include\s*['\"]" r"(?P[\w\d./\\]+[.]src)['\"]", re.I) Which should knock off any whitespace before #include. This is not a fix, but it might help us find the problem. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Tue Apr 29 12:42:29 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 29 Apr 2008 09:42:29 -0700 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> Message-ID: <48174FF5.50005@noaa.gov> Alan G Isaac wrote: > For useful reference, these are now here: > Thanks Alan, Here are a few comments on that text (I know, I really should go dig up my scipy password...) > 1. Do we want to be able to extract a single row or column from a > matrix, and have it be indexable like a 1-d object? (Answer: certainly > yes for rows, but columns could be gotten from the transpose. I don't get this -- transpose of what? if you get a 1-d ndarray from: M[i,:], that can't be transposed -- it's 1-d. You could do M.T[0] but then you've lost the concept of it being a column. If we stick with M[i,:] being a (n,1) matrix, then it doesn't index like a 1-d object, and it doesn't need transposing anyway. > This is most useful, allows the most generic code, and breaks fewest > expectations.) > 2. Do we want to be able to do (1) without calling the Array > attribute: M.A[i]? (Yes: this is not less useful, allows the most > generic code, and breaks fewest expectations.) > 3. Do we want to have those 1-d objects to have an orientation so > that they act like Row or Column vectors in linear algebra operations > (and broadcasting)? (Unclear: the resulting functionality could be > provided by rows and columns attributes that yield corresponding > matrices.) no, it couldn't -- we can already use M[:,i] (or M[:,i:i+1] to get a matrix that is essentially a column -- however that can't be indexed with a scalar (see (1). That's kind of the point of this discussion. Or did you mean that you get a column by: np.matrix(M[:,i]).T ? yow! Maybe we need to separate the row and column questions -- particularly with Charles' discovery of how other parts of numpy expect rank reduction with indexing, I think we probably have a consensus that we need to be able to extract 1-d objects from matrixes with indexing -- at least rows. That leaves the column question: -- do we need an object that represents a column, that otherwise acts like a 1-d array (returns a scalar with scalar indexing)? I guess my key point is that the fact that seeing a Matrix as a collection of rows rather than columns is an accident of C memory arrangement -- if numpy (and python) had been written in Fortran, we'd have the opposite. I'd rather that not be built into the API more than required. We can only have one definition for M[i], it might as well be a row, but other than that, we can have symmetry: M[i,:] is row M[:,i] is a column and rows and columns behave the same way everywhere they "should": indexing, converting to ndarrays, ???, but not in linear algebra operations. Then, or course there are implementation details... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From aisaac at american.edu Tue Apr 29 13:11:36 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 29 Apr 2008 13:11:36 -0400 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <48174FF5.50005@noaa.gov> References: <20080426100938.GD24606@phare.normalesup.org><48161F61.5090204@noaa.gov> <48174FF5.50005@noaa.gov> Message-ID: >> On Tue, 29 Apr 2008, Christopher Barker apparently wrote: > Here are a few comments on that text In response, I have modified to try to clarify what was meant and perhaps to address your core concerns. Alan PS Perhaps I should note that I have come around to what I take to be Tim H's view that matrix indexing should should work just like ndarray indexing, except that when it produces a 2d result then a matrix is returned. (This means giving up the current behavior of ``x[0,:]``.) I also agree with those who want ``rows`` and ``cols`` attributes for iteration. A small difference: I would have these yield matrices. (I.e., no need for a new "vector" type.) From Glen.Mabey at swri.org Tue Apr 29 13:19:11 2008 From: Glen.Mabey at swri.org (Glen W. Mabey) Date: Tue, 29 Apr 2008 12:19:11 -0500 Subject: [Numpy-discussion] failure building numpy using icc In-Reply-To: References: <20080228192138.GA21482@swri.org> <3d375d730803010017r1505e28fwd68bc554060b5ba3@mail.gmail.com> <20080429152151.GI23710@swri.org> <20080429155032.GA4718@swri.org> Message-ID: <20080429171911.GB4718@swri.org> On Tue, Apr 29, 2008 at 11:24:56AM -0500, Charles R Harris wrote: > Hmm, something must be mucking with the sources. You can run conv_template on the pure file from the src directory with > > python ../../distutils/conv_template.py scalartypes.inc.src > > However, when called from within python the routine also processes the include files and I suspect that may be the problem. Try going into distutils/conv_template and find the line > > include_src_re = re.compile(r"(\n|\A)#include\s*['\"]" > r"(?P[\w\d./\\]+[.]src)['\"]", re.I) > > > and replace it with > > include_src_re = re.compile(r"(\n|\A)\s*#include\s*['\"]" > r"(?P[\w\d./\\]+[.]src)['\"]", re.I) > > Which should knock off any whitespace before #include. This is not a fix, but it might help us find the problem. Humm. I couldn't see any difference in the behavior. Is there something that I should be looking for? Glen From Chris.Barker at noaa.gov Tue Apr 29 13:29:15 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 29 Apr 2008 10:29:15 -0700 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <48174FF5.50005@noaa.gov> Message-ID: <48175AEB.3030008@noaa.gov> Alan G Isaac wrote: > Perhaps I should note that I have come around to what > I take to be Tim H's view that matrix indexing should should > work just like ndarray indexing, except that when it > produces a 2d result then a matrix is returned. (This means > giving up the current behavior of ``x[0,:]``.) I also agree > with those who want ``rows`` and ``cols`` attributes for > iteration. +1 on all of that. > A small difference: I would have these yield > matrices. (I.e., no need for a new "vector" type.) So the only thing we "give up" (which we don't have now anyway) is something that acts like column and can be indexed like a 1-d object. I still think that would be nice, but maybe we're coming close to a consensus about the rest of it. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From kwgoodman at gmail.com Tue Apr 29 13:40:23 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 29 Apr 2008 10:40:23 -0700 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <48174FF5.50005@noaa.gov> Message-ID: On Tue, Apr 29, 2008 at 10:11 AM, Alan G Isaac wrote: > indexing should should work just like ndarray indexing, > except that when it produces a 2d result then a matrix > is returned. (This means giving up the current behavior > of ``x[0,:]``.) I often use x[i,:] and x[:,i] where x is a matrix and i is a scalar. I hope this continues to return a matrix. From charlesr.harris at gmail.com Tue Apr 29 14:01:55 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 29 Apr 2008 12:01:55 -0600 Subject: [Numpy-discussion] failure building numpy using icc In-Reply-To: <20080429171911.GB4718@swri.org> References: <20080228192138.GA21482@swri.org> <3d375d730803010017r1505e28fwd68bc554060b5ba3@mail.gmail.com> <20080429152151.GI23710@swri.org> <20080429155032.GA4718@swri.org> <20080429171911.GB4718@swri.org> Message-ID: On Tue, Apr 29, 2008 at 11:19 AM, Glen W. Mabey wrote: > On Tue, Apr 29, 2008 at 11:24:56AM -0500, Charles R Harris wrote: > > Hmm, something must be mucking with the sources. You can run > conv_template on the pure file from the src directory with > > > > python ../../distutils/conv_template.py scalartypes.inc.src > > > > However, when called from within python the routine also processes the > include files and I suspect that may be the problem. Try going into > distutils/conv_template and find the line > > > > include_src_re = re.compile(r"(\n|\A)#include\s*['\"]" > > r"(?P[\w\d./\\]+[.]src)['\"]", re.I) > > > > > > and replace it with > > > > include_src_re = re.compile(r"(\n|\A)\s*#include\s*['\"]" > > r"(?P[\w\d./\\]+[.]src)['\"]", re.I) > > > > Which should knock off any whitespace before #include. This is not a > fix, but it might help us find the problem. > > Humm. I couldn't see any difference in the behavior. Is there > something that I should be looking for? > Not really, it was just a shot in the dark. The thing is, if conv_python is being run on the same source each time, it should give the same result. So I suspect that something is changing the sources and that is what I'm trying to track down. One of the build guys should know... Hmmm..., I notice that there is a line break in line 18 of scalartypes.inc.src that doesn't belong there. Could you fix that? And the line number of the loop is 497, which is the loop before the one with PREFIX defined. For some reason the parser is missing the /**end repeat**/ on line 509. Hmmm, try putting a blank line before it. Just fishing here... Why that should happen with icc and not with gnu is a mystery. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Tue Apr 29 14:11:43 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 29 Apr 2008 20:11:43 +0200 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <4817439E.9020002@enthought.com> References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <9457e7c80804290624n754e74a6w6a1bb0295ae4a79b@mail.gmail.com> <4817439E.9020002@enthought.com> Message-ID: <20080429181143.GG24600@phare.normalesup.org> On Tue, Apr 29, 2008 at 10:49:50AM -0500, Travis E. Oliphant wrote: > As the number of special-case work-arounds grows the more I'm convinced > the conceptualization is wrong. So, I now believe we should change the > a[i] for matrices to return a 1-d array. > The only down-side I see is that a[i] != a[i,:] for matrices. > However, matrix(a[i]) == a[i,:], and so I'm not sure there is really a > problem, there. I think we have simply replaced one problem by another one. I think this is a bad route to go. I think the only proposition that makes sens so far is the RowVector/ColumnVector. Yes you return a 1D object, but one that knows it is a linear algebra object, and not an array. I _really_ don't like a[i] != a[i,:]. I also don't like loosing the information that you are doing linear algebra. Ga?l From aisaac at american.edu Tue Apr 29 14:21:34 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 29 Apr 2008 14:21:34 -0400 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org><48161F61.5090204@noaa.gov><48174FF5.50005@noaa.gov> Message-ID: On Tue, 29 Apr 2008, Keith Goodman apparently wrote: > I often use x[i,:] and x[:,i] where x is a matrix and i is > a scalar. I hope this continues to return a matrix. 1. Could you give an example of the circumstances of this use? 2. Would any or all of the following be just as good a result? a. 1d matrix b. row and column vectors (1d but "oriented" for linear algebra purposes) 3. If 2a. and 2b. are no good for you, a. would e.g. ``x[i:i+1,:]`` be too burdensome? b. would ``rows`` and ``columns`` attributes that yielded matrics be an offsetting convenience? Thank you, Alan Isaac PS It is nice to have another matrix user in the conversation! From aisaac at american.edu Tue Apr 29 14:41:45 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 29 Apr 2008 14:41:45 -0400 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <20080429181143.GG24600@phare.normalesup.org> References: <20080426100938.GD24606@phare.normalesup.org><48161F61.5090204@noaa.gov><9457e7c80804290624n754e74a6w6a1bb0295ae4a79b@mail.gmail.com><4817439E.9020002@enthought.com> <20080429181143.GG24600@phare.normalesup.org> Message-ID: On Tue, 29 Apr 2008, Gael Varoquaux apparently wrote: > I really don't like a[i] != a[i,:]. Tim H's proposal avoids that problem. What do you think of it? > I also don't like loosing the information that you are > doing linear algebra. Hmmm. Is it a given that asking for ``a[i]`` is "doing linear algebra"? Is ``a[i:i+1,:]`` so bad if you want a matrix? I think some more use cases, like Bill's, might prove clarifying. Alan From kwgoodman at gmail.com Tue Apr 29 14:46:41 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 29 Apr 2008 11:46:41 -0700 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <48174FF5.50005@noaa.gov> Message-ID: On Tue, Apr 29, 2008 at 11:21 AM, Alan G Isaac wrote: > On Tue, 29 Apr 2008, Keith Goodman apparently wrote: > > I often use x[i,:] and x[:,i] where x is a matrix and i is > > a scalar. I hope this continues to return a matrix. > > 1. Could you give an example of the circumstances of this > use? In my use i is most commonly an array (i = M.where(y.A)[0] where y is a nx1 matrix), sometimes a list, and in ipython when debugging or first writing the code, a scalar. It would seem odd to me if x[i,:] returned different types of objects based on the type of i: array index idx = M.where(y.A)[0] where y is a nx1 matrix x[dx,:] --> matrix list index idx = [0] x[idx,:] --> matrix? scalar index idx = 0 x[idx,:] --> not matrix > 2. Would any or all of the following be just as good > a result? > a. 1d matrix > b. row and column vectors (1d but "oriented" for > linear algebra purposes) For me, having arrays and matrices is confusing enough. Adding row and column objects sounds even more confusing. > 3. If 2a. and 2b. are no good for you, > a. would e.g. ``x[i:i+1,:]`` be too burdensome? Yes. > b. would ``rows`` and ``columns`` attributes that > yielded matrics be an offsetting convenience? I like the idea, from earlier in the thread, of row and col being iterators for those time when you want to work on one column or row at a time. From charlesr.harris at gmail.com Tue Apr 29 14:46:55 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 29 Apr 2008 12:46:55 -0600 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <9457e7c80804290624n754e74a6w6a1bb0295ae4a79b@mail.gmail.com> <4817439E.9020002@enthought.com> <20080429181143.GG24600@phare.normalesup.org> Message-ID: On Tue, Apr 29, 2008 at 12:41 PM, Alan G Isaac wrote: > On Tue, 29 Apr 2008, Gael Varoquaux apparently wrote: > > I really don't like a[i] != a[i,:]. > > Tim H's proposal avoids that problem. > What do you think of it? > > Of course, now we will have to remove all that special casing code... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Tue Apr 29 14:55:28 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 29 Apr 2008 20:55:28 +0200 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: Message-ID: <20080429185528.GA5130@phare.normalesup.org> I will answer this question too, because I feel like Keith. On Tue, Apr 29, 2008 at 02:21:34PM -0400, Alan G Isaac wrote: > On Tue, 29 Apr 2008, Keith Goodman apparently wrote: > > I often use x[i,:] and x[:,i] where x is a matrix and i is > > a scalar. I hope this continues to return a matrix. > 1. Could you give an example of the circumstances of this > use? x[i, :]*A*x[:, i] I find this cleaner, because more explicit, than x[i]*A*x[:, i]. I'd be _very very_ upset if this broke. Breaking the explicit for the implicit is a _bad_ idea, and if you go down this path you go down the path of inconsistency. > 2. Would any or all of the following be just as good > a result? > a. 1d matrix Yes. > b. row and column vectors (1d but "oriented" for > linear algebra purposes) Yes. > 3. If 2a. and 2b. are no good for you, > a. would e.g. ``x[i:i+1,:]`` be too burdensome? No way. This is hard to read and the best way to make long calculations impossible to follow and debug. > b. would ``rows`` and ``columns`` attributes that > yielded matrics be an offsetting convenience? Hum, I don't love this because it introduces noise in the formulas, but at least it is explicit and readable. I would really like us to go down the way of the row/vector columns. This is consistent at least. Ga?l From gael.varoquaux at normalesup.org Tue Apr 29 15:12:32 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 29 Apr 2008 21:12:32 +0200 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080429181143.GG24600@phare.normalesup.org> Message-ID: <20080429191232.GC5130@phare.normalesup.org> On Tue, Apr 29, 2008 at 02:41:45PM -0400, Alan G Isaac wrote: > On Tue, 29 Apr 2008, Gael Varoquaux apparently wrote: > > I really don't like a[i] != a[i,:]. > Tim H's proposal avoids that problem. > What do you think of it? Sorry, I guess I am lost. Could you remind me which one it is? Given how well you are summing up the discussion on http://www.scipy.org/MatrixIndexing , I suspect it is already in the list of proposal. > > I also don't like loosing the information that you are > > doing linear algebra. > Hmmm. Is it a given that asking for ``a[i]`` is "doing > linear algebra"? Is ``a[i:i+1,:]`` so bad if you want > a matrix? Very, very bad. This is completly unreadable. Actually to the beginner, this looks like a 2*N matrix, not a 1*N matrix. To me you are really replacing a problem by another. We should fix the problem, not replace it. Ga?l From aisaac at american.edu Tue Apr 29 15:20:02 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 29 Apr 2008 15:20:02 -0400 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <20080429185528.GA5130@phare.normalesup.org> References: <20080429185528.GA5130@phare.normalesup.org> Message-ID: On Tue, 29 Apr 2008, Gael Varoquaux apparently wrote: > x[i, :]*A*x[:, i] > I find this cleaner, because more explicit, than x[i]*A*x[:, i]. >> a. would e.g. ``x[i:i+1,:]`` > No way. So *if* ``x[0]`` is going to be 1d to fix the many anomalies that have surfaced, then you would still want ``x[0]==x[0,:]``, but to have those be 1d and have ``x[i, :]*A*x[:, i]`` work would require (oriented) "vector" objects, so you would want those. Is that a fair summary of your view? Cheers, Alan Isaac From wright at esrf.fr Tue Apr 29 15:19:48 2008 From: wright at esrf.fr (Jon Wright) Date: Tue, 29 Apr 2008 21:19:48 +0200 Subject: [Numpy-discussion] Matrix: potential usage case? In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org><48161F61.5090204@noaa.gov><9457e7c80804290624n754e74a6w6a1bb0295ae4a79b@mail.gmail.com><4817439E.9020002@enthought.com> <20080429181143.GG24600@phare.normalesup.org> Message-ID: <481774D4.7030406@esrf.fr> Hi everyone, Despite being a bit lost in the matrix debate, today I was working on something which might want to use what is being described. You can see an array only version at: http://fable.svn.sourceforge.net/viewvc/fable/ImageD11/trunk/test/demo/latred.py?revision=2741&view=markup The problem is in diffraction from crystals. We measure "scattering vectors" which are in "reciprocal space" and use these to find crystal structures in "real space". These spaces are a covariant/contravariat pair*. The purpose of the code in the script is to construct a lattice class which can work with vectors that are directly measured, or come from an FFT of those vectors, which averages lots of peaks into fewer peaks. [nb, for a single lattice this is a solved problem in many software packages]. I used a keyword argument space="real" or space="reciprocal" to indicate which space a vector is in. It might be a good case to think about putting in RowVector and ColumnVector and trying to deal with the consequences. It is not in this code yet, but the lattice symmetry should show up soon. I have not understood how to distinguish a metric tensor (flips RowVector to ColumnVector) from a reciprocal metric tensor (flips back) from a simple rotation matrix (applies a symmetry operator, like 2-fold rotation, so doesn't flip at all). I fear the limitation is in my maths? Best, Jon * Some heretics have been known to scale reciprocal space by 2pi. See "Vectors and Tensors in Crystallography" by Donald Sands for an overview. From tim.hochberg at ieee.org Tue Apr 29 15:22:18 2008 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Tue, 29 Apr 2008 12:22:18 -0700 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <48174FF5.50005@noaa.gov> Message-ID: Let me throw out a couple of more thoughts: First, there seems to be disagreement about what a row_vector and column_vector are (and even if they are sensible concepts, but let's leave that aside for moment). One school of thought is that they are one-dimensional objects that have some orientation (hence row/column). They correspond, more or less, to covariant and contravariant tensors, although I can never recall which is which. The second view, which I suspect is influenced by MatLab and its ilk, is that they are 2-dimensional 1xN and Nx1 arrays. It's my view that the pseudo tensor approach is more powerful, but it does require some metainformation be added to the array. This metadata can either take the form of making the different objects different classes, which leads to the matrix/row/column formulation, or adding some sort of tag to the array object (proposal #5, which so far lacks any detail). Second, most of the stuff that we have been discussing so far is primarily about notational convenience. However, there is matrix related stuff that is at best poorly supported now, namely operations on stacks of arrays (or vectors). As a concrete example, I at times need to work with stacks of small matrices. If I do the operations one by one, the overhead is prohibitive, however, most of that overhead can be avoided. For example, I rewrote some of the linalg routines to work on stacks of matrices and inverse is seven times faster for a 100x10x10 array (a stack of 100 10x10 matrices) when operating on a stack than when operating on the matrices one at a time. This is a result of sharing the setup overhead, the C routines that called are the same in either case. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Tue Apr 29 15:27:21 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 29 Apr 2008 21:27:21 +0200 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080429185528.GA5130@phare.normalesup.org> Message-ID: <20080429192721.GE5130@phare.normalesup.org> On Tue, Apr 29, 2008 at 03:20:02PM -0400, Alan G Isaac wrote: > On Tue, 29 Apr 2008, Gael Varoquaux apparently wrote: > > x[i, :]*A*x[:, i] > > I find this cleaner, because more explicit, than x[i]*A*x[:, i]. > >> a. would e.g. ``x[i:i+1,:]`` > > No way. > So *if* ``x[0]`` is going to be 1d to fix the many anomalies that > have surfaced, then you would still want ``x[0]==x[0,:]``, > but to have those be 1d and have ``x[i, :]*A*x[:, i]`` work > would require (oriented) "vector" objects, so you would want > those. > Is that a fair summary of your view? That an extremely good summary of my views. Better than I could have formulated it. Thank you. Given this good short paragraph summing up the problem, as I see it, this is why I think the only correct option is the oriented 1D objects. The second best option is not to change anything in order not to break backward compatibility. Ga?l From aisaac at american.edu Tue Apr 29 15:32:17 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 29 Apr 2008 15:32:17 -0400 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <20080429191232.GC5130@phare.normalesup.org> References: <20080429181143.GG24600@phare.normalesup.org> <20080429191232.GC5130@phare.normalesup.org> Message-ID: On Tue, 29 Apr 2008, Gael Varoquaux apparently wrote: > Could you remind me which one it is? Proposal #6 at is my effort to summarize what Tim was proposing. It does not include the oriented 1d vectors that I think you need (if I have understood you). Alan PS For beginners, perhaps ``x[[i],:]`` would be better than ``x[i:i+1,:]``. It has other advantages too... From gael.varoquaux at normalesup.org Tue Apr 29 15:30:59 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 29 Apr 2008 21:30:59 +0200 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <48174FF5.50005@noaa.gov> Message-ID: <20080429193059.GF5130@phare.normalesup.org> On Tue, Apr 29, 2008 at 12:22:18PM -0700, Timothy Hochberg wrote: > First, there seems to be disagreement about what a row_vector and > column_vector are (and even if they are sensible concepts, but let's leave > that aside for moment). One school of thought is that they are > one-dimensional objects that have some orientation (hence row/column). > They correspond, more or less, to covariant and contravariant tensors, > although I can never recall which is which. The second view, which I > suspect is influenced by MatLab and its ilk, is that they are > 2-dimensional 1xN and Nx1 arrays. It's my view that the pseudo tensor > approach is more powerful, but it does require some metainformation be > added to the array. This metadata can either take the form of making the > different objects different classes, which leads to the matrix/row/column > formulation, or adding some sort of tag to the array object (proposal #5, > which so far lacks any detail). Good summary. I support the 1D object with orientation, rather than the 2D object with special indexing. I would call the row_vector/column_vectors bras and kets rather than tensors, but that because I come from a quantum mechanics background. > Second, most of the stuff that we have been discussing so far is primarily > about notational convenience. However, there is matrix related stuff that > is at best poorly supported now, namely operations on stacks of arrays (or > vectors). As a concrete example, I at times need to work with stacks of > small matrices. If I do the operations one by one, the overhead is > prohibitive, however, most of that overhead can be avoided. For example, I > rewrote some of the linalg routines to work on stacks of matrices and > inverse is seven times faster for a 100x10x10 array (a stack of 100 10x10 > matrices) when operating on a stack than when operating on the matrices > one at a time. This is a result of sharing the setup overhead, the C > routines that called are the same in either case. Good point. Do you have an idea to move away from this problem? Ga?l From gael.varoquaux at normalesup.org Tue Apr 29 15:34:07 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 29 Apr 2008 21:34:07 +0200 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080429191232.GC5130@phare.normalesup.org> Message-ID: <20080429193407.GG5130@phare.normalesup.org> On Tue, Apr 29, 2008 at 03:32:17PM -0400, Alan G Isaac wrote: > On Tue, 29 Apr 2008, Gael Varoquaux apparently wrote: > > Could you remind me which one it is? > Proposal #6 at > > is my effort to summarize what Tim was proposing. > It does not include the oriented 1d vectors that I think you > need (if I have understood you). I prefer proposal number 1. > PS For beginners, perhaps ``x[[i],:]`` would be better > than ``x[i:i+1,:]``. It has other advantages too... I think this will get beginners lost. From my experience, people take a long time to understand indexing with an iterable. It is actually a source of confusion, but I am not even remotely thinking of removing it, because it is soooo nice, when you know how to use it. Ga?l From kwgoodman at gmail.com Tue Apr 29 15:50:12 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 29 Apr 2008 12:50:12 -0700 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <48174FF5.50005@noaa.gov> Message-ID: On Tue, Apr 29, 2008 at 11:46 AM, Keith Goodman wrote: > On Tue, Apr 29, 2008 at 11:21 AM, Alan G Isaac wrote: > > On Tue, 29 Apr 2008, Keith Goodman apparently wrote: > > > I often use x[i,:] and x[:,i] where x is a matrix and i is > > > a scalar. I hope this continues to return a matrix. > > > > 1. Could you give an example of the circumstances of this > > use? > > In my use i is most commonly an array (i = M.where(y.A)[0] where y is > a nx1 matrix), sometimes a list, and in ipython when debugging or > first writing the code, a scalar. It would seem odd to me if x[i,:] > returned different types of objects based on the type of i: > > array index > idx = M.where(y.A)[0] where y is a nx1 matrix > x[dx,:] --> matrix > > list index > idx = [0] > x[idx,:] --> matrix? > > scalar index > idx = 0 > x[idx,:] --> not matrix Here's another use case: for i in xrange(x.shape[1]): xi = x[:,i] idx = M.where(M.isfinite(xi).A)[0] xi = xi[idx,:] # more code Also the functions I have written work on matrices. If x[i,:] did not return a matrix, let's say it returned some other vector type object that you index with a scalar, then my functions will have to test whether the input is a full matrix or a vector since the indexing is different. From bsouthey at gmail.com Tue Apr 29 16:14:41 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Tue, 29 Apr 2008 15:14:41 -0500 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <48174FF5.50005@noaa.gov> Message-ID: Hi, On Tue, Apr 29, 2008 at 2:22 PM, Timothy Hochberg wrote: > > Let me throw out a couple of more thoughts: > > First, there seems to be disagreement about what a row_vector and > column_vector are (and even if they are sensible concepts, but let's leave > that aside for moment). One school of thought is that they are > one-dimensional objects that have some orientation (hence row/column). They > correspond, more or less, to covariant and contravariant tensors, although I > can never recall which is which. The second view, which I suspect is > influenced by MatLab and its ilk, is that they are 2-dimensional 1xN and > Nx1 arrays. It's my view that the pseudo tensor approach is more powerful, > but it does require some metainformation be added to the array. This > metadata can either take the form of making the different objects different > classes, which leads to the matrix/row/column formulation, or adding some > sort of tag to the array object (proposal #5, which so far lacks any > detail). > Actually I think that this simply stems from the different variants of linear (and more specifically matrix) algebra. My background is very heavy in the statistical favor of this (what the heck are tensors... http://mathworld.wolfram.com/Tensor.html). It is also clear (from that definition) that others are from more physics and engineering backgrounds. Hence all the back and forth because people have slightly different usage, terminology and expectations. The ability to treat vectors as matrices would be sufficient for my needs because these are almost always used in the context of vector-matrix multiplication. There is no additional benefit from having row or column shapes or metadata because the row/column nature is usually predetermined and would be represented by the shape of the corresponding matrix. It really would be annoying to find that for an n by 1 vector/matrix, the product of X.T*X is a scalar or an error rather than an n by n matrix. Or requires additional code to get the desired effect. Regards Bruce From kwgoodman at gmail.com Tue Apr 29 16:15:21 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 29 Apr 2008 13:15:21 -0700 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <48174FF5.50005@noaa.gov> Message-ID: On Tue, Apr 29, 2008 at 12:50 PM, Keith Goodman wrote: > On Tue, Apr 29, 2008 at 11:46 AM, Keith Goodman wrote: > > On Tue, Apr 29, 2008 at 11:21 AM, Alan G Isaac wrote: > > > On Tue, 29 Apr 2008, Keith Goodman apparently wrote: > > > > I often use x[i,:] and x[:,i] where x is a matrix and i is > > > > a scalar. I hope this continues to return a matrix. > > > > > > 1. Could you give an example of the circumstances of this > > > use? > > > > In my use i is most commonly an array (i = M.where(y.A)[0] where y is > > a nx1 matrix), sometimes a list, and in ipython when debugging or > > first writing the code, a scalar. It would seem odd to me if x[i,:] > > returned different types of objects based on the type of i: > > > > array index > > idx = M.where(y.A)[0] where y is a nx1 matrix > > x[dx,:] --> matrix > > > > list index > > idx = [0] > > x[idx,:] --> matrix? > > > > scalar index > > idx = 0 > > x[idx,:] --> not matrix > > Here's another use case: > > for i in xrange(x.shape[1]): > xi = x[:,i] > idx = M.where(M.isfinite(xi).A)[0] > xi = xi[idx,:] > # more code > > Also the functions I have written work on matrices. If x[i,:] did not > return a matrix, let's say it returned some other vector type object > that you index with a scalar, then my functions will have to test > whether the input is a full matrix or a vector since the indexing is > different. And here's a use case that currently doesn't work but would be nice (for me) if it did: x[idx,i] = M.rand(2,1) where x is a matrix, say 4x3, idx is an array with 2 elements, and i is a scalar From andrea.gavana at gmail.com Tue Apr 29 16:48:25 2008 From: andrea.gavana at gmail.com (Andrea Gavana) Date: Tue, 29 Apr 2008 21:48:25 +0100 Subject: [Numpy-discussion] Linear Interpolation 2 Message-ID: Hi All, as I didn't get any answer, I thought I might repost with some more analysis the trouble I am having with interp2d. The attached script produces 3 different results depending on the value of the parameter numColumns, i.e.: 1) numColumns = 20 Traceback (most recent call last): File "C:\MyProjects\LinearInterpolation.py", line 27, in function = interp2d(xx, yy, z, kind="linear") File "C:\Python25\Lib\site-packages\scipy\interpolate\interpolate.py", line 113, in __init__ self.tck = fitpack.bisplrep(self.x, self.y, self.z, kx=kx, ky=ky, s=0.) File "C:\Python25\Lib\site-packages\scipy\interpolate\fitpack.py", line 702, in bisplrep tx,ty,nxest,nyest,wrk,lwrk1,lwrk2) ValueError: Invalid inputs. 2) numColumns = 200 Traceback (most recent call last): File "C:\MyProjects\LinearInterpolation.py", line 27, in function = interp2d(xx, yy, z, kind="linear") File "C:\Python25\Lib\site-packages\scipy\interpolate\interpolate.py", line 113, in __init__ self.tck = fitpack.bisplrep(self.x, self.y, self.z, kx=kx, ky=ky, s=0.) File "C:\Python25\Lib\site-packages\scipy\interpolate\fitpack.py", line 702, in bisplrep tx,ty,nxest,nyest,wrk,lwrk1,lwrk2) MemoryError (MemoryError? With a 372x200 matrix???) 3) numColumns = 2000 Traceback (most recent call last): File "C:\MyProjects\LinearInterpolation.py", line 27, in function = interp2d(xx, yy, z, kind="linear") File "C:\Python25\Lib\site-packages\scipy\interpolate\interpolate.py", line 113, in __init__ self.tck = fitpack.bisplrep(self.x, self.y, self.z, kx=kx, ky=ky, s=0.) File "C:\Python25\Lib\site-packages\scipy\interpolate\fitpack.py", line 702, in bisplrep tx,ty,nxest,nyest,wrk,lwrk1,lwrk2) OverflowError: long int too large to convert to int Is there anyone who could tell me, please, what I am doing wrong? This is on Windows XP 1GB RAM, 3.7 GHz, Python 2.5, scipy 0.6.0, numpy 1.0.4. Thank you for your suggestions. Andrea. "Imagination Is The Only Weapon In The War Against Reality." http://xoomer.alice.it/infinity77/ -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: LinearInterpolation.py URL: From aisaac at american.edu Tue Apr 29 16:55:53 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 29 Apr 2008 16:55:53 -0400 Subject: [Numpy-discussion] the path forward In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org><48161F61.5090204@noaa.gov><48174FF5.50005@noaa.gov> Message-ID: On Tue, 29 Apr 2008, Bruce Southey apparently wrote: > There is no additional benefit from having row or column > shapes or metadata because the row/column nature is > usually predetermined and would be represented by the > shape of the corresponding matrix. The problem is that ``x[0]`` being 2d has produced a variety of anomalies, and the natural fix is for ``x[0]`` to be 1d. Gael has argued strongly that she should be able to use the following notation: ``x[0,:]*A*x[:,0]``. But this will work only if ``x[0,:]`` is 2d or if it is 1d but has an "orientation". So *if* you think ``x[0] == x[0,:]`` is desirable, *and* you want to satisfy Gael, *then* it seems you must introduce 1d "oriented" vectors. I believe Travis is also suggesting that we travel that road, taking a first step as follows: for now let ``x[0]`` be a 1d array to quickly fix the anomalies, but let ``x[0,:]`` continue to be a matrix until the vector code is written, at which point ``x[0]`` and ``x[0,:]`` we be the same "row vector". Or so I have understood things. Cheers, Alan Isaac From aisaac at american.edu Tue Apr 29 16:56:03 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 29 Apr 2008 16:56:03 -0400 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <20080429192721.GE5130@phare.normalesup.org> References: <20080429185528.GA5130@phare.normalesup.org> <20080429192721.GE5130@phare.normalesup.org> Message-ID: On Tue, 29 Apr 2008, Gael Varoquaux apparently wrote: > I think the only correct option is the oriented 1D > objects. The second best option is not to change anything > in order not to break backward compatibility. Maybe the extant ``matrix`` class should be available for a while as ``oldmatrix``? Alan From aisaac at american.edu Tue Apr 29 16:56:05 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 29 Apr 2008 16:56:05 -0400 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org><48161F61.5090204@noaa.gov><48174FF5.50005@noaa.gov> Message-ID: On Tue, 29 Apr 2008, Keith Goodman apparently wrote: > here's a use case that currently doesn't work but would be nice > (for me) if it did: > x[idx,i] = M.rand(2,1) I think this is another example of the kind of thing we are trying to fix ... Cheers, Alan Isaac From peridot.faceted at gmail.com Tue Apr 29 17:03:58 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Tue, 29 Apr 2008 23:03:58 +0200 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <4817439E.9020002@enthought.com> References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <9457e7c80804290624n754e74a6w6a1bb0295ae4a79b@mail.gmail.com> <4817439E.9020002@enthought.com> Message-ID: On 29/04/2008, Travis E. Oliphant wrote: > I'm quite persuaded now that a[i] should return a 1-d object for > arrays. In addition to the places Chuck identified, there are at > least 2 other places where special code was written to work-around the > expectation that item selection returns an object with reduced > dimensionality (a special-cased .tolist for matrices and a special-cased > getitem in the fancy indexing code). > > As the number of special-case work-arounds grows the more I'm convinced > the conceptualization is wrong. So, I now believe we should change the > a[i] for matrices to return a 1-d array. > > The only down-side I see is that a[i] != a[i,:] for matrices. > However, matrix(a[i]) == a[i,:], and so I'm not sure there is really a > problem, there. I also don't think that whatever problem may actually > be buried in the fact that type(a[i]) != type(a[i,:]) is worse than the > problem that several pieces of NumPy code actually expect hierarchical > container behavior of multi-dimensional sequences. I am puzzled by this. What is the rationale for x[i,:] not being a 1-d object? Does this not require many special-case bits of code as well? What about a[i,...]? That is what I would use to make a hierarchical bit of code, and I would be startled to find myself in an infinite loop waiting for the dimension to become one. > I don't think making the small change to have a[i] return 1-d arrays > precludes us from that 1-d array being a bit more formalized in 1.2 as a > RowVector should we choose that direction. It does, however, fix > several real bugs right now. +1 What's more, I think we need to have a matrix-cleanliness test suite that verifies that all basic numpy tools behave identically on matrices and preserve matrixiness. But that's a big job, and for 1.2. Anne From gael.varoquaux at normalesup.org Tue Apr 29 17:07:29 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 29 Apr 2008 23:07:29 +0200 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <9457e7c80804290624n754e74a6w6a1bb0295ae4a79b@mail.gmail.com> <4817439E.9020002@enthought.com> Message-ID: <20080429210729.GD15538@phare.normalesup.org> On Tue, Apr 29, 2008 at 11:03:58PM +0200, Anne Archibald wrote: > I am puzzled by this. What is the rationale for x[i,:] not being a 1-d > object? It breaks A*B[i, :] where A and B are matrices. Ga?l From peridot.faceted at gmail.com Tue Apr 29 17:16:15 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Tue, 29 Apr 2008 23:16:15 +0200 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <20080429210729.GD15538@phare.normalesup.org> References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <9457e7c80804290624n754e74a6w6a1bb0295ae4a79b@mail.gmail.com> <4817439E.9020002@enthought.com> <20080429210729.GD15538@phare.normalesup.org> Message-ID: On 29/04/2008, Gael Varoquaux wrote: > On Tue, Apr 29, 2008 at 11:03:58PM +0200, Anne Archibald wrote: > > I am puzzled by this. What is the rationale for x[i,:] not being a 1-d > > object? > > It breaks A*B[i, :] where A and B are matrices. Really? How? In [26]: A = np.matrix([[1,0],[0,1],[1,1]]) In [28]: A*np.ones(2) Out[28]: matrix([[ 1., 1., 2.]]) In [29]: np.ones(3)*A Out[29]: matrix([[ 2., 2.]]) I guess you don't get dot products with *: In [30]: np.ones(3).T*np.ones(3) Out[30]: array([ 1., 1., 1.]) Since the whole point of matrices it to avoid having to type "dot" I guess this should be convincing. Anne From tim.hochberg at ieee.org Tue Apr 29 17:18:46 2008 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Tue, 29 Apr 2008 14:18:46 -0700 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <20080429210729.GD15538@phare.normalesup.org> References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <9457e7c80804290624n754e74a6w6a1bb0295ae4a79b@mail.gmail.com> <4817439E.9020002@enthought.com> <20080429210729.GD15538@phare.normalesup.org> Message-ID: On Tue, Apr 29, 2008 at 2:07 PM, Gael Varoquaux < gael.varoquaux at normalesup.org> wrote: > On Tue, Apr 29, 2008 at 11:03:58PM +0200, Anne Archibald wrote: > > I am puzzled by this. What is the rationale for x[i,:] not being a 1-d > > object? > > It breaks A*B[i, :] where A and B are matrices. Shouldn't that be B[i,:] * A? Or am I just confused? In any event this wouldn't be a problem under row/column scheme or any of the other results that end up tagging the results with some metainformation. B[i] would be tagged as a row vector somehow and __mul__ would "do the right thing" even though it was 1D. -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Tue Apr 29 17:18:23 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Tue, 29 Apr 2008 23:18:23 +0200 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <48174FF5.50005@noaa.gov> Message-ID: On 29/04/2008, Keith Goodman wrote: > On Tue, Apr 29, 2008 at 11:21 AM, Alan G Isaac wrote: > > On Tue, 29 Apr 2008, Keith Goodman apparently wrote: > > > I often use x[i,:] and x[:,i] where x is a matrix and i is > > > a scalar. I hope this continues to return a matrix. > > > > 1. Could you give an example of the circumstances of this > > use? > > > In my use i is most commonly an array (i = M.where(y.A)[0] where y is > a nx1 matrix), sometimes a list, and in ipython when debugging or > first writing the code, a scalar. It would seem odd to me if x[i,:] > returned different types of objects based on the type of i: > > array index > idx = M.where(y.A)[0] where y is a nx1 matrix > x[dx,:] --> matrix > > list index > idx = [0] > x[idx,:] --> matrix? > > scalar index > idx = 0 > x[idx,:] --> not matrix It is actually pretty unreasonable to hope that A[0] and A[[1,2,3]] or A[[True,False,True]] should return objects of the same rank. Anne From charlesr.harris at gmail.com Tue Apr 29 17:26:09 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 29 Apr 2008 15:26:09 -0600 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <20080429210729.GD15538@phare.normalesup.org> References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <9457e7c80804290624n754e74a6w6a1bb0295ae4a79b@mail.gmail.com> <4817439E.9020002@enthought.com> <20080429210729.GD15538@phare.normalesup.org> Message-ID: On Tue, Apr 29, 2008 at 3:07 PM, Gael Varoquaux < gael.varoquaux at normalesup.org> wrote: > On Tue, Apr 29, 2008 at 11:03:58PM +0200, Anne Archibald wrote: > > I am puzzled by this. What is the rationale for x[i,:] not being a 1-d > > object? > > It breaks A*B[i, :] where A and B are matrices. A*B[:,i], surely. So it's a notational problem, what is the easiest way to get row/col vectors out of a matrix. And it will be helpful if the notation is consistent and easy to understand. In any case, dot doesn't care, so the fault above comes from the matrix class itself. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Tue Apr 29 17:33:13 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 29 Apr 2008 16:33:13 -0500 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <20080429181143.GG24600@phare.normalesup.org> References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <9457e7c80804290624n754e74a6w6a1bb0295ae4a79b@mail.gmail.com> <4817439E.9020002@enthought.com> <20080429181143.GG24600@phare.normalesup.org> Message-ID: <48179419.2030608@enthought.com> Gael Varoquaux wrote: > On Tue, Apr 29, 2008 at 10:49:50AM -0500, Travis E. Oliphant wrote: > >> As the number of special-case work-arounds grows the more I'm convinced >> the conceptualization is wrong. So, I now believe we should change the >> a[i] for matrices to return a 1-d array. >> > > >> The only down-side I see is that a[i] != a[i,:] for matrices. >> However, matrix(a[i]) == a[i,:], and so I'm not sure there is really a >> problem, there. >> > > I think we have simply replaced one problem by another one. I think this > is a bad route to go. The problem is that I know exactly what the problem we are replacing, but I have no idea only vagueries about the problem we are getting. So, in the short term, it seems like the right thing to do, because we aren't preventing a change to RowVector/ColumnVector in the future. -Travis From Chris.Barker at noaa.gov Tue Apr 29 17:40:53 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 29 Apr 2008 14:40:53 -0700 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <9457e7c80804290624n754e74a6w6a1bb0295ae4a79b@mail.gmail.com> <4817439E.9020002@enthought.com> <20080429210729.GD15538@phare.normalesup.org> Message-ID: <481795E5.3020802@noaa.gov> > It breaks A*B[i, :] where A and B are matrices. > > A*B[:,i], surely. right. which is a really good argument for: A * B.col[i] (and .row, or course) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From gael.varoquaux at normalesup.org Tue Apr 29 17:42:24 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 29 Apr 2008 23:42:24 +0200 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <9457e7c80804290624n754e74a6w6a1bb0295ae4a79b@mail.gmail.com> <4817439E.9020002@enthought.com> <20080429210729.GD15538@phare.normalesup.org> Message-ID: <20080429214224.GJ15538@phare.normalesup.org> On Tue, Apr 29, 2008 at 11:16:15PM +0200, Anne Archibald wrote: > On 29/04/2008, Gael Varoquaux wrote: > > On Tue, Apr 29, 2008 at 11:03:58PM +0200, Anne Archibald wrote: > > > I am puzzled by this. What is the rationale for x[i,:] not being a 1-d > > > object? > > It breaks A*B[i, :] where A and B are matrices. > Really? How? > In [26]: A = np.matrix([[1,0],[0,1],[1,1]]) > In [28]: A*np.ones(2) > Out[28]: matrix([[ 1., 1., 2.]]) > In [29]: np.ones(3)*A > Out[29]: matrix([[ 2., 2.]]) Yes, sorry, I am shameful. I should have thought a bit more before posting. There is no big problem with x[i,:] not being a 1-d object. The problem is for x[:, i]. However I would find it nice that, for linear algebra, x[i, :] == x[:, i].T This is the kind of behavior I expect, and we won't be getting it with 1D arrays. Cheers, Ga?l From Chris.Barker at noaa.gov Tue Apr 29 17:43:42 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 29 Apr 2008 14:43:42 -0700 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <48174FF5.50005@noaa.gov> Message-ID: <4817968E.5010402@noaa.gov> Timothy Hochberg wrote: > However, there is matrix related > stuff that is at best poorly supported now, namely operations on stacks > of arrays (or vectors). Tim, this is important, but also appears to be an orthogonal issue to me -- whatever we do with matrices, rows, columns, whatever, we still need to solve this problem, and any of the schemes being proposed strikes me as amenable to having a "array or matrices", and the LA functions that act on them. What am I missing? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Glen.Mabey at swri.org Tue Apr 29 17:43:09 2008 From: Glen.Mabey at swri.org (Glen W. Mabey) Date: Tue, 29 Apr 2008 16:43:09 -0500 Subject: [Numpy-discussion] failure building numpy using icc In-Reply-To: References: <20080228192138.GA21482@swri.org> <3d375d730803010017r1505e28fwd68bc554060b5ba3@mail.gmail.com> <20080429152151.GI23710@swri.org> <20080429155032.GA4718@swri.org> <20080429171911.GB4718@swri.org> Message-ID: <20080429214309.GA11807@swri.org> On Tue, Apr 29, 2008 at 01:01:55PM -0500, Charles R Harris wrote: > Not really, it was just a shot in the dark. The thing is, if conv_python is being run on the same source each time, it should give the same result. So I suspect that something is changing the sources and that is what I'm trying to track down. One of the build guys should know... > > Hmmm..., I notice that there is a line break in line 18 of scalartypes.inc.src that doesn't belong there. Could you fix that? Okay, got that one. > And the line number of the loop is 497, which is the loop before the one with PREFIX defined. For some reason the parser is missing the /**end repeat**/ on line 509. Hmmm, try putting a blank line before it. Putting a blank line before it didn't do the trick. I have been looking at the python code that should have found the /**end repeat**/ mark and I can't understand why it didn't find it. In fact, watch this great trick in pdb: ipdb> text[265:281] '/**end repeat**/' ipdb> text.find( text[265:281] ) -1 Isn't that cool? I can only assume that it is a compiler bug and I will have to upgrade to a newer version of icc (I'm using 10.0.025, actually it's cce). After I do that, I'll post again if I have trouble. Thanks for your help. Glen From oliphant at enthought.com Tue Apr 29 17:43:48 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 29 Apr 2008 16:43:48 -0500 Subject: [Numpy-discussion] the path forward In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org><48161F61.5090204@noaa.gov><48174FF5.50005@noaa.gov> Message-ID: <48179694.9010304@enthought.com> > The problem is that ``x[0]`` being 2d has produced a variety > of anomalies, and the natural fix is for ``x[0]`` to be 1d. > > Gael has argued strongly that she should be able to use the > following notation: ``x[0,:]*A*x[:,0]``. But this will work > only if ``x[0,:]`` is 2d or if it is 1d but has an "orientation". > > So *if* you think ``x[0] == x[0,:]`` is desirable, *and* you > want to satisfy Gael, *then* it seems you must introduce 1d > "oriented" vectors. > > I believe Travis is also suggesting that we travel that > road, taking a first step as follows: > for now let ``x[0]`` be a 1d array to quickly fix the > anomalies, but let ``x[0,:]`` continue to be a matrix > until the vector code is written, at which point ``x[0]`` > and ``x[0,:]`` we be the same "row vector". > > Or so I have understood things. > You've characterized my current thinking pretty well. I'm less concerned that x[0] != x[0,:] than I think Gael is. -Travis From oliphant at enthought.com Tue Apr 29 17:48:06 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 29 Apr 2008 16:48:06 -0500 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <9457e7c80804290624n754e74a6w6a1bb0295ae4a79b@mail.gmail.com> <4817439E.9020002@enthought.com> Message-ID: <48179796.8080404@enthought.com> > I am puzzled by this. What is the rationale for x[i,:] not being a 1-d > object? Does this not require many special-case bits of code as well? > What about a[i,...]? That is what I would use to make a hierarchical > bit of code, and I would be startled to find myself in an infinite > loop waiting for the dimension to become one. > The rationale is so you can write x[i,:] * A * x[:,i] and have it work correctly without the RowVector / ColumnVector objects. Maybe it's just easier to create them and be done with it. But, nobody has yet, so I'm not sure. -Travis From Chris.Barker at noaa.gov Tue Apr 29 17:49:22 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 29 Apr 2008 14:49:22 -0700 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <48174FF5.50005@noaa.gov> Message-ID: <481797E2.7080504@noaa.gov> Keith Goodman wrote: > It would seem odd to me if x[i,:] > returned different types of objects based on the type of i: I think this is where matrices should act like arrays: >>> a array([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) >>> a[:,[1]] array([[1], [4], [7]]) >>> a[:,1] array([1, 4, 7]) in this case, it's not really different types, but totally different concepts -- indexing with a sequence is different than a scalar, just like indexing with a slice is different than a scalar. There are many, many places in python code where we have to make distinctions between a single object and a sequence of objects that happens to be of length one. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Tue Apr 29 17:55:38 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 29 Apr 2008 14:55:38 -0700 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <48174FF5.50005@noaa.gov> Message-ID: <4817995A.2090207@noaa.gov> Bruce Southey wrote: > The ability to treat vectors as matrices would be sufficient for my > needs because these are almost always used in the context of > vector-matrix multiplication. There is no additional benefit from > having row or column shapes or metadata because the row/column nature > is usually predetermined and would be represented by the shape of the > corresponding matrix. The benefit is that they can be indexed with a scalar and converted to a 1-d array with r.A, and no reshaping. Also that indexing a matrix reduces its rank, which is expected in a lot of places. > It really would be annoying to find that for an > n by 1 vector/matrix, the product of X.T*X is a scalar or an error > rather than an n by n matrix. yes, it would, which is the whole point of the matrix object in the first place, and the point of the proposed row/column objects. They would provide 1-d object that behave as row an column vectors with * and **. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From bressert at gmail.com Tue Apr 29 18:00:21 2008 From: bressert at gmail.com (eli bressert) Date: Tue, 29 Apr 2008 18:00:21 -0400 Subject: [Numpy-discussion] accuracy issues with numpy arrays? Message-ID: Hi, I'm writing a quick script to import a fits (astronomy) image that has very low values for each pixel. Mostly on the order of 10^-9. I have written a python script that attempts to take low values and put them in integer format. I basically do this by taking the mean of the 1000 lowest pixel values, excluding zeros, and dividing the rest of the image by that mean. Unfortunately, when I try this in practice, *all* of the values in the image are being treated as zeros. But, if I use a scipy.ndimage function, I get proper values. For example, I take the pixel that I know has the highest value and do x = scipy.ndimage.maximum(image) print x 1.7400700016878545e-05 The script is below. Thanks for the help. Eli import pyfits as p import scipy as s import scipy.ndimage as nd import numpy as n def flux2int(name): d = p.getdata(name) x,y = n.shape(d) l = x*y arr1 = n.array(d.reshape(x*y,1)) temp = n.unique(arr1[0]) # This is where the bug starts. All values are treated as zeros. Hence only one value remains, zero. arr1 = arr1.sort() arr1 = n.array(arr1) arr1 = n.array(arr1[s.where(arr1 >= temp)]) val = n.mean(arr1[0:1000]) d = d*(1.0/val) d = d.round() p.writeto(name[0,]+'fixed.fits',d,h) From Chris.Barker at noaa.gov Tue Apr 29 18:03:51 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 29 Apr 2008 15:03:51 -0700 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <20080429214224.GJ15538@phare.normalesup.org> References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <9457e7c80804290624n754e74a6w6a1bb0295ae4a79b@mail.gmail.com> <4817439E.9020002@enthought.com> <20080429210729.GD15538@phare.normalesup.org> <20080429214224.GJ15538@phare.normalesup.org> Message-ID: <48179B47.4010007@noaa.gov> Gael Varoquaux wrote: > However I would find it nice that, for linear algebra, > x[i, :] == x[:, i].T > > This is the kind of behavior I expect, and we won't be getting it with 1D > arrays. But you WOULD get it with 1-d row/column objects. I'm going to try to summarize what I think is clear from the discussion: 1) There are significant problems with the fact that the current matrix object does not reduce rank with indexing. 2) people want a "row" or "column" extracted from a matrix to behave like a linear algebra object, for example: M[:,i] * M[i,:] is an matrix As far as I can see, the ONLY way to satisfy both of these is with a row/column object of some sort. You can get almost there by having M[i] return a 1-d array, while M[:,i] and M[i,:] continue to return Matrices, but this gives an asymmetry between extracting rows and columns, and, probably more important, breaks: M[i] == M[i,:] plus you don't get the benefit of being able to extract a row or column and index the result with a scalar. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From tim.hochberg at ieee.org Tue Apr 29 18:07:19 2008 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Tue, 29 Apr 2008 15:07:19 -0700 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <4817968E.5010402@noaa.gov> References: <48161F61.5090204@noaa.gov> <48174FF5.50005@noaa.gov> <4817968E.5010402@noaa.gov> Message-ID: On Tue, Apr 29, 2008 at 2:43 PM, Christopher Barker wrote: > Timothy Hochberg wrote: > > However, there is matrix related > > stuff that is at best poorly supported now, namely operations on stacks > > of arrays (or vectors). > > Tim, this is important, but also appears to be an orthogonal issue to me > -- whatever we do with matrices, rows, columns, whatever, we still need > to solve this problem, and any of the schemes being proposed strikes me > as amenable to having a "array or matrices", and the LA functions that > act on them. What am I missing? Ah, that's a very good question. I'll do my best to answer it, it's been a long time since I dug into this seriously, so I may make some missteps: holler if you see something odd. Basically, I believe that you need two things: 1. The ability to make matrix/row/column objects of arbitrary dimensionality, where the extra dimensions contain the matrices/rows/columns. Current matrix objects are always 2D. 2. A way to tell whether a given object represents a row, column or matrix. For instance, given a 2D object is it matrix or a stack of vectors. You might think that there's enough information implicit in the shape of the arrays in conjunction with the nature of the function called that this wouldn't be necessary, but it turns out not to be so. As I recall, I had to fudge things when I was trying to containerize linalg.solve, because there wasn't enough information available to figure out unambiguously the kinds of the arguments. Any of the solutions that attach information to the array, such as your matrix/row/col approach, or the quasi-tensor approach that I (and someone else) have mentioned but never detailed, should work for this. However, any of the solutions that involve trying to encode the information into the array/matrix shape will probably not work. At least not well. Fortunately, they seem to be losing traction. -- . __ . |-\ . . tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Tue Apr 29 18:10:26 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Wed, 30 Apr 2008 00:10:26 +0200 Subject: [Numpy-discussion] accuracy issues with numpy arrays? In-Reply-To: References: Message-ID: On 30/04/2008, eli bressert wrote: > I'm writing a quick script to import a fits (astronomy) image that has > very low values for each pixel. Mostly on the order of 10^-9. I have > written a python script that attempts to take low values and put them > in integer format. I basically do this by taking the mean of the 1000 > lowest pixel values, excluding zeros, and dividing the rest of the > image by that mean. Unfortunately, when I try this in practice, *all* > of the values in the image are being treated as zeros. But, if I use a > scipy.ndimage function, I get proper values. For example, I take the > pixel that I know has the highest value and do I think the bug is something else: > import pyfits as p > import scipy as s > import scipy.ndimage as nd > import numpy as n > > def flux2int(name): > d = p.getdata(name) > x,y = n.shape(d) > l = x*y > arr1 = n.array(d.reshape(x*y,1)) > temp = n.unique(arr1[0]) # This is where the bug starts. All values > are treated as zeros. Hence only one value remains, zero. Actually, since arr1 has shape (x*y, 1), arr1[0] has shape (1,), and so it has only one entry: In [82]: A = np.eye(3) In [83]: A.reshape(9,1) Out[83]: array([[ 1.], [ 0.], [ 0.], [ 0.], [ 1.], [ 0.], [ 0.], [ 0.], [ 1.]]) In [84]: A.reshape(9,1)[0] Out[84]: array([ 1.]) The python debugger is a good way to check this sort of thing out; if you're using ipython, typing %pdb will start the debugger when an exception is raised, at which point you can poke around in all your local variables and evaluate expressions. > arr1 = arr1.sort() > arr1 = n.array(arr1) > arr1 = n.array(arr1[s.where(arr1 >= temp)]) > > val = n.mean(arr1[0:1000]) > > d = d*(1.0/val) > d = d.round() > p.writeto(name[0,]+'fixed.fits',d,h) Good luck, Anne From gael.varoquaux at normalesup.org Tue Apr 29 18:10:49 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 30 Apr 2008 00:10:49 +0200 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <48179B47.4010007@noaa.gov> References: <9457e7c80804290624n754e74a6w6a1bb0295ae4a79b@mail.gmail.com> <4817439E.9020002@enthought.com> <20080429210729.GD15538@phare.normalesup.org> <20080429214224.GJ15538@phare.normalesup.org> <48179B47.4010007@noaa.gov> Message-ID: <20080429221049.GM15538@phare.normalesup.org> On Tue, Apr 29, 2008 at 03:03:51PM -0700, Christopher Barker wrote: > Gael Varoquaux wrote: > > However I would find it nice that, for linear algebra, > > x[i, :] == x[:, i].T > > This is the kind of behavior I expect, and we won't be getting it with 1D > > arrays. > But you WOULD get it with 1-d row/column objects. Yes. I agree 100% with you that this is the best solution. I fully agree with your complete message. Ga?l From kwgoodman at gmail.com Tue Apr 29 18:23:58 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 29 Apr 2008 15:23:58 -0700 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <48174FF5.50005@noaa.gov> Message-ID: On Tue, Apr 29, 2008 at 2:18 PM, Anne Archibald wrote: > On 29/04/2008, Keith Goodman wrote: > > In my use i is most commonly an array (i = M.where(y.A)[0] where y is > > a nx1 matrix), sometimes a list, and in ipython when debugging or > > first writing the code, a scalar. It would seem odd to me if x[i,:] > > returned different types of objects based on the type of i: > > > > array index > > idx = M.where(y.A)[0] where y is a nx1 matrix > > x[dx,:] --> matrix > > > > list index > > idx = [0] > > x[idx,:] --> matrix? > > > > scalar index > > idx = 0 > > x[idx,:] --> not matrix > > It is actually pretty unreasonable to hope that > > A[0] > > and > > A[[1,2,3]] > or > A[[True,False,True]] > > should return objects of the same rank. Why it unreasonable to hope that x[0,:] and x[0, [1,2,3]] or x[0, [True,False,True]] where x is a matrix, continue to return matrices? From aisaac at american.edu Tue Apr 29 18:36:33 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 29 Apr 2008 18:36:33 -0400 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <48179796.8080404@enthought.com> References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <9457e7c80804290624n754e74a6w6a1bb0295ae4a79b@mail.gmail.com> <4817439E.9020002@enthought.com> <48179796.8080404@enthought.com> Message-ID: On Tue, 29 Apr 2008, "Travis E. Oliphant" apparently wrote: > x[i,:] * A * x[:,i] This need is also fully met by Proposal 5. It is just a matter of syntax and sticking with matrices. E.g., x.rows(i) * A * x.cols(i) Cheers, Alan Isaac From aisaac at american.edu Tue Apr 29 18:36:38 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 29 Apr 2008 18:36:38 -0400 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: <481795E5.3020802@noaa.gov> References: <20080426100938.GD24606@phare.normalesup.org><48161F61.5090204@noaa.gov><9457e7c80804290624n754e74a6w6a1bb0295ae4a79b@mail.gmail.com><4817439E.9020002@enthought.com><20080429210729.GD15538@phare.normalesup.org> <481795E5.3020802@noaa.gov> Message-ID: On Tue, 29 Apr 2008, Christopher Barker apparently wrote: > a really good argument for: > A * B.col[i] Also see the syntax discussed in Proposal 5. (I am not expressing an opinion.) One possibility is to let the rows and cols methods take an argument with a default of None. The default is to return an iterator over the rows/columns. An iterator over subsets of rows/columns could be specified with a sequence. A single row/column could be specified with a scalar argument. Cheers, Alan From aisaac at american.edu Tue Apr 29 18:44:11 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 29 Apr 2008 18:44:11 -0400 Subject: [Numpy-discussion] the path forward In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org><48161F61.5090204@noaa.gov><48174FF5.50005@noaa.gov> Message-ID: On Tue, 29 Apr 2008, Alan G Isaac apparently wrote: > Gael has argued strongly that he should be able to use the > following notation: ``x[0,:]*A*x[:,0]``. By the way Gael, is x.rows(0) * A * x.cols(0) a good replacement in your view? I find it nicely explicit and, to meet another of your concerns, usable by newbies. Cheers, Alan From peridot.faceted at gmail.com Tue Apr 29 18:41:35 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Wed, 30 Apr 2008 00:41:35 +0200 Subject: [Numpy-discussion] dot/tensordot limitations Message-ID: Timothy Hochberg has proposed a generalization of the matrix mechanism to support manipulating arrays of linear algebra objects. For example, one might have an array of matrices one wants to apply to an array of vectors, to yield an array of vectors: In [88]: A = np.repeat(np.eye(3)[np.newaxis,...],2,axis=0) In [89]: A Out[89]: array([[[ 1., 0., 0.], [ 0., 1., 0.], [ 0., 0., 1.]], [[ 1., 0., 0.], [ 0., 1., 0.], [ 0., 0., 1.]]]) In [90]: V = np.array([[1,0,0],[0,1,0]]) Currently, it is very clumsy to handle this kind of situation even with arrays, keeping track of dimensions by hand. For example if one wants to multiply A by V "elementwise", one cannot simply use dot: In [92]: np.dot(A,V.T) Out[92]: array([[[ 1., 0.], [ 0., 1.], [ 0., 0.]], [[ 1., 0.], [ 0., 1.], [ 0., 0.]]]) tensordot() suffers from the same problem. It arises because when you combine two multiindexed objects there are a number of ways you can do it: A: Treat all indices as different and form all pairwise products (outer product): In [93]: np.multiply.outer(A,V).shape Out[93]: (2, 3, 3, 2, 3) B: Contract the outer product on a pair of indices: In [98]: np.tensordot(A,V,axes=(-1,-1)).shape Out[98]: (2, 3, 2) C: Take the diagonal along a pair of indices: In [111]: np.diagonal(np.multiply.outer(A,V),axis1=0,axis2=3).shape Out[111]: (3, 3, 3, 2) What we want in this case is a combination of B and C: In [106]: np.diagonal(np.tensordot(A,V,axes=(-1,-1)),axis1=0,axis2=2).T.shape Out[106]: (2, 3) but it cannot be done without constructing a larger array and pulling out its diagonal. If this seems like an exotic thing to want to do, suppose instead we have two arrays of vectors, and we want to evaluate the array of dot products. None of no.dot, np.tensordot, or np.inner produce the results you want. You have to multiply elementwise and sum. (This also precludes automatic use of BLAS, if indeed tensordot uses BLAS.) Does it make sense to implement a generalized tensordot that can do this, or is it the sort of thing one should code by hand every time? Is there any way we can make it easier to do simple common operations like take the elementwise matrix-vector product of A and V? The more general issue, of making linear algebra natural by keeping track of which indices are for elementwise operations, and which should be used for dots (and how), is a whole other kettle of fish. I think for that someone should think hard about writing a full-featured multilinear algebra package (might as well keep track of coordinate transformations too while one was at it) if this is intended. Anne From peridot.faceted at gmail.com Tue Apr 29 18:52:27 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Wed, 30 Apr 2008 00:52:27 +0200 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <48161F61.5090204@noaa.gov> <48174FF5.50005@noaa.gov> Message-ID: On 30/04/2008, Keith Goodman wrote: > On Tue, Apr 29, 2008 at 2:18 PM, Anne Archibald > wrote: > > It is actually pretty unreasonable to hope that > > > > A[0] > > > > and > > > > A[[1,2,3]] > > or > > A[[True,False,True]] > > > > should return objects of the same rank. > > Why it unreasonable to hope that > > x[0,:] > > and > > x[0, [1,2,3]] > > or > > x[0, [True,False,True]] > > where x is a matrix, continue to return matrices? Well, I will point out that your example is somewhat different from mine; nobody is arguing that your three examples should return objects of different ranks. There is some disagreement over whether x[0,:] should be a rank-1 or a rank-2 object, but x[0,[1,2,3]] and x[0, [True, False, True]] should all have the same rank as x[0,:]. Nobody's questioning that. What I was pointing out is that x[0,0] should not have the same rank as x[0,:] or x[0,[0]]. In this case it's obvious; x[0,0] should be a scalar. But the same logic applies to x[0,:] versus x[[0,1],:] or even x[[0],:]. For arrays, if x has rank 2, x[0,:] has rank 1. if L is a list of indices, x[L,:] always has rank 2, no matter what is actually in L (even if it's one element). It is perfectly reasonable that they should yield objects of different ranks. With matrices, we have the surprising result that x[0,:] is still a two-dimensional object; to get at its third element I have to specify x[0,:][0,2]. It seems to me that this is a peculiar hack designed to ensure that the result still has "*" defined in a matrix way. Anne From gael.varoquaux at normalesup.org Tue Apr 29 18:56:17 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 30 Apr 2008 00:56:17 +0200 Subject: [Numpy-discussion] the path forward In-Reply-To: References: Message-ID: <20080429225617.GN15538@phare.normalesup.org> On Tue, Apr 29, 2008 at 06:44:11PM -0400, Alan G Isaac wrote: > On Tue, 29 Apr 2008, Alan G Isaac apparently wrote: > > Gael has argued strongly that he should be able to use the > > following notation: ``x[0,:]*A*x[:,0]``. > By the way Gael, is > x.rows(0) * A * x.cols(0) > a good replacement in your view? > I find it nicely explicit and, to meet > another of your concerns, usable by newbies. I don't really this proposal. I find this harder to read and make the matrices different than the arrays for no good reason. We have a good solution (the row/column), why not use it? Hum, I guess the answer is "talk is cheap, show me the code". So as I have hands tied with other work, I should rather shut up :) and let people who are actually doing the work decide. Ga?l From dalcinl at gmail.com Tue Apr 29 19:16:25 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 29 Apr 2008 20:16:25 -0300 Subject: [Numpy-discussion] Starting to work on runtime plugin system for plugin (automatic sse optimization, etc...) In-Reply-To: <4815E2E9.5020506@ar.media.kyoto-u.ac.jp> References: <4815E2E9.5020506@ar.media.kyoto-u.ac.jp> Message-ID: David, I briefly took a look at your code, and I have a very, very important observation. Your implementation make uses of low level dlopening. Then, your are going to have to manage all the oddities of runtime loading in the different systems. In this case, 'libtool' could really help. I know, it is GPL, but AFAIK it has some special licencing that let's you ship it with your code in the same licence terms than your code. But, I definitely think that a betther approach would be using a stubs mechanism ala TCL, or wath is currently used in some extension modules in core python, like cStringIO. In short, you access all your functions from a pointer do a struct (statically or heap allocated) where each struct member is filled with a pointer to a function. This is pretty much similar to C++ virtual tables, or the Cython cdef's classes with cdef's methods. And then you just let Python do the work of dynamic loading of extension modules. The numpy C/API uses a similar approach, but uses an array. IMHO, the struct approach is cleaner. What do you think about this? On 4/28/08, David Cournapeau wrote: > Hi, > > I've just started working on a prototype for a plugin system for > numpy. The plugin aims at providing a framework for the following user > cases: > - runtime selection of blas/lapack/etc...: instead of harcoding in > the binary one blas/lapack implementation, numpy could choose the SSE > optimized if the CPU supports SSE, etc... > - this could also be used for core numpy, for example ufuncs: if we > want to start implementing some tight loop with aggressively optimized > code (SSE, etc...), we could again ship with a default pure C > implementation, and choose the best one at runtime. > - we could even have a system to choose a different implementation > (for example, right now, scipy is shipped with a slow fft for licensing > issues mainly, and people installing fftw could then tell scipy to use > fftw instead of the included one). > > Right now, the prototype does not do much, and only works for linux; I > mainly focused on automatic generation of the plugin from a list of > functions, and transparent use from numpy point of view. It provides the > plugin api through pure function pointers, without the need for the user > to be aware of it. For example, if you have an api with the following > functions: > > void foo1(); > int foo2(); > int foo3(int); > int foo4(double* , double*); > int foo5(double* , double*, int); > > The current implementation would build the boilerplate to load those > functions, etc... and you would just use those functions in numpy like > the following: > > init_foo(); > > /* all functions are prefixed with npyw, for numpy wrapper */ > npyw_foo1(); > npyw_foo2(n); > etc... > > The code can be found there: > > https://code.launchpad.net/~david-ar/+junk/numplug > > And some thinking (pretty low content for now): > > http://www.scipy.org/RuntimeOptimization > > cheers, > > David > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From kwgoodman at gmail.com Tue Apr 29 20:01:11 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 29 Apr 2008 17:01:11 -0700 Subject: [Numpy-discussion] the path forward In-Reply-To: <48179694.9010304@enthought.com> References: <48174FF5.50005@noaa.gov> <48179694.9010304@enthought.com> Message-ID: On Tue, Apr 29, 2008 at 2:43 PM, Travis E. Oliphant wrote: > > > The problem is that ``x[0]`` being 2d has produced a variety > > of anomalies, and the natural fix is for ``x[0]`` to be 1d. > > > > Gael has argued strongly that she should be able to use the > > following notation: ``x[0,:]*A*x[:,0]``. But this will work > > only if ``x[0,:]`` is 2d or if it is 1d but has an "orientation". > > > > So *if* you think ``x[0] == x[0,:]`` is desirable, *and* you > > want to satisfy Gael, *then* it seems you must introduce 1d > > "oriented" vectors. > > > > I believe Travis is also suggesting that we travel that > > road, taking a first step as follows: > > for now let ``x[0]`` be a 1d array to quickly fix the > > anomalies, but let ``x[0,:]`` continue to be a matrix > > until the vector code is written, at which point ``x[0]`` > > and ``x[0,:]`` we be the same "row vector". > > > > Or so I have understood things. > > > You've characterized my current thinking pretty well. I'm less > concerned that x[0] != x[0,:] than I think Gael is. I hope that changing x[0,:] is considered a major change since it will break most any package based on matrices (mine). And so I hope that such a change wouldn't show up, if at all, until 2.0. From aisaac at american.edu Tue Apr 29 20:18:39 2008 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 29 Apr 2008 20:18:39 -0400 Subject: [Numpy-discussion] the path forward In-Reply-To: References: <48174FF5.50005@noaa.gov><48179694.9010304@enthought.com> Message-ID: On Tue, 29 Apr 2008, Keith Goodman apparently wrote: > I hope that changing x[0,:] is considered a major change since it will > break most any package based on matrices (mine). And so > I hope that such a change wouldn't show up, if at all, > until 2.0. What if the extant matrix class would continue to be available for awhile as "oldmatrix", which you could then import as "matrix" if you wished. Would that meet your needs? (I am simply assuming this would be feasible, without being sure how a lot of the special casing for matrices has been done.) Cheers, Alan Isaac From kwgoodman at gmail.com Tue Apr 29 20:40:01 2008 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 29 Apr 2008 17:40:01 -0700 Subject: [Numpy-discussion] the path forward In-Reply-To: References: <48179694.9010304@enthought.com> Message-ID: On Tue, Apr 29, 2008 at 5:18 PM, Alan G Isaac wrote: > On Tue, 29 Apr 2008, Keith Goodman apparently wrote: > > I hope that changing x[0,:] is considered a major change since it will > > break most any package based on matrices (mine). And so > > I hope that such a change wouldn't show up, if at all, > > until 2.0. > > What if the extant matrix class would continue to be > available for awhile as "oldmatrix", which you could then > import as "matrix" if you wished. Would that meet your > needs? (I am simply assuming this would be feasible, > without being sure how a lot of the special casing for > matrices has been done.) I never import matrix directly (is that what you are suggesting?). I usually create it with M.ones, M.rand, x * y, etc., where M is import numpy.matlib as M If a big change is made to matrix behavior could it be accessed in 1.X from the __future__ and, if successful, switched to the "present" in 2.X? I, obviously, have no idea what would be involved to make that happen. In my use, changing x[0] is not a big deal, but changing x[0,:] is. From lists at informa.tiker.net Tue Apr 29 21:26:44 2008 From: lists at informa.tiker.net (Andreas =?iso-8859-1?q?Kl=F6ckner?=) Date: Tue, 29 Apr 2008 21:26:44 -0400 Subject: [Numpy-discussion] Starting to work on runtime plugin system for plugin (automatic sse optimization, etc...) In-Reply-To: References: <4815E2E9.5020506@ar.media.kyoto-u.ac.jp> Message-ID: <200804292126.47685.lists@informa.tiker.net> On Dienstag 29 April 2008, Lisandro Dalcin wrote: > Your implementation make uses of low level dlopening. Then, your are > going to have to manage all the oddities of runtime loading in the > different systems. Argh. -1 for a hard dependency on dlopen(). At some point in my life, I might be forced to compile numpy on an IBM Bluegene/L, which does *not* have dynamic linking at all. (Btw, anybody done something like this before?) Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From david at ar.media.kyoto-u.ac.jp Tue Apr 29 21:59:30 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 30 Apr 2008 10:59:30 +0900 Subject: [Numpy-discussion] Can anyone look at #759 In-Reply-To: <4817384F.9000406@enthought.com> References: <4816E630.2040407@ar.media.kyoto-u.ac.jp> <4817384F.9000406@enthought.com> Message-ID: <4817D282.7060407@ar.media.kyoto-u.ac.jp> Travis E. Oliphant wrote: > David Cournapeau wrote: > >> Hi, >> >> I have a patch for an issue which had bugged me for some time, but >> since I am not familiar with that part of numpy code, I would prefer >> someone checking I am not doing anything stupid before checking in, >> >> > It looks good to me. > Ok, I included it in r5112. Thanks, David From david at ar.media.kyoto-u.ac.jp Tue Apr 29 22:29:05 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 30 Apr 2008 11:29:05 +0900 Subject: [Numpy-discussion] Starting to work on runtime plugin system for plugin (automatic sse optimization, etc...) In-Reply-To: References: <4815E2E9.5020506@ar.media.kyoto-u.ac.jp> Message-ID: <4817D971.8040204@ar.media.kyoto-u.ac.jp> Lisandro Dalcin wrote: > David, I briefly took a look at your code, and I have a very, very > important observation. > > Your implementation make uses of low level dlopening. Then, your are > going to have to manage all the oddities of runtime loading in the > different systems. In this case, 'libtool' could really help. I know, > it is GPL, but AFAIK it has some special licencing that let's you ship > it with your code in the same licence terms than your code. Ok, there are several issues here: 1 cross platform runtime loading 2 how to access the plugin capabilities (function pointer, interfaces, etc...) 3 how to build 1: the implementation is not cross platform, but the API is; It took me ~ 2 hours to refactor symbol loading, and getting an implementation for posix/win32/Mac os X. I don't know any OS with runtime loading capabilities and without the ability to load a file and a symbol from it; if it does not, it cannot be used by python anyway, everything would have to be build statically at the same time as python, which we do not support in numpy anway, AFAIK. 2: I studied quite a bit several approaches before using this one. That was my main concern at first. For plugins, you have the following possibilities I am aware of: - raw function pointer - COM - pre-defined API Raw function pointer are the simplest, but is not really scalable. COM is this big monstrosity, extremely ackward to use, but can be extended ad nauseum, without pre-defined interface. By pre-defined API, I mean something like VST plugins and the co (used for music softwares, where a host can load may plugins to provide sound effectsl it is the de-facto standard, Mac OS X has its built-in thing called AudioUnit, which is the same thing for what matters here). For the usage I have in mind (blas, fft, lapack, etc...), the API cannot be pre-defined (each one is totally different), so I quickly dismiss the VST-like approach. Then there is the COM thing, which is really complicated. Although each plugin interface is totally different (blas vs lapack), they are relatively fixed in stone, so I thought that by using generated code, the scalability problem of raw pointers could be alleviated. 3: libtool does not know about windows, and I think it is way too overkill. We can't use libtool for building (which is one of the big thing libtool provides). dlopen-like approach is not as portable as libtool, but it is as portable as python, which is good enough for numpy :) > But, I definitely think that a betther approach would be using a stubs > mechanism ala TCL, or wath is currently used in some extension modules > in core python, like cStringIO. In short, you access all your > functions from a pointer do a struct (statically or heap allocated) > where each struct member is filled with a pointer to a function. This > is pretty much similar to C++ virtual tables, or the Cython cdef's > classes with cdef's methods. And then you just let Python do the work > of dynamic loading of extension modules. The numpy C/API uses a > similar approach, but uses an array. IMHO, the struct approach is > cleaner. > > What do you think about this? It may be cleaner, but I am not convinced it buys us much. With my approach, all is needed for the existing code (numpy.core, numpy.linalg, etc...) is a renaming of the used function (which can be done in 5 minutes with sed), because in C, calling a function or a function pointer is exactly the same thing. With the approach of function pointers, you will have to replace all the function calls for blas, lapack, etc... by a system to pass the array of function pointers. That sounds like nightmare to me, because I don't see how to do that automatically. Maybe I just don't see it; in that case, what would be your approach ? I have been convinced that the function pointer approach is usable by looking at liboil, which does exactly the thing we need: http://liboil.freedesktop.org/wiki/ (you can look at the liboil/liboilfuncs* and liboil/liboilfunc.* files). Liboil's approach is more complicated, because the function pointer is provided by a function factory (so that each function can be initialized differently), but I don't think we need the factory in our case (and if we need, we can do it without changing anything in the code which uses the plugin). cheers, David From david at ar.media.kyoto-u.ac.jp Tue Apr 29 22:33:27 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 30 Apr 2008 11:33:27 +0900 Subject: [Numpy-discussion] Starting to work on runtime plugin system for plugin (automatic sse optimization, etc...) In-Reply-To: <200804292126.47685.lists@informa.tiker.net> References: <4815E2E9.5020506@ar.media.kyoto-u.ac.jp> <200804292126.47685.lists@informa.tiker.net> Message-ID: <4817DA77.2030409@ar.media.kyoto-u.ac.jp> Andreas Kl?ckner wrote: > > Argh. -1 for a hard dependency on dlopen(). There is no hard dependency on dlopen, there is a hard dependency on runtime loading, because well, that's the point of a plugin system. It should not be difficult to be able to disable the plugin system for platforms who do not support it, though (and do as today), but I am not sure it is really useful. > At some point in my life, I might > be forced to compile numpy on an IBM Bluegene/L, which does *not* have > dynamic linking at all. (Btw, anybody done something like this before?) How will you build numpy in the case of a system without dynamic linking ? The only solution is then to build numpy and link it statically to the python interpreter. Systems without dynamic linking are common (embedded systems), though. cheers, David From lists at informa.tiker.net Tue Apr 29 22:58:35 2008 From: lists at informa.tiker.net (Andreas =?iso-8859-1?q?Kl=F6ckner?=) Date: Tue, 29 Apr 2008 22:58:35 -0400 Subject: [Numpy-discussion] Starting to work on runtime plugin system for plugin (automatic sse optimization, etc...) In-Reply-To: <4817DA77.2030409@ar.media.kyoto-u.ac.jp> References: <4815E2E9.5020506@ar.media.kyoto-u.ac.jp> <200804292126.47685.lists@informa.tiker.net> <4817DA77.2030409@ar.media.kyoto-u.ac.jp> Message-ID: <200804292258.37209.lists@informa.tiker.net> On Dienstag 29 April 2008, David Cournapeau wrote: > Andreas Kl?ckner wrote: > > Argh. -1 for a hard dependency on dlopen(). > > There is no hard dependency on dlopen, there is a hard dependency on > runtime loading, because well, that's the point of a plugin system. It > should not be difficult to be able to disable the plugin system for > platforms who do not support it, though (and do as today), but I am not > sure it is really useful. As long as it's easy to disable (for example with a preprocessor define), I guess I'm ok. > > At some point in my life, I might > > be forced to compile numpy on an IBM Bluegene/L, which does *not* have > > dynamic linking at all. (Btw, anybody done something like this before?) > > How will you build numpy in the case of a system without dynamic linking > ? The only solution is then to build numpy and link it statically to the > python interpreter. Systems without dynamic linking are common (embedded > systems), though. Yes, obviously everything will need to be linked into one big static executable blob. I am somewhat certain that distutils will be of no help there, so I will need to "roll my own". There is a CMake-based build of Python for BG/L, I was planning to work off that. But so far, I might not end up having to do all that, for which I'd be endlessly grateful. Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From david at ar.media.kyoto-u.ac.jp Tue Apr 29 22:52:26 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 30 Apr 2008 11:52:26 +0900 Subject: [Numpy-discussion] Starting to work on runtime plugin system for plugin (automatic sse optimization, etc...) In-Reply-To: <200804292258.37209.lists@informa.tiker.net> References: <4815E2E9.5020506@ar.media.kyoto-u.ac.jp> <200804292126.47685.lists@informa.tiker.net> <4817DA77.2030409@ar.media.kyoto-u.ac.jp> <200804292258.37209.lists@informa.tiker.net> Message-ID: <4817DEEA.3010606@ar.media.kyoto-u.ac.jp> Andreas Kl?ckner wrote: > > Yes, obviously everything will need to be linked into one big static > executable blob. I am somewhat certain that distutils will be of no help > there, so I will need to "roll my own". There is a CMake-based build of > Python for BG/L, I was planning to work off that. > You will have to build numpy too. Not that I want to discourage you, but that will be a hell of a work. cheers, David From david at ar.media.kyoto-u.ac.jp Tue Apr 29 23:21:06 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 30 Apr 2008 12:21:06 +0900 Subject: [Numpy-discussion] Starting to work on runtime plugin system for plugin (automatic sse optimization, etc...) In-Reply-To: <200804292258.37209.lists@informa.tiker.net> References: <4815E2E9.5020506@ar.media.kyoto-u.ac.jp> <200804292126.47685.lists@informa.tiker.net> <4817DA77.2030409@ar.media.kyoto-u.ac.jp> <200804292258.37209.lists@informa.tiker.net> Message-ID: <4817E5A2.90206@ar.media.kyoto-u.ac.jp> Andreas Kl?ckner wrote: > But so far, I might not end up having to do all that, for which I'd be > endlessly grateful. > If you really need it, note that numpy can be built with scons instead of distutils, and the scons scripts are now available in numpy svn (and will be included in the releases sources starting from 1.1.0). Scons severely lacks in the cross-compilation departement, but I think scons scripts are easier to adapt to cmake than distutils setup.py files if you need to use cmake :) I would actually be quite interested in making numpy build in a cross-compilation environment with scons cheers, David From lists at informa.tiker.net Tue Apr 29 23:50:35 2008 From: lists at informa.tiker.net (Andreas =?iso-8859-1?q?Kl=F6ckner?=) Date: Tue, 29 Apr 2008 23:50:35 -0400 Subject: [Numpy-discussion] Starting to work on runtime plugin system for plugin (automatic sse optimization, etc...) In-Reply-To: <4817DEEA.3010606@ar.media.kyoto-u.ac.jp> References: <4815E2E9.5020506@ar.media.kyoto-u.ac.jp> <200804292258.37209.lists@informa.tiker.net> <4817DEEA.3010606@ar.media.kyoto-u.ac.jp> Message-ID: <200804292350.36777.lists@informa.tiker.net> On Dienstag 29 April 2008, David Cournapeau wrote: > Andreas Kl?ckner wrote: > > Yes, obviously everything will need to be linked into one big static > > executable blob. I am somewhat certain that distutils will be of no help > > there, so I will need to "roll my own". There is a CMake-based build of > > Python for BG/L, I was planning to work off that. > > You will have to build numpy too. Not that I want to discourage you, but > that will be a hell of a work. Good news is that Bluegene/P (the next version of that architecture) *does* support dynamic linking. It's probably broken in some obscure way, but that's (hopefully) better than not exsitent. :) In any case, if I can't dodge porting my code to BG/L, you'll hear from me. :) Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: From nadavh at visionsense.com Wed Apr 30 00:18:08 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Wed, 30 Apr 2008 07:18:08 +0300 Subject: [Numpy-discussion] dot/tensordot limitations References: Message-ID: <710F2847B0018641891D9A216027636029C12C@ex3.envision.co.il> You open here a Pandora box: What should I do if I want to run an operation on elements of an array which are not in the library. The usual answers are either use more memory ( (A*B).sum(axis=1) ), or more time ([dot(A[i],B[i]) for i in len(A)]). Would your issue can be rephrase like this: A way to iterate over array in the C level, where the function/operation is defined in the C level, without the overhead of python? Nadav. -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? Anne Archibald ????: ? 30-?????-08 01:41 ??: Discussion of Numerical Python ????: [Numpy-discussion] dot/tensordot limitations Timothy Hochberg has proposed a generalization of the matrix mechanism to support manipulating arrays of linear algebra objects. For example, one might have an array of matrices one wants to apply to an array of vectors, to yield an array of vectors: In [88]: A = np.repeat(np.eye(3)[np.newaxis,...],2,axis=0) In [89]: A Out[89]: array([[[ 1., 0., 0.], [ 0., 1., 0.], [ 0., 0., 1.]], [[ 1., 0., 0.], [ 0., 1., 0.], [ 0., 0., 1.]]]) In [90]: V = np.array([[1,0,0],[0,1,0]]) Currently, it is very clumsy to handle this kind of situation even with arrays, keeping track of dimensions by hand. For example if one wants to multiply A by V "elementwise", one cannot simply use dot: In [92]: np.dot(A,V.T) Out[92]: array([[[ 1., 0.], [ 0., 1.], [ 0., 0.]], [[ 1., 0.], [ 0., 1.], [ 0., 0.]]]) tensordot() suffers from the same problem. It arises because when you combine two multiindexed objects there are a number of ways you can do it: A: Treat all indices as different and form all pairwise products (outer product): In [93]: np.multiply.outer(A,V).shape Out[93]: (2, 3, 3, 2, 3) B: Contract the outer product on a pair of indices: In [98]: np.tensordot(A,V,axes=(-1,-1)).shape Out[98]: (2, 3, 2) C: Take the diagonal along a pair of indices: In [111]: np.diagonal(np.multiply.outer(A,V),axis1=0,axis2=3).shape Out[111]: (3, 3, 3, 2) What we want in this case is a combination of B and C: In [106]: np.diagonal(np.tensordot(A,V,axes=(-1,-1)),axis1=0,axis2=2).T.shape Out[106]: (2, 3) but it cannot be done without constructing a larger array and pulling out its diagonal. If this seems like an exotic thing to want to do, suppose instead we have two arrays of vectors, and we want to evaluate the array of dot products. None of no.dot, np.tensordot, or np.inner produce the results you want. You have to multiply elementwise and sum. (This also precludes automatic use of BLAS, if indeed tensordot uses BLAS.) Does it make sense to implement a generalized tensordot that can do this, or is it the sort of thing one should code by hand every time? Is there any way we can make it easier to do simple common operations like take the elementwise matrix-vector product of A and V? The more general issue, of making linear algebra natural by keeping track of which indices are for elementwise operations, and which should be used for dots (and how), is a whole other kettle of fish. I think for that someone should think hard about writing a full-featured multilinear algebra package (might as well keep track of coordinate transformations too while one was at it) if this is intended. Anne _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion From oliphant at enthought.com Wed Apr 30 00:29:39 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 29 Apr 2008 23:29:39 -0500 Subject: [Numpy-discussion] dot/tensordot limitations In-Reply-To: <710F2847B0018641891D9A216027636029C12C@ex3.envision.co.il> References: <710F2847B0018641891D9A216027636029C12C@ex3.envision.co.il> Message-ID: <4817F5B3.8070501@enthought.com> Nadav Horesh wrote: > You open here a Pandora box: > What should I do if I want to run an operation on elements of an array which are not in the library. The usual answers are either use more memory ( (A*B).sum(axis=1) ), or more time ([dot(A[i],B[i]) for i in len(A)]). > > Would your issue can be rephrase like this: > A way to iterate over array in the C level, where the function/operation is defined in the C level, without the overhead of python? > My plan for this is the general function listed on the NumPy Trac area along with a weave-like kernel creation from Python code. http://www.scipy.org/scipy/numpy/wiki/GeneralLoopingFunctions I'd love to get time to do this or mentor someone else in doing it. -Travis From oliphant at enthought.com Wed Apr 30 00:31:30 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Tue, 29 Apr 2008 23:31:30 -0500 Subject: [Numpy-discussion] dot/tensordot limitations In-Reply-To: <710F2847B0018641891D9A216027636029C12C@ex3.envision.co.il> References: <710F2847B0018641891D9A216027636029C12C@ex3.envision.co.il> Message-ID: <4817F622.7090907@enthought.com> Nadav Horesh wrote: > You open here a Pandora box: > What should I do if I want to run an operation on elements of an array which are not in the library. The usual answers are either use more memory ( (A*B).sum(axis=1) ), or more time ([dot(A[i],B[i]) for i in len(A)]). > > Would your issue can be rephrase like this: > A way to iterate over array in the C level, where the function/operation is defined in the C level, without the overhead of python? > Quoting from the GeneralLoopingFunction wiki page: There seems to be a general need for looping over functions that work on whole arrays and return other arrays. The function has information associated with it that states what the basic dimensionality of the inputs are as well as the corresponding dimensionality of the outputs. This is in addition to information like supported data-types and so forth. Then, when "larger" arrays are provided as inputs, the extra dimensions should be "broadcast" to create a looping that calls the underlying construct on the different sub-arrays and produces multiple outputs. Thus, let's say we have a function, basic_inner, that takes two vectors and returns a scalar where the signature of the function is known to take two 1-d arrays of the same shape and return a scalar. Then, when this same function is converted to a general looping function: loopfuncs.inner(a, b) where a is (3,5,N) and b is (5,N) will return an output that is (3,5) whose elements are constructed by calling the underlying function repeatedly on each piece. Perl has a notion of threading that captures a part of this idea, I think. The concept is to have a more general function than the ufuncs where the signature includes dimensionality so that the function does more than "element-by-element" but does "sub-array" by "sub-array". Such a facility would prove very useful, I think and would abstract many operations very well. -Travis From charlesr.harris at gmail.com Wed Apr 30 01:24:08 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 29 Apr 2008 23:24:08 -0600 Subject: [Numpy-discussion] dot/tensordot limitations In-Reply-To: <4817F622.7090907@enthought.com> References: <710F2847B0018641891D9A216027636029C12C@ex3.envision.co.il> <4817F622.7090907@enthought.com> Message-ID: On Tue, Apr 29, 2008 at 10:31 PM, Travis E. Oliphant wrote: > Nadav Horesh wrote: > > You open here a Pandora box: > > What should I do if I want to run an operation on elements of an array > which are not in the library. The usual answers are either use more memory ( > (A*B).sum(axis=1) ), or more time ([dot(A[i],B[i]) for i in len(A)]). > > > > Would your issue can be rephrase like this: > > A way to iterate over array in the C level, where the function/operation > is defined in the C level, without the overhead of python? > > > > Quoting from the GeneralLoopingFunction wiki page: > > > > > There seems to be a general need for looping over functions that work on > whole arrays and return other arrays. The function has information > associated with it that states what the basic dimensionality of the > inputs are as well as the corresponding dimensionality of the outputs. > This is in addition to information like supported data-types and so forth. > > Then, when "larger" arrays are provided as inputs, the extra dimensions > should be "broadcast" to create a looping that calls the underlying > construct on the different sub-arrays and produces multiple outputs. > > Thus, let's say we have a function, basic_inner, that takes two vectors > and returns a scalar where the signature of the function is known to > take two 1-d arrays of the same shape and return a scalar. > > Then, when this same function is converted to a general looping function: > > loopfuncs.inner(a, b) > > where a is (3,5,N) and b is (5,N) will return an output that is (3,5) > whose elements are constructed by calling the underlying function > repeatedly on each piece. Perl has a notion of threading that captures a > part of this idea, I think. The concept is to have a more general > function than the ufuncs where the signature includes dimensionality so > that the function does more than "element-by-element" but does > "sub-array" by "sub-array". > > Such a facility would prove very useful, I think and would abstract many > operations very well. > Basically, element wise operations on elements that are subarrays; generalized ufuncs, if you will. Sorting would be a unary type operation in that context ;) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Apr 30 03:16:07 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 30 Apr 2008 01:16:07 -0600 Subject: [Numpy-discussion] dot/tensordot limitations In-Reply-To: References: Message-ID: On Tue, Apr 29, 2008 at 4:41 PM, Anne Archibald wrote: > Timothy Hochberg has proposed a generalization of the matrix mechanism > to support manipulating arrays of linear algebra objects. For example, > one might have an array of matrices one wants to apply to an array of > vectors, to yield an array of vectors: > > In [88]: A = np.repeat(np.eye(3)[np.newaxis,...],2,axis=0) > > In [89]: A > Out[89]: > array([[[ 1., 0., 0.], > [ 0., 1., 0.], > [ 0., 0., 1.]], > > [[ 1., 0., 0.], > [ 0., 1., 0.], > [ 0., 0., 1.]]]) > > In [90]: V = np.array([[1,0,0],[0,1,0]]) > > Currently, it is very clumsy to handle this kind of situation even > with arrays, keeping track of dimensions by hand. For example if one > wants to multiply A by V "elementwise", one cannot simply use dot: > Let A have dimensions LxMxN and b dimensions LxN, then sum(A*b[:,newaxis,:], axis=-1) will do the trick. Example: In [1]: A = ones((2,2,2)) In [2]: b = array([[1,2],[3,4]]) In [3]: A*b[:,newaxis,:] Out[3]: array([[[ 1., 2.], [ 1., 2.]], [[ 3., 4.], [ 3., 4.]]]) In [4]: sum(A*b[:,newaxis,:], axis=-1) Out[4]: array([[ 3., 3.], [ 7., 7.]]) Chuck Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From lmnop345 at gmail.com Wed Apr 30 04:11:36 2008 From: lmnop345 at gmail.com (a g) Date: Wed, 30 Apr 2008 01:11:36 -0700 Subject: [Numpy-discussion] very simple iteration question. Message-ID: <9a771bf70804300111s749747b8we8a4393a3c68fd02@mail.gmail.com> Hi. This is a very basic question, sorry if it's irritating. If i didn't find the answer written already somewhere on the site, please point me to it. That'd be great. OK: how do i iterate over an axis other than 0? I have a 3D array of data[year, week, location]. I want to iterate over each year at each location and run a series of stats on the columns (on the 52 weeks in a particular year at a particular location). 'for years in data:' will get the first one, but then how do i not iterate over the 1 axis and iterate over the 2 axis instead? thanks, alex. From peridot.faceted at gmail.com Wed Apr 30 04:21:57 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Wed, 30 Apr 2008 04:21:57 -0400 Subject: [Numpy-discussion] should outer take an output argument? In-Reply-To: References: Message-ID: 2008/4/29 Alan G Isaac : > As I was looking at Bill's conjugate gradient posting, > I found myself wondering if there would be a payoff > to an output argument for ``numpy.outer``. (It is fairly > natural to repeatedly recreate the outer product of > the adjusted residuals, which is only needed during a > single iteration.) I avoid np.outer, as it seems designed for foot shooting. (It forcibly flattens both arguments into vectors no matter what shape they are and then computes their outer product.) np.multiply.outer does (what I consider to be) the right thing. Not that it takes an output argument either, as far as I can tell. But trying to inspect it suggests a few questions: * Why are ufunc docstrings totally useless? * Why are output arguments not keyword arguments? As for your original question, yes, I think it would be good if .outer() and .reduce() methods of ufuncs took output arguments. This would let one use add.reduce() as a poor man's sum() even when dealing with uint8s, for example. (Coming up with a problem for which subtract.reduce(), divide.reduce(), or arctan2.reduce() are the solutions is left as an exercise for the reader.) I can definitely see a use for divide.outer() : Int -> Int -> Float, though. Anne From nadavh at visionsense.com Wed Apr 30 04:17:35 2008 From: nadavh at visionsense.com (Nadav Horesh) Date: Wed, 30 Apr 2008 11:17:35 +0300 Subject: [Numpy-discussion] very simple iteration question. References: <9a771bf70804300111s749747b8we8a4393a3c68fd02@mail.gmail.com> Message-ID: <710F2847B0018641891D9A216027636029C12E@ex3.envision.co.il> for i in range(52): week_data = data[:,i,:] OR for week_data in data.transpose(1,0,2): ... Nadav -----????? ??????----- ???: numpy-discussion-bounces at scipy.org ??? a g ????: ? 30-?????-08 11:11 ??: numpy-discussion at scipy.org ????: [Numpy-discussion] very simple iteration question. Hi. This is a very basic question, sorry if it's irritating. If i didn't find the answer written already somewhere on the site, please point me to it. That'd be great. OK: how do i iterate over an axis other than 0? I have a 3D array of data[year, week, location]. I want to iterate over each year at each location and run a series of stats on the columns (on the 52 weeks in a particular year at a particular location). 'for years in data:' will get the first one, but then how do i not iterate over the 1 axis and iterate over the 2 axis instead? thanks, alex. _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion From peridot.faceted at gmail.com Wed Apr 30 04:33:49 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Wed, 30 Apr 2008 04:33:49 -0400 Subject: [Numpy-discussion] very simple iteration question. In-Reply-To: <9a771bf70804300111s749747b8we8a4393a3c68fd02@mail.gmail.com> References: <9a771bf70804300111s749747b8we8a4393a3c68fd02@mail.gmail.com> Message-ID: 2008/4/30 a g : > Hi. This is a very basic question, sorry if it's irritating. If i > didn't find the answer written already somewhere on the site, please > point me to it. That'd be great. > > OK: how do i iterate over an axis other than 0? > > I have a 3D array of data[year, week, location]. I want to iterate > over each year at each location and run a series of stats on the > columns (on the 52 weeks in a particular year at a particular location). > 'for years in data:' will get the first one, but then how do i not > iterate over the 1 axis and iterate over the 2 axis instead? Well, there's always for i in xrange(A.shape[1]): do_something(A[:,i,...]) But that's kind of ugly in this day and age. I would be tempted by for a in np.rollaxis(A,1): do_something(a) It's worth mentioning that traversing an array along an axis not the first usually results in subarrays that are not contiguous in memory. While numpy makes working with these convenient, they may not be very efficient for cache reasons (for example, modern processors load from memory - an extremely slow operation - in blocks that may be as large as 64 bytes; if you only use eight of these before moving on, your code will use much more memory bandwidth than it needs to). If this is a concern, you can usually transparently rearrange your array's memory layout to avoid it. Anne From eads at soe.ucsc.edu Wed Apr 30 04:40:44 2008 From: eads at soe.ucsc.edu (Damian Eads) Date: Wed, 30 Apr 2008 01:40:44 -0700 Subject: [Numpy-discussion] very simple iteration question. In-Reply-To: <9a771bf70804300111s749747b8we8a4393a3c68fd02@mail.gmail.com> References: <9a771bf70804300111s749747b8we8a4393a3c68fd02@mail.gmail.com> Message-ID: <4818308C.6050808@soe.ucsc.edu> Hi Alex, a g wrote: > Hi. This is a very basic question, sorry if it's irritating. If i > didn't find the answer written already somewhere on the site, please > point me to it. That'd be great. You should look at any of the documents below and read up on array slicing. It is perhaps the most important and pervasive concept of Numpy and should be understood by all users. Numpy Tutorial: http://www.scipy.org/Tentative_NumPy_Tutorial Numpy for MATLAB users: http://www.scipy.org/NumPy_for_Matlab_Users Guide to Numpy > OK: how do i iterate over an axis other than 0? > > I have a 3D array of data[year, week, location]. I want to iterate > over each year at each location and run a series of stats on the > columns (on the 52 weeks in a particular year at a particular location). > 'for years in data:' will get the first one, but then how do i not > iterate over the 1 axis and iterate over the 2 axis instead? It is not clear to me whether you want to slice or iterate over an array. Assuming you are fixing the year and location, the following code iterates over data for fixed year and location. for week in xrange(0, 52): data[year, week, loc] Slicing is more efficient and you should use it if you can. Fixing the year and location, the following computes the mean and standard deviation across all weeks. All of the statements below yield scalars. data[year, :, loc].mean() -- takes the mean of the data across weeks data[year, :, loc].std() -- takes the standard deviation of the data across weeks You should download IPython and type help(numpy.array) to see one set of functions you can call on the result of a slice (sum, min, etc.). Although I don't know what statistics you are computing for sure, the following code might be useful since it computes a statistic across all weeks for each year and location value. data.mean(axis=1) It yields a num_years by num_locations array mu where mu[y, l] is the average data value across all weeks for year y and loc l. I hope this helps. Damian From wfspotz at sandia.gov Wed Apr 30 09:49:43 2008 From: wfspotz at sandia.gov (Bill Spotz) Date: Wed, 30 Apr 2008 07:49:43 -0600 Subject: [Numpy-discussion] the path forward In-Reply-To: References: <48174FF5.50005@noaa.gov> <48179694.9010304@enthought.com> Message-ID: <72F90B08-74F2-42ED-A943-38173E586A33@sandia.gov> On Apr 29, 2008, at 6:01 PM, Keith Goodman wrote: > I hope that changing x[0,:] is considered a major change since it will > break most any package based on matrices (mine). And so I hope that > such a change wouldn't show up, if at all, until 2.0. The only code that should break would be indexing the extracted row/ column with two indexes. Multiplication with the extracted row/column would still work as expected (and that sounded to me like what the application typically was . . . maybe I'm wrong). ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From travis at enthought.com Wed Apr 30 10:47:30 2008 From: travis at enthought.com (Travis Vaught) Date: Wed, 30 Apr 2008 09:47:30 -0500 Subject: [Numpy-discussion] [ANN] EuroSciPy Abstracts Deadline Reminder Message-ID: Greetings, Just a reminder: the abstracts for the EuroSciPy Conference in Leipzig are due by midnight tonight (CST, US [UTC -6]) April, 30. If you'd like to present, please submit your abstract as a PDF, MS Word or plain text file to euroabstracts at scipy.org. For more information on the EuroSciPy Conference, please see: http://www.scipy.org/EuroSciPy2008 From Chris.Barker at noaa.gov Wed Apr 30 12:26:39 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 30 Apr 2008 09:26:39 -0700 Subject: [Numpy-discussion] the path forward In-Reply-To: <72F90B08-74F2-42ED-A943-38173E586A33@sandia.gov> References: <48174FF5.50005@noaa.gov> <48179694.9010304@enthought.com> <72F90B08-74F2-42ED-A943-38173E586A33@sandia.gov> Message-ID: <48189DBF.9050307@noaa.gov> Bill Spotz wrote: > On Apr 29, 2008, at 6:01 PM, Keith Goodman wrote: >> break most any package based on matrices (mine). And so I hope that >> such a change wouldn't show up, if at all, until 2.0. > > The only code that should break would be indexing the extracted row/ > column with two indexes. It will also break code that expects M[i,:] to be the same shape as M[[i],:] and M[i:i+1,:], though it sounded like that was only really expected during testing. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Wed Apr 30 12:48:01 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 30 Apr 2008 09:48:01 -0700 Subject: [Numpy-discussion] very simple iteration question. In-Reply-To: <9a771bf70804300111s749747b8we8a4393a3c68fd02@mail.gmail.com> References: <9a771bf70804300111s749747b8we8a4393a3c68fd02@mail.gmail.com> Message-ID: <4818A2C1.5010208@noaa.gov> a g wrote: > OK: how do i iterate over an axis other than 0? This ties in nicely with some of the discussion about interating over matrices. It ahs been suggested that it would be nice to have iterators for matrices, so you could do: for row in M.rows: ... and for column in M.cols: ... If so, then wouldn't it make sense to have built in iterators for nd-arrays as well? something like: for subarray in A.iter(axis=i): ... where axis would default to 0. This is certainly cleaner than: for j in range(A.shape[i]): subarray = A[:,:,j,:] Wait! I have no idea how to spell that generically -- i.e. the ith index, where i is a variable. There must be a way to build a slice object dynamically, but I don't know it. How often would i be a variable, rather than hard coded? I have no idea. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From peridot.faceted at gmail.com Wed Apr 30 13:24:43 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Wed, 30 Apr 2008 19:24:43 +0200 Subject: [Numpy-discussion] very simple iteration question. In-Reply-To: <4818A2C1.5010208@noaa.gov> References: <9a771bf70804300111s749747b8we8a4393a3c68fd02@mail.gmail.com> <4818A2C1.5010208@noaa.gov> Message-ID: 2008/4/30 Christopher Barker : > a g wrote: > > OK: how do i iterate over an axis other than 0? > > This ties in nicely with some of the discussion about interating over > matrices. It ahs been suggested that it would be nice to have iterators > for matrices, so you could do: > > for row in M.rows: > ... > > and > > for column in M.cols: > ... > > > If so, then wouldn't it make sense to have built in iterators for > nd-arrays as well? something like: > > for subarray in A.iter(axis=i): > ... > > where axis would default to 0. > > This is certainly cleaner than: > > for j in range(A.shape[i]): > subarray = A[:,:,j,:] > > Wait! I have no idea how to spell that generically -- i.e. the ith > index, where i is a variable. There must be a way to build a slice > object dynamically, but I don't know it. Slices can be built without too much trouble using slice(), but it's much easier to just write for subarray in np.rollaxis(A,i): ... rollaxis() just pulls the specified axis to the front. (It doesn't do what I thought it would, which is a cyclic permutation of the axes). Anne From Chris.Barker at noaa.gov Wed Apr 30 13:52:55 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 30 Apr 2008 10:52:55 -0700 Subject: [Numpy-discussion] recarray fun Message-ID: <4818B1F7.904@noaa.gov> Hi folks, Someone on the wxPython list posted a nifty recarray example that I don't quite understand. The idea is to have an array for an RGBA image: rgbarec = numpy.dtype({'r':(numpy.uint8,0), 'g':(numpy.uint8,1), 'b':(numpy.uint8,2), 'a':(numpy.uint8,3)}) A = numpy.zeros(shape, dtype=(numpy.uint32, rgbarec) ) what I don't understand is having BOTH numpy.uint32 and rgbrec as the dtype. How does that work? Actually, with a bit of testing, it's pretty cool -- if you index like: A[i,j] ==> uint32 A['r'][i,j] ==> uint8 pretty cool really. it seems that numpy is treating it as both a recarray and a regular uint32 array, which makes some sense, but I'm still confused by the semantics. also: >>> A array([[4278190080, ... [4278190080, 4278190080, 4278190080, 4278190080, 4278190080]], dtype=uint32) but: >>> A.dtype dtype(('>u4', [('r', '|u1'), ('g', '|u1'), ('b', '|u1'), ('a', '|u1')])) so what is the dtype? Also, I see advantages and disadvantages to either way. If you do: A = numpy.zeros(shape, dtype=(numpy.uint32, rgbarec) ) then you can do: A[i,j]['r'] to get the red value of a pixel A[i,j] = (red, green, blue, alpha) to set a pixel A[:,:] = (red, green, blue, alpha) to make the whole image one color With the "dual dtype" approach: A = numpy.zeros(shape, dtype=(numpy.uint32, rgbarec) ) You need to set the pixels separately: A['r'][i,j] = red B['r'][i,j] = green ... or construct uint32 values: C = numpy.uint32(red) C += numpy.uint32(green) << 8 C += numpy.uint32(blue) << 16 Which is really ugly! (and I may not even have it right!). Is there a better way? One more idea -- is there a way to have two arrays, each with a different dtype, that point to the same data? maybe: RGBImage = numpy.zeros(shape, dtype=rgbarec ) IntImage = RGBImage.view() IntImage.dtype = numpy.uint32 That way you could work with it either way, whichever was easier in the context. So -- what are folks' thoughts about how best to deal with images as numpy arrays? (I'll put it in a Wiki page if I get some good comments) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From doutriaux1 at llnl.gov Wed Apr 30 14:34:00 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Wed, 30 Apr 2008 11:34:00 -0700 Subject: [Numpy-discussion] numpy.ma question Message-ID: <4818BB98.1000004@llnl.gov> Hello i have a quick question about MA ans scalar the following works: import numpy.ma as MA a0= MA.array(0)/1 a1= MA.array((0,0))/1 but not: import numpy.ma as MA a0= MA.array(0)/0 a1= MA.array((0,0))/0 Is that a bug ? I'm using numpy 1.0.5.dev4958 and also whats in trunk right now (1.1.0.dev5113) C. From doutriaux1 at llnl.gov Wed Apr 30 14:35:38 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Wed, 30 Apr 2008 11:35:38 -0700 Subject: [Numpy-discussion] oldnumeric.MA behaviour broken ? Message-ID: <4818BBFA.9040000@llnl.gov> HI, import numpy.oldnumeric.ma as MA a0= MA.array(0)/0 sh0=list(a0.shape) sh0.insert(0,1) b0=MA.resize(a0,sh0) Does not work anymore, I believe it used to work It does works using numpy.ma (but i can't subclass these yet...) C. From Chris.Barker at noaa.gov Wed Apr 30 14:57:44 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 30 Apr 2008 11:57:44 -0700 Subject: [Numpy-discussion] very simple iteration question. In-Reply-To: References: <9a771bf70804300111s749747b8we8a4393a3c68fd02@mail.gmail.com> <4818A2C1.5010208@noaa.gov> Message-ID: <4818C128.7030908@noaa.gov> Anne Archibald wrote: >> it's much easier to just write > > for subarray in np.rollaxis(A,i): > ... cool, thanks! So the answer to the OPs question: > OK: how do i iterate over an axis other than 0? > > I have a 3D array of data[year, week, location]. I want to iterate > over each year at each location ... for loc in np.rollaxis(data, 2): for year in np.rollaxis(data, 0): # rollaxis not required here, but # for symmetry's sake... .... I think I still like the idea of an iterator (or maybe making rollaxis a method?), but this works pretty well. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant at enthought.com Wed Apr 30 14:58:18 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Wed, 30 Apr 2008 13:58:18 -0500 Subject: [Numpy-discussion] recarray fun In-Reply-To: <4818B1F7.904@noaa.gov> References: <4818B1F7.904@noaa.gov> Message-ID: <4818C14A.2070107@enthought.com> Christopher Barker wrote: > Hi folks, > > Someone on the wxPython list posted a nifty recarray example that I > don't quite understand. The idea is to have an array for an RGBA image: > > rgbarec = numpy.dtype({'r':(numpy.uint8,0), > 'g':(numpy.uint8,1), > 'b':(numpy.uint8,2), > 'a':(numpy.uint8,3)}) > > A = numpy.zeros(shape, dtype=(numpy.uint32, rgbarec) ) > > what I don't understand is having BOTH numpy.uint32 and rgbrec as the > dtype. How does that work? > Basically, the rgbarec defines the fields, but the "base-type" for the numpy array is numpy.uint32 (rather than VOID which is the default data-type for arrays with fields defined). This is why it prints the way it does. I'm not sure what the real value is doing it that way as opposed to just having two views on the data: one as a uint32 and another as a "normal" recarray with a VOID data-type underlying it) like you suggest at the end. I think it's really just the "architecture" showing through to the user layer. -Travis From pgmdevlist at gmail.com Wed Apr 30 14:43:07 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 30 Apr 2008 14:43:07 -0400 Subject: [Numpy-discussion] numpy.ma question In-Reply-To: <4818BB98.1000004@llnl.gov> References: <4818BB98.1000004@llnl.gov> Message-ID: <200804301443.08121.pgmdevlist@gmail.com> Charles, > but not: > import numpy.ma as MA > a0= MA.array(0)/0 > a1= MA.array((0,0))/0 > > Is that a bug ? That a0 is MA.masked is not a bug. That a1 should be a (2,) array masked everywhere should work, but does not: that's the bug. Thanks for reporting. From gael.varoquaux at normalesup.org Wed Apr 30 15:09:30 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 30 Apr 2008 21:09:30 +0200 Subject: [Numpy-discussion] very simple iteration question. In-Reply-To: <4818C128.7030908@noaa.gov> References: <9a771bf70804300111s749747b8we8a4393a3c68fd02@mail.gmail.com> <4818A2C1.5010208@noaa.gov> <4818C128.7030908@noaa.gov> Message-ID: <20080430190930.GG1497@phare.normalesup.org> On Wed, Apr 30, 2008 at 11:57:44AM -0700, Christopher Barker wrote: > I think I still like the idea of an iterator (or maybe making rollaxis a > method?), but this works pretty well. Generally, in object oriented programming, you expect a method like rollaxis to modify an object inplace. At least that would be my expectation. Ga?l From stefan at sun.ac.za Wed Apr 30 15:20:55 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 30 Apr 2008 21:20:55 +0200 Subject: [Numpy-discussion] recarray fun In-Reply-To: <4818B1F7.904@noaa.gov> References: <4818B1F7.904@noaa.gov> Message-ID: <9457e7c80804301220p327be027sb0845b3e125fe31a@mail.gmail.com> Hi Chris 2008/4/30 Christopher Barker : > Someone on the wxPython list posted a nifty recarray example that I > don't quite understand. The idea is to have an array for an RGBA image: > > rgbarec = numpy.dtype({'r':(numpy.uint8,0), > 'g':(numpy.uint8,1), > 'b':(numpy.uint8,2), > 'a':(numpy.uint8,3)}) > I find manipulating data with records arrays highly intuitive; I had the same approach in mind when I wrote http://www.scipy.org/RecordArrays > One more idea -- is there a way to have two arrays, each with a > different dtype, that point to the same data? maybe: > > RGBImage = numpy.zeros(shape, dtype=rgbarec ) > > IntImage = RGBImage.view() > IntImage.dtype = numpy.uint32 > > That way you could work with it either way, whichever was easier in the > context. That's the way, or just rgba_image.view(numpy.int32). Cheers St?fan From doutriaux1 at llnl.gov Wed Apr 30 15:25:43 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Wed, 30 Apr 2008 12:25:43 -0700 Subject: [Numpy-discussion] numpy.ma question In-Reply-To: <200804301443.08121.pgmdevlist@gmail.com> References: <4818BB98.1000004@llnl.gov> <200804301443.08121.pgmdevlist@gmail.com> Message-ID: <4818C7B7.90102@llnl.gov> that's exactly my understanding thanks for confirming C. Pierre GM wrote: > Charles, > > >> but not: >> import numpy.ma as MA >> a0= MA.array(0)/0 >> a1= MA.array((0,0))/0 >> >> Is that a bug ? >> > > That a0 is MA.masked is not a bug. That a1 should be a (2,) array masked > everywhere should work, but does not: that's the bug. Thanks for reporting. > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From pgmdevlist at gmail.com Wed Apr 30 15:37:25 2008 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 30 Apr 2008 15:37:25 -0400 Subject: [Numpy-discussion] numpy.ma question In-Reply-To: <4818C7B7.90102@llnl.gov> References: <4818BB98.1000004@llnl.gov> <200804301443.08121.pgmdevlist@gmail.com> <4818C7B7.90102@llnl.gov> Message-ID: <200804301537.26294.pgmdevlist@gmail.com> On Wednesday 30 April 2008 15:25:43 Charles Doutriaux wrote: > that's exactly my understanding thanks for confirming Fixed in 1.1.0dev5114 From dalcinl at gmail.com Wed Apr 30 15:44:43 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 30 Apr 2008 16:44:43 -0300 Subject: [Numpy-discussion] a possible way to implement a plogin system Message-ID: David, in order to put clear what I was proposing to you in previous mail regarding to implementing plugin systems for numpy, please take a look at the attached tarball. The plugins are in charge of implementing the action of generic foo() and bar() functions in C. The example actually implements two different plugins in Cython. There is also an include file with some support declarations and some macro definitions wrap_foo and wrap_bar Next, in mymodule.pyx, I define python side foo() and bar() functions (calling funtins pointers in a struct via the macros wrap_xxx) and a use() for selecting the plugin at runtime. Finally, the test.py script is pure python code showing all this in action. If you want to give a try, then install Cython, next do 'make', and finally 'python test.py' Please note that I've implemented this is Cython just for convenience, it does not actually depend in any special Cython feature, and could be translated to pure C code with Python C/API calls. IMHO, this is the easier and cleaner way to deal inside python with plugins written in low-level C. It does not depend explicitely on 'dlopen' stuff from your side. Now tell me. What is in your mind that this is not general/robust enough as to have to explicitely deal with dlopen? It even works on systems with no dynload, provided that all is put inside the python executable. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Wed Apr 30 15:45:57 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 30 Apr 2008 16:45:57 -0300 Subject: [Numpy-discussion] a possible way to implement a plogin system In-Reply-To: References: Message-ID: Sorry, I forgot to attach the code... On 4/30/08, Lisandro Dalcin wrote: > David, in order to put clear what I was proposing to you in previous > mail regarding to implementing plugin systems for numpy, please take a > look at the attached tarball. > > The plugins are in charge of implementing the action of generic foo() > and bar() functions in C. The example actually implements two > different plugins in Cython. There is also an include file with some > support declarations and some macro definitions wrap_foo and wrap_bar > > Next, in mymodule.pyx, I define python side foo() and bar() functions > (calling funtins pointers in a struct via the macros wrap_xxx) and a > use() for selecting the plugin at runtime. > > Finally, the test.py script is pure python code showing all this in > action. If you want to give a try, then install Cython, next do > 'make', and finally 'python test.py' > > Please note that I've implemented this is Cython just for convenience, > it does not actually depend in any special Cython feature, and could > be translated to pure C code with Python C/API calls. > > IMHO, this is the easier and cleaner way to deal inside python with > plugins written in low-level C. It does not depend explicitely on > 'dlopen' stuff from your side. > > Now tell me. What is in your mind that this is not general/robust > enough as to have to explicitely deal with dlopen? It even works on > systems with no dynload, provided that all is put inside the python > executable. > > > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -------------- next part -------------- A non-text attachment was scrubbed... Name: stubs.tar Type: application/x-tar Size: 20480 bytes Desc: not available URL: From Chris.Barker at noaa.gov Wed Apr 30 16:07:11 2008 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 30 Apr 2008 13:07:11 -0700 Subject: [Numpy-discussion] recarray fun In-Reply-To: <9457e7c80804301220p327be027sb0845b3e125fe31a@mail.gmail.com> References: <4818B1F7.904@noaa.gov> <9457e7c80804301220p327be027sb0845b3e125fe31a@mail.gmail.com> Message-ID: <4818D16F.7090502@noaa.gov> St?fan van der Walt wrote: > That's the way, or just rgba_image.view(numpy.int32). ah -- interestingly, I tried: rgba_image.view(dtype=numpy.int32) and got: Traceback (most recent call last): File "", line 1, in TypeError: view() takes no keyword arguments Since it is optional, shouldn't it be keyword argument? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From charlesr.harris at gmail.com Wed Apr 30 18:06:51 2008 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 30 Apr 2008 16:06:51 -0600 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <20080426100938.GD24606@phare.normalesup.org> <48161F61.5090204@noaa.gov> <48174FF5.50005@noaa.gov> Message-ID: On Tue, Apr 29, 2008 at 1:22 PM, Timothy Hochberg wrote: > > Let me throw out a couple of more thoughts: > > First, there seems to be disagreement about what a row_vector and > column_vector are (and even if they are sensible concepts, but let's leave > that aside for moment). One school of thought is that they are > one-dimensional objects that have some orientation (hence row/column). They > correspond, more or less, to covariant and contravariant tensors, although I > can never recall which is which. The second view, which I suspect is > influenced by MatLab and its ilk, is that they are 2-dimensional 1xN and > Nx1 arrays. It's my view that the pseudo tensor approach is more powerful, > but it does require some metainformation be added to the array. This > metadata can either take the form of making the different objects different > classes, which leads to the matrix/row/column formulation, or adding some > sort of tag to the array object (proposal #5, which so far lacks any > detail). > > Second, most of the stuff that we have been discussing so far is primarily > about notational convenience. However, there is matrix related stuff that is > at best poorly supported now, namely operations on stacks of arrays (or > vectors). As a concrete example, I at times need to work with stacks of > small matrices. If I do the operations one by one, the overhead is > prohibitive, however, most of that overhead can be avoided. For example, I > rewrote some of the linalg routines to work on stacks of matrices and > inverse is seven times faster for a 100x10x10 array (a stack of 100 10x10 > matrices) when operating on a stack than when operating on the matrices one > at a time. This is a result of sharing the setup overhead, the C routines > that called are the same in either case. > Some operations on stacks of small matrices are easy to get, for instance, +,-,*,/, and matrix multiply. The last is the interesting one. If A and B are stacks of matrices with the same number of dimensions with the matrices stored in the last two indices, then sum(A[...,:,:,newaxis]*B[...,newaxis,:,:], axis=-2) is the matrix-wise multiplication of the two stacks. If B is replaced by a stack of 1D vectors, x, it is even simpler: sum(A[...,:,:]*x[...,newaxis,:], axis=-1) This doesn't go through BLAS, but for large stacks of small matrices it might be even faster than BLAS because BLAS is kinda slow for small matrices. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Wed Apr 30 18:11:04 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 1 May 2008 00:11:04 +0200 Subject: [Numpy-discussion] recarray fun In-Reply-To: <4818D16F.7090502@noaa.gov> References: <4818B1F7.904@noaa.gov> <9457e7c80804301220p327be027sb0845b3e125fe31a@mail.gmail.com> <4818D16F.7090502@noaa.gov> Message-ID: <9457e7c80804301511g51b43d56p70c57f2f12ddf516@mail.gmail.com> 2008/4/30 Christopher Barker : > St?fan van der Walt wrote: > > That's the way, or just rgba_image.view(numpy.int32). > > ah -- interestingly, I tried: > > rgba_image.view(dtype=numpy.int32) > > and got: > > Traceback (most recent call last): > File "", line 1, in > TypeError: view() takes no keyword arguments > > Since it is optional, shouldn't it be keyword argument? Thanks, fixed in r5115. Cheers St?fan From cournapeau at cslab.kecl.ntt.co.jp Wed Apr 30 23:03:46 2008 From: cournapeau at cslab.kecl.ntt.co.jp (David Cournapeau) Date: Thu, 01 May 2008 12:03:46 +0900 Subject: [Numpy-discussion] a possible way to implement a plogin system In-Reply-To: References: Message-ID: <1209611026.23684.19.camel@bbc8> On Wed, 2008-04-30 at 16:44 -0300, Lisandro Dalcin wrote: > David, in order to put clear what I was proposing to you in previous > mail regarding to implementing plugin systems for numpy, please take a > look at the attached tarball. Thanks for looking at this Lisandro. The problem I see with the approach of struct vs direct function pointers is of ABI compatibility: it is easy to mess things up with structures. There is the advantage of using only one dlsym (or equivalent) with the struct, which may be much faster than using hundreds of dlsym for each function. Linux does not seem to have problems with that, but mac os X for example felt slow when I tried doing this for several thousand functions. I did not go really far on that, though. I don't really see why using a struct is cleaner, though. That's really the same thing (function pointers), in both cases the name namespace pollution will be the same, and in both cases there will be a need to generate source code. > > IMHO, this is the easier and cleaner way to deal inside python with > plugins written in low-level C. It does not depend explicitely on > 'dlopen' stuff from your side. Concerning the loading mechanism, I don't understand the point of using PyCObject_Import. By quickly looking at the code of the function, the plugin needs to be a python module in that case, which is not needed in our case; I don't like the fact that it is not documented either. The code for dlopen/dlclose is really short. For each platform, it is like 50 lines of code, and we have a better control on what we can do (which may be needed; for example, you want want to load symbols from a dll A, but the symbols are only in dll B, which dll A depends on; that's a problem that does not happen for python extensions, I think; I don't really know to be honest). The one thing which may have been useful by using python mechanism is the binary location detection, but that won't be needed in our case, since the plugins will always be distributed at the same time as numpy/scipy, hence we will have tight control on their location. > It even works on > systems with no dynload, provided that all is put inside the python > executable. That won't be needed: the point is to have several implementations with the same functions. If you link everything statically, you will get name clashes anyway (if you want to stay portable). On those platforms, plugins should be disabled when building numpy/scipy. cheers, David From peridot.faceted at gmail.com Wed Apr 30 23:16:22 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Wed, 30 Apr 2008 23:16:22 -0400 Subject: [Numpy-discussion] untenable matrix behavior in SVN In-Reply-To: References: <48161F61.5090204@noaa.gov> <48174FF5.50005@noaa.gov> Message-ID: 2008/4/30 Charles R Harris : > Some operations on stacks of small matrices are easy to get, for instance, > +,-,*,/, and matrix multiply. The last is the interesting one. If A and B > are stacks of matrices with the same number of dimensions with the matrices > stored in the last two indices, then > > sum(A[...,:,:,newaxis]*B[...,newaxis,:,:], axis=-2) > > is the matrix-wise multiplication of the two stacks. If B is replaced by a > stack of 1D vectors, x, it is even simpler: > > sum(A[...,:,:]*x[...,newaxis,:], axis=-1) > > This doesn't go through BLAS, but for large stacks of small matrices it > might be even faster than BLAS because BLAS is kinda slow for small > matrices. Yes and no. For the first operation, you have to create a temporary that is larger than either of the two input arrays. These invisible (potentially) gigantic temporaries are the sort of thing that puzzle users when as their problem size grows they suddenly find they hit a massive slowdown because it starts swapping to disk, and then a failure because the temporary can't be allocated. This is one reason we have dot() and tensordot() even though they can be expressed like this. (The other is of course that it lets us use optimized BLAS.) This rather misses the point of Timothy Hochberg's suggestion (as I understood it), though: yes, you can write the basic operations in numpy, in a more or less efficient fashion. But it would be very valuable for arrays to have some kind of metadata that let them keep track of which dimensions represented simple array storage and which represented components of a linear algebra object. Such metadata could make it possible to use, say, dot() as if it were a binary ufunc taking two matrices. That is, you could feed it two arrays of matrices, which it would broadcast to the same shape if necessary, and then it would compute the elementwise matrix product. The question I have is, what is the right mathematical model for describing these arrays-some-of-whose-dimensions-represent-linear-algebra-objects? One idea is for each dimension to be flagged as one of "replication", "vector", or "covector". A column vector might then be a rank-1 vector array, a row vector might be a rank-1 covector array, a linear operator might be a rank-2 object with one covector and one vector dimension, a bilinear form might be a rank-2 object with two covector dimensions. Dimensions designed for holding repetitions would be flagged as such, so that (for example) an image might be an array of shape (N,M,3) of types ("replication","replication","vector"); then to apply a color-conversion matrix one would simply use dot() (or "*" I suppose). without too much concern for which index was which. The problem is, while this formalism sounds good to me, with a background in differential geometry, if you only ever work in spaces with a canonical metric, the distinction between vector and covector may seem peculiar and be unhelpful. Implementing such a thing need not be too difficult: start with a new subclass of ndarray which keeps a tuple of dimension types. Come up with an adequate set of operations on them, and implement them in terms of numpy's functions, taking advantage of the extra information about each axis. A few operations that spring to mind: * Addition: it doesn't make sense to add vectors and covectors; raise an exception. Otherwise addition is always elementwise anyway. (How hard should addition work to match up corresponding dimensions?) * Multiplication: elementwise across "repetition" axes, it combines vector axes with corresponding covector axes to get some kind of generalized matrix product. (How is "corresponding" defined?) * Division: mostly doesn't make sense unless you have an array of scalars (I suppose it could compute matrix inverses?) * Exponentiation: very limited (though I suppose matrix powers could be implemented if the shapes are right) * Change of basis: this one is tricky because not all dimensions need come from the same vector space * Broadcasting: the rules may become a bit byzantine... * Dimension metadata fiddling Is this a useful abstraction? It seems like one might run into trouble when dealing with arrays whose dimensions represent vectors from unrelated spaces. Anne